Slow backup speed a Synology bad configuration of rsync?

IF your questions pertains to using the file server performance, transfer issues, jumbo frames tuning, post it here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form and check this FAQ:
https://account.synology.com/support/suppo ... p?lang=enu
https://www.synology.com/en-global/know ... se/faq/586
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Sat May 06, 2017 10:29 am

After conducting many tests I came to the conclusion that Synology low backup speeds are related to the Hyper backup and Shared Sync Folder implementation by Synology.

I have opened a ticket with them and waiting for their second reply, the first reply as is usual with Synology is that they don't really understand the reported issue.

The setup:
Copy/sync from DS214 (source) to DS115j (destination).

Environment:
Gbit ethernet wired to all devices under test with CAT 5e or CAT 6 cables.
MTU of both NASs set to auto.
No encryption nor compression to any of the transfers.

FIle transferred: large video file of 13GB.

Tests:
1. Transfer the file from DS214 to PC, using windows explorer -> 100MB/s
2. Transfer the file from PC to DS115j using windows explorer -> 80MB/s
3. Transfer the file from DS214 to DS115j using windows explorer -> 45MB/s
4. Transfer the file from DS214 t DS115j using file station (mounted remote folder) ->45Mb/s
6. Transfer the file from DS214 to DS115j using Shared Folder Sync -> 13MB/s
7. Transfer the file from DS214 tp DS115j using Hyper Backup data copy remote rsync ->13MB/s
8. Transfer the file from DS214 to DS115j using the PC application SyncToy ->13MB/s.

Conclusions:
The HW (CPU, memory) of the NASs is not a bottleneck, it can handle at least 45MB/s for large files transfer.
The Drives used is not a bottleneck, individually for direct transfers to windows explorer they can achieve at least 80MB/s
The network environment is not a problem.

The only problem I can think of is the implementation of rsync protocol in the NASs, or at least in one of the NASs.

I hope Synology finaly does some serious checking of the issue and inform all (and I see a lot of people complaining in here) of us having such problems with the backup speeds.

Personally I spent a lot of money to have 2 synology NASs doing an automatic backup between them without having to run the PC and doing a manual task.

Synology do you hear?
imforumman
I'm New!
I'm New!
Posts: 4
Joined: Sat May 06, 2017 4:32 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby imforumman » Mon May 08, 2017 10:10 pm

Just some brainstorming concerning your tests:

1. Makes sense
2. Makes sense
3. Makes sense
4. Hmm. What protocol does filestation use for mounted remote folders? Seems slow, shouldn't this use a protocol as efficient as SMB?
(What happend to 5? :wink: )
6-8. This is desastrous. Does Shared Folder Sync also use rsync? Hardly I would think.

Something is amiss here, but I'm not sure it's rsync.
fuu
I'm New!
I'm New!
Posts: 1
Joined: Mon May 08, 2017 10:49 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby fuu » Mon May 08, 2017 11:10 pm

RSYNC,

my testing shows the RSYNC to be poorly written,

the NAS quite happily copes with much higher (throughputs using FTP, SMB, NFS) and RSYNC is just consuming CPU and memory,

my testing is focused around 3TB being copied from NAS to NAS (same DSM), Giga BIT switch, and various MTU settings, both the machines are dedicated, and running IP4, the source machine has plenty of head room on IOPS, MEMORY, CPU, and the target (DS115J) running 6.1.1-151010 Update 1. Given the indexing is off, there is no AV, and no DPI on Firewall), and the much higher speed seen with FTP, SMB, NFS, >25MB/sec the RSYNC is struggling @ 13MB/Sec. (the source content is static and file types are mixed).

more of a concern is the Rsync deamon seems to consume the GUI resources (DSM on the DS115J) web page becomes unresponsive as though IO bound, the Rsync is still chugging away as my source system is still uploading, the only way to recover back the DSM web page is to disconnect the DS115J from the LAN let if flush all the pending writes, stabilise, isolate the source then re-introduce the DS115J, then the WEB page responds.

the exhaustive RSYNC should be easily repeatable

Suspicion from me - the RSYNC code is inefficiently using a mixture of (TCPIP, EXT4, cache,) -

it may be robust, but it is slow and un-needingly resource hungry, maybe its doing too much CRC checking?

Fuu
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Tue May 09, 2017 7:48 am

On request of synology I did more tests with similar results.
I sent them the debug file and waiting for a reply.

Shared folder sync uses RSYNC also.

What ever happened to 5? :lol:
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Fri May 12, 2017 10:27 pm

So I run more specific tests and sent the results to synology support (see in italic below):

Test environment for all tests:
Source NAS: DS214
Destination NAS: DS115j
Transfer video file of 13GB
Time from NTP server from google time.
No data compression
No data encryption
No metadata information
1 Gb network (all adapters all switches)
MTU of both NAS set to auto. Reported from network settings tab as: 1000Mbps, full duplex, 1500 MTU
All devices including the PC are wired with cat5e cables



Test 1: From DS214 to DS115j using Shared Folder Sync
Time start: 16.22.00
Time End: 16.35.00
Duration: 780 sec
Transfer rate: 16.6MB/s

Test 2: From DS214 to DS115j using Hyper Backup (remote data copy)
Time Start: 14.41.00
Time End: 16.57.15
Duration: 975 sec
Transfer rate: 13.3MB/s

Test 3: From DS214 to DS115j using File Station (mounted remote folder on DS214).
Time Start: 17.11.11
Time End: 17.17.29
Duration: 378 sec
Transfer rate: 34.4MB/s

Test 4: From DS214 to DS115j using windows PC (windows explorer) mapped directories for each NAS
Time Start: 17.21.03
Time End: 17.24.20
Duration: 197 sec
Transfer rate: 65.9 MB/s


I also send the debug file of the source NAS (DS214) as requested by synology support.

Within a day I received the answer:

Unfortunately the debug file hasn't provided us with a much greater insight into the reported issue I'm afraid.
So that we can investigate this further, please can I ask you to provide me with remote access to the NAS via the instructions below


Giving remote access to anyone with Admin root rights is out of company policy and also not very prudent anyway, so I dont know how we will prioceed from now on.

What is quit strange is that they seem to be convince the problem lays with the source NAS, since they only make mention of NAS instead of NASs
when they ask for debug files and remote access.
Further I dont understand why dont they try to reproduce the issue we are having at their lab instead of asking remote access.
If they can reproduce the issue then they must solve it with an update for everyone rather than try to support a single customer.

I will keep on updating
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Mon May 15, 2017 1:42 pm

I received the reply from Synology.

They dont want remote access anymore, however after studying again the debug file (the one that initially did not give them any clues) I sent them they outlined 4 issues that need fixing before they continue with their investigation:

1. Remove "Advanced Power Manager" from the NAS.
They claim that this third party package is potentially causing some issues on the DSM. They cannot be sure but they want it uninstalled.
2. Reset and reinstall DSM on the source NAS.
Due to the installation of the third party package they want the NAS reset to its original state.
3. Replace the network cable connected to the source NAS.
There were some dropped packages so they advise the change of the CAT5e cable from source NAS to Gbit switch.
4. Replace the desktop graded HDDs with NAS graded ones.
I am using 2 desktop graded disks that they want replaced with NAS graded disks.

I replied to synology support:

1. I will uninstall the third party package. (Advanced Power Manager). However, with reference to the test results (see below) I sent you at the start of the troubleshooting, how is it possible that the package does not affect SMB transfers using windows as well as file station transfers to achieve 65MB/s and 34MB/s respectively but it affects the transfer rates using rsync?
2. I am not able to reset the NAS on basis of a third party package installed (that will be uninstalled) on the machine, the downtime will be too high. How could this possibly solve a problem just on rsync related transfer rates?
3. I will replace the network cable. However, with reference to the test results I sent you at the start of the troubleshooting, how is it possible that on the same Cat5e cable and infrastructure; SMB transfers using windows explorer as well as file station transfers achieve in excess of 34MB/s and up to 65MB/s?
4. These HDDs, ST2000DM001, are stated as compatible from Synology
(https://www.synology.com/en-uk/compatib ... mode=false)
I cannot understand your suggestion against Synology’s compatibility list and I cannot understand how would a replacement solve any issue of transfer rates? Again I make reference to the test results; see below.

Based on my answers, I ask you to have a look again at the below results and try to work together to figure out why the performance is so poor only on rsync related transfers
.

So I am expecting their reply again.
I was quite surprised that they introduced conditions to investigate further, since they havent proposed any meaningful action from the start of the investigation.
The fact that they ask for disk replacement (against their own compatibility list) makes me quite suspicious of the level of competence.

I will keep on updating, the stoty goes on.
mervincm
Versed
Versed
Posts: 270
Joined: Wed Jul 30, 2014 6:08 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby mervincm » Tue May 16, 2017 5:35 pm

I am sorry but just don't see your position as reasonable. I understand that you would want it to be otherwise, but I don't see it as reasonable from a low-mid tier NAS company perspective. Synology responses seem correct to me, and frankly they are what I would expect. I want them to limit their support to known supported configs as otherwise they would have to spend more on support and I already think their prices are high enough. spending time testing 3rd party apps on how or why they might impact performance is not a good use of their resources IMO. Also, your hard disks are supported, and your NAS is working. You want it to work faster/better, surely you can admit that some disks will work better that others? (personally I doubt it will make the difference but ....)
as far as the cable goes, its just a standard response. you can search the forum and you will find issues fixed by cable replacement that didn't seem cable related based on the symptoms. I agree with you that it is unlikely given SMB performance, but that being said, different protocols can make a difference in situation where you have unreliable media, and asking them to spend the time to research/explain why is not a good use of their limited support resources.


Do you not have a spare HDD that you can use to satisfy their suggestions? pull out all of your disks, install a single spare HDD, reinstall from scratch, and try that? once your testing is done, throw your disks back in and away you go. Good luck solving the issue, and thanks for documenting your issue here for reference.
1815+ (factory patched board) 4GB 5x6TB WD RED
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Wed May 17, 2017 11:04 am

mervincm wrote:I am sorry but just don't see your position as reasonable. I understand that you would want it to be otherwise, but I don't see it as reasonable from a low-mid tier NAS company perspective. Synology responses seem correct to me, and frankly they are what I would expect. I want them to limit their support to known supported configs as otherwise they would have to spend more on support and I already think their prices are high enough. spending time testing 3rd party apps on how or why they might impact performance is not a good use of their resources IMO. Also, your hard disks are supported, and your NAS is working. You want it to work faster/better, surely you can admit that some disks will work better that others? (personally I doubt it will make the difference but ....)
as far as the cable goes, its just a standard response. you can search the forum and you will find issues fixed by cable replacement that didn't seem cable related based on the symptoms. I agree with you that it is unlikely given SMB performance, but that being said, different protocols can make a difference in situation where you have unreliable media, and asking them to spend the time to research/explain why is not a good use of their limited support resources.


Do you not have a spare HDD that you can use to satisfy their suggestions? pull out all of your disks, install a single spare HDD, reinstall from scratch, and try that? once your testing is done, throw your disks back in and away you go. Good luck solving the issue, and thanks for documenting your issue here for reference.


Hi,
Good to hear your point of view, I respect it however I disagree.

Synology made conditions that were irrelevant to the problem of slow transfer speeds.
They admitted it in their later reply to me, where they state that these 3 suggestions were for the general good working condition of my NAS.

So I still stand behind my stance that I am surprised that they came up with these type of issues for a transfer speed problem.

Especially for the disks they stated that I should always use disks that best match my needs!
What are my needs?
I bought a NAS and in the compatibility list I chose the specific seagate desktop disks. Why do I need to change my needs now and replace them with new NAS graded disks?
BTW the seagate desktop disks are still in the compatibility list.

I will issue update later today as there are more developments.
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Wed May 17, 2017 12:08 pm

Synology replied to me that the 3 of the 4 conditions (third party optware, reinstallation of DSM, NAS graded HDDs) stated in their previous email were general advice, in their words:

"advice relative to your current NAS setup in general so that we bring to your attention that your version of DSM maybe vunerable to threats if changes have been made to the core config, that there are dropped packets occurring and that you have optware installed."

and

"As per my previous advice I would still recommend performing the DSM reinstallation and replacing the desktop graded HDDs to provide you with a secure DiskStation which is utilising the correct drives"

To me it was clear that the synology generic advise was not related to the slow transfer speeds so I pursued the issue.
I suggested to Synology that we concentrate on the log entries of the debug files during the specific timestamps that we had from the performed tests and results.

Synology agreed and wrote back to me that the log entries relevant to the specific test results were fine and there were no rsync errors in these tests.
Synology asked my to perform command line rsync tests via SSH and send them the results.
I am at the moment expecting the detailed instructions from Synology to perform the rsync tests.

In the meantime I have done some more digging and found plenty of articles related to slow linux rsync transfer rates. There seems to be a "-z" parameter that compresses the data during transmision. I have an intuition that Synology has implemented the -z parameter in their backup data copy code and thus even if the user chooses to "not encrypt" not compress" the rsync does the compression and then decompression regardless.
We will only know whether this is the problem when we do the specific rsync command line tests (without the -z parameter).

PS: Regarding HDDs.
When anyone is purchasing a NAS, he/she is advised to choose HDD from the compatibility list.
The list contains Desktop, NAS, Enterprise NAS and Surveilance grade HDDs.
In my specific case:
https://www.synology.com/en-uk/compatib ... mode=false
So I chose Desktop grade HDDs (compatible for both DS214 and DS115j).

Synology actually stated to me that while they stand behind the compatibility list they publish, they in parallel advise people to replace HDDs from said list with NAS graded HDDs to "better suit their needs".
The logical question I assume everyone would have is: Why do Synology publish desktop graded HDDs as compatible in the first place if they would advise customers to replace them for NAS graded ones? Why dont synology drop these HDDs from the list, so at least new customers buy directly the advised NAS graded HDDs?
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Sat May 20, 2017 8:04 pm

Update

I run the command line I received from Synlogy, it was a "rsync -avz" command.
The whole test equipment and methodology was identical to the set of previous tests conducted and outlined above in thie thread.

Predictably I recorded very slow transfer rate of 3MB/s.

I proceeded and removed the "-z" parameter from the rsync command and I recorded transfer speed of 9MB/s.
So at that stage I gave up and sent the results to synology, asking them why did they send me command line that was basically utilising the slowest implementation of rsync (rsync -avz compresses and encrypts the data, so the CPU load and protocol overhead due to SSH encryption are high => slow transfer speeds)?

Synology Support contacted me and stated that using their "share Folder sync" and "hyperbackup" GUIs, the user has the choice to use rsync:
1. rsync module mode
2. rsync module +ssh
3. rsh


So my first set of test results using "share Folder sync" and "hyperbackup" were using rsync with "rsh" protocol which is without encryption and compression (I selected the options within the GUI for no compression no encryption), thus the test results recorded were 13.3MB/s and 16.6MB/s respectively for "share Folder sync" and "hyperbackup".
The second set of tests using the command line was using rsync with the "SSH" protocol.
Therefore recording better results with the GUI than the command line but still way slower than expected for data copy without any compression and encryption.

I contacted Synology again and pointed out that since their GUI's already support using "rsh" then the users (upon selecting to tick the boxes for No Encryption No Compression) should expect that the data copy transfers would be free of overheads due to compression and encryption and CPU usage should be low => thus why do we have still slow transfer rates instead of similar transfer rates to SMB of NFS?

Synology replied that:
Unfortunately rsync isn't something our developers can change as it is a technology implemented within our own application. However, we are always looking to improve our applications where possible and this is something our developers are always working on but as I'm sure you're aware, the software development lifecycle is a long process and such an immediate change wouldn't be implemented as any updates are rolled out globally


So between the lines synology admits that their developers must look into the "rsync rsh" commands they use behind their GUI and figure out how to make them more efficient in order to achieve better transfer rates. They will not say it out clearly but it seems to me that they are looking at the issue.

The internet is abundant with rsync examples where people use rsh and rsync daemon and achieve up to 100MB/s rates over Gbit LANs.
Most people seem to agree that rsh (which incidentaly was the original protocol for rsync) adds very little overhead to the transfer and load to the CPUs, thus low-end HW shouldn't be suffering badly due to small CPU resources.

I will wait for Synology to reply to me and update. Hopefuly we will get updated packages that offer faster rates for data backup in the near future.
Dave_O
Rookie
Rookie
Posts: 36
Joined: Sat Feb 13, 2016 3:45 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Dave_O » Sun Jun 11, 2017 4:52 am

I just got a DS216J to backup my DS415play. I see exactly the same issue when using "Remote Copy" (rsync) to do file backups. NAS to NAS with rsync (no encryption) is about 35 MBs. PC to DS216J is twice that at over 70 MBs. The DS216j has 2 disks of the same model as the DS415play, configured in Basic volumes. Hope Synology takes this seriously. All devices are on Gigabit wired ethernet.
mervincm
Versed
Versed
Posts: 270
Joined: Wed Jul 30, 2014 6:08 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby mervincm » Tue Jun 13, 2017 10:30 pm

I use my 1813+ mostly as a backup destination. Last week I rebuild my 1813+ from scratch (wanted to change disks, raid type and use BTRFS) and thus needed to recopy my data to the 1813+ from my main NAS. I used the synology remote mount point feature and it copied smoothly, for more than 24hrs straight, and at over 80MB/sec. This was despite the fact that it was doing the consistency check at the same time. considering there was 10+TB and a ton of tiny>large files, I was happy with that.
1815+ (factory patched board) 4GB 5x6TB WD RED
Ravelin
I'm New!
I'm New!
Posts: 1
Joined: Thu Jul 27, 2017 10:07 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Ravelin » Thu Jul 27, 2017 10:17 am

Any news from synology?
Geky
Experienced
Experienced
Posts: 119
Joined: Sat Oct 25, 2014 12:17 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Geky » Sun Jul 30, 2017 10:30 pm

Ravelin wrote:Any news from synology?


Yes, they concluded that the 115j has limited CPU resources to run rsync faster.
I disputed that on the basis that the first-time data copy has nothing to compare to at the destination thus the destination CPU shouldn't be at 100% so that it bottlenecks the data copy.
After that I didn't get any more answers from synology.

So I send an email to synology to advise that they either implement a hybrid data copy that for the first data transfer it uses a fast protocol (like SMB) then uses rsync to only sync the new and chambged files.
Further I think synology should be clear in the specs of each DS regarding the expected rsync copy speeds just like they do for transfers with SMB and NFS.
Again I didn't get any reply from synology.
khirsah
I'm New!
I'm New!
Posts: 2
Joined: Tue Aug 01, 2017 5:29 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby khirsah » Tue Aug 01, 2017 5:42 am

Also an 1817+ here and Hyperbackup via rsync compatible transfer is crazy slow. I monitor my box via SNMP and found that when rsync is running SNMP shows all 4 cores at 100% CPU usage, while the Resource Monitor inside DSM does not change and remains around 20%. If I stop rsync, SNMP goes back down to expected levels. Seems to be an extremely CPU intensive process to copy large amounts of files that eventually kills every other process on my box. My iSCSI mounts drop out and my Dockers grind to a halt..

There has to be a better backup process for large amounts of files.

Return to “File Server Performance”

Who is online

Users browsing this forum: No registered users and 8 guests