Slow backup speed a Synology bad configuration of rsync?

IF your questions pertains to using the file server performance, transfer issues, jumbo frames tuning, post it here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form and check this FAQ:
https://myds.synology.com/support/suppo ... p?lang=enu
https://www.synology.com/en-global/know ... se/faq/586
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
mervincm
Versed
Versed
Posts: 270
Joined: Wed Jul 30, 2014 6:08 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby mervincm » Sat Sep 02, 2017 12:29 am

adding another data piece here.
416j (4disk shr ext4) replicating to 1813+ (6 disk shr btrfs) using shared folder sync (rsync)
The load is gen backups, photos and some video average speed is 55 MB/sec.

Spun up a second rcync thread, same source volume same destination volume, different share, now I see a total of 75-95 MB/sec
looks like running a couple simultaneous replications can help with RSYNC. I suspect it would hurt unless you have enough disks to keep up, but its really helping here!.
Last edited by mervincm on Sat Sep 02, 2017 1:14 am, edited 1 time in total.
1815+ (factory patched board) 4GB 5x6TB WD RED
Noah
Seasoned
Seasoned
Posts: 532
Joined: Sat Apr 21, 2012 3:20 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby Noah » Sat Sep 02, 2017 12:34 am

khirsah wrote:Also an 1817+ here and Hyperbackup via rsync compatible transfer is crazy slow. I monitor my box via SNMP and found that when rsync is running SNMP shows all 4 cores at 100% CPU usage, while the Resource Monitor inside DSM does not change and remains around 20%. If I stop rsync, SNMP goes back down to expected levels. Seems to be an extremely CPU intensive process to copy large amounts of files that eventually kills every other process on my box. My iSCSI mounts drop out and my Dockers grind to a halt..

There has to be a better backup process for large amounts of files.


How do you monitor your box via SNMP, if I might ask?
Somebody here sucks big time. BIG time. Small...
User avatar
PaulS
Enlightened
Enlightened
Posts: 415
Joined: Thu May 02, 2013 1:52 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby PaulS » Sat Sep 02, 2017 4:04 am

khirsah wrote:There has to be a better backup process for large amounts of files.

It's called "rsync."

Not Synology crappy co-opt GNU cop-out rsync. Just rsync.
alan sh
Apprentice
Apprentice
Posts: 85
Joined: Wed Oct 02, 2013 3:25 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby alan sh » Tue Sep 05, 2017 10:53 pm

I'm running shared folder sync from a 916+ to a 416j. I'm getting around 50mbytes/sec and neither CPU is running at 100%.

The files are mostly large MP4's.

Alan
mervincm
Versed
Versed
Posts: 270
Joined: Wed Jul 30, 2014 6:08 am

Re: Slow backup speed a Synology bad configuration of rsync?

Postby mervincm » Tue Sep 05, 2017 11:20 pm

alan sh wrote:I'm running shared folder sync from a 916+ to a 416j. I'm getting around 50mbytes/sec and neither CPU is running at 100%.

The files are mostly large MP4's.

Alan



did you try splitting up the job into multiple shared folder sync jobs, then run both at the same time? When I did that I was able to see much higher simultaneous performance.
1815+ (factory patched board) 4GB 5x6TB WD RED
alan sh
Apprentice
Apprentice
Posts: 85
Joined: Wed Oct 02, 2013 3:25 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby alan sh » Wed Sep 06, 2017 10:29 am

I think that's the max for my transfer. But I am doing another two at the moment and it's still maxing out at 50Mbytes/sec

Alan
User avatar
PaulS
Enlightened
Enlightened
Posts: 415
Joined: Thu May 02, 2013 1:52 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby PaulS » Wed Sep 06, 2017 3:07 pm

Geky wrote:Yes, they concluded that the 115j has limited CPU resources to run rsync faster.

The truth is (the part they don't tell you) their developers have limited resources preventing them from effectively layering a bloated, inefficient GUI on top of rsync.
sincarne
Navigator
Navigator
Posts: 973
Joined: Wed Feb 15, 2017 9:57 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby sincarne » Wed Sep 06, 2017 6:09 pm

PaulS wrote:
Geky wrote:Yes, they concluded that the 115j has limited CPU resources to run rsync faster.

The truth is (the part they don't tell you) their developers have limited resources preventing them from effectively layering a bloated, inefficient GUI on top of rsync.


u must not be in tech if you not know rsync very CPU heavy. 115j have very weak CPU. not sure you post when all your post bile
User avatar
PaulS
Enlightened
Enlightened
Posts: 415
Joined: Thu May 02, 2013 1:52 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby PaulS » Wed Sep 06, 2017 9:17 pm

sincarne wrote:u must not be in tech if you not know rsync very CPU heavy. 115j have very weak CPU. not sure you post when all your post bile

Like the Syno so-called "backup" program developers? I wonder if they are in tech? :roll:

Welcome to my ignore list, skippy! :lol:
timespinner
I'm New!
I'm New!
Posts: 1
Joined: Wed Aug 14, 2013 8:34 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby timespinner » Thu Oct 12, 2017 1:58 am

For anyone still having this issue -

I've been using my own rsync .sh script rather than Synology's backup tools.

I telnet into the Synology as an admin user and then execute it directly - I've been able to achieve speeds up to 70MB/s, with an average of ~40-45MB/s. Using Hyper Backup - the best I was able to get before was 12-16MB/s.

The script contents are as follows - the ip address needs to be updated to point to the destination RSync server.

#!/bin/sh

# Script Options
#
NOW="$(date +"%Y%m%d-%H%M%S")"
HOME="/volume1/Backup"
LOG_HOME="${HOME}/logs"

#rsync options
#
SRC="/volume1"
DEST="192.168.99.99::syno_backup"
OPTIONS="-rltDWOv --delete"
EXCLUSIONS="--exclude surveillance --exclude=@ThumbConvertPro.core --exclude=@optware --exclude=@updater.core --exclude=@MailScanner --exclude=@syno_alias_impo.core --exclude=@maillog --exclude=@clamav --exclude=@spool --exclude=@S2S --exclude=@synoaudiod.core --exclude=@dms.core --exclude=synoquota.db --exclude=Plex --exclude=photo --exclude=@webdav --exclude=@ssbackup --exclude=@database --exclude=@eaDir --exclude=@cloudstation --exclude=docker --exclude=@autoupdate --exclude=@download --exclude=@docker --exclude=aquota.group --exclude=aquota.user --exclude=@appstore --exclude=@tmp"
LOG="${LOG_HOME}/rsync-${NOW}.log"
RSYNC="/bin/rsync"


RSYNC_COMMAND="${RSYNC} ${OPTIONS} ${EXCLUSIONS} ${SRC} ${DEST}"

log() {
echo "$(date +'%D %T') $1" >> ${LOG}
}

#Execute commands
#
log "Starting backup ${NOW}"
log "${RSYNC_COMMAND}"

${RSYNC_COMMAND} 2>&1 >>${LOG}

log "Ending backup ${NOW}"
lucetius
I'm New!
I'm New!
Posts: 2
Joined: Mon Oct 16, 2017 5:29 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby lucetius » Mon Oct 16, 2017 7:49 pm

Add me to the list of poor transfer speeds between 2 Diskstations using Cloud Station ShareSync.

I have a DS3612xs and a DS3617xs. Both are loaded with 12 x 12TB enterprise drives. Raid 0/BTRFS for both. Both units have an intel 2port 10gig nic. Ports are bonded (LACP) to a 10gig switch. I've been attempting to get files to sync/mirror from the 3617 to the 3612 but the transfer is incredibly slow. It's averaging around 6-8MB/s. Both these units have fresh installs with minimal packages running. For the moment, they are just basic file servers.

From a fast workstation with a 10gig nic and M2 drives, I'm able to hit sustained 800MB/s transfers to the 3617 and 600-700MB/s to the 3612. The bandwidth is there but there is a problem with the communication between the units.

My data is backed up to a server running a raid 1/0 via another 10gig nic. Raw file transfers are in the 500 - 600MB/s range. Backups average 350MB/s.

I've seen some people mention they are using Shared Folder Sync with Success. I'll be giving that a try while I wait for a response from Synology support.
lucetius
I'm New!
I'm New!
Posts: 2
Joined: Mon Oct 16, 2017 5:29 pm

Re: Slow backup speed a Synology bad configuration of rsync?

Postby lucetius » Mon Oct 16, 2017 7:57 pm

As a followup,

Synology support steered me towards a Cloud Station white paper https://global.download.synology.com/download/Document/WhitePaper/Synology_Cloud_Station_White_Paper-Based_on_DSM_6.0.pdf

After reading this, I've determined this solution is not for me. My test of 3 folders contained 1.5mil files which I think puts me outside the realm of what this app is capable of. (My total file count is much higher.)

It was suggested I look in to Shared folder sync which I am testing now. Results are much better with my file transfers averaging around 180MB/s with bursts up to 250MB/s. It's a one way Sync which for this situation works. I still think I should be able to get better speeds but I can live with this if needed.

In the past I have used ViceVersa Pro to sync files from one of my Synologys to a 3rd server and had good success / throughput with a once a day sync. I may do some additional testing with it to see how it performs in real time. Was hoping to keep this contained to just the two diskstations but that may not be possible with large amounts of data.

Return to “File Server Performance”

Who is online

Users browsing this forum: No registered users and 2 guests