Rebuild/Resync Performance Improvements

Questions and mods regarding system management may go here
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:

https://account.synology.com/support/support_form.php?lang=enu



2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
witchdoc
I'm New!
I'm New!
Posts: 8
Joined: Fri Mar 30, 2012 9:08 pm

Re: Rebuild/Resync Performance Improvements

Postby witchdoc » Sat Sep 28, 2013 12:20 pm

Anyone tried the additional parameters from http://www.cyberciti.biz/tips/linux-rai ... speed.html yet?

Tip #1 (/proc/sys/dev/raid/{speed_limit_max,speed_limit_min} kernel variables) is what this topic was about.

He also mentions tips #2 through #5;
Tip #2: Set read-ahead option
Tip #3: Set stripe-cache_size for RAID5 or RAID 6
Tip #4: Disable NCQ on all disks
Tip #5: Bitmap Option

A lot more details and the exact commands can be found on his blog by the way.
Hossy
Trainee
Trainee
Posts: 19
Joined: Fri Oct 14, 2011 2:34 pm

Re: Rebuild/Resync Performance Improvements

Postby Hossy » Wed Apr 02, 2014 5:51 am

witchdoc wrote:Anyone tried the additional parameters from http://www.cyberciti.biz/tips/linux-rai ... speed.html yet?

Tip #1 (/proc/sys/dev/raid/{speed_limit_max,speed_limit_min} kernel variables) is what this topic was about.

He also mentions tips #2 through #5;
Tip #2: Set read-ahead option
Tip #3: Set stripe-cache_size for RAID5 or RAID 6
Tip #4: Disable NCQ on all disks
Tip #5: Bitmap Option

A lot more details and the exact commands can be found on his blog by the way.


So, firstly: what didn't work. I had already tried 4 & 5 previously (see earlier thread entries).

Tip #2, the read-ahead option for md2 and md3, didn't seem to change much of anything, but I left it set to 65536. I also checked the read-ahead for each disk (default: 384), but that also didn't seem to change much of anything.

Tip #3, the stripe_cache_size, did seem to have an effect, but not a huge one.

Previously, I was getting about 11MB/sec. The stripe_cache_size default is set to 1024. After playing around with the settings (NOTE: I have 3GB RAM in my DS2411+), I ended up setting both md2 and md3 to 12288. I'm now getting anywhere between 15MB/s and 20MB/s.

I'm concerned that the number of read requests are where my downfall is. I have over 2000 r/s per disk with an average request size of 8 sectors (or 4096 bytes). I'd like to figure out how to increase the size of each request. I'm hoping this will increase the throughput. Anyone have any ideas?
Synology DS2411+ 12 x 3TB Seagate ST3000DM001 SHR2 | Remember: RAID is not a backup! I use CrashPlan+.
https://github.com/Hossy | http://serverfault.com/questions/2888/why-is-raid-not-a-backup
Hossy
Trainee
Trainee
Posts: 19
Joined: Fri Oct 14, 2011 2:34 pm

Re: Rebuild/Resync Performance Improvements

Postby Hossy » Fri Apr 25, 2014 2:55 pm

For those of you who are wondering, here is the script I've come up with to speedup the RAID resync process.

NOTE: My Synology has 3GB RAM installed. If yours doesn't have that much, you'll need to adjust the stripe cache size setting.

Disclaimer: Run this at your own risk. If you don't understand what this is doing, don't run it.

Code: Select all

FIRSTMD=2
LASTMD=4

## Sanity check on variables
[ $LASTMD -lt $FIRSTMD ] && exit

display_settings()
{
        for f in min max; do echo "speed_limit_$f: `cat /proc/sys/dev/raid/speed_limit_$f`"; done
        n=$FIRSTMD; while [ $n -le $LASTMD ]; do echo "md$n stripe_cache_size: `cat /sys/block/md$n/md/stripe_cache_size`"; n=$(( $n + 1 )); done
        blockdev --report /dev/md[0-$LASTMD]
}

## Display all settings before changes
echo "BEFORE:"
display_settings

## Set minimum RAID resync speed to 20000K/sec
echo 20000 > /proc/sys/dev/raid/speed_limit_min

## Set read-ahead to 2560
blockdev --setra 2560 /dev/md[$FIRSTMD-$LASTMD]

## Set stripe cache size to 12288
n=$FIRSTMD; while [ $n -le $LASTMD ]; do echo 12288 > /sys/block/md$n/md/stripe_cache_size; n=$(( $n + 1 )); done

## Display all settings
echo
echo "AFTER:"
display_settings


Commands I use to watch things:
watch cat /proc/mdstat
iostat -dkx 2
top
Synology DS2411+ 12 x 3TB Seagate ST3000DM001 SHR2 | Remember: RAID is not a backup! I use CrashPlan+.
https://github.com/Hossy | http://serverfault.com/questions/2888/why-is-raid-not-a-backup
absi
I'm New!
I'm New!
Posts: 1
Joined: Thu May 12, 2016 2:45 pm

Re: Rebuild/Resync Performance Improvements

Postby absi » Thu May 12, 2016 2:50 pm

Apologies for the bump - I came here to try and speed my resync up, thank you to onelesd for the tips.

The only thing I've added is to 'renice' the process priority, in the hope that the rebuild process doesn't get interrupted.

# renice 5 -p <<PID>>

I've done that to both md2_raid5, and md2_resync processes. This may differ depending on how your RAID is setup. I'm using Synology's Hybrid RAID.

I've now got my resync running at ~95000K/sec, and it should be done in a few hours. This is a much better ETA than the X number of days I was scheduled.
Myflag
I'm New!
I'm New!
Posts: 5
Joined: Thu Aug 18, 2016 4:27 am

Re: Rebuild/Resync Performance Improvements

Postby Myflag » Sat Aug 20, 2016 4:30 am

Hello,

I did not have any problems increasing the speed_limit. However, when I tried to increase the strip_cache_size it just responds with "-ash: echo: write error: Invalid argument". Can anybody assist?

I am using SHR with 2 disk fault tolerance.

Blessings!
angelfilms
I'm New!
I'm New!
Posts: 2
Joined: Tue Nov 05, 2013 11:20 am

Re: Rebuild/Resync Performance Improvements

Postby angelfilms » Wed Aug 31, 2016 4:42 pm

Myflag wrote:Hello,

I did not have any problems increasing the speed_limit. However, when I tried to increase the strip_cache_size it just responds with "-ash: echo: write error: Invalid argument". Can anybody assist?

I am using SHR with 2 disk fault tolerance.

Blessings!


Hi, had the same problem when I tried to input numbers above 32678. Maybe you did the same?
angelfilms
I'm New!
I'm New!
Posts: 2
Joined: Tue Nov 05, 2013 11:20 am

Re: Rebuild/Resync Performance Improvements

Postby angelfilms » Wed Aug 31, 2016 5:05 pm

Hi,

While trying to speed up the rebuild of a 12 disk SHR 2, I found the advice about increasing speed_limit_min/max very helpful, and it took the rebuild from 7-10 MB/s to 45-50 MB/s. However, I noticed that running the DSM Storage Manager while rebuilding, would slow the speed back down to around 8 MB/s! The CPU load is only 10% and RAM usage 60%, so that is not the problem. Also, just having the main DSM window open in my browser would slow the speed to 17 MB/s.

I'm running a RS3412xs with a RX1211, DSM 6.0.2-8451.
JeffQc
Novice
Novice
Posts: 55
Joined: Wed Jul 25, 2012 2:14 am

Re: Rebuild/Resync Performance Improvements

Postby JeffQc » Sun Sep 04, 2016 10:07 pm

Hi!

I did try the renice command but got a permission denied...

"renice: failed to set priority for 22436 (process ID): Operation not permitted"


I am currently rebuilding my array and... have to recover my data but 10mb/sec for rebuilding is really slow..

Is there any trick I have to do to gain access to that renice command?

Thank you!

90mb/sec for rebuilding... NICE!
Myflag
I'm New!
I'm New!
Posts: 5
Joined: Thu Aug 18, 2016 4:27 am

Re: Rebuild/Resync Performance Improvements

Postby Myflag » Sun Sep 04, 2016 10:31 pm

angelfilms wrote:
Myflag wrote:Hello,

I did not have any problems increasing the speed_limit. However, when I tried to increase the strip_cache_size it just responds with "-ash: echo: write error: Invalid argument". Can anybody assist?

I am using SHR with 2 disk fault tolerance.

Blessings!


Hi, had the same problem when I tried to input numbers above 32678. Maybe you did the same?


Angelfilms,

I attempted to input the number 512, to test it at first. Mine was set to 128 by default. Do you have any advice?

Blessings!
david_
I'm New!
I'm New!
Posts: 4
Joined: Fri Mar 04, 2016 4:54 pm

Re: Rebuild/Resync Performance Improvements

Postby david_ » Wed May 17, 2017 3:55 am

I let a SSH session run with the following command:

Code: Select all

while true; do echo 32768 > /sys/block/md2/md/stripe_cache_size; sleep 2.0; done


That kept the stripe_cache_size from reducing back down (if you only set it once and do cat /sys/block/md2/md/stripe_cache_size, you'll find the synology will make it smaller and smaller until its back to the default)

The above drastically improved rebuild time for a RAID-6 array.
farrsight
I'm New!
I'm New!
Posts: 2
Joined: Mon Jul 03, 2017 4:08 am

Re: Rebuild/Resync Performance Improvements

Postby farrsight » Tue Jul 04, 2017 8:35 am

Welp.

I tried changing the stripe cache size to 8192 and got a nice reduction in time and decided to see if the performance was better at 16k or 32k. I tried 16k and saw better performance then decided to test 32768. I entered the following:

echo 32768 > /sys/block/md2/md/stripe_cache_size

My SSH connection dropped and now I cannot get back in via SSH or access the web interface to the diskstation and it has dropped it's SMB shares. However, I do still hear the hard drives chunking away and there is plenty of activity so I think I have no choice but to wait it out and hope it finishes.

I'm also hoping that what other posters have said about the synology reducing the stripe cache size back down towards 256 will happen and allow me to SSH back in. I didn't use the 'while true' version of the command that was previously posted.

EDIT: After about 2 hours I was able to SSH back in. stripe_cachse_size remained at 32768 so I wonder if that setting is 'sticking' now in the last version of DSM? I've dropped back down to 16k which seems stable on the 416j
bpwl
I'm New!
I'm New!
Posts: 2
Joined: Fri Dec 08, 2017 10:46 am

Re: Rebuild/Resync Performance Improvements

Postby bpwl » Sat Dec 09, 2017 1:17 am

Adding a second 4TB disk to a DS218+ with version 6.1.4 starts a very long recovery process to make it RAID 1.
At 10MB/s for a 4TB drive, just count ...

I managed to increase that speed with all the hints in this forum. Played around with the raid rebuild speed limit, and checked via the resource monitor in the GUI for the effects. Speed is up to 170MB/s but dropping to zero every 5 seconds, still an average of arround 100MB/s . But could not find a stable value, where the process is not interrupted. There is no other load on this brand new device.

login as: admin
admin@192.168.3.141's password:
admin@bpacds:~$ ls
CloudStation
admin@bpacds:~$ cat /proc/sys/dev/raid/speed_limit_min
10000
admin@bpacds:~$ cat /proc/sys/dev/raid/speed_limit_max
10000
admin@bpacds:~$ echo 100000 > /proc/sys/dev/raid/speed_limit_max
-sh: /proc/sys/dev/raid/speed_limit_max: Permission denied

admin@bpacds:~$ sudo -i

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.


Password:
root@bpacds:~# echo 100000 > /proc/sys/dev/raid/speed_limit_max
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_max
100000
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_min
10000
root@bpacds:~# echo 100000 > /proc/sys/dev/raid/speed_limit_min
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_min
100000
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_max
100000
root@bpacds:~# echo 120000 > /proc/sys/dev/raid/speed_limit_min
root@bpacds:~# echo 120000 > /proc/sys/dev/raid/speed_limit_max
root@bpacds:~# ls /proc/sys/dev/raid
speed_limit_max speed_limit_min
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_max
120000
root@bpacds:~# cat /proc/sys/dev/raid/speed_limit_min
120000
root@bpacds:~# echo 90000 > /proc/sys/dev/raid/speed_limit_min
root@bpacds:~# echo 90000 > /proc/sys/dev/raid/speed_limit_max
root@bpacds:~# echo 80000 > /proc/sys/dev/raid/speed_limit_max
root@bpacds:~# echo 80000 > /proc/sys/dev/raid/speed_limit_min
root@bpacds:~# echo 200000 > /proc/sys/dev/raid/speed_limit_min
root@bpacds:~# echo 200000 > /proc/sys/dev/raid/speed_limit_max

Return to “System Managment Mods”

Who is online

Users browsing this forum: No registered users and 4 guests