Diskstation expanding array - adding disks is slow - How to make expansion faster

Questions and mods regarding system management may go here
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:

https://account.synology.com/support/support_form.php?lang=enu



2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
User avatar
B9677A61
Beginner
Beginner
Posts: 22
Joined: Sun Feb 02, 2014 9:08 pm

Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby B9677A61 » Tue Aug 09, 2016 12:43 am

Background:
When adding new devices to a RAID array the rebuild is throttled to ensure the array still performs well in the NAS. The idea is to minimize the effect on users so they don't notice the rebuild.

Taking a long time may not be a problem for you. Especially if your users are sensitive to slow down. But there are a couple of scenarios you might not mind performance impact on users.
  • When the array is at risk of a second hard disk failure before the rebuild completes (causing loss of the array)
  • When volume changes need to take place but are blocked until the rebuild completes.

For me, after two days waiting and only at 60% I searched and found the information required to speed things up. I wanted to summarize here for the benefit of the community.

Solution:
The solution is to increase the minimum speed the RAID array will use to rebuild. Diskstation does not provide a simple way to tune this. You will have to SSH to the NAS and execute a few commands.

Once you SSH to the NAS you can execute a couple of commands to see the current values for the rebuild throttle. The values are stored in speed_limit_min and speed_limit_max. Check them using the "cat" command. Change them by echoing new values into them. You can also check the status of the rebuild (including speed) by the command "cat /proc/mdstat".

Step 1: Logging into the diskstation via SSH (putty):
Image

Step 2: Check current speed:
Image

Step 3: Check current throttle values:
Image

Step 4: Increase the throttle values to 250MB:
Image

Step 5: See results
Image

Proof!
Image

References:
More background information: http://www.ducea.com/2006/06/25/increase-the-speed-of-linux-software-raid-reconstruction/
Thanks to @lethalwp for the response to this thread which had the original solution.
titanax
I'm New!
I'm New!
Posts: 8
Joined: Sun Jan 29, 2017 12:08 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby titanax » Fri Feb 10, 2017 1:39 am

Hi,

So I have successfully migrated from a DS213 to DS916+. I did not mess around with the original SHR. It took me about 2.5 days to the migration, it was the bad sector disk repair that took the most time, this was despite me using the HDD with lower bad sector.

Anyhow the migration completed successfully , & I have 2x6TB running. Yesterday morning I put in the 3rd HDD (6TB also) to expand the RAID, its been expanding ever since.

Its only at Expanding (Adding disk 16.51%).

But no success. I get access denied when I try to:

admin@DS916:/$ cat /proc/sys/dev/raid/speed_limit_min
10000
admin@DS916:/$ cat /proc/sys/dev/raid/speed_limit_max
200000

admin@DS916:/$ echo 250000> /proc/sys/dev/raid/speed_limit_min
-sh: /proc/sys/dev/raid/speed_limit_min: Permission denied
admin@DS916:/$ echo 250000> /proc/sys/dev/raid/speed_limit_max
-sh: /proc/sys/dev/raid/speed_limit_max: Permission denied

Some help please
blondangel
Beginner
Beginner
Posts: 25
Joined: Wed Feb 01, 2017 5:08 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby blondangel » Fri Feb 10, 2017 5:27 pm

titanax wrote:admin@DS916:/$ echo 250000> /proc/sys/dev/raid/speed_limit_min
-sh: /proc/sys/dev/raid/speed_limit_min: Permission denied
admin@DS916:/$ echo 250000> /proc/sys/dev/raid/speed_limit_max
-sh: /proc/sys/dev/raid/speed_limit_max: Permission denied

Some help please
You will need to be 'root' -- not 'admin.' After you log in as 'admin', then run the following command to log in as root:
sudo su -
and enter your admin password. Then run your 'echo' commands.
titanax
I'm New!
I'm New!
Posts: 8
Joined: Sun Jan 29, 2017 12:08 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby titanax » Tue Feb 21, 2017 2:44 pm

Thank you! Got it.
franckjl
I'm New!
I'm New!
Posts: 3
Joined: Wed Jun 21, 2017 9:01 pm
Location: Paris

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby franckjl » Wed Jun 21, 2017 9:15 pm

Thanx for the tip, B96

I didn't notice much improvement, but still about 10% less time for parity check on a 5x2To RAID5 array filled at 98% (720min to 600min).
I expect a more noticeable improvement for the next step (migration to 5x4To RAID5... one disk at a time...).

@Titanax and others: rather than login in as admin, then su to root, just login as root when you putty|telnet|ssh your NAS. Password is same as admin's


(using DSM 5.2-5967 Update 3)
sgreadly
I'm New!
I'm New!
Posts: 1
Joined: Thu Jun 29, 2017 9:02 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby sgreadly » Wed Jul 19, 2017 9:51 pm

I recall similar experience with my old custom built NAS (Slackware + mdraid). There are 2 steps you need to do to improve the speed.


==================
Enable SSH
==================

If you already have skip this. But for those of you who struggled, make sure you "Enable user home service" in User > Advanced.

You can then ssh with your own username (ensure it has Administrator group privilege) and `sudo su -`


==================
speed_limit_min
==================

This is the first step.

Code: Select all

root:~# cat  /proc/sys/dev/raid/speed_limit_min
1000


Change that from 1,000 to say 50,000


Code: Select all

root:~# echo 50000 >/proc/sys/dev/raid/speed_limit_min


You shouldn't see an immediate noticeable difference. Mine is still as such:


Code: Select all

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc5[2] sda5[0] sdb5[1]
      3902187456 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [===>.................]  reshape = 19.9% (778964736/3902187456) finish=2910.0min speed=17887K/sec
     
md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
      2097088 blocks [4/3] [UUU_]
     
md0 : active raid1 sdc1[2] sda1[0] sdb1[1]
      2490176 blocks [4/3] [UUU_]
     
unused devices: <none>



You can keep an eye out with this line from another SSH session:


Code: Select all

 while true; do clear; cat /proc/mdstat; sleep 1; done


To stop that just ctrl+c


==================
stripe_cache_size
==================

This is where the magic is. It seems the cache size is limited to 256 bytes.

Code: Select all

root:~# cat /sys/block/md2/md/stripe_cache_size
256


Push that up to say 16K

Code: Select all

root:~# echo 16384 >/sys/block/md2/md/stripe_cache_size


Immediate improvement. It took about 2 minutes to reach its new limit. The average time (finish=) dropped by half from ~3000 minutes to ~1400, and speed doubled.


Code: Select all

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc5[2] sda5[0] sdb5[1]
      3902187456 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [====>................]  reshape = 20.1% (784766864/3902187456) finish=1437.1min speed=36152K/sec
     
md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
      2097088 blocks [4/3] [UUU_]
     
md0 : active raid1 sdc1[2] sda1[0] sdb1[1]
      2490176 blocks [4/3] [UUU_]
     
unused devices: <none>
staxim
I'm New!
I'm New!
Posts: 9
Joined: Tue Nov 19, 2013 5:04 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby staxim » Wed Oct 11, 2017 5:04 pm

I just tried using the method above to speed up an 8TB expansion on a brand new DS418. Below are the steps I used to accomplish it, and my results. I posted the specific commands that I used. I thought this might be helpful since some of the screenshots in the original post were deleted, making this thread was a bit difficult to follow.

Summary: I adjusted the raid speed limit max/min, as well as the stripe cache size, but it did not seem to have a significant effect on overall time to complete the expansion. YMMV

My setup: DS418 with just shy of 2TB of data on an SHR 4TB+8TB+8TB volume. No packages installed, virtually everything left default. System is idle aside from the expansion and resource monitor + me running ssh to monitor mdstat in the background. DSM 6.1.3-15152 Update 6.

Details:

I'm moving from a DS414 to a DS418 (I finally hit the 16TB volume size limit and needed to upgrade to something with a 64-bit processor). As part of the build I hit the point where I have a ~10TB SHR volume consisting of 1x 4TB drive and 2x 8TB drives. Adding a new 8TB drive should get me to ~17TB. I've done 8TB expansions on my DS414 in the past and they took several days (more than 3 days, but less than 10 from what I remember), so I wanted to see if I could speed things up a bit.

I started the expansion and let it run for 45 mins. I haven't touched any settings yet, and mdstat tells me that I have 1611 mins remaining (about 26 hours):

Code: Select all

admin:/volume1$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdb5[0] sda5[2] sdd5[1]
      7804374912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     
md3 : active raid5 sdc6[2] sdb6[0] sda6[1]
      3906998912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [>....................]  reshape =  3.3% (129672364/3906998912) finish=1611.2min speed=39071K/sec
     
md1 : active raid1 sdc2[3] sda2[2] sdb2[0] sdd2[1]
      2097088 blocks [4/4] [UUUU]
     
md0 : active raid1 sdc1[3] sda1[2] sdb1[0] sdd1[1]
      2490176 blocks [4/4] [UUUU]


OK, at least it's just a day. But let's see if we can get it to go faster.

I ssh to the box and run the following commands to tweak the settings:

Code: Select all

admin:/volume1$ sudo su -
Password:
root:~# echo 250000 > /proc/sys/dev/raid/speed_limit_max
root:~# echo 250000 > /proc/sys/dev/raid/speed_limit_min
root:~# echo 16384 > /sys/block/md2/md/stripe_cache_size
root:~# cat /proc/sys/dev/raid/speed_limit_max
250000
root:~# cat /proc/sys/dev/raid/speed_limit_min
250000
root:~# cat /sys/block/md2/md/stripe_cache_size
16384


Give it some time to digest, and check it 40 minutes later:

Code: Select all

root:~#  cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdb5[0] sda5[2] sdd5[1]
      7804374912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     
md3 : active raid5 sdc6[2] sdb6[0] sda6[1]
      3906998912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=>...................]  reshape =  5.8% (230455092/3906998912) finish=1559.5min speed=39290K/sec
     
md1 : active raid1 sdc2[3] sda2[2] sdb2[0] sdd2[1]
      2097088 blocks [4/4] [UUUU]
     
md0 : active raid1 sdc1[3] sda1[2] sdb1[0] sdd1[1]
      2490176 blocks [4/4] [UUUU]


Hmm, not much change.

Now watching the entire process, I did notice that the speed reported by mdstat started around 40,000K/sec (3% done), and throughout the day (over the course of 12+ hours) it slowly dropped until I went to bed and it was around 25,000K/sec (48% done). When I woke up in the morning it had sped up to 71,000K/sec (98% done). I don't have an explanation for this odd slow-down / speed-up behavior, but the end result was that it took around 25 hours as the original estimate predicted.

After this first process ended, the system essentially repeated the "checking parity consistency" process for md2, and followed a similar timeline and slow-down / speed-up process. After both md2 and md3 had finished, it took a few minutes to expand the space and the volume was back to normal.

By the way, I followed almost the exact same procedure with the previous 8TB expansion that I did 2 days ago.... and had the same outcome. So, the result/timeline after tweaking the settings seems fairly repeatable.

Although it didn't seem like the tweaked settings had any material effect on the overall timeline, the volume expansion was pretty quick overall even by the "default" estimate of ~1d each for md2 and md3. So I can't complain, but wanted to post my results here for the record.
viper359
I'm New!
I'm New!
Posts: 2
Joined: Fri Sep 08, 2017 3:58 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby viper359 » Fri Oct 13, 2017 1:52 pm

When i run the command echo 16384 >/sys/block/md2/md/stripe_cache_size it works for about 15 seconds then it returns to 256!!

Is there something new in DSM6.1.3 that is stopping this from taking effect permanently?

In the interim I have just created a task scheduler in the control panel to run every minute, to echo the change. it is working
Thor68
Experienced
Experienced
Posts: 112
Joined: Sun Oct 09, 2011 8:09 pm
Location: Oslo, Norway

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby Thor68 » Sun Oct 15, 2017 1:06 am

viper359 wrote:When i run the command echo 16384 >/sys/block/md2/md/stripe_cache_size it works for about 15 seconds then it returns to 256!!

Is there something new in DSM6.1.3 that is stopping this from taking effect permanently?

In the interim I have just created a task scheduler in the control panel to run every minute, to echo the change. it is working


I only set the value to 4096, but it has stayed at that value for two hours now (running 6.1.3 U7).
DS1815+ [16GB RAM / SHR-1 (2x10TB + 5x4TB, btrfs) / DSM 6.1.4-15217-U5] - Primary server (online since june 2016, no patch)
DS1812+ [3GB RAM / SHR-1 (mixed, btrfs) / DSM 6.1.4-15217-U5] - Backup for the 1815+
DS1511+ [3GB RAM / SHR-1 (2x2TB, ext4) / DSM 6.1.4-15217-U5] - BT Client / HDD tester/ HDD eraser
pgotze
Novice
Novice
Posts: 49
Joined: Thu Oct 03, 2013 8:43 am

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby pgotze » Wed Nov 29, 2017 10:03 pm

Check in Storage Manager / Storage Pool / Configurations

Here you can a bit increase performance of such operations.
DS916+(3x2TB SHR, 1x1TB single) primary, DS213j(1x1 TB, 1x 500GB) off-site backup
brimur
Experienced
Experienced
Posts: 101
Joined: Sat Jan 12, 2013 8:22 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby brimur » Thu Jan 18, 2018 5:41 pm

@staxim In case you are ever doing this again note that you updated MD2 when MD3 was the array that was repairing.


staxim wrote:I just tried using the method above to speed up an 8TB expansion on a brand new DS418. Below are the steps I used to accomplish it, and my results. I posted the specific commands that I used. I thought this might be helpful since some of the screenshots in the original post were deleted, making this thread was a bit difficult to follow.

Summary: I adjusted the raid speed limit max/min, as well as the stripe cache size, but it did not seem to have a significant effect on overall time to complete the expansion. YMMV

My setup: DS418 with just shy of 2TB of data on an SHR 4TB+8TB+8TB volume. No packages installed, virtually everything left default. System is idle aside from the expansion and resource monitor + me running ssh to monitor mdstat in the background. DSM 6.1.3-15152 Update 6.

Details:

I'm moving from a DS414 to a DS418 (I finally hit the 16TB volume size limit and needed to upgrade to something with a 64-bit processor). As part of the build I hit the point where I have a ~10TB SHR volume consisting of 1x 4TB drive and 2x 8TB drives. Adding a new 8TB drive should get me to ~17TB. I've done 8TB expansions on my DS414 in the past and they took several days (more than 3 days, but less than 10 from what I remember), so I wanted to see if I could speed things up a bit.

I started the expansion and let it run for 45 mins. I haven't touched any settings yet, and mdstat tells me that I have 1611 mins remaining (about 26 hours):

Code: Select all

admin:/volume1$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdb5[0] sda5[2] sdd5[1]
      7804374912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     
md3 : active raid5 sdc6[2] sdb6[0] sda6[1]
      3906998912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [>....................]  reshape =  3.3% (129672364/3906998912) finish=1611.2min speed=39071K/sec
     
md1 : active raid1 sdc2[3] sda2[2] sdb2[0] sdd2[1]
      2097088 blocks [4/4] [UUUU]
     
md0 : active raid1 sdc1[3] sda1[2] sdb1[0] sdd1[1]
      2490176 blocks [4/4] [UUUU]


OK, at least it's just a day. But let's see if we can get it to go faster.

I ssh to the box and run the following commands to tweak the settings:

Code: Select all

admin:/volume1$ sudo su -
Password:
root:~# echo 250000 > /proc/sys/dev/raid/speed_limit_max
root:~# echo 250000 > /proc/sys/dev/raid/speed_limit_min
root:~# echo 16384 > /sys/block/md2/md/stripe_cache_size
root:~# cat /proc/sys/dev/raid/speed_limit_max
250000
root:~# cat /proc/sys/dev/raid/speed_limit_min
250000
root:~# cat /sys/block/md2/md/stripe_cache_size
16384


Give it some time to digest, and check it 40 minutes later:

Code: Select all

root:~#  cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdb5[0] sda5[2] sdd5[1]
      7804374912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     
md3 : active raid5 sdc6[2] sdb6[0] sda6[1]
      3906998912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=>...................]  reshape =  5.8% (230455092/3906998912) finish=1559.5min speed=39290K/sec
     
md1 : active raid1 sdc2[3] sda2[2] sdb2[0] sdd2[1]
      2097088 blocks [4/4] [UUUU]
     
md0 : active raid1 sdc1[3] sda1[2] sdb1[0] sdd1[1]
      2490176 blocks [4/4] [UUUU]


Hmm, not much change.

Now watching the entire process, I did notice that the speed reported by mdstat started around 40,000K/sec (3% done), and throughout the day (over the course of 12+ hours) it slowly dropped until I went to bed and it was around 25,000K/sec (48% done). When I woke up in the morning it had sped up to 71,000K/sec (98% done). I don't have an explanation for this odd slow-down / speed-up behavior, but the end result was that it took around 25 hours as the original estimate predicted.

After this first process ended, the system essentially repeated the "checking parity consistency" process for md2, and followed a similar timeline and slow-down / speed-up process. After both md2 and md3 had finished, it took a few minutes to expand the space and the volume was back to normal.

By the way, I followed almost the exact same procedure with the previous 8TB expansion that I did 2 days ago.... and had the same outcome. So, the result/timeline after tweaking the settings seems fairly repeatable.

Although it didn't seem like the tweaked settings had any material effect on the overall timeline, the volume expansion was pretty quick overall even by the "default" estimate of ~1d each for md2 and md3. So I can't complain, but wanted to post my results here for the record.
DiskStation: 413j,1815+
DSM 6.2-22259
jespermart
I'm New!
I'm New!
Posts: 2
Joined: Thu Feb 18, 2016 2:44 pm

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby jespermart » Sun Jan 28, 2018 3:15 am

viper359 wrote:When i run the command echo 16384 >/sys/block/md2/md/stripe_cache_size it works for about 15 seconds then it returns to 256!!

Is there something new in DSM6.1.3 that is stopping this from taking effect permanently?

In the interim I have just created a task scheduler in the control panel to run every minute, to echo the change. it is working


How did you create this task?
Johan_A_M
I'm New!
I'm New!
Posts: 2
Joined: Fri Jan 05, 2018 11:52 am

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby Johan_A_M » Sun Feb 04, 2018 2:32 pm

Hi,

I'm completely new here, and an absolute newbie when it comes to everything ssh, linux etc. I've just added a second WD Red 8Tb to my 418play and in Putty it says I have 160000 minutes left... I'm trying all tips in here and the only thing that doesn't work is the stripe cache, because I get an "permission denied". I have logged in as root. I'm sure there's an easy explanation but as I said, I know nothing in this area. Would really appreciate some advice.

Very best
J
Johan_A_M
I'm New!
I'm New!
Posts: 2
Joined: Fri Jan 05, 2018 11:52 am

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby Johan_A_M » Tue Feb 06, 2018 2:32 pm

Johan_A_M wrote:Hi,

I'm completely new here, and an absolute newbie when it comes to everything ssh, linux etc. I've just added a second WD Red 8Tb to my 418play and in Putty it says I have 160000 minutes left... I'm trying all tips in here and the only thing that doesn't work is the stripe cache, because I get an "permission denied". I have logged in as root. I'm sure there's an easy explanation but as I said, I know nothing in this area. Would really appreciate some advice.

Very best
J


I can answer myself and tell that it all works now.

Another question though: When everything is done, should I restore the original values?
User avatar
Exsosus
Trainee
Trainee
Posts: 12
Joined: Thu Oct 14, 2010 8:59 am

Re: Diskstation expanding array - adding disks is slow - How to make expansion faster

Postby Exsosus » Mon Feb 12, 2018 10:27 am

Hi Johan,

Your question is just as good as a yes or no. If you wish, you can restore the settings to Synologys DSM main settings manually over SSH which is something I did. If you're feeling more experienced, and when you learn the linux commands even better, experiment with keeping some settings.

Remember, Synology set the RAID rebuild optimized for you to be able to use your NAS while its repairing / re-syncing / re-whatever disk doing.
----
Windows 10 64-bit
Ryzen 7 clocked @ 3500Mhz
32GB memory clocked @ 2900Mhz
Synology NAS solution: Synology Disk Station DS1815+ with lots of disks + in the background Synology DS411J NAS

Return to “System Managment Mods”

Who is online

Users browsing this forum: No registered users and 3 guests