How long to reshape

Questions pertaining to Power settings, Auto Block, Permissions, User Quotas and Email alerts may be posted here.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
kentvv
Trainee
Trainee
Posts: 14
Joined: Tue Oct 07, 2008 7:45 am

How long to reshape

Postby kentvv » Wed Oct 08, 2008 7:32 am

What determines the duration of a reshaping operation? Is it the capacity of the drives, how full the drives are, or the number of files present on the volume?

Does the reshaping operation vary in its rate or does the rate stay fairly consistent throughout the entire process?

I started a reshaping operation to take three 1TB drives to four 1TB drives and at the current rate it seems the reshaping operation will require eight days. Does that see reasonable? The original volume was nearly full and contained ~3 million files.

Thank you

The contents of /proc/mdstat are shown below. Does 1415K/sec seem reasonable? (The drive lights are barely flashing.) Is there anything that can be done to speed this process up? The volume still appears mounted from the perspective of the shell. Is it safe to manipulate the files while reshaping? Can the sharing processes be brought back up while reshaping? Read-only would be fine if that is required.

Code: Select all

DiskStation> cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md2 : active raid5 sdd3[3] sda3[0] sdb3[1] sdc3[2]
      1947238400 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  1.0% (10032768/973619200) finish=11328.4min speed=1415K/sec

md1 : active raid1 sdd2[3] sda2[0] sdb2[1] sdc2[2]
      522048 blocks [5/4] [UUUU_]

md0 : active raid1 sdd1[3] sda1[0] sdb1[1] sdc1[2]
      2489920 blocks [5/4] [UUUU_]

unused devices: <none>
kentvv
Trainee
Trainee
Posts: 14
Joined: Tue Oct 07, 2008 7:45 am

Re: How long to reshape

Postby kentvv » Wed Oct 08, 2008 4:21 pm

Increasing /proc/sys/dev/raid/speed_limit_min to 40000 and setting /proc/sys/dev/raid/speed_limit_max to zero has increased the reshaping rate by tenfold. Why not set this parameters by default when adding a disk? After all, if the share services are down the system is not going to be used for anything during the reshaping.
vger
I'm New!
I'm New!
Posts: 3
Joined: Wed Oct 29, 2008 4:02 am

Re: How long to reshape

Postby vger » Fri Nov 07, 2008 12:15 am

Many thanks to Kentvv for the hint on those 2 parameters !!! :D
it makes the reorg of the RAID come down from 8 days to 22H on my DS508 going from 3 to 4 disks of 1TB

From my experience here is how it goes :
those 2 parameters can be manually and dynamicly modified by doing a simple echo command redirected to the name of the file
I have entered 0 for speed_limit_max as suggested by Kentvvv and entered 40000 for speed_limit_min
In firmware 722 (full re-installed), values are by default : min = 1000 and max = 200000, and the reorg tends to dramaticly aligned on the min value... I was around 1200 KB/s...
After dynamicly changed min and max, I get almost immediately 11500 KB/s on around 50% of the reorg, with even +23000 KB/s in the second half of the reorg...
I already did a RAID increase in the firmware 640 in the past months and the timing was also 22H... so I guess something has disappeared in the 722 release...
Also don't underestimate the duration of the second operation : the filesystem resizing !

Here is the final statistics I can give everyone (based on my experience and the command and parameters given by kentvv) :
DS-508, going from 3 to 4 disks of 1 TB in RAID5, firmware 722, free space less than 1GB on the current space
to know default values, see content of following files (under root user, do a cat on the following files in /proc/sys/dev/raid) : speed_limit_min and speed_limit_max
before : min = 1000 and max = 200000 (those values are KB by seconds)
to see the current value of progress of your reorg of data : cat /proc/mdstat -> in my case, rate was around 1200 KB/s and time expected was around 15000 min (8 days)
modifying values to :
min = 40000, go into the directory, and type : echo 40000 > speed_limit_min
max = 0, go into the directory, and type : echo 0 > speed_limit_max
to see the changes, issue a command : cat /proc/mdstat, you will notice the difference almost immediately -> in my case 11500 KB/s, then 22000 to 25000 KB/s later
So after 22H, the reorg is done... now starts automatically the resize operation of the filesystem to take benefit of your new disk
This final step will take around 2 hours : when you start, the progress rate is increasing in a way you may expect 15mn would be enough to complete but... it would be too easy !!! In fact, when around 50% of progress, the progress rate will increase much more slowly.. who knows why... and at the end it represents around 2 hour for this step...
So globally, adding a 4th disk of 1 TB ends represents around 25 hours of unavailability of your shares ! :)

Final hint for all : regarding the duration of that kind of critical operations, I encourage everyone to use a UPS... I can't imagine what would result from a power cut during the reorg...a mess for sure... :shock:
User avatar
Franklin
Synology Inc
Synology Inc
Posts: 6664
Joined: Sat Oct 14, 2006 11:33 pm
Location: Washington, USA

Re: How long to reshape

Postby Franklin » Fri Nov 21, 2008 9:59 pm

@all

I'll consult with our RAID developers to see about this thread; however I wouldn't advise changing the RAID settings, as it could produce unforeseen errors.
**Franklin is not available**
**Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!**
Library ~ SynologyWiki ~ Synology FAQ ~ Compatibility Lists
Forum Links ~ Forum Policy ~ 3rd-party forums ~ Help us help you ~ Posting Images
Demo Links ~ DSM GUI ~ Photo Station
Downloads ~ Firmware Downloads ~ Beta Program
Support ~ Support Form ~ Submit Kernel ~ Synology eNews
kentvv
Trainee
Trainee
Posts: 14
Joined: Tue Oct 07, 2008 7:45 am

Re: How long to reshape

Postby kentvv » Mon Nov 24, 2008 6:38 pm

Unless Synology has messed with kernel components, I suspect using a normal Linux interface to increase the rate of reshaping has less chance of producing unforeseen errors than allowing reshaping to take over a week to execute during which the RAID is unprotected. Surely no one would find this length of time acceptable.
User avatar
Franklin
Synology Inc
Synology Inc
Posts: 6664
Joined: Sat Oct 14, 2006 11:33 pm
Location: Washington, USA

Re: How long to reshape

Postby Franklin » Tue Nov 25, 2008 12:49 am

@all

Our engineering team has noticed a performance abnormality with the Linux Kernels 2.6.24 and reshaping the volume. Thus far, it appears to be affecting just this kernel, and our engineering team is currently investigating this issue.

For those who wish to have a faster method of getting their RAID system online, this thread contains useful information.

Hope this helps
**Franklin is not available**

**Please do not Private Message me for support questions; leave it on the forum so all members can learn. Thanks!**

Library ~ SynologyWiki ~ Synology FAQ ~ Compatibility Lists

Forum Links ~ Forum Policy ~ 3rd-party forums ~ Help us help you ~ Posting Images

Demo Links ~ DSM GUI ~ Photo Station

Downloads ~ Firmware Downloads ~ Beta Program

Support ~ Support Form ~ Submit Kernel ~ Synology eNews

carloc
Trainee
Trainee
Posts: 11
Joined: Wed Jun 02, 2010 7:27 am

Re: How long to reshape

Postby carloc » Sat Apr 07, 2012 5:33 pm

Hi All,

It has been a long while sins 2008, but this problem (still?) exists in DSM 4.0 on a DS412+!

I just bought a new Synology because my old 210j couldn't handle my demand anymore !.
I bought a brand new DS412+ with 2 brand new Hitachi 3TB discs.

First i installed the nas with the 2 new drives with the last DSM4.0, then i copied over all my data from my old nas and placed my old WD 3TB drive in my new drive. i added that one to my Hybr Raid. After a day it was still on 15% !

Than i used this link :
http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

My speed was around 1000kb a second ! After the changes it went to 77.000 kb/second !
It changed the 18 day's into 1 day !

I'm a technical guy, but a lot of the Synology buyers aren't !
Why is synology using this setting in a extremely low level (1000)? :?
I really don't understand why ! Linux distro's usealy use arround 50.000 en 100.000 for there min and max.

Please synology : correct this in future releases ! :roll:
scarface
Trainee
Trainee
Posts: 13
Joined: Mon May 07, 2012 6:54 am

Re: How long to reshape

Postby scarface » Sat May 19, 2012 1:43 pm

It seems they listened to some degree.

Currently I am expanding a RAID5 3x3TB to 4x3TB.

Code: Select all

DATAVAULT> ls
speed_limit_max  speed_limit_min
DATAVAULT> cat *
200000
1000


The low number is still pretty damn low, but from what I understand this means that rebuilds will just stand way back in priority when there is actual activity on the disks from users.

Without that I get:

Code: Select all

cat /proc/mdstat
      5851088512 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [==========>..........]  reshape = 52.5% (1536863104/2925544256) finish=772.1min speed=29973K/sec


Doing file reads and copies (read from NAS, write to NAS) in rebuild I get:

Code: Select all

cat /proc/mdstat
      5851088512 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [==========>..........]  reshape = 53.0% (1550766848/2925544256) finish=6245.7min speed=3668K/sec


But after canceling those it goes back up very quickly.

Unfortunately:

Code: Select all

echo 50000 > /proc/sys/dev/raid/speed_limit_min
-sh: can't create /proc/sys/dev/raid/speed_limit_min: Permission denied
[...]
sysctl -w dev.raid.speed_limit_min=50000
error: permission denied on key 'dev.raid.speed_limit_min'


Even though I am logged into SSH as admin, which should be giving me total control (at least that is my experience with my former QNAP) it seems the files are locked down. The files are owned by root and -rw-r--r--. chown and chmod don't seem to work.

It puzzles me that it's "only" close to 30MB/sec, though the disks should be a tad bit faster than that. The CPU is not even lose to being on the edge, however the load averages are quite high...

Code: Select all

CPU:  1.4% usr 10.7% sys  0.0% nic 86.0% idle  0.0% io  0.0% irq  1.7% sirq
Load average: 2.34 2.33 2.27 2/122 11549


Just FYI. And for mine: were you able to simply change the values in your NAS model under DSM4? Or did you have to some extra way? I also use 3TB Hitachi drives and while close to 30MB ist not awful, 77MB sure sounds better :D

(EDIT: Never mind, I am an idiot. I could login as root with my known pw as well and change the settings. Raising the max value seems to do something. 200000 is a lot already, went carefully to 250000. Gonna see if I can push it any further.)

One last edit: Pushing the max does not seem to do a lot. It got me from 27MB to 29 or 30 (varies). I put max back to 200000 and set min to various numbers between 50000 and 150000 and it still maxs out in the 30MB area...
Erhnam
Trainee
Trainee
Posts: 14
Joined: Thu Aug 12, 2010 12:44 pm

Re: How long to reshape

Postby Erhnam » Sun Sep 15, 2013 2:30 pm

Here another way to speed it up:

Change the value of the stripe_cache_size

Code: Select all

echo 32768 > /sys/block/md2/md/stripe_cache_size


I also changed the speed_limit_min (to 70000) and speed_limit_max (to 0). Resync speed is now above 70.000 k/sec, see below:

Code: Select all

DiskStation> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc5[2] sdb5[1] sda5[0]
      5851063296 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [>....................]  resync =  3.0% (88676992/2925531648) finish=675.3min speed=70003K/sec
     
md1 : active raid1 sda2[0] sdb2[1] sdc2[2]
      2097088 blocks [8/3] [UUU_____]
     
md0 : active raid1 sda1[0] sdb1[1] sdc1[2]
      2490176 blocks [8/3] [UUU_____]
     
unused devices: <none>
DiskStation>
dclive
I'm New!
I'm New!
Posts: 2
Joined: Sun Nov 11, 2012 6:07 pm

Re: How long to reshape

Postby dclive » Wed Mar 16, 2016 4:12 am

Thanks all for posting this - it's really helpful.

I have a 414. I had 2 6TB disks in an SHR mirror set. I then added another 6TB disk to make a SHR 'raid5' set.

So far it's been 5 days, and it's at 48% done:

DS2> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid1 sdd2[4] sdc2[2] sdb2[1] sda2[0]
2097088 blocks [4/3] [UUU_]
resync=DELAYED

md4 : active raid5 sdd5[2] sda5[0] sdc5[1]
5855787648 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[=========>...........] reshape = 47.7% (2794081920/5855787648) finish=8394.9min speed=6077K/sec

md3 : active raid1 sdb5[0]
972027648 blocks super 1.2 [1/1] [U]

md0 : active raid1 sdd1[0] sda1[3] sdb1[2] sdc1[1]
2490176 blocks [4/4] [UUUU]

I have done the max and min modifications:

DS2> sysctl dev.raid.speed_limit_min
dev.raid.speed_limit_min = 50000
DS2> sysctl dev.raid.speed_limit_max
dev.raid.speed_limit_max = 90000

...but my speed hasn't changed a bit (should it start going up right away?)...CPU utilization hasn't changed, nothing has changed.

What should I be looking at here?

Return to “System Management”

Who is online

Users browsing this forum: No registered users and 1 guest