DS509+ and DX5 - 10 Disk RAID 6 array?

[alex11561]

How can I create a raid 6 array encompassing all the drives on a fully populated DS509+ linked to a DX5?

[alex11561]

This is not a reply, but rather a clarification.
It would be nice if you could just select all 10 disks in a DS509+ DX5 combo and choose "create Raid6", but that's not an option.
All the menu system allows is choosing all the disks in the DS509+ OR all the disks in the DX5 and choose to create a Raid6 volume. I thought, Okay, I'll create a Raid6 on the DS509+ and then expand it to the DX5.
It takes about 16 hours to create a Raid6 on the DS509+ using 5 disks. Okay, come back the next day and choose Expand, select all 5 disks on the DX5, and the dialog pops up asking you what type of volume to create. basic is greyed out, but I want a Raid6 volume, so I choose Raid6 and go away for another day.
Next day I come back to find what I ended up with was some weird volume with the capacity of only 6 disks, and I assume 4 parity disks. Not good. Okay, start over and create a Raid6 on the DS509+ from scratch. There is no Unexpand option.
That was yesterday. Today I figure I should try adding one disk from the DX5 to the existing volume at a time. Looking at the progress so far, I figure a best case scenario of about 4 to 5 hours to add each disk. I'm still working on that first disk addition, so I don't know if it will even work.
I was hoping some Synology engineer would be monitoring this board, and might save me what's turning out to be many days of trial and error on this. There is no mention of this in any manual I have found and so far it has turned out to be far from intuitive.
I figure another two days to add all the disks from the DX5, assuming the first disk adds correctly. I'll check back with my progress.
Does anybody know what the basic raid system is? Could I go to a command line and do all this with mdadm?

[alex11561]

No go. After you've expanded one disk to the DS509+, it won't allow further expansion to that volume. All it allows is expansion of the DX5 volume. Just for a lark I chose Raid5. Do you wanna bet I end up with 3 parity disks?
Judging by the progress so far, I won't know until tomorrow.

[alex11561]

I was right, I ended up with 3 parity disks. Okay, I've deleted the volume, SSH'd in and I'm using mdadm to create a raid6 volume (/dev/md101) using the available partitions, /dev/sd[a-e]3 and /dev/sdf[a-e]. mdstat says it'll take 50 hours to create it. I'll format it ext3 when it's done. It looks like it will just mount itself. Anybody have any experience with this? Will I have to add an fstab entry?

[jekyll]

How can I create a raid 6 array encompassing all the drives on a fully populated DS509+ linked to a DX5?

To my knowledge the DX5 can not be part of the RAID array on a DS509+

[Derf Elot]

Maybe it's true what jekyll says and its not possible. Could also be that you found a bug in DSM seeing how you are probably one of the first who are playing with the DX5.

As for the fstab, I think it is not possible (afaik) to manually edit it, as it is rewritten from flash at every reboot. This is how I understand it anyways. A possible workaround you could experiment with would be:

1) create Raid6 array in DSM
2) use telnet/ssh to --stop the array (should be /dev/md2)
3) recreate the array manually using the --create option (back onto /dev/md2)
4) format it using "mke2fs -j" (not entirely sure if this would be needed even, but I guess it wont hurt)
5) reboot & pray it worked

This might just work, but I can't give you any guarantees of course. This way, the /dev/md2 is properly stored in the flash and wont have to manually enter it anywhere - also, it would be properly shown in the DSM.

Good luck,
Derf

[jekyll]

5) reboot & pray it worked

To correct my last post

Using a DS509+ with a DX5, yes, you can have a 10x2TB HDD volume, if you wanted a 20TB Volume in RAID-0 format, then yes, you could go for that. However, I wouldn't recommended it.

There is a small performance drop while using the DX5 - however the benefits of having scalable storage outweight the performance drop (and again, it's close to being not perceivable by users)

Sorry about this...

[alex11561]

Here I am again. It finally finished creating the raid6 array. During the build DSM listed the array as 'crashed', but that didn't worry me because I hadn't formatted the volume yet. BTW, I had chosen the array device as /dev/md101 because all my previous attempts through DSM had that ID.
> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md101 : active raid6 sdfe[9] sdfd[8] sdfc[7] sdfb[6] sdfa[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
15602969600 blocks level 6, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU]

md1 : active raid1 sde2[3] sdd2[4] sda2[0] sdb2[1] sdc2[2]
522048 blocks [5/5] [UUUUU]

md0 : active raid1 sde1[3] sdd1[4] sda1[0] sdb1[1] sdc1[2]
2489920 blocks [5/5] [UUUUU]

unused devices: <none>
It appeared to build the volume Okay, so I typed 'mke2fs -L (something) -j /dev/md101' That told me that the volume was too big to format as ext3, which has a maximum size of 8TB.
> mke2fs -j /dev/md101
mke2fs 1.39 (29-May-2006)
mke2fs: Filesystem too large. No more than 2**31-1 blocks
(8TB using a blocksize of 4k) are currently supported.

'Franklin' claims the current firmware recognises a 16tb partition, which would be just right for me, so I enter 'mke2fs -L (something) -j -F /dev/md101'. The '-F' should force it to format the array anyway, though I find it suspicious that the version of mke2fs on the system is from 2006.

> mke2fs -j -F /dev/md101
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1950384128 inodes, 3900742400 blocks
25600 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3904897024
119042 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.


It reports no errors, and I reboot.
Well, after the reboot the array comes up with only 5 drives out of ten, listing only the drives on the DS509+.
> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md101 : active raid6 sda3[0] sdb3[1] sdc3[2] sdd3[3] sde3[4]
15602969600 blocks level 6, 64k chunk, algorithm 2 [10/5] [UUUUU_____]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3]
522048 blocks [5/5] [UUUUU]

md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[4] sde1[3]
2489920 blocks [5/5] [UUUUU]

unused devices: <none>


The system still sees the DX5 partitions:
> cat /proc/partitions
major minor #blocks name

8 0 1953514584 sda
8 1 2490043 sda1
8 2 522112 sda2
8 3 1950371325 sda3
8 16 1953514584 sdb
8 17 2490043 sdb1
8 18 522112 sdb2
8 19 1950371325 sdb3
8 32 1953514584 sdc
8 33 2490043 sdc1
8 34 522112 sdc2
8 35 1950371325 sdc3
8 48 1953514584 sdd
8 49 2490043 sdd1
8 50 522112 sdd2
8 51 1950371325 sdd3
8 64 1953514584 sde
8 65 2490043 sde1
8 66 522112 sde2
8 67 1950371325 sde3
129 192 1953514584 sdfa
129 208 1953514584 sdfb
129 224 1953514584 sdfc
129 240 1953514584 sdfd
130 0 1953514584 sdfe
9 0 2489920 md0
9 1 522048 md1
9 101 15602969600 md101

mdadm refuses to add any of the drives from the DX5

> mdadm -a /dev/sdf[a-e]
mdadm: /dev/sdfa does not appear to be an md device


Okay, let's make some md devices:

> fdisk /dev/sdfa
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 243201.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-243201, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-243201, default 243201):
Using default value 243201

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Same for all 5 disks on the DX5.
Then start making the array from scratch:

> mdadm --create --verbose /dev/md101 --level=6 --raid-devices=10 /dev/sd[a-e]3 /dev/sdf[a-e]1
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sda3 appears to contain an ext2fs file system
size=-1591578632K mtime=Mon Jan 28 07:21:55 2030
mdadm: /dev/sda3 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sun May 10 12:40:01 2009
mdadm: /dev/sdb3 appears to contain an ext2fs file system
size=-1576899584K mtime=Wed Dec 31 19:00:00 1969
mdadm: /dev/sdb3 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sun May 10 12:40:01 2009
mdadm: /dev/sdc3 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sun May 10 12:40:01 2009
mdadm: /dev/sdd3 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sun May 10 12:40:01 2009
mdadm: /dev/sde3 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sun May 10 12:40:01 2009
mdadm: size set to 1950371200K
Continue creating array? y
mdadm: array /dev/md101 started.


and this time I won't wait to format it:

> mkfs.ext3 -F /dev/md101
mke2fs 1.39 (29-May-2006)
Filesystem label=1.39-Apr132009
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1950384128 inodes, 3900742400 blocks
25600 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3904897024
119042 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432

Writing inode tables:


I mount it manually:

>mount -t ext3 /dev/md101 /volume1

And DSM sees it as a Raid6 volume that is checking parity consistency and has the capacity of 8 disks. It sounds like success, butwhen I check to see which disks are a part of the volume, it only shows the five disks of the DS509+. This makes me think the volume might not be there when I reboot. I won't find out until the parity check is done, which won't be for a while...

[Derf Elot]

Hi,

i believe the OS as well as swap partition need to be present on every disk, meaning that your md0 and md1 raid array should spread across all 10 disks, not just the first 5.

The volume creation in the DSM will take care of this partitioning and raid creation, so I would still suggest you try the workaround I mentioned above. I dont think your current attempt is going to work either, unfortunately.

Best,
Derf

[alex11561]

That sounds like the right answer to me. I'll wait to see what happens after the array is synchronised, but if it doesn't work then I'll expand md0 and md1 to the DX5 and recreate the array that way. I bet that will make DSM see all the disks.

Thanks Derf.

[alex11561]

Okay, that didn't work. I'm trying Derfs suggestion. I'll check back when it's done.

[alex11561]

Okay, I've been working on redoing the build. I've noticed that the DS509+ is using LVM to join the DS509+ to the DX5. The automatic raid creation did not expand the system or swap partitions to the DX5. If you've got some time to kill, below is what I've done so far. Right now it's synchronising, so I don't know if this setup will survive a reboot.

After building a raid6 on the DS509+ and expanding it to the DX5:
command> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md101 : active raid6 sdfe[4] sdfd[3] sdfc[2] sdfb[1] sdfa[0]
5860543488 blocks level 6, 64k chunk, algorithm 2 [5/5] [UUUUU]

md2 : active raid6 sda3[0] sdb3[1] sdc3[2] sdd3[3] sde3[4]
5851113600 blocks level 6, 64k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3]
522048 blocks [5/5] [UUUUU]

md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[4] sde1[3]
2489920 blocks [5/5] [UUUUU]


command> mount
/dev/md0 on / type ext3 (rw,data=ordered)
/tmp on /tmp type tmpfs (rw)
/sys on /sys type sysfs (rw)
/proc/bus/usb on /proc/bus/usb type usbfs (rw)
/dev/vg1/lv on /volume1 type ext3 (usrquota,grpquota)

command> umount /volume1

command> pvdisplay /dev/md101
--- Physical volume ---
PV Name /dev/md101
VG Name vg1
PV Size 5.46 TB / not usable 3.00 MB
Allocatable yes
PE Size (KByte) 16384
Total PE 357699
Free PE 357699
Allocated PE 0
PV UUID PxCGZ8-syOZ-oR9A-Hr60-zwD3-3HyD-Q21Htt


command> vgdisplay /dev/vg1
--- Volume group ---
VG Name vg1
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 10.91 TB
PE Size 16.00 MB
Total PE 714822
Alloc PE / Size 714822 / 10.91 TB
Free PE / Size 0 / 0
VG UUID tFyR2t-c4Z3-Cr3L-vena-WoH4-XGF6-bD2fNU

command> lvremove /dev/vg1/lv
Do you really want to remove active logical volume "lv"? [y/n]: y
Logical volume "lv" successfully removed

command> vgreduce /dev/vg1 /dev/md101
Removed "/dev/md101" from volume group "vg1"

command> vgremove /dev/vg1
Volume group "vg1" successfully removed

command> mdadm -S /dev/md101
mdadm: stopped /dev/md101

command> mdadm -S /dev/md2
mdadm: stopped /dev/md2

command> mdadm --create /dev/md2 --level=6 --raid-devices=10 /dev/sd[a-e]3 /dev/sdf[a-e]
mdadm: /dev/sda3 appears to be part of a raid array:
level=raid6 devices=5 ctime=Thu May 14 15:43:26 2009
mdadm: /dev/sdb3 appears to be part of a raid array:
level=raid6 devices=5 ctime=Thu May 14 15:43:26 2009
mdadm: /dev/sdc3 appears to be part of a raid array:
level=raid6 devices=5 ctime=Thu May 14 15:43:26 2009
mdadm: /dev/sdd3 appears to be part of a raid array:
level=raid6 devices=5 ctime=Thu May 14 15:43:26 2009
mdadm: /dev/sde3 appears to be part of a raid array:
level=raid6 devices=5 ctime=Thu May 14 15:43:26 2009
mdadm: /dev/sdfa appears to be part of a raid array:
level=raid6 devices=5 ctime=Fri May 15 15:02:27 2009
mdadm: /dev/sdfb appears to be part of a raid array:
level=raid6 devices=5 ctime=Fri May 15 15:02:27 2009
mdadm: /dev/sdfc appears to be part of a raid array:
level=raid6 devices=5 ctime=Fri May 15 15:02:27 2009
mdadm: /dev/sdfd appears to be part of a raid array:
level=raid6 devices=5 ctime=Fri May 15 15:02:27 2009
mdadm: /dev/sdfe appears to be part of a raid array:
level=raid6 devices=5 ctime=Fri May 15 15:02:27 2009
Continue creating array? y
mdadm: array /dev/md2 started.

command> pvcreate /dev/md2
Physical volume "/dev/md2" successfully created

command> vgcreate /dev/vg1 /dev/md2
Volume group "vg1" successfully created

command> pvdisplay /dev/md2
--- Physical volume ---
PV Name /dev/md2
VG Name vg1
PV Size 14.53 TB / not usable 3.00 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 3809318
Free PE 3809318
Allocated PE 0
PV UUID hgozSn-2EGv-kWyp-HVMU-1EWC-CleJ-oHDUli

command> lvcreate -l 3809318 vg1 -n lv
Logical volume "lv" created

command> lvdisplay
--- Logical volume ---
LV Name /dev/vg1/lv
VG Name vg1
LV UUID 0y56Rp-yA3M-Sows-WV31-cUJC-WUq1-Yu80M6
LV Write Access read/write
LV Status available
# open 0
LV Size 14.53 TB
Current LE 3809318
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

command> mkfs.ext3 -F /dev/vg1/lv
mke2fs 1.39 (29-May-2006)
Filesystem label=1.39-Apr132009
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1950384128 inodes, 3900741632 blocks
25600 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3904897024
119042 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

command> mount -t ext3 /dev/vg1/lv /volume1


DSM shows a raid6 array with the proper capacity, but the 'get info' window is showing the error:

# "Failed to access the system partition of some disks. But, the data is not effected. This failure could be resulted by that the bad sectors or older system are detected on the system partition. We suggest that you replace the hard disk with the status [System Partition Failed] with a healthy one, or reboot to repair system partition automatically.

I guess I should have expanded md0 & md1, even though the status of all disks shows normal. I'll see if this setup survives reboot.

[alex11561]

Failed again.
Okay, I've removed the failed volume and partitioned the DX5 disks to match the DS509+ disks:


command> fdisk /dev/sda
Command (m for help): p

Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 310 2490043+ fd Linux raid autodetect
/dev/sda2 311 375 522112+ fd Linux raid autodetect
/dev/sda3 392 18446744073709527469 18446744073512439293 fd Linux raid autodetect

Command (m for help): q

command> fdisk /dev/sdfa
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 243201.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-243201, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-243201, default 243201): 310

Command (m for help): t
Selected partition 1
Hex code (type L to list codes):
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (311-243201, default 311): 311
Last cylinder or +size or +sizeM or +sizeK (311-243201, default 243201): 375

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (376-243201, default 376): 392
Last cylinder or +size or +sizeM or +sizeK (392-243201, default 243201):
Using default value 243201

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): p

Disk /dev/sdfa: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdfa1 1 310 2490043+ fd Linux raid autodetect
/dev/sdfa2 311 375 522112+ fd Linux raid autodetect
/dev/sdfa3 392 18446744073709527469 18446744073512439293 fd Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Same for all 5 disks.
Then I expanded md0 & md1:

command> mdadm /dev/md0 --add /dev/sdf[a-e]1

command> mdadm -G /dev/md0 -n 10

command> mdadm /dev/md1 --add /dev/sdf[a-e]2

command> mdadm -G /dev/md1 -n 10

command> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdfe2[5] sdfd2[6] sdfc2[7] sdfb2[8] sdfa2[9] sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3]
522048 blocks [10/10] [UUUUUUUUUU]

md0 : active raid1 sdfe1[5] sdfd1[6] sdfc1[7] sdfb1[8] sdfa1[9] sda1[0] sdb1[1] sdc1[2] sdd1[4] sde1[3]
2489920 blocks [10/10] [UUUUUUUUUU]

unused devices: <none>


I'll be back when I've finished rebuilding the raid6 array as before. Should be around four days. If this fails, I guess I'll give it up as a bad job.
BTW, I have communicated with Synology tech support, and they seem to think it's fine that the raid6 array has 4 parity disks. They had no further suggestions. I suggested this was a bug in the firmware, and should be addressed. They had no comment.

[Derf Elot]

Hi Alex,

thanks for letting us know. I was curious as to what happened after the reboot at the end of your last post

It's really unfortunate that you have to go through so much pain. A RAID across all 10 disks sure would have been nice. Still hoping that last step is going to work out for you though. This way, the Dx5 is really not an extension, more of a very big RAID capable external drive, which is unfortunate.

Keep us updated on your last try.
Best,
Derf

[alex11561]

Well, Nothing I've tried has worked out. Expanding the boot & swap partitions was no good because the unit refused to recognise them after a reboot. I couldn't even add them back manually. There doesn't seem to be any hardware reason for it not to work, and it's too much work right now to dig into the boot & init scripts. Meanwhile the DX5 is just as Derf described it, a raid capable external drive. I'm certain Synology could fix this with a firmware revision, if only they had the will. Meanwhile, I have better things to do.