RAID6 or RAID10

Topics pertaining to volume setup, boot/shutdown, initialization, DSM update, HDD migration.
Forum rules
We've moved! Head over to Synology Community (community.synology.com) to meet up with our team and other Synology enthusiasts!
cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

RAID6 or RAID10

Unread post by cfuttrup » Sun Sep 14, 2014 3:12 pm

Hi there

Since DS415+ is coming up, I'm looking into buying this device and configure it with 4 x 4 TB WD RED disks.

Now I'm starting to think about which setup I should use, either RAID6 (with the capability of data scrubbing) or RAID10. Does RAID10 support scrubbing? I'd like the added write speed.

What about mdadm checkarray. Does anyone have experience with this?
http://www.thomas-krenn.com/en/wiki/Mdadm_checkarray

If this could work on a RAID10 array, this may satisfy my desire to check on the disks.

/Claus

SDirickson
Guru
Guru
Posts: 1462
Joined: Sun Mar 16, 2014 10:32 pm

Re: RAID6 or RAID10

Unread post by SDirickson » Sun Sep 14, 2014 8:52 pm

RAID 6 makes no sense for a 4-drive array; it is slower and (statistically) less safe in the presence of a failed drive. And rebuilds after a failed drive take longer and work the rest of the drives significantly harder, making the second point a much more important difference.
RAID is not Backup.

RAID has nothing to do with Backup.

RAID is about (and only about) continuity of service: your data remains available even in the presence of a drive failure.

"Backup" means a completely separate copy of the data in a completely separate system, preferably in another building (another town is even better). If your "backup" is sitting on the desk next to the "main", and your house floods or burns down, how useful is that "backup"?

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Mon Sep 15, 2014 6:30 pm

Hi SDirickson

Thanks for your opinion and help.

1) What's the number of disks when RAID6 starts to make sense?
2) Can you document the statistical side of it (an URL to a text is OK, I can read ... )

/Claus

SDirickson
Guru
Guru
Posts: 1462
Joined: Sun Mar 16, 2014 10:32 pm

Re: RAID6 or RAID10

Unread post by SDirickson » Tue Sep 16, 2014 12:16 am

If you poke around the Web, you'll find opinions across the entire spectrum, from "parity RAID is evil and must never be used" to "whaddya mean, I shouldn't run a RAID 5 array of 24 drives?"

For me
1) RAID 5 starts earning its keep at 4 drives, and is good up to 6
2) RAID 6 is good from 6 to 9 or maybe 10
Though, if 'enterprise' grade lower-URE-rate drives aren't used, I'd knock the upper limit down by one for each. Same for drive capacities over 4TB, because drives of that size haven't been around that long and (like any new technology/design) are likely to have lower reliability figures for a while, and because the URE-rate factor becomes a larger concern. Robin Harris' alarmist "death by RAID" prophecies haven't panned out, but the concern is real; it's just the magnitude that may need to be adjusted.

The statistics for RAID 10 vs RAID 5 are easy:
1) Assume that any drive is equally likely to fail during any chosen time interval; call that probability X
2a) For a 4-drive RAID 10 array with one failed drive, the failure of exactly one specific drive--the mirror of the failed drive--will cause the loss of the array. So the probability of losing the array during the rebuild is the probability of losing one drive during the rebuild: X
2b) For a 3-drive RAID 5 array with one failed drives, the failure of another drive--any drive--means the loss of the array. So the probability of losing the array during a rebuild is triple the probability of losing any drive during the rebuild interval, 3*X

Except that the probability of losing the array during these specific rebuild scenarios isn't 3 times as bad for RAID 5 as for RAID 10. Because
1) 'Rebuilding' the RAID 10 array simply means copying the failed drive's mirror to the replacement
2) Rebuilding the RAID 5 array mean reading every byte on every drive in the array (except the replacement, of course), calculating the data or parity value that should be on the replacement drive, and putting it there.

So a RAID 5 rebuild takes longer--a lot longer--than a RAID 10 rebuild, and works the existing drives--the ones you can't afford to lose--a lot harder. So the probability ratio of losing the array during a RAID 5 rebuild vs. a RAID 10 rebuild is a lot worse than 3X. Is it 10X? 100X? Don't know.

Also, keep in mind that that second "failure" doesn't need to be an actual drive failure; an unrecoverable read error during the rebuild downs the array. With a given error-probability-per-sector-read, the RAID 5 array is 3 times as likely to get a read error, because it's reading 3 times as much data.

"That's all well and good, Steve, except that the question is about RAID 6, not RAID 5".

True. What RAID 6 does is make it less likely that you'll end up in the situation where another drive failure causes the loss of the array. If one drive fails, you're now in RAID 5, and the calculations above apply if another drive fails before you get the first one replaced and the array rebuilt. But it takes a second drive failure to get there, so the probability of getting into the one-down-all-gone scenario is significantly less. The 'obvious' how-much-less factor is the drive non-failure probability, but that isn't correct. Because
1) RAID 6 rebuilds take a long time (like, days, or even weeks for large arrays)
2) RAID 6 rebuilds work the array a lot harder during the rebuild; the whole point of RAID is to maintain availability in the presence of a drive failure, so the read-every-sector-on-every-drive and write-every-sector-on-the-replacement-drive work is in addition to the normal workload on the array during the rebuild

the probability of some other drive failing during the rebuild is significantly higher than the probability that that drive would fail during 'normal' operations.

Even if you decide to make an emergency backup before replacing the failed drive, RAID 6 is more likely to not get there because--you guessed it--it takes longer and works the array harder.

So
RAID 6 keeps you out of one-down-all-gone territory much better than RAID 10*
But, once you're in that mode, RAID 6 is more likely to fall over the edge

The expected frequency of a RAID 6 "oops" is much lower than that of a RAID 10 "oops". But the chance that a RAID 6 "oops" will turn into an "aw sh**" is much higher.


* RAID 6's advantage is quite a bit less
1) With new drives, where the infant-mortality or bad-batch factors significantly increase the probability of overlapping failures
2) With old drives, where increased wear due to age does the same thing
In both cases, the longer rebuild time and higher workload of RAID 6 increase the probability of finding "that second bad drive in the batch" or "that second on-its-last-legs" drive, exactly when you can't afford to find it.
RAID is not Backup.

RAID has nothing to do with Backup.

RAID is about (and only about) continuity of service: your data remains available even in the presence of a drive failure.

"Backup" means a completely separate copy of the data in a completely separate system, preferably in another building (another town is even better). If your "backup" is sitting on the desk next to the "main", and your house floods or burns down, how useful is that "backup"?

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Tue Sep 16, 2014 6:25 pm

Hi SDirickson

Thanks for your input on RAID vs. number of disks. Although I don't understand the big picture and the statistics a bit dull to me, I'll listen to this advice.

Anyway, I like RAID10 for the same reasons you do, and I'm aware of the intensity required when rebuilding RAID5/6 arrays. Since I'm a private user, I don't need high-availability ... I could let the array rebuild without using the DiskStation (I could consider unplugging the ethernet cable during the rebuild).

If I can choose RAID10, then I'll do it.

The only issue is, can I scrub a Syno-DiskStation RAID10 array or can I use "checkarray" (see initial post)? I'd like a check feature to prevent bit rot. Drive failure is not my only concern.

/Claus

trauts
Sharp
Sharp
Posts: 192
Joined: Tue Nov 06, 2012 10:34 pm

Re: RAID6 or RAID10

Unread post by trauts » Tue Sep 16, 2014 11:32 pm

Hi Claus,
As I understand it, the problem with running a scrub on a raid 10 volume is that raid 10 writes are not buffered in the kernel (i.e. not atomic at this level) and as such it's possible to read what looks like mis-synched data off the drives, leading to a false fail (if the same read was made a little later it would be a pass as the data would then be physically written).
1512+ with DX513
3612xs

SDirickson
Guru
Guru
Posts: 1462
Joined: Sun Mar 16, 2014 10:32 pm

Re: RAID6 or RAID10

Unread post by SDirickson » Wed Sep 17, 2014 1:14 am

"Can you document the statistical side of it"->"the statistics a bit dull to me"

Ouch ;-)

Yes, the issue with a RAID 1/10 scrub is "what if the mirrors don't match?"

If one of the pair fails a read, and the other succeeds, the RAID controller or OS will write the 'good' data again on the 'bad' mirror, which will either 'fix' a transient read error or cause the drive to relocate away from a sector/track that's actually bad and write it there; in either case, as far as everyone is concerned, the data was successfully written, and the mirror is 'fixed'. That's why you'll usually see recommendations that yes, you should scrub your mirrored arrays.

But what happens if both drives report a successful read, but the returned data from one drive doesn't match that returned from the other half of the pair? If it's a transient mismatch like the one mentioned, where the read happens after the write to one drive is complete but the other isn't, the OS should be (which doesn't mean that it is) smart enough to recognize the situation and retry the match. But what if the data physically on the drives is different (cosmic ray, quantum bit flip, whatever)? Which one is "right"? The good news is that actual corruption of at-rest data on the media is (despite what the ZFS/BTRFS people would have you believe) *really* rare on non-defective drives.

Fortunately, my next RAID setup will be on an Adaptec 7805, so the transient-false-mismatch error path won't be an issue.
RAID is not Backup.

RAID has nothing to do with Backup.

RAID is about (and only about) continuity of service: your data remains available even in the presence of a drive failure.

"Backup" means a completely separate copy of the data in a completely separate system, preferably in another building (another town is even better). If your "backup" is sitting on the desk next to the "main", and your house floods or burns down, how useful is that "backup"?

MrFixit
Sharp
Sharp
Posts: 180
Joined: Thu Jul 11, 2013 12:57 pm

Re: RAID6 or RAID10

Unread post by MrFixit » Wed Sep 17, 2014 12:09 pm

It seems to me if you are concerned about data loss you really ought to consider how you are going to (regularly) back up your data, as RAID is primarily about data availability and not data backup.

Personally, for my home DS1813+, I went RAID 6 (8 3TB drives, same size, make and model). I scrub monthly which takes about two days (the array is 57% full). I do daily Quick SMART tests and Weekly Full SMART tests and check these regularly. In 12 months I have had two separate drive failures - both drives were replaced under RMA - the RAID 6 rebuilds went fine (I didn't time them, sorry, but ~two days I recall). I keep a spare drive handy (same size, make and model) in case of drive failure. I also backup (daily) all important, can't be replaced, data (~4TB of photographs plus ~1TB of personal stuff) on another RAID 5 NAS and several disks in a PC. I use Retrospect Desktop (running on a Windows 8 PC) as it does incremental backups and works with Windows, OSX and Linux (I use Retrospect clients on each platform). I run two separate backups of all important data. But then, of course, I am paranoid about data loss :D . YMMV.
Synology NAS Model: DS1813+ (upgraded to 4GB)
DSM Firmware: Version: DSM 6.2-23739
Hard Disks: 8 Toshiba 3TB DT01ACA300 in RAID 6, plus a cold spare

Synology NAS Model: DS1817+ (8GB)
DSM Firmware: Version: 6.2-23739
Hard Disks: 5 Seagate 8TB Ironwolf in RAID 6, 1 Seagate 8TB Ironwolf in Basic

UPS: APC Smart-UPS 1000 (SMT1000I) with AP9630 Management Card and SNMP UPS mode

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Wed Sep 17, 2014 6:41 pm

MrFixit wrote:It seems to me if you are concerned about data loss you really ought to consider how you are going to (regularly) back up your data, as RAID is primarily about data availability and not data backup.
Hi MrFixit :D

I'm not that "paraniod" - but actually this will be my second active diskstation and it will mirror another one as backup.

The twist is the old DS413 is located in one country, while I am working in another country, so I need both to run and be used, while somehow I also need to setup that they backup each other ... cross-country so to say :D

The old DS413 was initially with 2 x 3 TB disks, RAID1 on volume 1, then one year later expanded with another pair of 3 TB, RAID1 on volume 2 (reserved for video).

I'd still like the new diskstation setup in a smarter way, with just one volume... and I'd like to be able to check my data on it regarding the quality. This requires parity - so that parity checks can be performed, or maybe as initially suggested mdadmin checkarray ... (I don't know much about it)

RAID10 is striped and mirrrored, so it's not an array with parity either, but I like the speed. I wonder if mdadmin checkarray can be used on a RAID10 and if anyone here could tell me how to do the setup. Alternatively have another suggestion what I should do.

/Claus

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Wed Sep 17, 2014 6:45 pm

SDirickson wrote:"Can you document the statistical side of it"->"the statistics a bit dull to me"

Ouch ;-)
I meant dull = matte, the correct word would be unclear, sorry about that. :oops:

/Claus

MrFixit
Sharp
Sharp
Posts: 180
Joined: Thu Jul 11, 2013 12:57 pm

Re: RAID6 or RAID10

Unread post by MrFixit » Thu Sep 18, 2014 12:47 am

cfuttrup wrote:RAID10 is striped and mirrrored, so it's not an array with parity either, but I like the speed. I wonder if mdadmin checkarray can be used on a RAID10 and if anyone here could tell me how to do the setup. Alternatively have another suggestion what I should do.
Will RAID 10 - given that access is over Gigabit Ethernet - be really that much faster than RAID 5 or RAID 6?
Synology NAS Model: DS1813+ (upgraded to 4GB)
DSM Firmware: Version: DSM 6.2-23739
Hard Disks: 8 Toshiba 3TB DT01ACA300 in RAID 6, plus a cold spare

Synology NAS Model: DS1817+ (8GB)
DSM Firmware: Version: 6.2-23739
Hard Disks: 5 Seagate 8TB Ironwolf in RAID 6, 1 Seagate 8TB Ironwolf in Basic

UPS: APC Smart-UPS 1000 (SMT1000I) with AP9630 Management Card and SNMP UPS mode

SDirickson
Guru
Guru
Posts: 1462
Joined: Sun Mar 16, 2014 10:32 pm

Re: RAID6 or RAID10

Unread post by SDirickson » Thu Sep 18, 2014 5:04 am

There's one other factor, not that relevant to a 4-drive box, but something to know: In general, Linux RAID 10 arrays can't be expanded--what you had when you created the array is what you get. mdadm is (was?) getting some capabilities that make it look like the RAID 10 array was expanded, but I doubt that Synology's version has those or will get them. DSM definitely won't even talk to you about expanding a RAID 10 array. There are hardware RAID controllers that can add drives to a RAID 10 array, but software RAID generally can't, and Synology definitely can't.
RAID is not Backup.

RAID has nothing to do with Backup.

RAID is about (and only about) continuity of service: your data remains available even in the presence of a drive failure.

"Backup" means a completely separate copy of the data in a completely separate system, preferably in another building (another town is even better). If your "backup" is sitting on the desk next to the "main", and your house floods or burns down, how useful is that "backup"?

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Thu Sep 18, 2014 3:56 pm

MrFixit wrote:Will RAID 10 - given that access is over Gigabit Ethernet - be really that much faster than RAID 5 or RAID 6?
I think Synology knows this is a bottle neck. That's why newer machines have two Ethernet ports.

/Claus

MrFixit
Sharp
Sharp
Posts: 180
Joined: Thu Jul 11, 2013 12:57 pm

Re: RAID6 or RAID10

Unread post by MrFixit » Fri Sep 19, 2014 7:16 pm

cfuttrup wrote:
MrFixit wrote:Will RAID 10 - given that access is over Gigabit Ethernet - be really that much faster than RAID 5 or RAID 6?
I think Synology knows this is a bottle neck. That's why newer machines have two Ethernet ports.

/Claus
True. But you only get a benefit in combining two Ethernet ports (e.g. using LACP) if you have multiple workloads. A single user will not in general see any additional bandwidth unless running two separate disk intensive processes. For LACP you also need a compliant smart switch.
Synology NAS Model: DS1813+ (upgraded to 4GB)
DSM Firmware: Version: DSM 6.2-23739
Hard Disks: 8 Toshiba 3TB DT01ACA300 in RAID 6, plus a cold spare

Synology NAS Model: DS1817+ (8GB)
DSM Firmware: Version: 6.2-23739
Hard Disks: 5 Seagate 8TB Ironwolf in RAID 6, 1 Seagate 8TB Ironwolf in Basic

UPS: APC Smart-UPS 1000 (SMT1000I) with AP9630 Management Card and SNMP UPS mode

cfuttrup
Virtuoso
Virtuoso
Posts: 1346
Joined: Fri May 09, 2008 8:01 pm
Location: Denmark
Contact:

Re: RAID6 or RAID10

Unread post by cfuttrup » Fri Sep 19, 2014 8:26 pm

MrFixit wrote:True. But you only get a benefit in combining two Ethernet ports (e.g. using LACP) if you have multiple workloads. A single user will not in general see any additional bandwidth unless running two separate disk intensive processes. For LACP you also need a compliant smart switch.
A single user probably doesn't have dual-lan ports on his computer anyway. IMHO high speed connection only makes sense with multiple workloads from several users. In my situation that might be quite comon (but probably only one which is disk intensive, sometimes both my spouse and I may do something).

... but then again, the NAS itself can be responsible for some disk intensive work. High activity in the RAID doesn't have to see the ethernet bottleneck.

/Claus

Locked

Return to “Installation, Configuration, Migration, Expansion”