Hey guys,
I need your help!
My Setup:
4 Disks: Western Digital Red Series (WD40EFRX) from February 2016 running in an raid 5 (Slot 1,2,3,4); Slot 5 is empty; DS515+
- Let me explain what happend and what I did in steps:
- On 17.11.2017 (early in the morning) my DS did an automatic update (I have had set the box to do automatic updates).
- On 18.11.2017 at around 00:30 I wanted to access my data and could not reach the web interface, nor the mountpoint on my windows and also couldn't ping it. So I looked at the DS and saw that the blue led light was flashing. If I remeber correctly every disk led was dead. It looked similiar to this one here: https://forum.synology.com/enu/viewtopi ... 7&t=128497 - So I did (sadly, should have waited...) a restart via the power button: Hold it for a few seconds until the beep. The DiskStation did a normal shutdown with a flashing blue led. After that I started it again via Power Button.
- After the start had completed, the status led was orange and all disk leds were on (no beeping, just the normal one, which says that the station has booted). I checked to access web, mountpoint, but could not reach my data. So I checked with Synology Assistant and found it with issue "Configuration lost" and an dhcp ip adress.
- I searched the web and it said that it should not be a problem to do a reinstall. My data would not be touched. Only the system partitions will be touched. So I did a connect and after that a reinstall (web showed reinstall, if I remember correctly) with manual downloaded .pat file (DSM_DS1515+_15217.pat). If the data would be wiped I should get a red text, which wasn't shown. So the reinstall started and failed between 40% and 50% with "failed formatting system partition [35]" (I don't remember the correct wording, but it was a "[35]" there. After that the nas showed me to install, not to reinstall.
- So I did a manual shutdown and start via power key, like in step 2 explained again. This time it came up with an orange led and a constant beeping (beep, beep, beep...). Ironically the station came up now with my old configuration (with the manual ip-adress i had set for example). But all my installed packages were gone. It showed me that volume1 crashed. volume1 is now shown with disk3 and disk4 but missing disk1 and disk2. Also it is showing no bytes. On the disk side every disk is on "normal", but has the error "system partition failed" (In German: "Systempartitionierung fehlgeschlagen). It looks like that:
- So now I can't do a reapair of the raid 5, because of 2 lost disks, my raid 5 is gone (well at least i thought that
). Now I did shut it down and gone on research. I will explain that down below. My goal is to get the data back. It doesn't matter if I have to recreate settings and the volume later. I just want to get my data back if it's possible...
Yes ist's correct: "I/O-Error on disk 3 on NAS01" means the disk in the phisycally third slot. "I/O-Error on disk 5 on NAS01" would be the phyisically fifth slot and "I/O-Error on disk 1 on NAS01" would be the first phisycally slot. Same goes for "disk[3]", "disk[5]" or "disk[1]" and so on. Don't mess this up with the "raid slot" mentioned down below, because mdadm counts from 0. So in my example, I have 4 disks, which are physically mentioned with 1,2,3,4 and in mdadm (the tool to manage software raids in Linux), they are mentioned as 0,1,2,3 - just keep that in mind! (Btw: NAS01 is just the name of the nas-system - so on yours it will most likely be another name of course).
I checked outputs from various files in /etc/space as well, so I did not change the disk order by accident (I paniced a little at that day...). I also checked some logs. It seems to be, that on a boot the diskstation tries to stop the /dev/md2 and wants to do a an force assemble, but fails because it can't stop the /dev/md2 (Why? I don't know...)
As I said, panic is the worst thing... Every step you do, write 'em down and make photos and videos, so you can remember correctly! For the peace of your heart, if you changed the disk order by accident, it's not that big issue at all. I tried this on a test case as well and I could get everything back. Mdadm is really robust on such things. Just keep reading.
I did run a few mdadm commands to check what is going on:
root@NAS01:/dev# cat /proc/mdstat
This shows the raid information. As you can see there are three raids: md0, md1 and md2. md0 and md1 are just system partitions in Synology Systems (md0 is normally the system partition and md1 is the swap partition). The following numbers (md2, md3 and so on) contain your created raids and therefore your actual data. Note that md0, md1, md2, md3 and so on are just names for the raids. It could be that md0 is your data and md2 your system partition for example. It's not likely like that in Synology systems, but you'll never know if you don't check that. Again: Test as much as you can and get as much information before you do anything! You can get to know which raid contains your data in some ways, I will explain on next post. However, you can see U and _ in the output. A U means this device is still in your raid and up. A _ means this device is missing in your raid (This has not to mean that your disk is dead!). Above of this blocks you can see which devices are in the raid. For example on md2 you see that sdc3 (/dev/sdc3) and sdd3 (/dev/sdd3) are still present on slot 2 and 3. But slot 0 and slot 1 are missing, you discover that because of the leading two missing _ and because the raid will start to count on 0, when it is created. Note here, that the slot number of the raid doesn't have to be your acutally physical disk order in your device! So the slot 0 in the raid doesn't have to be the first(1) physical disk in your device. It could also be the second (2) or fifth or whatever. Normally Synology devices use phyiscal disk slot 1 for raid slot 0, physical disk slot 2 for raid slot 1 and so on and also names the devices like that (first disk is /dev/sda, second disk is dev/sdb etc.), but just keep in mind, that it isn't always like this! Another thing you can see, is that md0 and md1 have five disks (UU_ _ _) - but I only have 4 disks in my case
This is shown, because Synology creates the system and swap partition on all disks, no matter if they exist or not. It's done like this to expand this raids on all new disks you put in your enclosure, to provide always a working DSM, regardless which disk fails. Note also that md2 is raid5 and md1 and md0 are raid1. As I did had an raid5 with 4 disks, I know at this point, that md2 is the raid which contains my data and has missing two devicdes (0 and 1) - I assume that this are /dev/sda3 and /dev/sdb3, but at this point I still can't be shure! So let's check it out
Code: Select all
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdc3[2] sdd3[3]
11706589632 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/2] [__UU]
md1 : active raid1 sdc2[0] sdd2[1]
2097088 blocks [5/2] [UU___]
md0 : active raid1 sda1[0] sdb1[1]
2490176 blocks [5/2] [UU___]
unused devices: <none>
root@NAS01:/dev# mdadm --detail /dev/md0
With this commands you can get more details about the existing raids. You can see here, that as I mentioned before, indeed this raid contains 5 disks, although I only have 4 disks. The funny thing is that the synology system raids are always degraded (see State at md0 and md1), if you don't use every physically slot
Code: Select all
/dev/md0:
Version : 0.90
Creation Time : Sat Nov 18 00:20:00 2017
Raid Level : raid1
Array Size : 2490176 (2.37 GiB 2.55 GB)
Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Nov 23 18:30:08 2017
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 5c6bb801:f95aa80d:3017a5a8:c86610be
Events : 0.2264
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 0 0 2 removed
3 0 0 3 removed
4 0 0 4 removed
root@NAS01:/dev# mdadm --detail /dev/md1
Code: Select all
/dev/md1:
Version : 0.90
Creation Time : Sat Nov 18 01:11:06 2017
Raid Level : raid1
Array Size : 2097088 (2048.28 MiB 2147.42 MB)
Used Dev Size : 2097088 (2048.28 MiB 2147.42 MB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Thu Nov 23 18:27:25 2017
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : d9a4606d:3e753320:8467a421:84d4f733 (local to host NAS01)
Events : 0.30
Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 0 0 2 removed
3 0 0 3 removed
4 0 0 4 removed
root@NAS01:/dev# mdadm --detail /dev/md2
So let me explain a bit more with the output of md2, as for md0 and md1 we don't really care. What you can see is the devices the raid normally contains (Raid Devices : 4) and the total devices which are in the raid at the moment (Total Devices : 2). You can compare this with the output of the previous command (cat /proc/mdstat). Another important thing is that on Persistence is shown: Superblock is persistent. However if not, there are also possibilities to solve from this problem, so don't lose hope if so and keep reading. Basically Linux raid reserves a bit of space (called a superblock) on each component device. This space holds metadata about the RAID device and allows correct assembly of the array. As it is an raid 5 with two missing disks, the state of the raid is in fact FAILED (see State). Also you see when the raid was created,the version of the Suberblock format etc. If you need more information take a look here: https://github.com/tinganho/linux-kerne ... ion/md.txt
Code: Select all
/dev/md2:
Version : 1.2
Creation Time : Sat May 21 17:13:58 2016
Raid Level : raid5
Array Size : 11706589632 (11164.27 GiB 11987.55 GB)
Used Dev Size : 3902196544 (3721.42 GiB 3995.85 GB)
Raid Devices : 4
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Nov 23 18:27:43 2017
State : clean, FAILED
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : NAS01:2 (local to host NAS01)
UUID : eadf7c1e:9c3e4ac3:3f35ab65:118a3497
Events : 156
Number Major Minor RaidDevice State
0 0 0 0 removed
1 0 0 1 removed
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
root@NAS01:/dev# fdisk -l
So now it get's interesting, because here you can see what actually happend to my raid. This command lists all partitions of all disks. Let's take /dev/sdd for example. On /dev/sdd you can se three existing partitions: /dev/sdd1, /dev/sdd2 and /dev/sdd3 with their size. So let's be honest, it's really obvious, that /dev/sdd3 is part of my raid 5, because of the size (3.6 TB * 4 disks = 14,8 - 3.6 (minus one disk for raid 5) = 10.8, which is my volume size). But don't find your raid because it seems to be obvious. Just compare the output with the output of the commands cat /proc/mdstat and mdadm --detail /dev/mdx. From this commands we see, which partitions are in the raids:
- For md0 it is /dev/sda1 and /dev/sdb1
- For md1 it is /dev/sdc2 and /dev/sdd2
- For md2 it is /dev/sdc3 and /dev/sdd3
If we compare this with the output here, we can see that on /dev/sda there is missing the partition /dev/sda2 and on /dev/sdb there is missing the partition /dev/sdb2 and therefore they both are missing in md1. Another thing you should see, is what I mentioned above: That Synology normally works like this /dev/sdx1 is in md0, /dev/sdx2 is in md1 and /dev/sdx3 is in md2 - or if you have more raids, /dev/sdx4 would be in md3 and so on. As I said, you can never be shure but it looks clearly like that. So let's take this as presume fact to go on. With this, it explains, why md1 has two missing disks, which should be there: /dev/sda2 and /dev/sdb2. They just don't exist anymore. So the error "system partition failed", which occured on reinstall was at least correct - in fact two parts of the swap raid are gone
I still don't know why this happens, but I can only imagine that the CPU has something to do with it... Recently Intel admited, that the Intel C2000 Series has Failures. While other resellers took back products, which have guarantee, Synology just extended the support of the affected devices - see Announcement:https://www.synology.com/en-global/comp ... %20Updatet. I had enabled the automatic updates on my case and could see in the logs, that a day before my case wasn't accessible, an update was made. Either this failed and deleted the two partitions, or my reinstall... I still don't know, but will contact Synology for a replacement. My trust in this case is gone, as this should not happen and I also tested my first disk (which is in fact /dev/sda) with an extended S.M.A.R.T. test, which completed with no errors (I also analyzed the raw values).Ok back to topic. If all is OK, you should see, that your partitions of your raid which crashed (failed), are still there. As we presume, that everything with a 3 at the end belongs to /dev/md2 and we find them all in the ouptut (/dev/sda3, /dev/sdb3, /dev/sdc3 and /dev/sda3), we know that the partitions of the raid are still there - so my reinstall didn't wipe my data partitions (As I said before, make photos. I paniced and wasn't really shure which errors were shown and was afraid, the reinstall wiped all
; This step was a rlly good point to take a deep breath ^^).
Code: Select all
Disk /dev/sdd: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2F83CF67-C568-4D90-8884-75ACFBA7954D
Device Start End Sectors Size Type
/dev/sdd1 2048 4982527 4980480 2.4G Linux RAID
/dev/sdd2 4982528 9176831 4194304 2G Linux RAID
/dev/sdd3 9437184 7813832351 7804395168 3.6T Linux RAID
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 6E557084-F62E-4FB3-AC85-1A88A1D150DE
Device Start End Sectors Size Type
/dev/sda1 2048 4982527 4980480 2.4G Linux RAID
/dev/sda3 9437184 7813832351 7804395168 3.6T Linux RAID
Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2BAD3013-6B28-4528-8EE9-36FF2DEC9B68
Device Start End Sectors Size Type
/dev/sdb1 2048 4982527 4980480 2.4G Linux RAID
/dev/sdb3 9437184 7813832351 7804395168 3.6T Linux RAID
Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 3A38788A-4BE1-4CAC-BBE1-A135FF1AFF6C
Device Start End Sectors Size Type
/dev/sdc1 2048 4982527 4980480 2.4G Linux RAID
/dev/sdc2 4982528 9176831 4194304 2G Linux RAID
/dev/sdc3 9437184 7813832351 7804395168 3.6T Linux RAID
Disk /dev/md0: 2.4 GiB, 2549940224 bytes, 4980352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/md1: 2 GiB, 2147418112 bytes, 4194176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/zram0: 2.4 GiB, 2522873856 bytes, 615936 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/zram1: 2.4 GiB, 2522873856 bytes, 615936 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/zram2: 2.4 GiB, 2522873856 bytes, 615936 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/zram3: 2.4 GiB, 2522873856 bytes, 615936 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/synoboot: 120 MiB, 125829120 bytes, 245760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfd900657
Device Boot Start End Sectors Size Id Type
/dev/synoboot1 * 63 32129 32067 15.7M 83 Linux
/dev/synoboot2 32130 224909 192780 94.1M 83 Linux
root@NAS01:/dev# mdadm --examine /dev/sd[abcd]1
Code: Select all
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : 5c6bb801:f95aa80d:3017a5a8:c86610be
Creation Time : Sat Nov 18 00:20:00 2017
Raid Level : raid1
Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
Array Size : 2490176 (2.37 GiB 2.55 GB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 0
Update Time : Thu Nov 23 18:29:58 2017
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : abbbeb63 - correct
Events : 2258
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : 5c6bb801:f95aa80d:3017a5a8:c86610be
Creation Time : Sat Nov 18 00:20:00 2017
Raid Level : raid1
Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
Array Size : 2490176 (2.37 GiB 2.55 GB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 0
Update Time : Thu Nov 23 18:29:58 2017
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : abbbeb75 - correct
Events : 2258
Number Major Minor RaidDevice State
this 1 8 17 1 active sync /dev/sdb1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : c07f7981:9727785b:3017a5a8:c86610be
Creation Time : Sat Nov 18 00:28:02 2017
Raid Level : raid1
Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
Array Size : 2490176 (2.37 GiB 2.55 GB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 0
Update Time : Sat Nov 18 00:43:38 2017
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : ad94f06a - correct
Events : 4936
Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1
0 0 8 33 0 active sync /dev/sdc1
1 1 8 49 1 active sync /dev/sdd1
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
/dev/sdd1:
Magic : a92b4efc
Version : 0.90.00
UUID : c07f7981:9727785b:3017a5a8:c86610be
Creation Time : Sat Nov 18 00:28:02 2017
Raid Level : raid1
Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
Array Size : 2490176 (2.37 GiB 2.55 GB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 0
Update Time : Sat Nov 18 00:43:40 2017
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : ad94f07f - correct
Events : 4937
Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1
0 0 8 33 0 active sync /dev/sdc1
1 1 8 49 1 active sync /dev/sdd1
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
root@NAS01:/dev# mdadm --examine /dev/sd[abcd]2
Code: Select all
/dev/sdc2:
Magic : a92b4efc
Version : 0.90.00
UUID : d9a4606d:3e753320:8467a421:84d4f733 (local to host NAS01)
Creation Time : Sat Nov 18 01:11:06 2017
Raid Level : raid1
Used Dev Size : 2097088 (2048.28 MiB 2147.42 MB)
Array Size : 2097088 (2048.28 MiB 2147.42 MB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 1
Update Time : Thu Nov 23 18:27:25 2017
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : 7ec7fead - correct
Events : 30
Number Major Minor RaidDevice State
this 0 8 34 0 active sync /dev/sdc2
0 0 8 34 0 active sync /dev/sdc2
1 1 8 50 1 active sync /dev/sdd2
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
/dev/sdd2:
Magic : a92b4efc
Version : 0.90.00
UUID : d9a4606d:3e753320:8467a421:84d4f733 (local to host NAS01)
Creation Time : Sat Nov 18 01:11:06 2017
Raid Level : raid1
Used Dev Size : 2097088 (2048.28 MiB 2147.42 MB)
Array Size : 2097088 (2048.28 MiB 2147.42 MB)
Raid Devices : 5
Total Devices : 2
Preferred Minor : 1
Update Time : Thu Nov 23 18:27:25 2017
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 3
Spare Devices : 0
Checksum : 7ec7febf - correct
Events : 30
Number Major Minor RaidDevice State
this 1 8 50 1 active sync /dev/sdd2
0 0 8 34 0 active sync /dev/sdc2
1 1 8 50 1 active sync /dev/sdd2
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
root@NAS01:/dev# mdadm --examine /dev/sd[abcd]3
Ok just a few words to this comamnds. The ouput shows information of the partitions /dev/sda3, /dev/sdb3, /dev/sdc3 and /dev/sdd3. The commands before equivalent to the partitons with a 1 and 2 at the end. The key information here is the Device UUID and the Events (Event Count). The UUID (Universally Unique Identifier) is a 128-bit number which is randomly generated to identify the device here. The Event Count, basically counts every major event what happens to the whole raid. If you boot up your nas, mdadm "reassembles" your raid and at this process the events of all containing disks will be counted up (normally it is current events +1) - More information: https://raid.wiki.kernel.org/index.php/Event. If one disk differs by the others, mdadm will not automatically inlcude this disks in the raid. And this is exactly what is happening here on every boot. As you can see, the events of /dev/sda3 and /dev/sdb3 are lower than the others. So mdadm doesn't reassemble them on boot, which leads to the crashed volume. Why this works like that? Imagine one disk of four disks had failed and was thrown out of the raid 5. As it is a raid 5 with one lost disk then, you could still access and change the data on the raid, because it is just degraded. If you now change a lot of data and would try to assemble the lost disk later on, it will possible lead to data loss, because the data on the thrown out disk is outdated... This is why in this case you do a rebuild and not an assemble.
- /dev/sda3 -> 144
- /dev/sdb3 -> 144
- /dev/sdc3 -> 156
- /dev/sdd3 -> 156
Note also, that the raid level of the partition is shown. As we know, that md0 and md1 are raid 1 and md2 is raid 5, this is another indication, that /dev/sda3, /dev/sdb3, /dev/sdc3 and /dev/sdd3 is in fact my crashed raid 5.
Code: Select all
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : eadf7c1e:9c3e4ac3:3f35ab65:118a3497
Name : NAS01:2 (local to host NAS01)
Creation Time : Sat May 21 17:13:58 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 23413179264 (11164.27 GiB 11987.55 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5a742e05:cd4dc858:cc89ff46:afe66cbc
Update Time : Fri Nov 17 05:48:35 2017
Checksum : 6caaf75c - correct
Events : 144
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : eadf7c1e:9c3e4ac3:3f35ab65:118a3497
Name : NAS01:2 (local to host NAS01)
Creation Time : Sat May 21 17:13:58 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 23413179264 (11164.27 GiB 11987.55 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 808a7ce1:02b4f1bd:e1e366ff:0a4cd1a8
Update Time : Fri Nov 17 05:48:35 2017
Checksum : 52ee333a - correct
Events : 144
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : eadf7c1e:9c3e4ac3:3f35ab65:118a3497
Name : NAS01:2 (local to host NAS01)
Creation Time : Sat May 21 17:13:58 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 23413179264 (11164.27 GiB 11987.55 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 91949959:f4bafdcb:5b756116:9ae3f29e
Update Time : Thu Nov 23 18:27:43 2017
Checksum : e63a082d - correct
Events : 156
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : eadf7c1e:9c3e4ac3:3f35ab65:118a3497
Name : NAS01:2 (local to host NAS01)
Creation Time : Sat May 21 17:13:58 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7804393120 (3721.42 GiB 3995.85 GB)
Array Size : 23413179264 (11164.27 GiB 11987.55 GB)
Used Dev Size : 7804393088 (3721.42 GiB 3995.85 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1626944f:cc7b07bc:a62141ac:3024989c
Update Time : Thu Nov 23 18:27:43 2017
Checksum : 5fc3476d - correct
Events : 156
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)
I read about how to get the disks back in to the arrray and thought about running this commands:
mdadm --stop /dev/md2
mdadm --assemble --run /dev/md2 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 -v
My problem is, that I don't know if the order of the disks depends in the last command? Because I don't know if /dev/sda3 or /dev/sdb3 should be on first or second position. Furthermore I don't know if the array is in case build of /dev/sda3 ; /dev/sdb3? I think it can only be this two? I am also scared now if the reinstall process just wiped my data partitions on disk1 and disk2. Could that be possible? Or is this not possible because there exists the /dev/sda3 and /dev/sdb3 on this disks? So If I would run with force I would kill my last hope which would be a data rescue comapany... (If I am right)?
Yes this is really a problem. You should know the correct order! Well mdadm is very robust as I said and when you reassemble the raid it will check which disk was on which logically slot in the raid, no matter where the disk is now and was before physically (CAUTION! THAT ONLY APPLIES TO AN --ASSEMBLE). But I would not recommend to just count on the robustness and forget the bain. I tested it twice on another case and it went well, but a bird in the hand is worth two in the bush. So check the correct disk order before - I will explain how, in the next post.
More problematic is, that the Event Count differs:
/dev/sda3 144
/dev/sdb3 144
/dev/sdc3 156
/dev/sdd3 156
I heard if I run it without --force , then it will fail because of "possible out-of-date" issue. But I think the Event Count differs, because of the boots I did (did like 4, after step 6 I think). Also I read, that running this command with --force can be dangerous..
Command with force:
mdadm --assemble --force --run /dev/md2 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 -v
Yes this is true, it would fail with "possible out-of-date" error on /dev/sda3 and /dev/sdb3 and would start the raid as crashed with /dev/sdc3 & dev/sdd3. It would not hurt, but also nothing would be achieved - this is exactly what happened on every boot. By the way, I assuemed correctly. The event count, counted +2 on /dev/sdc3 and /dev/sdd3 on every boot - this is why the event count differs on my partitions.
So what can I do at best? I don't think my disks failed, because a double fail at the same time is very unlikely. A little bit later yes, but not on the same time on reboot... But if they have a failure, I could break even more if I'm doing stuff... So I'm asking me if I should clone one of the failed disks, or every disk...
I will explain you in the next post, what I did and what you should do.
I appreciate any help.
Thank you in advance!
So there we are, I hope you could follow me, if not just ask
. I think it is really important, that you know what you are doing and not unthinkingly run commands, which are recommended on the internet. This is why I write in such detail. So research the web and take the time you need. I know it is hard, if there is valuable data on the raid and not knowing of getting it back. But as I said in the beginning: KEEP CALM! Every step you do could be the step which leads to full data loss. If you do nothing, you can't destroy anything. So think and research before running the commands. Make shure you understood what I or others did and trying to explain before going to act. And if you have another case to test with, even better. Ok so let's move on and let me explain in steps what I did to get all back.
here, because I would not stake my life on that! I tested it with my test case twice with different mixed orders and it worked, but I can't promise that it would work with another version of mdadm or DSM. So it's better to get back the correct physical disk order if you mixed it, or even better never mix the order and number your disks (Step one 
I can only explain the behaviour to me, that the disk will be thrown out of all raids when something happens to one array (e.g. /dev/md1), because it is assumend something abnormal happened.
(fire or water can be dangerous... 
) 