[FIXED]crashed volume after power outage - disks appear fine

Questions pertaining to Power settings, Auto Block, Permissions, User Quotas and Email alerts may be posted here.
Forum rules
We've moved! Head over to Synology Community (community.synology.com) to meet up with our team and other Synology enthusiasts!
hawkinsst7
I'm New!
I'm New!
Posts: 2
Joined: Sat May 23, 2015 4:19 am

[FIXED]crashed volume after power outage - disks appear fine

Unread post by hawkinsst7 » Tue May 26, 2015 3:22 am

So there was a power outage here, and when my DS413 came back alive, it was beeping at me.


I've cut off the beeping, and found that Storage manager reports that Volume 1 has crashed.

I'm technical, comfortable with the command line, but i'm really new to RAID, LVM and Volume terminology. I'm realizing I need to learn, but I'd rather not have my crash course with all my data at stake!

So here's some info I've dug up that can possibly help, but I really don't know what steps to take going forward.

Its a DS413 running 5.2
4 disk RAID (SHR with 1 disk fault tolerance)
SMART status reports everything is OK with all 4 disks, and an extended test turned up no problems.

Code: Select all

DiskStation> vgs vg1 -v
    Using volume group(s) on command line
    Finding volume group "vg1"
  VG   Attr   Ext   #PV #LV #SN VSize VFree VG UUID
  vg1  wz--n- 4.00M   2   1   0 8.17T    0  xFF48s-dGxM-aSLd-Xai6-B9dD-u32X-z4fMy6

Code: Select all



DiskStation> df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.4G  712M  1.6G  32% /
/tmp            506M  760K  505M   1% /tmp
/run            506M  2.5M  503M   1% /run
/dev/shm        506M     0  506M   0% /dev/shm

Code: Select all


DiskStation> mdadm --query /dev/md2
/dev/md2: 5575.51GiB raid5 4 devices, 0 spares. Use mdadm --detail for more detail.
DiskStation> mdadm --query /dev/md3
/dev/md3: 2794.49GiB raid5 4 devices, 0 spares. Use mdadm --detail for more detail.
DiskStation> mdadm --query /dev/md1
/dev/md1: 2047.94MiB raid1 4 devices, 0 spares. Use mdadm --detail for more detail.
DiskStation> mdadm --query /dev/md0
/dev/md0: 2.37GiB raid1 4 devices, 0 spares. Use mdadm --detail for more detail.

Code: Select all


DiskStation> mdadm --detail /dev/md3
/dev/md3:
        Version : 1.1
  Creation Time : Tue Mar 26 04:00:11 2013
     Raid Level : raid5
     Array Size : 2930231424 (2794.49 GiB 3000.56 GB)
  Used Dev Size : 976743808 (931.50 GiB 1000.19 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Mon May 25 20:14:20 2015
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : DiskStation:3  (local to host DiskStation)
           UUID : ca1018f3:e073bf56:75248a9f:134cae50
         Events : 392399

    Number   Major   Minor   RaidDevice State
       0       8       54        0      active sync   /dev/sdd6
       1       8       38        1      active sync   /dev/sdc6
       2       8        6        2      active sync   /dev/sda6
       3       8       22        3      active sync   /dev/sdb6

Code: Select all




DiskStation> mdadm --detail /dev/md2
/dev/md2:
        Version : 1.1
  Creation Time : Sat Mar 23 20:17:16 2013
     Raid Level : raid5
     Array Size : 5846342016 (5575.51 GiB 5986.65 GB)
  Used Dev Size : 1948780672 (1858.50 GiB 1995.55 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Mon May 25 20:14:20 2015
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : DiskStation:2  (local to host DiskStation)
           UUID : dbd840f9:dd275cdc:0c7c11e5:41e0a854
         Events : 11443196

    Number   Major   Minor   RaidDevice State
       5       8       21        0      active sync   /dev/sdb5
       4       8        5        1      active sync   /dev/sda5
       3       8       53        2      active sync   /dev/sdd5
       2       8       37        3      active sync   /dev/sdc5

Code: Select all


DiskStation> mdadm --assemble --scan
mdadm: No arrays found in config file or automatically

Code: Select all


DiskStation> mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/dm-0
mdadm: cannot open device /dev/md3: Device or resource busy
mdadm: cannot open device /dev/md2: Device or resource busy
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: no recogniseable superblock on /dev/mtdblock5
mdadm: no recogniseable superblock on /dev/mtdblock4
mdadm: no recogniseable superblock on /dev/mtdblock3
mdadm: no recogniseable superblock on /dev/mtdblock2
mdadm: no recogniseable superblock on /dev/mtdblock1
mdadm: no recogniseable superblock on /dev/mtdblock0
mdadm: cannot open device /dev/sdd6: Device or resource busy
mdadm: cannot open device /dev/sdd5: Device or resource busy
mdadm: cannot open device /dev/sdd2: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: cannot open device /dev/sdc5: Device or resource busy
mdadm: cannot open device /dev/sdc2: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb6: Device or resource busy
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: cannot open device /dev/sdb2: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: cannot open device /dev/sda5: Device or resource busy
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: No arrays found in config file or automatically



I'm not realy sure what other information people might need, but would very much appreciate some hints/thoughts on going forward (and also perhaps a high level explaination of whats going on - i don't want to just fix this, i really do want to learn some of this stuff.)

hawkinsst7
I'm New!
I'm New!
Posts: 2
Joined: Sat May 23, 2015 4:19 am

Re: [FIXED]crashed volume after power outage - disks appear

Unread post by hawkinsst7 » Fri Jun 12, 2015 12:53 pm

FIXED (Sort of)
So just for the sake of Googlers in the future ( https://xkcd.com/979/ )

Synology support finally contacted me and remoted in, and was able to mount the drives in a R/O state. Afterwards, I checked the command history, which is good because apparently the NAS got an update and auto-rebooted which unmounted everything again. Comments are my observations. These might be useful troubleshooting steps.

Code: Select all

mount /dev/vg1 /volume1    # does it mount?
ll /volume1  # lv -> /dev/mapper/vg1-lv

umount /volume1   # causes beeps
mount /dev/vg1/lv /volume1   # appears to fail (mount: mounting /dev/vg1/lv on /volume1 failed: No such device) 
tail /var/log/messages   # I think these are the lines he was looking for
# Jun 11 06:50:54 DiskStation kernel: [13008.925928] JBD2: journal transaction 28430572 on dm-0-8 is corrupt.
# Jun 11 06:50:54 DiskStation kernel: [13008.932286] EXT4-fs (dm-0): error loading journal


mount -o ro,noload /dev/vg1/lv /volume1/      # remount it read only, without loading the journal. Genius!!
df -h  # Confirm its mounted
synospace --map_file -d    # returned "Success to dump space info into '/etc/space,/tmp/space' " but not really sure what that means
synocheckshare # Ran some diags I guess
grep volume1 /usr/syno/etc/smb.conf #my share names pop up, not sure why he was grepping for this.

at this point, the FS was accessable via FileStation and FTP so I grabbed what I needed, and then had to delete my vg and rebuild.

incidentally: synocheckshare has no --help, but synospace has some documentation there.

So, future Googlers, I hope this helps somewhat!

Phylum
Student
Student
Posts: 76
Joined: Sat Oct 15, 2011 5:58 pm

Re: [FIXED]crashed volume after power outage - disks appear

Unread post by Phylum » Fri Dec 11, 2015 5:17 am

You. Are. My. Hero.

My Sad Story.
I received an alert that a drive failed the SMART test. I was a bit surprised since I replaced most of the drives in the last year - whatever. The next day we had a power outage on one side of the apartment and the NAS was not happy when it came up.
  • I have 1 disk group made up of 4 disks in RAID5
  • I have 2 volumes on said disk group.
  • Volume 1 is the primary with all of the good stuff
  • Volume 2 really serves no purpose anymore but its there.
  • Volume 1 was listed as degraded so I followed the advice in the NAS and let it run through its check.
  • The following morning the status of Volume 1 changed from degraded to crashed.
  • Volume 2 however was fine.
  • I ended up bouncing it again at some point to verify the physical connections since the disk that allegedly failed was now no longer detected.
Not that there's ever good timing for something like this but this was not ideal as I was moving in 3 days.
I moved in 1 day and got the NAS reconnected by the end of the weekend

I ended up opening a case with Synology support via the Synology website and hoped for a speedy response. I eventually received a response and they asked me to email them the special support key and admin password.
They wanted the key and the admin password. In an Email. In clear text.
smh
I called support to explain the situation and the guy pointed me to the Support Center app on the NAS. I ASSUME that uses HTTPS to submit info but you never know. I put all the details in the support ticket I opened through the support app on the NAS and again waited for a response.
I think I was getting a response in about 24 hours which when you're in "dire straits" feels like an eternity. Again they asked for the password information in an email to which I asked if there was a secure web portal where I could update the ticket etc.
I ultimately called the support line again and asked the same question and this time the person said yes. HOWEVER, it wasn't a support site in the traditional sense. It was access to one of their NAS' where I could upload the debug.dat as well as any additional information. The guy said he'd update the ticket with a note stating that the credentials would be in the folder that way the tech working on my NAS could get the information he needed. Sounded good!

I get in there and add the files to my folder. (I don't recall if I created the folder OR if it already existed) After adding all the details it dawned on me that the credentials they gave were very very very generic and super easy to figure out making me fear what was going on. I started browsing the structure and sure enough, could get into every other ticket. folder on the NAS. Every. One. If someone had been inclined to peruse the folders, they would have found a lovely package by me containing all the details they would need to get into my NAS. Great.

Ok so its December 10th and they have yet to be able to start working on the NAS. All this week I've been fielding emails from Synology Support about not providing the key in the email, not providing the admin password in the email, the NAS not being accessible etc. I had to send screenshots from my phone while at work to prove that it was indeed online and accessible. What nonsense.

Earlier in the week, knowing a little about lvm, I poked around the system to see what the issue was and that's where I discovered the problem was with a corrupt journal transaction. For some crazy reason, although volume 1 wasn't mounted (per df & mount) I couldn't run fsck against it. I get an email from Synology support asking about the commands that were run on the box and why etc. I told them plainly I was poking around trying to understand the structure and left it at that. (Curious, I thought they couldn't acces the NAS though?)

Well, its been way too long without any appreciable action and I'm getting antsy.
So this evening I ran

Code: Select all

fsck.ext4 -fvpc on /dev/mapper/vol1-origin
Then by God's good grace was finally able to mount it in read-only where previously I couldn't and the error escapes me.
I googled something fairly obscure like

Code: Select all

"JBD2: journal transaction" on dm-3-8 "is corrupt"
That found this post which allowed me to get all the shares up and running again.
I'm pulling everything down now and will likely redo everything this weekend.


I'm going to add more data to this post that will hopefully lead more would be Synology Googlers / Bingers in a similar predicament here.
I've also added loads of interesting reads at the bottom.


Excerpt of dmesg

Code: Select all

[    0.180731] raid6: sse2x1     113 MB/s
[    0.201468] raid6: sse2x2     218 MB/s
[    0.221403] raid6: sse2x4     433 MB/s
[    0.225005] raid6: using algorithm sse2x4 (433 MB/s)
[    0.230004] raid6: using ssse3x2 recovery algorithm
[    5.246323] ahci 0000:00:1f.2: version 3.0
[    5.246669] ahci 0000:00:1f.2: irq 43 for MSI/MSI-X
[    5.246796] ahci: SSS flag set, parallel bus scan disabled
[    5.252430] ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[    5.260665] ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ccc ems sxs
[    5.269412] ahci 0000:00:1f.2: setting latency timer to 64
[    5.317588] scsi0 : ahci
[    5.324795] scsi1 : ahci
[    5.329774] scsi2 : ahci
[    5.334874] scsi3 : ahci
[    5.342832] scsi4 : ahci
[    5.347462] scsi5 : ahci
[    5.351217] ata1: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b100 irq 43
[    5.358661] ata2: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b180 irq 43
[    5.366081] ata3: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b200 irq 43
[    5.373502] ata4: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b280 irq 43
[    5.380902] ata5: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b300 irq 43
[    5.388303] ata6: SATA max UDMA/133 abar m2048@0xfe87b000 port 0xfe87b380 irq 43
[    5.395702] Delay 10 seconds to wait for disk 1 ready.
[   15.707036] ata1: SATA link down (SStatus 0 SControl 300)
[   15.712477] Delay 10 seconds to wait for disk 2 ready.
[   26.024035] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   26.030832] ata2.00: ATA-9: WDC WD30EFRX-68EUZN0, 80.00A80, max UDMA/133
[   26.037545] ata2.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[   26.045288] ata2.00: configured for UDMA/133
[   26.050223] scsi 1:0:0:0: Direct-Access     WDC      WD30EFRX-68EUZN0         80.0 PQ: 0 ANSI: 5
[   26.060436] sd 1:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   26.068213] sd 1:0:0:0: [sdb] 4096-byte physical blocks
[   26.070764] Delay 10 seconds to wait for disk 3 ready.
[   26.078850] sd 1:0:0:0: [sdb] Write Protect is off
[   26.083673] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[   26.083800] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   26.147914]  sdb: sdb1 sdb2 sdb3
[   26.154214] sd 1:0:0:0: [sdb] Attached SCSI disk
[   36.377037] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   36.384786] ata3.00: ATA-8: TOSHIBA DT01ACA300, MX6OABB0, max UDMA/133
[   36.391332] ata3.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[   36.400124] ata3.00: configured for UDMA/133
[   36.405841] scsi 2:0:0:0: Direct-Access     TOSHIBA  DT01ACA300               MX6O PQ: 0 ANSI: 5
[   36.416134] sd 2:0:0:0: [sdc] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   36.423944] sd 2:0:0:0: [sdc] 4096-byte physical blocks
[   36.429474] sd 2:0:0:0: [sdc] Write Protect is off
[   36.434296] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[   36.434383] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   36.492639]  sdc: sdc1 sdc2 sdc3
[   36.498097] sd 2:0:0:0: [sdc] Attached SCSI disk
[   36.734037] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   36.740920] ata4.00: ATA-8: ST3000DM001-9YN166, CC9D, max UDMA/133
[   36.747123] ata4.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[   36.765868] ata4.00: configured for UDMA/133
[   36.770705] scsi 3:0:0:0: Direct-Access     ATA      ST3000DM001-9YN166       CC9D PQ: 0 ANSI: 5
[   36.780826] sd 3:0:0:0: [sdd] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   36.788631] sd 3:0:0:0: [sdd] 4096-byte physical blocks
[   36.794159] sd 3:0:0:0: [sdd] Write Protect is off
[   36.798967] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[   36.799067] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   36.866594]  sdd: sdd1 sdd2 sdd3
[   36.871856] sd 3:0:0:0: [sdd] Attached SCSI disk
[   37.097036] ata5: SATA link down (SStatus 0 SControl 300)
[   37.407028] ata6: SATA link down (SStatus 0 SControl 300)
[   37.413461] sata_sil24 0000:02:00.0: version 1.1
[   38.415188] scsi6 : sata_sil24
[   38.420917] ata7: SATA max UDMA/100 host m128@0xfeafdc00 port 0xfeafe000 irq 17
[   38.428333] ata7: limiting SATA link speed to 1.5 Gbps
[   40.453041] ata7: SATA link down (SStatus 0 SControl 10)
[   40.525638] md: linear personality registered for level -1
[   40.531177] md: raid0 personality registered for level 0
[   40.536521] md: raid1 personality registered for level 1
[   40.541841] md: raid10 personality registered for level 10
[   40.547468] md: raid6 personality registered for level 6
[   40.552806] md: raid5 personality registered for level 5
[   40.558131] md: raid4 personality registered for level 4
[   40.563878] device-mapper: ioctl: 4.25.0-ioctl (2013-06-26) initialised: dm-devel@redhat.com
[   41.792950] md: Autodetecting RAID arrays.
[   41.860661] md: invalid raid superblock magic on sdb3
[   41.865732] md: sdb3 does not have a valid v0.90 superblock, not importing!
[   41.931504] md: invalid raid superblock magic on sdc3
[   41.936579] md: sdc3 does not have a valid v0.90 superblock, not importing!
[   42.011564] md: invalid raid superblock magic on sdd3
[   42.016634] md: sdd3 does not have a valid v0.90 superblock, not importing!
[   42.023613] md: Scanned 9 and added 6 devices.
[   42.028074] md: autorun ...
[   42.030882] md: considering sdb1 ...
[   42.034481] md:  adding sdb1 ...
[   42.037723] md: sdb2 has different UUID to sdb1
[   42.042279] md:  adding sdc1 ...
[   42.045524] md: sdc2 has different UUID to sdb1
[   42.050078] md:  adding sdd1 ...
[   42.053323] md: sdd2 has different UUID to sdb1
[   42.057893] md: created md0
[   42.060704] md: bind<sdd1>
[   42.063457] md: bind<sdc1>
[   42.066206] md: bind<sdb1>
[   42.068950] md: running: <sdb1><sdc1><sdd1>
[   42.073725] md/raid1:md0: active with 3 out of 4 mirrors
[   42.079098] md0: detected capacity change from 0 to 2549940224
[   42.085242] md: considering sdb2 ...
[   42.088859] md:  adding sdb2 ...
[   42.092108] md:  adding sdc2 ...
[   42.095375] md:  adding sdd2 ...
[   42.099646] md: created md1
[   42.102482] md: bind<sdd2>
[   42.105298] md: bind<sdc2>
[   42.108058] md: bind<sdb2>
[   42.110802] md: running: <sdb2><sdc2><sdd2>
[   42.115658] md/raid1:md1: active with 3 out of 4 mirrors
[   42.121091] md1: detected capacity change from 0 to 2147418112
[   42.127262] md: ... autorun DONE.
[   42.160627]  md0: unknown partition table
[   45.991142] EXT4-fs (md0): barriers disabled
[   45.999122] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts:
[   48.452862] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: barrier=1
[   50.658451] EXT4-fs (md0): re-mounted. Opts: (null)
[   56.186358] sd 1:0:0:0: Attached scsi generic sg0 type 0
[   56.191918] sd 2:0:0:0: Attached scsi generic sg1 type 0
[   56.197487] sd 3:0:0:0: Attached scsi generic sg2 type 0
[   56.451316]  md1: unknown partition table
[   56.503910] Adding 2097084k swap on /dev/md1.  Priority:-1 extents:1 across:2097084k
[   57.981887] Btrfs loaded
[   58.055705] 2015-12-4 4:3:34 UTC
[   58.058976] synobios: load, major number 201
[   58.063296] Brand: Synology
[   58.066154] Model: DS-411+II
[   62.993923] md: md2 stopped.
[   63.012258] md: bind<sdc3>
[   63.015352] md: bind<sdd3>
[   63.018384] md: bind<sdb3>
[   63.023658] md/raid:md2: device sdb3 operational as raid disk 1
[   63.029630] md/raid:md2: device sdd3 operational as raid disk 3
[   63.035616] md/raid:md2: device sdc3 operational as raid disk 2
[   63.042488] md/raid:md2: allocated 4346kB
[   63.046887] md/raid:md2: raid level 5 active with 3 out of 4 devices, algorithm 2
[   63.054434] RAID conf printout:
[   63.054447]  --- level:5 rd:4 wd:3
[   63.054456]  disk 1, o:1, dev:sdb3
[   63.054462]  disk 2, o:1, dev:sdc3
[   63.054468]  disk 3, o:1, dev:sdd3
[   63.054560] md2: detected capacity change from 0 to 8987271954432
[   63.089666]  md2: unknown partition table
[   67.300535] EXT4-fs (dm-4): barriers disabled
[   67.414398] EXT4-fs (dm-4): mounted filesystem with writeback data mode. Opts: usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,data=writeback,oldalloc
[   68.172962] EXT4-fs (dm-3): barriers disabled
[   68.992074] JBD2: journal transaction 9300484 on dm-3-8 is corrupt.
[   68.998374] EXT4-fs (dm-3): error loading journal
[   70.205127] ata2.00: configured for UDMA/133
[   70.209506] ata2: EH complete
[   70.215114] ata4.00: configured for UDMA/133
[   70.219465] ata4: EH complete
[   70.222530] ata3.00: configured for UDMA/133
[   70.226879] ata3: EH complete
[   74.608244] ata2.00: configured for UDMA/133
[   74.612659] ata2: EH complete
[   74.864621] ata3.00: configured for UDMA/133
[   74.868967] ata3: EH complete
[   75.224228] ata4.00: configured for UDMA/133
[   75.228584] ata4: EH complete

Code: Select all

> fdisk -l
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdb: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdc: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdc1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdd: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdd1               1      267350  2147483647+ ee EFI GPT

Disk /dev/synoboot: 128 MB, 128450560 bytes
255 heads, 63 sectors/track, 15 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

        Device Boot      Start         End      Blocks  Id System
/dev/synoboot1   *           1           2       16033+ 83 Linux
/dev/synoboot2               3          14       96390  83 Linux

Code: Select all

> pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               vg1
  PV Size               8.17 TB / not usable 2.44 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              2142732
  Free PE               438793
  Allocated PE          1703939
  PV UUID               ki4s01-uXuM-IDOl-x2Tb-s9dX-y2pf-iHDE3K

Code: Select all

> vgdisplay
  --- Volume group ---
  VG Name               vg1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               8.17 TB
  PE Size               4.00 MB
  Total PE              2142732
  Alloc PE / Size       1703939 / 6.50 TB
  Free  PE / Size       438793 / 1.67 TB
  VG UUID               VtDoSS-Rc3x-2q4q-DOig-ellf-p1sS-bkKR61

Code: Select all

> lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg1/syno_vg_reserved_area
  VG Name                vg1
  LV UUID                hoRzk1-zeP0-FMM1-fPvu-gGKe-Bt5W-FQy1rk
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                12.00 MB
  Current LE             3
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     384
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg1/volume_1
  VG Name                vg1
  LV UUID                lWYX0l-lth9-u8y1-M4ce-g5c9-XaUH-h880ic
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                5.50 TB
  Current LE             1441792
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/vg1/volume_2
  VG Name                vg1
  LV UUID                kI1EpE-oFiQ-va02-ABJG-f9WA-7U5X-UBPi31
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.00 TB
  Current LE             262144
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:2

Code: Select all

> lvs
  LV                    VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  syno_vg_reserved_area vg1  -wi-a- 12.00M
  volume_1              vg1  -wi-ao  5.50T
  volume_2              vg1  -wi-ao  1.00T

Code: Select all

> cat /etc/fstab
none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/mapper/vol1-origin /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0
/dev/mapper/vol2-origin /volume2 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0

Code: Select all

> mount
/dev/root on / type ext4 (defaults)
/sys on /sys type sysfs (0)
none on /dev/pts type devpts (gid=4,mode=620)
/tmp on /tmp type tmpfs (0)
/run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /sys/fs/cgroup type tmpfs (uid=0,gid=0,mode=0755,size=4k)
/dev/bus/usb on /proc/bus/usb type bind (bind)
none on /sys/kernel/debug type debugfs (0)
/dev/mapper/vol2-origin on /volume2 type ext4 (usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl)
securityfs on /sys/kernel/security type securityfs (0)

Code: Select all

> df
Filesystem               1K-blocks     Used  Available Use% Mounted on
/dev/root                  2385528   888008    1378736  40% /
/tmp                        507440      656     506784   1% /tmp
/run                        507440     2804     504636   1% /run
/dev/shm                    507440        0     507440   0% /dev/shm
none                             4        0          4   0% /sys/fs/cgroup
/dev/bus/usb                503400        4     503396   1% /proc/bus/usb
/dev/mapper/vol2-origin 1056758196 39557092 1017082320   4% /volume2

Code: Select all

> fsck.ext4 -n /dev/vg1/volume_1
e2fsck 1.42.6 (21-Sep-2012)
Warning!  /dev/vg1/volume_1 is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
1.42.6-4482: is cleanly umounted, 639158/184549376 files, 1075425181/1476395008 blocks

Code: Select all

> mount -t ext4 -o ro,noload /dev/vg1/volume_1 /vol1tst/
mount: mounting /dev/vg1/volume_1 on /vol1tst/ failed: Device or resource busy

Code: Select all

> dmesg | tail
[   74.612659] ata2: EH complete
[   74.864621] ata3.00: configured for UDMA/133
[   74.868967] ata3: EH complete
[   75.224228] ata4.00: configured for UDMA/133
[   75.228584] ata4: EH complete
[   75.238476] loop: module loaded
[   83.593195] findhostd uses obsolete (PF_INET,SOCK_PACKET)
[ 1616.734365] EXT4-fs (dm-3): barriers disabled
[ 1617.366499] JBD2: journal transaction 9300484 on dm-3-8 is corrupt.
[ 1617.372799] EXT4-fs (dm-3): error loading journal

Here are some of the more interesting finds on my search for recovering from this nightmare.
http://forum.synology.com/wiki/index.ph ... s_on_Linux
https://serverfault.com/questions/4906/ ... sk-cloning
https://superuser.com/questions/605435/ ... ng-journal
http://forum.synology.com/enu/viewtopic ... 39&t=61747
https://serverfault.com/questions/16702 ... r-bad-bloc

May no one else have to suffer through something like this.

tommylux
Novice
Novice
Posts: 42
Joined: Fri Jun 22, 2012 10:22 am

Re: [FIXED]crashed volume after power outage - disks appear fine

Unread post by tommylux » Fri May 05, 2017 1:39 pm

Have a crashed volume on my NAS at the moment, Been waiting for a few days now from Synology with just basic questions such as obtain the logs (which I can't seem to complete in this state anyway)

Not getting a speedy answer, I want see if I can be of assistance and see if I can mount the volume myself.

I just want to try and mount the volume:
mount /dev/vg1 /volume1

But when I try and log in via SSH, I get disconnected because the user accounts do not have a home dir.

I tried logging in as root - the only account that doesn't have a home directory. Apparently, this should be the same passwords as the default admin account, but this does not appear to be the case.

Anyone know how I can get back into SSH so I can try the mount commad?

Thanks

Locked

Return to “System Management”