Expanding Volume - errors in the filesystem

Topics pertaining to volume setup, boot/shutdown, initialization, DSM update, HDD migration.
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu
2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
jameskirl
I'm New!
I'm New!
Posts: 8
Joined: Tue Jan 11, 2011 9:36 pm

Expanding Volume - errors in the filesystem

Postby jameskirl » Sun Aug 10, 2014 12:39 am

Need some help please.

I have logged a support ticket but reading through the forum i am a little nervous.

I started with 5 disks
Three 2TB Disks and two 3TB Disks
I bought 2 new 6TB WD Red disks.

I followed the steps in the user guide
https://www.synology.com/en-us/support/tutorials/560
How to Expand Volumes or Disk Groups by Replacing Hard Disks on Synology NAS

My volume is SHR
I replaced the first 2TB Disk with the first 6TB
Then Manage in Storage and repaired the Array
Completed - no issue
I replaced the second 2TB Disk with the second 6TB
Then Manage in Storage and repaired the Array
Completed - no issue
I rebooted
Then Manager in Storage Manger - select Expand the volume with unallocated disk
I get the following error.

Code: Select all

Aug 6 19:43:52 DiskStation volumehandler.cgi: space_expand_unallocated.c:82 [Info] Success Dump map file
Aug 6 19:44:02 DiskStation kernel: [ 144.375040] EXT4-fs warning (device dm-0): ext4_resize_begin:32: There are errors in the filesystem, so online resizing is not allowed
Aug 6 19:44:02 DiskStation kernel: [ 144.375047]
Aug 6 19:44:02 DiskStation volumehandler.cgi: (fs_vol_expand.c)ExtFSExpand(88):Failed to '/sbin/resize2fs -fpF /dev/vg1/lv > /dev/null 2>&1', WEXITSTATUS(r) = 1
Aug 6 19:44:03 DiskStation volumehandler.cgi: volume_manage_with_temp_dev.c:279 Failed to expand file system on /dev/vg1/lv
Aug 6 19:44:03 DiskStation volumehandler.cgi: volume_lib.cpp:1017 Failed to expand file system on /dev/vg1/lv
Aug 6 19:44:03 DiskStation volumehandler.cgi: volumehandler.cpp:1444 failed to expand unallocated file system: /volume1


I raised the support ticket on Friday night but no answer yet.
Since then i tried the following
I ran Data Scrubbing under the DSM GUI - this completed in 24 hours with no issue but when i try to expand the volume i get exactly the same issue.
While Data Scrubbing was running i ran an Extended Smart Test on all 5 disks. Maybe not the best idea to kick everything off at the same time. After 24 hours, 3 of the disks are stuck at 90%, one of the new 6TB disks passed the smart test with no issues and the remaining 2TB disk is at 80%.
Update - SMART finished for all the disks finally - no issues

Aside from not being able to expand the volume, everything is showing normal and the unit is fully functional.

I have been busy reading the forum on what next steps might be.
http://forum.synology.com/enu/viewtopic.php?f=7&t=56468
http://forum.synology.com/enu/viewtopic.php?f=39&t=66550

Looks like the following is what i need to do.

1) run the command syno_poweroff_task

Code: Select all

 $syno_poweroff_task

2) unmount /volume1

Code: Select all

$umount /volume1

3) Run your check on the mount point

Code: Select all

$fsck.ext4 -pvf /dev/vg1/lv


I will wait for support to reply but any assistance or advice much appreciated.
For example what is the difference between Data Scrubbing and fsck


Results from cat /proc/mdstat

Code: Select all

...tore/webconsole/app/wc_work> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid1 sdb7[0] sdc7[1]
      2930247808 blocks super 1.1 [2/2] [UU]
     
md3 : active raid5 sde6[0] sdc6[3] sdb6[2] sda6[1]
      2930231424 blocks super 1.1 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
     
md2 : active raid5 sdb5[6] sde5[5] sdd5[3] sdc5[7] sda5[4]
      7795122688 blocks super 1.1 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
     
md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3]
      2097088 blocks [5/5] [UUUUU]
     
md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4]
      2490176 blocks [5/5] [UUUUU]
     
unused devices: <none>


Results from df

Code: Select all

...tore/webconsole/app/wc_work> df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0               2451064    700360   1648304  30% /
/tmp                    507872      1120    506752   0% /tmp
/dev/vg1/lv          8634218576 8085722412 548185548  94% /volume1
jameskirl
I'm New!
I'm New!
Posts: 8
Joined: Tue Jan 11, 2011 9:36 pm

Re: Expanding Volume - errors in the filesystem

Postby jameskirl » Sun Aug 10, 2014 7:20 pm

I did more research and found this http://boomkicker.wordpress.com/2013/02/14/scrub-synology-raid-disks/

Going through the steps now

Code: Select all

echo check > /sys/block/md2/md/sync_action
echo check > /sys/block/md0/md/sync_action
echo check > /sys/block/md1/md/sync_action
echo check > /sys/block/md3/md/sync_action
echo check > /sys/block/md4/md/sync_action


Then

Code: Select all

cat /sys/block/md[x]/md/mismatch_cnt


When i run this i get

Code: Select all

DiskStation> cat /sys/block/md2/md/mismatch_cnt
16


I tried the final step of

Code: Select all

echo repair > /sys/block/md2/md/sync_action


But no change to mismatch_cnt so i assume i must wait until the Scrub is finished first
jameskirl
I'm New!
I'm New!
Posts: 8
Joined: Tue Jan 11, 2011 9:36 pm

Re: Expanding Volume - errors in the filesystem

Postby jameskirl » Tue Aug 12, 2014 1:55 pm

Any help please?

I went through the steps above but still the same error.
Synology helpdesk have responded and asked for remote access. I will do that tomorrow.

Code: Select all

echo check > /sys/block/md0/md/sync_action
echo check > /sys/block/md1/md/sync_action
echo check > /sys/block/md2/md/sync_action
echo check > /sys/block/md3/md/sync_action
echo check > /sys/block/md4/md/sync_action



Code: Select all

Mismatches were reported
cat /sys/block/md0/md/mismatch_cnt
cat /sys/block/md1/md/mismatch_cnt
cat /sys/block/md2/md/mismatch_cnt
cat /sys/block/md3/md/mismatch_cnt
cat /sys/block/md4/md/mismatch_cnt



Code: Select all

After running this and Data Scrubbing complete - i tried to extend the voluem again and same error reported
echo repair > /sys/block/md0/md/sync_action
echo repair > /sys/block/md1/md/sync_action
echo repair > /sys/block/md2/md/sync_action
echo repair > /sys/block/md3/md/sync_action
echo repair > /sys/block/md4/md/sync_action


Code: Select all

No errors reported at disk level
cat /sys/block/md0/md/rd?/errors
cat /sys/block/md1/md/rd?/errors
cat /sys/block/md2/md/rd?/errors
cat /sys/block/md3/md/rd?/errors
cat /sys/block/md4/md/rd?/errors



Is the next step going to be...

Code: Select all

1) run the command syno_poweroff_task
Code: Select all
 $syno_poweroff_task

2) unmount /volume1
Code: Select all
$umount /volume1

3) Run your check on the mount point
Code: Select all
$fsck.ext4 -pvf /dev/vg1/lv
jameskirl
I'm New!
I'm New!
Posts: 8
Joined: Tue Jan 11, 2011 9:36 pm

Re: Expanding Volume - errors in the filesystem

Postby jameskirl » Wed Aug 27, 2014 10:51 pm

Here are the findings from Synology support.

We found NAS has file system error,

We found file system error in your NAS, the file system errors may be caused by the sudden power failure without the protection of UPS or disk errors. We strongly recommend you back up your data to another place first, and run a file system check and try to fix the errors. Please enable the ssh/telnet function in DiskStation > Control Panel > Terminal, and use Putty http://the.earth.li/~sgtatham/putty/lat ... /putty.exe to ssh/telnet to your DiskStation with the user 'root' and admin's password. Then, run the following commands to repair the file system. Before you do this, you must back up your important data, as there are risks of data loss:


Code: Select all

syno_poweroff_task -d [Press Enter] << This will bring down all services except the SSH.

vgchange -ay [Press Enter] << This will enable the volume

fsck.ext4 -pvf -C0 /dev/vg1/lv [press Enter] << This will try to fix the error, pop out yes/no to let you chose when error show up.

fsck.ext4 -yvf -C0 /dev/vg1/lv [Press Enter] << This will try to fix the error, system will automatic chose yes to fix error.


If -pvf can't work, please use -yvf instead. After the command fsck is completed, please reboot your DS with the reboot command.


That worked :D
Conq
Trainee
Trainee
Posts: 14
Joined: Fri May 16, 2014 9:38 am

Re: Expanding Volume - errors in the filesystem

Postby Conq » Sat Dec 13, 2014 3:48 am

jameskirl wrote:Here are the findings from Synology support.

We found NAS has file system error,

We found file system error in your NAS, the file system errors may be caused by the sudden power failure without the protection of UPS or disk errors. We strongly recommend you back up your data to another place first, and run a file system check and try to fix the errors. Please enable the ssh/telnet function in DiskStation > Control Panel > Terminal, and use Putty http://the.earth.li/~sgtatham/putty/lat ... /putty.exe to ssh/telnet to your DiskStation with the user 'root' and admin's password. Then, run the following commands to repair the file system. Before you do this, you must back up your important data, as there are risks of data loss:


Code: Select all

syno_poweroff_task -d [Press Enter] << This will bring down all services except the SSH.

vgchange -ay [Press Enter] << This will enable the volume

fsck.ext4 -pvf -C0 /dev/vg1/lv [press Enter] << This will try to fix the error, pop out yes/no to let you chose when error show up.

fsck.ext4 -yvf -C0 /dev/vg1/lv [Press Enter] << This will try to fix the error, system will automatic chose yes to fix error.


If -pvf can't work, please use -yvf instead. After the command fsck is completed, please reboot your DS with the reboot command.


That worked :D


I tried the same procedure on my mount point, but I am getting the following error when I run fsck.ext4:
"/dev/vg1000/lv is in use"

Couldn't find a way to stop the volume from showing "in use", even disabling services manually before disabling everything through PuTTY didn't help.

Tried Googling, everybody's successful resolution was through an undocumented method applied by Synology Support team, in which they gained access to the system remotely for fixing the issue.
Conq
Trainee
Trainee
Posts: 14
Joined: Fri May 16, 2014 9:38 am

Re: Expanding Volume - errors in the filesystem

Postby Conq » Sun Dec 14, 2014 6:10 pm

Conq wrote:
jameskirl wrote:Here are the findings from Synology support.

We found NAS has file system error,

We found file system error in your NAS, the file system errors may be caused by the sudden power failure without the protection of UPS or disk errors. We strongly recommend you back up your data to another place first, and run a file system check and try to fix the errors. Please enable the ssh/telnet function in DiskStation > Control Panel > Terminal, and use Putty http://the.earth.li/~sgtatham/putty/lat ... /putty.exe to ssh/telnet to your DiskStation with the user 'root' and admin's password. Then, run the following commands to repair the file system. Before you do this, you must back up your important data, as there are risks of data loss:


Code: Select all

syno_poweroff_task -d [Press Enter] << This will bring down all services except the SSH.

vgchange -ay [Press Enter] << This will enable the volume

fsck.ext4 -pvf -C0 /dev/vg1/lv [press Enter] << This will try to fix the error, pop out yes/no to let you chose when error show up.

fsck.ext4 -yvf -C0 /dev/vg1/lv [Press Enter] << This will try to fix the error, system will automatic chose yes to fix error.


If -pvf can't work, please use -yvf instead. After the command fsck is completed, please reboot your DS with the reboot command.


That worked :D


I tried the same procedure on my mount point, but I am getting the following error when I run fsck.ext4:
"/dev/vg1000/lv is in use"

Couldn't find a way to stop the volume from showing "in use", even disabling services manually before disabling everything through PuTTY didn't help.

Tried Googling, everybody's successful resolution was through an undocumented method applied by Synology Support team, in which they gained access to the system remotely for fixing the issue.


After several attempts with multiple restarts, I finally was able to run fsck.ext4, the check took a while, it was a bit slow (took like 4-5 hours) on 7 drives (3-4 TB Each).

The test did fix a lot of errors, and after the reboot, I was finally able to expand the storage successfully, after having the issue for months since I upgraded the hard drives back in May 2014.

viewtopic.php?f=7&t=85947 (Related post with my issue)
Tolis77
I'm New!
I'm New!
Posts: 4
Joined: Tue Aug 27, 2013 12:22 pm

Re: Expanding Volume - errors in the filesystem

Postby Tolis77 » Wed Dec 17, 2014 12:30 pm

Can you please tell us the solution.
There are a lot of people stuck with the warning "/dev/vg1000/lv is in use".
Thanks
hurlyhunk
Trainee
Trainee
Posts: 11
Joined: Thu Feb 19, 2015 12:35 am

Re: Expanding Volume - errors in the filesystem

Postby hurlyhunk » Sat Apr 25, 2015 4:24 am

Hello,

I am getting this error,

Code: Select all

e2fsck 1.42.6 (21-Sep-2012)
fsck.ext4: No such file or directory while trying to open /dev/vg1/lv
Possibly non-existent device?


any ideas ??
ksmithaz1
I'm New!
I'm New!
Posts: 3
Joined: Sat May 23, 2015 4:21 pm

Re: Expanding Volume - errors in the filesystem

Postby ksmithaz1 » Sat May 23, 2015 4:28 pm

Your volume may not be vg1. Use 'ls -d' to show the correct name eg. . .

nas-3> ls -d /dev/vg*
/dev/vg1000 /dev/vga_arbiter

So you would use the volume group vg1000

fsck.ext4 -pvf /dev/vg1000/lv
Keith Smith
DS1815+ / 22TB / Plex + Media Server + Snivel
RS2212+ / 25TB / Video Station + File Services
julianwood
I'm New!
I'm New!
Posts: 4
Joined: Tue Dec 29, 2009 8:46 pm

Re: Expanding Volume - errors in the filesystem

Postby julianwood » Wed May 27, 2015 11:05 pm

Pulling my hair out trying to get this working.
I have a failed file system and can't seem to run fsck as the volume is always I use even though it is unmounted.

I stopped all services.
The only one I couldn't find to stop via a .sh file was:
NAS> lsof |grep volume1
s2s_daemo 10795 root 8u REG 253,1 11264 454270979 /volume1/@S2S/event.sqlite

I killed the process which then worked.

I can't run:
NAS> syno_poweroff_task -d
device-mapper: remove ioctl failed: Device or resource busy
Command failed

Nothing seems to be mounted on /volume1:
NAS> lsof /volume1

Volume is unmounted successfully.
NAS> umount /volume1
umount: can't umount /volume1: Invalid argument

Can't run fsck:
NAS> fsck.ext4 -pvf /dev/vg1000/lv
/dev/vg1000/lv is in use.
e2fsck: Cannot continue, aborting.

Any ideas?
kungfuman
Trainee
Trainee
Posts: 13
Joined: Wed Mar 07, 2012 12:53 pm

Ds1515+ new drive. Can't expand volume as file system error

Postby kungfuman » Sat Dec 05, 2015 3:31 pm

I'm running a ds1515+ with 2 x 4tb hgst drives. I wanted to increase storage by adding another identical 4tb hgst drive. However when I go to the storage manager and try to expand the volume I get a warning stating the file system error and won't increase the volume. The volume is stated as healthy and running smart check on all the drives shows pass. Where do I go from here? Thanks for any advice given!
User avatar
maxxfi
Compiler
Compiler
Posts: 6794
Joined: Sun Dec 27, 2009 12:13 pm
Location: Espoo, Finland

Re: Ds1515+ new drive. Can't expand volume as file system er

Postby maxxfi » Sun Dec 06, 2015 10:41 am

Is the disk you tried to add brand new, or has it been used before?
DS-106j > DS-210j > DS-411
kungfuman
Trainee
Trainee
Posts: 13
Joined: Wed Mar 07, 2012 12:53 pm

Re: Ds1515+ new drive. Can't expand volume as file system er

Postby kungfuman » Sun Dec 06, 2015 5:32 pm

the new drive is brand new.

Everything is stated as normal in DSM: system health, hard disk checks.
kungfuman
Trainee
Trainee
Posts: 13
Joined: Wed Mar 07, 2012 12:53 pm

Re: Ds1515+ new drive. Can't expand volume as file system er

Postby kungfuman » Sun Dec 13, 2015 5:38 pm

still not having much luck with this. I've backed up all my data in anticipation I may need yo do a fresh rebuild. Can anyone offer any steps I can try to fix the system errors?
mike42dk
Proficient
Proficient
Posts: 2681
Joined: Sun Jun 06, 2010 7:45 am

Re: Ds1515+ new drive. Can't expand volume as file system er

Postby mike42dk » Sun Dec 13, 2015 7:56 pm

What kind of volume have you created with the first 2 disks?
¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
DS 415+ DSM 6 7321
2 WD red 2 TB RAID 1
2 Seagate nas 4 TB basic
DX 213
1 WD purple 1 TB basic

Return to “Installation, Configuration, Migration, Expansion”

Who is online

Users browsing this forum: grouchy, sjeph8je and 2 guests