DS918+ root partition at 100%

Questions and mods regarding system management may go here
Forum rules
Please note the disclaimer before modifying your Synology Product.
User avatar
pecus
Trainee
Trainee
Posts: 19
Joined: Tue Feb 17, 2009 12:19 am

DS918+ root partition at 100%

Postby pecus » Wed Dec 06, 2017 12:28 am

I am trying to troubleshoot a root partition at 100%. I can connect via SSH, but had to disable 2FA in order to access the web shell
This is a dump from df

Code: Select all

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/md0          2385528    2348224          0 100% /
none              1938084          4    1938080   1% /dev
/tmp              1941932       1136    1940796   1% /tmp
/run              1941932       3180    1938752   1% /run
/dev/shm          1941932         12    1941920   1% /dev/shm
none                    4          0          4   0% /sys/fs/cgroup
cgmfs                 100          0        100   0% /run/cgmanager/fs
/dev/vg1000/lv 2803775572 2758217960   45557612  99% /volume1


I have no idea of what filled up the partition and can't seem to find any culprit to delete. Any help would be appreciated.
This is a 918+
I am Xposting here after a day or so gone unanswered in System Management

TIA
nixjps
Novice
Novice
Posts: 47
Joined: Fri Jun 09, 2017 1:39 pm

Re: DS918+ root partition at 100%

Postby nixjps » Wed Dec 06, 2017 11:49 am

Hi,

Guess /var is filling up for some reasons....
Please post result of "du -xhs --exclude /proc --exclude /dev /*"
You may want to look at /root and /var (actually, post result of "du -xhs /var/*).

Rgds,
DS916+ (8G) - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 3 - 1 Disk Group SHR - 2 volumes - Home Usage
DS216Play - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 2 - Basic Disks - 2 Volumes - Off site backup of DS916+ and local browsing of Photos, Musics and Videos
DS916 and DS216Play MAN link (1Gb FFTH same ISP).
DS916 S2S -> DS216Play, HyperBackup of DS916 config & critical data to DS216Play. HyperBackup of DS216Play configuration and local data to DS916
User avatar
pecus
Trainee
Trainee
Posts: 19
Joined: Tue Feb 17, 2009 12:19 am

Re: DS918+ root partition at 100%

Postby pecus » Thu Dec 07, 2017 12:40 am

Hi nixjps,
thanks for trying to solve this out.
Here are the dump from du (all run as root):

Code: Select all

du -xhs --exclude /proc --exclude /dev /*

0   /1
0   /bin
0   /config
8.0M   /etc
7.2M   /etc.defaults
4.0K   /initrd
0   /lib
0   /lib32
0   /lib64
4.0K   /lost+found
4.0K   /mnt
48K   /root
3.3M   /run
0   /sbin
0   /sys
1.1M   /tmp
727M   /usr
32M   /var
5.8M   /var.defaults
647M   /volume1
22M   /volumeSATA1
1.5G   /volumeUSB1

and

Code: Select all

du -xhs /var/*

19M   /var/cache
4.0K   /var/crash
12K   /var/db
2.5M   /var/dynlib
4.0K   /var/empty
5.6M   /var/lib
0   /var/lock
3.4M   /var/log
720K   /var/packages
0   /var/run
4.0K   /var/services
532K   /var/spool
8.0K   /var/state
4.0K   /var/synobackup
12K   /var/target
100K   /var/tmp
nixjps
Novice
Novice
Posts: 47
Joined: Fri Jun 09, 2017 1:39 pm

Re: DS918+ root partition at 100%

Postby nixjps » Thu Dec 07, 2017 5:48 pm

Hi,

I don't see anything particular.... /var content looks ok,
Your root looks good too (unexpected folder /1, but it's empty).

Can you post result of "du -xhs /dev" and ls -la /?

I would think about two things: a) File system corruption, b) a couple of huge files in a mount point [core dump under /volume1 for instance).

To clear a) you can run a file system check.

Code: Select all

sudo e2fsck -nvf /dev/md0

The above command will not fix file system error, just report issues.
If issues..... Need to find out out to fix.... I know on traditional linux box how to force a fsck on a mounted root file system.... Don't know how to do this on DSM.... May be some reader can.
We must remount root as readonly then run fsck.......

As for b).
You must umount /volume1, /volumeSATA1 and /volumeUSB1 to check whether you have unexpected files in these directories.

Umounting SATA1 and USB1 is not an issue, eject it with DSM then "cd /volumeSATA1; ls -la" to identify unexpected files, same for for /volumeUSB1.
For /volume1 it's slightly different. umount will probably tell it can't because filesystem is busy....

Code: Select all

# So assuming you are logged as admin
# change working directory to make sure umounting won't raise busy file system exception
cd /
# now replace current admin shell process with sudo... 
exec sudo -
syno_poweroff_task -d
# it gracefully terminates all application/process and unmount volumes.
# if will take some time (eventually your ssh session will be closed if for some reasons you missed one of the above steps (cd /; exec).
# reconnect
# to check /volume1 has been unmount.
df
# if not umount it.
cd /; cd /volume1
ls -la
# should be empty.... If not this is eating space on you root file system....
# once every thing checked reboot
reboot


Rgds,
PS: edited post to add stuff about current working directory
DS916+ (8G) - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 3 - 1 Disk Group SHR - 2 volumes - Home Usage
DS216Play - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 2 - Basic Disks - 2 Volumes - Off site backup of DS916+ and local browsing of Photos, Musics and Videos
DS916 and DS216Play MAN link (1Gb FFTH same ISP).
DS916 S2S -> DS216Play, HyperBackup of DS916 config & critical data to DS216Play. HyperBackup of DS216Play configuration and local data to DS916
User avatar
pecus
Trainee
Trainee
Posts: 19
Joined: Tue Feb 17, 2009 12:19 am

Re: DS918+ root partition at 100%

Postby pecus » Sat Dec 09, 2017 12:42 pm

Hi nixjps,

thanks again for troubleshooting this with me. Indeed, there were files on the external drives mountpoints!
I cleaned everything up and went down to / at 36%.

I also ran fsck and there are minor errors (I think I can live with it), but if anyone knows how to trigger fsck on the root partition either at reboot or by remounting the volume rw, I won't mind fixing this too.

Code: Select all

e2fsck 1.42.6 (21-Sep-2012)
Warning!  /dev/md0 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'conv_progress_photo.CkoLd9' in /var/spool (25560) has deleted/unused inode 29927.  Clear? no

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +40268 -40271
Fix? no

Free blocks count wrong (9517, counted=9829).
Fix? no

Inode bitmap differences:  -29927 +29993
Fix? no

Free inodes count wrong (124119, counted=124074).
Fix? no


1.42.6-15091: ********** WARNING: Filesystem still has errors **********


       31529 inodes used (20.26%, out of 155648)
          88 non-contiguous files (0.3%)
          10 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 28723/24
      613027 blocks used (98.47%, out of 622544)
           0 bad blocks
           1 large file

       25087 regular files
        3353 directories
           2 character device files
           0 block device files
           0 fifos
        1227 links
        3123 symbolic links (2817 fast symbolic links)
           0 sockets
------------
       32793 files
       
nixjps
Novice
Novice
Posts: 47
Joined: Fri Jun 09, 2017 1:39 pm

Re: DS918+ root partition at 100%

Postby nixjps » Mon Dec 11, 2017 5:49 pm

pecus wrote:[...]
I also ran fsck and there are minor errors (I think I can live with it), but if anyone knows how to trigger fsck on the root partition either at reboot or by remounting the volume rw, I won't mind fixing this too.


Indeed, looks like minor errors. Actually, I believe they aren't really errors.

I might be wrong, but looks like you were running efsck while there was activity on /var/spool.
The activity seems related to photostation indexing or thumbnail generation.
During indexing or thumbnail generation, temporary files are created for short time /var/spool.
If such files are created and deleted between fsck test phases we may get error/warning messages....

For the time being, don't worry about fsck on rootfs. It's checked at each boot.... So you want/need to run fsck on your rootfs? Just reboot ;-D.

Over the weekend I extracted the initrd files of my ds216play and a DS916+ to check what's done during booting phases.
Wanted to find out if a mechanism was used (like ubuntu's /forcefsck file) to force root filesystem fsck.
According to "/linuxrc.syno", it is done each time system boots (before root files system of DS hard drive is mounted).

You can check by looking at output of "sudo tune2fs -l". See line "Last checked:". Should be close to last time your DS booted.

Rgds,
DS916+ (8G) - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 3 - 1 Disk Group SHR - 2 volumes - Home Usage
DS216Play - DSM 6.1.3-15152u4 - ST4000DM000-1F2168 x 2 - Basic Disks - 2 Volumes - Off site backup of DS916+ and local browsing of Photos, Musics and Videos
DS916 and DS216Play MAN link (1Gb FFTH same ISP).
DS916 S2S -> DS216Play, HyperBackup of DS916 config & critical data to DS216Play. HyperBackup of DS216Play configuration and local data to DS916

Return to “System Managment Mods”

Who is online

Users browsing this forum: No registered users and 2 guests