No left space on device

Discussion room for Docker, a lightweight virtualization application that gives you the ability to run thousands of containers created by developers from all over the world on DSM
Forum rules
We've moved! Head over to Synology Community (community.synology.com) to meet up with our team and other Synology enthusiasts!
hhppaa
I'm New!
I'm New!
Posts: 4
Joined: Fri Sep 21, 2018 8:33 am

No left space on device

Unread post by hhppaa » Fri Sep 21, 2018 8:39 am

Hi,

On drive with lot of free space, I get "OSError: [Errno 28] No space left on device" error during docker build.

Code: Select all

$ stat -f /volume1/
  File: "/volume1/"
    ID: d09e2494c028130d Namelen: 255     Type: btrfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1874206065 Free: 1836018764 Available: 1836018764
Inodes: Total: 0          Free: 0
It seams there is no inodes free.

Any ideas ?

User avatar
Rusty1281
Sagacious
Sagacious
Posts: 3246
Joined: Fri Jun 03, 2011 10:51 pm

Re: No left space on device

Unread post by Rusty1281 » Fri Sep 21, 2018 5:51 pm

What image is in question? Could be that you have no permission on the root of your volume1, docker that is.
Synology DS918+ (4x4TB WD RED - RAID 5 with 2x250GB 960EVO NVMe) | Synology DS412+ (4x3TB WD RED - RAID 5) | RT1900AC

mightbetrue
Versed
Versed
Posts: 273
Joined: Sun Oct 02, 2016 9:41 am

Re: No left space on device

Unread post by mightbetrue » Fri Sep 21, 2018 6:30 pm

Google consulting says that btrfs has a dynamic iNode allocation and supports up to 2^64 files in the FS.

Docker tends to suffer from iNodes when the "Docker Root Dir" is on an ext4 filesystem. On Synology the root dir must be on a btrfs device. Thus, it should be impossible to suffer from iNode exhaustion.

Please post the result of the command "docker info".

hhppaa
I'm New!
I'm New!
Posts: 4
Joined: Fri Sep 21, 2018 8:33 am

Re: No left space on device

Unread post by hhppaa » Sun Sep 23, 2018 8:20 am

Could be that you have no permission on the root of your volume1, docker that is.
I already created some containers. And the error is about space on device, not permissions.
Thus, it should be impossible to suffer from iNode exhaustion.
There just one disk in the synology (DS281+), I let the default parameters, so it's a full brtfs disk.

Code: Select all

/volume1/docker/odoo$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 16
Server Version: 17.05.0-ce
Storage Driver: btrfs
Logging Driver: db
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: fd189da3e13a3ef3d6d9eb73c5cd4697b4536cdd (expected: 9048e5e50717ea4497b757314bad98ea3763c145)
runc version: a2d6e07aab95ff37fb63cf5dec3c40d29940194f (expected: 9c2d8d184e5da67c95d601382adf14862e4f2228)
init version: 7a83305 (expected: 949e6fa)
Security Options:
 apparmor
Kernel Version: 4.4.59+
Operating System: <unknown>
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.785GiB
Name: syno
ID: CKNV:V2II:EFQ5:AUSP:XPB2:CNAP:3SMV:RSZC:NV3C:7FN2:VCRW:2HMI
Docker Root Dir: /volume1/@docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled


mightbetrue
Versed
Versed
Posts: 273
Joined: Sun Oct 02, 2016 9:41 am

Re: No left space on device

Unread post by mightbetrue » Sun Sep 23, 2018 10:07 am

The output of docker info confirms that your docker root is on a btrfs filesystem.

The classical iNode exhaustion (=tiny files eating up all the possible iNode entries) is not your problem, though it appears that brfs has a comparable mess going on.

<update>
the problem is the unallocated space on the btrfs volume: the output of the command 'btrfs filesystem usage /volume1' directly shows the problem.

The command 'btrfs filesystem df -H /volume1' shows how the "Device allocated" actualy is used.

Now the question is why there is unallocated storage and how to make the system use it!
</update>

Thanks for putting your finger into that issue. Eventually this will be a problem for Synology Docker users.

Thus said, I am confused why Synology even introduced btrfs as a filesystem. They should have gone with XFS as the Filesystem, providing Overlay2 for the Docker installation and everyting would be perfect. I am pretty sure, that we are not going to see that...

mightbetrue
Versed
Versed
Posts: 273
Joined: Sun Oct 02, 2016 9:41 am

Re: No left space on device

Unread post by mightbetrue » Sun Sep 23, 2018 1:35 pm

Seems like btrfs automaticly claims unallocated storage.
I just used dd to ocupy some space and as a restult the allocated data grew and the unallocated shrinked.

I am glad I came to wrong conclusions before.

Thus said, I guess it is high likely that hhppaa's btrfs volume needs a rebalancing.

@hhppaa: can you please post the results of 'btrfs fi df /volume1' and 'btrfs fi usage /volume1'

hhppaa
I'm New!
I'm New!
Posts: 4
Joined: Fri Sep 21, 2018 8:33 am

Re: No left space on device

Unread post by hhppaa » Sun Sep 23, 2018 2:33 pm

Code: Select all

btrfs filesystem df -H /volume1
Data, single: total=161.07GB, used=156.40GB
System, DUP: total=8.39MB, used=16.38kB
Metadata, DUP: total=3.22GB, used=740.39MB
GlobalReserve, single: total=268.43MB, used=0.00B

Code: Select all

sudo  btrfs filesystem usage /volume1
Password: 
Overall:
    Device size:		   7.27TiB
    Device allocated:		 156.02GiB
    Device unallocated:		   7.12TiB
    Device missing:		     0.00B
    Used:			 147.04GiB
    Free (estimated):		   7.12TiB	(min: 3.56TiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 256.00MiB	(used: 0.00B)

Data,single: Size:150.01GiB, Used:145.66GiB
   /dev/vg1000/lv	 150.01GiB

Metadata,DUP: Size:3.00GiB, Used:706.09MiB
   /dev/vg1000/lv	   6.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/vg1000/lv	  16.00MiB

Unallocated:
   /dev/vg1000/lv	   7.12TiB

Thanks @mightbetrue for your investigation.

mightbetrue
Versed
Versed
Posts: 273
Joined: Sun Oct 02, 2016 9:41 am

Re: No left space on device

Unread post by mightbetrue » Sun Sep 23, 2018 3:27 pm

Even though "Data, Single" seem to be almost full, btrfs allocates storage space when needed and grows the total.

I would have expacted that one of the other types (like MetaData, Dup) are close to full. Though, even this is not the case.

It is worrying to see a filesystem, where roughly 1/50th of the capacity is used, reports back that there is no space left.
I wish I knew what is causing it, but I don't. Since i am using the filesystem myself, it terrifies me somehow.

Thus said, I guess it's time to ask Synology for support or ask the btrfs community for assistance.

hhppaa
I'm New!
I'm New!
Posts: 4
Joined: Fri Sep 21, 2018 8:33 am

[Solved] Re: No left space on device

Unread post by hhppaa » Sun Sep 23, 2018 10:00 pm

It's not related to btrfs.

The inode "problem" shown by "stat -f /volume1/" is, I guess a tooling problem. See also https://wiki.archlinux.org/index.php/Bt ... free_space
General linux userspace tools such as df will inaccurately report free space on a Btrfs partition. It is recommended to use btrfs filesystem usage to query Btrfs partitions.
docker build was, in fact, working well. docker-compose build did not.

I guess it's related to this issue : https://github.com/docker/compose/issues/3775
docker-compose use a RAM filesystem. RAM is very limited on my DS218+ 2Gb.

Solution : replace docker-compose#build: by docker-compose#image:

Thanks for your help mightbetrue.

mightbetrue
Versed
Versed
Posts: 273
Joined: Sun Oct 02, 2016 9:41 am

Re: No left space on device

Unread post by mightbetrue » Mon Sep 24, 2018 6:10 am

I am glad you sorted it out!

Also I feel relieved that my hunch about btrfs was wrong all along :)

Locked

Return to “Docker”