/mnt/data/docker/aufs/diff is full


#1

Hey, we are running into a weird issue, one of our container filled up on /mnt/data/docker/aufs/diff

Similar issue reported on /mnt/data full; what to do?

It’s at about 29GB at the moment and the apps won’t start. Would you know why things would endup in that state?

Thanks,
Brice


#4

Hi @brice,

Can you please let us know what resinOS version your device is running?

Thanks!


#5

@lekkas sure, the box was on Resin OS 2.12.5+rev1


#10

Same here.

One of our devices running Resin OS 2.12.7+rev1 is filling up on /mnt/data/docker/aufs/diff. Then the folder seems to purge itself after a while, only to start filling up again.

All the other devices in our fleet behave as expected and run the same version of ResinOS, as can be seen here:

firefox_2018-06-21_09-37-10

I just confirmed with a mix of df and du commands that /mnt/data/docker/aufs/diff is the only folder filling up. All other things are equal across the fleet.


#11

Hey @lv82,

Thank you very much for reporting this. This is an issue we’ve been chasing for a while, we’ll follow up over email so that you can share some more information about your application.


#12

@brice I’m also reaching out to you over email as well (since you reported the original issue :slight_smile: so that you can give us more information about your application. Thanks!


#13

Hey Laurent, what are you using to graph disk usage?


#14

A slightly customized version of the monitoring dashboard from https://resin.io/blog/prometheusv2/

The Disk Fullness metric is provided out of the box with the shipping dashboards.


#16

Awesome, thanks


#17

Hi @brice, @lv82 (and anyone who may bump into this)

After checking @lv82’s affected device, we found out that the underlying cause was an application script that failed to mount an SD card. The application code continued writing data anyway, but that data ended up in the container’s filesystem, (instead of the SD, because of the failed mount), which caused /mnt/data/docker/aufs/diff to fill up.

So it looks like similar application-specific issue patterns are definitely worth looking out for, in cases where device storage is filling up unexpectedly.

Thank you @brice and @lv82 for helping us with the investigation.


#19

@lekkas nice one, yea we do have a script to mount an external drive and record video, probably related.