Resin OS Host Update Not Enough Space Beaglebone Black

beaglebone

#1

A clean upgrade from 2.7.5+rev2 to 2.12.3+rev1 cannot be performed on beaglebone black due to not enough space being available on the device.

root@788b700:/mnt/data/resinhup# cat upgrade-2.x.sh.20180509_145201.log
[000000001][LOG]Loading info from config.json
[000000001][LOG]Loading info from device-type.json
[000000001][LOG]Device type check: OK
[000000001][LOG]Target OS version "2.12.3+rev1" OK.
[000000001][LOG]VARIANT_ID: prod
[000000001][LOG]Host OS version "2.7.5+rev2" OK.
[000000001][LOG]Checking for manifest of registry.hub.docker.com/resin/resinos:2.12.3_rev1-beaglebone-black
[000000003][LOG]Manifest found, good to go...
[000000003][LOG]No resin-device-progress fix is required...
[000000003][LOG]No supervisor updater fix is required...
[000000003][LOG]hostapp-update command exists, use that for update
[000000004][LOG]No device-specific pre-update fix for beaglebone-black
[000000004][LOG]Do not have docker-host running; legacy mode
[000000004][LOG]Clean inactive partition
[000000004][LOG]Balena migration
[000000004][LOG]Stopping supervisor and related services...
2.12.3_rev1-beaglebone-black: Pulling from resin/resinos
cbf5856c503a: Pulling fs layer
cbf5856c503a: Ready to download
cbf5856c503a: Pull complete
Digest: sha256:7816c5a0128a341b5349359eea428c0df91bbc3b18ea010388f25e2b2165e566
Status: Downloaded newer image for registry.hub.docker.com/resin/resinos:2.12.3_rev1-beaglebone-black
/mnt/sysroot/active is not a mountpoint
[000000222][LOG]Starting hostapp-update within a container
time="2018-05-09T14:55:50.879506270Z" level=info msg="libcontainerd: new containerd process, pid: 21"
time="2018-05-09T14:55:52.964986341Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-05-09T14:55:53.018783065Z" level=info msg="Loading containers: start."
time="2018-05-09T14:55:54.274736858Z" level=info msg="Loading containers: done."
time="2018-05-09T14:55:54.845440697Z" level=info msg="Docker daemon" commit=73136d1985f28a34705f537e90636cf208734048 graphdriver(s)=aufs version=17.06.0-dev
time="2018-05-09T14:55:54.851482637Z" level=info msg="Daemon has completed initialization"
time="2018-05-09T14:55:54.969681667Z" level=info msg="API listen on /var/run/balena-host.sock"
Starting hostapp-update
Error processing tar file(exit status 1): write /607b01dee0d4e1d7337eccb9e9285b105223ce093f0c3ec46660de1e71e3a337/layer.tar: no space left on device
invalid reference format
[000000434][ERROR]Update based on hostapp-update has failed...
root@788b700:/mnt/data# du -sh *
4.0K    apps.json
753M    docker
16K     lost+found
124K    resin-data
32K     resinhup
root@788b700:/mnt/data# df -h
Filesystem                         Size  Used Avail Use% Mounted on
devtmpfs                           218M     0  218M   0% /dev
/dev/disk/by-partuuid/3a3bb6c2-03  295M  252M   23M  92% /
/dev/disk/by-label/resin-state      19M  199K   17M   2% /mnt/state
tmpfs                              244M  108K  244M   1% /dev/shm
tmpfs                              244M  4.6M  239M   2% /run
tmpfs                              244M     0  244M   0% /sys/fs/cgroup
tmpfs                              244M     0  244M   0% /tmp
tmpfs                              244M   68K  244M   1% /var/volatile
/dev/mmcblk1p1                      40M  506K   40M   2% /mnt/boot
/dev/mmcblk1p2                     295M  2.2M  273M   1% /mnt/sysroot/inactive
/dev/mmcblk1p6                     1.1G  756M  240M  76% /mnt/data

#8

Hi,

Could you share with us the device’s UUID and, as well, activate support access so we can take a look on the device?

Regards,
Andrei


#9

Hi, the issue is that /data (mmcblk1p6) doesn’t have enough space as it is for the 2.7.5->latest update (which requires some migrations, and has to use space on /data during the process, unfortunately, but only temporarily). In this case your device is a very low storage space BBB (2G eMMC, by the look of it?).

It can be fixed up, updated, and in future updates this won’t be an issue. I’ll send you a private message, can send us the UUID of the device, and we’ll update it for you.

Hope this helps!


#10

Thanks so much for your help. I’ve followed up via private message. I’d be helpful for the forum if you’d share the manual steps taken to resolve this issue in case any other users have the same problem.


#11

Is it possible to use beaglebone black with external SD storage instead of eMMC using resin OS? I’m not aware of how to accomplish that, but I’d gladly use SD storage as the boot volume if this was possible.


#12

The device is now fixed up, and should be all good. I think only people running into this will be:

  • very small storage device
  • updating from <2.9.0 to >=2.9.0 version

Neither is very common, and the later should not be the case for everyone who sets up devices now (and start at >=2.9.0)

The manual steps were:

  • start the device up, and have it in “online” state
  • log into the HostOS
  • stop the supervisor and updater:
systemctl stop resin-supervisor
systemctl stop update-resin-supervisor.timer
  • remove the running containers and images
docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -q)
  • then finally start the update from the dashboard as before, this time it should work.
  • upon reboot, the device will redownload the user application image, and :checkered_flag:

Hope this helps!

Unfortunately at the moment there’s no way to run the BeagleBone from external storage, but we do investigate things…


#13

The device is now fixed up, and should be all good. I think only people running into this will be:

  • very small storage device
  • updating from <2.9.0 to >=2.9.0 version

Neither is very common, and the later should not be the case for everyone who sets up devices now (and start at >=2.9.0)

The manual steps were:

  • start the device up, and have it in “online” state
  • log into the HostOS
  • stop the supervisor and updater:
systemctl stop resin-supervisor
systemctl stop update-resin-supervisor.timer
  • remove the running containers and images
docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -q)
  • then finally start the update from the dashboard as before, this time it should work.
  • upon reboot, the device will redownload the user application image, and :checkered_flag:

Hope this helps!

Unfortunately at the moment there’s no way to run the BeagleBone from external storage, but we do investigate things…


#14

Thanks for all your help. The device appears to be operating correctly now. It’s good to hear that this issue will no longer occur in future updates. Running off the SD card on BBB would resolve a number of issues I’ve had with the platform, namely that you cannot deploy moderately sized containers due to the amount of space taken up by the host OS. But I understand that this is a peculiar device which is not top priority.


#15

One thing you can try, if you are deploying images that are relatively large (i.e. larger than ~50% of the data partition, that is ~1.1GB in your case), to use the delete-then-download update strategy. That has a longer update cycle if you deploy an application, and more network traffic, but then you can use pretty much all the data partition (minus the supervisor, that is ~50MB at the moment) and your user data.

Just some hints :slight_smile: let us know if you try it out!


#16

That’s a great suggestion and not something i’ve considered yet because my current workload does function under download-then-kill (the default strategy) but I will keep that in mind in case I run into this issue again.