Fresh Resin OS 2.14.3+rev1 fails to install container


#1

A fresh downloaded image of Resin OS 2.14.3+rev1 on the QEMU (64bit) architecture is failing to download our container. An older OS (such as v2.13.6) continues to work fine. It infinitely loops with the following error:

21.08.18 17:25:46 (+1000) Downloading image 'registry2.resin.io/v2/589b1a2881c03d18feb4d466dba14bc8@sha256:a0676f63ca0a7224befd87d5cb6643b517b546705f3f05e66151e48dd0d0d70e'
21.08.18 17:31:10 (+1000)  Failed to download image 'registry2.resin.io/v2/882eaff7631f943eb125e80d91abea1d@sha256:b225dd37708baf14700773ee599e260784836e739d52d45af105df3e7c4f567e' due to 'failed to register layer: Error processing tar file(exit status 1): write /usr/lib/openssh/ssh-keysign: cannot allocate memory'
21.08.18 17:35:09 (+1000) Failed to download image 'registry2.resin.io/v2/589b1a2881c03d18feb4d466dba14bc8@sha256:a0676f63ca0a7224befd87d5cb6643b517b546705f3f05e66151e48dd0d0d70e' due to 'failed to register layer: Error processing tar file(exit status 1): write /usr/lib/postgresql/9.4/bin/postgres: cannot allocatememory

An example device is available 7f0fee0c43e2e7660976435880e2eea5 with support access granted. This device has been provisioned with 700mb memory.

We also tested at 2GB which failed with a similar message of:

Failed to download image 'registry2.resin.io/v2/3a10b846c241512b28fc2d08ecbc1e4d@sha256:c8396cd326f154987f6eb3494b641f6756a164b6815ba6c4eae94e5bdbf1cc92' due to 'failed to register layer: Error processing tar file(exit status 1):'

Update to Resin OS version 2.14.3+rev1 does not loads the app container
Failed to download image from registry
Failed to download image on Resin OS 2.14.3+rev1 factory build
#4

After 5 hours of looping with this error it did eventually install.


#6

Please note this is not “resolved” new QEMU devices showing this error and taking 5+ hours to complete their first install is not desirable behavior.


#10

Hey @SplitIce I tried to replicate this, our QEMU device downloaded the application right away.

Do you have a device that is currently experiencing the issue? The device that you mentioned doesn’t seem to exist anymore.

Also, for the 2GB layer one: how big is your image? The QEMU image by default has ~1GB space for the docker storage. The message seems as if it was running out of space. If it’s running out of space, you can take QEMU image that you use (when it’s not running) and add more space, e.g. adding ~5GB storage:

qemu-img resize resin-appname-qemux86-64-2.14.3+rev1-v7.16.6.img +5G

where you replace the resin-appname-qemux86-64-2.14.3+rev1-v7.16.6.img with your image’s name. But this is just the hunch, and only relevant to the second part of the issue you mentioned.


#12

The container is around 800mb and it’s complaining about being out of memory not ram not disk (not that it actually seems to be using all that much ram when checked after the errors). The disk on one test device was 32GB.

@brdrcol (colleague) has also tested some VM’s with stupidly enlarged specifications with no luck.

The device (7f0fee0c43e2e7660976435880e2eea5) also definitely still exists, I just confirmed as much in the dashboard and is running.


#13

(Responding to Failed to download image from registry)

Hi @imrehg,

Previously I converted the QEMU disk to VirtualBox vdi with qemu-img, increased the disk size to 8GB and memory to 6000MB and let it download/fail for 7 hours. I can’t run that VM at the moment.

I have started a new device (a64a7fb0f4d50f882bd2d44d808a94f6) with plain QEMU on Linux and granted support access. Commands were:

qemu-img resize resin-testappqemu-qemux86-64-2.14.3+rev1-dev-v7.16.6.img +8G
qemu-system-x86_64 -drive file=resin-testappqemu-qemux86-64-2.14.3+rev1-dev-v7.16.6.img,media=disk,cache=none,format=raw -net nic,model=virtio -net user -m 8192 -nographic -machine type=pc,accel=kvm -smp 4 -cpu host

It has done two download/fail iterations so far:

`24.08.18 00:19:08 (+1000) Failed to download image [...] due to 'failed to register layer: Error processing tar file(exit status 1): '`
`24.08.18 00:24:52 (+1000) Failed to download image [...] due to 'failed to register layer: Error processing tar file(signal: killed): '`

#14

@brdrcol have you tried using the Intel NUC image used with VirtualBox?

On the actual issue, we are checking it out, and have some leads, but not quite there yet.


#17

@imrehg no we havent. The process however is the same as converting a QEMU image. We also use KVM (QEMU compatible cpu) fyi so the QEMU distribution is a natural fit.


#18

@imrehg has any progress been made this week?


#19

Hey @SplitIce, I think a new resinOS version is coming out soon with the fix included for this issue, fixed in balena over the weekend. Will keep you posted!