What happens during a resin host OS update?

resinhup

#1

The ability to remotely update the entire host operating system (OS) of your device, rather than just the user application, is a core feature of resin.io. Performing updates may feel like an opaque process, so we thought it would be useful to break it down.

Since resinOS 1.x and 2.x behave somewhat differently in how they manage updates, we’ll examine the three main use cases. In all three cases, an updater script gets transferred to the device to handle the update process, as described in detail below.

Upgrading between resinOS 2.x versions

The first step the updater script performs includes a number of sanity checks, so that an update does not progress if something isn’t right. The checks include, among other things, whether the device is running a resinOS version that it can work with (i.e., not on 1.x), whether the script is running on the right device type, whether it can reach the Docker registry to get the target resinOS image, and whether that image exists.

Next, the supervisor is stopped so it does not inadvertently interfere with the update process. The user application is kept running to minimize downtime. The updater then uses Docker to pull the new resinOS image. When the pull succeeds, the spare root partition is formatted and the contents of the image are exported onto that partition. The modifications to the boot partition are also applied.

The updater script checks to see if the new resinOS version ships with a newer supervisor version than what the device currently runs, and, if so, the supervisor is updated.

Finally, the boot settings are modified so that on the next reboot the new root file system is used. As a last step, the device is rebooted.

Upgrading from resinOS 1.x to 2.x

There are quite a few changes between resinOS 1.x and 2.x, which makes this type of update the most complex of the three. The main differences that the updater has to account for are a different partition layout, a different Docker file system, and the switch from ConnMan to NetworkManager.

As mentioned above, the updater script starts with a number of cross-checks, the most important being whether the OS is running from the main or the secondary root partition. The latter would happen if the device has already been updated between 1.x versions. If it’s running from the secondary root partition, as a first step the script stops the user application, switches to read only mode, copies over the data to the main partition, switches the boot settings, and reboots the device to run from the main partition. It is then ready to retry the update process.

After the checks are run, both the supervisor and the user application are stopped and the containers and images are removed.

Next, the script downloads a couple of binary tools that are not present in 1.x systems but are required for the transition (mainly tools handling the ext4 file system).

The contents of the resin-data folder, where the contents of your application’s /data folder are stored, are backed up on the secondary root partition temporarily. This limits the amount of data that can be backed up automatically to about 170MB (compressed). If there’s more data than this, the update process will stop, but the device hasn’t been materially modified yet.

Next, the device’s file system is modified to match the setup required by resinOS 2.x. Docker is then restarted and pulls a new resinOS image, which is exported into a compressed archive. Finally, the new rootfs is populated, and resin-data is restored.

The WiFi settings are then migrated from the 1.x setup (in config.json) to the 2.x setup (as NetworkManager files). The migration also tries to create configurations from any WiFi Connect settings found.

The boot settings are switched so that the next time the device boots, it will do so from the resinOS 2.x system. The supervisor version is checked, and if the target resinOS version has a newer supervisor, then the device is marked for a supervisor update as well.

Finally, the device is rebooted. On the first boot it will necessarily come online without any Docker images (as the update process needed to remove the images). So when the device starts, it will first redownload the required supervisor, which then redownloads the user application. After this all the heavy lifting is done you should have a brand new resinOS 2.x device, with all the improvements from its former life as a 1.x device.

Upgrading between resinOS 1.x versions

The process for upgrading between 1.x versions mirrors the process used for 2.x versions, though the tooling is somewhat different.

The updater script first does a couple of cross-checks, stops the supervisor and user containers, and if everything looks okay, starts by checking if any supervisor updates are needed.

Then a resinhup Docker image is pulled, which contains all the tools and the secondary Python updater scripts for the main migration. Since the resinOS 1.x root filesystem is read-write, the resinhup updater scripts start with a fingerprint check to see if any files have been modified. If there have been any modifications, it will stop the update as not to unexpectedly overwrite any modifications.

If the fingerprint check succeeds, the updater will use Docker to pull the target resinOS image and export the relevant contents onto the spare root partition and the boot partition. Next, the updater runs any required data migrations (for example, to account for any changes to the location and contents of the config.json file). As a final step, this Python-based updater will switch the boot settings, so that next time the device boots, it will boot from the updated system.

From here, the original updater script takes back control, and finishes up any other changes required by the update. When all this is done, the device is rebooted into the new system.


Remotely update your host operating system on resin.io
#2

What happens regarding the bandwidth? How should I consider my data consumption on a sim card for GPRS connection? Can you give a hint what amount of Mb (Gb) data would be used to update the OS on the connected devices?
From my application I can assume the traffic they will generate. To see the whole picture of data consumption would be great. Since the devices will be connected through GPRS connections the data consumption needs to be planed.
Thanks.


#3

The amount of data transferred depends on the device types (the size of the hostOS varies somewhat between the versions)

  • the resinOS image is between 70-120MB
  • the supervisor is 40-70MB (though some updates downoad less, as docker layers between versions are cached, and newer versions are on the smaller end)
  • the 1.x->1.x updater tool is ~50MB (only downloaded for those updates)

We are working on reducing this constantly. For example with the release of balena we’ll be able to use delta updates for the host OS and the supervisor updates too, which should be better under all circumstances than the current base level.

Also, this other topic might be relevant to your interests, if you haven’t seen it yet: Updated bandwidth usage numbers


#4

I haven’t seen yet that topic, but it seems a good start for me. Thank you.


#5

Difficult to me:frowning_face:


#6

Some real life numbers: I just updated a Beaglebone Black on 4G including downloading our application container (339 MB) and the data consumption is :
550 MiB received
25 MiB send

This was an update from Resin OS 2.0.0+rev2 (prod)
to Resin OS 2.12.3+rev1 (prod)


#7

Hm, that is more than I’ve expected, and will be looking into it. The 2.12.3+rev1 image compressed is ~110MB, and shouldn’t + the supervisor is less than 50MB.
Have to see what else happened there, e.g. were there any retries due to network (if on 4G and not totally reliable network for some reason).

One thought is that the user application might have been downloaded as well, as 2.0.0+rev2 should be pulling differently than 2.12.3 (which now can use the authenticated registry). But e.g. that would not matter much with deltas, without deltas it might be substantial contribution.

Thanks for the real-life number update, and we’ll update it here after we check it again too.


#8

I think it is quite within margin, right? 110+50+339MB= 500Mb that is 476 MiB. Maybe the something downloaded twice. Anyway, If I encounter such a clear update another time, I’ll post it.

Another question: How can we see what is updated in the OSes. Other than Resin stuff, like which Yocto build is used and security patches etc.

Thanks!


#9

Yeah, good point regarding the numbers!

Regarding the changes, you mean something like the meta-resin/resinOS changelog at https://github.com/resin-os/meta-resin/blob/master/CHANGELOG.md ? For the individual device types there are specific changelogs as well, for example BeagleBone at https://github.com/resin-os/resin-beaglebone/blob/master/CHANGELOG.md, whihc are much smaller, as most changes occur on the resinOS level.
Is this the sort of thing you had in mind?