Lock file is not preventing updates


#1

Hi!

We use /data/resin-updates.lock to have more control over when to run updates on devices (see for example https://resin.io/blog/scheduling-updates-with-resin-io/).

But the last couple of times we updated (did a git push), we saw devices being updated which had the /data/resin-updates.lock file.

We checked and the “ENABLE LOCK OVERRIDE” button was not enabled (it was yellow). Another time we had some with “enable lock override” enabled and some not, and they were all updated regardless.

Is this a bug, or did the functionality for this change?

Thanks!!
Flavia


#2

Hi @flavia,
We’ve had some issues with the update lock in the past, especially if the devices reboot. What host OS and supervisor versions are your devices using?
It also depends on how you’re taking the lock - if you’re not using an exclusive open it’s possible that the supervisor has taken the lock before. What procedure are you using to take the lock? Could you maybe share the part of your code that handles that?

We’re rolling out a new update lock with supervisor 4.0.0, the lock location will change but it will make it way more robust.


#3

Hi Pablo,

Ah, sorry, forgot to mention the versions we are using :slight_smile:

On two devices where it happened we are using Resin OS 2.0.0-rc1.rev1, supervisor version 4.

But I think the other one we are using Resin 1, not 100% sure.

I’m not sure what you mean by this:

It also depends on how you’re taking the lock - if you’re not using an exclusive open it’s possible that the supervisor has taken the lock before.

To enable locking, we create a file: /data/resin-updates.lock, that’s all. It’s always there, and to perform an action we Enable lock override -at least that’s the idea.

So for what we did in this case there is no code involved.

Does it make sense?

Thanks,
Flavia


#4

@pablo @flavia I believe the other device was Supervisor 2.7.1 and Resin OS 1.19.0.


#5

@flavia @fokko can try out lockfile-create (from mail-lock, it should be available from lockfile-progs on Debian for example):

#!/bin/bash

# For Supervisor >=4.0.0
LOCKFILE=/tmp/resin/resin-updates
# For supervisir <4.0.0
# LOCKFILE=/data/resin-updates
lockfile-create ${LOCKFILE}
lockfile-touch ${LOCKFILE} &
# Save the PID of the lockfile-touch process
LOCKING_PID="$!"

# Do the things that that shouldn't be interrupted
while : ; do
  echo "Important stuff with lock (PID ${LOCKING_PID})"
  sleep 5
done

Would kill/remove lockfile with:

kill "${LOCKING_PID}"
lockfile-remove ${LOCKFILE}

There are some more examples in the documentation, but we’ll add more info in the future too, locking should become better & more usable.


#6

@flavia, @fokko check the recently updated docs - in supervisor v4.0.0 there was a breaking change and the lock is now at /tmp/resin/. The old lock location will only be supported in ResinOS 1.X. So if you’re switching to 2.0, you should try the new lock location as the old one (/data/) will be ignored. Hope this clarifies the mystery!

I also recommend you follow @imrehg’s advice. Simply creating the file causes the risk that the supervisor can be trying to take the lock at the same time, so you can’t be sure that the lock is yours unless you use something to create the lockfile exclusively. Also ensure that your app always takes the lock immediately after startup, as the lock is always cleared automatically on reboot.


#7

Thanks! That explains a lot :slight_smile:

I have a question about the way we are using the lock.

We have many devices in one application, and we want to be able to have full control about when we update them. So we want to update them one by one, make sure everything goes alright, go on to the next, etc.

So what we (want to) do: all the devices have the resin-updates.lock by default. We set it up in the Dockerfile with a command like this:

CMD touch /data/resin-updates.lock;*

Then we run a (python) script which uses the Resin API to iterate over an application’s devices and prompts the user to update each device. For that, it calls supervisor.update forcing a lock override to update each device… And we do this one device at a time.

This is actually based on a recommendation from @petrosagg if I remember well, but maybe things have changed in the Resin architecture for Resin 2, and this is not the right way to go any more.

So is this a good way of solving this, or should we be doing something else? Is it ok to have the lock permanently except if we want to update? What is the Resin-recommended way of updating devices one by one?

  • Actually from what I read from a previous message in this post, I learned we have to acquire the lock in another way. But is it possible to do it from the Dockerfile?

(I can post this a separate thread if that’s better :slight_smile:)

Thanks!!


#8

@flavia,
Yeah that way of having control of updates is still the way to go (only with the new lock at /tmp/resin/ now). We’re thinking of better ways of enabling this, but the update lock is still the best mechanism for now.

The only alternative that might be useful is to use two separate apps, and always push to the other app and then move the devices to the other app when you want to trigger the update. (If you do this my recommendation is to not use the lock at all, as it’s been known to cause trouble).

If you stick to using the lock, it’s okay to keep the lock permanently, and doing it from the Dockerfile is fine. Using touch will probably work in your case as it would be unusual for the Supervisor to try to take the lock immediately after starting the app - but it’s still safer if you use the method recommended by @imrehg. If it’s too complex to put inline in the CMD, you can always have a custom bash start script that does that (and starts your app), and use that as the CMD.