Unreliable time


#1

Hey folks,

Seeing some weird stuff on resinOS - I hope someone can help!

Sometimes, one of my device restarts (not sure why!) and it’s system time is totally off. E.g.

09.03.17 11:03:08 (-0800) Starting application 'registry.resin.io/...'
09.03.17 11:02:13 (-0800) Systemd init system enabled.
09.03.17 11:02:13 (-0800) systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
09.03.17 11:02:13 (-0800) Detected virtualization 'other'.
09.03.17 11:02:13 (-0800) Detected architecture 'arm'.
09.03.17 11:02:13 (-0800) Set hostname to <010e05e-010e05e>.
// sometimes later
17.07.17 10:49:43 (-0700) Application is already running registry.resin.io/...'

Notice the date - 09-03-2017!

The system time is corrected sometime later, but we’ve built some logic around the datetime in our initialization script that gets tripped because of this. Any idea why this might be happening on few of our devices?

The HostOS version for this particular device is “Resin OS 2.0.0-rc1.rev2”


#3

What device is that? It probably doesn’t have a battery, and the hardware clock resets itself if the board is powered down. Then, some time after boot, NTP sets the time over the network to a correct value.


#4

@daniel.lee and @michal I think it is also probably related to https://github.com/resin-os/resinos/issues/280 , since some of the necessary data for time keeping is not persisted through reboots.


Beta testers: resinOS 2.x updates!
#5

Hey @michal and @shaunmulligan thanks for your replies! I’m running my apps in RPi 3 - I do wonder if this issue is on resinOS not the Pi based on @shaunmulligan’s comment. Do you guys have additional thoughts?


#6

Hi @daniel.lee , as Shaun said, we believe this has been fixed starting with version v2.0.3.
Can you use the latest available version for your RPI3 and let us know if this issue is still persisting?


#7

@floion I can test it out in few days

Meanwhile, I’m still looking for a way around this! Few ideas

  1. Send d-bus command (https://docs.resin.io/runtime/time/) and wait until host OS completes at least one NTP sync
  2. Run ntpd on the container?

Is there any risk I should be aware of in running (2)?


#8

Hello Daniel,

There should be no problem with running ntpdate or ntpd in the container.


#11

Hey @daniel.lee would probably suggest running the check through dbus-send (direct link in the docs: Checking if device time is NTP synchronized), either with dbus-send directly or with another language like Python or Node.js, or anything that dbus communication libraries.

Would not recommend running other ntp clients inside the application, having multiple ntp services trying to modify time potentially may cause all kinds of issues.

What resinOS version are you running now on your devices?


#12

@imrehg Most of our devices are running resin OS 2.0.0.

We are using the dbus-send command to check whether NTP synchronization has completed or not. Usually, NTP sync within reasonable amount of time, thought it would be cool if we can FORCE the sync (I haven’t figured out how this can be done :frowning: ).


#13

We are working on resinOS 2.x device updates, and we have experimental tools to do that (e.g. potentially update your device from 2.0.0 to a newer version, likely the latest one, which has these fixes). Would you be interested maybe trying it out on some of your devices?

I’ve checked briefly, as a workaround might be able to set the time directly from the container, and be close enough for your use case so time syncing can happen on its own pace. E.g. using the trick of querying a public server, and getting the time from the HTTP response:

date -s "`curl -v --silent http://google.com/ 2>&1 \ | grep Date | sed -e 's/< Date: //'`"

This is obviously a hacky workaround and it’s not something I’d recommend, the best thing is using the newer version of resinOS, if that’s possible for you…


#14

@imrehg Really cool ideas there - thanks for suggesting them!

The resinOS update is a really cool idea - I’ve love to hear more about it. The way we are using Resin has two constraints I can think of:

  1. Our devices are connected via 3G network - we care about the amount of data it uses ($$)
  2. Our devices are located far away from us and from each other - an update gone wrong is going to cost a lot of man hours to fix!

Do you think our setup is still a good case to try out the experimental tool?


#15

Sure thing, we love coming up with ways to help you get your work done. :slight_smile:

Regarding the resin host OS updates - under these circumstances I would be hesitant to suggest to do that at this point if the network traffic amount is really sensitive. The update would be about ~150MB data transferred at the moment. We are working on reducing that, and apply delta updates to the host OS as well (ie. only transferring the changes), but it’s not there yet. :tractor:

Regarding the 2. point, updating resinOS 2.x is should be pretty robust, it was designed with updates in mind, and it’s much simpler than how resinOS 1.x updates work at the moment. We are doing a writeup of what exactly happens when an update is run, so it’s more transparent and easier to make choices of update or not to update.

Hope this helps, and let us know what do you think! Will keep you posted about future host OS updates as well, that would suit your circumstances!


#16

Hey that’s great - yall’s blog posts are always fun and informative. Looking forward to it.

Do keep us posted - ability to remotely update host OS will be a blast for us (and suuuper great if it’s gonna be small deltas).


#17

Confirmed, that time kept well on newer resinOS devices, just tried it on a 2.2.0+rev1 Raspberry Pi 3, and the host OS logs show this after a reboot:

Jul 28 05:23:00 raspberrypi3 systemd-timesyncd[560]: System clock time unset or jumped backwards, restoring from recorded timestamp: Fri 2017-08-04 10:16:53 UTC
Aug 04 10:16:53 raspberrypi3 systemd[1]: Time has been changed
Aug 04 10:16:53 raspberrypi3 systemd[1]: Started Network Time Synchronization.

And as mentioned, this is available since resinOS 2.0.3. If you have any devices to update to newer version, we are beta-testing updates, see more in the announcement :slight_smile:


#18

Hi,

Does anyone has a simpler method (like a Python script) that returns 1 or 0 depending if the NTP synchronization has completed? (“NTPSynchronized” parameter)?

Thanks!


#20

Hi @ymaia
You can use the code found in https://github.com/resin-io-playground/resin-python-ntp-dbus that internally uses dbus-send command and parses the result, as noted in the respective documentation page:
https://docs.resin.io/runtime/runtime/#checking-if-device-time-is-ntp-synchronized

The repo by default runs an sh script that sets a var and then runs the python script but you can make your code invoke the NTPQuery().get_ntp_syncronized() method directly.


#23

This issue should have been fixed in OS versions >2.12.4