NTP and GPS timing details

The Raspberry Shake’s timestamping is based on NTP, where the NTP daemon program governs the local Raspberry Pi computer clock on a permanent basis, assuming a connection to the Internet.

Case 0 (normal mode of operation): When the Raspberry Shake does have a connection to an NTP server at boot-up

Under normal operating circumstances, a Raspberry Shake will have access to the internet, and thus provide constant (more or less) access to an NTP server. The computer clock will be properly set and the NTP daemon started. The NTP daemon keeps the computer clock in sync with real time over all time, constantly making tiny adjustments that are hardly noticable. It does not do something like wake up every so often to check the clock and reset it if necessary. All you really need to know is that if the unit is connected to the internet, and the NTP daemon has a connection to an NTP server, the clock will remain in sync within a few milliseconds of accuracy at all times. But here are some more details:

Once the system is started, the program responsible for timestamping the actual data packets reads the local Raspberry Pi clock, using this time to stamp the actual data packets delivered by the micro-chip for onward forwarding.

During ongoing data processing, constant adjustments between the data packet processing program and the local clock are maintained, guaranteeing that any data sample will be within one sample interval of time.

For the different types of Raspberry Shakes, this means the timestamps will be within a margin of error based on the instrument’s sample rate, defined for each as:

  • The original Raspberry Shake (RS1D V4/V5) - (data delivered @ 50Hz) - 20 Msecs
  • Raspberry Shake (RS1D V6+) - (data delivered @ 100Hz) - 10 Msecs
  • RS3D, RS4D, RJAM, RBOOM, RS&BOOM models - (data delivered @ 100Hz) - 10 Msecs

Some things to keep in mind:

  • This accuracy very much depends on NTP maintaining access to the Internet to keep the local Raspberry Pi clock as close to real-time as possible. That is to say that timing is only good as NTP’s ability to govern the local clock itself.
  • After the NTP daemon has been running for a week, it has “learned” how the local Raspberry Pi clock drifts and becomes very good at maintaining good time even when not connected to the Internet for long periods of time. (of course, this assumes conditions affecting clock drift remain mostly constant when not connected to an NTP server).

Case 1: No Internet connection at startup

When the Raspberry Shake starts, the computer clock will be set to the same time as when it was last powered down, either one minute ago (in case of a simple reboot command), or many months ago (in case of receiving the Raspberry Shake for the first time, for example).

If an NTP server is not available, the timing will start incorrect and will remain incorrect until an internet connection is found and the Raspberry Shake rebooted. When the computer time is very much different from real time, the computer must be rebooted in order for NTP to reset the clock successfully, i.e., resetting the clock can only happen during the computer’s boot-up sequence. Please note: this is a Raspberry Pi feature, not a Shake feature.

Data will still flow locally and use the system time for time stamping (It always uses the computer clock, whether NTP is running or not. The difference is if the clock disciplined by NTP). If there is no NTP, however, Raspberry Shake will recognize that the timing quality has been compromised and will not stream the data to the community server.

If at some later time an internet connection is detected, the unit will reboot. After reboot, NTP timing will behave as described above in Case 0. Note: internet detection and system reboot is automatic, i.e., it is not necessary for the end-user to manage this state change themselves.

Note

This also applies to: stand-alone mode of operation (see Direct connections for offline applications like classroom demos) where the unit is installed in the field with no GPS antenna and no Internet

Case 2: Connection to NTP lost for some time

It is not necessary for the NTP client to maintain a permanent connection to the internet for it to do its job. The reason for this is that the NTP client has become very smart since its general adoption back in 1985. In that, once it’s been up and running for about a week, it determines the average drift on the local clock and will continue to apply adjustments to keep it accurate even in the absence of a connection to an NTP server. In other words, if connection to an NTP server is lost for some time, timestamping will drift exactly in step with the drift of the local Raspberry Pi clock itself and the NTP client will continue to govern the local Raspberry Pi clock according to what it has learned over time in terms of its specific drift rate, keeping relatively good time. And then once a connection is re-established, the clock will be adjusted again absolutely, while the average drift calculation is further refined. What this means is that the longer NTP executes, the more accurate it becomes when it cannot talk to a server.

When an Internet connection is restored, and NTP makes a large adjustment to the local Raspberry Pi clock (large defined as >= one sample interval of time), the data processing program will do a hard reset of the data packet timestamp, thus forcing a timestamp re-sync with the local clock. This will necessarily result in a time-tear, either a gap or an overlap, depending on the direction of drift, which will be reported at /opt/log/odf_SL_plugin.info.

We commonly are asked, how much drift can one expect when there is no external clock that can be referenced to govern the shake computer’s clock? Unknown. The Raspberry Pi clock is governed by a 20 ppm crystal, so ~2 seconds/ day.

Case 3: Connecting a GPS module to the computer via USB

This is done also in conjunction with NTP, where:

  • the GPS module sends a PPS to the computer across some hardware channel
  • a GPS daemon, gpsd, receives the PPS command, processes it and sets a time inside of shared memory (RAM)
  • the NTP daemon, in the meantime, has been configured to query the same location in shared memory to retrieve the time and set the clock and to keep it in sync, just as it does when querying an NTP server

So, to the end-user, there is no difference in terms of using the NTP daemon whether connecting to an NTP server, or to shared memory (which is being updated by a PPS generated by an attached GPS module). The only difference is the NTP configuration file.

If either or both situations can happen, (that an NTP server might be available, and / or that a GPS module is attached), it is also standard to configure NTP to retrieve its time from both the GPS PPS and an NTP server as a fallback. This is desirable in the case when the GPS module may lose contact with the satellite for a time, where NTP can then fall back to using an NTP server.

ver.9 added the functionality needed to use the PPS from an attached GPS module (where the user will only be required to do the physical attachment of the module).

You can check that GPS is being used by the NTP client / daemon with the following command:

$ ntpq -p

Which will return a list of the NTP servers the daemon knows about (both local GPS and internet-based NTP servers), also indicating which server it is using.

For example:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 SHM(1)          .PPS.            0 l    -   16    0    0.000    0.000   0.000
+173.0.156.209   152.2.133.53     2 u   29   64  377   99.865    0.380  13.118
 resolver2.skyfi .STEP.          16 u    -  512    0    0.000    0.000   0.000
*cake.aseriesoft .PPS.            1 u   54   64  377  113.915    2.277   2.854
+li216-154.membe 45.56.123.24     3 u   45   64  377  100.474    3.520   3.611
 SHM(0)          .GPS.            0 l    -   16    0    0.000    0.000   0.000

the SHM() entries are the GPS module servers

the line with * is the server that the NTP daemon is currently using

the line with + is a server that is a valid candidate if the NTP daemon needs to change to another

the order of the servers listed is irrelevant

the ‘st’ column is the stratum level number, the lower the better (GPS = 0)

In this example, the GPS module is configured, but not connected, so it is not being used. if GPS were connected, the * would be next to the SHM(1) entry.

Details for NTP power users

There are two services that make up NTP:

  • ntpdate
  • ntpd

where ntpdate is started first (to set the clock) and ntpd is started after ntpdate completes. You can stop and restart the ntp daemon at will using the systemctl commands stop, start, and restart. It is not necessary to reboot the machine. For example, you can restart ntpd and ntp services using the following sequence of commands:

$ sudo systemctl stop ntp
$ sudo systemctl restart ntpd
$ sudo systemctl start ntp

NTP servers can be specified at /etc/ntp.conf. To have the best chance of consistently maintaining accurate time, it’s worth editing ntp.conf to choose at least four time servers in the pool that’s closest to you. See https://www.ntppool.org for a list of the zones, so you can pick the nearest one. Why four time servers, you might ask?- If the NTP daemon has access to one clock, it will know exactly what time it is, but that clock might be wrong. If there are two clocks, it can determine whether they agree to a reasonable precision, but if they disagree, it cannot tell which one is right. If it has three clocks, it can tell when one of them is out of sync and disregard it, as long as the other two are well-synchronized. With four clocks, it can throw out the outlier, and still have three closely clustered clocks, so it can sync with the median one, or maybe the one with the lowest jitter. The more clocks the NTP daemon is looking at, and the closer they are to your Raspberry Shake, the more likely it is that NTP’s algorithms will give a good result. Having said that, the NTP algorithms are good, and the demands of this type of seismology are reasonably low, so even with the default ntp.conf, you are likely to be OK as long as you have a reasonably good network connection. When there is a good internet connection, the time should be fine within the 10ms range. Especially, once ntpd runs for a number of days, it will average out most short-time variances.

NTP works on port 123 and it needs to have both TCP and UDP protocols open in both directions.

Diagnostic information you can provide us if things go wrong

  1. Contents of /etc/ntp.conf
  2. Output of command ‘ntpq -p’
  3. Output of command ‘sudo journalctl -xn’ if and when ‘restart ntpd’ fails
  4. The log files which can be downloaded from the web front-end

Is GPS timing necessary?

Where GPS timing is understood to mean +/- 0 ms, or no timing error.

The answer to this question is application-specific and ultimately determined by you, the end user. But let’s put this into perspective. If you are locating earthquakes a Geophysical Institute or Observatory setting, your earthquake location errors will come from these sources:

Source of Location error Estimated error Comments
Instrument timing

GPS: 0

Raspberry Shake NTP: +/- 1 sample, or +/-10 ms at 100 sps

 
Picking of onsets 100’s of ms A conservative estimate for someone who is an expert picker and is not tired, so already an order of magnitude more than the uncertainty above
Velocity Model 1000’s of ms Another order of magnitude more

Other sources of error include metadata (latitude, longitude, elevation).

For the Raspberry Shake, since all models operate at 100 sps, the timing quality is +/- 10 ms for all products. To give you an idea what this means, consider an earthquake’s P-wave traveling at a very fast 6000 m/s. With an uncertainty of +/- 10 ms, this translates into a location uncertainty of only +/- 60 meters. For lower P-wave velocities, the uncertainties diminish. This uncertainty is well within a reasonable margin of error for locating earthquakes where even for the best locations, the error is measured in kilometers. So, if everything else was perfect, an error of 60 meters would be a stellar result.

If you need excellent relative timing for active seismic studies, for example, then you can improve the relative timing precision by setting up a Raspberry Shake as the daemon NTP server and slaving all other Raspberry Shakes to it.