NTP and GPS timing details

The Raspberry Shake’s timestamping is based on NTP, where the NTP daemon program governs the local Raspberry Pi computer clock on a permanent basis, assuming a connection to the Internet.

Contents

• NTP and GPS timing details

  • Case 0 (normal mode of operation): When the Raspberry Shake does have a connection to an NTP server at boot-up

  • Case 1: No Internet connection at startup

  • Case 2: Connection to NTP lost for some time

  • Case 3: Connecting a GPS module to the computer via USB

  • Details for NTP power users

  • GPS Timing Basics

  • Leap seconds and the Raspberry Shake

  • Diagnostic information you can provide us if things go wrong

  • Is GPS timing necessary?

  • mini-SEED Data Timing Quality Parameter

Warning

Seismic data standards require that data be collected in UTC time only. Do not attempt to change the Raspberry Shake’s timezone or system time (outside of NTP) as it could result in unforeseen problems down the road.

Case 0 (normal mode of operation): When the Raspberry Shake does have a connection to an NTP server at boot-up

Under normal operating circumstances, a Raspberry Shake will have access to the internet, and thus provide constant (more or less) access to an NTP server. The computer clock will be properly set and the NTP daemon started. The NTP daemon keeps the computer clock in sync with real time over all time, constantly making tiny adjustments that are hardly noticable. It does not do something like wake up every so often to check the clock and reset it if necessary. All you really need to know is that if the unit is connected to the internet, and the NTP daemon has a connection to an NTP server, the clock will remain in sync within a few milliseconds of accuracy at all times. But here are some more details:

Once the system is started, the program responsible for timestamping the actual data packets reads the local Raspberry Pi clock, using this time to stamp the actual data packets delivered by the micro-chip for onward forwarding.

During ongoing data processing, constant adjustments between the data packet processing program and the local clock are maintained, guaranteeing that any data sample will be within one sample interval of time.

For the different types of Raspberry Shakes, this means the timestamps will be within a margin of error based on the instrument’s sample rate, defined for each as:

• The original Raspberry Shake (RS1D V4/V5) - (data delivered @ 50Hz) - 20 Msecs

• Raspberry Shake (RS1D V6+) - (data delivered @ 100Hz) - 10 Msecs

• RS3D, RS4D, RJAM, RBOOM, RS&BOOM models - (data delivered @ 100Hz) - 10 Msecs

Some things to keep in mind:

• This accuracy very much depends on NTP maintaining access to the Internet to keep the local Raspberry Pi clock as close to real-time as possible. That is to say that timing is only good as NTP’s ability to govern the local clock itself.

• After the NTP daemon has been running for a week, it has “learned” how the local Raspberry Pi clock drifts and becomes very good at maintaining good time even when not connected to the Internet for long periods of time. (of course, this assumes conditions affecting clock drift remain mostly constant when not connected to an NTP server).

Case 1: No Internet connection at startup

When the Raspberry Shake starts, the computer clock will be set to the same time as when it was last powered down, either one minute ago (in case of a simple reboot command), or many months ago (in case of receiving the Raspberry Shake for the first time, for example).

If an NTP server is not available, the timing will start incorrect and will remain incorrect until an internet connection is found and the Raspberry Shake rebooted. When the computer time is very much different from real time, the computer must be rebooted in order for NTP to reset the clock successfully, i.e., resetting the clock can only happen during the computer’s boot-up sequence. Please note: this is a Raspberry Pi feature, not a Shake feature.

Data will still flow locally and use the system time for time stamping (It always uses the computer clock, whether NTP is running or not. The difference is if the clock disciplined by NTP). If there is no NTP, however, Raspberry Shake will recognize that the timing quality has been compromised and will not stream the data to the community server.

If at some later time an internet connection is detected, the unit will reboot. After reboot, NTP timing will behave as described above in Case 0. Note: internet detection and system reboot is automatic, i.e., it is not necessary for the end-user to manage this state change themselves.

Note

This also applies to: stand-alone mode of operation (see Offline and stand-alone applications (like classroom demos)) where the unit is installed in the field with no GPS antenna and no Internet

Case 2: Connection to NTP lost for some time

It is not necessary for the NTP client to maintain a permanent connection to the internet for it to do its job. The reason for this is that the NTP client has become very smart since its general adoption back in 1985. In that, once it’s been up and running for about a week, it determines the average drift on the local clock and will continue to apply adjustments to keep it accurate even in the absence of a connection to an NTP server. In other words, if connection to an NTP server is lost for some time, timestamping will drift exactly in step with the drift of the local Raspberry Pi clock itself and the NTP client will continue to govern the local Raspberry Pi clock according to what it has learned over time in terms of its specific drift rate, keeping relatively good time. And then once a connection is re-established, the clock will be adjusted again absolutely, while the average drift calculation is further refined. What this means is that the longer NTP executes, the more accurate it becomes when it cannot talk to a server.

When an Internet connection is restored, and NTP makes a large adjustment to the local Raspberry Pi clock (large defined as >= one sample interval of time), the data processing program will do a hard reset of the data packet timestamp, thus forcing a timestamp re-sync with the local clock. This will necessarily result in a time-tear, either a gap or an overlap, depending on the direction of drift, which will be reported at /opt/log/odf_SL_plugin.info.

We commonly are asked, how much drift can one expect when there is no external clock that can be referenced to govern the shake computer’s clock? Unknown. The Raspberry Pi clock is governed by a 20 ppm crystal, so ~2 seconds/ day.

Case 3: Connecting a GPS module to the computer via USB

This is done also in conjunction with NTP, where:

• the GPS module sends a PPS (pulse per second) to the computer across some hardware channel

• a GPS daemon, gpsd, receives the PPS command, processes it and sets a time inside of shared memory (RAM)

• the NTP daemon, in the meantime, has been configured to query the same location in shared memory to retrieve the time and set the clock and to keep it in sync, just as it does when querying an NTP server

So, to the end-user, there is no difference in terms of using the NTP daemon whether connecting to an NTP server, or to shared memoryodf (which is being updated by a PPS generated by an attached GPS module). The only difference is the NTP configuration file.

If either or both situations can happen, (that an NTP server might be available, and / or that a GPS module is attached), it is also standard to configure NTP to retrieve its time from both the GPS PPS and an NTP server as a fallback. This is desirable in the case when the GPS module may lose contact with the satellite for a time, where NTP can then fall back to using an NTP server.

ver.9 added the functionality needed to use the PPS from an attached GPS module (where the user will only be required to do the physical attachment of the module).

You can check that GPS is being used by the NTP client / daemon with the following command:

$ ntpq -p

Which will return a list of the NTP servers the daemon knows about (both local GPS and internet-based NTP servers), also indicating which server it is using.

For example:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 SHM(1)          .PPS.            0 l    -   16    0    0.000    0.000   0.000
+173.0.156.209   152.2.133.53     2 u   29   64  377   99.865    0.380  13.118
 resolver2.skyfi .STEP.          16 u    -  512    0    0.000    0.000   0.000
*cake.aseriesoft .PPS.            1 u   54   64  377  113.915    2.277   2.854
+li216-154.membe 45.56.123.24     3 u   45   64  377  100.474    3.520   3.611
 SHM(0)          .GPS.            0 l    -   16    0    0.000    0.000   0.000

the SHM() entries are the GPS module servers

the line with * is the server that the NTP daemon is currently using

the line with + is a server that is a valid candidate if the NTP daemon needs to change to another

the order of the servers listed is irrelevant

the ‘st’ column is the stratum level number, the lower the better (GPS = 0)

In this example, the GPS module is configured, but not connected, so it is not being used. if GPS were connected, the * would be next to the SHM(1) entry.

Note

Many (but not all) GPS devices have PPS (pulse per second) capability. The Shake works best with devices that do have this capability, as it works to keep the clock from drifting exactly once per second.

Details for NTP power users

There are two services that make up NTP:

• ntpdate

• ntpd

where ntpdate is started first (to set the clock) and ntpd is started after ntpdate completes. You can stop and restart the ntp daemon at will using the systemctl commands stop, start, and restart. It is not necessary to reboot the machine. For example, you can restart ntpd and ntp services using the following sequence of commands:

$ sudo systemctl stop ntp
$ sudo systemctl restart ntpd
$ sudo systemctl start ntp

NTP servers can be specified at /etc/ntp.conf. To have the best chance of consistently maintaining accurate time, it’s worth editing ntp.conf to choose at least four time servers in the pool that’s closest to you. See https://www.ntppool.org for a list of the zones, so you can pick the nearest one. Why four time servers, you might ask?- If the NTP daemon has access to one clock, it will know exactly what time it is, but that clock might be wrong. If there are two clocks, it can determine whether they agree to a reasonable precision, but if they disagree, it cannot tell which one is right. If it has three clocks, it can tell when one of them is out of sync and disregard it, as long as the other two are well-synchronized. With four clocks, it can throw out the outlier, and still have three closely clustered clocks, so it can sync with the median one, or maybe the one with the lowest jitter. The more clocks the NTP daemon is looking at, and the closer they are to your Raspberry Shake, the more likely it is that NTP’s algorithms will give a good result. Having said that, the NTP algorithms are good, and the demands of this type of seismology are reasonably low, so even with the default ntp.conf, you are likely to be OK as long as you have a reasonably good network connection. When there is a good internet connection, the time should be fine within the 10ms range. Especially, once ntpd runs for a number of days, it will average out most short-time variances.

NTP works on port 123 and it needs to have both TCP and UDP protocols open in both directions.

GPS Timing Basics

When a GPS is connected to any Shake model, NTP data is acquired and processed as follows:

1. if the GPS module is locked to at least one satellite, then the computer will be using the GPS time to set the computer clock

2. the computer clock is adjusted by the NTPD daemon, not by any Shake software

3. if the GPS module loses contact with satellite(s) for too long, and then after some time regains the connection, it will return to a locked state, where NTPD will adjust the computer clock accordingly

4. this may, or may not, result in a time tear in the data. If a time-tear does occur, this will be noted in the output log file /opt/log/odf_SL_plugin.err

Leap seconds and the Raspberry Shake

Since the Raspberry Shakes are controlled by NTP, the occasional leap second adjustment will occur according to the rules dictated by both the interaction between the NTP daemon (client program running on the Raspberry Pi computer) and the operating system (Raspbian, in this case), where there are three options: jump, hold or slew (details on these methods here).

Regardless the method used, the effect of adding a leap second will cause a discontinuity in the data, i.e., one or more gaps that will bring both the data timestamp and clock time back in-line with each other.

Diagnostic information you can provide us if things go wrong

1. Contents of /etc/ntp.conf

2. Output of command ‘ntpq -p’

3. Output of command ‘sudo journalctl -xn’ if and when ‘restart ntpd’ fails

4. The log files which can be downloaded from the web front-end

Is GPS timing necessary?

Where GPS timing is understood to mean +/- 0 ms, or no timing error.

The answer to this question is application-specific and ultimately determined by you, the end user. But let’s put this into perspective. If you are locating earthquakes a Geophysical Institute or Observatory setting, your earthquake location errors will come from these sources:

Source of Location error	Estimated error	Comments
Instrument timing	GPS: 0 Raspberry Shake NTP: +/- 1 sample, or +/-10 ms at 100 sps
Picking of onsets	100’s of ms	A conservative estimate for someone who is an expert picker and is not tired, so already an order of magnitude more than the uncertainty above
Velocity Model	1000’s of ms	Another order of magnitude more

Other sources of error include metadata (latitude, longitude, elevation).

For the Raspberry Shake, since all models operate at 100 sps, the timing quality is +/- 10 ms for all products. To give you an idea what this means, consider an earthquake’s P-wave traveling at a very fast 6000 m/s. With an uncertainty of +/- 10 ms, this translates into a location uncertainty of only +/- 60 meters. For lower P-wave velocities, the uncertainties diminish. This uncertainty is well within a reasonable margin of error for locating earthquakes where even for the best locations, the error is measured in kilometers. So, if everything else was perfect, an error of 60 meters would be a stellar result.

If excellent relative timing for active seismic studies is needed, for example, then it is possible to improve the relative timing precision by setting up a computer as a daemon NTP server and slaving all other Raspberry Shakes to it.

The computer that would run this local NTP server can be any computer whatsoever, not necessarily a Raspberry Shake, and not necessarily a Raspberry Pi even.

The basic idea follows these steps:

1. a computer with a connected GPS device that governs its HW clock

2. running an NTP server on the same computer (not the client daemon) to which a Raspberry Shake will be connected to use as its NTP server

In this way, it is possible to have a single NTP server that will regulate the timing of all the devices connected to it, instead of having multiple NTP connections for each device. This will provide an improved level of accuracy, ensuring that all connected devices are synchronized with each other, and all their data can be promptly used together without the need to check for the aforementioned syncronization.

mini-SEED Data Timing Quality Parameter

In summary,

For all cases described below, a timing quality (TQ) value is set when constructing the data packets and forwarded to the data destination(s). This parameter is used to describe the timing accuracy of the data and can change over time. For Raspberry Shake instruments, the Raspberry Pi computer clock is always governed by NTP, even when a GPS module is attached. When NTP is determined to be ON, this value is set to 90 (normal values are between 0 and 100, where 100 = time-stamping is governed by GPS). In the case when NTP is OFF, (i.e., when NTP is unavailable at system boot), this value will be 0.

Since NTP goes from OFF to ON exactly once, (and remains ON, regardless connect / disconnect actions with NTP servers), this means that the timing parameter value will also go from 0 to 90 in the case when NTP is unavailable at system boot, but at some later time comes ON. Also to note, if NTP is OFF at system boot, no data will be forwarded to the data server until NTP comes ON, when the system is configured to forward data.

For additional details read on,

• 100 = GPS LOCKED

  If GPS unlocks, the TQ value will reduce by 1 every 24 hours, until it reaches 91, where it will “stick”. Once GPS ever re-locks, TQ will return to 100.

• 90 = NTP LOCKED

  NTP daemon is locked and the Raspberry Shake data acquisition software has successfully sync’ed with the MCU (microcontroller unit) clock, meaning that data timestamps are guaranteed to be within one sample interval of time, i.e., 10msecs for 100Hz data.

• 40 = NTP UNLOCKED (previously 45)

  NTP daemon is locked, but the Raspberry Shake data acquisition software is unable to sync with the MCU clock. For this case, you should assume the timestamp might be close, but not necessarily accurate. This should be a very rare case since this implies communications between the Raspberry Pi computer and Raspberry Shake board are not stable

• 0 = NO LOCK

  There is no lock to an external clock, neither GPS nor NTP. In this case, the Raspberry Shake data acquisition software will only forward the data to the local SeedLink server on the Raspberry Pi computer itself, and will refuse to forward this to the data server. As such, you should not see any incoming data with the TQ value = 0 on the Raspberry Shake Community Server.