Raspberry Shake Data Producer UDP Port Output

Description

Currently, the data producer service of the Raspberry Shake sends the seismometer data to a couple of destinations: a local Seedlink server, used to support both Swarm and the local Heli-plots, and to the cloud data server, used as part of back-end processing of Raspberry Shake network data and to support the online global map of Raspberry Shake stations, viewable here.

With release v0.8, a new destination type is being added that will be of general interest to the Shake community of end-users: one or more UDP ports located on any computer your Shake unit is able to communicate with.

The Benefits

What this means is that you, the end-user, can connect a program directly to this UDP port and receive the data coming off the seismometer as quickly as it is being produced, i.e., in real-time. This has at least the following advantages: 1) this is now the quickest way of reading the data coming off the seismometer; 2) connecting to a real-time stream is now completely trivial; 3) and since there is no limit to the number of UDP ports that can be configured for output, multiple systems can receive the data in real time, e.g., a web-page and another program meant to analyze the data, for example.

So, for all the scientists out there, amateur and professional alike, who want direct and instant access to the data for your own purposes, this enhancement is for you!

Have fun, enjoy, and make something cool! (A new EQ detection algorithm, maybe?) And when you do, let everyone know about it, here: https://github.com/raspishake.

All sections below detail how to use this functionality, please let us know if you have any questions.

Implementation Details

The updated data producer service, by default, now writes to this new data stream on the local Shake Pi, Port 8888. If you want to attach to a single data stream on the Shake Raspberry Pi computer itself, no further configuration is necessary to take advantage of this functionality. Output format and connection details are further described below.

Output Format

Individual data packets are sent to port 8888 on the Shake Pi computer by default. The output format is straightforward:

  • Each data packet contains data for a single channel only
  • Entire data packet is wrapped with open and closing braces: { }
  • All fields are separated by a comma
  • First element defines the channel name - string in single quotes
  • Second element defines the timestamp, in epoch seconds, down to milliseconds, of the first data point - float
  • All remaining elements are the data points themselves - integer

At first glance, it appears there is missing information, but no! To keep data packet size to an absolute minimum, all other derivable values are omitted. If any of the following values is necessary for your follow-on processing, it is left as an exercise for the reader to compute these as part of program start-up when it first connects to the UDP port:

  • Rate of Data Packet Transmission - can be calculated from the timestamps of two subsequent packets for the same channel
  • Sample Rate - can be obtained by counting the data points of a single data packet, in conjunction with the derived data packet transmission rate
  • Total Number of Channels - can be calculated by identifying the unique channel names being received, your program is guaranteed to receive all channels once in a sequence of packets representing the same timestamp

To avoid having to compute these values programmatically, for the currently available three types of instruments, these values are defined here and can be hard-coded vs. a particular instrument type:

_images/udp-outputFormat.png

The following record is a sample data packet produced by a Raspberry Shake 1D instrument:

{'SHZ', 1507760140.530, 614, 916, 1095, 1156, 839, 923, 861, 856, 861, 789, 568, 823, 965, 788, 835, 991, 1028, 1225, 1142, 828, 682, 635, 771, 978, 834, 1167, 1116, 888, 627, 564, 944, 994, 780, 652, 811, 915, 832, 1134, 1020, 594, 756, 782, 748, 810, 864, 936, 977, 1014, 676, 502}

Connecting to a Data Stream

Connecting to the UDP port to read the values can be done using any programming language with the ability to make socket connections, which is basically all of them. Without getting into the details of socket programming, the choice of UDP was made to minimize the programming requirements to make a connection, get the data, and then do something interesting with it.

Below is a sample Python program that does only three things: connects to the default socket on the Shake Pi, reads the data off the socket, and prints it to the terminal.

import socket as s
host = ""                               # can be blank for localhost
port = 8888                             # Port to bind to
sock = s.socket(s.AF_INET, s.SOCK_DGRAM | s.SO_REUSEADDR)
sock.bind((host, port))
print "Waiting for data on Port:", port
while 1:                                # loop forever
    data, addr = sock.recvfrom(1024)    # wait to receive data
    print data

This sample program can be found in the directory ’/opt/settings/user’, named ‘shake-UDP.py’. To execute from the command line (control-C to quit):

bash> python shake-UDP.py

In a real-world scenario, your program should separate out the individual fields of the data packet, compute any needed derivable values at program startup, place the data points into an array, and process the incoming data.

Configuration of Additional UDP Ports

If you would like to create additional data streams, either on the Shake Pi or a different computer altogether, a simple configuration file can be created to define these. Once the system has been updated to v0.8, a new directory will exist, named ‘/opt/settings/user’. Contained in this directory is a file named ‘UDP-data-streams.conf.tpl’, which serves as the template for the actual configuration file used by the data producer service.

The format of this UDP configuration file is JSON and must conform by specifying the following entries:

{
    "UDP-destinations" : [
        { "dest" : "UDP-NAME-1"},
        { "dest" : "UDP-NAME-2"},
        { "dest" : "UDP-NAME-..."}
    ],

    "UDP-NAME-1" : {
        "Hostname" : "IP address or Fully-qualified DNS Name",
        "Port" :         "Port Number"
    },

    "UDP-NAME-2" : {
        "Hostname" : "IP address or Fully-qualified DNS Name",
        "Port" :         "Port Number"
    },

    "UDP-NAME-..." : {
        "Hostname" : "IP address or Fully-qualified DNS Name",
        "Port" :         "Port Number"
    }
}

The first section, “UDP-destinations”, defines the name of each UDP port to send data to. Subsequent sections must be named and represent each of the entries of the first section and define only two elements: the computer to send the data to, either by IP address or fully-qualified DNS name, and the port number. Create one or as many destination entries as you like.

Note

Since JSON format is very sensitive to mistakes, take care when creating the configuration file, making sure that it conforms absolutely to the above. For example, notice that each section is separated by a comma, but that each last entry does not have a comma following. To confirm it is valid JSON, there are numerous validators on the web, such as this one, that will tell you if it passes, or tell you where it doesn’t.

An example configuration file sending data to two additional ports, one located on the local Shake Pi and another located on a different computer located on the local network might look like:

{
    "UDP-destinations" : [
        { "dest" : "UDP-SHAKE-PI"},
        { "dest" : "UDP-NETWORK-PI"}
    ],

    "UDP-SHAKE-PI" : {
        "Hostname" : "UDPIPFILE:/opt/settings/sys/ip.txt",
        "Port" :         "54321"
    },

    "UDP-NETWORK-PI" : {
        "Hostname" : "192.168.1.203",
        "Port" :         "11335"
    }
}

(Notice that for the local Shake Pi entry, the value “UDPIPFILE:/opt/settings/sys/ip.txt” is provided instead of an IP address or hostname. This is because for some Shake units their IP address is not fixed and can change when rebooted. This entry guarantees that the data producer will always output the data to the Shake Pi regardless if the IP address changes between reboots. If your Shake Pi has a fixed IP, the IP address can be provided directly if so desired.)

Once you have defined the computer and port combinations, and have created a configuration file, copy this to the file named ‘UDP-data-streams.conf’, located in the same directory where the template file itself is located: /opt/settings/user

Once the configuration file exists, the data producer service will automatically begin writing data to the configured hosts and ports, there is no need to restart the data producer service. As well, any time this file is modified, the new configuration will be taken on by the data producer service immediately, including if the file is removed, and will stop data transmission to all previous configurations.