Raspberry Shake Data Producer UDP Port Output#
Contents
Description#
This is now known as “DATACAST”.
Currently, the data producer service of the Raspberry Shake sends the seismograph data to a couple of destinations: a local Seedlink server, used to support both Swarm and the local helicorder plots, and to the cloud data server, used as part of back-end processing of Raspberry Shake network data and to support the online global map of Raspberry Shake stations, viewable here.
There is one more destination type that may be of general interest to the Raspberry Shake community: one or more UDP ports located on any computer your Raspberry Shake unit is able to communicate with.
UDP functionality is not intended for production purposes but for hobbyist applications. This is because UDP by definition does not guarantee 100% receipt of data packets.
Preamble: UDP versus TCP protocols#
Because of the way the UDP protocol is designed, the Raspberry Shake must forward packets to a specific IP address (e.g., your desktop computer), rather than your your desktop computer requesting the packets from the Raspberry Shake. This is different from most of the other data transfer methods used in Seismology (SeedLink, etc.) which rely on handshakes and confirmations that data has been delivered. These methods use the more standard internet protocol TCP.
Since UDP is designed for real-time systems which prioritize sending data to arrive quickly, and do not really care about “dropped packets” like TCP does, there are no requests or handshaking, there is only sending and receiving. The sending computer is the Raspberry Shake. The receiving computer is a port on another computer.
The Benefits#
What this means is that you, the end-user, can connect a program directly to this UDP port and receive the data coming off the seismograph as quickly as it is being produced, in real-time. This has at least the following advantages: 1) this is now the quickest way of reading the data coming off the seismograph; 2) connecting to a real-time stream is now completely trivial; 3) and since there is no limit to the number of UDP ports that can be configured for output, multiple systems can receive the data in real time, e.g., a web-page and another program meant to analyze the data, for example. Please see our open-source client program rsudp for a fully functional (and very useful) example of what you can do with UDP datacasting here.
So, for all the scientists out there, amateur and professional alike, who want direct and instant access to the data for your own purposes, this enhancement is for you!
Have fun, enjoy, and make something cool! (A new EQ detection algorithm, maybe?) And when you do, let everyone know about it, here: raspishake.
All sections below detail how to use this functionality, please let us know if you have any questions.
Implementation Details#
The data producer program, by default, now writes to this new data stream on the local Shake Pi, Port 8888. If you want to attach to a single data stream on the Shake Raspberry Pi computer itself, no further configuration is necessary to take advantage of this functionality. Output format and connection details are further described below.
Output Format#
Individual data packets are sent to port 8888 on the Shake Pi computer by default. The output format is straightforward:
Each data packet contains data for a single channel only
Entire data packet is wrapped with open and closing braces: { }
All fields are separated by a comma
First element defines the channel name - string in single quotes
Second element defines the timestamp, in epoch seconds, down to milliseconds, of the first data point - float
All remaining elements are the data points themselves - integer
At first glance, it appears there is missing information, but no! To keep data packet size to an absolute minimum, all other derivable values are omitted. If any of the following values is necessary for your follow-on processing, it is left as an exercise for the reader to compute these as part of program start-up when it first connects to the UDP port:
Rate of Data Packet Transmission - can be calculated from the timestamps of two subsequent packets for the same channel
Sample Rate - can be obtained by counting the data points of a single data packet, in conjunction with the derived data packet transmission rate
Total Number of Channels - can be calculated by identifying the unique channel names being received, your program is guaranteed to receive all channels once in a sequence of packets representing the same timestamp
To avoid having to compute these values programmatically, for the currently available three types of instruments, these values are defined here and can be hard-coded vs. a particular instrument type:
The following record is a sample data packet produced by a Raspberry Shake RS1D instrument:
{'SHZ', 1507760140.530, 614, 916, 1095, 1156, 839, 923, 861, 856, 861, 789, 568, 823, 965, 788, 835, 991, 1028, 1225, 1142, 828, 682, 635, 771, 978, 834, 1167, 1116, 888, 627, 564, 944, 994, 780, 652, 811, 915, 832, 1134, 1020, 594, 756, 782, 748, 810, 864, 936, 977, 1014, 676, 502}
Configuring a Data Stream - The Easy Way#
Since version 0.15, it is possible to configure destination machines and ports through the front-end configuration interface http://rs.local/. Go to Settings icon >> DATACAST and define the combination of Target Host IP address(es) and Target Port(s) you wish to send the data to. Sample programs are available for download which can be run on the destination machine(s); these will provide a good tutorial and starting point for doing something interesting with the data.
In the meantime a full-bodied example Raspberry Shake data client, contributed by our chief scientist Ian Nesbitt can be downloaded directly from here: shake-UDP-live
Configuring and Connecting to a Data Stream - By Hand#
Connecting to the UDP port to read the values can also be done using any programming language with the ability to make socket connections, basically all of them. Without getting into the details of socket programming, the choice of UDP was made to minimize the programming requirements to make a connection, get the data, and then do something interesting with it.
Below is a sample Python program that does only three things: connects to the default socket on the Shake Pi, reads the data off the socket, and prints it to the terminal.
import socket as s
with open('/opt/settings/sys/ip.txt', 'r') as file:
host = file.read().strip()
port = 8888 # Port to bind to
sock = s.socket(s.AF_INET, s.SOCK_DGRAM | s.SO_REUSEADDR)
sock.bind((host, port))
print "Waiting for data on Port:", port
while 1: # loop forever
data, addr = sock.recvfrom(1024) # wait to receive data
print data
This sample program can be found in the directory ’/opt/settings/user’, named ‘shake-UDP-local.py’. To execute from the command line (control-C to quit):
bash> python shake-UDP-local.py
In a real-world scenario, your program should separate out the individual fields of the data packet, compute any needed derivable values at program startup, place the data points into an array, and process the incoming data.
Configuration of Additional UDP Ports#
If you would like to create additional data streams, either on the Shake Pi or a different computer altogether, a simple configuration file can be created to define these. Once the system has been updated to v0.8, a new directory will exist, named ‘/opt/settings/user’. Contained in this directory is a file named ‘UDP-data-streams.conf.tpl’, which serves as the template for the actual configuration file used by the data producer service. Editing this file will do nothing, as it is only a template. Read on for how to edit and save this as a working config file.
The format of this UDP configuration file is JSON and must conform by specifying the following entries:
{
"UDP-destinations" : [
{ "dest" : "UDP-NAME-1"},
{ "dest" : "UDP-NAME-2"},
{ "dest" : "UDP-NAME-..."}
],
"UDP-NAME-1" : {
"Hostname" : "IP address or Fully-qualified DNS Name",
"Port" : "Port Number"
},
"UDP-NAME-2" : {
"Hostname" : "IP address or Fully-qualified DNS Name",
"Port" : "Port Number"
},
"UDP-NAME-..." : {
"Hostname" : "IP address or Fully-qualified DNS Name",
"Port" : "Port Number"
}
}
The first section, “UDP-destinations”, defines the name of each UDP port to send data to. Subsequent sections must be named and represent each of the entries of the first section and define only two elements: the computer to send the data to, either by IP address or fully-qualified DNS name, and the port number. Create one or as many destination entries as you like.
Note
Since JSON format is very sensitive to mistakes, take care when creating the configuration file, making sure that it conforms absolutely to the above. For example, notice that each section is separated by a comma, but that each last entry does not have a comma following. To confirm it is valid JSON, there are numerous validators on the web, such as this one, that will tell you if it passes, or tell you where it doesn’t.
An example configuration file sending data to two additional ports, one located on the local Shake Pi and another located on a different computer located on the local network might look like:
{
"UDP-destinations" : [
{ "dest" : "UDP-SHAKE-PI"},
{ "dest" : "UDP-NETWORK-PI"}
],
"UDP-SHAKE-PI" : {
"Hostname" : "UDPIPFILE:/opt/settings/sys/ip.txt",
"Port" : "54321"
},
"UDP-NETWORK-PI" : {
"Hostname" : "192.168.1.203",
"Port" : "11335"
}
}
(Notice that for the local Shake Pi entry, the value “UDPIPFILE:/opt/settings/sys/ip.txt” is provided instead of an IP address or hostname. This is because for some Shake units their IP address is not fixed and can change when rebooted. This entry guarantees that the data producer will always output the data to the Shake Pi regardless if the IP address changes between reboots. If your Shake Pi has a fixed IP, the IP address can be provided directly if so desired.)
Once you have defined the computer and port combinations, and have created a configuration file, copy this to the file named ‘UDP-data-streams.conf’, located in the same directory where the template file itself is located: /opt/settings/user
Once the configuration file exists, the data producer service will automatically begin writing data to the configured hosts and ports, there is no need to restart the data producer service. As well, any time this file is modified, the new configuration will be taken on by the data producer service immediately, including if the file is removed, and will stop data transmission to all previous configurations.