Building a Raspberry Pi Stratum 1 NTP Server


After the nice reception my last article on NTP got, I decided that it was about time I pulled a project off the shelf from a few years ago and get a stratum 1 NTP server running on a Raspberry Pi again.

So this article is going to be my notes on how I built one using a cheap GPS module off eBay as the reference time source and a Raspberry Pi. (a 2B in this case, but hopefully the differences for newer models will be minor; if there are any, they're probably around the behavior of the serial port since I think things changed on the Pi3 with bluetooth support?)

So anyways, before we jump into it, let's spend some time talking about why you would and wouldn't want to do this:

Pro: NTP is awesome, and this lets you get a stratum 1 time server on your local network for much lower cost that traditional COTS NTP time standards, which tend to be in the thousands of dollars price range.

Con: This is a Raspberry Pi, booting off an SD card, with a loose PCB plugged into the top of it, so I wouldn't classify it as "a robust appliance". It isn't a packaged enterprise grade product, so I wouldn't rely on this NTP server on its own for anything too critical, but then again, I would say that about NTP in general; it's only meant to get clocks to within "pretty close" of real time. If you're chasing individual milliseconds, you should probably be using PTP (Precision Time Protocol) instead of NTP... Totally depends on what you're doing. I'm just being a nerd.

Pro: The pulse-per-second driver for the Raspberry Pi GPIO pins is pretty good, so once you get this working, the GPS receiver will set the Pi's clock extremely well!

Con: The Raspberry Pi's Ethernet interface is actually hanging off of a USB hub which is hanging off of the USB interface on the SoC (system on chip) that powers the Raspberry Pi, so there is an intrinsically higher level of latency/jitter on the Pi's Ethernet interface vs a "real" PCIe NIC. This means that your bottleneck for getting crazy good time sync on all of your computers is ironically going to be getting the time off of the Raspberry Pi onto the network...

Pro: I would expect this USB jitter to be on the order of 500 microseconds, which is still less than a millisecond, and remember what I said about chasing individual milliseconds in NTP? This should be fine for any reasonable user of NTP.

Conceptual Overview

So we are going to be building an NTP server, which sets its time off of a GPS receiver, which is setting its time off of the global constellation of GPS (et al) satellites, which have cesium atomic clocks on-board which the US government puts a lot of effort into setting correctly, so that is where our time is actually coming from.

I'm specifically using a u-blox NEO-6M GPS receiver, which I got on eBay for a few dollars on a breakout board. This module isn't the "Precision timing" specific NEO-6T variant, but the 6M module is still completely sufficient for the level of time accuracy that we're looking for, and much cheaper/more available than the 6T module which is specifically designed for this sort of static time-keeping application (to the extent where the 6T will even lose GPS lock if you start moving it!)

The NEO-6M GPS receiver, like many GPS receiver modules, has a USB interface, which we will not be using, a UART serial interface, which we will be using, and a "Pulse-per-second" PPS output that it asserts high at the very top of every second, which we will definitely be using!

The reason that we will be using both the UART serial port and the PPS outputs on the GPS is because the UART tells us the current date, wall time, etc, but it does it over a 9600 baud serial port, so the latency between when the module starts sending that report and when we receive all of it will depend on how long the message is, and that depends on things like which NMEA sentences you have turned on, how many satellites the receiver see, etc, so while it usually takes about 130ms to receive this report, you can't be sure exactly how long it was.

The PPS output, on the other hand, is extremely precise on telling us exactly when the top of the second is by taking a single pin and asserting it high right on the top of each second, but all it tells us is when the top of the second is, not which second, or hour, or even what day it is! We want to know both what second we're currently on in history, and exactly when that second starts, so we need both the UART and PPS inputs to correctly set the time.

Looking at conceptually how we're going to use the UART and PPS outputs of the module, we're going to feed both of them into the Raspberry Pi's GPIO header (more on this in the hardware section), and do the following:

  • We're going to use the gpsd daemon to parse the /dev/ttyAMA0 UART text stream, which is usually NMEA sentences, but gpsd might put the receiver into its proprietary binary mode if it can figure out what kind of GPS receiver it is. The gpsd developers have put a lot of work into writing a GPS data stream parser, so we want to benefit from it. 
  • Gpsd will then write the time information it gets from the UART to a shared segment of memory, which the NTP daemon is able to connect to using its shared memory segment driver, to get a rough idea of what date/time it is.
  • The PPS output of the receiver will be fed into the Linux kernel PPS driver, which watches for the positive edges on the pin and time stamps them relative to the local system clock. The Linux kernel makes these events available for other applications via the /dev/pps0 device, which NTP will read using the NTP PPS Clock Discipline driver. 

Required Hardware

For this project, I used a generic GPS breakout board off eBay, which I soldered to a piece of 4x6cm perf prototype board, and soldered wires to make the required connections. That being said, there are several Raspberry Pi GPS "hats" available off the shelf, and it will likely be much easier for you to just buy the one part from someone like AdaFruit instead of trying to build your own and chase both hardware issues and software issues at the same time when trying to get this working. But hey, do what you want; I'm not the boss of you. I just built my own because I happened to have all of these parts on hand already.
The one key difference depending on which GPS hat you use or how you build your own, is which of the many GPIO pins the GPS PPS pin is attached to. Popular PPS pins are GPIO4 (pin 7) and GPIO18 (pin 12), but I suspect that there's no reason most of the GPIO pins can't support the PPS input, so if you're building your own board, you can pick a different GPIO pin all together. The only thing you need to make sure is that the physical GPIO pin that you use matches the GPIO pin that you configure in the /boot/config.txt file when you're enabling the PPS overlay driver. I used GPIO18 on my build.

Nevermind the LEDs on the side of the perf board; those were left over from the last project I did on this perf board, and I figured they might come in handy if I wanted to program some status indicators for this NTP server.

The Raspberry Pi UART is tied to pins 8 and 10 on the header, so the UART should work regardless of what HAT you buy. What might vary is the exact baud rate. My modules happened to run at 9600 baud, but I've worked with GPS receivers that by default use 4800, 19200, or even 115200. Check the documentation for your receiver, and GPSD does a good job of trying to figure these things out on its own as well so it might not even matter.

So to summarize:

  • 5V on Pi header to Vcc on GPS module
  • GND on Pi header to GND on GPS module
  • TX on Pi header to RX on GPS module
  • RX on Pi header to TX on GPS module
  • GPIO18 on Pi header to PPS on GPS module

Initial Raspberry Pi Setup

I'm going to mostly gloss through this part, because setting up Raspbian on a Raspberry Pi is a pretty well documented process which greatly depends on what OS you're trying to do it on, and if you're trying to use something like NOOBS or write a raw IMG file directly onto an SD card.

I will just say that my preferred method of creating a new Raspbian SD card from my Linux desktop is using the "dd" command, so assuming that my SD card reader came up as /dev/sdf (check the last few lines in "dmesg" to check") I do the following:


kenneth@thor:~/data/operatingSystems/rasberrypi$ sudo umount /dev/sdf*
umount: /dev/sdf: not mounted
kenneth@thor:~/data/operatingSystems/rasberrypi$ sudo dd if=./2019-07-10-raspbian-buster-lite.img of=/dev/sdf
4292608+0 records in
4292608+0 records out
2197815296 bytes (2.2 GB, 2.0 GiB) copied, 843.348 s, 2.6 MB/s
kenneth@thor:~/data/operatingSystems/rasberrypi$ sync

I should probably download a more recent image of Raspbian Lite... It all sorts itself out when I eventually run "sudo apt update; sudo apt upgrade -y".

Once you're created a Raspbian SD card, put it in your Pi, and boot it. I was having some difficulty getting the Pi to boot initially with the GPS hat plugged in (likely since by default the serial port is configured as a login console, and Linux didn't like the GPS receiver spewing a bunch of NMEA sentences into it), so I left the GPS unplugged until I got a chance to disable the login console on the serial port and reboot the Pi. 

Install SD card in Pi, plug in monitor, keyboard, Ethernet, and power, and watch Raspbian automatically grow the filesystem to fill the card, and boot. Login with the default "user: pi" "password: raspberry" and do my usual new Raspbian host setup using the raspi-config tool (most of which probably isn't critical to getting NTP working):

From the command line, run "sudo raspi-config" to open the Raspbian config tool. Up/down arrows to navigate, enter to select, right arrow to jump down to the select/finish options.

1 Change User Password -- Pick a new password so you can later enable SSH
2 Network Options - N1 Hostname -- Pick a meaningful hostname for the Pi, like... ntp1pi... or something...
3 Boot Options - B2 Wait for Network at Boot -- I usually turn this off for all my projects, but it likely doesn't matter for a network time server...
4 Localisation Options - I1 Change Locale -- I scroll down and turn off en_GB.UTF-8 and turn on en_US.UTF-8 with the space bar. Page up/down are your friend on this dialog. I pick C.UTF-8 as my default locale. This change takes some time while Raspbian generates the new locale. 
4 Localisation Options - I2 Change Timezone -- I go in and pick my local timezone.
4 Localisation Options - I3 Change Keyboard Layout -- This one is actually pretty important! Mainly because the default GB keyboard layout moves around a few keys like the | and @ I think? The one issue I have is that if I try and open this menu now, it displays as gibberish, so I need to reboot the Pi again before making this change.
4 Localisation Options - I4 Change Wi-Fi Country -- Doesn't matter on the Pi2, but on Pi3 and above, I set the WiFi to us.
5 Interfacing Options - P2 SSH -- I enable SSH, because this initial set up is both the first and the last time I plan to have a monitor and keyboard plugged into the Pi. I'll log in remotely over SSH going forward.
5 Interfacing Options - P6 Serial -- This is the important setting to change in here for this project! You would not like a login shell to be accessible over serial (answer no) but you would still like the serial port hardware to be enabled (answer yes)
7 Advanced Options - A3 Memory Split -- Since I'm never going to run a GUI on this Pi, or even ever plan to plug a monitor into it, I like to turn the GPU RAM down to the minimum of 16MB to free up a little more RAM for Linux itself.

When you right arrow to finally select "finish", it will probably prompt you to reboot, and you probably should, particularly because you need to do so to come back in to raspi-config a second time to change the keymap.

SUMMARY: So this was a bunch of my usual quality of life settings on new Raspbian images, and the one crucial step of turning off the login shell on the serial port, but leaving the serial port itself enabled.

Configuring GPSD

At this point, plug in the GPS hat. 

The GPS receiver will take some time to gain a lock and start reporting valid location/time information (Ranging from 15 seconds to 20 minutes, depending on how recently the receiver last had a fix, how good its skyview it, etc), but most receivers will still spew out NMEA sentences, even when they don't have a lock. 

To perform our first sanity check, I installed minicom (sudo apt install minicom) and used it to open the /dev/ttyAMA0 serial port at 9600 baud to confirm that I had the GPS wired up correctly and was getting at least something that looked like this:

$GPRMC,064605.00,V,,,,,,,,,,N*7C
$GPVTG,,,,,,,,,N*30
$GPGGA,064605.00,,,,,0,00,99.99,,,,,,*67
$GPGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99*30
$GPGSV,1,1,01,29,,,35*75
$GPGLL,,,,,064605.00,V,N*4B

At this point, we can install gpsd and configure it to handle the /dev/ttyAMA0 serial port!

pi@ntp1pi:~ $ sudo apt install gpsd

The GPSD daemon doesn't have a configuration file, but you pass it a few options via the /etc/defaults/gpsd file to tell it what you want.

pi@ntp1pi:~ $ sudo nano /etc/default/gpsd

We need to change two things in this file: 
  1. Add "/dev/ttyAMA0" to the empty DEVICES list
  2. Add the "-n" flag to the GPSD_OPTIONS field so GPSD will always try and keep the GPS receiver running. When gpsd is running on something like a phone, it makes sense to try and minimize when the GPS is running, but we don't care. I think GPSD also doesn't count NTP as a client, since NTPd is using the shared memory segment to talk to GPSD, so GPSD will randomly just stop listening to the receiver if you don't add this flag.
This is what my /etc/defaults/gpsd file looks like in the end:
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
DEVICES="/dev/ttyAMA0"
# Other options you want to pass to gpsd
GPSD_OPTIONS="-n"

To apply these changes, restart GPSD:

pi@ntp1pi:~ $ sudo service gpsd restart

To perform a sanity check that this configuration is working, we can install the gpsd clients (sudo apt install gpsd-clients) and use one of them like "gpsmon" or "cgps" to see if GPSD is even talking to the receiver, and hopefully reading good time/location information if you've left it running long enough already. You can also run "sudo service gpsd status" and should see a green dot that indicates that systemd has successfully started gpsd. (q quits out of that display)

Configuring the PPS Device

The next step is to tell the Linux kernel which GPIO pin we've attach the PPS output from the GPS receiver to, so Linux can capture the pulse events and timestamp them. This is done by editing the "/boot/config.txt" file, and adding a new line with the driver overlay for pps-gpio (I usually add it at the end of the file), and specifying which GPIO pin you used at the end of the line (in my case, 18):

dtoverlay=pps-gpio,gpiopin=18

At this point, you will have to reboot again to apply this change to the /boot/config.txt file. 

Once the Pi has rebooted, you should see a new /dev/pps0 device, and can test it using the ppstest tool (sudo apt install pps-tools):

pi@ntp1pi:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1583046422.047708052, sequence: 66 - clear  0.000000000, sequence: 0
source 0 - assert 1583046423.047205387, sequence: 67 - clear  0.000000000, sequence: 0
source 0 - assert 1583046424.046730740, sequence: 68 - clear  0.000000000, sequence: 0
source 0 - assert 1583046425.046282831, sequence: 69 - clear  0.000000000, sequence: 0
source 0 - assert 1583046426.045858285, sequence: 70 - clear  0.000000000, sequence: 0

If you instead see error messages about timeouts, it's possible the GPS receiver hasn't gained GPS lock yet, since many won't output PPS until they have a lock, or you've got a problem with your connections between the GPS and your selected GPIO pin...

One thing to note about the output of ppstest is that it timestamps each PPS event with the local time down to the nanosecond resolution. If you notice in the output above, each pulse seems to be happening 500us sooner than the pulse before, which shows that the local system clock speed is grossly off, since pulse per second events should be happening... once per second... not once per every 0.9995 seconds! Once you get ntpd running and disciplining the local clock off of this PPS, you should see the assert timestamp get close to all zeros after the decimal point, which means that your system clock is well aligned to the actual top of the second. So once we're all done here, it should look something like this:

pi@ntp1pi:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1583098783.999996323, sequence: 46821 - clear  0.000000000, sequence: 0
source 0 - assert 1583098784.999995254, sequence: 46822 - clear  0.000000000, sequence: 0
source 0 - assert 1583098785.999995642, sequence: 46823 - clear  0.000000000, sequence: 0
source 0 - assert 1583098786.999994935, sequence: 46824 - clear  0.000000000, sequence: 0
source 0 - assert 1583098787.999994174, sequence: 46825 - clear  0.000000000, sequence: 0

Notice how the local time stamps relative to the PPS input is pretty dang extremely close to once per second, near the top of each second!

Configuring NTP

At this point, we should have both GPSD and the kernel PPS driver pulling information from the GPS receiver, so now we need to install the NTP server and edit its config file to tell it to use both of these time sources!

pi@ntp1pi:~ $ sudo apt install ntp

One thing that is a little unusual about NTP is that for local time sources, it's about the only system I've ever seen that takes advantage of the fact that the IPv4 loopback address space is an entire /8 (127.0.0.0/8), so each different type of time source, and each instance of each time source, is actually defined by a different 127.127.x.x IP address!

Looking at the NTP documentation for time sources, the two that we are interested in are the PPS clock discipline (driver 22) and the shared memory driver (driver 28). 

Since we are interested in using the 0th PPS device (/dev/pps0), the server address we want for the PPS clock source is 127.127.22.0. Likewise, for the SHM(0) driver, we want address 127.127.28.0. Notice how the third octet in both of these IPv4 addresses correspond to the number of the driver we're using, and the fourth octet is the instance of that driver that we're using (the 0th/first for both). For example, if we were monitoring multiple PPS devices for some reason, we could configure multiple server addresses: 127.127.22.0 for /dev/pps0, 127.127.22.1 for /dev/pps1, 127.127.22.2 for /dev/pps2, etc. For this blog post, we're just looking at one of each...

We also need to configure a few flags for each of these time sources, so the new chunk of text we're adding to /etc/ntp.conf should look like this (thanks to this blog post for this snippet):

# pps-gpio on /dev/pps0
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS
fudge 127.127.22.0 flag3 1  # enable kernel PLL/FLL clock discipline

# gpsd shared memory clock
server 127.127.28.0 minpoll 4 maxpoll 4 prefer  # PPS requires at least one preferred peer
fudge 127.127.28.0 refid GPS
fudge 127.127.28.0 time1 +0.130  # coarse offset due to the UART delay

The first half configures the pulse per second device:
  • The minpoll 4 maxpoll 4 options on the server line tell NTP to always poll the PPS device every 16 seconds (2 raised to the power of 4) instead of the default "start at 64 second intervals and back off to 1024 second intervals" that ntpd uses by default, since we're not sending queries to remote NTP servers here! We're just looking at events from a local piece of hardware.
  • The "fudge 127.127.22.0 refid PPS" line assigns a human readable identifier of ".PPS." to this time source. Again, if you were doing something squirrely like monitoring multiple PPS devices (i.e. "PPS1", "PPS2", etc), or just wanted to assigned a different name to this server than "PPS", you could change that here.
  • The "fudge 127.127.22.0 flag3 1" line enables the kernel Phase Locked Loop clock discipline... which is about all I can say about it... It sounds important!
  • Same thing for the "server 127.127.28.0 minpoll 4 maxpoll 4 prefer" line with regards to the minpoll/maxpoll options; query the shared memory driver every 16 seconds. The "prefer" option tells ntpd to prefer this time source, which according to the comment seems to be required, but I don't quite follow why, since this is a stratum 0 time source, so I'd expect ntpd to end up preferring it anyways.
  • Again, "fudge 127.127.28.0 refid GPS" is assigning a human readable refid to this time source, which in this case is ".GPS." to indicate that this is the NMEA data over the serial port, vs the PPS coming in over the GPIO pin.
  • The oddest line is probably "fudge 127.127.28.0 time1 +0.130" which adds a 130ms offset to the exact time reported from the UART. Remember how I said that the precise time from the UART tends to be pretty bad, since it takes a while to deliver the data over the serial port at 9600 baud, and the exact length of the message will vary second to second? This 130ms offset is a crude approximation of how long it takes to send the NMEA report on the second, so that this clock will at least not be grossly off from true time. You will still see a few ms offset, and plenty of jitter, but at least the offset won't be huge!
So given this chunk of configuration, we add that to /etc/ntp.conf. Granted, even though we're setting up a stratum 1 time server here, it will likely still be wise to leave some other NTP servers in the configuration, so in case our GPS receiver dies for some reason, our server can still get its time from another server and will simply increment its stratum from 1 to one more than whichever other server it has selected as its system peer. Why the stock Raspbian ntp.conf comes with four pools configured (the pool gets expanded into multiple servers, so I don't think you need four of them) is beyond me...

My /etc/ntp.conf file ends up looking like this!
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift
leapfile /usr/share/zoneinfo/leap-seconds.list
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

pool 0.debian.pool.ntp.org

# pps-gpio on /dev/pps0
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS
fudge 127.127.22.0 flag3 1  # enable kernel PLL/FLL clock discipline

# gpsd shared memory clock
server 127.127.28.0 minpoll 4 maxpoll 4 prefer  # PPS requires at least one preferred peer
fudge 127.127.28.0 refid GPS
fudge 127.127.28.0 time1 +0.130  # coarse offset due to the UART delay

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Needed for adding pool entries
restrict source notrap nomodify noquery

To apply this new configuration, you need to tell Linux to restart ntp:

pi@ntp1pi:~ $ sudo service ntp restart

Checking Our Work

So now the BIG question is if this all actually worked. I didn't see any signs of life right away, so I did try rebooting the Pi, which might be required.

The tool you can use to interrogate the local NTP server with regards to what peers it's monitoring and what offsets it has calculated is "ntpq".
Here you can see the output of the "ntpq -p" command. Notice how the reach for the SHM(0) remote is no longer zero! This might take a few minutes, and once the NTP server can reach the shared memory segment, it will wait a few more minutes before it starts also polling the PPS, so don't freak out if you don't start seeing that reach value incrementing as well. It seems to typically take my server about 5-10 minutes of just monitoring the SHM(0) source before it starts also querying the PPS(0) source. If after 15-30 minutes you still see a 0 reach for both local clocks, you should investigate all of the sanity checks we did above (is the pps event being seen by the kernel, is gpsd running and happy, etc)
After allowing my NTP server to run for several hours, we can see how the PPS(0) offset and jitter have gone to essentially zero. The SHM(0) offset and jitter are to be expected, since like I said, that precise timing is based off how long it takes to read the data over the serial port.

And with that, we have a working NTP server! The last time I built one of these, I didn't have access to a cabinet in a datacenter, so I will need to play around with this and see how it performs under load, but that's another project on its own...

Comments

Popular Posts