Sunday, January 12, 2014

NTP Multicast Servers

Anyone who's followed this blog or gone back and read my past posts has probably picked up on my somewhat unhealthy obsession with time. Every time I get my hands on a new display or microcontroller, the first thing I do is build a clock out of it (one, two, three, four, five, six, seven, eight, nine). I've also dabbled in high precision time with my rubidium frequency standard and the multitude of frequency counters in my apartment. One area that I haven't had as much success with time in the past is keeping all of my Linux systems synchronized, and that's been bothering me.

I finally sat down this week and spent a few hours figuring out how to get multicast network time protocol working on Debian, since I've already put so much work into getting multicast routing working on my apartment's LAN and want to actually use it for something. There were a few critical pieces that all came from different places online, so I figured I'd collect them all in one place.

Typically, NTP operates by configuring each client with a server address to periodically poll for the current time. During each poll, the client tries to determine the remote server's time and how long it took for the server's response to get back to the client. For remote Internet servers, this time delay can be in the order of 20-100ms and change wildly based on network conditions. The traditional solution to this is to have a single local system poll several remote NTP servers, and then manually configure each local client to poll just this one local server, which in my case is my WNDR3800 router. This works acceptably well, but just isn't complicated enough to meet my desire to make life hard on myself.

Multicast NTP is based on multiple servers beaconing their local time towards a multicast address, which clients listen for to discover these servers. When a client discovers a new server, it first performs a standard unicast query with the server to determine network delay between the two systems, and then sits back and silently listens for future multicast beacons from the server to keep the client's local time in sync. All this does for me is allow my NTP clients to dynamically find new and forget old NTP servers on my network when I swap in and out computers. Another advantage where multicast really shines on huge computer networks is that after the initial propagation measurement, clients don't generate any additional network traffic, but only passively listen for the multicast beacons from the servers.


Configuring NTP Multicast Servers

On my systems that I've designated as my NTP servers, they do the typical pulling of four unique servers from *.pool.ntp.org DNS entries, but then act as both multicast servers and clients so they can discover and synchronize with each other.

Sample Config and ntpq output:


A few tripping points in the configuration file:

  • The default restrict statements prevent discovering peers, so you need to remove the nopeer statement from the IPv4 and/or IPv6 restrict statements.
  • The broadcast statement defaults to "ttl 0", which means the 0th entry in the eight value ttl array used by manycasting. This means that the argument to ttl isn't actually a time to live value, but an index into the default array of ttl values {1, 32, 64, 96, 128, 160, 192, 224}, which you can re-declare if that doesn't meet your needs.
  • Unlike broadcast clients, for multicastclient you need to specify the address to listen to.
  • /etc/ntp.conf isn't the config file that NTP actually uses. If you look at /etc/init.d/ntp, you'll discover that by default, ntpd uses the config file /var/lib/ntp/ntp.conf.dhcp, which is built by appending /etc/ntp.conf to any NTP server advertisements from the system's DHCP server. /var/lib/ntp/ntp.conf.dhcp isn't rebuilt when you run "service ntp restart", which is why your changes to /etc/ntp.conf don't appear to do anything until you reboot the entire system! I haven't bothered to figure out which init script triggers this config rebuild, but have been doing testing on the /var/lib/... file and then manually copying the config back to /etc/ntp.config
The selection of the multicast address is pretty arbitrary. What I have shown isn't the actual address I use, but anything in the 239.192.0.0/14 or ffx8::/16 organizationally-local multicast address subnets are technically correct and should work fine as long as all of your systems match.

Monday, January 6, 2014

My First Adventure in Homebrewing Beer

Yes, I am in fact alive. Last quarter ended up being more work than I had expected, and teaching a section of Electronics Lab for Non-Majors sapped away a lot of the instructional energy that I normally blow off on this blog.

Every year, I try and find a new hobby to broaden my life experience with. 2012, it was getting back into Amateur Radio; 2013, it was shooting (mostly trap, though I'm slowly getting into rifle...); 2014, I'm thinking it's going to be homebrewing beer.

I've been toying with fermentation for a while. Most notably, last winter/spring I had a few false starts that ended with me throwing up in my bathroom and my friends laughing their asses off. After that, two experiences really convinced me to go all in and re-approach the hobby more rigorously this year.

  • I helped my buddy Robbie brew a batch of beer using a grossly expired Mr. Beer brew kit, and the beer still came out drinkable. Tell-tale soapy tastes of a 4 year old extract malt, but not the worst beer I've had.
  • Which was the second experience. Killing time with another buddy, Sean, late last quarter, we stopped at a liquor store and bought a variety pack of "Brewer's Mystery" beer from some local brewery that I had never heard of, mainly because it was the only case of beer they had that we hadn't already been drinking in the last week. It was awful. It was the worst beer I had ever experienced in my life. It wasn't a "eh, this isn't very good" bad; it was a "I immediately regret opening this bottle and not leaving it for someone else" bad. We eventually managed to pawn it off at some random college party, but the trauma was already done.
So, when Robbie and I manage to brew a better beer with four year old malt and yeast than someone seriously sold us in a liquor store, it's hard not to figure that this is a hobby I can't be too unsuccessful at. Queue a trip to the local homebrew shop here in San Luis Obispo (Doc's Cellar), where I buy a copy of Palmer's How to Brew. It's a very nicely laid out book:
  • The first chapter is a crash course in turning the crank to brew your first batch of beer, without focusing on understanding too much of what's happening.
  • The rest of the book is then divided into three major sections for extract, partial grain, and full grain brewing. This means that you get to start simple and make the brewing hobby as complicated as you want to.
  • It is also available in website form, although I definitely don't regret having a hard copy to read in bed and have in the kitchen.
Thanks to my apartment kitchen already doubling as a wet lab for various projects (PCB etching, etc.), I can already cobble together most of what I needed to brew my first batch. What little I still needed (mainly a hydrometer and bottle capper), I put on my Christmas list for my parents, who always appreciate when I ask for more tactile things than a textbook about knots for holidays. 


Starting My First Baseline Batch

Being the engineer that I am, the first thing to do is run a small batch of the most generic beer recipe I can come up with as a baseline to compare against. This is based on Palmer's Cincinnati Pale Ale recipe, with substitutions based on availability at Doc's Cellar and only doing a half batch at a time (2.5 gallons).
  • 1.5 lbs pale malt syrup ($3.38)
  • 1.5 lbs Munich malt syrup ($3.38)
  • 7g Safale US-05 yeast ($1.75)
  • 0.5oz Cascade hops for 60 minutes for bittering ($1)
  • 0.5oz Cascade hops for 10 minutes for finishing ($1)
Putting the expense of consumables for the first batch at $10.51, not counting the ~$100 of equipment I've collected/bought/been gifted over the last six months.
First thing is to boil the malt and hops. This pasteurizes the wort so that the only thing growing in it ends up being the yeast we want, changes the chemical composition of the malt, and steeps the hops. I did a partial boil with 1 gallon of water, which I then diluted to 2.5 gallons in the final fermentation chamber, which isn't ideal for steeping the hops, but the best my largest pot allows.
By adding half of the Cascade hops at the beginning of the boil, and holding the second half until the last ten minutes, I get to reduce half of it to just the bittering agents, while still retaining some of the aroma and more volatile flavoring from the second half.
Picture well into the boil. I didn't bother tracking boil-off water loss, so I'm likely going to end up with about 2 gallons of beer in the end, once you factor in water loss here and in the bottom sediment when I bottle in a few weeks.
The hour of boiling the wort breaks down lots of complex proteins that you want to settle out of the beer. Once that happens (which makes it look a lot like egg drop soup), you want to cold shock the wort to pull out more proteins, which mainly cause a hazy appearance when you chill the beer, but also do affect the long-term stability of the beer (which is unlikely to be a problem around college students...)
The pot is placed in one sink filled with cold water, while ice water is circulated through the copper tubing coil I fashioned from left-over copper tubing from a boiler experiment I did in the past.
Once the wort is cooled down to below 70°F, I dilute it to 2.5 gallons, pitch 7g of rehydrated yeast, and split the wort between a 3 quart carboy and a 5 gallon bucket so I can visually track a small sample of the batch.

The initial specific gravity of the batch ended up being 1.047, using only the malt syrup and no added sugar. This, minus the final gravity in a few weeks, can tell us how much alcohol my little yeasty buddies here created, which isn't going to be anything to phone home about. Then again, if I were doing this just to try and get drunk, I'd save the trouble and just drink more rum from my liquor closet...

The first definite signs of life took about 16 hours. These last two pictures were taken 41 hours after pitching the yeast. Lots of churning going on in the wort, the airlock is bubbling every ~12 seconds, and all that nasty looking foam on the top is slowly growing (that is totally normal).

Now I need to wait three weeks, bottle this batch with a small addition of sugar for carbonation, and let it bottle condition for another few weeks. Come about mid-February, I'll get to test this first batch. Until then, I just need to leave it alone and figure out how I want to specialize in this hobby.