Wednesday, March 26, 2014

Range Testing NanoBridge M5s

In preparation for the Wildflower Triathlon in May, I've been testing many of the major components of the computer network I'm helping build for communications. One of the big unknowns this year is the unusually large number of mid-range microwave links, which I've never had too much experience with. WDS links across buildings or small parks I'm comfortable with, but we are looking at streaming HD video through WiFi hops that are multiple kilometers long, and I've just never played with links like that before.
To gain some experience before the final deployment, I bought a pair of Ubiquiti NanoBridge M5 nodes. These are 200mW 5GHz WiFi access points with optional high-gain dishes, and at $80 a piece are pretty reasonable for what you get.

The NanoBridges come with either 22dBi or 25dBi dishes, pole mounting hardware, and a passive PoE injector. Luckily, Ubiquiti uses the kind of standard passive PoE injector pinout, so I often use the cheap 2.1mm barrel PoE injectors you can get on eBay for $2 instead of using the OEM injectors from Ubiquiti so I can power them off 12V batteries for portable operations. I selected the lower gain 22dBi dishes because I plan to be moving these around a lot so the smaller dishes are a nice convenience.
I think the NanoBridges claim to have a 20km range, which isn't an unreasonable number if you look at the link margin calculations, but since I've never done a deployment like this before I did want to do a real-life deployment before trying to do it across lake San Antonio. I gave one of my buddies, Robbie, a six pack of beer and one of the two dishes and told him to point it at Cuesta Ridge north of SLO. I then drove up the horrific road to the top of the ridge and pointed the second dish back at his apartment. Once we got both of them turned on and roughly pointed in the right direction, the link came right up at full speed.

At 6.5km, this test is longer than any of the links that we're going to need for Wildflower, so I was very happy when the link came up without any fine tuning of either dish's aim. I was even able to aim one of the dishes 30-40 degrees off center and the link stayed up, although with a smaller link margin.
AirOS, which is what most Ubiquiti nodes run, has a really neat spectrum analyzer mode that I used to help select channels for the test. The first screen shot is from Cuesta Ridge looking at the second node down the hill beaconing its SSID.
This is a capture of my MacBook Air transferring files from my apartment's AP using a 40MHz channel.
This is another spectrum scan from the radio site on top of Cuesta. Interestingly, none of these have the distinctive 802.11 spectrums, but are other modulation types from non-802.11 equipment using the 5GHz ISM band. I thought they were really interesting.

So in short, I'm really happy with my NanoBridge M5s, which couldn't be much easier for setting up a ~80Mbps effective throughput link between any two points with line of sight. 6.5km was no problem, and shorter links will only enjoy a larger power margin to allow for non-perfect aiming, rain fade, etc.

Thursday, March 20, 2014

Pushing VLAN Tags Through Unmanaged Switches

Now that it's Spring Break and I'm in San Luis Obispo, it's full speed ahead on building the communications network for the Wildflower Triathlon that CPARC supports every year. Wildflower is a very big event in the middle of nowhere (Lake San Antonio), so we have to build quite a bit of infrastructure to support the operation.

This year, I was designated as the computer network zonie, so it's my job to make sure that there's IP connectivity between all the major sites in the network. This involves building a computer network that includes a couple Internet hand-off points, multiple routers, several medium-range (2-5km) microwave links, QoS enforcement for a few hundred devices to support VoIP and streaming video while sharing a 15Mbps Internet uplink, a couple 802.1Q VLAN trunks, etc.

Needless to say, we are building a network beyond the budget we're being given, so duct tape and Linksys devices are being applied liberally throughout this project.

One problem we've encountered this year is that we need a few network devices to be on the same layer two network while being two miles apart. These two sites don't have line of sight, so we're using two microwave links to bounce off a third site between them, while these links also need to carry a few other L2 domains. A perfect application for VLAN tagging.

The problem is that this middle site needs to run several repeaters and all of it's network gear off of a generator and batteries for all weekend. The traditional technique of using managed rack-mount switches on every hop of a VLAN trunk is problematic since a single rack-mount switch exceeds our power budget for all the network gear at the middle radio site. Ideally, we find a small low-power managed switch to use, but I really want to just use a 5 port dumb workgroup switch in the middle since it runs straight off of 12V and only consumes a few watts.

Conventional wisdom dictates that you can NOT move 802.1Q VLAN tagged traffic through unmanaged network switches.


Plot twist: apparently this is wrong. I took the time to set up a test where I used two L2 managed switches to tag and untag Ethernet traffic, and then put various unmanaged switches between them on their trunk line, and the VLAN tunnel kept working... This is really unexpected; I've had several networking techs tell me prior that what I did wasn't possible, since the MTU of Fast Ethernet switches is only 1514 and the extra four bytes added by 802.1Q will break things.

As far as I can tell, none of the possible failure conditions we came up with cropped up during testing:

  • Dropping the 1514+4 frames.
  • Crashing
  • Truncating the last four bytes
  • Severely lowered throughput (The switches even continued to perform MAC learning)
Taking my experiment a set further, I plugged the unmanaged switches between a pair of GigE Linux systems and bisected the maximum L2 MTU that the switches could handle. The minimum needed for standard Ethernet is 1514, for VLAN tags 1518:
  • SD216 v2.1 - 16 port Linksys Fast Ethernet switch - 1532
  • SR224 - 24 port Linksys switch - VLANs worked, but physically destroyed before maximum MTU could be measured.
  • ASW308P vA2 - 8 port AirLink 101 PoE switch - 1532
  • FS608 v3 - 8 port NetGear switch - 1532
  • DS104 - 4 port dual-speed hub (!) - at least 4014. NIC MTU limited further testing
So it would appear that the standard MTU for Fast Ethernet switches isn't 1514, but actually 1532, which leaves a comfortable margin for the extra four bytes needed for 802.1Q tagging. Am I missing something, because I really thought this wouldn't work before I tested it.


For reference, here is the MTU's of the rest of the hardware I used for these experiments, on layer 3:
  • RTL-8139 Fast NIC: 1500 L3
  • BCM5722 GigE: 1500 L3
  • RTL-8169 GigE: 7152 L3
  • Intel 82571EB dual-GigE: 9216 L3
  • RTL-8111G GigE: 4080 L3
  • NanoBridge M5: 2024 L3 (set via web interface)
I didn't bother testing if anything allowed a larger than 14 byte Ethernet header with these L3 MTUs, so you may have a bad time trying to run VLANs with the MTU cranked all the way up. I also didn't bother researching device drivers, so you may be able to push these higher with not stock Debian drivers.

I also never saw ANY device successfully send ICMP responses for MTU discovery, so jumbo frames appear to still definitely be a thing for specially designed networks, for the record.



My testing process involved increasing the Linux system MTUs via "ifconfig eth0 mtu ####" until I got a "SIOCSIFMTU: Invalid argument" error, then placing the unmanaged switch between the two systems running iperf and lowered the MTU on one of them until the TCP connection stopped black-holing into the switch, which all silently dropped over-sized jumbo frames.

Sunday, January 12, 2014

NTP Multicast Servers

Anyone who's followed this blog or gone back and read my past posts has probably picked up on my somewhat unhealthy obsession with time. Every time I get my hands on a new display or microcontroller, the first thing I do is build a clock out of it (one, two, three, four, five, six, seven, eight, nine). I've also dabbled in high precision time with my rubidium frequency standard and the multitude of frequency counters in my apartment. One area that I haven't had as much success with time in the past is keeping all of my Linux systems synchronized, and that's been bothering me.

I finally sat down this week and spent a few hours figuring out how to get multicast network time protocol working on Debian, since I've already put so much work into getting multicast routing working on my apartment's LAN and want to actually use it for something. There were a few critical pieces that all came from different places online, so I figured I'd collect them all in one place.

Typically, NTP operates by configuring each client with a server address to periodically poll for the current time. During each poll, the client tries to determine the remote server's time and how long it took for the server's response to get back to the client. For remote Internet servers, this time delay can be in the order of 20-100ms and change wildly based on network conditions. The traditional solution to this is to have a single local system poll several remote NTP servers, and then manually configure each local client to poll just this one local server, which in my case is my WNDR3800 router. This works acceptably well, but just isn't complicated enough to meet my desire to make life hard on myself.

Multicast NTP is based on multiple servers beaconing their local time towards a multicast address, which clients listen for to discover these servers. When a client discovers a new server, it first performs a standard unicast query with the server to determine network delay between the two systems, and then sits back and silently listens for future multicast beacons from the server to keep the client's local time in sync. All this does for me is allow my NTP clients to dynamically find new and forget old NTP servers on my network when I swap in and out computers. Another advantage where multicast really shines on huge computer networks is that after the initial propagation measurement, clients don't generate any additional network traffic, but only passively listen for the multicast beacons from the servers.


Configuring NTP Multicast Servers

On my systems that I've designated as my NTP servers, they do the typical pulling of four unique servers from *.pool.ntp.org DNS entries, but then act as both multicast servers and clients so they can discover and synchronize with each other.

Sample Config and ntpq output:


A few tripping points in the configuration file:

  • The default restrict statements prevent discovering peers, so you need to remove the nopeer statement from the IPv4 and/or IPv6 restrict statements.
  • The broadcast statement defaults to "ttl 0", which means the 0th entry in the eight value ttl array used by manycasting. This means that the argument to ttl isn't actually a time to live value, but an index into the default array of ttl values {1, 32, 64, 96, 128, 160, 192, 224}, which you can re-declare if that doesn't meet your needs.
  • Unlike broadcast clients, for multicastclient you need to specify the address to listen to.
  • /etc/ntp.conf isn't the config file that NTP actually uses. If you look at /etc/init.d/ntp, you'll discover that by default, ntpd uses the config file /var/lib/ntp/ntp.conf.dhcp, which is built by appending /etc/ntp.conf to any NTP server advertisements from the system's DHCP server. /var/lib/ntp/ntp.conf.dhcp isn't rebuilt when you run "service ntp restart", which is why your changes to /etc/ntp.conf don't appear to do anything until you reboot the entire system! I haven't bothered to figure out which init script triggers this config rebuild, but have been doing testing on the /var/lib/... file and then manually copying the config back to /etc/ntp.config
The selection of the multicast address is pretty arbitrary. What I have shown isn't the actual address I use, but anything in the 239.192.0.0/14 or ffx8::/16 organizationally-local multicast address subnets are technically correct and should work fine as long as all of your systems match.

Monday, January 6, 2014

My First Adventure in Homebrewing Beer

Yes, I am in fact alive. Last quarter ended up being more work than I had expected, and teaching a section of Electronics Lab for Non-Majors sapped away a lot of the instructional energy that I normally blow off on this blog.

Every year, I try and find a new hobby to broaden my life experience with. 2012, it was getting back into Amateur Radio; 2013, it was shooting (mostly trap, though I'm slowly getting into rifle...); 2014, I'm thinking it's going to be homebrewing beer.

I've been toying with fermentation for a while. Most notably, last winter/spring I had a few false starts that ended with me throwing up in my bathroom and my friends laughing their asses off. After that, two experiences really convinced me to go all in and re-approach the hobby more rigorously this year.

  • I helped my buddy Robbie brew a batch of beer using a grossly expired Mr. Beer brew kit, and the beer still came out drinkable. Tell-tale soapy tastes of a 4 year old extract malt, but not the worst beer I've had.
  • Which was the second experience. Killing time with another buddy, Sean, late last quarter, we stopped at a liquor store and bought a variety pack of "Brewer's Mystery" beer from some local brewery that I had never heard of, mainly because it was the only case of beer they had that we hadn't already been drinking in the last week. It was awful. It was the worst beer I had ever experienced in my life. It wasn't a "eh, this isn't very good" bad; it was a "I immediately regret opening this bottle and not leaving it for someone else" bad. We eventually managed to pawn it off at some random college party, but the trauma was already done.
So, when Robbie and I manage to brew a better beer with four year old malt and yeast than someone seriously sold us in a liquor store, it's hard not to figure that this is a hobby I can't be too unsuccessful at. Queue a trip to the local homebrew shop here in San Luis Obispo (Doc's Cellar), where I buy a copy of Palmer's How to Brew. It's a very nicely laid out book:
  • The first chapter is a crash course in turning the crank to brew your first batch of beer, without focusing on understanding too much of what's happening.
  • The rest of the book is then divided into three major sections for extract, partial grain, and full grain brewing. This means that you get to start simple and make the brewing hobby as complicated as you want to.
  • It is also available in website form, although I definitely don't regret having a hard copy to read in bed and have in the kitchen.
Thanks to my apartment kitchen already doubling as a wet lab for various projects (PCB etching, etc.), I can already cobble together most of what I needed to brew my first batch. What little I still needed (mainly a hydrometer and bottle capper), I put on my Christmas list for my parents, who always appreciate when I ask for more tactile things than a textbook about knots for holidays. 


Starting My First Baseline Batch

Being the engineer that I am, the first thing to do is run a small batch of the most generic beer recipe I can come up with as a baseline to compare against. This is based on Palmer's Cincinnati Pale Ale recipe, with substitutions based on availability at Doc's Cellar and only doing a half batch at a time (2.5 gallons).
  • 1.5 lbs pale malt syrup ($3.38)
  • 1.5 lbs Munich malt syrup ($3.38)
  • 7g Safale US-05 yeast ($1.75)
  • 0.5oz Cascade hops for 60 minutes for bittering ($1)
  • 0.5oz Cascade hops for 10 minutes for finishing ($1)
Putting the expense of consumables for the first batch at $10.51, not counting the ~$100 of equipment I've collected/bought/been gifted over the last six months.
First thing is to boil the malt and hops. This pasteurizes the wort so that the only thing growing in it ends up being the yeast we want, changes the chemical composition of the malt, and steeps the hops. I did a partial boil with 1 gallon of water, which I then diluted to 2.5 gallons in the final fermentation chamber, which isn't ideal for steeping the hops, but the best my largest pot allows.
By adding half of the Cascade hops at the beginning of the boil, and holding the second half until the last ten minutes, I get to reduce half of it to just the bittering agents, while still retaining some of the aroma and more volatile flavoring from the second half.
Picture well into the boil. I didn't bother tracking boil-off water loss, so I'm likely going to end up with about 2 gallons of beer in the end, once you factor in water loss here and in the bottom sediment when I bottle in a few weeks.
The hour of boiling the wort breaks down lots of complex proteins that you want to settle out of the beer. Once that happens (which makes it look a lot like egg drop soup), you want to cold shock the wort to pull out more proteins, which mainly cause a hazy appearance when you chill the beer, but also do affect the long-term stability of the beer (which is unlikely to be a problem around college students...)
The pot is placed in one sink filled with cold water, while ice water is circulated through the copper tubing coil I fashioned from left-over copper tubing from a boiler experiment I did in the past.
Once the wort is cooled down to below 70°F, I dilute it to 2.5 gallons, pitch 7g of rehydrated yeast, and split the wort between a 3 quart carboy and a 5 gallon bucket so I can visually track a small sample of the batch.

The initial specific gravity of the batch ended up being 1.047, using only the malt syrup and no added sugar. This, minus the final gravity in a few weeks, can tell us how much alcohol my little yeasty buddies here created, which isn't going to be anything to phone home about. Then again, if I were doing this just to try and get drunk, I'd save the trouble and just drink more rum from my liquor closet...

The first definite signs of life took about 16 hours. These last two pictures were taken 41 hours after pitching the yeast. Lots of churning going on in the wort, the airlock is bubbling every ~12 seconds, and all that nasty looking foam on the top is slowly growing (that is totally normal).

Now I need to wait three weeks, bottle this batch with a small addition of sugar for carbonation, and let it bottle condition for another few weeks. Come about mid-February, I'll get to test this first batch. Until then, I just need to leave it alone and figure out how I want to specialize in this hobby.

Wednesday, September 25, 2013

DTMF Decoder Test Circuit

 As part of a few projects I'm working on, I want circuits to be able to decode DTMF tones played over an amateur radio link for remote control. This can be done with a moderately powerful processor in DSP, but I opted to buy a handful of MT8870D dedicated DTMF decoder chips on eBay.

I wired together a quick little test circuit, and it worked right away.
Video:


The built circuit is straight out of the datasheet on page 15.
I'm feeding audio into it from a GM300, which is a common commercial radio that with some effort you can get working on the amateur bands. I haven't done any testing yet as to how noise immune this DTMF decoder is, but I expect most of my remote control commands to be done with enough power that radio link noise won't go beyond the datasheet specs.