Wednesday, March 26, 2014

Range Testing NanoBridge M5s

In preparation for the Wildflower Triathlon in May, I've been testing many of the major components of the computer network I'm helping build for communications. One of the big unknowns this year is the unusually large number of mid-range microwave links, which I've never had too much experience with. WDS links across buildings or small parks I'm comfortable with, but we are looking at streaming HD video through WiFi hops that are multiple kilometers long, and I've just never played with links like that before.
To gain some experience before the final deployment, I bought a pair of Ubiquiti NanoBridge M5 nodes. These are 200mW 5GHz WiFi access points with optional high-gain dishes, and at $80 a piece are pretty reasonable for what you get.

The NanoBridges come with either 22dBi or 25dBi dishes, pole mounting hardware, and a passive PoE injector. Luckily, Ubiquiti uses the kind of standard passive PoE injector pinout, so I often use the cheap 2.1mm barrel PoE injectors you can get on eBay for $2 instead of using the OEM injectors from Ubiquiti so I can power them off 12V batteries for portable operations. I selected the lower gain 22dBi dishes because I plan to be moving these around a lot so the smaller dishes are a nice convenience.
I think the NanoBridges claim to have a 20km range, which isn't an unreasonable number if you look at the link margin calculations, but since I've never done a deployment like this before I did want to do a real-life deployment before trying to do it across lake San Antonio. I gave one of my buddies, Robbie, a six pack of beer and one of the two dishes and told him to point it at Cuesta Ridge north of SLO. I then drove up the horrific road to the top of the ridge and pointed the second dish back at his apartment. Once we got both of them turned on and roughly pointed in the right direction, the link came right up at full speed.

At 6.5km, this test is longer than any of the links that we're going to need for Wildflower, so I was very happy when the link came up without any fine tuning of either dish's aim. I was even able to aim one of the dishes 30-40 degrees off center and the link stayed up, although with a smaller link margin.
AirOS, which is what most Ubiquiti nodes run, has a really neat spectrum analyzer mode that I used to help select channels for the test. The first screen shot is from Cuesta Ridge looking at the second node down the hill beaconing its SSID.
This is a capture of my MacBook Air transferring files from my apartment's AP using a 40MHz channel.
This is another spectrum scan from the radio site on top of Cuesta. Interestingly, none of these have the distinctive 802.11 spectrums, but are other modulation types from non-802.11 equipment using the 5GHz ISM band. I thought they were really interesting.

So in short, I'm really happy with my NanoBridge M5s, which couldn't be much easier for setting up a ~80Mbps effective throughput link between any two points with line of sight. 6.5km was no problem, and shorter links will only enjoy a larger power margin to allow for non-perfect aiming, rain fade, etc.

Thursday, March 20, 2014

Pushing VLAN Tags Through Unmanaged Switches

Now that it's Spring Break and I'm in San Luis Obispo, it's full speed ahead on building the communications network for the Wildflower Triathlon that CPARC supports every year. Wildflower is a very big event in the middle of nowhere (Lake San Antonio), so we have to build quite a bit of infrastructure to support the operation.

This year, I was designated as the computer network zonie, so it's my job to make sure that there's IP connectivity between all the major sites in the network. This involves building a computer network that includes a couple Internet hand-off points, multiple routers, several medium-range (2-5km) microwave links, QoS enforcement for a few hundred devices to support VoIP and streaming video while sharing a 15Mbps Internet uplink, a couple 802.1Q VLAN trunks, etc.

Needless to say, we are building a network beyond the budget we're being given, so duct tape and Linksys devices are being applied liberally throughout this project.

One problem we've encountered this year is that we need a few network devices to be on the same layer two network while being two miles apart. These two sites don't have line of sight, so we're using two microwave links to bounce off a third site between them, while these links also need to carry a few other L2 domains. A perfect application for VLAN tagging.

The problem is that this middle site needs to run several repeaters and all of it's network gear off of a generator and batteries for all weekend. The traditional technique of using managed rack-mount switches on every hop of a VLAN trunk is problematic since a single rack-mount switch exceeds our power budget for all the network gear at the middle radio site. Ideally, we find a small low-power managed switch to use, but I really want to just use a 5 port dumb workgroup switch in the middle since it runs straight off of 12V and only consumes a few watts.

Conventional wisdom dictates that you can NOT move 802.1Q VLAN tagged traffic through unmanaged network switches.

Plot twist: apparently this is wrong. I took the time to set up a test where I used two L2 managed switches to tag and untag Ethernet traffic, and then put various unmanaged switches between them on their trunk line, and the VLAN tunnel kept working... This is really unexpected; I've had several networking techs tell me prior that what I did wasn't possible, since the MTU of Fast Ethernet switches is only 1514 and the extra four bytes added by 802.1Q will break things.

As far as I can tell, none of the possible failure conditions we came up with cropped up during testing:

  • Dropping the 1514+4 frames.
  • Crashing
  • Truncating the last four bytes
  • Severely lowered throughput (The switches even continued to perform MAC learning)
Taking my experiment a set further, I plugged the unmanaged switches between a pair of GigE Linux systems and bisected the maximum L2 MTU that the switches could handle. The minimum needed for standard Ethernet is 1514, for VLAN tags 1518:
  • SD216 v2.1 - 16 port Linksys Fast Ethernet switch - 1532
  • SR224 - 24 port Linksys switch - VLANs worked, but physically destroyed before maximum MTU could be measured.
  • ASW308P vA2 - 8 port AirLink 101 PoE switch - 1532
  • FS608 v3 - 8 port NetGear switch - 1532
  • DS104 - 4 port dual-speed hub (!) - at least 4014. NIC MTU limited further testing
So it would appear that the standard MTU for Fast Ethernet switches isn't 1514, but actually 1532, which leaves a comfortable margin for the extra four bytes needed for 802.1Q tagging. Am I missing something, because I really thought this wouldn't work before I tested it.

For reference, here is the MTU's of the rest of the hardware I used for these experiments, on layer 3:
  • RTL-8139 Fast NIC: 1500 L3
  • BCM5722 GigE: 1500 L3
  • RTL-8169 GigE: 7152 L3
  • Intel 82571EB dual-GigE: 9216 L3
  • RTL-8111G GigE: 4080 L3
  • NanoBridge M5: 2024 L3 (set via web interface)
I didn't bother testing if anything allowed a larger than 14 byte Ethernet header with these L3 MTUs, so you may have a bad time trying to run VLANs with the MTU cranked all the way up. I also didn't bother researching device drivers, so you may be able to push these higher with not stock Debian drivers.

I also never saw ANY device successfully send ICMP responses for MTU discovery, so jumbo frames appear to still definitely be a thing for specially designed networks, for the record.

My testing process involved increasing the Linux system MTUs via "ifconfig eth0 mtu ####" until I got a "SIOCSIFMTU: Invalid argument" error, then placing the unmanaged switch between the two systems running iperf and lowered the MTU on one of them until the TCP connection stopped black-holing into the switch, which all silently dropped over-sized jumbo frames.