Monday, December 31, 2018

FCIX - State of the Exchange

The First Year of Running an Internet Exchange

It has been a little over a year since one of my friends challenged me on a whim to get an autonomous system number and stand up my own little corner of the Internet, and what a long slippery slope that has been. One of the advantages to running your own autonomous system is that you can blend your own connection to the Internet via peering, so as we continued to make more friends in the Hurricane Electric FMT2 data center who we wanted to peer with, the number of desired cross connects started to get out of hand; particularly since they aren't free, and we have all of about zero budget for... just about everything.

It's that quadratic growth of the number of interconnects in a full mesh that really get you.

But that's exactly the problem that Internet Exchange Points are meant to solve; I have N number of networks that want to interconnect in one place without running O(N2) Ethernet cables between each other, so everyone connects to one central Ethernet switch and it's just as effective at a much lower cost of entry.

So eight months ago, we jokingly registered the domain, grabbed a spare /24 + /64 of address space we had laying around, and founded the Fremont Cabal Internet Exchange.

We had a good laugh setting up a cheeky little website to make it look like we're a real Internet Exchange, which lasted all of about two weeks before the owner of the data center, Hurricane Electric, applied to our Internet Exchange, brought in the 75,000 prefixes from their customer cone, and put us on their advertising material for the building.
Well crap. That joke got out of hand rather quickly.

Membership Growth

For the first few months there, we were handling two or three new membership applications per week, so it became evident that we needed to get our act together rather quickly.

The first point of order was dealing with the fact that we were running this no-longer-just-between-friends Internet Exchange on borrowed address space, so we needed to get the exchange its own ASN and /24 + /64, but that's $550 for the ASN and $250 for the resource service agreement for the address space... A bit of a problem when we have zero budget...

But at that point we had the advantage of having ~15 current + pending members, so we passed around the hat and between our membership and some other very amused on-lookers, very quickly managed to scrape together the $800 needed to cover the registration costs of FCIX's resources.

Not only did this enable us to re-number onto a real ASN and IXP address space, but getting such a concrete signal of support from our members was touching. This thing wasn't much of a joke anymore; we're actually providing a service to networks which they value enough to throw a few hundred bucks our way to make it happen.

New membership applications have slowed down, but as of this writing we're up to 25 members, which I don't think is half bad for a less than one year old exchange in the east bay in a single site.


So after getting a round of donations to cover the start-up costs for the ARIN resources, the next question was how to handle getting networks actually connected to FCIX. Originally, we were just running FCIX as a VLAN on my Cisco 6506 which was powering my personal network (AS7034), but that suffered from a few issues, the largest of which is that the 6506 is so old that 10G Ethernet was a cutting edge feature at the time. At best, 10G line cards for the 6500 support 16 ports, but the 10G line card we had managed to scrounge for AS7034 only had four ports for XENPAKs, and burns 100W per port, so offering 10G for FCIX was going to be problematic at such a low port density, even ignoring the issue of sourcing line cards and such vintage optics.

This is where Arista stepped in and has contributed to FCIX in a huge way. I got a call from a long-time friend who works at Arista who liked what we were doing and was interested in getting us a real switch to run the exchange on. This means that we got a pair of Arista 7050S-64 switches, which have 48 SFP+ ports which can support either 1G or 10G optics, plus another four QSFP+ ports for 40G, because hey, maybe we'll need 40G at some point...

This now only left the issue of optics. Every member port that we turn up needs an LX or LR optic, which even from third party vendors start to add up quickly (remember how we have zero budget?), so we were very rapidly tapping out our junk bins of left-over optics we all had laying around. So while we sat there brain-storming ways to work around this sustainability problem, we got an awesome direct message on Twitter from Flexoptix!

Flexoptix is a third-party optical transceiver vendor who has the additional advantage that they sell what they call their "FLEXBOX" which allows you to insert one of their optics and over USB reprogram their optic for any vendor's switch which you need, so even though we've got that scrappy "we'll use whatever switch we can drum up" aesthetic to us, we only need to stock one tray of 1G and 10G optics to cover any possible switch we'd want to plug these optics into. Furthermore, as we moved from my Cisco 6506 to our shiny new Arista, we were able to simply reprogram the optics and reuse them, so already the flexibility of their universal transceivers have borne fruit.

The Cost of Entry

Having been started originally as mostly a joke, we've been very against charging any kind of membership fee to join FCIX, for multiple reasons:

  1. There is already several established pay-to-play IXPs in the bay area, so trying to charge for ours when we have none of the valuable peers that existing exchanges have would be kind of silly.
  2. We all have real day jobs, so if someone paid to join FCIX and it then stopped working during the day, they'd rightfully kind of expect us to get it working again ASAP. My day job boss probably wouldn't appreciate that, so we don't charge anything, and problems get fixed when we can get to them. Outages, of course, are refunded in full. (see what we did there?)
This sort of zero cost of entry and zero membership fee model definitely wouldn't have been possible without all the donations we've gotten, and particularly Flexoptix donating trays of optics so we can light our end of every new member's port without trying to deal with every new member somehow contributing an optic to FCIX to light their port.

This has meant a few issues with people trying to abuse our free model, so we very quickly needed to institute an informal "one port per cabinet" rule, since we at one point got six applications from different people sharing one cabinet, and although Hurricane Electric is sponsoring all the cross connects, I'm not going to abuse that deal to run six pieces of fiber to one cabinet. Charging a one time turn-up fee like SIX does probably would have been a good idea and prevented most of our issues with low quality membership applications.

Plans for Growth and Value Add

At this point the basic framework for the exchange has been set up. Adding new members is relatively painless and mainly involves generating them a letter of authorization to redeem for a free cross-connect from HE and adding them to a CSV which propagates to the website and route servers automatically.
The most exciting piece of news with regards to growth is that Hurricane Electric has agreed to sponsor FCIX with a second cabinet in their other building, FMT1. That, plus a pair of dark fiber between the two buildings (thanks HE!), plus a pair of LR4 40G optics (thanks Flexoptix!), plus a second switch (thanks Arista!), and FCIX will soon be multi-site! Membership applications from FMT1 are now open.

The other challenge has been coming up with projects to focus on to help increase the value of the exchange to existing members. Adding new members is easy, but we have also been working on getting things like cache appliances and DNS servers on-net. Verisign was kind enough to contribute a J root + .com/.net DNS server to the exchange, so anyone running their own recursive DNS resolvers get to enjoy direct access to J root and B.gTLD over the exchange. Work on other value-add appliances is on-going.

As we head into this new year, I couldn't be happier or more grateful with how far we've gotten with this project while keeping it sustainable. The annual expenses specific to the exchange are still below $500 between our ARIN fees and other misc fees, and contributions of hardware and money from sponsors and members have enabled us to grow much further than we would have been able to fund on our own.

So thank you again to everyone involved in FCIX, and I wish all of you a lovely 2019.

Saturday, November 24, 2018

Inspiration for Date Nights

My previous romantic relationship was recently classified as "not successful," which is one of those things that generally sucks. A lot.

After a few days of just being all around miserable about it, I moved on to the RCCA phase of me moping around after a relationship and tried to take a hard look at the Root Causes and Corrective Actions for my part in the relationship not being successful. One of the issues I identified was that my tendency as an introvert to not plan outings doesn't lend itself well to building healthy relationships, so I clearly need a more concrete framework for date night.

Putting a few days into the idea during these holidays, this is what I came up with: trawl as many click-bait "top 100 date ideas" articles as I can stomach cherry picking any that I thought I'd enjoy with someone or wouldn't be a terrible way to push the envelope with someone, and write each one on a 2.5"x3" index card. Each card isn't an actionable plan, but a category of activity (i.e. "Go for a hike"), with space below it to fill in specifics. Whether the most specifics are something I do on my own or I sit down with my partner to spend an evening workshopping "exactly where would we be interested in hiking?" to fill in the rest of the space is unclear, but I'm interested to see if this proves helpful to me in the future.

Since others have already asked for this list, I figured I could post my current collection of cards here. Some of them have pretty obvious "more specifics" to be filled in, but some of them do suffer from the difficulty of figuring out exactly how to find a place to do them:

  1. Visit a local tourist destination
  2. Visit an aquarium/zoo
  3. Go to a library together
  4. Go shopping at a thrift shop together
  5. Go shopping at a flea market together
  6. Go on a local walking tour
  7. Go to a movie
  8. Go for a hike
  9. Go camping
  10. Night of coloring/doodling
  11. Host a dinner party
  12. Host a board game night
  13. Go an art gallery
  14. Visit a museum
  15. Work on a home depot project/build something
  16. Have a picnic
  17. Go to a pottery class
  18. Go fruit picking
  19. Take a cooking class
  20. Sample drinks at a bar/brewery/vineyard
  21. Morning coffee date/brunch
  22. Day at the beach
  23. Go to a used book store
  24. Make dinner together
  25. Go to an amusement park
  26. Go to a county fair
  27. Take a boat/ferry ride
  28. Go to a farmers market
  29. Go play pool
  30. Go out bowling
  31. Go to an arcade
  32. Go to a shooting range
  33. Train for and run a race together
  34. Brew beer together
  35. Visit a local nursery for house plants
  36. Read the same book
  37. Visit a botanical garden
  38. Work on a puzzle together
  39. Go play miniature golf
  40. Go garage/estate sale hopping

Monday, July 16, 2018

A Quickstart Guide to IRR & RPSL

In case you are in the very limited pool of people in the world who are responsible for an Autonomous System and need to set up IRR for peer route filtering, I've written a whitepaper on the matter for FCIX.

If you aren't a network engineer for an ISP, I suspect that article will be less interesting on its own.

Sunday, July 8, 2018

Measuring Anycast DNS Networking Using the RIPE Atlas

As I have previously mentioned, I set up a RIPE Atlas probe in my autonomous system to help measure the Internet and accrue myself Atlas credits to spend on my own custom measurements. As I have been looking for various networks to invite into our newly created Internet Exchange, anycast DNS is a good candidate since they're networks that want to be at a lot of IXPs (Internet Exchange Points) and tend to be tiny (typically a single 1U server or VM) so we are able to just offer them free hosting as part of the exchange.

Anycast DNS is where you pick one global IP address that all your users query against, and instead of tolerating the nuisance of the quite limited speed of light (lol) back to a single server, you create multiple identical copies of your server (even down to using the same IP address) and distribute them across the entire Internet. So identical that it really doesn't matter which server the query goes to; they all come back with the same answer anyways. This ability to have the same IP address in multiple places is made possible by dedicating a whole /24 IPv4 block and a /48 IPv6 block to the server's public facing address (these size subnets are driven by those being the smallest block of IP addresses generally accepted in the global BGP routing table). You then have the server itself advertise those two IP blocks into BGP itself, and as networks see the same advertisement from each of the distributed servers, they use the typical BGP best path selection process to select the "closest" server (where closest is unfortunately in the sense of network topology and business policy, not necessary linear distance or latency like you'd hope).

So this is great; anycast is used to make it seem like a single server is beating the speed of light with how fast it's able to answer queries from across the globe, and is a popular technique used by practically all of the root DNS servers, as well as many other DNS servers, be they authoritative DNS servers for specific zones or public recursive DNS resolvers like those provided by Google or CloudFlare.

So my challenge to myself has been to find any of these anycast networks who would plausibly be interested in tying into our IXP, and would benefit from an anycast node added in California.

This. This is where my RIPE Atlas credits I've accumulated until now come in super handy. I have the entire global network of RIPE Atlas probes at my disposal to poke at anycast servers to measure their performance and see how good their global deployment really is.

The process consists of these major steps:
  1. Trawl the Internet for anycast DNS services and "Top [Number] DNS Servers You Should Use Instead of Whatever You're Using" articles listing popular DNS services (odd, often in a very similar order to the other "Top [Number] DNS Servers" articles...) and collate all of these into a list of potential DNS networks to measure.
  2. Craft a DNS query which can be sent to each instance of an anycast server and its response time measured to give a good indication of how well the anycast network is performing.
  3. Roll this query out to 500 RIPE Atlas probes and spend some super exciting evenings analyzing the data to try and discover what each DNS network's current topology is, and if they'd seemingly benefit from a node in Fremont. 
So that first step isn't too bad; that's just some Google-Fu and a willingness to really confuse ad networks with my sudden willingness to read click-bait articles. 

The second step is where it starts to get interesting, because we need to somehow measure how fast the closest anycast node can answer a DNS query. There are DNS benchmarking tools which run through a few hundred of the most popular websites and try and resolve those to see how quickly the DNS resolver responds, but those measurements are a little problematic for what I'm looking for, because they don't only measure the network round-trip-time to the anycast node, but also how hot its cache is and how well connected it is to the actual authoritative DNS servers. If I was an end-user looking to pick the best DNS server, how quickly it can answer actual real DNS queries is important, but that's not really what I'm interested in. I want to know how quickly a random place on the Internet can get a query to this server, and after the DNS server has gone off and done all the work of resolving the query into an answer, how long it takes to get that answer back to the client. (In the case of authoritative DNS servers which don't provide open recursive resolution, these "top 500 websites" benchmarks are worthless, since the authoritative servers will only answer queries for zone files they're hosting themselves)

So really, I need a DNS query which the anycast servers can definitely answer themselves locally, without (maybe) having to send off queries to other DNS servers and walk the DNS hierarchy trying to find the answer. And hang up the rotary phone guys, cause most servers actually support a query that meets this criteria, and it even usually returns a TXT answer which identifies which specific instance of the anycast network you queried, which is much easier than running a traceroute from each probe to the DNS server and trying to analyze 500 traceroutes to figure out which end-node they happen to be getting routed to. 

The "id.server" domain name (as documented in RFC4892) is a handy TXT record, which (for presumably very histericalorical reasons) is a CHAOS class domain instead of an IN class domain (like every other DNS query you're used to) and tells you a unique name for each instance of a DNS server. By convention, global things like backbone routers and anycast DNS servers identify themselves by the nearest IATA three letter airport code, so knowing that, it's often quite successful figuring out generally where a DNS answer is coming from.

A Specific Example

So as an example, let us study one of the anycast DNS resolvers out there; namely, UncensoredDNS, an open resolver based out of Denmark. They happen to have both a unicast node hosted in Denmark, as well as an anycast IP address (, 2001:67c:28a4::) hosted from multiple locations. 

The first thing to do is confirm that this DNS admin has happened to implement id.server, so we can use a local tool like nslookup to query CHAOS: id.server to see if it even works.
nslookup -class=chaos -type=txt id.server

Gives us an answer of: ""

Excellent! So this tell us two things:
  1. They answer to id.server, so it's possible for us to run that query using the Atlas network to get a good perspective on where they are.
  2. They seemingly put something useful in their id.server string, namely perhaps a hosting network name and a city designator (although LGB is Long Island, and based on the 153ms ping time from Fremont, California, I doubt it's really in the same state as me). So maybe not entirely IATA city designators, but something unique at least...
So now we need to create a custom Atlas measurement to run this query from everywhere and see what comes back and how quickly. 
From the "My Atlas" dashboard, I select "Create a New Measurement". The measurement creation wizard has three parts:
  1. Measurement definition: This is where we specify to query "CHAOS: id.server" against the DNS server's IP address
  2. Probe selection: This is where you decide how many probes you're willing to spend credits on to take this measurement. Since these measurements are relatively cheap, I max this out at the allowed 500 probes.
  3. Timing: Since I'm not trying to monitor these networks but just characterize them, I just make absolutely sure that I check the "This is a one-off" box so the measurement doesn't get re-run on a schedule.
For the test definition, add a DNS measurement for each DNS server address you're interested in querying (I kind of wish there was a "clone this other measurement definition" button, but regardless you can create multiple measurements to run against the same probe set). When filling out the test definition, the important fields are to set the target to the DNS server of interest, change the query class to CHAOS, the query type to TXT, and the query argument to "id.server". I also give the test a meaningful description so I stand a chance of finding it later, and tagged it "dns" so my DNS measurements are all generally grouped together. 

If you're scheduling a long-running measurement, the interval field would be important, but since I'll be checking the one time off measurement box in the timing options, this interval field will eventually disappear.
By default, measurements are run on ten probes randomly selected from anywhere in the world, but we want a larger test group than that, so press the X on the "Worldwide 10" selection and "+ New Set - wizard" to open the map to add more probes.
Ideally, you would be able to just type in "Worldwide", select it, say you want 500 probes, and be done with it, but the problem is since RIPE is the European RIR, the concentration of probes in Europe is unusually high, so an unspecified "worldwide" selection tends to have the majority of its probes in Europe and a thin select everywhere else. Since I'm particularly interested in the behavior across the continental US, but also curious what each anycast server's behavior looks like worldwide, I've settled on a bit of a compromise of selecting 250 probes from "worldwide" and 250 probes from "United States". This ensures a good density in the US where I need it, but still gives me a global perspective in the same measurement.
Type in worldwide, click on it in the auto-completion drop-down, and it should pop up a window on the right asking you how many probes you'd like to select. I choose 250, and press yes.
It will then ask you which tags you'd like to include or exclude when selecting probes. The system automatically tags Atlas probes with various categories like what hardware version they are, or how stable their IP addresses are, or if they're able to resolve AAAA records correctly, etc. I'm unsure if Atlas is smart enough to do this already, but particularly when measuring an IPv6 DNS server, I make sure to specify that probes tagged "IPv6 Works" or "IPv6 Stable 1d" should be used. For an IPv4 measurement, filtering on tags is probably not critical, but I usually select "IPv4 Stable 1d" just for good measure.
Click add and the 250 worldwide probes get added to the sidebar, and repeat the same process for the United States to get up to the maximum 500 probes. Once that's done, press OK and it returns to the measurement form.
The probe selection box should then be filled in with the 250 + 250 selection. In the Timing section, simply check the "This is a one-off" box, and the measurement(s) are ready to launch.

Press the big grey "Create My Measurement(s)" button.
At this point, if all goes well, it should pop up with a window giving you a hyperlink to your measurement results. The results won't be immediately available, but for one-off DNS measurements it should only take a minute or two before most of the results start populating on the results page. Link to the results of this example measurement.
For the measurements I'm doing, the most useful results page is the map, where it color codes the results by latency, and you can then click on any single dot to see which probe that is and what result it got for the TXT query we requested.
Looking at the map, you can see some green hotspots in Europe, so his coverage in Europe seems to be pretty good, and there's a green gradient across the US east to west, so there's seemingly a node on the east coast. Clicking on any of the green east coast probes tells us that their TXT result was "", where IAD is the IATA code for Washington DC, so chances are the east coast node is there.

Browsing around on the west coast, there's zero green measurements, and most of them are in the ~150ms range coming from servers located in Europe, so this result actually points to this being a reasonably good candidate of an anycast network to try and bring into our exchange...

Wednesday, June 13, 2018

Peering with Root DNS Servers

The Domain Name System is a recursive system for resolving host names to IP addresses and from IP addresses back to host names, which is really handy, since ideally no one interacts with IP addresses and instead refers to servers by names like or

When you're resolving a hostname like, it's actually a multi-step process where you first figure out the DNS server for the .com domain, then ask them where the name server for is, then you ask them what the address for the "blog" server is. This is a well documented process elsewhere, but what I'm particularly interested in is that first little step where you somehow find the first DNS server; this is done by asking one of the 13 root name servers, which are 13 specific servers (lettered A through M) hard coded into every recursive DNS implementation as a starting point to resolve any other address.

The reason that I'm interested is because I recently became part of the team running the Fremont Cabal Internet Exchange. IXPs often peer with root name servers to make the fabric more valuable since root name servers tend to be really important for the other networks connecting to the IXP. This is possible because many of the root servers aren't implemented as one enormous DNS server in a specific place like you'd imagine, but are actually many identical copies of the same server advertising the same anycast prefix from every instance.

This means that even though we're a small IXP in the bay area, we actually stand a chance of an instance of several of the root servers being close by, or being willing to ship us the equipment to host an instance of them. We have spare rack space, so hosting their hardware to be able to increase our value and make the Internet generally better is worth providing them the space and power.

For curiosity's sake, I've been stepping through the list of root DNS servers to try and find what information I can on them, and figured these notes would be useful for some small fraction of other people online.

  • A ROOT - Run by Verisign
    • Homepage
    • Status: Only hosted in six locations; Ashburn, Los Angeles, New York, Frankfurt, London and Tokyo
  • B ROOT - Run by University of Southern California, Information Sciences Institute
    • Homepage
    • Status: Only hosted in Los Angeles and Miami
  • C ROOT - Run by Cogent
    • Homepage
    • Status: Only hosted in 10 locations; LA, Chicago, New York, etc
  • D ROOT - Run by University of Maryland
    • Homepage
    • 136 Sites
    • Partially hosted by Woodynet (AS42), which means they're already in FMT2
  • E ROOT - Run by NASA Ames Research Center
    • Homepage
    • Status: 194 sites
    • Also partially hosted by Woodynet (AS42), which means they're already in FMT2
  • F ROOT - Run by Internet Systems Consortium
  • G ROOT - Run by Defense Information Systems Agency
    • Homepage
    • Status: Only 6 sites, none in California
  • H ROOT - Run by US Army Research Lab
    • Homepage
    • Status: Only 2 sites; San Diego and Aberdeen
  • I ROOT - Run by netnod
    • Homepage
    • Hosting Requirements: Contact info[at]netnod[dot]se
    • Peering Requirements
    • Status: 68 sites, including one somewhere in San Francisco 
  • J ROOT - Run by Verisign
    • Homepage
    • Requirements include:
      • 1U space, 2x power
      • 2x network, peering LAN and /29+/64 management interface
    • Status: Already somewhere in San Francisco 
  • K ROOT - Run by RIPE
    • Homepage
    • Hosting Requirements include:
      • Provide a Dell server with 16GB RAM, quad core, 2x500GB HDD, etc.
      • Public IPv6 address with NAT64
    • Status: Seems the physically closest one is on TahoeIX in Reno.
  • L ROOT - Run by ICANN
    • Homepage
    • Hosting Requirements:
      • Sign NDA
      • Purchase code named appliance to host inside own network
    • Status: Somewhere in San Jose, per their FAQ they are not joining any additional IXPs.
  • M ROOT - Run by WIDE Project
    • Homepage
    • Status: Somewhere in San Francisco, nine sites total

  • Roots that will never be in the bay area: A, B, C, G, H
  • Roots already in the bay area: D, E, F, I, J, L, M
My rationale for the first list is that several of the root servers only have 2-10 instances spread across the world, so they're presumably not in the business of deploying the 100-200 anycast nodes that several of the other ones are. If they don't happen to already be in the bay area, it's not like we can afford to lease fiber out to where they already happen to be. 

Root servers on the "already in the bay area" list is also problematic since our exchange currently is only in Hurricane Electric's building, so if they already have a local node, it's unlikely that we would be able to convince them to build another node in the east bay just for us unless they already happen to be co-located with us.

But you'll notice that between those two lists, there's only 12 roots... K root isn't in the bay area. 
So I did some more digging. Using the Atlas probe in my rack, I can see that K root is currently 70ms away from us, so it has a not quite optimal latency to the bay area. It looks like it's currently reachable via its node in Utah, but it has a physically closer node in Reno connected to the TahoeIX.

TahoeIX is interesting for two reasons:
  1. They're a fantastic example of another tiny IXP who has done a remarkably good job of collecting value-add peers to their network; Verisign, PCH/WoodyNet, Akamai, AS112, K root, and F root.
  2. Hurricane Electric is in "provisioning" with them, so presumably at some point soon, HE will have access to K root from Tahoe, dropping its latency to the bay area quite a bit.
So this opportunity posed by K root being the last root server not yet built out in the bay area very well might disappear soon. This is a bit of a drag since that might make RIPE less likely to entertain us hosting yet another California node, and the K roots don't come for free. We would need to provide a Dell server meeting all of their specifications, which I just priced out at $1475 for a Dell R230.


So, at this point, it's possible that I will be able to get F root to join FCIX, since they're just a cross connect away in the same building (and I happen to already be friendly with them), plus D and E if they happen to be on the local AS42 node. I, J, and M are in the area, so getting them on the fabric is conceivable except that they aren't in our building, so that problem would need to somehow be solved. And K is currently several states away, so I'd need to convince them that yet another west coast node is worth their bother, and we'd need to pony up $1500 to get the gear they require.

There are other value-add networks we can work on getting on our IXP, such as AS112 to trap bogon DNS requests, CDNs like Akamai, CloudFlare, and (maybe?) NetFlix, and it seems like there's also 13 DNS servers for gTLDs, but I can't find much information on who hosts those or how they're rolled out. Presumably they're hosted by just Verisign so one of those would come with a J root.

Overview of Fiber Optic Transceivers


Thanks again to Arista Networks and FlexOptix for helping make FCIX possible.

Thursday, May 24, 2018

Hamvention 2018

Hamvention is one of the largest amateur radio conferences in the world, and this year I finally decided to give it a try. It always falls on the same weekend as Maker Faire Bay Area, which I had always found more attractive than the idea of the largest ham con, so even I'm a little surprised that my Ham Radio Workbench Podcast buddies managed to convince me to come out and give it a shot.

In order to limit the number of days out on PTO (It's spring, so I've been taking a lot of personal time off for Wildflower Tri related activities), as well as save on travel / hotel expenses, I thought it would be a good idea to take a red-eye out to Dayton Wednesday evening to land there Thursday morning to help with set up Thursday.

Boy, was I wrong...

1. The 10:30PM to 9AM red-eye leaves you in a very odd mental state, so maybe not the best flight plan for me.
2. I had been expecting a Maker Faire-like set up day, where everyone is setting up their booths and hanging out and having a good time, which really wasn't the case.

Most, but not even all of the vendors, were there setting up their booths, but as soon as each one finished set up, they generally just threw a tarp over their stuff and rolled out. Not quite the "the event without all the crowds" experience I was hoping to get being there a day early.
What was exciting was that this was the first time I got to actually meet Jeremy, KF7IJZ, in person. We've been friends for a long time, since I'm a "friend of the podcast" for the Ham Radio Workbench Podcast, but since he's out in Ohio we had not met yet. George happens to be local to me in the Silicon Valley, so we've met a few times.

The meet-up went just as well as I could have hoped. We were using DMR for the weekend, so when Jeremy heard me call him on the radio, he was at first a little confused since they hadn't gotten the Internet-connected hotspot gateway running in the booth yet, until a moment later when he finally realized that he was able to hear me because I was in actual simplex range. I, of course, forgot to take a selfie with the two of us capturing this historic moment in HRWB history, but we all survive.
Set-up went as should be expected. Lots of people running around, lots of missing power drops, lots of not quite working Internet service, etc etc.

Friday and Saturday at Hamvention were both very similar agenda-wise, but clearly Friday was the day to be there. More stuff left at the flea market, more stuff left at the booths, and many more people to run into and share the experience with.

I started the day with Smitty, KR6ZY, checking out the flea market and generally walking around seeing the other vendors until the rain picked up, at which point we took shelter in our expo hall and I spent pretty much the rest of the day working in the Ham Radio Workbench booth talking to people about our podcast and what we're all about, as well as selling our bare PCB "kits", which I really like as a kit model: We sell you the board, and you're free to source all the other components yourself or click on the Digikey shopping cart link in our documentation to have them sell you all the parts, because Digikey is obviously much better at counting out resistors and LEDs into tiny bags than we are.

One of the exciting things for HRWB this year is that we've partnered with Digilent (including putting together an Analog Discovery 2 + accessories bundle specifically for HRWB listeners) and managed to convince them to come have a booth at Hamvention! We had a "visit both booths to get entered in a drawing" deal going with them, and it was all around great to have another group of people passionate about electronics education near-by. HRWB had Kaitlyn from Digilent on earlier this year on episode 41 to talk about the Analog Discovery 2.

After a full day of almost solid wall-to-wall people talking to us at the booth, we all went out to dinner with Digilent, and then headed back to the hotel to record the 50th (!) episode of this crazy Ham Radio Workbench podcast that was originally going to maybe run for six episodes if they were able to find enough to talk about. We are no longer concerned we'll find enough to talk about.

We all promptly went to our hotel rooms and passed out for the next day.

Saturday was about the same, but with what felt like fewer people and less chaos. I had already seen much of the convention, so I generally stayed in the booth for most of the day and continued to spread the good word of making and building your own projects. It was great.
One part of Saturday that was awesome was that my very good friend Jeff Glass, KK9JEF, drove down from Chicago and I got to hang out with him during the day. He had actually been my original exposure to amateur radio waaay back in middle school when he already had his license, so it was great fun to hang out at Hamvention with him 15 years later with us both as active hams.

A quiet evening doing some local shopping and dinner, passing out in our hotels once again, and then it was Sunday.
Sunday was laid back; breakfast with the HRWB entourage, a few of us dropped by the fairgrounds to kill another few hours, and I was off to catch my flight.

The Flea Market Haul

I was a little limited by having flown to Dayton, but if anything that was a very positive restriction on what I could buy at the flea market; no buying a boat anchor or 19" rack cabinet for me. Continues to avoid eye contact with Smitty who drove from California and had space in his car.

One of the first things I found was a neat little bayonet lamp holder, which includes a turnable ring to make it dimmer! I spent the rest of the weekend looking for the right bulb to put it in, but never did manage to find a booth selling the right lamp I was willing to spend money at. (The surplus dealer in the constructed tent had them, but their prices were so comically high I wasn't willing to buy anything from them)
 The Anderson PowerPole is the standard 12V DC power connector used by hams, myself included, so I've built hundreds of them, and have been meaning to get a real contact insertion/extraction tool for PowerPoles for a long time. For $15, I figured this was that moment when I'd finally own the right tool for the job. Life is too short to not have the right tools for the job.
 Of course, my favorite radios from Motorola don't use PowerPole, but the "bullet" or "RV" power connector, so I found some really nice 10AWG cables which I can cut in half and terminate with PowerPoles to be adapter cables from all of my 12VDC gear to Motorola.
 Little 20W 12VDC flood lamps are great for little work lamps when you've already got lots of 12VDC around. I'll be putting PowerPoles on this and throwing it in my DC power box.
 I finally broke down and got a DMR radio last week, so to make it generally more useful, I decided to get the ZumSpot kit, which has been a very slick hotspot experience. Take it out, put it all together (including a pre-imaged SD card), connect to the wifi AP coming from the Raspberry Pi, configure everything specific to how I want to use it through the web interface, reboot, and done. Now I can hang out on DMR talkgroup 31075 from my apartment or anywhere that I bring the hotspot tethered to my phone.
The hotspot now hangs out in my living room, pulling power off the USB port on one of my WiFi APs. HRO had a few cases for the ZumSpot, but those sold out early so shame on me for not buying it right away but waiting until the end of the day.
 This was one of those things at the flea market where I got super interested when I saw it on a table, and the vendor was happy enough to see someone interested in it to just give it to me. I'm a total sucker for vintage engineering textbooks. :)
 My second book for the weekend wasn't actually bought at Hamvention, but Saturday evening when I went to check out the local bookstores as a nice quiet way to spend the evening.
Probably my best deal for the weekend was talking one of the vendors down to $40 for a bag of ten SFP+ SR optics, which I can use to run 10Gbps Ethernet between my servers in my Hurricane Electric rack.
My favorite find was actually a mechanical camera shutter release timer, not because it's at all useful, but because it became a fun puzzle to hand to people all weekend and have them try and figure out what the hell it even was. I do technically have a digital camera with a threaded shutter release, and I did come home and prove that it works, but using the internal timer is much easier... Still $2 well spent.

My Final Thoughts

Processing all of Hamvention hasn't been easy, and it's taken me a few days to really decide how I felt about it. Granted, my thoughts in one or two years will likely be different from here, but this is what I've got now:

The Good

  • Hamvention itself was very well run. Garbage, Restrooms, Food vendors, facilities; it was all very well taken care of from what I saw. This even went as far as they clearly came out Friday evening and laid down more gravel + hay on what had turned into muddy soup Friday during the rain.
  • Getting to see my friends from Ham Radio Workbench and Digilent was great and made it a fun weekend.
  • The mobility scooter traffic wasn't nearly the disaster other's have always joked that it was at Hamvention. Granted, I did hear that there seemed to be notably fewer scooters than usual, so maybe the mud disaster of last year scared many of those people off. The coordinators clearly made an effort to keep the conference accessible, so even by the end of the weekend you could still drive out through the flea market on their nicely graveled pathways.
  • Getting to talk to many of our excited listeners and many new people interested in what we were talking about was very touching. I always appreciate the reminder of what a large impact a few of us with microphones can make.

The Bad

  • Would you believe I missed the APRS forum? I wrote my bloody masters thesis on APRS, and then totally lost track of time and missed it. So embarrassing. 
  • It sure must have been a good thing that there were fewer mobility scooters, because even still I nearly got side swiped by a few of them (why do they think they deserve to roll faster than the rest of us can walk through a crowded building?) and got rammed in the ankle by one that the operator totally lost control of.
  • I was still disappointed that the fairgrounds didn't feel particularly social Thursday. Everyone set up and then cleared out.
  • Which might have been partly because there was apparently a separate Four Days of May QRP conference in a near-by hotel Thursday night. This being my first time at Hamvention, and between putting little effort into investigation myself and not getting a clear understanding of what everyone else was doing, I later found out I missed out on a lot of these sorts of events during the weekend simply since I didn't know about them. I just wasn't well plugged into the unadvertised parts of the conference.
  • More than anything else, Hamvention felt like a place to come buy things and talk to commercial vendors. Ham Radio Outlet and Maine Trading Company and 100 other retailers were there to take your money, and the big radio vendors like Yaesu were there to talk about their new and up-coming rigs, but I wasn't particularly interested in that. It felt like even the few individuals with booths at Hamvention were still more interested in selling you one of their projects than talking about it. It might not be fair to compare the Hamvention experience to Maker Faire, but that's the sort of experience I was looking for with an amateur radio focus, and that's definitely not what I came away with from Hamvention. 

The Ugly

  • I hadn't really braced myself for dealing with amateurs still salty about the FCC dropping the morse code requirement, so that came as a bit of a shock this weekend when several times I caught flak for being one of those lowly "no code" amateurs who isn't a "real ham". One gentleman even came up to us at the booth and said he was glad that we were fulfilling the moral imperative of amateur radio by building things, so that at least a few of us and our listeners were finally true hams. Like everyone who does anything else with the hobby except building their own SSB rigs for HF aren't also valid amateurs. I know that these guys and their opinions shouldn't have been a damper on my weekend, but they can go fuck themselves with their elitist, exclusionary, opinions on what the hobby should be. 
  • Between our booth and Digilent's booth, we had a few ladies working our booths and talking to people, and disappointingly, I over-heard several inappropriate comments made about them either to their faces or out of their ear shot. Amateur radio has a serious diversity problem, and making inappropriate comments about the few ladies who do come out to an event like this isn't helping. I've even run out of patience for terms like YL or XYL, because I don't think they're cute or endearing to distinguish the females in the hobby like that, but I would have gladly dealt with a few awkwardly forced references to YLs over what was said this weekend. We as a community need to stop tolerating harassment and call each other out on it. 


So, would I come back to Hamvention? Right now, given that it's always the same weekend as Maker Faire Bay Area? 

Pfth. Not a chance.

It was absolutely fantastic to get to meet the rest of the HRWB podcast crew, and all our great listeners and fellow makers, but the small subset of people at Hamvention who were on the same wavelength as me with regards to why they were there just doesn't compete with the vast majority of the people at Maker Faire who are there for the exact same reasons as me. I consider myself a maker first and an amateur second, so I think it's reasonable that I should spend the third weekend of May at a conference that is all makers first and amateurs get a part of one aisle instead of a conference that is all hams and makers get a small fraction of the booths.

Now, if Hamvention weren't the same weekend as the largest annual gathering of my maker friends (which also happens to be much cheaper to attend since it's local to me) I could seriously see myself attending Hamvention on a regular if not annual basis. It's a perfectly fine event, and it should be on me to pony up and help form it into the event I wanted it to be, but it just can't compete with Maker Faire. I feel bad saying this since I do have several amateur radio friends who I got to see this weekend, but my priorities for this one weekend point to Maker Faire (which they should all come and experience themselves).

My original plan had been to alternate between Maker Faire and Hamvention, and I can still see that being a great plan for a tremendous number of hams more serious about amateur radio as a whole than me, but my interests in amateur radio are all just too niche and my interests in Maker Faire and the crafting and artwork and electronics that comes from that are all just too broad. I think every amateur should come experience Hamvention at least once to see if it's their thing, so I don't regret the experience, but every maker should also come experience Maker Faire.

Sunday, April 15, 2018

Creating an Internet Exchange for Even More Fun and Less Profit

Last quarter, I was pulled into the slightly odd underground of people running their own autonomous systems, and since then, our circle of friends running autonomous systems at Hurricane Electric's FMT2 has slowly been growing.
Which is great, except that we're all running autonomous systems, which means that we can set up peering links, and are you really friends with another network engineer if you're not running a cross connect between your two networks? This wasn't too bad for the first few networks joining our little cabal of networks, but due to that pesky quadratic growth issue, the number of new cross connects needed when the fifth or sixth person joined started getting ridiculous. (It's like, four or five!)
This is, of course, an issue that real networks have to deal with as well, so when we had an eighth friend sign a service agreement with Hurricane Electric this week, the idea was (half jokingly) floated that we should just start our own Internet Exchange Point to cut down on the number of cross connects we need for each new member.

An Internet Exchange is basically just a single L2 Ethernet switch which every network plugs into, such that every network can directly set up BGP peering / route packets to each other network on the fabric. Furthermore, to make it even easier to add new networks to an Internet Exchange, many IXs run "route servers," which are BGP peers which re-distribute all the connected routes. This is convenient because it means that only the IX operator and the new network need to adjust their BGP configuration when a network joins; everyone else is already peered with the route server and start getting the new routes (and which router on the switch to send that traffic to) as part of their already existing connection to the route server.

So we were all sitting there, contemplating the idea of ordering seven more cross connects and once again all logging into our routers to update our configs, and at that point, the idea of creating an Internet Exchange instead didn't seem too bad.

We could instead have all gotten cross connects into one of the existing Internet Exchanges in the HE FMT2 building, such as SFMIX, but they charge $995/year for a port on their fabric, which is more money than it's worth for all of us to cross connect for amusement's sake (most of us are amateurs and not making money on our networks). So screw it, hold my other beer, and away we go!

And that's how the Fremont Cabal Internet Exchange was born. 

We even made a website and everything.

We allocated a /64 IPv6 subnet from my /48 (which was originally allocated from another guy's /32), drummed up an IPv4 /24 that was currently between projects, and very carefully selected the private ASN 4244741280, and all that was left to get was a switch to all connect to.
Thankfully, my entire network in my cabinet is built on a Cisco 6506, which is technically a switch, so we called that close enough, and instead of having to find another piece of hardware, just allocated a VLAN on my 6506 as the switch fabric, and we were all set. Besides, we were getting a little worried that there were getting to be too few Internet Exchanges running on Cisco 6500s these days.

Now whenever someone wants to connect to the FCIX (Fremont Cabal Internet Exchange) fabric, they just get a cross connect to my cabinet, I set another port to be an access port to the FCIX VLAN, and they're hooked up to everyone.

It's only 1Gbps to each network, but most of us are only originating a few prefixes for a few servers, so we aren't really pushing the limits of single 1G links per participant yet, but just like in any real IX, as soon as someone starts saturating their link to FCIX, they can start setting up direct peering links to other networks to start shedding that traffic off their exchange links. You know... when that happens...

Ideally we would have applied for a public ASN for the exchange, but that $550 + $100/yr for a registered ASN kind of went against the objective of saving money on cross connects, and I figured the chances of someone connecting to FCIX already using one random 4 byte private ASN inside their network was pretty low. Since the IX ASN is never appended to any routes going through the exchange, there's also the fact that no one outside the exchange will ever see this ASN, so it seems like a pretty acceptable trade-off for a group of amateurs for now. (The biggest downside I can think of is that we might not be able to register this IX on peeringDB with a private ASN, to further prop up the facade that this is an Internet Exchange to be taken seriously)

Edit: OK, I stand corrected. peeringDB had no problem and we're now live on there as well. That was not expected.
The last piece to really make adding new members to this peering fabric convenient is setting up two route servers, so that each new member doesn't trigger everyone needing to log into their routers to add a new BGP peer. Instead, everyone peers with the route servers and they handle the full N-to-N exchange of routes. When a new member joins, they set up their router on the fabric's /24+/64, and peer with the two route servers, and the only other involvement needed is from one of the IX admins (which is really just me, currently) to add them to the route server. Every other member doesn't need to be involved and can just enjoy the new routes appearing on their router.
We have two BGP route servers so as I need to restart each one for maintenance reasons, everyone can still trade routes over the other one and I don't trigger a reconvergence every time I restart the daemon or VM. We even managed to get the second VM on a different hypervisor in Javier's cabinet instead of mine, for further fault tolerance.

We're still working to figure out exactly which route server software we want to use. I'm the most familiar with Quagga, but Quagga tries to emulate the Cisco model of all config changes are made on the fly through the console, where I don't want to be hand crafting config changes every time we add a member, so I'm currently taking a crash course in running BIRD as one of our route servers, and will likely be swapping various daemons in for each route server as we learn more.

Sunday, April 1, 2018

Measuring the Internet

Progress on the whole "running my own bit of the Internet" project has been going well. We've got a router, and some servers, and even a NAS, so one of the next questions is how well is our network behaving over time?

There are plenty of different ways to ask this question, and plenty of different metrics to look at. For example, to track my bandwidth usage, I'm using LibreNMS, which is a pretty good SNMP front-end to query my router every five minutes to see how many packets I'm moving.

One network monitoring tool that I've discovered as part of this project is the RIPE Atlas. It is a world-wide network of measurement probes spread across the Internet, which they use to measure the health of the Internet as a whole, but also allow others to request measurements on it.

To get started, you can request a probe and if approved, they mail you the simple hardware (clearly based on a TP-Link router running their custom firmware) to plug into your network. Once it's powered on, the probe starts taking measurements from your Internet connection, and you start earning credits to spend on your own custom measurements.
For example, I requested the probe, and although I never got any kind of email, a DHL package showed up about 6-8 weeks later with the probe + some cables inside. Once I plugged it in and registered its serial number to my account, I'm now accruing 21600 credits per day for keeping the probe online, plus another 5000-50000 credits per day for "results delivered" which I presume is running other people's custom measurements.
I haven't come up with any long term custom measurements yet, but to give you a sense of scale, a single traceroute costs 60 credits, so running a traceroute to my network from 10 random probes costs 600 credits, and RIPE's traceroute report is pretty slick.
The main reason I haven't programmed any periodic custom measurements yet is because the probe comes with a set of "built-in" measurements where it automatically measures the latency and packet loss to its first and second hop, all the root DNS servers, and some Atlas infrastructure, which already answers most of my questions on how well my network is doing. I really should set up some user-defined queries to monitor my HTTP servers, but for now I'm just accruing credits.

You can see all the public information coming from my probe here. You can even order Atlas measurements specifically from my network to your network if you specify the measurement should be sent to probe #34742, which I find rather amusing.

One thing I noticed right away is that I'm seeing 100% packet loss (solid red) to a few IPv6 root DNS servers... This is actually because Hurricane Electric and Cogent have been having a peering dispute over IPv6 for... pretty much forever, so the IPv6 Internet is actually split brain and I'm just not able to reach some parts of the IPv6 Internet from my Hurricane Electric transit...

One of the perks of running my own autonomous system is that I'm able to work on getting myself blended transit from someone other than Hurricane Electric and fix this problem myself... (Anyone with blended IPv6 transit on the west coast want to help me out?)

The probe uses about 3kbps to take its measurements, so the network load from it is what I would describe as "pretty much undetectable" considering my main transit link hovers around four orders of magnitude higher than that. This plot is from LibreNMS for my "DHCP" /29 subnet, which I use for my Atlas probe and plugging in my laptop to a spare port on my router when I'm standing in the datacenter working on my rack.

Monday, January 15, 2018

Off-the-Grid Raspbian Repositories

For one of my future projects, I'm looking at spinning up 15-25 Raspberry Pi desktops in the middle of nowhere, with somewhere between zero and awful Internet connectivity. We're making a best effort wag at what the OS image should be before-hand, but I expect we will inevitably get there and realize we want to install some other piece of software from the Raspbian software repo.

I figured the best solution to this was to pre-download ALL of the software available for Raspbian (it's only ~400GB) and sneaker-net it in. I also don't want to be making any changes to the config files on the Raspbian installs, since that would mean we'd need to remember to set things back to normal when we're done, so I want this local copy of the Raspbian repos to include some DNS trickery to make the Raspberry Pis think they're still actually talking to the actual repos on the Internet.

This means I need:

  1. A router to give the Pis IP addresses and respond to their DNS queries for and with the address of the local server instead of those server's actual public IP addresses.
  2. A laptop with at least 500GB of disk space running apache2 serving a local copy of and

My router of choice for these sorts of scratch projects where I need something to do DHCP/DNS/NAT but don't particularly care about performance or getting my router back afterwards is the WRT54GL. I've got a stack of them running mainline OpenWRT, not because OpenWRT has a nice interface, but because it is exceedingly flexible.

Beyond the basic configuration of picking a new subnet that isn't and setting a hostname for it, I made two changes:

  1. A static DHCP address allocation for my laptop running apache2.
  2. I sshed into the router and created a new hosts file that gives the laptop's IP address for the two repos I'm trying to fake.
Hosts files are the simplest way to override DNS records for single hostnames. I created /etc/hosts.rpimirror and it contains the two domains associated with my laptop's static IP address:
Once I created this extra hosts file, I added it to the web configuration via "Network > DHCP and DNS > Resolv and Hosts Files > Additional Hosts files", but of course your router's method to add additional hosts files will vary if you're not running OpenWRT. 
At this point, any Raspberry Pi plugged into this router will get faked answers back for the two repo domains, and will try and send all of those requests to the laptop, without having to edit any of the sources files on the Pis, so as soon as they get plugged into any other router, they'll behave just as normal.

Now all that's left to do is to create a local mirror of those two repos and make them available over http. 

To copy the mirror files to the laptop, I wrote a simple script which uses rsync to copy the Raspbian repo and debmirror to copy the needed parts of the repo (mainly because it doesn't support rsync for some reason). We're talking about something on the order of 400GB of files here, so strap in; this download is going to take a while regardless of how fast your Internet is. Once it's done, you should be able to rerun the same script and have it go relatively quickly since it will only be updating your existing local copy.

You will need to change the MIRRORBASE variable to somewhere on your computer that you want to store all of these files, since I have it hard-coded in my home directory.

I'm also only mirroring the stretch packages from, so you will need to edit the "release" variable if you're using a different release of Raspbian.

Script was based on the Ubuntu debmirror page and the rsync command at the bottom of the Raspbian mirror's list.

Once that download is done, you'll need to link the and folders into apache's /var/www/html root so both folders are visible:
sudo ln -s /home/kenneth/mirror/ /var/www/html/debian
sudo ln -s /home/kenneth/mirror/ /var/www/html/raspbian

At this point, you should be able to see the two repo's dists and pool folders by pointing your browser at "http://laptop's_address/raspbian" and "http://laptop's_address/debian", and any Raspberry Pis plugged into the router will be able to download any new software or updates entirely locally without a connection to the Internet.

While this does let you install any updates or software existing at the time of your repo snapshots, this doesn't get you any updates or software released after the snapshot. One option for that, assuming this local repo is somewhere entirely without Internet connectivity, is to carry it back to somewhere with Internet, rerun my repo mirror script above to download any changes to the repos, then carry the server back to where it's being used.