Wednesday, November 15, 2017

Creating an Autonomous System for Fun and Profit

At its core, the Internet is an interconnected fabric of separate networks. Each network which makes up the Internet is operated independently and only interconnects with other networks in clearly defined places.

For smaller networks like your home, the interaction between your network and the rest of the Internet is usually pretty simple: you buy an Internet service plan from an ISP (Internet Service Provider), they give you some kind of hand-off through something like a DSL or cable modem, and give you access to "the entire Internet". Your router (which is likely also a WiFi access point and Ethernet switch) then only needs to know about two things; your local computers and devices are on one side, and the ENTIRE Internet is on the other side of that network link given to you by your ISP.

For most people, that's the extent of what's needed to be understood about how the Internet works. Pick the best ISP, buy a connection from them, and attach computers needing access to the Internet. And that's fine, as long as you're happy with only having one Internet connection from one vendor, who will lend you some arbitrary IP address(es) for the extend of your service agreement, but that starts not being good enough when you don't want to be beholden to a single ISP or a single connection for your connectivity to the Internet.

That also isn't good enough if you *are* an Internet Service Provider so you are literally a part of the Internet. You can't assume that the entire Internet is that way when half of the Internet is actually in the other direction.

This is when you really have to start thinking about the Internet and treating the Internet as a very large mesh of independent connected organizations instead of an abstract cloud icon on the edge of your local network map.

Which is pretty much never for most of us.

Almost no one needs to consider the Internet at this level. The long flight of steps from DSL for your apartment up to needing to be an integral part of the Internet means that pretty much regardless of what level of Internet service you need for your projects, you can probably pay someone else to provide it and don't need to sit down and learn how BGP works and what an Autonomous System is.

But let's ignore that for one second, and talk about how to become your own ISP.

To become your own Internet Service Provider with customers who pay you to access the Internet, or be your own web hosting provider with customers who pay you to be accessible from the Internet, or your own transit provider who has customers who pay you to move their customer's packets to other people's customers, you need a few things:
  1. Your own public IP address space allocated to you by an Internet numbering organization
  2. Your own Autonomous System Number (ASN) to identify your network as separate from everyone else's networks
  3. At least one router connected to a different autonomous system speaking the Border Gateway Protocol to tell the rest of the Internet that your address space is accessible from your autonomous system.
Once your router tells other networks that you're now the home of some specific range of IP addresses, and that advertisement propagates out through the rest of the Internet, everyone else's routers will have an entry in their routing tables so if they see any packets with your address on them, they know which direction to send them so they eventually end up at your door step.

Wait, but why don't you need any of this for your home Internet?

So why doesn't your home router need to speak BGP or you need to own public IP space to be reachable from the rest of the Internet? Because your ISP takes care of that for you. In addition to funding the wiring from their data center to your house, the $50/month you pay to your ISP funds them getting address space allocated for you, advertising it to the rest of the Internet, and getting enough connectivity to the rest of the Internet that your packets can get where they're headed.

The same answer is true on the other end when you spin up a web server somewhere like Digital Ocean or Amazon Web Services; they handle IP addressing and BGP for you so all you need to worry about is setting up the software for your own corner of the Internet on the one specific address they set aside for you from their big pools of addresses that they manage.

Wait, but why am I blogging about BGP then?

If you've made it this far, you're probably pretty curious why I'm talking about BGP at all, and what this blog post is leading up to. 

So... I recently set up my own autonomous system... and I don't really have a fantastic justification for it...

My motivation was twofold:
  1. One of my friends and I sat down and figured it out that splitting the cost of a rack in Hurricane Electric's FMT2 data center marginally lowered our monthly hosting expenses vs all the paid services we're using scattered across the Internet which can all be condensed into this one rack.
And this first reason on its own is a perfectly valid justification for paying for co-location space at a data center like Hurricane Electric's, but isn't actually a valid reason for running it as an autonomous system, because Hurricane Electric will gladly let you use their address space for your servers hosted in their building. That's usually part of the deal when you pay for space in a data center: power, cooling, Internet connectivity, and your own IP addresses.
  1. Another one of my friends challenged me to do it as an Autonomous System.
So admittedly, my justification for going through the additional trouble to set up this single rack of servers as an AS is a little more tenuous. I will readily admit that, more than anything else, this was a "hold my beer" sort of engineering moment, and not something that is at all needed to achieve what we actually needed (a rack to park all our servers in).

But what the hell; I've figured out how to do it, so I figured it would make an entertaining blog post. So here's how I set up a multi-homed autonomous system on a shoe-string budget:

Step 1. Found a Company

You're going to need a legal entity of some sort for a few steps here, so you're going to need a business name. I already happened to have one from other projects, so at the minimum you'll want to go to your local city hall and get a business license. My business license cost me the effort to come up with a kick-ass company name and about $33/year, and I've never even gotten around to doing anything fancy like incorporating it, so it's really just a piece of paper that hangs in my hallway and allows me to file 1099-MISC forms on my tax returns within the city of Sunnyvale, CA. In the context of this project, this business license primarily just needs to look official enough to get me approvals when I apply for an autonomous system number needed to set up my own network.

Step 2. Get Yourself Public Address Space

This step is, unfortunately, probably also the most difficult. You need to get yourself a block of public IP addresses big enough to be advertised over BGP (there's generally agreed upon minimums to keep the global routing table from getting ridiculous) and allocated for you to advertise over BGP yourself. You'll probably want both IPv4 addresses, which have to be at least a /24 subnet (256 addresses) and IPv6 addresses, which have to be at least a /48 subnet (65536 subnets of /64).

The big problem is that there are no IPv4 addresses left. There was only 4 billion of them in the first place, and we've simply run out of them, so the "normal" procedure of going to your local Internet numbers organization like ARIN isn't that productive. If all you need is IPv6 space (which is unlikely) and you happen to be in North America, you actually can still go to ARIN and request resources, but IPv4 addresses are generally still really needed. There's other solutions like buying IPv4 space on the second hand market, but that's getting expensive, so here's probably the least helpful part of this whole blog post:

I just borrowed some address space from my friends.

For example, I've got another friend who, for a different project, got a /32 IPv6 allocation from ARIN, which is a metric TON of addresses, so I asked him if I could have a (relatively small) /48 sub-allocated from his /32, so he drafted me an all official looking "Letter of Authorization" on his company letterhead that literally just says:
"Dear Sirs,
"Please accept this letter of authority on behalf of [FRIEND'S COMPANY NAME] to permit the BGP announcement of [/48 IPv6 SUBNET INSIDE HIS /32 SUBNET] by [KENNETH'S COMPANY NAME].
It's not as impressive as having IP space with my name on it in ARIN's database, but it's also a whole hell of a lot cheaper than even the smallest address allocation you can get from ARIN (a couple beers vs $250/year).

This letter of authorization is also the first instance of where learning about how the Internet actually works gets a little weird. That letter is literally all it took for me to take control of a sub-block of someone else's public address space and get it routed to my network instead of theirs. Some of my network peers later asked for me to provide this LoA when we were setting up my network links, but that means I just sent them a PDF scan of a letter with my friend's signature on it. And I mean an actual signature; not some kind of fancy cryptographic signature, but literally a blue scribble on a piece of paper.

To be fair, the letterhead looked very official.

Step 3. Find Yourself Multiple Other Autonomous Systems to Peer With

So the name of the game with the Internet is that you need to be interconnected with at least one other part of it to be able to reach any of it, but that isn't necessarily good enough here. If you were only peering with one other autonomous system, you probably wouldn't even need to run BGP, and if you did, you could even do it using a "private" autonomous system number (ASN) which your upstream provider could just replace with their own before passing your routes on to the rest of the Internet.

But that's not good enough here. I didn't want to use some kind of lousy non-public ASN! This project was a personal challenge from a friend and the network engineering equivalent of driving a pickup with a lift kit, so we need a public ASN. We're going to later need to apply to ARIN to get one allocated and we'll need to provide at least two other autonomous systems we're going to be peering with to justify the "multi-homed" routing policy we're using to justify ARIN allocating us an ASN.

This multi-homed policy where we're peering with multiple other networks is also kind of neat because it means that if one of our upstream providers decides to take the day off, or only provide us a certain amount of bandwidth to the rest of the Internet, we have alternatives we can use from our peering links into other autonomous systems.

This whole concept of peering and all the different types of peering policies you might want for your network is a pretty deep rabbit hole, so I actually ended up buying a whole book just on peering, which was very helpful: The 2014 Internet Peering Playbook, Norton. He also has a website, which is a significant fraction of the content of his book in a less curated form.

Peering is definitely one of these "how the sausage gets made" sorts of topics that a lot of networks tend not to like to talk about. Exactly how well connected one network is to other networks is hugely important to their customer's quality of service, so everyone wants to make it appear that they're extremely well connected without showing their hand and letting others see their weaknesses. This means the peering community is rife with quiet backroom handshake deals that are never publicly disclosed, and you can spend hours digging through online looking glass servers that show you the global BGP tables trying to figure out what in the world networks are doing with their peering links.

Long story short, I'm getting a "paid transit" peering link from Hurricane Electric due to renting one of their cabinets, and managed to find a few friends in the Hurricane Electric FMT2 data center who had spare Ethernet ports on their router and were willing to set up free peering links for what little traffic happens to go directly between our own networks. Free peering links tend to be pretty common when both networks are at about the same level in the network provider / customer hierarchy, so tier 1 transit providers tend to peer for free to make the Internet happen, and lower tier small networks tend to peer for free to by-pass both needing to pay higher level ISPs to transit their traffic when they can move it directly, but if either network thinks they can charge the other for money, that might happen as well.

This is obviously where human networking becomes exceedingly important in computer networking, so start making friends with the peering coordinators for other networks which you expect to be trading a lot of traffic with. Every packet I'm able to shed off onto one of these lateral peering links with another AS is traffic that doesn't tie up my primary 1Gb hand-off from HE and makes my network faster.

Step 4. Apply for an Autonomous System Number

There are five Internet number organizations world-wide, and since I'm in North America the one I care about is ARIN, so I created an account on ARIN's website and:
  1. Created a Point of Contact Record for myself - Pretty much just a public profile for my identity: "Kenneth Finnegan, address, phone number, etc etc"
  2. Requested an Organization Identifier for "[MY COMPANY NAME]" and tied my point of contact record to it - This was by opening a ticket and attaching my business license to prove that my company actually exists. Since my company isn't its own legal identity, ARIN internally tracks it as "Kenneth Finnegan (doing business as) [MY COMPANY NAME]", but this doesn't show up on the public listing, so it wasn't a big deal.
  3. Requested an ASN for my Organization Identifier - This is where I needed to be able to list at least two other ASes I was planning on peering with. 
  4. Pay the $550 fee for getting an ASN issued per ARIN's current fee schedule for resources.
The whole process took about a week between setting up the orgID and requesting the ASN, mainly because I didn't quite get either support ticket request right on the first try due to me not quite knowing what I was doing, but in the end ARIN ultimately took my money and issued me an ASN all my own.

Step 5. Source a Router Capable of Handling the Entire Internet Routing Table

Remember how your home router only needs two routes? One for your local computers and one for the rest of the Internet, so the two routes are probably something like " (local) (WAN)"

Processing a full BGP Internet routing table is a little more involved than that, and means you need a much more powerful router than one you can go buy at Office Depot. You could also probably build a router yourself out of a server running a router operating system like pfsense or just your favorite Linux distro with the right iptables voodoo and a BGP daemon like quagga, but that wasn't part of the originally thrown gauntlet challenge for this project.

The challenge was to use a real Cisco router capable of handling the entire Internet routing table, and I wanted one that can switch it at line speed. Hurricane Electric alone is giving us a 1Gb Ethernet hand-off, not including all the additional bandwidth out of our rack available due to other peering links, so we wanted a router that could definitely handle at least 1Gbps.
Meet the Cisco Catalyst 6506. Yes, it's rack mount, on a 19" rack. And is 12U high, which since a U is 1.75", means that this router is almost two feet high. And 150 pounds. And burns about 1kW.

Yes. It's size is ridiculous. Which for this project, isn't entirely out of line.

But it's also kind of a really awesome router, particularly for being a nearly 2 decade old product. The 6500 series is a line of switch/router chassis which support 3,4,6,9, or 13 modular line cards/router supervisors. In the early 2000s this was the best switch that money could buy, and it is definitely showing its age now, but that's perfect. Network engineers love to hate their 6500s because they're so old, but its relatively limited "only" 30 million packets per second throughput is plenty for an autonomous system that fits in a single rack, and its age means I was able to pick up a 6506 configured with dual supervisors and three (3!) x 48 port Gigabit Ethernet line cards on eBay for $625 shipped!

I probably could have found a more reasonably sized router for what I needed, but the 6506 has the appeal that it definitely has more switching horsepower than I'll ever need for this project, and its biggest downsides are it's size and power, which are both not that big of issues since I've got a whole 44U rack for just a few servers and I don't get billed for my power usage. More desirable routers have the big downside that they're actually desirable, so other people are willing to spend a few thousand dollars on them, where I didn't really want to drop $2k on a well kitted out 3945.

The 6506 probably deserves blog posts of its own, but the main thing is that low end configurations of it like this are cheap on eBay, with the one disadvantage that they don't come with a supervisor card with enough memory to handle a full Internet table. This means I did need to scrounge a sup720-XL supervisor that can handle 1 million routes in its routing table. Another few hundred bucks on eBay, or a friend with access to the right kind of e-waste pile solves this problem.
Granted, a million routes isn't actually that much. The public IPv4 table is about 675,000 routes, and IPv6 is another 45,000, and they're both growing fast, so in another 2-3 years the Internet is going to exceed the capabilities of this router. When that happens, I'm going to need to either replace it with something more advanced than this ancient beast or start using some tricks to trim down the full table once we get there. If you'd like to follow along at home and watch the IPv4 routing table march towards the demise of the cat6500, you can find a link to a bunch of graphs here.
I also added a 4 port 10Gb line card, because 10Gb is awesome, and took one of the 48x1Gb line cards out because I really didn't need 144 Ethernet ports on this thing. That's just a ridiculous number of Ethernet ports for a single rack.

So the final configuration is:

  1. 48x1Gb line card for my four copper peering links with other autonomous systems, including Hurricane Electric
  2. 4x10Gb line card for my peering link with one of my friends who happened to have a spare 10Gb port on his router, and who also thinks 10Gb is awesome. This will probably also serve some local 10Gb links in the rack once I grow beyond one server.
  3. A blankoff plate instead of my third 48x1Gb line card to save power.
  4. 48x1Gb line card for the local servers in the cabinet. Since we've only got two servers installed so far, there's currently only a 2x1Gb bonded link to my server and a 4x1Gb bonded link + out of band management to my friend's server.
  5. The sup720-BXL which does the actual router processing and makes this whole mess a BGP router. The one cable from this card runs up to a console server letting me manage this beast remotely from the comfort of my apartment without standing in a cold data center.
  6. One of my spare sup720 (not XL) which can't handle the full Internet table pulled out an inch so it doesn't power up because this seemed like the best place to store it until I figure out what to do with it. 

Step 6. Turn it All On and Pray

Wait, I mean, carefully engineer your brand new network and swagger into the data center confident that your equipment is all correctly configured.

But seriously, I found a few textbooks on BGP network design and happened to have a 13 hour flight to China and back to take a crash course in BGP theory, and spent a week in my apartment with ear plugs in taking a crash course in how to interact with Cisco devices more sophisticated than just setting up VLANs on an Ethernet switch, which is about all my experience with Cisco IOS before this month.

After spending a week lovingly hand crafting my router configuration (while listening to networking podcasters bagging on how ridiculous it is that we still lovingly hand craft our routing configurations), I was ready to deploy my router plus all of our servers in the data center.

When I signed my service agreement with Hurricane Electric, it consisted of:

  • One full 44U rack with threaded posts.
  • Two single phase 208V 20A feeds.
  • A single 1Gbps copper network drop
The network operations center then also emailed me and asked how many of Hurricane's IP addresses I needed, which was two: one for my router's uplink interface, and a second for a serial port console server so if I ever manage to really bork my router's configuration I can still reach it's console port without having to trek over to the data center and stand there in the cold. This means that my hand-off from HE is a /29, so I actually have 5 usable addresses, but that Ethernet drop goes into a fixed eight port GigE switch which breaks out the console server, then plugs into the 6506 for the majority of my Internet traffic.

Once I confirmed that my network feed from HE was live, I then opened a support ticket with HE saying "My BGP router is on [IPv4 ADDRESS] and [IPv6 ADDRESS] and will be advertising these specific routes per attached letters of authorization" and waited for them to set it up on their side, which took less than an hour before I got an email from them saying "we turned it on, and your router connected, so it looks good from here"
And we're off to the races.

At this point, Hurricane Electric is feeding us all ~700k routes for the Internet, we're feeding them our two routes for our local IPv4 and IPv6 subnets, and all that's left to do is order all our cross-connects to other ASes in the building willing to peer with us (mostly for fun) and load in all our servers to build our own personal corner of the Internet.

In the end, setting up my own autonomous system wasn't exactly simple, it was definitely not justified, but some times in life you just need to take the more difficult path. And there's a certain amount of pride in being able to claim that I'm part of the actual Internet. That's pretty neat. 

And of course, thanks to all of my friends who variously contributed parts, pieces, resources, and know-how to this on-going project. I had to pull in a lot of favors to pull this off, and I appreciate it.

Tuesday, August 1, 2017

Building a Stand-Alone Raspberry Pi Dispatch Server

As part of my amateur radio hobby, I volunteer and provide communication support for a few events every year, the largest of which is the Wildflower Triathlon. Compared to most amateur radio communication support roles, the scale of Wildflower is pretty unprecedented with us having multiple active dispatchers on multiple repeaters handling calls at the same time. (Presentation on Wildflower)

One of the key pieces of technology which enables us to keep multiple dispatchers in sync with regards to what is going on is a system called Computer Aided Dispatch (CAD). Computer Aided Dispatch is a system which aids radio dispatchers by replacing what is typically local paper log sheets or even just a legal pad on the low end with a web application. This is powerful since it means that every dispatcher can see every log entry  and incident ticket as it is entered in real-time, and logged to the web server, with automatic time-stamping, etc.

There are plenty of CAD software packages out there, and the open source option that comes up most often is TicketsCAD, but my impression is that it is a relatively complex system to configure and use, so for events like Wildflower where the CAD system is only deployed for a few hours to days, that complexity is burdensome. Most of our users have never interacted with a CAD system before, so simplicity on the user side is also an important motivator, so we use a very simple and clean CAD system called BlackFlower.

I don't know the entire history of the BlackFlower CAD package, but in short, there was an overlap of staff between Wildflower and Burning Man in the early 2000s, and they decided to develop a CAD system specifically for these two events. Since then, those of us running Wildflower have ended up not being involved with the development of BlackFlower, but we still use the package and just don't get hung up on it including Burning Man logos.

For smaller events, where we're not spinning up a multi-seat dispatch center, the overhead of lugging a Linux server into the field is usually prohibitive, but I thought it would be interesting to see if I could build a minimum viable product using just a Raspberry Pi with some extra hardware to make it more suitable for getting pulled out of a box in the field and turned on for either training events or really really small events.

I'm still hesitant to actually recommend that you use this as a CAD system running off a microSD card in a Raspberry Pi for real events where you're handling real medical calls. I see a CAD system running off a Raspberry Pi most interesting as a way to lower the barrier to entry for live demonstrations and drills for evaluation of the concept. Once an organization decides that they would want to rely on using CAD, I'd still recommend moving to a platform with  more precedence for reliability than the Raspberry Pi (and things like a real hard drive and on-board RTC).

Bill of materials

Installing Raspbian

Install Raspbian Lite (or full Raspbian if you want a GUI) per any of the guides available. My personal favorite is to just copy NOOBS onto an SD card and download Raspbian Lite from that on the Pi.  Once installed and booted, you need to perform some of the configuration from the Pi itself with a monitor and keyboard since SSH is disabled by default. Once I get SSH set up, I typically switch to using the Pi headless and SSHing into it from one of my other systems, but all of this guide is doable from the Pi itself if you prefer. While having raspi-config open, I just do all of the needed changes on it on one go (disclaimer: raspi-config menus tends to change periodically, so you might have to wing it vs what I have here):
  • Log in with username pi and the default password raspberry.
  • Run sudo raspi-config
  • 1 - Change User Password: Set it to something you're remember. This is your "pi" account for logging into Linux on the system
  • 2 - Hostname: I set mine to gobox.
  • 3 - Boot Options → B2 Wait for Network at Boot → No
  • 4 - Localization Options: Set your locale, timezone, and Wi-Fi country code. I use en_US.UTF-8, America → Los Angeles, and US respectively.
  • 5 - Interfacing Options: Enable SSH so we can log into Linux remotely and I2C for the Real Time Clock hat.
  • 7 - Advanced Options → A3 Memory Split → 16 because we're about to unplug the monitor for the last time so why reserve more memory for the GPU? If you're using full Raspbian for a more user friendly Linux experience, you might not want to do this.

Set up and configure your RTC (Real Time Clock) hat following AdaFruit's tutorial. This is so when the Raspberry Pi boots somewhere without an Internet connection, it can still know what day and time it is, instead of defaulting to Jan 1, 1970, which is what it does if you don't set up an RTC hat. Since I'm using a DS3231 hat the line I ended up adding to my /boot/config.txt is "dtoverlay=i2c-rtc,ds3231"

Install software dependencies

# Quality of life utilities that I install on every Linux system
sudo apt install vim dstat git screen
# Basic LAMP Dependencies
sudo apt install apache2 php5 mysql-server mysql-client php5-mysql
# Set a password for the root MySQL user and remember it. This is the password to log into the MySQL server, and shouldn't be confused with your Linux user password set earlier (except I just use the same one because why not; I'm the same admin for both)
# BlackFlower dependencies
sudo apt install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl python apt-show-versions
# I also like to install a MySQL database management tool like phpmyadmin so I can reach into the BlackFlower database and tune a bunch of the options myself using a nice web frontend. Be sure to tell it to auto reconfigure apache2 since that's the web server we're using. It will ask for the database root password you set two commands ago when you installed mysql-server
sudo apt install phpmyadmin

We now have a web server up and running with phpmyadmin as a friendly MySQL frontend to manage databases (include blackflower's) available at http://gobox/phpmyadmin/ (or whatever you set your hostname to)

Next up is installing blackflower, which consists of copying all the web assests into the web server folder (/var/www/html/ in Debian-based systems) and running the included bash script to load information into the blackflower config file and copy a blank database into MySQL. The official released tarballs for BlackFlower are available here, but there's some issues with the official releases, so I actually maintain a mirror of those tarballs, plus a branch with my patches, on github.  For this deployment I'm using my "fix12" branch, which is all my latest patches against the most-recent-as-of-this-writing v1.12.3pre1 release.

In these instructions, I'm copying the BlackFlower files into /var/www/html/cad/, so that the cad system will be available at http://hostname/cad/, which means the same web server can be used for other things, but if you want to use a different URL, just change the part after /var/www/html/.

Install BlackFlower

mkdir -p ~/src
cd ~/src
git clone -b fix12
# Copy the app files into the web server
sudo mkdir -p /var/www/html/cad
sudo cp -r ~/src/blackflower/* ~/src/blackflower/.htaccess /var/www/html/cad/
sudo chown -R www-data:www-data /var/www/html/cad
# Configure BlackFlower for the local environment
cd /var/www/html/cad
sudo bash ./

This script prompts you for all the information to access your MySQL server and what your main admin account name should be.

Host for MySQL database: Accept default (localhost)
Name of host to allow apps: Accept default (localhost)
Enter MySQL admin password: whatever you entered during the apt install of mysql-server
Enter database name to create for CAD: Enter something unique if you want to run multiple instances of BlackFlower on one system, but for stand-alone systems, the default is fine (cad)
MySQL username: Accept default (cad)
New password for this user: Enter something good
Enter CAD administrator username: This is where we're now talking about users inside the CAD system instead of MySQL users. The default is good (Administrator)
The Administrator user starts with the password "default-admin-pw", but it will prompt you for a unique password once you log in. This is the password for just logging into the BlackFlower cad system, so depending on the scale of your event, you might want a unique password for the CAD admin vs the SQL admin vs Linux admin accounts, but I generally just use the same password for all three since our IT team is handling all computer systems at the same time.
At this point, you should have a functional CAD server. If you're only planning on using it from the local Raspberry Pi, or you've already got a functional LAN at your event to hang this server off of to make it available, you're done. Point a browser at it, log in with "Administrator:default-admin-pw" and reset the password to something unique, load in units for your event, create lower level user accounts for your dispatchers, and you're ready to run an event. Exact details for using BlackFlower as a CAD system and all the tricks / conventions we use for it deserves its own blog post, but even just using its event logger system can be a set up for an event vs a pad of paper. [Insert philosophical debate about the pros and cons of a computer based system vs the elegant simplicity of a legal pad here]

Note: The root http://hostname/ URL at this point still points to the default "Apache is alive!" index.html file in /var/www/html/. You probably want to replace that with some sort of homepage for the event LAN in case someone accidentally forgets to add the /cad/ to the end. I usually just have a list of bare links to the network resources for the event, which at the very least is the CAD server, the phpmyadmin page, and maybe a folder of files like a PDF of the event map.

Note: The BlackFlower documentation also says this, but it's worth mentioning that this is just a small CRUD webapp built by a few guys volunteering for an event, so don't expect BlackFlower to be particularly hardened against adversarial attackers, so exposing an instance of it to the whole Internet or even just the event's free WiFi may be a bad idea. Every event should distinguish between the trusted and the untrusted parts of their network, and this server should be sitting squarely in one of the trusted subnets.

WiFi Access Point

So this running CAD server plus a consumer grade router/AP like the classic WRT54GL or a newer router like my favorite the Asus 66U, and you have a reasonable CAD system. But we can still go deeper! What if we didn't need a separate WiFi access point and instead used the WiFi chipset on the Raspberry Pi itself?

The official documentation for setting up the Pi as an access point is pretty good, and I followed it up until setting up the bridge between WiFi and the Ethernet interface, since I don't need that and I'm most interested in just using it as a stand-alone offline server with no Internet connection, but your exact requirements for this server being a DHCP server or client on either WiFi or Ethernet and bridging vs routing between them will of course vary.

The way I have it set up, anything that connects to the Pi's WiFi AP will be issued an IP address and can reach the CAD server, but with no Internet access or routing for the clients. The Pi's Ethernet port will still request a DHCP lease from another router, so plugging it into any existing network will still work as usual. With a little more effort, I could at least set it up so that if the Pi had Internet via its Ethernet port, it could share it with any WiFi clients. There's plenty of good tutorials for building your own Linux-based router online, which is what you'd be doing at that point.

I was a little more assertive than the guide to get local DNS working, and picked a more distinctive subnet than, since and are two subnets which tend to already be used in places.

  • Raspberry Pi wlan0 IP:
  • Local domain:
  • WiFi SSID: "GoBox" with password "dispatch"
  • Update the /etc/hosts file so gobox correctly resolves to for other clients instead of, which is the default in Raspbian.

So now we really have an entirely stand-alone box that lets you power it up, connect from a few laptops, point their browsers at, and you're up and running.

NTP Server

One last thing that would a nice to have, but I haven't done for this project yet, is having the Pi run an NTP server to synchronize the clients to the Pi, which is a requirement for BlackFlower. This would consist of configuring an NTP server, then advertising it via DHCP by including "dhcp-option = option:ntp-server," in the dnsmasq.conf file.

Two options for how to configure the NTP server would be to copy its time from the RTC hat we've already installed and configured, or attach a GPS receiver and set up the Pi as a strata 1 server. The NTPsec guys have a good whitepaper on setting a Raspberry Pi up as a strata 1 server.

For serious use for an event, I'd say getting a working NTP server is pretty important, but since I'm only planning on using my server for demos thus far, I haven't bothered to go through the steps for this. If I do at some point do this, I'll come back and back-fill that information here.


At this point, this project is still in the workbench lab environment stage, so I have no experience deploying this minimalist of a CAD server for a real event, and can't make any guarantees for what that experience will look like until I try it.  I can't take responsibility for what comes out of anyone trying to use this setup for a real event, but I would be interested in hearing any stories from other amateur organizations on their experience using either BlackFlower or any other CAD system to support events.

Monday, July 17, 2017

Replacing Dead Capacitors in Consumer Electronics


Be warned, this video starts a little slow, but once I get around to it, I go into very fine detail on how to spec out replacement capacitors to buy to replace failing ones in your equipment. Today's patient was a Netgear FS116 Ethernet switch.

Links to most of the needed equipment is listed in my Getting Started in Electronics post.

Thursday, July 6, 2017

Working a Professional Fireworks Show

Since this last "weekend" was the 4th of July, fireworks were happening... pretty much everywhere. This is the story of what I did with my Monday/Tuesday.

This wasn't the first time I've worked a professional show. I actually worked one last year, then promptly never got around to writing about it, so sorry about that, but the main question people ask there is how did I get into working professional fireworks shows once a year?

The answer is pretty simple: I answered the phone.
Me: "Hello?"
Laura (friend who happens to be a licensed pyro): "Kenneth. I know you like to do crazy stuff. My friend needs more hands to work a fireworks show. Would you be interested?"
Me: "Yes. yes. Matter of fact, yes I would. Dear god, yes. Yes."

Fast forward to this year, and she hadn't heard anything too terrible about my performance last year, so she invited me to work the City of Cupertino show with her this year, which is what I'm detailing here.
Monday (day before show), 10AM: The box truck arrives on site. This truck is carrying the racks of tubes for the shells, but not the actual fireworks.
 The first day is pretty much all just hard labor. Each of these plywood + 2x4 racks holds five 3" HDPE guns, and they all need to go from the box truck to the lawn.
A few packets of the show plan are floated around showing how many racks needed to be in each of the nine clusters (A-H) and how they should be grouped (set of three, pairs, etc).
 Once put in vaguely the right location, all the racks need to be nailed together so they don't fall over when the shells fire. I'll freely admit that I'm certainly not a pyrotechnics expert yet, but I was pretty quick to get on-board with the desire for the shells to go up first, and then explode.
While nailing them together, we also accounted for the expected wind direction with a bit of a general tilt to the right and the desire to give the shells some level of spread, so notice how we mounted them with some angle spread in each block.
After lots and lots of nailing, it was time to clean out the guns so they'd be ready to get loaded the next day. The last show who used these inevitably left all sorts of debris in the gun which we don't want below our fireworks, so it's a matter of checking each tube and cleaning it out.
After a hard five hours, we were pretty much ready for the fireworks the next morning and called it a day.

Show day, 10AM: The product arrives in another box truck. 860 pounds of glorious 1.3G explosives.

Due to the limited size of the field we were firing this show in, the show was limited to 3" shells, so they weren't going to be going particularly high, but we were making up for it in shear quantity. 950 loose shells, plus another 150 in "cakes," which are 25 shell clusters that just fire one after another, generally as the final crescendo to and during the finale.
We now had 11 hours to unpack all the fireworks, sort out which guns in which clusters each one was loaded in, and set them all up to get launched during the ~20 minute show. The cakes are real easy since they come pre-packaged, but we've got 950 loose shells that need to be laid out.

Before loading them in the guns, we place them on top of each gun for the inevitable shuffling of shells when we come up short of one kind and have a long string of the same kind or color in one of the blocks which would be visually boring. Swapping some of these for some of those in the next block, etc etc.

Notice how each shell is pre-wired for both electronic firing  via wire (which is what this show is using) and manual firing via the quick match fuse.
Quick-match is neat stuff. It's gun powder infused cord, which on its own burns relatively slowly (kind of like how everyone expects "fuses" to burn), but once you wrap it in a paper/tape wrapping to contain the burning gun powder, it becomes a propagating explosion, which runs down the quick-match at several hundred feet per second. When you want to manually set off a shell, route this stuff out the top of the gun, light the few inches of unwrapped fuse, DUCK, and it almost immediately goes up. Last year, I helped fire a show entirely manually, which literally meant that when the show started, the operator handed me a 20 minute road flare, and told me to start on one end of the row and while waving the flare around try to only launch one shell at a time.
 This show was instead an electric fired show, so we were using the electric squib wired to each shell.
This consists of fine gauge zipcord and what looks like a match head which has a resistive element in it to set it off when you pass enough current through it.
So now we've got ourselves a bit of a problem. We have 950 shells with electric squibs, and we need to wire all of them back to one place to be able to fire them all off. Thankfully they're all getting fired off in groups of two, or three, or more, so we can wire each group in series, but we're still talking about a LOT of wires.
For the larger 5-10 shell clusters, we don't bother wiring a squib to each shell, but actually splice them together using quick-match and one or two squibs at the end.
 So we need to wire all the shells. This is done off of an address sheet where each channel on each breakout box is listed with how many of which type of shell should be attached to it.
So we wire, and wire, and wire. Thankfully, others on the team were smart enough to bring pop-ups, so we were able to do this in shade, which was important, since this was the vast majority of the day.
And wiring and wiring and wiring... Seriously, I meant it when I said a LOT of wires. This is just one of the breakout boards. Each breakout has 45 channels, which are simple spring clips, and best I understand they all share 5 common ground returns through 50 pin Centronics connectors to the trunk lines back to the firing panel. Some of my Twitter followers rightfully so gave me a hard time for falling short of my usual wire management standard here, but it helps when you know that you're literally going to be blowing it up in a few hours and then frantically shoving it in trash cans.
After running the trunk lines back to the firing panel, we can do a continuity check on all the shells to make sure between the series wire splices, spring clips, breakout boxes, trunk lines, and the firing panel we don't have any opens.

Of course, we had several opens (see channel 3 in the photo above), so it was several iterations of checking the panel, noting down the open channels, turning off the panel, running out and finding the broken wires, fixing them, then clearing the field, then checking the panel, and figuring out which other wires we broke during the last round, etc. etc.
So this takes us up right to the show. To start the show, we light off one, then a second shell, to get everyone's attention and then show where the fireworks are going to be coming from. We then give everyone a minute or two to get themselves settled in and facing the right direction, before starting the electronics and running the show.

Yes. That is a photo of my hand, with a road flare. I was picked to set off the two starter shells, so I got to walk out into the middle of a lawn, filled with explosives wired to go off, light a road flare, and touch it to the first two fuses.

You know, when I put it like that... it really makes you ponder the life decisions that have gotten you to the place where you stand...
Then again, it was a pretty good view...

So then the show is over, and we need to go clean up the little mess we've made in the middle of some grounds keeper's pride and joy setting off 1100 explosives.
 Fire up the lights, and the hunt is on!
The hunt for the duds. We put a lot of effort into making sure that all the fireworks went off during the show, since that's pretty much the whole point of a fireworks show - it going up and being entertaining. Now it is time to find the shells which didn't entirely subscribe to this same objective for this evening and hadn't felt compelled to launch themselves into the air and the whole blowing up thing.

Poke a stick into each tube, check to see that it's empty, and pour out the shells that are still left and collected them in a pile. I know. Very technical.

Of course, getting another truck out here which is placarded and licensed to transport 1.3G explosives for just the six duds we had would be a bit of a waste of time, particularly when we've got 950 perfectly good launching tubes already set up, a fire marshal who's being a good sport, and 15 pyros who came to set off some fireworks.

Remember how each shell is both wired for electronic and manual firing? This is when the manual fuse comes in handy for setting off these six duds. This is also where I was handed a second flare and sent out to fire off another six shells manually. I mean, I don't mind, but I'm starting to suspect there's some not-in-their-20s self-preservation impulse for everyone else that kept resulting in me being the only volunteer for the manual shots...

Have you ever wondered why shows set off a few fireworks about a half hour after the show ends? This is why. They were just missed during the show and the operator really doesn't want to have to carry them home.
So it's now about 10:30PM, the fire marshal and our licensed operator have agreed that the explosives are all gone, and it's now a frantic rush back onto the field with hammers to dismantle the carefully nailed together racks and load them all back into the box truck. This takes us to just shy of 1AM, at which point it's some hearty hand shakes, hugs, and a wave farewell until next year.

Sunday, June 25, 2017

YouTube Channel IoT View Counter

I've wanted an Internet connected read-out for some time now, inspired by the awesome shadow box IoT projects Becky Stern has been doing (weather, YouTube subscribers). I'm certainly not to the same level of packaging as her yet, but I've got a functional display working with a Hazzah and an eBay seven segment display module.


Bill of Materials:

I soldered the Hazzah and display onto a piece of perf board I had laying around, which didn't work so well since the display was longer than the grid of the perf board.

I programmed the ESP8266 through the Arduino environment, which means you need to install the Arduino IDE, install the ESP8266 board module following their instructions, and install the YouTube API library through the library manager.

You'll also need to generate a Google API key for yourself, which you can do following the instructions on the YouTube API library's README.

The CHANNEL_ID macro is the random characters at the end of the URL for any channel of interest. For example, for my channel you'd want the part in bold:

The hardware is relatively simple: solder the provided headers onto the Hazzah, and connect the five needed lines from the Hazzah to the MAX7219 display:
  • Hazzah - MAX7219
  • V+ - Vcc
  • GND - GND
  • Pin 12 - CS
  • Pin 13 - DIN
  • Pin 14 - CLK

If you wire up the chip select, clock, and data lines differently, you should only need to change the macros at the top of the source code to correctly reflect your setup.

Of course, the possibilities here for packaging this display better are endless, as is using either the ESP8266 and/or these $2 displays to display any other information than YouTube channel view counts. The fact that these displays are chainable even makes it possible to make large multi-line displays with them.

Tuesday, April 18, 2017

Mastr 3 UHF Low Pass Filter Testing

Last year, I got a good deal ($50) on a pallet of Mastr 3 repeaters. They were all T band (490MHz) and more or less clapped out, but I figured it would be a good learning experience (and I got a nice tear-down video out of it).  Being way off the amateur band and wrought with problems, most of the electronics have been of little use to me and has been little more than a learning opportunity, but one piece I see myself being able to use elsewhere is the final low-pass filters from the power amplifiers (Part number 19D902856G9).
I've built low pass filters before, but wouldn't trust my own constructions to actually being put in use for where you need these filters the most on VHF/UHF, which is on repeater systems. Repeaters tend to be key-down for long periods of time, so power handling in the filters is important, and since they're remote if something goes wrong there's no one there to notice and getting to it to service it is generally inconvenient.  Since these low-pass filters off the Mastr 3s are already designed for high power (90W) continuous duty, they're perfect for my repeater applications (Usually 10W-25W) with the one exception that they're tuned for 470-512MHz, where all the repeaters I tend to build are in the 440-470MHz range.

That being said, 440-470 is lower than 470-512, so it's conceivable that the impedance in the pass-band might still be acceptable, and then the only downside is that I wouldn't enjoy quite as much filtering of the second harmonic than if the filter was really designed for 440-470MHz.
One thing I found interesting about the filter design was that it isn't symmetric! It's a 7th order LC filter, but the capacitors to ground only reduce in value on one side and not the other! My best guess as to why this is the case is that the engineers designing this filter knew that the output of the power amplifier wasn't perfectly 50ohms resistive, so they added a little reactance here to better match the amplifier.

To muddy the waters, the schematic and the physical unit I took apart don't agree on which port has the lower capacitance, and I wasn't careful enough when disassembling the amplifiers to say which port was actually connected where. I also enjoy that this model of the filter has black boxes instead of the inductance values filled in like on the other LPFs in the spec sheet; I'm not the only one fudging the documentation at times!

In any case, I just want to check the pass-band behavior of these in the lower 440-470MHz range, and see if these are something worth keeping or not. To the ungodly expensive vector network analyzer!
So on the top we've got the S21 plot going through the filter, so you can see the flat, low insertion loss, region on the left for the desired signals, and then the filter starts rolling off as you go higher in frequency, to the point where, on the right side of the plot, 900MHz is attenuated 45dB. The two markers 1 and 2 are at the two limits of my frequencies of interest, so you can see how the filter stays flat for much longer than I need before starting to roll off, so that 900MHz attenuation number could be better.

On the bottom, we see smith charts for the J1 and J2 ports (way zoomed in to see the detail around the center), so there's certainly an asymmetry to the filter, but both ports are decently close to 50 ohms so this isn't a game stopper. In an ideal world these filters would be a dot at the middle of the smith charts, which would be 50Ω + 0jΩ (50 ohms resistance plus zero ohms reactance), but they start wandering off into the wilderness as you go higher in frequency, which makes sense since the whole point of a low pass filter is that the insertion loss goes up with frequency, so the impedance usually does weird things (you can design flat impedance diplex filters if you care, but we really don't here)
So now lets zoom into the pass band we're interested in and check the SWR to sanity check that this filter is an acceptable match for my transmitters at my lower frequencies. Annnnd they are! This filter actually happens to bottom out its SWR curve right at 470MHz, so this filter is going to be great for my part 90 stuff, and still find for amateur stuff with an SWR of <1 .20="" 90="" a="" amateurs="" for="" gear="" have="" lower="" nbsp="" of="" p="" part="" radio="" standard="" than="" thumb:="" ule="">
Notice that this check was pretty valid, since the SWR starts sliding upwards at lower frequencies. I probably should have done a sweep further down in frequency to demonstrate how bad it can get once you wander outside the band the filter was designed for, but these measurements were being taken on borrowed time on the company VNA.

Great. In any case, now lets take it apart!
Ten screws reveal about what is expected, four inductors and three capacitors to ground.
Of course, at UHF the needed inductance is about one turn, so these are pretty minimalist inductors! The capacitors are a little odd; I haven't seen these sorts of clad mica capacitor before, but they're designed to handle the high RF current seen in a filter like this without needing to be huge for the power dissipation from using a normal termination instead of these heavy wrap-around metal bits. The rated working voltage for these capacitors is just 100V, which was a bit surprising to me, but makes sense once you consider that this is a 50Ω 90W system.
My favorite part of the die-cast shield is the little cutouts in the walls leaving a gap where the strip line passes underneath the dividers between stages. I don't know if those are designed for a critical dimension or just "tiny gaps."

In any case, I've now got a pile of these great LPFs for various repeater projects. They're usable as-is, but I can always go in there and tweak inductor/capacitor values if I want to use the chassis for something else (or actually tweak them down to my frequencies of interest).

Tuesday, March 7, 2017

Podcast Interviews on APRS

When it comes to amateur radio, APRS is pretty near and dear to my heart. I did my masters thesis on it, and I'm now the maintainer of Aprx, which is one of the most popular digipeater software packages.

I recently started talking to Cale, K4CDN, and have recorded a pair of podcast episodes for HamRadio360 on APRS. Enjoy:

  1. All About APRS with W6KWF
  2. APRS Follow-up (Listener Q&A)

Tuesday, January 17, 2017

APRS Symbol Look Up Table

Click to Enlarge
In APRS, symbols on the map are encoded as two ASCII characters. The first character selects the table (primary or secondary) and the second character selects the symbol from that table. (The table selector can also be used as a alphanumberic overlay on top of the secondary table)

The symbol list is available online here, but I've always wanted an easy-to-use graphic with the symbol codes and the icons right next to each other. I finally broke down this week and created the image you see above based on Hessu's great APRS vector symbol set. Click on the table to see it in full resolution.

To use this table, simply find the icon you want to use, read off the symbol code above it, and use "/[symbol code]" if it's in the middle square and "\[symbol code]" if it's in the bottom square.

Sunday, January 1, 2017

2016 Retrospective

It's the end of the year, so I figured I'd do a general life recap and wax on what I'm looking at in 2017.

First and foremost, I changed jobs this year. I'm now an electrical engineer at Lam Research in the etch engineering group, where I'm working on their wafer plasma etchers used in semiconductor fabrication. This is particularly fun because I got this job through one of my Cal Poly Amateur Radio Club cohorts, so I get to see him nearly every day (thanks Sean!). It's an interesting group to work in since wafer processing spans a wide breath of fields including power systems, vacuum tech, RF power, embedded controls, etc. Most of my workload is related to my experience in RF systems, so designing RF filters, but the Lam Etch Engineering group is organized in such a way that I enjoy a high level of diversity in the problems I get assigned to, so I'm not concerned about ever getting pigeon-holed into just working on RF.

My group at Lam has been growing very quickly this year. Even just since I started we've hired three more EEs and looking for a few more. If you're in the bay area and looking for a job feel free to shoot me an email if you're interested.

My personal hobbies have continued to be involved and time consuming. My previous new year's resolution was to get back into public speaking and give four talks to radio clubs. I only ended up developing two new talks this year, but there was enough interest in them to take them on tour and give the same talks several times so I think it mostly counts as a success:
I've been having fun building and deploying radio repeaters. On the bench side, I've been putting together a collection of videos on repeater building [1][2][3][4]. Online information on repeaters is tough to come by, and I don't feel like I've even made a dent yet, so plenty more repeater videos are yet to come. On the repeater deployment side, I got to spin up several of my personal repeaters at events like the Wildflower Triathlon under my commercial radio license and at the Bay Area Maker Faire on the amateur repeater test pair.

In addition to working communications for the Wildflower Triathlon for TriCalifornia, I got talked into volunteering for several other of TriCal's events. A dedicated communications team isn't needed for their other events, so I instead got to work on the course team placing cones and traffic control around the race courses for the Pacific Grove Triathlon, the SF Triathlon at Alcatraz, and the Giant Race. Getting to drop traffic cones off the back of a moving box truck is a kick.

Sadly, TriCalifornia recently announced that they're not hosting any of their own events in 2017, so my event support in 2017 is going to need to be more diverse.

I've been having fun climbing towers for the Salinas Valley Repeater Group this year. They're a good group of guys with an impressive repeater network set up covering the bay area down to Fresno. I love tower climbing since it's a good day of exercise on top of a great excuse to get tours of radio sites.

In the category of "crazy adventures I never got around to writing about," one of my friends this year called me up at the last minute in the end of June:
"Hey Kenneth, I know you like to go on crazy adventures. You want to help one of my friends put on a commercial fireworks in Ft Bragg?"
Yes. Yes I do.

It ended up being a full day experience of get up at 5am, drive up to Ft Bragg, spend the afternoon unloading 650lb of fireworks and the corresponding gun racks out of a box truck, and then firing the show before crashing in a motel. The fun part about these "hand-fired" shows is that we don't use a mile and a half of electronic matches to light off the shells but I get handed a 20 min road flare at the beginning of the show and get to run around using the road flare to light off the fireworks.


So now for the question on what I want to try and do in 2017. My tradition of trying to pick up a new hobby every year started getting more difficult recently mainly because my hobbies (repeaters, beer brewing, tower climbing, etc) all tend to be so equipment intensive that my apartment is starting to get ridiculous with all my stuff.

I think I was on to something good with the 2016 "public speaking" objective, so I think I'm going to go on a similar vein this year and try and get four magazine articles out the door. I've been complaining about how disappointing the amateur radio magazines have been long enough that I just need to put up and start delivering the content I want to read. The ARRL also pays pretty decent money for accepted content too, so that's a nice perk that this year's hobby might actually be cash positive for once. We'll see at the end of the year as to how I do.

Here's to another year, and I hope all of your objectives and achievements find you in good health as well!