FCIX - State of the Exchange
The First Year of Running an Internet Exchange
It's that quadratic growth of the number of interconnects in a full mesh that really get you.
But that's exactly the problem that Internet Exchange Points are meant to solve; I have N number of networks that want to interconnect in one place without running O(N2) Ethernet cables between each other, so everyone connects to one central Ethernet switch and it's just as effective at a much lower cost of entry.
So eight months ago, we jokingly registered the domain FCIX.net, grabbed a spare /24 + /64 of address space we had laying around, and founded the Fremont Cabal Internet Exchange.
We had a good laugh setting up a cheeky little website to make it look like we're a real Internet Exchange, which lasted all of about two weeks before the owner of the data center, Hurricane Electric, applied to our Internet Exchange, brought in the 75,000 prefixes from their customer cone, and put us on their advertising material for the building.
Well crap. That joke got out of hand rather quickly.
Membership Growth
For the first few months there, we were handling two or three new membership applications per week, so it became evident that we needed to get our act together rather quickly.The first point of order was dealing with the fact that we were running this no-longer-just-between-friends Internet Exchange on borrowed address space, so we needed to get the exchange its own ASN and /24 + /64, but that's $550 for the ASN and $250 for the resource service agreement for the address space... A bit of a problem when we have zero budget...
But at that point we had the advantage of having ~15 current + pending members, so we passed around the hat and between our membership and some other very amused on-lookers, very quickly managed to scrape together the $800 needed to cover the registration costs of FCIX's resources.
Not only did this enable us to re-number onto a real ASN and IXP address space, but getting such a concrete signal of support from our members was touching. This thing wasn't much of a joke anymore; we're actually providing a service to networks which they value enough to throw a few hundred bucks our way to make it happen.
New membership applications have slowed down, but as of this writing we're up to 25 members, which I don't think is half bad for a less than one year old exchange in the east bay in a single site.
Sponsors
So after getting a round of donations to cover the start-up costs for the ARIN resources, the next question was how to handle getting networks actually connected to FCIX. Originally, we were just running FCIX as a VLAN on my Cisco 6506 which was powering my personal network (AS7034), but that suffered from a few issues, the largest of which is that the 6506 is so old that 10G Ethernet was a cutting edge feature at the time. At best, 10G line cards for the 6500 support 16 ports, but the 10G line card we had managed to scrounge for AS7034 only had four ports for XENPAKs, and burns 100W per port, so offering 10G for FCIX was going to be problematic at such a low port density, even ignoring the issue of sourcing line cards and such vintage optics.This is where Arista stepped in and has contributed to FCIX in a huge way. I got a call from a long-time friend who works at Arista who liked what we were doing and was interested in getting us a real switch to run the exchange on. This means that we got a pair of Arista 7050S-64 switches, which have 48 SFP+ ports which can support either 1G or 10G optics, plus another four QSFP+ ports for 40G, because hey, maybe we'll need 40G at some point...
This now only left the issue of optics. Every member port that we turn up needs an LX or LR optic, which even from third party vendors start to add up quickly (remember how we have zero budget?), so we were very rapidly tapping out our junk bins of left-over optics we all had laying around. So while we sat there brain-storming ways to work around this sustainability problem, we got an awesome direct message on Twitter from Flexoptix!
Flexoptix is a third-party optical transceiver vendor who has the additional advantage that they sell what they call their "FLEXBOX" which allows you to insert one of their optics and over USB reprogram their optic for any vendor's switch which you need, so even though we've got that scrappy "we'll use whatever switch we can drum up" aesthetic to us, we only need to stock one tray of 1G and 10G optics to cover any possible switch we'd want to plug these optics into. Furthermore, as we moved from my Cisco 6506 to our shiny new Arista, we were able to simply reprogram the optics and reuse them, so already the flexibility of their universal transceivers have borne fruit.
The Cost of Entry
Having been started originally as mostly a joke, we've been very against charging any kind of membership fee to join FCIX, for multiple reasons:- There is already several established pay-to-play IXPs in the bay area, so trying to charge for ours when we have none of the valuable peers that existing exchanges have would be kind of silly.
- We all have real day jobs, so if someone paid to join FCIX and it then stopped working during the day, they'd rightfully kind of expect us to get it working again ASAP. My day job boss probably wouldn't appreciate that, so we don't charge anything, and problems get fixed when we can get to them. Outages, of course, are refunded in full. (see what we did there?)
This sort of zero cost of entry and zero membership fee model definitely wouldn't have been possible without all the donations we've gotten, and particularly Flexoptix donating trays of optics so we can light our end of every new member's port without trying to deal with every new member somehow contributing an optic to FCIX to light their port.
This has meant a few issues with people trying to abuse our free model, so we very quickly needed to institute an informal "one port per cabinet" rule, since we at one point got six applications from different people sharing one cabinet, and although Hurricane Electric is sponsoring all the cross connects, I'm not going to abuse that deal to run six pieces of fiber to one cabinet. Charging a one time turn-up fee like SIX does probably would have been a good idea and prevented most of our issues with low quality membership applications.
The most exciting piece of news with regards to growth is that Hurricane Electric has agreed to sponsor FCIX with a second cabinet in their other building, FMT1. That, plus a pair of dark fiber between the two buildings (thanks HE!), plus a pair of LR4 40G optics (thanks Flexoptix!), plus a second switch (thanks Arista!), and FCIX will soon be multi-site! Membership applications from FMT1 are now open.
The other challenge has been coming up with projects to focus on to help increase the value of the exchange to existing members. Adding new members is easy, but we have also been working on getting things like cache appliances and DNS servers on-net. Verisign was kind enough to contribute a J root + .com/.net DNS server to the exchange, so anyone running their own recursive DNS resolvers get to enjoy direct access to J root and B.gTLD over the exchange. Work on other value-add appliances is on-going.
As we head into this new year, I couldn't be happier or more grateful with how far we've gotten with this project while keeping it sustainable. The annual expenses specific to the exchange are still below $500 between our ARIN fees and other misc fees, and contributions of hardware and money from sponsors and members have enabled us to grow much further than we would have been able to fund on our own.
So thank you again to everyone involved in FCIX, and I wish all of you a lovely 2019.
This has meant a few issues with people trying to abuse our free model, so we very quickly needed to institute an informal "one port per cabinet" rule, since we at one point got six applications from different people sharing one cabinet, and although Hurricane Electric is sponsoring all the cross connects, I'm not going to abuse that deal to run six pieces of fiber to one cabinet. Charging a one time turn-up fee like SIX does probably would have been a good idea and prevented most of our issues with low quality membership applications.
Plans for Growth and Value Add
At this point the basic framework for the exchange has been set up. Adding new members is relatively painless and mainly involves generating them a letter of authorization to redeem for a free cross-connect from HE and adding them to a CSV which propagates to the website and route servers automatically.
The other challenge has been coming up with projects to focus on to help increase the value of the exchange to existing members. Adding new members is easy, but we have also been working on getting things like cache appliances and DNS servers on-net. Verisign was kind enough to contribute a J root + .com/.net DNS server to the exchange, so anyone running their own recursive DNS resolvers get to enjoy direct access to J root and B.gTLD over the exchange. Work on other value-add appliances is on-going.
As we head into this new year, I couldn't be happier or more grateful with how far we've gotten with this project while keeping it sustainable. The annual expenses specific to the exchange are still below $500 between our ARIN fees and other misc fees, and contributions of hardware and money from sponsors and members have enabled us to grow much further than we would have been able to fund on our own.
So thank you again to everyone involved in FCIX, and I wish all of you a lovely 2019.
