Monday, January 15, 2018

Off-the-Grid Raspbian Repositories

For one of my future projects, I'm looking at spinning up 15-25 Raspberry Pi desktops in the middle of nowhere, with somewhere between zero and awful Internet connectivity. We're making a best effort wag at what the OS image should be before-hand, but I expect we will inevitably get there and realize we want to install some other piece of software from the Raspbian software repo.

I figured the best solution to this was to pre-download ALL of the software available for Raspbian (it's only ~400GB) and sneaker-net it in. I also don't want to be making any changes to the config files on the Raspbian installs, since that would mean we'd need to remember to set things back to normal when we're done, so I want this local copy of the Raspbian repos to include some DNS trickery to make the Raspberry Pis think they're still actually talking to the actual repos on the Internet.

This means I need:

  1. A router to give the Pis IP addresses and respond to their DNS queries for mirrordirector.raspbian.org and archive.raspberrypi.org with the address of the local server instead of those server's actual public IP addresses.
  2. A laptop with at least 500GB of disk space running apache2 serving a local copy of mirrordirector.raspbian.org/raspbian and archive.raspberrypi.org/debian


My router of choice for these sorts of scratch projects where I need something to do DHCP/DNS/NAT but don't particularly care about performance or getting my router back afterwards is the WRT54GL. I've got a stack of them running mainline OpenWRT, not because OpenWRT has a nice interface, but because it is exceedingly flexible.

Beyond the basic configuration of picking a new subnet that isn't 192.168.1.0/24 and setting a hostname for it, I made two changes:

  1. A static DHCP address allocation for my laptop running apache2.
  2. I sshed into the router and created a new hosts file that gives the laptop's IP address for the two repos I'm trying to fake.
Hosts files are the simplest way to override DNS records for single hostnames. I created /etc/hosts.rpimirror and it contains the two domains associated with my laptop's static IP address:
10.2.2.2    mirrordirectory.raspbian.org.
10.2.2.2    archive.raspberrypi.org.
Once I created this extra hosts file, I added it to the web configuration via "Network > DHCP and DNS > Resolv and Hosts Files > Additional Hosts files", but of course your router's method to add additional hosts files will vary if you're not running OpenWRT. 
At this point, any Raspberry Pi plugged into this router will get faked answers back for the two repo domains, and will try and send all of those requests to the laptop, without having to edit any of the sources files on the Pis, so as soon as they get plugged into any other router, they'll behave just as normal.

Now all that's left to do is to create a local mirror of those two repos and make them available over http. 

To copy the mirror files to the laptop, I wrote a simple script which uses rsync to copy the Raspbian repo and debmirror to copy the needed parts of the raspberrypi.org repo (mainly because it doesn't support rsync for some reason). We're talking about something on the order of 400GB of files here, so strap in; this download is going to take a while regardless of how fast your Internet is. Once it's done, you should be able to rerun the same script and have it go relatively quickly since it will only be updating your existing local copy.

You will need to change the MIRRORBASE variable to somewhere on your computer that you want to store all of these files, since I have it hard-coded in my home directory.

I'm also only mirroring the stretch packages from archive.raspberrypi.org, so you will need to edit the "release" variable if you're using a different release of Raspbian.


Script was based on the Ubuntu debmirror page and the rsync command at the bottom of the Raspbian mirror's list.

Once that download is done, you'll need to link the mirrordirector.raspbian.org/raspbian and archive.raspberrypi.org/debian folders into apache's /var/www/html root so both folders are visible:
sudo ln -s /home/kenneth/mirror/archive.raspberrypi.org/debian /var/www/html/debian
sudo ln -s /home/kenneth/mirror/mirrordirector.raspbian.org/raspbian /var/www/html/raspbian


At this point, you should be able to see the two repo's dists and pool folders by pointing your browser at "http://laptop's_address/raspbian" and "http://laptop's_address/debian", and any Raspberry Pis plugged into the router will be able to download any new software or updates entirely locally without a connection to the Internet.

While this does let you install any updates or software existing at the time of your repo snapshots, this doesn't get you any updates or software released after the snapshot. One option for that, assuming this local repo is somewhere entirely without Internet connectivity, is to carry it back to somewhere with Internet, rerun my repo mirror script above to download any changes to the repos, then carry the server back to where it's being used.