Linux on a Netgate SG-8860-1U

I recently obtained a second-hand Netgate SG-8860-1U firewall, which has now replaced my PC Engines apu2c4 board as the router in my home network. The apu2c4 is a perfectly good piece of hardware, but it only has three Ethernet ports compared to six on the Netgate machine. I was using all three ports (one connected to my modem and two bonded together and connected to my managed switch), so having some spare ports for expansion is useful – I’m already thinking about how I could rewire things to try to achieve 2 gigabits/sec with bonding between my desktop and server.

The Netgate machine comes bundled with pfSense, which is based on FreeBSD, but I’m not a particular fan of either – all my servers run Debian. I know a lot of people rave about how good the BSDs are at networking, but I’m not really convinced if you compare them to a modern Linux system. nftables is a significant improvement over iptables. Furthermore, people struggle to get the apu2c4 boards to do gigabit routing on BSDs, but Linux copes fine while using an insignificant amount of CPU power in comparison.

The specifications of the Netgate machine are:

The firmware is based on coreboot. It’s very bare bones – you can change the boot order and access the PXE console, but that’s about it. PXE only seems to work on the OPT1-4 ports, but I didn’t test this extensively. Booting from a USB stick also works.

It’s all fairly standard, so I thought it’d be quite easy to get Linux booting on it, but it did take me an entire evening trying to get the serial output to work.

On your desktop/laptop you’ll need to use a command like picocom -b 115200 /dev/ttyUSB0 to open the serial port. I didn’t have any problems getting output from the BIOS or GRUB, but I didn’t see any output after that point – making it very difficult to debug.

I discovered, after much trial and error, that the internal USB to serial adapter is actually connected to the COM2 port, so you need to pass console=ttyS1,115200n8 to the Linux command line. I’d been trying with ttyS0 instead, assuming it used COM1!

After that point everything went fairly smoothly, ignoring the slight annoyances that come with doing everything over a 115200 baud serial port instead of a high-speed SSH connection with a full-screen terminal.

Once you’ve installed Linux you need to make sure that your bootloader (e.g. GRUB) will continue using the serial port and pass the correct console= option to Linux. This varies between distributions. In Debian you need to set the following variables in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200 --parity=no --stop=1 --word=8"

and then run update-grub to update the /boot/grub/grub.cfg file. systemd automatically spawns a getty on ttyS1, so you don’t neeed to worry about fiddling around with /etc/inittab any more.

Modern Linux distributions assign network interface names predictably, and I spent a little bit of time mapping the predictable names to the labels on the front panel (hopefully this will save some people’s time!):

Network interface name Front panel label
enp4s0 WAN
enp3s0 LAN
enp0s20f0 OPT1
enp0s20f1 OPT2
enp0s20f2 OPT3
enp0s20f3 OPT4

With a fairly complicated nftables ruleset, including connection tracking, it’s capable of routing traffic between two subnets at around 930 megabits/sec, which isn’t too far from the theoretical maximum after accounting for Ethernet and TCP/IP overhead:

~$ iperf -c rictusempra
------------------------------------------------------------
Client connecting to 81.187.119.162, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.2 port 43218 connected with 81.187.119.162 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   934 Mbits/sec
~$