Firewall throughput measurements: OPNsense on APU4d4, OPNsense in a Proxmox VM, and OpenWRT on Turris Omnia

Why

For a few weeks, I have been struggling to make OPNsense perform well from a performance point of view on my low-power test box, an APU4d4. While OPNsense is very well done from a firewall rules management point of view (alhtough I am not happy that forwarding rules cannot specify both incoming and outgoing interfaces like it is possible with Linux Netfilter…) and has many features of expensive firewall products (including web interface based management for clustering/failover), the FreeBSD/HardenedBSD kernel seems to be struggling with higher throughputs. After not progressing with simple trial&error with various settings gathered from different howto guides, I decided to first measure my current status properly.

In the last ca. 10 years, I have running my home lab setup with OpenWRT based routers (for a long time on Mikrotik RB2011, which is extremely power efficient for what it can do), more recently a Turris Omnia for the automatic updates coupled with maximum flexbility (and the snapshot features are really well integrated). However, for teaching our course on “Network Security” at the Institute of Networks and Security at JKU Linz, we decided to use OPNsense because it comes with an easy-to-understand web interface and is open source. A direct comparison therefore seems useful.

All systems under test have a roughly equal IP (v4 and V6) and firewall rules configuration. For completeness, I compare the OPNsense installation on the APU4d4 to a similarly configured OPNsense instance inside a VM on the same Proxmox host.

How

My setup is pretty simple: a Proxmox server hosting a small number of VMs that are connected to a DMZ VLAN, attached through a Linux host bridge that connects virtio network interfaces for the VMs with a tagged VLAN on the hardware NIC as a trunk to the local Ethernet switch. On the same switch, I have a desktop connected through a 1Gbps link. The switch is configured as a pure L2 switch (with multiple VLANs), all routing is done through the firewalls under test. One VM on the Proxmox host runs the iperf3 service, the host itself (on a different VLAN) as well as the separate desktop run iperf3 clients.

The three systems to compare are:

OPNsense on the APU4d4 has the recommended settings from here and here applied. OPNsense inside the VM has NIC hardware offload features disabled and VM configured with recommended settings from here as well as all VLANs terminated on the Linux kernel host side and bridged into the VM as independent virtual network interfaces (the consensus seems to be that VLAN tag handling is faster on Linux than BSD in such a virtualized setting).

Results

First I took a baseline mearurement with an iperf3 client running on the VM host itself, connecting to an iperf3 server running within a Debian 10 VM without any of these test systems in the routing path, but simply a virtio network connection on a single VLAN / IP subnet. The limit was CPU bound, as my Proxmox host (wich a low-power CPU) ran around 90% load over both cores during this baseline test.

All measurements were taken with iperf3 in TCP mode with 1 or 4 parallel streams and 20 repetitions:

iperf3 -c <server IP> -P <number of streams> -t 20

Measuring from the VM host to the VM (but on different VLANs, forcing traffic to be routed through the firewall under test), first without IPsec active (all transfer rates in Mbps):

Note that the VM->VM measurements, when going through the VM OPNsense instance, are all on the same physical host and therefore not bound by any hardware network limits, but only CPU and efficiency of the 3 network stacks involved. It’s interesting that IPv6 traffic was faster than IPv4 in this case.

Then with two IPsec tunnels to external sites configured and up/routed, but with the traffic under test explicitly not being routed through the tunnels. That is, those tunnel policies are loaded in the kernel, but the test traffic should not be matched by those policies. As we can see in the results, there is however a very clear performance impact on OPNsense when we hit CPU limits (I have only measured 4 streams, as we already know that single stream performance is limited on OPNsense with low power CPUs):

Repeating the measurements from a physically separate client, with the test traffic going through a physical switch to the (physical or virtual) firewall under test, then (for the two physical firewalls, not the VM one) through the same switch (different VLAN) and to the Proxmox host, where the VLAN tagged traffic is bridged into the VM running the iperf3 server:

Conclusions

For standard routing and firewalling of multiple parallel streams, OPNsense on a low-power APU4d4 system performs a bit better (noticably better with IPv6) than a Turris Omnia with slightly less electrical power draw under load. OPNsense has the advantage of much nicer UI for firewall rules (including the possibility to define host objects and groups spanning IPv4 and IPv6), more control in terms of monitoring the firewall, nicely integrated modules like VPN protocols, and the beginnings of an API for automated configuration. Pretty much all of that can also be done with OpenWRT, but mostly on the shell or through config files of wide variety. None of these physical systems reach a full Gbps firewalling speed like the even lower powered Mikrotik systems with RouterOS and Fasttrack do.

However, there are currently two areas of concern:

• Single stream performance is worse, and this is a known problem for FreeBSD kernels. For a single stream (e.g. uploads/downloads to/from a local fileserver), OPNsense (both physical on the APU4d4 and virtual on a power efficient server CPU) is limited to about half the maximum Ethernet throughput. This may or may not be relevant to your use case.
• When IPsec is active - even if the relevant traffic is not part of the IPsec policy - throughput is decreased by nearly 1/3. This seems like a real performance issue / bug in the FreeBSD/HardenedBSD kernel. I will need to try with VTI based IPsec routing to see if the in-kernel policy matching is a problem.
• These tests intentionally deactivated some of the interesting OPNsense features such as traffic analysis with samplicate/flowd_aggregate. Enabling them will again cost around 150-200Mbps throughput on the APU4d4, stacked on top of the performance drop of IPsec if all are enabled.