Enabling per-device traffic analysis with separate VLANs, 802.1x MAC based authentication, and OpenWRT

Last updated on 2020-12-18 9 min read

Why

For analysing what devices do on a network - specifically the shared medium of a wireless LAN - just packet tracing based on IP address is often not sufficient. There are multicasts, the initial DHCP requests, and potentially other types of traffic not captured by that. Even MAC address based packet tracing is problematic given recent defaults of MAC address randomization e.g. on Android (default since Android 10, optional before). The best solution to capture a complete picture of all communication from a single device therefore seems to be a separate link. For wired Ethernet, network taps or switch mirror ports are the tool of choice, for WiFi the easiest solution seems to be assigning a separate VLAN on the access point for each device, which can then be traced individually on the central switch/router.

The point of this post is to set this up as automatically as possible. I use the excellent Turris Omnia access point / router running a custom version of OpenWRT (TurrisOS uses a different image format and recompiled packages, but is highly compatible in terms of configuration), Freeradius 3, and 802.1x authentication of devices to assign separate VLANs. MAC based authentication is used for the devices that don’t directly support other 802.1x authentication methods (e.g. IoT devices that don’t have a sufficiently capable API to enter username/password or other credentials).

How

These notes are a summary of my setup, with most of the inspiration taken from the official OpenWRT documentation as well as this howto for Freeradius PEAP and this howto for exporting netflows. Please thank the OpenWRT project for making this fairly easy already, and all mistakes in here are mine alone.

The following assumes TurrisOS 5.0.0 (on HBS branch) or newer. It has not been tested with TurrisOS 4 at the time of this writing.

opkg update

opkg install freeradius3-utils freeradius3-mod-sql-sqlite freeradius3-mod-sqlcounter freeradius3-mod-eap freeradius3-mod-eap-peap freeradius3-mod-eap-mschapv2 freeradius3-mod-eap-tls freeradius3-mod-pap freeradius3-democerts freeradius3-mod-files freeradius3-mod-preprocess freeradius3-mod-radutmp freeradius3-mod-attr-filter freeradius3-mod-always freeradius-mod-detail freeradius3-mod-expiration freeradius3-mod-logintime freeradius3-mod-expr
- For the default config without disabling any of the modules, optionally opkg install freeradius3-mod-eap-md5 freeradius3-mod-eap-leap freeradius3-mod-eap-gtc freeradius3-mod-eap-ttls freeradius3-mod-chap freeradius3-mod-digest freeradius3-mod-realm freeradius3-mod-detail freeradius3-mod-unix freeradius3-mod-exec
Enable the guest network in Foris (the TurrisOS web interface), but don’t enable the WiFi guest networks - these don’t seem to support dynamic VLAN tagging, though I have not managed to find out why in quite a few hours of digging. For the time being, we will change the main WiFi to use 802.1x.

Note: We can only use the 2.4GHz WiFi but not the 5GHz one because of a bug in the ath10k kernel driver. Until this is fixed, only use the 2.4GHz one (wlan1). If you still try to enable the same options on wlan0, you will see an error hostapd: Failed to create interface wlan0.<VLAN ID>: -95 (Not supported) in the system log.

Add the dynamic VLAN options for the 2.4GHz WiFi AP in /etc/config/wireless:

config wifi-iface 'default_radio1'
    option device 'radio1'
    option network 'lan'
    option mode 'ap'
    option disabled '0'
    option ssid 'AndroidDeviceSecurityLab'
    option encryption 'wpa2+ccmp'
    option auth_server '127.0.0.1'
    option auth_secret 'testing123'
    option acct_server '127.0.0.1'
    option acct_secret 'testing123'
    option acct_interval 600
    option dynamic_vlan '2'
    option vlan_file /etc/config/hostapd1.vlan
    option isolate '1'

The option dynamic_vlan '2' requires that the Radius server send VLAN tags and will reject authentication otherwise, so make sure all users have a VLAN tag. Also create the file /etc/config/hostapd1.vlan with a single line:

*           wlan1.#	br-guest_turris

This will cause new, tagged VLAN interfaces to be created with the dynamic name wlan1.<VLAN ID> and added to the single bridge br-guest_turris. That means all VLANs will be bridged together again instead of the default hostapd behavior of creating a new bridge for every VLAN ID.

Make sure there is a properly configured bridge interface with an IP subnet (which should have been set up by default by enabling the guest network option) in /etc/config/network:
```
config interface 'guest_turris'
    option enabled '1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '10.111.222.1'
    option netmask '255.255.255.0'
    option bridge_empty '1'
```
Note that I keep a single IP subnet for all devices, even though they will be put in separate VLANs. That makes it a lot easier from the IP point of view (only a single address space and DHCP range configuration is necessary). Nonetheless, each device (assuming every device uses its own, separate 802.1x credentials) is assigned to a dynamic VLAN interface – which in turn is automatically added to the single bridge device – and can therefore easily be isolated to trace all of its network packets on that VLAN device.

Make sure that the DHCP server hands out IP addresses on this bridge network (which should have been set up by default by enabling the Wi-Fi guest option) in /etc/config/dhcp:

config dhcp 'guest_turris'
	option interface 'guest_turris'
    option ignore '0'
    option start '100'
    option limit '150'
    option leasetime '3600'
    list dhcp_option '6,10.111.222.1'

Configure Freeradius for PEAP-MSCHAPv2 support

a. In /etc/freeradius3/sites-available/default, most of the defaults already work. I just disabled the modules chap, digest, and suffix in block authorize because we don’t use them and didn’t install the respective modules.

NOTE: The important change from the default config for VLAN-based device separation with e.g. Android clients (using PEAP-MSCHAPv2 authentication) is required in /etc/freeradius3/mods-enabled/eap: in the inner block eap { ... peap { <IN HERE> } ... } change to the option use_tunneled_reply = yes. According to comments in this version of freeradius3, this is actually deprecated, but still works at the time of this writing. If you forget to turn this on, you will get an error IEEE 802.1X: authentication server did not include required VLAN ID in Access-Accept in the system log when clients actually try to authenticate to a WiFi interface configured with enforced VLAN tagging.

If you want to keep traffic counters, then enable the sql module in block accounting: This and this howtos are good references for configuring Freeradius 3 in itself and this one for setting up SQLite as a backend.

b. (Optional) Add a test client machine to execute radtest from in /etc/freeradius3/clients.conf:
```
client test-client {
	ipaddr		= 192.168.1.149
	secret		= testing123
}
```
c. (Optional) Add a test user in /etc/freeradius3/mods-config/files/authorize:
```
pixel3a		Cleartext-Password := "andseclab"
	Tunnel-Type = "VLAN",
	Tunnel-Medium-Type = "IEEE-802",
	Tunnel-Private-Group-ID = "101"
```

(optional) To export netflows from this guest network to a collector

a. opkg install softflowd

b. Modify /etc/config/softflowd to look like this:

 config softflowd
     option enabled        '1'
     option interface      'br-guest_turris'
     option pcap_file      ''
     option timeout        'maxlife=600'
     option max_flows      '8192'
     option host_port      '<ip address:port> of your netflow collector'
     option pid_file       '/var/run/softflowd.pid'
     option control_socket '/var/run/softflowd.ctl'
     option export_version '5'
     option hoplimit       ''
     option tracking_level 'full'
     option track_ipv6     '1'
     option sampling_rate  '1'

(optional) Instead of just exporting flows, full analysis of packets (e.g. using Arkime) can also be done by creating s virtual switch mirror/tap port. While there may be multiple ways to do that (including OpenVSwitch instead of tc filters or other tunnel types, or using specific user-space tap/mirror software like fluxcap), the solution that turned out to be the first to actually work is the following.

Note: As GRETAP for a still unknown reason didn’t work (any packets sent into the gretap1 device simply vanished but were not encapsulated and sent out over the physical link), I instead set up an L2TP-Ethernet-over-UDP encapsulation tunnel (this has some overhead compared to GRETAP, but is still fairly fast due to its in-kernel support, at least compared to going through userspace as e.g. OpenVPN tunneling would):

a. Set up the l2tp-eth tunnel:
```
opkg install kmod-l2tp-eth
ip l2tp add tunnel tunnel_id 1 peer_tunnel_id 1 udp_sport 5000 udp_dport 5000 encap udp local 140.78.100.105 remote 140.78.100.108
ip l2tp add session tunnel_id 1 session_id 1 peer_session_id 1
ip link set l2tpeth0 up mtu 1428
```
The first line is only necessary once, the other 3 on every reboot - I just put them into /etc/rc.local (chmod +x to make it executable) as netifd doesn’t currently seem to properly support statically configured L2TP tunnels through /etc/config/network. If that changes in the future, it would be much cleaner.

b. Mirror traffic from the virtual VLAN tagged WiFi interfaces into that tunnel. The specific setup described here was heavily inspired by this post, and I learnt about the VLAN action from this paper:

opkg install kmod-sched-act-vlan to install the VLAN action module, and then set up mirroring for each VLAN individually to separate them:
```
for vlan in `seq 100 160`; do 
    tc qdisc add dev wlan1.$vlan handle ffff: ingress
    tc filter add dev wlan1.$vlan parent ffff: protocol all u32 match u32 0 0 action vlan push id $vlan pipe action mirred egress mirror dev l2tpeth0 pipe action vlan pop
    tc qdisc add dev wlan1.$vlan handle 1: root htb
    tc filter add dev wlan1.$vlan parent 1: protocol all u32 match u32 0 0 action vlan push id $vlan pipe action mirred egress mirror dev l2tpeth0 pipe action vlan pop
done
```
This cascades 3 actions for each packet in- and outgoing on each of the specific wlan interfaces: add the respective VLAN tag, then mirror (copy) it to the virtual tunnel interface, and remove the VLAN tag again to allow local processing of the packet (i.e. forwarding through NAT to the external Internet). This is not as nice as bridging a single interface that has those tags already, but it is the only method that I found working right now (and it took me over a day to get there).

As the virtual devices are created dynamically when a client connects, hotplug scripts can be used to set up this mirroring upon the device appearing, e.g. /etc/hotplug.d/iface/30-local-mirror-traffic:
```
#!/bin/sh

if [ "$ACTION" = ifup ] && [ `echo "$DEVICE" | cut -d. -f1` = "wlan1" ]; then
    vlan=`echo $DEVICE | cut -d. -f2`
    logger -t adsd-mirror "Starting (hotplug) mirroring traffic from $DEVICE (vlan id $vlan)..."
    tc qdisc add dev $DEVICE handle ffff: ingress
    tc filter add dev $DEVICE parent ffff: protocol all u32 match u32 0 0 action vlan push id $vlan pipe action mirred egress mirror dev l2tpeth0 pipe action vlan pop
    tc qdisc add dev $DEVICE handle 1: root htb
    tc filter add dev $DEVICE parent 1: protocol all u32 match u32 0 0 action vlan push id $vlan pipe action mirred egress mirror dev l2tpeth0 pipe action vlan pop
fi
```
Note: On the current TurrisOS 5.1.4, the hotplug script doesn’t execute when hostapd activates the new network interface when a client connects, and I don’t yet know why. Until this is clear, I just trigger this whenever a new DHCP address is assigned for all currently existing interfaces with /etc/hotplug.d/dhcp/50-local-mirror-traffic:
```
#!/bin/sh
logger -t adsd-mirror "New DHCP assignment: $ACTION $IPADDR $MACADDR"
ip link | egrep "wlan1\..*: <BROADCAST,MULTICAST,UP,LOWER_UP>" | cut -d: -f2 | while read dev; do ACTION=ifup DEVICE=$dev /etc/hotplug.d/iface/30-local-mirror-traffic; done
```
c. To receive that traffic on another (e.g. virtual) machine, create the corresponding L2TP-Ethernet interface. On Debian, the easiest (and clean) way to do that is through /etc/network/interfaces:
```
iface l2tpeth0 inet manual
      pre-up ip l2tp add tunnel tunnel_id 1 peer_tunnel_id 1 udp_sport 5000 udp_dport 5000 encap udp local 140.78.100.108 remote 140.78.100.105
      pre-up ip l2tp add session tunnel_id 1 session_id 1 peer_session_id 1
      down ip l2tp del tunnel tunnel_id 1
```

Testing / Debugging

If WLAN clients seem to be able to connect, but do not receive an IP address through DHCP or can’t communicate otherwise, verify the correct VLAN-interface-to-bridge assignment with brctl show. An example output should like
```
br-guest_turris		7fff.04f021232181	no		lan4
						wlan1.101
```
If freeradius doesn’t start, radiusd -f -X -C should give a reason why.