Linux GRE Tunnel to Cisco IOS with NAT

This is an example of a GRE tunnel between an Ubuntu 20 server and Cisco IOS 15 ISR 800 series router. Traffic from the remote LAN attached to the Cisco router is routed over the GRE tunnel to the remote server and NAT is used on the Ubuntu server, to allow the clients behind the Cisco router to access the public Internet with the servers IPv4 and IPv6 address.

 

GRE Linux to Cisco IOS

Load the GRE kernel modules. When ip_gre is loaded an interface called gre0 is created, this can not be deleted or modified in any way, it is like a kind of base/reference interface. It can be ignored. A new interface (tun0 in the example below) will be created.

root@remote:~# lsmod | grep gre
root@remote:~# modprobe gre
root@remote:~# modprobe ip_gre
root@remote:~# lsmod | grep gre
ip_gre                 28672  0
ip_tunnel              24576  1 ip_gre
gre                    16384  1 ip_gre

 

The Cisco router's public IP address 192.0.2.1 and the server public IPv4 address is 192.0.2.2, these are the GRE underlay. 192.168.254.0/29 and FD:0:0:FF::/64 are the subnets used over the GRE tunnel. Note that the Cisco router runs PPPoE on it's WAN router meaning that the WAN MTU is reduced by 8 bytes.

root@remote:~# ip tunnel add tun0 mode gre remote 192.0.2.1 local 192.0.2.2 ttl 255
root@remote:~# ip link set tun0 up
root@remote:~# ip addr add 192.168.254.2/29 dev tun0
root@remote:~# ip link set mtu 1468 dev tun0
root@remote:~# ip route change 192.168.254.0/29 dev tun0 advmss 1428
root@remote:~# ip r a 10.0.0.0/24 dev tun0 advmss 1428        # This is the remote LAN v4 subnet
root@remote:~# ip -6 addr add fd::ff:0:0:0:2/64 dev tun0
root@remote:~# ip -6 r add fd:0:0:3::/64 dev tun0 advmss 1408 # This is the remote LAN v6 subnet

Above the MTU is 1468. This is to match the low MTU on the remote Cisco router. The Cisco router has a 1500 byte WAN link MTU but it runs PPPoE which uses 8 bytes of the MTU. The GRE tunnel to the Ubuntu server is 20 bytes of IPv4 plus 4 bytes of GRE (RFCs state various sites from 8, to 16, to 20, to 32 bytes, but 4 is the minimum that can be used and since no features are used here such as sequencing, meaning the minimum is used). 1500 - (8+20+4) == 1468. This means that the IPv4 TCP MSS must be another 40 bytes lower, 20 bytes for IPv4 and 20 bytes for TCP.

Above the IPv6 MTU isn't set, nor the IPv6 TCP MSS, for the tun0 interface. The IPv6 MTU is assumed by Linux to be the same as the IPv4 MTU, which is true in this case. The IPv6 TCP MSS is set for the remote LAN subnet though, and one can check that the IPv4 MTU has been applied to this interface with the following.

root@remote:~# cat /proc/sys/net/ipv6/conf/tun0/mtu 
1468

Below it can be seen that the tunnel and routes are configured:

root@remote:~# ip -4 r
10.0.0.0/24 dev tun0 scope link advmss 1428 
192.168.254.0/29 dev tun0 scope link advmss 1428 

root@remote:~# ip -6 r
fd:0:0:3::/64 dev tun0 metric 1024 advmss 1408 pref medium
fd:0:0:ff::/64 dev tun0 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium

root@remote:~# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 11:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1464 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: tun0@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1468 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/gre 192.0.2.2 peer 192.0.2.1

root@remote:~# ip tun
gre0: gre/ip remote any local any ttl inherit nopmtudisc
tun0: gre/ip remote 192.0.2.1 local 192.0.2.2 ttl 255

 

Below is the config from the Cisco IOS router:

interface Vlan1
 description LAN
 ip address 10.0.0.1 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip virtual-reassembly in max-reassemblies 32
 ip verify unicast source reachable-via rx
 ip tcp adjust-mss 1428
 ip policy route-map clear-df-bit ! In case a LAN client doesn't understand or use PMTU, need to clear DF bit
 load-interval 30
 ipv6 address FD:0:0:3::1/64
 ipv6 enable
 ipv6 nd other-config-flag
 ipv6 nd router-preference High
 ipv6 nd ra dns server FD:0:0:3::1 4294967295
 no ipv6 redirects
 no ipv6 unreachables
 ipv6 tcp adjust-mss 1408
 ipv6 verify unicast source reachable-via rx
 ipv6 dhcp server LAN rapid-commit
 no ipv6 mld router
 ntp disable
!
interface Tunnel0
 description GRE
 ip address 192.168.254.1 255.255.255.248
 no ip redirects
 no ip proxy-arp
 ip mtu 1468
 ip tcp adjust-mss 1428
 ip policy route-map clear-df-bit
 load-interval 30
 ipv6 address FD:0:0:FF::1/64
 ipv6 enable
 ipv6 mtu 1468
 ipv6 nd ra suppress all
 no ipv6 redirects
 no ipv6 unreachables
 ipv6 tcp adjust-mss 1408
 no ipv6 mld router
 ntp disable
 tunnel source Dialer1  ! PPPoE WAN interface
 tunnel destination 192.0.2.2
!
interface Dialer1
 description WAN
 mtu 1492
 ip address 192.0.2.1 255.255.255.255
 encapsulation ppp
 ip tcp adjust-mss 1452
!

 

Dual-Stack NAT44/NAT66

IPv4 and Ipv6 forwarding needs to be enabled on the Ubuntu server:

root@remote:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@remote:~# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
root@remote:~# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
root@remote:~# echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf

Load iptables:

root@remote:~# modprobe iptable_filter
root@remote:~# modprobe iptable_nat
root@remote:~# modprobe ip6table_filter
root@remote:~# modprobe ip6table_nat
root@remote:~# echo "iptable_filter" >> /etc/modules
root@remote:~# echo "iptable_nat" >> /etc/modules
root@remote:~# echo "ip6table_filter" >> /etc/modules
root@remote:~# echo "ip6table_nat" >> /etc/modules

Add the iptables forwarding and NAT rules:

root@remote:~# iptables-save
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p gre -s 192.0.2.1/32 -m comment --comment "Allow GRE from Cisco" -j ACCEPT
-A INPUT -j DROP

-A FORWARD -d 10.0.0.0/24 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.0.0.0/24 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -j DROP
COMMIT


root@remote:~# ip6tables-save
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -j DROP
-A FORWARD -d fd:0:0:3::/64 -j ACCEPT
-A FORWARD -s fd:0:0:3::/64 -j ACCEPT
-A FORWARD -j DROP
COMMIT

 

Ping Tests

IPv4 ping from LAN to Ubuntu GRE tunnel interface IP:

bensley@ubuntu-laptop [2021-08-30 09:43:17] (~)
$ping 192.168.254.2 -M do -s 1440
PING 192.168.254.2 (192.168.254.2) 1440(1468) bytes of data.
1448 bytes from 192.168.254.2: icmp_seq=1 ttl=63 time=36.1 ms
1448 bytes from 192.168.254.2: icmp_seq=2 ttl=63 time=61.7 ms
^C
--- 192.168.254.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 36.103/48.892/61.682/12.789 ms

8 PPPoE, 20 IPv4, 4 GRE, 20 IPv4, 8 ICMP, 1440 ICMP Payload = 8+20+4+20+8+1440 == 1500
IPv4 MTU = 1500 - (8+20+4) = 1468
IPv4 TCP MSS = 1500 - (8+20+4+20+20) == 1428

IPv6 ping from LAN to Ubuntu GRE tunnel interface IP

bensley@ubuntu-laptop [2021-08-30 10:12:39] (~)
$ping6 fd:0:0:ff::2 -M do -s 1420
PING fd:0:0:ff::2(fd:0:0:ff::2) 1420 data bytes
1428 bytes from fd:0:0:ff::2: icmp_seq=1 ttl=63 time=34.9 ms
1428 bytes from fd:0:0:ff::2: icmp_seq=2 ttl=63 time=37.2 ms
1428 bytes from fd:0:0:ff::2: icmp_seq=3 ttl=63 time=64.2 ms
^C
--- fd:0:0:ff::2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 34.938/45.450/64.210/13.297 ms

8 PPPoE, 20 IPv4, 4 GRE, 40 IPv6, 8 ICMPv6, 1420 ICMPv6 Payload = 8+20+4+40+8+1420 == 1500
IPv6 MTU = 1500 - (8+20+4) = 1468
IPv6 TCP MSS = 1500 - (8+20+4+40+20) == 1408

IPv4 curl to Internet to check NAT44:

bensley@ubuntu-laptop [2021-08-30 12:10:40] (~)
$curl ipv4.53bits.co.uk
192.0.2.2

IPv6 curl to Internet to check NAT66:

bensley@ubuntu-laptop [2021-08-30 12:11:11] (~)
$curl ipv6.53bits.co.uk
2003::192:0:2:2

 

---