A router can have multiple outbound connections, in which case it may be useful to redirect the traffic out though multiple interfaces. This can be done efficiently on Linux by marking packets and routing them though different interfaces. Similarly, we can also route inbound connections to servers behind the router based on the same rules.

Given the following configuration:

• eth0 is the LAN interface.
• tap0 is an upstream ISP giving us the IP address 95.173.136.70 and gateway 23.206.132.1
• tap1 is an upstream ISP giving us the IP address 23.206.132.14 and gateway 95.173.136.1

First, we set-up two different routes for tap0, respectively tap1 by editing /etc/iproute/rt_tables and adding the lines:

500     tap0
501     tap1

Note that the number in the first column identifies the mark identifier and that tap0 and tap1 are just descriptive names - however, we call them tap0 and tap1 to conveniently distinguish between the two interfaces for which we mark packets.

The next step is to set-up the rules and route with iproute2. For the ISP on tap0 we enter via a script:

#!/bin/sh
 
ip route add default via 23.206.132.1 dev tap0 table tap0
ip rule add fwmark 500 lookup tap0
ip route flush cache

and for the ISP on tap1 we enter via a script:

#!/bin/sh
 
ip route add default via 95.173.136.1 dev tap0 table tap1
ip rule add fwmark 501 lookup tap1
ip route flush cache

These settings set the default gateway for both providers and creates the firewall mark 500 for tap0 and 501 for tap1 - which are defined in /etc/iproute2/rt_tables.

Now it is time to configure the firewall that will mark packets and route them out through the interfaces:

#!/bin/sh
 
# create input chains for both interfaces,
# mark packets through the interfaces,
# and save the connection mark
iptables -t nat -N IN_TAP0
iptables -t nat -A IN_TAP0 -j MARK --set-mark 500
iptables -t nat -A IN_TAP0 -j CONNMARK --save-mark
iptables -t nat -N IN_TAP1
iptables -t nat -A IN_TAP1 -j MARK --set-mark 501
iptables -t nat -A IN_TAP1 -j CONNMARK --save-mark
# from tap0 jump to the IN_TAP0 chain
iptables -t nat -A PREROUTING -i tap0 -j IN_TAP0
# from tap1 jump to the IN_TAP1 chain
iptables -t nat -A PREROUTING -i tap1 -j IN_TAP1
 
# create output chains for the interfaces,
# otherwise the IP address for each interface,
# will not be set properly when talking to servers.
iptables -t nat -N OUT_TAP0
iptables -t nat -A OUT_TAP0 -j SNAT --to-source 23.206.132.1
iptables -t nat -N OUT_TAP1
iptables -t nat -A OUT_TAP1 -j SNAT --to-source 95.173.136.1
# for tap0 jump to the OUT_TAP0 chain
iptables -t nat -A POSTROUTING -o tap0 -j OUT_TAP0
# for tap1 jump to the OUT_TAP1 chain
iptables -t nat -A POSTROUTING -o tap1 -j OUT_TAP1
 
# create mark chains for the interfaces
iptables -t mangle -N MARK_TAP0
iptables -t mangle -A MARK_TAP0 -j MARK --set-mark 500
iptables -t mangle -A MARK_TAP0 -j CONNMARK --save-mark
iptables -t mangle -N MARK_TAP1
iptables -t mangle -A MARK_TAP1 -j MARK --set-mark 501
iptables -t mangle -A MARK_TAP1 -j CONNMARK --save-mark
 
# Restore connection-marked packets for TCP and UDP
iptables -t mangle -A PREROUTING -i eth0 \
    -p tcp -m state --state ESTABLISHED,RELATED \
    -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -i eth0 \
    -p udp \
    -j CONNMARK --restore-mark

Now that we have a frameworks for routing and marking packets, we can conditionally load-balance outbound packets. For instance, let's load-balance outgoing FTP traffic on both interfaces:

#!/bin/sh
 
# statistically mark the first packet of every two new outbound FTP 
# connections and jump to the MARK_TAP0 chain which will route the 
# connection out through the tap0 interface.
iptables -t mangle -A PREROUTING -i eth0 \
        -p tcp -m multiport --dport 20,21  \
        -m state --state NEW \
        -m statistic --mode nth --every 2 --packet 0 \
        -j MARK_TAP0
 
# statistically mark the second packet of every two new outbound FTP 
# connections and jump to the MARK_TAP1 chain which will route the 
# connection out through the tap1 interface.
iptables -t mangle -A PREROUTING -i eth0 \
        -p tcp -m multiport --dport 20,21  \
        -m state --state NEW \
        -m statistic --mode nth --every 2 --packet 1 \
        -j MARK_TAP1

Note that we use -m state –state NEW in order to load-balance only new connections and that ensuing traffic (ESTABLISHED,RELATED) will be taken care of by the -j CONNMARK –restore-mark rule in our firewall.

Suppose that amongst clients we also have some servers that offer different services such as e-mail, web-services, etc… We can now use the setup that we created in order to transparently NAT those services and offer them externally over our outbound interfaces.

Suppose that we now have a mail-server behind the router with the following configuration:

• Internal server with IP address 192.168.1.7
  • Listens on ports 25, 465, 587 for incoming e-mails.
  • Connects to ports 25, 465, 587 for outgoing e-mails.

and that we want to expose this mail-server to the internet over the tap0 interface.

The rules are then as follows:

#!/bin/sh
 
# forward incoming ports 25, 465, 587 through MARK_TAP0 chain
iptables -t mangle -A PREROUTING -s 192.168.1.7 \
        -p tcp -m multiport --sport 25,465,587 \
        -j MARK_TAP0
# forward outgoing ports 25, 465, 587 through MARK_TAP0 chain
iptables -t mangle -A PREROUTING -s 192.168.1.7 \
        -p tcp -m multiport --dport 25,465,587 \
        -j MARK_TAP0
# NAT incoming ports 25, 465, 587 on tap0 to mail-server at 192.168.1.7
iptables -t nat -A PREROUTING -i tap0 \
        -p tcp -m multiport --dport 25,465,587 \
        -j DNAT --to 192.168.1.7

Now the mail-server behind the router is exposed on ports 25, 465, 587.

Now, suppose that we have an FTP server behind the router with the following configuration:

• Internal FTP server with IP address 192.168.1.7
  • Listens on ports 20, 21 for incoming e-mails.
  • Connects to passive ports 1024 to 1026 (must be at least two consecutive ports).

proFTP is the FTP server of choice and has the following directives configured in /etc/proftpd/proftpd.conf:

# Select at least two consecutive ports
PassivePorts       1024 1026

# Set to the external IP of tap0 interface
MasqueradeAddress  23.206.132.14

and that we want to expose this FTP to the internet over the tap0 interface.

In that case, we need to load the FTP modules:

nf_nat_ftp
nf_conntrack_ftp

and the server iptables rules will read as follows:

# Make sure that established connections from port 20 to 21 go out through tap0
# Note: there is no need for NEW here since the connection should have been 
# established by the connecting client.
iptables -t mangle -A PREROUTING -s 192.168.1.7 \
        -p tcp --sport 20:21 \
        -m conntrack --ctstate ESTABLISHED \
        -j MARK_TAP0
# The same as the former for all the configured passive ports in proFTPd
iptables -t mangle -A PREROUTING -s 192.168.1.7 \
        -p tcp --sport 1024:1026 \
        -m conntrack --ctstate ESTABLISHED,RELATED \
        -j MARK_TAP0
 
# Nat all incoming connections to ftp ports 20 to 21 to the proFTPd server.
iptables -t nat -A PREROUTING -i tap0 -p tcp --dport 20:21 \
        -m conntrack --ctstate NEW \
        -j DNAT --to 192.168.1.7
# Nat all incoming connections to passive ftp ports 1024 to 1026 to the proFTPd server.
iptables -t nat -A PREROUTING -i tap0 -p tcp --dport 1024:1026 \
        -m conntrack --ctstate NEW \
        -j DNAT --to 192.168.1.7

Now, with this in-place, both passive and active FTP is configured correctly on the server. However, the client itself must be configured to allow incoming ports for active FTP. The easiest solution, in case you are also running Linux on the client is to load the same modules:

nf_nat_ftp
nf_conntrack_ftp

which will do all the work for you - otherwise, rules will be required in INPUT and OUTPUT to allow connections.

With the rules established, we can now conditionally route all TCP and UDP packets out of multiple interfaces. While TCP is a statefull, UDP is stateless. That means that we can indeed load-balance outbound TCP connections but we have to be careful with UDP and pick an interface to send all packets through.

In other words, with the rules in effect, TCP connections will alternate between tap0 and tap1 whilst UDP connections will be processed only through tap1.

The rules will be added at the end such that they will apply only if the other rules previous to them (for example, FTP, etc…) did not match.

#!/bin/sh
 
# All load-balancing
# TCP, statefull -> load balance between tap0 and tap1
iptables -t mangle -A PREROUTING -i eth0 \
    -p tcp \
    -m state --state NEW \
    -m statistic --mode nth --every 2 --packet 0 \
    -j MARK_TAP0
iptables -t mangle -A PREROUTING -i eth0 \
    -p tcp \
    -m state --state NEW \
    -m statistic --mode nth --every 2 --packet 1 \
    -j MARK_TAP1
# UDP, stateless -> use either tap0 or tap1
iptables -t mangle -A PREROUTING -i eth0 \
    -p udp \
    -j MARK_TAP1 # or MARK_TAP0

Although this scheme should work perfectly for most applications, there is absolutely nothing stopping software from tracking connections: for instance, an application could assume that some IP that connected on some TCP port will be the same IP that will connect to an UDP port. In that situation, the application's needs must be addressed in the rule - for example by matching the connecting ports.

In cases where UPnP is necessary, a systemd unit file can be created in order to start linux-igd on all WAN interfaces thereby providing port mappings regardless of outbound packet balancing.