Linux Policy Base Routing

What happened in the first place?

When building EC2 on AWS, packets may not reach EC2 with multiple NICs in the following situations, so this is a memorandum. As a configuration, the EC2 instance has multiple NICs, each belonging to a different subnet.

(Quoted from the figure) http://blog.serverworks.co.jp/tech/2018/02/16/rhel-pbr-setting/

The routing table (main) has the following situation. The default GW is a router on the subnet (10.0.1.0/24).

[root@ip-10-0-1-5 etc]# ip route show table main
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.5 
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.5 
default via 10.0.1.1 dev eth0

In this situation, I tried to ping the 10.0.0.2.x NIC (eth1) from the NW belonging to 10.0.1.0/24, but there is no return packet. As a result of investigating why, PING has arrived at the EC2, but when returning the return packet, it is sending a packet from the default GW of the EC2, and as a result it does not reach the source. It was a thing.

For the same communication, communication in which the receiving NIC and the returning NIC are different is called asymmetric routing. In Linux, from some version, the following kernel parameters are set to drop packets if the NIC that communicates back is different from the received NIC. This setting is enabled at the packet destination, and only the return communication is filtered.

`Return route filter`


net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

There is a way to disable the above, but it is a security setting for the entire OS to prevent IP spoofing. In order to avoid this, there is also a function called Policy Base Routing, so implement it.

Policy Base Routing

The standard routing table routes packets to a specific router or default gateway by looking at the destination address. On the other hand, Policy Base Routing (PBR) is a function that determines the routing by looking at the source IP address and TCP / UDP port. Also, prioritize PBR as a rule over normal routing.

Therefore, in this case, it can be concluded that the PBR should be set so that packets with a source IP address of 10.0.2.0/24 as a reference will be skipped to 10.0.2.1. (Although it is the source IP address mentioned here, it refers to the part of the return packet of PING. As you can see by tcpdump, the return packet is 10.0.2.x ==> 10.0.1.x.)

--Since this is the current rule, set the priority to a smaller value than "32766: from all lookup main". The lower the priority, the higher the priority rule

[root@ip-10-0-1-5 etc]# ip rule show
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default

--Create route table rules for PBR

`temporary`


ip route flush table 2000
ip route add table 2000 to 10.0.2.0/24 dev eth1
ip route add table 2000 to default via 10.0.2.1 dev eth1
ip rule add from 10.0.2.0/24 table 2000 priority 200

`Permanent`


cat << EOF > /etc/sysconfig/network-scripts/pbr02-eth1
from 10.0.2.0/24 table 2000 priority 200
EOF

cat << EOF > /etc/sysconfig/network-scripts/route-eth1
10.0.2.0/24 dev eth1 table pbr02
default via 10.0.2.1 dev eth1 table 2000
EOF

--Setting confirmation system

[root@ip-10-0-1-5 network-scripts]# ip rule show
0:	from all lookup local 
200:	from 10.0.2.0/24 lookup pbr02 
32766:	from all lookup main 
32767:	from all lookup default 

[root@ip-10-0-1-5 network-scripts]# ip route show table 2000
10.0.2.0/24 dev eth1  scope link 
default via 10.0.2.1 dev eth1

Reference https://milestone-of-se.nesuke.com/nw-basic/routing/policy-based-routing/