I'm not for sure when they did it, but the RHEL folks made it a bunch easier to setup simple source policy routing. By using source policy routing, we fix the issue of firewalls freaking out when the reply packet to a host leaves a multihomed host on a different interface than what the request came in on. In prior versions, you had to setup some custom scripts, but that's no longer the case - all the hooks are there in the OS now.

In this example, imagine a CentOS host with two nics. 192.168.0.2/24 is on eth0, and 10.0.0.2/24 is on eth1. The default gateway is set to 192.168.0.1. Any host accessing 10.0.0.2 from any subnet that isn't on 10.0.0.0/24 will have it's reply packets sent out via 192.168.0.1. Some firewalls drop this type of traffic *cough* Cisco ASA's *cough*.

Thanks to the iproute2 package in Linux, this is easy enough to fix. RedHat has made it even easier now - we can do this in 3 steps (all performed as root):

Step 1: Create a table

We need to create a table for iproute2. Name it anything you want, and add it to /etc/iproute2/rt_tables, like so:

echo -e "200\tSecondSubnet" >> /etc/iproute2/rt_tables

Step 2: Create a route

We need to create a route for eth1 that says to use our CorpNet table defined in Step 1.

echo "default table SecondSubnet via 10.0.0.1" > /etc/sysconfig/network-scripts/route-eth1

Step 3: Create a rule

We need to create a rule for eth1 that says to use our route above for traffic received on eth1.

echo "from 10.0.0.2 table SecondSubnet" > /etc/sysconfig/network-scripts/rule-eth1

Step 4: Restart networking

/etc/init.d/network restart

That's it. Fire up a packet sniffer and verify your config!