Debian Configuration :: Multiple Gateways / Firewalls Pointed To One Interface?

I am having some troubles using iptable rules on two Servers that act as Gateways pointed to one backend server with only one interface.

To be more exact, i have 3 Servers, 2 of those have a public and a private interface, with different public ips but common private interface ( they connect to the same switch ), the last one only has 1 private interface and is connected to that same switch.

Those 2 servers also act as a gateway and a firewall for the private network.

My problem is that i cannot seem able to route traffic from both of those to the third one and back to the same public ip that the request came from ( effectivly using two gateways on the machine with only one interface ).

As a testing scenario i am using ferm for applying iptable rules that forward ssh traffic ( for example ) to the backend server, and it works well when i do it with one gateway.

When i apply something like this in /etc/network/interfaces on the backend server though:

Code: Select allauto eth0:0 eth0:1
# The primary network interface
allow-hotplug eth0
iface eth0:1 inet static
   address   192.168.9X.XXX
   netmask   255.255.255.0
   broadcast 192.168.9X.255
   network   192.168.9X.0

[Code] ....

Even though forcing selection of an interface from the backend server ( like curl --interface ) seems to work well, meaning that the request to the curl appear to happen from the correct public ip, i can still only use one of the public ips to access the server with the ferm rules. Ideally i should be able to ssh to the backend server from both public ips using their ferm rules for forwarding traffic to the backend server.

I feel like i am missing some details on routing that should happen on the firewalls as the backend server seems to be able to use both gateways to access the internet and receive replies from it.