Things I learned about VMs with two interfaces on OpenStack
Sometimes you want to attach two interfaces to a OpenStack virtual machine (VM). As discussed in my last post, I had to disable OpenStacks port security feature to allow my IPVS realservers to implement direct server return. Disabling port-security has the downside, that my realserver can participate in other OpenStack security groups. For my use case, my realservers have to be part of a security group to reach certain backends. To implement that, I added a second network interface to my VM. It turned out, it was not easy as expected. This picture shows the setup:
The underlying OpenStack network implemented with Linux bridges is very picky about which traffic can be sent over
which interface. By default, Linux is not configured for this setup. You
have to change certain defaults to make this work. To sum the linked article up, Linux will respond by default to an
ARP request received on any interface requesting an IP assigned to any local interface. Thus, in the example above,
an ARP request received on
ens3 requesting the IP
192.0.2.4 will be answered. This messes up the OpenStack network,
since it contradicts the security model and assumes the response is spoofing. OpenStack compares ARP responses with
the configured IP addresses in its network model. The second problem is, that for any packets originating on my
realserver, only the interface
ens3 is used. Even if the source IP is
192.0.2.4. This is due to the fact, that
by default the first matching route is used. We use policy-based routing to fix this problem.
For ARP, we use the same settings as in my last post
As explained in my last post, the
arp_ignore=1 sysctl settings will
instruct the linux kernel only to answer ARP requests for IPs attached to the receiving interface. Thus, in our
example, the kernel will only respond to ARP requests received via
ens3 requesting the IP
requests for the IP
192.0.2.4 are ignored on the interface
ens3 after this setting is set.
The next problem we have to solve, is that the Linux kernel will send traffic originating from the IP
ens3 interface. This traffic will be dropped by OpenStack, since it assumes this is spoofed traffic.
Furthermore, we want to use the security groups
ens4. To make sure, traffic originating from
192.0.2.4 is forwarded via the
ens4 interface, we
must use policy-based routing.
First, we create two new routing tables:
You can choose what ever name you want. I used
table_ensX for clarity. You must however, choose a different ID for
both tables. I have chosen the ID 200 and 201.
Next, we add the default routes to the two routing tables:
ip route add table table_ensX instructs the
ip route add command, to add the route to the routing table
If you have other routing requirements, e.g. allowing use
ens4 only for traffic within the subnet
you can do this:
The last step is to configure the policy-based routing. Therefore, we use the
ip rule command:
To show the rules/policies in effect, you can use the command
ip rule show. The output should look like this:
The first number is the priority. The rules are check from the lowest priority number (zero) to the highest (here 32767). The first matching rule is used.
To test your routing setup, the command
ip route get is very handy. This let you check the where a packet would be
Doing this should give you an output like:
As you can see,
ip route get shows you the device
dev ens3 and the used table to retrieve that information. The
downside of this set up is, you have to manage the routes in the tables
table_ens4 by yourself.
Thus, you should to this only if your routing and the IPs are pretty static.
To sum it up, we have learned how to configure Linux to run with two interfaces in the same subnet. First, we have to reconfigure the default ARP answering behaviour. Secondly, we have to use policy-based routing to steer the egress traffic to the proper interfaces.