Subject: [v6ops] Source address selection

From: Sander Steffann <[email-masked]>
Date: Mon, 11 Feb 2013 18:23:49 +0100

Hi,

This is a long message. It comes down to "is there a good way to make a box always use a specific default source address?"

The longer version is below to give you some more background information :-)

I am talking to a customer who has the following situation:

They have multiple dual stack networks. I'm only looking at the IPv6 part here. The servers (Linux boxes) all have multiple IPv6 addresses. Every box has its own address (iron address) and each service has a separate address (service address) so that it can be moved to a different box if necessary (unexpected high number of visitors etc). For some of the addresses load balancers are used to distribute the load.

All of the load balancing is done using Direct-Routing (one box responds to NDP for that address and then forwards the packet to the right back-end server by leaving the layer-3 addresses in place and using the layer-2 address of the chosen back-end server, the back-end server has the address on its loopback interface, accepts the packet and sends the reply directly to the original layer-3 source address) with VRRP between load balancers for redundancy.

Now the incoming requests are no problem at all. The problem is with the source address selection when the boxes initiate a connection. We want outgoing connections to always use the iron address as default source address. Using one of the service addresses is undesirable for several reasons:
- If the address is load balanced then initiating outbound connections from that address will break as return traffic might be load balanced to one of the other boxes
- If the application running service A creates an outbound connection using service B's address, then moving service B to another box will break application A's established connections
- Determining which machine sent which packet is harder
- Firewall policies are much harder to write if connections are initiated from an unpredictable source address

With IPv6 this is much harder to accomplish than I ever expected. Putting the addresses in a configuration file is not an option because the services can be migrated to another box (= another iron address).

When looking at the source address selection mechanism in RFC 6724 I see a few possibilities to adjust the algorithm:

Rule 1: Prefer same address
- N/A here: Localhost traffic isn't important here

Rule 2: Prefer appropriate scope
- All addresses involved have global scope

Rule 3: Avoid deprecated addresses
- Making all service addresses deprecated prevents them from being used as a source address by default, so this accomplishes what we want, but it feels like a hack...

Rule 4: Prefer home addresses.
- N/A here: Mobile IPv6 not in use

Rule 5: Prefer outgoing interface.
- Lots of addresses on the same interface (does work for addresses on the loopback interface)

Rule 5.5: Prefer addresses in a prefix advertised by the next-hop.
- N/A: Possible source addresses are in the same prefix

Rule 6: Prefer matching label.
- Giving all service addresses a different label from the iron addresses seemed a good choice. We considered renumbering all services to use addresses in a specific part of the /64 and then configure Linux to use a separate label for that prefix (i.e. `ip addrlabel add prefix 2001:db8:123:abc:1000::/68 label 1000`). This works for connections to the outside world, but if one service connects to another service then it still would choose a service address as source address. Giving every service address its own label would solve that, but with the potentially hundreds of addresses on each box (yes, this is a big server farm) this becomes unmanageable.

Rule 7: Prefer temporary addresses.
- N/A: temporary addresses are disabled on the boxes

Rule 8: Use longest matching prefix.
- When connecting to external services all addresses on the LAN have the same match-length (all shorter than a /64). When connecting to other services on the same LAN it becomes unpredictable again (depends on the other service addresses on the same box)

Based on this it seems that making all service addresses deprecated immediately and leaving the iron address as the only preferred address would be the only way to solve this. It does feel like a hack though. RFC 4862 says that "an address becomes "deprecated" in anticipation that its current interface binding will become invalid". These addresses are not anticipated to become invalid for a very long time (= indefinite). I can already imagine a kernel deciding to free up some resources by dropping addresses that are marked as deprecated for example.

So: does anybody have any advice on this? Is it safe/future-proof/etc to mark addresses as deprecated to avoid them being used as default source addresses? Do we need some other way to steer the default source address selection procedure? Or is the only 'official' way to give each and every /128 its own label (so that rule 6 becomes equal to rule 1 for service addresses)?

Cheers,
Sander

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 13 Feb 2013 10:50:24 -0800

> - If the address is load balanced then initiating outbound connections from that address will break as return traffic might be load balanced to one of the other boxes

I'm surprised that you want to use the common iron address for outbound 
connections. Where I've though about this it seems more natural for 
service A to use the A address as a source address for outbound 
connections. FWIW the second problem can be solved by using things like LXC.

> Based on this it seems that making all service addresses deprecated immediately and leaving the iron address as the only preferred address would be the only way to solve this. It does feel like a h…

I'd stay away from overloading the preferred/deprecated notions for 
these purposes. That information is typically under the control of the 
protocols, be it DHCPv6 or stateless address autoconfig.
Even if you use static IPv6 address assignment today (hence can "borrow" 
the deprecated state), you'd loose the option to use DHCPv6 or SLAAC 
tomorrow.
And there is a risk that implementations have issues where even static 
addresses might be affected by the router advertisements. Hence I think 
you should use something different.

> So: does anybody have any advice on this? Is it safe/future-proof/etc to mark addresses as deprecated to avoid them being used as default source addresses? Do we need some other way to steer the de…

How many applications need to do the right thing for outbound 
connections? Do they all need to do the same (use the iron address), or 
do some need to use the iron address and others the service address?

Depending on the above it may or may not make sense to tweak the 
applications and not rely on the stack to pick the source address.

Regards,
     Erik

---

From: Leo Baltus <[email-masked]>
Date: Fri, 15 Feb 2013 14:18:48 +0100

Op 14/02/2013 om 17:47:09 -0800, schreef Erik Nordmark:
> On 2/14/13 4:01 AM, Sander Steffann wrote:
> >>>connections.
> >>That was my thought too. Sander did write
> >>"- If the address is load balanced then initiating outbound connections from that address will break as return traffic might be
> >>load balanced to one of the other boxes"
> >>but that suggests a strange and stateless load balancer, that is attempting
> >>to balance return packets for sessions initiated from behind the load balancer,
> >>which is broken anyway. It's only incoming sessions for which the LB can
> >>pick a target.
> >
> >It's direct routing, where the outbound traffic does not pass
> >through the load balancer at all. It scales much better than pushing
> >all traffic through the load balancer, especially when the outbound
> >traffic is 99% of the total (content, streaming etc). The downside
> >is that multiple boxes have the same IP address configured and the
> >load balancer balances between them using layer-2. Which is why the
> >boxes must not use the load balanced addresses as source address for
> >their own connections, which is one of the reasons why I asked the
> >question :-)
> 
> Are you doing some live migration of the applications?
> Or just stopping one and restarting it on a different box?
> 
> If you want outbound traffic to work well I'd assert you'd want one
> IP address for every instance of your application. Thus if service A
> is load balanced to 3 servers, and service B to 4, you'd have 7 IP
> addresses that are unique per service instance. (In addition you'd
> have 2 for the services - A and B, plus one for each physical
> server.)
> 
> I think you can do that using LXC (or your favorite server
> virtualization technology) by assigning the service (e.g., B)
> address to the loopback interface in the container/VM, and the
> instance address to eth0 in the container/VM.
> 

Let me jump in here and try to explain what we are doing. This is a
lengthy post sorry for that.

We are Dutch Public Braodcasting, we host all sort of (web)applications
which more or less are are associated with national tv and radio shows.

We also host application during events like Tour de France, Olympics,
World Cup Soccer etc.

At very unpredictable moments we can serve >30.000hits/sec on webtraffic
streaming content exceeds 100Gb/sec. In order to accommodate this, each
application, be that php, java, ruby-on-rails etc, is hosted behind a
loadbalancer, a minimum of 4 apache reverse front-proxies and a minimum
of 4 backends running said php's java's etc.

Not every application needs to take the full load constantly so there
are a lots of vhosts in each apache instance.

The hosts are running a fairly run of the mill linux, but all
instances are started by our own scripts and each instance bring up its
own ipadress. We have several groups of hosts we call clusters (not HA)
a cluster can consist of some >30 servers. Each OS is
configured identical apart from its hostname and its IPv4adress we call
this its iron-address. Storage is either nfs-based or iscsi, in
the latter case if a service requires iscsi it mounts its own storage
on startup.

On such a cluster we run about 400 apache instances and some 300 other
instances like tomcat, mysql, memcached, postfix etc. serving some 700+
applications. As each host is identical the configuration of all
instances is on every host and we use rsync to keep it that way. The
information of what instance is running where is kept on nfs-storage.

The nice thing of this setup is that we can take an instance of, say, php
away from busy hardware en start it elsewhere with minimal impact. Also
we can easily deploy extra instance when needed. Needless to say that we
often need more capacity than a single host can offer so virtualization
is not an option.

As Sander has pointed out there is an issue with source address
selection in IPv6 because it defaults to using the most recently
added address. In our case, were are not using SLAAC, this is always an
address of some service that is started last.

That becomes an issue if we move a service to some other hardware. Long
running tcp connections break and also all traffic initiated from this
host appears to come from another IPv6 address, so that a change of
service A has impact for service B,C,D. This is not acceptable.

Of course there are applications like postfix which can bind() we use
that whenever possible. But things like ruby-on-rails, java, php or even
mod_balancer need to have this information from the instance running it,
as far as I know there is no facility to globally bind() on every
outgoing connection, and there will always be some application which do
not so we have to have a plan.

The only way we have found is to bring up each service address with
preferred_lft 0 so it will not be chosen, the only one left is its iron-
address which is exactly what we want and we have been using this for
the last 2 years.

So far so good. The thing that bothers me however is that this use of
'preferred_lft 0' is not very known and accepted and in linux at least if
you list ip addresses it shows up 'deprecated':

$ ip a
<snip>
    inet6 2a02:458:101:28:100:28:0:72/128 scope global deprecated 

This is what Sander means with 'feels like a hack'. I can live with the
fact that this may just be a wording issue and if we could developers
to persuade to call it 'preferred_lft 0' that may take some of this
feeling away.

On the other side I would also feel a bit better if the use
'preferred_lft 0' would be documented better, indeed seeing
pages like http://biplane.com.au/blog/?p=30 (thanks for the link BTW) do
help but this is just a page. I would feel much more comfortable if the
use of this option would be part of some BCP or so, so I could refer to
it to others, like the developers of keepalived which I have sent a
patch but never got accepted

http://marc.info/?l=keepalived-devel&m=130200733315039

Keepalived keeps track of members of an IPVS loadbalanced setup, when
using direct routing you either must bind() or use preferred_lft 0 to
avoid the loadbalanced address to be used as source address. Buts this
is just an example.

Then there is another issue. If we set-up a ipv4 and ipv6-with-preferred_lft-0
and on the same host an connection is initiated to a 'name' which
resolves to both addresses, it seems on linux that it is falling
back to IPv4. We think this is a bug, but in order to discuss this
isssue we need to have a firm reference to the use of preferred_lft 0 in
order to make any chance of have it accepted as a bug.

I hope this explains a little bit more, please shoot :)

-- 
Leo Baltus, internetbeheerder                         /\
NPO ICT Internet Services                            /NPO/\
Sumatralaan 45, 1217 GP Hilversum, Filmcentrum, west \  /\/
[email-masked], 035-6773555                    \/

---

From: sowmini varadhan <[email-masked]>
Date: Fri, 15 Feb 2013 20:46:54 -0500

> address of some service that is started last.

Are you sure this is not just an implementation artifact of whatever OS
you are using? There's nothing that I can see in RFC6724 that mandates
that the "most recently added address" should be selected first.

However Rule 5 does prefer outgoing interface over some other cases
that may apply in your use-case (e.g., longest matching prefix), so trying
Erik's suggestion of adding service-address on the loopback (and having
the services explicitly bind() to that address) might solve your problem?

--Sowmini