Subject: interop problems with getaddrinfo() address selection

From: Tony Finch <[email-masked]>
Date: Thu, 6 Dec 2007 19:03:31 +0000

Rule 9 of RFC 3484 specifies that the IP addresses (v4 and v6) returned by
getaddrinfo() should be sorted according to the size of their common
prefix with the local host's chosen source IP address. This defeats DNS
round robin load balancing which has led to some severe operational
problems. DNS round robin needs to be documented in an RFC, and RFC 3484
should be updated with rule 9 deleted or substantially modified for
compatibility with DNS round robin.

Ian Jackson has written a thorough description of the problem from the
perspective of a proposal that Debian's version of glibc should have the
troublesome sorting behaviour removed:

http://lists.debian.org/debian-ctte/2007/11/msg00029.html

FreeBSD's getaddrinfo() only implements Rule 9 if the kernel's address
selection policy is configured, which is not done by default so DNS round
robin is honoured.

[I originally posted this to the dnsop WG list, but they don't seem to
understand or care about the problem...]

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE LUNDY FASTNET: WEST VEERING NORTHWEST 6 TO GALE 8, INCREASING GALE 8 TO
STORM 10 FOR A TIME. VERY ROUGH OR HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR
GOOD.

---

From: der Mouse <[email-masked]>
Date: Thu, 6 Dec 2007 15:18:52 -0500 (EST)

> Rule 9 of RFC 3484 specifies that the IP addresses (v4 and v6)
> returned by getaddrinfo() should be sorted according to the size of
> their common prefix with the local host's chosen source IP address.
> This defeats DNS round robin load balancing which has led to some
> severe operational problems.

I'm interseted in what sort of problems it causes.  It sounds
reasonable to me, which means there's something operationally important
I'm missing.  Can you explain, or do you have a reference to somewhere
I can read about it?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Tony Finch <[email-masked]>
Date: Fri, 7 Dec 2007 17:46:01 +0000

On Thu, 6 Dec 2007, der Mouse wrote:
>
> I'm interseted in what sort of problems it causes.

DNS round robin is effective at balancing load evenly across multiple
servers, because each client is (in effect) directed to one of the servers
at random. If the client sorts the DNS response before using it then the
randomization is defeated and the load on the servers is no longer
balanced.

Ian's message described the problem briefly in point 12, but I forgive you
for not reading it :-)

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER THAMES DOVER: WEST OR NORTHWEST 7 TO SEVERE GALE 9,BACKING
SOUTHWEST 5 OR 6. VERY ROUGH DECREASING ROUGH. SHOWERS, RAIN LATER. GOOD
BECOMING MODERATE.

---

From: der Mouse <[email-masked]>
Date: Fri, 7 Dec 2007 13:27:33 -0500 (EST)

> servers, because each client is (in effect) directed to one of the
> servers at random.  If the client sorts the DNS response before using
> it then the randomization is defeated and the load on the servers is
> no longer balanced.

Doh!  I knew I was missing something.  (What I was missing was looking
at the problem from the point of view of the servers, rather than the
clients.)

Thanks for the explanation.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Keith Moore <[email-masked]>
Date: Sat, 08 Dec 2007 13:21:00 -0800

> randomization is defeated and the load on the servers is no longer
this seems like an almost-inherent consequence of IPv6's expecting hosts
to have multiple addresses and asking hosts initiating a communication
to do address selection.  it's not specific to getaddrinfo() - any API
that tried to solve that problem would be likely to reorder addresses to
some degree.

though it should be possible for the client implementation to go through
the list of addresses, assign a preference value to each one (based on
address type and number of leading bits in common with one of the
hosts's local addresses), then perform a stable sort by preference value.

---

From: Tony Finch <[email-masked]>
Date: Sun, 9 Dec 2007 20:10:03 +0000

> that tried to solve that problem would be likely to reorder addresses to

The problem isn't specific to IPv6 - for example see the comments about
address selection for "multihomed" MXs in RFC 2821. However in practice we
ignore this possibility and treat host names with multiple IP addresses as
equally reachable at any address. I don't think IPv6 substantially changes
this situation, because most of its common extra addresses have limited
scope and will not be published in the DNS.

> hosts's local addresses), then perform a stable sort by preference value.

That's essentially what RFC 3484 recommends, and what causes the problem.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTIES: CYCLONIC 7 TO SEVERE GALE 9 BECOMING NORTHERLY 5 TO 7. ROUGH OR VERY
ROUGH, OCCASIONALLY HIGH AT FIRST. RAIN OR SHOWERS. MODERATE OR GOOD.

---

From: Keith Moore <[email-masked]>
Date: Sun, 09 Dec 2007 12:29:59 -0800

> scope and will not be published in the DNS.
I think it's a stretch to assume that in the future that all IP
addresses associated with a domain and advertised in DNS are equally
reachable.

As a start IPv4 isn't going away.  So a site may need to advertise both
IPv4 and IPv6 addresses in order to be widely reachable, and the party
initiating a connection gets to decide which is better.

>From my own experience it is a good idea for a site to advertise both
6to4 and native IPv6 addresses.  A 6to4 site initiating a connection
will probably do better to use the destination's 6to4 address; similarly
for a native site using the destination's native address.  An
IPv4-to-IPv4 connection is almost certainly better than a connection
between a native and 6to4 IPv6 address.

Then you have ULIAs which are globally unique and globally scoped but
which might not be globally reachable.  As long as there's an
expectation that IPv6 addresses are PA, there will be a need for ULIAs. 
But networks using ULIAs will interconnect with other networks including
those using PA space and other networks using ULIAs. 

Then you have mobile IP which creates additional choices, and a growing
tendency for hosts to have multiple network interfaces where more than
one interface can be connected to the Internet.

The notion that you can control address visibility using DNS in this
world is pretty much a fantasy.

> That's essentially what RFC 3484 recommends, and what causes the problem.
What causes the problem is the assumption that a destination can have
multiple addresses, a source can have multiple addresses, and the party
initiating the connection can somehow make a good decision about which
address pair to use without any information from the network.

As for DNS randomization, it's never been reasonable to expect an
application to try each of several addresses in the order returned by
DNS.  The fact that many apps happened to work this way (because it was
often the easiest way to do things) was an accident, not something
required by the DNS protocol or the defined protocol for most of those apps.

Keith

---

From: Tony Finch <[email-masked]>
Date: Mon, 10 Dec 2007 07:49:52 +0000

> addresses associated with a domain and advertised in DNS are equally

The examples you listed, based on different kinds of connectivity, are
reasonable; what it not is de-randomizing equivalent classes of address.

> address pair to use without any information from the network.

Yes :-)

> DNS.

That has been the de facto standard behaviour for over 10 years (though
before getaddrinfo they only tried the first). A lot of sites are going to
be rather upset if wide deployment of RFC 3484 suddenly breaks their
service architecture.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOVER: NORTHWESTERLY 6 TO GALE 8, DECREASING 4 LATER. MODERATE OR ROUGH,
OCCASIONALLY VERY ROUGH. SQUALLY SHOWERS. GOOD.

---

From: der Mouse <[email-masked]>
Date: Mon, 10 Dec 2007 13:10:27 -0500 (EST)

> The examples you listed, based on different kinds of connectivity,
> are reasonable; what it not is de-randomizing equivalent classes of
> address.

But whose definition of "equivalent" applies?  From a client point of
view, close addresses are not equivalent to distant addresses - the
client wants a nearby server, not unreasonably - making sorting by
match length at least moderately reasonable.  (Not entirely; routing
topology doesn't necessarily parallel matching prefix length.  But,
absent something like AS path information, it's probably the best
available approximation.)

>> application to try each of several addresses in the order returned
>> by DNS.
> A lot of sites are going to be rather upset if wide deployment of RFC
> 3484 suddenly breaks their service architecture.

Any architecture that depends on the DNS preserving record ordering
(never mind depending on clients doing anything in particular with the
order the DNS happens to give them) is already broken.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Tony Finch <[email-masked]>
Date: Mon, 10 Dec 2007 19:46:52 +0000

On Mon, 10 Dec 2007, der Mouse wrote:
> match length at least moderately reasonable.

In practice, services advertise server-specific names as well as umbrella
names, so that clients that care about which server they use can ask for
it.

> order the DNS happens to give them) is already broken.

Services that use round robin DNS do not depend on preserved record
ordering: quite the opposite, they depend on disordering. (Bind does
round-robin when acting as a cache as well as an authoritative server.)
You can claim it's broken until you are blue in the face, but it works in
practice and it is widely documented (though not by the IETF) without
caveats.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
TYNE DOGGER: NORTHERLY 5 TO 7 BECOMING VARIABLE 4. ROUGH OR VERY ROUGH
DECREASING MODERATE OR ROUGH. SHOWERS. MAINLY GOOD.

---

From: der Mouse <[email-masked]>
Date: Mon, 10 Dec 2007 18:11:26 -0500 (EST)

> ordering: quite the opposite, they depend on disordering.

Hmm?  It appears we're talking about differnet things.  To me, DNS
round-robin refers to the sort of thing the NTP pool project does:
rotating the answer(s) returned in response to a query through a larger
(usually much larger) set of possible answers.  It sounds as though
you're talking about simply returning all the answers and hoping that,
somewhere along the line, someone will end up choosing one in a way
that looks random with respect to the property you care about.

For example, see this next quote, which makes sense only for the latter
meaning of "round-robin":

> (Bind does round-robin when acting as a cache as well as an
> authoritative server.)

Thanks for making my point for me :-) - this indicates that you can't
depend on a caching server preserving the order (or disorder, for that
matter) of records it receives from elsewhere.

Round-robin - this kind - is not broken per se.  Depending on it for
correctness (as opposed to efficiency) is - you cannot count on
arm's-length DNS servers doing anything you want, be it preserving or
disturbing, with the order of RRs handed to it.  If your setup depends
on spreading the load over multiple servers, then you really need to
return different addresses to different queries, not just the same
addresses in different orders.

> You can claim it's broken until you are blue in the face, but it
> works in practice and it is widely documented (though not by the
> IETF) without caveats.

Whey I say it's broken to depend on it, what I mean is, it may "work",
but if/when it fails, it's _your_ bug, and it's unreasonable to expect
others to change to work around it.  Documentation implying otherwise
may exist, but, if so, it's wrong.  1034 section 3.6: "The order of RRs
in a set is not significant, and need not be preserved by name servers,
resolvers, or other parts of the DNS.".  5.2.1 item 1: "Since the DNS
does not preserve the order of RRs," - the rest of this particular
quote is actually relevant to the issue that started this thread:
"...[name to address translation] may choose to sort the returned
addresses...".

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Keith Moore <[email-masked]>
Date: Mon, 10 Dec 2007 16:32:39 -0800

> that looks random with respect to the property you care about.
mumble.  if what the site really wants it for the client to pick an
address at random, then having something besides the DNS server
randomize the order of the results doesn't matter so much.   (but if the
client sorts the addresses, the randomness is lost regardless of how
many random reorderings were imposed between DNS and the client).

on the other hand, if the site is trying to do dynamic load-balancing
with DNS by having DNS favor lightly-used servers, then reordering does
matter.
> addresses..."
you and I (and I expect Tony also) agree on that.  but practically
speaking it makes little difference.  if a feature that used to "work"
in IPv4 (even if by coincidence or accident) fails to work in IPv6, IPv6
will get blamed.  and this may hinder adoption of IPv6.  most of the
people affected by this won't bother to read through the RFCs to see who
is really to blame.

Keith

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 11 Dec 2007 10:47:46 +0100

On Mon, Dec 10, 2007 at 04:32:39PM -0800,
 Keith Moore <[email-masked]> wrote 
 a message of 47 lines which said:

> on the other hand, if the site is trying to do dynamic
> load-balancing with DNS by having DNS favor lightly-used servers,

There is a standard way for a site to do this sort of "DNS
engineering" and it is SRV (RFC 2782). Everything else can be only a
hack since, as explained, the DNS and the subsequent libraries cannot
guarantee anything about the order of addresses.

---

From: Tony Finch <[email-masked]>
Date: Tue, 11 Dec 2007 10:39:57 +0000

On Mon, 10 Dec 2007, der Mouse wrote:
>
> It appears we're talking about differnet things.  To me, DNS round-robin
> refers to the sort of thing the NTP pool project does: rotating the
> answer(s) returned in response to a query through a larger (usually much
> larger) set of possible answers.

That's incompatible with RFC 2181 section 5 and DNSSEC.

> It sounds as though you're talking about simply returning all the
> answers and hoping that, somewhere along the line, someone will end up
> choosing one in a way that looks random with respect to the property you
> care about.

Right.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
VIKING NORTH UTSIRE SOUTH UTSIRE FORTIES: NORTHWEST 5 OR 6, BECOMING VARIABLE
3 OR 4, THEN BECOMING SOUTH 5 TO 7, PERHAPS GALE 8 LATER IN VIKING. MODERATE
OR ROUGH. SHOWERS. GOOD.

---

From: der Mouse <[email-masked]>
Date: Tue, 11 Dec 2007 09:26:31 -0500 (EST)

>> It appears we're talking about differnet things.  To me, DNS
>> rotating the answer(s) returned in response to a query through a
>> larger (usually much larger) set of possible answers.
> That's incompatible with RFC 2181 section 5

I read through 2181 section 5 and the only thing I see that seems
plausible to me as what you're talking about is 5.1: "A query for a
specific (or non-specific) label, class, and type, will always return
all records in the associated RRSet".

All you need to do here is hold your mind a little differently - think
of it as changing the RRset instead of returning only some of the
records from an unchanging RRset.

> and DNSSEC.

(a) I don't see how it could be, assuming the mental attitude mentioned
above (a small changing RRset instead of part of a large static RRset).

(b) So, don't sign such records. :-)

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Keith Moore <[email-masked]>
Date: Tue, 11 Dec 2007 10:31:38 -0800

> engineering" and it is SRV (RFC 2782).
Per RFC 2782, SRV only applies to protocols that are specified to use
it.  Even then, it's not clear that SRV is good for dynamic load balancing.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 11 Dec 2007 10:33:29 -0800

> records from an unchanging RRset.
which is fine if and only if it increments the zone serial # each time.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 11 Dec 2007 10:33:19 -0800

Keith

---

From: der Mouse <[email-masked]>
Date: Tue, 11 Dec 2007 14:40:34 -0500 (EST)

>> All you need to do here is hold your mind a little differently -
>> think of it as changing the RRset instead of returning only some of
>> the records from an unchanging RRset.

Agreed.

But since almost all queries do not see the zone serial, this is not
relevant to whether doing such things in normal operation is
reasonable.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Tony Finch <[email-masked]>
Date: Wed, 12 Dec 2007 15:34:06 +0000

> it.  Even then, it's not clear that SRV is good for dynamic load balancing.

That's OK: we're talking about static load balancing. The SRV and MX
specifications both require the client to randomize the RRs, so they
achieve this goal.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: SOUTHWESTERLY 3 OR 4 BACKING SOUTHERLY 6 TO GALE 8, PERHAPS
SEVERE GALE 9 LATER. VERY ROUGH, OCCASIONALLY HIGH LATER. OCCASIONAL RAIN.
MODERATE OR GOOD.

---

From: Keith Moore <[email-masked]>
Date: Mon, 10 Dec 2007 11:46:07 -0800

> before getaddrinfo they only tried the first).
IIRC, very old versions of gethostbyname() only returned a single
address, but this was fixed in most platforms long before IPv6.  It was
still up to the application to reference h_addr_list rather than h_addr,
and to choose from the list of returned addresses.
>  A lot of sites are going to
yes, they'll be upset,  though I wouldn't pin the blame on RFC 3484. 
and actually I'd be surprised if the sites pinned the blame on RFC 3484
either.  it seems more likely that they'd just blame IPv6.

as for their "service architecture", expecting everyone else in the
Internet to do things your way has always been risky.

Keith

p.s. I wouldn't have much of a problem with clarifying a successor to
RFC 3484 in this regard, though IMHO getaddrinfo() is so messed up that
it really needs to be scrapped.  yes, I realize this would be a PR disaster.