Subject: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

From: Erik Nygren <[email-masked]>
Date: Fri, 15 Jun 2018 11:45:19 -0400

A number of folks have been bitten by a bug in bind 9.12 where it silently
changes the default sorting of rrsets to always be sorted (even if the
authoritative response wasn't sorted).  This causes problems for services
assuming at least some degree of round-robin behavior by clients as now
many clients would sent all traffic to only the lowest IP.  Bug details:
https://gitlab.isc.org/isc-projects/bind9/issues/336   If you are upgrading
to or have upgraded to bind 9.12 you likely want to take a fix or override
in config.

This raises the question of whether there would be value in a more modern
BCP covering round-robin expectations for recursive resolvers?  I suspect
many (most?) service operators take at least some degree of DNS round-robin
behavior by recursive resolvers as a default.

I suspect starting assumptions are roughly in the range of:

* Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of
round-robin in RRset responses.

* There are a variety of ways to implement round-robin (randomize, permute,
etc).

* Server operators need to be aware that round-robin may be a part of a
load balancing scheme (especially if capacity is far greater than average
demand) but should not be relied on exclusively.  (Perhaps with examples of
why...)

Am I missing something in-terms of an existing BCP to this effect?  There's
RFC 1794, but I couldn't find much else (but given the sheer number of DNS
RFCs it's very likely I missed one).

Best, Erik

---

From: Jared Mauch <[email-masked]>
Date: Fri, 15 Jun 2018 12:08:22 -0400

> On Jun 15, 2018, at 11:45 AM, Erik Nygren <[email-masked]> wrote:
> 
> A number of folks have been bitten by a bug in bind 9..12 where it silently changes the default sorting of rrsets to always be sorted (even if the authoritative response wasn't sorted).  This causes problems for services assuming at least some degree of round-robin behavior by clients as now many clients would sent all traffic to only the lowest IP.  Bug details:  https://gitlab.isc.org/isc-projects/bind9/issues/336   If you are upgrading to or have upgraded to bind 9.12 you likely want to take a fix or override in config.
> 
> 
> This raises the question of whether there would be value in a more modern BCP covering round-robin expectations for recursive resolvers?  I suspect many (most?) service operators take at least some degree of DNS round-robin behavior by recursive resolvers as a default.
> 
> * Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of round-robin in RRset responses.

Stub is in your desktop (and possible application, but may vary depending on OS, etc).

> * There are a variety of ways to implement round-robin (randomize, permute, etc).

Sure, but services should also be sufficiently robust should there be some amount of polarization in the hash algorithm.  I recall when the entropy went into bind4 to assist with balancing of some web services as well as mail/ftp stuff.

Perhaps this is something that could be done better with a flavor of happy-eyeballs in the client application, whereby you race not just v4 vs v6 but also some limited subset of answers the application gets back as part of getaddrinfo()

> * Server operators need to be aware that round-robin may be a part of a load balancing scheme (especially if capacity is far greater than average demand) but should not be relied on exclusively.  (Perhaps with examples of why...)
> 
> Am I missing something in-terms of an existing BCP to this effect?  There's RFC 1794, but I couldn't find much else (but given the sheer number of DNS RFCs it's very likely I missed one).

Have you checked the DNS Camel :-)

https://powerdns.org/dns-camel/

I didn’t find anything else looking real quick, and it gave me a reminder of some of the folks that I used to interact with at ANS back in the day, so at least for a Friday it’s a fun and quick re-read to hit up 1794.

- Jared

> 
> Best, Erik
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Tony Finch <[email-masked]>
Date: Fri, 15 Jun 2018 17:53:57 +0100

Erik Nygren <[email-masked]> wrote:

> authoritative response wasn't sorted).

Huh, I noticed this and put the workaround in my config but I didn't
realise it counted as a bug.

Anyway, there's a related issue, RFC 3484 address sorting. I believe glibc
used to derandomize addresses in getaddrinfo(), but it seems to have
stopped doing so. (However I can't find a changelog item saying so...)

Windows had a similar issue (I think it was fixed in Windows 7?):

https://support.microsoft.com/en-us/help/968920/windows-vista-and-windows-server-2008-dns-clients-do-not-honor-dns-rou

I never understood how RFC 3484 sorting could possibly make sense when
there's no way for an edge device (or even an expert sysadmin) to know
anything meaningful about how IP addresses relate to network topology.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Sole, Lundy, Fastnet: West or southwest 3 or 4, increasing 5 or 6. Moderate or
rough. Occasional rain or showers. Good, occasionally poor.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 15 Jun 2018 13:12:31 -0400

On Fri, Jun 15, 2018 at 11:45:19AM -0400, Erik Nygren wrote:
> A number of folks have been bitten by a bug in bind 9..12 where it silently

I believe that RRsets are unordered sets by definition.  So I supect
that if people are relying on the order in which they come off the
wire, they're making a mistake.

This is a common mistake with unordered data sets; it results in lots
of complaints about SQL data responses, too, for instance.  (Of
course, in that case you always have the option of using an ORDER BY
clause, which we don't have in the DNS.)

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Paul Vixie <[email-masked]>
Date: Fri, 15 Jun 2018 20:23:58 +0300

> wire, they're making a mistake.

that's what i've always said. especially after adding round-robin and 
getting all the complaints about it. famously one complaint came from 
TGV, who had a terminal server product that used TXT RRsets to describe 
possible "hosts" that could be reached through their product. for their 
internal corporate network, they had encoded carroll's _Jabberwocky_, 
one verse per internal server. the bug report that came to me as the 
BIND4 maintainer was, "you're scrambling jabberwocky" or words to that 
effect. my response when closing the report was, "how can you tell?"

> clause, which we don't have in the DNS.)

i think round-robin does more good than harm. another famous bug report 
from the BIND4 era when i first added round-robin was from UC Berkeley, 
who had two NS RRs for berkeley.edu, and the second one had never 
worked, and after i added round-robin, it started to receive queries for 
the first time ever. to that complaint i said, "fix your NS RRset."

so while there's no way to please everybody, and while depending on the 
order (or in today's case, on the lack of order) of data whose ordering 
is not guaranteed is certainly an error, it's very implicit one, and we 
need sensible defaults rather than interminable whichness-of-what arguments.

-- 
P Vixie

---

From: "Peter van Dijk" <[email-masked]>
Date: Fri, 15 Jun 2018 20:01:51 +0200

Hello Andrew,

> wire, they're making a mistake.

This. One hundred million times, this.

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: Shumon Huque <[email-masked]>
Date: Fri, 15 Jun 2018 14:28:35 -0400

> wire, they're making a mistake.

This is all true.

Yet, large numbers of application services on the Internet already rely on
the assumption that address RRsets are returned from resolvers in a
shuffled or randomized order for load balancing reasons. So changing this
behavior will elicit many fierce complaints (I already observed it).

-- 
Shumon.

---

From: Bob Harold <[email-masked]>
Date: Fri, 15 Jun 2018 14:38:00 -0400

> behavior will elicit many fierce complaints (I already observed it).

Round-robin is a documented feature that many applications use.  Removing
it from DNS resolvers, and then having to add it to a much larger number of
applications, does not seem like a good trade-off.

-- 
Bob Harold

---

From: Jared Mauch <[email-masked]>
Date: Fri, 15 Jun 2018 14:59:47 -0400

> On Jun 15, 2018, at 2:01 PM, Peter van Dijk <[email-masked]> wrote:
> 
> Hello Andrew,
> 

Yes, so because they’re unordered, they could also be ordered, and that should be seen as a version of unordered vs strictly ordered.  Just because the last 10m times I asked it came in the same order doesn’t mean that for 10m+1 it will be or should be.

I’m reminded of some large content providers telling me how they routed their ICMP bits differently than their non-ICMP because they had dedicated hosts to just respond to the ICMP requests of www.example.com folks used as a measurement point to see if their connectivity was working :-)

- Jared

---

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 16 Jun 2018 01:22:32 +0530

> applications, does not seem like a good trade-off.

The _default_ in BIND 9.12 was changed from order random to order
none. It seems to be missing from the release notes by mistake, but the
administrator manual mentions what the default is:

         <para>
	    By default, records are returned in indeterminate but
            consistent order (see <command>none</command> above).
          </para>

As before, an admin can configure it to any of {fixed, random, cyclic,
none} per class/type/domain, in named.conf. See the section "RRset
Ordering" in the administror manual.

No sorting functionality was removed. The default was changed.

		Mukund

---

From: Erik Nygren <[email-masked]>
Date: Fri, 15 Jun 2018 16:17:00 -0400

On Fri, Jun 15, 2018 at 3:52 PM, Mukund Sivaraman <[email-masked]> wrote:

> administrator manual mentions what the default is
>

We have many years of software that relies on emergent behaviors from the
current default.
While pedantically it may be true that these should be treated as unordered
sets and that
applications or stub resolver libraries should do some permutations or
randomized selection,
that doesn't match the current reality for widely used software (eg, curl
and ssh, which I'm
sure is just the tip of the iceberg).

Software should have safe defaults that matches common expectations.
Those common expectations, as demonstrated by the configuration of all
of the large public resolvers I've tested, as well as by how common
software behaves,
is that the order of results is NOT consistent.  In many environments, this
lack
of consistency is relied upon for systems to work properly.  Switching to
consistent
order is no big deal on a small scale, but a widespread shift (eg, as would
happen
due to a change in default in popular software) would almost certainly have
significant operational impact and is something that warrants significant
discussion
about the practical implications.

This ambiguity in the current specifications results in this mismatch
between the pedantic (rrsets are explicitly unordered, and a consistent
order is a subset of that) and the current reality (applications and
services
rely on resolvers-at-scale to be explicitly inconsistent in the ordering of
rrsets)
is why I started off by proposing that we may need a BCP or informational
RFC
that describes the currently assumed defaults and best-practices
(ie, round-robin is assumed in many places so don't consistently order
at-scale by default).

        Erik

---

From: Mukund Sivaraman <[email-masked]>
Date: Sat, 16 Jun 2018 01:51:22 +0530

> of the large public resolvers I've tested, as well as by how common

I agree with this.

Perhaps this should be done by default when recursion is on.

		Mukund

---

From: Paul Vixie <[email-masked]>
Date: Sat, 16 Jun 2018 00:42:51 +0300

Erik Nygren wrote:
> ...
>
> This ambiguity in the current specifications results in this
> mismatch between the pedantic (rrsets are explicitly unordered, and a
> consistent order is a subset of that) and the current reality
> (applications and services rely on resolvers-at-scale to be
> explicitly inconsistent in the ordering of rrsets) is why I started
> off by proposing that we may need a BCP or informational RFC that
> describes the currently assumed defaults and best-practices (ie,
> round-robin is assumed in many places so don't consistently order
> at-scale by default).

i am +1 to this view, which is why i didn't suggest you move it to 
bind-workers@.

-- 
P Vixie

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 15 Jun 2018 14:54:31 -0700

Just a question on this: was the old/classic behavior really
random/shuffled? Or was it that bind would "rotate" through iterations
where the order was the same each time if you think of the rrset list as a
ring, but with a different start and end point within that ring? (That's
what's described here:
https://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm)

On Fri, Jun 15, 2018 at 1:17 PM, Erik Nygren <[email-masked]> wrote:

> While pedantically it may be true that these should be treated as
> unordered sets and that
> is that the order of results is NOT consistent.  In many environments,
> this lack
> of consistency is relied upon for systems to work properly..  Switching to
> consistent
> order is no big deal on a small scale, but a widespread shift (eg, as
> would happen
> rely on resolvers-at-scale to be explicitly inconsistent in the ordering
> of rrsets)

-- 
Colm

---

From: Mark Andrews <[email-masked]>
Date: Sat, 16 Jun 2018 08:40:14 +1000

What we should be asking is why a service that needs multiple machines isn’t using SRV. It has randomisation between servers explicitly defined along with proportional load balancing. 

It also addresses CNAME at the apex which quite frankly is only used to find the name of the HTTP server for the zone which SRV  is quite capable of doing.

-- 
Mark Andrews

> On 16 Jun 2018, at 01:45, Erik Nygren <[email-masked]> wrote:
>

---

From: Shumon Huque <[email-masked]>
Date: Fri, 15 Jun 2018 19:54:56 -0400

On Fri, Jun 15, 2018 at 5:55 PM Colm MacCárthaigh <[email-masked]> wrote:

> https://docstore..mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm
> <https://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm>)
>

ISC veterans can confirm, but my recollection is that the earliest
implementations were indeed as described above - the response RRset was
cycled/rotated, rather than randomized.

Shumon.

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 15 Jun 2018 17:01:15 -0700

I think so too; and I wouldn't be so strict on backwards compatibility
there.

That behavior is a side-channel that defeats DNS privacy in some cases.
E.g. I can query a record, watch you send an encrypted query, then query
the record again, and tell what you queried. Within some probability at
least.

For that reason, It'd be worth experimenting with an implementation that
does shuffle the results each time.

On Fri, Jun 15, 2018 at 4:54 PM, Shumon Huque <[email-masked]> wrote:

>> what's described here: https://docstore..mik.ua/
>> orelly/networking_2ndEd/dns/ch10_07.htm

-- 
Colm

---

From: Shumon Huque <[email-masked]>
Date: Fri, 15 Jun 2018 20:12:37 -0400

On Fri, Jun 15, 2018 at 6:40 PM Mark Andrews <[email-masked]> wrote:

> What we should be asking is why a service that needs multiple machines
> isn’t using SRV. It has randomisation between servers explicitly defined
> along with proportional load balancing.
>

That would be nice, but sadly major applications like the web, have (in my
assessment) never seriously shown any interest in using SRV records. To
quote some of the discussion of HTTP2 on this subject:

from https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0674.html:

--- excerpt

"We discussed this, but decided not to support SRV in HTTP/2 - mostly
because HTTP/2 needs to be backwards-compatible with the existing Web,
which doesn't use SRV. In discussion, it also became pretty clear that if
we were to use an additional record type for HTTP, SRV may not suit,
because we need to do things like protocol version negotiation.

There were also concerns about latency and interoperability (especially
considering how many DNS requests a page load can make, and the unfortunate
limitations of some DNS gateways/proxies)."

--- end excerpt

The latter point about additional latency, due to the need for additional
follow-on queries to resolve the address records for the SRV targets (if
the resolver hasn't provided them), has been a pretty potent blocker for
SRV adoption in the web.

Also, in my experience, many implementations of client applications that do
employ SRV don't even bother to implement the load balancing aspect, just
the priority aspect. This was particularly true in Kerberos from my
recollection.

> It also addresses CNAME at the apex which quite frankly is only used to
> find the name of the HTTP server for the zone which SRV  is quite capable
> of doing.
>

Yup ..

Shumon.

---

From: Shumon Huque <[email-masked]>
Date: Fri, 15 Jun 2018 20:14:34 -0400

Yeah, good point about side channels. Let's stick to recommending
randomization!

Shumon.

---

From: Erik Nygren <[email-masked]>
Date: Fri, 15 Jun 2018 20:53:01 -0400

> quote some of the discussion of HTTP2 on this subject:

Agreed that it would be nice (and a good recommendation for protocols to
support SRV records),
but for all of the other protocols we're stuck operating today today with
the implementations we have,
not just the ones we wish we had.

(Side-note, I do recommend folks provide feedback on the ALTSVC draft:
    https://tools.ietf.org/html/draft-schwartz-httpbis-dns-alt-svc-01
It adds enough value and extensibility beyond just SRV that
browsers show more interest in implementing it.)

randomization!

Good point!  Agreed.  I hadn't thought about that and had otherwise been
thinking of shuffling and randomization as roughly equivalent as long as you
pick a random initial offset for the shuffling.
(More reasons to write current best practices down in a draft.)
Shuffling may also have some weird patterns showing up depending on the
relation between
TTLs and number of clients querying, but I don't know if this is ever
problematic
beyond the side-channel issue.

        Erik

---

From: bert hubert <[email-masked]>
Date: Sat, 16 Jun 2018 07:54:07 +0200

> wire, they're making a mistake.

A data point here may be useful.  PowerDNS has in many cases not done rrset
shuffling on a query by query basis.  All our three products employ whole
packet caches which send back binary identical responses to equivalent
questions, by default for one hour or the lowest TTL entry in the response,
whichever is lower.

We've been doing this since "forever" and it has indeed led to a single
complaint by a cable company with set top boxes that always connect to the
first IP address, no matter what.

Within PowerDNS Recursor it is possible to override the packet cache on a
per subdomain basis, and we know this has been deployed in a very limited
number of cases. Setting the TTL to 5 seconds also does it.

But otherwise over more than ten years, it appears the reliance on recursors
shuffling for each and every answer has not been there for our users (which
number in the hundreds of millions of IP addresses served).

Based on this, if we write a BCP, I'd be inclined to say that given what is
deployed, operators should not count on their resolving nameserver to do the
shuffling for them, because there is no guarantee that it will. 

Crucially, the new DoH spec enables or even encourages HTTP-level caching of
DNS messages by existing HTTP infrastructure, which guarantees there will be
no shuffling.

As an aside, another data point I'd like to mention is an obscure BIND
feature we had to clone called 'sort list'.  I mention it as obscure since
at a recent ISC/CZNIC/NLNetLabs/PowerDNS lunch, no one present of ISC could
recall why it was there: the sort list. 

https://ftp.isc.org/isc/bind9/cur/9.9/doc/arm/Bv9ARM.ch06.html

https://doc.powerdns.com/recursor/lua-config/sortlist.html?highlight=sortlist

This is a very powerful feature that makes BIND and the PowerDNS Recursor
sort RRSETs in very specific ways, depending on who asks. It turns out that
this feature is used in 3GPP environments to route roaming traffic (!). These
people very much do not rely on shuffling in the stub, but rely on the first
resource record being the one they should use.

At that lunch, we could not figure out who originally required such a
detailed ordering configuration in BIND, and it might be interesting to find
out.

But anyhow - as stated above, for non-trivial amounts of resolvers services,
shuffling isn't happening that frequently today, so maybe we should not BCP
reliance on this shuffling.

	Bert

---

From: "Peter van Dijk" <[email-masked]>
Date: Sat, 16 Jun 2018 10:30:23 +0200

> Yeah, good point about side channels. Let's stick to recommending

Unbound has interesting middle ground here:

        rrset-roundrobin: <yes or no>
               If yes, Unbound rotates RRSet order in response (the 
random number is taken from the query ID,
               for speed and thread safety).  Default is no.

It rotates, but you cannot predict (easily) by how much. It keeps the 
implementation simple but mostly avoids (as far as I can judge) the side 
channel.

I do want to point out that the default is ‘no’, suggesting it is 
getting away with no ‘round robin’ at all in many deployments.

Kind regards,

---

From: Philip Homburg <[email-masked]>
Date: Sat, 16 Jun 2018 14:04:46 +0200

>detailed ordering configuration in BIND, and it might be interesting to find

What I remember from a very long time ago is the following network setup:
- a collection of NFS servers each with multiple ethernet interface cards
  connecting to different subnets
- a collection of NFS clients that would connect to the first address in 
  the returned RRset (i.e. that would not locally sort the RRset)
- a DNS resolver in a completely different subnet that had incomplete knowledge
  of the network. I think bind treated the 'class B' network as a single
  network, not as a collection of /24s.

Without the sortlist feature the DNS resolver had not enough information
to move the best address to the start of the list. And the router was slow
enough that you didn't want NFS traffic to go through the router.

---

From: Florian Weimer <[email-masked]>
Date: Mon, 18 Jun 2018 10:22:37 +0200

On 06/15/2018 05:45 PM, Erik Nygren wrote:
> load balancing scheme (especially if capacity is far greater than 
> average demand) but should not be relied on exclusively.  (Perhaps with 
> examples of why...)
> 
> Am I missing something in-terms of an existing BCP to this effect?

Unless all addresses happen to have identical shared prefix length with 
the client address (that is, count-leading-zeros(client-address XOR 
server-address) is the same), RFC 6724 Rule 9 requires that clients do 
*not* perform random server selection.

I think this was a mistake in RFC 3484, and it is still wrong.  But I 
think procedurally, you cannot change this in a BCP.

Thanks,
Florian

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 18 Jun 2018 11:01:57 -0400

> We have many years of software that relies on emergent behaviors from the

Well, I think more accurately we have years of software that relies on
emergent behaviours of the prior default of certain implemnetations.

> sets

I don't think that's being pedantic.  That's literally the definition
of an RRset, and (as I guess has been shown by others in this thread)
relying on an existing order behaviour to which one has become
accustomed is going to be a problem even if people just switch out
their software or change the configuration of their systems.  This is
a bug in the relying software, because the network _never_ made the
promise that software was relying upon.

> Software should have safe defaults that matches common expectations.

I think this is true only if the common expectations are reasonable
ones, and given what RRsets are the expectation in this case is not a
reasonable one.  What you're really saying is that, if there was ever
one dominant system on the Internet (in this case, BIND), then the
standards all need to be rewritten to conform not only to what that
software implemented (a position with which I have some sympathy) but
also to conform with the default settings of that software (a position
I think needs rather more support than you've offered).  

> is that the order of results is NOT consistent.

Sure.  "Unordered sets."

>  In many environments, this
> lack
> of consistency is relied upon for systems to work properly.

To me, this is like saying that, in many environments, the order of
TCP packets (which very frequently do come in order) is relied upon
for systems to work properly.  This is true, but still broken.

> This ambiguity in the current specifications

What is the ambiguity?  There is only an ambiguity if you think that
people's expectation of something nobody ever promised them is part of
the specification, and it isn't.

Best regards,

A
-- 
Andrew Sullivan
[email-masked]

---

From: Paul Vixie <[email-masked]>
Date: Mon, 18 Jun 2018 18:45:27 +0100

to the extent that the dns technical community has a choice about 
default behaviour, we should consider the costs to the rest of the 
internet community of each default.

in my prior e-mail to this thread i gave examples of assumptions of 
ordering that were violated by the first round-robin implementation. in 
both cases, the assumption was dangerous -- led to fragility. (leaving a 
bad NS in an RRset, and putting verses of poetry in TXT RRsets.)

assumptions of non-ordering are less dangerous. (expect load balancing 
and don't get it.)

we should, if we can engage on the topic of defaults at all, recommend a 
default that is compatible with less-dangerous assumptions.

in other words we should re-order rrsets by default, so that very few 
people or agents are ever prone to think their order is stable. the spec 
says they are unordered, but human nature says, expect more of what 
you're seeing.

vixie

---

From: Florian Weimer <[email-masked]>
Date: Mon, 18 Jun 2018 23:14:01 +0200

> says they are unordered, but human nature says, expect more of what 

But the client has to sort them again based on shared prefix length
with its own address.  I think we should fix that as well, otherwise
the overall protocol disadvantages new entrants who cannot get a
contiguous prefix in which they can place all their load balancer
endoints, so that they are immune from that mandatory client sorting.

---

From: Shumon Huque <[email-masked]>
Date: Mon, 18 Jun 2018 17:51:26 -0400

On Mon, Jun 18, 2018 at 5:14 PM Florian Weimer <[email-masked]> wrote:

> with its own address.

I assume you're talking about the default destination address selection
rules specified in RFC 6724 (Section 6, Rule 9). Yeah, I agree this doesn't
seem wise to me. I'm not sure what the goal here was - this doesn't get
the client the topologically closest destination (in the general case; even
for IPv6 addresses unless the Internet was using solely provider aggregated
hierarchical routing, which isn't the case).

> I think we should fix that as well, otherwise

Client applications delegate address sorting to name resolution functions
like getaddrinfo() - correct?

Doing some quick tests of getaddrinfo() just now on a recent *NIX machine,
it appears to return addresses sorted roughly in accordance with RFC
6724, but rule 9 (longest prefix match) seems to be only applied to
IPv6 addresses. For IPv4 addresses (using an upstream resolver
that randomizes the response RRset), the order returned by getaddrinfo()
is ever changing - I assume presented in the order received.

I haven't actually read the code, so perhaps someone can confirm this, or
share what portions of the default address selection algorithm common
name resolution libraries actually implement.

Shumon.

---

From: "Darcy Kevin (FCA)" <[email-masked]>
Date: Mon, 18 Jun 2018 23:04:21 +0000

RFC 6724 specifically says: "Rules 9 and 10 MAY be superseded if the implementation has other
means of sorting destination addresses. For example, if the
implementation somehow knows which destination addresses will result
in the 'best' communications performance."

So, technically, if an implementation chooses a method of "the exact order in which I received the address records from my upstream resolver" as a way to produce the "'best' communication performance", given the circumstances, then that is technically not a violation of the standard. The local optimization is to trust the upstream resolver to Do The Right Thing. It's not always a wise choice, but most of the time it's better than sorting based on prefix-length matching (right?)

RFC 6724 is, after all, about *default* address selection (that word is even in the title of the RFC). Defaults are made to be superseded -- that's kind of the definition of what a default *is*.

											- Kevin

-----Original Message-----
From: DNSOP <[email-masked]> On Behalf Of Florian Weimer
Sent: Monday, June 18, 2018 4:23 AM
To: Erik Nygren <[email-masked]>; dnsop WG <[email-masked]>
Subject: Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

On 06/15/2018 05:45 PM, Erik Nygren wrote:
> * Server operators need to be aware that round-robin may be a part of 
> a load balancing scheme (especially if capacity is far greater than 
> average demand) but should not be relied on exclusively.  (Perhaps 
> with examples of why...)
> 

Unless all addresses happen to have identical shared prefix length with the client address (that is, count-leading-zeros(client-address XOR
server-address) is the same), RFC 6724 Rule 9 requires that clients do
*not* perform random server selection.

I think this was a mistake in RFC 3484, and it is still wrong.  But I think procedurally, you cannot change this in a BCP.

Thanks,
Florian

---

From: Shumon Huque <[email-masked]>
Date: Mon, 18 Jun 2018 19:21:03 -0400

On Mon, Jun 18, 2018 at 7:05 PM Darcy Kevin (FCA) <[email-masked]>
wrote:

> So, technically, if an implementation chooses a method of "the exact order
> in which I received the address records from my upstream resolver" as a way
> to produce the "'best' communication performance", given the circumstances,
> then that is technically not a violation of the standard. The local
> optimization is to trust the upstream resolver to Do The Right Thing. It's
> not always a wise choice, but most of the time it's better than sorting
> based on prefix-length matching (right?)
>
> RFC 6724 is, after all, about *default* address selection (that word is
> even in the title of the RFC). Defaults are made to be superseded -- that's
> kind of the definition of what a default *is*.
>

This whole thread is about "defaults"  though!

Application deployers attempting to rely (rightly or wrongly) on load
balancing of addresses presented by name resolution APIs will have
preferences for a particular default that allows them to achieve that goal.

(I'm well aware that the spec allows and that OSes provide knobs to tweak
the address selection algorithm - I've lost track of how many times I've
edited my gai.conf file to do various kinds of tests!)

Shumon.

---

From: "Petr Spacek" <[email-masked]>
Date: Tue, 19 Jun 2018 08:42:05 +0200

> edited my gai.conf file to do various kinds of tests!)

Hello dnsop,

I believe that discussion about protocol level/resolver behavior is going to be
purely academic because DoH is going to cache whole messages and that's it.

Do you think that having different behavior on "classic" DNS and DoH is a good idea?

-- 
Petr Špacek  @  CZ.NIC

---

From: Paul Vixie <[email-masked]>
Date: Tue, 19 Jun 2018 08:58:10 -0400

> But the client has to sort them again based on shared prefix length

happy eyeballs says so, but some clients won't implement anything like 
that, or will have local knowledge (access to a routing table via an API 
of some kind) that gives them even better information. the question of 
what a client should do, and whether a given client can or should, is 
clearly independent of what order the response comes to them in.

> I think we should fix that as well, otherwise the overall protocol
> disadvantages new entrants who cannot get a contiguous prefix in
> which they can place all their load balancer endoints, so that they
> are immune from that mandatory client sorting.

i have no strongly held view on whether clients should sort. but i'm 
sure a client will have better information to base such sorting on than 
the dns servers will have. therefore i'd like to pursue a simple rule 
whereby the server perturbs the order somehow. simplistic clients, and 
clients who aren't nearby any shard of a load balanced service, should 
see some kind of chaos in the ordering, not just know it can be there.

-- 
P Vixie

---

From: 神明達哉 <[email-masked]>
Date: Tue, 19 Jun 2018 11:51:18 -0700

At Mon, 18 Jun 2018 17:51:26 -0400,

> is ever changing - I assume presented in the order received.

The very original implementation of BSD getaddrinfo() conformed to the
RFC, but FreeBSD seems to have changed the behavior a few years ago so
rule 9 wouldn't apply to IPv4 addresses:
https://github.com/freebsd/freebsd/commit/1390d13ae69089142f6c6465dbb24438f295edee
The commit log suggests the rrset round-robin was exactly the
motivation for the change (whether it was good or bad).

--
JINMEI, Tatuya

---

From: Shumon Huque <[email-masked]>
Date: Tue, 19 Jun 2018 15:22:24 -0400

> motivation for the change (whether it was good or bad).

Very useful information. Thanks for confirming!

My tests were on a Linux machine, so apparently they did the same ..

Shumon.

---

From: Shumon Huque <[email-masked]>
Date: Tue, 19 Jun 2018 17:25:58 -0400

> motivation for the change (whether it was good or bad).

I took a quick look at glibc upstream source (from which I presume most
Linux distros take code) and see the following:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/posix/getaddrinfo.c;h=553833d1f22d6ea4ae24c68fe14e2140aa059d50;hb=HEAD#l1559

/* Rule 9: Use longest matching prefix.  */
[...]
          /* Outside of subnets, as defined by the network masks,
             common address prefixes for IPv4 addresses make no sense.
             So, define a non-zero value only if source and
             destination address are on the same subnet.  */

So again, no Rule 9 implementation for IPv4, unless there is a local subnet
destination involved, because it makes no sense otherwise (with which I
agree, as I previously remarked).

Shumon.