Subject: [DNSOP] What should ANAME-aware servers do when target records are verifiably missing?

From: Richard Gibson <[email-masked]>
Date: Tue, 9 Apr 2019 13:56:29 -0400

Copied from https://github.com/each/draft-aname/issues/54 per Tony Finch.

The current draft specifies

> We treat missing address records (i.e. NXDOMAIN or NODATA) the same 
> successfully resolving as a set of zero address records, and distinct 
> from "failure" which covers error responses such as SERVFAIL or REFUSED.

This is both undesirable for customers of DNS service providers (whose 
active sites will occasionally be inaccessible to some clients for 
$SOA_MINIMUM seconds), and operationally cumbersome because resolvers 
are not in a good position to synthesize the necessary SOA records for 
NXDOMAIN responses (e.g., example.com. ANAME example.invalid. alongside 
example.com. A 192.0.2.1). Tony suggested that this was to be "as much 
like CNAME as possible", but I disagree because unlike CNAME, ANAME can 
have sibling records which are therefore available for use.

---

From: Bob Harold <[email-masked]>
Date: Tue, 9 Apr 2019 16:04:25 -0400

> have sibling records which are therefore available for use.

If it gets an authoritative answer saying that there are no address
records, then it should respect that answer.  If that is incorrect, then
whatever gave that answer is broken or misconfigured and should be fixed.

Perhaps I am missing something.  In what cases can you imagine getting a
response with no errors and no records?

Perhaps a load balancer that has probed all the servers and determined that
none are working?  If you want a fall-back, it should be configured in the
load balancer in that case.

-- 
Bob Harold

---

From: Richard Gibson <[email-masked]>
Date: Wed, 10 Apr 2019 16:42:45 -0400

Responses below.

On 4/9/19 16:04, Bob Harold wrote:
> records, then it should respect that answer.  If that is incorrect, 
> then whatever gave that answer is broken or misconfigured and should 
> be fixed.
>
> Perhaps I am missing something.  In what cases can you imagine getting 
> a response with no errors and no records?

Misconfiguration is precisely the case, but quite possibly 
misconfiguration in the zone of /target/ records as opposed to the zone 
containing the ANAME. And there are two problems with respecting that 
[negative] answer.

The first problem is for the owner of the ANAME-containing zone, for 
whom the upstream misconfiguration will result in downtime and be 
extended by caching to the MINIMUM value from their SOA, which in many 
cases will be one to three orders of magnitude greater than the TTL of 
the ANAME itself.

The second problem is for the query originator and their ANAME-aware 
resolver, which will be forced to resolve the SOA of the 
ANAME-containing zone in order to issue a proper RFC 2308 Type 2 NODATA 
response <https://tools.ietf.org/html/rfc2308#section-2.2.1>—and such 
resolution must take place synchronously, tying up resolver resources 
and increasing end-user latency (insignificantly when the SOA is already 
cached, significantly when it is not).

Both of these problems can be addressed by 
allowing/recommending/requiring ANAME-aware servers to preserve ANAME 
siblings when resolution of ANAME targets results in NXDOMAIN or NODATA 
responses, rather than replacing them with an empty RRSet... which, to 
be honest, seems to be always-undesirable behavior anyway—if anyone can 
think of a scenario where it would be /beneficial/ to dynamically remove 
ANAME siblings, please share it.

---

From: Bob Harold <[email-masked]>
Date: Thu, 11 Apr 2019 09:48:21 -0400

On Wed, Apr 10, 2019 at 4:42 PM Richard Gibson <[email-masked]>
wrote:

> misconfiguration in the zone of *target* records as opposed to the zone
> The first problem is for the owner of the ANAME-containing zone, for whom
> the upstream misconfiguration will result in downtime and be extended by
> caching to the MINIMUM value from their SOA, which in many cases will be
> one to three orders of magnitude greater than the TTL of the ANAME itself.
>
> resolver, which will be forced to resolve the SOA of the ANAME-containing
> zone in order to issue a proper RFC 2308 Type 2 NODATA response
> <https://tools.ietf.org/html/rfc2308#section-2.2.1>—and such resolution
> must take place synchronously, tying up resolver resources and increasing
> end-user latency (insignificantly when the SOA is already cached,
> significantly when it is not).
>
> Both of these problems can be addressed by allowing/recommending/requiring
> ANAME-aware servers to preserve ANAME siblings when resolution of ANAME
> targets results in NXDOMAIN or NODATA responses, rather than replacing them
> with an empty RRSet... which, to be honest, seems to be always-undesirable
> behavior anyway—if anyone can think of a scenario where it would be
> *beneficial* to dynamically remove ANAME siblings, please share it.
>
Fair enough.  I would rather solve the long MINIMUM problem in general,
rather than only for this specific case.  But here was have a fairly easy
source of backup answers, so it will be easier than a general solution.  I
don't like it, but I see your point.

As for when we might want an NXDOMAIN answer, perhaps:
CDN is overwhelmed or web servers are all down, so stop users from trying
to connect.
Or server changes and we don't want users to fall back to old, incorrect
addresses.

-- 
Bob Harold

---

From: Matthew Pounsett <[email-masked]>
Date: Thu, 11 Apr 2019 18:50:11 -0400

On Wed, 10 Apr 2019 at 16:43, Richard Gibson
<[email-masked]> wrote:
>
>
> The first problem is for the owner of the ANAME-containing zone, for whom the upstream misconfiguration will result in downtime and be extended by caching to the MINIMUM value from their SOA, which in many cases will be one to three orders of magnitude greater than the TTL of the ANAME itself.

I think I'm missing something here.  If, for example, the TTL of the
ANAME is 1 hour, what mechanism results in caching holding onto a
negative answer for a broken target name for a minimum of 10 hours,
and to 40 days?

>
> Both of these problems can be addressed by allowing/recommending/requiring ANAME-aware servers to preserve ANAME siblings when resolution of ANAME targets results in NXDOMAIN or NODATA responses, rather than replacing them with an empty RRSet... which, to be honest, seems to be always-undesirable behavior anyway—if anyone can think of a scenario where it would be beneficial to dynamically remove ANAME siblings, please share it.

I feel like this is creating an even bigger potential problem.  What
happens when the owner of the ANAME target legitimately wants that
name to go away, but some other zone owner is leaving an ANAME in
place pointing to that now-nonexistent name?  Continuing to serve the
sibling records indefinitely seems like serve-stale gone horribly
wrong.

---

From: Richard Gibson <[email-masked]>
Date: Thu, 11 Apr 2019 20:02:26 -0400

Responses inline.

> negative answer for a broken target name for a minimum of 10 hours,
Demonstrative example zone:

example.com.  3600  IN    SOA  ns.example.net. hostmaster.example.net. 1 (
                                   7200   ; REFRESH
                                   3600   ; RETRY
                                  86400   ; EXPIRE
                                   3600  ); MINIMUM
example.com.    60  IN  ANAME  example.invalid.
example.com.    60  IN      A  192.0.2.1

When an ANAME-aware resolver queries an ANAME-aware authoritative server 
for example.com. A, it will receive the A record in the answer section 
and the ANAME in the additional section. If it then chases the ANAME 
target to an NXDOMAIN and accepts that as justification for replacing 
the sibling A RRSet with nothing as currently specified in the draft, 
then the appropriate response will be a Type 2 NODATA in which the 
answer section is empty and the additional section contains the SOA. But 
this suffers from both of the problems I have been complaining about—the 
resolver does not necessarily /have/ the zone SOA, possibility 
necessitating an inline lookup, and per RFC 2308 the negative response 
will be cached according to values from the SOA that are unrelated to 
and far exceed the TTL of the ANAME.

> sibling records indefinitely seems like serve-stale gone horribly

In such a configuration, the owner of the ANAME will be able to see that 
clients are using its static sibling records rather than its target (and 
therefore that they are getting no benefit from the ANAME), and can 
react accordingly. If your concern is excess queries for the ANAME 
target, then this seems no different from e.g. CNAME—the owner of the 
target can issue long-lived negative responses while performing whatever 
other exploration and/or mitigation they deem fit.

But this seems like it will be much more rare and frankly much less of a 
problem than stretching out misconfiguration at an ANAME target into 
extended downtime for an ANAME owner. It must be possible for the latter 
to execute a recovery plan as quickly as possible, and if ANAME is 
specified well then that the first step of recovery can be literally 
instant and automatic.

---

From: Matthew Pounsett <[email-masked]>
Date: Thu, 11 Apr 2019 23:45:10 -0400

> The first problem is for the owner of the ANAME-containing zone, for whom the upstream misconfiguration will result in downtime and be extended by caching to the MINIMUM value from their SOA, which…

[snip]

> But this suffers from both of the problems I have been complaining about—the resolver does not necessarily have the zone SOA, possibility necessitating an inline lookup, and per RFC 2308 the negative response will be cached according to values from the SOA that are unrelated to and far exceed the TTL of the ANAME.

Ah, I see the confusion.  You're using definitive statements such as
"will" when what you actually mean is "may."   There's no specific
mechanism that causes the client to cache the negative response "for
one to three orders of magnitude greater than the TTL of the ANAME."
And, the TTL of the SOA doesn't necessarily "far exceed" the TTL of
the ANAME.  You're just presupposing that will be a common
configuration?

I think we're still talking about misconfigurations here, and
designing a protocol around people who misconfigure their DNS at the
expense of people who configure it properly seems like a bad path to
take.

> Both of these problems can be addressed by allowing/recommending/requiring ANAME-aware servers to preserve ANAME siblings when resolution of ANAME targets results in NXDOMAIN or NODATA responses, r…

Yes, I can think of a case where it would be beneficial to remove
ANAME siblings: when the target of the ANAME is removed from the DNS.

> In such a configuration, the owner of the ANAME will be able to see that clients are using its static sibling records rather than its target (and therefore that they are getting no benefit from the ANAME), and can react accordingly. If your concern is excess queries for the ANAME target, then this seems no different from e.g. CNAME—the owner of the target can issue long-lived negative responses while performing whatever other exploration and/or mitigation they deem fit.

If the target of the ANAME disappears, the owner of the ANAME will
similarly be able to recognize the problem and deal with it.  If the
administrator of the name owning the ANAME is concerned about downtime
due to misconfigurations by the target, then that administrator can
either select a different target (presumably by selecting a different
service provider) or set their TTLs appropriately to not be subject to
the potential issue you identified above.

However, if the spec requires preserving the target in the DNS despite
the administrator of the target zone removing it, then that is a path
for abuse by the administrator of the zone containing the ANAME, and
the owner of the target will have no recourse.  This is what I meant
by my reference to serve-stale.

---

From: Tony Finch <[email-masked]>
Date: Fri, 12 Apr 2019 12:05:24 +0100

> sibling records indefinitely seems like serve-stale gone horribly

It's worth noting that Oracle's ANAME model does not couple the sibling
addresses to the ANAME target addresses. As I understand it, they have
additional "fallback" infrastructure (web servers and whatnot) which is
used when the ANAME target isn't available.

I'm not sure how this would work as a replacement for CNAME, when the
request from the user comes without any information about how to set up a
fallback server.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Biscay: Variable 3 or less, becoming southeast 3 or 4. Moderate, becoming
slight. Fair. Moderate or good.

---

From: Matthijs Mekking <[email-masked]>
Date: Fri, 12 Apr 2019 13:34:14 +0200

> request from the user comes without any information about how to set up a

I think the logic suggested for ANAME is given this example:

1. Have ANAME and A and AAAA sibling address records.
2. Look up ANAME target A and AAAA target records.
3. If there is no positive answer (SERVFAIL, NXDOMAIN, NODATA) keep
sibling address records.
4. Otherwise replace sibling address records with ANAME target records.

This is different than NS1, that will never replace sibling address
records.  Instead, if there is no sibling address record, it will
perform ANAME resolution.  If that response is a SERVFAIL, NXDOMAIN or
NODATA, that is what will be given to the client (although returning
SERVFAIL won't be a thing in the ANAME specification).

In order to fit both use cases I think we need to relax the steps in the
ANAME resolution logic, but am hesitant to do that: If you make steps
optional it will be unclear what the optional resolver's behavior is
going to do.  I would very much like to see an agreement on the ANAME
resolution logic, especially so that customers can have multiple
providers or switch providers and can expect the same thing in both places.

Best regards,

Matthijs

---

From: Richard Gibson <[email-masked]>
Date: Fri, 12 Apr 2019 11:47:31 -0400

> the ANAME.  You're just presupposing that will be a common
I am indeed claiming that will be a common configuration, and I have 
access to data supporting that claim from existing use of Oracle+Dyn 
ALIAS functionality. Also, please note that those "will" statements are 
properly understood in the context of the examples they are describing.
> expense of people who configure it properly seems like a bad path to
You're pretty much making my point... it is a bad path to design a 
protocol around people who misconfigure their [ANAME-targeted] DNS at 
the expense of people who configure [ANAMEs with static sibling records] 
properly.
> ANAME siblings: when the target of the ANAME is removed from the DNS.
That would take the ANAME-owning domain offline, rather than supporting 
it with its static A and/or AAAA records. How exactly is that beneficial?
> service provider)
It seems awfully insensitive to bake into a protocol an unnecessary 
requirement of its users for all-or-nothing commitment to external 
service providers.
> or set their TTLs appropriately to not be subject to
At the unnecessary expense of reducing cache lifetime (and therefore 
undesired traffic) for /all/ negative responses, rather than just those 
associated with the ANAME.
> by my reference to serve-stale.
The spec requires nothing like "preserving the target in the DNS", with 
or without my proposed changes. The abuse path you mention is already 
present with CNAME, and mitigable by owners of ANAME targets in exactly 
the same way—increase negative caching TTL (and unlike the above, this 
is a scenario where it /would/ make sense to increase it broadly rather 
than for specific RRSets).

---

From: Joe Abley <[email-masked]>
Date: Fri, 12 Apr 2019 13:37:04 -0400

> the ANAME.  You're just presupposing that will be a common

You can't tell how long to cache a negative answer for unless you know data from closest-enclosing SOA RRSet (the RRSet TTL and the SOA.MINIMUM field). In this case potentially there are two SOA RRSets to consider, the one enclosing the ANAME and the one enclosing the ANAME target, and there's the additional complication that the ANAME RR also has a TTL.

I don't see text in the working draft (the one on github) that makes the expected behaviour clear in the case that Richard outlined. I think it makes sense to specify this carefully, otherwise there is a risk of negative answers being cached for longer than expected, which might be especially bad in the case (perhaps the expected use case) where availability and deterministic cache behaviour are kind of important.

I have not been following this closely enough, quite possibly.

Joe

---

From: Richard Gibson <[email-masked]>
Date: Fri, 12 Apr 2019 14:46:00 -0400

In further support of preserving sibling records when target chasing 
comes back negative, I'd like to further explore my offhand mention of 
"A and/or AAAA records".

For a domain owner wanting to use a currently IPv4-only service provider 
(names withheld) while still supporting IPv6, the logical approach would 
be configuring an ANAME record with one or more AAAA siblings. But that 
won't work if the AAAA NODATA response from chasing the ANAME target is 
allowed to override those siblings with nothing. This seems to force a 
needless choice between abandoning the provider or abandoning IPv6.

---

From: Richard Gibson <[email-masked]>
Date: Fri, 12 Apr 2019 14:52:00 -0400

> providers or switch providers and can expect the same thing in both places.
The strategy of never replacing siblings seems ill-advised to me, but 
not really harmful because it's functionally equivalent to being 
ANAME-ignorant and therefore matches the behavior of all resolvers and 
nearly all authoritative servers currently on the Internet, and also the 
behavior of future resolverless ANAME-aware authoritative servers.