Subject: [DNSOP] Caching of negative zone (non-authoritative) responses

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 16:42:39 +0000

This is something that has bugged me for a long time, and I'd love to see a good solution to.

If a record is requested from an authoritative server, where the zone exists, but the records does not exist, the negative response is cached for <minimum> period of time.

If a record is requested from an authoritative server, where the zone does not exist, generally the response is REFUSED, but *this is not cached* by the requesting server. This results in a nearly continuous stream of retries, which continue to result in the same response. Our authoritative servers see no less than 15%, and sometimes as much as 25% of our worldwide traffic as these non-authoritative responses.

There needs to be a means to signal to a recursive server that it should not requery a REFUSED response for a specified period of time. Given that these responses to not have ANSWER records to put a TTL on, return a (new) EDNS record?

Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 12:50:49 -0400

> If a record is requested from an authoritative server, where the zone does not exist, generally the response is REFUSED, but *this is not cached* by the requesting server. This results in a nearly…

A zone that doesn’t exist is actually a name that doesn’t exist under the enclosing zone that does exist, which may be the root zone.  Are you saying that if I look up a name that is a subdomain of a name that doesn’t exist, that is handled differently than a name that is a subdomain of a name that is a zone, or something different?   I’m not disputing the observed behavior—I’m just not clear on what that is.

To put it another way, if you get a REFUSED from a server, that server is not authoritative for the name that you requested.   Is the situation that you have a delegation from one server to another where the other is not actually configured to be authoritative for the delegated zone?   If so, that is indeed an interesting conundrum.

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 17:04:39 +0000

On 7/8/19 9:50 AM, Ted Lemon wrote:
> Notice: This email is from an external sender.
> 
>  
> 
> On Jul 8, 2019, at 12:42 PM, Michael J. Sheldon <[email-masked]
> <mailto:[email-masked]>> wrote:

> To put it another way, if you get a REFUSED from a server, that server
> is not authoritative for the name that you requested.   Is the situation
> that you have a delegation from one server to another where the other is
> not actually configured to be authoritative for the delegated zone?   If
> so, that is indeed an interesting conundrum.

This is exactly the situation. A domain owner has discontinued their
services, but left the domain pointing to our DNS Servers, or sometimes,
just pointed to us for no apparent reason.

There is no mechanism for Authoritative DNS Server owners to have lame
delegations removed by the registries, so I either have to put up with
the continuous query/retry traffic, or I have to actually create a zone
just so there's a means to return NXDOMAIN with a TTL. Neither solution
is good, and the second one, while probably justifiable, does not feel
"legit" to me, and results in longer-term data maintenance issues.

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 13:13:08 -0400

On Jul 8, 2019, at 1:04 PM, Michael J. Sheldon <[email-masked]> wrote:
> Neither solution

So this is a former customer who stopped paying but still has a valid registration?   This seems like it would be straightforward to automate.  I think it’s legit to configure your server to answer authoritatively for the zone as long as the delegation exists.

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 17:27:21 +0000

On 7/8/19 10:13 AM, Ted Lemon wrote:
> On Jul 8, 2019, at 1:04 PM, Michael J. Sheldon <[email-masked]
> So this is a former customer who stopped paying but still has a valid
> registration?   This seems like it would be straightforward to automate.
>  I think it’s legit to configure your server to answer authoritatively
> for the zone as long as the delegation exists.
> 

I agree it's somewhat legit to answer for it, but it's a literal
maintenance nightmare when you're dealing with a very large number of
zones. Things like that tend to get put in place, then never removed.

And it still leaves the issue that recursives should not just keep
hammering the lame delegations when they've gotten a REFUSED response.
That is a definitive legitimate response, and should be honored for a
reasonable period of time.

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jul 2019 17:56:49 +0000

On Monday, 8 July 2019 16:42:39 UTC Michael J. Sheldon wrote:
> ...
> 
> If a record is requested from an authoritative server, where the zone does
> not exist, generally the response is REFUSED, but *this is not cached* by
> the requesting server. This results in a nearly continuous stream of
> retries, which continue to result in the same response. Our authoritative
> servers see no less than 15%, and sometimes as much as 25% of our worldwide
> traffic as these non-authoritative responses.
> 
> There needs to be a means to signal to a recursive server that it should not
> requery a REFUSED response for a specified period of time. Given that these
> responses to not have ANSWER records to put a TTL on, return a (new) EDNS
> record?

i've always sent back SERVFAIL when the zone isn't loaded, on either a primary 
or secondary (authoritative, that is) server. and i cache SERVFAIL on the 
recursive/iterative side with a holddown timer equal to the negative TTL 
interval (SOA.MINIMUM).

but i didn't realize that the standard doesn't say to do this, until i read 
the above.

-- 
Paul

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 13:59:57 -0400

> zones. Things like that tend to get put in place, then never removed.

Right, but as I said, it’s dead easy to automate.   It’s only a maintenance nightmare because you haven’t done that automation.   I could write you a python script to do it in a week.   And registrations cost money, which means that they will go away on their own.

> reasonable period of time.

That’s not a solvable problem.  The right solution to this problem is actually to set up a way that the operator of an auth server can signal to the registrar that the delegation is refused.   If you want to spend some effort on process, that’s the place I’d suggest putting it.   Getting every resolver on the Internet to stop doing something that every resolver on the Internet currently does is not likely to actually change what you experience as an operator in the relatively short term.

Also, of course, you mentioned that it didn’t seem legit to respond authoritatively, but if we take REFUSED as an authoritative answer, then that’s not legit either.

BTW, it would also be perfectly legitimate for an authoritative server that doesn’t provide recursion to respond with NXDOMAIN for any query within a domain that’s delegated to it, and this would be quite easy to implement, although perhaps harder than my python script.   Just do the lookup for the delegation, and if it points to you, respond authoritatively.

Although now that I think about this, another problem we’d face here is that DNSSEC would break this completely.   If you have a secure delegation, but don’t have the ability to sign the zone, you can’t respond authoritatively even if the delegation is pointing at your server.   So here the only real fix is at the registrar.

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 18:02:32 +0000

> but i didn't realize that the standard doesn't say to do this, until i read

BIND returns REFUSED, so that's what I've always used, for maximum
consistency/compatibility.

What SOA.minimum are you returning? Which SOA? And on what record would
it be returned? The issue is that there is no matching zone.

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 14:05:56 -0400

The parent zone TTL would work fine. 

Sent from my iPhone

> On Jul 8, 2019, at 2:02 PM, Michael J. Sheldon <[email-masked]> wrote:
> 
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 18:11:33 +0000

> The parent zone TTL would work fine.

What parent zone???
. ?
com. ?

I'm not authoritative for those. Any response I send for the parent
should be ignored completely.
And it still leaves the issue that I can't return a TTL without a
record, and I don't have a record to return it on.

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Mon, 8 Jul 2019 18:12:01 +0000

On 7/8/19 10:59 AM, Ted Lemon wrote:
> BTW, it would also be perfectly legitimate for an authoritative server
> that doesn’t provide recursion to respond with NXDOMAIN for any query
> within a domain that’s delegated to it, 

But again, since you have no SOA to return, you have no record to attach
a TTL to, so retries, retries, retries.

> Although now that I think about this, another problem we’d face here is
> that DNSSEC would break this completely.   If you have a secure
> delegation, but don’t have the ability to sign the zone, you can’t
> respond authoritatively even if the delegation is pointing at your
> server.   So here the only real fix is at the registrar.

Not gonna argue that the best fix is at the registrar. As an owner of a
nameserver, I really should have the right to say "no, you can't point
that to me."

But there's other reasons for REFUSED as well (security, ddos filtering,
etc) and I should have a means to add to the reply, "don't ask again for
X seconds"

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 14:37:11 -0400

> record, and I don't have a record to return it on.

In the case of a delegation, I think you respond as if the zone still exists.   You are definitely authoritative for that zone—that’s what a delegation means.   Responding with REFUSED doesn’t solve your problem, so doing that isn’t the right move.

> X seconds”

I don’t think there’s a way to do this that will work, because DNSSEC.   If you don’t have the key for the zone, you can’t reply with a signed answer.   This also makes my solution above fail.   Of course, you can do it for un-signed zones, which are the majority, but if we want a real solution, it’s going to require some new work.   And if it requires updating all recursive resolvers, you aren’t going to see any benefit from it this decade.   And since your proposed solution doesn’t work with DNSSEC, it again doesn’t work this decade.

I think if we really wanted a solution to this problem that was secure, we’d need an analog of the DS record, only for the resolver.  This would have to exist at the delegation point, and would be owned by the server operator, not the domain owner.   A response signed by the owner of the name would be taken to indicate that the delegation is no longer valid, and would result in deferred retries for the lifetime of the zone.   Much handwaving, etc., but you’d need something like this in order to securely repudiate a delegation without having the ZSK or the KSK of the delegated zone.

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jul 2019 18:38:48 +0000

On Monday, 8 July 2019 18:02:32 UTC Michael J. Sheldon wrote:
> > i've always sent back SERVFAIL when the zone isn't loaded, on either a
> > primary or secondary (authoritative, that is) server. and i cache
> > SERVFAIL on the recursive/iterative side with a holddown timer equal to
> > the negative TTL interval (SOA.MINIMUM).
> > 

REFUSED means, in my reading (and coding) that there is no zone declaration at 
the authority. SERVFAIL means the zone is declared/configured, but not loaded. 
i now realize that both have to have a holddown timer, not just SERVFAIL.

> it be returned? The issue is that there is no matching zone.

closest enclosing zone, on the assumption that the authority who sent me the 
SERVFAIL (or REFUSED) may be an ancestor of the missing or not-loaded zone. if 
this means the SOA.MINIMUM of the root priming metadata, then so be it.

this timing information isn't returned with the SERVFAIL or REFUSED -- as you 
say, those don't have records, so there's noplace to put a negative TTL. 
however, my negative cache has a holddown timer for SERVFAIL, just to suppress 
query storms. i think i need to add one for REFUSED. the idea of a holddown 
timer is, within that interval, you assume that the error is still present, 
and so the iterator just does what it would do if it asked the same question 
and got the same answer.

-- 
Paul

---

From: Mark Andrews <[email-masked]>
Date: Tue, 9 Jul 2019 07:34:05 +1000

The problem here is that there is no post delegation maintenance occurring.  If you are no longer under contract to serve the zone you should be able to request the NS records pointing to your servers be removed.  If the zone operator continues to list your servers in the zone you are able to request that the entire delegation be removed with documented attempts to correct the delegation first.

There where checks and balances built into the system. See RFC 1033. The problem is that people are ignoring those checks and balances.

Mark

> On 9 Jul 2019, at 2:42 am, Michael J. Sheldon <[email-masked]> wrote:
> 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Mon, 8 Jul 2019 18:15:10 -0400

There’s a reason people are ignoring them. There’s no mechanism for validating such change requests nor checking that they are authorized. RFC 1033 was written in a rather different time. 

Sent from my iPhone

> On Jul 8, 2019, at 5:34 PM, Mark Andrews <[email-masked]> wrote:
>

---

From: Mark Andrews <[email-masked]>
Date: Tue, 9 Jul 2019 14:00:33 +1000

Actually if a DNS operator is requesting that NS records pointing to them be removed then the TLD only need to look at the enclosing SOA of NS’s address records to find a valid contact.

As was pointed out to ICANN, the DNS operator is a party that needs to be listened to, and yes they have acknowledged that publicly.  This was with respect to .CM (delegation) and RIPE NCC (operator) and cm.cctld.authdns.ripe.net.  They are still working through the details of how to correct it.

See the thread https://lists.dns-oarc.net/pipermail/dns-operations/2019-June/018816.html

I’m also pretty sure the Courts would also agree that pointing NS records to operators that don’t want the requests is something that should be corrected by all parties involved in the delegation (registrant, registrar, registry).  I’d actually like to see some case law here.  Parent zone operators might stop this BS that they can’t take action when it has been expected from the very beginning.  Yes, there is a chance that the DNS operator would be in breach of contract by requesting the removal, but a little bit of due diligence on behalf of the registry should be able to detect that and result in push back to the Courts.

If you are already returning REFUSED when you request the removal of the NS record pointing to yourself you can’t fake the SOA record of the zone in question.

Mark

> On 9 Jul 2019, at 8:15 am, Ted Lemon <[email-masked]> wrote:
>

---

From: Shane Kerr <[email-masked]>
Date: Tue, 9 Jul 2019 10:46:06 +0200

Paul,

Minor nit, just to be pedantic.

> i now realize that both have to have a holddown timer, not just SERVFAIL.

ServFail means "something bad happened". There are 10 possible reasons 
listed in the extended errors draft:

https://datatracker.ietf.org/doc/draft-ietf-dnsop-extended-error/

And there are of course many, many more. So it _could_ be zone is 
declared/configured, but not loaded, but not necessarily.

This doesn't change your perfectly correct observation that a hold-down 
timer is needed for both Refused and ServFail.

Cheers,

--
Shane

---

From: Ted Lemon <[email-masked]>
Date: Tue, 9 Jul 2019 08:53:12 -0400

> Actually if a DNS operator is requesting that NS records pointing to them be removed then the TLD only need to look at the enclosing SOA of NS’s address records to find a valid contact.

And how do they validate that any communication that follows is actually with that contact?

---

From: Tony Finch <[email-masked]>
Date: Tue, 9 Jul 2019 15:07:07 +0100

Michael J. Sheldon <[email-masked]> wrote:
>
> If a record is requested from an authoritative server, where the zone
> does not exist, generally the response is REFUSED, but *this is not
> cached* by the requesting server. This results in a nearly continuous
> stream of retries, which continue to result in the same response. Our
> authoritative servers see no less than 15%, and sometimes as much as 25%
> of our worldwide traffic as these non-authoritative responses.

Yuck :-(

BIND's default lame-ttl is 10 minutes; I don't know if other resolvers
have a similar feature. It might be better from your point of view if the
lame-ttl matched the delegation TTL, but I bet that would be a bit
frustrating for operators who set up a new delegation in the wrong order!

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Dogger: Variable 2 to 4, becoming south 3 to 5. Slight. Rain. Good, becoming
moderate or poor.

---

From: "Michael J. Sheldon" <[email-masked]>
Date: Tue, 9 Jul 2019 15:49:20 +0000

> frustrating for operators who set up a new delegation in the wrong order!

10 min seems pretty reasonable to me. Allows for a reasonably short
delay where someone didn't set up the DNS zone before the delegation,
but prevents the hammering of a server with a true lame delegation.

While in the case of a true bad delegation, I wouldn't mind the typical
48 hour TTL from the TLDs, It seems a bit punitive for the guy who just
forgot to do a restart of his server before changing delegation.

-- 
Michael Sheldon
Dev-DNS Services
GoDaddy.com

---

From: Mark Andrews <[email-masked]>
Date: Wed, 10 Jul 2019 09:58:39 +1000

> On 9 Jul 2019, at 10:53 pm, Ted Lemon <[email-masked]> wrote:
> 

They email the address and ensure they get back something unique from that email.

1) Check the NS is returning REFUSED for the delegated zone.
2) Email the SOA contact with a unique confirmation URL with a validity interval.
3) When the URL is clicked remove the NS record from the delegation.

Optionally allow for confirmation via email.

If you want to check with delegated zone’s administrators do that between steps 1 and 2.

If you are worried about the SOA contact being forged require that the SOA record be signed
and that it validates as secure.

The DNS is a good enough introducer especially when it is signed.

Mark

---

From: Ted Lemon <[email-masked]>
Date: Tue, 9 Jul 2019 21:06:10 -0400

Yes, something like that could work, but you’d have to document it. 

Sent from my iPhone

> On Jul 9, 2019, at 7:58 PM, Mark Andrews <[email-masked]> wrote:
> 
> 
>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 22 Jul 2019 17:15:45 -0400

On Mon, Jul 08, 2019 at 05:27:21PM +0000,
 Michael J. Sheldon <[email-masked]> wrote 

> hammering the lame delegations when they've gotten a REFUSED

Sorry for being late in the discussion but
draft-ietf-dnsop-extended-error-06 has a code to make REFUSED crystal-clear:

4.4.1.  REFUSED Extended DNS Error Code 1 - Lame

   An authoritative server that receives a query (with the RD bit clear)
   for a domain for which it is not authoritative SHOULD include this
   EDE code in the SERVFAIL response.  A resolver that receives a query
   (with the RD bit clear) SHOULD include this EDE code in the REFUSED
   response.  Implementations should set the R flag in this case
   (another nameserver or resolver might not be lame).