Subject: Re: [DNSOP] I-D Action: draft-ietf-dnsop-serve-stale-03.txt

From: Dave Lawrence <[email-masked]>
Date: Fri, 1 Mar 2019 12:33:14 -0500

Bob Harold writes:
> Will the "resolution recheck timer" cause ttl's less than the timer
> to be effectively lengthened, by refusing to look them up again?  I
> think 'serve-stale' should focus on the situation where the auth
> server is not available, and not change the handling of short ttl's.
> Or am I mis-reading that?

Mildly misreading, though it does point out that we could make it
clearer that this is akin to existing mechanisms that resolvers have
to cache that an authority is problematic (eg, RFC 2308 Section 7).
It is NOT about the feature that some resolvers also have, a minimum
cache TTL, which is what you're concerned about and we don't intend to
address in this document.  As you say, and unfortunately we somehow
managed to not make clear, the purpose is only to deal with failures
to refresh authoritative data.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 1 Mar 2019 17:50:49 +0000

> to refresh authoritative data.

FWIW, I read it like Bob did. The following paragraph from the current draft makes it clear that the draft is proposing that the resolution recheck timer be lengthened to at least 30 seconds.

   If iterative lookups will be, done then the resolution recheck timer
   is consulted.  Attempts to refresh from the authorities are
   recommended to be done no more frequently than every 30 seconds.  If
   this request was received within this period, the cache may be
   immediately consulted for stale data to satisfy the request.

I'm not sure a standards track document that updates RFC 1034/1035 should be recommending a minimum TTL.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 1 Mar 2019 19:51:01 +0000

Following up on my previous message:

The document is actively confusing about recommendations.

- Section 4 has the actual update to the RFC 1035, and that update contains MAY and SHOULD statements.

- Section 5 is called "Example Method" but also contains recommendations.

- Section 6, "Implementation Caveats" also has recommendations. A subsection, "6.1. Implementation Considerations" has more recommendations.

Proposal: Put all recommendations in Section 4, and talk about them (instead of introducing them) in the other sections. That way, a lazy developer who only reads Section 4 will know all the recommendations.

--Paul Hoffman

---

From: Dave Lawrence <[email-masked]>
Date: Fri, 1 Mar 2019 15:54:39 -0500

> should be recommending a minimum TTL. 

As previously noted, we're making no such recommendation and that will
be clarified.  The first definition of "resolution recheck timer" in
section 5 does already say that it regards failed lookups, but it
seems that adding that distinction later is also warranted.

> The document is actively confusing about recommendations.

Before we go pushing around whole sections of text, could anyone
please comment on whether they find it "actively confusing about
recommendations"?  Yes, they appear in a couple of different sections,
for different reasons -- as described in the text the example method
presented is just one possible way of doing things, with the core idea
and the standards-relevant changes being distinct.

The recommendations, however, are not in conflict with each other and
I'm really not clear myself on where the criticism that they are
confusing comes from.  The mention of the explicit word "recommended"
in section 6/6.1 is just reiterating the value described in section 5,
and in the broader sense of recommendations section 6.1 is meant as
how the recommendations were made.  (Personally I'd renumber 6.1 -> 7,
7 -> 8, etc, but  that's a minor separate issue.)  Putting that
discussion in a separate discussion doesn't bog down the basic
description of the sample implementation.

I don't object to reorganization in principle, if it makes notable
gains, but I can't really fix what I don't understand the problem is.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 1 Mar 2019 21:10:57 +0000

> On Mar 1, 2019, at 12:54 PM, Dave Lawrence <[email-masked]> wrote:
> 
> Paul Hoffman writes:
> be clarified.  

"Attempts to refresh from the authorities are recommended to be done no more frequently than every 30 seconds" feels like a recommendation because of the words "are recommended".

> confusing comes from.  

What is confusing is whether or not they are recommendations. If they are recommendations, why don't they appear in Section 4 with SHOULD and MAY languages? If they are not, how is an implementer meant to read them?

> (Personally I'd renumber 6.1 -> 7,
> 7 -> 8, etc, but  that's a minor separate issue.)

+1

--Paul Hoffman

---

From: Tim Wicinski <[email-masked]>
Date: Sun, 3 Mar 2019 20:33:48 -0500

"Proposal: Put all recommendations in Section 4, and talk about them
(instead of introducing them) in the other sections. That way, a lazy
developer who only reads Section 4 will know all the recommendations."

I totally agree here.  It also makes it easier if the document passed WGLC
and moves along the standards process train.

Tim

On Fri, Mar 1, 2019 at 2:51 PM Paul Hoffman <[email-masked]> wrote:

> - Section 6, "Implementation Caveats" also has recommendations. A
> subsection, "6.1. Implementation Considerations" has more recommendations.
>
> Proposal: Put all recommendations in Section 4, and talk about them
> (instead of introducing them) in the other sections. That way, a lazy
> developer who only reads Section 4 will know all the recommendations.
>
> --Paul Hoffman
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Holger Freyther <[email-masked]>
Date: Mon, 4 Mar 2019 06:19:06 +0000

> On 23. Feb 2019, at 18:54, [email-masked] wrote:
> 

> There are also htmlized versions available at:
> https://tools.ietf.org/html/draft-ietf-dnsop-serve-stale-03
> https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-serve-stale-03

Section 5:

   If so, it adds that data to the response message; it MUST
   set the TTL of each expired record in the message greater than 0,
   with 30 seconds recommended.

Can we recommend to return a value randomly from an interval to add jitter instead? This is a common mitigation to a "thundering herd".

Section 8:

I might have missed some of the discussion but has the point of view of authoritative servers been considered?

What I really liked about a resolver signaling its' intent to stale serve to an authoritative nameserver is that it allows to (dynamically) lower the TTL. This allows an operator to react more quickly to changes without decreasing the resilience by relying on the resolver to cache. It seems equally attractive to both CDNs and "hobbyist" deployments.

Without signaling operators of authoritative nameservers either need to (non exhaustive):

a.) Keep the trade-off between nameserver unavailability and rate of change.
b.) Discriminate based on the resolver address
c.) Wait for further large scale experiments to determine the adoption rate of stale serving.

---

From: Puneet Sood <[email-masked]>
Date: Mon, 4 Mar 2019 18:24:57 -0500

Tim, Paul,

Thanks for the feedback. Working on clarifying the recommendations.

-Puneet

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 4 Mar 2019 21:21:42 -0500

can I ask, what happens when a domain is intentionally down though? For
instance, take .eg... ~4yrs back? (maybe 5?) Someone requested that the
master/shadow NS go down, hard. All public auth servers eventually (in a
day or so) went dark too.

If someone is 'ordered' to make a zone dark, there may be reasons for that
action, and real penalties if the request is not honored.
Is this draft suggesting that the DNS operations folk go against the wishes
of the domain owner by keeping a domain alive after the auth servers have
become unreachable? How would a recursive resolver know that the auth is
down: "By mistake" vs: "By design" ?

-chris

On Mon, Mar 4, 2019 at 6:25 PM Puneet Sood <puneets=
[email-masked]> wrote:

>> I totally agree here.  It also makes it easier if the document passed
>> WGLC and moves along the standards process train.
>>
>
>> Tim
>>
>>

---

From: Paul Vixie <[email-masked]>
Date: Tue, 05 Mar 2019 02:26:05 +0000

> day or so) went dark too.

i already raised that question, very far up-thread. got no answer.

> down: "By mistake" vs: "By design" ?

this the essence of the argument against utility for this entire proposal. no 
data should be served beyond its TTL unless some new leasing protocol is first 
defined, to obtain permission and to provide a cache invalidation method.

vixie

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 4 Mar 2019 21:36:09 -0500

> i already raised that question, very far up-thread. got no answer.
apologies for repeat-asking :( whoops! It'd still be good to hear an answer.
I can think of simple product and not 'life safety' reasons that I might
darken a zone as well, so if you don't want to answer for an abundance of
caution for your fellow folken... think of the product marketing folk who
mistakenly launch a day early, or who would like to retire a service in a
clearly delineated manner.

:)

> defined, to obtain permission and to provide a cache invalidation method.

this is what I was thinking as well. thanks for more clarity.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 5 Mar 2019 14:35:44 +1100

> On 5 Mar 2019, at 1:26 pm, Paul Vixie <[email-masked]> wrote:
> 

And one can to that if we add 2 TTLs to each DNS record. One for total time to
live and one for freshness (old client get this).  It will require EDNS to signal
that multiple TTLs are desired and are present in the response and may require
using the last DNS flag bit to move the OPT record to in front of the question
section to make parsing easier (no trial and error).

Yes, this is radical but it will work and is incrementally deployable.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Paul Wouters <[email-masked]>
Date: Mon, 4 Mar 2019 22:59:58 -0500 (EST)

>>> day or so) went dark too.

 	"If the recursive server is unable to contact the
          authoritative servers for a query "

So make the DNS server reachable, but return ServFail or NXDOMAIN. If
the owner doesn't cooperate and there is legal standing, talk to the
parent to do this for the delegation.

I don't think this draft stops a domain from being brought down
intentionally.

>>> down: "By mistake" vs: "By design" ?

The DNS resolvers who want to accomodate their governments need to
manually override their resolvers anyway with new (forged) data. This
draft does not change that.

If the owner itself wants to bring the domain down, they just need to
make its auth servers reachable.

If the DNS hoster wants to bring it down, they just need to modify the
data it serves resulting in NXDOMAIN, ServFail or 127.0.0.1 A records.

>> defined, to obtain permission and to provide a cache invalidation method.

I don't really follow this reasoning. Are you saying that:
1) if the domain owner wants their domain to be reachable
2) and they have lost their auth servers due to a DDOS attack
3) they might prefer to be down over extending the TTL a bit

In the non-DDOS case, the auth server is reachable and none of the data
is getting additional TTL added:

    Answers from authoritative servers that have a DNS Response Code of
    either 0 (NOERROR) or 3 (NXDOMAIN) MUST be considered to have
    refreshed the data at the resolver.  In particular, this means that
    this method is not meant to protect against operator error at the
    authoritative server that turns a name that is intended to be valid
    into one that is non-existent, because there is no way for a resolver
    to know intent.

Although perhaps it should also explicitely state this regarding
ServFail ?

> Yes, this is radical but it will work and is incrementally deployable.

See above. It seems a bit overkill for a strange corner case. But even
so, one could specify the maximum ever allowed TTL to be like 3 days? Which
I think is kind of enforced by most DNS resolvers anyway?

Adding more TTLs would just make this more complicated and more error
prone and not lead to reduced outage times which is the goal of this
draft.

I don't think the "4 years" is a realistic problem case.

I can see how people want to get a few hours or a few days of usage
beyond the TTL to accomodate for errors. Although, it is likely that
moving up the error this way will also delay the error from being
detected before the extra time has expired, and we are just moving
the goal post with no effective gain. But in the case of a DDOS
attack, the draft's feature is surely useful.

Paul

---

From: Mark Andrews <[email-masked]>
Date: Tue, 5 Mar 2019 15:33:47 +1100

> On 5 Mar 2019, at 2:59 pm, Paul Wouters <[email-masked]> wrote:
> 

For those with long existing TTLs it would allow them to refresh the records
earlier allowing them to exist in caches longer during the DoS event.

For those with short existing TTLs it would allow them to control how long
the records exist past reachability of the DNS servers.  Most of the time
short lived TTLs are there “just in case we need to change them" or they
are doing short term load balancing between a set of servers which are
actually at stable addresses.  This would allow those functions to continue
but keep the addresses in caches longer under error conditions.

> attack, the draft's feature is surely useful.

Monitoring will detect problems.

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 4 Mar 2019 23:52:04 -0500

> parent to do this for the delegation.
this wasn't, near as I could tell, an option for the ccTLD in question.
They lost their IP access, and were 'required' to disable their dns zone,
their master was down hard from the internet's perspective,
keeping anything else up wasn't really an option.

I don't know how long it takes to get ICANN to 'do something creative' for
the root zone, but I'm guessing this isn't always feasible in normal
timelines (hours to a day or so).

> I don't think this draft stops a domain from being brought down
i think it prolongs the data being available in a bunch of the cases I can
imagine/have-seen.
I think there's at least one case where it's likely grave harm could have
come to the dns operator. :(

I think what the draft does is attempt to guess at WHY a thing is changed,
without an real data to back up the reasoning. this will mean the decision
is wrong more than a small percent of the time.

> make its auth servers reachable.
sometimes that's not possible :( and the expectation is that 'when the
right ttl sets expire all of our records disappear as you requested"

> data it serves resulting in NXDOMAIN, ServFail or 127.0.0.1 A records.

this is also not always possible :(

> I don't think the "4 years" is a realistic problem case.
is the '4 years' here in reference to the .eg problem?
in my example the '4 yrs' was: "when the incident happened" not "please
keep my dns alive for 4 yrs without talking to me"

> attack, the draft's feature is surely useful.
seems unlikely to be useful... even in the case of dos...really.
(to me anyway)

-chris

---

From: Paul Wouters <[email-masked]>
Date: Tue, 5 Mar 2019 09:25:02 -0500 (EST)

On Mon, 4 Mar 2019, Christopher Morrow wrote:

> They lost their IP access, and were 'required' to disable their dns zone, their master was down hard from the internet's perspective,

We should not base our DNS domains on scenarios where a DNS administrator
loses access to all their IP addresses and therefor could not operate
the DNS as required.

> i think it prolongs the data being available in a bunch of the cases I can imagine/have-seen.
> I think there's at least one case where it's likely grave harm could have come to the dns operator. :(

Grave harm beyond being already completely disconnected from the
internet? I think that kind of harm is not something we can address
in protocol design.

> I think what the draft does is attempt to guess at WHY a thing is changed, without an real data to back up the reasoning. this will mean the decision is wrong
> more than a small percent of the time.

You will have to provide a few scenarions that we can actually evaluate
to back that statement. To me, if your presence has been basically cut
from the internet, then this is no longer an internet protocol issue.

> sometimes that's not possible :( and the expectation is that 'when the right ttl sets expire all of our records disappear as you requested"

If your servers are connected to the internet, that remains the case.
The case we seem to talk about is only when all authoritative servers
are down. If you get non-technical pressure to bring things down without
getting TTL extensions, explain to those cutting you of that they can
get a shorter disconnect time by letting you run a modified DNS
configuration on your servers?

> this is also not always possible :(

Sorry, again you have to substantiate your answer here. Make up an
imaginary story to illustrate this if you cannot use a real example.

> is the '4 years' here in reference to the .eg problem?

I have no idea what the ".eg problem" is.

> in my example the '4 yrs' was: "when the incident happened" not "please keep my dns alive for 4 yrs without talking to me"

Ahh.

> seems unlikely to be useful... even in the case of dos...really.

If I need a few hours up to a few days to try and work around a DDOS
attack by adding capacity or some external provider to help me deal
with the DDOS, then buying those few hours are what is keeping a domain
online during such an attack. So while that might not apply to you, it
is an actual real use case.

Paul

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 5 Mar 2019 10:57:11 -0500

> i already raised that question, very far up-thread. got no answer.

While my answer to it was significantly late for when the query was
made, and for that I have previously and will again apologize, it was
responded to:

https://mailarchive.ietf.org/arch/msg/dnsop/2o62EmB35OiBKNq9YZLfSMpqL2U

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 5 Mar 2019 10:59:44 -0500

> Although perhaps it should also explicitely state this regarding

I personally have a very strong opposition to including servfail.
Servfail is an extremely clear indication that the authority that was
contacted is having some sort of structural problem.  It is a very
distinct condition from being told by the authority that the name does
or does not exist.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 5 Mar 2019 16:18:55 +0000

> distinct condition from being told by the authority that the name does

I agree with David on this. This has been clear since RFC 1035.

--Paul Hoffman

---

From: Paul Wouters <[email-masked]>
Date: Tue, 5 Mar 2019 12:06:58 -0500 (EST)

> I agree with David on this. This has been clear since RFC 1035.

I am a bit confused here. The goal of the draft is to keep data past the
TTL in case you cannot reach the authoritative servers during a DDOS
attack.

If you reach the authoritiatve server, and it gives ServFail, you did
reach the server. You might have "structural problems" but not in the
way of a denial of service attack. Unless you are saying DNS
implementations that are overloaded return ServFail instead of dropping
the packets?

Misconfiguring your authoritative server by removing the zone is not
meant to be covered by this draft if I understood it correctly. If it
is, then introduction will need to add text to cover that use case.

Paul

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 5 Mar 2019 12:39:17 -0500

So, sorry I added an example set and we rat-holed on those.

My point is that the recursive reoslver has no idea why an authoritative
is unreachable and that doing anything like sending stale records is
going to cause unintended problems.

-chris

<elided all previous to avoid confusion.. which i caused, sorry>

---

From: Joe Abley <[email-masked]>
Date: Tue, 5 Mar 2019 12:50:42 -0500

On 4 Mar 2019, at 23:52, Christopher Morrow <[email-masked]> wrote:

> I don't know how long it takes to get ICANN to 'do something creative' for the root zone, but I'm guessing this isn't always feasible in normal timelines (hours to a day or so).

The IANA created an official, supported mechanism for emergency changes to the root zone back in 2010, as part of the project to deploy DNSSEC. The goal was to accommodate the needs of TLD managers to do quick changes to DS RRSets in the event that some bad signing thing happened. Even without that emergency provision, there were examples way back when of out-of-cycle changes were pushed through by the root zone maintainer (e.g. a third serial in a single day) because of some operational concern. When it comes down to it, all the people involved are operational and are good at what they do.

I think TLDs are a red herring here, though. The TTLs on referral responses from TLD servers tend to be long and there is no shortage of options for diversity and redundancy in the NS set of TLD zones. Developing TLDs that have not yet reached a level to be able to engineer in that kind of diversity tend not to be the TLDs that are relied upon at the scale of those that have (and, I would suggest, serve-stale is not going to save them out from outage anyway). Structural instability in such TLDs is probably better addressed by technical outreach, support and education than by protocol extensions.

Enterprise zones with low TTLs and with reduced options for authority server diversity due to the response-time tricks used to manage their traffic are far more likely to be interested in something like serve-stale, especially if their revenue is closely correlated with being reachable, and especially if they use lots of response-time tricks and want to understand what happens to client traffic when there's a DNS blip. From the other side, resolver operators for whom DNS non-reachability means a support burden have already implement these things. Describing how they work using outside voices seems like a good thing for everybody.

Joe

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 5 Mar 2019 17:51:23 +0000

> going to cause unintended problems.

Those problems were not intended by the authoritative servers. Not sending stale records will also cause problems not intended by the users of the resolver that is serving stale data.

Maybe the draft should add a sentence about this shift of "unintended problems" from authoritative to resolvers in the end of the Abstract, and again in the Introduction. It seems like people in the WG may be missing this shift.

--Paul Hoffman

---

From: Tony Finch <[email-masked]>
Date: Tue, 5 Mar 2019 18:02:01 +0000

> TTL in case you cannot reach the authoritative servers during a DDOS

Right.

There's a tricky interaction between lameness and serve-stale.

Say you have a partially-lame zone, where some servers might have an
expired copy (returning SERVFAIL) and some might not know about the zone
at all (returning REFUSED or referrals to the root). Typically (without
serve-stale) a resolver will react by adding a lame server cache entry and
re-trying other hopefully working servers.

I think serve-stale should only take effect after this point, if a zone
has at least one non-lame server, and all the non-lame servers do not
respond.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
a fair, free and open society

---

From: Tony Finch <[email-masked]>
Date: Tue, 5 Mar 2019 18:13:38 +0000

> has at least one non-lame server, and all the non-lame servers do not

D'oh, sorry, I missed out the important case:

If the zone has a mixture of lame and working servers, the lame servers
should not trigger serve-stale, because the resolver still needs to try
asking the working servers.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Irish Sea: Southwest, veering east for a time, 5 to 7, perhaps gale 8 later.
Slight or moderate, occasionally rough in south. Rain or showers. Good,
occasionally poor.

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 5 Mar 2019 20:45:42 -0500

Paul Wouters writes:
> I am a bit confused here. The goal of the draft is to keep data past
> the TTL in case you cannot reach the authoritative servers during a
> DDOS attack.

There are many different failure modes in operating the DNS and
the goal of this draft has been to accommodate the ones that are clear
failures.  I, for one, have never put forth that it is only about
resiliency against DDoS and don't recall hearing Warren or Puneet say
that either.  It can include when there are other clear errors in
the system, even when self-inflicted by the authoritative operator.

> is, then introduction will need to add text to cover that use case.

I can sort of see how someone might infer from "It is predicated on
the observation that authoritative server unavailability can cause
outages ..." that it means this whole idea is constrained to DDoS, and
presumably you would include as well other network and server outages
not caused by DDoS.  It doesn't only mean that though.  The intention
is that this applies to any inability to get a proper authoritative
response, one which has AA set in a protocol-meaningful way.

This can be edited to be clearer, perhaps as simply as changing
"authoritative server unavailability" to "authoritative answer
unavailability".  We'd be happy to consider alternative text.

Realistically only rcodes NoError and NXDomain apply for being
authoritative answers, each being an explicit assertion regarding the
name/type in the query and legitimately supplanting whatever previous
data was known about that name and type.

ServFail is a clear signal that something is going wrong with the
authoritative server itself has something going wrong.  If you send a
ServFail then AA is completely irrelevant.

REFUSED is slightly murkier as to its exact meaning, thanks to
overloading, but in its most commonly seen usage for lameness
indicates a clear problem with the delegation.  Even in its other use
cases, notably an EDNS Client Subnet error or an actual "I am
authoritative for the name but administratively denying your
resolution of it", I submit that if the resolver has a stale answer
then serving it is reasonable.  In that administrative denial case
it'd be better to issue NXDomain anyway, which is exactly what split
horizon authorities do.

Other lesser seen rcodes are largely similar in not indicating
anything at all about the legitimacy of the name and whatever data you
might have previously associated with it.  Only the dynamic update
rcodes come close to being relevant, but they are not part of the
resolution process covered by serve-stale.

Despite the unfortunate RFC 1035 nomenclature of NXDomain as "Name
Error" it is called out explicitly because it isn't really an error,
not in the database lookup sense.  There's no way of knowing whether
the NXDomain is happening because of operator fault or the far more
likely case that it just doesn't exist.  That's why it is called out
separately in the doc, with an explicit note about why it has to be
treated as replacing any stale data associated with the name.

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 5 Mar 2019 20:47:41 -0500

> going to cause unintended problems.

Given the operational benefit that has already been observed in
production with serve-stale, I think it needs more than vague fears of
"is going to cause unintended problems" to argue against it.  If we
let unspecified concerns of possible unintended problems be our
guidance we'd never get anything done around here.

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 5 Mar 2019 20:49:09 -0500

> asking the working servers.

Yes, complete agreement.  That is the thrust of what the document says.

---

From: Paul Wouters <[email-masked]>
Date: Tue, 5 Mar 2019 21:55:04 -0500 (EST)

> unavailability".  We'd be happy to consider alternative text.

Ok, then that needs to be clarified in the draft. And you should discuss
exactly which kind of failures are valid for extending the TTL and which
are not and which should still try another auth server.

> treated as replacing any stale data associated with the name.

So put some text similar to this in the draft.

Paul

---

From: Tony Finch <[email-masked]>
Date: Wed, 6 Mar 2019 13:03:28 +0000

Dave Lawrence <[email-masked]> wrote:

> then serving it is reasonable.

This sounds like it will lead to stale answers being given instead of
re-trying other potentially working servers. I think this is wrong, and
it's inconsistent with your other reply, so I am confused.

https://mailarchive.ietf.org/arch/msg/dnsop/HIUK2ME8uHbA-cwztnrNVYRtqLc

I think serve-stale should only cover cases where servers are unreachable
or unresponsive.

If all a zone's servers start to reply REFUSED, that's a deliberate
decision to disable the zone, and resolvers should not try to keep it
alive beyond its TTL. (This is important for the take-down situations that
Paul Vixie is concerned about.)

I'm more ambivalent about the SERVFAIL case but it seems simpler to treat
it the same as REFUSED, i.e. server is working but not for this zone.

There's a big difference between the RFC 4697 (Observed DNS Resolution
Misbehavior) section 2.2 (Repeated Queries to Lame Servers) lame server
cache time of 30 minutes and the serve-stale retry time of 30 seconds,
which makes me think the serve-stale spec should explicitly update RFC
4697.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Lyme Regis to Lands End including the Isles of Scilly: South, veering west or
northwest 5 to 7, occasionally gale 8 later. Moderate or rough, becoming very
rough for a time in far west. Thundery showers. Good, occasionally poor.

---

From: Joe Abley <[email-masked]>
Date: Wed, 6 Mar 2019 09:37:35 -0500

> I think serve-stale should only cover cases where servers are unreachable

That phrase sounds concise and definitive, but it's not really.

How recently should a server have been checked and found not to respond to conclude that it's unresponsive? What does unresponsive mean? Presumably this involves a timeout; how long? Perhaps these questions are already answered in practice by existing implementations, but I don't know that they are written down anywhere.

If a server responds to some queries but not others (e.g. same QNAME, different QTYPEs) is that unresponsive across the board? Or does responsiveness depend on the precise (QNAME, QCLASS, QTYPE) tuple?

This is not only a pedantic, annoying observation (you're welcome) but I think the last question provides an attack vector; if you can find a set of DNS authority servers that silently discards a particular kind of query, sending such queries through resolvers that are known to support serve-stale might suppress other queries and trigger the serve-stale behaviour even though the authority servers are not actually unresponsive for them.

I think it's a fair observation that such authoritative servers are broken, but I seem to think they exist e.g. due to the ongoing existence of poorly-conceived middleboxes. If I'm mistaken about that last part then I guess we're back to pedantic and annoying.

Joe

---

From: Daniel Stirnimann <[email-masked]>
Date: Wed, 6 Mar 2019 16:09:48 +0100

On 06.03.19 15:37, Joe Abley wrote:

> if you can find a set of DNS authority servers that silently discards
> a particular kind of query, sending such queries through resolvers
> that are known to support serve-stale might suppress other queries
> and trigger the serve-stale behaviour even though the authority
> servers are not actually unresponsive for them.
I can provide some statistics for this behavior.

There are currently around 2 million delegations in .ch. 568 domain
names are hosted on authoritative name servers where all of them do not
respond to some query types. A little more common is that only some do
not respond to some query types.

I had the feeling that this only facilitates kaminsky style cache
poisoning attacks but your example is indeed another attack vector.

Daniel

---

From: Tony Finch <[email-masked]>
Date: Wed, 6 Mar 2019 15:11:18 +0000

Joe Abley <[email-masked]> wrote:
>
> How recently should a server have been checked and found not to respond
> to conclude that it's unresponsive? What does unresponsive mean?
> Presumably this involves a timeout; how long? Perhaps these questions
> are already answered in practice by existing implementations, but I
> don't know that they are written down anywhere.

I think it's covered in the draft already.

> If a server responds to some queries but not others (e.g. same QNAME,
> different QTYPEs) is that unresponsive across the board? Or does
> responsiveness depend on the precise (QNAME, QCLASS, QTYPE) tuple?

Best response to this kind of brokenness is to tell them to fix it, not
to work around it.

https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
West Forties, Cromarty, Forth: Cyclonic becoming northerly or northeasterly, 5
to 7, occasionally gale 8 in Cromarty and Forth. Moderate or rough, becoming
very rough in Cromarty. Rain, fog patches. Good, occasionally poor or very
poor.

---

From: Dave Lawrence <[email-masked]>
Date: Wed, 6 Mar 2019 11:16:30 -0500

Tony Finch writes:
> re-trying other potentially working servers.

The document is explicit that you need to keep trying to get an
answer, so if an implementation is not retrying other potentially
working servers that is its own defect.

> I think serve-stale should only cover cases where servers are
> unreachable or unresponsive.

You are of course free to write your own implementation that way.
Having worked for operations where the authorities were concerned
about the possibility of accidental ServFails, I know that their
preference is that resolvers would serve-stale then too and enhance
the overall resiliency of the system.

If you think it would help, I can add some text to Implementation
Considerations about this, something like:

   Consider whether serve-stale should kick in for only the case of
   all servers being unresponsive, or whether authoritative servers
   responding with DNS RCODEs other than NoError and NXDomain also
   trigger it.  Some authoritative servers operators would prefer
   stale answers to be used in the event of their server failures,
   while other implementers see any answer from the authoritative
   server as being sufficient indication that any previously available
   answer for the question is superseded.

The implications of that are a transition from good answers to
failure answers to unavailable means that the stale answers will never
be available when they otherwise could have been, but so be it.

> alive beyond its TTL.

You cannot know that it is a deliberate decision to disable the zone.
In fact, I have direct operational experience of why it's a terrible
way to disable a zone.

One of my own servers was slammed for queries in a zone I was not
authoritative for.  (It was a well-known zone, too, and one which is
not DNSSEC-signed.)  My server was dutifully returning Refused to the
queries, and yet they kept coming very frequently, maxing the
link. Arguably the clients should have applied the techniques of RFC
2308 for negative caching of those Refused answers, but it was not
until I added the zone in question to send back an authoritative
answer with a proper caching signal that the queries really went away.

While it is obviously the best approach for a takedown to update the
delegation, in the situation where you have a delegation pointing to
server(s) that cannot be updated but you have control over the
servers, it is far better to provide an affirmative answer to the
question than to send Refused.  Positive caching is much better
understood than negative caching by the wide variety of DNS
implementations out there.

---

From: Dave Lawrence <[email-masked]>
Date: Wed, 6 Mar 2019 11:27:18 -0500

Joe Abley writes:
> if you can find a set of DNS authority servers that silently
> discards a particular kind of query, sending such queries through
> resolvers that are known to support serve-stale might suppress other
> queries and trigger the serve-stale behaviour even though the
> authority servers are not actually unresponsive for them.

RFC 2308 section 7.2 made it clear that when caching a dead server
indication, "The indication MUST be stored against query tuple <query
name, type, class, server IP address> unless there was a transport
layer indication that the server does not exist, in which case it
applies to all queries to that specific IP address."

The scenario described in one of a resolver that is modern enough to
have implemented standards-compliant serve-stale behavior but is still
not standards-compliant with a two decade old RFC, interacting over a
path with a delegation to authoritative servers that are all
exhibiting erratic behavior.  This needs some supporting evidence that
it could really be a real-world problem that needs to be addressed.

---

From: Joe Abley <[email-masked]>
Date: Wed, 6 Mar 2019 11:41:28 -0500

> applies to all queries to that specific IP address."

Ahh, I forgot about that text in 2308! Good, then.

Joe

---

From: 神明達哉 <[email-masked]>
Date: Wed, 6 Mar 2019 12:40:36 -0800

At Fri, 1 Mar 2019 15:54:39 -0500,
Dave Lawrence <[email-masked]> wrote:

> recommendations"?

FWIW: "actively confusing" may be too strong, but I also found it
confusing on my fresh re-read of serve-stale-03 in that the "example
method" section contains normative descriptions using RFC2119
keywords.

So I support Paul's proposal:

>> (instead of introducing them) in the other sections. That way, a
>> lazy developer who only reads Section 4 will know all the
>> recommendations.

--
JINMEI, Tatuya

---

From: Dave Lawrence <[email-masked]>
Date: Thu, 7 Mar 2019 12:28:57 -0500

Jinmei:
> I also found it confusing on my fresh re-read of serve-stale-03 in
> that the "example method" section contains normative descriptions
> using RFC2119 keywords.

You're in good company in that co-author Puneet has voiced the same
opinion.  

My own take is that it's appropriate because if someone were to go
ahead and directly implement from the example method, then the BCP 14
normative language is appropriate because they are requirements of the
example method.  But that's only my viewpoint without knowing of any
further existing editorial guidance on the matter, and your point is
also sensible in favor of downcasing them as not being BCP 14
"requirements in the specification" in a broader sense.

I'm inclined to leave it to the RFC Editor to adjudicate as to what's best.
Ultimately I'm not passionate about it either way.