Subject: [DNSOP] comments on draft-ietf-dnsop-serve-stale-03

From: 神明達哉 <[email-masked]>
Date: Thu, 7 Mar 2019 12:47:45 -0800

I've read draft-ietf-dnsop-serve-stale-03.  In addition to the
high-level draft organization matter I mentioned in another thread,
here are my other comments on this version:

- Section 4:

   The definition of TTL in [RFC1035] Sections 3.2.1 and 4.1.3 is
   amended to read:

   TTL  [...]  If the authority for the data is unavailable
      when attempting to refresh, the record MAY be used as though it is
      unexpired.

  On understanding that this is the only real normative description,
  I'd suggest making some more points explicit to prevent abusing of
  this leniency:
  - explicitly say "all authoritative servers" instead of just "the
    authority"
  - also explicitly note that this MUST NOT be allowed if at least one
    authoritative server is available
  - clarify whether this means a 0-TTL record can be cached and reused
    under this condition (I assume it must not, but it's not very clear
    to me)

- Section 5

   If it
   finds no relevant unexpired data and the Recursion Desired flag is
   not set in the request, it SHOULD immediately return the response
   without consulting the cache for expired records.

  It would be nice if it clarified *what* to return in this case (if
  it's intentionally left open, explicitly say so).

- Section 5

   Outside the period of the resolution recheck timer, the resolver
   SHOULD start the query resolution timer and begin the iterative
   resolution process.

  It's not clear to me how this timer is related to the 'server-stale'
  behavior; this draft doesn't explain what happens when this timer
  expires, for example.  Also, in my understanding unbound doesn't
  have this timer - it eventually gives up a resolution if all
  possible external query fails with a per-query timeout, but it
  doesn't cap the overall resolution time.  That may not matter much
  as this section doesn't seem to be normative and it's just an
  implementation detail of a particular implementation, but if the
  role of this timer doesn't matter either, we might rather simplify
  the text by just omitting it.

- Section 6

   Stale data is used only when refreshing has failed, in order to
   adhere to the original intent of the design of the DNS and the
   behaviour expected by operators.

  I agree on this statement, but I wonder how widely this behavior is
  actually implemented.  As noted in Section 7, unbound doesn't behave
  this way, and in my understanding it's intentional, mainly due to
  a concern about related IPR.  If that's more common for other open
  source implementors (BIND 9 seems to work as described here, but I
  don't know about others), the description won't match the actual
  implementation behavior very well in reality.  So I'm curious about
  implementation status about this point, and if many different
  implementations intentionally ignore this "caveat" for the same
  reason, I think we should adjust the text to match the reality.

- Section 7

   Unbound has a similar feature for serving stale answers, but will
   respond with stale data immediately if it has recently tried and
   failed to refresh the answer by pre-fetching.

  If I understand the implementation correctly, this is not 100%
  accurate: unbound always return the stale data if it's found in the
  cache as long as the "serve-expired" option is enabled.  So I
  suggest revising the text to:

   Unbound has a similar feature for serving stale answers, but will
   respond with stale data immediately whenever the feature is
   enabled.

--
JINMEI, Tatuya

---

From: Dave Lawrence <[email-masked]>
Date: Fri, 8 Mar 2019 15:29:48 -0500

Thank you very much for the feedback, Jinmei.  

Combined with previous changes we made following the other messages on
the draft we expect to republish it before the Monday IETF 104
submission deadline, after one last review by all of the co-authors.

>     authoritative server is available

I can see how the phrasing of "if the authority" could imply to a
novice in the DNS space that only one server would be tried, so I
updated the wording to:

     If the data is unable to be authoritatively refreshed when the
     TTL expires, the record MAY be used as though it is unexpired.

I think this phrasing is sufficient without needing to explicitly say
all servers and must not if at least one responds.  Existing resolver
implementations are extremely thorough in trying to get authoritative
answers and it is extremely hard to imagine that anyone would take
this draft to mean that they should be any less thorough.

>     under this condition (I assume it must not, but it's not very clear

Added this to the caveats section:

     The continuing prohibition against using data with a 0 second TTL
     beyond the current transaction explicitly extends to it being
     unusable even for stale fallback, as it is not to be cached at all.

>>   If it finds no relevant unexpired data and the Recursion Desired
>>   flag is not set in the request, it SHOULD immediately return the
>>   response without consulting the cache for expired records.

>   it's intentionally left open, explicitly say so).

Added:

     Typically this response would be a referral to authoritative
     nameservers covering the zone, but the specifics are implementation
     dependent.

I was surprised to discover when testing against BIND 9.12 (without
serve-stale in play) that dig +norec for an unknown example.com name
gave a referral to com, even when it knew the NS for example.com
either via the parent delegation or even from the apex.

>   behavior; [...]

I think it's main utility in the example method is to emphasize that
even if you send a stale answer to the client while a lengthy
resolution attempt is still playing out, you've got to keep trying.
Admittedly capping the work of that lengthy attempt is not
specifically relevant, but as you noted this is an example.
I can see your point about possibly simplifying by removing a few
sentences related to it, but as I also think that capping work is an
important aspect of resiliency I'm inclined to leave it in.

>   this draft doesn't explain what happens when this timer
>   expires, for example.

Based on "This timer bounds the work done by the resolver
when contacting external authorities" I'd have thought it was
implicitly clear, but I have added:

     If this timer expires on an attempted lookup that is still
     being processed, the resolution effort is abandoned.

> Also, in my understanding unbound doesn't have this timer - it
> eventually gives up a resolution if all possible external query
> fails with a per-query timeout, but it doesn't cap the overall
> resolution time.

Interesting.  I know of an Unbound-derived server that definitely caps
work, though that may have been local changes and not incorporated
into mainline.  Tarpitting was a significant issue for the people
involved.

>   a concern about related IPR.

Huh, My understanding from a hallway conversation with Benno was that
the immediate response is only sent for names that would have been
subject to pre-fetching, such that the immediate response in this case
is sufficiently covered under the guidance of a recent attempt being
made.  If that is not the case, and you can get stale answers from
Unbound even without a recent refresh attempt, then I personally think
that is an error in Unbound and not this document.

Also, the IPR is effectively released. (... but I am not a lawyer.)

> So I'm curious about implementation status about this point, and if
> many different implementations intentionally ignore this "caveat"
> for the same reason, I think we should adjust the text to match the
> reality.

I have strong objections to doing that.  Imagine having data that
expired days ago sitting in a cache but unrefreshed because no one
asked for it, then poof sending possibly obsolete information to the
first person who asks even though the authorities were perfectly well
responsive.  I think it is very important that implementations do
their best to honor TTLs as a refresh signal, and removing text that
emphasizes that is a bad idea.

>   cache as long as the "serve-expired" option is enabled. 

In addition to the note above, which needs clarification regarding the
Unbound implementation, note that section 7 will be removed entirely
once the document moves on.

Thanks again.

---

From: Benno Overeinder <[email-masked]>
Date: Sun, 24 Mar 2019 12:03:30 +0100

Hi,

> that is an error in Unbound and not this document.

The current implementation of serve stale in Unbound is closely related
with the pre-fetching process.  It works well for most cases, that is
names that are frequently queried for, so the pre-fetch assures for
fresh and correct entries in the cache.

For names with relative short TTLs and that are not frequently queried
for (i.e. less frequent than covered by the TTL), the entry from the
cache is stale and only after serving the reply, a pre-fetch (resolve)
is initiated to update/re-fresh the entry in the cache.

We acknowledge this behavior is not optimal in some situations and will
reimplement a part of the re-fresh strategy of cache entries.

-- Benno

-- 
Benno J. Overeinder
NLnet Labs
https://www.nlnetlabs.nl/

---

From: Paul Vixie <[email-masked]>
Date: Sun, 24 Mar 2019 04:36:50 -0700

i object to serve-stale as proposed. my objection is fundamental and 
goes to the semantics. no editorial change would resolve the problem.

i would withdraw that objection if this draft incorporates section 2 of 
https://tools.ietf.org/html/draft-vixie-dnsext-resimprove-00, to wit:

> 2. Delegation Revalidation Upon NS RRSet Expiry
> 
>    2.1. Because the delegating NS RRset at the bottom of the parent zone
>    and the apex NS RRset in the child zone are unsynchronized, the TTL
>    of the parent's delegating NS RRset is meaningless. A child zone's
>    apex NS RRset is authoritative and thus has a higher cache
>    credibility than the parent's delegating NS RRset, so, the NS RRset
>    "below the cut" immediately replaces the parent's delegating NS RRset
>    in cache when an iterative caching DNS resolver crosses a zone cut.
> 
>    2.2. The lowest TTL found in a parent zone's delegating NS RRset
>    should be stored in the cache and used to trigger delegation
>    revalidation as follows.  Whenever a cached RRset is being considered
>    for use in a response, the cache should be walked upward toward the
>    root, looking for expired delegations. At the first expired
>    delegation encountered while walking upward toward the root,
>    revalidation should be triggered, putting the processing of dependent
>    queries on hold until validation is complete.
> 
>    2.3. To revalidate a delegation, the iterative caching DNS resolver
>    will forward the query that triggered revalidation to the nameservers
>    at the closest enclosing zone cut above the revalidation point. While
>    searching for these nameservers, additional revalidations may occur,
>    perhaps placing an entire chain of dependent queries on hold,
>    unwinding in downward order as revalidations closer to the root must
>    be complete before revalidations further from the root can begin.
> 
>    2.4. If a delegation can be revalidated at the same node, then the
>    old apex NS RRset should be deleted from cache and then the new
>    delegating NS RRset should be stored in cache. The minimum TTL from
>    the new delegating NS RRset should also be stored in cache to
>    facilitate future revalidations. This order of operations ensures
>    that the RRset credibility rules do not prevent the new delegating NS
>    RRset from entering the cache. It is expected that the child's apex
>    NS RRset will rapidly replace the parent's delegating NS RRset as
>    soon as iteration restarts after the revalidation event.
> 
>    2.5. If the new delegating NS RRset cannot be found (RCODE=NXDOMAIN)
>    or if there is a new zone cut at some different level of the
>    hierarchy (insertion or deletion of a delegation point above the
>    revalidation point) or if the new RRset shares no nameserver names in
>    common with the old one (indicating some kind of redelegation, which
>    is rare) then the cache should be purged of all names and RRsets at
>    or below the revalidation point. This facilitates redelegation or
>    revocation of a zone by a parent zone administrator, and also
>    conserves cache storage by deleting unreachable data.
> 
>    2.6. To make the timing of a revalidation event unpredictable from
>    the point of view of a potential cache-spoof attacker, the parent's
>    delegating NS RRset TTL should be reduced by a random fraction of its
>    value before being stored for use in revalidation activities.

in other words, we'd be negotiating for the right to re-interpret 
existing signaling (the authority's TTL no longer purely governs the 
data's lifetime) by insisting that the parent zone's delegating TTL be 
given absolute power for revocation.

vixie

---

From: bert hubert <[email-masked]>
Date: Sun, 24 Mar 2019 13:04:44 +0100

On Sun, Mar 24, 2019 at 04:36:50AM -0700, Paul Vixie wrote:
> i object to serve-stale as proposed. my objection is fundamental and goes to
> the semantics. no editorial change would resolve the problem.

I too object.  This is partially due to the apparently unresolved IPR issue
from Akamai, who are known not to be shy asserting their IPR.

https://datatracker.ietf.org/ipr/3014/ notes an Akamai IPR claim and does
not yet provide a license suitable for use on an open internet.

https://patents.google.com/patent/US8583801B2/en & 

https://en.wikipedia.org/wiki/Akamai_Techs.,_Inc._v._Limelight_Networks,_Inc.
have some context. 

The mechanics are that once something is an RFC, operators require adherance
to it. This in turn is a boon for the IPR holder, and hurts everyone else.

My secondary objection is that the draft contains an example implementation
that then however uses normative words. This leads to pain with operators
demanding serve-stale compliance. Example algorithms should clearly be
examples and not tell us what we SHOULD do.

	Bert

---

From: Ray Bellis <[email-masked]>
Date: Sun, 24 Mar 2019 14:50:56 +0100

> given absolute power for revocation.

+lots

Ray

---

From: Puneet Sood <[email-masked]>
Date: Mon, 25 Mar 2019 16:07:48 +0100

Hi Paul,

> https://tools.ietf.org/html/draft-vixie-dnsext-resimprove-00, to wit:

I went back and read the discussion on this draft and I could not find
consensus on adopting it at that time. I do not think adding it to the
serve-stale draft will make path for adoption for either the
serve-stale draft or these recommendations easier.

https://tools.ietf.org/html/draft-ietf-dnsop-serve-stale-04 section 5
(page 6, paragraph 2) talks about refreshing the delegation. Quote:

+ When no authorities are able to be reached during a resolution
+   attempt, the resolver SHOULD attempt to refresh the delegation and
+   restart the iterative lookup process with the remaining time on the
+   query resolution timer.  This resumption should be done only once
+   during one resolution effort.

Maybe there are fewer, specific bits from your draft which would be
appropriate in the serve-stale context? Would you be willing to
discuss those?

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Puneet Sood <[email-masked]>
Date: Mon, 25 Mar 2019 16:08:35 +0100

> > given absolute power for revocation.

you mean lots of changes or lots of agreement with the quoted text?
They mean very different things.

-Puneet

---

From: Ray Bellis <[email-masked]>
Date: Mon, 25 Mar 2019 16:30:01 +0100

> They mean very different things.

I was agreeing with the quoted text - I believe that any serving of 
stale records must be predicated on the presence of a valid delegation 
from the parent zone.

Serve stael must not become a vector whereby malware can keep its C&C 
systems artificially alive even if the parent has removed the C&C domain 
name.

Ray

---

From: Frederico A C Neves <[email-masked]>
Date: Mon, 25 Mar 2019 13:24:10 -0300

> systems artificially alive even if the parent has removed the C&C domain 

+1

Fred

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 25 Mar 2019 15:45:16 -0400

> https://tools.ietf.org/html/draft-vixie-dnsext-resimprove-00, to wit:

I always liked resimprove.  Warren and I were talking about it, and if
you would like we'd be quite happy to pick it up and get it moving in
dnsop.  

This document already has text, however, for refreshing the delegation
and I don't believe it really needs to much detail as to what that
means.  "Delegation Revalidation Upon NS RRset Expiry" is an issue
orthogonal to serve-stale, and in fact most often applies when
serve-stale isn't even being triggered; it's a regular occurrence
under normal operations.  Ballooning the standards action section here
to nearly three times its existing size is unnecessary bloat.

Please let us know if you'd like us to take up the charge for
resimprove.

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 25 Mar 2019 15:58:17 -0400

> from Akamai, who are known not to be shy asserting their IPR.

This is definitely a problem.  Even though Akamai had previously
agreed to specify under what IETF-acceptable terms the IPR would be
made available, it clearly hasn't yet specified them.  I've contacted
them to get a timeline on when the legal department can take care of
this, and the first order response is that the DNS team is trying to
get the ball rolling again with legal this week.

> My secondary objection is that the draft contains an example
> implementation that then however uses normative words. This leads to
> pain with operators demanding serve-stale compliance. Example
> algorithms should clearly be examples and not tell us what we SHOULD
> do.

As previously noted in this very thread, at least one of the authors,
Puneet, agrees with you.  When I wrote the text that way it was
because of the also not-unreasonable viewpoint that if someone were to
be implementing the example then the text could be considered
normative as to how to do that.  It's even softened by having no MUSTs
at all, just SHOULD.  In addition, I'm dubious as to the claim that
people would cause meaningful pain to demand compliance with an
example, and not be adequately refuted when it is pointed out to them
that it is a clearly marked as an example.

That said, since I had waffled on it myself at time of composition and
I don't actually have a very strong feeling about it, in the end
wouldn't fight over downcasing it.  Yet I think it should be settled
at a level above dnsop because ultimately it's an issue that should be
consistent across IETF documentation.  Unless there's already an
explicitly stated IETF policy about this, and not just ad hoc past
cases to point to, I think it is best to sort out with the RFC editor.

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 25 Mar 2019 19:28:57 -0400

> systems artificially alive even if the parent has removed the C&C domain 

I wholeheartedly agree with this ideal, and am very open to
considering text improvements.

The document already has guidance on this point, but it is admittedly
in a considerations section and not in standards action, and is a
weaker "SHOULD" versus "MUST" right now.

Would the WG prefer that a line like this be put into the Standards
Action section?

  When no authorities for a name are able to be reached, the resolver
  MUST attempt to refresh the delegation.

I like the basic idea but am a little stuck on the wording because of
the endless loop it implies.  This is probably why it appears as
"SHOULD" already (but I honestly don't remember, so there's that).

It seems to me that the risk is very low, even as currently written in
the draft.  Not only do I have a lot of confidence in the implementers
of the most widely used resolvers in the world, as they're right here
participating too and have in the past shown good conscientiousness in
this area, but the practical attack is still hard to make meaningful.
If "the parent has removed the C&C domain name" as you said,
serve-stale shouldn't even kick in.  NxDomain, problem solved.

Various other scenarios come to mind, even with obstinate parents that
won't remove the delegation and the zone's NSs have gone dark, but
those scenarios have other possible remedies.  And fast flux using
long TTL NS RRsets are an issue no matter whether serve-stale is in
play or not.

So text regarding refreshing delegations could be given even more
words to describe backoff intervals and such, but to what end?  What's
the scenario?  And wouldn't it be handled better by reviving
resimprove to talk about the generalized problem?

(To be clear, I'm quite okay with politely being shown that I'm wrong
and there is a vector by which serve-stale becomes uniquely
interesting, and would certainly endeavour to make sure it is addressed.)

---

From: Paul Vixie <[email-masked]>
Date: Tue, 26 Mar 2019 02:00:29 -0700

Puneet Sood wrote on 2019-03-25 08:07:
> consensus on adopting it at that time.

noone understood it. there was no reply, positive or negative, to my 
proposal that the WG adopt it. it dropped like a brick, in silence.

i believe that the issues are, now, nine years later, better understood.

> ... I do not think adding it to the

understood. we disagree, but, i understand your position.

> appropriate in the serve-stale context? Would you be willing to

yes, certainly. the text i quoted from the 2010 resimprove draft was the 
result of implementation experience, and touches on necessary details, 
which are at the moment missing from serve-stale. i'll be very happy to 
discuss these, though regrettably, i could only be in prague for sunday, 
and so i'll miss any side meetings or working group discussions this 
time. no disrespect is intended by my absence, i had other obligations.

-- 
P Vixie

---

From: Tony Finch <[email-masked]>
Date: Tue, 26 Mar 2019 11:47:25 +0000

> considering text improvements.

I think the suggested max stale timer of 7 days is excessive. The aim is
to cope with an outage, so I think 1 day is much more reasonable (though I
have configured my servers with a 1 hour limit).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Shetland Isles: West or southwest 5 or 6, decreasing 3 or 4 for a time,
occasionally 7 at first. Moderate in sheltered east, otherwise rough,
occasionally very rough at first in west. Occasional rain. Moderate or good,
occasionally poor.

---

From: Olli Vanhoja <[email-masked]>
Date: Tue, 26 Mar 2019 13:18:50 +0100

> have configured my servers with a 1 hour limit).

I agree. At least based on my own experience, all the network or other
downtime issues I have experienced last only few minutes. If there is
a downtime longer than that and it's only affecting DNS, I would
seriously consider changing my service providers and vendors, whatever
is the issue.

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 26 Mar 2019 09:19:10 -0400

Olli Vanhoja writes:
> downtime issues I have experienced last only few minutes. 

Okay, I agree a little that 7 days is probably excessive as a
recommendation, though not harmful.  I also agree that in most
instance where serve-stale has already proven itself to be useful, 
the events are fairly short-lived.

On the other hand I have direct operational experience that says if a
problem is being caused not by a generalized DOS or other transient
network issue, then it can indeed take multiple days to resolve.
Start of a long weekend?  Trying to reach the right people to fix it?
Surely you've experienced customers not responding quite as quickly to
fix their problems as you'd like.

So I'm not so keen on one day, but could see dropping the
recommendation to 3.  It is, after all, still just a recommendation
and one that should be configurable.

> If there is a downtime longer than that and it's only affecting DNS,
> I would seriously consider changing my service providers and
> vendors, whatever is the issue.

Right!  But in the meantime, until that change is done ...

---

From: Olli Vanhoja <[email-masked]>
Date: Tue, 26 Mar 2019 14:34:21 +0100

> and one that should be configurable.

Yes, I remember that this has happened a few times at large scale during
the last few years. I'm a bit worried if it would still cause more problems
than it would solve.