Subject: Re: [DNSOP] New draft on delegation revalidation

From: Shumon Huque <[email-masked]>
Date: Mon, 4 May 2020 08:19:57 -0400

On Wed, Apr 29, 2020 at 11:57 AM Daniel Migault <[email-masked]> wrote:

> Hi,
>
> I discovered this draft during the interim meeting. We had similar
> thoughts in our "Recommendations for DNSSEC Resolvers Operators". Our
> motivation for supporting this work are that it  1) improves the
> reliability of the resolution as well as 2) removes the temptation to
> (inadvertently) break resolution by fixing in appearance a
> misconfiguration. In other words it eases the operation.
>

Thank you Daniel. Will read your draft shortly, but in the meantime ..

Your note reminded me of the question you asked during the interim
meeting about why we can't just use the DS TTL as the revalidation
interval. Our response was that the main impediment was that DNSSEC
deployment is far from ubiquitous, but also that there is no reason that it
could not be factor, when available.

Here's what the draft already says about DS:

   Technically, if both parent and child zone are DNSSEC [RFC4033]
   [RFC4034] [RFC4035] signed with a corresponding secure delegation
   between them, then expiration of the DS record will cause
   revalidation of the current child zone's DNSKEY set, so responses
   from the orphaned child nameservers would no longer be trusted.
   However, delegation revalidation is still necessary to locate the
   current nameserver addresses.

In practice, the DS and delegating NS TTL do not always match.
COM for example uses 2 day NS and 1 day DS TTL for all its
delegations. So where DS is lower, that could be used as the upper
bound. We'll queue up some text on this subject for the next revision
of the draft.

Shumon.

---

From: Daniel Migault <[email-masked]>
Date: Mon, 4 May 2020 08:43:58 -0400

Thanks, that will be appreciated. I will make sure the two documents are
synchronised.
Yours,
Daniel

-- 
Daniel Migault
Ericsson

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Mon, 11 May 2020 14:59:24 +0200

>>  Do you plan to maintain the parent/child disjoint NS 
>> domain (marigliano.xyz <http://marigliano.xyz>) going forward? And what
>> about the test
>> domains for other types of misconfigurations?
> 
> Great idea. Let me look into this, will get back to with that.

Done. Check http://superdns.nl :)

Marco and I (mostly Marco, I've got say) set up this website and all the
delegations/records that replicates the setup of the paper.

We did under a diff domain for sake of simplicity for us and differently
from the paper, we create 4 delegations, each one corresponding to one
of the scenarios (in the paper we change the NS configurations in
between experiments, we want a static setup here for folks to test).

Hope it helps and if you need any help, let me know.

/giovane

ps: Raffaele, the first author of our paper, will present the study on
RIPE80 on Tuesday's plenary:
https://ripe80.ripe.net/programme/meeting-plan/plenary/#tue4 , in case
you want to check it out

---

From: Shumon Huque <[email-masked]>
Date: Wed, 13 May 2020 11:31:34 -0400

> delegations/records that replicates the setup of the paper.

Thanks Giovane (and Marco)!

Looks pretty good at first glance.

A few tangential questions though:

The HTTPS site goes to a different and mostly empty page - and
Chrome doesn't like the certificate because it has a wildcard Subject
CN. Are you planning to fix that?

I know DNSSEC is likely not the focus of your experiment, but the
zones do seem to be signed - but with algorithm 16 (Ed448), which
not a lot of resolvers or debugging tools support yet. Any reason you
didn't choose a more widely supported algorithm?

> https://ripe80.ripe.net/programme/meeting-plan/plenary/#tue4 , in case

Thanks for the pointer. I missed it, but will try to view the recording
soon.

Shumon.

---

From: Petr Špaček <[email-masked]>
Date: Thu, 21 May 2020 14:23:37 +0200

On 10. 04. 20 15:45, Shumon Huque wrote:
> Hi folks,
> 
> Paul Vixie, Ralph Dolmans, and I have submitted this I-D for
> consideration:
> 
>    https://tools.ietf.org/html/draft-huque-dnsop-ns-revalidation-01

I would appreciate a practical example of changes envisioned in the following paragraph:

>    A common reason that zone owners want to ensure that resolvers place
>    the authoritative NS RRset preferentially in their cache is that the
>    TTLs may differ between the parent and child side of the zone cut.
>    Some DNS Top Level Domains (TLDs) only support long fixed TTLs in
>    their delegation NS sets, and this inhibits a child zone owner's
>    ability to make more rapid changes to their nameserver configuration
>    using a shorter TTL, if resolvers have no systematic mechanism to
>    observe and cache the child NS RRset.

Could someone please post an example in steps? Something like:
- time 0, NSSET parent = {P0}, NSSET child = {C0}
- time 1, NSSET parent = {P1}, NSSET child = {C1}
... along with textual description what operator is hoping to achieve?

It would help me to appreciate the value of the proposal.

Ad 4.  Delegation Revalidation:

I agree with author's note "we would prefer to discard the extensive mechanism" but the simple mechanism has simple description for me to understand consequences.

>    The simple mechanism:
> 
>    o  Cap the time to cache the child NS RRset to the lower of child and
>       parent NS RRset TTL.  The normal iterative resolution algorithm
>       will then cause delegation revalidation to naturally occur at the
>       expiration of the capped child NS TTL, along with dispatching of
>       the validation query to upgrade NS RRset credibility.

So far so good, but it does not specify what should happen with RRsets other than NS. Even if nothing is prescribed please state that explicitly.

Most importantly:
- Does the NS affect maximum TTL of _other_ data in the zone?
- If it does, doesn't it increase risk of thundering herd behavior?
- If it does not, is it even worth the effort if attacker can put week long TTL for A/AAAA and keep using that?
- How should resolver handle RCODE=NXDOMAIN? Should it have different effect than changing NS set to different set of servers? Or change in DS record value?

For me personally mixing two problems (GHOST domains and NS inconsistency) in single proposal does not help me to understand reasoning behind the proposal and its intended effects.

Take care.

-- 
Petr Špaček  @  CZ.NIC

---

From: Shumon Huque <[email-masked]>
Date: Sun, 24 May 2020 23:23:42 -0400

> ... along with textual description what operator is hoping to achieve?

I'll try to come up with a step by step example later. But in the example
cited, what the operator is trying to achieve is to change their nameserver
configuration (e.g. switch to another set of nameservers for their zone) in
such a way that (1) they can make that change visible to resolvers
reasonably quickly, and (2) to make sure they are able to backout that
change quickly if things go wrong. They can do this by lowering the TTL
in the child NS set which is under their control -- assuming that resolvers
are
preferentially caching the child NS set, in accordance with the data ranking
rules of the DNS protocol (the child NS set is authoritative).

Ad 4.  Delegation Revalidation:
>
> I agree with author's note "we would prefer to discard the extensive
> mechanism" but the simple mechanism has simple description for me to
> understand consequences.
>

I've heard the same from other implementers too. Paul V has mentioned that
there are some subtle corner cases that are dealt with more precisely by
the extensive algorithm (I can't recall the details right now, but maybe he
will elaborate for us). Even if we end up on the simple path, it would be
good to have a better understanding of those cases.

> So far so good, but it does not specify what should happen with RRsets
> other than NS. Even if nothing is prescribed please state that explicitly.
>

Thanks, yes I agree we should discuss all of these. Some quick thoughts for
now ..

> - Does the NS affect maximum TTL of _other_ data in the zone?

I think there are probably different views on what should happen here.
Folks who want very prompt takedown of "bad" domains, will probably prefer
a complete pruning of the cache at the delegation point at the revalidation
interval, if the NS set has changed or disappeared. When this topic has
come up in the past, there has been pushback from some implementers that
it's difficult to do this because they use a non-tree data structure for
the cache (a hash table most commonly). Most resolvers these days already
enforce a max cache TTL parameter, so that typically prevents too much
abuse. But at the very least, they should probably use the revalidation
interval as a signal to stop "pre-fetching" records below the cut.

Possibly, depending on how popular the zone is, and what we decide is the
answer to the previous question. At any rate, implementers should always
employ strategies that bound how much work resolvers can be caused to do.

It might be reasonable to suggest that resolvers enforce a lower limit on
the child NS TTL (5 minutes?). If they see something less than that, it
would be set to the limit instead.

- If it does not, is it even worth the effort if attacker can put week long
> TTL for A/AAAA and keep using that?
>

Answered above ...

> - How should resolver handle RCODE=NXDOMAIN? Should it have different
> effect than changing NS set to different set of servers?

If resolvers are following RFC 8020 strictly, they are pruning their cache
at the delegation - that would be the ideal. Otherwise, they should allow
cached records below to live on, subject to max-cache TTL and disabling of
pre-fetching.

Or change in DS record value?

Which value? TTL or RDATA?

DS TTL expiration would automatically trigger a revalidation of the child
SEP keys at the parent. RFC 4035 says DS and delegating NS TTL SHOULD
match, so NS revalidation should happen on the same time scale. In reality
though, they are often different (COM/NET uses 2d for NS, 1d for DS). if NS
> DS, a resolver could just decide to explicitly fetch the expiring DS
first and wait out the delegating NS TTL if the DS rdata has not changed.
But it seems much simpler to just say that  if there is a secure
delegation, then the DS TTL should be used as the NS revalidation interval
too.

For me personally mixing two problems (GHOST domains and NS inconsistency)
> in single proposal does not help me to understand reasoning behind the
> proposal and its intended effects.
>

Ok, we'll think about how to make this clearer. But the two topics are
related. If resolvers prefer the authoritative child NS set, then timely
revalidation of the delegation at the parent is also necessary.

Shumon.

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Mon, 25 May 2020 08:18:13 +0200

Hi Shumon,

> Thanks Giovane (and Marco)!
Sure thing.

> CN. Are you planning to fix that?
fixed.

> didn't choose a more widely supported algorithm?

No particular reason, we can also change later on.

/giovane

---

From: Petr Špaček <[email-masked]>
Date: Thu, 28 May 2020 16:38:11 +0200

On 25. 05. 20 5:23, Shumon Huque wrote:
> On Thu, May 21, 2020 at 8:24 AM Petr Špaček <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     >
> in the child NS set which is under their control -- assuming that resolvers are
> I've heard the same from other implementers too. Paul V has mentioned that there are some subtle corner cases that are dealt with more precisely by the extensive algorithm (I can't recall the details right now, but maybe he will elaborate for us). Even if we end up on the simple path, it would be good to have a better understanding of those cases.
> 
>     >    The simple mechanism:
>     >
> Thanks, yes I agree we should discuss all of these. Some quick thoughts for now ..
> 
>     Most importantly:
> I think there are probably different views on what should happen here. Folks who want very prompt takedown of "bad" domains, will probably prefer a complete pruning of the cache at the delegation point at the revalidation interval, if the NS set has changed or disappeared. When this topic has come up in the past, there has been pushback from some implementers that it's difficult to do this because they use a non-tree data structure for the cache (a hash table most commonly). Most resolvers these days already enforce a max cache TTL parameter, so that typically prevents too much abuse. But at the very least, they should probably use the revalidation interval as a signal to stop "pre-fetching" records below the cut.
> 
> Possibly, depending on how popular the zone is, and what we decide is the answer to the previous question. At any rate, implementers should always employ strategies that bound how much work resolvers can be caused to do.

I'm not concerned about any single resolver instance, I'm more concerned about large number of resolver instances doing the same thing at the same time.

E.g. if NS TTL was short (say 30 s) and it was used as cap on TTL of all other records in the zone, then each resolver instance would clear zone from its cache each 30 seconds. That might cause interesting behavior when NS TTL is shortened e.g. before NS set change etc.

I do not know if there really is a problem, I'm just trying to explain why potential for thundering herd needs to be be seriously analyzed.

> 
> It might be reasonable to suggest that resolvers enforce a lower limit on the child NS TTL (5 minutes?). If they see something less than that, it would be set to the limit instead.
> 
>     - How should resolver handle RCODE=NXDOMAIN? Should it have different effect than changing NS set to different set of servers?
> 
> 
> If resolvers are following RFC 8020 strictly, they are pruning their cache at the delegation - that would be the ideal. Otherwise, they should allow cached records below to live on, subject to max-cache TTL and disabling of pre-fetching.
> 
> DS TTL expiration would automatically trigger a revalidation of the child SEP keys at the parent. RFC 4035 says DS and delegating NS TTL SHOULD match, so NS revalidation should happen on the same time scale. In reality though, they are often different (COM/NET uses 2d for NS, 1d for DS). if NS > DS, a resolver could just decide to explicitly fetch the expiring DS first and wait out the delegating NS TTL if the DS rdata has not changed. But it seems much simpler to just say that  if there is a secure delegation, then the DS TTL should be used as the NS revalidation interval too.
> 
> Ok, we'll think about how to make this clearer. But the two topics are related. If resolvers prefer the authoritative child NS set, then timely revalidation of the delegation at the parent is also necessary.

Yeah, you are right. It is probably good idea to discuss both at once because it will force us to find compromise between the two. E.g. most aggressive approach where resolver prunes cache on mere NS set _change_ might increase fragility when someone makes mistake while doing NS set change etc.

Having said that, I still think it would help if arguments/requirements for each use-case (GHOST vs. NS inconsistency / malicious vs. well-behaved operator) were laid out separately and then document reconciled arguments from these two camps and reached single recommendation for resolver implementers.

-- 
Petr Špaček  @  CZ.NIC

---

From: Paul Vixie <[email-masked]>
Date: Thu, 28 May 2020 15:55:15 +0000

On Thursday, 28 May 2020 14:38:11 UTC Petr Špaček wrote:
> > ...
> > Folks who want very prompt takedown of "bad" domains, will probably
> > prefer a complete pruning of the cache at the delegation point at the
> > revalidation interval, if the NS set has changed or disappeared. ...

yes, that was the original motive for revalidation itself, noting that it also 
facilitates emergency redelegation, for example, after a registrant/registrar 
account compromise. so the domain might not be bad, but the cached content 
might be poisonous in other ways.

> > When
> > this topic has come up in the past, there has been pushback from some
> > implementers that it's difficult to do this because they use a non-tree
> > data structure for the cache (a hash table most commonly). ...

i don't think we should argue the computer science behind implementing this. 
if the cache doesn't facilitate "rm -r" behaviour directly, then it might 
choose to bookend the retrievals, such that retrieving data having a bailiwick 
whose timestamp is older than the revalidation event would cause deletion and 
refetch at the time of the encounter. these details should not enter into a 
discussion of whether the system should have a capability or not.

> I'm not concerned about any single resolver instance, I'm more concerned
> about large number of resolver instances doing the same thing at the same
> time.
> 
> E.g. if NS TTL was short (say 30 s) and it was used as cap on TTL of all
> other records in the zone, then each resolver instance would clear zone
> from its cache each 30 seconds. That might cause interesting behavior when
> NS TTL is shortened e.g. before NS set change etc.
> 
> I do not know if there really is a problem, I'm just trying to explain why
> potential for thundering herd needs to be be seriously analyzed.

if we can think of a way that the intervals can become synchronized, then we 
would treat this with random subtractive revalidation. to get a thundering 
herd every member of the herd would have to start their interval at the same 
time. such synchronicity is hard to trigger.

-- 
Paul

---

From: Gavin McCullagh <[email-masked]>
Date: Sat, 30 May 2020 10:13:10 -0700

Hi,

> > > revalidation interval, if the NS set has changed or disappeared. ...

Those last few words seem very critical.  "If the NS set has changed or
disappeared".

> > potential for thundering herd needs to be be seriously analyzed.

I think Petr's point might be that someone could interpret the draft
(perhaps especially the simple mechanism) to mean that the NS TTL becomes
an effective cap on the TTL of all records below the zone cut even where
the NS set has not changed.

If lowering a delegating NS TTL to 300 for a very large zone (or one high
up the tree) meant a lot of resolvers might expire every record below the
zone cut every 300 seconds (even unsynchronized), then it seems like
lowering that TTL could produce a thundering herd.   I think the intent of
the draft and Shumon's quote above are that a resolver should only expire
cached subdomain records when the delegating NS changes.   Would it be
useful to be explicit about that?

I imagine even the NS revalidation means that if an operator lowers a
delegating NS TTL to e.g. 300 in advance of a change in order to "make that
change visible to resolvers reasonably quickly", they might well see a 300
second long period of thunder immediately after the change while every
resolver's cache expires that part of the tree and then fills it back up.

One minor other question:

      To revalidate a delegation, the iterative caching DNS resolver
      will forward the query that triggered revalidation to the
      nameservers at the closest enclosing zone cut above the
      revalidation point.

Does "the query that triggered revalidation" refer to the original client
query which had RD=1?  If so, does this fit with qname minimization?

Gavin

---

From: Shumon Huque <[email-masked]>
Date: Sat, 30 May 2020 13:45:38 -0400

> useful to be explicit about that?

Gavin - yes, I agree. I was indeed deliberate in my wording about "If the
NS set has changed or disappeared" to avoid creating unnecessary work for
resolvers. This topic is not yet discussed in the draft, but assuming the
draft is adopted and we cover it, we can make it explicit.

> query which had RD=1?  If so, does this fit with qname minimization?

Yeah, good catch. We need to adjust that text slightly to be compatible
with qname minimization, which should be pretty straightforward - it's just
pruning the requisite number of left most labels for the enclosing zone.

Shumon.

---

From: Paul Vixie <[email-masked]>
Date: Sat, 30 May 2020 21:52:05 +0000

> the NS set has not changed.

i hadn't seen that possible interpretation, and never intended that potential 
outcome, so thank you for explaining that the draft language is unclear in 
that way.

> lowering that TTL could produce a thundering herd.

i think there are two discussable matters there. first, the thundering herd 
would still have to be externally synchronized somehow, or else each member of 
the herd would begin their 300-second cycle independently, and never thunder.

second, the only condition under which "rm -r $apex" would occur in the cache 
is if the delegation point disappears or if it rolls over 100% (no NS RRs in 
common between two successive NS RRsets seen during delegation.) so, very 
rare.

we can clarify the language to remove the possible misinterpretation you're 
describing.

> I think the intent of

yes, apparently so :-).

> resolver's cache expires that part of the tree and then fills it back up.

until and unless we have a way for authorities to notify existing caches that 
a delegation has changed (either in content of NS or DS RRset, or A/AAAA glue 
under the delegation point, or TTL of any of that), then the change in your 
example would not be seen by all caches at the same time, which means they 
could not synchronize their revalidation cycles into a thundering herd.

as an interesting corner case, an RFC2308 negative proof that included a 
delegation section could cause an updated delegation to be seen immediately by 
a large number of caches, for example during a randomized subdomain attack. it 
seems possible that this latent attack vector deserves mention in the 
revalidation draft, but i'm not sure how to describe "synchrony should be 
avoided" other than to say that caches practicing delegation revalidation 
ought to begin this process at a random time during the last 20% of the 
delegating TTL, to avoid becoming part of a thundering herd if their original 
delegation reception time was somehow broadly synchronized.

> query which had RD=1?  If so, does this fit with qname minimization?

that language predates Q-M, so thanks for noticing the mismatch. we'll have to 
say "whatever query would normally be forwarded to the authority". what we're 
trying to avoid is making an NS query for the delegation, since that's 
considered by many authority operators as a diagnostic query that they can't 
or won't answer. but i agree, this has to be updated to cover the Q-M method.

-- 
Paul