Subject: [DNSOP] draft-tale-dnsop-serve-stale

From: Dave Lawrence <[email-masked]>
Date: Mon, 27 Mar 2017 17:56:58 -0400

One of the two drafts I wanted to talk about at dnsop today for WG
adoption was "Serving Stale Data to Improve DNS Resiliency":
https://datatracker.ietf.org/doc/draft-tale-dnsop-serve-stale/

In short, this describes a method for increasing DNS resiliency by
treating the inability to refresh data after TTL expiration as a soft
error, eventually becoming a hard error if the authoritative server
failures are not remedied.

This basic algorithm has been in use at Akamai for six years now and
helped us avoid numerous incidents.  I'd implemented it in BIND and
the patches were recently contributed to ISC.

There are relevant patents in the area held by Google and
Akamai/Xerocole.  I'm still waiting for the official statement from
Akamai lawyers about it, but given that we contributed to the code to
ISC for release under the Mozilla Public License I don't expect any
really issue here.

Warren and I are hoping for WG adoption.

---

From: P Vix <[email-masked]>
Date: Mon, 27 Mar 2017 21:59:58 +0000

I agree to review and comment. Note that I am provisionally negative to the idea itself, and my review may reflect that. Vixie

On March 27, 2017 4:56:58 PM CDT, Dave Lawrence <[email-masked]> wrote:
>_______________________________________________
>DNSOP mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/dnsop

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Jared Mauch <[email-masked]>
Date: Mon, 27 Mar 2017 18:19:09 -0400

> I agree to review and comment. Note that I am provisionally negative to the idea itself, and my review may reflect that. Vixie

I will note there are other implementations out there as well, such as in unbound.  serve-expired configuration directive is available there as well.

- Jared

---

From: Robert Edmonds <[email-masked]>
Date: Mon, 27 Mar 2017 18:46:54 -0400

> I will note there are other implementations out there as well, such as in unbound.  serve-expired configuration directive is available there as well.

Though, the algorithm described in this document is a much different
algorithm than the one in Unbound.

If I understand Unbound's serve-expired algorithm correctly, it always
serves from cache if available (regardless of expiration status), and if
what it served to the client happened to be expired, it triggers a
post-response fetch to update the cache asynchronously. That can end up
serving a lot more stale bread than is strictly necessary if your
Unbound server only serves a few clients.

(I guess Unbound could sort of be said to implement this draft, but with
the client response timer hardcoded to 0 and the maximum stale timer
hardcoded to ∞.)

I support adoption of this document.

-- 
Robert Edmonds

---

From: Jared Mauch <[email-masked]>
Date: Mon, 27 Mar 2017 18:49:41 -0400

> On Mar 27, 2017, at 5:56 PM, Dave Lawrence <[email-masked]> wrote:
> 

[clarification]

I support adoption when the chairs request it.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Mon, 27 Mar 2017 18:55:36 -0400

> On Mar 27, 2017, at 6:46 PM, Robert Edmonds <[email-masked]> wrote:
> 
> Jared Mauch wrote:

At least the initial implementation is documented (via code) here:

https://github.com/jedisct1/unbound/commit/e03d89343e4031be15b2ee78bd432f83cdc79889

> Unbound server only serves a few clients.

I see the perceived damage here as very low due to the "few clients" you already
commented on.

- Jared

---

From: Pieter Lexis <[email-masked]>
Date: Tue, 28 Mar 2017 13:20:50 +0200

On Mon, 27 Mar 2017 18:19:09 -0400

> I will note there are other implementations out there as well, such as in unbound.  serve-expired configuration directive is available there as well.

I feel that the authors should attempt to describe the goal of the algorithm and suggest possible limits and describe pitfalls rather than describing the exact algorithm to use.
This will allow implementers to come up with new and innovative algorithms based on continued measurements.
I do support an adoption of such a draft in the WG.

Best regards,

Pieter

-- 
Pieter Lexis
PowerDNS.COM BV -- https://www.powerdns.com

---

From: Shumon Huque <[email-masked]>
Date: Tue, 28 Mar 2017 07:08:43 -0500

On Mon, Mar 27, 2017 at 5:19 PM, Jared Mauch <[email-masked]> wrote:

> > I agree to review and comment. Note that I am provisionally negative to
> the idea itself, and my review may reflect that. Vixie
>
>
> I will note there are other implementations out there as well, such as in
> unbound.  serve-expired configuration directive is available there as well.
>

OpenDNS also has had a similar feature (exact protocol unpublished AFAICT)
for a while now (Smartcache, ~ 2009 I think).

-- 
Shumon Huque

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 28 Mar 2017 10:42:27 -0400

Pieter Lexis writes:
> I feel that the authors should attempt to describe the goal of the
> algorithm and suggest possible limits and describe pitfalls rather
> than describing the exact algorithm to use.

I confess I'm a bit flummoxed by this comment, as I believe the
document already does describe goals and pitfalls.  If you believe
there is something missing, please let us know what else should be
included.

> This will allow implementers to come up with new and innovative
> algorithms based on continued measurements.

As with how the process went for refuse-any, we'd be happy to consider
including alternative mechanisms.

---

From: Warren Kumari <[email-masked]>
Date: Tue, 28 Mar 2017 10:31:11 -0500

> I feel that the authors should attempt to describe the goal of the algorithm and suggest possible limits and describe pitfalls rather than describing the exact algorithm to use.

Noted.
As one of the authors of both this, and an earlier document with a
different algorithm ( draft-wkumari-dnsop-ttl-stretching ), I think
that the actual algorithm specified is a secondary consideration --
first we need to agree on if the concept / idea is good for general
use.

Assuming that it is a good idea to standardize (perhaps only for
certain use-cases / environments)...

This document has an algorithm which has been deployed, and tuned and
fettled, and works nicely -- however, the environment in which is
being used in may different attributes to the general case, so...

It is entirely possible that:
A: the described algorithm and parameters are perfect for general use, or
B: the described algorithm with different parameters is perfect for general use
C: we need a different algorithm for general use
D: we could describe multiple algorithms, and implementors can select
E: we can just describe what the expected behavior is, and
implementors can determine how to do this.
F: something else

I personally think that B is reasonable -- the algorithm seems to
work, but different (or adjustable) timers might be better for general
recursive servers. But, I'm happy to be wrong -- if the WG adopts the
document it owns it and can choose any of the above...

> I do support an adoption of such a draft in the WG.

Thank you.
W

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: "Peter van Dijk" <[email-masked]>
Date: Tue, 28 Mar 2017 19:20:37 +0200

Hello Shumon,

> for a while now (Smartcache, ~ 2009 I think).

There is some background at https://www.youtube.com/watch?v=Gt9VUPDoZk0 
starting at 14:33. Slides at 
https://indico.dns-oarc.net/event/24/session/8/contribution/13/material/slides/2.pdf 
(slide 7).

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: Paul Vixie <[email-masked]>
Date: Tue, 28 Mar 2017 10:20:40 -0700

Warren Kumari wrote:
> ... I think

that will depend on the algorithm. for example if it's only stretchy for
timeouts and servfail, but nxdomain is authoritative as before, then the
best-case and worst-case outcomes are suitable for eachother, and
general use could not be worse than googledns, opendns, and akamai all
doing it with apparent success.

since it allocates no code point and the method requires no interop,
this draft feels a bit like resimprove, which died on the vine for no
reason i can now recall. it's harmless as an FYI, but does not belong on
the standards track.

speaking of resimprove, i hope you'll include in this draft the idea of
using delegation-TTL as a delegation-recheck interval, and using an
authoritative NXDOMAIN from the delegator as proof that you need to run
an "rm -rf" in your cache.

i bring this up because we need to be able to kill more cached data
faster-- the opposite of stretchiness-- for abuse control reasons. if
you're going to soften the signaling for cache expiration, you really
ought to balance that out with this simple method of also hardening it.

-- 
P Vixie

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 28 Mar 2017 17:44:16 -0400

> an "rm -rf" in your cache.

I definitely like the latter idea as wholly on-point for the main
issue this draft is trying to address, and will get it into the next
rev.

I'm slightly less sure about dragging delegating rechecking into it,
but open enough to being convinced that it is also on-point and
doesn't unduly complicate what is conceptually very straightforward in
the current draft.

I'm also in favour of revivificating resimprove.  I don't know why it
languished either, but don't recall substantial objection.  If it were
brought back that somewhat obviates needing to bring up the delegation
recheck here.

---

From: Shumon Huque <[email-masked]>
Date: Tue, 28 Mar 2017 20:31:27 -0500

> ought to balance that out with this simple method of also hardening it.

Hi Paul,

Perhaps you've forgotten (since you participated in the discussions), but
the portion of resimprove that dealt with expunging cached data below the
NXDOMAIN boundary was rescued, expanded on, and published as
RFC 8020 ("NXDOMAIN: There Really is Nothing Underneath") late last
year.

-- 
Shumon Huque

---

From: Paul Vixie <[email-masked]>
Date: Wed, 29 Mar 2017 12:10:28 -0700

Shumon Huque wrote:
> On Tue, Mar 28, 2017 at 12:20 PM, Paul Vixie <[email-masked]
>     ...
> 

yes, i know that RFC 8020 includes both the idea of stopping downward
searches when a cached NXD is encountered, and the idea of pruning (like
"rm -rf" in UNIX) existing cache entries when an NXD is received. those
changes, in addition to QM, will do much to pacify the DNS data model.

however, it's equally vital for malicious DNS content takedown, that the
part about remembering the delegation NS TTL and using it to
periodically revalidate the existence of that delegation, also be
brought forward. this becomes even more vital if we're making TTL
stretchy, and i would be happy to see tale's document cover this topic.

in other words, the prune-and-stop on NXD must be given a new source of
NXD, which is delegation revalidation. i'd love it if delegations all
had a one hour or less TTL, at least for public suffixes.

-- 
P Vixie

---

From: Shumon Huque <[email-masked]>
Date: Wed, 29 Mar 2017 14:19:55 -0500

> had a one hour or less TTL, at least for public suffixes.

Yes, I forgot to comment on that piece in the previous email. I agree
that delegation revalidation should be published also. Among other things,
I think it effectively solves the ghost domain problem that was discussed
during the last IETF and DNS-OARC.

-- 
Shumon Huque