Subject: [DNSOP] Call for Adoption: draft-huston-kskroll-sentinel

From: tjw ietf <[email-masked]>
Date: Thu, 16 Nov 2017 16:23:32 +0800

All

The author has rolled out a new version addressing comments from the
meeting on Monday, and we feel it’s ready to move this along.

This starts a Call for Adoption for draft-huston-kskroll-sentinel

The draft is available here:
https://datatracker.ietf.org/doc/draft-huston-kskroll-sentinel/

Please review this draft to see if you think it is suitable for adoption by
DNSOP, and comments to the list, clearly stating your view.

Please also indicate if you are willing to contribute text, review, etc.

This call for adoption ends: 30 November 2017 23:59

Thanks,
tim wicinski
DNSOP co-chair

---

From: George Michaelson <[email-masked]>
Date: Thu, 16 Nov 2017 16:34:05 +0800

I support adoption of this work. Its a sensible, simple proposal which
has immediate benefit, and can be used by anyone to test the ability
of their nominated resolver to recognise specific keys, and their
trust state.

I believe as a community, at large,  we need code deployed into the
resolvers in the wild, and we need a document specifying the behaviour
we need deployed into those resolvers. We can use this to inform
ourselves of operational risk under keychange. We can know as
individuals, as organizations what we will see, if keys change. I
think this is quite powerful. compared to measurement of what
resolvers see, or what authoritatives or roots see, going back to
these service-providers themselves. This method informs the client
side of the transaction. Thats big.

I'm not saying we shouldn't do other things, or measure. I'm saying
that this proposal has a qualitative aspect which I think is
different, and good.

-George

On Thu, Nov 16, 2017 at 4:23 PM, tjw ietf <[email-masked]> wrote:
> All
>
> The author has rolled out a new version addressing comments from the meeting
> on Monday, and we feel it’s ready to move this along.
>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Joe Abley <[email-masked]>
Date: Thu, 16 Nov 2017 17:18:32 +0800

I support adoption and am willing to contribute and review.

> On Nov 16, 2017, at 16:23, tjw ietf <[email-masked]> wrote:
> 
> All
> 
> The author has rolled out a new version addressing comments from the meeting on Monday, and we feel it’s ready to move this along.
> 
> Please review this draft to see if you think it is suitable for adoption by DNSOP, and comments to the list, clearly stating your view.
>

---

From: Paul Wouters <[email-masked]>
Date: Thu, 16 Nov 2017 06:05:40 -0500 (EST)

> Please review this draft to see if you think it is suitable for adoption by DNSOP, and comments to the list, clearly stating your view.

In favour of adopting, but still confused about parts of the draft. But
I agree with the idea.

> Please also indicate if you are willing to contribute text, review, etc.

willing to review.

Paul

---

From: Petr Špaček <[email-masked]>
Date: Thu, 16 Nov 2017 13:23:26 +0100

I support adoption, will review.

Petr Špaček  @  CZ.NIC

On 16.11.2017 09:23, tjw ietf wrote:
> Please review this draft to see if you think it is suitable for adoption
> by DNSOP, and comments to the list, clearly stating your view.
>

---

From: "A. Schulze" <[email-masked]>
Date: Thu, 16 Nov 2017 14:56:56 +0100

> https://datatracker.ietf.org/doc/draft-huston-kskroll-sentinel/

the draft uses Vnew Vold Vleg and nonV without description.
that makes it hard for me as I still do not fully understand the idea ...

Andreas

---

From: manu tman <[email-masked]>
Date: Thu, 16 Nov 2017 14:38:55 +0000

> that makes it hard for me as I still do not fully understand the idea ...

Well it is defined/described in section 3 but I agree that a high level
explanation in the terminology section would not hurt.

Manu

---

From: Matthew Pounsett <[email-masked]>
Date: Thu, 16 Nov 2017 09:58:00 -0800

> This starts a Call for Adoption for draft-huston-kskroll-sentinel
I support adoption, and I'm willing to review.  This seems like an
incredible valuable mechanism.

In fact, I already have some comments/questions:

In the last paragraph of section 2,
"This mechanism is to be applied only by resolvers that perform DNSSEC
validation ..."
I seem to recall there was some ambiguity in the signal from RFC 8145
validators caused by some implementations that might send a signal even if
they weren't actively validating.  I think it would be worthwhile to make
this statement stronger and more explicit that an implementation should
only enable this mechanism if it is actually configured to validate, and
not simply capable of validation.

The records being queried seem to be anchored at the root. Given that a
server in state Vleg is expected to return A responses that implies that
the root zone needs to include these records, but I don't see that stated
anywhere.. perhaps I just missed it?

At the bottom of section 3, the document gives two definitions for a Vleg
response.  One seems to be a typo and should be Vold.
"A Vleg response pattern says that the nominated key is not yet trusted by
the resolver in its own right."

---

From: Benno Overeinder <[email-masked]>
Date: Fri, 24 Nov 2017 14:53:00 +0100

I support adoption and will review the draft.

-- Benno

-- 
Benno J. Overeinder
NLnet Labs
http://www.nlnetlabs.nl/

---

From: "Paul Hoffman" <[email-masked]>
Date: Fri, 24 Nov 2017 08:41:08 -0800

I would like to see this draft adopted and worked on by the WG. Some of 
Ed's observations ring true for me as well, but I can see ways forward 
for all the ones that concern me.

--Paul Hoffman

---

From: tjw ietf <[email-masked]>
Date: Sat, 25 Nov 2017 20:13:34 -0500

FYI,  I'm taking notes of all the issues raised by folks in this thread
(thank you!) and will hold the authors accountable in addressing them.

tim

On Fri, Nov 24, 2017 at 11:41 AM, Paul Hoffman <[email-masked]>
wrote:

> Ed's observations ring true for me as well, but I can see ways forward for
> all the ones that concern me.
>
> --Paul Hoffman
>
>

---

From: Daniel Shaw <[email-masked]>
Date: Mon, 27 Nov 2017 15:33:10 +0400

> This starts a Call for Adoption for draft-huston-kskroll-sentinel

Support adoption too.

Thanks,
Daniel

---

From: Richard Barnes <[email-masked]>
Date: Mon, 27 Nov 2017 13:10:24 -0500

George, you should know better than to claim that a mechanism that requires
resolver updates will have "immediate benefit" :)

I do not find this mechanism terribly compelling.  It is not useful in the
short run, as noted above.  And it has the wrong architecture for the long
run.

What zone operators need, for KSK roll-overs and other evolution decisions,
is telemetry about the capabilities of the resolvers they serve.  In order
for an approach like this to provide that telemetry, one would need a
broad-scale client-side measurement system.  While such systems exist
(Geoff and George being expert practitioners), they have a lot of problems
-- they're expensive to operate at scale; they're extremely limited in
terms of what they can measure and how reliably; and they impose much more
overhead than is needed here.  We shouldn't be building a telemetry system
for the DNS that has hard-coded assumptions about web ads or dedicated
probes.

It would be far better to build a telemetry mechanism that operated
directly between resolvers and authoritative servers.  There are a variety
of ways you could do this.  In today's world, you could have some record by
which an authoritative server could advertise a telemetry submission
point.  In a DOH world, you could have the resolver provide a Link header
telling the authoritative server where it could pick up information about
resolver capabilities.  None of these are hard to build (and they don't
interfere with the "fast path" of the resolver) and they provide much more
high quality information.

If you need data for the KSK roll that we're already a decade late for,
gather it in a way that doesn't require a resolver upgrade.  (Deploy a
dedicated temporary TLD if you need to.)  If you're trying to solve the
long-run telemetry problem, then build it properly.

--Richard

On Thu, Nov 16, 2017 at 3:34 AM, George Michaelson <[email-masked]> wrote:

> meeting
> by

---

From: Richard Barnes <[email-masked]>
Date: Mon, 27 Nov 2017 13:36:17 -0500

Well, that's what I get for providing drive-by feedback.  Someone pointed
me off-list to RFC 8145 and the operational issues with that.  I still
think that that calls for a better authoritative/resolver telemetry
interface, not some client-side thing.

On Mon, Nov 27, 2017 at 1:10 PM, Richard Barnes <[email-masked]> wrote:

> George, you should know better than to claim that a mechanism that
> requires resolver updates will have "immediate benefit" :)
>
> What zone operators need, for KSK roll-overs and other evolution
> decisions, is telemetry about the capabilities of the resolvers they
> serve.  In order for an approach like this to provide that telemetry, one
> would need a broad-scale client-side measurement system.  While such
> systems exist (Geoff and George being expert practitioners), they have a
> lot of problems -- they're expensive to operate at scale; they're extremely
> limited in terms of what they can measure and how reliably; and they impose
> much more overhead than is needed here.  We shouldn't be building a
> telemetry system for the DNS that has hard-coded assumptions about web ads
> or dedicated probes.
>

---

From: Joe Abley <[email-masked]>
Date: Mon, 27 Nov 2017 14:06:31 -0500

As I imagine you've heard, part of the problem with resolver-authoritative telemetry interfaces is that the deployed infrastructure is not so simple; it also includes forwarders, changed resolvers, caches, middleware that interferes with the query path and/or drops queries that don't look normal... It's a jungle out there. Looking at the closest tree from just outside the edge of the jungle is actually much easier than speculating about what bizarre horrors lie within it.

> On 27 Nov 2017, at 13:36, Richard Barnes <[email-masked]> wrote:
> 
> Well, that's what I get for providing drive-by feedback.  Someone pointed me off-list to RFC 8145 and the operational issues with that.  I still think that that calls for a better authoritative/resolver telemetry interface, not some client-side thing.
> 
> George, you should know better than to claim that a mechanism that requires resolver updates will have "immediate benefit" :)
> 
> I do not find this mechanism terribly compelling.  It is not useful in the short run, as noted above.  And it has the wrong architecture for the long run.
> 
> What zone operators need, for KSK roll-overs and other evolution decisions, is telemetry about the capabilities of the resolvers they serve.  In order for an approach like this to provide that telemetry, one would need a broad-scale client-side measurement system.  While such systems exist (Geoff and George being expert practitioners), they have a lot of problems -- they're expensive to operate at scale; they're extremely limited in terms of what they can measure and how reliably; and they impose much more overhead than is needed here.  We shouldn't be building a telemetry system for the DNS that has hard-coded assumptions about web ads or dedicated probes.
> 
> It would be far better to build a telemetry mechanism that operated directly between resolvers and authoritative servers.  There are a variety of ways you could do this.  In today's world, you could have some record by which an authoritative server could advertise a telemetry submission point.  In a DOH world, you could have the resolver provide a Link header telling the authoritative server where it could pick up information about resolver capabilities.  None of these are hard to build (and they don't interfere with the "fast path" of the resolver) and they provide much more high quality information.
> 
> If you need data for the KSK roll that we're already a decade late for, gather it in a way that doesn't require a resolver upgrade.  (Deploy a dedicated temporary TLD if you need to.)  If you're trying to solve the long-run telemetry problem, then build it properly.
> 
> --Richard
> 
>

---

From: Richard Barnes <[email-masked]>
Date: Mon, 27 Nov 2017 14:47:36 -0500

As tempting as it may be to do the easy thing, it's not always the best use
of resources.  Looking at the closest tree might be easy for one observer,
but when you try to do it with enough observers to have a result that's
useful for the King of the Jungle, you end up with similar tangles.

I don't think that it make sense to infer from the failure of RFC 8145 that
resolver/authoritative telemetry isn't possible -- just that it's not
possible with the heavy-weight machinery in that mechanism.  To the degree
that the DNS still works at all, there must be some channel by which
information can be faithfully passed from authoritative to resolver, which
can presumably be used to bootstrap telemetry.  Maybe it's a TXT record
with an HTTP URL; maybe it's a funny CNAME.

Maybe you can't build a road through the jungle, but there are still rivers
that make it through, which can carry a message in a bottle.

On Mon, Nov 27, 2017 at 2:06 PM, Joe Abley <[email-masked]> wrote:

> As I imagine you've heard, part of the problem with resolver-authoritative
> telemetry interfaces is that the deployed infrastructure is not so simple;
> it also includes forwarders, changed resolvers, caches, middleware that
> interferes with the query path and/or drops queries that don't look
> normal... It's a jungle out there. Looking at the closest tree from just
> outside the edge of the jungle is actually much easier than speculating
> about what bizarre horrors lie within it.
>
> > Well, that's what I get for providing drive-by feedback.  Someone
> pointed me off-list to RFC 8145 and the operational issues with that.  I
> still think that that calls for a better authoritative/resolver telemetry
> > I do not find this mechanism terribly compelling.  It is not useful in
> the short run, as noted above.  And it has the wrong architecture for the
> long run.
> >

---

From: Joe Abley <[email-masked]>
Date: Mon, 27 Nov 2017 14:59:50 -0500

On 27 Nov 2017, at 14:47, Richard Barnes <[email-masked]> wrote:

> As tempting as it may be to do the easy thing, it's not always the best use of resources.  Looking at the closest tree might be easy for one observer, but when you try to do it with enough observers to have a result that's useful for the King of the Jungle, you end up with similar tangles.
> 
> I don't think that it make sense to infer from the failure of RFC 8145 that resolver/authoritative telemetry isn't possible -- just that it's not possible with the heavy-weight machinery in that mechanism.  To the degree that the DNS still works at all, there must be some channel by which information can be faithfully passed from authoritative to resolver, which can presumably be used to bootstrap telemetry.  Maybe it's a TXT record with an HTTP URL; maybe it's a funny CNAME.

In the case of the root zone, the additional complications include the procedural difficulty in installing experimental records in the zone itself, the amount of effort which has gone into reducing legitimate reasons for resolvers to send queries to the root servers at all, e.g. through the use of aggressive negative caching and even the TTLs on delegation NS sets, and the effort required to do data collection on the root servers, whose twelve operators manage many hundreds of instances. The existence of caching (and the degree to which people take liberties with what they can or should cache) also adds complication, not all of which is deterministic.

Although it is intuitively obvious that collecting data from millions of end-users is harder than collecting data on hundreds of root server instances, it turns out that there is a whole industry that more or less exists to collect data from end-users already and no corresponding mechanism for authority servers. Also counter-intuitively, some root server operators view the traffic they receive as containing PII whilst the privacy concerns with kskroll-sentinel seem benign.

When it comes down to it, you can more often rely upon an end-user's query being sent and a response being generated by a recursive server than you can rely upon queries and responses being exchanged within a (potentially complex) graph of resolvers and authority servers. That potential complexity means that understanding whatever data you do collect can be difficult.

> Maybe you can't build a road through the jungle, but there are still rivers that make it through, which can carry a message in a bottle.

You might see a river go in and a river come out and be tempted to assume that the bit in the middle is river-shaped, or that the two parts you can see are connected, but you won't always be right.

Joe

---

From: "Wessels, Duane" <[email-masked]>
Date: Mon, 27 Nov 2017 21:47:10 +0000

> On Nov 27, 2017, at 11:47 AM, Richard Barnes <[email-masked]> wrote:
> 
> I don't think that it make sense to infer from the failure of RFC 8145 ...

Do you really think RFC 8145 is a failure (even having only recently learned about its existence)?

No doubt there are complications and implementation hiccups but I think failure is a bit harsh.

DW

---

From: Edward Lewis <[email-masked]>
Date: Mon, 27 Nov 2017 22:12:50 +0000

A couple of reactions to this message:

On 11/27/17, 10:10, "DNSOP on behalf of Richard Barnes" <[email-masked] on behalf of [email-masked]> wrote:

> [one] should know better than to claim that a mechanism that requires resolver updates will have "immediate benefit" :)

I have been impressed by the impact of "Signaling Trust Anchor Knowledge in DNS Security Extensions (DNSSEC)".  Although the document was published in April, it seems that it has worked its way into operating servers quite quickly.  I see this as a testament to operators being able to pull off tech-refreshes more quickly as time progresses.  That is one aspect of this I am walking away with.

But "benefit."?  One can debate the benefit of the quickly adopted mechanism.  There's a lot of uncertainty in what is being seen, that is what I mean.  Not meaning to demean the new mechanism, it does raise concerns about new proposals - which may also come with uncertainty.

I recall a conversation from early in 2015, when I lamented the lack of feedback related to Automated Updates (RFC 5011's mechanism).  I was told that if we had something, given a lack of a baseline, we would get confused with the results.  Later, the "Signaling Trust Anchor Knowledge..." effort (RFC 8145) began and, well, that prophecy has been fulfilled.

That's not a failure, just a learning curve.  I'd not label it a failure as it hasn't lessened where we are, it has helped lead more investigation even if it hasn't given us something we fully understand yet.

>wrong architecture for the long run.

I'm unsure of that.  I commented earlier that having the query/response mechanism be something the operator of the trust anchor store (connected to a DNSSEC validator that is part of a DNS cache, etc.) can use is a good first step.  In the constrained use case of "me owning the test target" (by owning, I mean credentials to access the configuration files of the target) there's a lot we can get from a query/response mechanism.  With that, whatever can be generalized for a replying party to glean and then report via a higher-layer test would be good.

A (as opposed to "the") test harness (the higher-layer) could then be designed a few different ways.  Describing the one the draft has in mind is a good idea, what I mean is that there could be other ways.  I can understand reluctance to accept a particular way to perform the test, but to throw out the under layer with that bathwater isn't the best way to go.

I had originally set out to agree with the "wrong architecture" for this reason though - I'd wish that there was a better description of managing secure entry points and trust anchors along the lines of a three-way handshake protocol.  This begins with - how does a trust anchor manager know that it should be managing (pinning) keys for a domain, etc.?  I'm not sure we can get there though.

---

From: Wes Hardaker <[email-masked]>
Date: Mon, 27 Nov 2017 16:30:05 -0800

> meeting on Monday, and we feel it’s ready to move this along.

FYI,

I hate this document.  I hate the idea of introducing yet another magic
keyword into the DNS protocol that requires special handling.

That being said, even with the above grumpiness, I DO SUPPORT the
creation of the document (believe it or not) as I realize we don't have
much of a choice in order to solve the very-important signaling problem
we're current stuck with.  I'll review (already have; comments coming
later).
-- 
Wes Hardaker
USC/ISI

---

From: David Conrad <[email-masked]>
Date: Tue, 28 Nov 2017 09:48:24 -0800

On Nov 27, 2017, 11:47 AM -0800, Richard Barnes <[email-masked]>, wrote:

> I don't think that it make sense to infer from the failure of RFC 8145 that resolver/authoritative telemetry isn't possible

Huh? RFC 8145 wasn’t a failure — it was stunningly successful. Within a few months of publication it provided us insights we hadn’t before had into how the infrastructure was actually working.  This was unexpected.

> To the degree that the DNS still works at all, there must be some channel by which information can be faithfully passed from authoritative to resolver, which can presumably be used to bootstrap telemetry.  Maybe it's a TXT record with an HTTP URL; maybe it's a funny CNAME.

Maybe it is, and when we see another viable channel, we’ll undoubtedly make use of it. Until then, adding something like sentinel would seem to be a useful way of gathering information about the state of reality.

> Maybe you can't build a road through the jungle, but there are still rivers that make it through, which can carry a message in a bottle.

I don’t want to let the perfect be the enemy of the … well, perhaps not good, but functional.

Regards,
-drc

---

From: Matt Larson <[email-masked]>
Date: Wed, 29 Nov 2017 08:55:54 -0500

> On Nov 16, 2017, at 3:23 AM, tjw ietf <[email-masked]> wrote:
> 

I support the document and will review and provide comments.

Matt