Subject: [DNSOP] raising the bar: requiring implementations

From: Job Snijders <[email-masked]>
Date: Sat, 24 Mar 2018 11:07:56 +0000

Dear DNSOP,

I'd like to share some pointers from the working group that governs the
BGP protocol, IDR, on requiring implementations before drafts can
advance towards RFC publication. Raising the bar for publication will
take weight off the camel's back.

The IDR policy and rationale is outlined here: https://trac.ietf.org/trac/idr/wiki

An example of an implementation report is available here:
https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-large-community%20implementations
Every normative (RFC 2119 / RFC 8174) term should expand into a
compliance test, and it is particularly good to document where
implementations deviate from what is required or suggested in the
proposed specification. The work listed on this wikipage was done
_before_ the draft was submitted to the IESG, and we intentionally
sought to create more than the minimum amount of implementations
required.

In the case of the BGP large communities project the working group was
particularly lucky because Pier Carlo Chiodi created an open 'regression
testing'-style environment into which BGP speakers could be plugged in
to assess their compliance:
https://github.com/pierky/bgp-large-communities-playground
https://github.com/pierky/bgp-large-communities-playground/blob/master/tests/README.md

The DNS world would benefit from such environments and automated
compliance testing. Manual testing for a specification (that is still
being worked on) can be tedious, having a 'playground' can pay huge
dividend. We did catch bugs in implementations thanks to this
environment, so it not only helped the specification, but increased the
quality of the participating implementations.

RFC 7942 suggest that During the development of a draft people are
encouraged to document what implementations exist, an example is here:
https://tools.ietf.org/html/draft-ietf-idr-large-community-12#section-7

Closed source implementations as part of such a list is not an issue
(just a little bit more work), at IETF meetings we'd meet up, plug
laptops with virtual machines into each other and run compliance tests.
Sidenote: when IANA codepoints are needed but not yet assigned, don't
forget to keep everything on separate beta/feature branches; don't ship
software with squatted codepoints :-)

The DNSOP working group will have to figure out what constitutes a
meaningful implementation in what context. For example, a tcpdump or
wireshark decoder would hardly count as an implementation of a proposed
DNS feature, but those decoders are _incredibly_ useful to have
alongside real implementations, especially during development. DNS
people will probably want to see multiple implementation reports in
context of packet decoding, stub, resolver, and authoritative.

Kind regards,

Job

---

From: Petr Špaček <[email-masked]>
Date: Mon, 26 Mar 2018 14:45:14 +0200

> sought to create more than the minimum amount of implementations

I very much support this approach.

BTW we already have test framework which was used to test BIND, Unbound,
Knot Resolver, and PowerDNS recursor! I.e. we have one tool which is
able to test all of these using the same test data, so the remaining
work here is writting tests, not creating test tools from scratch.

Petr Špaček  @  CZ.NIC

---

From: Willem Toorop <[email-masked]>
Date: Wed, 28 Mar 2018 16:52:03 +0200

I would love to see a hard requirement for implementations &
implementation reports (like IDR has) in the charter or in the working
group house rules.  Early implementations (perhaps even during the
hackathon) can reveal implications that might have been missed while
designing the draft.  In addition it is a good way to see if everybody
interprets a draft the same way, by interoperability testing.

Op 24-03-18 om 12:07 schreef Job Snijders:
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: tjw ietf <[email-masked]>
Date: Wed, 28 Mar 2018 10:55:13 -0400

I would say that most things we have adopted in the past few years do have
some implementations to reference.
Not when drafts are adopted, but generally before we head to WGLC I've
always wanted to see someone
who implemented the option in some manner.

But yes, agree.

---

From: Mukund Sivaraman <[email-masked]>
Date: Wed, 28 Mar 2018 20:49:39 +0530

> who implemented the option in some manner.

I'd raise the bar even higher, to see complete implementation in a major
open source DNS implementation when it applies. Sometimes implementation
problems are very revealing (client-subnet should have gone through
this).

		Mukund

---

From: bert hubert <[email-masked]>
Date: Wed, 28 Mar 2018 17:24:33 +0200

> problems are very revealing (client-subnet should have gone through

Well to allow the one remaining closed source DNS implementation some room,
I think we could live with a 'demo' from them if they'd want to. This would
lead to an implementation report, much like is customary in the BGP WGs.

But otherwise, +100. 

This might go for MIXFR which we are discussing now btw.  It looks nice in
theory, but I wonder about the practice, and if the people who want this
(TLD operators I guess) would be willing to test it in simulated production
to see if it fits their needs.

	Bert

---

From: Matthijs Mekking <[email-masked]>
Date: Wed, 28 Mar 2018 17:29:18 +0200

> problems are very revealing (client-subnet should have gone through

As mentioned in the meeting, I am in favor of requiring implementations 
before drafts become standards.

However, I would be opposed to limit acceptable implementations to the 
few major open source DNS implementations (define major). It should be 
acceptable for other organizations or just persons to contribute a 
reference implementation.

Best regards,

Matthijs

---

From: tjw ietf <[email-masked]>
Date: Wed, 28 Mar 2018 11:33:04 -0400

An enterprise company with rather large zone which update often are "highly
interested" in MIXFR.
But we may be the exception rather than the rule.

Tim

---

From: Tony Finch <[email-masked]>
Date: Wed, 28 Mar 2018 16:46:52 +0100

> Well to allow the one remaining closed source DNS implementation some room,

authoritative services: Akamai Amazon Cloudflare Dyn Google Verisign
recursive services: Google OpenDNS Quad9
COTS: Nominum

 ... I have probably missed several ...

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Humber, Thames, Dover: Cyclonic becoming west or northwest 5 or 6. Slight or
moderate. Rain then showers. Moderate or poor, becoming good.

---

From: Job Snijders <[email-masked]>
Date: Wed, 28 Mar 2018 15:55:10 +0000

> recursive services: Google OpenDNS Quad9

Those can just as well participate in implementation compatibility
matrices. While open source can make interop testing easier, it is
certainly not a requirement and shouldn't be treated as such.

Kind regards,

Job

---

From: Paul Vixie <[email-masked]>
Date: Wed, 28 Mar 2018 11:45:58 -0700

Matthijs Mekking wrote:
>>> I would say that most things we have adopted in the past few
>>> years do have some implementations to reference. Not when drafts
>>> are adopted, but generally before we head to WGLC I've always
>>> wanted to see someone who implemented the option in some manner. >>>
>>> But yes, agree.
>>

i'm in general agreement with each of the assertions made at each layer 
of quoting above, but i have two quibbles.

first, they aren't reference implementations. not even BIND, which for 
many years i called a reference implementation, is not one. a reference 
implementation is a special kind of beast, it's something that if you 
don't interoperate with it, you are in the wrong. we have a 
specification, and we judge the quality of that specification by the 
ease with which interoperable non-reference implementations can be made.

second, i think it's 2018, and we can require that at least one of the 
demonstrated interoperable implementations be source-available. (not 
open source; we don't care about license, only transparency.)

-- 
P Vixie

---

From: Frederico A C Neves <[email-masked]>
Date: Wed, 28 Mar 2018 16:28:15 -0300

Bert,

> to see if it fits their needs.

We'll definitely test this on real data with open-source clients and
with our own server implementation.

> 	Bert

Fred

---

From: Frederico A C Neves <[email-masked]>
Date: Wed, 28 Mar 2018 16:30:24 -0300

>  ... I have probably missed several ...

MS

---

From: Mukund Sivaraman <[email-masked]>
Date: Thu, 29 Mar 2018 01:18:36 +0530

On Wed, Mar 28, 2018 at 05:29:18PM +0200, Matthijs Mekking wrote:
> However, I would be opposed to limit acceptable implementations to the few
> major open source DNS implementations (define major). It should be
> acceptable for other organizations or just persons to contribute a reference
> implementation.

It depends on the topic of the draft of course, esp. where in operations
it applies. If it is nameserver territory, I absolutely want to see an
implementation in *any* of the major DNS implementations. By major, I
mean the popular ones (e.g., PowerDNS, NSD, Unbound, Knot, etc.) This is
because:

* A full-fledged nameserver is somewhat different from a toy
  implementation in performance and scalability (this point is from
  experience with a bad implementation of a draft)

* The rest of us want to see proof that it can be implemented (not just
  a promise or mention of implementation) and play with it and observe
  operational characteristics _freely_, and determine whether a draft
  will really improve things in the way it says it will. E.g., take all
  the multiple-answer drafts that are making the rounds.. in Singapore
  there was a presentation of a grand matrix of them, but who knows how
  they actually perform?

It's 2018. We aren't living in the dark ages with a single DNS
implementation.

If a draft is for nameserver software to implement, and if the authors
cannot implement it by themselves, they can persuade one of the open
source vendors to do so. If they are unable to persuade any, that should
be enough consensus about how significant that draft is. Speaking for
myself, we in our DNS implementation add support for several drafts
early in the draft stage because they look necessary or interesting, or
because we want to know how they behave early on.

		Mukund

---

From: Mukund Sivaraman <[email-masked]>
Date: Thu, 29 Mar 2018 01:21:43 +0530

> mean the popular ones (e.g., PowerDNS, NSD, Unbound, Knot, etc.) This is

Sorry, that should read major *open source* DNS implementations! That's
the point.

		Mukund

---

From: Mukund Sivaraman <[email-masked]>
Date: Thu, 29 Mar 2018 01:32:23 +0530

Also, this "Camel" in the presentation may have been a mythological
beast, but the real and existing DNS Camels that struggle with their
backs breaking under the weight of DNS are the DNS implementations,
especially the open source ones that are underfunded and understaffed.

If someone wants to add more weight on top, do so by helping by
implementing it for one or more of the majorly used open source
implementations. I can't agree that a closed-source or an example
implementation of itself will suffice, when the burden eventually comes
onto us.

Other than that, the "Camel" that everyone feels sorry for doesn't
exist.

		Mukund

---

From: Arsen STASIC <[email-masked]>
Date: Thu, 29 Mar 2018 09:59:46 +0200

>recursive services: Google OpenDNS Quad9

Quad9 is using powerdns dnsdist as loadbalancer and powerdns recursor and unbound [0], [1]

[0] https://www.makeuseof.com/tag/quad9-dns-overview/
[1] 
https://www.theinquirer.net/inquirer/news/3021536/ibm-teams-with-global-cyber-alliance-to-launch-quad9-a-free-public-domain-name-service-system

-arsen

---

From: Tony Finch <[email-masked]>
Date: Thu, 29 Mar 2018 13:11:57 +0100

> >  ... I have probably missed several ...

D'oh!

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Lundy, Fastnet: Cyclonic 4 or 5. Moderate or rough, occasionally very rough in
southwest Fastnet. Thundery showers. Good, occasionally moderate.

---

From: Tony Finch <[email-masked]>
Date: Thu, 29 Mar 2018 13:13:49 +0100

Arsen STASIC <[email-masked]> wrote:
> [1] https://www.theinquirer.net/inquirer/news/3021536/ibm-teams-with-global-cyber-alliance-to-launch-quad9-a-free-public-domain-name-service-system

Thanks for the correction!

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
South Utsire, Forties, Cromarty: Easterly 5 to 7, occasionally gale 8 at first
except in Cromarty. Moderate or rough. Occasional sleet. Good, occasionally
moderate.

---

From: "Jan Komissar (jkomissa)" <[email-masked]>
Date: Thu, 29 Mar 2018 18:23:39 +0000

Cisco

On 3/29/18, 8:12 AM, "DNSOP on behalf of Tony Finch" <[email-masked] on behalf of [email-masked]> wrote:

    D'oh!
    
    Tony.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 29 Mar 2018 15:05:19 -0400

On Wed, Mar 28, 2018 at 2:45 PM, Paul Vixie <[email-masked]> wrote:

>
> i'm in general agreement with each of the assertions made at each layer of
> quoting above, but i have two quibbles.
>
> implementation is a special kind of beast, it's something that if you don't
> interoperate with it, you are in the wrong. we have a specification, and we
> judge the quality of that specification by the ease with which
> interoperable non-reference implementations can be made.
>
> second, i think it's 2018, and  can require that at least one of the
> demonstrated interoperable implementations be source-available. (not open
> source; we don't care about license, only transparency.)

​Quite, a reference implementation is not a production implementation. In
fact it may well not even be standards compliant because the most important
parts of an implementation to test are what happens when messages are not
as expected.

Production implementations should be forgiving in what they accept and
conservative in what they generate. Reference implementations should be the
opposite.

I generate my deployment and reference implementations from the same source
but they are not the same thing.

---

From: Suzanne Woolf <[email-masked]>
Date: Thu, 29 Mar 2018 15:18:45 -0400

> problems are very revealing (client-subnet should have gone through

This is actually quite a good example of another problem: Client-subnet was documented for Informational purposes; it was already in wide use in the public internet and had an EDNS opt code codepoint allocated from the IANA registry.

Nothing that happened in DNSOP or the IETF changed that, and I strongly suspect nothing that happened in DNSOP or the IETF could have changed it.

Other documents we’ve considered on features or modifications to the protocol include refuse-any and, I believe, serve-stale, and the stated purpose of the localhost draft is to clarify the existing documentation on the reserved name “localhost”.

Should we refuse to document such things because they’re not in well-known open source codebases, or have otherwise not passed tests of goodness?

It’s not a rhetorical question. Given that people are extending the protocol outside of the IETF or any other formal process, in order to solve their own technical and business problems, it makes sense to ask ourselves periodically whether we should be encouraging them, trying to stop them, or something in between.

Suzanne

---

From: Mukund Sivaraman <[email-masked]>
Date: Fri, 30 Mar 2018 01:35:35 +0530

On Thu, Mar 29, 2018 at 03:18:45PM -0400, Suzanne Woolf wrote:
> This is actually quite a good example of another problem:
> Client-subnet was documented for Informational purposes; it was
> already in wide use in the public internet and had an EDNS opt code
> codepoint allocated from the IANA registry.
> 
> Nothing that happened in DNSOP or the IETF changed that, and I
> strongly suspect nothing that happened in DNSOP or the IETF could have
> changed it.

We found issues in the client-subnet description (in the draft stages)
that were corrected before it became RFC - this involved some behavioral
changes. However, the opportunity was not given to address fundamental
design issues, so what's in the RFC largely documents the
implementations preceding it and still has issues. I didn't think
client-subnet was in wide use when it came to dnsop. Even today it
doesn't look like it's in wide use.

You have asked an interesting question, because what happens if
something is already being used and there's no chance to update/correct
problems because that would make it a different protocol?

IMO, we should not put anything through dnsop that does not allow
reviewing and correcting problems. It is pointless for dnsop to shepherd
a draft to RFC when there's no possibility to influence it.

During later stages of the draft via dnsop, we implemented client-subnet
(resolver side) and found various problems. Some of these were addressed
in the draft, but it was revealing how poor the state of the
design/draft was. This is why I strongly suggest: A code contribution of
a _complete implementation_ of everything described in the draft to one
of the open source DNS implementations should come sometime during
adoption, so that

(a) issues in production implementation are revealed

(b) it can be tried in the real world to understand how it behaves.

As you're a co-chair:

When Bert did that Camel presentation, I felt hooray, here's one of us
who's talking about what the steady churn of ideas and RFCs is doing to
DNS implementations. We finish implementing one draft and at almost
breathless pace, a few more drafts queue up. It's not sustainable,
esp. as all the existing features also need to be maintained for years.

Introducing a new RR type that doesn't involve additional processing is
relatively trivial and nobody objects about it.

Introducing something like TCP pipelining, or rather, out-of-order query
processing, may seem trivial when describing the change, but
implementing it may need fundamental restructuring of the
implementation. Proper implementation of client-subnet needs change
everywhere within a nameserver from the client message parsing code to
the data structures used for cache, and how cache lifetime is managed.

Implementations are being stretched and bent 5 different ways to adapt
to the length and breadth of all these newfangled DNS items that
literally are showing up at an alarming pace.

Bert really hit the spot at the right time. Something needs to be done
to check what becomes an RFC. A good way is to require an open source
implementation. If draft authors cannot supply that or convince an open
source implementation to write support for it, it can go back to where
it came from.

		Mukund

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 29 Mar 2018 16:59:56 -0400

On Thu, Mar 29, 2018 at 3:18 PM, Suzanne Woolf <[email-masked]>
wrote:

>
> Should we refuse to document such things because they’re not in well-known
> open source codebases, or have otherwise not passed tests of goodness?
>
> It’s not a rhetorical question. Given that people are extending the
> protocol outside of the IETF or any other formal process, in order to solve
> their own technical and business problems, it makes sense to ask ourselves
> periodically whether we should be encouraging them, trying to stop them, or
> something in between.
>

​There are in fact precedents for doing something of the sort. But not ones
that I think we should follow.

DNSSEC would have been implemented as part of the VeriSign ATLAS roll out.
The reason it was not was it simply could not be deployed using the tech
available with the spec as it was at the time without a major increase in
cost and complexity.

A blanket requirement for open source implementations may well have the
same effect.

Operating systems at Internet scale is vastly more complex than running
systems at network scale. It is not just the genius of the original
Internet designers that has allowed the Internet to scale from 100 nodes to
10 billion. There is a vast amount of unseen engineering work that is
equally heroic and some of that infrastructure comes with some very
specific constraints.

Some of the most useful work in IETF is making closed systems more open.
Often those projects require some extra slots to carry information that is
needed for some legacy system. I see no value to saying folk can't have a
smooth transition path from proprietary to open.

---

From: Matthijs Mekking <[email-masked]>
Date: Fri, 30 Mar 2018 09:44:39 +0200

I can agree with the argument that if implemented in a major open-source 
DNS implementation it would weigh in more to the discussion, but 
limiting to is far too restricting in my opinion.

Best regards,
   Matthijs