Subject: [Doh] Proposal to close off these threads

From: Paul Hoffman <[email-masked]>
Date: Sun, 10 Jun 2018 00:29:20 +0000

Greetings. Yeah, this one is difficult. Well-intentioned folks with decades of experience disagree, and yet we need to move forward. It would be good to do so using definitive language that does not presuppose particular understanding of the DNS protocols. The best way to do this is to make changes to the specific media type defined in this document, not to the entire protocol. (We were sometimes sloppy in the draft about differentiating the protocol from this specific media type.)

Proposal:

7.  Definition of the application/dns-message media type

   The data payload in this media type is the DNS on-the-wire format defined
   in [RFC1035]. The format is for DNS over UDP.  Although [RFC1035] says
   "Messages carried by UDP are restricted to 512 bytes", that was later
   updated by [RFC6891]. This media type restricts the size of the DNS
   message to 65535 bytes. Note that the wire format used in this media type
   is different than the wire format used in [RFC7858].

   DNS API clients using this media type MAY have one or more EDNS options
   [RFC6891] in the request. DNS API servers using this media type MUST ignore
   the value given for the EDNS UDP payload size in DNS requests.

   The use of the AXFR and IXFR QTYPEs with this media type is not yet
   defined.

   When using the GET method, the data payload for this media type MUST be
   encoded with base64url [RFC4648] and then provided as a variable named
   "dns" to the URI Template expansion. Padding characters for base64url MUST
   NOT be included.

   When using the POST method, the data payload for this media type MUST NOT
   be encoded and is used directly as the HTTP message body.

People can later define new media types, such as to handle larger DNS messages, to handle new QTYPEs or Meta-TYPEs, and so on. People can also later update this media type to define how to use AXFR and/or IXFR in a predictable and interoperable fashion.

--Paul Hoffman

---

From: bert hubert <[email-masked]>
Date: Mon, 11 Jun 2018 00:08:41 +0200

On Sun, Jun 10, 2018 at 12:29:20AM +0000, Paul Hoffman wrote:
> particular understanding of the DNS protocols.  The best way to do this is
> to make changes to the specific media type defined in this document, not
> to the entire protocol.  (We were sometimes sloppy in the draft about
> differentiating the protocol from this specific media type.)

While I understand it will not make everyone happy, I heartily support the
wording below, especially in the interest of getting this transfer method
for DNS out there in 2018.

	Bert

---

From: Patrick McManus <[email-masked]>
Date: Sun, 10 Jun 2018 19:31:07 -0400

I think we can make progress here. I'm pleased to see the text restricted
to the media type. I don't think it should explicitly restrict axfr, but
rather be about size.

This both makes the reasoning clearer and hits the target more accurately
in edge cases (doesn't apply to small axfr and does apply to a huge rr set
of key types and txt records or something). I'm happy if we also mention
many [ai]xfr require more than one traditional wireformat message and so
won't fit in this media type. Its good to note, non-normatively, that a
server that cannot satisfy a request because of this media type's size
restriction can use 413 to signal that.

make sense?

On Sun, Jun 10, 2018 at 6:08 PM, bert hubert <[email-masked]>
wrote:

> _______________________________________________
> Doh mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/doh
>

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 10 Jun 2018 23:53:15 +0000

On Jun 10, 2018, at 4:31 PM, Patrick McManus <[email-masked]> wrote:
> 
> I think we can make progress here. I'm pleased to see the text restricted to the media type. I don't think it should explicitly restrict axfr, but rather be about size.
> 
> This both makes the reasoning clearer and hits the target more accurately in edge cases (doesn't apply to small axfr and does apply to a huge rr set of key types and txt records or something). I'm happy if we also mention many [ai]xfr require more than one traditional wireformat message and so won't fit in this media type.

I would rather not mention stuff we can't back up. If a developer wants to do ?XFR over this transport and have others interoperate with them, it will probably take much less time for them to write the (hopefully very short) Internet Draft than it will for them to eventually have to debug and patch their code when someone else makes different assumptions.

> Its good to note, non-normatively, that a server that cannot satisfy a request because of this media type's size restriction can use 413 to signal that.

"can use"? "SHOULD use" except in situation X? "MUST use"?

--Paul Hoffman

(PS: The above proves that the document authors are not carefully coordinating their responses off-list...)

---

From: Patrick McManus <[email-masked]>
Date: Sun, 10 Jun 2018 19:08:35 -0500

On Sun, Jun 10, 2018 at 6:53 PM, Paul Hoffman <[email-masked]>
wrote:

> > I think we can make progress here. I'm pleased to see the text
> restricted to the media type. I don't think it should explicitly restrict
> axfr, but rather be about size.
> >
> > This both makes the reasoning clearer and hits the target more
> accurately in edge cases (doesn't apply to small axfr and does apply to a
> huge rr set of key types and txt records or something). I'm happy if we
> also mention many [ai]xfr require more than one traditional wireformat
> message and so won't fit in this media type.
>
> I would rather not mention stuff we can't back up. If a developer wants to
> do ?XFR over this transport and have others interoperate with them, it will
> probably take much less time for them to write the (hopefully very short)
> Internet Draft than it will for them to eventually have to debug and patch
> their code when someone else makes different assumptions.
>
>
I'm working very hard to keep this in the realm of concrete.

There has a been a convincing case that a > 64KB axfr responses uses 2
wireformat records in TCP today and therefore won't fit in the DoH MTI
wireformat media type. AIUI that's not because its AXFR, but because it is
>64KB, right? At the same time tale convincingly argues he has plenty of
<64KB zones that only use one message and match our MTI fine.

There are also, admittedly fringe, cases where you can create > 64KB
responses not using ?XFR at all.

Therefore, match the restriction to what the problem is.. in other words
the problem is that the MTI wireformat can only carry 64KB.

And then, as I mentioned, its useful to have non-normative text mentioning
that ?XFR is of particular concern here.

> > Its good to note, non-normatively, that a server that cannot satisfy a
> request because of this media type's size restriction can use 413 to signal
> that.
>
sorry about suggesting 413 - that's wrong as it is meant to apply to
request bodies. We're looking for 406.

"HTTP defines status code 406 for cases where the server cannot generate a
representation suitable for the client". We can probably just say that.

> --Paul Hoffman
>
> (PS: The above proves that the document authors are not carefully
> coordinating their responses off-list...)

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 11 Jun 2018 00:50:21 -0400

As should surprise absolutely nobody, I don't think this would be good
to adopt.  It's trying to address a legacy issue that hasn't been
demonstrated and instead introduces a new one.  If it is what is
ultimately adopted I want it to be clear for the archives that the
decision was not made based on technical merits but rather
philosophical grounds that have been externalized when they could be
handled internally by an implementation.  It's kicking the can down
the road; we expect better of our politicians (who consistently fail
to come though) and I believe we can do better.

The legacy issue is fundamentally different from other cases where
we've had to be conservative in the DNS because we were rebuilding the
airplane in flight.  We had to interoperate with existing production
software in a heterogeneous environment using only the tools that were
given to us by RFC 1035, because messages would be pumped directly
into software that had been written under a different understanding of
the standards.

DoH is a new tool though that changes nothing at all that happens on
DNS/UDP or DNS/TCP.  There is no good reason why legacy code should
end up dealing accidentally with wire format messages larger than 64k
because it doesn't have any standard way of getting them that isn't
the result of new code.

I totally get that there are these legacy functions that you want to
reuse with DoH.  It makes completely rational sense, as does the fact
that they were coded with an understanding that they wouldn't deal
with messages larger than 64k. Yet even if DoH specifies no limits,
you can still reuse them.  The new code that you have calling those
functions can trivially enforce the limits that you want, either
encoding or decoding.

For encoding, nothing at all changes from your existing code.  DNS
server software already has to deal with the possibility that the data
producer might have asked for more data to be encoded than can
actually fit in the message.  It just keeps filling up the buffer
until it hits the 64k limit and it's done.

For decoding, you're going to have to be sanitizing Content-Length
anyway.  I'm confident that none of you would just blindly pass the
message body into a parser without checking the length, especially if
the draft says there's a 64k max.  You can make the exact same check
even if DoH says messages can be greater than 64k.  Your end result is
the same.  

Without a clear technical objection, the other issues raised are
philosophical.

You don't think DNS messages should ever be greater than 64k, because
who needs messages that big anyway?  Fair enough, I respect that point
of view even despite having previously stated that I could make use of
messages that big.  I don't think email messages should ever be bigger
than a couple of megabytes either, but I can just set my server to
reject those messages.  As noted above, a DoH implementation can
easily do similar on either side of the transaction.

You don't want your resources used up on potentially large messages?
I get that, I hate when my resources get used up too.  Of course
that's not a new risk of unlimited-DoH; if you're telling me you have
insufficient cache control tools to manage that, I've just been told
of an attack vector exploitable by non-DoH DNS.

A message greater than 64k wouldn't be representable on DNS/UDP or
DNS/TCP?  This is an issue that goes back the founding of the DNS;
there's always been the potential for a message to only be fully
available on one transport.  Not a new problem.

Setting a limit on DoH, even constrained to one particular media type,
creates a new legacy problem because if larger messages are eventually
deemed acceptable then you'll have an installed base that all wrote in
a limit per this standard.  I posit that even despite Paul's attempt to
make more clear the media type versus the substrate, the chances that
the limit is incorrectly associated as a property of https versus a
property of the media type are high.

Putting in the limit on this media type sends a signal that, despite
the ability of a DNS wire format to handle messages greater than 64k
and a new transport that is not hamstrung by the three decade old
design decisions, people should continue to write software that uses
this arbitrary limit in all contexts.

We already expect a new media type, JSON, that will not have natively
have any sense of the size of a DNS wire format encoding by which to
adjudge the propriety of its own encoding.  Are we going to hamstring
that too?  If not, why not?

The limits you need for your antique network of T1s and PDP-11s
shouldn't be foist on people writing software for the 21st century.

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 11 Jun 2018 01:15:36 -0400

Patrick McManus writes:
> Therefore, match the restriction to what the problem is.. in other words the
> problem is that the MTI wireformat can only carry 64KB.

Spit-balling here, but what about two mandatory-to-implement wire
format types, with the only difference being that one, dns-message-64k
puts a hard cap at 64k, while the other, dns-message(-unlimited)?, has
basically a soft cap at 64k for people who are fronting legacy
handlers, until such time as that code is, if ever, updated.

Yeah I don't expect that to make the people who want the limit very
happy either, but there it is.  I'm not in love with it, but that
seems like the closest thing to a compromise on the issue that I can
envision at the moment if a limit really has to be hung on a media
type.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 11 Jun 2018 11:59:17 +0200

On Sun, Jun 10, 2018 at 12:29:20AM +0000,
 Paul Hoffman <[email-masked]> wrote 

> Proposal:

Alternate proposal, because the original one was not clear to me:

    The data payload in this media type is the DNS on-the-wire format defined
    in section 4.2.1 of [RFC1035]. The format was for DNS over UDP.
    Although [RFC1035] says
    "Messages carried by UDP are restricted to 512 bytes", that was later
    updated by [RFC6891]. This media type restricts the size of the DNS
    message to 65535 bytes. Note that the wire format used in this media type
    is different than the wire format used in [RFC7858] (which uses
    the format defined in section 4.2.2 of [RFC1035]).

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 11 Jun 2018 14:00:20 +0000

>    the format defined in section 4.2.2 of [RFC1035]).

Thanks, this is clearer than what I wrote.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 11 Jun 2018 15:26:00 +0000

Cherry-picking some places to respond to this...

> On Jun 10, 2018, at 9:50 PM, Dave Lawrence <[email-masked]> wrote:
> 
> It's trying to address a legacy issue that hasn't been
> demonstrated

This feels dismissive of the other developers in the WG who said that they do see the issue in the code they are writing. You might not see it in the code you are writing, but that doesn't mean it hasn't been seen.

> and instead introduces a new one.

It does not introduce a legacy issue: it introduces a restriction for a completely new media type.

> because it doesn't have any standard way of getting them that isn't

Multiple developers on the list have talked about creating gateways. You are correct that there is no standard way to create a DNS transport gateway, but that's true of nearly every protocol gateway used across the Internet. The times that the IETF has tried to create standards for gateways have often ended up pretty terribly; ample data is available by looking in the archives of the MIXER and IPP WGs.

> available on one transport.  Not a new problem.

Yes, it is. There were two transports defined with no explicit provision for more. This WG has created a new problem.

> property of the media type are high.

Developers of other data formats have successfully dealt with differing media types for 25 years. If the DNS folks screw this up worse than everyone else, well, there is nothing we can do about that. I would prefer to think that we can learn from them.

> adjudge the propriety of its own encoding.

No, we absolutely do not expect a JSON format that has the same capabilities of the wire format. I have tried repeatedly for many years to get interest in standardizing one, and there was a complete lack of interest in anything other than a small subset of DNS messaging. This is well-documented in the DNSOP WG archives.

>  Are we going to hamstring

Because, if it exists, it will be an new media type. Each media type has its own rules. There is nothing in current DOH spec that limits what a media type can do.

--Paul Hoffman

---

From: Tony Finch <[email-masked]>
Date: Mon, 11 Jun 2018 16:54:18 +0100

> <64KB zones that only use one message and match our MTI fine.

It's more complicated than that :-)

Small zones can be transferred using multiple messages too.

A DoH proxy might be able to defragment a small multi-message transfer,
provided there isn't any TSIG authentication.

Large zones with more than 64K records must use multiple messages, not
because of the 2 byte message size, but because of the 2 byte ANCOUNT.

> representation suitable for the client". We can probably just say that.

My DoH server sends a 200 OK HTTP response containing a NOTIMP DNS response.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Bailey, Fair Isle, Faeroes: West or northwest 3 or 4, increasing 5 at times in
Fair Isle. Slight or moderate. Rain at times. Good, occasionally moderate.

---

From: Dave Lawrence <[email-masked]>
Date: Mon, 11 Jun 2018 12:42:00 -0400

Paul Hoffman writes:
> This feels dismissive of the other developers in the WG who said
> that they do see the issue in the code they are writing. You might
> not see it in the code you are writing, but that doesn't mean it
> hasn't been seen.

I'm not being dismissive of them.  I am observing that I haven't seen
a description of a real problem that they have where a message larger
than 64k is getting into existing software that isn't a result of new
code being written, code which can trivially check to make sure there
isn't a problem.

I am really quite okay with being corrected on this point, being shown
a real, practical problem that exists that is solved by insisting on a
limit for all implementations.  I am also really quite okay with being
gently reminded that such an example has already been provided, so
that I can have the opportunity to evaluate it.

> for a completely new media type. 

I described further down in my message how this creates a new legacy
issue, but causing new code to be written with the unnecessary limit
and further solidifying the notion that this maximum length is an
immutable characteristic of DNS messages.  My contention is that even
though you're trying to attach it to a specific media type, if that
media type is the one-and-only media type that starts off being
defined for DoH it will be very easy to programmers to confuse it as
an attribute of the transport rather than an attribute of only one
media type, leading to deployment problems in the future.

> Yes, it is. There were two transports defined with no explicit
> provision for more. This WG has created a new problem. 

I really don't get how this remark is in disagreement with my
assertion.  You're disputing that the UDP/TCP DNS already has
situations where some messages can only be representable on one (TCP)
transport?  That's what my remarks were about, that philosophically
this is not a new concept and not a technical reason to claim that
messages over 64k should be verboten.

> >  Are we going to hamstring [JSON] too?  If not, why not?
> 
> Because, if it exists, it will be an new media type. Each media type
> has its own rules. There is nothing in current DOH spec that limits
> what a media type can do. 

I agree with you on that last bit, but it shoots right past the point.
If JSON is going to be allowed to carry DNS data that would ultimately
represent DNS wire format messages in excess of 64k, then why should
an explicitly DNS wire format media type be unnecessarily constrained?
I need to use an even more voluminous media type in order to get the
large messages that I want?

---

From: Patrick McManus <[email-masked]>
Date: Mon, 11 Jun 2018 17:36:48 -0700

> Small zones can be transferred using multiple messages too.
it doesn't need to be more complicated. The mti wireformat is defined as
one message (or if its not - it should be clarified) that might be limited
to 64KB. I thought one message was already clear and we were debating the
64KB point.

so if you can't meet those constraints, you can't use that media format.
but its not inherently about the query type.

> My DoH server sends a 200 OK HTTP response containing a NOTIMP DNS
Personally, I think you're better off with the HTTP error in this case as
the problem is with finding a matching HTTP transport.

---

From: Tony Finch <[email-masked]>
Date: Tue, 12 Jun 2018 14:31:44 +0100

> to 64KB. I thought one message was already clear and we were debating the

I was talking about how to handle query meta-types rather than response
sizes. My point was that there are several reasons an AXFR or IXFR query
can break the single message requirement.

> but its not inherently about the query type.

I think I would prefer it if the client could predict the response media
type based on the query, rather than the server choosing a media type
based on criteria outside the client's control. This is a programmer
interface so I think predictability is more important than DWIMmery.

> the problem is with finding a matching HTTP transport.

I don't think that is supported by the current draft text: it explicitly
says that DNS errors should be returned as 200 OK application/dns-message
responses, and there's no other defined media type that we're failing to
negotiate.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: Northerly or northwesterly 4 or 5, occasionally 6 in east. Slight
or moderate. Fair. Good.

---

From: Patrick McManus <[email-masked]>
Date: Tue, 12 Jun 2018 10:23:05 -0700

> responses, and there's no other defined media type that we're failing to

we're more aligned that you think :)

DNS errors should indeed be 2xx responses (the draft doesn't specify the
media type, correctly)

I would just argue that the situation you are describing is a HTTP error,
not a DNS one. Let's assume a response that requires multiple messages -
you've got that information at the DNS layer, you just can't encode
transport it at the HTTP layer using a media type the client can
understand. So that's an http level failure.

---

From: Tony Finch <[email-masked]>
Date: Wed, 13 Jun 2018 14:28:46 +0100

> understand. So that's an http level failure.

The reason I think it's a DNS-level error is that AXFR and IXFR are just
examples of the more general problem of query meta-types.

The DNS machinery needs special-case code to handle a meta-type: if it's a
proxy it won't (in general) be able to parse the response it gets from
upstream; if it's an integrated DoH server, it won't even know how to
begin generating a response. So the problem occurs at DNS query dispatch
time, well before it gets to the point of thinking about rendering the
response for HTTP.

My code looks at the qtype and generates a DNS error if it is an unknown
meta-type (128 <= qtype <= 254).

There are similar considerations for DNS opcodes.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
promote human rights and open government