Subject: [DNSOP] draft-ietf-dnsop-no-response-issue-03

From: Edward Lewis <[email-masked]>
Date: Thu, 18 Aug 2016 14:34:54 +0000

##   A Common Operational Problem in DNS Servers - Failure To Respond.
##                 draft-ietf-dnsop-no-response-issue-03

I have a lot of high-level concerns with this document.

##1.  Introduction
##
##   The DNS [RFC1034], [RFC1035] is a query / response protocol.  Failure
##   to respond to queries or to respond incorrectly causes both immediate
##   operational problems and long term problems with protocol
##   development.

While the latter is true, operationally the DNS is not strictly query -
response.  It's more like "query - if I want to respond".  I was
"enlightened" during a project 10-15 years ago when I realized that some
DNS implementations choose silence when deciding how to respond.

Based on this premise, the prescriptive language in sections 3 and 10
are very problematic in my opinion.

##2.  Common queries kinds that result in non-responses.

This section is not built on data or a document study, making it seem
flimsy, to wit:

##   Some servers fail to respond to ...

This doesn't establish a need to react to the situation.  "Some" might be
one operator's code, etc.

##3.  Remediating

This section steers far from the purpose of defining interoperability of the
protocol.  This section gets too specific regarding the current business
of DNS registration (registry and registrars) for technical needs.  I don't
think this section belongs in an IETF document.

##7.  Response Code Selection
##
##   NOERROR (no data) are the expected response codes.  A server is not
##   supposed to serve a zone which contains unsupported types ([RFC1034])
##   so the only thing left is return if the QNAME exists or not.  NOTIMP

But we have "Handling of Unknown DNS Resource Record (RR) Types" which
contradicts that "not supposed to".  This is the closest I'll come to a "nit."

##10.  IANA Considerations
##
##   IANA / ICANN needs to consider what tests, if any, from above that it
##   should add to the zone maintenance procedures for zones under its
##   control including pre-delegation checks.  Otherwise this document has
##   no actions for IANA.

There is a lot wrong with that paragraph.  (As in, I'm not sure where to
start.)  The IANA functions operator manages according to community defined
processes, there is no internal consideration of what to test.  Further,
only delegations from the root zone are considered, not anything lower in
the tree.  (I.e., most of the issues observed wouldn't be touched.)

This document would be more useful if it clarified the use of RCODEs in the
face of queries described here, in the vein of "Negative Caching of DNS
Queries (DNS NCACHE)".  We lack any document describing the circumstances
in which RCODE values are returned and what the appropriate reaction to them
is.  As the document stands, it is unspecific regarding issues, particularly
the operational scale, and ventures into policy instead of technical
directions regarding operations.

As an afterthought - Perhaps a whitepaper (non-IETF) can be produced to describe testing methodology and results observed would be an avenue to publicize the situation seen here.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 19 Aug 2016 00:58:54 +1000

> This section is not built on data or a document study, making it seem

There is plenty of data.  It's just not cited.

There is years of data at https://ednscomp.isc.org/compliance/summary.html
 
> contradicts that "not supposed to".  This is the closest I'll come to a "nit."

Go and read that RFC carefully.  It does not apply to meta queries.
Meta queries had reserved ranges when it was published.

> --B_3554361294_203519420
> Content-Type: application/pkcs7-signature; name="smime.p7s"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment; filename="smime.p7s"
> 
> MIIH2QYJKoZIhvcNAQcCoIIHyjCCB8YCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC
> BaIwggWeMIIEhqADAgECAhAE64fxtFjS2DdV8JfouoFSMA0GCSqGSIb3DQEBCwUAMGUxCzAJ
> BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
> dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IFNIQTIgQXNzdXJlZCBJRCBDQTAeFw0xNjA4MDkw
> MDAwMDBaFw0xOTA4MDkxMjAwMDBaMIG9MQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZv
> cm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxPDA6BgNVBAoTM0ludGVybmV0IENvcnBvcmF0
> aW9uIGZvciBBc3NpZ25lZCBOYW1lcyBhbmQgTnVtYmVyczEeMBwGA1UEAxMVRWR3YXJkIExl
> d2lzIDkgQXVnIDE2MSUwIwYJKoZIhvcNAQkBFhZlZHdhcmQubGV3aXNAaWNhbm4ub3JnMIIB
> IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxGXAQjOhBCDPz1+sGOERDMvSCFWbOUws
> GUrXAHLGeEAGYTCpni8f7kYPH7RMalvbep2aVtkMSUn6HR1uY84b437ZZuHCviUn7Aw6itGE
> wrDyq7Pb7zTlqE/1kLVhKYrHA4sjhsQRHhBHevxWbb3SYU2IMNxJd4QFgRJIp4zDmAR7bzbH
> 1ZazFGfo/op0QRsfcpFYmBotbj/4SnldtFwZasr7zTK9wJRSXa9sspLXtQhBe9itxTHJRg0H
> BH66VcPX7iRra9XFzkM5JLXOT2nBDIxeqDsLKoTkRWiNoSWoDZylAqS3BfBppwq7eremR7zD
> LIaPN4Tbb8TCpORMCZuvswIDAQABo4IB7zCCAeswHwYDVR0jBBgwFoAU5wIjgABP2Ne8lAvZ
> P3Q5STI8inkwHQYDVR0OBBYEFBiOkLRCGoHjShiTxeM3n94XHm+2MAwGA1UdEwEB/wQCMAAw
> IQYDVR0RBBowGIEWZWR3YXJkLmxld2lzQGljYW5uLm9yZzAOBgNVHQ8BAf8EBAMCBaAwHQYD
> VR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEMGA1UdIAQ8MDowOAYKYIZIAYb9bAQBAjAq
> MCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy5kaWdpY2VydC5jb20vQ1BTMIGIBgNVHR8EgYAw
> fjA9oDugOYY3aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJ
> RENBLWcyLmNybDA9oDugOYY3aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hB
> MkFzc3VyZWRJRENBLWcyLmNybDB5BggrBgEFBQcBAQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6
> Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcwAoY3aHR0cDovL2NhY2VydHMuZGlnaWNl
> cnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJRENBLmNydDANBgkqhkiG9w0BAQsFAAOCAQEA
> zhEJp9X6tm36lKKaxCdyjyETxL3oVPDmv2JHBD/T/xbqRH0vQfR34VkRnSO+5d8AqHGdhYTK
> 7yjB/vW52KjiQUiW9WnQFYdY5Q/Yv3cnVgi4zuuye9BPvyr87HbmJ0uafYjATnYziT71njO7
> xKIQP6w0MFv8ugT/fXIZsV4NU2eGQEHhvPkR+WOt0KfFa5jEY1qXoj4iZmE21j+f0OhSTA9K
> EofM5chR6FCaOyomKPIYU1mcoQwcistCwfcdVhUCpmgazxn6fR89ZOOiKTXxhOHrILdO0pCI
> dlJ3xO6EBvFKBCOyRBkD2z7jcWm/U3GkkYBLyRMvH1Ki+q4vwKnFZjGCAf8wggH7AgEBMHkw
> ZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRp
> Z2ljZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQgU0hBMiBBc3N1cmVkIElEIENBAhAE64fx
> tFjS2DdV8JfouoFSMAkGBSsOAwIaBQCgXTAjBgkqhkiG9w0BCQQxFgQU9we5Ko7uvDTaAky6
> aRtnlxHIYIMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTYw
> ODE4MTQzNDU0WjANBgkqhkiG9w0BAQEFAASCAQC49M9tdiQy66uHxpY8/4HF2UmggY9a7nl/
> qLAYkr8XRp0PBuEo8In6zmjavFWEoEaDIzVAVnXFv3BIkKADU/LAgvL8pf69MxZKGoclJwzy
> /ySVhTo3H9mKEPDvYg26Lfva2YNY5YliY77rfL26EXLdm0k6sv+zmq/bFb21ILgOZ+NUt036
> NNHkn4e5z2Dc2SwqVZyQXodLDVHvMhFZapswm6FBipH29YRCNpnrNyw/QWUFrGGK67e0pPvd
> IEU8Go40Wvq86F+tezMapUu8F4xuUZ1vbE7MHMsfPyqzRtkLZHLruCY7n7+6jO5gzfaauRkp
> jA8QN1zvZppw4xLqrYTB
> 
> --B_3554361294_203519420--
> 
> 
> --===============6037154100842559261==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
> 
> --===============6037154100842559261==--
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Tim Wicinski <[email-masked]>
Date: Thu, 18 Aug 2016 13:39:56 -0400

Mark and I sat down and talked about this draft in Berlin, and I have 
some strong concerns about specific sections (3 and 10), but I also love 
large parts of the draft.  I have a (rather) large sheet of edits for 
him that I promised him I would get to him.  I have failed him.  I will 
effort to get these out by the weekend.

tim

(Ed - now you may being mocking)

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 25 Aug 2016 04:35:52 +0000

> DNS implementations choose silence when deciding how to respond.

Of course nobody can compell a server to respond, and dropping some
queries may be unavoidable/necessary when under attack.  That said,
routine silence is substantially sub-optimal.  Especially (as is
typical) when said silence is the result of misguided "security
features" in middleboxes.

When a nameserver consistently fails to respond:

    * Queries will needlessly be sent to the remaining nameservers,
      increasing the traffic load.
    * For lack of a cached negative reply, queries will also be sent
      much more frequently.
    * DNS clients will experience significant latency retrying the
      query.
    * Security-aware applications may deduce a possible downgrade
      attack and defer or fail data transmission.

Similar considerations apply when inappropriate RCODEs are used in
replies for unsupported RRTYPEs that don't appear in the zone, and
thus a NODATA or NXDOMAIN would be the right response.  Compare:

    $ time dig +noall +ans +auth +comment +qu +cmd +nocl +nottl -t tlsa _25._tcp.pilot.jhuapl.edu
    ; <<>> DiG 9.10.3-P4 <<>> +noall +ans +auth +comment +qu +cmd +nocl +nottl -t tlsa _25._tcp.pilot.jhuapl.edu
    ;; global options: +cmd
    ;; connection timed out; no servers could be reached

    real    0m15.081s

with:

    $ time dig +noall +ans +auth +comment +qu +cmd +nocl +nottl -t aaaa _25._tcp.pilot.jhuapl.edu

    ; <<>> DiG 9.10.3-P4 <<>> +noall +ans +auth +comment +qu +cmd +nocl +nottl -t aaaa _25._tcp.pilot.jhuapl.edu
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40442
    ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ;; QUESTION SECTION:
    ;_25._tcp.pilot.jhuapl.edu. IN AAAA

    ;; AUTHORITY SECTION:
    jhuapl.edu.             SOA     infoblox-prod-gm.jhuapl.edu. nic-contact.jhuapl.edu. 13899920 10800 1080 2592000 600

    real    0m0.120s

Which would you say is the correct nameserver behaviour?
Given:

    $ dig +short -t mx jhuapl.edu | sort -n
    40 pilot.jhuapl.edu.
    40 piper.jhuapl.edu.

and that the same behaviour is observed for the TLSA records of
both MX hosts, the jhuapl.edu domain is a blackhole for email via
DANE-enabled sending MTAs.

> one operator's code, etc.

While the situation is improving, non-response is still a real
issue, with some domains having all non-responsive nameservers,
and others having a subset of non-responsive nameservers.

-- 
	Viktor.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 25 Aug 2016 14:51:47 +1000

See also https://ednscomp.isc.org/compliance/alexa-report.html

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 25 Aug 2016 09:25:45 +0200

On Thu, Aug 25, 2016 at 04:35:52AM +0000,
 Viktor Dukhovni <[email-masked]> wrote 

> When a nameserver consistently fails to respond:

Add "it may make easier for a third-party to inject bogus
responses". See
<http://www.ssi.gouv.fr/uploads/IMG/pdf/DNS-OARC-2013-Blocking_DNS_Messages_Is_Dangerous.pdf>

---

From: william manning <[email-masked]>
Date: Thu, 25 Aug 2016 03:47:41 -0700

I'm with Ed here,  A valid response is silence.  The resolver/client has no
way to determine if the lack of a reply is due to the server has chosen
silence, or if there was something in-path which dropped the packet.  In
this case, client misbehaviour is panicking and sending many queries to try
and gain the information it things it needs.  WindowsXP had this
behaviour.  Servers can and do "blackhole" queries the operator deems
irrelevant/excessive with hold-down and supression capabilities.  The fix,
if there is one needed, needs to sit at the resolver/client side.

/Wm

---

From: Tony Finch <[email-masked]>
Date: Thu, 25 Aug 2016 12:10:37 +0100

william manning <[email-masked]> wrote:

> I'm with Ed here,  A valid response is silence.

I think it is important for people producing and deploying DNS server
software and DNS-interfering middleboxes to understand the bad
consequences of dropping queries or responses. If you understand these
effects and still think you can improve things by dropping packets, then
maybe go ahead. But it isn't a simple valid / invalid binary choice.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Southeast Fitzroy: Northerly or northeasterly 5 or 6, occasionally 7 later.
Moderate or rough. Thundery showers. Good occasionally poor.

---

From: william manning <[email-masked]>
Date: Thu, 25 Aug 2016 08:59:21 -0700

On Thursday, 25 August 2016, Tony Finch <[email-masked]> wrote:

> william manning <[email-masked] <javascript:;>> wrote:
>

> Where does the "badness" occur? The server or resolver?    The rational
> for a server to silently ignore a query often revolves around malformed
> queries ...  Should a server attempt to answer malformed queries or
> silently drop them?

What about client that will not shut up? Should rate limiting be frowned on
> and the server attempt to answer all received queries?

> Some food for thought

---

From: Tony Finch <[email-masked]>
Date: Thu, 25 Aug 2016 17:23:59 +0100

william manning <[email-masked]> wrote:

> Where does the "badness" occur? The server or resolver?

Both. The resolver suffers extra latency; the server suffers extra traffic
- even a well-behaved resolver has to retry in this situation.

> The rational for a server to silently ignore a query often revolves
> around malformed queries ...  Should a server attempt to answer
> malformed queries or silently drop them?

See section 7 of the draft. It would be reasonable to rate-limit
responses.

This kind of nuance is what this draft should discuss.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Trafalgar: In southeast, cyclonic, mainly easterly 6 to gale 8. In northwest,
northerly or northeasterly 5 or 6, occasionally 7 later. Moderate or rough. In
southeast, showers. In northwest, thundery showers. In southeast, moderate or
good. In northwest, good occasionally poor.

---

From: Marek Vavruša <[email-masked]>
Date: Thu, 25 Aug 2016 11:11:22 -0700

> This kind of nuance is what this draft should discuss.

+1, there are other implications besides performance. For example
attacker can silence
the NS for victim (either on path or off path with spoofed source
subnet). If successful,
the attacker doesn't have to race NS->victim RTT anymore for
successful cache poisoning.

Marek

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 25 Aug 2016 21:19:28 +0200

On Thu, Aug 25, 2016 at 11:11:22AM -0700,
 Marek Vavruša <[email-masked]> wrote 

> successful cache poisoning.

Which is exactly the attack explained in the OARC talk I cited a few
emails before.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 26 Aug 2016 07:56:35 +1000

Not answering queries has effects on OTHER servers.  Because there
are servers out there that don't answer EDNS queries or only answer
the first EDNS query resolvers have to ASSUME that no answer to a
EDNS quere means "NO EDNS".  They then make a plain DNS query and
get a answer.  Those servers then get marked for a while as not
supporting EDNS.  This isn't a big issue until those servers are
serving a signed zone.  Once that server serves a signed zone you
have real issues as DNSSEC responses rely on EDNS queries and you
stop getting EDNS queries.  The client then doesn't get the RRSIGs
and can't validate.

PMTUD from the server to the client has a similar loss pattern to
drop on EDNS, answer on DNS.  Do you see the problems caused to
others by deciding to drop packets in a query / response protocol.

The same thing happens with servers that don't answer EDNS options.
The recursive server has to decide is this the EDNS option or EDNS
or just packloss.  Add in servers that don't like particular options
(yes there are servers that don't answer NSID queries but do answer
other EDNS options) and the problem for the recursive servers becomes
bigger for the recursive server.

We already have DNS zones that will not work with BIND 9.10.x and
BIND 9.11.x because the servers for those zones drop queries with
EDNS options present and the zones are signed.

The CORRECT behaviour for a recursive server is to retry the SAME
query.  No answer means PACKET LOSS, usually for load but sometimes
for corruption.  It does NOT mean that I don't like the query.  We
have a rcode for that: REFUSED.

When named drops packet (too many queries for the same question)
the expected behaviour from the client is to timeout and retry.
Its load shedding the same way as a link sheds load by dropping
packets.  It also isn't based on a EDNS option, EDNS flag, EDNS
version, QTYPE, QCLASS, DNS flag, opcode or similar.  It purely
load based.

For completeness named can also be configured to drop queries based
on address but is drops all queries so it behaves like a dead server
to that client.

A nameserver / firewall should not be looking at query content and
deciding not to answer based on that content.

If you don't want to implement a EDNS than don't implement it.  If
you don't want to use a EDNS option with some client then just
ignore the option.  Similarly for EDNS flags.  The client is expecting
that unsupported options and flags will be ignored not used to
decide to drop a query.

Mark

---

From: Edward Lewis <[email-masked]>
Date: Fri, 26 Aug 2016 11:19:03 +0000

If this is your intended message, stick to that in the draft.  I.e., don't go beyond the message by requiring responses to queries, reinforce how to respond if the server isn't implementing EDNS0.  Don't direct operators to perform maintenance checks, that has little to do with properly implementing the EDNS mechanism, stick with making sure implementers know what to code up.  Maybe this is "clarifications on EDNS response behavior" and not "no response issue".

---

From: Mark Andrews <[email-masked]>
Date: Fri, 26 Aug 2016 22:14:33 +1000

In message <[email-masked]>, Edward Lewis write
s:
> If this is your intended message, stick to that in the draft.  I.e., don't go
>  beyond the message by requiring responses to queries, reinforce how to respo
> nd if the server isn't implementing EDNS0.  Don't direct operators to perform
>  maintenance checks, that has little to do with properly implementing the EDN
> S mechanism, stick with making sure implementers know what to code up.  Maybe
>  this is "clarifications on EDNS response behavior" and not "no response issu
> e".

It's not just EDNS.  "dig +ad +noedns" -> no response.

If you would answer any query from a address you need to answer ALL
query types from a address.  Resolvers shouldn't have to play 50
queries to get a answer.

The draft doesn't require EDNS.  It requires that EDNS be fully
implemented if you implement EDNS.

Mark