Subject: Re: [DNSOP] [dns-operations] dnsop-any-notimp violates the DNS standards

From: Paul Wouters <[email-masked]>
Date: Mon, 9 Mar 2015 10:48:15 -0400 (EDT)

On Mon, 9 Mar 2015, D. J. Bernstein wrote:

> My "qmail" software is very widely deployed (on roughly 1 million SMTP
> server IP addresses) and, by default, relies upon ANY queries in a way
> that is guaranteed to work by the mandatory DNS standards.

And you've been told for two decades that this was wrong?

> Specifically, query type ANY "matches all RR types" for that node on
> that server.

Wrong, query type ANY "matches all RR types CURRENTLY IN THE CACHE". So
the result of qmail's ANY query is completely meaningless and qmail
cannot derive any conclusion from the absence of any record from that
query.

So if the MX or AAAA record has expired from the cache but another RRtype
with larger TTL (say NS) is still in there, your ANY query will fail to
find records. qmail with this feature is broken.

Additionally, Tony Finch did a write up of qmail's ANY problems too:
https://fanf.livejournal.com/122220.html

> In new software today I would sacrifice these efficiency
> benefits for the sake of simplicity, but this doesn't mean that I'm
> going to frivolously inflict retroactive punishment upon administrators
> who have installed standards-compliant software and done nothing wrong.

You have had 10 years to fix it. Luckilly, I believe most distributions
shipping qmail add the patch to fix this already.

> I understand how a sufficiently large site might acquire the impression
> that it can safely take radical action at its own whim, violating the
> existing protocol standards

Uhm, we pointd out qmail's _bug_ for a decade. I'm quite sure even you
do not need to interop with BIND4 anymore.

> Apparently Firefox recently deployed ANY queries. I haven't looked at
> the details but I gather that they're related to the well-known
> annoyances of handling AAAA etc. Firefox was browbeaten into reverting
> this change on the basis of highly questionable claims regarding
> amplification: "It can return enormous result sets, and some
> authoritative servers have taken to refusing ANY queries because of the
> frequency with which such queries show up in amplification attacks" ->
> "I'm concerned about amplification and the perception thereof by
> security monitors."

No, they were also told that ANY queries only return data from the
cache, and using ANY queries means you might miss actual A or AAAA
records. This has nothing to do with ANY queries and amplification.

> The common theme of CNAME/MX/A and A/AAAA is that there's widepread
> interest in being able to easily retrieve multiple record types. What
> I'm saying is not that query type ANY is the ultimate answer (clearly it
> can be improved); what I'm saying is that these are protocol issues, and
> that protocol changes need to be handled by an appropriately chartered
> IETF working group.

I agree there is a use for this. I tried a few years ago to introduce a
new EDNS0 option that would allow you to query for a bitmap number of
RRsets, but people did not like it. Perhaps the WG is ready for
something like this now.

Paul

---

From: Tony Finch <[email-masked]>
Date: Mon, 9 Mar 2015 14:54:26 +0000

> that is guaranteed to work by the mandatory DNS standards.

There are three bugs in the way qmail uses ANY queries.

(1) qmail uses ANY queries for domain canonicalization on outgoing
messages, as specified by RFC 1123. But canonicalization is not required
by the current SMTP specification. It is a waste of time. Fixing this bug
would make bug (3) moot.

(2) qmail's DNS response buffer is too small to accommodate a complete DNS
message, so it fails if it gets a large response. It uses the low-level
libc resolver API which can easily handle large responses, including
fallback to TCP, so it is a pity that qmail breaks this part of the
resolver's functionality. This bug means it is not guaranteed to work.

(3) Using an ANY query suppresses alias processing, so qmail makes a
series of queries to follow CNAME chains. This is inefficient and
wasteful. If you make an A or MX query, the DNS server will chase the
CNAME chain for you, so you only need to make one query to get the
canonical name.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: Easterly 5 or 6 in far southeast, otherwise northerly 4 or 5.
Moderate or rough. Mainly fair. Good.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 9 Mar 2015 16:08:38 +0100

On Mon, Mar 09, 2015 at 11:08:03AM -0000,
 D. J. Bernstein <[email-masked]> wrote 
 a message of 111 lines which said:

>    * First: The proposed protocol modification has to be taken to an
>      IETF working group chartered to modify the protocol, so that
>      stakeholders will have a proper chance to evaluate and comment on
>      the proposal.

djb defending the IETF processes. It made my day :-)

---

From: Edward Lewis <[email-masked]>
Date: Mon, 9 Mar 2015 15:16:30 +0000

>that protocol changes need to be handled by an appropriately chartered

(I removed the dns-operations list from this because this needs to be
discussed on DNSOP.)

Dan,

You're right.  But.

Operators are not bound to comply with what the IETF documents.

As much as operators shouldn't bully the IETF nor the public for which the
serve, the street goes two ways.  The IETF is not able to and should not
bully them.  The IETF is supposed to provide us with an interoperable way
to operate and is supposed to have documents that reflect "running code."

>  * Second: The merits of the protocol modification have to be properly
>    discussed in that working group, to evaluate the costs and benefits
>    of the protocol modification---and to consider whether there are
>    better ways to achieve the same benefits.

If operators find that a protocol needs to be modified to be properly
operated, the IETF ought to adjust the protocol definition.  Having a
migration path would be a plus too.  (Said tongue in cheek.)

But - "finding that a protocol needs to be modified" is not as easy as
that - hence my quoting of your words above.

Ed Lewis

---

From: bert hubert <[email-masked]>
Date: Mon, 9 Mar 2015 16:18:12 +0100

> that is guaranteed to work by the mandatory DNS standards.

Hi Dan,

The way I read RFC 1034 4.3.2, this is not true. In step 4 we match whatever
we find in the cache, put it in the packet, and move on to step 6.

This means the algorithm might terminate returning only an A record or only
an AAAA record.  Or a TXT record for that matter.

This reading of 4.3.2 makes 'ANY' queries to resolvers fragile. It might not
return what you need.

Do you think I read 4.3.2 wrong? Or is there another RFC that updates the
algorithm?

	Bert

---

From: Jared Mauch <[email-masked]>
Date: Mon, 9 Mar 2015 11:23:29 -0400

> On Mar 9, 2015, at 10:54 AM, Tony Finch <[email-masked]> wrote:
> 

Even ignoring if qmail is “broken”.  (I would rather classify it as, could do
better), depreciating the ANY qtype is going to have some significant side
effects of users troubleshooting DNS problems.

I’m very sensitive to the abuse of ANY queries, but this is something that
I feel there are sufficient controls that exist to mitigate the issues,
namely using TC=1 to direct well behaving clients to receive a valid response.

dnsop-any-notimp violates the principle of least surprise in technology by
returning NOTIMP where Paul Vixie suggested NOERROR/ANCOUNT=0 would be more
appropriate with the existing definitions.

Much of this is triggered by bad coding practices and bad networking examples
that are littered around codebases, e.g.: gethostbyname() vs getnameinfo() and
by broken behaviors by nscd and other OS/LIBC implementations that also violate
the principle of least surprise.

- Jared

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Mar 2015 11:08:03 -0000

Specifically, query type ANY "matches all RR types" for that node on
that server. There's an example in RFC 1034 of how a CNAME record is
returned by "a type CNAME or * query". There's nothing telling clients
to avoid this query type; it's perfectly valid for a client to treat a
server that refuses this query type as a broken server, because that's
exactly what the server is. Of course, there's no guarantee of which RR
types for a node are available at a cache, but a client is guaranteed to
be able to detect CNAME records from responses to query type ANY (as
qmail does), since a CNAME type prohibits all regular RR types.

I started using these standard ANY queries for interoperability reasons
(working around a BIND bug and matching sendmail behavior at the time)
but the choice continues to provide some efficiency benefits, larger
than many of the efficiency benefits used as rationales in IETF
protocols. In new software today I would sacrifice these efficiency
benefits for the sake of simplicity, but this doesn't mean that I'm
going to frivolously inflict retroactive punishment upon administrators
who have installed standards-compliant software and done nothing wrong.

Let's now take a look at draft-ogud-dnsop-any-notimp-00.txt with this
background in mind. The I-D specifies behavior that

   * violates the existing mandatory DNS standards and that
   * breaks interoperability with this standards-compliant use of ANY.

An accompanying blog post says that "in a few weeks" the author's
organization will begin deploying this standards violation---with the
effect of immediately damaging Internet email delivery. The blog post
and ID contain a number of dubious assertions attempting to justify this
change.

I don't mean to suggest that protocols must never change in incompatible
ways. However, modifications to IETF's standard protocols need to be
handled through the established IETF procedures, with appropriate
respect for the existing standards and the installed base:

   * Third: _If_ the benefits of the modification are judged to outweigh
     the costs, a sunset period---in this case, a timeline for the
     client to stop using ANY queries---has to be specified, to avoid
     interoperability problems. This period has to be several years,
     recognizing the time required for client administrators to hear
     about and carry out the necessary redeployment. (It's not as if
     we're talking about an emergency security change.)

   * Fourth: After the sunset period expires, the server will be free to
     use the modified protocol---in this case, to refuse ANY queries.

I understand how a sufficiently large site might acquire the impression
that it can safely take radical action at its own whim, violating the
existing protocol standards and as a result creating interoperability
problems---but this is making a mockery of the IETF standardization
process. This is _not_ a mere operational change within the existing
protocol; it is _not_ a private extension using the standard negotiation
mechanisms; it is a flagrant violation of the required DNS standards.

My understanding is that [email-masked] is not chartered to make DNS
protocol changes, so any discussion here will have to be repeated in an
appropriate working group, but let me nevertheless comment on the two
benefits claimed for having servers refuse ANY queries:

   * Refusal would reduce DNS amplification: This argument already seems
     to have been dismissed by various people, and doesn't seem to be
     defended. It's obviously less effective than standards-compliant
     approaches such as limiting UDP responses to 512 bytes.

   * "Attempting to handle ANY queries creates enormous complexity in
     our DNS server code base": This is a quite puzzling claim,
     especially since the specified features (load balancing, geoip,
     etc.) have been supported for many years by other software that has
     no trouble handling ANY. What exactly is the claimed difficulty in
     copying records from, e.g., the "A" key to the "*" key in the
     underlying database?

---Dan

---

From: Jared Mauch <[email-masked]>
Date: Mon, 9 Mar 2015 11:36:01 -0400

> On Mar 9, 2015, at 11:16 AM, Edward Lewis <[email-masked]> wrote:
> 
> to operate and is supposed to have documents that reflect "running code.”

I would be interested in hearing the results you had from disabling ANY
queries, or anyone else results.  

This isn’t standing in opposition to change, but trying to understand the true
scope and nature of this problem.  Perhaps I’m missing parts of the problem, but
the qmail issue has existed for 10 years.  Firefox recently turned on and back off
ANY queries in 36.0 and 36.0.1 to try to keep performance what it should be
for websites that have low TTLs vs being ‘sticky’ to an old A/AAAA record.

> that - hence my quoting of your words above.

I’m concerned a change of this sort will cause a number of people to see
something as ‘broken’ where it really is not.  If we are dealing with code
that will break things for existing users, giving them broad warning is
important, even if they are using/abusing this capability of the DNSs
system today.

- Jared

---

From: Tony Finch <[email-masked]>
Date: Mon, 9 Mar 2015 16:03:23 +0000

Jared Mauch <[email-masked]> wrote:
> better)

Yes.

> appropriate with the existing definitions.

This would have the amusing effect of disabling qmail's address
canonicalization without patching qmail.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rockall, Malin: Cyclonic in north Rockall at first, otherwise westerly or
southwesterly severe gale 9 to violent storm 11, decreasing 5 or 6. High or
very high, becoming very rough. Squally showers. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Mon, 9 Mar 2015 16:09:44 +0000

> This reading of 4.3.2 makes 'ANY' queries to resolvers fragile. It might not

Dan is aware of that, but this particular oddity isn't a problem for
qmail. Later in his message he wrote:

> > Of course, there's no guarantee of which RR types for a node are
> > available at a cache, but a client is guaranteed to be able to detect
> > CNAME records from responses to query type ANY (as qmail does), since
> > a CNAME type prohibits all regular RR types.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Fair Isle: Southeast veering west, severe gale 9 to violent storm 11. Very
rough or high, becoming very high for a time. Rain then showers. Moderate or
poor, becoming good.

---

From: David C Lawrence <[email-masked]>
Date: Mon, 9 Mar 2015 12:23:44 -0400

RFC 1035 explicitly allows for a server to indicate that a kind of
query is not implemented.

Whether it is a good idea to respond to ANY this way is a separate
argument that is worth having.  You just won't win on the foundation
that it is a violation of the standard.

---

From: "John Levine" <[email-masked]>
Date: 9 Mar 2015 16:44:23 -0000

>that is guaranteed to work by the mandatory DNS standards.

All the qmail installations I know patched that about a decade ago to
be a CNAME lookup, because the ANY query caused interop problems.

It's not a very strong argument in favor of ANY.

R's,
John

---

From: Edward Lewis <[email-masked]>
Date: Mon, 9 Mar 2015 16:55:58 +0000

>queries, or anyone else results.

We got a few complaints to the help desk.  To quiet this, we resumed
answering with a mind to stopping again in the future.  The next was to
take on messaging towards a resumption of disabling ANY responses, this
did happen (and there are presentations in some archives as a result of
this) including the set of slides I refer to as my manifesto (DNS-OARC,
Dublin workshop).

Then events overcame concern about the ANY load.  Our system became
over-provisioned enough to handle the extra load we experienced and we
developed a solution to handling DDos flood attacks.  Inside operations,
the ANY queries were no longer a concern.

I'll add that what never rose up as an issue, as is mentioned in the blog
post, was the complication in assembling the ANY response in face of
tailored responses. From this I cannot conclude anything, either we didn't
see the complication or we never looked hard enough.  But I can see that
this could be an issue.

>for websites that have low TTLs vs being ‘sticky’ to an old A/AAAA record.

I think you are correct is saying that there are missing parts of the
problem.  Not saying "you are missing" but I believe that there hasn't
been a formal tradeoff analysis documented and made public.  I'm a bit
concerned that some have proposed expanding the scope to include other
queries (meta- and RRSIG) without having the problem fully discussed.

And this opens the door to whether "Firefox was brow-beaten" or observed
load caused Firefox to be convinced to reverse course.

>important, even if they are using/abusing this capability of the DNSs

While I am of the opinion that disabling ANY queries would be a good thing
to do, I'll admit that I am not certain.  I could be wrong.  I haven't
undertaken any such study because my motivation to spend time on this has
evaporated (no longer being an operator for one).  But I will contribute
my experiences and hypothesis to anyone performing the study.

---

From: Suzanne Woolf <[email-masked]>
Date: Mon, 9 Mar 2015 13:07:33 -0400

Hi,

(chair hat on)

To the question of what's on charter for DNSOP:

On Mar 9, 2015, at 7:08 AM, D. J. Bernstein <[email-masked]> wrote:

> appropriate working group

From our charter as approved (http://datatracker.ietf.org/wg/dnsop/charter/):

"4. Publish documents to address operational issues with the DNS
protocols by extending or performing protocol maintenance 
on them. Act as focal-point for operator discussion and provide 
advice to the Ops ADs and other WGs on EDNS0 options, new 
RRTYPEs, DNSSEC, record synthesis, or other mechanics of 
extending DNS to support other applications.

5. Serve as a home for drafts that document the problem space 
around existing or new DNS issues. The group, with the advice 
and consent of the responsible AD in coordination with other areas, 
will then decide whether these issues belong in DNSOP under 
the existing charter and, if not, will work with the authors and 
appropriate ADs to determine the proper place for the work to be 
carried out. "

It seems to your chairs at least that discussion is in scope for DNSOP, in that we can decide any of: 

a) there's no work here, per open discussion in the WG with operators and implementers;

b) there's work here, per open discussion, and it's within item 4 above, so we can do it in DNSOP;

c) there's work here but it needs to go to another WG, in accordance with item 5, perhaps one to be spun up for the purpose, as was done with DPRIVE

At the very least, the draft seems to have generated some interest, and we're happy to host this initial discussion in DNSOP as an initial step in order to decide whether/how to proceed.

thanks,
Suzanne & Tim

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Mar 2015 23:18:37 -0000

> Operators are not bound to comply with what the IETF documents.

As I said before, this is making a mockery of the IETF standardization
process. Instead of

   * obeying the existing mandatory standards,
   * giving due respect to the installed base relying on the standards,
   * trying to build consensus for a transition that demands action from
     the installed base, and
   * taking the slow steps necessary to maintain interoperability during
     a transition if any transition happens,

a large operator is using its market position to violate the standards
and _create_ interoperability failures as a tool to enforce a protocol
change that it wants. Furthermore, a few weeks before this standards
violation is going to be deployed, the stated rationale for the change
is undergoing massive revisions, making serious discussion difficult and
leading observers to wonder how carefully the change was thought through
in the first place.

If you want IETF standards to be taken seriously---if you think that the
basic rules of Internet communication should be established by consensus
in IETF, and not simply overridden by future developers and operators
who think they know better, including cases where you _don't_ agree with
them---then you have to stop endorsing standards violations.

Tony Finch writes:
> qmail uses ANY queries for domain canonicalization on outgoing
> messages, as specified by RFC 1123. But canonicalization is not
> required by the current SMTP specification. It is a waste of time.

I fully agree that this was made optional in SMTP---that qmail is no
longer required to do this. But how do you leap to the wild conclusion
(stated twice in your message) that this is a "bug" in qmail?

More importantly, why do you think this is relevant to anything that I
said? I didn't say that qmail's behavior is currently required for SMTP.
I said that qmail is very widely deployed and relies upon ANY queries in
a way that is guaranteed to work by the mandatory DNS standards. The
dnsop-any-notimp proposal ignores those standards and will create real
interoperability problems with mail delivery.

> Using an ANY query suppresses alias processing, so qmail makes a
> wasteful.

No, you have the efficiency picture backwards. CNAME chains have always
been an extremely small fraction of the DNS queries inside mail, while
the QTYPE=ANY side effect of preloading MX/A records has always produced
a significantly larger reduction in DNS queries.

This item is something else that you explicitly label as a "bug". You
keep using that word; I do not think that word means what you think it
means.

> qmail's DNS response buffer is too small to accommodate a complete DNS
> message, so it fails if it gets a large response.

Precisely how many bytes do you believe are in "a complete DNS message"?
65535, the TCP limit? Do you seriously believe that 65535-byte responses
work reliably today, and that any failure to handle such responses is a
"bug"? How about 512, given the fact that the mandatory DNS standards do
_not_ require TCP support? Or maybe 1280, the recommended safe size for
EDNS0 UDP (depending on the network etc.)?

Even in an imaginary world where 65535-byte responses work, what do you
think happens if a server administrator puts _more_ than 65535 bytes of
records at a single node? Do you blame the server administrator? If DNS
implementors handle this use case by introducing a DNS transport capable
of handling 4-gigabyte responses, will you then claim that there's a
"bug" in every DNS client that has less RAM available or that simply
insists on a smaller limit to control its resource use?

In fact, all DNS client and server deployments have size limits on DNS
responses, and these limits have always varied, making the system as a
whole increasingly fragile for the unfortunate administrators pushing
the limits (notably DNSSEC administrators). The only sane way out is for
the protocol to declare a single reasonable size limit to be respected
by all clients and servers---but this implies reengineering large DNS
record types to split data across nodes (the same way that TCP splits
streams across packets), and nobody seems willing to do this work. It's
much easier to stuff all data into one node, pretend that the system
works, and blame the administrators for all of the resulting failures.

I'd be happy to discuss this issue further if anyone is interested in
fixing these aspects of the DNS protocol. I would, however, suggest
starting a separate thread for that, since this really isn't relevant to
how dnsop-any-notimp violates the DNS standards.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Mar 2015 12:59:13 -0000

Paul Wouters writes:
> So if the MX or AAAA record has expired from the cache but another
> RRtype with larger TTL (say NS) is still in there, your ANY query will
> fail to find records.

The client is behaving correctly. The ANY query isn't guaranteed to find
the MX, but you're wrong in claiming that the client is relying on this.
I realize that you don't understand how this type of DNS client works,
so let me go through the details, slowly, covering

   (1) why the queries reliably produce the desired information and
   (2) why QTYPE=ANY ends up reducing the number of queries that the
       authoritative server has to handle for producing this information.

The client begins with an ANY query to the cache. There are several
possibilities for the cache state at this moment:

   * maybe there's nothing;
   * maybe there's an MX record;
   * maybe there's an A record;
   * maybe there's some other regular record, such as NS;
   * maybe there's a mix of regular records;
   * maybe there's a CNAME record;
   * etc.

In the "nothing" case, the cache forwards the query to the server. This
could end up retrieving, e.g., an MX set, an A set, and an NS set; it
could end up retrieving a CNAME; there are many possibilities.

The cache then returns whatever it has to the client. The client notices
failure cases (such as SERVFAIL) and, in those cases, defers mail
delivery. Let's now focus on what happens in the success cases.

If the cache returns a CNAME: The client sees the CNAME, replaces the
original name with the CNAME, and starts over.

If the cache returns, e.g., an NS record: The client sees this record
and concludes that there _isn't_ a CNAME. This conclusion is justified
by the rule that a name can't have CNAME together with regular records.
An administrator who sets up a single name with both CNAME and NS (or
CNAME and MX, etc.) is entirely at fault for the resulting confusion.

The client then sends an MX query to the cache. Again failures are
caught and defer mail delivery. The most common success case is that
the cache has an MX set at this point; the client sees the MX set and
acts accordingly. (The A that the MX points to is normally cached too.)
If there's a successful response without an MX (case 5 described in
http://cr.yp.to/djbdns/notes.html#response-parsing) then the client
correctly concludes that there isn't an MX and falls back to an A query
for the original name.

To summarize, this type of ANY->MX->A client correctly sees whether
there's a CNAME, correctly sees whether there's an MX, and (when there
isn't an MX) correctly sees whether there's an A. If there's a server
failure (e.g., NOTIMP) or cache failure, the client correctly defers
mail delivery.

A comprehensive efficiency analysis requires detailed measurements for
many years at many sites, but spot checks have consistently shown that
the following cases are most important:

   * Most common: MX in server and in cache. The ANY->MX->A client ends
     up generating 0 queries to the server: the cache answers ANY with
     MX, and answers MX with MX.
     
     For comparison, a CNAME->MX->A client would also end up sending 0
     queries to the server _if_ the cache were smart enough to use the
     MX as a reason to deny CNAME. But typical caches aren't that smart,
     so this type of client would end up sending 1 query to the server.

     An MX->A client uninterested in CNAME would end up sending 0
     queries to the server.

   * MX in server but not in cache. The ANY->MX->A client ends up
     generating 1 query to the server: the server answers ANY with MX
     (and typically A), and then the cache answers MX with MX.

     For comparison, a CNAME->MX->A client would end up generating 2
     queries to the server: the server answers CNAME with no data, then
     answers MX with MX.

     An MX->A client uninterested in CNAME would end up sending 1 query
     to the server: the server answers MX with MX.

   * A in server and in cache, no MX in server: The ANY->MX->A client
     ends up sending 1 query to the server. The cache answers ANY with
     A; the server answers MX with no data; the cache answers A with A.

     For comparison, a CNAME->MX->A client would end up generating 2
     queries to the server (or 1 if the cache is smarter): the server
     answers CNAME with no data; the server answers MX with no data; the
     cache answers A with A.

     An MX->A client uninterested in CNAME would end up sending 1 query
     to the server. The server answers MX with no data, and the cache
     answers A with A.

   * A in server, nothing in cache: The ANY->MX->A client ends up
     sending 2 queries to the server. The server answers ANY with A; the
     server answers MX with no data; the cache answers A with A.

     For comparison, a CNAME->MX->A client would end up generating 3
     queries to the server: the server answers CNAME with no data; the
     server answers MX with no data; the server answers A with A.

     An MX->A client uninterested in CNAME would end up sending 2
     queries to the server. The server answers MX with no data, and then
     answers A with A.

To summarize, in all of these common cases, the ANY->MX->A client sends
fewer queries than the CNAME->MX->A client (despite acquiring the same
information), and just as few as the MX->A client (despite acquiring
more information). The CNAME->MX->A client can perform fewer queries
when there _is_ a CNAME, but that's a vastly less frequent situation.

Packet size is harder to analyze. ANY often pulls some records that
aren't used, and if the site isn't configured carefully then ANY can
even end up falling back to TCP, costing bytes _and_ packets. On the
other hand, there are a huge number of Internet sites that don't have a
noticeable volume of unusual records and don't need TCP, and there's a
clear traffic win for every skipped query and skipped no-data response.

If IETF is serious about considering DNS protocol changes, the right way
forward is to recognize that QTYPE=ANY _does_ have efficiency benefits
and to consider protocol extensions that would do even better. In any
event, IETF has a multi-stage procedure for creating mandatory standards
in the first place, and needs to follow the same procedures for
modifying those standards.

> cannot derive any conclusion from the absence of any record from that

False. As above, _any_ regular record asserts the non-existence of a
CNAME. You're wrong in your claim that the result is meaningless _and_
you're wrong in your claim that qmail is misbehaving.

> qmail with this feature is broken
  [ etc., etc., etc. ]

There's an impressive volume of misinformation being presented here in
an attempt to justify Cloudflare unilaterally and intentionally creating
interoperability problems by violating IETF's mandatory DNS standards.
It's not reasonable to expect all the errors to be corrected within "a
few weeks".

---Dan

---

From: Michael Graff <[email-masked]>
Date: Thu, 12 Mar 2015 16:11:57 +0000

My guess is that with DNSSEC, this will be common, as often times the domain apex is where the email would be sent.  For my personal domain, that’s @flame.org<http://flame.org>, and weighs in at 1758 bytes to an ANY query right now.

Once this is done, the MX target then needs to be followed of course (or targets in the case of a failure to connect.)

In the following, I’m using ESND0.  If this isn’t true, we all know anything > 512 bytes as a response was a TCP hit.  I’m not as scared of TCP hits as others may be, but I do think they should be avoided when practical.

ANY comes in as 1769 with or without DNSSEC.  Had it asked for the MX directly, it would have gotten 60 bytes without DNSSEC, and 229 with.

If there was no MX record, I assume then another query would be issued for the AAAA and A records.  That’s two more queries, but both of which would be smallish in comparison to the ANY query.  The DNSSEC keys nearly always dominate ANY queries at the apex.

I’m happy we are discussing issues with ANY queries, and the fairly small number of clients that use them.  I would have to see hard numbers collected over a lot of data showing where the ANY query was actually better than following the <MX, AAAA, A> path.  Until data is collected from how people set up zones today, I’m not sure I can say one is better than the other, other than as a feeling that it might help reduce queries but I’m not sure it reduces bandwidth.

What problem are we specifically trying to solve here again?

—Michael

---

From: Mark Andrews <[email-masked]>
Date: Fri, 13 Mar 2015 10:09:59 +1100

> What problem are we specifically trying to solve here again?

A non-problem for most of us.

> Michael

If one really wants to reduce the number of packets required with
SMTP processibg just write a RFC that says A and AAAA records should
be returned in the additional section if no MX records exist at the
qname.  This is currently permitted so vendors could do this today.

Alternatively change SMTP processing to not fallback to A / AAAA
on no MX.  Set a date several years out my which all sites that
accept email need to publish a MX record.  This really should have
been done years ago.  Instead we have null-MX to signal that there
isn't a SMTP server.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Paul Vixie <[email-masked]>
Date: Fri, 13 Mar 2015 19:56:01 +0900

note, i am limiting my reply to [email-masked]; it's my view that
dns-operations@ and dnsop@ should each prohibit (by mail filter) any cc
to any other list.

Randy Bush wrote:
> not break things that are working

everything is broken, depending on whom you ask.

-- 
Paul Vixie

---

From: Randy Bush <[email-masked]>
Date: Fri, 13 Mar 2015 19:27:56 +0900

randy

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 14 Mar 2015 01:02:48 +0900

> not break things that are working

Yup. Qmail or any software produced by djb adhering the existing
standards of the Internet.

> everything is broken, depending on whom you ask.

The worst broken thing in DNS is DNSSEC.

As a person who have been saying DNSSEC has been broken from the
beginning, after which, as certain amount of operational experiences,
it was revised several times along ways to fix some (but not all),
IMHO, broken parts, may I volunteer to fix not ANT but DNSSEC entirely?

Before replying me, remember that you have been saying, from the
beginning, that DNSSEC was OK if it were properly implemented.

I may temporally ignore fundamental operational impossibility of
DNSSEC and try to make it least harmful w.r.t. DDOS.

						Masataka Ohta

---

From: Paul Vixie <[email-masked]>
Date: Sat, 14 Mar 2015 01:28:15 +0900

> Masataka Ohta <mailto:[email-masked]>
> Saturday, March 14, 2015 1:02 AM
> Randy Bush wrote:
>

you say "adhering to ... standards".
i say "depending on corner cases".

ultimately what matters is whatever works. if cloudflare decides to stop
answering QTYPE=ANY then it would take all million or so qmail customers
complaining to cloudflare's NOC to get cloudflare to change its mind. i
don't think that's going to happen, for a number of reasons, one of
which is that the corner case qmail is depending on was a bad idea
originally and has gotten nothing but worse since then. but let's run
the experiment, shall we?
> The worst broken thing in DNS is DNSSEC.

thank you for amplifying my point.
> beginning, that DNSSEC was OK if it were properly implemented.

i probably said that fifteen years ago and maybe i said it again ten
years ago. here is me, on record:

> DNSSEC is a colossal flop, but not a mistake. It's an embarrassment,
> but we'd do it all again if we had to. It's late -- it was started
> years before the IPv6 effort but is (believe it if you can) even less
> finished and less deployed than IPv6. It's ugly and complicated and if
> we knew then what we know now we'd've scrapped DNS itself and started
> from scratch just to avoid the compromises we've made. But we didn't
> know then, etc., and what we have to do now is avert our gaze and
> fully deploy this ugly embarrassing thing.
>
> Let me explain.
>
> ... 
(http://www.dnssec.net/why-deploy-dnssec)

see also:

> At the time of this writing DNSSEC mostly does not work. 
(http://www.circleid.com/posts/defense_in_depth_for_dnssec_applications/)
> DNSSEC and try to make it least harmful w.r.t. DDOS.

uh, thanks?

-- 
Paul Vixie

---

From: Paul Wouters <[email-masked]>
Date: Fri, 13 Mar 2015 12:42:23 -0400 (EDT)

> the experiment, shall we?

I bet most qmail installs run from distributions that have included the
CNAME patch. I'm not sure if this is going to break more than 1 server.

All debian qmail packages come with:

http://ftp.de.debian.org/debian/pool/non-free/q/qmail/qmail_1.03-49.2.diff.gz

+  * Applied patch to dns.c to allow e-mail to deliver correctly to systems where
+    their DNS size is greater > 512.  Fixes "CNAME Lookup Failure" error when
+    delivering mail to aol.com
+
+ -- Jon Marler <[email-masked]>  Sat,  29 May 1999 12:33:00 +0100

This ship sailed 16 years ago. The only people at risk are those who
compiled from source without applying patches.

Paul

---

From: Morizot Timothy S <[email-masked]>
Date: Fri, 13 Mar 2015 17:21:31 +0000

From: Paul Vixie

ultimately what matters is whatever works. if cloudflare decides to stop answering QTYPE=ANY then it would take all million or so qmail customers complaining to cloudflare's NOC to get cloudflare to change its mind. i don't think that's going to happen, for a number of reasons, one of which is that the corner case qmail is depending on was a bad idea originally and has gotten nothing but worse since then. but let's run the experiment, shall we?

[snip]

DNSSEC is a colossal flop, but not a mistake. It's an embarrassment, but we'd do it all again if we had to. It's late -- it was started years before the IPv6 effort but is (believe it if you can) even less finished and less deployed than IPv6. It's ugly and complicated and if we knew then what we know now we'd've scrapped DNS itself and started from scratch just to avoid the compromises we've made. But we didn't know then, etc., and what we have to do now is avert our gaze and fully deploy this ugly embarrassing thing.

I agree that what works will ultimately win the day. Cloudflare can do what it wants and if it only breaks qmail's unusual implementation they may decide they are fine with the results. That's a process independent of operational recommendations or policy work. Nor do I see that either will ultimately impact the outcome.

Many of the discussions about DNSSEC tend to focus on who has or hasn't signed their authoritative zones and people tend to point to a low percentage of signed zones as some sort of failure in the deployment of the protocol. (I'm not sure I've seen an analysis that attempts to determine how many of  those unsigned zones are basically just parked domains, but that's a side issue.) But signing is only part of the story and for those of us who have signed our zones, not the most important part. For instance, the statistic I monitor and care the most about is this one:

http://stats.labs.apnic.net/dnssec/US

It's been steadily increasing for years now and gives me an idea what percentage of the US public is protected against certain types of attacks involving our zones. DNSSEC validation is not a panacea, but in a layered approach toward combating fraud and certain sorts of attacks, it does provide a particular sort of protection not available through any other means. Whether or not ISPs sign their authoritative zones matters much less to us than whether or not they implement DNSSEC validation on their recursive nameservers. And that's not a failure at all. By the measure above (which isn't perfect, but the best one available) roughly a fifth to a quarter of the US public, the primary consumers of our zones, exclusively use validating nameservers. That's significant. Would I like to see it higher? Sure. But I'll take it.

Enabling validation supports everyone who cares enough about their zones to sign them, whatever percentage of the total authoritative namespace that might be. Signing authoritative DNS is a decision each organization must make for themselves. I would encourage it, but that's something the organization must weigh. Enabling validation is a public good and much less problematic. And the concerns that have led to the need for negative trust anchors during early deployment mostly go away once most nameservers validate. Then if an organization breaks their authoritative DNS through a DNSSEC error, it breaks for most people instead of just ones with certain providers. It's an incomplete deployment of validation that's toxic and we'll see the most significant benefit from widespread validation. Yet when people talk about DNSSEC 'being broken' (a characterization with which I don't really agree), they mostly focus on the authoritative side.

Just a few thoughts spurred by the discussion.

Scott

---

From: Paul Wouters <[email-masked]>
Date: Fri, 13 Mar 2015 13:54:25 -0400 (EDT)

On Fri, 13 Mar 2015, Paul Vixie or Morizot Timothy S wrote:

[not sure of the quoting in this message]

> DNSSEC is [...] even less finished and less deployed than IPv6.

I have to disagree with this continued claim by opponents of DNSSEC
that it is not widely deployed. The fact that Apple had an outage that
was caused by DNSSEC actually shows it has been deployed widely! All of
Google DNS and Comcast clients were affected! That's probably more than
all IPv6 users.

Also, while the DNSSEC opponents screamed (well, tweeted)  murder and
called for throwing out DNSSEC, when the next day all the online Apple
Stores were down because of Apple's "internal DNS error", those DNSSEC
opponents were very quiet and not screaming to rip DNS out of the
internet :P

> then what we know now we'd've scrapped DNS itself and started from scratch just to avoid the
> compromises we've made.

And we would not be able to avoid some of the same issues. For example,
any newly made solution would have the same choice of deciding in the
browser to soft fail or hard fail (or do a pop-up). People now complain
about the "hard fail" nature of DNSSEC in browsers, but DNSSEC has
actually been designed to allow a soft fail. All the browsers need to
do in resposne to a ServFail is to re-issue the query with the CD bit
set and if they receive an answer on the second go, pop-up to the user
for an override.

Paul

---

From: David Conrad <[email-masked]>
Date: Fri, 13 Mar 2015 10:56:53 -0700

On Mar 13, 2015, at 10:54 AM, Paul Wouters <[email-masked]> wrote:
> that it is not widely deployed.

I believe this was Paul Vixie quoting himself from 10 years ago.

Regards,
-drc

---

From: Morizot Timothy S <[email-masked]>
Date: Fri, 13 Mar 2015 17:59:31 +0000

> -----Original Message-----
> From: Paul Wouters [mailto:[email-masked]]
>
>

The comments you referenced were all Paul's. I tried to make that clear in the structure of the email even though my work email system does not make it easy to quote intelligently. My comments were all at the bottom. I tried a different approach with this email. Hopefully it's clearer.

Scott

---

From: Robert Edmonds <[email-masked]>
Date: Fri, 13 Mar 2015 14:16:26 -0400

> http://ftp.de.debian.org/debian/pool/non-free/q/qmail/qmail_1.03-49.2.diff.gz

This is the "qmail" source package [0] that was shipped in prior Debian
releases.  In the current stable release and newer, the "qmail" binary
package is provided by the "netqmail" source package [1].

[0] https://tracker.debian.org/pkg/qmail

[1] https://tracker.debian.org/pkg/netqmail

> + -- Jon Marler <[email-masked]>  Sat,  29 May 1999 12:33:00 +0100

There are at least two DNS-related qmail patches in circulation, an
"ANY-to-CNAME" patch [2], and a "large DNS packet" patch [3].  This
changelog entry for the old Debian "qmail" source package appears to be
referring to only the latter patch, because the ANY-to-CNAME patch does
not appear to be present [4].

The binary package you currently get from "apt-get install qmail" is
built with a dns.c [5] that appears to have identical behavior to the
original qmail 1.03, i.e., neither of the two patches are included.

[2] http://www.memoryhole.net/qmail/any-to-cname.patch

[3] http://www.memoryhole.net/qmail/qmail-103.patch

[4] https://sources.debian.net/src/qmail/1.03-49.2/dns.c/#L214

[5] https://sources.debian.net/src/netqmail/1.06-5/dns.c/

-- 
Robert Edmonds

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 13 Mar 2015 13:07:43 -0700

> On Mar 13, 2015, at 10:21 AM, Morizot Timothy S <[email-masked]> wrote:
> It’s been steadily increasing for years now and gives me an idea what percentage of the US public is protected against certain types of attacks involving our zones. DNSSEC validation is not a panacea, but in a layered approach toward combating fraud and certain sorts of attacks, it does provide a particular sort of protection not available through any other means. Whether or not ISPs sign their authoritative zones matters much less to us than whether or not they implement DNSSEC validation on their recursive nameservers. And that’s not a failure at all. By the measure above (which isn’t perfect, but the best one available) roughly a fifth to a quarter of the US public, the primary consumers of our zones, exclusively use validating nameservers. That’s significant. Would I like to see it higher? Sure. But I’ll take it.
> 

The problem is validation by the recursive resolver is nearly useless for security, but one heck of an effective DOS attack (NASA, HBO, etc)...

Lets look at what real world attacks on DNS are.

a:  Corrupt the registrar.  DNSSEC do any good?  Nope.

b:  Corrupt the traffic in-flight (on-path or in-path).  DNSSEC do any good?  Only if the attacker is not on the path for the final traffic, but just the DNS request.

c:  The recursive resolver lies.  Why would you trust it to validate?

d:  The NAT or a device between the recursive resolver and the user lies.  Again, validation from the recursive resolver works how?

Overall, unless you are validating on the end host rather than the recursive resolver, DNSSEC does a lot of harm from misconfiguration-DOS, but almost no good.

--
Nicholas Weaver                  it is a tale, told by an idiot,
[email-masked]                full of sound and fury,
510-666-2903                                 .signifying nothing
PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc

---

From: Morizot Timothy S <[email-masked]>
Date: Fri, 13 Mar 2015 21:02:33 +0000

Nonsense.

I'm not sure exactly what sort of attack profile you have in mind at the registrar with a, but given that the TTL for DS records is generally 24 hours, most attacks at that level will create pretty widespread DNSSEC validation errors for at least that initial day. DNSSEC validation helps a great deal.

B & d are issues securing the first hop if validation is not done on the endpoint itself. Those are valid, but do not mean that DNSSEC validation provides no protection. It certainly protects against an array of cache poisoning attacks even in that configuration. And that's protection the clients would not otherwise have. It definitely makes it a lot harder to use DNS as an attack vector with nobody noticing. One layer of a layered approach.

C is certainly a problem if you don't validate on the end point and trust any random nameserver on any network to which you connect.

However, most enterprise clients and ISP users do tend to have a reliable and reasonably secure path to their first hop recursive nameserver. It's not nearly as secure as validating on the client, but it's much more secure than having no validation whatsoever.

Nor is DNSSEC validation a DOS vector. That's a non sequitur and frankly a pretty silly assertion. Yes, an organization can break their own authoritative DNS (which is related to signing not validation), but frankly DNSSEC is just one of many ways an organization can screw up DNS or anything else in their network. It's best to know what you're doing. Organizations will learn. If you haven't implemented DNSSEC validation yourself, you may not have noticed, but US government agency management of DNSSEC has improved greatly with experience. Outages due to error are less and less common and usually limited in scope when they occur. Since we've been validating all Internet responses for four years and counting now (and tend to interact quite a bit with other agencies), we have noticed the improvement. Refusing to return results when the authoritative DNS response fails validation is good thing, not a bad thing, even when it's the authoritative zone administrators who screwed up their own zone.

DNSSEC validation is not a panacea, but if you refuse to implement it you are denying your users one layer of protection you could pretty easily provide. And given that in the US the large majority of federal agency DNS authoritative zones are signed, you also can't claim there's no benefit to the US public from validation. Implementing validation on recursive nameservers does not protect users from every attack. Nothing does. Nor is it as good as performing validation at the client. But it is a solid first step with real security benefits. And it's a step that can be followed and built upon with further enhancements later.

Scott

---

From: Mark Andrews <[email-masked]>
Date: Sat, 14 Mar 2015 08:25:04 +1100

In message <[email-masked]>, Morizot Timothy S writes:
>
> DNSSEC validation is not a panacea, but if you refuse to implement it you
> are denying your users one layer of protection you could pretty easily
> provide. And given that in the US the large majority of federal agency
> DNS authoritative zones are signed, you also can't claim there's no
> benefit to the US public from validation. Implementing validation on
> recursive nameservers does not protect users from every attack. Nothing
> does. Nor is it as good as performing validation at the client. But it is
> a solid first step with real security benefits. And it's a step that can
> be followed and built upon with further enhancements later.

And validating in the recursive server is required for DNSSEC to
work reliably when the client is validating as it doesn't talk
directly to the authoritative servers.  Turning on DNSSEC validation
in the recursive servers is the first step in turning on validation
in the client.

---

From: "Darcy Kevin (FCA)" <[email-masked]>
Date: Fri, 13 Mar 2015 22:21:37 +0000

According to their own statement, Cloudflare perceived the "problem" to be the code-complexity of their DNS implementation -- in particular, they characterized the complexity of their (former) QTYPE=*-handling code as "enormous". Their "fix" was to feign ignorance (RCODE=NOTIMP) of QTYPE=* and thus -- as I and others interpret it -- fall out of compliance of any reasonable reading of RFC 1034/1035.

IANAL, but I think this might have legal ramifications. If they are advertising/selling "DNS" services and what they are delivering is not "DNS", then Truth in Advertising and/or Bait-and-Switch statutes, regulations and/or treaty provisions may apply. They could avoid this fate, of course, by rebranding their name-resolution service as something other than "DNS" (Cloudnameserviceflare?), even though coincidentally it runs on port 53 and in all respects other than QTYPE=* responses looks and quacks a lot like "DNS".

Of course, IETF is not the FTC, nor is it the WTO. What can we do? There seems to be a diversity of opinion on this:

The standards-purists want to render an opinion that Cloudflare's implementation has forsaken standards-compliance, and let those chips fall where they may, legally or otherwise.

The accommodationists want to come up with a "smarter" or "cleverer" way for Cloudflare (and undoubtedly others to follow) to frustrate QTYPE=* queries in a way that causes as little wreckage as possible. Not sure how they hope to achieve that, if anything beyond "return(DNS_RCODE_NOTIMP)" qualifies as "enormous" code-complexity to the Cloudflare folks...

Cloudflare justifies their action, in part, by making the questionable claim "The original reason for adding the ANY to DNS was to aid in debugging and testing". Whatever other action may or may not be taken by the IETF, since only IETF has the institutional memory to definitively confirm or deny this claim, I think it is worthy of a response.

												- Kevin

-----Original Message-----
From: DNSOP [mailto:[email-masked]] On Behalf Of Randy Bush
Sent: Friday, March 13, 2015 6:28 AM
To: Michael Graff
Cc: [email-masked]; D. J. Bernstein; [email-masked]
Subject: Re: [DNSOP] [dns-operations] dnsop-any-notimp violates the DNS standards

randy

_______________________________________________
DNSOP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsop

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Mar 2015 22:58:11 -0000

I remain puzzled at the entire technological motivation that CloudFlare
claims for this deliberate creation of interoperability problems.

In particular, what exactly is the programming difficulty that they
claim they're encountering in implementing QTYPE=*? Are they also having
trouble implementing NXDOMAIN, which from a programming perspective is a
very similar unification of data across all types?

The rest of this message identifies specific rules that CloudFlare's
threatened plan will violate in IETF's mandatory DNS standards.

David C Lawrence writes:
> query is not implemented. Whether it is a good idea to respond to ANY
> this way is a separate argument that is worth having. You just won't
> win on the foundation that it is a violation of the standard.

Let's look at what the standards actually say.

RFC 1034 clearly defines QTYPE=* to match "all RR types", along with,
e.g., QTYPE=A to match "just that type". It explicitly says that "the
name server looks for matching RRs".

CloudFlare's stated plans will violate this rule. This "matching" for
QTYPE=* is precisely what CloudFlare claims is too hard to implement!

You claim that this violation of a rule in IETF's mandatory DNS
standards doesn't constitute a violation of the standards. Evidently
you believe that the standards contain some relevant exception to the
rule. What exactly do you claim that this exception is?

The foundation for your argument, apparently, is the fact that RFC 1035
defines a syntax for a NOTIMP response. But why do you think this is of
any relevance to the "matching RRs" rule? The rule doesn't say "the name
server looks for matching RRs, except for types that the server doesn't
want to bother implementing". Where exactly, and what exactly, is the
CloudFlare exception?

Do you believe that the availability of a NOTIMP syntax overrides all
other rules and frees the server to use this syntax for anything that it
doesn't want to implement? Here's a hypothetical example to consider:

   * A very large cache operator is opposed to the usage of DNSSEC. (The
     operator's reason for this isn't relevant to this example.)

   * To deter usage of DNSSEC, the cache operator decides to create
     large-scale DNSSEC interoperability problems, augmenting DNSSEC's
     existing fragility.

   * Specifically, the cache operator issues DNSSEC queries to servers;
     and then, if a server response shows DNSSEC support, the cache
     operator returns NOTIMP to clients for _all_ of the server's data.

   * To avoid any sudden changes, the cache operator slowly ramps up
     this behavior, turning on the DNSSEC queries with higher and higher
     probability as time passes, but jumping immediately to probability
     1 for servers that don't show DNSSEC support.

   * To justify the use of NOTIMP, the cache operator claims that it
     _wanted_ to implement actually returning DNSSEC records to clients
     but found this too complicated given geoip blah blah blah, so it
     had to return NOTIMP. It quotes your claim that RFC 1035
     "explicitly allows" returning NOTIMP.

Would you call this cache behavior compliant with the mandatory DNS
standards? No? Why not? Why isn't the cache free to use NOTIMP whenever
it hasn't implemented something? Are you quite sure that you've thought
through what you're claiming?

Let's continue looking at the mandatory DNS standards. RFC 1034
explicitly allows not-implemented queries in _some_ cases, such as
inverse queries:

   Implementation of this service is optional in a name server, but all
   name servers must at least be able to understand an inverse query
   message and return a not-implemented error response.

RFC 1035 is also quite clear about this:

   Inverse queries are an optional part of the DNS. Name servers are
   not required to support any form of inverse queries. If a name server
   receives an inverse query that it does not support, it returns an
   error response with the "Not Implemented" error set in the header. 
   While inverse query support is optional, all name servers must be at
   least able to return the error response.

If the RFCs had meant to say that _all_ DNS features are optional
(leaving interoperability up to the whim of bullies, apparently what
you're endorsing), then why didn't the RFCs simply say that? Why did
they explicitly highlight particular features as being optional?

Furthermore, RFC 1123 explicitly requires DNS software to "support all
well-known, class-independent formats". This is another mandatory rule
that CloudFlare's plan clearly violates. What exactly do you think this
"support" requirement means, if servers are free to use NOTIMP whenever
they want, for example for QTYPE=*?

RFC 1034 explicitly says that a server "is free to refuse to perform
recursive services for any or all clients" (and also explicitly says
that an AXFR "may cause an error, such as refused") but explicitly says
that "All name servers must implement non-recursive queries". Do you
think a server is allowed to use NOTIMP for non-recursive queries? Do
you think that "implement" prohibits NOTIMP while "support" doesn't?

Frankly, your position looks _extremely_ difficult to defend.

---Dan

---

From: Paul Vixie <[email-masked]>
Date: Sat, 14 Mar 2015 11:59:30 +0900

> Nicholas Weaver <mailto:[email-masked]>
> Saturday, March 14, 2015 5:07 AM
>
> ...
>
> Overall, unless you are validating on the end host rather than the
> recursive resolver, DNSSEC does a lot of harm from
> misconfiguration-DOS, but almost no good.

several of us jumped for joy in 2008 when kaminsky showed rdns poisoning
to be a trivial exercise, because it finally provided justification for
what was at that time 12 years of apparently-wasted effort on DNSSEC.

you're entirely right that the end-system case (for example, DANE) is
the only actual justification for the added costs and risks of Secure
DNS, and you'd be right if you said that the current system is both
over- and under-engineered for this sole actual use case.

because it takes so many years to get everybody's permission to make any
change to the DNS protocol, and so many more years to get everybody's
permission to make any change of this magnitude to the root zone, the
cost of giving up and starting over necessarily includes not just the
tear-down but the re-negotiation. not practical.

so we'll keep pushing the crap system we have, uphill all the way, noone
loving it, and almost everyone in fact hating it. we've now spent more
calendar- and person-years on DNSSEC than was spent on the entire IPv4
protocol suite (including DNS itself) as of 1996 when the DNSSEC effort
began. ugly, ugly, ugly.

-- 
Paul Vixie

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 13 Mar 2015 21:42:33 -0700

> qname.  This is currently permitted so vendors could do this today.

For some data; Route 53 does do this for MX, NS, SRV and CNAME. It's
never been a problem, and does seem to speed up processing a little.

-- 
Colm

---

From: Paul Vixie <[email-masked]>
Date: Sat, 14 Mar 2015 14:50:14 +0900

i am restricting my CC to "dnsop@", for reasons previously given.

> never been a problem, and does seem to speed up processing a little.

i think this shows the need for an RFC on this topic. (and returning
AAAA as additional for A queries, and A as additional for AAAA queries.)

-- 
Paul Vixie

---

From: Nicholas Weaver <[email-masked]>
Date: Sat, 14 Mar 2015 12:44:50 -0700

> On Mar 13, 2015, at 7:59 PM, Paul Vixie <[email-masked]> wrote:

> >	Nicholas Weaver	Saturday, March 14, 2015 5:07 AM
>> 
>>> ...
>>> 
> several of us jumped for joy in 2008 when kaminsky showed rdns poisoning to be a trivial exercise, because it finally provided justification for what was at that time 12 years of apparently-wasted effort on DNSSEC.

But it didn't justify DNSSEC, even at the time.

Between actually adding in a bit more entropy in the request through 0x20 and port randomization, and more importantly cleaning up the glue policy for recursive resolvers (which Unbound did), you close the door on off-path attackers: both making races harder AND eliminating the "race until win" property.

In fact, several have viewed the glue policy cleanup which gets to the root cause of the Kaminski problem as detrimental specifically because of the desire to force DNSSEC adoption.

> so we'll keep pushing the crap system we have, uphill all the way, noone loving it, and almost everyone in fact hating it. we've now spent more calendar- and person-years on DNSSEC than was spent on the entire IPv4 protocol suite (including DNS itself) as of 1996 when the DNSSEC effort began. ugly, ugly, ugly.

At which point is it sunk cost fallacy?

"DNS is insecure, live with it" may be the best answer.  Why keep throwing good effort after bad?

It certainly is a hell of a lot better than the DOS attack that is recursive resolver validation which provides almost no meaningful security gain.

If I was Comcast, after the HBO DNSSEC mess-up, on top of previous mess-ups where Comcast inevitably gets the blame, I'd be really really tempted to turn OFF DNSSEC validation.  It has failed.

---

From: Paul Vixie <[email-masked]>
Date: Sun, 15 Mar 2015 16:23:39 +0900

> Sunday, March 15, 2015 4:44 AM

no, of course not. dnssec was well justified by the need for new
dnssec-aware applications such as DANE. all we got from kaminsky was a
whip to frighten the crowds with.
> Between actually adding in a bit more entropy in the request through 0x20 and port randomization, and more importantly cleaning up the glue policy for recursive resolvers (which Unbound did), you c…

i wish this were so. but 0x20 didn't add enough bits on average-sized
names, and furthermore, a lot of RDNS servers are behind NAT. (CGN is a
reality, in japan.) NAT's source port derandomization is as dangerous as
we thought. far too many RDNS servers were never patches, or were
patched but NAT'ed.

> At which point is it sunk cost fallacy?

it always has been. we needed end-to-end authentication, because of
bullshit like SOPA, and ad-insertion, and DNS Changer. the DNS
resolution path looks like raw meat for the BBQ's of every ISP and many
ASP's and governments and criminals and oh my. but DNSSEC was the wrong
answer, architecturally, because it tried to secure the middle of the
data path rather than the ends, and no dnssec-aware endpoint can get
DNSSEC if they are behind a non-dnssec-aware RDNS or CPE. it's been an
inarguable clusterfsck for the last ten years. we are so far into the
sunk cost fallacy on DNSSEC that we can't even see or remember our
starting conditions any more.

> "DNS is insecure, live with it" may be the best answer.  Why keep throwing good effort after bad?

it's not, though, the best answer. we have to secure the DNS resolution
path. what's in doubt is whether DNSSEC can do this, or if it can,
whether it's the best way to have done this.
> It certainly is a hell of a lot better than the DOS attack that is recursive resolver validation which provides almost no meaningful security gain.

don't get bent about dnssec as an amplifier. TXT is also a fine
amplifier. but in many DDoS victim data paths, the bottleneck is defined
in terms of number of packets processed, not the number of bits
processed, so a non-amplifying reflector is still quite valuable to
attackers. even an attenuator at the bit level is quite valuable. (an
attenuator at the packet level is not valuable to attackers, which is
why RRL uses SLIP=2 as a default.)

in other words we had to, and still have to, solve the reflector problem
-- with or without DNSSEC. andrews/eastlake "dns cookies" will do that.
and once we've done that, "DNSSEC as a DoS vector" will be just another
dead meme.
> If I was Comcast, after the HBO DNSSEC mess-up, on top of previous mess-ups where Comcast inevitably gets the blame, I'd be really really tempted to turn OFF DNSSEC validation.  It has failed.

i know what you mean and i'd face the same temptation. i wonder if there
has ever been a validation failure in the comcast resolver complex that
wasn't due to a keying/signing cockup by the domain holder -- that is,
if comcast has to have negative trust anchors to protect its validation
investment, what's the upside? could they defend their users just as
well by not running validation at all, so that keying/signing problems
don't have to be managed by the comcast NOC?

what matters for DNSSEC is the end-to-end case. as long as comcast is
running DNSSEC-aware resolvers, they don't need to validate anything in
order to make DNSSEC applications like DANE workable for their
customers. and i'd rather see them turn off validation than see negative
trust anchors added to the specification.

-- 
Paul Vixie

---

From: Mark Andrews <[email-masked]>
Date: Sun, 15 Mar 2015 21:14:54 +1100

> trust anchors added to the specification.

Can we kill this myth that recursive servers do not need to validate
because they do need to validate for DNSSEC to work reliably.  DNSSEC
only work without validation in the middle if no one is spoofing, dropping
RRSIGs etc.  The moment there is anything other than only good answers
being cached things will go wrong.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 15 Mar 2015 08:09:30 -0400

> being cached things will go wrong.

+1

Of course, what goes wrong is that the response can't be validated, so DNSSEC is still doing its job, but it can prevent cache poisoning if validation is done in the cache, and cannot if it is not.

---

From: David Conrad <[email-masked]>
Date: Sun, 15 Mar 2015 09:22:02 -0700

Hi,

> it's not, though, the best answer. we have to secure the DNS resolution path.

Probably a terminology issue, but I think we need to secure the data, not the resolution path.

I'm not a particular fan of DNSSEC for a number of reasons, however what the Kaminsky thing demonstrated to me was that as long as the data is not protected, there are going to be path-based attacks that are going to allow for compromise. I see DNSSEC as a way of being able to stop playing whack-a-mole with those path-based attacks.

>  i'd rather see them turn off validation than see negative trust anchors added to the specification.

Simply: I believe without NTAs, DNSSEC will not get deployed except by a few 'true believers'.

Regards,
-drc

---

From: bert hubert <[email-masked]>
Date: Mon, 16 Mar 2015 15:23:50 +0100

> > that is guaranteed to work by the mandatory DNS standards.
(...)
> Do you think I read 4.3.2 wrong? Or is there another RFC that updates the

Thanks to some clarification from Dan, I now understand that one can indeed
rely on ANY queries to resolvers to either deliver a CNAME or no CNAME, in
which case there is no CNAME. 

Separately, I fail to see why we actually need to outlaw ANY queries when we
can happily TC=1 them. 

I realize it is nice to do house cleaning in DNS, but I also realize that
having a document that says we deprecated ANY queries is not going to change
a lot about the real world. People will continue to perform them and expect
them to work.

A TC=1 answer does not generate any UDP packet amplification that can be
used for reflection attacks, and in my experience, TCP/IP performance these
days is stunning enough that the few legit ANY queries that come back pose
no issue.
http://blog.powerdns.com/2013/06/25/simple-tcpip-dns-benchmarking-tool/ has
some numbers.

Incidentally, our new tool 'dnsdist' [1] implements any-to-tcp as a setting,
proving that it could be added to any setup w/o too much work.

[1] http://dnsdist.org/ 

	Bert

---

From: Paul Vixie <[email-masked]>
Date: Mon, 16 Mar 2015 23:53:17 +0900

> bert hubert <mailto:[email-masked]>
> Monday, March 16, 2015 11:23 PM

i'd like to see those clarifications. if any non-CNAME RRset had existed
and been cached, and then replaced by a CNAME, then ANY would not see
the CNAME until the last non-CNAME RRset had expired from that cache.

which is why, when i want to know if there is a CNAME, i ask if there's
a CNAME.

i realize that this only matters when the non-CNAME TTL's are one to two
weeks long, and weren't trimmed before replacement with the CNAME.
however, that situation is common enough that i dispute the phrase
quoted above, "guaranteed to work by mandatory DNS standards."

> Separately, I fail to see why we actually need to outlaw ANY queries when we

TC=1'ing them would be a way to prevent them from being used as an
amplifying reflector. that is not the use case for this. the updated
document makes clear that the iteration complexity in split-authority
systems having a lightweight front end, is the situation where ANY is
painful.

there never was an ANY-related problem for which TC=1 was the solution.

-- 
Paul Vixie

---

From: bert hubert <[email-masked]>
Date: Mon, 16 Mar 2015 16:05:11 +0100

On Mon, Mar 16, 2015 at 11:53:17PM +0900, Paul Vixie wrote:
> that is not the use case for this. the updated document makes clear that
> the iteration complexity in split-authority systems having a lightweight
> front end, is the situation where ANY is painful.

Sorry? We solve implementation hardship by standards action now?

   "Some modern Authoritative servers, such as those used by CDN's, do
   not have DNS zones.  For those servers answering ANY query truthfully
   is hard work.  Thus ignoring ANY queries simplifies the
   implementation."

Is this really all there is to the story? Seriously? 

I have lots of respect for Olafur, but this does seem to turn a local
challenge into a global standards action..

	Bert

---

From: Paul Vixie <[email-masked]>
Date: Tue, 17 Mar 2015 00:15:50 +0900

> Tuesday, March 17, 2015 12:05 AM
>

as with client-subnet, we recognize that people will do what they want,
or stop doing what they don't want, especially if they are CDN providers
with a lot of revenue and a lot of expense riding on their choices. i
don't love this situation but i can tell you that quoting specifications
at folks and using words like "mandatory" isn't the way to change their
minds (or their deeds.)

noting that there's a more-than-ten-years-old CNAME patch to qmail that
just about everybody is supposedly running, i expect the operational
impact of phasing out ANY to be ~0. also, a lot of operators foolishly
patched their BIND servers to stop answering ANY because they considered
it a DDoS risk (which is patently insane but please don't shoot the
messenger) and not a single qmail user was heard from on the matter.

the internet works by cooperation. often, first mover advantage is
sticky. but almost as often, somebody like the mozilla dev team decides
that something like ANY is the solution to their API layering problem,
and the rest of us rip the bandaids off and study the underlying wound.
so it is in this case. now, mozilla has backed off, but the underlying
wound remains a visible topic of conversation.

to me the use case is, it's an information leak, and i don't want my
cache probed, and i can't tell the difference between a cache prober and
qmail, so into the same stew pot they both must go. (along with RD=0 on
an RA=1 server.)

-- 
Paul Vixie

---

From: Ray Bellis <[email-masked]>
Date: Mon, 16 Mar 2015 15:16:08 +0000

> On 16 Mar 2015, at 15:05, bert hubert <[email-masked]> wrote:
> 

Hypothetically, if you're using one of those funky NoSQL-style backends where RRs are looked up in a key-value store directly from a (QNAME, QTYPE) tuple I can see how supporting QTYPE == ANY would be tricky.

The fact that "ANY" exists can put some significant implementation constraints on your system :(

Ray

---

From: bert hubert <[email-masked]>
Date: Mon, 16 Mar 2015 16:22:38 +0100

On Mon, Mar 16, 2015 at 03:16:08PM +0000, Ray Bellis wrote:

> Hypothetically, if you're using one of those funky NoSQL-style backends
> where RRs are looked up in a key-value store directly from a (QNAME,
> QTYPE) tuple I can see how supporting QTYPE == ANY would be tricky.

At DNS query rates, you could just query purely based on the name as the
key. You'd have to do so anyhow to determine what kind of NXDOMAIN/NOERROR
response to generate!

Or are we going to flatten that distinction too to ease implementation?

	Bert

---

From: Ray Bellis <[email-masked]>
Date: Mon, 16 Mar 2015 15:25:58 +0000

> On 16 Mar 2015, at 15:22, bert hubert <[email-masked]> wrote:
> 

Yes, that's a good point :)

> Or are we going to flatten that distinction too to ease implementation?

No, we had that argument already - NOERROR is definitely the right error for an empty non-terminal ;-)

Ray

---

From: P Vixie <[email-masked]>
Date: Tue, 17 Mar 2015 08:15:50 +0900

On March 17, 2015 7:42:09 AM GMT+09:00, Michael Sinatra <[email-masked]> wrote:
>
>
>On 03/16/15 07:23, bert hubert wrote:
>
>If the public recursives also support TC=1 on all ANY queries, then
>this
>works.  If not, the issue arises where just-below-the-radar attacks are
>using many public recursives, in which case you're not stopping much.

Michael, what attacks do you think we can stop by limiting ANY? Paul
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Michael Sinatra <[email-masked]>
Date: Mon, 16 Mar 2015 17:50:44 -0700

> Michael, what attacks do you think we can stop by limiting ANY? Paul

The attack that I have had to grapple with is this:

* Someone sets up a bot to query public recursives (google, opendns,
level3, etc.) for a particular domain whose ANY response is large.
(This _usually_ means DNSSEC-signed.)

* The query from each <client,domain,qtype> tuple is just barely slow
enough not to trigger rate limiting from the public recursive service.

* The backend of the public recursive service queries my authoritatives
for some of the involved domains.  Suppose the response is just under
the usual typical default EDNS0 buffer size of 4096.

* These domains are DNSSEC-signed with NSEC3.  Many tools set the TTL of
NSEC3PARAM to 0 when signing zones with NSEC3.  The NSEC3PARAM RR is
part of the ANY response.

* The public recursive servers use an implementation that clears all
records from the cache when the TTL of one record expires, so the next
time the recursive server gets an ANY query, it must re-query the
authoritative server.

In this situation, if I set TC=1 for all ANY queries on my authoritative
server, but the public recursives don't, then the victim still gets hit
with a pretty big amplification attack, and my authoritative servers get
hammered with TCP queries.  It's annoying for me--not insurmountable,
but annoying, as the thousands of simultaneous TCP connections require
some tuning to manage reasonably.  But for the victim?  Who knows--I
can't see who the victim is in this case.  The more I tune my servers,
the more data gets likely thrown at the victim.

I have seen this in the wild, even where the response is bigger than
4096, so the TC bit should be set all around.  Note also that if my
response is bigger than 4096, I'll send an empty response back with TC=1
(I am using BIND-latest).  I have seen some recursive implementations (e.g.
unbound) that will dutifully send the victim everything right up to the
next RRset that would push it over 4K and set TC=1 for good measure.  So
the victim still gets a ~4000-byte UDP response, even with TC set.

So my point is that if we're going to specify TC=1 for ANY queries, it
has to be mandatory, and all implementations have to handle it the same:
Send an empty NOERROR and set TC=1.  If I am the only one setting TC=1,
it won't doing any good for the attack described above, even if my
domains are the ones being used in the attack.

The other option is to allow the authoritative servers to control what
gets set out in response to QTYPE=ANY.  But I see devils in the details,
just as with NOTIMP and other proposals.

michael

---

From: Mark Andrews <[email-masked]>
Date: Tue, 17 Mar 2015 12:10:50 +1100

> just as with NOTIMP and other proposals.

Lets get DNS cookies finalised so that TC=1 isn't needed for repeat
legitimate clients.  There is a open question about whether the
error field is needed or not and I'm of the opinion that it isn't
needed.

TC=1 for amplification suppression should be triggered by response
size and whether you are a known repeat client or not rather than
{meta} query type.

Mark

---

From: Michael Sinatra <[email-masked]>
Date: Mon, 16 Mar 2015 18:16:37 -0700

On 03/16/15 18:07, Yunhong Gu wrote:
> 
> 
> On Mon, Mar 16, 2015 at 8:50 PM, Michael Sinatra <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     > On March 17, 2015 7:42:09 AM GMT+09:00, Michael Sinatra <[email-masked] <mailto:[email-masked]>> wrote:
>     >>
>     >>
> Sounds to me this is the root cause of the problem and ANY is the just a
> scapegoat.

Giving NSEC3PARAM a positive TTL would prevent my headache, but it
wouldn't help the victim of the attack, and would probably make it worse
for the victim.

michael

---

From: Paul Vixie <[email-masked]>
Date: Tue, 17 Mar 2015 15:15:07 +0900

note: replying only to dnsop@. no thread is ever appropriate for dnsop@
plus some other mailing list. please stop cc'ing dns-operations@ on your
replies; this is not an operational thread, and the people in the dns
community who care about protocol development, are probably on both lists.

> Mark Andrews <mailto:[email-masked]>
> Tuesday, March 17, 2015 10:10 AM
>
> legitimate clients. ...

to be clear, response rate limiting will still be necessary even with
dns cookies in place.

without dns cookies, the requests don't have to have correct source-ip
addresses, and thus, a dns server can be made to attack the apparent
source of those queries. rrl helps with this.

with dns cookies, the requests have to have correct source-ip addresses,
and thus, a dns server can be made to attack its own upstream
infrastructure. rrl helps with this, also.

there should of course be more strict rate limits in place for the
former, than for the latter.

-- 
Paul Vixie

---

From: Paul Vixie <[email-masked]>
Date: Tue, 17 Mar 2015 15:31:07 +0900

removing dns-operations@ as a cc. one mailing list at a time, please?

> part of the ANY response.

well, that's part of your problem right there. but i can see how ANY is
involved, now.

> time the recursive server gets an ANY query, it must re-query the

this is an internet-affecting bug and i hope you report it as such.
RRsets are to be purged when the lowest TTL therein expires, but the
other RRsets sharing that name should not be affected. can i ask which
public facing dns service has mandated that the rest of us juggle their
chainsaws for them in this way? (this is even worse than the jerk who
decided that EDNS could use IP fragmentation -- but i think that guy has
apologized at least.)

> hammered with TCP queries.

"hammered" sounds like a volume greater than that which would be
detected and strangled with DNS RRL. what am i missing here, that makes
this your problem, rather than the public recursive dns operator's problem?

> just as with NOTIMP and other proposals.

i think there's no saving ANY at this point. we're deciding how it dies
and where to bury it, that's all.

however, you've provided an example of an ANY attack that can't be
trivially switch to TXT, so, thank you.

-- 
Paul Vixie

---

From: Tony Finch <[email-masked]>
Date: Tue, 17 Mar 2015 10:28:58 +0000

> just as with NOTIMP and other proposals.

What do you think of this?
http://www.ietf.org/mail-archive/web/dnsop/current/msg13730.html

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Bailey: Cyclonic becoming south for a time, 4 or 5, occasionally 6 in
northwest. Rough becoming moderate. Occasional rain. Good, occasionally poor.

---

From: Paul Wouters <[email-masked]>
Date: Tue, 17 Mar 2015 09:34:36 -0400 (EDT)

On Tue, 17 Mar 2015, Yunhong Gu wrote:

> The reason that this response can be used for an amplification attack is its size, not the ANY type. A responses
> with 200 A records can be used for the same purpose. The (even deeper) root cause is the use of UDP in DNS protocol.
> I just do not think banning ANY touches any of these fundamental issues.

Right, so require tcp or eastlake cookies, or allow padding the ANY
request so the request/response ratio is close to 1 before allowing
the answer. Make the dig command default to tcp. That should cover
the vast majority of valid ANY queries.

Paul

---

From: Yunhong Gu <[email-masked]>
Date: Mon, 16 Mar 2015 21:07:34 -0400

> dns-operations mailing list
> [email-masked]
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> dns-jobs mailing list
> https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
>

---

From: Yunhong Gu <[email-masked]>
Date: Tue, 17 Mar 2015 09:17:23 -0400

On Mon, Mar 16, 2015 at 9:16 PM, Michael Sinatra <[email-masked]>
wrote:

> >     > On March 17, 2015 7:42:09 AM GMT+09:00, Michael Sinatra <
> [email-masked] <mailto:[email-masked]>> wrote:
> >     >>
> >     >>
> >     >>> Separately, I fail to see why we actually need to outlaw ANY
> queries
> >     >> when we
> >     >>> can happily TC=1 them.
> >     >>

The reason that this response can be used for an amplification attack is
its size, not the ANY type. A responses with 200 A records can be used for
the same purpose. The (even deeper) root cause is the use of UDP in DNS
protocol. I just do not think banning ANY touches any of these fundamental
issues.

---

From: Michael Sinatra <[email-masked]>
Date: Tue, 17 Mar 2015 09:40:29 -0700

On 3/17/15 6:17 AM, Yunhong Gu wrote:

> its size, not the ANY type. A responses with 200 A records can be used
> for the same purpose. The (even deeper) root cause is the use of UDP in
> DNS protocol. I just do not think banning ANY touches any of these
> fundamental issues.

Ah, this is a different argument.

I don't believe that I ever suggested "banning" ANY.  What I suggested
is that if you want to use TC=1 to stop the current en vogue attack, you
need to apply it everywhere and make sure that all implementations
handle it the same way: All ANY responses get TC=1 with a *minimal*
response.  Not all implementations currently do that.  Another option
is cookies, although I haven't fully thought through how that would mesh
with the current attack dynamics.

I recognize you can do reflection attacks with A records, although there
are even better attacks out there that don't use ANY or A.  But ANY has
basically three uses: reflection attack, troubleshooting, and supporting
old qmail.  I'd be interested in methods that can be implemented in a
matter of weeks or months, which can reduce the attack vector, while
retaining as much of the "legitimate functionality" as possible, knowing
that the threat can't be fully eliminated.

As it stands, operators are taking matters into their own hands and
simply blocking ANY responses (which makes my headache bigger because I
slave for some of those operators and they're pushing more of the
iterative ANY queries at me).

michael

---

From: Paul Vixie <[email-masked]>
Date: Wed, 18 Mar 2015 00:32:46 -0700

Paul Wouters wrote:
>> The reason that this response can be used for an amplification attack
>> is its size, not the ANY type. A responses with 200 A records can be
>> used for the same purpose. The (even deeper) root cause is the use of
>> UDP in DNS protocol. I just do not think banning ANY touches any of
>> these fundamental issues.
>
> Right, so require tcp or eastlake cookies,

that would protect third parties, but not the server itself.

> or allow padding the ANY
> the answer.

that would not prevent the unfortunate information leak that allows
third parties to scan our caches.

> Make the dig command default to tcp. That should cover

my proposal is, limit ANY to a selected set of source-ip addresses, as
is commonly done with AXFR/IXFR.

-- 
Paul Vixie

---

From: Michael Sinatra <[email-masked]>
Date: Wed, 18 Mar 2015 01:11:50 -0700

> removing dns-operations@ as a cc. one mailing list at a time, please?

Sorry, saw this response late...

> well, that's part of your problem right there. but i can see how ANY is

Right, there have been discussions on this in the past (I think on
dns-operations@), but both BIND (dnssec-signzone) and the OpenDNSSEC
tools set the NSEC3PARAM TTL to 0.  I am not sure what the "right"
answer is, but it is contributing to the problem below.

> decided that EDNS could use IP fragmentation -- but i think that guy has

I know unbound did do this--I think it still does, but it's late and I
haven't had a chance to test it.  The public recursive DNS service was
google, and again, my tests six months ago showed that they were purging
the entire ANY pseudo-RRset and requerying the authoritative servers.

> this your problem, rather than the public recursive dns operator's problem?

I think there are a few issues at play.  google and other public
recursives will sometimes have multiple backend servers fetch a given RR
when they receive a single query for that RR on one instance of, say,
8.8.8.8.  I am basing this both on observed behavior and on Geoff
Huston's research (recently presented at NANOG).  And since nothing is
cached, I get the same servers asking the same query over and over
again.  Writ large, the result is that I end up with 1-2k of
simultaneous TCP sessions, per server, per domain.  Nothing I can't
handle, since usually only 2-3 of my domains are involved,  But it does
mean that I have to tweak BIND's defaults, since the number of allowed
simultaneous TCP sessions for that implementation is much lower.
Otherwise I deny legitimate clients, since the TCP limit is applied
across the entire named process, without regard to QTYPE or anything else.

So if someone is good at figuring out where the rate limits lie, and
what tuple(s) they're based on, they can try to sneak just under the
radar of any public cache, not just google.  If they spread the tuple
values out enough, they might have an effective attack.  I have no idea
how debilitating it is, as I have no visibility into who the actual
victim is in this case.

> trivially switch to TXT, so, thank you.

I am pretty small-time, but given some of the domains I slave are viewed
as useful for reflection attacks, I will usually see these odd things
when they crop up.

michael

---

From: Michael Sinatra <[email-masked]>
Date: Wed, 18 Mar 2015 01:15:35 -0700

On 03/18/15 01:11, Michael Sinatra wrote:

> simultaneous TCP sessions, per server, per domain.  

Just a quick qualification: This is during an active attack, not as a
normal course of events.  However, the attacks can and will last for
several weeks.

michael

---

From: Paul Vixie <[email-masked]>
Date: Wed, 18 Mar 2015 01:22:37 -0700

> Michael Sinatra <mailto:[email-masked]>
> Wednesday, March 18, 2015 1:11 AM

let's assume for the moment that we're going to work together to find
any implementation of RDNS which purges all RRsets when one RRset
expires, and then use every means at our disposal to correct that
misbehaviour, and further, that we will be successful.
> simultaneous TCP sessions for that implementation is much lower.

and not unfairly so, given the unfortunate logic of RFC 1035's TCP
connection management text. however, for the purpose of this discussion,
i ask you to assume that you and i will jointly and severally discuss
this matter with google, so that afterward, their cache-miss servers
will not "pile on" in this way.
> across the entire named process, without regard to QTYPE or anything else.

to be fair to the laws of physics, there's no way to refuse a TCP
connection based on the QNAME it has not had a chance to tell you yet.
> how debilitating it is, as I have no visibility into who the actual

i think it's fair to say that there's no one bogey man here. we're going
to have to get the purge behaviour fixed wherever it's broken, and we're
going to have to get the refill behaviour fixed wherever it's broken,
and we're going to have to do this even while knowing all the while that
ANY is a zombie, dead but still walking around looking for a place to
lay down.
> as useful for reflection attacks, I will usually see these odd things

yes. and it's great that you pay attention to, and share, your experiences.

-- 
Paul Vixie

---

From: Paul Wouters <[email-masked]>
Date: Wed, 18 Mar 2015 09:58:04 -0400 (EDT)

On Wed, 18 Mar 2015, Paul Vixie wrote:

>             The reason that this response can be used for an amplification attack is its size, not the ANY type.
>             A responses with 200 A records can be used for the same purpose. The (even deeper) root cause is the
>             use of UDP in DNS protocol. I just do not think banning ANY touches any of these fundamental issues.
>

It offers some protection, but it is not perfect.

> that would not prevent the unfortunate information leak that allows third parties to scan our caches.

So there are two clear distinct use cases. ANY against authoritative and
ANY against caches. I agree that one could limit ANY queries of caches
to a sysadmin only ACL. We lose some remote debugging capability but
gain some privacy. Putting an ACL in on the authoritative server is
much more complex - it could backfire as a long discussion in the last
week on how to refuse those queries show.

> my proposal is, limit ANY to a selected set of source-ip addresses, as is commonly done with AXFR/IXFR.

Which I answered before by saying that is basically killing the ANY
query. The proposed solution merely pretends to not kill it by saying
"acl". But let me clarify that I think putting an ACL on the recursive
is fine - legitimate applications should not use use ANY queries and
this is a fine mechanism to ensure the qmail's and firefoxes of the
future won't make the same mistake.

The trick of requiring ANY comes via verified sourc IP or a question
packet bigger than the response ensures this query type will cease to
be used for amplification. If we want to do this we should start changing
our legitimate tools to do this so at some point in the future we
could enforce this without breaking our own toolbox.

Paul

---

From: Paul Vixie <[email-masked]>
Date: Wed, 18 Mar 2015 08:55:10 -0700

> Paul Wouters <mailto:[email-masked]>
> Wednesday, March 18, 2015 6:58 AM
> "acl".

i don't think there's any pretense here about not wanting to kill, or
not killing, ANY.

the history of DNS is replete with examples of information leaks which
had to be stopped, either by ad-hoc action or by standards action.
limiting who can do zone transfers was first (BIND4 "King James
Edition", 1989-ish). preventing DNSSEC zone walking was next (DNSEXT,
NSEC3, 2001-2014). now it's ANY. many things which made sense on an
academic research Internet do not make sense on a world-wide commercial
internet.

we need a document that says "If you don't want to answer ANY, here's
how to do it interoperably." we don't need to say "you should not answer
ANY", but we do need to say "if you want to query for ANY, here's what
might happen." that, sir, is a killing. we are killing ANY. there's no
pretense.

-- 
Paul Vixie

---

From: Olafur Gudmundsson <[email-masked]>
Date: Sat, 21 Mar 2015 19:33:09 -0400

> On Mar 18, 2015, at 11:55 AM, Paul Vixie <[email-masked]> wrote:
> 
> we need a document that says "If you don't want to answer ANY, here's how to do it interoperably." we don't need to say "you should not answer ANY", but we do need to say "if you want to query for ANY, here's what might happen." that, sir, is a killing. we are killing ANY. there's no pretense.

This is exactly what my goal is tell the upstream the type ANY will not be answered. 
I do not care if that is RCODE=X or Rcode=0 + NULL record in answer section or something else. 

Olafur

---

From: Florian Weimer <[email-masked]>
Date: Sun, 22 Mar 2015 19:58:02 +0100

* Olafur Gudmundsson:

>> we need a document that says "If you don't want to answer ANY,
>> here's how to do it interoperably." we don't need to say "you
>> should not answer ANY", but we do need to say "if you want to query
>> for ANY, here's what might happen." that, sir, is a killing. we are
>> killing ANY. there's no pretense.
>

Upstream?

Is reducing the number of ANY queries an immediate goal of yours?
(Discounting any indirect effect that useless ANY answers will
have on future application changes.)