Subject: DNS Server DoS Attacks

From: "Alec H. Peterson" <[email-masked]>
Date: Fri, 22 Nov 2002 17:37:31 -0700

These attacks seem to be on the rise.  There were the well-publicized ones 
on the root servers a few weeks ago, and yesterday UltraDNS was hit with 
one.  It occurs to me that these attacks have the potential to completely 
shut down the DNS system if they are structured properly.  I have a thought 
on how to make the system more resiliant to authoritative servers getting 
hammered, and was interested in some thoughts on it.

My thought is to add another TTL to DNS responses, similar to the SOA 
maximum parameter.  The current TTL would be similar to the SOA minimum. 
This would still allow for records to expire in a reasonable amount of 
time, but it would also allow for DNS responses to be answered in the event 
that servers in the hierarchy are unreachable for some reason.  It occurs 
to me that it is possible to retrofit existing DNS servers to have a static 
maximum timeout without any protocol modifications.

Anyway, the way I see it since DNS already has a caching infrastructure 
built in it makes sense to take extra advantage of that infrastructure when 
things are under attack.

Alec

--
Alec H. Peterson -- [email-masked]
Chief Technology Officer
Catbird Networks, http://www.catbird.com

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>

---

From: "Alec H. Peterson" <[email-masked]>
Date: Fri, 22 Nov 2002 19:07:00 -0700

--On Saturday, November 23, 2002 1:54 AM +0000 Paul Vixie <[email-masked]> 
wrote:

>
> if people would just cache what they already receive, then 98% of the
> queries seen by the root servers would never be sent.  so, i think
> there's ample evidence to refute the idea that the internet's caching
> infrastructure is a model we can build on.

That is a good point, that had already been made to me before I posted that 
actually.

I see your point that a sustained attack should be easy to track down, but 
I think it would be really nice if we could find a way to preemptively come 
up with a way to help mitigate these attacks, instead of operating in a 
reactive mode.

Alec

---

From: Bill Manning <[email-masked]>
Date: Fri, 22 Nov 2002 19:49:35 -0800 (PST)

% --On Saturday, November 23, 2002 2:33 AM +0000 Paul Vixie <[email-masked]> 
% wrote:
% > well, like i said, we could secure the edge.
% 
% While we're at it why don't we deploy IPv6.
% 
% Alec
% 
	we are. :)  just not as quickly or as transparently
	as some would like..

--bill

---

From: "John M. Brown" <[email-masked]>
Date: Sat, 23 Nov 2002 13:33:23 -0700

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Alec H. Peterson
> Sent: Saturday, November 23, 2002 7:49 AM
> To: [email-masked]; [email-masked]
> Subject: RE: DNS Server DoS Attacks
> 
> 
> --On Saturday, November 23, 2002 12:47 AM -0700 "John M. Brown" 
> <[email-masked]> wrote:
> 
> > packet forwarding engines won't tell the difference between
> > a good query and a bad query without serious penalty to the PFE 
> > performance.
> 
> I don't think I understand, what does my suggestion have to 
> do with packet 
> forwarding engines distinguishing between good and bad queries?

long flight, coffee == empty, it was part of a different conversation
that slipped into this thread.  sorry.
 
Some people where arguing that you could filter via ACL's and such,
that works until the poison (bad packets) looks, smells and tastes like
what you think lunch should be.
 
> But if your authoritative DNS servers aren't even reachable 
> to have the cache get re-populated then what good is it to 
> have the cache get aged?  

keeps the system cleaner.  see pauls answer

> My proposal doesn't change the current cache aging system, you 
> can still have 
> a 10 minute TTL and have an authoritative server re-query after the 
> original 10 minutes has expired.  This just _allows_ caching 
> nameservers to 
> keep stuff for longer if it is not possible to re-populate 
> the cache due to 
> unreachable nameservers.

SO how does this caching change handle a zone going away ?  If I remove
a zone from service, is that not like a DOS ??

what I'm hearing is " lets have a static value that keeps data in a
cache
longer than what the owner wants, just incase things break"...  is that
correct ??
 
> > c)  signing the root zone, more for layer 8 reasons than others.
> 
> I fail to see how signing the root zone would keep somebody 
> from flooding me with packets.

It doesn't.  It ""protects"" the zone, its a layer 8 warm and fuzzy
thing.  It does help make sure that the anycast box you are asking is
serving the same data as all of the others, and if its not, you can
ignore that server.

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 07:00:39 -0800

Why not go the whole hog and transport the entire DNS system over
Napster?

Of course this would have some minor inconveniences, not least the fact
that the assumed resilience of the other infrastructure might be
imagined and none of the existing software would work.

Anything at the application layer has a severe bootstrap problem. Hint,
I don't type a raw IP address into my newsreader, I type
nntp.attbi.com...

Plus DoS attacks against NNTP are quite easy, particularly at the local
level. The services are not configured as critical infrastructure. All
it takes is one news server configured with a guest login account and
that server canbe used to take down UESnet. Flood fill breaks both ways.
Usenet is the biggest packet amplifier out there.

Before chasing off with 'fixes' that change the whole architectural
model I would like to know some information about the puported success
of the attack. Specifically did any machines actually go down or was the
reported 'unavailability' due to ISPs and others turning off ICMP
responses? Several of the 'measurement' sites were using ping packets.

This is a classic example of a reputation attack. We had one at the
Whitehouse once. The router blew a fuse and before it was fixed a bunch
of Russian hackers were claiming to Wired.com that they were
responsible.

First we need to get the measurement sites reconfigured so that they use
an actual DNS query and not a PING packet.

Second it would be useful to know which systems (if any) went down. To
date I know the identity of 5 of the 4 servers that stayed up and do not
know the identity of a single machine that went down.

Then we should consider architectural approaches such as using anycast
that allow DNS to be DoS proofed without disruption.

		Phill

> From: D. J. Bernstein [mailto:[email-masked]]
> Sent: Sunday, November 24, 2002 4:27 AM
> To: [email-masked]
> Subject: Re: DNS Server DoS Attacks
> 
> 
> [ post by non-subscriber.  with the massive amount of spam, 
> it is easy to
>   miss and therefore delete mis-posts.  your subscription address is
>   [email-masked], please post from it or
>   fix subscription your subscription address! ]
> 
> Rob Payne writes:
> > You want to turn the root zone into a signed "hosts.txt" (RFC 952,
> > 953), and how, exactly does that scale
> 
> I already answered that: ``Effects on load: Everybody will receive the
> entire zone, rather than just the parts they need. On the other hand,
> any sensible format would be much smaller than DNS packet format. More
> importantly, the data will be cached much more effectively than it is
> with the current root-zone protocol. Most importantly, the 
> load will be
> very widely distributed.''
> 
> The last factor is, as I said, the most important one. USENET wouldn't
> notice if ten copies of the root zone---or ten thousand copies---were
> sent out every day.
> 
> > it did not scale the last time
> 
> Nobody really tried to make it scale, but this is beside the point.
> ``Root zone'' does not mean ``complete list of Internet hosts.''
> 
> ---D. J. Bernstein, Associate Professor, Department of Mathematics,
> Statistics, and Computer Science, University of Illinois at Chicago
> 
> 
> 
> --

---

From: "John S. Quarterman" <[email-masked]>
Date: Mon, 25 Nov 2002 10:36:57 -0600

[ post by non-subscriber.  with the massive amount of spam, it is easy to
  miss and therefore delete mis-posts.  so fix subscription addresses! ]

> know the identity of a single machine that went down.

All 13 root DNS servers were up during the DDoS attack of 22-23 October 2002.
3 of them turned off ICMP ECHO responses, but were responding to DNS requests.
There were side effects on Internet performance elsewhere.

-jsq

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 09:04:31 -0800

I thought that was the most likely situation.

There may also have been measurement problems due to ISPs turning off
transport of ICMP pings and due to ICMP packets being preferentially
dropped which would explain some of the measurements.

I don't think the answer is to redesign DNS for several reasons, not
least the fact that there are other critical infrastructures susceptible
to DoS attack.

What we need is some sort of mechanism for providing a trusted
advertisement that a critical site is under DoS attack from specific (or
non-specific) sources. Then we can start implementing successive
blocking of the attack packets in the infrastructure.

For example we could have DocSIS III have a filtering capability that
could be turned on by the cable provider to block packets on specific
ports to specific destinations. This would provide a cheap and
lightweight handle for turning off machines that have been compromised
and turned into a DDoS drone.

		Phill

> From: John S. Quarterman [mailto:[email-masked]]
> Sent: Monday, November 25, 2002 11:37 AM
> To: Hallam-Baker, Phillip
> Cc: John S. Quarterman; 'D. J. Bernstein'; [email-masked]

---

From: "John S. Quarterman" <[email-masked]>
Date: Mon, 25 Nov 2002 11:13:13 -0600

>dropped which would explain some of the measurements.

Do you have evidence for either of those things?
If not, it would be best not to base architecture on speculation.

-jsq

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 09:14:50 -0800

Case in point as to why Prof Bernstein's hack is out to lunch.

Like many folk Dan is running a spam filter that requests a respond
receipt to verify an email. Unlike most folk however Dan's filter
requests a response on every single message.

One callback request is OK, every time is bad manners.

Now I know that there are spams that hijack mailing list addresses to
bypass such filters, but all my mail is signed for that reason. Whatever
you think of the VeriSign CPS I think we can all agree that even a self
signed cert should be sufficient in combination with a confirmation
email and a VeriSign class 1 should be sufficient for spam filtering
purposes.

Dan's hack solves his problem fine but does not solve the problem for
anyone else because it only solves the problem from one perspective.

		Phill

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 09:27:45 -0800

John,

	I assert that unless you know for certain that the measurements
are not contaminated in the manner hypothesised that you would be
operating on speculation by accepting them at face value.

	The onus of proof is generally held to be on the observation.

	The only exception being cases like holocaust denial, moon
landing hoax conspiracy theories and claims that OJ was innocent where
the objections to the evidence require a vast number of other
assumptions to be made.

	The figures you give, 3 servers turned off ICMP do not match the
measurements reported in the press (9 servers down) or my reading of the
measurement sites at the time. Ergo it appears that either more sites
turned off ICMP than you report or some network operations decision or
network related effect, probably due to congenstion may have occured.

	Of course there may be yet another reason.

		Phill

> Sent: Monday, November 25, 2002 12:13 PM

---

From: "John S. Quarterman" <[email-masked]>
Date: Mon, 25 Nov 2002 11:42:37 -0600

Phill,

> 	The onus of proof is generally held to be on the observation.

I'm having a little trouble with this logic.  You made two speculations,
with no evidence, and now you wish to shift the burden of proof for
those speculations to someone else?

> turned off ICMP than you report

I have data showing regular ping responses from the other 10 DNS roots
during that period, as well as regular ping attempts to the 3 that stopped
responding to ping, plus direct correspondence with the operators of those 3.

> or some network operations decision or

> 	Of course there may be yet another reason.

Indeed.  One could speculate any number of potential causes for any
effect.  The burden of proof would be on the person so speculating.
If you have concrete evidence for your speculations, please cite it;
otherwise, we might as well move on.

-jsq

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 09:58:16 -0800

> those speculations to someone else?

It depends what you consider the hypothesis to be. The one I thought
we were considering was the statements made by the measurement sites.

I was pointing out that they can be contaminated by a number of sources.
That is why I said that "There may also have been measurement
problems..."

If I had intended to claim that the effects hypothesised definitively 
happened I would have said "There were also". 

Try reading before you flame.

The measurements from the measurement sites plus your own observations
show a discrepancy. The measurement sites I saw had something like 7
sites
offline. You say you have data showing responses from 10 of the servers.

Thus there is a discrepancy of 4 servers to be accounted for and an 
indication that the effect was dependent on the observation point.

The precise cause of the result is irrelevant, the only conclusions we 
can draw are that the measurements from the sites were not sound and
that we should build measurement systems that are not subject to the 
possible sources of contamination described.

		Phill

---

From: "John S. Quarterman" <[email-masked]>
Date: Mon, 25 Nov 2002 12:48:25 -0600

Jon Postel used to have a rule that no one should post to namedroppers
more than once a day.  The wisdom of that rule has become evident in
this thread.  This will be my last post on this subject today.

> indication that the effect was dependent on the observation point.

As I said, I have regular responses during that period from 10 DNS root
servers and regular pings with no responses for the other 3.
I think 10 + 3 = 13 total root DNS servers.

The data I am referring to was publicly visible at the time via
 http://average.matrix.net/dns-root/

That data for each DNS root server was collected from half a dozen
different observation points, not one.

It's hard to tell what you mean by "the measurement sites".
I'd be happy hold a conversation about them, but that's not
possible without any specifics.

> possible sources of contamination described.

Indeed, we took such considerations into account when we constructed
our services.  I don't know about "the sites".

-jsq

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 25 Nov 2002 12:33:10 -0800

> our services.  I don't know about "the sites".

In which case we have to work out why the press either got hold of the 
wrong figures or misinterpreted the figures and used those as the basis 
of the Associated Press and Reuters reports.

By design or chance someone managed a reputation attack against the
system
that reached the mainstream media and resulted in senior policy people
calling CEOs etc. That is a major problem. Media management must be a 
part of the solution.

		Phill

---

From: Danny Mayer <[email-masked]>
Date: Mon, 25 Nov 2002 22:28:07 -0500

>of the Associated Press and Reuters reports.

This has nothing to do with namedroppers. You can't control where the
agencies and the newspapers get their information.

>calling CEOs etc. That is a major problem. Media management must be a

I've almost never seen a newspaper article on something for which I have
direct personal knowledge get it right. However, this is not a TECHNICAL
problem, it's a management problem. You're asking in the wrong forum.

Danny