Subject: [dnsop] what scares me about doug...

From: Edward Lewis <[email-masked]>
Date: Tue, 14 Nov 2006 11:52:14 -0500

...'s presentation last week...

(I'm being sarcastic about being scared of Doug, btw.  But it gives a 
good title.)

The issues of SPF, DKIM, other SPAM prevention technics and in 
general slapped-on security approaches is not a topic for this list. 
If and when I refer to a case, it's for example only, not for 
comparison, and maybe even to expose my relative ignorance of the 
reality.

What I find potentially troubling is that an application might choose 
to refer to the DNS in an unwise way to find ancillary data to verify 
that a received message unit is genuine and is authorized to be 
processed.  I have had some experience in building such systems, the 
experience being part of research projects, and have seen first hand 
some of the startling performance issues that may ensue.  This work 
is about 6 years old now, but it didn't really hit home until now.

The first ingredient in the problem is that the processing unit 
performing the security checks is the end receiver.  This means that 
there is probably a large population base from which traffic can be 
sourced (legitimately).  These sources are prone to "abuse" because 
it is their job to accept data and then protect themselves.  What 
else can they do but do their job?

The second ingredient is a malicious traffic load that is sent to the 
above processing units just to kick off the protection mechanism. 
The fact that the malicious load can be generated is both what 
creates the need for the corrective measures and is also what causes 
the corrective measures to be abused.  Spam is not just a problem 
because it exists, it is a problem because the defenses against it 
are just as much of a problem.  (Not to mention the false positives.)

The third ingredient is the amplification factor that is a result of 
robustness of the security.  When slapping security onto any existing 
system, some level of robustness is lost.  If the security can only 
find data via one path, then robustness is brought down to a single 
point of failure.  E.g., if all certificates are on one server, knock 
out that server and the system will cease to function (securely).  To 
counteract this, security can be made to be multipath - like using 
multiple X.509 certificates for the same key, each certificate from a 
different issuer, etc.  But each added path may prove to be merely an 
amplification of an abuse.

The fourth ingredient is the complexity of the security operation. 
If the robustness approach is hard-to-get-right, then the sender will 
unintentionally cause the receiver or a third party a lot of pain. 
(Of course a malicious sender will be able to do this intentionally 
too.)

Keep in mind that the impact to the DNS in this situation is a 
backdrop to another abuse.  It might not even be apparent that there 
is an attack on the DNS or on some other system via the DNS, when 
everyone is staring at yet another infusion of spam.

My experience in this (albeit a prototype) was a response to a 
network flooding attack.  As repair code flowed to routers to stem 
the flood, the repair code required certification to let the router 
know it was legit.

Because of the flood, the routers couldn't get to the DNS to get the 
certificates.  We solved that by eventually building a second 
"control plane" network.  In building the data, because we were 
battling a deadline, I implemented two ways to retrieve the 
certificate revocation list and installed two places for the CRLS to 
be found.  During the one-plane approach, I watched the prototype run 
and saw the 4 DNS attempts (each location times each retrieval 
method) time out, as nothing got though the flood.  Instead of 
helping matters, the security added to the flood - 4 DNS queries 
times the retries because of time out.  Maybe this is too short to 
convey the issue, but I have seen security make a matter worse.

In conclusion, there isn't really much the DNSOP community can do 
about this.  If we try to protect the DNS, well, there's nothing that 
can really be done in the protocol about this, other than to ask 
"please use the DNS less."  The recommendation is to whomever is 
judging the architectures to first consider the load on the DNS and 
more over to consider whether there is any tractable way to make 
operating the security system free from an abuse.

What I kind of am getting to here is that if the TXT RR is used to 
convey security policy or a search list of servers, then it would be 
hard to enforce reasonable approaches.  If an new RR is required for 
the approach, then the protocol can be built in such a way that 
number of searched sources can be limited.  Further, if there is no 
fallback from one RR to another, no "alternate" paths to take, I 
THINK (can't prove) that it would make it easier on an operator to 
know what is the appropriate level of security response to the 
receipt of a message.  If the RR RDATA can only have two servers and 
there can only be one RR considered per event, the load has a chance 
of being managed.

This is the FUD that I took away from the meeting last week.

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

I didn't miss the meeting, I left it before I arrived.
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Paul Wouters <[email-masked]>
Date: Tue, 14 Nov 2006 20:27:56 +0100 (CET)

On Tue, 14 Nov 2006, Edward Lewis wrote:

> What I find potentially troubling is that an application might choose to refer
> to the DNS in an unwise way to find ancillary data to verify that a received
> message unit is genuine and is authorized to be processed.

(Un)fortunately, the DNS is the best distributed database we have.

>  I have had some
> experience in building such systems, the experience being part of research
> projects, and have seen first hand some of the startling performance issues
> that may ensue.  This work is about 6 years old now, but it didn't really hit
> home until now.

Trust me, comparing the performance of some DNS lookups to avoid the need
to run them through anti-virus and anti-spam processing is moot. DNS lookups
are cheap. Without using various DNS checks (and greylisting, which i
personally despise) I'd have to add a few Xeons to our anti-spam setup.

> The first ingredient in the problem is that the processing unit performing the
> security checks is the end receiver.  This means that there is probably a
> large population base from which traffic can be sourced (legitimately).  These
> sources are prone to "abuse" because it is their job to accept data and then
> protect themselves.  What else can they do but do their job?

Whether the anti-spam postfix, or my real mailserver's postfix is accepting
the message hardly matters from a security point of view. And I think
experience has proven that a unix based anti-spam server's security risk
outweights 500+ employees running malware within your network.

> Spam is not just a problem because it exists, it is a problem because
> the defenses against it are just as much of a problem.  (Not to mention the
> false positives.)

And as I argued with people like Gilmore in the past - I can't afford
not to do spam filtering. Even if my customers wanted to see all spam
(they don't) and even if I felt morally obligated to deliver email even
if it is spam (I don't) and provided I had the hours per day to manually
read the spam (I don't), then my personal error rate on messages would
be higher then my current technical setup.

> robustness of the security.  When slapping security onto any existing system,
> some level of robustness is lost.

and gained?

> certificates.

Sounds like a design mistake.

> In conclusion, there isn't really much the DNSOP community can do about this.

There are things we can do, mostly involved with BCP's and sane software
defaults and things like AS112.

> If we try to protect the DNS, well, there's nothing that can really be done in
> the protocol about this, other than to ask "please use the DNS less."

Why would we ask that? DNS serves many good purposes. If "protecting" comes
down to "don't use", then you also have no more need to protect it.

> What I kind of am getting to here is that if the TXT RR is used to convey
> security policy or a search list of servers, then it would be hard to enforce
> reasonable approaches.  If an new RR is required for the approach, then the
> protocol can be built in such a way that number of searched sources can be
> limited.  Further, if there is no fallback from one RR to another, no
> "alternate" paths to take, I THINK (can't prove) that it would make it easier
> on an operator to know what is the appropriate level of security response to
> the receipt of a message.  If the RR RDATA can only have two servers and there
> can only be one RR considered per event, the load has a chance of being
> managed.

Having a seperate RR makes it easier to make custom decisions based on that
RR set.

As for the mentioned EHLO attack, this is easilly prevented in postfix.
Add to main.cf:

	smtpd_helo_restrictions = check_helo_access hash:/etc/postfix/helo_access

Add to /etc/postfix/helo_access:

	# reject ehlo string with more then 5 host parts
	.*.*.*.*.*	"550 HELO string too long, rejecting possible DOS"

And if you think SMTP spam filtering email is bad, wait for your phones
to need to do SIP spam filtering. How long before we can't trust our
phones anymore?  I give it two years top.

Paul
.

---

From: Edward Lewis <[email-masked]>
Date: Tue, 14 Nov 2006 15:08:42 -0500

>>  some level of robustness is lost.

No, robustness is lost - in the sense that if you examine the state 
machine of the protocol, once legal states are no longer legal.  In 
many state machines there are paths that travel from safe states 
through questionable states and then back to safe states.  It is when 
travelling though a questionable state (sometimes known as a 
vulnerability) that bad things could happen; so slapped-on security 
makes these states unreachable.  When the number of possible good 
paths are reduced, robustness is lost.

When I say that robustness is lost, I'm looking at the protocol 
design.  I suppose that if your concern is capacity (i.e. the length 
of queues), you could say that security is improving robustness by 
avoiding hitting limits.  That may be, but the point I was trying to 
make above is that if you take an operating system working without a 
problem and then crimp it where you think there might be a 
vulnerability, you would be wise to crimp it in ways that do not 
choke it.

>Sounds like a design mistake.

Well, something was lost in the translation.  The case was that a 
router had to pull from a flood of packets some information that 
would stem the flow of bad packets.  To validate the good 
information, credentials needed to be judged.  Even if the X.509 was 
sent along in entirety, the revocation list could not be sent 
reliably.  The error is in relying on remote security data when 
reaching for it may be impossible.  (How do you put all needed 
security data local to where an attack can be fended off?  If you 
push ahead of time, then you are using resources needlessly in 
"peacetime."  And - in an open population, to where do you send 
stuff?)

>>about this.

And we could encourage better protocol design.  Note that I refer to 
"slapped-on" security.  Not all security is doomed - it's when we 
slap it on we are risking doing a worse job.  E.g., if in DNSSEC we 
get a signature, it is simple to judge the security of the data.  If 
we don't get the signature, we have to go see if the signature was 
supposed to be there, etc.  Same for the spam defense approaches.

Think of what it would mean to DNS if all records came with a 
signature field built in. (For some of is, think "slabs.")  The 
prove-able non-existence issue wouldn't be the same.  Replays may 
declare a zone empty, but you couldn't poison cache with false data 
by downgrading a zone's security.

>>  If we try to protect the DNS, well, there's nothing that can really be done
>>  in the protocol about this, other than to ask "please use the DNS less."
>

Exactly my point.

>phones anymore?  I give it two years top.

Fortunately for phones there is heavy regulation. ;)  Do-not-call 
lists and the like.  (The non-SIP phones I mean.)
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

---

From: Paul Vixie <[email-masked]>
Date: 15 Nov 2006 01:22:36 +0000

> (Un)fortunately, the DNS is the best distributed database we have.

"Not for very much longer." (Columbia, RHPS)
-- 
Paul Vixie
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: bert hubert <[email-masked]>
Date: Wed, 15 Nov 2006 08:39:36 +0100

> "Not for very much longer." (Columbia, RHPS)

Can you elaborate on this intriguing suggestion? Googling on that phrase
only brings up pictures of a thing called "the rocky horror picture show".

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html

---

From: Robert Story <[email-masked]>
Date: Wed, 15 Nov 2006 17:42:03 -0500

On Wed, 15 Nov 2006 08:39:36 +0100 bert wrote:
BH> On Wed, Nov 15, 2006 at 01:22:36AM +0000, Paul Vixie wrote:
BH> > > (Un)fortunately, the DNS is the best distributed database we have.
BH> > 
BH> > "Not for very much longer." (Columbia, RHPS)
BH> 
BH> Can you elaborate on this intriguing suggestion? Googling on that phrase
BH> only brings up pictures of a thing called "the rocky horror picture show".

RHPS == Rocky Horror Picture Show, and he is quoting a line from "The
Time Warp", a song from the show/movie. Columbia was one of the
characters, but it was actually Magenta who had this line. In the Time
Warp, one line is "time is fleeting", to which the response is "not for
very much longer."

Since I don't see any upcoming replacement for the DNS, I can only
assume that he's saying we're all mad (ie crazy) and caught in a time
warp. I for one welcome our new sequined lingerie wearing transvestite
overlords. :-)

-- 
Robert Story
SPARTA

---

From: wayne <[email-masked]>
Date: Wed, 15 Nov 2006 22:46:52 -0600

In <[email-masked]> Paul Wouters <[email-masked]> writes:

> [large amounts of spam discussion snipped.  I think that is
>  off-topic here ]

> 	.*.*.*.*.*	"550 HELO string too long, rejecting possible DOS"

I'm afraid you have almost completely missed the point of Williams
"HELO attack".

The first and most important part for this WG is that you don't need
lots of labels or anything.  If I, as an attacker, can make you do a
DNS lookup on a name that I choose, I can make you attack an innocent
third party.

Simply doing a DNS lookup on the HELO name will do.  As will checking
to see if email's From: address is a valid domain, or having an <img>
tag in an HTMLized email, or sending email through large ISP with a
To: address of the attacker's special domain name.  

Let's look at William's attack again:

  EHLO big.bad.example.net

and he previously set up the following in DNS:

 $ORIGIN bad.domain.example.net
   big IN NS victim-dns1.example.com
   big IN NS victim-dns2.example.com
   big IN NS victim-dns3.example.com
   big IN NS victim-dns4.example.com
   big IN NS victim-dns5.example.com
   big IN NS victim-dns6.example.com
   big IN NS victim-dns7.example.com
   big IN NS victim-dns8.example.com

There aren't 5 labels in the EHLO, and the attack could be done with
only 2 labels.

The whole problem here is that your local resolver will chug through
that list of name servers and try asking each one for an answer.

Actually, it is worse...  We talk about "recursive resolvers" for a
reason, they are recursive.  Most of the time, you won't find lots of
name servers for a victim, but that doesn't make any difference.
"victim-dnsX" can be replaced with any string and example.com's name
servers will be asked for the A record for it so that the recursive
resolver can, in theory, ask it about big.bad.domain.example.net.

I'm not sure how many name servers resolvers will try checking, I
suspect it will check all of them.  I've checked, and my version of
bind9 will certainly send up to 20 queries.

So, name server records are actually worse than SPF.  With SPF, I made
sure that there were some limits on the number of queries that can be
sent. 

-wayne
.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 16 Nov 2006 06:19:54 -0800

On Wed, 2006-11-15 at 22:46 -0600, wayne wrote:
> > As for the mentioned EHLO attack, this is easilly prevented in 
> > postfix. Add to main.cf:
> >
...
> sure that there were some limits on the number of queries that can be

Wayne, you have missed the point as well. In addition to a possible 100
victim targeted queries, SPF provides the attacker more than 111
opportunities per execution of an SPF script to introduce an NS chaining
scheme as well.  SPF makes an NS chaining problem 111 times worse!  In
addition, early termination of an SPF script coupled with initiation of
another script defeats exponential back-off.

-Doug

.

---

From: "william(at)elan.net" <[email-masked]>
Date: Thu, 16 Nov 2006 08:17:16 -0800 (PST)

> another script defeats exponential back-off.

Like I said in my original post on this subject it all comes down to
the timeout (either the one from SPF processor or the one from
resolver itself or application one) as there are enough opportunities
with & without SPF to do chaining and in the end the end its going
to be about same amplification.

The thing I'm seriously afraid of are "advanced" libraries designed
for high-end multi-threaded systems that would try to send multiple
requests asynchronously. For SPF that means multiple listed "exists" 
and/or "mx" instead of trying first and then 2nd, etc - but I don't
think there are any doing it this way right now even though SPF allows
for it in its design. More serious issue maybe mail servers as there
are some that would try to get resolution for all listed MX in a
asynchronous manner. Lucky for us the number of such deployments
are few where as attack depends on great number of systems doing it.

P.S. I'm not sure this discussion is good to be held on public mail
list. Think about who it helps most when we publicly brainstorm most 
useful way to accomplish this sort of attack?

===
William Leibzon
Elan Networks
[email-masked]
.

---

From: wayne <[email-masked]>
Date: Thu, 16 Nov 2006 12:15:06 -0600

In <[email-masked]> "william(at)elan.net" <[email-masked]> writes:

> resolver itself or application one) [...]

Yes, the SPF RFC4408 does talk about timeouts, but it is limited to a
"MAY time out", rather than a SHOULD or a MUST.  RFC4408 does say that
"SPF implementations SHOULD limit the total amount of data obtained
vis DNS queries", which again helps limit this general problem for the
specific case of SPF.

Unfortunately, I not so sure that the recursive name servers have a
timeout.  Resolver libraries and things that use /etc/resolv.conf do,
but from my initial checking, it appears that things like bind's named
continue to process request.

> useful way to accomplish this sort of attack?

I agree with this.  That is why clinte.schlitt.net has a short TTL and
I didn't make it trivial to determine the exact NS records.  (e.g.,
"dig clinte.schlitt.net ns" will return SERVFAIL rather than tell you
want is going on.)

Basically, when I saw a problem like this in SPF, I took steps to
change things to try and make it safer.  I'm not sure if I managed to
put all appropriate limits into the spec, but it is better now than
when I started.  (And, no where near as bad as Doug makes out.)

So, if I was in charge of the DNS RFCs like I ended up in charge of
the SPF RFC, I would be working on finding a good fix for this.

-wayne
.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 16 Nov 2006 11:19:46 -0800

On Nov 16, 2006, at 8:17 AM, william(at)elan.net wrote:
>>> So, name server records are actually worse than SPF.  With SPF, I  
>>> made sure that there were some limits on the number of queries  
>>> that can be sent.
>>
>> Wayne, you have missed the point as well. In addition to a  
>> possible 100 victim targeted queries, SPF provides the attacker  
>> more than 111 opportunities per execution of an SPF script to  
>> introduce an NS chaining scheme as well.  SPF makes an NS chaining  
>> problem 111 times worse!  In addition, early termination of an SPF  
>> script coupled with initiation of another script defeats  
>> exponential back-off.
>
> Like I said in my original post on this subject it all comes down  
> to the timeout (either the one from SPF processor or the one from  
> resolver itself or application one) as there are enough  
> opportunities with & without SPF to do chaining and in the end the  
> end its going to be about same amplification.

In some SPF routines, a timeout is as short as 5 seconds and DNS  
transaction potentials for SPF scripts remains at multiples of 100!   
Attacks may leverage spam sent from bot-nets which represent more  
than 70% of the sources.  An SPF related attack demands little of the  
bot-net's resources.

> requests asynchronously.

When a victim is targeted by the distribution of messages referencing  
malicious SPF scripts, then "advanced" libraries are not required  
before abrupt timeouts become highly problematic.

> For SPF that means multiple listed "exists" and/or "mx" instead of  
> trying first and then 2nd, etc - but I don't think there are any  
> doing it this way right now even though SPF allows for it in its  
> design. More serious issue maybe mail servers as there are some  
> that would try to get resolution for all listed MX in a  

There seems to be agreement an SPF exploit as described can be  
constructed that exceeds the number of DNS transactions indicated in  
the spf-dos-exploit draft.  Your techniques simply increase this  
number.  Why quibble over the gains achieved by a difference between  
the TTL of attacking records with that of negative caching?  One day  
MX record retention with 5 minute negative cache retention (which may  
not be controlled by the victim) means 288 negative address  
references might be made for each malicious MX record reference.

There are similar attacks that simply chain 11 wildcard SPF records  
without relying upon negative caching.  SPF record chaining removes  
an ability to detect suspicious records as well.  SPF scripts  
introduce many avenues of attack.  : (

The problem you are currently describing exists without "advanced"  
SPF libraries.  SPF script execution must be considered in  
aggregate.  Thousands of such libraries will be employed by an  
attacker.  SPF scripts can be used that completely flood a network  
backbone interconnect.

> list. Think about who it helps most when we publicly brainstorm  
> most useful way to  accomplish this sort of attack?

In the spf-dos-exploit draft, this statement was made:

"Other techniques can further increase the severity of such an  
attack, but are not reviewed."

Your efforts to point out other DNS weaknesses only strengthens  
concerns related to SPF scripts.  This is not a general problem  
related to DNS.  SPF scripts dramatically increase the impact  
especially, when other weaknesses are also exploited.

-Doug

.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 17 Nov 2006 09:31:44 +1100

> continue to process request.

	All versions of named will timeout their requests.
 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]
.
dnsop resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html