Subject: Re: [Asrg] draft-levine-iprangepub-01

From: David Nicol <[email-masked]>
Date: Tue, 4 Jan 2011 10:27:58 -0600

I've been meaning to share a summary of December's discussions as they
seemed to me, so here it is:

The problem is extending listing protocol to support listing of a range as
well as a point, hopefully without sacrificing too many data bits.

The current DNSxL protocol SHOULD support ranges of no-data information,
given properly implemented NXDOMAIN handling and presentation of the
appropriate glue.

Alternate listing protocols (such as the ones I presented, and Levine's
b-tree proposal, which could easily extend to other information domains that
can be represented as a b-tree, which is, any mapping with sortable keys,
such as (not to encourage black-hat behavior, but it's a good example of
data as such) a rainbow table)

I pursued a way of packing queries using natural bit flipping instead of
explicit leaf ranges until it became clear that my end result was equivalent
to rfc5782 representation of ipv6 address names, with the addition of
supporting shorter names for ranges.

This resulted in my envisioning what could be a protocol to compete with
draft-levine-iprangepub-01<http://tools.ietf.org/html/draft-levine-iprangepub-01>
in
this space, amounting to a couple of extensions to the 5782 protocol.

In short, when there is a range at a.b.c.rangelister.example, all queries in
subdomains of the range-representing entry, such as
t.m.i.a.b.c.rangelister.example, get responded to with an NXDOMAIN error,
while the range-representing entry holds the data for the range.

This means that a list query library would start with the full query and
work its way up until it gets a result, and the NXDOMAIN at the appropriate
part of the glue determination process will prevent the too-long queries
from reaching the server, as they will be handled by the cache server.

As a policy enforcement measure, a server should identify query sources not
respecting NXDOMAIN responses and block them.

The above is intended as a "representing of ranges" extension to RFC 5782.

On Wed, Dec 29, 2010 at 10:48 AM, John R. Levine <[email-masked]> wrote:

> I've done another version of it, with mostly editorial changes.  Comments
> as always welcome.  I guess now I should implement it.
>
> http://tools.ietf.org/html/draft-levine-iprangepub-01
>
-- 
“The aeroplane is fatally defective. It is merely a toy—a sporting
play-thing.  It can never become commercially practical." -- Nikola Tesla

---

From: John Levine <[email-masked]>
Date: 4 Jan 2011 21:03:24 -0000

Thanks for the summary.

>The current DNSxL protocol SHOULD support ranges of no-data
>information, given properly implemented NXDOMAIN handling and
>presentation of the appropriate glue.

I don't think this is right, unless you're assuming that DNS caches
will do stuff they don't do now.

>from reaching the server, as they will be handled by the cache server.

You might want to try this with a few caches.  I'd be surprised if
they synthesized answers for subdomains of NXDOMAIN entries.  I just
tried it with dnscache from djbdns, and it doesn't.

R's,
John

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 4 Jan 2011 17:12:59 -0500

> tried it with dnscache from djbdns, and it doesn't.

After having used dnscache for quite a while, with, er, not very 
satisfactory performance results[+], it might be worth while retrying 
the experiment with bind and/or, say, unbound (which is what I use now) 
which are better suited to heavy iron caching.

While dnscache may indeed be quite popular at the smallish scale it 
don't do well at the medium to very large end.  Which means that the 
problems it may represent for IPv6 caching may be common on a per-site 
level, but not common at a sheer volume level, and may not matter _that_ 
much overall.

[+] The reasons as to why are OT, but I can elaborate if necessary.

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 01:55:03 -0000

>which are better suited to heavy iron caching.

Grumble.  I'll build unbound and see what it does, but I would be
rather surprised if the results were any different.  The DNS crowd
has dislikes synthesizing answers, even in cases like this where
you can argue that the synthesized answer should be correct.

R's,
John

---

From: "John R Levine" <[email-masked]>
Date: 4 Jan 2011 21:15:32 -0500

>>> tried it with dnscache from djbdns, and it doesn't. ...

> rather surprised if the results were any different.

I wasn't surprised.  I looked for foo.gurus.org, got an NXDOMAIN, then 
queried bar.foo.gurus.org, and it went back to the authoritative server to 
get the answer.

Nice cache, though.  Maybe I'll give it a try.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 4 Jan 2011 21:24:30 -0500

> you can argue that the synthesized answer should be correct.

I get the impression that the unbound writers were a bit more pragmatic 
w.r.t. performance versus personal theories of "purity" than, say, DJB.

[The dnscache performance issue is just plain dumb.]

You may well be right, but I'd hesitate to use dnscache's behaviour as 
representative without actually testing something else.

I have several unbound cache servers that might be easy to make visible 
on the net if you can't get your own unbound up quick.

Unbound seems to be _really_ easy to set up tho.

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 4 Jan 2011 23:39:32 -0500

On 1/4/2011 9:15 PM, John R Levine wrote:

> I wasn't surprised.

Oh well.

> I looked for foo.gurus.org, got an NXDOMAIN, then

I guess I must have misunderstood.  If _that_ is how you test it, I 
can't imagine a DNS server doing otherwise.

> Nice cache, though.  Maybe I'll give it a try.

Isn't it?  _Highly_ recommended.

dnscache has a fixed size outstanding query buffer (compile-time 100 
queries IIRC), and throws queries away if you overflow it.  If you do 
the obvious thing and recompile it with a larger buffer, as the match 
between query and response is a _linear search_ (!), it degrades 
atrociously.  Especially if the admin naively recompiles for several 
thousand entries.

[It's apparently even worse than that, but I disremember all the details.]

Unbound's buffer is run-time allocated, and reply/query matching is 
hashed.  I don't know anything about the guts of bind but it don't 
degrade like dnscache either.

[I exiled myself off the Corporate DNS infrastructure (bind) for my mail 
servers/traps occasionally causing one of their servers to drop _other_ 
queries for minutes at a time.  I was DDOS'ing it. Hence the initial 
foray into a server-coresident dnscache which behaved worse (but only 
affected me), and then finally to a server-coresident unbound (which has 
been 100% perfect so far).]

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 05:13:14 -0000

>can't imagine a DNS server doing otherwise.

It's not a totally ridiculous idea.  There's a difference between
NODATA, which means that the name exists but has no records, and
NXDOMAIN which means the name doesn't exist.  Assuming the data are
consistent, if you see NXDOMAIN there should be no subdomains of the
name with the NXDOMAIN.  But the DNS crowd hates synthesizing answers
for reasons that strike me as mostly religious, so nobody does.

>queries IIRC), and throws queries away if you overflow it.

For the handful of computers that use my cache, it hasn't been a problem.

R's,
John

---

From: Steve Atkins <[email-masked]>
Date: Tue, 4 Jan 2011 21:49:57 -0800

On Jan 4, 2011, at 9:13 PM, John Levine wrote:

> It's not a totally ridiculous idea.  

It is.

> There's a difference between
> NXDOMAIN which means the name doesn't exist.

Yes.

>  Assuming the data are
> name with the NXDOMAIN.  

No.

> But the DNS crowd hates synthesizing answers

They're generally not "religious" so much as they're the opinions
of people who know DNS.

NXDOMAIN for c.b.a simply means that there is no RR for c.b.a.

It tells you absolutely nothing about the existence of an RR for d.c.b.a,
and any recursive resolver that synthesized results for such based on
any RRset for c.b.a would be, simply, broken.

platter:test steve$ host 0.0.127.sbl.spamhaus.org
Host 0.0.127.sbl.spamhaus.org not found: 3(NXDOMAIN)
platter:test steve$ host 2.0.0.127.sbl.spamhaus.org
2.0.0.127.sbl.spamhaus.org has address 127.0.0.2

Or a little more complex:

platter:test steve$ host c.b.a.tupid.org
Host c.b.a.tupid.org not found: 3(NXDOMAIN)
platter:test steve$ host d.c.b.a.tupid.org
d.c.b.a.tupid.org has address 127.0.0.1

Cheers,
  Steve

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 5 Jan 2011 01:29:17 -0500

> for reasons that strike me as mostly religious, so nobody does.

Er.  But isn't this (a) whether you interpret NODATA (aka No Error) as 
NXDOMAIN or otherwise a DNS client library and/or DNSBL query function 
decision, and (b) does DNS really return "No Error" for b.a if there's 
no RR for b.a, but there is one for c.b.a?

All my DNSBL query code is written well "below" the gethostbyname() 
level of abstraction, and I always have to make a conscious decision (so 
far) to treat NOERROR as NXDOMAIN on DNSBLs.  On DNSWLs I treat anything 
but an affirmative NXDOMAIN (ie: server fail, timeout, no error) as listed.

[Most of my DNSBLs and all of my DNSWLs are served locally via a single 
combined A query, so a DDOS on an external DNSWL doesn't cause me to 
accept everything, and I get 10 DNSxL queries for (almost as little) as 
the price of one.]

> For the handful of computers that use my cache, it hasn't been a problem.

Note "handful".  I'd also suggest relatively low DNS-consumptive.  In my 
case, I'm talking about a single mail server doing 4-8 DNS queries per 
email, and 20-100 emails/second blowing a dnscache instance's brains out.

dnscache is fine for the average small DNS user.  But if you're doing 
lots, it leaves something to be desired.

When I crank up the process to ASNify the past week's worth of seem IPs 
(6+ million) via a local copy of the routeviews zone for global metrics, 
it matters if you want it done in a reasonable time (I'm doing 10 
minutes now).  Udns or ParaDNS is a great way to break nearby caches 
(and consumer level routers as some people have found out to their 
dismay) if you're not careful.

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 06:44:24 -0000

>It tells you absolutely nothing about the existence of an RR for
>d.c.b.a, and any recursive resolver that synthesized results for such
>based on any RRset for c.b.a would be, simply, broken.

You might want to look at RFC 4592, particularly section 2.2.2,
which explains this corner of DNS arcana and the difference between
NOERROR and NXDOMAIN.

But you're in good company, since a lot of people who write alleged
DNS software haven't read it either.

>2.0.0.127.sbl.spamhaus.org has address 127.0.0.2

The guy who wrote rbldnsd didn't read RFC 4592.

>d.c.b.a.tupid.org has address 127.0.0.1

Nor did the guy who wrote whatever free01.editdns.net is running.  I
can't tell what it is, since the usual chaos version.bind query
produces a bogus result that suggests he didn't read section 3.2 of
RFC 1035, either.

I agree that synthesizing results would be risky, since there is a
substantial amount of DNS software that doesn't properly report the
difference between NOERROR and NXDOMAIN.  If DNS servers followed the
specs, it should work, perhaps in a more perfect world than this one.

With that said, I still like my b-tree hack, which makes queries that
shouldn't get either NOERROR or NXDOMAIN, a lot better as a way to
publish ranges of addresses in a DNSxL.

R's,
John

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 06:54:26 -0000

>no RR for b.a, but there is one for c.b.a?

Per my previous message, the answer to (b) is yes, according to
RFC 4592, but there's a lot of buggy software that gets it wrong.

>Note "handful".

Indeed.  If I find that I have to run the mail system for a major
multinational corporation, I expect I'll use something else.  djbdns
hasn't been updated since 2001, when its query limits were reasonable.
But that's Dan for you.

>... I'm talking about a single mail server doing 4-8 DNS queries per 

That reminds me, any chance of getting running some IP/timestamp logs
through my DNSBL analyzer and see how much a cache helps a busy setup
like yours?

R's,
John

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 5 Jan 2011 02:11:08 -0500

> through my DNSBL analyzer and see how much a cache helps a busy setup

I'll run it on something moderately large tomorrow.

Send me the script.

I'll also see if I can get it run on something humungous.

---

From: Scott Howard <[email-masked]>
Date: Tue, 4 Jan 2011 23:17:45 -0800

> which explains this corner of DNS arcana and the difference between

That's the joy of standards - you can never have too many of them.

RFC 1034 implies that NXDOMAIN means that the domain does not exist, and
thus it would not be possible for any sub-domains to exist (although if you
squint hard enough whilst reading it you can actually make it say the
opposite).

RFC 2308 states that a response of NXDOMAIN does NOT means that the domain
doesn't exist, and thus it's feasible for a there to be sub-domains even on
an NXDOMAIN response.

Then there's RFC 2535, 4592 and probably a few others which even further
muddy the waters.

At best, this is a vague area, and the only correct thing to do is to not
presume any specific behavior - the only guarantee is that different
nameservers are going to act differently.

  Scott.

---

From: Steve Atkins <[email-masked]>
Date: Tue, 4 Jan 2011 23:41:53 -0800

> which explains this corner of DNS arcana and the difference between

4592 is contradicted by some other RFCs and, more importantly, by
common behavior.

What's "correct" behaviour would be an interesting thing to bring
up with the IETF DNS folks (pick a standard, any standard), but
what's implemented is what's important when it comes to relying
on that behaviour.

> specs, it should work, perhaps in a more perfect world than this one.

The authoritative nameservers I've used recently all do the "right thing"
there: no RRs => NXDOMAIN. And the recursive servers I have handy
also do the "right thing" and don't synthesize fake answers.

For layering something on top of DNS the deployed implementation is
the only spec that matters - what's implemented trumps what would be nice.

> publish ranges of addresses in a DNSxL.

If you're going to the effort of adding that much functionality
at the client end, switching to a better matched protocol instead
might be better. All the high traffic DNSBL users already use
a push protocol of sorts. Moving to a better, more standardized
one might be a win. A local server for protocol X could still offer
a DNS interface to the MTA to ease implementation

Cheers,
  Steve

[1] and potentially CNAMEs and suchlike.

[2] or maybe a delegation, if there's a domain cut somewhere above
c.b.a.

---

From: Bill Cole <[email-masked]>
Date: Wed, 05 Jan 2011 12:52:55 -0500

> RFC 4592, but there's a lot of buggy software that gets it wrong.

FWIW, a very quick randomish check of the DNSBL's I find interesting shows 
exactly one (tor.dnsbl.sectoor.de) that answers such queries correctly other 
than my own local ones which are served by BIND 9. As a cache, BIND 9 will 
cache and serve a NXDOMAIN answer that should really be NODATA/NOERROR, even 
if it has cached records under the supposed NXDOMAIN.

The point: getting this wrong is the norm in the DNSBL space. It may be due 
to buggy software, but the bug is endemic.

---

From: David Nicol <[email-masked]>
Date: Wed, 5 Jan 2011 13:04:52 -0600

> tried it with dnscache from djbdns, and it doesn't.

okay, so the list query library has to start from the top and drill down.
Anyway, the interesting generalization is that once you start supporting
ranges, there's no reason to have a particular length or granularity of
"items in the list"; the other side of this is imagining other problem
domains where a-Btree-in-DNS would be useful.

I wasn't able to find a paragraph about ranges with holes in them, or the
representation of the data beyond this point is in/not in the list, in
iprangepub-01.

by "ranges with holes" I mean, the range is all listed except for the
sublist.

also the concept of increasing resolution with finer-grained querying,
useful for reputation-by-address kinds of databases, seemed absent.

---

From: David Nicol <[email-masked]>
Date: Wed, 5 Jan 2011 13:21:47 -0600

> the only spec that matters - what's implemented trumps what would be nice.

it depends if we are layering over "DNS: the installed infrastructure" or
"DNS: the protocol." If the second, we are free to demand and enforce a
particular semantics.

---

From: Steve Atkins <[email-masked]>
Date: Wed, 5 Jan 2011 11:27:26 -0800

On Jan 5, 2011, at 11:21 AM, David Nicol wrote:

> it depends if we are layering over "DNS: the installed infrastructure" or "DNS: the protocol." If the second, we are free to demand and enforce a particular semantics.

If the second... we could do much, much better than DNS for the WAN part of any protocol.

If we're building the infrastructure from scratch, we can demand any semantics we like, but unless the existing software happens to implement them, we're still out of luck unless we're planning on implementing replacements from scratch.

The only reason DNS was originally used was because it was easy to jam into sendmail. The only reason it's used now is that the infrastructure for it is ubiquitous (where infrastructure includes auth servers, caches and MTA support).

I think DNS still makes sense as a protocol to use between the MTA and a local server. It makes much less sense for WAN distribution.

Cheers,
  Steve

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 20:02:34 -0000

>there: no RRs => NXDOMAIN.

FYI, BIND 9, which must be the most popular authoritative server on
the net, does what 4592 says, NOERROR at an empty nonterminal:

$ host -t txt foo.bar.sp.am light.lightlink.com
Using domain server:
Name: light.lightlink.com
Address: 64.57.176.2#53
Aliases: 

foo.bar.sp.am descriptive text "Hi there"

[johnl@joyce ~]$ host -t txt bar.sp.am light.lightlink.com
Using domain server:
Name: light.lightlink.com
Address: 64.57.176.2#53
Aliases: 

bar.sp.am has no TXT record

[johnl@joyce ~]$ host -t txt har.bar.sp.am light.lightlink.com
Using domain server:
Name: light.lightlink.com
Address: 64.57.176.2#53
Aliases: 

Host har.bar.sp.am not found: 3(NXDOMAIN)

> And the recursive servers I have handy also do the "right thing" and
>don't synthesize fake answers.

No argument there, no cache synthesizes answers, even the ones that
have enough DNSSEC data to do so reliably.

>the only spec that matters - what's implemented trumps what would be

Well, for the third time, yeah. 

>> With that said, I still like my b-tree hack, ...
>
>one might be a win.

Huh?  The functionality to the client is the same, it looks up an IP
address and finds out whether or not it's listed.  The internal
representation is different, and it should cache a lot better.  The
extra work at the client is actually quite small, a simple loop to
walk through some easily parsed records.

R's,
John

---

From: Matthias Leisi <[email-masked]>
Date: Wed, 5 Jan 2011 21:07:17 +0100

> walk through some easily parsed records.

The current proposal is not really a text format, so "easily parsed"
does not apply under all circumstances.

-- Matthias

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 20:08:20 -0000

>representation of the data beyond this point is in/not in the list, in

It doesn't handle ranges with holes.  If you want a hole, you divide
the range.  I don't understand what you mean by "data beyond this
point is in/not in the list."  If you mean checking that an address is
less than the first entry in the list or greater than the last entry,
the lookup process handles that automatically, as it notes in section
3.

>useful for reputation-by-address kinds of databases, seemed absent.

Sorry, don't understand what that means, either.  If you're asking
whether it could return the prefix in which an entry was found rather
than just saying the entry was there, of course it can.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: David Nicol <[email-masked]>
Date: Wed, 5 Jan 2011 15:00:19 -0600

> than just saying the entry was there, of course it can.

Differentiating between a good in a range of bad and a bad in a range of
good, or getting some composite information about a range before drilling
down further. This feature is most likely a YAGNI.

---

From: John Levine <[email-masked]>
Date: 5 Jan 2011 23:21:02 -0000

>down further. This feature is most likely a YAGNI.

That sounds like an interesting problem, but it's not one that DNSBLs
have tried to address, and it's not one that I'm trying to solve.

Back when we were running DAC, after we did VBR which lets you publish
what is basically a single bit saying "this signer is good", we looked
at ways to publish more complex reputation info, and what we found was
that we don't understand the issues well enough to try to standardize
anything.

Regards,

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 5 Jan 2011 19:00:22 -0500

> a DNS interface to the MTA to ease implementation

I must confess to a bit of confusion.  We seem to be talking about 
really naive IPv6 DNSBL implementations blowing out caches on high 
volume, and yet, at the same time, we acknowledge that the large DNSBL 
users already download zone files.

Rsync is essentially the defacto standard for bulk DNSBL transfer, and 
as you say it's "not awful".  So, we don't seem to have a significant 
difficulty with that.

I do know of one (commercial) blacklisting system that does use a more 
explicitly "incremental" distribution mechanism, but it's probably not 
that much better than rsync, and in fact it may be worse.

Even if we were to do something as simplistic as chop IPv6 queries at 
the /64, given that the number of spammers and bots doesn't magically go 
up simply because there's more bits to hide in, the caching problem 
appears to not that much worse than it already is with IPv4.

IOW, some of the discussion threads here seem to be solutions looking 
for problems.

Clearly things are going to shift somewhat.  But, it doesn't look like 
the real future is much more than:

1) Some mechanism for CBL/XBL single-IP DNSBLs to remain useful (eg: 
hardcoded /64 truncation or some mechanism like John's) for Internet 
query from small sites.

2) Zone download (Rsync or perhaps something better) becoming more 
prevalent.

3) DNSBL operators will be more conscious of query load and will more 
forcefully terminate abusers.

Perhaps we might do more work in (2), to specify zone formats for download.

While the tradeoff volumes for query versus zone downloads/incrementals 
may well shift, it will just about be never advantageous for small sites 
doing a few dozen emails per day to take a whole zone of something as 
big as the XBL.  Besides, in many cases, that introduces latency delays.

---

From: Steve Atkins <[email-masked]>
Date: Wed, 5 Jan 2011 17:07:04 -0800

On Jan 5, 2011, at 4:00 PM, Chris Lewis wrote:

> I must confess to a bit of confusion.  We seem to be talking about really naive IPv6 DNSBL implementations blowing out caches on high volume, and yet, at the same time, we acknowledge that the large DNSBL users already download zone files.

IPv6 is more likely to blow out caches than IPv4, I think. That's going to increase the load an end-site places on a central server quite significantly. And _that_ is going to mean that an IPv6 based list may end up with more "large" DNSBL users than current IPv4 based systems, as the same end-site might generate significantly more IPv6 queries than IPv4.

Or maybe it won't. Heck, I'm expecting there to be very little IPv6 mail, legitimate or otherwise, for quite a long time.

> Rsync is essentially the defacto standard for bulk DNSBL transfer, and as you say it's "not awful".  So, we don't seem to have a significant difficulty with that.

AIUI, rsync is reasonably expensive to offer (in terms of CPU load), and it adds significant latency to listing and delisting (as you say below). Do you have any numbers on system load vs users vs update rate conveniently to hand?

> I do know of one (commercial) blacklisting system that does use a more explicitly "incremental" distribution mechanism, but it's probably not that much better than rsync, and in fact it may be worse.

Google push a large chunk of their entire domain blacklist to every end-user browser (using a mix of incremental updates, broad blocks pushed to the browser, with queries to a central server for more specific checks when there's a broad block match, explicit invalidation of locally stored entries and so on). It's not the same problem being solved as IP based blacklist distribution, but it's something that gives some good ideas.

I don't think that looking at an alternate WAN distribution protocol is something we *have* to do, but if we're looking at changing behaviour at the MTA location anyway, now's a good time to think about whether we do want to do that or not, and what the tradeoffs might be. We're a research group, are we not? :)

> Even if we were to do something as simplistic as chop IPv6 queries at the /64, given that the number of spammers and bots doesn't magically go up simply because there's more bits to hide in, the caching problem appears to not that much worse than it already is with IPv4.

The number of spammers may not go up, but the number of addresses they use may. Or may not, it depends on their behaviour. But I'm not betting that it won't.

> IOW, some of the discussion threads here seem to be solutions looking for problems.
> 
> Clearly things are going to shift somewhat.  But, it doesn't look like the real future is much more than:
> 
> 1) Some mechanism for CBL/XBL single-IP DNSBLs to remain useful (eg: hardcoded /64 truncation or some mechanism like John's) for Internet query from small sites.

I think this bit is really tricky, and has lots of possibilities for over-thinking the problem. Blocking by /64 seems both a good idea, and probably good enough for small sites.

> 2) Zone download (Rsync or perhaps something better) becoming more prevalent.

Yup. I think zone download is going to become more prevalent (to the extent that I think even rsync is going to get too painful to offer in some cases, and something that looks more like IXFR is worth another look).

> 3) DNSBL operators will be more conscious of query load and will more forcefully terminate abusers.
> 
> While the tradeoff volumes for query versus zone downloads/incrementals may well shift, it will just about be never advantageous for small sites doing a few dozen emails per day to take a whole zone of something as big as the XBL.

No, it wouldn't. (Then again, there's no such thing as a site doing a few dozen emails a day, I don't believe. The spam alone is going to be a couple of orders of magnitude more than that :) ).

>  Besides, in many cases, that introduces latency delays.

It only introduces latency delays because rsync is not a good protocol for it - it's not dreadful, but it's not great.

Cheers,
  Steve

---

From: John Levine <[email-masked]>
Date: 6 Jan 2011 01:07:13 -0000

>appears to not that much worse than it already is with IPv4.

Everyone tells me that in IPv6 we're going to be using a lot of
whitelists.  I don't think it's a good idea to whitelist /64s, and
people are checking individual IPs and spammers change addresses which
IPv6 makes easy, you'll have load problems.

I agree that blacklists can probably do OK just by truncating to /64s,
but my thinking is that if everyone is going to change their filtering
software to handle IPv6, they'll do it by using some lookup library or
other, so this is the time to develop libraries that will work for all
the tasks they're likely to be used for.

R's,
John

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 7 Jan 2011 02:19:24 -0500

> AIUI, rsync is reasonably expensive to offer (in terms of CPU load), and it adds significant latency to listing and delisting (as you say below). Do you have any numbers on system load vs users vs…

AIUI, rsync really isn't that bad on large zone files (with one caveat, 
see below), unless compression is turned on (CPU), and even then, you 
can support several hundred downloaders (mixed compression and 
non-compression) on reasonably modest hardware.  The rsyncd can also 
force non-compression if deemed necessary.

Of course bandwidth can also be an issue.  I wouldn't want to do the 
above with anything less than a 20Mb/S Internet link...

Rsync behaves quite badly, however, on large zone files if you're 
updating random locations in the zone.  Eg: if you're doing time-based 
expiration, you damn well better sort the file in entry "first seen" 
timestamp order, so the bulk part of the adds and deletes are 
contiguous.  Don't change, say, existing TXT entries - once the entry 
goes in, it shouldn't change until it disappears.

The performance numbers I have are from a strangeish case, so, you'll 
want to get representative numbers from a more straightforward one. 
Like Spamhaus.

> I don't think that looking at an alternate WAN distribution protocol is something we *have* to do, but if we're looking at changing behaviour at the MTA location anyway, now's a good time to think…

No.  Other than, maybe, zone file formats.  This can encourage the 
development of common ancilliary tools to manipulate the zone files 
before internal publication.

For example, as a performance improvement, we download almost all of our 
DNSBLs and merge them into one zone, with locally assigned return codes 
for each of the originating DNSBLs.  One query, and you can get many A 
records back, each of which is uniquely identifiable (by the A value) as 
to which DNSBL it came from, so you can score them independently (which 
we do).

It would have been nice if the DNSBL file formats was better 
standardized.  The merge code is a bit smelly.  It'll work on (most) 
rbldnsd-compatible zones, but everything else gives the coder (me) 
heartburn ;-)

> The number of spammers may not go up, but the number of addresses they use may. Or may not, it depends on their behaviour. But I'm not betting that it won't.

Given the experience and impressions I've acquired - at least in the 
botspam space, IP hopping is actually quite rare in IPv4.  The cases 
I've seen are actually hard to tell whether they're due to load balanced 
multiple exit NATs, or real and explicit IP hopping by the bot.  As it's 
been explained to me, "real" IP hopping is usually just about impossible 
(on ISPs) with more modern hardware that enforces which IP you've 
authenticated to on _that_ wire, and you lose connectivity altogether if 
you change your IP (and this is even _within_ the subnet your local ISP 
switch is operating in).

If each IPv6 user is getting a, say, /64 for their allocation, but the 
ISP hardware capabilities don't change, they can easily change their IP 
within the /64, but not outside of it.  Hence, if we fix the DNSBL 
cutpoint at the same level, they can't hop far enough to get out from 
under the listing...

This doesn't necessitate that the cutpoint is 100% applicable, only that 
it's the majority.

Mind you, if IPv6 is so plentiful, snow-shoers will likely be able to 
get far more IP blocks than they can now, and get harder to block.

I'm really beginning to think that it's still quite premature to do much 
in this space.  We have to wait and see how the environment evolves 
before we can predict what will work.  And as you say, it's probably 
still quite some time off before it matters.

It's nice to see public experiments in IPv6 DNSBLs, but, I really have 
to wonder about the usefulness of it at the present time.  Of the 
hundreds of DNSBLs that exist, only a dozen or two block that much on 
their own.  Couple that with the dearth of IPv6 MTAs, I can't help 
thinking that such DNSBLs are essentially useless except as 
proof-of-concept.

> I think this bit is really tricky, and has lots of possibilities for over-thinking the problem. Blocking by /64 seems both a good idea, and probably good enough for small sites.

It strikes me that it may well be advantageous to pick an arbitrary 
cut-point and encourage even the medium to large sites to continue using 
it at that level.  You'd be building an additional (but small) factor of 
possible FPs into it, giving the hosters more incentive to clean up the 
bad IPs faster, because they may well have someone else yelling at them.

Consider the following thought experiment: if the XBL rounded off every 
entry to a /31 (in IPv4), every listing has a (actually quite small) 
possibility of blocking an innocent IP that someone will be impacted by. 
  I think providers would have a disproportionally higher incentive to 
fix the bad IP.  But the real FP impact would be extremely low.

> Yup. I think zone download is going to become more prevalent (to the extent that I think even rsync is going to get too painful to offer in some cases, and something that looks more like IXFR is wo…

I thought IXFR was abandoned because it was too expensive.  Or perhaps 
that support was pretty poor.  AXFR is obviously out of the question.

> No, it wouldn't. (Then again, there's no such thing as a site doing a few dozen emails a day, I don't believe.

Sure there are servers that small.  Mine - it does even less ;-)

> The spam alone is going to be a couple of orders of magnitude more than that :) ).

Spammers haven't found it yet.  Open relay probers have tho.

> It only introduces latency delays because rsync is not a good protocol for it

Rsync's batch-ish nature isn't quite why I mentioned latency.  The 
latency is zone rebuild time at the DNSBL server.  With a zone file as 
large as the XBL, say, even if you did some sort of incremental update 
directly off a core DBMS in realish-time, I think you'd run into 
performance problems.

I've been doing experiments aimed towards UDP-based DNSBL adds, and even 
that starts to get nasty at the levels I think major DNSBLs would see.

I know of systems that can get "bad IPs" from detectors thru to 
moderately large numbers of clients within a handful of seconds.  But I 
don't know how big the system is (clients nor magnitude of bad IPs), nor 
am I confident of its stability in a heterogenous client environment 
with multiple code authors.  These are commercial for-pay environments, 
who are obviously much smaller audiences than the good "free" ones.

---

From: John Levine <[email-masked]>
Date: 7 Jan 2011 22:08:50 -0000

>I thought IXFR was abandoned because it was too expensive.  Or
>perhaps that support was pretty poor.  AXFR is obviously out of the
>question.

Does anyone know if any DNS software other than BIND supports IXFR?

Supported or not, it's not very clever, just ships a list of NS
records to delete and a list to add, relative to the version of the
zone that the client says it has.  The server has to have (or be
prepared to generate) the diffs from all the stale versions of the
zone that a client might ask for.

If we know that a file is a list of IP addresses or a list of prefixes
and lengths, we can presumably design an efficient representation of
the differences.  I'm more interested in how we'd distribute it.  For
example, the oft-cited ClamAV distributes via a very large network of
HTTP mirrors.  That's not awful, but I wouldn't want to have to come
up with such a network for every BL in the world.

It seems to me that there's always going to be a point below which
it's better to query for individual answers, and above which it's
better to maintain a mirror, so whatever we do we should be prepared
for both.  I also think that it wouldn't be hard to come up with
something that caches a lot better than current BLs, which would move
the query/mirror line up considerably.

R's,
John

---

From: Franck Martin <[email-masked]>
Date: Sat, 8 Jan 2011 10:42:58 +1200 (FJT)

Seems GIT with puppet could be a better fit? Or some kind of revision control system...

Git allows you to share patches (between repositories), and puppet can make it automatic...

http://en.wikipedia.org/wiki/Git_%28software%29

Franck Martin 
http://www.avonsys.com/ 
http://www.facebook.com/Avonsys 
twitter: FranckMartin Avonsys 

Check your domain reputation: http://gurl.im/b69d4o 

----- Original Message -----
From: "John Levine" <[email-masked]>
To: [email-masked]
Sent: Saturday, 8 January, 2011 11:08:50 AM
Subject: Re: [Asrg] download approaches

R's,
John

_______________________________________________
Asrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/asrg

---

From: Steve Atkins <[email-masked]>
Date: Fri, 7 Jan 2011 14:59:39 -0800

> Does anyone know if any DNS software other than BIND supports IXFR?

I'm not even convinced that bind really *supports* it. :)

IXFR isn't useful unless you're maintaining your entire zone
in a transactional (database or dynamic update driven) way,
and that's not that common a case for a typical zone. (It is
a common case for a DNSBL, or could be quite easily.)

> zone that a client might ask for.

Yes. For any decently designed server that's going to be a *lot*
cheaper to do than rsync, especially for zones that change a lot.

There are some obvious tweaks you can do to make those deltas
more cacheable, but they'll be pretty easy to generate (and cache) on the fly, if
you're using a database backend, anyway (blacklist operators
using flat files and scripts to process their data, rather than something
transactional, are likely to be out of luck).

> up with such a network for every BL in the world.

There are a bunch of ways to do that, but there's one distinction
between them - that's access control. Distributing the list widely
and cheaply without access control is one problem. Distributing it
with access control, along a controlled distribution path is another.

I've not thought about the second case much, but there are some
ways to layer the first case on top of a (cacheable, CDN-able)
http distribution layer fairly efficiently, and that's a content distribution
platform that "everybody" already supports cheaply.

> something that caches a lot better than current BLs,

Yes, it would. I'm less convinced that it's easy to come up with
something that's both significantly better (in terms of query load and
implementation complexity tradeoffs) and also layered on top of a
simple key-value store such as DNS. Well worth a try, though.

> which would move

Yup. OTOH, a cheap, efficient way of running a local mirror would
move it in the other direction.

Cheers,
  Steve

---

From: Matthias Leisi <[email-masked]>
Date: Sat, 8 Jan 2011 22:00:48 +0100

On Fri, Jan 7, 2011 at 8:19 AM, Chris Lewis <[email-masked]> wrote:

> It's nice to see public experiments in IPv6 DNSBLs, but, I really have to
> wonder about the usefulness of it at the present time.  Of the hundreds of
> DNSBLs that exist, only a dozen or two block that much on their own.  Couple
> that with the dearth of IPv6 MTAs, I can't help thinking that such DNSBLs
> are essentially useless except as proof-of-concept.

Even having proof-of-concepts is valuable -- otherwise, everybody will
wait for everybody else. Chicken, meet Egg :)

There has been quite some discussion on whether (more-or-less-)
arbitrary limits should be chosen ("only allow /64" etc), and some
proposals have appeared.

I now implemented an extremely simple proposal (note that this is
experimental only and not useful for production purposes). The basic
idea is that the server will *always* return multiple answers (up to
3):

1) The listed entry "below" the one requested
2) The listed entry "above" the one requested
3) If available, a "matching" entry

If two of the above overlap, only one will be returned. In general,
always at least two RRs should be returned. Each of the responses will
have a netmask, and additional data (ie, trust levels, reputation
score etc).

With that information, a "intelligent" application / query library can
greatly reduce the number of lookups required (eg when a spammer
enumerates all IPs in a IPv6-/64). On the other hand, this makes it
mathematically feasible to enumerate a complete DNSxL zone -- but I do
not consider this as an issue, at least not for dnswl.org.

The experimental server supports TXT and SRV records with essentially
the same information. It is single-threaded and not very efficiently
coded. But please feel free to test it:

:~> dig @ns-exp1.dnswl.org -t txt 2.0.0.127.exp.dnswl.org
;; ANSWER SECTION:
2.0.0.127.exp.dnswl.org. 3600   IN      TXT     "32" "10" "0"
34.244.255.125.exp.dnswl.org. 3600 IN   TXT     "32" "8" "2"
254.254.254.127.exp.dnswl.org. 3600 IN  TXT     "32" "10" "0"

:~> dig @ns-exp1.dnswl.org -t srv 2.0.0.127.exp.dnswl.org
;; ANSWER SECTION:
2.0.0.127.exp.dnswl.org. 3600   IN      SRV     32 10 0 2.0.0.127.exp.dnswl.org.
2.0.0.127.exp.dnswl.org. 3600   IN      SRV     32 8 2
34.244.255.125.exp.dnswl.org.
2.0.0.127.exp.dnswl.org. 3600   IN      SRV     32 10 0
254.254.254.127.exp.dnswl.org.

The information for the two identical examples above is to be
interpreted as follows:

* Request for 127.0.0.2
* 127.0.0.2 has an exact match in the first of the three return RRs.
* 125.255.244.34/32 is the next entry "below", with category 8 and trustlevel 2
* 127.254.254.254/32 is the next entry "above" (a further test entry),
with cat 10 and trust 0

Another example:

:~> dig @ns-exp1.dnswl.org -t srv 254.5.210.128.exp.dnswl.org
254.5.210.128.exp.dnswl.org. 3600 IN    SRV     24 11 2
0.5.210.128.exp.dnswl.org.
254.5.210.128.exp.dnswl.org. 3600 IN    SRV     32 11 2
121.9.210.128.exp.dnswl.org.

* Request 128.210.5.254
* 128.210.5.254 is contained in the range 128.210.5.0/24, category 11, trust 2
* Next entry "above" is 128.210.9.121/32, cat 11, trust 2
* No next entry "below" is listed, because the range already serves
that purpose.

The list has two synthetical records to indicate the lower and upper bands:

:~> dig @ns-exp1.dnswl.org -t srv 0.0.0.0.exp.dnswl.org
:~> dig @ns-exp1.dnswl.org -t srv 255.255.255.255.exp.dnswl.org

Since I just hacked this together using existing Perl modules, the
data is only in IPv4, but the concept can easily be extended to IPv6
payload. This server will serve current dnswl.org data.

This approach has a number of benefits:

* No arbitrary cut-points
* Enables "intelligent" applications to avoid a lot of unnecessary requests
* Does require an application to walk down the tree from a root point
(and does not require DNSxL operators to artificially create such a
tree with synthesized records)

Some disadvantages:

* I don't know of a way to serve this off of a generic nameserver
(other than that nameserver being proxy to a "hidden master").
* DNSxL operators have less valuable logs because less specific
traffic hits their infrastructure.

I approached naturally from a whitelist point-of-view, so someone with
more experience in blacklists may have different approaches.

> Rsync's batch-ish nature isn't quite why I mentioned latency.  The latency
> is zone rebuild time at the DNSBL server.  With a zone file as large as the
> XBL, say, even if you did some sort of incremental update directly off a
> core DBMS in realish-time, I think you'd run into performance problems.

The latency is also in TTL. The lower the latency requirement, the
less is DNS the right tool for the job.

-- Matthias

---

From: John Levine <[email-masked]>
Date: 8 Jan 2011 21:26:18 -0000

>3) If available, a "matching" entry

Interesting approach, and as always, experiments are informative.

>greatly reduce the number of lookups required 

Well, that's the problem.  Some MTAs keep shared context for all
incoming connections, some (particularly on Unix-ish systems) don't.
Even on systems that share context, it's common to have a cluster of
servers each with their own context.  If you don't have the shared
context, this has triple the queries of a naive system.

That's why I designed my B-tree scheme so that the shared context is
in the DNS cache. Nearby lookups will fetch exactly the same records
and get good cache behavior, without having to remember anything in
the MTA.

With a system like yours, you could certainly posit a shared local
daemon that caches the info, but if you're going to do that, you might
as well use a daemon-to-server query system that gets 40 ranges per
query rather one that only gets two or three.

R's,
John

>(other than that nameserver being proxy to a "hidden master").

Oh. that's easy, you just have some offline process make up the zone
files and poke the server to load them.  It's conceptually no different
from putting files of CIDRs into rbldnsd.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 8 Jan 2011 19:16:33 -0500

Yes.

> context, this has triple the queries of a naive system.

Consider the situation with many larger server infrastructures.  Not 
only do you have multiple clusters of machines, but each machine may be 
using threads.  While there are ways to share server context, in the 
large, they can still be bottlenecks even on a single machine.

I can't help thinking that divide and conquer - as in, a highly 
efficient distinct cache (whether DNS or not) is better.

> from putting files of CIDRs into rbldnsd.

That's how my local rbldnsd infrastructure works.  Single query to a 
zone that's a merge of the DNSBLs we use.

Spfilter is old (now impractical) code that can do this.  I don't know 
of any free implementation of anything better - so I had to write my own.

---

From: Matthias Leisi <[email-masked]>
Date: Mon, 10 Jan 2011 14:16:19 +0100

> from putting files of CIDRs into rbldnsd.

No, it's very much different. It will always return the "nearest"
records, regardless of whether you have a match in the list data or
not (ie, querying for none-existing "a" will return "a-1" and "a+1" if
they exist).

-- Matthias

---

From: John Levine <[email-masked]>
Date: 10 Jan 2011 20:57:17 -0000

>not (ie, querying for none-existing "a" will return "a-1" and "a+1" if

It's not that hard.  The concept is that you precompute all of the
possible answers and put them in the zone.  To keep the zone size down
you use wildcards to cover ranges with the same answer.

On the other hand, if there's something I'm missing and this approach
really does need a stunt DNS server, how about making a b-tree out of
the ranges and serving it from a bog standard DNS server instead?

R's,
John