Subject: [Asrg] RFC 6471 and "listing the Internet" as a punishment

From: Martijn Grooten <[email-masked]>
Date: Tue, 24 Jan 2012 15:07:50 +0000

It was nice to see the RFC being published. Good work.

Then I came across this:

http://blog.vamsoft.com/2012/01/24/ub-black-uribl-com-url-blacklist-started-to-block-everything/

(Vamsoft ORF is a spam-filter.) Basically uribl.com was returning 127.0.0.1 to _all_ queries from nameservers that are sending high volumes (presumably without paying for it) as some kind of punishment. http://uribl.com/ confirms that.

Now, as Vamsoft mentions, it is not a good idea to use third-party nameservers on a server you're making DNS requests from. (Although, unlike openDNS, Google's nameservers do return NXDOMAIN when they can't resolve a domain.) Moreover, it does seem Google's nameservers are now getting REFUSED as a response to any uribl.com request. I was just wondering whether the RFC says anything about this kind of behaviour ('listing' everything as a punishment). To my reading it doesn't.

Martijn.

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: "Emanuele Balla (aka Skull)" <[email-masked]>
Date: Tue, 24 Jan 2012 16:19:00 +0100

On 1/24/12 4:07 PM, Martijn Grooten wrote:
>  (Vamsoft ORF is a spam-filter.) Basically uribl.com was returning
> 127.0.0.1 to _all_ queries from nameservers that are sending high
> volumes (presumably without paying for it) as some kind of
> punishment. http://uribl.com/ confirms that.
> 
> Now, as Vamsoft mentions, it is not a good idea to use third-party
> nameservers on a server you're making DNS requests from. (Although,
> unlike openDNS, Google's nameservers do return NXDOMAIN when they
> can't resolve a domain.) Moreover, it does seem Google's nameservers
> are now getting REFUSED as a response to any uribl.com request. I was
> just wondering whether the RFC says anything about this kind of
> behaviour ('listing' everything as a punishment). To my reading it
> doesn't.
> 
> Martijn.

In truth there's
in point 3.3:

«  Note: In Section 3.4, it is noted that some DNSBLs have shut down in
   such a way to list all of the Internet.  Further, in Section 3.5,
   DNSBL operators MUST NOT list 127.0.0.1.  Therefore, a positive
   listing for 127.0.0.1 SHOULD indicate that the DNSBL has started
   listing the world and is non-functional. »

and again, in point 3.5:

«  A functioning DNSBL MUST NOT list 127.0.0.1.  There are a number of
   mail server implementations that do not cope with this well, and many
   will use a positive response for 127.0.0.1 as an indication that the
   DNSBL is shut down and listing the entire Internet.»

That is not clearly against "listing everything as a punishment", but
means uribl.com is technically "non-functional"... ;-)

-- 
Paranoia is a disease unto itself. And may I add: the person standing
next to you may not be who they appear to be, so take precaution.
-----------------------------------------------------------------------------
http://bofhskull.wordpress.com/

---

From: David Romerstein <[email-masked]>
Date: Tue, 24 Jan 2012 10:28:13 -0500

On 1/24/12 10:07 AM, Martijn Grooten wrote:
> I was just wondering whether the RFC says anything about this kind of behaviour ('listing' everything as a punishment). To my reading it doesn't.

Only thing in the RFC close to this is 3.4, regarding shutting down 
gracefully (MUST NOT list the entire Internet).

Listing the world for folks overloading your system is unlikely to have 
the effect that you want, and is most likely going to impact folks who 
have no say in the configuration of the receiving mail server.

-- D, who was just investigating a spate of URIBL-related blocks 
yesterday...

---

From: David Romerstein <[email-masked]>
Date: Tue, 24 Jan 2012 10:29:30 -0500

>     DNSBL is shut down and listing the entire Internet.»

"Listing 127.0.0.1" is not the same as "returning 127.0.0.1".

-- D

---

From: "Emanuele Balla (aka Skull)" <[email-masked]>
Date: Tue, 24 Jan 2012 16:30:36 +0100

> "Listing 127.0.0.1" is not the same as "returning 127.0.0.1".

Damm. FAIL. :-D

---

From: Rich Kulawiec <[email-masked]>
Date: Tue, 24 Jan 2012 10:35:33 -0500

On Tue, Jan 24, 2012 at 04:19:00PM +0100, Emanuele Balla (aka Skull) wrote:
> and again, in point 3.5:
> 
> ?  A functioning DNSBL MUST NOT list 127.0.0.1.  There are a number of
>    DNSBL is shut down and listing the entire Internet.?
> 

I may well be misreading this report AND the RFC (coffee level: alarmingly
low) but it appears to me that this DNSBL is not listing "127.0.0.1",
but is returning 127.0.0.1 in response to queries for "example.com"
(and all other domains) when those queries are issued from certain hosts.
(And I presume those hosts are the set which have issued excessive
and/or unwelcome queries in the opinions of the DNSBL's operators.)

---rsk

---

From: "Emanuele Balla (aka Skull)" <[email-masked]>
Date: Tue, 24 Jan 2012 16:52:42 +0100

> and/or unwelcome queries in the opinions of the DNSBL's operators.)

You (and David) are absolutely right: I simply messed up the whole
point. ;-)

The only point that directly applies, here, is IMHO 3.3:

«
3.3.  DNSBLs SHOULD Provide Operational Flags

   Most IP address-based DNSBLs follow a convention of query entries for
   IP addresses in 127.0.0.0/8 (127.0.0.0-127.255.255.255) to provide
   online indication of whether the DNSBL is operational.  Many, if not
   most, DNSBLs arrange to have a query of 127.0.0.2 return an A record
   (usually 127.0.0.2) indicating that the IP address is listed.  This
   appears to be a de facto standard indicating that the DNSBL is
   operating correctly.  See [RFC5782] for more details on DNSBL test
   entries.

   If this indicator is missing (query of 127.0.0.2 returns NXDOMAIN),
   or any query returns an A record outside of 127.0.0.0/8, the DNSBL
   should be considered non-functional.
»

Somehow, this seems to allow URIBL to do what it does: they state that
returning 127.0.0.1 means you're an undesired client and you have the
chance to trap that flag (with the right software, at least)...

On the other side, there's point 3.4:

«
   The DNSBL operator MUST issue impending shutdown warnings (on the
   DNSBL web site, appropriate mailing lists, newsgroups, vendor
   newsletters, etc.), and indicate that the DNSBL is inoperative using
   the signaling given in Section 3.3.
[...]

   The shutdown procedure should have the following properties:

   1.  MUST NOT list the entire Internet
»

One could argue they're not shutting down (just listing the Internet
:-D) so the point does not apply, but IMHO the underlying concept is.

After all, they're trying to force the offending customers to change
their configurations, just as during shutdowns...

---

From: [email-masked]
Date: Tue, 24 Jan 2012 13:23:49 -0500

As I tried to say in the past, having a value to return for all
queries from a DNS server that has been deemed abusive is *useful* to
black/whitelist providers.  Enough that it's looking like it'll be done
whether the ASRG likes it or not.  If you'd prefer something other than
127.0.0.1 to be used, document it somewhere.

Also, as the linked article said, "...the 127.0.0.1 response indicates
that uribl.com does not accept any queries from the DNS server".
SpamAssassin had this handled as URIBL defined, no false positives
resulted.

-- 
"Go forth, and be excellent to one another." - http://www.jhuger.com/fredski.php
http://www.ChaosReigns.com

---

From: David Romerstein <[email-masked]>
Date: Tue, 24 Jan 2012 13:33:17 -0500

> 127.0.0.1 to be used, document it somewhere.

Following with RFC 6471, would it be possible to do a split zone 
(abusive/non-abusive), sending abusive IPs to do their loopups from IP 
addresses in TEST-NET RFC-5735 addresses?

-- D

---

From: "John Levine" <[email-masked]>
Date: 24 Jan 2012 18:46:19 -0000

>addresses in TEST-NET RFC-5735 addresses?

In principle, although it's a little tricky since the setup is usually
something like this:

(at TLD)
example.net NS foo.example.net <-- main name server that does split horizon

(at foo.example.net)
dnsbl.example.net NS rbldnsd.example.net <-- the server that's getting hammered

So the server that's getting overloaded has to tell the server above
what lies to tell.

R's,
John

---

From: Dave Warren <[email-masked]>
Date: Tue, 24 Jan 2012 11:44:17 -0800

On 1/24/2012 7:28 AM, David Romerstein wrote:
> Listing the world for folks overloading your system is unlikely to 
> have the effect that you want, and is most likely going to impact 
> folks who have no say in the configuration of the receiving mail server.

It will, however, impact folks who know how to get in touch with whoever 
does have a say in the configuration of said mail server.

It's not spectacularly ideal, but lacking in some way to tell the 
software to reconfigure itself, it's sometimes the only way to get 
attention.

-- 
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

---

From: "Emanuele Balla (aka Skull)" <[email-masked]>
Date: Tue, 24 Jan 2012 20:50:59 +0100

> 127.0.0.1 to be used, document it somewhere.

I fully agree with you, FWIW...

> SpamAssassin had this handled as URIBL defined, no false positives

Yes, and somehow that's the point: SW (like spamassassin) that deal with
return values correctly, will not encounter FPs but this means it also
gives the BL operator no advantage.

While any other return value outside 127/8, while more opportune,
probably will affect bad implementations like 127.0.0.1 or any other code.

IMHO there's no winner here...

---

From: Derek Diget <[email-masked]>
Date: Tue, 24 Jan 2012 16:09:06 -0500 (EST)

On Jan 24, 2012 at 20:50 +0100, Emanuele Balla (aka Skull) wrote:
=>On 1/24/12 7:23 PM, [email-masked] wrote:
=>> As I tried to say in the past, having a value to return for all
=>> queries from a DNS server that has been deemed abusive is *useful* to
=>> black/whitelist providers.  Enough that it's looking like it'll be done
=>> whether the ASRG likes it or not.  If you'd prefer something other than
=>> 127.0.0.1 to be used, document it somewhere.
=>
=>I fully agree with you, FWIW...
=>
=>
=>> Also, as the linked article said, "...the 127.0.0.1 response indicates
=>> that uribl.com does not accept any queries from the DNS server".
=>> SpamAssassin had this handled as URIBL defined, no false positives
=>> resulted.
=>
=>Yes, and somehow that's the point: SW (like spamassassin) that deal with
=>return values correctly, will not encounter FPs but this means it also
=>gives the BL operator no advantage.
=>
=>While any other return value outside 127/8, while more opportune,
=>probably will affect bad implementations like 127.0.0.1 or any other code.

ASRG List,

See the SpamAssassin bugs ids related to this issue from just a few 
weeks ago.

Bug 6724 - DNSxL returning purposefully wrong answers as part of 
Anti-Abuse / Free for Some Policies 
<https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6724>

Bug 6728 - DNSBLs need a way to turn off queries based on BLOCKED rules 
triggering <https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6728>

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************

---

From: Chris Lewis <[email-masked]>
Date: Tue, 24 Jan 2012 16:42:05 -0500

> 127.0.0.1 to be used, document it somewhere.

As I've said more than once, there are two methods of doing it 
documented already that are preferable to listing the world:

- return TEST NET DNS server addresses, OR
- return something outside of 127.0.0.0/8

The first one has the advantage of doing something useful EVEN IF the 
DNSBL client has never heard of RFC6471.

---

From: SM <[email-masked]>
Date: Wed, 25 Jan 2012 10:37:52 -0800

>punishment. http://uribl.com/ confirms that.

   "After investigating this further, it seems the affected ORF users
    all use Google public DNS servers for the queries (or use such servers
    as forwarders in their local DNS configuration)."

Anyone using open recursive DNS servers or their ISP's DNS server for 
DNSBL queries is asking for trouble.  The listing is to get the 
attention of the sender.  It's "antisocial".

Regards,
-sm

---

From: Dave Warren <[email-masked]>
Date: Wed, 25 Jan 2012 11:23:24 -0800

> attention of the sender.  It's "antisocial".

I have to wonder, if a DNSBL were being operated entirely on a free 
basis (rather than a freemium model), might it not be better to take 
advantage of the large caches out there rather than having thousands of 
individual servers performing the same mundane set of lookups individually?

Obviously this negates the DNSBL's ability to try and pull cash out of 
the larger entities, but purely from a resource management point of 
view, if someone wants to offer a front-line cache for free, surely that 
should reduce load.

---

From: SM <[email-masked]>
Date: Wed, 25 Jan 2012 12:11:21 -0800

At 11:23 25-01-2012, Dave Warren wrote:
>advantage of the large caches out there rather than having thousands 
>of individual servers performing the same mundane set of lookups individually?

There are no large caches out there.  That's not how DNS works in my opinion.

>Obviously this negates the DNSBL's ability to try and pull cash out 
>of the larger entities, but purely from a resource management point 
>of view, if someone wants to offer a front-line cache for free, 
>surely that should reduce load.

It is expensive to provide such a cache for free.  Why would someone 
want to offer such a service for free?

Regards,
-sm

---

From: Douglas Otis <[email-masked]>
Date: Wed, 25 Jan 2012 13:12:09 -0800

> There are no large caches out there.  That's not how DNS works in my 
Dear SM,

Many providers offer non-authoritative DNS servers.  SOPA and PIPA bills 
attempt to regulate which domains are to be excluded, and ignore 
authoritative DNS traffic not cached by providers.  Of course, only 
accepting results from authoritative servers would reduce performance.  
Infrequently requested records might be dropped prior to expiry to 
liberate cache.
> want to offer such a service for free?
Use of these caching services generate valuable information and reduce 
network traffic between peers.

Regards,
Doug Otis

---

From: SM <[email-masked]>
Date: Wed, 25 Jan 2012 14:57:47 -0800

Hi Doug,
At 13:12 25-01-2012, Douglas Otis wrote:
>Many providers offer non-authoritative DNS servers.  SOPA and PIPA 
>bills attempt to regulate which domains are to be excluded, and 
>ignore authoritative DNS traffic not

I do not have any opinion about SOPA and PIPA bills.

BTW, do you have an opinion about Pippa? :-)

>Use of these caching services generate valuable information and 
>reduce network traffic between peers.

Could you elaborate on the "generate valuable information"?

Are there any examples of DNS "caching services" set up to reduce 
network traffic and why it is being done?

Regards,
-sm

---

From: Paul Smith <[email-masked]>
Date: Wed, 25 Jan 2012 23:26:15 +0000

> network traffic and why it is being done?

Hang on - isn't that the whole point of DNS caching? If people didn't 
care about network traffic, no one would cache anything, they'd just 
resolve the addresses every time. Apart from increased network traffic, 
resolving every time would be much better, as there'd be no stale data.

So, pretty much every caching DNS server is there to reduce network 
traffic. Isn't it?

---

From: Brendan Hide <[email-masked]>
Date: Thu, 26 Jan 2012 02:01:46 +0200

On 01/26/12 01:26, Paul Smith wrote:
> resolve the addresses every time. Apart from increased network 
> traffic, resolving every time would be much better, as there'd be no 
> stale data.
>
Pretty much every ISP has caching DNS servers *partly* for this exact 
reason, yes.

The other reason is speed. Cached entries result in less latency when 
the authoritative DNS server is on the other side of the world. :)

-- 
__________
Brendan Hide
Mobile: +27 83 448 3867
Mobile: [email-masked] (plain text only please)
Web Africa - Internet Business Solutions
http://www.webafrica.co.za/?AFF1E97

---

From: "John Levine" <[email-masked]>
Date: 26 Jan 2012 00:02:55 -0000

>want to offer such a service for free?

Perhaps you might ask Google why they run 8.8.8.8 and 8.8.4.4.

I run a free DNSBL, korea.services.net.  The public caches have
never been a problem.  The people I have to block hammer on it
directly from their own clients.  The query streams suggest that
they aren't caches, since they tend to ask the same questions
over and over.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 26 Jan 2012 00:03:40 -0000

>So, pretty much every caching DNS server is there to reduce network 

Except for the "pretty much" part, yes.

R's,
John

---

From: Steve Atkins <[email-masked]>
Date: Wed, 25 Jan 2012 16:08:57 -0800

On Jan 25, 2012, at 3:26 PM, Paul Smith wrote:

>> Are there any examples of DNS "caching services" set up to reduce network traffic and why it is being done?
> 
> Hang on - isn't that the whole point of DNS caching? If people didn't care about network traffic, no one would cache anything, they'd just resolve the addresses every time. Apart from increased network traffic, resolving every time would be much better, as there'd be no stale data.
> 
> So, pretty much every caching DNS server is there to reduce network traffic. Isn't it?

Partly, yes. It also reduces latency significantly (helped by very high cache hit rates near the root of the tree).

(Queries to DNSBLs and similar trees - e.g. in-addr.arpa - do damage that somewhat, by creating a large number of different queries few of which are reused, hence tending to evict higher value records from the cache. But that's orthogonal to what we're discussing here, really.)

Cheers,
  Steve

---

From: SM <[email-masked]>
Date: Wed, 25 Jan 2012 16:33:21 -0800

>Perhaps you might ask Google why they run 8.8.8.8 and 8.8.4.4.

Vertical integration? :-)

Regards,
-sm

---

From: Douglas Otis <[email-masked]>
Date: Thu, 26 Jan 2012 10:27:10 -0800

Dear SM,

To support touted performance feature, chrome aggressively resolves 
links in advance.  This strategy makes chrome sensitive to DNS 
performance.  Cache in their recursive resolvers is kept current ahead 
of requests.  People addicted to speed.  ;^)

Regards,
Doug Otis

---

From: Dave Warren <[email-masked]>
Date: Fri, 27 Jan 2012 16:51:32 -0800

> of requests.  People addicted to speed.  ;^)

I might be being argumentative for it's own sake (sorry), but doesn't 
this make Chrome less sensitive to DNS performance rather than more 
sensitive?

With a browser that doesn't pre-cache DNS lookups, the user is aware of 
DNS latency every time they click a link and with every resource that 
the browser loads from a different domain. Conversely, with caching, in 
most cases pre-caching every link on a page will take longer than it 
takes the user to find a link and click on it even if the DNS cache is 
extremely slow.

Now I'd agree that faster DNS servers makes a noticeable difference when 
browsing since many websites load content from a dozen or more hostname, 
but I'd argue that Chrome's precaching makes it less sensitive to slow 
DNS queries.

---

From: Dave Warren <[email-masked]>
Date: Fri, 27 Jan 2012 16:51:31 -0800

(I'm a bit late getting back to this, my apologies)

> (Queries to DNSBLs and similar trees - e.g. in-addr.arpa - do damage that somewhat, by creating a large number of different queries few of which are reused, hence tending to evict higher value reco…

And, one might argue, an artifact of poor cache expiry policies. At 
least in my version of an ideal world, I'd want to keep records in the 
cache based on frequency of use over nearly anything else (within the 
TTL lifetime, of course)

However, in the context of DNSBLs, you may well have the same problem as 
in-addr.arpa in that there are a lot of records that will have limited 
cache re-use. Still, if a DNSBL is overloaded, increasing TTLs and 
encouraging (rather than discouraging or prohibiting) use of public 
caches would probably decrease load on the DNSBL servers.

For example, with a DNSBL negative-caching at, say, 150 seconds, my 
servers check Gmail's outbound IPs for DNSBL listings, on average, every 
180 seconds or so. Were I and 10 of my best friends running similarly 
sized mail servers to start querying 8.8.8.8 instead of using our own 
internal resolvers, a DNSBL might see one hit every 150 seconds instead 
of 1 every 18 seconds (10 every 180 seconds).

Now that being said, as a matter of practice I wouldn't suggest we start 
suggesting mail server operators start using Google's public DNS as 
their primary DNS. However, the reality of it is that the majority of 
people hit by "listing the Internet for over-quota usage" policies were 
using shared (or public) DNS resolvers, most weren't actually hitting 
any sort of limit due to their own traffic.

Obviously if a DNSBL keeps their TTLs (positive and negative) too low 
then aggregating queries does little good, and there does need to be 
some level of responsiveness. However, if a DNSBL recommends a hourly or 
daily rsync for rsync users, that might suggest a starting point for TTLs.

At the end of the day though, it's not about stopping abuse or people 
hammering the DNSBLs, but rather, it's about making it more convenient 
for larger players to pay money for a valuable service. That's not 
really unfair, and the freemium model is always a complicated one with 
potential holes for abuse, but it's disingenuous to declare that 
listing-the-internet is the only way to cut down query volume.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 31 Jan 2012 13:50:01 -0800

On 1/27/12 4:51 PM, Dave Warren wrote:
>> performance.  Cache in their recursive resolvers is kept current 
>> ahead of requests.  People addicted to speed.  ;^)
>
> With a browser that doesn't pre-cache DNS lookups, the user is aware 
> of DNS latency every time they click a link and with every resource 
> that the browser loads from a different domain. Conversely, with 
> caching, in most cases pre-caching every link on a page will take 
> longer than it takes the user to find a link and click on it even if 
> the DNS cache is extremely slow.
>
> Now I'd agree that faster DNS servers makes a noticeable difference 
> when browsing since many websites load content from a dozen or more 
> hostname, but I'd argue that Chrome's precaching makes it less 
> sensitive to slow DNS queries.
Dear Dave,

When DNS transactions attempt to include all possible choices, necessary 
transactions are at a greater risk of being queued, rather than being 
ready in advance.  Chrome offers a switch setting to disable this 
behavior that may actually result in reduced performance.

Regards,
Doug Otis