Subject: [Asrg] How well do DNSxL lookups cache ?

From: "John R. Levine" <[email-masked]>
Date: 4 Jan 2011 22:56:57 -0500

Over on the Spamassassin users' list, we've been having a spirited debate 
about the design of DNSBLs and WLs for IPv6, starting with my draft.

One of the key questions is how DNSxL lookups will play with DNS caches 
and servers, which of course depends on the pattern of IP addresses that 
send mail.  David Skoll collected a few stats from his servers, which 
suggested that for low volume mail servers, only about 20% of lookups are 
answered from the cache, and for a largish server handling 1M 
connections/day, it was only about 2/3.

That seemed hard to believe, so I took a week of logs from my mail server, 
did a similar analysis, and by golly, he's right.  The analysis takes 
timestamps and IP addresses, and computes how many DNSxL queries would be 
answered from a DNS cache under various scenarios.  Each group shows the 
number of cache hits and total queries by 24 hour period. (The first and 
last are short, since my logs didn't start and end at midnight.)  Then it 
shows the number of unique IPs queried, and a list showing cache 
effectiveness, with the number of times each cache entry was used, e.g., 
in the first dataset, there was one cache entry used 514 times, down to 
536,009 entries used only once.

To see how sensitive these answers were to DNSxL design, I tried it with a 
15 minute and 60 minute TTL, and I tried with /24 rather than 
/32 granularity.  Neither made much difference: the cache effectiveness 
rarely got above 20% and the vast majority of cache entries were never 
reused.

So now I'm scratching my head.  If these numbers are typical, I don't 
understand why DNSBL servers don't all fall over now.

My analyses use a small perl script which you can have if you have logs 
you'd like to try it on.  All it needs for input is a series of IP 
addresses and timestamps.  It extracts them from the peculiar log format 
that mailfront and daemontools produce, but adjusting the input format is 
easy.  I'll see if I can get some data from larger mail systems and see 
how different the answers are.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

--- 15 minute TTL ---
   7078 of  56237 ( 12 %) Sat Dec 25 00:00:00 2010
   6146 of  45213 ( 13 %) Sun Dec 26 00:00:00 2010
   9398 of  67167 ( 13 %) Mon Dec 27 00:00:00 2010
  12155 of  76827 ( 15 %) Tue Dec 28 00:00:00 2010
  10845 of  83165 ( 13 %) Wed Dec 29 00:00:00 2010
   8781 of  73894 ( 11 %) Thu Dec 30 00:00:00 2010
  10240 of  59080 ( 17 %) Fri Dec 31 00:00:00 2010
   7753 of  51736 ( 14 %) Sat Jan  1 00:00:00 2011
   7918 of  53362 ( 14 %) Sun Jan  2 00:00:00 2011
   8440 of  63016 ( 13 %) Mon Jan  3 00:00:00 2011
   9022 of  56342 ( 16 %) Tue Jan  4 00:00:00 2011
Total IPs = 358782
  hits  count
  514       1
  423       1
  420       1
  415       1
  403       1
  378       1
  373       1
  369       1
  368       1
  366       2
  309       1
  283       1
  280       1
  269       1
  267       1
  266       1
  265       1
  257       2
  251       1
  247       1
  246       1
  244       1
  239       1
  223       1
  220       1
  218       1
  208       1
  203       1
  199       1
  197       1
  195       1
  193       1
  181       2
  175       1
  173       1
  168       1
  166       1
  162       1
  160       1
  159       2
  157       1
  156       1
  149       1
  148       1
  145       1
  141       1
  135       1
  134       1
  133       2
  132       1
  131       1
  127       1
  124       3
  122       1
  120       1
  119       1
  118       1
  115       1
  113       1
  111       1
  110       1
  106       1
  105       1
  104       1
  103       1
  100      12
   99       4
   98       4
   97       2
   95       3
   94       1
   93       2
   92       2
   91       1
   90       1
   89       1
   87       1
   86       2
   81       1
   79       2
   77       1
   75       1
   73       1
   72       1
   68       4
   67       1
   66       2
   65       1
   64       1
   62       2
   61       1
   59       5
   58       4
   57       2
   56       5
   54       2
   53       3
   52       2
   51       3
   50       7
   48       1
   46       1
   45       3
   43       1
   42       3
   41       3
   40       1
   39       1
   38       2
   37       2
   36       3
   32       1
   31       4
   30       9
   29       9
   28       8
   27       6
   26       2
   25       5
   24      10
   23       6
   22       7
   21       5
   20      53
   19       9
   18       5
   17      10
   16      17
   15      19
   14      35
   13      26
   12      49
   11      60
   10      69
    9     113
    8     227
    7     185
    6     411
    5     762
    4    1918
    3    6599
    2   41434
    1  536009

--- 60 minute TTL ---
   9652 of  56237 ( 17 %) Sat Dec 25 00:00:00 2010
   8364 of  45213 ( 18 %) Sun Dec 26 00:00:00 2010
  13141 of  67167 ( 19 %) Mon Dec 27 00:00:00 2010
  15976 of  76827 ( 20 %) Tue Dec 28 00:00:00 2010
  15708 of  83165 ( 18 %) Wed Dec 29 00:00:00 2010
  12886 of  73894 ( 17 %) Thu Dec 30 00:00:00 2010
  13266 of  59080 ( 22 %) Fri Dec 31 00:00:00 2010
  10442 of  51736 ( 20 %) Sat Jan  1 00:00:00 2011
  10391 of  53362 ( 19 %) Sun Jan  2 00:00:00 2011
  11559 of  63016 ( 18 %) Mon Jan  3 00:00:00 2011
  11876 of  56342 ( 21 %) Tue Jan  4 00:00:00 2011
Total IPs = 358782
  hits  count
  514       1
  423       1
  420       1
  415       1
  411       1
  403       1
  378       1
  369       1
  368       1
  366       2
  309       1
  283       1
  280       1
  270       1
  269       2
  267       1
  266       1
  265       1
  259       1
  257       2
  251       1
  248       1
  246       1
  244       1
  239       1
  223       1
  220       1
  219       1
  208       1
  203       1
  199       1
  197       1
  195       1
  193       1
  186       1
  181       2
  175       1
  173       1
  168       1
  162       1
  160       1
  159       2
  157       1
  154       1
  149       1
  148       1
  145       1
  141       1
  135       1
  134       1
  133       2
  132       1
  131       1
  124       3
  122       1
  120       1
  119       1
  117       1
  115       1
  111       1
  110       1
  106       1
  104       1
  103       1
  100      12
   99       4
   98       4
   97       2
   95       2
   94       1
   93       1
   92       2
   91       1
   90       1
   89       1
   87       1
   86       2
   85       1
   83       1
   81       1
   79       2
   77       1
   75       1
   74       1
   73       1
   72       1
   69       1
   68       3
   67       1
   66       2
   65       2
   64       1
   62       2
   61       2
   59       2
   58       4
   57       2
   56       6
   54       2
   53       2
   52       2
   51       3
   50       7
   48       1
   47       1
   46       2
   45       4
   43       1
   42       3
   41       2
   40       2
   39       2
   38       1
   37       3
   36       4
   35       2
   34       1
   33       2
   32       2
   31       2
   30       8
   29      10
   28       8
   27       3
   26       3
   25       6
   24       9
   23       8
   22       9
   21      12
   20      60
   19       8
   18      13
   17      19
   16      20
   15      31
   14      42
   13      37
   12      58
   11      65
   10      89
    9     149
    8     302
    7     311
    6     656
    5    1367
    4    3209
    3   10400
    2   58038
    1  477633

--- 15 minute TTL, combine /24s ---
   7828 of  56237 ( 13 %) Sat Dec 25 00:00:00 2010
   6837 of  45213 ( 15 %) Sun Dec 26 00:00:00 2010
  10614 of  67167 ( 15 %) Mon Dec 27 00:00:00 2010
  13518 of  76827 ( 17 %) Tue Dec 28 00:00:00 2010
  12430 of  83165 ( 14 %) Wed Dec 29 00:00:00 2010
  10142 of  73894 ( 13 %) Thu Dec 30 00:00:00 2010
  11045 of  59080 ( 18 %) Fri Dec 31 00:00:00 2010
   8433 of  51736 ( 16 %) Sat Jan  1 00:00:00 2011
   8712 of  53362 ( 16 %) Sun Jan  2 00:00:00 2011
   9342 of  63016 ( 14 %) Mon Jan  3 00:00:00 2011
   9830 of  56342 ( 17 %) Tue Jan  4 00:00:00 2011
Total IPs = 186854
  hits  count
  514       1
  423       1
  420       1
  415       1
  403       1
  378       1
  373       1
  369       1
  368       1
  366       2
  309       1
  283       1
  280       1
  269       1
  267       1
  266       1
  265       1
  257       2
  251       1
  247       1
  246       1
  244       1
  239       1
  223       1
  220       1
  218       1
  208       1
  203       1
  199       1
  197       1
  195       1
  193       1
  181       2
  175       1
  173       1
  168       1
  166       1
  162       1
  160       1
  159       2
  157       1
  156       1
  149       1
  148       1
  145       1
  141       1
  135       1
  134       1
  133       2
  132       1
  131       1
  127       1
  124       3
  122       1
  120       1
  119       1
  118       1
  115       1
  113       1
  111       1
  110       1
  106       1
  105       1
  104       1
  103       1
  100      12
   99       4
   98       4
   97       2
   95       3
   94       1
   93       2
   92       3
   90       1
   89       1
   87       1
   86       2
   81       1
   79       2
   77       1
   75       1
   73       1
   72       1
   68       4
   67       1
   66       2
   65       1
   64       1
   62       2
   61       1
   59       5
   58       4
   57       2
   56       5
   54       2
   53       4
   52       1
   51       3
   50       7
   48       1
   46       1
   45       3
   43       1
   42       4
   41       3
   40       1
   39       1
   38       2
   37       2
   36       3
   35       1
   32       3
   31       3
   30       9
   29       9
   28      10
   27       6
   26       5
   25       3
   24      10
   23       7
   22       8
   21       7
   20      55
   19      12
   18      13
   17      17
   16      21
   15      27
   14      43
   13      35
   12      66
   11      86
   10      93
    9     165
    8     288
    7     264
    6     549
    5     954
    4    2253
    3    7508
    2   45131
    1  519465

--- 60 minute TTL, combine /24s ---
  11210 of  56237 ( 19 %) Sat Dec 25 00:00:00 2010
   9595 of  45213 ( 21 %) Sun Dec 26 00:00:00 2010
  15383 of  67167 ( 22 %) Mon Dec 27 00:00:00 2010
  18675 of  76827 ( 24 %) Tue Dec 28 00:00:00 2010
  18714 of  83165 ( 22 %) Wed Dec 29 00:00:00 2010
  15380 of  73894 ( 20 %) Thu Dec 30 00:00:00 2010
  14832 of  59080 ( 25 %) Fri Dec 31 00:00:00 2010
  11827 of  51736 ( 22 %) Sat Jan  1 00:00:00 2011
  11987 of  53362 ( 22 %) Sun Jan  2 00:00:00 2011
  13420 of  63016 ( 21 %) Mon Jan  3 00:00:00 2011
  13543 of  56342 ( 24 %) Tue Jan  4 00:00:00 2011
Total IPs = 186854
  hits  count
  514       1
  423       1
  420       1
  415       1
  411       1
  403       1
  378       1
  370       1
  368       1
  366       2
  309       1
  283       1
  280       1
  270       1
  269       2
  267       1
  266       1
  265       1
  259       1
  257       2
  251       1
  248       1
  246       1
  244       1
  239       1
  223       1
  220       1
  219       1
  208       1
  203       1
  199       1
  197       1
  195       1
  193       1
  186       1
  181       2
  175       1
  173       1
  168       1
  162       1
  160       1
  159       2
  157       1
  154       1
  149       1
  148       1
  145       1
  141       1
  135       1
  134       1
  133       2
  132       1
  131       1
  125       1
  124       3
  121       1
  120       1
  119       1
  115       1
  111       1
  110       1
  106       1
  104       1
  103       1
  100      12
   99       4
   98       4
   97       2
   95       2
   94       1
   93       1
   92       3
   90       1
   89       1
   87       2
   86       2
   85       1
   81       1
   79       2
   77       1
   75       1
   74       1
   73       1
   72       1
   69       1
   68       3
   67       2
   66       2
   65       2
   64       1
   62       2
   61       2
   59       2
   58       4
   57       2
   56       5
   54       3
   53       3
   52       1
   51       3
   50       7
   48       2
   47       1
   46       3
   45       4
   44       1
   43       1
   42       4
   41       3
   40       6
   39       2
   38       3
   37       4
   36       6
   35       5
   34       2
   33       1
   32       5
   31       7
   30      10
   29      19
   28       9
   27      11
   26      10
   25       9
   24      12
   23      13
   22      17
   21      19
   20      70
   19      16
   18      24
   17      36
   16      40
   15      47
   14      63
   13      62
   12      79
   11      84
   10     125
    9     203
    8     346
    7     394
    6     859
    5    1666
    4    4022
    3   12676
    2   64013
    1  446298

---

From: Steve Atkins <[email-masked]>
Date: Tue, 4 Jan 2011 20:06:16 -0800

On Jan 4, 2011, at 7:56 PM, John R. Levine wrote:

> Over on the Spamassassin users' list, we've been having a spirited debate about the design of DNSBLs and WLs for IPv6, starting with my draft.
> 
> One of the key questions is how DNSxL lookups will play with DNS caches and servers, which of course depends on the pattern of IP addresses that send mail.  David Skoll collected a few stats from his servers, which suggested that for low volume mail servers, only about 20% of lookups are answered from the cache, and for a largish server handling 1M connections/day, it was only about 2/3.
> 
> That seemed hard to believe, so I took a week of logs from my mail server, did a similar analysis, and by golly, he's right.  The analysis takes timestamps and IP addresses, and computes how many DNSxL queries would be answered from a DNS cache under various scenarios.  Each group shows the number of cache hits and total queries by 24 hour period. (The first and last are short, since my logs didn't start and end at midnight.)  Then it shows the number of unique IPs queried, and a list showing cache effectiveness, with the number of times each cache entry was used, e.g., in the first dataset, there was one cache entry used 514 times, down to 536,009 entries used only once.
> 
> To see how sensitive these answers were to DNSxL design, I tried it with a 15 minute and 60 minute TTL, and I tried with /24 rather than /32 granularity.  Neither made much difference: the cache effectiveness rarely got above 20% and the vast majority of cache entries were never reused.

Assuming an infinite cache, that'll get much better as the traffic gets higher, but it'll still be pretty ugly.

> So now I'm scratching my head.  If these numbers are typical, I don't understand why DNSBL servers don't all fall over now.

Because high-volume DNSBL consumers don't use them.

They use a local copy of the database, pushed out to them (typically via rsync, which is not ideal, but not *dreadful*) and they only use DNS as a LAN protocol to communicate between their MTA and their local authoritative DNS server, dedicated to serving DNSBL data.

Moving away from running DNS for DNSBLs across the public internet has been an obvious move for a long time. IPv6 does move us further in that direction.

DNS is still a perfectly good local protocol, though.

Cheers,
  Steve