Subject: [ietf-dkim] Interesting recent statistics

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 8 Feb 2011 11:31:14 -0800

A few statistics OpenDKIM captured recently that might amuse some of you:

1) There's a slow but steady increase in signed message rates: http://www.opendkim.org/stats/report.html#signing_trend (ignore the first row, that month was skewed; also the current month is, of course, incomplete)

2) "To:" is now far and away the most munged header field invalidating signatures: http://www.opendkim.org/stats/report.html#broken_headers

3) Overall pass rates are pretty high, even when they transit MLMs: http://www.opendkim.org/stats/report.html#passfail

4) Oddly, signed messages with huge numbers of Received: header fields almost always pass verification, including one with 39 of them: http://www.opendkim.org/stats/report.html#received_correlation (let's see SPF do that!), though that message was ultimately rejected for having too many Received: fields.

-MSK

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Tue, 08 Feb 2011 23:53:17 +0100

Hi, Murray,

> A few statistics OpenDKIM captured recently that might amuse some of you:

Interesting figures!

> 1) There's a slow but steady increase in signed message rates: 
> http://www.opendkim.org/stats/report.html#signing_trend (ignore the 
> first row, that month was skewed; also the current month is, of 
> course, incomplete)
>

Good news! Although... we don't know whether the increase is due to 
spammers that adopt DKIM or legit senders adopting DKIM.

> 2) "To:" is now far and away the most munged header field invalidating 
> signatures: http://www.opendkim.org/stats/report.html#broken_headers
>
> 3) Overall pass rates are pretty high, even when they transit MLMs: 
> http://www.opendkim.org/stats/report.html#passfail
>

What I don't understand here, what's what. I see four figures:

Signatures: 1362786 Ignored: 482 Pass: 1275158 Failed (body): 16027

But this row does not provide the correct pass/fail rate in total, does 
it? I thought the pass rate was (1,362,786 - 16,020) / (1,362,786), or 
if we subtract the ignored: (1,362,786 - 482 - 16,020) / (1,362,786). 
This, however, is not correct: the number of failed is not the total 
number of failed, only the body failed (as is indicated). Is it possible 
to add a fifth figure: failed (header), to this row in order to have a 
complete list of figures?

> 4) Oddly, signed messages with huge numbers of Received: header fields 
> almost always pass verification, including one with 39 of them: 
> http://www.opendkim.org/stats/report.html#received_correlation (let's 
> see SPF do that!),
>

+1

> though that message was ultimately rejected for having too many 
> Received: fields.
>

Too bad ;-)

/rolf

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 8 Feb 2011 16:28:35 -0800

> -----Original Message-----
> From: Murray S. Kucherawy
> Sent: Tuesday, February 08, 2011 4:00 PM
> To: 'Rolf E. Sonneveld'
> Cc: [email-masked] WG
> Subject: RE: [ietf-dkim] Interesting recent statistics
> 
> > Good news! Although... we don't know whether the increase is due to spammers that adopt DKIM or legit
> > senders adopting DKIM.
> 
> We have some hints in the data, since I have some correlation between
> DKIM-specific message data and SpamAssassin results.  I just need to
> build the right SQL query.  I'll post again when I have that.

Here you go (remember, February is incomplete and August was probably skewed as the collection system rolled out):

+----------+--------+------+-------+
| count(*) | spammy | year | month |
+----------+--------+------+-------+
|       56 |      0 | 2010 |     8 |
|       19 |      1 | 2010 |     8 |
|      282 |      0 | 2010 |     9 |
|      548 |      1 | 2010 |     9 |
|      133 |      0 | 2010 |    10 |
|      212 |      1 | 2010 |    10 |
|       54 |      0 | 2010 |    11 |
|       80 |      1 | 2010 |    11 |
|       60 |      0 | 2010 |    12 |
|       51 |      1 | 2010 |    12 |
|       35 |      0 | 2011 |     1 |
|      114 |      1 | 2011 |     1 |
|       23 |      0 | 2011 |     2 |
|       76 |      1 | 2011 |     2 |
+----------+--------+------+-------+

So spammer adoption is outpacing legit adoption (except for December), according to the criterion that a domain is "spammy" if SpamAssassin thought at least 25% of signed mail from that domain is spam.  (Changing it to 50% barely changed the count(*) column at all.)  This is a count of new domains only, per month.

That report was based on the From: domain; keying instead on the "d=" domain, the result is almost exactly the same:

+----------+--------+------+-------+
| count(*) | spammy | year | month |
+----------+--------+------+-------+
|       37 |      0 | 2010 |     8 |
|       17 |      1 | 2010 |     8 |
|      284 |      0 | 2010 |     9 |
|      538 |      1 | 2010 |     9 |
|      122 |      0 | 2010 |    10 |
|      203 |      1 | 2010 |    10 |
|       44 |      0 | 2010 |    11 |
|       78 |      1 | 2010 |    11 |
|       54 |      0 | 2010 |    12 |
|       51 |      1 | 2010 |    12 |
|       31 |      0 | 2011 |     1 |
|      112 |      1 | 2011 |     1 |
|       18 |      0 | 2011 |     2 |
|       74 |      1 | 2011 |     2 |
+----------+--------+------+-------+

In any case, there is a constant stream of new legit domains as well, which is what we really need.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 8 Feb 2011 16:00:10 -0800

> From: Rolf E. Sonneveld [mailto:[email-masked]] 
> Sent: Tuesday, February 08, 2011 2:53 PM
> To: Murray S. Kucherawy
> Subject: Re: [ietf-dkim] Interesting recent statistics
> 
> [...]
> 
>> http://www.opendkim.org/stats/report.html#signing_trend (ignore the first row, that month was skewed;
>> also the current month is, of course, incomplete)
> 

We have some hints in the data, since I have some correlation between DKIM-specific message data and SpamAssassin results.  I just need to build the right SQL query.  I'll post again when I have that.

> But this row does not provide the correct pass/fail rate in total, does it? I thought the pass rate was
> (1,362,786 - 16,020) / (1,362,786), or if we subtract the ignored: (1,362,786 - 482 - 16,020) / (1,362,786).
> This, however, is not correct: the number of failed is not the total number of failed, only the body failed
> (as is indicated). Is it possible to add a fifth figure: failed (header), to this row in order to have a

Pass/fail is an indication of whether or not the signature verified.  A signature fails to verify if there was a header hash mismatch or a body hash mismatch.  Thus:

	failed(header) = total - ignored - pass - failed(body)

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 8 Feb 2011 16:37:42 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
> To: Rolf E. Sonneveld
> Pass/fail is an indication of whether or not the signature verified.  A
> signature fails to verify if there was a header hash mismatch or a body
> hash mismatch.  Thus:
> 

Actually, you probably would also have to subtract all of the cases where validation failed because of things like syntax errors.  I think we might need to add an 'errors' column as well.