Subject: [Asrg] Blacklist metrics FYI

From: "Chris Lewis" <[email-masked]>
Date: Thu, 20 Mar 2003 14:31:37 -0500

Got a number of signs of interest in my metrics, so, I thought I'd spend 
a little more time on them.  Added a few of the more famous BLs we don't 
use, and "fixed" our whitelist metrics so they're useful in determining 
relative effectiveness in getting entries removed (which includes _our_ 
effectiveness in being able to request a retest).

We're not using the "not used" blacklists, and they're not included in 
the "TOTAL BLOCKED".  They're included for "relative performance" only. 
I'll only annotate those that have changed.

Blacklist effectiveness spamtrap only:
BOPM                      3907738  51.57
DYNABLOCK (not used)        77373   1.02 [1]
Flonetwork                    283   0.00
IP, NOT BL                  94011   1.24
MONKEYPROXY               4943111  65.24
NTblack                    880901  11.63
NTmanual                   362157   4.78
OBproxies                 1556609  20.54
OBrelays                   442915   5.85
OK                             65   0.00 [2]
OK MONKEYPROXY                  0   0.00 [3]
OK NTblack                      0   0.00 ...
OK NTmanual                     6   0.00
OK OBproxies                    0   0.00
OK OBrelays                     0   0.00
OK OSinputs                     0   0.00
OK OSsocks                      0   0.00
OK SBL                          0   0.00
OSinputs                   830086  10.95
OSproxy                    109974   1.45
OSsocks                   1923861  25.39
PERMBLOCK (not used)      5477053  72.28 [4]
SBL                        578476   7.63
SPEWS (not used)          1521310  20.08 [5]
TOTAL                     7577266 100.00
TOTAL BLOCK               6365664  84.01

[1] Wirehub DHCP pool (we were using this until recently, but due to 
technical problems with the zone builder, we've had to back off)
[2] Emails corresponding to current whitelist entries.
[3] Emails actually _needing_ a whitelist, and which one
[4] Wirehub PERMBLOCK is a consolidation of several other blacklists,
including SPEWS and SBL.  Not using it because of SPEWS.
[5] SPEWS level one.  SPEWS' relative aggressiveness is because they've 
escalated to whole class Bs (eg: Brazil proxy problems etc).  Their 
escalation/listing criteria are a little too unpredictable for our 
taste.  Worthy of note that SPEWS is vastly more famous than, say, BOPM, 
despite being only 1/4 as "effective".

Blacklist effectiveness excluding spamtrap:
BOPM                        99250   5.25
CONTENT                     54312   2.87
DYNABLOCK (not used)         6088   0.32
Flonetwork                   6054   0.32
IP, NOT BL                  29551   1.56
MONKEYPROXY                139631   7.39
NTblack                     38278   2.03
NTmanual                    26689   1.41
OBproxies                   47601   2.52
OBrelays                    16545   0.88
OK                           6139   0.32
OK MONKEYPROXY                 53   0.00
OK NTblack                     93   0.00
OK NTmanual                   443   0.02
OK OBproxies                 2221   0.12
OK OBrelays                    67   0.00
OK OSinputs                   296   0.02
OK OSsocks                     15   0.00
OK SBL                         37   0.00
OSinputs                    32263   1.71
OSproxy                      2032   0.11
OSsocks                     54823   2.90
PERMBLOCK (not used)       245617  12.99
SBL                         54785   2.90
SPEWS (not used)            80014   4.23
TOTAL                     1890155 100.00
TOTAL BLOCK                314863  16.66

The extremely high number on "OK OBproxies" is because of a single 
mailing list server whose "now closed" determination hasn't made it 
through the processing cycle yet.  Also, there has been some trouble 
with retest request submission with OB*.  Contrast that with BOPM - 
removal is sufficiently quick and reliable that NONE of its whitelist 
entries are required anymore.  The open relay blacklists have longer 
"retention" periods and higher FP rates as expected.

The OK versus sumof(OK *) entries indicates that we're tending to get 
60% or so of blacklist entries cleared (as per volume of email). Once 
the atypical OBproxies entry out of the way, it'd be closer to 95%.

The high numbers on NTmanual are due to the fact we block some whole 
ranges with NTmanual (NTmanual is a local blacklist), but punch holes 
for individual "good" sites.  And secondly, because we can't actually 
_do_ a removal on NTmanual yet.

Have to figure out what's happening with NTblack.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked] (Justin Mason)
Date: Thu, 20 Mar 2003 12:00:12 -0800

> effectiveness in being able to request a retest).

While we're at it, then, I'll post the SpamAssassin effectiveness
measurements from the most recent GA run (Feb 16 2003).

These are taken from a mail corpus of 63203 spam messages and 83465
non-spam messages, taken as a representative, hand-sorted sample from
about 15-20 people's mail feed over the 6 months before Feb 16 (spam from
only 3 months).  So it's not live, but it's reasonably close.

Each line consists of several figures, like so:

  total%  spam%    nonspam%  soratio rank    score name
  2.752   6.3763   0.0072    0.999   0.94    4.29  RCVD_IN_OPM

spam% and nonspam% are the most important ones in this case; it indicates
the percentages of the spam or nonspam corpora that were hit by the rule.
"soratio" is the "spam over overall" ratio -- so the probability that a
mail hit by this rule is spam.  A rule with an soratio of 1.0 is a perfect
rule for finding spam, and 0.0 is a perfect rule for nonspam.

They're sorted by the rule's "rank", which is effectively accuracy (as
per s/o ratio) and number of hits combined.  "score" is the score assigned
inside SpamAssassin 2.50.

First, the DNSBL effectiveness ratings:

  2.752   6.3763   0.0072    0.999   0.94    4.29  RCVD_IN_OPM
  3.468   8.0139   0.0264    0.997   0.94    1.10  RCVD_IN_SBL
  0.751   1.7420   0.0012    0.999   0.93    4.29  RCVD_IN_DSBL
  1.328   2.9097   0.1306    0.957   0.82    0.18  RCVD_IN_RELAYS_ORDB_ORG
 18.118  33.6297   6.3715    0.841   0.60    1.21  RCVD_IN_NJABL
 29.728  53.1889  11.9619    0.816   0.58    0.51  RCVD_IN_OSIRUSOFT_COM
  5.883   8.7290   3.7273    0.701   0.33    1.25  RCVD_IN_RFCI
 12.092  13.6607  10.9040    0.556   0.18    0.00  RCVD_IN_UNCONFIRMED_DSBL

and some others of general interest.  Here's Razor 2's effectiveness
on our corpora.  RAZOR2_CHECK is Razor's default check, while the CF_RANGE
tests are for Razor's "confidence factor" of 0 to 100.

 19.503  45.1039   0.1162    0.997   0.99    1.11  RAZOR2_CF_RANGE_91_100
 23.614  54.1794   0.4685    0.991   0.99    0.79  RAZOR2_CHECK
  0.200   0.4367   0.0216    0.953   0.81    0.58  RAZOR2_CF_RANGE_31_40
  0.147   0.3323   0.0072    0.979   0.88    0.85  RAZOR2_CF_RANGE_51_60
  0.168   0.3797   0.0084    0.978   0.88    0.33  RAZOR2_CF_RANGE_81_90
  0.822   1.8448   0.0467    0.975   0.87    0.98  RAZOR2_CF_RANGE_21_30
  1.293   2.8986   0.0779    0.974   0.87    0.59  RAZOR2_CF_RANGE_11_20
  0.278   0.6171   0.0216    0.966   0.84    0.37  RAZOR2_CF_RANGE_41_50
  0.162   0.3528   0.0168    0.955   0.81    0.00  RAZOR2_CF_RANGE_61_70
  0.595   1.2230   0.1198    0.911   0.71    0.27  RAZOR2_CF_RANGE_01_10
  0.130   0.2468   0.0419    0.855   0.58    1.86  RAZOR2_CF_RANGE_71_80

next, Pyzor and DCC on our corpora:

  4.535  10.4900   0.0264    0.997   0.94    1.25  PYZOR_CHECK
  5.736  13.2367   0.0563    0.996   0.94    2.76  DCC_CHECK

Finally, Bayesian learning results, using SpamAssassin's auto-learning
alone:

 27.135   0.0190  47.6679    0.000   1.00   -6.60  BAYES_01
 20.542  47.5832   0.0659    0.999   1.00    2.85  BAYES_90
  8.457   0.0016  14.8601    0.000   0.95   -6.40  BAYES_00
  6.427  14.8996   0.0120    0.999   0.95    2.81  BAYES_80
  7.458   0.0142  13.0953    0.001   0.95   -5.80  BAYES_10
  3.479   8.0613   0.0084    0.999   0.94    2.79  BAYES_99
  4.613  10.5660   0.1054    0.990   0.92    2.19  BAYES_70
  3.376   0.0490   5.8959    0.008   0.92   -3.10  BAYES_20
  3.417   0.2753   5.7952    0.045   0.82   -1.60  BAYES_30
  3.560   7.4395   0.6218    0.923   0.74    1.16  BAYES_60

Notes:

SBL's effectiveness is 8.01% for us; for Chris it's 7.63%.  Pretty
close.  But our effectiveness for OPM is much lower, at 6.3% instead
of 51%.

Chris notes an accuracy rate of 20% for SPEWS.  We get 53% (as
relays.osirusoft.com), but with 11% FPs.

--j.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 20 Mar 2003 15:57:20 -0500

> close.  But our effectiveness for OPM is much lower, at 6.3% instead

Two reasons for the OPM issue.  One is that OPM's coverage has had a 
huge jump over the past month or two.  Second is "I'd have to kill you" 
(which is also related to reason #1) :-)

Too bad you didn't have MONKEYS in there too.  That'd have answered the 
question of whether the "I'd have to kill you" factor predominates or not.

> relays.osirusoft.com), but with 11% FPs.

My numbers for SPEWS and yours for relays.osirusoft.com aren't 
comparable.  relays.osirusoft.com is a combination of several things in 
addition to SPEWS.  OSIRUS relays (inputs.relays.osirusoft.com) and 
Socks (socks.relays.osirusoft.com) are very large (OSsocks larger than 
SPEWS), and together they skew the relays.osirusoft.com values far away 
from SPEWS alone. However, I'd suggest that SPEWS was the main 
contributor to the "nonspam" value (tho, "spamsites" may have 
contributed too).  It'd have been more useful if you differentiated the 
contributing lists to OSIRUS.

Strangely enough, we did try DSBL, and weren't very happy with it.  No, 
we weren't using "unconfirmed".

Ditto NJABL.  It may be better now.

Be interesting to try SpamAssassin on our spamtrap.  Maybe one day.

---

From: Matt Sergeant <[email-masked]>
Date: Thu, 20 Mar 2003 21:07:21 +0000 (GMT)

> relays.osirusoft.com), but with 11% FPs.

It's worth noting that those 11% FPs are a little skewed by SPEWS listing 
a netblock containing the Linux Kernel Mailing List. Which is, of course, 
rather high traffic (even higher than this list!). Of course this is one 
of the major reasons not to use SPEWS, so the end result (low score in 
SA), is appropriate.

Matt.

---

From: Steven F Siirila <[email-masked]>
Date: Thu, 20 Mar 2003 15:03:48 -0600

> Ditto NJABL.  It may be better now.

We use DSBL and NJABL to find open relays/proxies (yes, NJABL now tests
for open proxies as of recently).  Both have been very effective for us.

We also use SBL to find known spammers and find it is very effective.
-- 

Steven F. Siirila			Office: Lind Hall, Room 130B
Internet Services			E-mail: [email-masked]
Office of Information Technology	Voice: (612) 626-0244
University of Minnesota
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked] (Justin Mason)
Date: Thu, 20 Mar 2003 13:08:20 -0800

> question of whether the "I'd have to kill you" factor predominates or not.

Yeah -- basically, there was a time when the Monkeys BL was down, so it
didn't seem quite reliable enough for distribution in SpamAssassin by 
default.

> contributing lists to OSIRUS.

oops, should have posted this to break that out:

  0.682   1.4714   0.0839    0.946   0.79    0.00  X_OSIRU_OPEN_RELAY
  4.438   8.6452   1.2520    0.873   0.63    0.00  X_OSIRU_SPAMWARE_SITE
  3.277   1.4857   4.6343    0.243   0.41    0.00  X_OSIRU_DUL_FH
  0.000   0.0000   0.0000    0.500   0.12    0.00  X_OSIRU_SPAM_SRC
  1.522   1.4224   1.5983    0.471   0.10    0.00  X_OSIRU_DUL

There's those ratings.  However I think Osirusoft's "Confirmed Spam
Source" (127.0.0.4) still includes more than SPEWS anyway....

> Be interesting to try SpamAssassin on our spamtrap.  Maybe one day.

with a big machine ;)

--j.

---

From: [email-masked] (Justin Mason)
Date: Thu, 20 Mar 2003 14:14:35 -0800

> Source" (127.0.0.4) still includes more than SPEWS anyway....

<doubletake>

hmm.  That's 0 hits for X_OSIRU_SPAM_SRC.  I don't know why...
but that makes that break-out of those stats somewhat less helpful. ;)

--j.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 21 Mar 2003 22:42:15 -0700 (MST)

> From: Steven F Siirila <[email-masked]>

> We also use SBL to find known spammers and find it is very effective.

The SBL is very different from some other DNS blacklists that send
unsolicited and unprovoked bulk email to innocent, virolently
anti-spam outifts in the name of various sorts of testing.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg