Subject: Re: [Asrg] 2. Analysis (specifically, trap addresses)

From: "Terry Sullivan" <[email-masked]>
Date: Thu, 02 Oct 2003 08:38:35 -0500

On Thu, 02 Oct 2003 00:07:24 -0400, Yakov Shafranovich wrote:

>Terry Sullivan wrote:
[snip] 
>> Ultimately, it seems to me that there is a need for a some sort of 
>> active (though not necessarily particularly time-consuming) trap 
>> address maintenance effort.  Otherwise, the only real opportunity
>> for *research* (as opposed to debate) lies in 
>> analysis/characterization of archival data.
>
> What is the bottom line? Can you elaborate on your desired steps 
> and what resources you require?

Actually, I was sorta hoping to inspire others to contribute their 
superior expertise to the discussion/effort.  Compared to several 
folks on here, my experience with trap addresses is minimal (and 
almost entirely accidental).  

I can help to identify/clarify/support the analytical goals that a 
trap address group might be asked to fulfill, along with helping to 
scope out the data/metadata requirements.  But I'm really neither 
qualified nor equipped to spearhead such an effort.  (Remember, I'm a 
statistician, not a mail admin.)

So, I guess the first step is for folks who're both able and willing 
to coordinate (or contribute to) a trap address effort to "raise 
their hands" (failing that, to write me/the chairs off-list).  For 
security purposes, the group should be small (again, no more than a 
handful of individuals) and thus needs to be composed entirely of 
folks both willing and able to actually contribute 
time/effort/expertise (and to sign NDAs).  If there's sufficient 
interest, the group could assemble and begin work almost immediately.  
If not, then the ASRG research agenda may need to be trimmed back.  

> As for archival data, SpamArchive and others provide lots of it.
> The FTC also maintains a spam archive and they might be open to 
> the idea of running something against it if we ask them.

Hey, I'm all for analysis of passive data.  But many of the 
questions/proposals, etc. that have been floated on this list, from 
greylisting to 550/CR, can't be answered by analyzing archival data.  

- Terry

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Terry Sullivan" <[email-masked]>
Date: Thu, 02 Oct 2003 09:09:52 -0500

On Wed, 1 Oct 2003 20:03:18 -0400, Kee Hinckley wrote:

>At 12:21 PM -0500 10/1/03, Terry Sullivan wrote:
>>Data from multiple independent sources (including Liam Meany, Scott
>>Nelson, and myself) indicate that "otherwise identical" trap
>>addresses receive *vastly* different amounts of spam traffic.

[snip]

>It's frequently been said that there are really only about 200 major 
>spammers.  And certainly there are fewer types of spamming software 
>than that, and fewer sources of target addresses.
[snip]
>...And it's definitely not a random population.

As you suggest, there are lots of possible/plausible explanations 
that may actually account for the "unexplained" variance in _ad lib_ 
spam volume.  And I absolutely agree with your core point, that said 
variance is so obscenely large that it cannot possibly be truly 
"random" in nature.  Clearly, one or more systematic forces are at 
play; but those systematic forces lie wholly outside experimental 
control.

Still, it remains statistically "bad" variance, precisely because it 
isn't subject to experimental control.  By implication, any "live" 
experiment simply *must* be able to factor it out at analysis time.  
Otherwise, there's no point in running the experiment.

(All of which underscores the need for a trap-address effort.)

- Terry

---

From: "Kurt Magnusson" <[email-masked]>
Date: Thu, 02 Oct 2003 14:35:37 -0200

[snip]
>greylisting to 550/CR, can't be answered by analyzing archival data.

A comment, since I've communicated some with Terry about this.
Not only we need more active data, we really need to run different
methods on the same accounts/site at the same time, it might help
to analyze some situations.

I have been running my proof-of-concept for the Earnest method
(filtering only on include URL's/Call center numbers, for you missing
it) on an old spam-ridden e-mail account, but when Blaster hit in
mid-Aug. my ISP gave up RBL and started up SpamAssassin in
baysian mode. I hade both one the same account.

It gave some really interesting effects and comparisons. My
data shows that both methods have ca 85 % hitrate each,
Earnest just above and SpamAssassin just below, but combined
they are incredibly effective, they stop well over 99% of the
spams with a low false positive rate, 2 out of 180 real mails
or ca 2 in 900 of the total amount of traffic. One because the
sender where from a domain, where the Earnest data file
extraction filter had extracted out a non-related URL-domain
from a spam, like a Hotmail link. The other because HTML in the
mail got SpamAssassin to lable it as spam.

The limitation of Earnest, as pointed out here before, is that
the data file with URL's and numbers needs to be often and
correctly updated, while a method as Spamassassin with baysian,
seems to be unable to recognize some types of spams (haven't
yet had time to check that).

A method like SpamAssassin also need more resources to handle
it's analysis. Earnest have ca 19.000 URL's, ca 400K, and works
with a simple grep, while SpamAssassin uses a number of criterias
and have typical ca 4-6 Mb data.

But the longer I have this combination, the more I lean on to
a multi-function process situation, two or three methods that
go through the mails, the most lightweighted first, leaving the
trickier cases to the more competent (and most resource prone)
method last.

Unfortunatly the figures above is on a bit low amount data, one
mailbox with ca 900 mails, 180 real and rest spams from Aug.
18 to Sept. 23. SpamAssassin was trained on ca 1500 active
mailboxes, where the av. spam level is ca 4 spam/day/box, while
Earnest data was manually extracted from 130.000 spams and
some hundreds more "live" through an automatic honeytrap
proof-of-concept tool.

Therefore it would be interesting to get other sites with more
volume and a number of mailboxes, that gets about the same
spams, also to run different methods in parallel, as well as some
combinations, in order to get more input.

Kurt Magnusson

_________________________________________________________________
Tired of spam? Get advanced junk mail protection with MSN 8. 
http://join.msn.com/?page=features/junkmail