Subject: Re: [Asrg] A paper/project worth considering (found it!)

From: Rich Kulawiec <[email-masked]>
Date: Fri, 2 Jan 2009 10:51:47 -0500

On Sun, Dec 14, 2008 at 05:30:24PM -0500, Chris Lewis wrote:
> I don't understand this.  I tried to explain this phenomena before.
> Didn't you take statistics somewhere?

Given that most of my graduate work was in statistical pattern recognition,
I think that's safe to presume. ;-)

> Let's say for sake of argument, AOL's users have a 5% error rate.  5% of
> what they report via TIS isn't spam.  That means, on average, 95 out of
> 100 reports are accurate and it is spam.
> 
> You have a FBL.  But you don't send any spam, right?  You only get your
> share of the error rate, and none of the accurate ones - because you
> don't send any spam.
> 
> So, from your perspective, the TIS button is 100% wrong.  For _you_ it
> is.  But it's NOT reflective of TIS hits against a network that sends spam.

Of course not: you're correct.  But it is reasonable to presume that a
user population which has generated a 100% error rate on the FP side has
also generated a substantial error rate on the FN side.  (That is, there's
no reason to think they're any more accurate one direction or the other.)

> Are you contending that Comcast's or Yahoo's FBLs are yielding correct
> TIS hits?  Or do you have FBLs with them at all?

Some of the operations I manage/consult to do.  I haven't completed
analysis of all those yet, so I'm quasi-reserving judgment.  But so far,
out of the data I *have* analyzed: 100% FPs.  It'll be months before
I'm done, I expect, because that's what it took to go through the AOL
results with a mix of automated/manual processes, and to cross-check
against logs, and so on.

> What the TIS button does is help highlight situations where the
> anti-spam filters aren't working.

Perhaps.  But I think all such instances need to be passed by a clueful,
experienced human for manual review.  That is, I think aggregating the
data and presenting to a person with a note that says "there may be
a problem here" is reaonable, but automated action based on end-user
reports alone is a bad idea.

I also think a much better approach -- which allows a higher degree
of automation because it removes users from the equation -- is to
run a large number of local and remote spamtraps.  After all, if spammer X
targets A local "real" users, then it seems reasonable to guess that X
will also target B local spamtraps and perhaps C remote spamtraps.
Correlation of data between all these makes it possible to identify at
least some spammers before users ever get a chance to use the TIS button.
(Yes, this is a methodology I use, and I use it based on connecting IP
address alone -- that is, I ignore everything else.  Any IP address
connecting to a sufficient number of sufficiently-diverse MXs and
attempting delivery to a sufficient number of spamtraps is up to no good
and is treated as such.)

---Rsk
_______________________________________________
Asrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/asrg

---

From: John Levine <[email-masked]>
Date: 2 Jan 2009 16:40:13 -0000

>no reason to think they're any more accurate one direction or the other.)

Depends on your model of the sending population.

It appears that you're in roughly the same situation I am,
well-behaved individual users, perhaps some COI lists.  With a
population like that, the scenarios for valid complaints are pretty
marginal, maybe a 'bot that takes its relay config from the local MUA,
maybe a few list members who treat the spam button as unsubscribe.
You're not going to get a lot of FNs if there's no spam to report.
What that tells me is that the main use to us is that if there were a
sudden burst of complaints, it would be a hint that something broke.

You can't generalize from that to a sending stream which has enough
complaint-worthy mail to generate a non-zero number of real
complaints.

R's,
John

---

From: Michael Thomas <[email-masked]>
Date: Fri, 02 Jan 2009 08:40:26 -0800

> reports alone is a bad idea.
That makes lots of presumption about what the "automated action"
encompasses. For me alone, hitting the TIS button is always 100%
correct (modulo the occasional clumsy fingers). Therefore "automatically"
putting the mail into the junk folder is 100% correct. For a larger 
population,
your FP rate will clearly go up. Whether the FP rate is acceptable depends
a whole lot on the population in question. One thing to consider here is
that the population does not need to be monolithic on the order of a
vast service provider's population. It can be subdivided into much smaller
populations that are context relevant for *me*. That is, people who
classify things like I do are likely to have a small FP rate from *my*
point of view. If I get enough of those similar-classifiers, and 
especially with
some diversity, their FP and FN rates are likely to be very good.

Think of this as "people who hate junk like I hate junk hate this mail".
I really wish people would stop with this absolutist nonsense that there
is single definition of junk that if only we bred more spam-Mentats
the world would be saved. People differ, as do their tastes, and
dissing them because of that entirely misses the point.

          Mike

---

From: der Mouse <[email-masked]>
Date: Fri, 2 Jan 2009 11:49:01 -0500 (EST)

> But it is reasonable to presume that a user population which has
> generated a 100% error rate on the FP side has also generated a
> substantial error rate on the FN side.  (That is, there's no reason
> to think they're any more accurate one direction or the other.)

Actually, I disagree.  If you send no spam, every report you receive
will be a FP; that is, you are guaranteed to see a 100% FP rate
regardless of how accurate your recipient population is.  (Well, unless
they're 100% accurate, in which case you'll see no reports at all and
your FP rate is undefined.)  All your user accuracy will affect is the
quantity of those reports.  The only data you have on which to base
even a wild guess at the FN rate for those users is the rate at which
you receive FP reports as a fraction of the rate at which you send
mail.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B
_______________________________________________
Asrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/asrg