Subject: RE: [Asrg] Fwd: Returned mail: see transcript for details

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 1 Apr 2003 09:41:29 -0500

On Monday, March 31, 2003 11:12 PM, Vernon Schryver 
[SMTP:[email-masked]] wrote:
8<...>8
> So say you and I, but not everyone else.  There is no canon that
> sanctifies any particular definition of a "spam ratio," at least
> not yet.
>
I think (and maybe you are inferring this here) we must develop an acceptable 
definition for our work, with out one how will the effectiveness of a solution 
be evaluated?
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 1 Apr 2003 10:16:17 -0500

http:.//www.paulgraham.com/better.html

-e

On Tuesday, April 01, 2003 3:26 AM, Jon Kyme [SMTP:[email-masked]] wrote:
> > Vernon Schryver <[email-masked]> wrote:
> > > As others have pointed out, there is a genuine technical ambiguity in
> > > "false positive."  Is it RD/TOTAL or RD/SPAM?  (for RD=rejected but
> > > desired by recipient, TOTAL=total mail of all sorts whether rejected
> > > or not, and SPAM=whatever that means).
> > 
> > 
> >   As an absolute number, false positive should be easy to define.  "I
> > went through my 'spam' folder, and I decided that 3 messages should
> > not have been put there."
> > 
> >   As a ratio or percentage, false positives are taken relative to the
> > desired signal, not to the total data, or to the noise.  So we have:
> > 
> >   MAIL   = number of emails you want to be marked as ok (not spam)
> >   JUNK   = number of emails you want to be marked as spam
> > 
> >   MARKED = number of emails actually marked as ok
> >   SPAM   = number of emails actually marked as spam
> > 
> >   FP     = false positives (number of "MAIL" in "SPAM", which should
> >            have been in "MARKED")
> >   FN     = false negatives (number of "MAIL" in "MARKED", which should
> >            have been in "SPAM")
> > 
> >   The ratio of false positives = FP / MAIL
> >   The ratio o false negatives  = FN / MAIL
> > 
> >   MARKED = MAIL - FP + FN
> >   SPAM   = JUNK + FP - FN
> > 
> >   Alan DeKok.
> I wonder if it might be useful to adopt the widely used ideas of 
> specificity and sensitivity.
> 
> so if 
> TP = number of emails you want to be marked as spam that actually are so
> marked
> and 
> FP = number of emails marked as spam that shouldn't have been
> and
> FN = number of emails not marked as spam that should have been
> 
> 
> specificity = TP/(TP + FP)
> sensitivity = TP/(TP + FN)
> 
> (I think I've got that right - I'm sure there are lot's of references out
> there)
> These measures are commonly used for comparing diagnostic tests in clinical
> applications.
> 
> 
> 
> 
> --

---

From: "Alan DeKok" <[email-masked]>
Date: Tue, 01 Apr 2003 11:14:39 -0500

Vernon Schryver <[email-masked]> wrote:
>> [ false positives ...]

  The ideas of false positives, signal to noise, etc. have been around
for decades.  I fail to see how the spam problem is so unique that
those methods and definitions don't apply.

  I will go through my old textbooks, and add text to the 'spam
measurement' document I'm working on, which will reference existing
definitions for these terms, and contain some simple examples.

  Alan DeKok.

---

From: "Eric D. Williams" <[email-masked]>
Date: Tue, 1 Apr 2003 11:16:14 -0500

http://www.stat.psu.edu/~lsimon/stat250/sp00/solutions/diagtests.htm - gives an 
example.
http://www.medal.org/ch39.html - gives def's (under anchor links).

-e

On Tuesday, April 01, 2003 9:17 AM, Jon Kyme [SMTP:[email-masked]] wrote:
> Correction - apologies.
> Oops, no, I don't think that's right - it should be:
> Specificity = TN/(TN+FP)
>
> positive predictive value = TP/TP+FP
>
> It's been a very long time since I used this part of my brain.
>
> Isn't this all used in computer vision and other AI stuff?
>
> Has anyone got a good reference?
>
>
>
>
> --

---

From: Kee Hinckley <[email-masked]>
Date: Tue, 1 Apr 2003 11:33:28 -0500

At 10:49 AM -0600 3/31/03, wayne wrote:
>  > You skipped from "humans" to "senders of bulk advertising" very nicely
>>  there.
>
>"Senders of bulk advertising" are humans also.

A limited subclass :-)

>The bounce message from the block *was* a "I'm sorry, I don't want to
>talk to you any more" message.  Ok, maybe it wasn't as polite as it
>should be, but it is the same thing.

I can buy that.  But it's a little more than just a politeness issue.
1. lack of trust -- it says that you don't want to talk anymore, and 
you don't trust the respondent to honor that request
2. topic - it says that you not only don't want to talk any more 
about this topic, but that you don't think the correspondent will 
*ever* have anything you want to hear.  That's the difference between 
saying, "I don't want to talk about this." and "I don't ever want to 
see you again."

>Yes, UI issues of spam filter need much more attention paid to them.
>In particular, bounce messages are often far more confrontational,
>cryptic and impolite than they should be.

Yes.

>One of the things that following this list has convinced me of is that
>if you are going to block email, you should try very hard to make the
>sender's MTA generate the bounce.  (You certainly shouldn't just

Yes.

>I think a large part of the objections to challenge-response systems
>can be eliminated by only generating a challenge if a reasonably good
>spam filter detects it as probable spam.  Most personal emails would
>make it through unchallenged.  In the case that a challenge is
>generated, you can give specific reasons why sender's email looks like
>spam.  Different types of challenges could be generated depending on
>how spammy the email looks.

I agree, this is a good compromise.  And again makes me think that 
someone should be coming up with a standard for extended bounce 
messages that can communicate those facts.
-- 
Kee Hinckley
http://www.messagefire.com/          Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg