Subject: Re: Careful with those spamtools.....

From: Shelby Moore <[email-masked]>
Date: Mon, 15 Sep 2003 08:21:00 +0800

> Indeed. These open relay blacklist sites were always a highly questionable
> source for mail filtering. Quite obviously, open relays have no relationship to
> spam...

Agreed and already in public domain:

http://www.imc.org/ube-relay.html

A similar criteria conceptually more correlated to spam filtering, would be a blacklist of relays that are dishonest about the previous IP address in the Received header chain.  I do not think http://www.rfc-ignorant.org/ currently databases such non-compliance.

Then again such a hypothetical database would be mostly useless in implementation, because dishonest proxies come and go faster than we could database them.  Could test in real-time, but tests can be lied to.

There are (some proprietary) reliable way to detect the dishonest proxies, but I agree with Dean, much better to just detect the spam directly.

In terms of detecting spam directly, per message filters which are based solely on content, have such as high false positive cost and are subvertable with content:

http://citeseer.nj.nec.com/androutsopoulos00learning.html (See Page 9 of the PDF linked at top)

Filters based on bulk correlation (DCC) of content, require whitelist maintenance and are subvertable with content.  Filters which required your senders to opt-in are inherently expensive to the email system, as well as generate many false positives, and are subvertable by forged headers (not to mention being patented).  A brief taxonomy is here:

http://www.imc.org/ube-sol.html

Even if these above filter types haven't been subverted in high rates yet, they can be:

http://www1.ietf.org/mail-archive/ietf/Current/msg22190.html

We are working on a filtering mechanism which does not suffer from these sorts of issues, because it actually looks at what it unique about spam, not just some sometimes correlated side effects as other filters above do.

I agree with Dean and I think conceptually that ALL existing anti-spam (that is currently in public domain that I am aware of) is useless and even harmful as Dean points out (in long run) because they filter things which are not spam, just sometimes (even if most of time so far) correlated to spam.

I've been making points like this for a long time:

http://ixazon.dynip.com/pipermail/nilsimsa/2002-December/000041.html
(my warnings on dangers of Bayesian anti-spam filtering, which imo caused Paul Graham to eventually add a disclaimer to his web page)

Shelby Moore
http://AntiViotic.com

---

From: Dean Anderson <[email-masked]>
Date: Sun, 14 Sep 2003 21:50:06 -0400 (EDT)

On Mon, 15 Sep 2003, Shelby Moore wrote:

>
> I agree with Dean and I think conceptually that ALL existing anti-spam
> (that is currently in public domain that I am aware of) is useless and
> even harmful as Dean points out (in long run) because they filter things
> which are not spam, just sometimes (even if most of time so far)
> correlated to spam.

I do not think that _all_ anti-spam is useless.  Only that some of the
less well considered ideas have been harmful.

I think that content analysis holds much promise. Only a few years ago, we
thought that speaker-independent voice recognition was science fiction.
Along similar lines, I think it will be possible for intelligent agents to
decide whether email is likely to be interesting to us.

		--Dean

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 15 Sep 2003 09:27:59 +0200

On maandag, sep 15, 2003, at 03:50 Europe/Amsterdam, Dean Anderson 

> thought that speaker-independent voice recognition was science fiction.

And in the '60s we thought we'd all be going to work in a rocket by now.

Throwing away spam after the fact has the problem that there is no 
feedback to the spammer, so they happily continue to spam the same 
email address. And because their spamming gets less successful, they're 
forced to spam even more.

But I guess content filtering can be used to blacklist spammers by 
their IP address in close to real time. That would be useful.

---

From: [email-masked]
Date: Mon, 15 Sep 2003 11:15:53 -0400

> their IP address in close to real time. That would be useful.

But has to be done *very* carefully.  The closer to real-time you try to do it,
the more likely that a false positive will block somebody you didn't want to
block.

I've run into a few domains that run local blacklists that forget to to leave
some way to get in contact with them so you can tell them that you're dropping
50 of their users because their mail is bouncing because their filter can't
tell the difference between spam and a mailing list....

One thing often overlooked when designing things - the designers and early
adopters probably have a clue.  The second wave is full of idiots.

---

From: Dean Anderson <[email-masked]>
Date: Mon, 15 Sep 2003 14:38:14 -0400 (EDT)

Why would you want to give feedback to the spammer?

Anyway, it seems to me that real spammers are getting more successful, not
less successful.  They have won in court, and the radicals on spam-l have
given up on lawful methods.

And it also seems that real spammers have T3s, opt-in lists, working
opt-out addresses, and tend to be fairly responsible, so far as I can
tell. They are clearly not the ones sending out the garbage that is most
annoying---most of that isn't genuinely commercial, or even genuinely
fraudulent. (To commit fraud, you have to take money). For example, none
of the Nigerian scam mails that I've responded to try to get information
or money, nor responded at all, except for one person who said his address
was forged.

Indeed, the DMA (Direct Mail Association), a historical advocate of junk
mail, and also of spam, has taken the position, essentially, that legal
controls are necessary; Not on genuine spam, but on the abusers sending
garbage to annoy people. In other words, the radical antispammers.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Tue, 16 Sep 2003 10:41:18 -0400 (EDT)

See, I get feedback from anti-spammers.... Last night, I got a bunch of
these...  Someone in australia is trying to get my address blacklisted by
spamming out emails forging my address.  I wonder who that could be?

		--Dean

Received: from 61.88.124.155 ([210.124.48.16]) by
exchange2.the-leading-edge.com.au with Microsoft
    SMTPSVC(5.0.2195.6713);
         Tue, 16 Sep 2003 20:53:25 +1000
from:mbc
Subject:  --
Content-Type:text/html
Content-Transfer-Encoding:7bit
Bcc:
Return-Path: [email-masked]
Message-ID:
<[email-masked]>
X-OriginalArrivalTime: 16 Sep 2003 10:53:25.0933 (UTC)
FILETIME=[C14D5DD0:01C37C40]
Date: 16 Sep 2003 20:53:25 +1000

<