Subject: RE: IETF announce list and spam filtering

From: "Christian Huitema" <[email-masked]>
Date: Wed, 14 Aug 2002 09:29:53 -0700

> Err, no.  That's not how SpamAssassin works.  It works based on
> looking at characteristics of mail messages, to see if they "look"
> like Spam.  It's what many of us do automatically.  For example if
> there is a HTML only body, and no text, I'm very likely to hit the 'D'
> key and go on, unless there's some very good reason why I think it
> might not be spam.  Similarly, if the character set is Korean or
> Chinese or Japanese, I will also assume that it is spam, and likely
> hit the 'd' key very quickly.

And here goes the baby with the bathwater: a lot of our Japanese
colleagues send mail to IETF working groups with:
	Content-Type: text/plain; charset=iso-2022-jp
This is a very natural set-up: they select "plain text" in their local
mail software, and they type their messages in English. They should be
fine, right? Well, not according to your rule. They would also have to
go tweak the preferred charset, a much harder proposition. I guess the
problem is the same from pretty much anyone who has to use a different
alphabet than ASCII.

The real reason why the IETF should study the countermeasures is
precisely that: it is damn easy to write spam filters that have many
"false positives". We need careful engineering, not slash and burn
tactics.

-- Christian Huitema

---

From: "Theodore Ts'o" <[email-masked]>
Date: Wed, 14 Aug 2002 14:52:29 -0400

> problem is the same from pretty much anyone who has to use a different

I didn't say I would automatically trash their messages.  Just that
the chances I would trash it is much, much, much higher.  In my
experience, the **vast** majority of the messages I receive with
Japenese, Chinese, and Korean charsets are SPAM.  If our Japanese,
Chinese, and Koreans colleages don't like that, then maybe they ought
to pass laws which prohibit SPAM from being originated in their
countries.....

> "false positives". We need careful engineering, not slash and burn

I wasn't describing an automated spam filter.  I was describing what
my figures have trained themselves to do, over multiple years of
receiving garbage in my e-mail.  It's only recently that I've started
using SpamAssassin, due to the increasing deluge of SPAM mail.  If
folks have better systems to propose, I'd be happy to look into using
them.

							- Ted

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 14 Aug 2002 16:56:35 -0700

At 2:52 PM -0400 8/14/02, Theodore Ts'o wrote:
>If our Japanese,

And until the US does the same, should we suggest to folks in those 
countries that they should use filters that throw away most mail from 
the US?

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Theodore Ts'o" <[email-masked]>
Date: Wed, 14 Aug 2002 21:40:56 -0400

> countries that they should use filters that throw away most mail from 

If that rule is effective for them, then maybe they should try it.
Unfortunately, they might find that the rule might have too high of a
false positive rate.

The reality is right now, the vast majority of the mail with non-ASCII
charsets, and the vast majority of mail that is HTML only, is SPAM.
So this is an effective filter for me, at least when combined with a
bit of human intelligence.  This may be that many of my correspondents
don't use Microsoft MUA's that have a tendency to declare charsets and
instead use MUA's that send plain-text e-mail messages.  But at least
for me, I haven't had a significant false-positive problem by keying
of the charset field.  Over time, I may find that I start getting more
false positives, and I'll stop using that hueristic.  But at least for
now, it's about as reliable as automatically trashing messages that
begin "This isn't spam..." or "You have requested this message as part
of AdSmart's marketing program" or "MAKE MONEY FAST".  (Although I
don't see much of that last one, as of late; spammers have figured
out it's a bad phrase to use.  :-)

							- Ted

---

From: Einar Stefferud <[email-masked]>
Date: Wed, 14 Aug 2002 19:31:50 -0700

I really have difficulty expecting that they do not already spam 
their own people with all those fonts and languages that they can 
read but we cannot.

I find it a great mystery why they even bother to spam us, but on the 
other hand, it is really easy to identify their spam and dev/null it.

Cheers...\Stef

At 4:56 PM -0700 8/14/02, Paul Hoffman / IMC wrote:
>countries that they should use filters that throw away most mail 
>from the US?
>
>--Paul Hoffman, Director

---

From: Bruce Campbell <[email-masked]>
Date: Thu, 15 Aug 2002 10:02:11 +0200 (CEST)

> Unfortunately, they might find that the rule might have too high of a

You've just insinuated that mail originating from the USA somehow has a
higher value than mail originating from Korea or Japan.  Given that most
of the cool stuff happening on the 'net is driven in part by people from
this area, I find that most surprising.

--==--
Bruce.