Subject: [Asrg] Thoughts so far

From: "David F. Skoll" <[email-masked]>
Date: Thu, 13 Mar 2003 21:34:18 -0500 (EST)

This forum has been pretty depressing, actually. :-(

I have a few thoughts about the spam problem after reading the postings.

1) Time is of the essence

If Alan DeKok and Chris Lewis's domains are harbingers of things to
come, we really need to act fast.  Some of the protocol proposals on
this forum will take years, if not decades, to become widely-used
enough to have any effect.  If spam increases to "striker" levels, the
'Net will collapse before then.

Perversely, the spammers have given Alan DeKok an awesomely-powerful
weapon: With a few edits of his DNS zone file, he can direct a
powerful DDoS attack at the server of his choosing.  Perhaps rotating
this traffic among various government mail servers will convince
lawmakers there's a problem.  (They'll probably pass legislation making
redirection of mail to servers outside your control a felony.)

2) Legislation is needed, soon

Technology alone won't solve this problem.  Laws with real teeth are required.
However, I'm not holding my breath.

3) In the mean time, we must do whatever we can to force behavior on
spammers

Filtering works -- for now.  DCC works -- for now.  But both are easily
defeated.  The DCC fuzzy checksum, for example, is an astonishing piece
of work, and I can't imagine how many hours of development and testing
went into it.  Unfortunately, the effort required to defeat it is no more
than half an hour of a creative person's time.  Similar comments apply
to Razor, content-filtering and Bayesian analysis.  Even Razor's
clever (but not very scalable) "Ephemeral Signatures" can be defeated
with appropriate message mutations (left as an exercise for the reader.)

There is exactly one piece of information the spammer cannot control:
The reply code returned by the victim's mail server.  In spite of some
postings, I believe that spam runs contain many invalid addresses
(which other postings seem to confirm), and a clearing house mapping
number of bad destination addresses to source IP address would be
useful.  Again, in spite of some postings, storing this information is
no more difficult than storing the information already kept in DCC or
Razor.

The spammer can control the source IP address of his spam, but is
much more constrained.  Whereas the number of purported source e-mail
addresses is practically unlimited, as is the number of possible message
mutations, the number of possible source IP addresses is probably quite
small, in the millions at most.

So I see the only way forward as a combination of filtering, distributed
data-gathering and reporting, DNS-based blacklists, silly tricks like
tempfailing first-time senders, and so on.  All the proposals for
authentication, and the complex schemes for new mail protocols, are,
I believe, missing the point.  In the early days of the Internet,
everyone changed from NCP to TCP, because the Internet was small enough
that everyone went along.  Today, the Internet is too big to force a major
protocol change on everyone.  (Just look at the adoption rate of IPv6.)

Depressing.

--
David.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Matt Sergeant <[email-masked]>
Date: Fri, 14 Mar 2003 10:05:01 +0000

> However, I'm not holding my breath.

I have strong hopes for the FTC spam workshop to make some inroads into 
this. Anyone else attending?

Matt.

---

From: [email-masked]
Date: Fri, 14 Mar 2003 00:58:31 -0800 (PST)

The forum is not depressing, what is depressing is the fact that many want 
to see a solution but all have different thoughts on what it should be and 
none are probably correct (damn, I think I'm depressed now as well...)

Now I'll not comment on legislation (I personally do not think it'll help 
unless its global) and as a geek I prefer to focus on things technical or
fantastic (like solution to spam :) ...

I think as far as technical part we need to do the following:
 1. Good classification of possible solutions and proposals with all pros 
    and cons clearly listed (I'm attempting to do so now with my notes)
 2. Separate solutions into two threads
  a) solutions that can be done fairly quickly (1-2 years to adaption) to 
     partially control spam
  b) solutions that need more extensive changes and will take long time 
     to adapt but have a lot better chance of actually stopping spam

BOTH should be done at the same time, 2nd one will take longer to adapt 
but by the time solution(s) a) begin to fail we should be ready with at 
least some adaption for b). And yes, it may take 4-10 years for long-term 
solutions to become mainstream depending on how much change they require.

And I personally would prefer b) to actually be new Unified Messaging 
protocol to combine mail/news/im. The reason is that its adaption will 
depend not only on how well it'll stop spam but if it offers other 
benefits and the more benefits it offers, the better and easier it is to 
push it through. Besides new mail protocol would go just in time for new 
ip protocol (I'm curious how many email messages actually come through 
ipv6 and if there is anyone on this list who's doing it regularly, 
please respond if you do).

Hope it helps (though I'm depressed myself now...)

William

---

From: Jason Hihn <[email-masked]>
Date: Fri, 14 Mar 2003 09:33:44 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of David
> F. Skoll
> Sent: Thursday, March 13, 2003 9:34 PM
> To: [email-masked]
> Subject: [Asrg] Thoughts so far
>
>

It's technology's problem. Tech created it, tech will have to fix it. As
long as it is technically possible to do, you can bet we'll have it. We are
going to have to replace the email delivery system sometime.

> with appropriate message mutations (left as an exercise for the reader.)

Filtering works now, and is immediate. No one (even close to a mesuarble
percentage uses it) Plus it only hides the problem. The bandwidth is still
wasted. Not enough email clients do a good enough job. Maybe we need a
intermediate peice that will filter it on the server before the client pics
it up? But it'd have to be under user control. Defaulting to on I think
would be ok.

Filtering can buy us time though, and I think we should only see it as a way
to buy time and not as an end.

---

From: Kee Hinckley <[email-masked]>
Date: Fri, 14 Mar 2003 10:20:03 -0500

>percentage uses it) Plus it only hides the problem. The bandwidth is still

It's more widespread than you think--you just don't see it.  All AOL 
and MSN users use filtering, even if they don't know it.  On the 
desktop all the major email clients either already have builtin 
Bayesian filtering, or will in their next release.

>to buy time and not as an end.

Agreed.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "David F. Skoll" <[email-masked]>
Date: Fri, 14 Mar 2003 10:48:08 -0500 (EST)

> It's technology's problem. Tech created it, tech will have to fix it.

I don't agree.  It's a social and economic problem as much as a
technological problem.

> As long as it is technically possible to do, you can bet we'll have
> it. We are going to have to replace the email delivery system
> sometime.

We don't have the luxury of waiting.  Changing from NCP to TCP when
the Internet had under 1,000 hosts was easy.  Changing a major
protocol on today's Internet is well-nigh impossible.

> it up?

That's what our software does.  (End-user-controlled server-side
filtering.)

> But it'd have to be under user control.

Yes.

> to buy time and not as an end.

Filtering will always be useful because there are other reasons to want
to filter e-mail besides stopping spam.

--
David.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 14 Mar 2003 09:46:21 -0700 (MST)

> >to buy time and not as an end.

That pair of messages demonstrates a need for better terminology,
because I suspect that "filter" is used in two ways in the two messages
and that both of which are as misleading as the use of "forged" by
others to mean "sender has not yet jumped through my confirmation
hoop." Both uses of "filter" conflict with the historic technical and
non-technical meaning of "filter" as anything that removes some of a
stream.  Some messages sent to this list have used "filter" to include
and others have intended it to exclude checksumming mechanisms.  Some
people would say that looking for good or bad domain names among SMTP
headers is "filtering," while others disagree and say it has something
to do with "detecting forgery."

The confusion in "filter" is shown by the fact that many of the people
who distinguish "filter" from "DNS blacklist" (or "blocklist") consider
SpamAssassin a "filter" although it can use DNS blacklists.

I'd like to see definitions and to-be-determined ("TBD") words
something like these:

   TBD1 (address mismatch?): envelope and/or header values that are
     inconsistent or apparently contradictory, such as mismatches
     between From and Reply-To headers or between envelope Mail_From
     domain names and SMTP client IP address reverse DNS names.  Some
     uses of TBD1 are necessary and desirable.

   forged:  envelope or header values involving forgery and that are
     fraudulently intended to evade filters or confuse mail recipients.
     Header or envelope forgery involves false and fraudulent TBD1.

   filter:  any mechanism that filters or rejects or discards mail 
     whether based on IP addresses, envelope values, header absence,
     presence or values, words, punctuation, or other patterns in STMP
     bodies, STMP authentication mechanims, time of day, or anything
     else.

   TBD2 (word filter?): a particular kind of filter that involves
     detecting patterns of words or punctuation in message bodies.

   TBD3: ("content filter"? perhaps covered by TBD3) a particular sort
     of filter that attempts to characterize messages by checksums or
     semantic content.

   Bayesian filter: any of many mechanisms often unrelated to Bayesian
      statistics that involve detecting or count words in the body
      and sometimes headers of the message.  Bayesian filters are a
      special type of TBD2.

   TBD4 (address or domain filter?): a particular kind of filter that
     detects client IP addresses or domain names.  "Blocklist" and
     "blacklist" are commonly used for this notion.  Many people think
     that DNS mechanisms must be involved, but others include local
     lists such as the sendmail access_DB.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Jason Hihn <[email-masked]>
Date: Fri, 14 Mar 2003 13:47:24 -0500

>      uses of TBD1 are necessary and desirable.

=consistency filter

>      bodies, STMP authentication mechanims, time of day, or anything

broaden it to anything that categorizes something based on content.
(categorized for deletion is one example)

>      detecting patterns of words or punctuation in message bodies.

=lexigraphical filter (Lexigraphy)(lexicon=dictionary)

>      of filter that attempts to characterize messages by checksums or

=Statistical content filter

>      lists such as the sendmail access_DB.

=network filter

---

From: [email-masked]
Date: Fri, 14 Mar 2003 13:54:07 -0500

> I don't agree.  It's a social and economic problem as much as a

Don't expect to win on social/economic points unless you have a really
well-bankrolled team of vicious attack lawyers.

http://yro.slashdot.org/article.pl?sid=03/03/11/223248&mode=nocomment

Executive summary:  Microsoft doesn't like Washington's anti-spam laws.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 14 Mar 2003 12:32:57 -0700 (MST)

> From: Jason Hihn <[email-masked]>

> >      uses of TBD1 are necessary and desirable.

"Consistency something" is fine, but it makes no sense to "filter" for such
consistency except to try to stuff the Internet back into the BBS box.
Sanction "consistency filter" and you'll have people insisting on the
nonsense of rejecting mail from an example.com IP address because it
has an envelope sender of domain.com.

> (categorized for deletion is one example)

"Based on content" is one of the phrases used in this mailing list and
elsewhere that bugs me.  It means too many contradictory things to
people, and its literal definition is useless.  What isn't "based on
content" in some sense?

"Filter" should be consistent with the broad definition that long
predates the Internet.  That standard definition applies to the RBL+
and any other mechanism that "filters."

> =lexigraphical filter (Lexigraphy)(lexicon=dictionary)

good.

> =Statistical content filter

Statistics do not necessarily have anything to do with checksums or
semantic content (e.g. full-out Chomsky or AI efforts).

> =network filter

No, that has a trouble like "Bayesian filter" or the common use of "filter."
Everything that "filters" including the RBL+ is a "filter."  Everything
that rejects mail via a network is a "network filter."

---

From: Kee Hinckley <[email-masked]>
Date: Fri, 14 Mar 2003 18:03:57 -0500

>I'd like to see definitions and to-be-determined ("TBD") words

Good idea.

>      Header or envelope forgery involves false and fraudulent TBD1.

What would you call it when spam software adds headers that make it 
look like the message was sent by a particular email client?

>      bodies, STMP authentication mechanims, time of day, or anything

I'd replace "filters or rejects" with "categorizes or rejects" to 
avoid the self-reference.

There's another kind of filter.  One which, for lack of a better 
term, looks for "lies in the headers".  These can be forged routing 
information, fake MUA headers or anything else that makes the message 
different from real mail sent from a real user.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 14 Mar 2003 17:47:06 -0700 (MST)

> From: Kee Hinckley <[email-masked]>

> look like the message was sent by a particular email client?

If the intent is fraudulent, then it is "forgery."  Otherwise it
is no worse than sending an HTTP server other than a completely
accurate and specific name for the browser you use or adjusting
your HTTP, FTP, rlogin, telnet, BIND or other banner or version
string to be less than completely revealing.

> avoid the self-reference.

That would be fine, except that I'm trying to appeal to the standard
meaning of "filter" and fight the common use of "filter" in spam
circles to mean either "wonderful" or "garbage" depending on prejudices.
How about

   mail filter: any mechanism that filters by rejecting or discarding ...

> different from real mail sent from a real user.

That is not "another kind of filter," but "another *specialized* or
*specific* kind of filter."

How about "malicious forgery filter" to make clear it is intended to
be looking for malicious nonsense instead of innocent and proper
apparent inconsistencies such as legitimate differences between envelope
sender and reverse DNS name?

Picking honest terms instead of "forgery=not yet validated" and
"filter=whatever I think can't work/is wonderful" seems to me to be the
first or at least parallel step toward a taxonomy of bad mail defenses.

To finesse the unending controversy over whether "spam" means "unsolicited
bulk," don't use the word "spam."  When "unsolicited bulk email" is
intended, say so.  When something else is meant, then say that instead.

The idiosyncratic definitions of "filter," "forgery," and "spam" and
the resulting arguments are unnecessary and wrong.  If a position is
valid, then there is no need to redefine words in order to co-opt
their connotations.  Some people don't want to see "spam" defined as
"unsolicited bulk," because they fear that their favorite evil will
get short shrift.  So let's take "spam" off the table.

---

From: Kee Hinckley <[email-masked]>
Date: Fri, 14 Mar 2003 20:09:18 -0500

>If the intent is fraudulent, then it is "forgery."  Otherwise it

Well then we've just moved the question off to a different level and 
have to define "fraudulent".  In this case obviously they are doing 
it to try and trick spam filters--but that's not technically 
fraudlent.

>    mail filter: any mechanism that filters by rejecting or discarding ...

The only thing that I see as confusing is that some people use filter 
to mean "decide to accept or discard" and others mean "decide to file 
in the good pile or the bad pile".  I think we've seen some of that 
confusion here.

>sender and reverse DNS name?

That sounds good to me.

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 15 Mar 2003 09:22:52 -0500

> this. Anyone else attending?

I am.  I'm also on the panel list, but they haven't made their final 
decisions yet.  I spoke to a FTC staffer on Thursday...

---

From: [email-masked] (John R. Levine)
Date: 15 Mar 2003 09:52:10 -0500

>decisions yet.  I spoke to a FTC staffer on Thursday...

I spoke to them too and I'll also be going.  The FTC staffers are very
well informed about spam issues, even though they have frustratingly
few legal tools to bash spammers.

Regards,
John Levine, [email-masked], http://www.abuse.net, Trumansburg NY
abuse.net postmaster
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: mathew <[email-masked]>
Date: Sat, 15 Mar 2003 11:36:10 -0500

> Executive summary:  Microsoft doesn't like Washington's anti-spam laws.

This is one of the two unarguable reasons why legislation almost 
certainly won't work in the USA.

The other is that politicians will inevitably draft the laws to give 
themselves permission to spam.

mathew