Subject: Re: covert channel and noise -- was Re: proposal ...

From: Vernon Schryver <[email-masked]>
Date: Mon, 16 Feb 2004 10:09:30 -0700 (MST)

> From: "Robert G. Brown" <[email-masked]>

> ...
> I agree.  However, all of the observations made so far regarding
> spam/virus filtering in general still apply to filtering at the SMTP
> level. I would say that NO keyword filtering could take place.  The
> best that one could do at the protocol level would be to reject messages
> that fail to meet a tighter criterion than is currently required.

What is "the SMTP level"?  Is that during SMTP transactions between MTAs? 
Is "the protocol level" the same or something else?

If both of those "levels" refer to SMTP transactions between MTAs, then
the conclusion is wrong.  Except for local administrative hassles, all
spam and virus filtering that can be done later can instead be done
during SMTP transactions between MTAs.  Your SMTP server may need to
wait to until the end of the DATA command to object to messages
containing viruses, missing or bad SMTP headers or other objectionable
content, but that works fine.  I know of many millions of spam that
are filtred during the DATA command every day, and I don't claim to
know about any really big sites.

The only problems are:
  - local administrative choices that keep bastion SMTP servers ignorant
      of per-user filter preferences
  - filtering at the DATA command requires either (1) rejecting for
     all or no targets or (2) accepting for all targets and siliently
     discarding the message for those targets that want it filtered.

In theory the second problem could be fixed if the DATA command could
accept a vector of 250-OK/4yz-try-later/5yz-fatal responses, one for
each target named with a Rcpt_To command.  In practice the spam problem
will be solved one way or another long before such a protocol change
would be sufficiently widely deployed to matter.

Vernon Schryver    [email-masked]

---

From: John Leslie <[email-masked]>
Date: Tue, 17 Feb 2004 14:10:14 -0500

Vernon Schryver <[email-masked]> wrote:
> 
> I know of many millions of spam that are filtred during the DATA command
> every day, and I don't claim to know about any really big sites.
> 
> The only problems are:

   This is a feature, not a problem. If the end user wants a filtering
process individualized that much, s/he should choose to use a SMTP
server which agrees to do so.

>      discarding the message for those targets that want it filtered.

   Alternatively, the receiving SMTP server could reject any multiply-
addressed email.

   Is it actually that unreasonable to apply the most-restrictive
filtering rules in the case of multiply-addressed email?

   (Silently discarding _is_ a bad idea, when done by the SMTP server
itself. IMHO, it's better to mark for later discard -- which actually
could be done in such a way as to mark only for those recipients who
requested the more restrictive filtering.)

> would be sufficiently widely deployed to matter.

   Agreed: that radical a change in SMTP wouldn't percolate through
quickly enough.

--
John Leslie <[email-masked]>

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 17 Feb 2004 13:45:46 -0700 (MST)

> From: John Leslie 

> server which agrees to do so.

That is a feature only if the user accepts the consequences of discarding
mail without generating bounces, including not informing senders of false
positives.  Bounces from internal spam filters (either in MUAs or MTAs
inside organizations) are a major source of unsolicited bulk mail or spam.

>    Alternatively, the receiving SMTP server could reject any multiply-

People running SMTP servers that handle 100K or more msgs/day have
been uniformly horrified when I've suggested that.  I don't really
understand why, but I have given up on the idea.

> requested the more restrictive filtering.)

A better positition is that everything should be logged, particularly
including discarded mail, and in that case, enough of bodies to allow
targets to identify senders and the nature of the discarded messages.
Of course, one should assume users won't normally look at those logs.
Spam you read is not filtered, but at most categorized and stigmatized.

---

From: "Robert G. Brown" <[email-masked]>
Date: Tue, 17 Feb 2004 17:47:32 -0500 (EST)

> > requested the more restrictive filtering.)

Or, mark for later accept/reject decisioning AFTER the SMTP server per
se, in the filter pipeline between the server and the mail spool of the
addressee.  Spam assassin does the right thing already (and this is
exactly what it does).

> Spam you read is not filtered, but at most categorized and stigmatized.

Logging a message you reject is nearly a waste of time.  In order to
recover the message (as you note, nobody ever looks at the logs, which
are VERY LARGE for a busy mailer and beyond human capacity to scan), it
really requires an out-of-band message telling you there is a message to
be recovered.  As in calling somebody up and asking them why you haven't
paid your e-bills (rejected as spam).  In most cases it will require a
retransmission and out-of-band communications to complete the
transaction, because what you have done is made email an unreliable
messaging service for valuable, wanted, messages.

This is where, and why, I take issue with filtering and discarding at
the level of the SMTP server, unless the accept/reject decision can be
made with 100% precision (no false positives, no false negatives, and it
may not be good even then because MY idea of the correct basis for the
decision may not be the same as YOURS).

Imagine a networking transport protocol such as TCP, that discarded
packets not based on their header information (according to any
protocol-level criterion you like) but on their CONTENT, using a
moderately arcane ruleset designed to identify high-level human patterns
such as spam from the content information, applied to the content in a
complex and highly multivariate decision tree.  Hmmm, pretty ugly for a
TRANSPORT LAYER.

It's not that filtering based on non-header-linked aspects of content is
or isn't a good idea in some cases.  It is that it has no business being
in the specification of TCP.  TCP is about reliable delivery and
sequencing of packets.  It (as a protocol) has nothing to do with the
content of those packets outside of information contained in its
headers.  Indeed, rejecting packets based on any mechanical ruleset
applied to packet contents will almost certainly REDUCE the reliability
of the network, because (for example) an encrypted document might by
pure chance have a byte sequence like SEX that caused it to be rejected
even though its actual contents were quotes from the stock market.  One
doesn't write filters looking for buffer overwrite attacks into the TCP
stack in the kernel -- one fixes the application.

For nearly all filtering programs, it is too easy to create a message
that is filtered but shouldn't be.  Even rejecting viruses on the basis
of their signatures makes it difficult to SEND a virus you've just
received to an email drop where somebody can identify it.  As we've
seen, a naive application of rules mkes it impossible for this
most-certainly-not-spam discussion to penetrate various "protected"
sites.  It is often similarly easy to craft a message that should be
filtered but isn't because it slips through your rules.  But we've had
this discussion; the point is about reliability.

SMTP was designed to permit reasonably RELIABLE (simple) transport of
addressed mail on top of a TCP connection.  In most cases, ethernet
forms one of the lowest level protocols that the mail message is wrapped
in (ethernet header), then IP (IP header), then TCP (with its header).
SMTP pays relatively little attention to the message headers at the
transport and network protocol levels as it doesn't matter much which
ethernet address was responsible for the last packet hop (likely a
router/gateway and the same for all packets and messages).  It doesn't
care MUCH about IP or TCP, except that the packets have to be correctly
wrapped according to the protocols and addressed to SMTP listening on
the appropriate port.

SMTP does care about packet content, but only to a certain level.  Part
of the packet content it cares about is the negotiation phase where it
manages a connection, executing "commands" based on content in a very
structured way designed to communicate key information recursively along
the mail message's delivery trajectory, as it builds its own mail header
containing the addressee, the nominal sender, and the delivery route
complete with timestamps.  It is designed to be quite trusting, hence
easy to spoof, at least for mail to someone that doesn't read the rest
of the header.  Part of the packet content it views as "data" -- a
message to be reliably delivered to a uniquely specified spool file,
encapsulated within its MAIL header information (the TCP, IP and
ethernet level header information having been discarded).  There a user,
or a program, may or may not read it and take actions based on its
content.

It seems to me to be highly unacceptable to attempt to insert
content-based accept/reject decisioning in at this PROTOCOL level in the
delivery process.  It also seems to be highly risky and possibly legally
actionable.  I expect, in good faith, that email addressed to me will be
delivered if it is deliverable.  Otherwise I cannot rely on email as a
reliable transport mechanism for important messages.  Filtering it "for"
me according to ANY CONTENT-BASED RULESET risks discarding at least some
messages that are not correctly classified when they are rejected.
Important messages can be lost.  Bad things can result.  Who is
responsible when this occurs?  Who do I get to sue?

Imagine the post office (the real one) opening your mail and examining
content -- for most of us this alone would be a nightmare and invasion
of privacy -- so fine, with automated anonymous "safe" machines, to
eliminate advertisement mailings (only) but pass everything else.  One
day it rejects and shreds a sweepstakes check (a real one) thinking it
is one of the many fake checks sent out by loan companies.  Another day
it shreds a warning by your bank that a loan payment is due because it
also looks like an advertisement.  Even humans make this sort of
(sometimes very expensive) mistake, but at least if you make it it is
"your fault".  How would you feel, and react, if the mistake were
utterly beyond your control?  How long would it be before banks and
other businesses rejected the post office as a reliable transport
agency?

It is perfectly reasonable for you to add content filters that YOU
control ABOVE this transport layer.  If you want to hire a secretary to
open all of your mail for you and sort it and reject all the
advertisements, you can.  If the secretary makes a mistake and throws
away a megamillion dollar contract offer that you subsequently lose, you
also bear the responsibility, or at least can direct your anger at
something you control and take steps to ensure that similar things don't
happen again.

Now, all that it would take to control this end stage filtering and make
it much more reliable would be a federal law mandating that advertising
communications be sent in envelopes clearly labelled as such.  No more
sending out loan offers in envelopes that strangely resemble official
government communications unless they were clearly labelled
"advertisement".  No more writing the address in by hand to convince you
that you are opening personal mail.  If somebody violated the law, you
would have the envelope and offer in hand and could recover damages in
small claims court with very little effort.  Otherwise, you could safely
discard all advertising.

Note that this doesn't really help someone forced to do the final
accept/reject step themselves by hand.  They still have to look at each
envelope to look for the mandatory classification mark.  Advertisers
would of course try to exploit this by making the envelopes themselves a
major part of their messages -- "open ME".  It would make it fairly
simple to institute automated blocks, though, both for human secretaries
and real mail and for e.g. procmail and email.

With that all said, there are tools that ALREADY provide the kind of
content level filtering mentioned above.  The better ones do not
themselves discard or bounce any mail to any user.  They simply SCORE
THE CONTENT with regard to its likelihood of being spam (or a virus) on
the basis of a whole battery of tests.  Scores that exceed a given
threshold can easily and automatically be rejected or binned for a
second stage pass by humans later looking for lost checks and bills.
The USER (the one that ultimately knows the value TO THEM of a lost
message) can set the threshold to whatever level they are comfortable
with.  Some will prefer to play it loose and never risk losing a
message, even though it means that a lot of spam gets through.  Others
will reject on a fairly low threshold, not caring that they blindly miss
out on discussions of spam on ietf.org because those discussions
inevitably contain phrases like "Buy Viagra From Us Today" (oops, lost
one whole block of recipients), or "Have the Wildest Sex Ever" (now I
lost all the elementary and high schools in the country).  It is THEIR
DECISION, and they can pay the expectation cost "penalty" in lost
messages vs lots of spam at whatever level they select.

This doesn't require protocol level modifications of SMTP.  In fact, it
is generally desirable not to alter the SMTP MTA except in VERY
carefully thought out ways, although it is fine to graft in content
rating/filtering systems after the reliable delivery is complete to the
receiving host but before putting the message in a user's mailbox (with
the score in a special header where it can be used to drive user-level
decisions.  Just as spam assassin actually does now.

I repeat, I see little for the IETF to do about spam at the protocol
level, although it could be a powerful force at the political level,
communicating to our lawmakers the real costs of spam and urging them to
adopt stringent legal controls and penalties.  We could help draft those
legal controls and penalties.  For example, make it illegal to collect
and resell lists of email addresses for commercial purpose.  Require all
advertisements to be clearly labelled as advertisements in a form that
permits them to be automatically and reliably filtered.  One thing that
we COULD do is create a new mail header line marking mass produced
advertising mail as such, and match that up with laws requiring all
legitimate corporate clients to use it (with hefty penalties for those
that don't).

This is not the case for bidirectional encryption of email content.
There it won't happen unless and until the IETF works out a practical
way to make it work at the protocol level, since clearly ALL MTA's have
to be able to manage the encryption.

At this point I see no practical way to require or enforce point to
point encryption of all mail at the user level without a nearly complete
reengineering of mail transport (basically replacing smtp altogether).

I CAN imagine all hosts having their own RSA public and private keys,
and having all the public keys be part of their domain nameservice
registration information and hence automatically available to
"everybody".  I can imagine both tools and service providers that could
generate suitable key pairs and manage distribution of the pairs onto
registered systems either for a fee or as a part of routine systems
management (as we already do now, for the most part, for ssh).

I simply canNOT imagine that process extending down to the level of
private users at this time.  I don't believe that there is yet a
suitable vehicle for the required nameservice that would scale to
billions of registered entities and many trillions of bytes of served
data (as good keys aren't small) and that has at least the robustness of
the existing DNS.  

It is worth imagining such a service, but we are not terribly close, I
think, to being able to engineer it at this point.  It also has
interesting social costs and risks that I suspect would significantly
affect the engineering, as obviously protecting one's private key then
becomes an "interesting" problem all by itself.  A well managed computer
is already a bit of a fortress with facilities that permit protecting
data.  Users are not fortresses at all (quite the opposite) and if
anything are terribly lax with important data.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 18 Feb 2004 01:28:35 +0100

On 17-feb-04, at 23:47, Robert G. Brown wrote:

> made with 100% precision

> protocol-level criterion you like) but on their CONTENT,

Actually stuff like this exists today, sometimes called "layer 7 
switches". They're used to filter out HTTP worms and so on. But the 
comparison makes little sense as in TCP segments are part of an 
explicitly created session. When someone accepts the session, it's 
reasonable to assume they'll want to receive TCP segments that are part 
of this session.

The issue with email is that the only reasonable moment to reject 
messages is when you're still talking to the sender, so that a failure 
notice can be communicated back. Once the message has been accepted, 
you either have to deliver it, or you run the risk of false positives. 
Since non-false positives contain false return addresses most of the 
time sending back an indication that a message is rejected to the 
return address does more harm than good.

Note that filtering while a transport session exists ISN'T the same 
thing as filtering the transport stream itself: modifying SMTP such 
that any mention of our favorite medicine starting with a V is blocked 
makes it impossible for SMTP to function. However, rejecting messages 
that contain this word may or may not be appropriate, if the user isn't 
going to read them anyway rejecting them at the SMTP is the right thing 
to do as both silently dropping them or sending back failure notices is 
much worse.

> that is filtered but shouldn't be.

That's why information that a message is filtered must find its way 
back to the sender, so that the sender can take appropriate action.

> SMTP was designed to permit reasonably RELIABLE (simple) transport

SMTP isn't and has never been reliable.

> of privacy --

What world exactly are you living in? Never heard of bomb or anthrax 
letters?

And you're assuming some external entity is imposing an arbitrary 
policy here. Fact is that most people _want_ their service provider to 
filter out spam even using today's relatively unreliable mechanisms. I 
don't think worrying about a world where legitimate pharmaceutical 
messages are filtered out with no way to disable the filters is 
warranted at this point.

> communications be sent in envelopes clearly labelled as such.

Yes, and all that we need to do to make sure people don't use 
psychotropic substances is make them illegal.

If we can't stop denial of service attacks that bring down entire 
networks causing very real monetary damage, how can we ever expect to 
be able to do something about spam, where any invidivual message is at 
most a slight nuisance, using laws that only apply to a few percent of 
the internet anyway?

> level

If you were a spammer (or selling bandwidth to spammers, or maybe even 
selling anti-spam software or services) you'd be saying the same thing. 
So this claim is very suspect. You need to back it up with *very* solid 
facts.

> to be able to manage the encryption.

You really don't get it, do you? Encrypting the transport between 
multiple hops is useless. Encryption needs to be end-to-end. But we 
both have per-hop and end-to-end encryption standardized, implemented 
and used *today*. (It's too late for me to go out and find all the RFC 
numbers.)

-- 
PGP key: 0x1B1FC4E6  --->  http://www.muada.com/iljitsch.gpgkey
Fingerprint: CABE 3C2C 55D0 4D31 ED1A  2B6D 37E7 8439 1B1F C4E6

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 17 Feb 2004 18:27:44 -0700 (MST)

> From: "Robert G. Brown" 

> addressee.  Spam assassin does the right thing already (and this is

***NO***!  Except when run as a milter or otherwise during the SMTP
transaction, SpamAssassin does the WRONG thing.  As run almost everywhere,
after the SMTP transaction, SpamAssassin can only iether silently discard
spam or generate new spam by sending bounces to innocent people.

> > including discarded mail, ...

> Logging a message you reject is nearly a waste of time.  

Based on my experience, people running ISPs or other large mail system
strongly disagree with your position.  Besides, I intentionally wrote
about logging ***discarded** mail.

Many institutions do turn off logging of greylisted messages, reduce
the default per-message logging limit of 32K Bytes, or delete log files
far sooner than the 14 default in the DCC source.  Still, logging
is seen as vital to be able to answer questions about which messages
were filtered and why, including being able to say "that message was
never sent" or "substantially identical copies of that message were
sent to 310 other users here and 433,797 users elsewhere; it was spam."

>                                                          In order to
> are VERY LARGE for a busy mailer and beyond human capacity to scan),

I said nothing about humans scanning everything.  Besides, giving
users the sense that they can see what's happening with spam filtering
on their mail and control it is a requirement for getting users to
accept filtering.

> >..

What you describe is a broken version of what I advocate, if you
consistently look at your personal log of rejected mail.  Your version
is broken because a "reject decision" after the SMTP transaction must
at least sometimes result in sending spam to innocent people.

> in the specification of TCP.  ...

> >..

Nice straw man.  I've never heard anyone with a taint of technical
clues talk about looking for "SEX" in raw TCP segments.

> that is filtered but shouldn't be.  ...

You evidently lack experience with the filters used by commercial
institutions.  Commersical SMPT servers cannot tolerate false positives
(legitimate rejected/total legitimate) of more than 0.01%, and even
that's pushing it.  The design requirement for filtering mail on which
money depends is that false positives must not be much worse than the
underlying SMTP error rate due to problems such as full disks and
broken DNS servers--not to mention mail recipients too quick to delete.

A lot of current talk about false positives is self-serving nonsense
from such as the Direct Marketing Association.  Manual spam filtering
also has false positives.  A human suffering a common spam load of 100
spam/day has trouble not deleting legitimate mail.  My filters are
rejecting about 300 spam/day sent in my direction, 12227 in the last
40 days.  Mechanical filtering even with a significant false positive
rate can reduce the overall false positive rate.

> SMTP was designed to permit reasonably RELIABLE (simple) transport of

It would be good to skip the networking 101 tutorials.  Those of us who
don't know all of that about TCP, SMTP, CSMA/CD, etc. often thanks to
decades of personal experience should apply elsewhere to learn the basics.
It is may hard to imagine how old farts like me see such tutorials, but
please try.  I've been receiving email as "vjs" since 1968.

> delivery process. 

That use of "level" confuses me.  It does not seem to conform to ISO
OSI architecture.

> responsible when this occurs?  Who do I get to sue?

You don't get too sue anyone, because a reasonably designed system
lets you choose to do all of your spam filtering manually or in
your personal MUA.  The only caveat is that your account should be
terminated for network abuse if whatever mechanisms you choose
bounce very much spam to innocents.

That many filtering systems are junk including not allowing per-user
preferences is as illuminating as the fact that spam flogging spam
filters is common.  All I can see from either is that hard problems
attract charletans and would be gurus.

> advertisements, you can.  ....

You are welcome to require manual filtering for your mail, but you
have no standing to dictate to others at other institutions.

"Transport layer" seems like a misuse of the term.  It certainly
conveys nothing to me.

> themselves discard or bounce any mail to any user.  

People with experience in this field never say never.

>                                                     They simply SCORE

All of that can be done during the SMTP transaction just as effectively
as after.  Rejecting spam after the SMTP transaction should be and in
many quarters already is seen as network abuse requiring at least a
stern warning.

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 17 Feb 2004 20:08:37 -0700 (MST)

Thn enclosed example of how not to filter spam is offered for those
who might want to preemptively add accuspam.com or downloadfast.com
to their blacklists.

It is also a classic example of what is wrong with the MUA filtering
tactics Robert Brown advocates.

I certainly did not try to contact anyone at 3dize.com.  A few readers
of this mailing list might recall that Shelby H. Moore III and I don't
see eye to eye.
I do not know what the From: header of [email-masked] is about.
Perhaps it was forged.  Or perhaps someone at that address wired a
subscription to this mailing list through an unusually braindea)
challenge/response spam filter at DownloadFAST.com.  If so, the
Secretariat should blacklist accuspam.com and/or downloadfast.com from
subscriptions to IETF mailing lists.  (It would need to be unusually
braindead to send me the challenge instead of the envelope sender.)

> From [email-masked]  Tue Feb 17 19:26:14 2004
> Received: from www.DownloadFAST.com (www.downloadfast.com [65.61.155.11])
> 	by calcite.rhyolite.com (8.12.11/8.12.11) with ESMTP id i1I2QDiM048994
> 	for <[email-masked]> env-from <[email-masked]>;
> 	Tue, 17 Feb 2004 19:26:13 -0700 (MST)
> Received: from www.downloadfast.com (localhost.downloadfast.com [127.0.0.1])
> 	by www.DownloadFAST.com (8.12.10/8.12.10) with ESMTP id i1I1hIQf002492
> 	for <[email-masked]>; Tue, 17 Feb 2004 19:43:18 -0600 (CST)
> Received: (from root@localhost)
> 	by www.downloadfast.com (8.12.10/8.12.6/Submit) id i1I1hITA002491;
> 	Tue, 17 Feb 2004 19:43:18 -0600 (CST)
> Date: Tue, 17 Feb 2004 19:43:18 -0600 (CST)
> Message-Id: <[email-masked]>
> To: [email-masked]
> Subject: Re: Re: covert channel and noise -- was Re: proposal ...
> From: [email-masked]
> Reply-To: [email-masked]

> You attempted to contact us via email, and our anti-spam system is returning your email below.
>
> Please kindly resend your email below, using our Contact Form on our web page:
>
> http://accuspam.com?kM,vbrJT
>
> After using our Contact Form only once, your future emails will not be returned.
>
> If you do not use our Contact Form to re-send your email below, then we can not read it.
>
> ______________________________
> Powered by http://AccuSpam.com. Signup instantly for FREE!
>
>
> Your Returned Message:
> From: "Robert G. Brown" 
>
> > ...

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 17 Feb 2004 20:36:03 -0800 (PST)

> It is also a classic example of what is wrong with the MUA filtering

You certain dont assume that there is nothing wrong with the filtering
system you use and others may try duplicate as well. Otherwise how would 
you explain that you have Elan and completewhois.com listed as filtered
on your site. Do you honestly believe we ever sent you any SPAM? Or maybe 
you're making certain assumptions about envelope from or normal "From:" 
headers and complaining when others are making the similar assumptions?
 
-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 17 Feb 2004 21:03:19 -0700 (MST)

> From: "william(at)elan.net" 

> headers and complaining when others are making the similar assumptions?

Mail from Elan and completewhois.com is unwelcome at rhyolite.com in
patt because of a message that said:

] Elan.Net Internet
] T.1 T.3 Frame Relay
] If you need more information about us or are interested in network services 
] (managed hosting, collocation, dedicated servers, t1, t3), please send email to [email-masked] 
] 
] For More info 
] http://www.elan.net
] [email-masked]

There are additional, independent, sufficient reasons for that listing
that we do not need to explore.  If you will read my web pages, you'll
see that my list of unwelcome domains is not only about senders of
unsolicited bulk email.

An advantage of a vanity or other tiny domain is that it can use
blacklists that would have intolerable false positive rates at other
or larger outfits but that have 0.000% local false positive rates.

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 17 Feb 2004 21:26:12 -0800 (PST)

> patt because of a message that said:
You might want to post headers that show it being sent from some open-proxy
in pacific and showing use of email accounts that were never there.
 
> that we do not need to explore. 
Except that we did not send that message and anti-spam community was 
informed it was a joe-job (95% guessed as much on their own). Considering 
you knew who I'm am, I suspect you knew all this as well and if you did 
not you might have notice a warning about this on the website. 

> or larger outfits but that have 0.000% local false positive rates.
That is unlikely considering what I know about your list. 

Unless you make a good effort to research anything you probably have as
many problems as some of the most agressive ip-based filtering sites (like 
spews), you might make a number of good hits but would have number of misses
as well. Besides that as has been shown many times, for spammers its a 
lot easier and less expensive to get new domain that to get new ip block - 
and many do and setup hundreds of new domains on monthly basis. As such 
the value of domain-based filtering is very minimal indeed.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Tony Hain" <[email-masked]>
Date: Wed, 18 Feb 2004 13:18:31 -0800

So if you had received the mail sent here yesterday claiming to be from
Alain Durand would you block Sun or IBM? I am sure Alain did not send a
random executable file to a non-existent account. It appears someone figured
out he had responded to me on this list in the past, and plenty of times
daily there are messages with the same content sent to half a dozen account
names as a cc set. Correlating Durand to Hain is completely in line with
typical spammer behavior. The fact this message got here is not a Sun
problem (but someone at IBM might want to send me a note). The point is that
it really doesn't matter which proxy was used what shows up here looks like
a legitimate message from someone I have corresponded with in the past. The
only way to detect a fraud at the MUA would be to have a verifiable
signature from Alain (this was trapped at my MTA due to the exe file). 

Tony

192.35.***.***:43014;4.65.25.155:25;Tue, 17 Feb 2004 15:12:51 -0800
tndh.net
S471B7
MAIL FROM:<[email-masked]>
RCPT TO:<[email-masked]>
<<MAIL-DATA>>
Date: Tue, 17 Feb 2004 17:10:17 -0600
To: [email-masked]
Subject: ID qfp... thanks
From: [email-masked]

----------443488178303183
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Yours ID fscyygroiei
--
Thank 
----------443488178303183
Content-Type: application/x-msdownload; name="pcrceynyu.exe"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="cdv.exe"

> 192.35.***.***
Non-authoritative answer:
***.***.35.192.in-addr.arpa     name = ***.***.ibm.com

*** - if someone from IBM wants to contact me off list I will provide the
missing name/number

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Vernon
> Schryver
> Sent: Tuesday, February 17, 2004 8:03 PM
> To: [email-masked]
> Subject: Re: How Not To Filter Spam
>

---

From: Dave Crocker <[email-masked]>
Date: Wed, 18 Feb 2004 14:10:28 -0800

Tony,

TH> a legitimate message from someone I have corresponded with in the past. The
TH> only way to detect a fraud at the MUA would be to have a verifiable
TH> signature from Alain (this was trapped at my MTA due to the exe file). 

 yes, but no.

 first, there is an increasingly heated debate between folks who want to
 sign the message (TEOS, DomainKeys), versus others who want to secure the channel between
 sender and receiver (RMX, LMAP, SPF, etc.).

 Once that debate is resolved, there is still the matter of compromised
 system. The message might actually come from the purported author's
 system, but still not be from the author because it has been taken over
 by evil forces. So, even with perfect automated validation, the content
 still might not be valid.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: "Robert G. Brown" <[email-masked]>
Date: Wed, 18 Feb 2004 16:33:39 -0500 (EST)

> tactics Robert Brown advocates.

Vernon, I think you misunderstand what I was saying in my previous note
-- the problem was likely mine as I may not have been very clear.  If
so, I apologize.  I actually agree with much of what you say (and
thought that was clear on the basis of my earlier posts on the subject).

I was objecting to precisely this kind of filtering, and almost
forwarded it to the list myself.  You might recall that I predicted that
my message to the ietf would generate bounces just like this because it
contains a few key phrases -- obviously your reply contained the same
phrases. In fact, it was in objection to this sort of automated bounce
generation for both spam and viruses (especially the latter) that I
joined this list and have been participating in this discussion.

I also think you misunderstand spamassassin, at least the way it is used
here.  We as a department don't "filter" (in the sense of rejecting
email addressed to them before they have the opportunity to see it)
anybody's email at all.  Duke as an institution does filter some
incoming mail (not mine) for specific viruses (not spam) as viruses have
a comparatively unambiguous signature.

In the department, where we do USE spam assassin, no bounce messages are
generated except when mail fails for one of the standard reasons
unrelated to filtering of any sort.  SA is implemented so that it
passively scores each piece of mail as it arrives in fields added to the
message header -- in fact this one (to which I am replying) has:

Subject: How Not To Filter Spam
Sender: [email-masked]

Spam assassin is pretty smart -- it correctly identified this as
non-spam (negative total hit count) with a very high level of
confidence.  Empirically VERY few messages that score +5 or better are
not spam, although a few percent of the spam messages that SA sees end
up with scores of between 3 and 5 and get through.  SA is smart, but
cannot compare with the human eye.

When it DOES identify a message as spam it STILL does nothing.  It does
not act as a mail accept/reject mechanism per se. It just adds a header
such as:

Subject: Learn about this investment and get the results you are seeking
tgm
Sender: ylxArnold J <[email-masked]>
Date: Sun, 15 Feb 2004 00:25:53 -0800
network
        *  3.0 MSGID_FROM_MTA_SHORT Message-Id was added by a relay
        *  0.1 RCVD_IN_SORBS RBL: SORBS: sender is listed in SORBS
        *      [24.99.103.83 listed in dnsbl.sorbs.net]
        *  0.7 RCVD_IN_DSBL RBL: Received via a relay in list.dsbl.org
        *      [<http://dsbl.org/listing?ip=24.99.103.83>]
        * 1.5 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in
bl.spamcop.ne
t
        * [Blocked - see <http://www.spamcop.net/bl.shtml?24.99.103.83>]
        * 2.6 RCVD_IN_DYNABLOCK RBL: Sent directly from dynamic IP
address
        *      [24.99.103.83 listed in dnsbl.sorbs.net]
X-Spam-Status: Yes, hits=14.7 required=5.0 tests=FORGED_RCVD_NET_HELO,
        MSGID_FROM_MTA_SHORT,NOT_ADVISOR,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DSBL,
        RCVD_IN_DYNABLOCK,RCVD_IN_SORBS autolearn=spam version=2.63
X-Spam-Level: **************

(from another message in my rejected folder).  I then use a separate
tool, in my case procmail, to sort ALL my mail out into folders (which
might be my inbox or might be Spam or might be /dev/null).  No bounces
are ever generated, no spammer gets any hint that my address is or isn't
a good address, and I can filter on the basis of hits or spam level
under MY control by simply changing my procmail entry.  I can either
keep the rejects (I actually do save them for a week in a rotated file)
and go through them visually very quickly and delete them (I generally
don't, but others I know do, often people who set their personal
threshold for spam at 3 or 4 and a smaller fraction of spam delivered
into their mail spool but have a higher risk of accidentally rejecting
messages that aren't spam) or discard them altogether.  I do a fair bit
of discussion with hardware vendors and textbook vendors that can easily
get a few spam points (but isn't) so I keep mine fairly high.

I still have reservations about attempting to apply this or any other
content based filtering that isn't looking for highly specific
signatures (things like viruses or messages originating from blacklisted
networks that one can identify with very few false positives) at the
SMTP level.  I also have reservations about decisions like the
spam-level that triggers a reject being in somebody's hands other than
my own.  A systems administrator who got a lot of complaints about spam
within a LAN might set it down to 3 or 4 to lower the fraction that
makes it through for everybody, not realizing (or perhaps not caring)
that it might make mail conversations with vendors very difficult.

If that rejection occurred during the original transaction and generated
a bounce -- well, that's the kind of thing we see above, a cure that can
easily be worse than the disease, MUCH worse if the spam is being sent
from a breakout address or bounced through many hops to get there with a
forged or irrelevant return address.

If I understand what you are saying, perhaps there is a way to "do it
correctly" -- reject the spam at the original smtp transaction but with
a message that goes back to the original sender (only) in spite of the
fact that both the From and Return Path header entries might well be
forged and the message relayed through one or more open relays.  I don't
see how, but I admit that I may well be ignorant and there may be a way.

Either way, let me make it very clear.  I do not like any sort of
content filtering that generates automated bounce messages that may or
may not be directed to the original senders.  In the case of viruses
especially (which ALWAYS have forged headers) the bounce message ALWAYS
goes to an innocent and acts as a virus amplifier.  I use linux -- I
CAN'T get most current viruses -- but I receive ten or so messages every
day telling me that some message that "I" sent contained a virus.
Sometimes the bounce messages THEMSELVES are social engineering wrappers
designed to transmit the virus -- bounce messages are so ubiquitous that
viruses routinely use them (with an appropriately forged header) to try
to transmit themselves.  Spam bounces are to my experience no less
braindead and are a significant source of "network abuse" in their own
right.

As was recently pointed out on this list, the automatically generated
bounce messages are also a vehicle for DDOS attacks -- a virus that
forged the SAME return header as it tried to mail itself all over could
arrange for a flood of bounce messages to be directed to some target
address.  The only bounce messages I like to see are ones that tell me
that the addressee is unknown (I mistyped an address), the message could
not be delivered (because it was too large), or the message has been
held up in a spool en route (a destination host is down somewhere).
Having ANY message I send identified as spam, with a bounce or not, is
very annoying.

   rgb

---

From: Dean Anderson <[email-masked]>
Date: Wed, 18 Feb 2004 17:49:51 -0500 (EST)

>  sender and receiver (RMX, LMAP, SPF, etc.).

What does it matter what the resolution is?  Neither solution eliminates
the spam channel, and solves the problem, as information theory shows.  
Its just a silly debate over two schemes to thread a needle that can't be
threaded.

>  still might not be valid.

Right.  And it might also be valid and still spam.

		--Dean

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 18 Feb 2004 15:48:58 -0700 (MST)

> From: "Tony Hain" 
> To: "'Vernon Schryver'" <[email-masked]>, <[email-masked]>

> Alain Durand would you block Sun or IBM?  ...

I should not have responded specifically (if at all) to the other
gentleman's complaint about my blacklists.  Whatever I do to mail
directed at stuff I control is irrelevant here, provided I do not
affect any third parties.  My freedom to filter access to port 25 (SMTP)
and port 23 (telnet) is equally and completely unfettered.

Two groups oppose that principle.  Some people demand SPEWS and other
filters with what they consider too many false positives be outlawed,
because those filters might affect their outgoing mail.  They are
unmoved by users knowingly choosing their own filters.  They feel their
right to be heard by whomever they choose overrides the rights of their
targets to be left alone.

Other people see nothing wrong in spewing junk at third parties if it
might reduce their own spam loads.  These people include users of
systems based on challenge/responses, bounces after the initial SMTP
transaction (sometimes from within MUAs), "bitch lists" that send
complaints to dozens of third parties.  These people feel their right
to consent to whatever appears in their mailbox overrides the similar
right of others.

As I see it, both groups suffer the same pathology as spammers.

 .................

] From: "Robert G. Brown" <[email-masked]>

] ...
] In the department, where we do USE spam assassin, no bounce messages are
] generated except when mail fails for one of the standard reasons
] unrelated to filtering of any sort.  ...

On today's Internet, innocents are almost certainly receiving bounced
spam and viruses from your system that could not be delivered for
reasons unrelated to filtering, such as bogus target addresses.

] ...
] If that rejection occurred during the original transaction and generated
] a bounce -- well, that's the kind of thing we see above, a cure that can
] easily be worse than the disease, ...

The idiotic messages from that stupid challenge/response system are
not generated during the original SMTP transaction.  It is possible
to do challenge/responses that do not involve separate messages, but
they suffer from MUAs and MTAs that do not handle SMTP rejections
properly and users who cannot understand them.

Somehow making SMTP rejections understandable to users is something
that the IETF might attempt.  I think that is something the ASRG is
considering.  I also think that is nearly impossible.  such is life.

] If I understand what you are saying, perhaps there is a way to "do it
] correctly" -- reject the spam at the original smtp transaction but with
] a message that goes back to the original sender (only) in spite of the
] fact that both the From and Return Path header entries might well be
] forged and the message relayed through one or more open relays.  ...

Headers and the SMTP envelope, forged or not, are irrelevant to SMTP
5yz and 4yz rejections, as far as the rejecting SMTP server is concerned.
If the spam came through an open relay, then a proper SMTP rejection
might cause the relay to send a bounce to an innocent mailbox, but
SMTP relays are out of favor among spammers compared to open proxies.

---

From: "Tony Hain" <[email-masked]>
Date: Wed, 18 Feb 2004 15:18:14 -0800

Dave Crocker wrote:
> Tony,
> 
> TH> a legitimate message from someone I have corresponded with in the
past.
> The

Is there an obvious reason not to do both? Neither is a total solution, but
either ties a message to an identity object. It is time to stop fighting
over which is better and put both approaches out there. The market will
decide if one is easier for Joe-sixpack to use. 

>  still might not be valid.

Compromised systems are a problem, but the scope of the bogus mail
originators is limited to the users of the compromised system. Without
traceability it comes from everywhere, but with traceability at least one
knows where to go to correct the problem. If we are stuck fighting over the
'perfect' solution, we will never get anywhere. The engineering community is
particularly bad at figuring out what will catch on for the lowest
denominator consumer, and a committee of engineers takes a bad situation and
makes it hopeless.

Tony

---

From: Dave Crocker <[email-masked]>
Date: Wed, 18 Feb 2004 15:37:27 -0800

Tony,

>> first, there is an increasingly heated debate between folks who want
>> to sign the message (TEOS, DomainKeys), versus others who want to
>> secure the channel between sender and receiver (RMX, LMAP, SPF,
>> etc.).

TH> Is there an obvious reason not to do both?

Cost of effort. Distraction of scarce resources. Public damage from
unmet expectations. Etc., etc.

TH>  It is time to stop fighting
TH> over which is better and put both approaches out there.

Recently, I have become fond of the phrase "ready, fire, aim", because
so many people support your suggestion.

As you may have noticed, I consider timely response to urgent need a
very good thing, indeed. But there is a difference between wasting time
working on an ideal solution, to the detriment of an adequate one,
versus ignoring basic questions of efficacy and cost.

>>  still might not be valid.

TH> Compromised systems are a problem, but the scope of the bogus mail
TH> originators is limited to the users of the compromised system.

Yup. That's why accountability is inherently good. At least the ability
to track down the source is accurate, even if the "identity" of the
source might be at issue.

d/

---

From: "Robert G. Brown" <[email-masked]>
Date: Wed, 18 Feb 2004 19:18:35 -0500 (EST)

> reasons unrelated to filtering, such as bogus target addresses.

If a message comes in incorrectly addressed, yes, it will bounce.  It
should, shouldn't it?  This has nothing to do with whether or not it is
spam or a virus or any other kind of message.  If it is a bad thing, it
is a very fundamental bad thing...

As far as spam itself and my filters are concerned, there ARE NO BOUNCES
generated at any point from when the MTA accepts the message as being
RFC-compliant and deliverable through my reading (or not) any message.
The mail comes in (to postfix).  Postfix is configured to use procmail
as a mailbox delivery command.  procmail pipes the mail through spamc.
spamc (the spamassassin client) scores/tags it passively and delivers
it, in my case to procmail again, but now in userspace.  Procmail
applies the rule:

:0:
* ^X-Spam-Level: \*\*\*\*\*
SpamSpamSpam

and presumptive spam is appended to the SpamSpamSpam folder.  

All post-MTA tools are configured to generate no bounce messages (to the
extent that they are capable of such a thing, which is actually not
terribly great). As far as the sender was concerned, the message was
successfully delivered, and no, zero, none, ziltch email messages are
generated to any human or agent as a side effect of their delivery.

There may well be MUA's or post-MTA filters that generate a bounce or
C/R message on "identifying" a message as spam -- I could probably write
a trivial filter that would do this in a few minutes of work, and I get
bounces that appear to come from them from time to time.  I think that
we are in complete agreement that this is a bad idea for so many reasons
-- waste of system resources, annoying the innocent, and the fact that
such tools tend to be written by idiots and hence use appallingly poor
rules to identify "spam" (like the appearance of vi@gra anywhere in any
message even one time -- he says, resigning himself to another stupid
bounce message from whoever it is on this list that apparently is using
such an MUA:-).  However, it seems to me to be just as bad an idea at
the MTA level, and I have little patience with automated C/R systems
because they I have yet to encounter a message for which they were
appropriate or useful or anything but a PITA.

> SMTP relays are out of favor among spammers compared to open proxies.

I will wait with great interest to see if the "nearly impossible" comes
to pass -- it certainly has before in the world of IT;-) Even so, I
personally would want a degree of control over what the MTA rejects out
of mail sent to me.  A spamassassin-like tool (or SA itself) integrated
with the MTA so that the agent uses user-specified rules in place of or
on top of global/default rules, and a fair degree of control over what
is done with regard to the rejected mail.

I actually think that the spamassasin/procmail combination above is
nearly ideal on the MUA side, as I can see no particular reason to want
to bounce to spammers or anybody else, and MOST of the mail that MIGHT
be mis-scored could easily be either whitelisted or confirmed even
without a bounce with a definitely non-spam out of band message.  By
spooling my rejects for a week, I don't even have to have a
retransmission, just a message saying "why didn't you respond to XYZ".
And SA is pretty smart, and actually getting smarter...new rules seem to
be gradually eliminating a lot of the vi@gra-style messages, for
example.

   rgb

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Feb 2004 20:41:36 -0500

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:
    Dave> first, there is an increasingly heated debate between folks who
    Dave> want to sign the message (TEOS, DomainKeys), versus others who want
    Dave> to secure the channel between sender and receiver (RMX, LMAP, SPF,
    Dave> etc.).

  Why are these mutually exclusive?

  Can't we do both?

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email-masked]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 18 Feb 2004 19:39:24 -0700 (MST)

> is a very fundamental bad thing...

If the envelope sender was forged as is common in spam, universal in
worms, and practically nonexistent in legitimate mail, then your bounce
will afflict third party's mailbox.  My mailbox receives enough worm
bounces to make me say it is an awfully bad thing.

The only fix is to have your external MX servers know all valid
addresses and so reject junk before it can be accepted and later
need to be bounced.  That fix is often impractical or impolitic.

No, SPF, RMX, TOES, etc. etc. etc. cannot fix this problem unless you
assume frictions (deployment resistence and delays) do not exist
or you discard SMTP design goals including transporting messages among
complete strangers.

People who know each other's crypto keys are not strangers.  

If you could someday trust organizations to vouch for strangers and
not sell spam-for-a-day certs to Ralsky/Ricther/&co, then today you
could trust the same outfits to not sell spam-for-a-day/week/years IP
bandwidth accounts.

---

From: Dean Anderson <[email-masked]>
Date: Wed, 18 Feb 2004 21:58:31 -0500 (EST)

> and port 23 (telnet) is equally and completely unfettered.

You have a right to make your own decisions.  You don't have the right to 
congregate and act as a group unlawfully.

> targets to be left alone.

This isn't quite accurate. Not by a longshot.  Very frequently, its not
"users"  choosing their own filters. Frequently, ISPs choose the filters
for the users and without their knowledge or consent. Users give consent
to blocking spam, not for participation in alternate non-spam boycotts.  
Even in the relatively rare case when it is chosen by the user, it is
often the case that blacklists misrepresent their purposes to unsuspecting
users. Often, blacklists aren't about blocking abuse or spam at all.  
SPEWS and certain groups violate antitrust law, which prohibits group
boycotts from harming business.  SPEWS and some other blacklists aren't
about preventing abuse or spam. They are about harming business, and often
not even spam businesses, and such harm has been made illegal by law.

For example, Paul Vixie hosts SORBS, an Australian blacklist on his
ISC.ORG company.  SORBS has been booted from other American ISPs, and the
operator, Matthew Sullivan has had other sites (isux.com) booted for AUP
violations such as threats of mailbombing "spammers".  While the claimed
purpose of the SORBS blacklist is to block spam, one thing that
distinguishes this list is that it claims that Av8 Internet's IP address
space is hijacked.  No reasonable person could (or has) made this mistake.  
When this is brought to the attention of SORBS' users, they invariably
quit using SORBS. This isn't what they thought SORBS was blocking.  When
it was brought to Mr.  Sullivan's attention, he first replied that 'he is
not responsible for SORBS'.  When this was refuted, he said that he had no
assets, and challenged me to sue or contribute.  

As most know, Paul Vixie has his own blacklist at Mail-abuse.org. So what
is his interest in hosting SORBS at ISC.ORG after it was booted by other
ISPs for abuse reasons?  The reason could be seen in exactly the response
given by Mr. Sullivan: Vixie has assets, and therefore is motivated to
comply with civil law. But it would seem that Mr. Vixie would like to
support Mr.  Sullivan's irresponsible efforts without entangling himself
or Mail-abuse.org.  Perhaps he expects he can claim that SORBS is just a
customer of ISC, and that he has an arm's-length relationship and isn't
responsible for its defamation.  Complaints have been made to ISC and
ISC's upstream, EP.NET for SORBS continuing unlawful defamatory activity,
which is typically an AUP violation. ISC.ORG uses SORBS to filter its
abuse email.  Bill Manning (EP.NET) didn't want to forward or accept a
complaint about ISC or ISC's customer.  These complaints have been
ignored. But the lack of responsible enforcement allows some statements to
be made:  It just shows that Mr. Vixie is also irresponsible. It also
shows that Mr. Manning is irresponsible as well.

Mr. Sullivan, like many of the disreputable blacklist operators claims he
has no assets which can be taken by lawsuit, and thus has nothing to lose,
and no reason to comply with civil law. Instead of blocking spam, SORBS
has gone to extreme lengths to defame and interfere with Av8 Internet's
business, and probably the business of other companies for reasons that
have nothing whatsoever to do with spam.  None of this is made clear to
potential SORBS users until they subscribe to the list and block email
they don't want to block.

For another example, consider Alan Brown of ORBS.  ORBS was booted by
Canadian ISP for abuse even before Mr. Brown took control of it.  Mr.  
Brown has lost 3 separate lawsuits involving issues of defamation and
false statements. 2 of the lawsuits involved ORBS making false statements
about ISPs that Mr.  Brown did not like.  None of this was made clear to
ORBS users when they subscribed, and the blacklists blocked email they
didn't want to block.

The list of disreputable blacklists goes on and on.  The common thread is
a desire to mislead subscribers into thinking they are blocking spam, and
then abusing their subscribers by blocking things their subscribers didn't
agree to block.

A unique characteristic of SPEWS is that it's operators are anonymous,
apparently in order to prevent themselves from being held legally
responsible.

Mail-abuse (MAPS) was sued in 2000 by a permission-based emailer. This
wasn't a spammer, in the abusive sense. It was just a company that sent
commercial email to people who had given their addresses for that purpose.
MAPS was forced to stop blocking them.  It has frequently been said that 
this company should never have been blocked by MAPS--the company didn't 
meet the criteria that MAPS was supposed to be using for its blacklist.

Besides deceiving their users about their criteria, or violating the
criteria, the blacklists seek to be immune from lawsuits by transferring
their operations between countries or by having operators with no assets
take responsibility. This is just a scam.  Abusers and virus operators
don't sue. Genuine businesses sue.  That's who these abusive blacklists
are hiding from.  But their users don't want email from these businesses
blocked.

A number of blacklists have shutdown by blocking all of their subscribers
email. This is an irresponsible act, and widely criticized by their users
as being irresponsible.  No users want to use a blacklist that might
shutdown suddenly and block all of their email.  These are not responsible
organizations, and mostly they are misleading their subscribers about
their goals and what email they might block.

And things detailed in this message isn't even a fraction of all the abuse
that has been conducted by blacklists.  There is no reason to have
sympathy for blacklists. They have very little to do with fighting spam,
and those opposed to blacklists are not trying to be heard by those who
want to be left alone.  Just the opposite. People quit using the
blacklists, and then new blacklists pop up, to mislead them again.

		--Dean

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 19 Feb 2004 11:39:56 +0100

On 19-feb-04, at 1:18, Robert G. Brown wrote:

> should, shouldn't it?

Yes, but only by ejecting the message immediately during the SMTP 
session. Accepting the message, then realize it can't be delivered and 
sending a bounce message back to the email address listed in the "from" 
header is NOT the right thing to do for reasons that should be apparent 
by now.

> nearly ideal on the MUA side,

It is not, because:

1. Bandwidth is used up by spam (which is fortunately usually not that 
big) and worms (which tend to be much bigger)
2. A lot of processing time is used on your system(s)
3. Either:
3a. Legitimate senders who are tagged as spam are blackholed and are 
unaware that you don't read their message
3b. You must manually sort through all messages that are flagged as spam

---

From: "Robert G. Brown" <[email-masked]>
Date: Thu, 19 Feb 2004 08:47:55 -0500 (EST)

> big) and worms (which tend to be much bigger)

Close to a MB/day (on average) for me personally -- html is not compact.
Bouncing these requires (at the MTA or beyond) requires examining them
and applying tests to the entire message, generally, so bouncing at the
MTA saves no wasted bandwidth -- it close to doubles it.  

More than doubles it if the bounce generates further autogenerated mail
as it caroms off of bogus return addresses.  Note that server/netowrk
load is likely dominated by latency, not bandwidth -- most email
messages are at order of a single packet or two of data with ethernet
MTUs, but negotiating a transaction requires several rounds of small
packets.

Not enormous, agreed (although that is partly a matter of the size of
your organization and number of users and their internet "profile" to
spammers) but a significant fraction of all mail and not trivial.

> 2. A lot of processing time is used on your system(s)

This is the same for MTA-side and MUA-side processing as well.  In fact
it might be the same core tool used in the two cases.  Processing power
is linked to how sophisticated a filter you apply, and a stupid filter
is a cure FAR worse than the disease.  Either way you have to receive
the entire message at least to memory and apply the filter.

The processing power required, BTW, while non-trivial is easily within
the reach of modern CPUs for up to perhaps a thousand users per server
(I don't really know how much beyond -- we're not close to a boundary
here with hundreds of users).  A fully userspace solution permits it to
even be distributed to the user's processors and offload it altogether
from the mail server.

The one thing I can think of that an MTA-side bounce without any sort of
spooling or significant logging of the rejected messages saves relative
to user side sorting is disk spool, and this is really at the option of
the user, who can spool rejects into /dev/null.

> 3b. You must manually sort through all messages that are flagged as spam

Instead you have legitimate readers who may be unaware that a message to
them was bounced or blacklisted, you have one size fits all message
sorting, you have the aforementioned doubling of bandwidth consumption,
you have the possibility of your site being used as a "reflector" in a
DDOS attack.  And you (the user) CAN'T sort through all messages that
are flagged as spam because they aren't spooled, they are rejected!

In MUA-side sorting with no bounce you control the sorting process more
or less completely and can set rejection thresholds high enough that
your false positive rate is (in YOUR judgement, not mine, the IETF's, or
your local system's administrator) negligible and then neglect them.  At
5, SA's false positive rate is maybe one message in 100 days, from my
spot checks of it, and that rate actually decreases in time as they
refine the tests.  Its false negative rate is high -- a few percent of
the spam it sees makes it through -- but so does more or less all of the
spam-like legitimate mail I get including mail from vendors and vendor
quotes.  The spam that does make it through is generally "stealth spam"
(the kind with relatively short messages with lots of random words
intended to confuse word count filters, few or no graphics, and no
embedded html) and is easily rejected.  A weekly catalog newsletter from
e.g. musician's friend and a local bookstore make it through (I'm
subscribed on their lists but have NOT whitelisted their sites) and act
as coal miner birds for other desired spam-like traffic.

Those of my friends who use a lower spam threshold (one that lets almost
NO spam through to their primary spool at the expense of a larger false
positive rate) do manually sort their presorted spam folder.  However,
BECAUSE the folder is presorted, all they functionally have to do is
scan the subject/from lines and use their "internal" whitelists to pick
out the false positives, click a button or two to delete the rest en
masse, and move on.  As in it takes them maybe a minute or two to
process hundreds of rejects with very high reliability and without
having to look at message contents at all in almost all cases.  If you
like, they prefer a two state sort with human judgement used before
final rejection but using a computer to do 80% of the work (winnowing
all the spam out of their mainline mail spool where they DO read each
message and think about it one at a time.  I'm talking systems people,
mostly, who have a very low false positive threshold and who DON'T want
to explain to their user base (which might include their employer, for
example) why a message sent to them is flagged and bounced as SPAM.

With MUA-side processing, you can even do a three way sort -- sort all
spam-level 4-5 stuff into a "maybe" folder, and all 5+ stuff into a spam
folder or /dev/null.  At 5+ your false positive rate would be one to
three a year -- far lower than the human false positive or "oops"
factor, especially when dealing with a junk-filled mailbox.  Almost no
spam would make it into your mailbox.  The 4-5 "borderline" messages
would constitute maybe 10% of all spam, would have maybe 2% false
positives mixed in (mostly things like musician's friend catalogs that I
often don't read anyway) and would take a very short period every day to
terminally sort and reject.

Note well the fineness of control.  I'm NOT totally against MTA-side
processing but would want an MTA-side agent to basically be "spam
assassin" and permit spooling of an intermediate class of "maybe" spam
according to its necessarily fuzzy sorting logic.  The bounce issue is
endlessly debateable.  

At the moment, bounces of viruses are a pat peeve of mine -- they ARE a
form of abuse, this should NEVER be done, and the IETF would do
everybody a favor if they articulated this, and the reasons for it very
clearly, combined them with a statistical study that indicates that 0%
of the bounce messages actually reach the infected party and no more
than 1 or 2% reach somebody that might be on the same network as the
infected party and capable of passing the word to them, and took it to
the worst AV vendors and get them to deprecate the use of bounce
features in their existing products and remove them altogether in future
ones because they do no good, do a fair bit of harm in the steady state,
and have the potential to do a great deal of harm all at once if all the
AV products suddenly become reflector points in a timed, virus-driven
DDOS attack.  Or we can just tool along and wait for this to happen and
I can continue to live with the ten or so messages a day, NONE of them
flagged as virus or spam, that are "virus bounces".

The big questions are then: what fraction of SPAM has forged headers and
is a form of "virus" in its own right? and could the MTA filter be made
smart enough NOT to generate a bounce unless the from/return addresses
matched the registered domain of the originating host. I actually think
that this sort of thing would be possible -- one way that I identify
viruses now WITHOUT the use of a virus scanner loaded with signatures is
to look at the headers and look for lines like:

X-Authentication-Warning: moorcock.acpub.duke.edu: cyrus set sender to
  [email-masked] using -f

or a clear indication that the message has forged headers.  spamassassin
does the same thing:

        * 2.6 FORGED_MUA_OUTLOOK Forged mail pretending to be from MS Outlook

Here is a message for which there is no point in generating a bounce,
for example:

---

From: Dean Anderson <[email-masked]>
Date: Thu, 19 Feb 2004 15:42:12 -0500 (EST)

All quite sensible.

What I do on my personal mailbox, is 

	1) refile all mailing lists and well-known corrspondents
	2) Select all of the remaining mail not to dean@ and not from mail
delivery and give it a once over for non-spam messages. These would be
wildcards from certain domains or spam.  Refile this in a spam sortfolder.
	3) refile all bounced mailed according to bounce reason.  I need
to separate spam bounces to incoming undeliverable addresses from real
bounces that may need attention.
	4) refile all the spam to dean@
	5) What remains will be personal messages to dean@

	6) at my leisure, I read the rest of the email lists and other 
mail.

Steps one through four can usually be done in about 15 to 20 minutes
without interruption on about 1500 messages per day.

On Thu, 19 Feb 2004, Robert G. Brown wrote:

> From [email-masked]  Thu Feb 19 07:42:40 2004
> Return-Path: <[email-masked]>
> Delivered-To: [email-masked]
> Received: from ipserv.phy.duke.edu (ipserv.phy.duke.edu [152.3.182.5])
>         by mail.phy.duke.edu (Postfix) with ESMTP id CD802A77CF
>         for <[email-masked]>; Thu, 19 Feb 2004 07:42:40 -0500 (EST)
> Received: from 152.3.182.5 (unknown [62.150.67.242])
>         by ipserv.phy.duke.edu (Postfix) with SMTP id 48A443AFFB
>         for <[email-masked]>; Thu, 19 Feb 2004 07:42:38 -0500 (EST)
> Received: from 188.106.194.65 by web365.mail.yahoo.com; Wed, 18 Feb 2004 12:33:04 -0300
> 
> The mail originates from an anonymous client on an unregistered network
> and is sent to one of the spam-oozing pustules on the internet where
> anybody can get an email account in seconds and abuse it for minutes,
> and then throw it away.  It has a forged from and return path (ones that
> clearly bear no resemblance to the originating more or less open yahoo
> proxy).  It is also OTHERWISE identified as spam -- this particular
> message has a spam level of 24.6 (5 required for ~0.01% false positive).
> 
> It should be (and was, until now:-) rejected unread and without tainting
> the human mind.  It should NOT generate a bounce message, as a bounce is
> utterly pointless and will actually create a "reverberation" as
> [email-masked] is doubtless not a valid address any more (it is
> in multiple blacklists) and the earthlink.net MTA will undoubtedly
> generate a second bounce of your bounce or harrass the hapless user
> bobby20 whose fraternity brothers thought that they'd prank him as a
> nerd.
> 
> Going through my spam reject spool (and this is a REAL reject spool,
> filled with current messages like the one above from this very morning)
> I have a very hard time finding a single message for which a bounce is
> appropriate.  "Real" vendor mail (even mass mail) has a from address and
> delivery path that correspond and other things of that ilk such that
> sure, a bounce would work but for a real vendor mailing, even a mass
> mailing, a bounce isn't really necessary.  They will generally have the
> required-by-law unsubscribe at the bottom, and will generally honor it
> if you unsubscribe.  They MAY remarket your email address if you
> unsubscribe.  With SA you are free to /dev/null their message and let
> them continue to waste their resources talking to the hand or you can
> actively unsubscribe.  A bounce message might or might not serve to get
> you unsubscribed.  Finally, I shudder when I think of bounces of any
> sort that make it through lists.  They drive me nuts on the beowulf list
> although they frequently ARE caught -- some come "from" a whitelisted
> user.
> 
> Overall, it boils down to a question of degree of specificity, utility,
> and statistics.  Numbers, I truly do love them.  Looking at my own
> admittedly anecdotal numbers (although I'm pleasantly exposed to a
> horrifically wide range of spam as I seem to be in a few thousand
> address books of complete strangers and have a middling large web
> presence, so my anecdotal numbers are actually probably not a terrible
> sample), a "blind" MTA bounce of spam is more or less totally useless as
> not 1 bounce message in 100 will actually go back to the originator.
> 1% efficiency is a joke.
> 
> SO, if y'all really want to push MTA-side bounces:
> 
>   a) Use one of the very good MUA-side sorters such as spamassassin for
> a few months and accumulate a nice, fat spool full of presumptive spam.
> Do this with a few hundred volunteers, actually to get better
> statistics.
> 
>   b) Sort out the message headers into malformed and not in some way
> that leads you to believe that there is nobody home at the return
> addresses of the malformed.  Go ahead, test the return addresses to be
> sure!  Just do it from YOUR account and not mine...:-).  
> 
>   c) Email the well-formed header return addresses as well.  For yuks,
> do so from a new, pristine email address (one that is in NOBODY'S lists)
> and measure the time required for the new address to appear in new spam.
> For even more yuks, generate an email address for each reply and get
> per-return statistics on same.
> 
>   d) Determine the statistics from the data.  What is the probability
> that a bounce will reach the human or even the organization that
> originated the spam?  What is the probability of nucleating ten new
> messages from new spammers per bounce (even bounce) reply or
> unsubscribe?
> 
> THEN you can do two things:  Propose an MTA bounce with a proper
> foundation, and propose that foundation.  Presumably it will be one that
> does NOT bounce to pretty much the entire class of spam with malformed
> headers and may have other rules that you discover by sorting through
> all the patterns -- spamassassin gives you a very DETAILED breakdown of
> the rules and results used to tag a message spam, so you can actually
> look for very subtle multivariate correlates if you want to do a
> thorough job.
> 
> I personally am a great believer in the central limit theorem, and in
> spite of possible biases in my limited sample I suspect that it is
> actually very likely highly representative of the current profile of
> spam today.  On the basis of my own sample, it is not WORTH it to winnow
> out the one message in 10 (to be generous) or more likely one message in
> 100 (to be realistic) that might, and I say might, make it back to an
> originating human.  It definitely isn't worth my own time to try "hand
> bouncing" lots of messages to validate my anecdotal impressions.
> 
> So there's the gauntlet.  You propose that bounces are important.  Are
> they important enough to justify bouncing 9 messages out of 10 to false
> return addresses?  How about 99 messages out of 100?  You propose that
> bounces will be effective.  Well, starting with a 1% to 10% MAXIMUM
> success rate off the top you can't be VERY effective...:-) Perhaps
> you'll be effective in warning the even smaller percentage (one that
> depends on rejection threshold, to be sure) of false positives that
> their message didn't get through.  Here is where one has to look hard at
> the cost benefit equation.  Given the above first-approximation
> statistics, IS it worth the very substantial costs in misdirected and
> useless bounces to catch that small and controllable percentage of false
> positives?  If you disagree with my statistics, come up with better
> ones.  If you think that the answer is a matter of principle and that we
> must leave No Legitimate Mail Behind (however tiny a fraction it is),
> well, I would disagree, and we'd have to try to work out a consensus of
> some sort.
> 
> At the moment, I'm open minded.  Come up with an actual bounce algorithm
> (based on spamassasin ratings unless you can come up with something
> better) that bounces no MORE than one message in 100 to a non-existent
> or false address and that leaves no room for a DDOS attack to propagate.
> You can publish the same tests to the AV crowd so that they can fix
> their bounce programs.  That would be an important first step in
> convincing me that MTA-side bounces are feasible/desireable, especially
> if they are driven by a truly intelligent agent that leaves a user with
> PERSONALIZED control over their filtering controls.
> 
>    rgb
> 
>

---

From: Ed Gerck <[email-masked]>
Date: Thu, 19 Feb 2004 13:29:32 -0800

> bounces to make me say it is an awfully bad thing.

Yes. However, if your mailbox could automatically handle confirmation
requests based on messages that were actually sent by you (in much
the same way that NAT boxes work -- you only get a reply to a request 
you send), then you would not be bothered by the C-R traffic at all. 

> need to be bounced.  That fix is often impractical or impolitic.

Yes, also because "all valid addresses" is a dynamic list.
 
> or you discard SMTP design goals including transporting messages among

Messages among complete strangers is a necessary feature, IMO, but  
shouldn't it behave in cyberspace as we learned to do it in the 
social space? Trust is earned. When a complete stranger calls me, 
I usually ask who or what introduced me to him before I start any 
conversation. If the complete stranger has no satisfactory answer, 
I ask him to take me off his database and not call again.

> People who know each other's crypto keys are not strangers.

It is possible for my MUA to automatically provide a complete stranger 
with my PK if I receive an email from him. The barrier to have my 
crypto keys does not have to be any higher than the barrier to have 
my email address.

> could trust the same outfits to not sell spam-for-a-day/week/years IP

Yes. TTPs cannot be trusted per se. The answer is not PKI as we know it.

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 19 Feb 2004 15:16:03 -0700 (MST)

> From: Ed Gerck <[email-masked]>

> you send), then you would not be bothered by the C-R traffic at all. 

As long as you are wishing for things with no prospect of reality
in the foreseeable future, why not wish for long jail terms for the
ROKSO 200?

Automatic C-R handling in MUAs would solve the spam problem much
as NAT boxes have solved the address shortage and routing table
size problems, by creating other problems that are worse in the
long run.  For example, C-R handling in MUAs would do nothing for
the problems C-R systems have with mail that is not simplistic
messages between individuals.

Someone recently wrote that challenge/response systems would be practical
if there were a way for C-R systems to identify and not challenge
mailing list traffic.  That made me choke, because all spam is mailing
list traffic.  Perhaps what was intended was making C-R systems recognize
solicited mailing list traffic.  If your C-R system could do that,
there would be no need for any challenging or responding.  You would
challenge neither non-bulk nor solicited bulk mail, and would simply
reject all unsolicited bulk or spam mailing list traffic.

> I ask him to take me off his database and not call again.

If that's good enough for you, then you already have it.  The start
of a phone call from a stranger corresponds to the initial mail
message.  The asking to be added to a DNC list corresponds to adding
an entry to your email blacklist.

You probably want PKI magic that will tell your MTA or MUA whether
substantially identical copies of an incoming message from a complete
stranger will soon be sent to 30,000,000 of your intitmate friends.
That magic would happen before you do the equivalent of answering
a phone call from a stranger.

If you are among those who configure their telephones to reject calls
with caller-ID values not in whitelist, then you can configure your
email system to do the same with IP addresses.  That will eliminate
essentially all spam.  It also eliminates messages from strangers.

> crypto keys does not have to be any higher than the barrier to have 

If a complete stranger is the sender of an incoming message, then
crypto keys are irrelevant to determining the message is unsolicited
bulk.  If the sender of spam is not a stranger, then you made a mistake
in handling keys.

The PGP mantra that a good key does not imply that the sender or the
message is good applies here.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 20 Feb 2004 00:35:42 +0100

> MTA saves no wasted bandwidth -- it close to doubles it.

I agree that rejecting the message based on content during the session 
won't help here, but I wasn't arguing that doing it on the MTA is 
ideal, just that filtering on the MUA isn't an ideal solution.

If you reject the message during the SMTP session you don't need to 
generate a bounce message, the other side will do this. So the 
bandwidth waste is the same in both cases.

> the reach of modern CPUs for up to perhaps a thousand users per server

Fortunately things aren't as bad as this (that means services like 
Hotmail would need mail server clusters of super computer proportions) 
but doing content based mail filtering is certainly "non-trivial", as 
you say.

> them was bounced or blacklisted,

Requiring that the intended recipient is informed of mail transmission 
problems doesn't make much sense since if the failure can be 
communicated, then the message itself could also have been 
communicated.

Anyway, I think it's more important to provide feedback to the sender 
as the sender knows how important the message is.

> And you (the user) CAN'T sort through all messages that

I'm not interested in flagging messages as spam, I'm interested in 
rejecting a significant percentage of unwanted mail. If the mail isn't 
unwanted, then by all means, have it delivered, regardless of it being 
spam or not.

> return addresses?

I believe sending back feedback when a message is filtered for some 
reason is important, because this way a limited amount of false 
positives is acceptable as the sender gets to retry the communication 
in some way. However, I fully agree that sending back bounces to the 
"from" address does much more harm than good here. But rejecting the 
message DURING THE SMTP SESSION (sorry for shouting) doesn't have these 
drawbacks as you are by definition (still) talking to the actual source 
of the message.

(Ignoring the corner case where someone accepts a message with forged 
headers and then tries to deliver it to a third party here.)

I believe our current email systems and protocols are broken so 
fundamentally, given the current environment, that it's no use trying 
to fix them. We need something new.

---

From: Ed Gerck <[email-masked]>
Date: Thu, 19 Feb 2004 16:31:57 -0800

Vernon Schryver wrote:
> bulk.  

No. In PGP, for example, I accept a key based on who signed it and
when. If I can trust the signer(s), I may use a key from a stranger.

> message is good applies here.

Define "good key" and you'll define what the key is good for.

---

From: Doug Royer <[email-masked]>
Date: Thu, 19 Feb 2004 17:31:19 -0700

> bandwidth waste is the same in both cases.

Not only that,  bulk spammers (hacked or not) keep it in their queue and 
not yours when
it is not delivered. They might retry later, however they do that anyway.

-- 

Doug Royer                     |   http://INET-Consulting.com
-------------------------------|-----------------------------
[email-masked]                 | Office: (208)520-4044
http://Royer.com/People/Doug   | Fax:    (866)594-8574
                               | Cell:   (208)520-4044

              We Do Standards - You Need Standards

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 19 Feb 2004 18:15:35 -0700 (MST)

> when. If I can trust the signer(s), I may use a key from a stranger.

That sounds like the old "authentication solves spam" hope.  It was
wrong before SMTP-AUTH and it is still wrong.  If the sender is a
stranger, then by the definition of "stranger" you can know nothing
more than that the key works.  You cannot know whether the stranger
is one of Alan Ralsky's myriad of aliases delivering spam.

> Define "good key" and you'll define what the key is good for.

The ancient PGP mantra refers to keys that "work," as in the results
of decoding using the indicated public keys yield a valid messages.
The key can be good, but a good key tells you nothing more than that
the sender of the message knows the corresponding private key. 

Would you trust every PGP key from the IETF key signings to guarantee
that a message is not spam?  Some IETF participants have been unashamed
senders of unsolicited bulk commercial advertisements.  The person I'm
thinking of objected to his entry in my blacklist by insisting that
although he had sent the triggering message, it was not spam because
he had not sent more than one copy per mailbox.  He might have since
changed his definition and stopped sending unsolicited bulk mail, but
it would be silly to think everyone who gets a PGP key signed at an
IETF key signing party is someone from whom you want to receive mail.

Given who will pay certifiers, the IETF key signings are far less
bad guarantors of non-spam than commercial certifiers.  Consider
privacy policy certifiers and see one of the several versions of
http://enterprise-security-today.newsfactor.com/story.xhtml?story_title=Online_Privacy_Policies_Misleading

] An analysis of Web sites carrying those seals found that the
] companies running them ask for more personal information -- and
] protect it less -- than sites that have no seals.

---

From: Ed Gerck <[email-masked]>
Date: Thu, 19 Feb 2004 18:54:38 -0800

Vernon Schryver wrote:
> more than that the key works. 

It seems that you're not a PGP user. A signed PGP key has more useful 
information than just the key value. PGP keys can and should be signed 
by the key-holder and by one or more introducer(s). If you can trust 
those signer(s) as introducer(s), you may use a key from a stranger.  

BTW, this has nothing to do with "authentication solves spam". Spam is a 
complex problem that can only be solved by an array of measures where, 
IMO, PK encryption is more useful than PK signatures.

> of decoding using the indicated public keys yield a valid messages.

No, this is not how PGP keys should be accepted and considered "good".
Of course, since the rules of PGP are user-centric, you may define
whatever you want as "good keys".

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 20 Feb 2004 10:24:25 +0100

On 20-feb-04, at 2:15, Vernon Schryver wrote:

> wrong before SMTP-AUTH and it is still wrong.

Guess what, it is impossible to "solve" spam the same way it is 
impossible to "solve" burglary. At least with authentication you get to 
have whitelists that work. If you get a message with my email address 
in the "from" line it could be from anyone. If it is signed with my PGP 
key you know it came from me personally or someone went through a LOT 
of trouble to get access to my private key and the key phrase.

The usefulness of authentication could be further extended by building 
a web of trust where people vouch for the fact that others aren't 
spammers. Obviously spammers will slip through from time to time, but 
anyone who spams or keeps vouching for spammers will be removed from 
the web of trust. But even if this part doesn't work authentication is 
still useful.

---

From: "Robert G. Brown" <[email-masked]>
Date: Fri, 20 Feb 2004 09:32:03 -0500 (EST)

On Fri, 20 Feb 2004, Iljitsch van Beijnum wrote:

> Hash: SHA1

It is useful only if you only get mail from a small, closed group of
people, almost by definition, as I think Vernon and others have pointed
out.  I get mail from complete strangers all the time -- I'm a "known
expert" on at least one of the lists that I'm on and random people mail
me for advice and free technical consults.  I also have extensive
physics teaching resources up on the web and get email from random
students all over the country.  I have a couple of GPL projects in use
by various people that I don't know, and get random mail from the
unknown users.  I get mail from friends of friends, from relatives who
are completely clueless about technology in general and who would run
screaming at the very mention of the words "electronic signature" or
"encryption" (unless of course it were integrated into Microsoft
Outlook).  Most of this is mail that I could care less about being
signed -- I wouldn't bother to validate the signature at all if it were
and there was ANY WORK AT ALL involved in doing so.

Nobody is arguing that electronic signatures aren't useful.  Of course
they are.  So are bulletproof vests, armored cars, and bars on windows.
So are armed couriers who carry messages in booby trapped briefcases
handcuffed to their wrists.  They are useful when your body, your money,
your house are in a very hazardous environment or there is something of
great value at stake.  They are just very >>expensive<< measures, in
both money and human time and hassle.  In some cases, expense and hassle
or not, they don't work. Bars on windows are all very well but not when
somebody kicks in your door, copies your key when the car is in the
shop, or when there is a fire and you need to get out of a window or
die.  And they're ugly.

This is all about cost-benefit and the realities of the messy, chaotic,
ignorant world of mail users the world around.  In nearly all cases the
cost-benefit of signing or encrypting all messages and maintaining
strict, reliable lists of ALL your correspondants' keys is
overwhelmingly negative.  The work involved vastly exceeds the work
required to merely delete spam that makes it through ordinary
intelligent filters, and the filters don't require any sort of massive
database of keys that would probably neither succeed in its purpose or
scale if it were built.

Even when it isn't obviously negative -- for example when I'd very much
like to be able to send in grades at Duke via email, which is obviously
only possible if the messages are electronically signed, as this would
definitely save me a few minutes of work a semester -- surprise
surprise, turns out that the registrar's office wouldn't know an
electronic signature if one was dangled in front of their face, the
university has no centralized database set up for keys, if it DID have
such a database and the registrar's office DID understand electronic
signatures, it would still need mind-numbingly transparent tools for its
essentially untrained office staff to be able to read my email
containing grades and authenticate the signature.  Number of clicks <=
1.  Training required = none.  Signing and/or validating a signature to
anyone on campus, known or unknown, would have to be reduced to one
click (or none at all).  We could do this but we don't.  Why not?

It is not because Duke is a "backwards institution" in terms of its IT
-- quite the contrary, I think we do rather well and have even won
awards that suggest that others agree.  It is because the cost of
setting all of this up EVEN with (at this point) a decent campus wide IT
infrastructure, a SISS database from peoplesoft, and campus wide
authentication mechanisms and user-specific control over access to
database fields, the OPPORTUNITY COST of making all of this work is
greater than the cost of making me find a fax machine or walk the grades
over to the registrar's office.  They might do it somedayfor other
reasons, but it is of limited utility in the grand scheme of things
relative to its cost for NEARLY ALL PURPOSES so it won't be soon.

Now, if it is not going to happen right away in this relatively small,
relatively advanced technical environment because of a lack of apparent
cost-benefit at an institutional scale, how exactly is this going to
scale to the entire Internet?  There one doesn't even have the
underlying universal kerberos-based authentication scheme (so that we
can know who is who from the point of view of accessing resources) or
integrated database (so a known individual can access data they are
entrusted with).  And if one DID, what database is going to robustly
scale to 10^8 entries distributed globally, at what cost?  And finally,
IF you built it, would it work or would somebody metaphorically just
kick your door down, copy or steal a key, or cut a hole through your
cheap wooden walls? Would you find yourself unable to see through your
windows for the bars and shutters and deprived of light and air from
outside, would you be unable to get out in case of a fire, would you
discover that your non-tech wife and kids can't figure out how to solve
the puzzle lock you installed on the front door and are forced to live
in a tent on the front lawn?  These are metaphors, sure, but they are
pretty good ones.  My wife, at least, has a hard time dealing with email
at all, given that she types with two fingers and is somewhat luddite in
her general world view, and she is CAPABLE relative to a lot of people
who manage to use email.

Note that this is NOT ABOUT PROTOCOLS.  As you have so aptly
demonstrated, it is entirely possible to sign messages already with open
source tools that are nearly universally available.  NOTHING prevents
people from developing new tools and integrating existing tools to make
all of the above happen automagically and transparently except that
commercial vendors don't think that they'll make any money for it if
they do, and open source software developers (who WILL eventually solve
this problem without any sort of guidance) tend to be driven by at least
a degree of personal need.  So far the tools have advanced to where a
computer expert can use them, and nobody has invested the energy to make
those tools idiot proof and fully integrated, although there are some
who are working on it.

I'm not about to sign messages to this list because (lacking those
tools) it is a PITA to do so, nor do we have each other's key data so in
any event I cannot verify that this is actually from you or you from me.
We may never meet.  I may not know anybody that you know that I would
trust to "sign your keys".  This doesn't matter.  WITHOUT keys, I trust
that I'm replying to mail that really is from you (whoever you are:-)
and am glad that we can communicate clearly and "reasonably" confidently
without the cumbersome burden of keys.  We could even get to be quite
good friends without meeting and without key exchange (I certainly have
many good friends I only "know" from email exchanges).

The only advantage of signed mail is for me to be sure that mail from
you is really from you.  Lacking a universal and all-encompassing list
of keys of possible correspondants, however, I still have to look at all
my mail as I might get unsigned mail or mail from people that whose
signature I cannot validate or whose signature it is too much trouble to
validate.  Furthermore, it is generally pretty obvious when mail from
people I DO "know" is really from those people -- my wetware neural
network is really, really good at reading their "signature" from what
they type, especially when I can look at headers, call them on the
phone, carry on a continued conversation with them if I am in any doubt.
I have a harder time recognizing people I know on the phone than I do in
an email message.

Signed messages have a purpose that is worth the effort for a tiny
fraction of all email communications, under the same general conditions
as the metaphor of the briefcase and handcuffs above or when a real
signature would be appropriate.  When information that MUST be kept
private is mailed, when data is mailed that MUST be validated against a
particular individual with legal force.  If anything does drive the
development of a reasonably scalable (institutional level, at least)
electronic signature/encryption scheme it will likely be things like
HIPPA.  Duke may not care much about protecting/validating my gradesheet
when there are simple alternative secure channels for its delivery, but
it is likely to be forced by federal law to care about
protecting/validating hospital data in transit, and the cost of hand
delivery and moving paper everywhere (which has security risks of its
own) is much higher.  The nonlinear constraint of a federal mandate and
possibility of legal suit creates an opportunity cost advantage to
making signatures and encryptions work which, in turn, is likely to
drive real developers, both commercial and noncommercial, towards
engineering a real solution.  Laws DO matter and CAN drive technical
innovation.

   rgb

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (Darwin)
> 
> iD8DBQFANdJTN+eEORsfxOYRAkk1AKCuNlZC3Te7VsC7UTiVcHu9CTGrBgCdFjAJ
> 5v4Y06Kl7UosCH6d9OJpvII=
> =eFuO
> -----END PGP SIGNATURE-----
> 
>

---

From: "Tony Hain" <[email-masked]>
Date: Fri, 20 Feb 2004 11:49:05 -0800

The problem with both of your arguments is in the assumption that a full set
of keys needs to be maintained locally, or that a web of trust will solve
the problem. While the web of trust idea works among a small set where
people can trust someone to vouch for someone else, it doesn't scale. What
are the liabilities for not removing the abuser from the web of trust? Who
is actually responsible for making sure it happens? What is the recourse
mechanism when revocation doesn't happen? 

Like it or not, trust is a political issue, and something the IETF is
particularly inept at dealing with. The only way this works at scale is to
have well established international legal process involved in the key
infrastructure. There may be technology issues involved in making signature
checking a <=1 click process, but without the appropriate political
infrastructure in place, there is no value in doing that work. 

It is time for the IETF to move past the adolescent petty BS, and involve
the ITU in developing a workable PKI, from both the technical and political
perspectives. 

Tony 

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert
> G. Brown
> Sent: Friday, February 20, 2004 6:32 AM
> To: Iljitsch van Beijnum
> Cc: Vernon Schryver; [email-masked]

---

From: jfcm <[email-masked]>
Date: Fri, 20 Feb 2004 23:14:10 +0100

I am interested to know if someone/group works/developped on a simple fast 
and reliable remote memory server link protocol? The need I investigate 
concerns small size secure transactions to read and store formated data. 
Thank you.
jfc

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 21 Feb 2004 16:02:26 +0100

On 20-feb-04, at 15:32, Robert G. Brown wrote:

> out.

Well, I don't know about you, but I _do_ get mail from a small, closed 
group of people. I also get mail from a large, open group of people, 
and I grant you authenticating them is too much trouble and leads to 
too little gain to bother. But I think authentication mechanisms would 
be very useful for interactions that fall between those two groups. For 
instance, just now I got a few more spam messages that claim to be 
security notices from my ISP. It's pretty obvious these aren't, but 
that's not always the case. I don't understand why ISPs and ecommerce 
sites such as Amazon don't sign their mail. Or closer to home: the 
IESG, IAB, RFC Editor and so on.

> I get mail from friends of friends, from relatives who
> Outlook).

While the math behind public key cryptography is pretty intense, and 
the protocols aren't too pretty look at, the way it is applied doesn't 
necessarily have to be all that hard to understand.

Obviously implementations (and some of the protocols) need to mature, 
but this is happening as we speak. For instance, Apple's mail 
application automatically handles incoming S/MIME signatures without 
the need for the user to do anything special. Microsoft has a system in 
place where they authenticate certain software.

> Most of this is mail that I could care less about being

With Apple Mail there is a big problem of how to get a certificate, but 
once you do it simply works unless something bad happens and then a big 
old yellow banner pops up warning the recipient that the message 
couldn't be authenticated.

> overwhelmingly negative.

Hm, this is basically what happens for HTTPS... Seems to work most of 
the time.

---

From: "Robert G. Brown" <[email-masked]>
Date: Mon, 23 Feb 2004 09:06:12 -0500 (EST)

> Hm, this is basically what happens for HTTPS... Seems to work most of 

For roughly $100/certificate/year, sure.  But nobody registers
certificates for all their hosts, just their server(s).  And clients
don't need or get certificates -- you are validating/encrypting based on
a one-sided connection.  And you still have lots of people who, when
they get the little box that says that the host key isn't recognized and
should you connect anyway say "sure".  Please, just take my VISA card
number and shoot me now...

Now scale that by maybe 50-100 unregistered clients per registered
server (maybe more -- an entire organization might have only a handful
of servers and thousands of workstations or personal computers).  Now
scale THAT by anywhere from 1-10 people per station/computer, globally
(or start at the other end and simply count the number of entities that
can send or receive electronic mail -- hundreds of millions, surely).

What this adds up to is money.  Real money.  At $100/year/entity tens of
billions of dollars a year.

It makes sense, nay, it is essential to secure a connection between a
random client and a place of internet commerce or business with value.
Credit card and personal data is "valuable" and the cost of
certification is trivial compared to the gain at the corporate level.

But, how many individuals are going to pay this kind of money to secure
their personal email?  Especially when they can secure it NOW, for free,
with readily available (if not always well integrated) tools, and only
secure it where it really matters and not bother when it is a message to
a coworker down the hall that you want to meet them for lunch in fifteen
minutes?

So I agree, I agree.  It would be lovely to have a universal
certification mechanism not unlike DNS or the existing SSL key
certification agencies that would scale to the same kind of level that
DNS scales to today -- a keypair for every human on the planet that uses
email or other key-driven network services, uniquely tied to the
individual, and accessible GLOBALLY via a directory service.

Am I the only person who thinks that this a) would be hell to engineer
so that it would work at all; b) would have an "interesting" impact on
civil liberties? On the one hand, in principle it would let everybody
encrypt everything and increase privacy.  On the other, the mind reels
at the thought of all the new avenues of attack and abuse this would
open up -- key server spoofing, NSA-driven decryption attacks, the
continuing vulnerability of private keys on otherwise poorly secured
systems.  And some people don't WANT to publish directory information at
all -- one side effect of registering all humans and email entities on
the planet is that ALL THEIR ADDRESSES become immediately accessible via
lookup functions, or the system cannot work.

Encryption is a good thing, but it also offers a somewhat illusory
privacy.  At the moment I KNOW my email and computer are not secure, so
I do not send the details of my cocaine transactions and terrorist
activities out in cleartext on open lines, nor do I store them on a hard
disk.  Both are easily compromised.  Ask both the Japanese and the
German high commands of WWII -- they'll tell you that historically
"unbreakable" encryption schemes inevitably aren't, and Moore's law plus
the advent of cluster supercomputers makes this doubly true today.

I'm willing to be convinced that this is feasible at the scale of
billions of users (the design would need to in principle be able to
accomodate "all" of the inhabitants of the globe, and all their
automated agents or corporate entities capable of sending or receiving
mail -- say 2^33 to 2^36 addresses, so better use a 64-bit or better
UID) with terabytes of associated data in the full registry database.
However, Duke has had plenty of problems just flattening out its UID
space on an INSTITUTIONAL level, with maybe 20-30K total users (many of
them highly transient, many of them "entities" with different identities
in different contexts, e.g. a single student might have 3 or 4 accounts
by which they are known and can login or receive main in different
departments or networks).  This is a relatively controlled, relatively
well managed environment.  Trying to extend this to the entire Internet
seems like madness to me, at least at the moment.  At the very least,
come up with something first that can manage an entire University sized
domain, is built on fully open source and is not proprietary, is secure
"enough" in actual application and with bad guys hammering on it (and
not just in theory), and from what you learn there MAYBE you can come up
with a way of scaling it out to multiple organizations and eventually
the world.

   rgb