Subject: Re: Why spam is a problem.

From: "Brian Bisaillon" <[email-masked]>
Date: Wed, 14 Aug 2002 12:50:28 -0400

Hello,

In terms of technology,

There must be a secure method that would allow a receiver to verify whether or not the sender actually exists as a user on the mail server for the domain the e-mail is coming from.

Advantages:

What I described above would allow mail servers to reject e-mail coming from randomly generated e-mail addresses and e-mail sent from anonymous remailers because the user has to exist on the mail server for the sender domain before the recipient's mail server is allowed to receive the message. So instead of receiving 1000 messages, the exchange above has to happen before any messages are received. This helps with the bandwidth problem.

Disadvantages:

What was described above does not prevent a user who actually exists on a mail server to send spam.

It also doesn't prevent a sender from intercepting packets from the receiver and forging a response back to say that the user actually exists on the system when in reality the user does not.

I am eager to hear everyone's comments on this.

---

From: Keith Moore <[email-masked]>
Date: Wed, 14 Aug 2002 14:50:40 -0400

> There must be a secure method that would allow a receiver to verify whether 
> or not the sender actually exists as a user on the mail server for the 
> domain the e-mail is coming from.

this is already possible.  it is not sufficient.

the recipient doesn't care whether the sender address exists, the recipient
cares whether he gets spam.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 15 Aug 2002 08:32:43 +0300 (EEST)

> this is already possible.  it is not sufficient.

It's possible but it's useless as one can't depend on it: too many MTA's 
are configured to refuse EXPN/VRFY requests if they were implemented in 
the first place.

-- 
Pekka Savola                 "Tell me of difficulties surmounted,
Netcore Oy                   not those you stumble over and fall"
Systems. Networks. Security.  -- Robert Jordan: A Crown of Swords

---

From: Andrew Rutherford <[email-masked]>
Date: Thu, 15 Aug 2002 15:30:19 +0930

>are configured to refuse EXPN/VRFY requests if they were implemented in

It'd still be next to useless if everyone did implement it and allow 
its use - it's not sufficient because checking if the email address 
is correct won't help you if the header is forged.

(A quick check on spam I've received today indicates around 70% has a 
forged envelope address - and of those around 15% have *my* address 
as the source.)

Also, in many cases there are security issues (real and imagined) 
with having an external mail relay machine (as part of a firewall 
system) know what addresses are and are not valid.

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 15 Aug 2002 00:17:54 -0600 (MDT)

> From: Pekka Savola <[email-masked]>

> are configured to refuse EXPN/VRFY requests if they were implemented in 

That might be why spammers don't use EXPN/VRFY but instead use Rcpt_To
to verify addresses in their lists.  If you watch an SMTP server that
gets much spam, you'll see a lot of SMTP transactions aborted after
Rcpt_To, even when the server answered with a 200-series status value.

I don't know which of various other mechanisms Keith Moore meant, but
I doubt he meant EXPN/VRFY requests or Rcpt_to, because all three
are wrecked by common uses of MX secondaries.

Note that "[verifying] whether or not the sender actually exists as
a user on the mail server for the domain the e-mail is coming from"
as stated does not make a lot of sense in the real world.  "The mail
server" suggests a single SMTP server per domain, which is often false.
"The domain the e-mail is coming from" suggests that there is something
wrong with sending mail from one ISP with a return address (envelope
and header From value) of a mailbox at some other ISP.  "Actually
exists as a user" suggests that aliases and forwarding are not kosher.
Then there are the complications of "virtual hosts."

Vernon Schryver    [email-masked]

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Aug 2002 09:53:21 -0400

> Rcpt_To, even when the server answered with a 200-series status value.

there's no way to know whether the verification is being done by a
spammer or for legitimate purposes.
 
> are wrecked by common uses of MX secondaries.

no I meant RCPT.  a 2xx response doesn't guarantee that the address
is valid, but a 5xx response is a reasonable assurance that the address
won't be able to receive mail.

> as stated does not make a lot of sense in the real world.  

in many (not all) cases it's fairly safe to assume that a message from 
an unreplyable address is not of interest to a recipient.  for instance, 
I use this to filter traffic that is sent to a mail robot autoresponder,
because there's no point in having the robot process a message and 
generate a response if the reply is going to bounce anyway.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 15 Aug 2002 09:02:03 -0500

> are configured to refuse EXPN/VRFY requests if they were implemented in 

And in the case of a spammer with his own domain, what's to stop him from
responding in the positive to everything? Useless.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 15 Aug 2002 09:18:30 -0600 (MDT)

> From: Keith Moore <[email-masked]>

> spammer or for legitimate purposes.
  
That may true in general, if you can figure out a legitimate purpose
for the hack.  I can't think of one that is not marginally abusive,
including your autoresponder.  It is marginally abusive because it
wastes the resourses of innnocents.

It is clearly false in about half the cases I see in my logs.  For
example, when I see a such an RCPT from well known spammers or followed
few minutes or hours later by spam from the same SMTP client, you know
what was going on.

I'm not sure what is going on in the many cases I see where such a
probe happens a few minutes after spam from the same SMTP client that
is clearly the spammer.

> generate a response if the reply is going to bounce anyway.

That's distinctly different from the other person's purpose.  
Knowing that an address is invalid is distinct from knowing that
it is valid.  In other words, how often do you figure you see 
a 2xx response to a RCPT command for an invalid address?  My guess
is at least 50% of invalid addresses will yield 2xx responses.

---

From: John Stracke <[email-masked]>
Date: Thu, 15 Aug 2002 11:17:52 -0400

Keith Moore wrote:

>an unreplyable address is not of interest to a recipient. 
>  
>
Definitely not all.  Counterexample: when Amazon sends out order 
confirmation messages, they say that the From: address will not accept 
mail.  (Personally, I think this is a bad thing, customer-service-wise, 
but that's another question.)

-- 
/===========================================\
|John Stracke      [email-masked]  |
|Principal Engineer|http://www.centivinc.com|
|Centiv            |My opinions are my own. |
|===========================================|
|Know your limits, then destroy 'em.        |
\===========================================/

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Aug 2002 13:43:54 -0400

> wastes the resourses of innnocents.

It doesn't waste their resources, because it improves the level of 
service that they see.    Giving someone immediate feedback that their 
address is invalid is far preferable to trying to send them a bounced 
mail message that will never arrive; and keeping our mail servers 
from being bogged down with bounces that will never get delivered 
helps free them up to deliver legitimate traffic.

However, just that because I see fit to use SMTP address validation 
in a few cases doesn't mean I use it everywhere, or that I recommend it 
as a general-purpose mechanism.... quite the contrary, I haven't made
much mention of it until now because it was clear that it had a lot of
potential for misuse, and that it was only a matter of time before it
would become less effective anyway.  For many years this was a nearly
perfect filter for blocking spam from mailing lists, not so any longer.

> a 2xx response to a RCPT command for an invalid address?  

I haven't tried to measure that.  For my purposes it is sufficient if
it provides early notification of a significant fraction of bad
email addresses, and it certainly does that.  

Keith

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 15 Aug 2002 11:53:16 -0600 (MDT)

> ...

That is the same sort of reasoning that spammers use to justify
their activities.

  - bounces would not bog down your mail servers, since you can discard
   or otherwise deal with bounces with no more "bogging" than you are
   spending poking at other people's systems.  This aint rocket science,
   because even the spammers that want to avoid bounces understand how
   to do that.  (Many spammers want to receive bounces to clean their
   lists, because some big ISPs and others automatically black-list
   SMTP clients after they've sent to too many bad addresses.)

  - a mechanism that could really determine that addresses are valid
   could be useful in a web page to provide immediate feedback to
   users.  The reality of MX servers, firewalls, sendmail "catch-all"
   maps, and other things make the RCPT command too unreliable to use
   for that purpose.

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Aug 2002 14:55:20 -0400

> That is the same sort of reasoning that spammers use to justify

it's also the same sort of reasoning that you used to justify 
sending this mail to the IETF list.

>    spending poking at other people's systems.  

for a variety of reasons, that's simply not true for the cases where 
I'm using it.

>    (Many spammers want to receive bounces to clean their

perhaps, but even the spammers who clean their recipient lists don't 
necessarily use valid return addresses, and it's the return addresses
that I'm checking.

>    maps, and other things make the RCPT command too unreliable to use

BS.  a simple syntax check isn't perfect either, but it's useful to 
catch some errors.  this catches some more errors, without exhibiting 
false positives.  it also has a cost, which is why I haven't 
recommended it for general use.  but that doesn't mean it is useless.

Keith

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 15 Aug 2002 13:34:55 -0600 (MDT)

> sending this mail to the IETF list.

You know that is a false statement.  You know that subscribers to this
list ask to receive all of the good stuff with the drivel, except for
the out-right spam and the messages from banned individuals.  You also
know that no one asks to have their SMTP servers do a useless cycle
in addition to receiving an auto-response.

I do agree I send too much mail to this list.  On the third hand, you
have me and everyone not in my filters beat by a factor more than 10.

> for a variety of reasons, that's simply not true for the cases where 

How did I guess you would see things that way?  It is, of course,
nonsense, because you could arrange to not receive any bounces just
as some spammers do.

> necessarily use valid return addresses, and it's the return addresses

That's nonsense that you would see as such if you were not stuck
on rationalizing that which you know is dubious.  Spamemrs who want
bounces to clean their lists do indeed use valid return addresses.

> recommended it for general use.  but that doesn't mean it is useless.

Charges of "BS" from someone who admits never having measure the
effectiveness of the mechanism are ironic.

I'll shut up now about this.

---

From: Keith Moore <[email-masked]>
Date: Thu, 15 Aug 2002 18:03:34 -0400

> You know that is a false statement.  

it's as true as your statement was, and for the same reason.

> You know that subscribers to this

they don't ask to have their DNS servers queried either.  nor do they ask
to have their SYNs ACKed.  actually very few users try to specify how a 
service should be implemented, they just want it to work well.

> nonsense, because you could arrange to not receive any bounces just

I *want* to receive some kinds of bounces, because they give me clues
about how well the system is working.  But I see absolutely no reason
to send large files to people who don't supply a valid return address,
nor to do database searches for them, nor to try to process commands
that they send to a command interpreter.

> bounces to clean their lists do indeed use valid return addresses.

You must get spam from different folks than I do, then.  If the spammers
use SMTP RCPT to cull bad addresses from their lists I'm not at all 
surprised, but most of the spam I see still comes from people who want 
to hide their tracks for the actual spam.  

> effectiveness of the mechanism are ironic.

No, you're trying to insist that I define "effectiveness" in your terms.

Keith