Subject: [Asrg] filtering at connect time

From: Michael Marking <[email-masked]>
Date: Mon, 3 Mar 2003 21:28:41 -0800 (PST)

One problem with text filters of various kinds is that the spammer is
unaware that the spam won't be delivered. There is no incentive for the
spammer not to continue to send mail to an address which is filtering,
possibly even discarding all such spam.

Blacklists, at least those which operate at the time connection is made,
at least offer some notice to the spammer that resources are being
wasted. They also allow the possibility of an additional legal measure:
it could be legislated that a spammer which continues to send to an
address after some proper connection refusal might be guilty of a theft
of service of some kind, which could have civil and criminal penalties.

At least one hashcash system (http://ssl.dizum.com/hashcash/) adds a new
header to the e-mail:

	X-Hashcash: <token>

This has a similar problem to that of text filtering: the spammer isn't
aware that the mail is unwelcome.

Properly implemented, an ephemeral address (expiring after a short
period), a single-user address, or a signed address can work at the time
connection is made or attempted, giving notice to the spammer that the
mail is not wanted, and reducing resource use in subsequent operations.

It should be possible to combine these mechanisms in a way which works
with current mailers, and still preserves some compatibility in case the
protocols are extended. For example, the current systems allow mail
addresses of the form

	<username>@<dnsname>

These addresses are seen at the time SMTP connections are established,
in MAIL and RCPT messages. Suppose we define some new tokens, so an
address becomes

	<prefix><mechanism><value><separator><username>@<dnsname>

This should be done in such a way that the combination

	<prefix><mechanism><value><separator><username>

is a valid <username> by itself.

<prefix> can be some unusual sequence, and can even be done on a domain-
by-domain basis. Suppose we use "AAA", assuming that no ordinary
username in our domain begins with "AAA".

<mechanism> is the type of additional information used. For example, we
might have

	HC	hashcash
	EP	ephemeral prefix
	SU	single user address

and so on.

<value> is whatever is required, as long as it is not the separator, and
consists of valid username characters.

<separator> can be a simple sequence; assume an underscore for this
example.

An address with an added hashcash value might be

	[email-masked]

This could be deemed equivalent to adding the header

	X-Hashcash: 01234567

to the body of the e-mail, where the "normal" recipient might simply be
[email-masked].

Of course, this is clumsier than changing the protocols, but it can be
made to work by simple modifications to existing software. It is fairly
easy to get the MAIL and RCPT addresses at connect time; that is how
most blacklists work. In time, there could be a migration to a better
protocol, but this could work quickly as in interim solution, and
wouldn't break existing software.

The error message passed back to the sender could make it clear what
would be required for the mail to go through. (The sender might be
directed to prefix a hashcash value, use an ephemeral address, or use
a one-time key.)

An ISP or systems administrator could implement this automatically
without work by the users: the prefix/mechanism/value could be added and
parsed by the MTA without the user seeing it. The policy could be
selective, based (for example) by finding the sender on a blacklist or
greylist, or by being able to authenticate the sender. The MTA might in
some circumstances learn the mechanism/value by looking through the
headers (for X-Hashcash, for example).

The sender's MTA could also implement automatic recovery. If an e-mail
were rejected for lack of postage (hashcash), the MTA might
automatically retry sending the message after computing the proper
value, and the sender need not be aware of the operation.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Mark Delany" <[email-masked]>
Date: 4 Mar 2003 05:39:27 +0000

On Mon, Mar 03, 2003 at 09:28:41PM -0800, Michael Marking allegedly wrote:

> made to work by simple modifications to existing software.

What about using esmtp-params?

Mail From: <[email-masked]> hashcash=01234567

It's codified in rfc2821.

Regards.

---

From: wayne <[email-masked]>
Date: 04 Mar 2003 00:00:13 -0600

> possibly even discarding all such spam.

Spammers don't care about undelivered spam.  Most of the time, they
won't even know because they use open proxies, forged from addresses
and forged reply-to addresses.

It is not worth the spammer's time to filter out bogus email
addresses.  Spamming is so cheap, that they would need to filter out
tens of thousands or even hundreds of thousands of bogus email
addresses per hour to make it worth their time.

-wayne

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 08:24:53 -0500 (EST)

On Mon, 3 Mar 2003, Michael Marking wrote:

> unaware that the spam won't be delivered.

Not necessarily.  Text filtering and how you handle rejection are
orthogonal; you can use (for example) Sendmail's Milter API to
fail MAIL FROM:, RCPT TO: or DATA commands.

> aware that the mail is unwelcome.

Not necessarily; you can fail the DATA command.

I would really like to see other MTA software (qmail, Postfix, Exim)
implement "real-time" filtering during the SMTP conversation.  As far
as I know, Sendmail is the only MTA that does this currently.

--
David.

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 08:26:05 -0500 (EST)

> and forged reply-to addresses.

However, there is a benefit to replying to spam with a 554 SMTP code.
If the spammers have been using an open relay, the open relay will generate
a bounce message that typically bounces, causing mail to go to
postmaster@open-relay.  This provides some incentive to close open relays.

---

From: Craig Hughes <[email-masked]>
Date: Tue, 4 Mar 2003 07:05:33 -0800

Correct.  In fact, most spam filters do not affect mail routing 
themselves at all, they rely on the integration into the MTA paths to 
deal with re-routing.  They are in the process of being integrated more 
substantially into the MTAs/MUAs though, and with those integrations, 
they are beginning to affect routing as well.  This is basically being 
driven by "easy of use" considerations, where an admin doesn't want to 
have to spend several hours figuring out how to re-write their 
sendmail.cf to integrate SpamAssassin having downloaded some tarball 
from a website -- they want to just double-click install.exe on their 
Exchange server and have spam automatically stop getting through.  At 
that point, the default configuration from the vendor has to be 
thinking about the issues which Michael mentioned.

C

---

From: [email-masked] (Justin Mason)
Date: Tue, 04 Mar 2003 17:20:35 +0000

One issue BTW that filter-during-SMTP-transaction systems have to think
about, is that if you teergrube or tempfail on an entire ESMTP connection
basis, this could result in affecting legit mail -- since legit mail may
be mixed in with spam, if (a) the spam was delivered to a mailing list, or
(b) the spam was sent to your secondary MX at around the same time as
some legitimate mail was.

David, I'm sure you've already considered this in the RoaringPenguin stuff
;)   But this technique is becoming more widely implemented, so it's worth
noting for the folks who haven't. ;)

--j.

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 12:52:38 -0500 (EST)

On Tue, 4 Mar 2003, Justin Mason wrote:

> basis, this could result in affecting legit mail

We do it on a per-message basis.  We only use tempfail codes; we don't
attempt to tarpit.  Tarpitting can open your machine up to DoS attacks,
because even a tarpitted process consumes some resources on your end.

Yes, sometimes legit mail is held up, but thanks to the fine authors
of SpamAssassin :-), that doesn't happen too often.

---

From: [email-masked] (Justin Mason)
Date: Tue, 04 Mar 2003 18:01:25 +0000

> because even a tarpitted process consumes some resources on your end.

As a matter of interest, have you collected any stats on the effectiveness
of tempfailing suspected spam?   I hear it's very effective at blocking
spam from spamware which doesn't support requeue-and-retry, but I have
no idea how effective "very" is. ;)

> of SpamAssassin :-), that doesn't happen too often.

Cheers! ;)

--j.

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 13:28:24 -0500 (EST)

On Tue, 4 Mar 2003, Justin Mason wrote:

> of tempfailing suspected spam?

The stats are online:

	http://www.roaringpenguin.com/canit/reports.php?what=hit-n-run

(You have to authenticate as demo/demo first.)  I tempfail all mail
from unknown senders the first time -- this happens at the RCPT TO:
phase, because I tempfail first-time sender-recipient pairs.

If you look at:

	http://www.roaringpenguin.com/canit/stats.php

you'll see that so-called "one-shot" messages make up a large percentage
of the daily message volume -- around 20-25%.

By the way, tempfailing senders the first time (or better yet,
tempfailing on the basis of sender-recipient pairs) is a solution that
fits transparently[*] into the existing e-mail infrastructure, uses
very few resources, and is quite effective.  I highly recommend it.

Regards,

David.

[*] Except for delays the first time a stranger sends you e-mail.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Mark Delany" <[email-masked]>
Date: 4 Mar 2003 19:27:36 +0000

> very few resources, and is quite effective.  I highly recommend it.

How do you differentiate between list-washing and spam delivery attempts? It's
an anecdote, but about half of the email to invalid sender-recipient pairs I
see appears to be testing addresses (in alphabetically order no less) without
issuing a DATA command.

If you issue the deferral before the DATA phase, then your 20-25% needs
to consider the list-washers/address testers.

Nonetheless, very interesting data.

Regards.

---

From: Brad Templeton <[email-masked]>
Date: Tue, 4 Mar 2003 11:30:06 -0800

> If the spammers have been using an open relay, the open relay will generate

More to the point, while spammers indeed don't care about getting
bounces on underlivered spam, it is simply very poor software
engineering to design a system that simply drops mail on the floor
and informs nobody who would care about it.  Any false positive in
such a system is a lost piece of mail with no diagnostic to tell
people about it.

Obviously the volume of spam makes it inappropriate to report errors
one email at a time the way you would for one to one mail.  So the
challenge is to build error aggregation systems which allow people
to view digests or statistics on errors that are relevant to them.

(Ideally you also store the mails somewhere that they can be
fetched by the target if desired.  Certainly disk is cheap enough
that if you have the message in ram, you can store it to a database
for a week or so for fetch-by-url.  However, if your technique
depends on blocking the mail before it has even finished arriving,
this is difficult, though you can still store as much as you did
receive.  Simply appending a block of data to disk is not so expensive
an operation.)

Reporting errors back to legit bulk mailers -- again, in aggregate --
is easy because they use legit headers and a unified from.  In
fact most MTAs do this already.

Giving up the principles of good software engineering and protocol
design, so that messages don't vanish into black holes with no
diagnostics should be one of our last-resort choices.  There are many
stories from the anti-spam wars of people whose (valid) mail was just been
silently discarded who only found out about it by accident.

And let's not forget that open relay operators are victims of the
spammers.  It may be necessary to close those relays, but they are
still the victim, not the perp.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 14:57:48 -0500 (EST)

On Tue, 4 Mar 2003, Brad Templeton wrote:

> one email at a time the way you would for one to one mail.

Right.  That's why you want to go to great lengths to force the
sending relay to generate the bounce.  All you want to do is return a
5XX code.

Once you've accepted the spam (that is, returned a 2XX code after DATA),
the spammer has won.

---

From: "David F. Skoll" <[email-masked]>
Date: Tue, 4 Mar 2003 15:05:05 -0500 (EST)

On Tue, 4 Mar 2003, Mark Delany wrote:

> How do you differentiate between list-washing and spam delivery attempts?

I don't, and I don't care.  Either way, it fails at RCPT TO:

However, grepping through my mail logs, I don't see any obvious
dictionary attacks.  I know other users who have reported this, but I
don't see it.  Besides, this technique has the pleasant side-effect of
stopping dictionary attacks too, unless the attacker makes a second attempt
for each recipient address.  Also, by watching the size of your "one-shot"
database, it quickly becomes apparent when you're under dictionary attack,
and you can block the offender.

Regards,

David.

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 4 Mar 2003 13:17:14 -0700 (MST)

> From: Brad Templeton <[email-masked]>

> ...

All conformant MTAs report errors, but not really in the aggregate, and
that is a good thing.   Legitimate bulk mail senders need to know
which addresses are broken to clean their lists.

> silently discarded who only found out about it by accident.

true.

> still the victim, not the perp.

That is not right.  People who operate relays may be less culpable than
the original senders of spam, but they are certainly blame worthy
and they are certainly not victims in the same class as spam targets.
Relay operators can suffer greater harm tham individual spam targets, but
relay operators must be held accountable.

Besides, it is impossible to distinguish willful spam relaying from
other sorts.  Spammers have been hiring relays since soon after the
beginning when Spamford recruited what he called his bandwidth partners.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Keith Moore <[email-masked]>
Date: Tue, 4 Mar 2003 19:51:11 -0500

> People who operate relays may be less culpable than

I don't see anything inherently wrong with operating an open relay, as long as
they don't alter messages and they're willing to pay for the bandwidth they
use.  It happens that we only have extremely poor mechanisms for tracking down
spammers, and that open relays defeat one of those mechanisms.  But the real
problem is not open relays and never was - it was the lack of tracability and
the corresponding lack of any means of making originators accountable for
their actions.  Shutting down open relays hasn't been a significant deterrent
to spammers, but it has cost us valuable functionality.  At most it's been a
needless distraction.

Keith

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 05 Mar 2003 10:05:56 -0500

> use.

Right.  There's nothing inherently wrong with operating one.  But the 
vast majority of cases are where they're inadvertent, and would be 
closed if the owner knew about what they're being used for. Furthermore, 
there's nothing inherently wrong about blocking email from them given an 
informed choice on the part of the recipients.

Yes, it's a loss of a specific means to a specific end.  But so are 
doors without locks.  Times change.  And these days, there's no need to 
run a "real" open relay because there are other techniques to obtain the 
same result. Doors _with_ locks.

---

From: Keith Moore <[email-masked]>
Date: Wed, 5 Mar 2003 10:27:37 -0500

> > I don't see anything inherently wrong with operating an open relay,
> > as long as they don't alter messages and they're willing to pay for
> > the bandwidth they use.
> 
> closed if the owner knew about what they're being used for.

perhaps.

> Furthermore, there's nothing inherently wrong about blocking email
> from them given an informed choice on the part of the recipients.

assuming, of course, that the recipients make the choice, and that they
have the expertise to do so.  neither of which tends to be the case.

Keith

p.s. I know several people who refuse to lock their doors.  They believe
that living in fear, and over-attachment to material things, are both
harmful.  As it happens I do lock my doors.  But I respect their right
to not lock theirs, and I don't think they're causing harm by not doing
so.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg