Subject: Re: [Asrg] A New Plan for No Spam / Velocity Indicator

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Sun, 27 Apr 2003 16:41:15 -0700

I don't see the validity of the argument.

The problem is mta filtering, so the mua issue is moot. If the message gets
to the mua this problem is solved.

Equaly people who upgrade their mta filters are going to upgrade their mta
if that is required.

I doubt that 2822 would have specified anything that broke legacy systems.
So it appears to me that we have a solution here.

Phill

 -----Original Message-----
From: 	J C Lawrence
Sent:	Sun Apr 27 16:15:18 2003
To:	[email-masked]
Subject:	Re: [Asrg] A New Plan for No Spam / Velocity Indicator 

On 27 Apr 2003 21:49:30 +0000 
Mark Delany <[email-masked]> wrote:
> On Sun, Apr 27, 2003 at 10:02:04AM -0700, J C Lawrence allegedly
> wrote:
>> On Sun, 27 Apr 2003 17:32:39 +0100 Jon Kyme <[email-masked]>
>> wrote:

>>>> My native suspicion is that RCPT TO bundling is going to have to
>>>> go.  Certainly for my own systems I'm more or less compleatly
>>>> disabled it via outbound rewriting via TMDA which unrolls any RCPT
>>>> TO bundles (only hold out is Mailman, and that only until I upgrade
>>>> to 2.1 and VERP).
 
>>> That's a shame, particularly given that the RFCs require that I must
>>> accept 100 rcpts.

>> Oh, I don't think it will be removed from the SMTP spec, but that
>> implementation of consent mechanisms will have the side effect of
>> causing single RCPT TO per message.

> Why? Couldn't consent information be sent as part of the RCPT TO:
> using the <rcpt-parameters> as defined in RFC2821?

Because the consent information needs to be encoded in a fashion that is
a) supported by currently deployed MTAs, and b) will be transparently
and auditably communicated to MUAs.  There are loads of approaches, like
RCPT TO encoding that could be used if we were also able to
significantly change deployed software both at the core and the edge.  I
don't see that happening.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
[email-masked]               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: J C Lawrence <[email-masked]>
Date: Sun, 27 Apr 2003 20:20:55 -0700

On Sun, 27 Apr 2003 16:41:15 -0700 
Phillip Hallam-Baker <Hallam-Baker> wrote:

> I don't see the validity of the argument.  The problem is mta
> filtering, so the mua issue is moot. If the message gets to the mua
> this problem is solved.

I disagree rather explicitly.

Spam is an inherently subjective definition that can easily be
subjective to and specific to a particular message.  More simply in
defining consent, it is the user of the MUA who defines, communicates
and arbitrarily waives or enforces his consent rules, not the MTA.  The
MTA can be an agent of the human, but the human trumps, and in all cases
is the final arbiter.  The human user might communicate his consent
rules to the MTA for more efficient processing, but the root remains the
human behind the MUA, not the service provider's.

These aspects convince me that consent definition and enforcement has to
occur at the MUA level.  Its the closet point to that fickle and
technology-ignorant luddite human.  Certainly the rules etc can be
communicated up to the upstream MTA for efficiency, but that's not
actually required for correct operation -- its just an attractive
option.

> Equaly people who upgrade their mta filters are going to upgrade their
> mta if that is required.

My current thoughts lead me to classify end users into two camps: Those
that can natively support a formalised consent definition, and those
that can't.  As that choice, as above, is to be made at the MUA level,
the only requirement of the MTA then is that it be configured such that
the consent expression "payload" (for want of a better term) can
actually get thru and reach the end user's MUA for appropriate
processing.

I've been beating on this particular point for weeks, and the only
reliable transport I can come up with is variants of plus addressing,
but I'll post more on this soon as I finish scraping time together for a
consent token protocol proposal which is layered atop SMTP and RFC 2822
without violating either.

> I doubt that 2822 would have specified anything that broke legacy
> systems.  So it appears to me that we have a solution here.

The problem is how a consent token expressed at initial MTA delivery
time can be received, extracted, verified, and analysed, umpty MTA
transitions later, perhaps on a disconnected system, when it finally
reaches the MTA immediately upstream of the human user (ie the one the
human user has the ability to communicate his consent preferences to).

Unless I'm blind or careless (very possible) I don't see that there is
any currently standard way for extended RCPT TO semantics to be encoded
in the resultant message or its envelope for subsequent downstream
receipt and processing, or that one could be defined and deployed
without requiring widescale NTA __and__ MUA upgrading.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 28 Apr 2003 20:30:09 +0200

On 2003-04-27 20:20:55 -0700, J C Lawrence wrote:
> and arbitrarily waives or enforces his consent rules, not the MTA.

I agree with that.

> technology-ignorant luddite human. 

One problem with those fickle and technology-ignorant luddite humans is
of course that they don't want to be bothered with "technical details"
like defining rules and expressing consent. They don't want to see any
spam (not even in a special Junk folder, if at all possible), they don't
want to lose any legitimate mail, and they expect that to "just work"
automatically. The ISP (or IT department) should (possibly using a
crystal ball or telepathy) filter out exactly the unwanted mails. 

Ok, that might be a bit exaggerated, but I think any mechanism which is
visible to the end user must be extremely simple. If they have to click
on a button once every hundred mails or so to inform "the computer" that
it has made a mistake, that's ok. Anything more complicated (like
deciding whether you want to whitelist an individual or a mailing-list,
or which BLs or SpamAssassin rules should be used) or work (like
C/R-systems) will probably be ignored.

(We will probably upgrade our users to Mozilla 1.4 when the final is
released, I am curious whether users will use the bayesian filters)

> Certainly the rules etc can be communicated up to the upstream MTA for
> efficiency, but that's not actually required for correct operation --
> its just an attractive option.
> 

If it has to get to the MUA, I don't think it is necessary or even
desirable to make the "consent expression payload" explicit in the SMTP
dialog. There could be an email header (e.g., Consent-Token:), which the
MUA can check before presenting the mail to the user. No change to
existing SMTP servers is necessary since they (usually) don't throw away any
unknown header lines in the messages they transport. OTOH,
plus-addresses are frequently mangled by forwarding or alias handling. 

However, I think rejecting unwanted messages immediately (instead of
accepting them and later generating a bounce or silently dropping them)
is preferrable for two reasons:

1) If I send somebody mail, I want to be notified if there is a problem. 
   Therefore, I think silently dropping mail is bad.

2) Because of forged sender addresses, bounces for spam are often not
    deliverable, so the spam ends up as double bounces in the
    postmaster's mailbox.

So the MTA should be able to check the consent token. This is certainly
possible even if it is contained in the message headers, as a message
can still be rejected at the end of the data phase. It "only" causes
additional traffic (not much, AFAICS: Most spams are short, and proxies
often send the whole smtp transaction without checking for 5xx replies).
The way the MUA can communicate with MTA to tell it which consent tokens
are valid and which aren't is a different (and non-trivial) problem, of
course.

	hp

-- 
   _  | Peter J. Holzer    | Latein ist das humanoide Äquivalent
|_|_) | Sysadmin WSR       | zu Fortran.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |    -- Alexander Bartolich in at.linux

---

From: J C Lawrence <[email-masked]>
Date: Mon, 28 Apr 2003 11:54:10 -0700

On Mon, 28 Apr 2003 20:30:09 +0200 

>> Certainly the rules etc can be communicated up to the upstream MTA
>> for efficiency, but that's not actually required for correct
>> operation -- its just an attractive option.  =20 > Equaly people who
>> upgrade their mta filters are going to upgrade their > mta if that is
>> required.  =20 My current thoughts lead me to classify end users into
>> two camps: Those that can natively support a formalised consent
>> definition, and those that can't.  As that choice, as above, is to be
>> made at the MUA level, the only requirement of the MTA then is that
>> it be configured such that the consent expression "payload" (for want
>> of a better term) can actually get thru and reach the end user's MUA
>> for appropriate processing.  =20 I've been beating on this particular
>> point for weeks, and the only reliable transport I can come up with
>> is variants of plus addressing, but I'll post more on this soon as I
>> finish scraping time together for a consent token protocol proposal
>> which is layered atop SMTP and RFC 2822 without violating either.

> desirable to make the "consent expression payload" explicit in the
> SMTP dialog. There could be an email header (e.g., Consent-Token:),
> which the MUA can check before presenting the mail to the user. No
> change to existing SMTP servers is necessary since they (usually)
> don't throw away any unknown header lines in the messages they
> transport. OTOH, plus-addresses are frequently mangled by forwarding
> or alias handling.

Encoding consent tokens in additional headers has a fairly simple
problem replies to sent messages will (almost guaranteed) not carry the
consent token.

  Bubba emails Boffo for help.

  Boffo replies.

  Bubba should receive Boffo's mail as de-facto consensual.

Ditto for transits across .forwards, aliases, mailing lists and courtesy
copies.  There are ways around this, for instance by encoding consent
tokens into Message-IDs and then looking for them in In-Reply-To and
References: headers, but this of course breaks for those MUAs that don't
generate In-Reply-To: or References: headers...

Next this doesn't at all address the issue of email relationships
established outside of SMTP, such as an email address entered on a web
form in the, "Please send me more information!" field, or even mailing
list subscriptions.

> 1) If I send somebody mail, I want to be notified if there is a
> problem.0 Therefore, I think silently dropping mail is bad.

There a great tendency to attempt to view the spam scene in terms of
rights and expectations.  What rights and expectations does a sender
have?  What rights and expectations does a recipient have?  What are the
guarantees?

Doesn't make sense to me.  Quite obviously senders are going to send
anything they damn well please; they have a wire and an urge to drop TCP
packets into it.  Equally obviously recipients are going to filter; they
have a wire and an urge to stem the flood of packets.  Senders may
cooperate with recipients in order to help get their packets thru, but
they have no obligation to.  Conversely, recipients have every
motivation and necessity to make their filtering mechanisms, of whatever
draconian and arbitrary form, as effective and annoyance removing for
themselves as possible.  This is pretty standard Darwinian stuff.

> deliverable, so the spam ends up as double bounces in the postmaster's
> mailbox.

I've found that many (most?) sites are configured to silently drop
double bounces.  Too noisy and not worth anybodies time.

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 28 Apr 2003 15:10:50 -0600 (MDT)

> From: J C Lawrence <[email-masked]>

> ...

Slightly to the contrary, since the very start of email spam I've found
that double bounces are useful in some ways.  One is for samples of 
unmitigated spam from soon to be blocked-on-sight-with-blackhole-routes
spammers. 

A second use of double bounces is to detect defunct internal addresses.
ISPs and small organizations don't have this problem but larger outfits
often have groups creating and deleting accounts without coordinating
things with the outer firewalls, MX servers, and so forth.  The
"backscatter" provided by spam is a handy check.

A third use is for potent spam traps.  For various reasons, spammers
often latch onto never-valid addresses, and spam them forever.  If
you sample double bounces, you're likely to find some good spam traps
for your body filtering systems, whether like the DCC or for training
"Bayesian" filters.  For example, anyone with a short username that
has been exposed for a while probably has not only it but it without
the first or last letters on those "30,000,000 address" CDROMs.  
"vj" has never been valid near me in the last 20 years at various
commercial domains, but it is one of my best spam traps.

When spammers give you manure, make compost.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 29 Apr 2003 13:01:21 +0200

> problem replies to sent messages will (almost guaranteed) not carry the

I thought it was your idea to modify the MUAs :-)

Seriously:

There are two problems: 

1) If Alice wants to send mail to Bob, she has to include the proof that
    Bob has consented to receiving mail from her in such a way that it
    actually reaches Bob's MUA.

2) If Bob wants to get mail from Alice, he has to send his "invitation"
    to her in a way she can actually use.

Ad 1)

    An additional header will get through almost always, as long as only
    mail transport (.forwards, aliases, mailing lists, ...) is involved.
    As soon as a MUA get's in the way (say the mail is manually
    forwarded) there is a good chance that unknown headers are lost and
    even the known headers are mangled.

    The RCPT TO: address is frequently rewritten at mail gateways. 

    Additional parameters to RCPT TO are not passed on by MTAs that
    don't understand them.

    The To: header is rarely, but sometimes rewritten. More importantly,
    for mailing lists, it usually doesn't contain the address of the
    final recipient, but that of the mailing list.

    The In-Reply-To: header and References are mostly equivalent to some
    new header in terms of "getting through". They have, however, a
    specific meaning - including a reference to an unrelated mail just
    to prove that we had mail contact before seems wrong to me.

Ad 2)

    An additional header will be ignored by any MUA which doesn't
    recognize it. So that requires both Alice and Bob to use a
    MUA which knows about consent tokens.

    A consent token encoded into the From: address will probably end up
    in the To: header and the RCPT of the sent mail. But note that many
    mailing-lists only accept mail from the address you subscribed with.

    A consent token encoded into the Message-Id will probably end up in
    the In-Reply-To or References header (though with lesser
    probability - even on this list there are people with MUAs incapable
    of generating these headers). But only if the mail is a direct reply
    to the mail in question. If the mail starts a new thread, it won't
    include such a header. Similarly, mails to a mailing list don't
    include an In-Reply-To header with the message id of the
    "subscribe" mail.

    A MUA which doesn't know about consent tokens, will of course never
    generate an extra parameter to a RCPT TO command.

In short, I don't think there is any scheme which won't break in at
least some current situations. So I think it will be necessary to 

* use several schemes in parallel

* change some of the infrastructure (over time)

* use the new schemes to whitelist mails, not to blacklist all others
  (because for at least some time, there will be lots of legitimate
  mails which won't comply to any scheme)

> Doesn't make sense to me. 

Call them "design goals", not "rights and expectations", then it makes
more sense. We are here to design a system which will dratically reduce or
maybe even stop spam, while preserving email as a useful tool. Stating
what properties of email are useful and hence worth preserving, and
which aren't useful and can be tossed is a necessary step.

To me, having a high probability that the mail was actually delivered
and will eventually be read unless I get a notification is useful. The
use of the bounce system has declined because of spam - today, if I get
neither a bounce nor a reply I don't know if the recipient has read the
mail and doesn't deem a reply necessary or if it was dropped into the
junk folder. But it is still useful, and I think that the reliability of
mail should be enhanced and not lowered by any measures we implement.

> double bounces.  Too noisy and not worth anybodies time.

I thought about doing this for some time. Occasionally they are useful
however.

	hp