Subject: [Asrg] draft-duan-smtp-receiver-driven-00.txt

From: Kartik Gopalan <[email-masked]>
Date: Thu, 5 May 2005 18:51:50 -0400

We'd like to solicit feedback from the SMTP community on 
our recent IETF draft titled "Receiver-Driven Extensions to SMTP".

http://www.ietf.org/internet-drafts/draft-duan-smtp-receiver-driven-00.txt

The abstract of the draft is attached below.

As we are new to the IETF process, we welcome any suggestions 
pointing out potential improvements as well as deficiencies in the draft.

The following DiffMail project website and our related paper in SRUTI'05 
might also be of interest.

http://www.cs.fsu.edu/~duan/projects/diffmail/diffmail.htm

Thanks,
Zhenhai Duan, Kartik Gopalan, Yingfei Dong

Abstract:

The Differentiated Mail Transfer Protocol (DMTP) provides simple 
extensions to the Simple Mail Transfer Protocol (SMTP) that 
enable receivers to exercise greater control over the email delivery 
process. The current SMTP-based email delivery architecture is 
fundamentally sender-driven and distinctly lacks receiver control over
the message delivery mechanism. This document describes DMTP 
that enables receivers to classify senders into three categories -- allowed, 

denied, and unclassified -- and process the delivery of messages 
from each category independently. As is the current practice, receivers 
may directly accept messages from senders in the allowed
category and decline senders in the denied category. In addition, 
DMTP receivers require senders in the unclassified category to 
store messages on the senders' own mail servers. Such messages 
are retrieved only if and when the end receivers wish to do so. 
By granting greater control over message delivery to receivers and
imposing greater message storage and maintenance overhead on 
senders, DMTP provides significant advantages in controlling spam. 
DMTP also easily operates in conjunction with (but does not require) 
many currently deployed anti-spam techniques.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Jon Kyme" <[email-masked]>
Date: Fri, 06 May 2005 11:28:44 +0100

> http://www.cs.fsu.edu/~duan/projects/diffmail/diffmail.htm

I'm surprised you don't cite IM2000

---

From: Matthew Elvey <[email-masked]>
Date: Fri, 06 May 2005 11:44:50 -0700

Given the initial crosspost, I'm doing the same (though they may be 
rejected) and suggesting all follow-up go to one list, only.  I suggest 
asrg.

FEEDBACK:
Yes, this is ~= IM2000, but with an improvement: it's potentially 
sometimes backward-compatible with SMTP.
Generally, this draft (which I'll refer to as DMTP or DiffMail) has 
features (the draft lists 3 advantages*) that, IMO, are better provided 
by other systems.
Addressing them in reverse order:
3)This is better provided by greylisting+.  That forces the sender to 
maintain something like a real mail queue.
2)Standard use of DNSRBLs (and sometimes RHSBLs with CSV or SPF) reduces 
bandwidth and storage costs more effectively and is compatible with all 
senders and most receivers today, with no code changes, and generally 
require just trivial configuration changes.
1)Again, greylisting allows a receiver a window of opportunity to allow 
the sender to get on a BL (or the message to enter a razor2 type DB) 
before receiving the message.  It is true, however, that DMTP does 
provide a potentially larger window, and does so without indroducing a 
significant delay.  The question is, is this worth the cost of requiring 
senders and receivers support new commands and a new reply code?

Confused by acronyms? See
<http://wiki.fastmail.fm/wiki/index.php/ClientSmtpValidation#.23Acronyms.2FGlossary>
for elucidation.

+See <http://greylisting.org>

*Purported Advantages:
   "1. By asking senders (non-regular contacts) to maintain messages on 
their mail servers, spammers are forced to keep their servers up. They 
cannot simply send a large number of spam messages, shut down their 
servers, and switch to another domain (and/or change IP addresses). In 
this way, DiffMail helps to improve the effectiveness of 
IP-address-based filtering schemes.
   2. Since a (complete) message is only retrieved by a receiver at his 
will, less bandwidth and storage resource will be consumed at the 
receiver side if the user does not retrieve the majority of messages 
from non-regular contacts.
   3. Spammers now have more responsibilities to maintain their outgoing 
messages[.]"

On 5/5/05 3:51 PM, Kartik Gopalan sent forth electrons to convey:
<snipped>

---

From: Kartik Gopalan <[email-masked]>
Date: Sat, 7 May 2005 00:35:37 -0400

> I'm surprised you don't cite IM2000

Thanks for pointing out. Yes we are aware of IM2000 and have indeed 
referred to it as the closest work to DMTP in our tech report on the website 
above. In the draft, actually there isn't much reference to any 
prior work as we perhaps got too focused detailing DMTP. 
Yes, we will add related citations to the draft as well mentioning 
IM2000 and others efforts.

We had first set out to design a new architecture from scratch 
but soon realized that incremental deployability 
would be a key factor in success. In our understanding, the key 
differences between DMTP and IM2000 are

(1) DMTP tries to maintain the advantage of the sender-push model 
of SMTP where its beneficial (in communication with trusted MTAs) 
and resorts to pull-based model only when the credentials are 
suspect (with unclassified MTAs). With IM2000,
pull-based model appears to underlie all communications.

(2) DMTP provides an incremental extension to SMTP, can interoperate 
with legacy SMTPs and there is no need to change any other entities 
such as MUAs or POP3/IMAP servers. IM2000 appears to re-design 
all components from scratch with a completely new protocol.
Latter is not necessarily a bad thing, but DMTP's focus has been 
incremental extension to SMTP.

In fact DMTP could act as a bridge in transitioning from the 
current simple push-based model of SMTP to completely 
new models (like pull-based IM2000) -- at present there appears a 
great chasm in crossing over from SMTP to any substantially 
different architecture.

- Kartik

---

From: Daniel Feenberg <[email-masked]>
Date: Sat, 7 May 2005 08:04:41 -0400 (EDT)

> > > pointing out potential improvements as well as deficiencies in the draft.

Can you describe how this is superior to "greylisting", which, as I
understand it amounts to refusing mail from unclassified sources with a
"try again later" (TEMPFAIL) code the first time it is offered, but
accepting it if offered again several hours later?

Could greylisting be made even more like your plan if the receiving MTA
only accepted mail on the second try if it came from the same IP address? 

The overall effect - requiring the sender to maintain their server for
several hours before mail is accepted - seems to be about the same. The
advantage for greylisting is that most legitimate senders are already able
to handle receivers using this technique, whereas DMTP would require an
upgrade to all senders to comply. I observe that MTA vendors and FOSS
sources of MTAs are extraordinarily resistant to new procedures.

I understand that greylisting works now because spammers haven't found it
worthwhile to record success or failure, and don't come back after a
TEMPFAIL. Of course, if many receiving MTAs started to use the technique,
spammers might respond by keeping such records. In that case, DMTP (or
greylisting, for that matter) would be effective only insofar as
blacklists could be expected to pick up new spammer addresses in a matter
of hours. Is that feasible? My impression is that while it may be
feasible, it is made necessary only by the tendency of DNSBLs to drop
entries rather quickly. If blacklist entries for dynamic addresses are
made very persistent, it isn't necessary to pick them up quickly, they
will already be on the list from the last spam run days or weeks before,
in which case the DNSBL itself is sufficient, and DMTP or greylisting is
not much additional help.

Daniel Feenberg
feenberg isat nber dotte org

---

From: Kartik Gopalan <[email-masked]>
Date: Sat, 7 May 2005 09:27:17 -0700

Matthew,

Your comments on greylisting are interesting.
Personally I think that greylisting is indeed a very clever idea.
At the same time, there is no denying that it does overload the 
"TEMPFAIL" semantics of SMTP with a semantics that was probably not
originally intended for use in this manner in the first place. So in the
long term, we'd want to move towards a more systematic mechanism.
One other concern is that spammers will adapt to greylisting as well
sooner than later once it starts affecting their bottomline. Greylisting
is still a purely sender push-based model that leaves the receivers 
in a reactive mode.

DMTP differs from greylisting in how receiving MTAs
(RMTAs) react to "unclassified" sending MTAs (SMTAs).
(unclassified = neither in whitelist or blacklist).
In greylisting, unclassified SMTAs are asked to push 
the message again later. With DMTP, the RMTA pulls the 
message from unclassified SMTA only if and when the receivers
are interested. 

In addition to being (1) a systematic receiver-pull 
mechanism, as you noted, (2) DMTP reduces the
delivery latency for legitimate senders (a small intent message 
can be delivered immediately) and (3) provides a larger time 
window for blacklists to be effective.

That said, we did carefully consider using greylisting itself as part of 
the incremental deployment phase of DMTP. Specifically,
when a non-DMTP-compliant unclassified SMTA attempts
message delivery to a DMTP-compliant RMTA,  the SMTA can be
discouraged by use of greylisting-like mechanism. We still think
it is a highly feasible alternative to the challenge-response
based mechanism currently described in Section 3.6 of the draft.

- Kartik

On 5/6/05, Matthew Elvey <[email-masked]> wrote:
<snipped>

---

From: Kartik Gopalan <[email-masked]>
Date: Sat, 7 May 2005 13:52:35 -0700

Since the discussion is focusing on greylisting vs. DMTP, we'd
like to emphasize that DMTP does not exclude the use of
greylisting. Greylisting can also be used with DMTP as
follows: unclassified sending MTAs can be asked to retry
delivery of DMTP intent messages at a later time.
This combines the benefit of greylisting with the benefit of
DMTP's pull-based architecture.

> accepting it if offered again several hours later?

I mentioned this in greater detail in earlier reply to Matthew's email.
Essentially greylisting is still a sender-push model that still leaves
receivers in
a reactive mode. DMTP advocates a hybrid model : allow
message "push" from "whitelisted" senders, "pull" messages
from any unclassified sender and, of course, block the blacklisted ones.

> The advantage for greylisting is that most legitimate senders are already able
> upgrade to all senders to comply.

This is not correct. DMTP does not require DMTP-compliance from all senders
at once. There are dfferent incremental deployment paths (Section 3.6) each of
which can be pursued by different vendors independently. Currently the
draft describes a challenge-response based scheme, though it can also
be easily replaced by other sender-discouragement mechanisms (e.g. greylisting).

> of hours.

The last sentence is not true for DMTP. There is a basic difference.
Greylisting will accept the spam from unclassified senders on second 
attempt if DNSBL cannot be updated within a time interval. DMTP does 
not depend on updating DNSBL within some time interval. If sender is 
unclassified, DMTP resorts to receiver-pull mode (whether it is a first 
try or second or third...).

- Kartik

---

From: Zhenhai Duan <[email-masked]>
Date: Sat, 7 May 2005 19:23:54 -0400 (EDT)

Hi Daniel,

Kartik has pointed out a few difference between DMTP and greylisting. Here 
I want to add some follow-ups.

One tricker difference between DMTP and greylisting is the time window 
that Greylisting and DMTP provide for RBLs to detect to a spammer and 
black list it. For Greylisting, if spammers indeed retry, the time window 
is from when the first receiver reads a spam message in a retry (or more 
precisely, the first user reports the spammer information to RBLs) to the 
time the spammer finishes sending the last spam message in this round of 
retry (assuming a more or less simple, synchronized model). It is key to 
note that this time window is not the retry window (time between retries). 
However for DMTP, the time window is related to the user email retrieval 
behavior. The later a receiver tries to retrieve a message from an 
unclassied sender, the more likely that the sender has been black listed 
(if it is a spammer).

Another difference is how DMTP and Greylist populuate the whitelist. 
Greylist does this based on the history of a triplet and this is done to a 
large extent automatically. However, DMTP's whitelist is populated based 
on if receivers want to communicate with the senders (for example, 
someone may prefer to only directly communicate with regular contacts, no 
matter how others behave, bad or good). I am not arguing which one is 
better, but I do think we need to pay attention to the way how 
we populate the whitelist, when we fight spam as a community, as spammers 
will certainly also eye it.

Cheers,
  -Zhenhai
============================================
Zhenhai Duan 
Assistant Professor 
Department of Computer Science 
Florida State University 
Tallahassee, FL 32306-4530

Phone: (850) 645-1561
Fax:   (850) 644-0058
Email: [email-masked]
URL:   http://www.cs.fsu.edu/~duan
=============================================

---

From: Tony Finch <[email-masked]>
Date: Mon, 9 May 2005 22:19:09 +0100

This draft is worthless.

The fundamental idea that it shares with IM2000 is based on a mis-targeted
optimisation. The main cost of spam is the human time it wastes, whereas
disk space on the recipient servers is very cheap. In fact DMTP makes
dealing with poorly-classified email MORE time-consuming than it is
already, because a recipient must deal with a content-free notification
(containing only a forged sender address and a message ID) as well as the
actual message.

If you enhance notification messages so that they are sufficiently
informative for a recipient to be able to decide whether to retrieve the
bulk message, then spammers will just put the entirety of their payload in
the notification message. This has already occurred with certain kinds of
mobile phone spam.

When in doubt about the spamminess of a message it is better to receive
the whole thing. This allows the full gamut of classification technology
can be applied to it, so that there is a greater chance of it being
classified before it wastes the time of a person.

IM2000 and DMTP are also based on an incorrect assumption, that spam
messages actually cost space on the senders disks. Most spam software is
stateless, and it can continue to be in IM2000 or DMTP: the spammer just
encodes enough information in the message ID in order to be able to
reconstruct the message when it is requested.

As well as being based on a fundamentally useless idea, the details of the
specification are incompetent.

It fails to use the SMTP extension model correctly. A client indicates
that it wishes to use an extension by using a command from the extenstion
or a MAIL or RCPT parameter, NOT by appending a keyword to its EHLO
command. It shows a misunderstanding of the difference between the SMTP
envelope and the message header. It has inexplicable colons in its command
names, in contradiction to SMTP's requirement that commands are
alphabetic.

It makes assumptions about IP routeing which compromise the usability of
DMTP with highly scaled MTA clusters, which often involve NAT and/or
multi-homed servers.

The attempt at cryptography in section 3.4 is laughable.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. MODERATE OR
GOOD.

---

From: Kartik Gopalan <[email-masked]>
Date: Tue, 10 May 2005 00:52:24 -0400

Before replying to your comments specifically, we'd like to stress
again that, unlike  IM2000, DMTP classifies sender MTAs into three
categories: allowed, denied, and unclassified. The receiver pull model
is only used for messages from the unclassified category. Just as
usual, whitelisted senders can send mail using normal SMTP and
blacklisted ones are summarily declined, with no additional overhead.

> disk space on the recipient servers is very cheap.

DMTP holds the unclassified senders responsible for *managing* their
own outgoing messages. Storage  is just one component of this
management. DMTP strongly discourages the current
"fire-forget-and-disappear" behaviour of  spammers who now need to
wait (potentially indefinitely) for receivers to retrieve their
messages. This goal can hardly be called a mis-targeted optimization.
Now spammers would face the dilemma: They should online longer if they
want more people to read, on the other hand, the longer they stay
online, the more likely that they will be black listed.

> In fact DMTP makes

DMTP gives solution to the scenario that none of the currently  used
anti-spam technologies can satisfactorily handle without receiving the
entire message. DMTP intent messages are used ONLY for unclassified
senders that are neither in receivers MTA's whitelists nor blacklists.
 It is only unclassified senders who are treated using the
receiver-pull model. And even to this category you can apply the same
sender-discouragement schemes that are currently being applied -- such
as greylisting, any other challenge-response schemes, etc.

> the notification message. This has already occurred with certain kinds of

First, this is inevitable in *ANY* form of communication where you
wish to allow untrusted parties to communicate with you (or to express
their intent to communicate). It is just like when your phone rings
and you can't recognize the caller ID, you either ignore it or pick it
up anyway. If its a wrong number, prank call, or a telemarketer, you
simply hang up. So the problem is not specific to DMTP.

Secondly, DMTP requires a configurable upper limit on the notification
size. It is effective to the extent that it limits the content of SPAM
delivered (just as phone caller ID helps in making an initial
decision). If one wishes, one could place additional restrictions like
not permitting HTML tags in the notification to make spammer's life
more difficult.

> classified before it wastes the time of a person.

One can still apply the whole gamut of classification technologies on
any complete emails in DMTP. Handling intent messages for a small
unclassified category of senders will not be too much of trouble. This
can even be implemented as transparently as retrieving the complete
message when *and only when* the receiver issues the open command on
the intent message at his/her MUA (again simply like deciding whether
to press talk/ignore upon receiving a caller-ID on the phone).
 
> reconstruct the message when it is requested.

As mentioned earlier, the primary goal is to discourage the
"fire-forget-and-disappear" behaviour that the current sender-push
model of SMTP encourages. Storage overhead is only one small part of
the picture. The bigger advantage is that spammer MTA needs to be up
and running for potentially indefinite duration if it wants the
receiver to read the entire message. In any event, storage becomes one
less problem for the receiver to deal with even if the spammer does
the kind of encoding you mention above.

> names, in contradiction to SMTP's requirement that commands are

The above is more a issue of syntax than of content and can be easily
addressed. We have noticed this issue already and are fixing it in our
next draft. As mentioned in the initial post,  being new to IETF
process we appreciate such feedback.
 
> DMTP with highly scaled MTA clusters, which often involve NAT and/or

There is no assumption about IP routing anywhere in the draft. It is
true if you mean making MTA clusters DMTP-compliant would require
changes depending on nature of the cluster. However these are fairly
simple and straightforward changes. Also note that DMTP-compliant MTAs
can interoperate with non-DMTP-compliant legacy MTAs, whether latter
is a scalable cluster or standalone MTA. If the latter is whitelisted
with the former, the transaction is same as regular SMTP. If the
latter is unclassified, it is subject to sender-discouragement
mechanisms at the same cost as today.

> The attempt at cryptography in section 3.4 is laughable.

There is no attempt at PKI whatsoever, if that is what you mean. The
terminology may be similar and can be easily clarified. If you read
that section carefully, the sole purpose of the related paragraph is
to make the mapping between a message id and a message itself less
guessable, and to speedup the lookup of the message when the message
id is presented. Moreover, as stated in the section, this is just one
of possible design choices. Different schemes can be deployed by
different MTAs, as long as it achieves the above purpose, given that
this is really a local matter at the sending MTAs.

- Kartik

---

From: Tony Finch <[email-masked]>
Date: Tue, 10 May 2005 11:07:17 +0100

On Tue, 10 May 2005, Kartik Gopalan wrote:
>
> DMTP classifies sender MTAs into three categories: allowed, denied, and
> unclassified.

So do SMTP servers.

> The receiver pull model is only used for messages from the unclassified
> category.

And this is the category of messages where it is least useful. You are
forcing people to guess whether they want to receive a message before you
have expended any automated effort to classify it.

> any complete emails in DMTP.

But only after you have wasted someone's time.

You have not addressed the user interface issues of DMTP. How is a user
supposed to decide whether to receive a message given only the sender
address and message ID? You suggest that message IDs should be
cryptographically generated, and therefore meaningless to people, and you
do not address the problem of forged sender addresses. Therefore DMTP's
notification messages are useless to the recipient.

> There is no assumption about IP routing anywhere in the draft.

You are assuming that each server's idea of its own IP address is the same
as the other TCP endpoint's idea of it. You are assuming that incoming and
outgoing connections to an MTA are symmetrical.

> There is no attempt at PKI whatsoever, if that is what you mean.

No, I mean that you are using a cryptographic hash incompetently.

Tony.

---

From: Kartik Gopalan <[email-masked]>
Date: Tue, 10 May 2005 12:35:20 -0700

John, 

Thanks for your note. In DMTP, we use coarser MTA classifiers at the
level of their IP address or domain names -- not the fine-grained
email-address classifiers (as you seem to mention below). Using the
latter would indeed have the problems you describe -- but DMTP I-D
does not use email-address classifiers. Its possible to do that, but
only with the kind of strong sender authentication you mention, which
is the reason we steered away from them in the first place.

- Kartik

On 5/10/05, John C Klensin <[email-masked]> wrote:
> 
> 
> --On Tuesday, 10 May, 2005 00:52 -0400 Kartik Gopalan
> <[email-masked]> wrote:
> 
> >...
> > DMTP gives solution to the scenario that none of the currently
> > used anti-spam technologies can satisfactorily handle without
> > receiving the entire message. DMTP intent messages are used
> > ONLY for unclassified senders that are neither in receivers
> > MTA's whitelists nor blacklists.  It is only unclassified
> > senders who are treated using the receiver-pull model. And
> > even to this category you can apply the same
> > sender-discouragement schemes that are currently being applied
> > -- such as greylisting, any other challenge-response schemes,
> > etc.
> >...
> 
> Kartik,
> 
> I want to reinforce, from a different perspective, one of Tony's
> points.
> 
> Ultimately, to let a message through, you are depending on a
> whitelist that is, in turn, based on sender addresses.  Your
> greylist technique, effective or not, is ultimately just a
> technique for determining which addresses get onto the
> whitelist. That causes you to inherit all of the issues with
> whitelists based on sender addresses, especially the fact that,
> in the absence of strong authentication, sender addresses are
> easily faked.  If, as a receiver, I've got sufficient
> infrastructure --probably, but perhaps not necessarily, PKI
> machinery-- in place to authenticate sender addresses, then I
> probably don't need your technique.  If I don't, then all the
> spammer needs to do is to send messages to me that are "from"
> people whom I know and with whom I have corresponded in the
> past.
> 
> Not only is that not hard, but they are doing it today.
> 
> That brings me to the serious downside of this, and many other,
> "maybe it would help at least a little bit" ideas.    Assume,
> for purposes of argument, that the details of your proposal,
> such as not understanding how to use the SMTP extension model,
> can be sorted out.   Assume you can get it widely enough
> deployed in sending and receiving SMTP servers to make it useful
> without putting huge burdens on legitimate, known, users trying
> to exchange mail with each other (for better or worse, this is a
> very high barrier).     Now, what we end up with is:
> 
> (1) Another list of really-good addresses that can presumably be
> easily harvested by an address-harvesting virus.   Note that
> your technique is of almost zero use against a spam generator
> who relies on compromising a machine and using the sender name
> to spam the people in the and address books or using the names
> in the address book to spam the machine's owner from other
> machines at a later date.  In either case, names intentionally
> put in the address book or whitelist are presumably already
> whitelisted.
> 
> (2) Another cycle in which: (i) A technique reduces the amount
> of spam that is received slightly. (ii) The spammers respond by
> increasing the amount of spam sent by, say, twice that amount,
> resulting in a slight increase in the amount of spam received.
> (iii) The spammers then devise a countermeasure.  In this case,
> the countermeasure is quite easy: address-pair lists are
> apparently already in heavy use by some spammer-enabling tools.
> Net result is an increase in the total amount of spam being
> delivered by around twice the amount that your method presumably
> stopped.
> 
> Those "arms race" cycles are not good for anyone concerned.
> 
>      john
> 
>

---

From: Zhenhai Duan <[email-masked]>
Date: Tue, 10 May 2005 16:03:26 -0400 (EDT)

Tony,

Thank you for the helpful discussions. Let me try to clarify some of
the intuition behind the schemes in the I-D.

>> DMTP classifies sender MTAs into three categories: allowed, denied, and

Our comparison above was to IM2000. Indeed the current SMTP servers
know how to handle senders that are denied (blacklisted), or allowed
(whitelisted). A message that falls in neither class is currently
subject to the content filters to decide if it is a spam message or
not. To put it another way, the current SMTP does not have an explicit
concept of unclassified senders. The one we are aware of that supports
the similar concept is Greylisting, which defines unclassified sources
as new triplets that are not in the whitelist and blacklist yet.
However, as other people on this list have pointed out already,
spammers haven't yet adapted to this scheme because it hasn't impacted
their bottomline. This would happen once greylisting becomes
widespread.

> have expended any automated effort to classify it.

This really is related to what kind of classification we are talking
about. As one example, the receiver can configure his/her MTA such
that most regular contacts are in the allowed class. As a consequence,
after a brief learning time period, most important messages fall in
the allowed category (that do not need the notification process) and
the receiver pays only infrequent attention to the notifications in
the unclassfied folder. Note that this definition of classification is
different and orthogonal from classifying a message based on its
content. Occasionally, an important message may lie unread for a long
time -- its not desirable but it is not a new problem. It happens even
now with automated content-based classifiers you mention above. You
get false positives all the time even now -- they are either tagged as
[SPAM] to be stored away in a junk folder or (worse yet) deleted
without receiver's knowledge. With DMTP, at least receiver gets to
periodically review the notifications from these unclassified senders
(if receivers so choose to).

> notification messages are useless to the recipient.

You missed one more piece of information in the notification: a
limited subject header summarizing the contents.  You may say that
"well subject can be misleading and sender address can be forged to
make it appear like a regular contact". Its possible and happens with
viruses all the time. But this technique is useless to spammers
because it doesn't help them to send the tremendous volume of
commercial traffic that makes spamming so attractive. Just imagine the
amount of state and association they need to maintain per spammed
receiver to make heavy volume mailing successful with this technique.

Also,
1) Your regular contacts will typically be in the allowed class. If
you see a notification from your regular contact that is in the
unclassified folder, it is easy to tell it is a faked message and you
need not spend time on it.

2) For the other legitimate senders that are not in your allowed
class, how often you hear from them? and how likely is it that a
spammer knows that this is not only your legitimate contact but ALSO
is not in your regular contact class?

> outgoing connections to an MTA are symmetrical.

This is a good point and Kartik's email addressed your question to
some extent by mentioning that some MTA modifications are inevitable
to make them DMTP-compliant. As it is in the current draft, the sender
MTA IP address is used for the RMTA to retrieve the complete message.
We made this choice because of security concerns and also for the
simplicity of the RMTA. It is possible that instead of relying on RMTA
to record the SMTA IP address, we can do a DNS MX lookup of sender
email address to get the MTA IP address at the sender side. However,
this indeed means that if sender side uses different MTA servers for
incoming and outgoing messages, they need to be in sync about state of
outgoing message queue (for example, shared file systems). In the long
run, its a tradeoff worth making.

> No, I mean that you are using a cryptographic hash incompetently.

We do not quite see you mentioning a specific concern here. As long as
the sender-side hash serves the purpose, it does not matter how
simple/complicated it is. For example, when you subscribe to a mailing
list, you normally receive a message asking you to confirm the intent
to subscribe. How complicated do you think the method to generate the
random text on the Subject line is?

Thank you,
  -Zhenhai

---

From: Tony Finch <[email-masked]>
Date: Tue, 10 May 2005 21:54:33 +0100

> limited subject header summarizing the contents.

A wonderful place to put a spam message and URL.

> need not spend time on it.

This statement is not generally true. For example, the IP addresses of the
sending site's MTAs may have changed, or the sender may be roaming.

> class, how often you hear from them?

These kinds of messages are frequently extremely valuable.

> It is possible that instead of relying on RMTA to record the SMTA IP
> address, we can do a DNS MX lookup of sender email address to get the
> MTA IP address at the sender side. However, this indeed means that if
> sender side uses different MTA servers for incoming and outgoing
> messages, they need to be in sync about state of outgoing message queue
> (for example, shared file systems).

Gee, that's a good idea. Imagine a shared filesystem supporting the whole
email infrastructure for millions of customers!

Tony.

---

From: Kartik Gopalan <[email-masked]>
Date: Tue, 10 May 2005 15:27:29 -0700

> A wonderful place to put a spam message and URL.

It says "limited subject header" above which can include restrictions
on placing URLs.
(I had mentioned this already earlier.)

> sending site's MTAs may have changed, or the sender may be roaming.

Its the sender that's is roaming, not the sending MTA -- I don't see
any problem. And if the sending MTA changes its IP address, the
currently used  IP-based blacklists and whitelists go for a toss
anyway. So its not a new problem here.

> email infrastructure for millions of customers!

Again, it said "for example" above. Its an engineering decision that
would be different for different situations that designers make every
day depending on the scale of the system. The point is to design the
MTA to fit the email architecture, not to design the email
architecture to fit the current MTA design. Else you are simply
putting the cart before the horse.

- Kartik

---

From: Brett Watson <[email-masked]>
Date: Thu, 12 May 2005 01:49:09 +1000

> our recent IETF draft titled "Receiver-Driven Extensions to SMTP".

I haven't managed to read the actual draft yet, due to a webserver that's
currently not responding. Others have mentioned Dan Bernstein's IM2000 as
prior work in the field. I might also add my BSc(Hons) thesis (July 2002),
which probably overlaps significantly with your work.

http://www.comp.mq.edu.au/~brett/bschons/index.html

One of the techniques proposed in the thesis is a method for recipients to
take control of the delivery process, opting to pull the message at the
time of their choosing, rather than accepting it immediately during the
SMTP dialogue. Sound familiar?

---

From: David hagel <[email-masked]>
Date: Wed, 11 May 2005 10:43:05 -0700

On 5/10/05, John C Klensin <[email-masked]> wrote:
>
> Unless you abandon mail relaying, and perhaps even if you do,
> domain names and IP addresses are as easily spoofed as full
> email addresses.  The spammers figured both out many years ago.
>

you're mistaken. udp spoofing is easy BUT smtp uses tcp.
its almost impossible to spoof tcp sessions now.
spoofer needs to correctly guess the randomly generated sequence/ack
numbers. even if you magically do this, you gotta control ALL
intermediate routers or else smtp server's responses will be misdirected.
meaning you cannot create a normal tcp session.
it's a common misconception that "ip spoofing"
can be used to hide your ip address while surfing the net,
chatting on-line, sending e-mail, and so forth.
this is not true unless you're on the same subnet as victim
where you can sniff their tcp seq. #'s.
if whay you say is true, all DNSBLs would stop working dude.

---

From: David hagel <[email-masked]>
Date: Wed, 11 May 2005 11:48:17 -0700

> That situation is precisely why I started my comment with
> "unless you abandon mail relaying".  If you insist on direct
> connections between the initial sender (or an MTA that will take
> responsibility for the sender and which you trust to do so) and
> the receiver/ delivery SMTP server, then you are quite right and
> the spoofing issue depends on control of intermediate routers
> (and/or much more sophisticated tricks).  But, as long as
> intermediate SMTP relays (not routers, but at the SMTP level)
> continue to be permitted, TCP is just not the issue because
> there is no end to end connection.
> 
> The workings of most blacklists have nothing to do with this
> because they depend on either most-recent-hop or on assumptions
> about [dis]trusted SMTP servers at intermediate points rather
> than, as this does, properties of the initiating sender system.

ah, then ip spoofing is not quite the same as hiding behind 
someone else's mail relays. i admit missing the connection 
you made there. but open mail relays are hated and justifiably blacklisted 
aggresively (i am sure you know).  its the right thing to do. ain't it?
no harm blocking someone who lets their mail server to be 
used by spammers.

david

---

From: Zhenhai Duan <[email-masked]>
Date: Wed, 11 May 2005 14:51:18 -0400 (EDT)

Tony,

> email infrastructure for millions of customers!

I think there is some misunderstanding. Note that this is for supporting 
MTA clusters. Shared file systems/NFS/Replications are used in today's MTA 
clusters.

-Zhenhai

---

From: Zhenhai Duan <[email-masked]>
Date: Wed, 11 May 2005 15:05:30 -0400 (EDT)

John,

> there is no end to end connection.

This (open relay) is one part that puzzled me from time to time. Correct 
me if I am wrong. In the early days, the Internet is not stable and we may 
have difficults sometime to deliver a message from the source MTA directly 
to the destination MTA. Under this kind of circumstance, I may appreciate 
open relays. However, as the Internet is getting so (reasonably) reliable. 
I think some assumption should be changed. I do not know how often we 
still encounter network problems when we (sender) try to deliver a message 
to a receiver. I assume it is rare. And even if this happens, I do not see 
it is a big problem for the original SMTA to hold the message and retry 
later.

In a nutshell, I believe all open relays should be disabled or otherwise 
blocked.

(I do not want to go into a fight:-), so I make it clear that we are 
talking about open relay, not MTA clusters, not the cases where a company 
have multiple MTAs, and one (or more) are used for external-internal 
interface, while others are used internals and depending on the interface 
MTA to do the relay. It can be seen that these situations can be addressed 
adequately.)

-Zhenhai
> And, of course, if they worked better, you presumably would not
> need the draft in question.
>
>       john
>
>
>
>
>
>

---

From: David Hagel <[email-masked]>
Date: Wed, 11 May 2005 12:15:37 -0700

On 5/11/05, Arnt Gulbrandsen <[email-masked]> wrote:
> David hagel writes:
> > its almost impossible to spoof tcp sessions now. spoofer needs to
> > correctly guess the randomly generated sequence/ack numbers.
> 
> Have a look at http://www.completewhois.com/hijacked/ for an
> alternative, simpler approach.

granted one can steal whole ip blocks. 
its still ok blacklisting those ip blocks if its in the hands of
spammers. right?

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 11 May 2005 21:19:28 +0200

John C Klensin wrote:

> I am, "dude" or not, somewhat familiar with how SMTP works.

:-)
   
Anybody listed in RfC 1123 is "guilty until proven innocent".
As soon as time travel is available I intend to discuss all
implications of 5.3.6 (a) and 5.3.3 with the authors.  Maybe
the net at large sponsors the trip back to 1989.  Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 May 2005 11:05:22 +0200

John C Klensin wrote:

> my involvement with ARPANET/ Internet email goes back to the
> decision to put a "MAIL" verb in the original FTP

Now that doesn't prove that you're guilty of any attempts to
twist the good old STD 10 reverse path into something like an
arbitrary "bounces to" construct.  Maybe it was somebody else.

For obscure reasons I'm almost always on the wrong list, 2822
issues here, because mail-arch mentions them, and 2821 on the
822 list when authors invent new header fields, last example:
<http://mid.gmane.org/[email-masked]>

Apparently Bruce found a minor problem in 2821, it's not 110%
clear what to put in the Return-Path for a null MAIL FROM:<>

> "former IAB Chair"

Do they discuss issues like "bounces-to" ?  <sarcasm> It's not
very serious, maybe some Terabytes, less than 1 billion USD
damage. </sarcasm>

When it hit me it was in total 150,000 bounces.  With an average
size of 4 KB and my V.90 line that's about 2000 minutes if I got
this right, or 20 EUR for the traffic to me.  Plus the time for
wading through this stuff, sending some complaints, inventing
filters to salvage good bounces and "good" spam (reportable via
SC, last year it didn't allow to report bounces), and solving
several hundreds of these funny C/R puzzles (UOL etc.).

> I'm pretty much about as implicated as one can get.  :-)

The crystal balls they sold 16 years ago were really bad, my
exemplar told me that today something like TCP or SMTP would
be ancient history.  All my prophecies were wrong, only SGML
was at least not completely wrong.

Maybe the author of STD 10 had a _working_ crystal ball, his
idea of a "reverse path" was actually correct.  But that was
23 years ago, the quality of crystal balls varies with the
years.  1982 I (ab)used DEV PUN on a VM/370 for local mail,
and I didn't know what a "modem", "ftp", or "smtp" might be.

  [1123]
>> As soon as time travel is available I intend to discuss
>> all implications of 5.3.6 (a) and 5.3.3 with the authors.
>> Maybe the net at large sponsors the trip back to 1989.

> Some of those issues where intended to be clarified in 2821.
> If you don't think the clarifications were adequate

It's clear from my POV, the only missing piece is to fix it,
back to the working concept of a "reverse path" in STD 10.

> this would be a good time to let me know.

That sounds like you're planning a 2821bis anytime soon (?)

In that case I'd like to discuss the 2821bis syntax for an
IPv6-address-literal and a General-address-literal, adding
"IPv6:" is odd, how about adopting the syntax in RfC 3986 ?

It was a pain to find a new syntax for the msg-id in USEFOR,
the "dcontent" in your syntax isn't allowed in news / NNTP,
and it's also "suboptimal" in news: URLs.  After some weeks
we found a working "maximal" syntax.  But your "dcontent" is
defined in RfC 2822, and as always I'm on the wrong list.

        Bye, Frank (only sent to the SMTP list, not ASRG)

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 May 2005 11:05:22 +0200

[1123]

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 May 2005 13:54:58 +0200

Hector Santos wrote:

>> it's not 110% clear
[...]       ^^^^ 
 
> hmm, you put exactly what the Reverse Path (MAIL FROM)
> provided.

Of course.  It's a discussion with Bruce, so if the syntax
somewhere says ( "<>" / Reverse-Path ) and later talks only
about the Reverse-Path, then the empty case is lost.

It's not about understanding what to do in reality... ;-)

                        Bye, Frank

P.S., sorry for the ASRG copy, I wanted to remove it.

---

From: Keith Moore <[email-masked]>
Date: Fri, 13 May 2005 11:46:51 -0400

> Also, the Return-Path is typically removed once the gate sees its a normal
> message (no bounce requiired) before final storage for the end-user.   It is
> not required any more at that point.

it is a violation of the standards for mail processors to remove Return-Path.
Return-Path is required even after delivery for proper processing of
message disposition notification requests and for generation of automatic
responses.