Subject: [lemonade] streamlined message submission

From: Tony Finch <[email-masked]>
Date: Wed, 22 Feb 2006 00:54:25 +0000

The talk about round-trip times on the main IETF list (combined with some
other things I have been thinking abou recently) got me interested. How
much can you streamline message submission? Current securely authenticated
message submission requires at least eleven round trips (including TCP
startup and shutdown). BURL can reduce this to nine. The non-standard but
popular smtps reduces it to six. A further tweak to SMTP's start sequence
could reduce it to five. This seems pretty good.

I've written this up in more detail. The following are all fairly short,
and where they aren't they are formulaic.
http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-submission-streamlined.txt
http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-tls-on-connect.txt
http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-quickstart.txt
http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-rcpthdr.txt

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. MODERATE OR
GOOD.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Ned Freed <[email-masked]>
Date: Tue, 21 Feb 2006 17:09:59 -0800 (PST)

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-submission-streamlined.txt

This is effectively a description of how to combine the remaining mechanisms
plus BURL.

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-tls-on-connect.txt

I have to say I agreed with the decision not to go with separate ports and with
STARTTLS instead. And although this does reduce round trips significantly I'm
fairly unenthusiastic it, but less because I agreed with the prior decision and
more because  I predict there will be opposition to moving back to the smtps
approach from the security folks. And although we've supported smtps in our
product for a long time, I don't in practice see the large amounts of lingering
use of it you do.

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-quickstart.txt

This extension seems quite reasonable to me. I would suggest that you point out
in the security considerations that logging of the client's IP address is more
useful in practice than logging of the EHLO value. (This is mentioned in
section 1 but bears repeating. Or perhaps some of the discussion in the
introduction should be moved to the security considerations section.)

You probably do need to include an SMTP service extension registration
template, even though as you say this doesn't really conform to the
EHLO extension announcement model.

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-rcpthdr.txt

First an editorial comment - the draft refers repeatedly to RFC 2476. This RFC
is obsolete, having been replaced by draft-gellens-submit-bis-02.txt, which has
already been approved as a draft standard.

As the draft points out, this doesn't reduce the number of round trips except
in the case where more than 4K of recipient data is involved, which is quite
rare in practice. I therefore see little value in it. I also see lots of
problems with it, including but not limited to:

(0) How are malformed or unparseable To:/Cc:/Bcc: fields supposed to be
    handled? There's no place to report such an error in the dialogue and
    a DSN isn't designed to report such issues.

(1) How are problems with specific recipient addresses supposed to be handled?
    The draft seems to say that this is covered in RFC 2476. But it isn't -
    in fact RFC 2476 specifically talks about using RCPT TO command
    responses to report this sort of thing - the facility you have eliminated.
    AFAICT the best that could be done is to return a DSN later. But after
    having seen this model used elsewhere I can tell you that users absolutely
    hate it. If there's a problem with a recipient they've specified they
    want to see it right there on the screen when they send a message.
    Getting back a DSN later is nowhere near as good, and as joe-jobs
    and blowback spam continues to grow DSNs stand an ever-increasing
    chance of being lost in the shuffle.

(2) The reliance on header ordering to handle Resent- is something I find
    to be quite problematic, language about it in RFC 2822 notwithstanding.

I also have to say I find the argument that traffic analysis of a pipelined
session often reveals the number of recipients and this mechanism protects
against it to be pretty dubious. There's a lot of variably sized stuff in the
various commands and it would be very difficult to track its cumulative effect.
Additionally, in the case where recipients are scattered around on different
destination systems, a message is going to be split and sent multiple times
to different destinations. In this case even if TLS is used for everything
tracking the overall size and temporal adjacency of the messages sent by a
given system will let you correlate things to the point where you'll know
at least how many different destinations for a given message was sent to,
regardless of whether or not this mechanism is used.

In short, I really don't think this one's a keeper.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 Feb 2006 14:14:18 +0000

On Tue, 21 Feb 2006, Ned Freed wrote:

Thanks for your comments.

> product for a long time, I don't in practice see the large amounts of lingering

I expect that what will continue to happen, whether this draft moves on or
not, is the STARTTLS and TLS-on-connect models will operate in parallel.
Most of the TLS-on-connect requirement comes from Microsoft Outlook.

> > http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-rcpthdr.txt

I'll check submit-bis. Thanks for reminding me about it.

> rare in practice.

Or if the number of recipients tips the combination of envelope and data
over this limit - assuming they are pipelined together via the BURL
extension.

>     handled?

By rejection of the message data, as in RFC 2476 sections 5.1 and 3.2.

> (1) How are problems with specific recipient addresses supposed to be handled?

Dave Cridland also said I should consider this more carefully. I didn't
worry about it too much because the wisdom I have received is that an MSA
should accept-and-bounce rather than reject, because that way the right
error message is more likely to get through to the user. I still get
complaints from users quoting the message "valid RCPT command must precede
DATA" rather than the RCPT rejection message, and I still see mad MUAs
repeatedly retrying a 550ed message every second for days, so I still
don't trust them to behave sensibly. </rant>

>     to be quite problematic, language about it in RFC 2822 notwithstanding.

The main reason for writing this down is to address a problem that arose
in the development of Exim: it isn't clear how an MSA should treat a
re-sent message, given the mismatch between RFC 822 and RFC 2822. In
particular, should Sender:, Message-ID: and Date: fields be fixed up or
should the Resent- versions be fixed up, and in the latter case, which set
should be fixed?

This part should probably be decoupled from the RCPTHDR idea. It may be
useful input to 2822-bis.

I agree that RCPTHDR by itself is not really compelling.

Tony.

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Feb 2006 06:20:12 -0800 (PST)

> By rejection of the message data, as in RFC 2476 sections 5.1 and 3.2.

I'm sorry, but RFC 2476's implication that messages should be rejected for
having invalid header syntax is not practical in many cases. There's just too
much cruft out there.

> error message is more likely to get through to the user.

This is almost the exact opposite of our experience. Immediate rejection is
handled well by most MUAs, which typically present (a) A messasge in the user's
own language saying there's a problem, (b) The address that failed, and (c) The
actual error message, which may not be in their language but doesn't really
have to be.

Now, there are a few MUAs that don't present the error message, or present the
wrong one, and this bothers technical types a lot. But the alternative - a
report generated later, appearing in the inbox, is far more difficult for
nontechnical users to deal with. Often as not they can't figure out what
message the report applies to, what went wrong, or what address was the one
that had a problem.

Language issues are often the root cause of some of the problems here. We go to
tremendous lengths in our product to try and make appropriate language choices
for DSNs in order to avoid these issues, but it is never enough. You cannot get
away from the fact that the DSN isn't attached to the message that had a
problem in the same way that immediate response is.

> I still get

But you trust them to display DSNs ssnsibly? Sorry, this makes no sense at all
to me. The only MUA I have ever seen that actually did DSN display localization
and correlation with sent messages was X.400 based. For whatever reason this
has not caught on in the Internet protocol world, likely due to the fact that
DSN and MDN formats were standardized so late in the game. (The fact that there
is ongoing resistance to using the standard formats in some quarters doesn't
help either.)

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Wed, 22 Feb 2006 14:43:48 +0000

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-submission-streamlined.txt

BURL doesn't allow clients to pipeline AUTH - that's just possible 
anyway with a server that supports PIPELINING. I have no idea why 
you'd want to pipeline QUIT - it's never going to be a good idea. You 
leave the connection open until it times out, in case you need to 
reuse it later. (Yes, this isn't nice to the server.)

Oh, and I believe that it's possible to include the banner in the TCP 
SYN+ACK packet coming from the server, but I couldn't swear to it. 
That one just got told to me in the pub once. I *think* Linux's TCP 
stack allows you to write the banner to a listening socket or 
something odd to get the behaviour, but I'd have to ask about.

What I really do like about this draft is the round-trip list - both 
this one and the TLS one. Those are actually really useful, and I for 
one would appreciate you publishing those on their own in at least 
I-D form, and include how it changes if we have TLS session cache 
hits, etc. It'd be useful to have a similar one for IMAP.

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-tls-on-connect.txt

The problem with removing the STARTTLS round-trip is that quite a bit 
of discussion and argument went into putting it there in the first 
place. The two arguments I personally find most convincing are:

1) Look at HTTP. Nobody uses TLS except when they absolutely have to, 
there, and generally, https: scheme URIs point to different content 
than http: scheme URIs. The two ports have mutated into two protocols 
- not good.

2) If unusual ports are used, and TLS is unavailable (something my 
own client is sometimes forced to deal with), then splitting the 
ports means that it's extremely difficult and ugly to deal with. 
Simply not issuing STARTTLS in order to have TLS-less access makes 
life much easier.

FWIW, back when SSL usage for IMAP was first being discussed on the 
IMAP mailing list, one of the arguments against a seperate port was 
that SSL was designed in such a way that it could identify whether it 
was TLS or not - a distinct inline switching command was not actually 
required, in other words. You *could* put such a capability in a 
banner, and observer the first octets to come through, seeing if they 
were valid TLS hello octets. I have virtually no idea what I'm 
talking about here, though.

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-quickstart.txt

A round-trip saved, that's good. This is a rough analogue of the IMAP 
CAPABILITY response code.

The question is, is it possible that some ESMTP clients might be 
confused by a multiline response? I'm mildly concerned by that.

There's an alternate model for this sort of thing - you can issue a 
server configuration indicator in the banner, which changes whenever 
the configuration does, such that if you recognise it, then the EHLO 
becomes redundant, because you know the capabilities. Moreover, 
you've cached them *after* the secure channel setup, of course, so 
you needn't ever fetch them until you see that change.

I'm thinking:

S: 220 foo.example.net ESMTP CONFIGMAGICNUMBER 
<[email-masked]>
C: // Elide EHLO, we know the thing supports TLS because we know that 
magic key.
C: STARTTLS
S: 250 Groovy.
// TLS exchange.
// No need to reissue EHLO, we have those cached.
C: AUTH DIGEST-MD5 fastreauthgunk
S: 2xx respauthgunk

etc.

Now... In theory we can also elide that AUTH, if TLS session caching 
hit, since that's presumably proof enough that we are the same source 
as last time. We can certainly use a AUTH EXTERNAL there on that 
premise, can't we? Has anyone been looking at BTNS, which ought to 
provide much the same thing with IPSec?

> http://www.cus.cam.ac.uk/~fanf2/hermes/doc/antiforgery/draft-fanf-smtp-rcpthdr.txt
Okay, I already commented on this to you. Ned's arguments are better 
than mine, too. :-)

> BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. 

If you could change that to Lundy, I'd actually find it useful. 
Biscay is quite a way off for me.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Antony Bowesman <[email-masked]>
Date: Wed, 22 Feb 2006 17:45:53 +0200

> and correlation with sent messages was X.400 based. For whatever reason this

Netscape some years ago had the option to move incoming reports to the sent 
folder which Thunderbird adopted.  This worked reasonalby well, but for some 
reason TB has now removed DSN requests completely and few people support MDN, so 
it's not so useful.

 > is ongoing resistance to using the standard formats in some quarters doesn't

Yes, MS' winmail.dat did plenty of harm to the idea of reports.  The privacy 
issue with MDN may also have affected adoption.

Antony

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Feb 2006 07:12:34 -0800 (PST)

> reuse it later. (Yes, this isn't nice to the server.)

The thing a lot of clients do is simply drop the connection without ever
sending a QUIT, rules to the contrary notwithstanding.

Of course the user can be told that the message was successfully sent the
instant the response to the final DATA/BURL/BDAT has been received. The QUIT
can then be handled asynchronously in the background. So if the goal is to
improve user experience, QUIT handling doesn't have to be relevant.

> something odd to get the behaviour, but I'd have to ask about.

Wow, you sure have more interesting conversations at the pub than I do ;-)

Anyway, I know of a couple of tricks in this space. One is the technique
described in RFC 1644, which eliminates the three way handshakes for repeated
connections between the same two machines. It isn't clear to me that this would
be useful in SMTP, however. Often as not there are a whole bunch of front end
machines a client can connect to, and arranging things so the client would have
an affinity for the last server would be tricky.

The other is to piggyback data on either the syn, syn-ack, or both. This has
been discussed many times on various lists but if there's an RFC discussing it
in detail I don't know the number. Piggyback data on syn isn't interesting in
SMTP since the server speaks first, but returning the server banner on the
syn-ack would represent some savings. The problem lies in implementation - most
(all?) TCP APIs don't allow data to be specified in the accept call. Some
stacks offer a way to delay sending acks in hopes they can be piggybacked on
returned data, but I don't know if this ever extends to syn-ack or not. The
trick you described of writing data to the socket before calling accept is not
one I've heard of before. I may try it and see if it works.

> confused by a multiline response? I'm mildly concerned by that.

This used to be a problem in practice, but several years ago some fairly
popular SMTP server started spitting out multiline banners (something sort of
"no trespassing" sign, I think) and refused to provide an option to disable it,
which then got a lot of the incompliant gunk cleaned up. (It is always nice
when something that's widely deployed pushes the envelope this way.) At this
point I think compliance is pretty good - as it happens I forgot to dismantle a
test a few months back and inadvertently ran with a mulitline banner for a
while and noticed no problems.

> you needn't ever fetch them until you see that change.

We actually use a similar trick in our SMTP in order to avoid having to
code the list of supported extensions into the proxy. This is one of those
things that sounds simple but it is actually surprisingly hard to implement
correctly. I forget exactly what bugs we had, but we definitely had a couple.
I actually worry more about implementors getting this right than I do about
getting multiline banner processing right.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 Feb 2006 17:16:51 +0000

On Wed, 22 Feb 2006, Dave Cridland wrote:

> anyway with a server that supports PIPELINING.

The BURL spec is the only document that gives permission to do it. The
AUTH spec doesn't say anything about PIPELINING, and PIELINING says

   Additional commands added by other SMTP extensions may only appear as
   the last command in a group unless otherwise specified by the
   extensions that define the commands.

RFC 3030 is a bit unclear about the interaction of BDAT and pipelining -
it doesn't describe things in terms of the "last command in a group" as
RFC 2920 does. It does allow BDAT to be pipelined with itself but doesn't
talk about pipelining it with the preceding or following envelopes. The
BURL spec clarifies that you can do this is the server supports BURL.

> I have no idea why you'd want to pipeline QUIT - it's never going to be
> a good idea. You leave the connection open until it times out, in case
> you need to reuse it later. (Yes, this isn't nice to the server.)

Pipelined QUIT is included in an example in RFC 2920. It shouldn't be a
problem if the client waits for all the server's responses. Note that in
my ladder diagrams I show the server initiating the connection close, so
this should Just Work. Some clients close the connection immediately after
QUIT, which is liable to cut off the server's response, especially if you
are also using TLS (which doesn't support half-closed connections).

Of course I am not precluding connection cacheing, and I should probably
add a mention of it to the drafts.

> SYN+ACK packet coming from the server, but I couldn't swear to it.

Maybe. From a look at Stevens and Comer I think it would work at the
client end of common TCP stacks, though I can't find any sign of support
at the server end.

> one would appreciate you publishing those on their own in at least I-D
> form, and include how it changes if we have TLS session cache hits, etc.
> It'd be useful to have a similar one for IMAP.

Yes, I need to add something about TLS session cacheing. It has
implications for what exchanges can be piggybacked.

> [TLS-on-connect is not kosher]

> there, and generally, https: scheme URIs point to different content than
> http: scheme URIs. The two ports have mutated into two protocols - not
> good.

We have this for SMTP too: message submission on port 587, usually with
TLS, and message transfer on port 25, usually without. In both cases the
general distinction is that TLS is used to protect authentication, and
anonymous connections go in the clear.

> 2) If unusual ports are used, and TLS is unavailable (something my own
> client is sometimes forced to deal with), then splitting the ports means
> that it's extremely difficult and ugly to deal with. Simply not issuing
> STARTTLS in order to have TLS-less access makes life much easier.

You seem to be arguing in favour of the downgrade attack vulnerability. In
this situation you must interact with the user because it indicates that
some intermediaty (firewall or attacker) is screwing with connectivity to
the configured submission server, and they need to find out from someone
who knows what is the right thing to do.

> [vague ideas about autodetecting TLS]

You can't reliably auto-detect plaintext SMTP (or IMAP) versus
TLS-on-connect on the same port, because there's a mismatch between who
talks first. If it was consistently the client that talked first then the
server could autodetect, but the SMTP or IMAP server banner prevents this.

> CAPABILITY response code.

Exactly.

> confused by a multiline response? I'm mildly concerned by that.

We should try it out. I have a multiline banner on my MX hosts which
doesn't cause interop problems, but MUAs are likely to be more
troublesome.

> There's an alternate model for this sort of thing - you can issue a server
> configuration indicator in the banner, which changes whenever the
> configuration does, such that if you recognise it, then the EHLO becomes
> redundant, because you know the capabilities.

Interesting idea, especially for use with STARTTLS.

The other thing I am considering is a way for the client to avoid waiting
for the server banner, which would save another round trip. This would
bring a STARTTLS connection down to 5 RTT, or 4 RTT with a cached TLS
session; cf. 3 RTT for TLS-on-connect, or 2 RTT with a cached TLS session
and server banner.

> Now... In theory we can also elide that AUTH, if TLS session caching hit,
> since that's presumably proof enough that we are the same source as last time.
> We can certainly use a AUTH EXTERNAL there on that premise, can't we?

Another interesting idea, though it doesn't belong in the QUICKSTART
document. I was also thinking about implicit authentication recently: if
the submission server trusts the IMAP server, the authorization in a
URLAUTH implies that the user was recently authenticated by the IMAP
server, so the submission server can treat it as an authentication token.

> Has anyone been looking at BTNS, which ought to provide much the same
> thing with IPSec?

I believe the motivation behind BTNS was ssh's known_hosts cache, which
avoids the need for a PKI or out-of-band trust management, provided that
you make a "leap of faith" and trust the server the first time you
connect. There's nothing to prevent you from using this model with TLS;
in fact RFC 3207 suggests doing so as a defence against man-in-the-middle
attacks when using optional STARTTLS between MTAs.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE LUNDY FASTNET: EAST OR NORTHEAST 5 TO 7. SHOWERS. GOOD.

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 Feb 2006 17:52:11 +0000

On Wed, 22 Feb 2006, Ned Freed wrote:
> described in RFC 1644, which eliminates the three way handshakes for
> repeated connections between the same two machines.

There are serious problems with T/TCP. See
http://tcp-impl.grc.nasa.gov/tcp-impl/list/archive/1292.html
I believe the only OS that ever shipped it is FreeBSD, and the support has
rotted to death over recent years as security hardening has broken it.

> The other is to piggyback data on either the syn, syn-ack, or both.

The former is the essence of T/TCP, and is really undesirable because it
makes you MUCH more vulnerable to syn floods.

> popular SMTP server started spitting out multiline banners (something
> sort of "no trespassing" sign, I think) and refused to provide an option
> to disable it, which then got a lot of the incompliant gunk cleaned up.
> (It is always nice when something that's widely deployed pushes the
> envelope this way.)

:-)

> getting multiline banner processing right.

It's hard to specify clearly, which is a bad sign.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
CAPE WRATH TO RATTRAY HEAD INCLUDING ORKNEY: NORTH OR NORTHEAST 4 OR 5,
OCCASIONALLY 3 AT FIRST. RAIN LATER. GOOD OCCASIONALLY MODERATE LATER. SLIGHT
OR MODERATE.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Feb 2006 01:14:47 +0000

> BURL spec clarifies that you can do this is the server supports 
Right, that's an interesting thing I'd not noticed before. As I've 
said before, I've always held the opinion that really, you want to 
know if you've misspelt an address before sending it to anyone at 
all, so I've always sent the envelope in one group, and the data in 
another.

> client end of common TCP stacks, though I can't find any sign of 
Okay... As far as I can find out, there's no API for this in the 
Linux implementation, but apparently, Phil Karn's KA9Q NOS (which I'm 
not familiar with at all) has done this for years. Between us, we 
must know someone who can do an API implementation on either FreeBSD 
or Linux, surely?

> anonymous connections go in the clear.
Ah, but we have the option to have anonymous MTA activity use TLS 
(there's plenty of MX machines supporting this), and the option to 
use, say, IP Sec etc for submission.

> You seem to be arguing in favour of the downgrade attack 
> vulnerability

No, I meant the client platform doesn't have TLS support available. 
Sadly, Polymer has to deal with that, too, sometimes. Generally it's 
older Python versions on Linux and other UNIXes, where the TLS code 
doesn't handle non-blocking sockets. So when given "imaps" URIs, I 
have to attempt to guess what the "imap" port might be, and hope it's 
equivalent. Obviously I issue a paranoia warning, too.

> server could autodetect, but the SMTP or IMAP server banner 
Well, you could in theory autodetect after the banner was sent, and 
the banner would need to indicate whether TLS autodetection were 
supported.

> doesn't cause interop problems, but MUAs are likely to be more
Test it out on those annoying students. :-)

> Interesting idea, especially for use with STARTTLS.
Ned seems to shoot it down based on experience, which I'll bow to. My 
gut feeling was that it really ought to be possible to specify this 
quite easily, and implement it easily, too, just by using a 
relatively strong hash of the capability listing.

> session; cf. 3 RTT for TLS-on-connect, or 2 RTT with a cached TLS 
*nods* Wrapping the banner into the SYN+ACK would seem to have the 
same effect for an easier shot, if we can. Means having platform 
level support, of course.

> server, so the submission server can treat it as an authentication 
No... Posession of a URLAUTH URL doesn't mean that you obtained it 
from the IMAP server, you might have obtained it from someone else, 
legally or otherwise.

> attacks when using optional STARTTLS between MTAs.
BTNS charter seemed to suggest to me it related to ascertaining that 
the peer was the same peer as previously. If the "previous" 
connection was authenticated, then it seems to me that provides 
useful information without any leaps of faith. Of course, nightmares 
ensue if you've multiple connections to the same peer using 
*different* authentication, and these mechanisms can't differentiate.

Dave.

---

From: Chris Newman <[email-masked]>
Date: Wed, 22 Feb 2006 19:27:37 -0800

Tony Finch wrote on 2/22/06 0:54 +0000:

> startup and shutdown). BURL can reduce this to nine.

I'm glad someone noticed that little feature I slipped into BURL :-)

As written, I'd have to strongly oppose this draft.  If you recast it as a 
"submits" port that is never to be used for MTA to MTA transfer, that addresses 
the most serious concern.  But I still think this is the wrong direction to 
move.  While I'd have to say TLS is the only truly useful security layer for 
applications available today, it's wrong to assume that will be true for the 
indefinite future.  If the IPsec folks ever come up with APIs and mechanisms 
that are useful for user authentication by applications, then that would 
probably be a preferable mechanism as it could resist DOS attacks that TLS 
can't.  Deploying a new port in submit clients is difficult (and deploying a 
new port in SMTP clients is borderline impossible).  We're just starting to see 
587 catch on despite years of ISPs blocking outbound port 25 and the other 
deployment reasons to use a separate port for submission.  Introducing another 
port now would just involve more disruption and confusion and make 
interoperability worse.

I recommend you drop this proposal and focus on a modification of quickstart 
(perhaps QUICKSTART+TLS), to avoid waiting for either the server success 
response to the STARTTLS command, or requiring a new EHLO command after the 
STARTTLS command.  That gets most of the benefit without assuming TLS is the 
only security layer ever used.

FWIW, some other problems with separate ports are documented here:

 http://www.apps.ietf.org/rfc/rfc2595.html#sec-7

Some of that text is dated.  The second bullet is correct in spirit but wrong 
in details due to changes in what was deployed then vs. now (40-bit is gone and 
GSSAPI+Kerberos 5 had security flaws).  The third bullet is wrong, but can be 
corrected by replacing "use TLS when available" with "require TLS if it's been 
advertised previously for that service".  Indeed, I'd argue the best clients 
should use that policy by default.

I like this idea.  IMAP made a similar move in the RFC 3501 revision and I 
believe it has worked well in practice.  I have no objections to adopting this 
as a lemonade WG item, or pursuing it as an individual submission.

The semantics of RFC 822/2822 headers and RFC 821/2821 envelope recipients are 
subtly different.  Those differences make attempts to derive the envelope from 
the headers a dangerous and error-prone thing to do.  Imagine what would happen 
if the post office opened letters and determined the recipient address from the 
letter headers?  It would probably work most of the time, but the failures 
would be spectacular and unpleasant.  I believe the same is true of this 
proposal.  It introduces a level of operational complexity into the system that 
is best avoided.

I know it's annoying for a mail client to construct an SMTP envelope and RFC 
2822 header which usually contain the same information, but it's also simple. 
If you really want to pursue something along these lines, I suspect the reverse 
approach is viable: construct the RFC 2822 headers from an annotated SMTP 
envelope at submit time.

                - Chris

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 12:56:17 +0000

On Wed, 22 Feb 2006, Chris Newman wrote:
> > Current securely authenticated message submission requires at least
> > eleven round trips (including TCP startup and shutdown). BURL can
> > reduce this to nine.
>

Yes it is nice, but on reflection I think the way it is specified isn't
ideal. In particular, to quote that bit from RFC 2920 again:

So I think the BURL spec is wrong to talk about pipelining AUTH, and if
you want to allow pipelined AUTH it should be specified as an update to
RFC 2554.

Furthermore, it isn't safe in all situations for a client to pipeline a
single round-trip AUTH. For example, it's common to have dual-purpose MSA
and MX servers which operate in MSA mode after AUTH and MX mode otherwise.
If the client pipelines an AUTH and an intended submission, but the auth
fails, bad things will happen - recipients will be rejected that would
have been OK, an incomplete message will suffer an anti-spam penalty
instead of being fixed up, etc.

Pipelined AUTH is OK if the server is going to reject any un-authenticated
submissions, in which case the server should explicitly say so, perhaps by
using an EHLO keyword line such as 250 PIPEAUTH PLAIN EXTERNAL.

I've also copied this message to Gregory Vaudreuil because of questions
about PIPELINING and BDAT, as well as BURL.

RFC 3030 implies that BDAT can be pipelined, though the way it says so
doesn't follow the "last command in a group" model of RFC 2920.
Furthermore, some of the descriptive text and the example in section 4.2
imply a synchronization point between the envelope and data, though
because all the commands in the sequence can be pipelined, this isn't a
forced sync point. It would be more correct to terminate the envelope
commands with a NOOP to force a sync.

The BURL spec requires a sync point after the "LAST" BDAT or BURL command,
instead of after the message envelope. I think it would be better for BDAT
and BURL to not require any synchronization at all: if the client wants a
sync, it can use the NOOP command. Note that the main reason a sync point
is required after DATA is so that the client and server agree about
whether the next utterance from the client is a command or is message
data. This isn't a problem for BDAT because the command and data are sent
together, with a single response from the server for both.

This would allow a client to pipeline multiple messages together, which
means you can move from using TCP in interactive mode to bulk data mode,
and therefore make the best use of the available bandwidth. For example,
consider a 1Mb/s link with a 100ms RTT. If the client sends a 5KB message,
this takes about 50ms to transmit, so by the time the server's replies
arrive the client will be one or two messages further on; if there was a
problem, the client can deal with it asynchronously, and it will have done
50ms extra useful work. If the client sends a 20KB message, the server's
replies will start arriving part-way through message transmission, so if
there is a problem the client can usefully RSET it before the next chunk
to save time and bandwidth.

I think it would be cool to allow this kind of high-bandwidth asynchronous
message transmission. It is also more friendly from the point of
congestion control than having multiple concurrent SMTP connections
perpetually in TCP slow start.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAIR ISLE FAEROES: NORTH OR NORTHEAST 4 OR 5, INCREASING 6 IN FAIR ISLE FOR A
TIME, BUT VARIABLE 4 IN FAEROES AT FIRST. FAIR. GOOD.

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 13:22:20 +0000

I think both Ned and Dave mentioned the irritation that can arise from a
message and receiving a bounce for some of the recipients. (Note to self:
[email-masked] is the wrong address). One advantage of a sync point
after the message envelope is that the client can decide not to proceed if
a subset of the recipients are rejected; this doesn't work if the envelope
and data are pipelined. (It is also limited by how strongly the server can
verify addresses.) What do you think of having a NOPARTFAIL parameter to
the MAIL command, which means that the client is asking the server to
reject the message data if any of the recipients are rejected - "no
partial failure".

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH UTSIRE: VARIABLE 4, BUT NORTHEAST 5 TO 7 AT FIRST IN NORTHWEST. SHOWERS
LATER. GOOD.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Feb 2006 18:51:29 +0000

On Thu Feb 23 13:22:20 2006, Tony Finch wrote:
> and data are pipelined.

This is what Polymer does already - it synchs after the envelope, 
before sending any data command (ie, DATA/BURL/BDAT).

>  (It is also limited by how strongly the server can

Hard to implement conclusively. Also, if the server rejects one 
recipient out of several, and the client doesn't wish the message to 
be sent, why on earth transmit the data? Transmission bears a very 
high cost, both in terms of battery power on a mobile phone, and in 
terms of financial cost.

Dave. (Currently hit 10.91k up, 38.45k down over expensive UK GPRS)
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 19:02:06 +0000

I have updated the QUICKSTART draft. The main difference is a description
of how to save the server greeting round-trip and how to skip EHLO after
STARTTLS. I've added packet traces to show the three RTT savings from
QUICKSTART plus one from TLS session cacheing.

I'm wondering if it might be safer for the client to indicate quickstart
using a QHLO command, which may be pipelined. Any opinions?

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MULL OF KINTYRE TO ARDNAMURCHAN POINT: NORTHEAST VEERING EAST 4 OR 5. RAIN AT
FIRST. MAINLY GOOD. MODERATE OR ROUGH OFFSHORE.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 Feb 2006 19:47:02 +0000

> QUICKSTART plus one from TLS session cacheing.
Skipping waiting for the greeting looks a little scary to me. That in 
itself would argue that this went Experimental, whereas other parts 
of it are on much firmer footing.

I also think there's a problem in ommitting the post-TLS EHLO - 
simply hoping the server's capabilities haven't changed seems a 
little vague.

I really would prefer to look again at some form of capability change 
indicator, and using that to elide EHLOs. I don't think that's 
mutually exclusive with QUICKSTART, unless QUICKSTART starts using 
more than the TCP window for capability data.

> using a QHLO command, which may be pipelined. Any opinions?
Six octets of signalling information which the submit server uses for 
what?

OTOH, you've suggested using this in MTA-MTA, where this is needed - 
spotting protocol misuse is used to block spammers sometimes. 
Personally, I don't think there's a major round-trip issue between 
MTAs.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 20:13:11 +0000

On Thu, 23 Feb 2006, Dave Cridland wrote:
> Six octets of signalling information which the submit server uses for what?
>
> indicator, and using that to elide EHLOs. I don't think that's mutually

I was partly wondering what would happen to mobile clients connecting to a
network with a hostile firewall which doesn't understand QUICKSTART. The
client needs a way to recover gracefully, and I thought an extra command
not supported by non-QUICKSTART servers might help.

OK, how about this. The QUICKSTART ehlo keyword gets a parameter which the
client echoes back in the QHLO command. If the server detects a mismatch,
it rejects everything until the client says EHLO (which would probably
happen anyway according to usual service extension rules). e.g.

 C: QHLO 35681d17 hunny.100acre.example
 C: STARTTLS
 S: 220-smtp.100acre.example ESMTP
 S: 220-QUICKSTART 35681d17
 S: 220-PIPELINING
 S: 220 STARTTLS
 S: 250 quickstart ok
 S: 220 go ahead
...
 C: QHLO 26f1aa9c hunny.100acre.example
 C: AUTH PLAIN AHBvb2gAYmVhcg==
 S: 220-smtp.100acre.example ESMTP
 S: 220-QUICKSTART 26f1aa9c
 S: 220-PIPELINING
 S: 220 AUTH PLAIN
 S: 235 auth ok

or

 C: QHLO 3c9e4e9b hunny.100acre.example
 C: STARTTLS
 S: 220-smtp.100acre.example ESMTP
 S: 220-QUICKSTART 35681d17
 S: 220-PIPELINING
 S: 220 STARTTLS
 S: 550 bad quickstart token
 S: 503 bad sequence of commands
 C: EHLO aaah.i.am.covered.in.bees
 S: 250-smtp.100acre.example ESMTP
 S: 250-QUICKSTART 35681d17
 S: 250-PIPELINING
 S: 250 STARTTLS
 C: STARTTLS
 S: 220 go ahead
...

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH UTSIRE: VARIABLE 4, BUT NORTHEAST 5 TO 7 AT FIRST IN NORTHWEST. SHOWERS.
GOOD.

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 20:16:34 +0000

>  C: AUTH PLAIN AHBvb2gAYmVhcg==

Er the server's response should have read

   S: 250 quickstart ok
   S: 235 auth ok

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTH UTSIRE: VARIABLE 3 OR 4, BECOMING SOUTHEAST 5 OR 6 FOR A TIME. SHOWERS.
GOOD.

---

From: Chris Newman <[email-masked]>
Date: Thu, 23 Feb 2006 15:58:24 -0800

Dave Cridland wrote on 2/23/06 19:47 +0000:
> Skipping waiting for the greeting looks a little scary to me. That in itself
> would argue that this went Experimental, whereas other parts of it are on
> much firmer footing.

Skipping waiting for the greeting is a complete non-starter in SMTP (25) -- 
it's a common MTA feature to flush the input buffer after sending the greeting 
to force spammers to wait for that round trip.  It's much more cost-effective 
to do that to _all_ transfers (legitimate or not) than to allow spammers to 
blast full speed.  I suppose it might be feasible in Submit (587), but it needs 
a separate capability from PIPELINING or QUICKSTART.

On the issue of TLS integration, I was thinking of something along these lines:

S: ...
S: 220-QUICKSTART TLS
S: ...
C: QUICKTLS [server-domain]
<TLS negotiation>
S: <EHLO-response>

This "TLS" advertisement is stronger than the "STARTTLS" advertisement. 
Instead of simply stating the server understands the command, this states the 
server is ready to negotiate TLS.  So the client can issue the command and 
immediately begin the TLS negotiation without waiting for the server response 
to the QUICKTLS command.  The ability for the client to tell the server the 
host name it is using to talk to the server permits virtual domains without 
burning IP addresses, another useful feature that STARTTLS is missing.

And why worry about all that complexity of caching server capabilities when the 
server can be explicit about it in the banner to save the round-trips in all 
cases?

                - Chris

---

From: Mark Crispin <[email-masked]>
Date: Thu, 23 Feb 2006 16:09:45 -0800 (Pacific Standard Time)

On Thu, 23 Feb 2006, Chris Newman wrote:
> it's a common MTA feature to flush the input buffer after sending the 
> greeting to force spammers to wait for that round trip.  It's much more 
> cost-effective to do that to _all_ transfers (legitimate or not) than to 
> allow spammers to blast full speed.

I agree.  I put that in the TOPS-20 SMTP server 23 years ago.

Of course, I think that you meant to say "before sending the CRLF that 
terminates the greeting" instead of "after sending the greeting".

> And why worry about all that complexity of caching server capabilities when 
> the server can be explicit about it in the banner to save the round-trips in 
> all cases?

I agree.  I was terribly disappointed when the NNTP people rejected this.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Chris Newman <[email-masked]>
Date: Thu, 23 Feb 2006 16:13:35 -0800

> instead of being fixed up, etc.

Well, if a server advertises all of BURL, PIPELINING, and AUTH PLAIN/EXTERNAL, 
then it must permit AUTH PLAIN/EXTERNAL to be pipelined in order to comply with 
the BURL spec.  A bit convoluted, but that's what the specs say.  Yes, this 
would probably be better done in an RFC 2554 update, but I don't have time to 
write one.

> using an EHLO keyword line such as 250 PIPEAUTH PLAIN EXTERNAL.

BURL is a submit-only extension with a normative reference to 
draft-gellens-submit-bis-02 which requires authentication.  So it's safe for 
clients to assume authentication is required if both BURL and AUTH are 
advertised.

                - Chris

---

From: Ned Freed <[email-masked]>
Date: Thu, 23 Feb 2006 16:36:56 -0800 (PST)

> a separate capability from PIPELINING or QUICKSTART.

We haven't released our code with this feature yet so we don't have a lot of
experience with it, but my guess is ISPs will want to use this trick to block
spam from infected PCs. And I doubt the authentication requirement for submit
will make much difference here: If these spambots don't already know how to dig
the necessary credentials out of the PC's configuration they soon will.

in other words, I'm dubious this makes sense even on the submit port.

> server can be explicit about it in the banner to save the round-trips in all

Full agreement here. Caching server capabilities is something we should
avoid it at all possible.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Fri, 24 Feb 2006 09:51:08 +0000

On Thu, 23 Feb 2006, Chris Newman wrote:
> then it must permit AUTH PLAIN/EXTERNAL to be pipelined in order to comply
> with the BURL spec.  A bit convoluted, but that's what the specs say.  Yes,
> this would probably be better done in an RFC 2554 update, but I don't have
> time to write one.

In that case I suggest that the BURL spec should have an explicit
Updates: RFC2554 annotation, since it changes the behavior of AUTH.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAIR ISLE: NORTHEAST 4 OR 5, OCCASIONALLY 6 LATER. RAIN LATER. MAINLY GOOD.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 24 Feb 2006 10:25:09 +0000

>Updates: RFC2554 annotation, since it changes the behavior of AUTH.
Indeed.

Also note that Rob Siemborski did a revision of RFC 2554. Even though 
his draft has expired, I thought it was in a good shape and probably it 
is time to Last Call it now.

---

From: Tony Finch <[email-masked]>
Date: Fri, 24 Feb 2006 10:43:09 +0000

On Thu, 23 Feb 2006, Chris Newman wrote:
> So the client can issue the command and immediately begin the TLS
> negotiation without waiting for the server response to the QUICKTLS
> command.

Nice! I suppose this is a fair replacement for anticipating the server's
greeting. Instead of

        C-->S: TCP SYN
        C<--S: TCP SYN/ACK
        C-->S: TCP ACK + SMTP QHLO, STARTTLS
        C<--S: SMTP quickstart greeting, 250 OK, 220 OK
        C-->S: TLS hello
        C<--S: TLS hello, certificate, etc.

you have

        C-->S: TCP SYN
        C<--S: TCP SYN/ACK
        C-->S: TCP ACK
        C<--S: SMTP quickstart greeting
        C-->S: SMTP QUICKTLS + TLS hello
        C<--S: TLS hello, certificate, etc.

(Hmm, I don't think you combine them because of the need for error
handling, and it might be tricky for an implementation to handle
pipelining of SMTP and TLS in both directions...)

> The ability for the client to tell the server the host name it is using
> to talk to the server permits virtual domains without burning IP
> addresses, another useful feature that STARTTLS is missing.

Excellent idea.

> the server can be explicit about it in the banner to save the round-trips in

The server banner after TLS loses you the benefit of TLS session cacheing.
Here's an uncached exchange:

        C-->S: SMTP QUICKTLS + TLS hello
        C<--S: TLS hello, certificate, etc.
        C-->S: TLS key exch, change cipher, finish handshake
        C<--S: TLS change cipher, finish handshake, SMTP greeting
        C-->S: SMTP AUTH + SASL PLAIN

And cached:

        C-->S: SMTP QUICKTLS + TLS hello
        C<--S: TLS hello, change cipher, finish handshake
        C-->S: TLS change cipher, finish handshake
        C<--S: SMTP greeting
        C-->S: SMTP AUTH + SASL PLAIN

This is why I wanted the client to be able to talk first after the TLS
handshake. I can't see how to allow this without a capability cache.

Tony.

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Feb 2006 13:33:58 +0000

On Thu, 23 Feb 2006, Arnt Gulbrandsen wrote:
> Tony Finch writes:
> > message transmission.
> Cool, yes, very, but useful? The only use case I can think of is a spammer
> using a mobile-phone botnet to send lots of mail.

It's probably more useful for MTA-MTA transmission than message
submission. Think of dealing with the queue after an outage, for example.
The MTAs I know well aren't capable of this level of performance because
of the faff they incur to switch between messages on the same connection,
but implementation limitations shouldn't restrict the protocol design.
Perhaps Ecelerity could make good use of the feature :-)

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MULL OF KINTYRE TO ARDNAMURCHAN POINT: NORTHEAST 4 OR 5, OCCASIONALLY 3 AT
FIRST. RAIN FOR A TIME. MODERATE OR GOOD. SLIGHT OR MODERATE.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Feb 2006 14:26:52 +0100

Tony Finch writes:
> I think it would be cool to allow this kind of high-bandwidth 
> asynchronous message transmission.

Cool, yes, very, but useful? The only use case I can think of is a 
spammer using a mobile-phone botnet to send lots of mail.

Arnt

---

From: "Rob Siemborski" <[email-masked]>
Date: Fri, 24 Feb 2006 08:14:11 -0800

I was holding that for when 2222bis finally got published.  It looks
like its making progress again so I'll have to go back through and
double check that 2554bis is still ok, then publish it.

-Rob

--
Rob Siemborski        |     PGP:0x5CE32FCC    |     [email-masked]
Software Engineer     |                       |            650-623-6925