Subject: SMTP compressed protocol...

From: Franck Martin <[email-masked]>
Date: 05 Dec 2003 15:29:54 +1200

While talking about HTML in e-mail messages that consume a lot of
bandwidth...

Why SMTP servers do not negotiate to send an 8bit compressed stream
between themselves. The same way HTTP negotiate a compressed stream
between client and server if the client has the capability...

Cheers

----
Franck Martin
[email-masked]
SOPAC, Fiji
GPG Key fingerprint = 44A4 8AE4 392A 3B92 FDF9  D9C6 BE79 9E60 81D9 1320
"Toute connaissance est une reponse a une question" G.Bachelard

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Dec 2003 01:11:13 -0500

--On Friday, 05 December, 2003 15:29 +1200 Franck Martin 
<[email-masked]> wrote:

> While talking about HTML in e-mail messages that consume a lot
> of bandwidth...
>
> Why SMTP servers do not negotiate to send an 8bit compressed
> stream between themselves. The same way HTTP negotiate a
> compressed stream between client and server if the client has
> the capability...

When the relevant WG looked briefly at the question a _long_ 
time ago (in Internet years), the conclusion was that it wasn't, 
in general, worth the trouble.  Or, if you prefer, worth the 
trouble often enough to justify the effort.   That conclusion 
was conditioned, if I recall, by the combination of several 
things:

(1) At the time, there was no obvious compression algorithm 
(given IPR encumbrances, etc.) and standards-conforming 8bit 
transport, having just been defined, was obviously not widely 
deployed.   That second condition has obviously changed.

(2) Relaying complicates everything with SMTP, since one could 
not guarantee that a negotiation was going on between sending 
and recipient machines.  If one had to compress between sender 
and initial relay, then have the relay decompress (and maybe 
recompress using a different algorithm) in order to pass the 
message along, it might cancel out any possible efficiencies.

(3) Some message body parts (usually "attachments"), especially 
large ones, are already compressed (e.g., zip files or 
equivalent) or not effectively compressible (e.g., almost 
anything encrypted), reducing the value of message transport 
compression techniques.  My very subjective and anecdotal 
impression is that fewer people are routinely compressing 
attachments these days, possibly because MUAs have gotten better 
at building attachments on a one-click basis that doesn't 
provide for compression at attachment time.  Similarly, I 
suspect that the amount of material that is being mailed that 
has very low information density per number of bits transmitted 
(HTML being only one example) is on the rise.   But others may 
have different experience and impressions.

(4) Operationally, the most important requirement for 
compression arises between the endpoints of a slow and/or 
expensive and/or intermittent point to point link (at SOPAC, you 
are probably very familiar with those).  For those situations, 
usually the right thing to do is to (i) set up MX records that 
force mail to end up on the top end of such a link, (ii) have 
that machine aggregate an entire data stream, presumably 
consisting of several messages, (iii) compress that stream and 
send it via some sort of "batch SMTP" or local protocol, (iv) 
decompress and disaggregate at the far end and either go back to 
SMTP or just distribute the results into a mail store, as 
appropriate.  That model compresses not only the message content 
but also the headers and envelope and, more important in many 
cases, eliminates all of the per-message command-reply 
turnarounds (which Pipelining merely reduces).   Of course, that 
approach has been in use over such links for years and years, 
starting long before the first discussions that led to MIME and 
8bit email transport.

(5) Finally, any scheme for compressing entire message bodies 
for transport purposes that doesn't also batch the messages 
themselves needs to deal with the inconveniences created by the 
incomplete separation of envelope and headers in SMTP, i.e., the 
fact that MTAs are required to insert trace ("Received:" and 
ultimately "Reply-to:") fields into message bodies without any 
knowledge of what else is going on with the message body, 
including with previously-applied "Received:" fields.  RFC 1869 
and its predecessors, and then 2821, raised the bar, but RFC 821 
does not explicitly require that message bodies have headers, or 
that those headers be in 822 format, as long as it is possible 
to prepend those trace fields.

Today, I would also worry a bit that compression might turn out 
to be the enemy of various strategies for early interception and 
repelling of spam.  One should at least think about that issue 
when contemplating compression schemes.

All of that said, if you think it would be worthwhile, 
especially after thinking about (4), I'd recommend proposing it. 
You would need to think carefully through the model and 
practical implications of (5) but, otherwise, an appropriate 
ESMTP extension wouldn't be hard to design and write up.  If you 
had such a proposal written up, even in outline, the right place 
to discuss it is probably the ietf-smtp list, hosted at imc.org. 
Only by starting from a specific writeup would you be likely to 
get a good handle on whether the idea would get enough traction 
to be worth pursuing further.

regards,
    john

p.s. Don't you know you aren't supposed to raise technical 
issues on the IETF list?  It might drop the noise to signal 
ratio below infinity, which many of those who seem to post the 
most messages to the list might find very disappointing.   :-(

---

From: stanislav shalunov <[email-masked]>
Date: Fri, 5 Dec 2003 01:38:19 -0500

On Fri, Dec 05, 2003 at 03:29:54PM +1200, Franck Martin wrote:
> between themselves.

In addition to John Klensin's excellent summary I might notice that one
more consideration would be the total amount of network capacity that is
consumed by email.  Looking at this backbone, all kinds of mail-related
protocols (SMTP, POP3, IMAP, etc.) combined use less than 0.5% of total
used capacity.

(I do realize that in some environments, the fraction might be very
different.)

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 5 Dec 2003 07:40:09 +0100

Nice discussion of the problem space.

> (1) At the time, there was no obvious compression algorithm (given IPR 
> encumbrances, etc.)

Let me quickly add that RFC3320 was designed to solve that specific 
problem (and the resulting deployment economy/interoperability 
problems).  The RFC was developed in the context of signalling (SIP), 
but certainly also can be applied to protocols like SMTP.

Gruesse, Carsten

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Dec 2003 02:15:43 -0500

> Nice discussion of the problem space.

thanks.

>> (given IPR  encumbrances, etc.)
>
> Let me quickly add that RFC3320 was designed to solve that
> specific problem (and the resulting deployment
> economy/interoperability problems).  The RFC was developed in
> the context of signalling (SIP), but certainly also can be
> applied to protocols like SMTP.

Yep.  As I tried to say, the circumstances now have evolved 
enough from the circumstances when ESMTP and MIME were being 
designed that the whole issue may be worth another look.  Of 
course, of the list I gave, the compression issues are among the 
easiest.  For me at least, the question of what problem is 
actually to be solved and, hence, whether the important 
questions are better approached by message aggregation and some 
sort of compressed batch SMTP mechanism, is the key one.   But I 
don't spend nearly as much time at the end of rotten links, or 
working really closely with those who do, as I did a decade or 
more ago. Those people who do --presumably including Franck-- 
are the ones who should sort those tradeoffs out.   Stanislav's 
message, and a bit of the analysis it stimulates, illustrates 
the importance of that quite effectively...

--On Friday, 05 December, 2003 01:38 -0500 stanislav shalunov 
<[email-masked]> wrote:

> In addition to John Klensin's excellent summary I might notice
> that one more consideration would be the total amount of
> network capacity that is consumed by email.  Looking at this
> backbone, all kinds of mail-related protocols (SMTP, POP3,
> IMAP, etc.) combined use less than 0.5% of total used capacity.

With the size of some of the things that go one in the Internet2 
backbone, perhaps precisely because of its capacity and design 
structures, it would be a bit surprising if it were otherwise. 
I'd also guess that, given the kind of connectivity Internet2 
implies, that you see very little mail relaying in practice 
(beyond initial submission).  But that is part of the analysis 
issue: given Internet2's bandwidth and performance 
characteristics, I'd think it would be pretty close to nuts to 
spend energy and processing cycles on email compression for 
individual messages, even if messaging traffic represented a 
more significant fraction of network traffic.  By responding to 
your comment and Carsten's in a single message, I save far more 
network resources on Internet2 and the machines attached to it 
(connection openings, waiting around for SMTP handshake/ 
turnarounds, etc.) than could possibly be saved by fussing with 
compression on a message of this size.

But compression, properly thought out, might still be very 
useful at the edges of the network with lower levels of 
resources and bandwidth... I just don't know.

> (I do realize that in some environments, the fraction might be
> very different.)

Yes.  And I don't mean to pick on you, and apologize for any 
appearance of doing so.  But the example just made part of the 
issue so clear to me that I thought putting in a different 
context might help others.

best,
    john

---

From: stanislav shalunov <[email-masked]>
Date: Fri, 5 Dec 2003 02:45:37 -0500

On Fri, Dec 05, 2003 at 02:15:43AM -0500, John C Klensin wrote:
> (beyond initial submission).

The backbone in question (Abilene) would see most traffic from a US
research university to a US research university (for example, this message
would not traverse Abilene because the IETF mail reflector does not have
Abilene connectivity).  Perhaps to put the fraction of traffic that is
email in a better perspective, the ratio of HTTP traffic to email traffic
is more than 20.  (I'm not claiming that inter-university traffic is a
representative sample of anything, but this might be a useful datapoint.)

> resources and bandwidth... I just don't know.

SMTP-level email compression might very well be useful for some edges.
I would think that the challenge would be to design it so that these
edges could take advantage of it, while better-connected sites would not
need to spend CPU cycles compressing and decompressing when exchanging
email among themselves.

One way to accomplish that could be to have two compression-related ESMTP
options: ``support compression'' and ``prefer compression.''  Any site
might support compression, while only capacity-starved edges with a large
fraction of traffic that is email might prefer compression (at the site
administrator's discretion).  Then, message SHOULD be compressed if and
only if both ESMTP peers support compression and at least one of the
peers prefers compression.  This way, only sites that need compression
would end up using it -- for both incoming and outgoing mail.

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Dec 2003 03:38:20 -0500

--On Friday, 05 December, 2003 02:45 -0500 stanislav shalunov 
<[email-masked]> wrote:

> The backbone in question (Abilene) would see most traffic from
> a US research university to a US research university (for
> example, this message would not traverse Abilene because the
> IETF mail reflector does not have Abilene connectivity).
> Perhaps to put the fraction of traffic that is email in a
> better perspective, the ratio of HTTP traffic to email traffic
> is more than 20.  (I'm not claiming that inter-university
> traffic is a representative sample of anything, but this might
> be a useful datapoint.)

Last time I looked at data from a public/ commercial backbone, 
the ratio was a _lot_ lower than that.  But things might have 
changed.

> SMTP-level email compression might very well be useful for
> some edges. I would think that the challenge would be to
> design it so that these edges could take advantage of it,
> while better-connected sites would not need to spend CPU
> cycles compressing and decompressing when exchanging email
> among themselves.
>
> One way to accomplish that could be to have two
> compression-related ESMTP options: ``support compression'' and
> ``prefer compression.''  Any site might support compression,
> while only capacity-starved edges with a large fraction of
> traffic that is email might prefer compression (at the site
> administrator's discretion).  Then, message SHOULD be
> compressed if and only if both ESMTP peers support compression
> and at least one of the peers prefers compression.  This way,
> only sites that need compression would end up using it -- for
> both incoming and outgoing mail.

>From a simple syntax standpoint, one option with an advertised 
"prefer/ accept" parameter would do the job, and would be much 
preferable architecturally to two options.  That would 
especially be the case if it were desired to support multiple 
compression options with one as a "must support" variation.

But I'd guess that, in practice, that much complexity might not 
be needed.  And, again, in a capacity-starved environment, 
especially one that was also connection-time-limited, individual 
message compression might be the wrong thing to do.  The fact 
that we could figure out how to do it doesn't make it the right 
solution.  And something batch-oriented also solves the "what to 
do about those Received headers" problem (it also just occurred 
to me that Received headers really need to be kept 
clear/uncompressed for loop control purposes, so compressing 
even some of them would not be a good idea, at least with some 
scenarios of how mail transport compression might work in 
practice.  And that would imply that the compressor needs to 
parse message headers.

So let me explicitly appeal for what my first message merely 
implied: let's do the problem analysis here, looking at the 
entire email transport and delivery system in the context of 
low-capacity and/or connect-time-constrained links, before we 
start leaping ahead to solutions.

regards,
    john

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 5 Dec 2003 07:12:10 -0600

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Franck Martin" <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, December 05, 2003 12:11 AM
Subject: Re: SMTP compressed protocol...

> most messages to the list might find very disappointing.   :-(

John can always count on me ... (heh, heh, heh)

This is actually a nice discussion (people thinking about how to make
the world a better place, other people remembering what was talked
about in the past while being realistic about whether things have
changed enough to make different decisions reasonable). I would have
thought a reasonable discussion would be *more* disaappointing than a
falling noise to signal ratio, based on the last week or so of
traffic.

Having said this, it seems to me that the people who are most likely
to care about mail compression are at the extremes - either people who
are working with what I consider extremely large files (as short as
the Problem Statement working group meeting was in Minneapolis, it
recorded as about 220 MB of MPEG1 audio/video), or working with
extremely slow/high latency access links (almost always wireless).

The people who are working with extremely large files have the most
reason to compress before transmission, including the part where mail
quotas are enforced before/after transmission, so that's the size that
really matters.

The people who are working with what PILC called either "lossy" or
"lousy" links (I believe this was a Vern Paxson Freudian slip)
probably do more with POP3 or IMAP4 than with SMTP (there are people
on this list who seem to send more e-mail than they receive, but I
don't think that's the general case!), so any compressing solution for
sending mail would need to think in terms of receiving mail as well.
This is probably why we seem to handle these links with per-hop
compression, rather than anything end-to-end.

But it is nice to see discussions like this taking place.

Now I can go back to installing a root server on my laptop for all of
North America (in other words, back to business as usual!)

Spencer

---

From: John Stracke <[email-masked]>
Date: Fri, 05 Dec 2003 08:43:55 -0500

>Why SMTP servers do not negotiate to send an 8bit compressed stream
Doesn't STARTTLS take care of that? Doesn't TLS support compression, to 
eliminate redundancy and resist cryptanalysis?

-- 
/=====================================================\
|John Stracke      [email-masked]              |
|Principal Engineer|http://www.centive.com            |
|Centive           |My opinions are my own.           |
|=====================================================|
|"The Reality Check's in the mail." --L. Peter Deutsch|
\=====================================================/

---

From: stanislav shalunov <[email-masked]>
Date: Fri, 5 Dec 2003 17:10:16 -0500

On Fri, Dec 05, 2003 at 03:38:20AM -0500, John C Klensin wrote:
> preferable architecturally to two options.

Yes.

> connection-time-limited

What's a connection-time-limited environment?

> And something batch-oriented also solves the "what to 
> do about those Received headers" problem

As well as the the extra round-trip times problem associated with SMTP.

A disadvantage of batch-oriented approaches is that mail rejection could
happen later.

---

From: [email-masked]
Date: Fri, 05 Dec 2003 19:31:06 -0500

> What's a connection-time-limited environment?

One that is either pay-by-the-minute or has a MTBF low enough to be a serious
hazard to any given transaction in progress.

---

From: [email-masked]
Date: Sat, 06 Dec 2003 10:03:03 -0800 (PST)

> eliminate redundancy and resist cryptanalysis?

Not quite at present, but we're getting close:

  http://www.ietf.org/internet-drafts/draft-ietf-tls-compression-06.txt

The last call for proposed for this document went out yesterday.

I also note in passing that object level compression in MIME is the
subject of this draft:

  http://www.ietf.org/internet-drafts/draft-freed-mime-newenc-00.txt

				Ned