Subject: [lemonade] Replying to or editing a big list

From: "Burger, Eric" <[email-masked]>
Date: Wed, 9 Nov 2005 14:31:03 -0500

There was a lot of discussion in the lemonade session at IETF 64 around
the issues with replying to or editing a big list of destinations.

A whole bunch of people thought the problem (a) is not real, (b) will go
away shortly, or (c) encourages stupid use of addresses over creating
lists.

How do we want to proceed?

1. Punt; try to convince OMA the requirement is not useful.

2. Take up the work item

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Wed, 09 Nov 2005 21:00:15 +0000

> How do we want to proceed?

Ted (I think) mentioned that trying to use technical limitations to 
people doing things which are, in our considered opinion, stupid, is 
not a good thing to do.

I'm inclined to agree. I think huge address lists in the headers are 
silly, too, but people do them, and the OMA wants a solution.

Since I've a small test implementation of a Lemonade submission 
server, I'll try implementing a couple of ideas and see whether they 
work in practise, and what the saving actually seems to be.

FWIW, my suggestion is (copied from a private mail to Alexey and 
Stéphane some time ago):

First problem: Header rewriting.

For this, it makes most sense to define a new pseudo-section, with 
the following syntax (or similar):

section-spec =/ section-addresses
section-addresses = "ADDRESS" SP "(" address-type *( SP address-type 
) ")" [ SP sequence-set ]
address-type = "ALL" / "FROM" / "TO" / "CC" / "BCC" / "REPLYTO"
	; My ABNF is probably wrong.

Where the sequence set element denotes which addresses, in the order 
they appear in the
message. This evaluates to a list of addresses as they'd appear in a 
header.

Then URLs work, and we're done. It's an utter waste having it 
supported in FETCH BODY[...] as well, but it makes the URL spec 
easier, and it might prove useful.

We have to specify that long lists MUST be folded if required to fit 
inside the 1000 octet line limit of MIME.

By the way, this looks *really* ugly in URL encoded form.

Second problem: ESMTP RCPT TO:

Perhaps a new command which extracts the recipient addresses from a 
URL?

In any case, this represents two extensions, one to Submission and 
one to IMAP.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Nov 2005 13:06:37 -0800

On Wed, 9 Nov 2005, Dave Cridland wrote:
> Ted (I think) mentioned that trying to use technical limitations to people 
> doing things which are, in our considered opinion, stupid, is not a good 
> thing to do.

FWIW, it was Dave Crocker.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Nov 2005 13:08:27 -0800

On Wed, 9 Nov 2005, Dave Cridland wrote:
> For this, it makes most sense to define a new pseudo-section

Instead of inventing yet another representation, why not a means to access 
ENVELOPE or portion thereof?

-- Mark --

---

From: "Stephane H. Maes" <[email-masked]>
Date: Wed, 9 Nov 2005 14:28:30 -0800

I would like to still investigate the different options. 

_____
Stephane H. Maes, PhD,
Director of Architecture - OCS & Mobile, Oracle Corporation.
Ph: +1-203-300-7786 (mobile/SMS); Fax / Office UM: +1-650-607-6296.
e-mail: [email-masked]
IM: shmaes (AIM/Y!/skype) or [email-masked] (MSN Messenger)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Burger, Eric
Sent: Wednesday, November 09, 2005 11:31 AM
To: [email-masked]
Subject: [lemonade] Replying to or editing a big list

2. Take up the work item

---

From: Dave Cridland <[email-masked]>
Date: Wed, 09 Nov 2005 23:06:59 +0000

On Wed Nov  9 21:08:27 2005, Mark Crispin wrote:
> Instead of inventing yet another representation, why not a means to 
> access ENVELOPE or portion thereof?

I was thinking that it made it easier to work into URLAUTH, and 
specifying the formatting as suitable for a header, it would make it 
easier to use it with CATENATE/BURL. Note that this isn't a "new 
representation", per se, but a reuse of an existing one.

Since the RCPT TO aspect to the problem requires a new verb anyway, I 
ignored the formatting issue there, and assumed that in practise, the 
URL would "resolve" to an abstract concept.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 10 Nov 2005 00:04:58 +0000

On Wed Nov  9 21:00:15 2005, Dave Cridland wrote:
> server, I'll try implementing a couple of ideas and see whether 
> they work in practise, and what the saving actually seems to be.
> 
> 
Okay, done in the server, although I need to do it in the client as 
well in order to test properly.

I decided to use the order of the addresses in the ENVELOPE (because 
it was easier at the time), and expanded section-addresses to include 
SENDER.

ALL expands to FROM SENDER REPLYTO TO CC BCC - the order in the 
Envelope.

I can't change the IMAP server, so I'm intercepting the requests as 
they pass through the API, and I have no URLAUTH support anyway, 
which is the only point at which I'd need support in the IMAP server.

For ESMTP, I've added a verb AURL, which does for RCPT TO what BURL 
does for BDAT. This currently fails on the first address it doesn't 
accept, which I don't like. It has a EHLO keyword of AURL, which 
operates the same way as BURL.

Should anyone want an account on the submit server to play with, and 
a corresponding account on the trusted IMAP server (it also supports 
BURL, BDAT, BINARYMIME, CHUNKING, etc) feel free to drop me a line.

Dave.

---

From: Randall Gellens <[email-masked]>
Date: Wed, 9 Nov 2005 19:06:27 -0800

I'd like to better understand the problem.

SMTP requires that servers support at least 100 recipients per 
message.  In the envelope, the phrase is not present, so the number 
of octets in each address is fairly limited.  In the text, the phrase 
can be present, so each address can be a bit longer.  In SMTP, the 
local part is limited to 64 octets, the domain 255, so 319.  In 
practice, actual addresses tend to be *much* shorter, since they are 
printed on business cards, memorized by people, etc.  I'd guess that 
it would be unusual to see addresses longer than 30 characters or so, 
but of course I could be mistaken.

So, for the envelope portion, 100 addresses can be a theoretical 
maximum of 31,900 octets, plus of course "RCPT TO:<" and ">CRLF" for 
each (an extra 12 octets per address), so a grand total of 33,100 
octets.  At 30 octets per address, plus the 12 octets of protocol, 
we'd have a maximum of 4,200 for the 100 recipients.

Transforming addresses from 2822 to 2821 form can be challenging. 
Currently, there is an expectation that clients do this when creating 
a reply.  Do we want to move this task to the submit server?

2822 forms can be lengthy (I'm not sure 2822 imposes any specific 
limit, other than the 998 octet limit on each line).  But still, are 
they that long in practice?

What I'm asking is: (1) How many octets are being sent now, and by 
how much are we trying to reduce this?  (2) What specific tasks are 
asking which entities to perform?
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The brain is a wonderful organ; it starts working the moment you get
up in the morning, and does not stop until you get to work.
                                                      --Robert Frost

---

From: Dave Cridland <[email-masked]>
Date: Thu, 10 Nov 2005 09:10:20 +0000

On Thu Nov 10 03:06:27 2005, Randall Gellens wrote:
> I'd guess that it would be unusual to see addresses longer than 30 
> characters or so, but of course I could be mistaken.
> 
> 
Over the 5773 addresses my client has captured (on read) to my 
capture addressbook, I see an average of 22.4 octets, a min of 7, and 
a max of 73.

> maximum of 31,900 octets, plus of course "RCPT TO:<" and ">CRLF" 
> for each (an extra 12 octets per address), so a grand total of 
> 33,100 octets.  At 30 octets per address, plus the 12 octets of 
> protocol, we'd have a maximum of 4,200 for the 100 recipients.
> 
> 
You're missing out the extra data coming back in the other direction, 
but yes.

> Currently, there is an expectation that clients do this when 
> creating a reply.  Do we want to move this task to the submit 
> server?
> 
> 
This would probably fall out of any design we come up with to solve 
this issue - if we do.

> limit, other than the 998 octet limit on each line).  But still, 
> are they that long in practice?
> 
> 
I'll write a program to analyze the phrase lengths as well.

> asking which entities to perform?

The other issue is how much complexity are we introducing by 
attempting to solve it. On the server side, it's not too bad at all. 
I'm fully expecting the client side to be a nightmare, though.

Dave.

---

From: Randall Gellens <[email-masked]>
Date: Thu, 10 Nov 2005 10:39:46 -0800

At 9:10 AM +0000 11/10/05, Dave Cridland wrote:

>> how much are we trying to reduce this?  (2) What specific tasks 
>> are asking which entities to perform?
>
> attempting to solve it. On the server side, it's not too bad at 
> all. I'm fully expecting the client side to be a nightmare, though.

I was trying to get to your questions by starting with mine.  Once we 
know the current cost and the expected reduction, we will be in a 
better position to evaluate if the complexity cost is greater than 
the octet savings.

In reply to an earlier thread on the same topic: I'm not sure we are 
trying to prohibit by technology users doing "something stupid", 
because I don't think we're trying to prohibit the use of lots of 
addresses.  We're trying to see if an expressed desire to reduce the 
bandwidth usage of using lots of addresses is worth the cost.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
All programmers are playwrights and all computers are lousy actors.

---

From: "Stephane H. Maes" <[email-masked]>
Date: Thu, 10 Nov 2005 10:43:14 -0800

Note the issue at hand is about editing and sending diff on any body part. The use case given and initially used by OMA is about long lists. But I do not believe that simply characterizing this as of little interest address the broader issue behind the expectation for this.

Stephane

_____
Stephane H. Maes, PhD,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Burger, Eric
Sent: Wednesday, November 09, 2005 11:31 AM
To: [email-masked]
Subject: [lemonade] Replying to or editing a big list

2. Take up the work item

---

From: Jonathan Clark <[email-masked]>
Date: Thu, 10 Nov 2005 13:58:25 -0500

> a reply.  Do we want to move this task to the submit server?

I wasn't in London, but I'm sure that this is what OMA has in mind.

Having downloaded the To and Cc lines of the message to the device,
they don't want to upload them again (plus having them duplicated,
in some sense, in the SMTP envelope). So they envisage a server-side
Reply-All and for good measure they have thrown in the kitchen-sink
ability to edit this list. I'm not sure that the editing is worthwhile,
but I can see the validity of building the address list on the server.

I note in passing that pulling the To and Cc lines out of the message
in the store also finesses the Bcc problem of storing the SMTP envelope
contents.

Jonathan Clark
Solutions Architect
Mirapoint Inc
--

---

From: Dave Cridland <[email-masked]>
Date: Thu, 10 Nov 2005 20:28:51 +0000

> protocol, we'd have a maximum of 4,200 for the 100 recipients.
Okay. I'm reasonably confident that your 30 octets is too high. The 
average seems to be around 22, so this figure drops to 34 octets 
typically per SMTP command, hence 3400 octets.

What I'm not so sure about is the responses - even assuming the 
entire recipient list is sent in a single pipeline, I suspect each 
response will be sent in a distinct packet in most cases. That in 
itself is probably not a problem, but what concerns me here is that 
this may incur a noticeable cost in terms of ACKs - I'm not terribly 
clear on whether each TCP packet would cause a seperate ACK.

For the header, a 100 address CC looks like it would take up 4198 
octets - the average length of a header (including MIME usage, etc) 
based on my capture addressbook appears to be 40 octets, plus a ", " 
between each one. For comparison, a CATENATE URL using my (probably 
awful) proposal takes around 93 octets:

/INBOX;UIDVALIDITY=1077560755/;uid=29808;section=address%20(from%20to%20cc)%201%3A78%2C80%3A*

The saving in bandwidth is quite straightforward there.

> creating a reply.  Do we want to move this task to the submit 
If we wanted to move that task, then moving the task of creating the 
In-Reply-To and References headers would also be targets, suggesting 
that both CATENATE and BURL require methods to generate the required 
headers. Unfortunately, the majority of clients still don't generate 
these headers correctly.

> are they that long in practice?
By the statistics I have to hand, 40.0 octets. (Minimum 18, maximum 
149 - since this includes this list, using the phrase part extracted 
from the List-Id header, I imagine that's the 149.)

> (1) How many octets are being sent now, and by how much are we 
> trying to reduce this? 

I hope this is beginning to answer that question. Loosely, we're 
looking at savings for headers containing more than 3 addresses, and 
for SMTP envelopes containing more than 4.

>  (2) What specific tasks are asking which entities to perform?

This question I'm really not sure I can answer.

Dave.

---

From: Philip Guenther <[email-masked]>
Date: Thu, 10 Nov 2005 13:08:17 -0800

Dave Cridland <[email-masked]> writes:
>response will be sent in a distinct packet in most cases.

If the RCPT commands are pipelined, I would expect the responses to be
pipelined back.  I know that both the sendmail MTA and our LMTP server
will do so.  I just traced pipelining multiple RCPTs and there were
three packets: the RCPTs in one, the ACK for that packet with the
responses in another, and then the ACK for that in a third.

>That in 

As a rule of thumb, in a one-way flow, every other packet will generate
an ACK.

Philip Guenther

---

From: Dave Cridland <[email-masked]>
Date: Thu, 10 Nov 2005 21:33:40 +0000

> responses in another, and then the ACK for that in a third.
Thanks. So that would suggest there would be the same number of 
transmissions from the mobile device's radio whether we use a single 
URL based RCPT replacement, or a bunch of RCPTs, or a pipelined 
mixture.

> >That in >itself is probably not a problem, but what concerns me 
> here is that >this may incur a noticeable cost in terms of ACKs - 
> I'm not terribly >clear on whether each TCP packet would cause a 
> seperate ACK.
> 
Again, thanks.

I'm working on the assumption - a possibly non-intelligent guess - 
that in terms of battery usage, there's a certain overhead to 
performing any transmission with a mobile device, hence the battery 
cost may tend toward per-packet rather than per-octet. I'd like some 
input on this, if possible.

Dave.

---

From: Ned Freed <[email-masked]>
Date: Thu, 10 Nov 2005 13:35:07 -0800 (PST)

> response will be sent in a distinct packet in most cases.

Why would you assume this? RFC 2920 explicitly discusses when a server can or
cannot buffer responses to commands. It doesn't require support for it (because
we wanted pipelining to work without forcing reasonable servers to change in
any way) but all the necessary information is there to tell imeplemtnros what
to do.

Most TCP stacks offer some control over when packets are or are not sent (e.g.,
the push bit) and it is hardly rocket science to check to see if there are
still commands in the input buffer and set the bit accordingly. (Local
buffering is also an option, of course. Our implementation uses both.)

Anyone who cares about latency issues implements server side pipelining. It is
actually considerably easier to implement than the client side, where
error handling is somewhat tricky.

If this really is important we can always specify that it needs to be
done as part of the lemonade SMTP profile.

> clear on whether each TCP packet would cause a seperate ACK.

I believe it depends on whether or not various optional TCP features can be
used. Given the quality issues I've seen in the mobile space I wouldn't assume
they're available.

> The saving in bandwidth is quite straightforward there.

The cost in terms of user astonishment may be large, however. The effect these
sorts of tricks have is to make it impossible for the SMTP server to report
per-recipient addressing errors to the client. The server has to fall back on
DSNs. I am very skeptical that mobile clients as a rule have high-quality
handling of DSNs.

And since not a day goes by that I don't get legitimate mail with bogus
addresses in the header (which then fail if I do a reply-all), I think this is
a fairly significant issue.

> > are they that long in practice?

FWIW, I've seen legitimate 10Mb headers in the wild listing tens of thousands
of addresses. (Once upon a time we have a parse stack overflow issue with such
headers, which is what broght them to my attention.) There used to be cases
where there was a legal requirement that all recipients of a message be listed
in every copy. I don't know if such cases still exist, but that was what
motivated the very large ones I've seen in the past.

> for SMTP envelopes containing more than 4.

I think the real question is whether or not clients will actually take
advantage of this if we specify it. The server side is, as you say, not
terribly difficult given that most servers already have the necessary code to
break down headers available. But for the forseeable future clients won't be
able to be sure that the extension is available, which means lots of duplicate
code paths. I don't write mobile clients myself so I really cannot say how much
code you can reasonably cram onto them.

				Ned

---

From: "Stephane H. Maes" <[email-masked]>
Date: Thu, 10 Nov 2005 15:38:23 -0800

Jonathan,

Having been involved in that OMA phase I can confirm that you are correct.

However as I pointed out this was just one of the use case. Desire is to cover diff updates of any body parts.

Stephane

_____
Stephane H. Maes, PhD,

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jonathan Clark
Sent: Thursday, November 10, 2005 10:58 AM
To: [email-masked]
Subject: Re[2]: [lemonade] Replying to or editing a big list

Jonathan Clark
Solutions Architect
Mirapoint Inc
--

---

From: Randall Gellens <[email-masked]>
Date: Thu, 10 Nov 2005 17:06:04 -0800

At 10:43 AM -0800 11/10/05, Stephane H. Maes wrote:

>  Note the issue at hand is about editing and sending diff on any 
> body part. The use case given and initially used by OMA is about 
> long lists. But I do not believe that simply characterizing this as 
> of little interest address the broader issue behind the expectation 
> for this.

Body parts and envelope addresses are two very different issues.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The idea that Bill Gates has appeared like a knight in shining armor
to lead all customers out of a mire of technological chaos neatly
ignores the fact that it was he who, by peddling second-rate technology,
led them into it in the first place.                    --Douglas Adams

---

From: Randall Gellens <[email-masked]>
Date: Thu, 10 Nov 2005 17:13:08 -0800

At 8:28 PM +0000 11/10/05, Dave Cridland wrote:

> based on my capture addressbook appears to be 40 octets, plus a ", 
> " between each one. For comparison, a CATENATE URL using my 
> (probably awful) proposal takes around 93 octets:
>
> 

This makes the question more clear, thanks: is the new protocol 
complexity worth cutting 4198 octets per worst-case message down to 
93?

Personally, since adding byte ranges to the URL helps in other areas, 
so there is very little incremental cost, I'd be inclined to say yes.

Note that we're only talking about the text here, not the envelope.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Power corrupts; Powerpoint corrupts absolutely.
            --Vint Cert

---

From: Randall Gellens <[email-masked]>
Date: Thu, 10 Nov 2005 17:38:31 -0800

>  motivated the very large ones I've seen in the past.

Since many servers now reject messages that have a large number of 
addresses (in the belief that this reduces spam), I'd suspect that 
this is less common than it may have once been.  I'd also guess this 
would tend to be less common on mobile devices, but I could easily be 
wrong here.

>  code you can reasonably cram onto them.

One of the goals for lemonade is to reduce the need for mobile 
clients to support alternative code paths.  By stamping a set of 
extensions as mandated by lemonade, mobile clients can be written to 
only support lemonade-compliant servers.  Of course, a primary goal 
is to see widespread deployment of the lemonade-mandated extensions, 
and including little-used ones in that set hurts this.  So we should 
only add things that really help clients and that both clients and 
servers are likely to implement.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Everyone knows that dragons don't exist.  But while this simplistic
formulation may satisfy the layman, it does not suffice for the
scientific mind.  The School of Higher Neantical Nillity is in fact
wholly unconcerned with what does exist.  Indeed, the banality of
existence has been so amply demonstrated, there is no need for us to
discuss it any further here.  The brilliant Cerebron, attacking the
problem analytically, discovered three distinct kinds of dragon: the
mythical, the chimerical, and the purely hypothetical.  They were all,
one might say, nonexistent, but each nonexisted in an entirely
different way......

---

From: "Stephane H. Maes" <[email-masked]>
Date: Thu, 10 Nov 2005 20:16:39 -0800

Agreed but that is the generic requirements from OMA.
S.

_____
Stephane H. Maes, PhD,

---

From: Dave Cridland <[email-masked]>
Date: Fri, 11 Nov 2005 08:51:35 +0000

On Thu Nov 10 21:35:07 2005, Ned Freed wrote:
> Why would you assume this?

Because I'm as much of a pessimist about server quality as the server 
implementors are about client quality?

> a fairly significant issue.
Yes, you'd need to indicate which addresses passed or failed in the 
response to the magic envelope command. The precision of the error 
would be particularly awkward to convey, by which I mean ugly rather 
than difficult.

> code you can reasonably cram onto them.

I'm not actually too fussed about the code path issue. I'm capable of 
writing multiple code paths, and while it's tricky to test, this is 
why I wrote a submission server I could flip the extension support on 
and off for. The much more complex code path to deal with is, in any 
case, the decision on whether to assemble the message over submission 
or IMAP.

What I am aware of is that some clients may choose to always 
implement only one code path (in the example above, always perform 
message assembly over IMAP). I believe I'm close to adequately 
arguing that this particular method for reduce bandwidth in some 
cases is sufficiently rarely going to lead to a saving that either:

a) It will make single-code-path clients less efficient in the 
majority of cases
b) It will make single-code-path clients ignore it.

Personally, I'd have to move various bits of message assembly 
relating to headers all over the place to take advantage of this, so 
I'm not terribly keen to do this unless people really care.

Dave.