Subject: [lemonade] checkpoint/resume, rfc1845bis

From: Tony Finch <[email-masked]>
Date: Wed, 8 Nov 2006 21:52:08 +0000

Following suggestions from Dave and Alexei, I've finally finished a draft
of rfc1845bis, rather later than I hoped...

This arose from the "REPLAY" draft I wrote back in March, though all it
has in common is that I wrote it. This new version is much more compatible
with RFC 1845, has better security, and clearer support for mobility.

http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/
draft-fanf-lemonade-rfc1845bis.html
draft-fanf-lemonade-rfc1845bis.txt
draft-fanf-lemonade-rfc1845bis.xml

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
IRISH SEA: NORTHWEST 5 TO 7 BECOMING VARIABLE 3. MODERATE BECOMING SLIGHT.
SQUALLY SHOWERS. GOOD.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: <[email-masked]>
Date: Thu, 9 Nov 2006 10:33:20 +0200

Good day Mr Finch,
I could not go to the meeting in San Diego, so I think I am missing something.
Would it be possible to clarify the intent with this draft? I assume the ultimate goal is to add it to the Profile-bis, right?
Thank You.

Best regards: Zoltán Ördögh
E-mail: zoltan dot ordogh at nokia dot com
Phone: +358 50 386 0566

>-----Original Message-----
>From: ext Tony Finch [mailto:[email-masked]] 
>Sent: 08 November, 2006 23:52
>To: [email-masked]
>Subject: [lemonade] checkpoint/resume, rfc1845bis
>
>Following suggestions from Dave and Alexei, I've finally 
>finished a draft of rfc1845bis, rather later than I hoped...
>
>This arose from the "REPLAY" draft I wrote back in March, 
>though all it has in common is that I wrote it. This new 
>version is much more compatible with RFC 1845, has better 
>security, and clearer support for mobility.
>
>f.a.n.finch  <[email-masked]>  http://dotat.at/ IRISH SEA: 
>NORTHWEST 5 TO 7 BECOMING VARIABLE 3. MODERATE BECOMING SLIGHT.
>SQUALLY SHOWERS. GOOD.
>

---

From: Dave Cridland <[email-masked]>
Date: Thu, 09 Nov 2006 09:30:05 +0000

On Thu Nov  9 08:33:20 2006, [email-masked] wrote:
> Would it be possible to clarify the intent with this draft? I 
> assume the ultimate goal is to add it to the Profile-bis, right?
> Thank You.

Several people at the WG meeting itself suggested that this wasn't 
actually wanted for Profile Bis. Essentially, this was based on 
concern about how difficult it would be to implement well and 
interoperably.

Personally, I think that this extension would be a welcome addition. 
There have been a few occasions where I've lost the connection after 
the message data has been sent but before the reply has been 
received, and while these are few, they represent a level of 
frustration. (If you develop your own mail client, and also work on 
the mailserver you use, it looks particularly bad if you send out a 
duplicate message, of course, so perhaps the problem is more acute 
for me than for most people).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: <[email-masked]>
Date: Thu, 9 Nov 2006 11:46:59 +0200

Hi Dave,
thank You for Your response.

>Several people at the WG meeting itself suggested that this 
>wasn't actually wanted for Profile Bis. Essentially, this was 
>based on concern about how difficult it would be to implement 
>well and interoperably.

Have those issues been expressed clearly?
Can those issues be addressed by updating the draft?

>There have been a few occasions where I've lost the connection 
>after the message data has been sent but before the reply has 
>been received, and while these are few, they represent a level 
>of frustration. (If you develop your own mail client, and also 
>work on the mailserver you use, it looks particularly bad if 
>you send out a duplicate message, of course, so perhaps the 
>problem is more acute for me than for most people).

I have felt this pain also while sending emails over unreliable (mobile) connections - while on the train, I could not send my email out until the train has stopped which actually jammed the whole connection due to constant retries.
Therefore, personally, I also agree with You, however I sense that the server vendors are not happy about such proposal. I would like to see those worries expressed before jumping into any conclusions.

---

From: Tony Finch <[email-masked]>
Date: Thu, 9 Nov 2006 10:32:02 +0000

> Have those issues been expressed clearly?

Have a look at the third paragraph of section 2.8, which I think is the
biggest problem. It's difficult to fix, even if you require big timeouts
as in RFC 1845 - and you don't want to do that for interactive mobile
clients.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER GERMAN BIGHT: NORTHWEST 6 TO GALE 8, OCCASIONALLY SEVERE GALE 9,
BACKING WEST 4 LATER. ROUGH TO VERY ROUGH, OCCASIONALLY HIGH. SQUALLY SHOWERS.
GOOD.

---

From: Tony Finch <[email-masked]>
Date: Thu, 9 Nov 2006 10:23:30 +0000

On Thu, 9 Nov 2006, [email-masked] wrote:

> Would it be possible to clarify the intent with this draft?

I became interested in improving RFC 1845 earlier this year so wrote a
replacement. Dave and Alexey recently expressed interest in an improved
version that would be more compatible with RFC 1845, so I re-wrote the
draft in the hope that it would get wider support.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HUMBER THAMES: NORTHWEST 5 TO 7, OCCASIONALLY GALE 8 IN HUMBER AT FIRST,
BACKING SOUTHWEST 4 OR 5 LATER. MODERATE TO ROUGH. SHOWERS. GOOD.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 09 Nov 2006 10:18:59 +0000

On Thu Nov  9 09:46:59 2006, [email-masked] wrote:
> Therefore, personally, I also agree with You, however I sense that 
> the server vendors are not happy about such proposal. I would like 
> to see those worries expressed before jumping into any conclusions.

As an aside here, I think it's possible to implement Tony's draft 
such that partially received transactions are dumped, and the server 
instead merely stores the result of transactions for a client to 
access in a subsequent session.

This would mean that a client losing a connection on a particularly 
large mail would have to resend in entirety, but on the other hand 
the problem I see as most prevelant - that of losing connectivity 
after sending the message, before receiving the result - is handled 
with minimal resource usage on the server.

Having the facility in the protocol for partial message resends would 
be useful, of course, but I believe the protocol as written would 
allow for a server to avoid the partial message storage that I 
suspect is one of the more onerous parts of the protocol.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Thu, 9 Nov 2006 10:36:35 +0000

On Thu, 9 Nov 2006, Dave Cridland wrote:
>
> As an aside here, I think it's possible to implement Tony's draft such that
> partially received transactions are dumped, and the server instead merely
> stores the result of transactions for a client to access in a subsequent
> session.

That is explicitly permitted in section 2.5 and there's some discussion in
section 4.1.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON: EAST 3 OR 4 VEERING SOUTH 5 TO 7, PERHAPS GALE 8 LATER. MODERATE TO
ROUGH. FAIR. GOOD.

---

From: Randall Gellens <[email-masked]>
Date: Thu, 16 Nov 2006 18:51:20 -0800

As we've come to expect, another well-written draft from Tony. 
Thanks for taking this on.

I do have a few comments, though:

In Section 2.4, under BURL, it says "The <octet-offset> SHALL count 
the size of the content retrieved from the URL parameter to the BURL 
command."  It might be difficult or at least prone to error for 
clients to have to calculate the exact offset of the final message 
data after BURL expansion.  Since the client isn't sending the data 
referenced by the BURL, I'm not sure of the benefit of including it.

It might be easier to say that the size includes the BURL itself (the 
reference) but not the data retrieved in response.  That way, if the 
client sends a BURL command and the connection drops before it sees 
the server's response, it will be clear on reconnection that the 
client doesn't need to send any data.

Typo in Section 3.2: "trandaction" instead of "transaction".

Typo in Section 3.3: parenthesized text starting "(using BDAT" has no 
close parenthesis.  (Second sub-bullet in first bullet.)

It occurs to me that there are weird potential edge cases where the 
response to a RCPT TO might be different when the message is resumed 
as when it was originally sent (assuming the server didn't receive 
the entire message and thus hasn't delivered it).  For example, if 
the recipient is local and there was a configuration problem with it, 
or it was over quota, the server might have issued a 4yz or 5yz code 
originally, but if the problem has since been cleared up, might be 
able to issue a 2yz when resumed.  But it is required to honor the 
original response.  In the case of a previous failed recipient, this 
is annoying.  In the case of a previous success recipient that now 
must be failed, the server must accept it and then generate a bounce. 
This is of course no different in RFrom [email-masked] Thu Nov 16 21:58:20 2006
Date: Thu, 16 Nov 2006 18:51:20 -0800
To: Tony Finch <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Re: [lemonade] checkpoint/resume, rfc1845bis
Cc: 

As we've come to expect, another well-written draft from Tony. 
Thanks for taking this on.

I do have a few comments, though:

In Section 2.4, under BURL, it says "The <octet-offset> SHALL count 
the size of the content retrieved from the URL parameter to the BURL 
command."  It might be difficult or at least prone to error for 
clients to have to calculate the exact offset of the final message 
data after BURL expansion.  Since the client isn't sending the data 
referenced by the BURL, I'm not sure of the benefit of including it.

It might be easier to say that the size includes the BURL itself (the 
reference) but not the data retrieved in response.  That way, if the 
client sends a BURL command and the connection drops before it sees 
the server's response, it will be clear on reconnection that the 
client doesn't need to send any data.

Typo in Section 3.2: "trandaction" instead of "transaction".

Typo in Section 3.3: parenthesized text starting "(using BDAT" has no 
close parenthesis.  (Second sub-bullet in first bullet.)

It occurs to me that there are weird potential edge cases where the 
response to a RCPT TO might be different when the message is resumed 
as when it was originally sent (assuming the server didn't receive 
the entire message and thus hasn't delivered it).  For example, if 
the recipient is local and there was a configuration problem with it, 
or it was over quota, the server might have issued a 4yz or 5yz code 
originally, but if the problem has since been cleared up, might be 
able to issue a 2yz when resumed.  But it is required to honor the 
original response.  In the case of a previous failed recipient, this 
is annoying.  In the case of a previous success recipient that now 
must be failed, the server must accept it and then generate a bounce. 
This is of course no different in RFC 1845, and perhaps is such an 
edge case that it isn't worth worrying about.  But if the client does 
resend the RCPT TO commands, perhaps the server could be permitted to 
send updated responses?  Presumably both the client and server have 
held the message in abeyance in the meantime, so both can act on the 
final results.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
When news of Alexander Graham Bell's invention reached the
United Kingdom, the chief engineer of the British Post Office
failed to be impressed.  "The Americans," he said loftily,
"have need of the telephone -- but we do not.  We have plenty
of messenger boys..."

---

From: Tony Finch <[email-masked]>
Date: Fri, 17 Nov 2006 11:46:29 +0000

On Thu, 16 Nov 2006, Randall Gellens wrote:

Thanks for looking through the draft! I've fixed the typos you spotted.

> In Section 2.4, under BURL, it says "The <octet-offset> SHALL count the size
> of the content retrieved from the URL parameter to the BURL command."  It
> might be difficult or at least prone to error for clients to have to calculate
> the exact offset of the final message data after BURL expansion.  Since the
> client isn't sending the data referenced by the BURL, I'm not sure of the
> benefit of including it.
>
> reference) but not the data retrieved in response.  That way, if the client
> sends a BURL command and the connection drops before it sees the server's
> response, it will be clear on reconnection that the client doesn't need to
> send any data.

Yes, this is an awkward design point. The alternative I considered was to
count a BURL as one octet, and your idea of counting the URL's length
would work equally well. However in 1845 the octet-offset nicely matches
the semantics of the SIZE parameter, and I wanted to preserve this. The
BURL spec isn't crystal clear about the client's use of SIZE, but since it
says that servers must apply SIZE checks (including the retrieved content
of the URL) during the SMTP transaction it would make sense for the same
to apply to the client.

> It occurs to me that there are weird potential edge cases where the response
> to a RCPT TO might be different when the message is resumed as when it was
> originally sent (assuming the server didn't receive the entire message and
> thus hasn't delivered it).

I decided that it would be too complicated to allow servers to change
their minds about their replies, especially because clients do not have to
re-send the corresponding commands (and cannot when using 1845
compatibility mode). If servers could change their minds then a client
implementation could not be based on simply freezing and restarting the
SMTP state machine. What if a client gets a 550 response to a subset of
the recipients originally, bounces the message immediately, then gets a
250 response after resuming the rest of the transaction? If the server
insists on changing its mind then it can still 450 after the message data
and force the client to retry later. This can be useful for changes from
OK to bad if you're really worried about unnecessary bounces from the
server.

I'm assuming that you will want the RESUME turn-around to be brief, so
you will be unlucky for significant changes in server state to happen in
that time.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
TYNE DOGGER: SOUTH VEERING SOUTHWEST 5 TO 7, INCREASING GALE 8 OR SEVERE GALE
9 FOR A TIME. MODERATE OR ROUGH, OCCASIONALLY VERY ROUGH IN DOGGER. RAIN OR
SHOWERS. MODERATE OR GOOD.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 17 Nov 2006 12:58:44 -0800

>  of the URL) during the SMTP transaction it would make sense for the same

But the profile document allows clients to be a little sloppy in 
their size calculation, saying that they must send a size at least as 
large as the actual size after expansion.  So a client can do a rough 
calculation, add a bit for slop, and send that.  In such a case it 
would be difficult for the client to know how to resume.

Likely the only time a client that used BURL would need to resume a 
message is if it lost the connection before seeing the result, and so 
wouldn't be sending any data anyway.  But I'm concerned that we have 
an ugly and potentially difficult or confusing area in the 
specification.

>  250 response after resuming the rest of the transaction?

It may be more likely for message submission clients to abort the 
transaction if any recipient fails (permanent or temporary), while a 
message relay client is likely to continue with the other recipients, 
and retry any that were temp-failed.

>   If the server

True, and perhaps we should mention this for cases where the server 
discovers during resumption (to its presumed shock and horror) that 
it is unable to deliver to previously OK recipients.

>  you will be unlucky for significant changes in server state to happen in

Also true.  And I'm Ok with this being a case of being very unlucky so too bad.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
One's need for loneliness is not satisfied if one sits at a table alone.
There must be empty chairs as well.                        --Karl Kraus

---

From: Dave Cridland <[email-masked]>
Date: Fri, 17 Nov 2006 21:52:59 +0000

On Fri Nov 17 20:58:44 2006, Randall Gellens wrote:
> their size calculation, saying that they must send a size at least 
> as large as the actual size after expansion.  So a client can do a 
> rough calculation, add a bit for slop, and send that.  In such a 
> case it would be difficult for the client to know how to resume.
> 
> 
Clients can usually not avoid having the size of a BURL chunk. In 
particular, for sending a leaf body part, they'll have the 
information in the BODYSTRUCTURE, which they have to have to know 
there *is* a leaf node to send.

For complete messages, they'll have to have requested RFC822.SIZE, 
unless they happened to APPEND the message themselves *and* did not 
use IMAP BINARY to do so.

I can't recall if they'll be able to find the size for non-leaf body 
parts, however.

> message is if it lost the connection before seeing the result, and 
> so wouldn't be sending any data anyway.  But I'm concerned that we 
> have an ugly and potentially difficult or confusing area in the 
> specification.
> 
> 
Another hiccup - clients will likely often have to resend BURL chunks 
with a different URL than they originally used, if they're using 
EXPIRES (which they should be) and they lost the connection for more 
than a few minutes.

But... Clients will, I suspect, resend a message using detirministic 
patterns of BDAT/BURL/DATA. Given a message they're trying to send, 
they will use the same pattern of commands to send it. The exception 
being if the capabilities are changed such that the pattern is 
changed, but I suspect that's pretty rare, especially with a resumed 
transaction.

If the offsets for RESUME where specified as command number and 
(optional?) offset, then we might side-step the issue quite neatly.

So RESUME would say something like "I remember up to the fourth 
command's fiftieth octet". Then the client looks at what it was 
trying to send, and it can presumably detirmine which the fourth 
command would have been, and go from there. If the fourth command was 
a BURL, then it would be able to assume that the fiftieth octet was 
at the end, and start with its original fifth command.

This doesn't prevent a client from resuming using a different 
pattern, of course, although I can't see that's likely. It must, 
however, be able to detirmine what the pattern it would have 
originally used is.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Sat, 18 Nov 2006 02:57:06 +0000

> On Fri Nov 17 20:58:44 2006, Randall Gellens wrote:

This SIZE discussion is exactly what the draft needs...

> > But the profile document allows clients to be a little sloppy in their size
> > calculation, saying that they must send a size at least as large as the
> > actual size after expansion.

Right, which is why I'm open to alternatives.

> Clients can usually not avoid having the size of a BURL chunk.

And I hoped this would be the case (not being familiar with the details of
BURL).

> Another hiccup - clients will likely often have to resend BURL chunks with a
> different URL than they originally used, if they're using EXPIRES (which they
> should be) and they lost the connection for more than a few minutes.

That implies that they might have to redo the GENURLAUTH depending on the
server's response to RESUME, which is a bit fiddly. I don't think it
demands any changes to the protocol, though. I'm not sure if it's worth
explaining this - I've tried to minimise the amount of rationale and other
chit-chat to keep the spec short.

> If the offsets for RESUME where specified as command number and (optional?)
> offset, then we might side-step the issue quite neatly.

Hmm, yes that is reasonably neat. It does mean that chunking clients have
to keep more information about how they have chunked the message data
(though perhaps it'll be the same for all messages from a given client)
and it isn't really 1845bis.

PS I have made some more updates to
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-lemonade-rfc1845bis.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HEBRIDES BAILEY: NORTHWEST BECOMING CYCLONIC THEN SOUTHWEST 6 TO GALE 8,
OCCASIONALLY SEVERE GALE 9 IN BAILEY. ROUGH OR VERY ROUGH. WINTRY SHOWERS.
MAINLY GOOD.