Subject: [imapext] Mail submission over IMAP

From: Jan Kundrát <[email-masked]>
Date: Sat, 21 Jul 2012 19:15:00 +0200

Hi,
there's been quite a few requests for adding e-mail submission to IMAP. 
Arnt's response last time was "Write draft-gondwana-imap-send, make it 
cover 99%, submit it for publication without support for all the corner 
cases", Mark's reaction to that was "That way, we can point to the 
document whenever anyone in the future brings up this bad idea...along 
with the fact that nobody implements it and why.". I've tried to follow 
(and re-read) the resulting discussion many times, looked at various 
concerns raised in the previous rounds, dug a bit into the history of 
various proposals and tried to come up with my version of the proposal. 
It is available for reading at [1], source at [2] (I don't know why that 
tool doesn't render in the "tools" HTML format of RFCs, sorry).

A few comments for that:

- Using Postfix, it can be trivially implemented using its `sendmail` 
interface
- It suppoorts Bcc just fine
- DSNs are specified
- When new ESMTP extensions appear, they can be integrated using 
traditional IMAP capabilities
- It doesn't require clients to speak (E)SMTP at all

I've implemented this version on a client side in Trojita [3] (but 
you'll need the git version for this feature). There are no server 
implementations that I'm aware of.

I have one open question -- right now, it can send a message only if it 
is in the currently opened mailbox. It's done this way to fit into the 
general IMAP picture (I suspect I'll receive quite some feedback for 
this comment, anyway). However, given how the APPEND command operates, 
at least my client would perform much more smoothly if the syntax was 
not "UID SUBMIT <uid>", but instead "UID SUBMIT <mailbox> <uidvalidity> 
<uid>" -- that way, I would not have to explicitly SELECT another 
mailbox just to send mail. If there's a consensus on this, I can easily 
change the definition.

I'd like to receive your opinion on this draft. I'm especially looking 
for server vendors who might want to implement it. I'm offering a 
special bounty of one $beverage to anyone who can first come up with a 
patch acceptable for upstream inclusion into any open source IMAP server 
project. Alternatively, for those proprietary serve vendors among us, a 
publicly available demo making this feature available qualifies as well.

However, even if your opinion is "your idea will never fly", I'd still 
like to hear it -- preferably with a reason about why it cannot work.

I'm afraid I'm not familiar with IETF's process when it comes to draft 
submission (I got confused by the bit saying that drafts aimed for the 
standards track cannot be individually submitted) -- that's why it's an 
unofficial page on my private web instead of a proper draft, although 
it's written in the officially recommended format for RFCs. I'd love to 
hear advice about proper process here as well.

With kind regards
Jan

[1] http://trojita.flaska.net/draft-imap-sendmail-00.html
[2] 
https://gitorious.org/trojita/trojita/blobs/drafts/draft-imap-sendmail-00/docs/proposed-extensions/draft-imap-sendmail.xml
[3] http://trojita.flaska.net/

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: "Adrien de Croy" <[email-masked]>
Date: Sat, 21 Jul 2012 23:41:46 +0000

Hi Jan

thanks for putting this together, I for one am very keen to see message 
submission in IMAP for the reasons you state (user email client 
configuration simplification, and support reduction), especially issues 
relating to inability to connect on normal submission ports.  I've 
fielded so many countless support queries of the kind "I can read mail 
but not send it" that I firmly believe IMAP is long overdue for this 
feature, and submission via IMAP is a zillion times easier for everyone 
(clients and servers) to implement than BURL over SMTP.

I have a few comments though.

1. name.  SENDMAIL is already a name of a product. I don't know if the 
name is trademarked.  There could be all sorts of issues calling it 
SENDMAIL, I'd recommend calling it SUBMIT instead.

2. FROM.  I'm not certain if this should be settable at submission time 
or not.  It's the sort of thing that may be best handled with 
server-side association of the account with a return path.

3. SENDER.  I'm not sure what SMTP envelope part this relates to, is it 
superfluous?  If it's already in the message headers, why do we need to 
specify it again here?  It's not needed for SMTP delivery (only need 
return path and forward path(s))

4. RECIPIENT processing.  Can't we just specify a list?

e.g. 

x UID SUBMIT 123 (FROM "[email-masked]" RECIPIENT 
("[email-masked]", "[email-masked]") DSN (delay failure))

5. $submitted and $submitpending.  Actually I would remove all of this.

One should not be prohibited from submitting a message more than once?  

It's not the end of the world if a message is sent more than once? 
Better than not at all?
Or is this to prevent loss of information about to whom a message has 
been sent?  E.g. one would need to copy first then submit the copy.  In 
which case COPY should NOT copy the $submitted flag.

6. BCC processing.  Presumably on submission, the BCC header should be 
stripped?

7. Atomicity.  I understand this is just when submitted, if there is 
any recipient checking.  Upon actual delivery, some recipients may work 
or not, but this would be handled by other mechanisms?

I can implement this if you want a server to test against, we don't do 
BURL, so don't have CATENATE though.

Regards

Adrien

------ Original Message ------
From: "Jan Kundrát" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 22/07/2012 5:15:00 a.m.
Subject: [imapext] Mail submission over IMAP
>Hi, 
>there's been quite a few requests for adding e-mail submission to 
>IMAP. Arnt's response last time was "Write draft-gondwana-imap-send, 
>make it cover 99%, submit it for publication without support for all 
>the corner cases", Mark's reaction to that was "That way, we can point 
>to the document whenever anyone in the future brings up this bad 
>idea...along with the fact that nobody implements it and why.". I've 
>tried to follow (and re-read) the resulting discussion many times, 
>looked at various concerns raised in the previous rounds, dug a bit 
>into the history of various proposals and tried to come up with my 
>version of the proposal. It is available for reading at [1], source at 
>[2] (I don't know why that tool doesn't render in the "tools" HTML 
>format of RFCs, sorry). 
>
>A few comments for that: 
>
>I have one open question -- right now, it can send a message only if 
>it is in the currently opened mailbox. It's done this way to fit into 
>the general IMAP picture (I suspect I'll receive quite some feedback 
>for this comment, anyway). However, given how the APPEND command 
>operates, at least my client would perform much more smoothly if the 
>syntax was not "UID SUBMIT <uid>", but instead "UID SUBMIT <mailbox> 
><uidvalidity> <uid>" -- that way, I would not have to explicitly 
>SELECT another mailbox just to send mail. If there's a consensus on 
>this, I can easily change the definition. 
>
>patch acceptable for upstream inclusion into any open source IMAP 
>server project. Alternatively, for those proprietary serve vendors 
>among us, a publicly available demo making this feature available 
>qualifies as well. 
>
>standards track cannot be individually submitted) -- that's why it's 
>an unofficial page on my private web instead of a proper draft, 
>although it's written in the officially recommended format for RFCs. 
>I'd love to hear advice about proper process here as well. 
>
>With kind regards 
>Jan 
>
>-- Trojita, a fast e-mail client -- http://trojita.flaska.net/ 
>
>_______________________________________________ 
>imapext mailing list 
>[email-masked] 
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Jan Kundrát <[email-masked]>
Date: Sun, 22 Jul 2012 15:48:37 +0200

Hi Adrien, thanks for your review. Answers below.

> SENDMAIL, I'd recommend calling it SUBMIT instead.

I don't have much of a preference -- given that the most obvious 
server-side implementation would call `sendmail` directly, the name "UID 
SENDMAIL" felt good. I can change this if more people suggest so.

> server-side association of the account with a return path.

The ESMTP allows for this, and I believe that there are legitimate 
reasons for envelope-From being different than header-From or even 
account-id (think mailing list managers). If this feature was out, I'd 
immediately get "you solution doesn't allow $foo" complaints, and I'd 
agree with them.

Anyway, if the server doesn't allow overriding the FROM parameter, it 
shall return a POLICYDENIED response code -- so the biggest problem is 
that such a server MUST check that the FROM header (if present) matches 
the account name. That sounds acceptable to me -- what about you?

Furthermore, there's currently no way for a client to see whether the 
denial was through "you've specified an envelope-FROM different from the 
header-From" or due to "your message looks like SPAM". Do you have a use 
case for making this a machine-readable (i.e. through the response code) 
identifier? (Both situations sounds very similar to me, though -- a 
spoofed sender can be a contributing factor for marking as spam, etc.)

> return path and forward path(s))

Oops, good catch -- the SENDER should not be there at all. I'll fix this 
in my next update.

> ("[email-masked]", "[email-masked]") DSN (delay failure))

This is to match how the ESMTP transaction looks like. Furthermore, if 
the grammar used a list syntax, some client would inevitably try 
"RECIPIENT one-address-here" without the parentheses, and if we wanted 
to plan for that, the grammar would have to allow for both types of 
syntax, which feels ugly. That's why I prefer the duplicate RECIPIENT bits.

> which case COPY should NOT copy the $submitted flag.

This is to conform with Lemonade's recommendations [1] and in order to 
guarantee that mail-submission is reasonably "atomic" from the client 
point of view -- sure, if there's a server crash, a message can be 
improperly flagged, but that's inherent problem in any multi-service 
environment.

Would you still strip out this section after reading what Lemonade has 
to say? It feels to me that these headers are meant to be manipulated by 
the party doing the actual submission, which is the IMAP server in the 
context of this extension.

> 6. BCC processing.  Presumably on submission, the BCC header should be

This is actually explicitly stated in the draft -- there's no 
requirement on payload manipulation in this case. I was afraid that 
mandating explicit removal of Bcc would prove to be hard to implement, 
but it looks like it's already mandated by RFC5321, appendix B [2].

I'll change the draft to include any requirements from RFC 5321/5322 and 
that the Bcc headers MUST be "handled".

> 7. Atomicity.  I understand this is just when submitted, if there is any
> recipient checking.  Upon actual delivery, some recipients may work or
> not, but this would be handled by other mechanisms?

Yes. The IMAP server shall either take the message with all recipients 
obtained from submission options or from the headers and try to deliver 
to all of them, or to none.

> BURL, so don't have CATENATE though.

CATENATE is very useful on its own, even without BURL. In fact, it is an 
excellent feature to have along this UID SUBMIT draft.

I'll be happy to test against your implementation even in absence of 
CATENATE, of course.

Cheers,
Jan

[1] http://tools.ietf.org/html/rfc5550#section-5.10
[2] http://tools.ietf.org/html/rfc5321#appendix-B
-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 22 Jul 2012 16:54:05 +0200

On 07/22/2012 03:48 PM, Jan Kundrát wrote:
> account-id (think mailing list managers).

Suggest some that are legitimate in the context of an IMAP client?

IMO, envelope from more or less has to match the header addresses. The 
exceptions are firmly in server or service land, not in IMAP client land.

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Sun, 22 Jul 2012 22:41:26 +0000

Hi Jan

------ Original Message ------
>server-side implementation would call `sendmail` directly, the name 
>"UID SENDMAIL" felt good. I can change this if more people suggest so. 
>
understand re mailing list manager.

I was thinking maybe the server needs to advertise whether FROM (smtp 
return-path) is settable, otherwise it is set by the mail server admin.

>that such a server MUST check that the FROM header (if present) 
>matches the account name. That sounds acceptable to me -- what about 
>you? 
could be difficult for the client to figure out what went wrong, unless 
you put special return codes in so the client can know why the 
submission was bounced.

>denial was through "you've specified an envelope-FROM different from 
>the header-From" or due to "your message looks like SPAM". Do you have 
>a use case for making this a machine-readable (i.e. through the 
>response code) identifier? (Both situations sounds very similar to me, 
>though -- a spoofed sender can be a contributing factor for marking as 
>spam, etc.) 
the alternative to machine-readable responses is either generic (e.g. 
pretty much useless) or localized which has other issues.  So I always 
prefer machine-readable.  There could be a number of reasons why 
submission is rejected.  However, things like spam (AV) filtering could 
be done at APPEND or CATENATE time?

>Oops, good catch -- the SENDER should not be there at all. I'll fix 
>this in my next update. 
I'm thinking if we have to parse and process headers anyway to deal 
with BCC, then Sender could be useful still.

>From my understanding, the main issue is that some record needs to be 
kept somewhere in the mail client about what email was sent to whom.  
This means if a message was BCCed to someone, then in the sent items 
folder, the BCC header should remain visible.  However, it needs to be 
removed for actual delivery.

     Therefore the server needs to parse and remove the BCC header. 

Other things should be done by the client, such as copying or moving to 
say sent items folder, 

>syntax, which feels ugly. That's why I prefer the duplicate RECIPIENT 

This is a new proposal, so I think we can assume people would implement 
it properly.

In fact having the tokens for RECIPIENT, FROM etc is just bloat.  Also, 
lists (even empty ones) are much easier to parse than token value 
pairs, and easier to deal with corner cases (e.g. missing "RECIPIENT" 
or missing recipient value)

If you look at the BODYSTRUCTURE response, it's perfectly fine to 
specify like

submit = "SUBMIT" SP submit-params
submit-params = "(" uniqueid SP forward-path [SP return-path] [SP 
dsn-spec] ")"
return-path = nstring
forward-path = "HEADERS" / "(" string [*(SP string)] ")"
dsn-spec = "(" dsn-token [*(SP dsn-token)] ")"
dsn-token = "DELAY" / "FAILURE" / "SUCCESS"

this solves several issues as well, e.g. the BNF doesn't allow multiple 
dsn-spec, or sub-option-from options (return-path).  

Since dsn-spec is optional, it makes no sense to allow a NIL value.  
Also, since it's a list, it is distinguishable from return-path.

If return-path is omitted, the server would use the return-path 
associated with the mail account.

in forward-path, if value is "HEADERS", then the forward path is taken 
from the email headers To and BCC, otherwise it's a non-empty list of 
non nil strings.

>to say? 
probably not.

Otherwise you'd need some meta-storage to keep information about all 
the submissions that have been performed on a message, and that doesn't 
really fit into most email client modes of operation - they just add a 
copy to sent items.

>It feels to me that these headers are meant to be manipulated by the 
>party doing the actual submission, which is the IMAP server in the 

It's not hard compared to implementing SEARCH.  So I wouldn't be 
concerned about that.

>
>
>I'll change the draft to include any requirements from RFC 5321/5322 
>and that the Bcc headers MUST be "handled". 
>
>>any 
>obtained from submission options or from the headers and try to 
>deliver to all of them, or to none. 
>
>CATENATE is very useful on its own, even without BURL. In fact, it is 
>an excellent feature to have along this UID SUBMIT draft. 
How would it be used without some way to deliver the result somewhere?  
Or maybe only useful in that case with shared folders or something?

Anyway, CATENATE will be on our roadmap soon, esp if SUBMIT / SENDMAIL 
gets any traction.

Regards

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 23 Jul 2012 07:35:20 +0000

------ Original Message ------
>"UID SENDMAIL" felt good. I can change this if more people suggest so. 

from http://www.sendmail.org/~ca/email/sm-X/LICENSE clause 4
The name "sendmail" is a registered trademark and service mark of
Sendmail, Inc.

I wouldn't use the name SENDMAIL, can possibly cause all sorts of 
problems for mail server vendors who may wish to advertise support for 
this extension etc etc etc

Adrien

---

From: Jan Kundrát <[email-masked]>
Date: Mon, 23 Jul 2012 10:25:26 +0200

> exceptions are firmly in server or service land, not in IMAP client land.

A group of friends is going on a vacation and for some reason they 
haven't set up a mailing list for coordination. Alice sent a mail to all 
of them, but Bob's mailbox was broken at the time of submission and his 
mail server refused to accept this message (or Alice just used Bob's old 
e-mail address which he no longer reads -- yes, I've seen such a situation).

Now Alice and the rest of the team has left for a short pre-vacation 
vacation and I'm the only one hard working guy left to "forward" the 
mail to Bob. I'd still like to retain the original Message-Id etc for 
threading, but I also want to be able to use DSN to see whether the 
message made it this time.

Hence I take the original message from Alice as delivered to my mailbox 
and issue `UID SUBMIT x (FROM "[email-masked]" RECIPIENT 
"[email-masked]")`. This requires support for return-path not matching 
the original From header.

This is admittedly a rather convoluted example, but I think that it is 
actually a valid use case. Or maybe it should be done by just taking the 
message as-is, prepending the Sender header and letting the MTA perform 
its extraction -- is that the case?

If the MTA always used a return-path matching my account credentials, 
that would work as well. That would have a disadvantage of requiring 
knowledge of the authenticated user at the MTA layer, though; I can 
imagine deployments where this would imply extra work for the sysadmin.

Opinions?

Cheers,
Jan

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 23 Jul 2012 10:28:11 +0200

> Opinions?

Yes, a strong one: Worry more about the common case and less about edge 
cases.

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Mon, 23 Jul 2012 11:11:43 +0200

Hi Adrien,

> return-path) is settable, otherwise it is set by the mail server admin.
[...]
> you put special return codes in so the client can know why the

s/bounced/rejected/, I guess (failing UID SUBMIT MUST NOT cause an 
SMTP-level bounce) -- just to prevent a possible misunderstanding, I'm 
sure you haven't meant an actual bounce.

The question is whether it's worth the cost of yet another response code 
to be able to tell the client the difference between "your message text 
looks like spam" and "you attempted to fake the envelope FROM". I tend 
to think this is not worth the cost (otherwise we'd have to add a 
special response code for "virus detected", "you used too much 
$random-drug advertisements", "looks like spam", "you've sent way too 
many messages in the last hour", "looks like you're faking the Received 
headers" etc) -- but I'm open to discussion and want to reach a 
consensus here. I just don't want to come up with something overly 
complex. If we add an explicit response code, clients will assume that 
servers will use it, and that raises the bar for servers to implement.

What shall the client do when it sees the proposed INVALIDRETURNPATH? Is 
there any difference in the desired behavior compared to handling of 
POLICYDENIED? Honestly, I don't know and am interested in other people's 
opinion.

> However, things like spam (AV) filtering could be done at APPEND or CATENATE time?

I strongly object to that. If this filtering was done "globally", how 
could I possibly import data from my previous account to my new 
"spam-collection-for-auto-learning" folder?

>>> 3. SENDER. I'm not sure what SMTP envelope part this relates to, is
>>> it superfluous? If it's already in the message headers, why do we
>>> need to specify it again here? It's not needed for SMTP delivery
>>> (only need return path and forward path(s))
>>
> I'm thinking if we have to parse and process headers anyway to deal with
> BCC, then Sender could be useful still.

You've suggested that in ESMTP, there's no such thing like a distinct 
"FROM" and "SENDER". I've amended my (currently unpublished) copy of the 
draft to only use a single "FROM" field for this purpose.

Arguably, if the client didn't use this submission option field, a 
compliant server implementation MUST set it to a value obtained from the 
message contents. A good order of where to get the value from might be 
Resent-From, Sender, From (but I won't try to pretend that I've read the 
relevant RFC about these headers yet, so please take this order with a 
big grain of salt).

> Other things should be done by the client, such as copying or moving to

Agreed. Client is responsible for building the actual message, including 
all of its headers. These headers should contain information about where 
it was sent to. This message body MUST be stored on the IMAP server 
completely unchanged (if only to guarantee the usual IMAP requirements 
about immutability).

OTOH, I agree that MTAs MUST "process Bcc headers correctly", i.e. the 
identity of any recipient listed in Bcc MUST NOT be revealed to any 
other recipient. This means that the party handling the UID SUBMIT 
command (be it the IMAP server, the `sendmail` binary or the first ESMTP 
MTA to which the IMAP server submits the message) MUST rewrite message 
headers.

I've verified that Postfix (RHEL's postfix-2.6.6-2.2.el6_1) behaves that 
way. I don't have any experience with any other MTA.

> lists (even empty ones) are much easier to parse than token value pairs,
> and easier to deal with corner cases (e.g. missing "RECIPIENT" or
> missing recipient value)
>

Please tell that to Google who still produces invalid BODYSTRUCTURE 
responses on e-mails with attached message/rfc822 parts (last checked a 
few months ago, anyway). As a client developer, I can assure you that it 
is NOT FUN to try to guess what five fields out of a sequence of twenty 
mandatory fields are missing today. I won't willingly inflict the same 
on any other programmer unless I'm in a really, really nasty mood.

> dsn-token = "DELAY" / "FAILURE" / "SUCCESS"

This command can indeed be specified in many possible ways. I don't see 
much value in putting the UID in the same parenthesized list as the rest 
of the data, for example, but that's just a very, very minor point. I 
will, however, always prefer a self-describing structure (i.e. with the 
superfluous "FROM", "DSN" and "RECIPIENT" headers) over something which 
is more compact for the reasons stated above.

It will take a well-explained technical reason to change my opinion on 
this one.

> dsn-spec, or sub-option-from options (return-path).

That would indeed be a nice benefit, but not something I consider a 
requirement.

> Also, since it's a list, it is distinguishable from return-path.

You have to be able to specify DSN of:

1) default, implementation defined behavior (i.e. "DSN not specified at 
all" in my proposal)
2) no DSN at all, even for failures (i.e. DSN NIL in my proposal)
3) a non-empty subset of the three available options with possibility to 
include all of them

Your grammar doesn't allow for option two. Maybe my draft should make 
this distinction more obvious, given you interpreted it in a wrong way.

> Or maybe only useful in that case with shared folders or something?

There are three steps in today's mode of e-mail sending where 
significant amounts of data are transferred:

1) create the message somehow (you might have to download attachments of 
existing messages if you're forwarding them)
2) upload the message to the IMAP server
3) send the message through SMTP

CATENATE can significantly reduce the amount of data in steps 1) and 2) 
-- if you're e.g. forwarding a big message, you can compose its copy for 
the Sent folder with almost no data transfers at all.  If the original 
message contained a small text/plain part with a huge attachment, a good 
client will not have to download said attachment at all.

Another case for CATENATE is being able to strip-out attachments from 
existing mail. If the user clicks on "delete this attachment", client 
can issue APPEND ... CATENATE that refers to all other message parts, 
again reducing the amount of data to transfer.

Cheers,
Jan

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 23 Jul 2012 10:51:54 +0000

------ Original Message ------
>haven't set up a mailing list for coordination. Alice sent a mail to 
>all of them, but Bob's mailbox was broken at the time of submission 
>and his mail server refused to accept this message (or Alice just used 
>Bob's old e-mail address which he no longer reads -- yes, I've seen 
>such a situation). 
>
>Hence I take the original message from Alice as delivered to my 
>mailbox and issue `UID SUBMIT x (FROM "[email-masked]" RECIPIENT 
>"[email-masked]")`. This requires support for return-path not 
>matching the original From header. 

I think what you would do is make a copy, edit the header so that From 
is from you, and edit the subject so it's obviously a forward from you 
of the original message.

>actually a valid use case. Or maybe it should be done by just taking 
>the message as-is, prepending the Sender header and letting the MTA 
>perform its extraction -- is that the case? 
>

I was planning on getting the IMAP server to edit the headers etc 
before submitting to the MTA.  It should have all the information it 
needs, since the submitter is logged into it.

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 23 Jul 2012 11:12:19 +0000

------ Original Message ------
>sure you haven't meant an actual bounce. 
yes sorry, I meant submission rejected by the IMAP server.  IMAP server 
can only do a certain amount of checking when validating submission, 
and the MTA will do some more.  The IMAP server can report errors to 
the SUBMIT command, whereas the MTA will deal with issues in another 
way (outside the scope of this)

>The question is whether it's worth the cost of yet another response 
>code to be able to tell the client the difference between "your 
>message text looks like spam" and "you attempted to fake the envelope 
>FROM". I tend to think this is not worth the cost (otherwise we'd have 
>to add a special response code for "virus detected", "you used too 
>much $random-drug advertisements", "looks like spam", "you've sent way 
>too many messages in the last hour", "looks like you're faking the 
>Received headers" etc) -- 
I don't think I'd bother with that many.  I'd use invariant tokens for 
the types of errors, and some may include some reason text.  What are 
the main expected failure cases?

* syntax error
* User may not submit
* return path rejected
* one or more forward paths rejected
* system error (e.g. out of disk).
* etc ?

>but I'm open to discussion and want to reach a consensus here. I just 
>don't want to come up with something overly complex. If we add an 
>explicit response code, clients will assume that servers will use it, 
>and that raises the bar for servers to implement. 
these are not hard things to do surely?

>
>What shall the client do when it sees the proposed INVALIDRETURNPATH? 
>Is there any difference in the desired behavior compared to handling 
>of POLICYDENIED? Honestly, I don't know and am interested in other 
>people's opinion. 
I foresee a setting in email clients which are like 

[x] submit messages using IMAP
[x] use return-path associated with your IMAP account

or similar type of concept (wording would likely be quite different).

So the response code has a big impact on what the user does next.  If 
it's policy denied, they need to see their sys admin, or turn off 
submission through IMAP.

If it's invalid return path, they need to alter their return-path 
settings.

Tech support will want to be able to distinguish for debugging / 
rectification of the issue.

>"spam-collection-for-auto-learning" folder? 

It's already done.  If you don't like it I guess you'd have to get it 
turned off on your mailbox.

It's not a particularly wide-spread requirement to store viruses in 
your mailbox, and I don't think we should deny others of AV protection 
so you can do so.

>"FROM" and "SENDER". I've amended my (currently unpublished) copy of 
>the draft to only use a single "FROM" field for this purpose. 
>
>compliant server implementation MUST set it to a value obtained from 
>the message contents. 
are you talking about the compliant server scraping return-path out of 
headers, or providing a Sender field into the headers?

>A good order of where to get the value from might be Resent-From, 
>Sender, From (but I won't try to pretend that I've read the relevant 
>RFC about these headers yet, so please take this order with a big 
>grain of salt). 
I would make it optional on the server maybe, where it can just 
optionally use some pre-determined value associated with the IMAP 
account.

e.g. user is logged into IMAP as bob, submits a message so return path 
is [email-masked]

>Agreed. Client is responsible for building the actual message, 
>including all of its headers. These headers should contain information 
>about where it was sent to. This message body MUST be stored on the 
>IMAP server completely unchanged (if only to guarantee the usual IMAP 
>requirements about immutability). 
>
>command (be it the IMAP server, the `sendmail` binary or the first 
>ESMTP MTA to which the IMAP server submits the message) MUST rewrite 
>message headers. 

that can be up to the implementation to decide how.  Our MTA doesn't do 
BCC processing, so the IMAP server would do it on behalf of the client. 
  It may mean multiple copies are made for delivery.

>
>
>I've verified that Postfix (RHEL's postfix-2.6.6-2.2.el6_1) behaves 
>that way. I don't have any experience with any other MTA. 
>
>>pairs, 
>responses on e-mails with attached message/rfc822 parts (last checked 
>a few months ago, anyway). As a client developer, I can assure you 
>that it is NOT FUN to try to guess what five fields out of a sequence 
>of twenty mandatory fields are missing today. I won't willingly 
>inflict the same on any other programmer unless I'm in a really, 
>really nasty mood. 

Even if I ask nicely?

It's servers that need to parse this.  I'm just saying it's easier to 
deal with a list of recipients than have to match token value etc.

Your BNF will be a lot more code for me, and more corner cases.  It 
shouldn't be any more or less difficult for a client to generate the 
request.
>This command can indeed be specified in many possible ways. I don't 
>see much value in putting the UID in the same parenthesized list as 
>the rest of the data
sure, it doesn't matter if it's in the list or not.

> for example, but that's just a very, very minor point. I will, 
>however, always prefer a self-describing structure (i.e. with the 
>superfluous "FROM", "DSN" and "RECIPIENT" headers) over something 
>which is more compact for the reasons stated above. 
problem is you use tags like that, it means order becomes variable.  
It's much easier to deal with fixed order of parameters.

Once you fix order of parameters, and use a list for recipients to 
remove corner cases about missing or duplicate tags etc etc, you end up 
with a structure something like I proposed.

Google may have trouble generating BODYSTRUCTURE (we did too for a long 
time), but this structure is much more simple, and furthermore, servers 
parse it, rather than generate it.

>1) default, implementation defined behavior (i.e. "DSN not specified 
>at all" in my proposal) 
OK I missed that case.

So Nil has some meaning by its presence that is different to its 
absence.  Absense = default behaviour, Nil = no DSN.  That's fine.

Could be more obvious maybe if you added some more tokens, such as 

dsn-spec = "NONE" / "DEFAULT" / "(" dsn-token [*(SP dsn-token)] ")" 
dsn-token = "DELAY" / "FAILURE" / "SUCCESS" 

>
>3) a non-empty subset of the three available options with possibility 
>to include all of them 
>
>1) create the message somehow (you might have to download attachments 
>of existing messages if you're forwarding them) 
>CATENATE can significantly reduce the amount of data in steps 1) and 
>2) -- if you're e.g. forwarding a big message, you can compose its 
>copy for the Sent folder with almost no data transfers at all. If the 
>original message contained a small text/plain part with a huge 
>attachment, a good client will not have to download said attachment at 
>all. 

OK, thanks for that.
Regards

Adrien

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 23 Jul 2012 18:14:53 +0200

On 07/23/2012 04:54 PM, Ned Freed wrote:
> Off the top of my head: MDNs (required to have a null MAIL FROM),

Good point. Sufficient, perhaps.

> some
> send-on-behalf-of scenarios, client-side mailing lists, some DSN
> correlation mechanisms, and some multi-author scenarios.

I've seen software to do all of that, but not IMAP clients.

"An IMAP extension to allow the current user to send mail as himself" is 
one thing. "An IMAP extension to do everything a SUBMIT server can" is 
quite another, and IMO problematic. Jan, what do you want your document 
to be?

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 24 Jul 2012 00:26:36 +0000

Hi all

was just reading through CATENATE RFC, and it mentioned hard limit of 
4GB for any message in any mailbox.

The time where this will be a problem is fast approaching.  It already 
is for some people apparently.  What is actually stopping current BNF 
from handling larger sizes?  Since we always transmit and receive 
(parse) textual representations of numbers, then moving to 64 bit is 
simply a matter of using a different conversion function?

Is this something we should write an extension for?

>From 3501 the issues are with 

number = 1*DIGIT
                 ; Unsigned 32 bit number

and

nz-number = 1*DIGIT
                 ; Non-zero unsigned 32-bit integer

which are used for msgnos, body parts / sections, ranges, searches 
(e.g. SMALLER), counts of things.

We could either upgrade number and nz-number to 64 bit, or introduce a 
big-number production which would be a lot more work but possibly lower 
impact on implementors.

In fact the 32 bit limit is only a comment in the BNF, I can't find it 
anywhere else.

Regards

Adrien

---

From: Randall Gellens <[email-masked]>
Date: Mon, 23 Jul 2012 19:10:04 -0700

At 4:54 PM +0200 7/22/12, Arnt Gulbrandsen wrote:

>  IMO, envelope from more or less has to match 
> the header addresses. The exceptions are firmly 
> in server or service land, not in IMAP client 
> land.

I rely heavily on Eudora's ability to allow 
free-form editing of the header 'From' field, 
because I use one-off addresses in many contexts, 
and don't want to bother creating new 
accounts/personalities for each.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
No man is a hero to his wife's psychiatrist.      --Eric Berne

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 24 Jul 2012 23:06:36 +0200

> quite another, and IMO problematic. Jan, what do you want your document

What I'd like to have is an ability to write a MUA submitting over IMAP 
which can do "everything that a MUA with support for ESMTP can do and 
would do under reasonable situations" -- so the goal is to make it 
possible to not use ESMTP at all for end-user clients.

Mail list management is not a reasonable situation. MDNs, on the other 
hand, looks like a reasonable use case. That use case with Alice, Bob 
and me that I mentioned fall into the "reasonable" use cases category 
for me (albeit maybe close to the edge); other people might disagree 
with that. That is fine with me and I really want to reach a consensus 
representing what the IMAP server vendors are willing to implement, as 
long as it is "enough" for a "typical client" to replace its ESMTP 
submission.

Thanks for your pointers, I agree that it's hard to find a balance 
between a bloated proposal and something which is too bare to be 
actually usable in real world. I admit that I do usually fall to the 
temptation of adding features "just because they are easy".

With kind regards
Jan

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 24 Jul 2012 23:31:50 +0200

Hi Adrien,

> the main expected failure cases?

-> BAD

> * User may not submit

-> NO [POLICYDENIED]

> * one or more forward paths rejected

-> NO [POLICYDENIED] in my current draft. You're proposing to make this 
fine grained. See below.

> * system error (e.g. out of disk).

-> NO (without any particular response code)

> [x] submit messages using IMAP

Agreed so far.

> [x] use return-path associated with your IMAP account

This is something for which I don't see the need. Why should the user be 
able to toggle this settings at all? What benefits would bring that to her?

> It's not a particularly wide-spread requirement to store viruses in your
> mailbox, and I don't think we should deny others of AV protection so you
> can do so.

Well, what I was objecting to was that I perceived your idea as "spam 
filtering should be done during APPEND, not during SUBMIT". That's 
something I don't agree with; in my opinion, servers WILL want to use 
the content filtering during UID SUBMIT. I don't care whether they do so 
during APPEND as well and I recognize that there are valid reasons for that.

> headers, or providing a Sender field into the headers?

I actually don't understand what you're asking here (at least in the 
context of the quoted discussion).

I've removed one superfluous item from my idea of the ESMTP envelope's 
equivalent in UID SENDMAIL -- an item which doesn't actually have a 
counterpart in ESMTP.

The draft says nothing about adding data to the Sender header of the 
outgoing message. Content manipulation is explicitly allowed, so I guess 
servers are free to do so -- but I don't see an obvious reason why they 
should do that provided the message already contains a proper From.

If there's a good reason for that, please tell me, I'll be happy to 
mandate that in my draft.

> optionally use some pre-determined value associated with the IMAP account.
>

(Speaking about the ways how the IMAP server determines the return-path.)

Yes, the account details can be used to set the return-path if it is not 
given explicitly by the client. The account details and the knowledge 
obtained through header scraping should both be used as an input, I guess.

>   It may mean multiple copies are made for delivery.

Agreed, that would be a compliant behavior when it comes to the wording 
of the draft and my intention. As long as your IMAP server guarantees 
that it won't leak the Bcc data, there's no difference in who does the 
header filtering.

> deal with a list of recipients than have to match token value etc.

To be honest, I have seen crappy clients and crappy servers alike. Given 
that a huge service provider weren't able to write a compliant 
implementation of their server side (where they absolutely control 
deployment schedule, any updates etc), I don't believe at all that small 
client developers will be able to follow the specification to the letter.

Anyway, the biggest advantage of the grammar I proposed is its future 
extensibility. I like that point and am afraid that this is something 
that breaks in the fixed-order grammar as soon as there are two future 
extensions.

> problem is you use tags like that, it means order becomes variable. It's
> much easier to deal with fixed order of parameters.

I see that it is easier in case the other side of the protocol is 
compliant. What I've seen when testing my client, however, is that the 
other side is usually very creative (and I know that I'm also very often 
"creative" as well -- members of this list against whose software I've 
run Trojita could probably tell).

I could live with the fixed-order syntax, but I believe that it is 
technically inferior (not extensible, more fragile to parse). If most of 
the people on this list object, it will be the fixed-order one, but what 
I'd like to try before that is persuading them that the varying-order 
identifier-based one is better technically. So far, it's a draw, I'll 
love to hear from other server vendors here (Timo, Bron, Alexey, others?)

> dsn-token = "DELAY" / "FAILURE" / "SUCCESS"

I'll probably make this explicit in my next draft; I suspect that this 
difference was not terribly obvious.

Thanks for your feedback,
Jan
-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 25 Jul 2012 09:23:13 +0200

> possible to not use ESMTP at all for end-user clients.

That was discussed in the Lemonade context. Someone (perhaps Ned or Pete 
Resnick?) pointed out that in that case, IMAP either needs an SMTP 
extension passthrough (so an IMAP client can use SMTP extensions) or 
parallel extensions (so for each SMTP extension there is an IMAP extension).

I think you'll have more luck by enumerating a requirement list needed 
by "typical clients", and then saying that only that list is in-scope.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Mon, 23 Jul 2012 07:54:31 -0700 (PDT)

> exceptions are firmly in server or service land, not in IMAP client land.

Off the top of my head: MDNs (required to have a null MAIL FROM), some
send-on-behalf-of scenarios, client-side mailing lists, some DSN
correlation mechanisms, and some multi-author scenarios.

All of these are firmly in client space, not server or service land. The first
two are quite common.

Oh, and submit servers are often configured to perform complex authorization
checks on messages, including verification of allowed send-on-behalf-of
combinations. The result is that trying to indicate what's allowed and
what's not in a single bit is effectively impossible.

				Ned

P.S. And doing this stuff "right", for some value of "right", doesn't make
submission facilities in IMAP a good idea.