Subject: Re: [imapext] The IMAP APPENDLIMIT Extension

From: Michael M Slusarz <[email-masked]>
Date: Sun, 15 Feb 2015 23:07:42 -0700

Quoting Jayantheesh S B <[email-masked]>:

> Please find the next version of the draft updated with APPENDLIMIT=0  
> case addressed.

Was working on a test server implementation of this, and ran into  
several issues that need to be addressed/clarified.

1) The APPENDLIMIT value is defined in ABNF as a "number", which is  
limited to 32-bits.  Seems like this should be 64-bits instead to  
better reflect modern resource limitations (or at least an unsigned  
63-bit number, such as MODSEQs).

2) I don't believe the APPENDLIMIT response code should be returned  
for an EXAMINE operation.  The value is irrelevant since you can't  
append to a read-only mailbox.  It's possible that APPENDLIMIT may be  
somewhat expensive to generate - i.e. a quota value that needs to be  
determined/parsed/updated - so there's no need to report it when it is  
not directly usable.

3) I believe this was mentioned by at least one other person, but the  
current handling of the response code in the mailbox is not correct,  
at least according to the draft specification.

Take this example:

<QUOTA active; user currently has 100 bytes left in account>
a1 SELECT INBOX
<...>
* OK [APPENDLIMIT 100]
a1 OK [READ-WRITE] Done
a2 STORE 1 +flags \deleted
* 1 FETCH (FLAGS (\Deleted))
a2 OK
a3 EXPUNGE
* 1 EXPUNGE
* 0 EXISTS
a3 OK

Assuming that message 1 was 100 bytes, the quota for the user is now  
up to 200 bytes.  This should be the new APPENDLIMIT value, but  
according to the spec this does not need to be reported/updated.

This needs to be handled similarly to the way that message counts are  
updated when in the selected state.  Although, as mentioned above, the  
appendlimit value may not be a trivial item to calculate so this can  
start to become a fairly expensive feature add.

It makes me wonder whether adding this response code this is the  
correct approach.  APPENDs will normally happen in an unselected state  
(there are at least 2 server implementations that I am aware of that  
don't allow APPENDs to a selected mailbox), so this information is  
better obtained by using one of the other two methods anyway.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 16 Feb 2015 11:06:45 +0000

> 63-bit number, such as MODSEQs).
A very good point. +1.
> append to a read-only mailbox.
Actually you are wrong on this one, because the APPEND command takes a 
mailbox name as a parameter, so it can be used from any mailbox (or no 
mailbox). So the currently selected mailbox and its state is irrelevant.
> It's possible that APPENDLIMIT may be somewhat expensive to generate - 
> i.e. a quota value that needs to be determined/parsed/updated - so 
> there's no need to report it when it is not directly usable.
>
I think I am agreeing with Cyrus who said that APPENDLIMIT should not be 
affected by outstanding quota. It is a hint anyway. Not linking it to 
quotas would make implementations easier (and clients need to handle 
errors anyway).
> _______________________________________________
> imapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/imapext

---

From: Michael M Slusarz <[email-masked]>
Date: Mon, 16 Feb 2015 16:09:45 -0700

Quoting Alexey Melnikov <[email-masked]>:

> Actually you are wrong on this one, because the APPEND command takes  
> a mailbox name as a parameter, so it can be used from any mailbox  
> (or no mailbox). So the currently selected mailbox and its state is  
> irrelevant.

I disagree with you here.

Agreed that APPEND has a mailbox argument, so it can be called from a  
non-selected context.

But APPEND is also allowed to be called from within a mailbox.  And if  
that current selected context is read-only via EXAMINE, my  
understanding/belief is that an APPEND cannot be successful since that  
would be an action that causes a "change[] to the permanent state of  
the mailbox".

Regardless, not much reason to discuss this further in this context  
due to below comments.

>> 3) I believe this was mentioned by at least one other person, but  
>> the current handling of the response code in the mailbox is not  
>> correct, at least according to the draft specification.
>>
>> Take this example:
>>
>> Assuming that message 1 was 100 bytes, the quota for the user is  
>> now up to 200 bytes.  This should be the new APPENDLIMIT value, but  
>> This needs to be handled similarly to the way that message counts  
>> are updated when in the selected state.  Although, as mentioned  
>> above, the appendlimit value may not be a trivial item to calculate  
>> so this can start to become a fairly expensive feature add.
> I think I am agreeing with Cyrus who said that APPENDLIMIT should  
> not be affected by outstanding quota. It is a hint anyway. Not  
> linking it to quotas would make implementations easier (and clients  
> need to handle errors anyway).

If this is the correct behavior, what's the purpose of APPENDLIMIT in  
the first place?  APPENDLIMIT then becomes an "it might be correct  
value, but it might not be", so a client might be able to use it to  
determine beforehand whether an APPEND but then again you might not.

If APPENDLIMIT is not an accurate representation of the maximum size,  
what useful purpose does it serve over what we currently have?  In  
other words: how is this better/more efficient than calling APPEND  
without LITERAL+ to catch oversized data.  (Note: as the spec is  
written, APPENDLIMIT is solely designed as a limitation to prevent  
unnecessary network traffic.  It is not meant to be used as a UI  
limitation device, e.g. ensure a user can only create a message less  
than a certain size.)

michael

---

From: [email-masked]
Date: Mon, 16 Feb 2015 23:46:23 +0000

Yes, isn't that the question.

If a client wants what appendlimit 
offers, it can just send an append without literal+, right?

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 17 Feb 2015 17:25:27 +0000

Hi Michael,

> On 16 Feb 2015, at 23:09, Michael M Slusarz <[email-masked]> wrote:
> 
>>> 2) I don't believe the APPENDLIMIT response code should be returned for an EXAMINE operation.  The value is irrelevant since you can't append to a read-only mailbox.
>> Actually you are wrong on this one, because the APPEND command takes a mailbox name as a parameter, so it can be used from any mailbox (or no mailbox). So the currently selected mailbox and its state is irrelevant.
> 
> Agreed that APPEND has a mailbox argument, so it can be called from a non-selected context.
> 
> But APPEND is also allowed to be called from within a mailbox.  And if that current selected context is read-only via EXAMINE, my understanding/belief is that an APPEND cannot be successful since that would be an action that causes a "change[] to the permanent state of the mailbox".

This is only talking about changes to flags/per-message state, EXAMINE was not suppose to prevent message injections and APPENDs.

---

From: Jayantheesh S B <[email-masked]>
Date: Tue, 17 Feb 2015 21:06:58 +0000

Hi Michael,

Thanks for your review comments. 

Please find my response inline. 

Regards,
Jay

-----Original Message-----
From: Michael M Slusarz [mailto:[email-masked]] 
Sent: Monday, February 16, 2015 1:08 AM
To: [email-masked]
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension

> Please find the next version of the draft updated with APPENDLIMIT=0 

Was working on a test server implementation of this, and ran into several issues that need to be addressed/clarified.

1) The APPENDLIMIT value is defined in ABNF as a "number", which is limited to 32-bits.  Seems like this should be 64-bits instead to better reflect modern resource limitations (or at least an unsigned 63-bit number, such as MODSEQs).

[Jay] Yes,  ABNF can be changed similar to that of MODSEQ to accommodate 64-bits. 

2) I don't believe the APPENDLIMIT response code should be returned for an EXAMINE operation.  The value is irrelevant since you can't append to a read-only mailbox.  It's possible that APPENDLIMIT may be somewhat expensive to generate - i.e. a quota value that needs to be determined/parsed/updated - so there's no need to report it when it is not directly usable.

[Jay] As per IMAP base RFC3501, If a client issues EXAMINE command, only permanent state and flags of the mailbox should not be changed. 

The EXAMINE command is identical to SELECT and returns the same
      output; however, the selected mailbox is identified as read-only.
      No changes to the permanent state of the mailbox, including
      per-user state, are permitted; in particular, EXAMINE MUST NOT
      cause messages to lose the \Recent flag.

Even I have observed that Gmail, AOL, Yahoo and Isode allows Appending the message in READ-ONLY mode.  Please find the transaction snippet below.

02-17 15:30:31.350  4433 12278 D MailTransport: >>> 25 EXAMINE "Drafts"
02-17 15:30:31.510  4433 12278 D Email   : <<< #null# ["OK", ["CLOSED"], "Previous mailbox closed"]
02-17 15:30:31.520  4433 12278 D Email   : <<< #null# ["FLAGS", ["\Answered", "\Flagged", "\Deleted", "\Draft", "\Seen"]]
02-17 15:30:31.520  4433 12278 D Email   : <<< #null# ["OK", ["PERMANENTFLAGS", []], "Flags"]
02-17 15:30:31.530  4433 12278 D Email   : <<< #null# ["0", "EXISTS"]
02-17 15:30:31.530  4433 12278 D Email   : <<< #null# ["0", "RECENT"]
02-17 15:30:31.540  4433 12278 D Email   : <<< #null# ["OK", ["UIDVALIDITY", "1398813396"], "UIDs valid"]
02-17 15:30:31.540  4433 12278 D Email   : <<< #null# ["OK", ["UIDNEXT", "20"], "Predicted next UID"]
02-17 15:30:31.540  4433 12278 D Email   : <<< #null# ["OK", ["URLMECH", "INTERNAL"], "URL mechs"]
02-17 15:30:31.540  4433 12278 D Email   : <<< #null# ["OK", ["HIGHESTMODSEQ", "67"], "Highest"]
02-17 15:30:31.550  4433 12278 D Email   : <<< #25# ["OK", ["READ-ONLY"], "EXAMINE Completed"]

02-17 15:30:31.660  4433 12278 D MailTransport: >>> 26 APPEND "Drafts" (\SEEN \DRAFT) {994}
02-17 15:30:31.820  4433 12278 D Email   : <<< #+# ["go ahead"]
02-17 15:30:32.430  4433 12278 D Email   : <<< #null# ["1", "EXISTS"]
02-17 15:30:32.450  4433 12278 D Email   : <<< #null# ["1", "RECENT"]
02-17 15:30:32.460  4433 12278 D Email   : <<< #26# ["OK", ["APPENDUID", "1398813396", "20"], "APPEND Completed"]

I believe APPENDLIMIT response code as part of EXAMINE command is fine.

3) I believe this was mentioned by at least one other person, but the current handling of the response code in the mailbox is not correct, at least according to the draft specification.

Take this example:

Assuming that message 1 was 100 bytes, the quota for the user is now up to 200 bytes.  This should be the new APPENDLIMIT value, but according to the spec this does not need to be reported/updated.

This needs to be handled similarly to the way that message counts are updated when in the selected state.  Although, as mentioned above, the appendlimit value may not be a trivial item to calculate so this can start to become a fairly expensive feature add.

It makes me wonder whether adding this response code this is the correct approach.  APPENDs will normally happen in an unselected state (there are at least 2 server implementations that I am aware of that don't allow APPENDs to a selected mailbox), so this information is better obtained by using one of the other two methods anyway.

michael

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 18 Feb 2015 13:23:15 -0700

Quoting Alexey Melnikov <[email-masked]>:

>> But APPEND is also allowed to be called from within a mailbox.  And  
>> if that current selected context is read-only via EXAMINE, my  
>> understanding/belief is that an APPEND cannot be successful since  
>> that would be an action that causes a "change[] to the permanent  
>> state of the mailbox".
>
> This is only talking about changes to flags/per-message state,  
> EXAMINE was not suppose to prevent message injections and APPENDs.

You are probably correct that this was what EXAMINE was supposed to  
mean.  But it isn't clear at all from any of the text in 3501 that  
this is the case (at least to me).  Probably user error on my part,  
but the documents seem to at least plausibly support my clueless  
position.

More concerning is that the ACL RFC explicitly ties APPEND rights in a  
mailbox to a READ-ONLY response, at least for SELECT commands.  So  
"READ-ONLY" arguably has a different meaning whether you are in a  
mailbox opened by SELECT or by EXAMINE, which is different from any  
other IMAP command I can think of in that you need to know the  
selection context of the current state in order to understand the  
meaning.

michael

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 18 Feb 2015 14:03:55 -0700

Quoting [email-masked]:

> If a client wants what appendlimit offers, it can just send an  
> append without literal+, right?

I tend to agree that APPENDLIMIT that doesn't provide a precise size  
doesn't provide anything better than the current situation of simply  
not using literal+ for APPENDs.

I think that if the append limit is static on a server and can be  
provided with the CAPABILITY string, that is a win.  And I have no  
issues with providing a client-side way of *proactively* requesting  
the limit for a mailbox via STATUS.

But passively including this information in SELECT/EXAMINE responses  
is a poor decision if that data is not accurate.  As a client author,  
I would ignore APPENDLIMIT if it is not 100% accurate since there is a  
chance that I might actually be able to add the message  
notwithstanding the value provided.

michael

---

From: Jayantheesh S B <[email-masked]>
Date: Thu, 19 Feb 2015 20:55:33 +0000

Hi Michael,

The intension of this draft to inform client about the supported APPENDLIMIT by the server beforehand to avoid any failure.
Most of the time, in normal uses cases, if an attachment with huge size is appended, server rejects with error response. 
Form the client side, it's not aware of the server side supported limit. Please find the failure transaction snippet below. 

>>>  APPEND "Drafts" (\SEEN \DRAFT) {50287114}
<<<  ["BAD", "parse error: request too long"]

>>> APPEND inbox (\SEEN) {50287114}
<<< BAD [ALERT] Message too large. http://support.google.com/mail/bin/answer.py?answer=8770

That too when LITERAL+ is used, then it consumes more battery power and mobile data in the mobile environment. 
To avoid these kind of issues, this draft proposes two recommendation. 

1.   Avoid using LITERAL+ for APPEND
2.   Advertise IMAP server supported APPENDLIMIT to client, so that client will not try appending the huge attachments.

By implementing the first recommendation, issue will get solved halfway. But still client will not know the exact size supported by the server.
IMAP Client has to do hit and trail to find out the exact size supported by the IMAP server. 

The second recommendation is similar to that of "SMTP Service Extension for Message Size Declaration" (https://tools.ietf.org/html/rfc1870).
Supported APPENDLIMIT value is advertised as part of CAPABILITY response. In case, if IMAP server has mailbox specific limit restriction,
 then it can be published as part of SELECT/EXAMINE, and LIST response.  This value is static in most of the cases, except the corner case,
 where server is critically running short of memory to accommodate new emails. 

To summarize , APPENDLIMIT is a recommendation to the client about the APPEND size supported by the server, rather than the real physical
 memory in server.  It gives an idea to the client that how much it can APPEND, rather than trying appending huge data and then fail.

Regards,
Jay

-----Original Message-----
From: Michael M Slusarz [mailto:[email-masked]] 
Sent: Wednesday, February 18, 2015 4:04 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension

Quoting [email-masked]:

> If a client wants what appendlimit offers, it can just send an append 
> without literal+, right?

I tend to agree that APPENDLIMIT that doesn't provide a precise size doesn't provide anything better than the current situation of simply not using literal+ for APPENDs.

I think that if the append limit is static on a server and can be provided with the CAPABILITY string, that is a win.  And I have no issues with providing a client-side way of *proactively* requesting the limit for a mailbox via STATUS.

But passively including this information in SELECT/EXAMINE responses is a poor decision if that data is not accurate.  As a client author, I would ignore APPENDLIMIT if it is not 100% accurate since there is a chance that I might actually be able to add the message notwithstanding the value provided.

michael

---

From: Stuart Brandt <[email-masked]>
Date: Thu, 19 Feb 2015 22:01:20 -0500

Jay -

Where things are getting muddled for me, at least, is the possible 
conflation of per-mailbox policy based limits (which I suspect is a rare 
case) with Quota.

If a server has a rarely changing, policy based Append limit that 
applies the same to all mailboxes of the authenticated user, would an 
implementation that simply advertises the server's global limit via the 
new Capability be sufficient, or are changes expected to 
Select/Examine/Status/List responses in order to cover the pending quota 
exhaustion case?

- Stuart

---

From: Jayantheesh S B <[email-masked]>
Date: Fri, 20 Feb 2015 16:21:35 +0000

Hi Stuart,

Yes you're right, server's global limit can be advertised via the new Capability (APPENDLIMIT=number), and it will be sufficient.

No, changes are not expected in Select/Examine/Status/List responses in most of the cases, except in cases where server wants to impose per mailbox limit.

We still give server the provision, that it can advertise APPENDLIMIT for per mailbox (if it really wants to).
Otherwise the value advertised by CAPABILITY is sufficient.

Regards,
Jay

-----Original Message-----
From: Stuart Brandt [mailto:[email-masked]] 
Sent: Thursday, February 19, 2015 10:01 PM
To: Jayantheesh S B
Cc: [email-masked]
Subject: Re: [imapext] The IMAP APPENDLIMIT Extension

Jay -

Where things are getting muddled for me, at least, is the possible conflation of per-mailbox policy based limits (which I suspect is a rare
case) with Quota.

If a server has a rarely changing, policy based Append limit that applies the same to all mailboxes of the authenticated user, would an implementation that simply advertises the server's global limit via the new Capability be sufficient, or are changes expected to Select/Examine/Status/List responses in order to cover the pending quota exhaustion case?

- Stuart

On 2/19/15 3:55 PM, Jayantheesh S B wrote:
> <<< BAD [ALERT] Message too large. 
> http://support.google.com/mail/bin/answer.py?answer=8770
>