Subject: Re: [imapext] [IMAP] APPEND Command Usage

From: Bron Gondwana <[email-masked]>
Date: Sat, 01 Nov 2014 00:24:27 -0700

On Fri, Oct 31, 2014, at 06:07 PM, Jan Kundrát wrote:
> On Saturday, 1 November 2014 01:37:15 CEST, Jayantheesh S B wrote:
> > I have created a short document “IMAP APPENDLIMIT extension” , 
> > which proposes the CAPABILITY based solution for the APPEND 
> > command usage issue.
> 
> I tend to think that using QUOTA for this would be a bit more intuitive, 
> but CAPABILITY will of cousre work as well. I don't think there's much 
> real-world need for per-mailbox APPEND size limits.

I agree, though QUOTA would allow them to be set if you really needed them.

> > capability    =/ "APPENDLIMIT=" size-param
> > 
> > A decimal number indicating the fixed maximum message size in bytes
> > that the server will accept. The syntax of the parameter is as
> > follows, using the augmented BNF notation of [RFC 822]:
> > 
> > size-param = [1*DIGIT]
> 
> You could probably use a nz-number from RFC3501 here. Also, you put the 
> grammar within square brackets which mean that this is an optional 
> production, and therefore you allow a capability named just "APPENDLIMIT=". 
> Is that what you mean? If so, how should a client interpret that?
> 
> With kind regards,
> Jan

Back to the original issue, which is Gmail append size limits.  I wonder if the abuse they were designed to deal with (uploading ISO images as emails?) is as much of an issue these days.

No matter what we agree to here, Gmail still won't be able to turn on LITERAL+, because the clients that broke before are probably the ones that won't check a new value anyway.

(I was thinking MAXLITERALSIZE as a name, but I'm not sure if it's the literals that are the issue, or the size of an appended message.  I think most people are thinking appended message, so you can't just CATENATE a bunch of literals)

Just to summarise my understanding of the options:

New quota type (MAXAPPENDSIZE or similar):
===
Pro: can do per-mailbox if required
Pro: it's already designed to be extensible with new fields
Pro: a well-behaving client already has to fetch the QUOTA to know if it can append a message of a given size, to be sure there is enough STORAGE space - so there's no additional round trip.
Pro: it kind of makes sense to be here, since it's a limit - certainly in Cyrus I would implement it as another quota field (there's a limit of UINT32_MAX anyway due to the index format).  We already do STORAGE, MESSAGE, ANNOTSTORAGE and NUMFOLDERS quotas.
Con: there's no real concept of "used" for a limit, unlike the above-mentioned.

New Command (MAXAPPENDSIZE or similar):
===
Pro: trivial to code on both server and client
Pro: no danger of confusing old clients with naive quota parsers
Con: extra roundtrip
Con: undiscoverable (unless you add a CAPABILITY as well)

Capability item (APPENDLIMIT=):
===
Pro: no extra roundtrip
Pro: very discoverable
Con: parsing issue (no current capability has a number in the value)
Con: capability string is getting very long!

Extra response line to SELECT:
===
Pro: not sure there really is any, because...
Con: you don't need to select a mailbox to append.  So let's not talk about this one any more.

I'm a fan of the quota, personally.  It feels like the right place to put limits - and just have a USED of zero.  On the server I would already be looking at the quota struct to see if the message was allowed to be appended, and I imagine that on a client I would be looking at the quota response already to see if I was allowed to upload a message.  It would just be a matter of treating the APPENDLIMIT item exactly the same was as the STORAGE item - identical code even, because with a 'used' of zero and a 'limit' of the allowed size for a single item, the maths would be identical.

Regards,

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 1 Nov 2014 12:41:56 +0000

Hi Bron,

> On 1 Nov 2014, at 07:24, Bron Gondwana <[email-masked]> wrote:
> 

Indeed. I think a new capability is needed to signal that LITERAL+ syntax is allowed, but not in APPEND.
> 
> (I was thinking MAXLITERALSIZE as a name, but I'm not sure if it's the literals that are the issue, or the size of an appended message.

IMHO, APPEND is the only place (other than possibly annotations) where big literals is an issue.
FETCH items can generally have much lower literal size limit.
> I think most people are thinking appended message, so you can't just CATENATE a bunch of literals)
> 

Good summary.

> Con: there's no real concept of "used" for a limit, unlike the above-mentioned.

Con: too much extra work for servers that don't support QUOTA. (My implementation would be happy with this.)
> Con: undiscoverable (unless you add a CAPABILITY as well)

I suppose this is similar to what ManageSieve has. But I think this is a non starter.
> Con: you don't need to select a mailbox to append.  So let's not talk about this one any more.
Yeah, good point. Although I am Ok with that as well. Maybe a new STATUS item then?

> I'm a fan of the quota, personally.  It feels like the right place to put limits - and just have a USED of zero.  On the server I would already be looking at the quota struct to see if the message…

My preference is a capability (easy to add, easy to handle in clients) + new QUOTA type for servers that support it.

Best Regards,
Alexey

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 1 Nov 2014 14:17:24 +0100

On Thursday, October 30, 2014 11:01:46 PM CEST, Brandon Long wrote:
> Sure, though you have to hit the limit first in that case.  I'm 
> fine with that, especially since we disabled LITERAL+ so that we 
> could respond to APPEND directly with a NO without having to eat 
> the overly large message or drop the connection.

Is dropping the connection very bad?

I ask because the $#@!$@# middlebox vendors seem to regard it as harmless, 
and perhaps they've even forced it to be mostly harmless.

Arnt

---

From: Jan Kundrát <[email-masked]>
Date: Sat, 01 Nov 2014 18:02:20 +0100

On Saturday, 1 November 2014 08:24:27 CEST, Bron Gondwana wrote:
> Pro: a well-behaving client already has to fetch the QUOTA to 
> know if it can append a message of a given size, to be sure 
> there is enough STORAGE space - so there's no additional round 
> trip.

The original RFC from 1997 is silent about this, but a newer draft from 
Alexey contains a very strong language that this actually MUST NOT be done 
[1] :).

With kind regards,
Jan

[1] http://tools.ietf.org/html/draft-melnikov-imapext-quota-00#section-5.1

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: Dave Cridland <[email-masked]>
Date: Sat, 1 Nov 2014 18:53:15 +0000

You can't have clients prune quota find out a message can stored because
quota isn't exact.

As a random collection of other thoughts:

Responding BAD to an spend for merely being too large seems wrong, there's
no way got the client to know a priori. So this should be a NO. A response
code on this NO seems sensible.

If a client tries sending a huge message with LITERAL+, issuing an untagged
NO, with a response code, and then a BYE seems fair.
On 1 Nov 2014 17:02, "Jan Kundrát" <[email-masked]> wrote:

>> Pro: a well-behaving client already has to fetch the QUOTA to know if it
>> can append a message of a given size, to be sure there is enough STORAGE
>> space - so there's no additional round trip.
>>
>
> _______________________________________________
> imapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/imapext
>

---

From: Jayantheesh S B <[email-masked]>
Date: Sun, 2 Nov 2014 03:26:08 +0000

Hi Jan,

Please find my response inline. 

Regards,
Jay
-----Original Message-----
From: Jan Kundrát [mailto:[email-masked]] 
Sent: Friday, October 31, 2014 9:08 PM
To: [email-masked]
Subject: Re: [imapext] [IMAP] APPEND Command Usage

On Saturday, 1 November 2014 01:37:15 CEST, Jayantheesh S B wrote:
> I have created a short document “IMAP APPENDLIMIT extension” , which 
> proposes the CAPABILITY based solution for the APPEND command usage 
> issue.

I tend to think that using QUOTA for this would be a bit more intuitive, but CAPABILITY will of cousre work as well. I don't think there's much real-world need for per-mailbox APPEND size limits.

> follows, using the augmented BNF notation of [RFC 822]:

You could probably use a nz-number from RFC3501 here. Also, you put the grammar within square brackets which mean that this is an optional production, and therefore you allow a capability named just "APPENDLIMIT=". 
Is that what you mean? If so, how should a client interpret that?

[Jay] Thanks for your review comments. This number is not the optional one, hence I will change the grammar accordingly. 
         As per your comments, I will update the value as well to nz-number. 

With kind regards,
Jan

---

From: Michael M Slusarz <[email-masked]>
Date: Sun, 02 Nov 2014 18:49:49 -0700

>> Con: capability string is getting very long!

What about a hybrid approach to an APPENDLIMIT (or whatever to call  
it) capability?

A server that has a constant APPEND limit regardless of mailbox (i.e.  
an admin settable option in the server config file for maximum append  
size allowed") could report this static limit as an argument to  
APPENDLIMIT:  APPENDLIMIT=10000000.  For this (most-likely common)  
server setup, it prevents the client from having to do roundtrips  
later to find the append limit for an individual mailbox.

A server that has variable APPEND limits (i.e. mailboxes are stored  
internally on different mailstore backends) would advertise  
APPENDLIMIT capability only, without any numeric argument.  If a  
client sees this bare APPENDLIMIT capability, it knows it can query  
the server - whether via STATUS, QUOTA, etc. - for an individual  
mailbox's limit before doing an APPEND.

So common use case requires 0 additional roundtrips, while leaving  
open extensibility for server configuration.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Brandon Long <[email-masked]>
Date: Mon, 3 Nov 2014 13:15:34 -0800

On Sat, Nov 1, 2014 at 12:24 AM, Bron Gondwana <[email-masked]> wrote:

> Back to the original issue, which is Gmail append size limits.  I wonder
> if the abuse they were designed to deal with (uploading ISO images as
> emails?) is as much of an issue these days.
>

http://en.wikipedia.org/wiki/GmailFS

But really, the problem is that, for better or worse, our current
implementation requires the whole message to be in memory, and its not
practical to allow someone to have a 2GB message in memory (which is an ask
we had from a Google Apps customer who's C-level execs apparently used to
trade videos via email).

> No matter what we agree to here, Gmail still won't be able to turn on
> LITERAL+, because the clients that broke before are probably the ones that
> won't check a new value anyway.
>

Most likely true.  We've had it off for years, now, so I'm a bit surprised
that its come up again.  With our support for UTF8=ACCEPT, I would think
the need for literals for non-APPEND would also be less.

I guess if new clients supported this value, we might be able to get away
with something less ideal for older clients, such as turning off
advertisement of LITERAL+ for a day or so if they try to append too large a
message.  Turning it off was the trivial fix, for us, and we didn't really
have an impetus for a better fix.

There was a suggestion for a "no literal+ for APPEND, ok for anything else"
capability.

I don't really have a strong opinion any which way on these.

Brandon

(I was thinking MAXLITERALSIZE as a name, but I'm not sure if it's the
> literals that are the issue, or the size of an appended message.  I think
> most people are thinking appended message, so you can't just CATENATE a
> bunch of literals)
>
> Pro: it kind of makes sense to be here, since it's a limit - certainly in
> Cyrus I would implement it as another quota field (there's a limit of
> UINT32_MAX anyway due to the index format).  We already do STORAGE,
> MESSAGE, ANNOTSTORAGE and NUMFOLDERS quotas.
> I'm a fan of the quota, personally.  It feels like the right place to put
> limits - and just have a USED of zero.  On the server I would already be
> looking at the quota struct to see if the message was allowed to be
> appended, and I imagine that on a client I would be looking at the quota
> response already to see if I was allowed to upload a message.  It would
> just be a matter of treating the APPENDLIMIT item exactly the same was as
> the STORAGE item - identical code even, because with a 'used' of zero and a
> 'limit' of the allowed size for a single item, the maths would be identical.
>
> Regards,
>
> Bron.
>
>
> --
>   Bron Gondwana
>   [email-masked]
>

---

From: Brandon Long <[email-masked]>
Date: Mon, 3 Nov 2014 13:20:47 -0800

On Sat, Nov 1, 2014 at 6:17 AM, Arnt Gulbrandsen <[email-masked]>
wrote:

>> Sure, though you have to hit the limit first in that case.  I'm fine with
>> that, especially since we disabled LITERAL+ so that we could respond to
>> APPEND directly with a NO without having to eat the overly large message or
>> drop the connection.
>>
>

In general, the IMAP spec is under-specified in terms of "what to do on
failure", and so clients don't really seem equipped to handle failure well.

If we just dropped the connection, the client would just re-connect and
re-try, infinitely, and never make forward progress uploading any other
sent messages.  It also caused bandwidth/battery usage, depending on how
much of the message they managed to upload before we dropped the
connection.  I don't recall now what the back-off or wait-time was on
retries.

The client seemed to handle our BAD response better.  YMMV.

I have noticed other clients which respond very poorly to dropped
connections, isn't there a known issue with older Outlook clients where a
dropped connection during IDLE would mean that it wouldn't notice the
connection was missing?

Brandon

---

From: Jayantheesh S B <[email-masked]>
Date: Thu, 6 Nov 2014 20:40:02 +0000

Hi All,

Please find the updated version of the Draft attached.

I have addressed the review comments related to ABNF and updated RFC reference section. 

Regards,
Jay
-----Original Message-----
From: Jayantheesh S B 
Sent: Saturday, November 01, 2014 11:26 PM
To: 'Jan Kundrát'; [email-masked]'
Subject: RE: [imapext] [IMAP] APPEND Command Usage

Hi Jan,

Regards,
Jay
-----Original Message-----
From: Jan Kundrát [mailto:[email-masked]]
Sent: Friday, October 31, 2014 9:08 PM
To: [email-masked]
Subject: Re: [imapext] [IMAP] APPEND Command Usage

With kind regards,
Jan

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 07 Nov 2014 11:28:42 +0000

> open extensibility for server configuration.

I like this approach as well.

---

From: Jayantheesh S B <[email-masked]>
Date: Mon, 10 Nov 2014 15:45:38 +0000

Hi All,

We are also fine with this approach. 

Our concern is, will there be any use cases, where folder specific append limit is needed ?

Please share your comments for this proposed dynamic approach. 

If everybody is ok, we can fix this approach as the solution for this problem.

Regards,
Jay

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 11 Nov 2014 16:57:55 -0700

> Our concern is, will there be any use cases, where folder specific  

A simple example would be an imap proxy, where each mailbox may point  
to a mailbox on another IMAP server (that may all have different  
append limits).

Another example would be an appendlimit that dynamically changes  
depending on the user's current quota usage.

Or an appendlimit that dynamically changes depending on the storage  
space currently available on the mail store backend.

michael

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 12 Nov 2014 08:48:26 +0000

That is why I suggested a response code. Those can be sent at any time. As part of login processing, of select, even of noop, and all the currents that know how to parse uidnext have the parsing support already.

(Sorry about the silence lately.)

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Wed, 12 Nov 2014 10:29:20 +0000

On 12 November 2014 08:48, Arnt Gulbrandsen <[email-masked]>
wrote:

> That is why I suggested a response code. Those can be sent at any time. As
> part of login processing, of select, even of noop, and all the currents
> that know how to parse uidnext have the parsing support already.
>

I would:

1) Add a response-code that indicates the append limit:

appendlimit-respcode = "APPENDLIMIT" SP nz-number
resp-text-code /= appendlimit-respcode

2) Make the nz-number parameter to the capability optional:

appendlimit-cap = "APPENDLIMIT" [ "=" nz-number ]
capability /= appendlimit-cap

3) Indicate that the capability append limit - if given - is a global
maximum, but may be overridden by individual mailboxes or other
circumstances. Specifically instruct clients to avoid the use of LITERAL+
during APPEND as a result; note that this advice is recommended behaviour
in general. (This draft will therefore update LITERAL+).

4) Indicate that the response code MAY be sent at any time, but suggest it
is most useful:

 - On an untagged OK response during SELECT, to advise clients of a
per-mailbox maximum.
 - On a tagged NO response to an APPEND.

5) Note that if a server receives a non-synchronizing literal which would
exceed the limit, it MAY immediately issue such a NO response and terminate
the session with BYE.

Dave.

---

From: Jayantheesh S B <[email-masked]>
Date: Wed, 12 Nov 2014 22:21:48 +0000

Dear Alexey/Michael/Dave/Arnt,

Thanks for the review and response.

All,

Kindly let me know, everybody agrees with the below proposed solution.

Regards,
Jay
From: Dave Cridland [mailto:[email-masked]]
Sent: Wednesday, November 12, 2014 5:29 AM
To: Arnt Gulbrandsen
Cc: Jayantheesh S B; Alexey Melnikov; [email-masked]; Michael M Slusarz
Subject: Re: [imapext] [IMAP] APPEND Command Usage

On 12 November 2014 08:48, Arnt Gulbrandsen <[email-masked]<mailto:[email-masked]>> wrote:
That is why I suggested a response code. Those can be sent at any time. As part of login processing, of select, even of noop, and all the currents that know how to parse uidnext have the parsing support already.

I would:

3) Indicate that the capability append limit - if given - is a global maximum, but may be overridden by individual mailboxes or other circumstances. Specifically instruct clients to avoid the use of LITERAL+ during APPEND as a result; note that this advice is recommended behaviour in general. (This draft will therefore update LITERAL+).

4) Indicate that the response code MAY be sent at any time, but suggest it is most useful:

 - On an untagged OK response during SELECT, to advise clients of a per-mailbox maximum.
 - On a tagged NO response to an APPEND.

5) Note that if a server receives a non-synchronizing literal which would exceed the limit, it MAY immediately issue such a NO response and terminate the session with BYE.

Dave.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 12 Nov 2014 13:55:59 -1000

Dave's proposal looks good to me.

> On 12 Nov 2014, at 12:21, Jayantheesh S B <[email-masked]> wrote:
> 
> On 12 November 2014 08:48, Arnt Gulbrandsen <[email-masked]> wrote:

---

From: Jayantheesh S B <[email-masked]>
Date: Mon, 17 Nov 2014 17:16:24 +0000

Hi All,

I have incorporated all the review comments and updated the next version of the draft.

Kindly review and share your valuable comments.

Regards,
Jay
From: Alexey Melnikov [mailto:[email-masked]]
Sent: Wednesday, November 12, 2014 6:56 PM
To: Jayantheesh S B
Cc: Dave Cridland; Arnt Gulbrandsen; [email-masked]; Michael M Slusarz
Subject: Re: [imapext] [IMAP] APPEND Command Usage

On 12 Nov 2014, at 12:21, Jayantheesh S B <[email-masked]<mailto:[email-masked]>> wrote:
Dear Alexey/Michael/Dave/Arnt,

All,

Regards,
Jay
From: Dave Cridland [mailto:[email-masked]]
Sent: Wednesday, November 12, 2014 5:29 AM
To: Arnt Gulbrandsen
Cc: Jayantheesh S B; Alexey Melnikov; [email-masked]<mailto:[email-masked]>; Michael M Slusarz
Subject: Re: [imapext] [IMAP] APPEND Command Usage

I would:

Dave.

---

From: Stuart Brandt <[email-masked]>
Date: Thu, 20 Nov 2014 22:04:02 -0500

Is it appropriate to cover interaction with the CATENATE extension 
(rfc4469) in the draft? That RFC already suggests that the server return 
TOOBIG if construction of the server-side message exceeds a 
server-specific limit, but I'd imagine APPENDLIMIT=<n> would be a more 
useful response for those servers/clients that support APPENDLIMIT.

- Stuart

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 24 Nov 2014 04:32:27 +0000

Hi Stuart,

> On 21 Nov 2014, at 03:04, Stuart Brandt <[email-masked]> wrote:
> 
> Is it appropriate to cover interaction with the CATENATE extension (rfc4469) in the draft? That RFC already suggests that the server return TOOBIG if construction of the server-side message exceeds a server-specific limit, but I'd imagine APPENDLIMIT=<n> would be a more useful response for those servers/clients that support APPENDLIMIT.

You have a very good point!
And yes, we should reuse the TOOBIG response code, if it can convey the per-mailbox limit.