Subject: Re: [Extra] [E] Extra Digest, Vol 48, Issue 3

From: Vikram Nagulakonda <[email-masked]>
Date: Wed, 29 Dec 2021 15:48:03 -0800

>
> ---------- Forwarded message ----------
> From: Timo Sirainen <[email-masked]>
> To: Vikram Nagulakonda <nvikram=[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Bcc:
> Date: Sun, 26 Dec 2021 17:27:40 +0200
> Subject: Re: [Extra] Suggestions on IMAP PARTIAL response
> On 23. Dec 2021, at 21.06, Vikram Nagulakonda <
> nvikram=[email-masked]> wrote:
>
>
> Hi All
> We are introducing MESSAGELIMIT capability and response code in IMAP
> PARTIAL RFC
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmelnikov-2Dimap-2Dpartial_&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=iZ14qCSBWgSfwV0j_3_IrghmBn31BHLbst--UCMBCA0&m=S0F2F4uimu2jnhGAKI2ZF0ai8VncRNMerPQ9xOS7I-GqMGCihJJ-eOz3XzvPo0aJ&s=cFHCG2qb4QOIZWWIbqjIe3hMjjAYn4_OUGHxeq1VjYw&e=>.
> IMAP PARTIAL RFC is enforcing message retrieval and update limits per imap
> request.
>
>
> "Interaction between PARTIAL, MIN, MAX and SAVE SEARCH return options"
> matches how Dovecot v2.3.11+ behave (I think older versions just returned
> everything in $).
>
> Should the non-UID FETCH command also support PARTIAL? It could be useful
> together with CHANGEDSINCE.
>
> Our thought was that sequence numbers were the same as what PARTIAL was
attempting to do. At least in my mind with a mailbox having 1000 messages,
1. UID FETCH 1:* (FLAGS) (PARTIAL 1:100) is equivalent to FETCH 1:100 (UID
FLAGS) and 2. UID FETCH 1:* (FLAGS) (PARTIAL -100:-1) is equivalent to
FETCH 901:1000 (UID FLAGS).

> As per RFC, IMAP server will return NO with MESSAGELIMIT response code to
> clients using PARTIAL return options on FETCH & SEARCH imap requests.
>
> For FETCH & SEARCH imap requests without PARTIAL return option, according
> to RFC the server will return messages up to the limit defined in
> capabilities along with an OK/MESSAGELIMIT response code.
>
>
> I don't see how you can implement this MESSAGELIMIT for FETCH (and maybe
> SEARCH) without breaking existing IMAP clients. They will not be happy
> about partial fetch results, regardless of whether it's returning OK or NO.
> I'd think NO would be at least a bit better since it indicates that
> everything didn't work right.
>
NO is what we would prefer. However, this breaks the end-user experience
completely. Most of our imap clients download partial mailboxes and mostly
never download older messages. Our intention was to indicate the response
is different without breaking the user experience.

> You could of course make it optional by requiring client to ENABLE
> MESSAGELIMIT, but then why would any client ever do that?
>
Completely agree with you on ENABLE that clients would never issue the call.

> I'm not sure what is the point of it anyway especially for FETCH? The
> server could just throttle the FETCH responses. Does it matter whether the
> client sends one FETCH command for 100k mails, vs 100 FETCH commands for 1k
> mails?
>
Our systems have a layered architecture and retrieving 100k messages in one
request would end up consuming lots of memory for a single request. We
prefer smaller payloads to less requests and keeping the service available
for all users. One of our biggest problems is clients over using 1:*
queries even when most of the response payload is thrown away by the client.

> And how is MESSAGELIMIT supposed to work for SEARCH? Is it applying for
> the set of messages being searched, or the number of search results? I
> guess the former? What is client supposed to do if it receives this error?
> Again just do the same SEARCH but just in multiple commands, each with an
> extra seqset parameter for pagination?
>
Yes. Partial or pagination is being applied on the messages being searched.
Our preference is for clients to repeat the search with smaller message
sets.

>
> FETCH and SEARCH can also return OK [EXPUNGEISSUED] if it skipped over
> some mails that were already expunged. This could conflict with the
> [MESSAGELIMIT].
>
Thanks for identifying the problem with EXPUNGEISSUED. We will review.

>
> We would like to keep the response for COPY, MOVE & STORE imap requests
> and wanted to get your opinions & suggestions. Do you think it is a good
> idea to perform a partial update and reply with OK/MESSAGELIMIT response
> code?
>
>
> COPY is supposed to be atomic and not perform partial copies. MOVE could I
> guess, if it returns NO [MESSAGELIMIT].
>
NO seems to be the right response here.

>
> If you agree that it is ok to do a partial update, how should we return
> the MESSAGELIMIT response code on COPY as tagged COPY response includes
> COPYUID response code?
>
>
> Existing clients aren't going to understand it anyway, so maybe * NO (TAG
> "...") [MESSAGELIMIT] similarly to ESEARCH. Or something similar.
>
Thanks. We will review and get back.

-Vikram

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 30 Dec 2021 10:03:08 +0200

On 30. Dec 2021, at 1.48, Vikram Nagulakonda <nvikram=[email-masked]> wrote:
> From: Timo Sirainen <[email-masked] <mailto:[email-masked]>>
> To: Vikram Nagulakonda <nvikram=[email-masked] <mailto:[email-masked]>>
> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> Bcc: 
> On 23. Dec 2021, at 21.06, Vikram Nagulakonda <nvikram=[email-masked] <mailto:nvikram=[email-masked]>> wrote:
>> 
>> Hi All
>> We are introducing MESSAGELIMIT capability and response code in IMAP PARTIAL RFC <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmelnikov-2Dimap-2Dpartial_&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=iZ14qCSBWgSfwV0j_3_IrghmBn31BHLbst--UCMBCA0&m=S0F2F4uimu2jnhGAKI2ZF0ai8VncRNMerPQ9xOS7I-GqMGCihJJ-eOz3XzvPo0aJ&s=cFHCG2qb4QOIZWWIbqjIe3hMjjAYn4_OUGHxeq1VjYw&e=>. IMAP PARTIAL RFC is enforcing message retrieval and update limits per imap request.
> 
> "Interaction between PARTIAL, MIN, MAX and SAVE SEARCH return options" matches how Dovecot v2.3.11+ behave (I think older versions just returned everything in $).
> 
> Should the non-UID FETCH command also support PARTIAL? It could be useful together with CHANGEDSINCE.
> 
> Our thought was that sequence numbers were the same as what PARTIAL was attempting to do. At least in my mind with a mailbox having 1000 messages, 1. UID FETCH 1:* (FLAGS) (PARTIAL 1:100) is equivalent to FETCH 1:100 (UID FLAGS) and 2. UID FETCH 1:* (FLAGS) (PARTIAL -100:-1) is equivalent to FETCH 901:1000 (UID FLAGS).

Right, that's why I was specifically mentioning CHANGEDSINCE. But now reading it again, I think you intended it to work differently than I first understood. I thought the example:

  C: 101 UID FETCH 25900:26600 (UID FLAGS) (PARTIAL -1:-30 CHANGEDSINCE 98305)

would return the last 30 messages where modseq >= 98305, but I guess you meant it first gets the last 30 messages from the UID range and out of those it returns the messages where modseq >= 98305?

>> As per RFC, IMAP server will return NO with MESSAGELIMIT response code to clients using PARTIAL return options on FETCH & SEARCH imap requests.
>> 
>> For FETCH & SEARCH imap requests without PARTIAL return option, according to RFC the server will return messages up to the limit defined in capabilities along with an OK/MESSAGELIMIT response code.
> 
> I don't see how you can implement this MESSAGELIMIT for FETCH (and maybe SEARCH) without breaking existing IMAP clients. They will not be happy about partial fetch results, regardless of whether it's returning OK or NO. I'd think NO would be at least a bit better since it indicates that everything didn't work right.
> NO is what we would prefer. However, this breaks the end-user experience completely. Most of our imap clients download partial mailboxes and mostly never download older messages. Our intention was to indicate the response is different without breaking the user experience.

I haven't actually looked recently at what the IMAP clients send, but I remember earlier most clients did FETCH 1:* FLAGS whenever opening a folder. Wouldn't this break the clients' flag syncing?

> You could of course make it optional by requiring client to ENABLE MESSAGELIMIT, but then why would any client ever do that?
> I'm not sure what is the point of it anyway especially for FETCH? The server could just throttle the FETCH responses. Does it matter whether the client sends one FETCH command for 100k mails, vs 100 FETCH commands for 1k mails?
> Our systems have a layered architecture and retrieving 100k messages in one request would end up consuming lots of memory for a single request. We prefer smaller payloads to less requests and keeping the service available for all users. One of our biggest problems is clients over using 1:* queries even when most of the response payload is thrown away by the client.
> 
> 
> And how is MESSAGELIMIT supposed to work for SEARCH? Is it applying for the set of messages being searched, or the number of search results? I guess the former? What is client supposed to do if it receives this error? Again just do the same SEARCH but just in multiple commands, each with an extra seqset parameter for pagination?
> Yes. Partial or pagination is being applied on the messages being searched. Our preference is for clients to repeat the search with smaller message sets.

Seems .. a bit weird to create a new RFC and require clients to change behavior just because of one server implementation. There's no benefit for clients to work this way. Also I don't see any benefit for server to work this way other than it being easier for your implementation. I don't really even see why the standard behavior is more difficult for you. Dovecot is already doing the same kind of FETCH pagination with its "imapc" storage driver (used e.g. for accessing shared folders). It converts incoming FETCH 1:* into a series of individual FETCH commands towards a remote IMAP server, and merges the replies so the original IMAP client sees a single response to its FETCH 1:* command. There's no extra memory usage required for that, since untagged replies are immediately forwarded to client while receiving them. So I don't see why you couldn't do the same thing. Just process the FETCH in smaller pieces internally rather than forcing clients to do it for you.

For COPY I could see MESSAGELIMIT being a helpful feature, since we've run into troubles with clients attempting to COPY millions of mails and then timeout & retry when it's not happening in 10 seconds or so. Although even there I wouldn't really want to just completely break existing clients, so I'd prefer doing it only when client acknowledges that it understands MESSAGELIMIT by issuing ENABLE. MOVE isn't so much of a problem because it can commit the progress periodically.