Subject: Re: [Extra] Suggestions on IMAP PARTIAL response

From: Alexey Melnikov <[email-masked]>
Date: Thu, 6 Jan 2022 11:14:35 +0000

Hi all,

On 26/12/2021 15:27, Timo Sirainen wrote:

> On 23. Dec 2021, at 21.06, Vikram Nagulakonda 
> <nvikram=[email-masked]> wrote:
>>
>> Hi All
>> We are introducing MESSAGELIMIT capability and response code in IMAP 
>> PARTIAL RFC 
>> <https://datatracker.ietf.org/doc/draft-melnikov-imap-partial/>. IMAP 
>> PARTIAL RFC is enforcing message retrieval and update limits per imap 
>> request.
>
> "Interaction between PARTIAL, MIN, MAX and SAVE SEARCH return options" 
> matches how Dovecot v2.3.11+ behave (I think older versions just 
> returned everything in $).
>
> Should the non-UID FETCH command also support PARTIAL? It could be 
> useful together with CHANGEDSINCE.
>
>> As per RFC, IMAP server will return NO with MESSAGELIMIT response 
>> code to clients using PARTIAL return options on FETCH & SEARCH imap 
>> requests.
>>
>> For FETCH & SEARCH imap requests without PARTIAL return option, 
>> according to RFC the server will return messages up to the limit 
>> defined in capabilities along with an OK/MESSAGELIMIT response code.
>
> I don't see how you can implement this MESSAGELIMIT for FETCH (and 
> maybe SEARCH) without breaking existing IMAP clients. They will not be 
> happy about partial fetch results, regardless of whether it's 
> returning OK or NO.

I think this would depend on the client and how the data is used. My 
client handles missing data, but it might request it again if it thinks 
it is missing from the client cache.

Anyway, it would be good to test this.

> I'd think NO would be at least a bit better since it indicates that 
> everything didn't work right.
I think FETCH returning NO is just weird and might break clients. I 
think it is generally more useful to omit data on a partially failed 
FETCH and possibly use untagged NO response.
>
> You could of course make it optional by requiring client to ENABLE 
> MESSAGELIMIT, but then why would any client ever do that?
>
> I'm not sure what is the point of it anyway especially for FETCH? The 
> server could just throttle the FETCH responses. Does it matter whether 
> the client sends one FETCH command for 100k mails, vs 100 FETCH 
> commands for 1k mails?
>
> And how is MESSAGELIMIT supposed to work for SEARCH? Is it applying 
> for the set of messages being searched, or the number of search 
> results? I guess the former? What is client supposed to do if it 
> receives this error? Again just do the same SEARCH but just in 
> multiple commands, each with an extra seqset parameter for pagination?
>
> FETCH and SEARCH can also return OK [EXPUNGEISSUED] if it skipped over 
> some mails that were already expunged. This could conflict with the 
> [MESSAGELIMIT].

I clarified this in the draft. Basically tagged OK [EXPUNGEISSUED] 
should be returned in this case, with untagged * NO [MESSAGELIMIT ...]

(We can switching this around. Either order should work, as long as one 
is recommended.)

>> We would like to keep the response for COPY, MOVE & STORE imap 
>> requests and wanted to get your opinions & suggestions. Do you think 
>> it is a good idea to perform a partial update and reply with 
>> OK/MESSAGELIMIT response code?
>
> COPY is supposed to be atomic and not perform partial copies.
Yeah, this is what is bothering me with returning MESSAGELIMIT on COPY. 
If the server needs to return MESSAGELIMIT, what is the best behaviour 
for clients that don't understand the MESSAGELIMIT response code? This 
is the choice between misleading the client that all messages were 
copied, when only some where (tagged OK [MESSAGELIMIT ...]) versa 
returning tagged NO [MESSAGELIMIT] and preventing any messages from 
being copied.
> MOVE could I guess, if it returns NO [MESSAGELIMIT].
>
>> If you agree that it is ok to do a partial update, how should we 
>> return the MESSAGELIMIT response code on COPY as tagged COPY response 
>> includes COPYUID response code?
>
> Existing clients aren't going to understand it anyway, so maybe * NO 
> (TAG "...") [MESSAGELIMIT] similarly to ESEARCH. Or something similar.

Yes, this is what the draft currently recommends, except it doesn't 
include the tag.

Best Regards,

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 6 Jan 2022 11:26:57 +0000

Hi all,

(I restored the subject for ease of searching in the mailing list archive.)

On 30/12/2021 08:03, Timo Sirainen wrote:
> On 30. Dec 2021, at 1.48, Vikram Nagulakonda 
>>     ---------- Forwarded message ----------
>>     From: Timo Sirainen <[email-masked]>
>>     To: Vikram Nagulakonda <nvikram=[email-masked]>
>>     Cc: "[email-masked]" <[email-masked]>
>>     Bcc:
>>     Date: Sun, 26 Dec 2021 17:27:40 +0200
>>     Subject: Re: [Extra] Suggestions on IMAP PARTIAL response
>>>     We are introducing MESSAGELIMIT capability and response code in
>>>     IMAP PARTIAL RFC
>>>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmelnikov-2Dimap-2Dpartial_&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=iZ14qCSBWgSfwV0j_3_IrghmBn31BHLbst--UCMBCA0&m=S0F2F4uimu2jnhGAKI2ZF0ai8VncRNMerPQ9xOS7I-GqMGCihJJ-eOz3XzvPo0aJ&s=cFHCG2qb4QOIZWWIbqjIe3hMjjAYn4_OUGHxeq1VjYw&e=>.
>>>     IMAP PARTIAL RFC is enforcing message retrieval and update
>>>     limits per imap request.
>>
>>     "Interaction between PARTIAL, MIN, MAX and SAVE SEARCH
>>     return options" matches how Dovecot v2.3.11+ behave (I think
>>     older versions just returned everything in $).
>>
>>     Should the non-UID FETCH command also support PARTIAL? It could
>>     be useful together with CHANGEDSINCE.
>>
>> Our thought was that sequence numbers were the same as what PARTIAL 
>> was attempting to do. At least in my mind with a mailbox having 1000 
>> messages, 1. UID FETCH 1:* (FLAGS) (PARTIAL 1:100) is equivalent to 
>> FETCH 1:100 (UID FLAGS) and 2. UID FETCH 1:* (FLAGS) (PARTIAL 
>> -100:-1) is equivalent to FETCH 901:1000 (UID FLAGS).
>
> Right, that's why I was specifically mentioning CHANGEDSINCE. But now 
> reading it again, I think you intended it to work differently than I 
> first understood. I thought the example:
>
>   C: 101 UID FETCH 25900:26600 (UID FLAGS) (PARTIAL -1:-30 
> CHANGEDSINCE 98305)
>
> would return the last 30 messages where modseq >= 98305, but I guess 
> you meant it first gets the last 30 messages from the UID range and 
> out of those it returns the messages where modseq >= 98305?
Yes, the latter.
>>
>>>     As per RFC, IMAP server will return NO with MESSAGELIMIT
>>>     response code to clients using PARTIAL return options on FETCH &
>>>     SEARCH imap requests.
>>>
>>     I don't see how you can implement this MESSAGELIMIT for FETCH
>>     (and maybe SEARCH) without breaking existing IMAP clients. They
>>     will not be happy about partial fetch results, regardless of
>>     whether it's returning OK or NO. I'd think NO would be at least a
>>     bit better since it indicates that everything didn't work right.
>>
>> NO is what we would prefer. However, this breaks the end-user 
>> experience completely. Most of our imap clients download partial 
>> mailboxes and mostly never download older messages. Our intention was 
>> to indicate the response is different without breaking the user 
>> experience.
>
> I haven't actually looked recently at what the IMAP clients send, but 
> I remember earlier most clients did FETCH 1:* FLAGS whenever opening a 
> folder. Wouldn't this break the clients' flag syncing?

I think we (I mean the EXTRA WG :-)) should write updated flag sync 
instructions for clients that don't care about the whole mailbox when it 
is over certain size, when the server supports something like PARTIAL 
extension.

Best Regards,

Alexey

/*
*/