Subject: [Extra] Working Group Last Call - draft-ietf-extra-imap-fetch-preview-08

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 02 Jul 2020 11:56:04 +1000

As discussed in the interim meeting, this document was brought back to the working group to be reverted to an earlier and simpler version.

This email announces a 2 week working group last call for any further comments on this version before I re-submit to IESG for publication. Please reply with any comments by Thursday, July 16th 2020. Let's get this one off our plates before IETF108.

Thanks,

Bron.

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 2 Jul 2020 12:20:10 +0300

> This email announces a 2 week working group last call for any further comments on this version before I re-submit to IESG for publication.  Please reply with any comments by Thursday, July 16th 202…

   It is possible that preview text is not available now, but might be
   available later -- perhaps the server's preview-generating resources
   are overloaded, there is a server-imposed timeout during preview
   generation, or there is some transient issue with fetching the
   message body.  In such cases, the server will return NIL as the
   preview response, and the client can try to retrieve the preview
   later.

This sounds like it's possible for server to return NIL even when the LAZY modifier isn't specified. Is this really wanted? I'd think client developers would prefer that only LAZY can return NILs. Either way, it's not clearly enough specified what is intended.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 2 Jul 2020 12:45:57 +0100

On 02/07/2020 02:56, Bron Gondwana wrote:

> As discussed in the interim meeting, this document was brought back to 
> the working group to be reverted to an earlier and simpler version.
>
> This email announces a 2 week working group last call for any further 
> comments on this version before I re-submit to IESG for publication.  
> Please reply with any comments by Thursday, July 16th 2020.  Let's get 
> this one off our plates before IETF108.

Other than what Timo reported, this version looks good to me.

---

From: Michael Slusarz <[email-masked]>
Date: Mon, 6 Jul 2020 21:22:08 -0600 (MDT)

>     On 07/02/2020 3:20 AM Timo Sirainen <[email-masked]> wrote:
> 
> 
>     On 2. Jul 2020, at 4.56, Bron Gondwana < [email-masked] mailto:[email-masked] > wrote:
> 
I was originally going to agree with Timo ... but then I remember we added this part as a direct response to a potential DoS situation - what happens if preview requests are (subjectively) taking too much time.

If a server can't produce a preview within a certain time threshold, it should somehow be able to abort the request.  NIL might not be the best solution, but what if it empty string ("") is returned?  Then a caching client may never have preview text for that message, even though the inability to return the text may be a temporary issue.

LAZY means a server should return preview only if it is pre-generated (or can be generated without noticeable delay to user).  non-LAZY request should return PREVIEW unless it exceeds some sort of server-defined timeout.  But in the latter case - unless the server knows it can never create a PREVIEW, it does seem that NIL is the semantically correct response, at least given the current definitions.  Empty string means the server has canonically decided that no reasonable preview can be generated.

Another suggestion might be to return NO response to the FETCH ... but semantics of that are a little tortured if the rest of the FETCH request could be returned, or if preview can be returned for all but one of the messages requested in the FETCH.

In the IMAP spec we have: "The LIST command SHOULD return its data quickly, without undue delay."  Maybe adopt this language and leave it to server to figure out what is "undue delay?"  But that doesn't address whether empty string or NIL is the correct response if undue delay is reached...

michael

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 07 Jul 2020 14:38:04 +1000

On Tue, Jul 7, 2020, at 13:22, Michael Slusarz wrote:
>> On 2. Jul 2020, at 4.56, Bron Gondwana < [email-masked]> wrote: 
>> 

I'm good with the server returning a NO to the whole fetch with text like "preview generation took too long, try again with LAZY". I mean - the Cyrus server does something like that with SEARCH results that take too long these days - it'll just tell you "NO try again with a less expensive search" if it takes over 30s with our config. I expect we'd do the same with FETCH - send a NO (possibly part way through the fetch results) if we hit this kind of error.

Maybe some client advice:

1) make the initial fetch with PREVIEW (LAZY) and the other fields you care about.
2) for items that got a NIL response, batch request non-lazy PREVIEW in sensible-sized batches (e.g. 64 at a time)

Bron.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 7 Jul 2020 09:30:55 +0100

On 07/07/2020 05:38, Bron Gondwana wrote:

>>> On 2. Jul 2020, at 4..56, Bron Gondwana < [email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>>> As discussed in the interim meeting, this document was brought back 
>>>> to the working group to be reverted to an earlier and simpler version.
>>>>
>>>> This email announces a 2 week working group last call for any 
>>>> further comments on this version before I re-submit to IESG for 
>>>> publication.  Please reply with any comments by Thursday, July 16th 
>>>> 2020.  Let's get this one off our plates before IETF108.
>>>
>>> This sounds like it's possible for server to return NIL even when 
>>> the LAZY modifier isn't specified. Is this really wanted? I'd think 
>>> client developers would prefer that only LAZY can return NILs. 
>>> Either way, it's not clearly enough specified what is intended.
>> I was originally going to agree with Timo ... but then I remember we 
>> added this part as a direct response to a potential DoS situation - 
>> what happens if preview requests are (subjectively) taking too much time.
>>
>> If a server can't produce a preview within a certain time threshold, 
>> it should somehow be able to abort the request. NIL might not be the 
>> best solution, but what if it empty string ("") is returned?  Then a 
>> caching client may never have preview text for that message, even 
>> though the inability to return the text may be a temporary issue.
>>
>> LAZY means a server should return preview only if it is pre-generated 
>> (or can be generated without noticeable delay to user).  non-LAZY 
>> request should return PREVIEW unless it exceeds some sort of 
>> server-defined timeout.  But in the latter case - unless the server 
>> knows it can never create a PREVIEW, it does seem that NIL is the 
>> semantically correct response, at least given the current 
>> definitions.  Empty string means the server has canonically decided 
>> that no reasonable preview can be generated..
>>
>> Another suggestion might be to return NO response to the FETCH ... 
>> but semantics of that are a little tortured if the rest of the FETCH 
>> request could be returned, or if preview can be returned for all but 
>> one of the messages requested in the FETCH.
>>
>> In the IMAP spec we have: "The LIST command SHOULD return its data 
>> quickly, without undue delay."  Maybe adopt this language and leave 
>> it to server to figure out what is "undue delay?"  But that doesn't 
>> address whether empty string or NIL is the correct response if undue 
>> delay is reached...
>
> I'm good with the server returning a NO to the whole fetch with text 
> like "preview generation took too long, try again with LAZY".  I mean 
> - the Cyrus server does something like that with SEARCH results that 
> take too long these days - it'll just tell you "NO try again with a 
> less expensive search" if it takes over 30s with our config.  I expect 
> we'd do the same with FETCH - send a NO (possibly part way through the 
> fetch results) if we hit this kind of error.

I think returning NO to FETCH is problematic and will cause problems 
with clients as they don't expect it. Untagged NO might be slightly better.

> 2) for items that got a NIL response, batch request non-lazy PREVIEW 
> in sensible-sized batches (e.g. 64 at a time)

These recommendations are good.

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 7 Jul 2020 11:54:30 +0300

On 7. Jul 2020, at 6.22, Michael Slusarz <michael.slusarz=[email-masked]> wrote:
>> On 2. Jul 2020, at 4.56, Bron Gondwana < [email-masked] <mailto:[email-masked]>> wrote: 

I don't really see why this would be an issue. IMAP protocol doesn't have this kind of a special "DoS-prevention feature" anywhere else. Especially now that the preview is always text, I don't think it can be excessively slow to generate it. At least in Dovecot's implementation it's basically as expensive as fetching BODYSTRUCTURE, and we don't have DoS prevention against fetching those. Maybe in theory if server wanted to analyze attachments in some way .. but is it really worth the added complexity for a problem that probably will remain theoretical?

FETCH is also good in that it returns the results after each mail. So the server is allowed to start throttling the responses to the client without actually failing it.

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 07 Jul 2020 21:00:55 +1000

> FETCH is also good in that it returns the results after each mail. So the server is allowed to start throttling the responses to the client without actually failing it.

You're right. Here we are overcomplexing it again!

Server MUST NOT return NIL to a non-lazy preview is fine by me.

Server SHOULD NOT return NIL is fine by me as well, along with clients SHOULD NOT cache a NIL response. Either way our server is always going to have one because it gets calculated on APPEND.

Bron.

---

From: Michael Slusarz <[email-masked]>
Date: Tue, 14 Jul 2020 21:29:33 -0600 (MDT)

It seems I was a day late to submit, as submissions are paused before the meeting...

I've finished the -09 draft.  If interested, I could post a copy.  Otherwise, I will just submit at end of the month when submission tool is reopened.

Changes in -09:
* PREVIEW without LAZY MUST NOT return NIL (kept it simple)
* Rewrote the LAZY modifier section a bit to break apart the semantics of LAZY from "Client Implementation Advice".  The latter is a combo of some of the text previously in the section with list recommendations.  Slightly tweaked Example 2 to request two previews in a single FETCH request in background.
Decided to make no mention of NO response in this document.  After discussion here, it seems like that is expected to be an extremely rare case - and that case is already adequately handled by FETCH command definition in base IMAP spec.

michael

>     On 07/07/2020 5:00 AM Bron Gondwana <[email-masked]> wrote:
>      
>      
>         > >         On 7. Jul 2020, at 6.22, Michael Slusarz <michael.slusarz=[email-masked] mailto:michael.slusarz=[email-masked] > wrote:
> > 
> >             > > >              
> > > 
> > >                 > > > >                 On 07/02/2020 3:20 AM Timo Sirainen <[email-masked] mailto:[email-masked] > wrote:
> > > >                  
> > > >                  
>     _______________________________________________
>     Extra mailing list
>     [email-masked]
>     https://www.ietf.org/mailman/listinfo/extra
>

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 24 Jul 2020 14:21:35 +1000

I'm ready to go with this on Monday :)