Subject: [Extra] Call for adoption - draft-slusarz-imap-fetch-snippet

From: Bron Gondwana <[email-masked]>
Date: Fri, 20 Jul 2018 07:13:01 +1000

Hi All,

As discussed in today's meeting, this is a formal call for adoption for:
https://datatracker.ietf.org/doc/draft-slusarz-imap-fetch-snippet/

It was clear in the room that this isn't the final form of the
document, so this is a call to adopt the document into the group for
ongoing work.  Please let the group know if you have any objection to
taking on this document.
I will give two weeks for this, so please file any objections by
August 3rd.
Kind regards,

Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Thu, 19 Jul 2018 19:14:30 -0400

I've reviewed this document. We have implemented the FUZZY algorithm in 
both a client and server (under an XSNIPPET capability). I support 
adoption of this document by the WG.

We have not implemented the LAZY= modifier. I reviewed our code and 
believe the LAZY= modifier has a logical server-side implementation that 
is reasonable. At this time, I do not see a use case for the LAZY= 
modifier in our client. Has there been client implementer interest in 
the LAZY= modifier? I don't personally object to the LAZY= modifier, but 
the bar for acceptance of a feature in a specification should preferably 
an implementation or active implementer interest.

		- Chris

---

From: Michael Slusarz <[email-masked]>
Date: Fri, 20 Jul 2018 12:01:25 -0600 (MDT)

Chris,

Thanks for the input.  Great news to hear about the implementation.

See below.

> On July 19, 2018 at 5:14 PM Chris Newman <[email-masked]> wrote:
> 
> 

The LAZY modifier was the final element added to the Snippet draft, and it was added specifically due to feedback from our internal *client* development team when implementing the base FETCH SNIPPET part.

The client in this case is a (heavily-modified) JavaMail library running in our middleware serving our webmail and mobile UIs.  The ask from the client team was for LAZY modifier, since without it the mailbox listing was blocked from an end-user UX perspective.  

I don't know if this blocking behavior could be worked around by altering the code/IMAP commands sent -- I have a feeling with JavaMail that this is not the easiest ask -- but the LAZY modifier was a solution that worked for them and was easily implementable in both client and server side code.

Obviously, it is up for discussion whether LAZY is generally useful to the community and/or the semantics of its optimal behavior, but it was added for a specific need and has been implemented in publicly released code. So this is not the case of a speculative API addition to solve a problem that might potentially occur, if that is the concern.

michael

p.s. Someone brought up the idea of LAZY FETCH results being sent as untagged FETCH responses at some point after the FETCH command was completed.  I dropped off the call at that point so didn't hear the reactions to that suggestion, but I disagree with that behavior because 1) it behaves differently than any other explicit FETCH request, where the data is immediately returned before the command is complete, and 2) this behavior is not desirable in a more disconnected mode of access that our client is implementing.

---

From: Bron Gondwana <[email-masked]>
Date: Sat, 21 Jul 2018 04:46:20 +1000

On Sat, Jul 21, 2018, at 04:01, Michael Slusarz wrote:
> The LAZY modifier was the final element added to the Snippet draft,
> and it was added specifically due to feedback from our internal
> **client** development team when implementing the base FETCH
> SNIPPET part.> 
> The client in this case is a (heavily-modified) JavaMail library
> running in our middleware serving our webmail and mobile UIs.  The ask
> from the client team was for LAZY modifier, since without it the
> mailbox listing was blocked from an end-user UX perspective.> 
> I don't know if this blocking behavior could be worked around by
> altering the code/IMAP commands sent -- I have a feeling with JavaMail
> that this is not the easiest ask -- but the LAZY modifier was a
> solution that worked for them and was easily implementable in both
> client and server side code.> 
> Obviously, it is up for discussion whether LAZY is generally useful to
> the community and/or the semantics of its optimal behavior, but it was
> added for a specific need and has been implemented in publicly
> released code. So this is not the case of a speculative API addition
> to solve a problem that might potentially occur, if that is the
> concern.
Because we always generate snippets (we call them preview) on delivery
in our server, we probably wouldn't need LAZY, but we could also
trivially implement it.
It seems a reasonable thing to add "I'd like snippets, but only if it's
not too much work".
> michael
> 
> p.s. Someone brought up the idea of LAZY FETCH results being sent as
>      untagged FETCH responses at some point after the FETCH command
>      was completed.  I dropped off the call at that point so didn't
>      hear the reactions to that suggestion, but I disagree with that
>      behavior because 1) it behaves differently than any other
>      explicit FETCH request, where the data is immediately returned
>      before the command is complete, and 2) this behavior is not
>      desirable in a more disconnected mode of access that our client
>      is implementing.
Yeah, the room came to the same conclusion "cute idea, but unlikely to
be useful in the real world".
Bron.
--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 20 Jul 2018 15:28:30 -0400

On 20 Jul 2018, at 14:01, Michael Slusarz wrote:
> *client* development team when implementing the base FETCH SNIPPET 
> part.
>
> mailbox listing was blocked from an end-user UX perspective.
>
> client and server side code.

Sounds good. Thanks for sharing the implementation experience!

> to solve a problem that might potentially occur, if that is the 

Since I want SNIPPET standardized, I wanted to prompt for implementation 
experience that will remove barriers to standardization. I expected a 
response like this, but getting it on record now makes advancement 
easier.

> untagged FETCH responses at some point after the FETCH command was 
> completed.  I dropped off the call at that point so didn't hear the 
> reactions to that suggestion, but I disagree with that behavior 
> because 1) it behaves differently than any other explicit FETCH 
> request, where the data is immediately returned before the command is 
> complete, and 2) this behavior is not desirable in a more disconnected 
> mode of access that our client is implementing.

I objected to the proposed untagged/deferred LAZY FETCH at the mike on 
the grounds that the server doesn't have enough information about what 
specific information the client wants to know. Unless we extended IMAP 
to with some sort of client state/interest mechanism, the server doesn't 
have enough information to share the right subset of data in the right 
order.

		- Chris

---

From: Michael Peddemors <[email-masked]>
Date: Wed, 25 Jul 2018 08:29:59 -0700

+1 on supporting the concept of retrieving a 'SNIPPET', however I also 
do not see that 'LAZY' is beneficial, and I think that by separating 
that out of this proposal, might help to bring about a quicker 
'consensus'..

I do however believe that the working group might assist in adding some 
more context to the proposal, eg defining exactly what a 'snippet' is, 
or whether to support 'types' of snippets.

It seems that in the draft, it strongly hints that the 'snippet' should 
be the first few lines of the first text part.  The idea of 'fuzzy' is 
such that the server decides what the text is should return, but I think 
it would be helpful to include further information on standardized 
algorithms..

Just tossing ideas out..

Eg, do we return 'No Plain Text Part, HTML Only' as a snippet?
Should HTML parts be parsed to return some context?
Should a standardized 'snippet' be returned, if the message only 
includes one part, say an image, an .ics, a document?

Maybe even just a couple of 'algorithm' conventions might help to 
increase it's value and acceptance..

We notice that it is recommended that the 'snippet-alg-ext' be 
registered, but maybe 'snippet-alg' and 'snippet-alg-ext' might 
converge.. eg all should be registered algorithms?

Eg, in the sense that the FLUFFY means, 'give me what the server thinks 
is a snippet', versus.. 'PLAINTEXT', meaning ask the server ONLY for the 
plain text parts..

And of course, letting my mind wander conceptually, is the 'snippet' 
really an array of items, that might include 
'FIRST_200_CHARS_FIRST_PART_TEXT'?

Any chance what we are looking for is really a retrieval of 'META' data 
for messages, in a method that should be standardized and one of the 
pieces is the 'FIRST_200_CHARS_FIRST_PART_TEXT'

I 100% agree that everyone who has been involved in doing any client 
design, understands the benefit of getting the 'meta data', in this case 
the 'snippet', rather than parsing the message at the client end to 
develop a 'snippet'.. and that it would be good to standardize the 
retrieval mechanism..

I would put myself in the camp of, 'I would not object' but maybe we can 
reach the submitters objective, in a more encompassing way?

I believe that client developers will have an 'expectation' of what the 
'FUZZY' would generate, and would not like to deal with support issues 
if a servers idea of what to return, is different than what they expect.

I also think that expecting IMAP implementations to support 'many' types 
of specific 'algorithms' may be un-realistic, so everyone will just not 
specify the algorithm, as they would prefer to receive something than 
nothing..

OR, is this really 'please return me meta data of message' known as 
'SNIPPET', which is strictly defined as 'FIRST_200_CHARS_PLAIN_TEXT_DATA'

Now of course, 'meta' would be confusing, given RFC 5464, but 
conceptually you might get what I am suggesting.. standard algorithms to 
return very specific forms of meta data.. one being the first 200 
characters of the plain text part, another could be returning the 
attachment names, another could be summary information..

I suggest that clients are going to want to see a very specific set of 
data returned, and should know what the returned 'snippet' is, without 
depending on the servers idea of what to return..

Make any sense?

Worried that this might end up being, get me snippet using algorithm 
one, and if I get no result, try algorithm two, etc .. And then, are we 
faced with having to include supported algorithms in the capability 
advertisement?

Maybe if the capability is advertised as '<NAME>' where support of 
'NAME>' means that a defined list of meta data be supported, where the 
first iteration requires the ability to return 'SNIPPET' as defined as 
'FIRST_200_CHARS_PLAIN_TEXT_DATA', and in the future additional 'meta' 
definitions can be added ..

C: E1 CAPABILITY
      S: * CAPABILITY IMAP4rev1 MSGMETA=SNIPPET SEARCHRES

On 18-07-20 11:46 AM, Bron Gondwana wrote:
>> **client** development team when implementing the base FETCH SNIPPET part.
>>
>> to solve a problem that might potentially occur, if that is the concern.
> 
> _______________________________________________
> Extra mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/extra
> 

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 25 Jul 2018 10:45:14 -0700

There was some previous discussion on SNIPPET. It turns out to be quite 
difficult to specify a precise algorithm. Should it be the first text 
part? What if the first text part is a text/vcard or 
text/tab-separated-values or text/vnd.ascii-art? What about 
multipart/alternative? It gets complicated quickly.

In the past when we've tried to get too precise with this sort of 
algorithm, we ended up with an algorithm that many in the community 
decided wasn't "good enough". The THREAD=REFERENCES algorithm in RFC 
5256 is an example of that.

For servers that operate at large scale, SNIPPET has to be extracted and 
cached to maximize performance. Having multiple algorithms is not 
desirable as it creates either performance problems or a growing cache 
size that has specialized content for different clients.

FUZZY leaves this as a "quality of implementation" issue. If servers 
generate lousy SNIPPETS, clients will go back to the pre-SNIPPET model 
which consumes more server resources so server implementations have 
incentive to improve the FUZZY algorithm to maximize deployment 
efficiency. The incentives are right for FUZZY SNIPPET to get better 
over time in an agile fashion, so I think it's important we don't 
over-specify.

		- Chris

On 25 Jul 2018, at 8:29, Michael Peddemors wrote:

> I do however believe that the working group might assist in adding 
> some more context to the proposal, eg defining exactly what a 
> 'snippet' is, or whether to support 'types' of snippets.
>
> It seems that in the draft, it strongly hints that the 'snippet' 
> should be the first few lines of the first text part.  The idea of 
> 'fuzzy' is such that the server decides what the text is should 
> return, but I think it would be helpful to include further information 
> on standardized algorithms..
>
> Just tossing ideas out..
>
> Eg, in the sense that the FLUFFY means, 'give me what the server 
> thinks is a snippet', versus.. 'PLAINTEXT', meaning ask the server 
> ONLY for the plain text parts..
>
> Any chance what we are looking for is really a retrieval of 'META' 
> data for messages, in a method that should be standardized and one of 
> the pieces is the 'FIRST_200_CHARS_FIRST_PART_TEXT'
>
> design, understands the benefit of getting the 'meta data', in this 
> case the 'snippet', rather than parsing the message at the client end 
> to develop a 'snippet'.. and that it would be good to standardize the 
> retrieval mechanism..
>
> I would put myself in the camp of, 'I would not object' but maybe we 
> can reach the submitters objective, in a more encompassing way?
>
> I believe that client developers will have an 'expectation' of what 
> the 'FUZZY' would generate, and would not like to deal with support 
> issues if a servers idea of what to return, is different than what 
> they expect.
>
> I also think that expecting IMAP implementations to support 'many' 
> types of specific 'algorithms' may be un-realistic, so everyone will 
> just not specify the algorithm, as they would prefer to receive 
> something than nothing..
>
> conceptually you might get what I am suggesting.. standard algorithms 
> to return very specific forms of meta data.. one being the first 200 
> one, and if I get no result, try algorithm two, etc .. And then, are 
> we faced with having to include supported algorithms in the capability 
> advertisement?
>
>>> running in our middleware serving our webmail and mobile UIs.  The 
>>> ask from the client team was for LAZY modifier, since without it the 
>>> altering the code/IMAP commands sent -- I have a feeling with 
>>> JavaMail that this is not the easiest ask -- but the LAZY modifier 
>>> was a solution that worked for them and was easily implementable in 
>>> both client and server side code.
>>>
>>> Obviously, it is up for discussion whether LAZY is generally useful 
>>> to the community and/or the semantics of its optimal behavior, but 
>>> it was added for a specific need and has been implemented in 
>>> publicly released code. So this is not the case of a speculative API 
>>> addition to solve a problem that might potentially occur, if that is 
>>> the concern.
>>
>> Because we always generate snippets (we call them preview) on 
>> delivery in our server, we probably wouldn't need LAZY, but we could 
>> also trivially implement it.
>>
>> It seems a reasonable thing to add "I'd like snippets, but only if 
>> it's not too much work".
>>
>>> michael
>>>
>>> request, where the data is immediately returned before the command 
>>> is complete, and 2) this behavior is not desirable in a more 
>>> disconnected mode of access that our client is implementing.
>>
>> Yeah, the room came to the same conclusion "cute idea, but unlikely 
>> to be useful in the real world".
>>
>> Bron.
>> --
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_extra&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=xWlYsjQlERK2oz4FpxCSba1t3D035GeMWvncBW3LLl8&s=h99Wzu74nqwqvPyYUaXRmExGLppz1oUvQR_XVanFols&e=
>>
>
>
>
> -- 
> Visit us at 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linuxmagic.com&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=xWlYsjQlERK2oz4FpxCSba1t3D035GeMWvncBW3LLl8&s=COb_ODe6Sw6kk_bnHVCsjdbHFxz-9_3NwBTalzWBDRo&e= 
> @linuxmagic
> A Wizard IT Company - For More Info 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.wizard.ca&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=xWlYsjQlERK2oz4FpxCSba1t3D035GeMWvncBW3LLl8&s=OFwh9Pjy5gLrc87gawbIdSwPqOOEQGHEQZCobBLJMp8&e=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_extra&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=xWlYsjQlERK2oz4FpxCSba1t3D035GeMWvncBW3LLl8&s=h99Wzu74nqwqvPyYUaXRmExGLppz1oUvQR_XVanFols&e=

---

From: Ned Freed <[email-masked]>
Date: Thu, 26 Jul 2018 08:04:39 -0700 (PDT)

> multipart/alternative? It gets complicated quickly.

This is similar in some respects to the algorithm for header/footer insertion.
Having implemented this, I can say that this is just the tip of the iceberg -
you also need to deal with things like multipart/alternative nested inside of
multipart/related, and vice versa. *

So far I've managed to deal with it algorithmically, but if things continue
down the current path it's going to require an extensible template scheme
and pattern matching to do properly.

				Ned

* This is actual observed traffic; nothing theoretical about it.

---

From: Michael Slusarz <[email-masked]>
Date: Thu, 26 Jul 2018 17:22:40 -0600 (MDT)

> On July 25, 2018 at 9:29 AM Michael Peddemors <[email-masked]> wrote:
>  

LAZY, as it currently exists in the draft, is essentially optional.

Clients don't need to use it.

Servers that don't want to implement it can always return a NIL response to requests with the LAZY modifier.  (Clients using SNIPPET will always need to fallback to the non-LAZY method on a LAZY miss.)  The only overhead is having to parse the modifier and add the code to always return NIL.

For our purposes, I believe adding full LAZY support required something like 50-100 lines of code.

> or whether to support 'types' of snippets.

Chris summarized my thoughts on FUZZY, so I will defer to his message as my response.

michael