Subject: [Extra] Discussing SNIPPET

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 17 Sep 2018 07:52:42 -0400

Hi All, 
 
I've just posted REPLACE off for publication after the expiry of the WGLC period and confirming the IPR status. 
 
In terms of the remaining documents, we have a couple of Sieve docs which Jiankang is busy Shepherding through their last calls, we have 64bit which we allowed to lapse (for now) and we have imap4rev2 which is a giant piece of work. 
 Finally, we have the recently adopted draft-ietf-extra-imap-fetch-snippet-00.
 
To summarise the discussion: 
 
1) there's nobody who thinks this is a bad idea 
2) there's general agreement that specifying an exact algorithm is bad 
3) while LAZY may or may not be that valuable, nobody minds it that much. Easy approach is to always generate regardless of the modifier. 
 
So much for the things everyone agreed on (please correct me if I missed anything!) Here's the issues under debate: 
 
a) Naming. The great difficulties, naming and cache invalidation! (and yep, we've got the second one too). Should it be called SNIPPET, or something else like PREVIEW? I'm asking largely because JMAP used "SearchSnippet" to refer to a fragment of matching text with terms hilighted. And JMAP used "preview" for the short piece of text (up to 256 characters) that represents the email content. 
 
b) Cache Invalidation. Ho hum. Is this an immutable property? If so, what happens if the server implementation changes? I'm quite happy to call it "softly immutable" myself - it might not be exactly the same bytes, but you can use the old version without major problems. 
 
c) Length:  
   The server SHOULD limit the length of the snippet text to 100
   characters.  The server MUST NOT output snippet text longer than 200
   characters.
 
I would prefer to bring this up to more like 200 for should and 256 for MUST NOT. 100 is a pretty short preview on big screens. I'd also be willing to have a way for the client to hint for shorter previews to save network bandwidth, but have servers generally store 256 chars. The syntax makes it quite easy to add a MAXLEN=80 or whatever to the fetch command. 
 
And yes, this would be trivial to implement in our server with its existing previews, and I would totally do it during last call :) 
 Bron. 

-- 
 Bron Gondwana, CEO, FastMail Pty Ltd 
 [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 17 Sep 2018 17:31:23 -0700

On 17 Sep 2018, at 4:52, Bron Gondwana wrote:
> Hi All,
>
> I've just posted REPLACE off for publication after the expiry of the 
> WGLC period and confirming the IPR status.
>
> In terms of the remaining documents, we have a couple of Sieve docs 
> which Jiankang is busy Shepherding through their last calls, we have 
> 64bit which we allowed to lapse (for now) and we have imap4rev2 which 
> is a giant piece of work.
> 3) while LAZY may or may not be that valuable, nobody minds it that 
> much. Easy approach is to always generate regardless of the modifier.

+1.

> So much for the things everyone agreed on (please correct me if I 
> missed anything!) Here's the issues under debate:
>
> a) Naming. The great difficulties, naming and cache invalidation! (and 
> yep, we've got the second one too). Should it be called SNIPPET, or 
> something else like PREVIEW? I'm asking largely because JMAP used 
> "SearchSnippet" to refer to a fragment of matching text with terms 
> hilighted. And JMAP used "preview" for the short piece of text (up to 
> 256 characters) that represents the email content.

I strongly feel that JMAP and IMAP should use consistent terminology 
when possible. Although the term 'snippet' is cute, the term 'preview' 
is more technically accurate for the desired functionality, particularly 
when producing a fuzzy preview for an image-only message (in which case 
the dictionary definition of 'snippet' is not what's desired). So I have 
a mild preference for changing SNIPPET to match JMAP terminology.

> b) Cache Invalidation. Ho hum. Is this an immutable property? If so, 
> what happens if the server implementation changes? I'm quite happy to 
> call it "softly immutable" myself - it might not be exactly the same 
> bytes, but you can use the old version without major problems.

The 'softly immutable' option works for me. However, I'll propose an 
alternative so we can contrast two reasonable options. We could add a 
way to fetch a FUZZY algorithm identifier for a given folder. Something 
like:

C: A043 STATUS blurdybloop (FUZZYALG)
S: * STATUS blurdybloop (FUZZYALG ("ocms:8.0"))

A cached PREVIEW=FUZZY (SNIPPET=FUZZY) from that folder is immutable 
unless the FUZZYALG string changes. This has some non-trivial server 
implications, but if clients really need predictable PREVIEW caching 
behavior, this is a protocol model I'd be willing to implement to 
provide that.

> I would prefer to bring this up to more like 200 for should and 256 
> for MUST NOT. 100 is a pretty short preview on big screens. I'd also 
> be willing to have a way for the client to hint for shorter previews 
> to save network bandwidth, but have servers generally store 256 chars. 
> The syntax makes it quite easy to add a MAXLEN=80 or whatever to the 
> fetch command.

No opinions here. Also easy for me to implement, but only useful if 
client implementers use it.

		- Chris

> And yes, this would be trivial to implement in our server with its 
> existing previews, and I would totally do it during last call :)
>  Bron.
>
>
> -- 

> _______________________________________________
> Extra mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_extra&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=Bzf8nG-Anmd7ybQFCYQd_kA5QfrsUFDlskrm11eWwSY&s=HOeED7FM1BpaoYj0D522AM5_Ke2XpfLSYfjTUoqhrPc&e=

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 18 Sep 2018 10:47:21 +0300

> On 17 Sep 2018, at 4:52, Bron Gondwana wrote:
...
> alternative so we can contrast two reasonable options. We could add a way to
> fetch a FUZZY algorithm identifier for a given folder. Something like:
> 
> A cached PREVIEW=FUZZY (SNIPPET=FUZZY) from that folder is immutable unless
> the FUZZYALG string changes. This has some non-trivial server implications,
> but if clients really need predictable PREVIEW caching behavior, this is a
> protocol model I'd be willing to implement to provide that.

For what it's worth, in Dovecot we cache the generated snippet/preview along
with a version number and take the 'softly immutable' approach.  For
example, an email with body of "foobar" would turn into "1foobar" in the
cache file.  When a client FETCHes it, we strip the version.  We haven't had
the need to up the version yet, but at least in theory it should provide us
with a way to tweak the snippet algo and know which cached values are stale.

I'm not fully caught up on the discussions here so maybe I'm missing
something, but I was under the impression that the algo can be selected per
*email* so wouldn't it really have to be a FETCH * FUZZYALG?.  I don't quite
understand what the above STATUS accomplishes; I guess it is something like
a uidvalidity for snippets in a folder but I'm not convinced that it is
worth it.  If you want to have the latest snippet, just FETCH it.

Jeff.

-- 
mainframe, n.:
  An obsolete device still used by thousands of obsolete companies serving
  billions of obsolete customers and making huge obsolete profits for their
  obsolete shareholders. And this year's run twice as fast as last year's.

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 18 Sep 2018 17:53:58 +1000

On Tue, Sep 18, 2018, at 17:47, Josef 'Jeff' Sipek wrote:
>>> b) Cache Invalidation. Ho hum. Is this an immutable property? If so,>>> what happens if the server implementation changes? I'm quite
>>> happy to>>> call it "softly immutable" myself - it might not be exactly the same>>> bytes, but you can use the old version without major problems.
>> 
>> The 'softly immutable' option works for me. However, I'll propose an>> alternative so we can contrast two reasonable options. We could add
>> a way to>> fetch a FUZZY algorithm identifier for a given folder.
>> Something like:>> 
>> A cached PREVIEW=FUZZY (SNIPPET=FUZZY) from that folder is
>> immutable unless>> the FUZZYALG string changes. This has some non-trivial server
>> implications,>> but if clients really need predictable PREVIEW caching behavior,
>> this is a>> protocol model I'd be willing to implement to provide that.
> 
> For what it's worth, in Dovecot we cache the generated
> snippet/preview along> with a version number and take the 'softly immutable' approach.  For
> example, an email with body of "foobar" would turn into
> "1foobar" in the> cache file.  When a client FETCHes it, we strip the version.  We
> haven't had> the need to up the version yet, but at least in theory it should
> provide us> with a way to tweak the snippet algo and know which cached values
> are stale.
We do something similar at FastMail - we have a keyword (currently $X-ME-Annot-
2) which is a version number for the logic which generated the preview
and a couple of other keywords like $HasAttachment by parsing the
message content.
> something, but I was under the impression that the algo can be
> selected per> **email** so wouldn't it really have to be a FETCH * FUZZYALG?.  I
> don't quite> understand what the above STATUS accomplishes; I guess it is
> something like> a uidvalidity for snippets in a folder but I'm not convinced
> that it is> worth it.  If you want to have the latest snippet, just FETCH it.

I agree with that!  It's small enough that re-fetching isn't a big deal
if you are worried that it might be old - and the cost in roundtrips to
check the version first plus the added complexity isn't worth it.
Bron.

---

From: Michael Slusarz <[email-masked]>
Date: Thu, 20 Sep 2018 18:03:08 -0600 (MDT)

> On September 17, 2018 at 5:52 AM Bron Gondwana <[email-masked]> wrote:
> 
I don't have a personal attachment to SNIPPET (other than we already implemented with that terminology, but that's easily worked around by some internal aliasing).

PREVIEW makes sense if we are looking to a future where additional algorithms would be developed.  As Chris points out, if you have algorithms that return both text blobs and something like image data, preview makes more sense semantically than snippet.

I have no objection if we wanted to change the label.

>     b) Cache Invalidation.   Ho hum.  Is this an immutable property?  If so, what happens if the server implementation changes?  I'm quite happy to call it "softly immutable" myself - it might not…
I spent quite a bit of time formulating the logic behind this topic when drafting the spec. Reproduced here for convenience:

-----

A server SHOULD strive to generate the same string for a given message for each request. However, since snippets are understood to be a representation of the message data and not a canonical view of its contents, a client MUST NOT assume that a message snippet is immutable for a given message. This relaxed requirement permits a server to offer snippets as an option without requiring potentially burdensome storage and/or processing requirements to guarantee immutability for a use case that does not require this strictness.

----

I would love to hear feedback what parts of this are weak or what parts need to be explained further.

I was thinking less about the situation of a server invalidating its cache rather than the scenario where a server loses its cache.  Demanding that server upgrades ensure that its algorithm stays exactly the same across versions, in order to rebuild exactly the preview data, is a burden I don't think is useful to enforce.  No user is going to be confused if their preview text suddenly differs across mail sessions.

Feedback so far doesn't seem to indicate this is too troubling of a design decision.  As mentioned above, I welcome feedback on how this can be explained better in the document.

>     I would prefer to bring this up to more like 200 for should and 256 for MUST NOT.  100 is a pretty short preview on big screens.  I'd also be willing to have a way for the client to hint for sh…
100 characters was selected via an informal survey of several different clients, both mobile and desktop, and how they currently display preview-like information in their UI.  It's probably a bit mobile-centric, since I had more of those applications available at the time.

More important, that analysis was probably too English-centric.  English is quite compact compared to many other languages, so it's probably a bad example to use when trying to determine ideal lengths (see, e.g., https://www.w3.org/International/articles/article-text-size).

200 characters seems just as arbitrary :)  But arguing over 100 extra characters stored per message is probably not a fight worth pursuing.  So I don't have a strong opinion on raising to 200 or max length as 256.  Although 256 isn't all that special since we are talking UTF-8 characters, not bytes, so landing on a power of 2 doesn't seem that necessary (just say 250 characters?).

MAXLEN seems like overkill though.  If a server is buffering output, we're not going to save that many network packets if a client requests partial text, since the use-case of preview is only to load data for the mailbox viewport.  And it adds more complexity on server side that doesn't seem to give us much benefit.

If we really wanted some sort of maxlen, for consistency sake it probably makes more sense to implement as <<partial>> syntax anyway.

michael