Subject: Re: [Extra] Barry Leiba's Discuss on draft-ietf-extra-imap-fetch-preview-03: (with DISCUSS and COMMENT)

From: Michael Slusarz <[email-masked]>
Date: Sat, 1 Jun 2019 09:46:17 -0600 (MDT)

My comments below.

> On May 13, 2019 8:22 PM Barry Leiba <[email-masked]> wrote:
> 
>  
> Hi, Michael,
> I'm sorry it took me so long to get back to this, and I won't let that
> happen again.
> 
> > > ----------------------------------------------------------------------
> > > DISCUSS:
> > > ----------------------------------------------------------------------
> > >
> > > — Section 3.1 —
> > >
> > > I don’t understand “the client’s priority decision”: what decision is that?
> > > And what’s the point of giving the server a list of algorithms here, given that
> > > they all have to be ones that are supported by the server?  Won’t the server
> > > always have to use the first one in the list?  If not, please add some text
> > > explaining what the server does.
> >
> > Consensus seems to be this section is confusing, or unclear, or unneeded, or some combination of this three.
> 
> But it's still there and still unexplained in version -05.  Let's see
> where we can go with it:
> 
> > I'll start with providing a (future) real-world example, and why this
> > behavior exists in its present form.  Given a hypothetical future
> > algorithm "IMAGE" (generate image/jpeg preview if image data exists in
> > message), I issue this preview command:
> >
> > a FETCH 1 (PREVIEW (LAZY=IMAGE FUZZY))
> >
> > This is asking the server: provide me an image preview of the message,
> > but ONLY if 1) the message actually contains image information and 2)
> > that image information is already generated or can be generated very
> > quickly.  Otherwise, fall back to FUZZY generation.
> 
> Hm.  That really seems like a contrived example.  How would the client
> software possibly know whether there's useful information in the
> image, or whether the information in the accompanying text is more or
> less important?  Suppose I have two messages:
> 
> 1. text/plain that basically says nothing; image/jpeg that has a
> diagram that explains everything
> 
> 2. text/plain that has the entire real content of the message;
> image/jpeg of the senders company logo
> 
> It's pretty clear from that explanation that for message 1 I want some
> sort of description of the image and for message 2 I want a preview of
> the text.  But how would the client software know that, and,
> therefore, know what to ask for?

Is that what a typical received message looks like?  To me, that looks like the edge case.

Seems that a server could figure out (through experience) when image preview data should be calculated or ignored.  Image size, MIME structure information, MIME order, etc. could all be used to reach some sort of 95% accuracy threshold.  Maybe that doesn't happen day 1 of implementation, but a competitive, reactive server implementation will work on this.

The same issue exists for text previews, FWIW.  What about a text/plain + text/html alternative message, where text/plain is "Please click on this URL to view the text part", and the text/html part is empty text (it's a single image link).  A server can be programmed to be smart enough to ignore that part re: preview.

As written, we allow a server to be given the latitude to figure these kind of issues out.  Closing down that opportunity because it might be a difficult logical problem isn't the right approach from my view.

> Apart from that, if we're going to do any sort of text-based preview
> of an image, I would think we'd just say that when FUZZY is applied to
> an image, that's what it does.

FUZZY is explicitly defined as text only output though, since that's generally mapping what clients already do.  Defining an algorithm that can potentially take any data input and produce any media type output (haha! I've now learned not to use MIME type anymore!) seems too broad when trying to map the default behavior with what the vast majority of clients currently do.

> > I find that use-case plausible, albeit not something useful given the
> > current landscape of a single algorithm.  The alternative would be to
> > have to send PREVIEW algorithms one at a time, non-pipelined, which is
> > precisely the kind of inefficient client/server interaction this
> > extension is trying to minimize.
> 
> We disagree on the plausibility.  Realistically, I can't think of
> *any* other plausible preview algorithms that a *client* might ask for
> without user interaction.  The most I can see is something about how
> to apply FUZZY to non-text media types.

Our UI client does "virtual attachment view", that shows graphical representations of all attachment data within messages within a mailbox, so PREVIEW is perfectly suited for that task.  (We currently have to do the preview conversion via a proprietary plugin.)

I'll agree that it may not be a common use-case, but it is entirely plausible.

> > I do agree that there is language that can surely be cleaned up here,
> > especially regarding error handling.  But I will hold off on rewriting
> > the section until we can have some discussion/consensus as to whether
> > the proposed usage discussed above is something that others believe we
> > should be supporting.
> 
> *If* we're going to keep the multi-algorithm syntax, I think the new
> paragraph you added has it covered, and the paragraph with the phrase
> "priority decision" is now unnecessary.
> 
> But I'd still like to see someone give a good argument for a case
> where having a client ask for multiple algorithms to be applied could
> really be useful and practical.
> 
> > > — Section 3.2 —
> > >
> > >    If the preview is not available, the server MUST return NIL as the
> > >    PREVIEW response.  A NIL response indicates to the client that
> > >    preview information MAY become available in a future PREVIEW FETCH
> > >    request.  Note that this is semantically different than returning a
> > >    zero-length string, which indicates an empty preview.
> > >
> > > I think the MUST here is hard to follow, because the text doesn’t make a clear
> > > enough distinction between “preview is not available” and “an empty preview”.
> > > Can you expand the text a bit to explain the distinction more clearly, as this
> > > is a protocol requirement?
> >
> > "Preview not available" examples =
> >   * Preview generation isn't available at the moment (previews are generated in a separate thread, and that pool is saturated)
> >   * Body text is not available at the moment, so preview can't be generated.
> >   * The preview text has not been generated within the alotted time (e.g. LAZY modifier)
> 
> OK, so you're saying that NIL means that there's some transient issue,
> and that a non-empty preview might be available later... and an empty
> string means that there's no preview text available for this message
> and never will be.  Yes?

Correct.

NIL = I asked for preview, I don't have that preview cached, and the message blob storage is acting up and taking 5 seconds to return data.  As a server, that's beyond my configured limit of requiring a message listing within 2 seconds, so the preview is "not available", but not necessarily empty

Empty String = As a server, I have analyzed the body and there is no data that I find worthy to display as "preview text" to the user, and this is a determinative evaluation.

Those are two entirely different scenarios (and the former scenario is expected, especially in a distributed storage system).  So they absolutely need to be handled separately.

> If that's right (and actually a necessary distinction), I suggest this:
> 
> OLD
>    If the preview is not available, the server MUST return NIL as the
>    PREVIEW response.  A NIL response indicates to the client that
>    preview information MAY become available in a future PREVIEW FETCH
>    request.
> 
>    Examples why a preview may not be available include: the preview
>    generation process is not available due to transient server resource
>    limitations, the message body text is unavailable, or a server-
>    imposed timeout was reached during generation.
> 
>    A NIL response is semantically different than returning a zero-length
>    string, which indicates that no meaningful preview text can be
>    generated for the message.
> 
> NEW
>    It is possible that preview text is not available now, but might be
>    available later -- perhaps the server's preview-generating resources
>    are overloaded, there is a server-imposed timeout during preview
>    generation, or there is some transient issue with fetching the
>    message body.  In such cases, the server will return NIL as the
>    preview response, and the client might possibly try again later.
> 
>    On the other hand, it's possible that the server has determined that
>    no meaningful preview text can be generated for a particular
>    message, and that won't change later.  Examples of this involve
>    encrypted messages, content types the server does not support
>    previews of, and other situations where the server is not able to
>    extract information for a preview.  In such cases, the server will
>    return a zero-length string.  Clients should not send another FETCH
>    for a preview for such messages.
> 
> END
> 
> But let's be sure we really think the distinction is important.

Thank you for these clarification suggestions.  Will adapt them in the next draft.

> > > — Section 8 —
> > >
> > > It seems like a bad idea to have to keep the IMAP Capabilities registry in sync
> > > with the two new registries: as it stands, when you add a new algorithm you
> > > have to add it to the Preview Algorithms registry, and also add a corresponding
> > > entry in the Capabilities registry... and similarly for a modifier, if I have
> > > that right above.
> > >
> > > Why not follow the model of AUTH= and RIGHTS=, and just reserve the PREVIEW=
> > > capability in the registry, allowing it to apply to entries from the two new
> > > registries?  That avoids inconsistencies in registrations if we later add
> > > algorithms or modifiers.
> >
> > See above.  With my proposal to remove priority modifiers, I believe this discussion
> > point becomes moot.
> 
> It does not; it's still an issue with algorithms.  If you register
> "PREVIEW=FUZZY" as a capability string and also register "FUZZY" as an
> algorithm, you're doing double registration and are prone to getting
> inconsistencies.
> 
> But if, instead, you just register "PREVIEW=" as a capability string
> and let that stand for any "PREVIEW=X", where X is a registered
> preview algorithm, you don't get into that sort of trouble.  That's
> what AUTH= and RIGHTS= did.  Is there a reason not to do that?

User error.  Combination of reading previous specs wrong, and mistaken ideas about how this should work.

100% agree with your reasoning.  Will clean this up for the next draft.

> 
> > > ----------------------------------------------------------------------
> > > COMMENT:
> > > ----------------------------------------------------------------------
> > >
> > > — Section 3.2 —
> > >
> > >    This relaxed requirement permits a
> > >    server to offer previews as an option without requiring potentially
> > >    burdensome storage and/or processing requirements to guarantee
> > >    immutability for a use case that does not require this strictness.
> > >
> > > That’s sensible, but can you include some text giving an example of a situation
> > > where the preview might change?  Given that the messages themselves are
> > > immutable, why would applying the same algorithm to the same text give
> > > different results?
> >
> > We discussed this on the list in the past, but one example would be
> > loss of cached preview data on the server and re-generation uses a
> > newer algorithm version which produces slightly different text.
> >
> > The consensus was that we should not be making extraordinary
> > efforts/costs to ensure this text never changes, where this text is not
> > being held out as being the canonical view of the message contents in
> > the first place.
> 
> That's fine.  So, as the first sentence of my comment says, can you
> include some text to explain the situation?  I'm not looking for a
> lot, just a sentence.  (If you think it's best not to, that's OK too:
> this is a comment, not a discuss point).

Fair point.  I will address this suggestion.  Maybe: "For example, the underlying IMAP server may change for an account due to a software upgrade; account state information may be retained in the migration, but the new server may support a different PREVIEW generation algorithm.  Thus, message state may remain the same but PREVIEW FETCH response may change."

Traveling, so give me a day or two and I will incorporate above edits, and discussion points previously raised by Alexey, in the next draft.

michael

---

From: Barry Leiba <[email-masked]>
Date: Sun, 2 Jun 2019 13:33:17 -0400

I think we're still in significant disagreement here; I'd like others
from the working group to comment, and see if that can move us in one
direction or the other.

> Is that what a typical received message looks like?  To me, that looks like the edge case.

It's not the "typical" message, but I don't think it's an edge case
either.  I do often get messages where the only significant content is
in an image.  (I've tried to dissuade those responsible from sending
things that way, but they persist.)  So, no, I wouldn't call it an
"edge case" either.  And even if it were, that's not really the point;
see below.

> Seems that a server could figure out (through experience) when image
> preview data should be calculated or ignored.  Image size, MIME
> structure information, MIME order, etc. could all be used to reach some
> sort of 95% accuracy threshold.  Maybe that doesn't happen day 1 of
> implementation, but a competitive, reactive server implementation will
> work on this.
>
> The same issue exists for text previews, FWIW.  What about a text/plain
> + text/html alternative message, where text/plain is "Please click on
> this URL to view the text part", and the text/html part is empty text
> (it's a single image link).  A server can be programmed to be smart
> enough to ignore that part re: preview.
>
> As written, we allow a server to be given the latitude to figure these
> kind of issues out.  Closing down that opportunity because it might be
> a difficult logical problem isn't the right approach from my view.

I note that in your response you say "server" four times, at least
once per paragraph, and "client" never.  But my point (see above) was,
"But how would the client software know that, and, therefore, know
what to ask for?"  These are things a *client* is requesting; how does
the client know whether to request that the preview be preferentially
done on the image or on the text?  I contend that it doesn't, and
can't, and that, therefore, it's strange to expect it to ask, and
strange to say that the server has to take the client's "priority
decision" into account.

I'm still looking for a real use case where the *client* is the entity
that's better suited to asking for any specific action with respect to
a message preview.

> FUZZY is explicitly defined as text only output though, since that's generally mapping
> what clients already do.

Well, but saying, "That's the way it works now," isn't really the
point.  We need to be looking at how we *want* this to work going
forward, and I think that having the *server* decide what the best way
is to give a preview of a message is the right answer.  And if that
means that the server decides to make a preview from the text/plain
part, the text/html part, the application/pdf part, or the image/jpeg
part (perhaps using OCR on the image), then I think that's fine.
FUZZY is explicitly defined as text only, only because that's how
we've defined it so far.

> Our UI client does "virtual attachment view", that shows graphical representations of all
> attachment data within messages within a mailbox, so PREVIEW is perfectly suited for
> that task.  (We currently have to do the preview conversion via a proprietary plugin.)

I don't understand this: this extension gives one preview for the
message, not separate previews for different message parts.  It looks
like you're talking about showing a graphic representation of the
message structure with each part having a preview.  Can you post a
screen capture of what you're talking about, showing how PREVIEW
helps?

> > really be useful and practical.

This comment of mine still stands.  Please, EXTRA working group, join
the discussion.

Barry

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 2 Jun 2019 21:23:40 +0200

> from the working group to comment, and see if that can move us in one

My two cents, then: Michael is right.

The correct wording, from my point of view, should specify the desired 
results in one or a few typical cases, suggest that implementations should 
do much the same in less common cases. Ideally it also contains an appendix 
might with some common/uncommon test messages _without_ specific results.

My reason for this is that  exact rules along the lines of e.g. 
THREAD=REFERENCES haven't been fortunate, while RFCs like FUZZY with more 
goal and less mechanism haven't led to complaints or calls for revision.

Some servers have the knowledge to thread better (e.g. because they had 
language-specific ways to compute BASESUBJECT, or could thread based on 
information from Exch^Wnon-822 mail systems) but T=R banned that. On the 
other hand, FUZZY has let implementers use their platform's abilities, and 
I haven't noticed interop problems caused by that.

Michael mentioned all-image HTML as an example. One of the platforms I know 
comes with OCR support, the others don't AFAICT. Now, this platform isn't 
typically a server platform... but I don't see a weighty reason to forbid 
using OCR for PREVIEW if the software has that available.

Arnt

---

From: "Neil Jenkins" <[email-masked]>
Date: Mon, 03 Jun 2019 10:27:19 +1000

I must admit to having skimmed this discussion, but it seems to me that this extension is getting way over-complicated. The key here is simply to answer the question: what is this for? The only purpose of this is to provide a way for clients to get a snippet of (plain) text to shown in the mailbox listing without downloading whole body parts and expecting a fast response from the server (like fetching common headers). This is a common feature and worth making more efficient. Trying to do anything else is making things more complicated for close to zero benefit.

I would contend:
 * No server is going to implement more than one algorithm. Like with fuzzy searching, we should not specify the algorithm in the spec. Servers should be free to implement the best solution they can, which is likely to change over time as new edge cases are found and new capabilities are available (for example, skipping salutations, OCRing an image-only body, skipping broken auto-generated plain text components and generating the preview from the text/html version instead, etc.). The server should be free to choose which parts it thinks it should use to generate the preview.
 * The client either accepts the server is going to give it a good preview or it will ignore the whole capability and generate its own, at the expense of being slower and more data hungry.
 * The output is always just plain text. This is what is shown in mailbox listings as the preview in Apple Mail, Gmail, FastMail, etc. If you think an image or text/html preview is useful, I disagree, but regardless that should be a different spec rather than overcomplicating one that solves an actual use case today. (To be clear, I think the spec already says this, but some of the comments in this thread seemed to indicate to me people thought you might want different typed outputs in the future).
 * (Perhaps controversial but…) If the server offers the capability it should always have the data readily available, so the LAZY modifier is not necessary. I would expect servers that implement this to generally calculate and cache a preview on delivery. If it's going to take a while, the benefit to the client is much less and it might as well just calculate its own. That way the client only has two code paths (server-preview or client-preview) rather than three (server delayed preview as well).

In the JMAP Mail spec, we simply specify the following for the equivalent property:

*preview*: `String`
*A plain text fragment of the message body. This is intended to be shown as a preview line on a mailbox listing, and may be truncated when shown. The server may choose which part of the message to include in the preview; skipping quoted sections and salutations and collapsing white-space can result in a more useful preview.**
*
*
*
*This MUST NOT be more than 256 characters in length.*

Neil.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Jun 2019 15:50:13 +0100

Ping... to both Michael and the EXTRA working group as a whole: Can we
keep this discussion moving and get this document resolved, please?

Thanks,
Barry