Subject: Re: [Extra] Barry Leiba's Discuss on draft-ietf-extra-imap-fetch-preview-03: (with DISCUSS and COMMENT)

From: Michael Slusarz <[email-masked]>
Date: Mon, 1 Jul 2019 01:29:11 -0600 (MDT)

Apologies on taking too long to respond.

I neglected to update the draft with the other changes discussed in previous messages; I will upload that now.

See below for additional responses.

> On June 2, 2019 11:33 AM Barry Leiba <[email-masked]> wrote:
> 
>  
> I think we're still in significant disagreement here; I'd like others
> from the working group to comment, and see if that can move us in one
> direction or the other.
> 
> > > > I'll start with providing a (future) real-world example, and why this
> > > > behavior exists in its present form.  Given a hypothetical future
> > > > algorithm "IMAGE" (generate image/jpeg preview if image data exists in
> > > > message), I issue this preview command:
> > > >
> > > > a FETCH 1 (PREVIEW (LAZY=IMAGE FUZZY))
> > > >
> > > > This is asking the server: provide me an image preview of the message,
> > > > but ONLY if 1) the message actually contains image information and 2)
> > > > that image information is already generated or can be generated very
> > > > quickly.  Otherwise, fall back to FUZZY generation.
> > >
> > > Hm.  That really seems like a contrived example.  How would the client
> > > software possibly know whether there's useful information in the
> > > image, or whether the information in the accompanying text is more or
> > > less important?  Suppose I have two messages:
> > >
> > > 1. text/plain that basically says nothing; image/jpeg that has a
> > > diagram that explains everything
> > >
> > > 2. text/plain that has the entire real content of the message;
> > > image/jpeg of the senders company logo
> > >
> > > It's pretty clear from that explanation that for message 1 I want some
> > > sort of description of the image and for message 2 I want a preview of
> > > the text.  But how would the client software know that, and,
> > > therefore, know what to ask for?
> >
> > Is that what a typical received message looks like?  To me, that looks like the edge case.
> 
> It's not the "typical" message, but I don't think it's an edge case
> either.  I do often get messages where the only significant content is
> in an image.  (I've tried to dissuade those responsible from sending
> things that way, but they persist.)  So, no, I wouldn't call it an
> "edge case" either.  And even if it were, that's not really the point;
> see below.

Agreed that this is not really the point of this discussion.  We aren't discussing how an image preview algorithm would work, since in practice it would likely need to be designed to be smart enough to not return image data for things like embedded company logos. (This is approaching "what is an attachment?" territory.)

> > Seems that a server could figure out (through experience) when image
> > preview data should be calculated or ignored.  Image size, MIME
> > structure information, MIME order, etc. could all be used to reach some
> > sort of 95% accuracy threshold.  Maybe that doesn't happen day 1 of
> > implementation, but a competitive, reactive server implementation will
> > work on this.
> >
> > The same issue exists for text previews, FWIW.  What about a text/plain
> > + text/html alternative message, where text/plain is "Please click on
> > this URL to view the text part", and the text/html part is empty text
> > (it's a single image link).  A server can be programmed to be smart
> > enough to ignore that part re: preview.
> >
> > As written, we allow a server to be given the latitude to figure these
> > kind of issues out.  Closing down that opportunity because it might be
> > a difficult logical problem isn't the right approach from my view.
> 
> I note that in your response you say "server" four times, at least
> once per paragraph, and "client" never.  But my point (see above) was,
> "But how would the client software know that, and, therefore, know
> what to ask for?"  These are things a *client* is requesting; how does
> the client know whether to request that the preview be preferentially
> done on the image or on the text?  I contend that it doesn't, and
> can't, and that, therefore, it's strange to expect it to ask, and
> strange to say that the server has to take the client's "priority
> decision" into account.

Client: "Please provide me the richest content you can - richest content is defined via the priority list I provided."

I don't see how a client providing a priority order of preview algorithms is any different than what the client would do if it had to download the message and parse it for preview data?

Also, how this is any different than issuing two consecutive FETCH PREVIEW commands?  If the first PREVIEW command returns the empty string, a client is just going to send another FETCH command using the next PREVIEW algorithm in the list.  Thus, the client is indicating its "priority decision" by the order it is sending its commands.  Why not do that in a single command?

> I'm still looking for a real use case where the *client* is the entity
> that's better suited to asking for any specific action with respect to
> a message preview.

See below.

> > > Apart from that, if we're going to do any sort of text-based preview
> > > of an image, I would think we'd just say that when FUZZY is applied to
> > > an image, that's what it does.
> >
> > FUZZY is explicitly defined as text only output though, since that's generally mapping
> > what clients already do.
> 
> Well, but saying, "That's the way it works now," isn't really the
> point.  We need to be looking at how we *want* this to work going
> forward, and I think that having the *server* decide what the best way
> is to give a preview of a message is the right answer.  And if that
> means that the server decides to make a preview from the text/plain
> part, the text/html part, the application/pdf part, or the image/jpeg
> part (perhaps using OCR on the image), then I think that's fine.
> FUZZY is explicitly defined as text only, only because that's how
> we've defined it so far.

How would a client handle this?  Does it need to provide a list of all media types it supports to the server, ala the HTTP Accept header?  That sounds overly complicated for a solution where text/plain is the primary use-case (see, e.g., JMAP preview).

More generally, I can provide this (real-world) example as my final argument as to the behavior at issue in this message.

Client:
  - images, videos, and URLs are commonly sent in messages
  - client wants to show small image preview if image is only content of message
  - client wants to show small image preview (or maybe a short GIF) if video is only content of message
  - client wants to show small image preview of website if a URL is only content of the message
  - server supports 3 PREVIEW extensions: IMG, VID, URL

The client, when listing (syncing) messages in the mailbox, will issue a FETCH PREVIEW for all of (IMG, VID, URL). These PREVIEW extensions describe mutually exclusive message data, so the client doesn't care which one is returned - it is identifying to the client what it supports display-wise.  For this client, it might make the decision whether to FETCH BODY text based on whether preview data is returned for a message.

This scenario is describing the UI behavior of every chat-like client out there: whether it be SMS/MMS, WhatsApp, iMessage, etc.  So it is extremely common, and generally reflects the experience a user expects when using messaging.  So a preview specification should theoretically be extensible to handle this kind of use-case.

To me it makes the most sense to allow this in a single FETCH call - one of the stated goals of the FETCH PREVIEW spec is to reduce roundtrips.  A "priority list" of algorithms is nothing more than a potential list of sequential FETCH calls the client might make.

michael

---

From: Michael Slusarz <[email-masked]>
Date: Mon, 1 Jul 2019 01:29:35 -0600 (MDT)

Neil,

Thanks for your comments.  Responses below.

On June 2, 2019 6:27 PM Neil Jenkins <[email-masked]> wrote:

I must admit to having skimmed this discussion, but it seems to me that this extension is getting way over-complicated. The key here is simply to answer the question: what is this for? The only purpose of this is to provide a way for clients to get a snippet of (plain) text to shown in the mailbox listing without downloading whole body parts and expecting a fast response from the server (like fetching common headers). This is a common feature and worth making more efficient. Trying to do anything else is making things more complicated for close to zero benefit.

I agree that a plaintext preview, to mimic what exists in current client UIs, was a main driving factor for creating this draft.  I disagree that this is the only potential use-case.

I would contend:

No server is going to implement more than one algorithm. Like with fuzzy searching, we should not specify the algorithm in the spec. Servers should be free to implement the best solution they can, which is likely to change over time as new edge cases are found and new capabilities are available (for example, skipping salutations, OCRing an image-only body, skipping broken auto-generated plain text components and generating the preview from the text/html version instead, etc.). The server should be free to choose which parts it thinks it should use to generate the preview.

We are in agreement here.  This is FUZZY.  We have defined some output requirements and limitations, but that output is up to the server.

The client either accepts the server is going to give it a good preview or it will ignore the whole capability and generate its own, at the expense of being slower and more data hungry.

The output is always just plain text. This is what is shown in mailbox listings as the preview in Apple Mail, Gmail, FastMail, etc. If you think an image or text/html preview is useful, I disagree, but regardless that should be a different spec rather than overcomplicating one that solves an actual use case today. (To be clear, I think the spec already says this, but some of the comments in this thread seemed to indicate to me people thought you might want different typed outputs in the future).

I disagree with this logic, as it is essentially "let's standardize what already exists in clients."  I'd rather design a standard that fits the current use cases AND can be extended to future use cases.

A real world example - we are currently working on an initiative to push email as a chat medium (Shameless plug:
https://www.coi-dev.org/).  In every chat client I have ever used, sending images and/or video is an important part of conversations.  When an image/video appears in a chat stream, the full contents are not generally displayed - it is a small preview of the image (or a small preview of a frame of the video).  That small image that is surely a "preview" of that message.  Mixed media previews are a very common paradigm in other messaging systems - why should email be any different?

We shouldn't have to come back in a year and propose a new "FETCH IMAGEPREVIEW" standard to support this - we should instead be able to propose a simple, optional extension to the base PREVIEW behavior.

(Perhaps controversial but…) If the server offers the capability it should always have the data readily available, so the LAZY modifier is not necessary. I would expect servers that implement this to generally calculate and cache a preview on delivery. If it's going to take a while, the benefit to the client is much less and it might as well just calculate its own. That way the client only has two code paths (server-preview or client-preview) rather than three (server delayed preview as well).

We index on delivery, but there's plenty of reasons why preview may not be available at a later time.  More important, a preview is not a fundamental part of the message data - it is data created via transformation of that data.  The original message cannot be recreated, but previews can - a server operator should be allowed to choose different storage and/retention policies based on that fact.

A client doesn't have to use LAZY.  A server can quickly support LAZY by returning NIL for every message requested - this doesn't add any appreciable overhead to the client/server interaction, as the client would need to issue a separate FETCH PREVIEW call regardless of whether LAZY was supported or not.  If LAZY is not something a client/server operator wants to deal with the complexities of, they can be avoided at almost zero development cost.

In the JMAP Mail spec, we simply specify the following for the equivalent property:

preview
:
String

A plain text fragment of the message body. This is intended to be shown as a preview line on a mailbox listing, and may be truncated when shown. The server may choose which part of the message to include in the preview; skipping quoted sections and salutations and collapsing white-space can result in a more useful preview.

This MUST NOT be more than 256 characters in length.

PREVIEW=FUZZY and JMAP preview can be serviced by the same data (Bron suggested changes to the preview spec in the past to ensure this), so I think we are good here.

michael

---

From: "Neil Jenkins" <[email-masked]>
Date: Tue, 02 Jul 2019 14:53:25 +1000

On Mon, 1 Jul 2019, at 17:29, Michael Slusarz wrote:
> A real world example - we are currently working on an initiative to push email as a chat medium (Shameless plug: https://www.coi-dev.org/). In every chat client I have ever used, sending images and/or video is an important part of conversations. When an image/video appears in a chat stream, the full contents are not generally displayed - it is a small preview of the image (or a small preview of a frame of the video). That small image that is surely a "preview" of that message. Mixed media previews are a very common paradigm in other messaging systems - why should email be any different? 
> 

I see, thanks for providing your motivation for this, it makes it much easier to discuss. This is certainly *a* solution to this problem, but I do wonder if it's the best one. If we start with the requirement that we must use IMAP for the chat message sync protocol, which admittedly does limit the options somewhat, the other main thing I would try is fetching the body structure with the COI client when getting a message listing. You would be able to see that, for example, a message has only an image/png part and how large it is. If it's small, the client may just fetch it directly. If it's large, you want a way for the client to ask the server for this image downsampled to a particular size; the client can specify exactly what it needs to optimise bandwidth usage.

Neil.