Subject: [imapext] IMAP SNIPPET extension (initial draft)

From: Michael M Slusarz <[email-masked]>
Date: Sun, 03 May 2015 23:56:25 -0600

A little late, but here's my efforts on an initial draft regarding  
SNIPPET so that it can be added to the WG.

Based on the conversation that occurred on this list a few months  
back, I made these editorial decisions:

   - Extend FETCH: several people said this would not be implemented  
if it used ANNOTATE/CONVERT
   - Allow multiple different algorithms

I think the main point of discussion going forward is whether to  
explicitly define an algorithm for snippet generation (i.e. threading)  
or to allow the default algorithm to be entirely server-defined (i.e.  
FUZZY searching).  For now, I went with the latter, and even named it  
FUZZY.

Another issue regards the size of snippet return.  In this draft, this  
is not client configurable.  For the FUZZY algorithm, I used 150  
octets as SHOULD and MUST NOT ever exceed 300 octets.  (Snippet data  
must be returned as UTF-8).

But hopefully this is a decent enough starting point where it can be  
used to continue the discussion going forward.

michael

----------------

Internet Engineering Task Force                          M. Slusarz, Ed.
Internet-Draft                                                   Dovecot
Intended status: Standards Track                                May 2015
Expires: October 31, 2015

                   IMAP4 Extension: Snippet Generation
                          draft-imap-snippet-00

Abstract

    This document specifies an IMAP protocol extension which allows a
    client to request that a server provide an abbreviated representation
    of a message (a snippet of text) that can be used by a client to
    provide a useful contextual preview of the message contents.

Status of this Memo

    This Internet-Draft is submitted in full conformance with the
    provisions of BCP 78 and BCP 79.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF).  Note that other groups may also distribute
    working documents as Internet-Drafts.  The list of current Internet-
    Drafts is at http://datatracker.ietf.org/drafts/current/.

    Internet-Drafts are draft documents valid for a maximum of six months
    and may be updated, replaced, or obsoleted by other documents at any
    time.  It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than as "work in progress."

    This Internet-Draft will expire on October 31, 2015.

Copyright Notice

    Copyright (c) 2015 IETF Trust and the persons identified as the
    document authors.  All rights reserved.

    This document is subject to BCP 78 and the IETF Trust's Legal
    Provisions Relating to IETF Documents (http://trustee.ietf.org/
    license-info) in effect on the date of publication of this document.
    Please review these documents carefully, as they describe your rights
    and restrictions with respect to this document.  Code Components
    extracted from this document must include Simplified BSD License text
    as described in Section 4.e of the Trust Legal Provisions and are
    provided without warranty as described in the Simplified BSD License.

Table of Contents

    1.  Introduction
    2.  Conventions Used In This Document
    3.  FETCH Data Item
      3.1.  Command
      3.2.  Response
    4.  SNIPPET Algorithms
      4.1.  FUZZY
    5.  Examples
    6.  Formal Syntax
    7.  TODO
    8.  Acknowledgements
    9.  IANA Considerations
    10. Security Considerations
    11. References
      11.1.  Normative References
      11.2.  Informative References
    Author's Address

1.  Introduction

    Many modern mail clients display small extracts of the body text as
    an aid to allow a user to quickly decide whether they are interested
    in viewing the full message contents.  Mail clients implementing the
    Internet Message Access Protocol (IMAP; RFC 3501 [RFC3501]) would
    benefit from a standardized, consistent way to generate these brief
    previews of messages (a "snippet").

    Generation of snippets on the server has several additional benefits.
    First, it allows caching of these snippets for use with both multiple
    mail clients and with clients that don't support client-side caching.

    Second, generation on the server is more efficient.  A client-based
    algorithm needs to issue, at a minimum, a FETCH BODYSTRUCTURE command
    in order to determine which MIME [RFC2045] body part should be
    displayed.

    Finally, server generation allows caching in a centralized location.
    Mail accounts are often accessed by multiple different clients, and
    many of these clients lack support for client-side caching.  Using
    server generated snippets allows snippets to be generated once and
    then cached indefinitely.

    A server that supports the SNIPPET extension indicates this with one
    or more capability names consisting of "SNIPPET=" followed by a
    supported snippet algorithm name as described in this document.  This
    provides for future upwards-compatible extensions and/or the ability
    to use locally-defined snippet algorithms.

2.  Conventions Used In This Document

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
    document are to be interpreted as described in RFC 2119 [RFC2119].

    "User" is used to refer to a human user, whereas "client" refers to
    the software being run by the user.

    In examples, "C:" and "S:" indicate lines sent by the client and
    server respectively.  If a single "C:" or "S:" label applies to
    multiple lines, then the line breaks between those lines are for
    editorial clarity only and are not part of the actual protocol
    exchange.

3.  FETCH Data Item

3.1.  Command

    To retrieve a snippet for a message, the "SNIPPET" FETCH attribute is
    used when issuing a FETCH command.

    If no algorithm identifier is provided, the server decides which of
    its built-in algorithms to use to generate the snippet text.

    Alternately, the client may explicitly indicate which algorithm
    should be used in a parenthesized list after the SNIPPET attribute
    containing the name of the algorithm.  This algorithm MUST be one of
    the algorithms identified as supported in the SNIPPET capability
    responses.  If a client requests an algorithm that is unsupported,
    the server MUST return a tagged BAD response.  The server SHOULD use
    this algorithm to generate the snippet.

    A client SHOULD NOT issue more than one SNIPPET attribute per FETCH
    command.  If more than one SNIPPET attribute is present in a FETCH
    command, the server SHOULD use the last attribute seen in the
    command.

3.2.  Response

    The server returns a variable-length string that is the generated
    snippet for that message.  This string MUST NOT be content transfer
    encoded and MUST be encoded in UTF-8.  The snippet text MUST be
    treated as text/plain MIME data by the client.

4.  SNIPPET Algorithms

4.1.  FUZZY

    The FUZZY algorithm directs the server to use any internal algorithm
    it desires, subject to the below limitations, to generate the snippet
    for a message.

    The server SHOULD limit the length of the snippet text to 150 octets.
    The server MUST NOT output snippet text longer than 250 octets.

    The server SHOULD remove any formatting markup that exists in the
    original text.

    The FUZZY algorithm MUST be implemented by any server that supports
    the SNIPPET extension.

5.  Examples

    Example 1: Requesting FETCH without explicit algorithm selection

         C: A1 CAPABILITY
         S: * CAPABILITY IMAP4rev1 SNIPPET=FUZZY
         S: A1 OK Capability command completed.
         C: A2 FETCH 1 (RFC822.SIZE SNIPPET)
         S: * 1 FETCH (RFC822.SIZE 20000 SNIPPET {61}
         S: This is the first line of text from the first text part.
         S: )
         S: A2 OK FETCH complete.

    Example 2: Requesting FETCH with explicit algorithm selection

           C: A1 CAPABILITY
           S: * CAPABILITY IMAP4rev1 SNIPPET=FUZZY
           S: A1 OK Capability command completed.
           C: A2 FETCH 1 (RFC822.SIZE SNIPPET (FUZZY))
           S: * 1 FETCH (RFC822.SIZE 20000 SNIPPET {61}
           S: This is the first line of text from the first text part.
           S: )
           S: A2 OK FETCH complete.

    Example 3: Requesting FETCH with invalid explicit algorithm selection

           C: A1 CAPABILITY
           S: * CAPABILITY IMAP4rev1 SNIPPET=FUZZY
           S: A1 OK Capability command completed.
           C: A2 FETCH 1 (RFC822.SIZE SNIPPET (X-SNIPPET-ALGO))
           S: A2 BAD FETCH contains invalid snippet algorithm name.

6.  Formal Syntax

    The following syntax specification uses the augmented Backus-Naur
    Form (BNF) as described in ABNF [RFC5234].  It includes definitions
    from IMAP [RFC3501].

        capability      =/ "SNIPPET=FUZZY"

        fetch-att       =/ "SNIPPET" *(SP "(" snippet-alg ")")

        msg-att-static  =/ "SNIPPET" SP nstring

        snippet-alg     =  "FUZZY" / snippet-alg-ext

        snippet-alg-ext =  atom
                           ; New algorithms MUST be registered with IANA

7.  TODO

    1.  More explicit algorithm for text/plain processing?

    2.  Interaction with CONDSTORE (MODSEQs)?

    3.  Allow algorithms to return non-text/plain data?

8.  Acknowledgements

    TODO

9.  IANA Considerations

    IMAP4 [RFC3501] capabilities are registered by publishing a standards
    track or IESG-approved experimental RFC.  The registry is currently
    located at:

       http://www.iana.org/assignments/imap-capabilities

    This document requests that IANA adds the "SNIPPET" capability to the
    IMAP4 [RFC3501] capabilities registry.

    This document also requests that IANA adds a new IMAP4 [RFC3501]
    snippet algorithms registry, which registers snippet algorithms by
    publishing a standards track or IESG-approved experimental RFC.  This
    document constitutes registration of the FUZZY algorithm in that
    registry.

10.  Security Considerations

    TODO; See RFC 3552

11.  References

11.1.  Normative References

    [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
               Requirement Levels", BCP 14, RFC 2119, March 1997.

    [RFC3501]  Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION
               4rev1", RFC 3501, March 2003.

    [RFC5234]  Crocker, D. and P. Overell, "Augmented BNF for Syntax
               Specifications: ABNF", STD 68, RFC 5234, January 2008.

11.2.  Informative References

    [RFC2045]  Freed, N. and N.S. Borenstein, "Multipurpose Internet Mail
               Extensions (MIME) Part One: Format of Internet Message
               Bodies", RFC 2045, November 1996.

Author's Address

    Michael Slusarz, editor
    Dovecot
    Denver, Colorado
    US

    Email: [email-masked]

___________________________________
Michael Slusarz [[email-masked]]

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 04 May 2015 10:18:25 -0400

Hi Michael,

--On May 3, 2015 at 11:56:25 PM -0600 Michael M Slusarz <[email-masked]> 
wrote:

> But hopefully this is a decent enough starting point where it can be used
> to continue the discussion going forward.

The spec should state somewhere that if the server is unable to extract a 
suitable snippet (e.g., message is encrypted, contains only an image), then 
it returns NIL in the FETCH SNIPPET response.

-- 
Cyrus Daboo

---

From: Stu Brandt <[email-masked]>
Date: Mon, 04 May 2015 12:09:54 -0400

Thanks, Michael, for putting this together.

On 5/4/15 1:56 AM, Michael M Slusarz wrote:
[snip]

>     2.  Interaction with CONDSTORE (MODSEQs)?

Is this where details on the question of the snippet being 
mutable/immutable will be or does the mention of CONDSTORE here imply 
that it's assumed to be mutable?

Assuming the latter, it would be good to cover the cases that demand a 
mutable snippet + offer recommendations for clients that don't use 
CONDSTORE/MODSEQ regarding snippet fetch frequency. I'd sure hate to see 
all the "FETCH 1:* (UID FLAGS)" polling turn into "FETCH 1:* (UID FLAGS 
SNIPPET)"

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 06 May 2015 13:11:06 -0600

Quoting Cyrus Daboo <[email-masked]>:

> --On May 3, 2015 at 11:56:25 PM -0600 Michael M Slusarz  
> <[email-masked]> wrote:
>
> The spec should state somewhere that if the server is unable to  
> extract a suitable snippet (e.g., message is encrypted, contains  
> only an image), then it returns NIL in the FETCH SNIPPET response.

Cyrus,

Thanks for the pointer.  I probably reminded myself five times to put  
explicit NIL return language in the server response part and,  
naturally, never did so.  I've added this language to the next draft.

As for your examples as when a server cannot extract a suitable  
snippet: those are actually great use cases as to why an extensible  
SNIPPET extension is useful.  As someone had mentioned in the initial  
conversation on the topic (apologies that I don't remember who brought  
the topic up), it may be useful to have a server generate a snippet like

   - "This message contains 2 images"
   - "This message contains 1 image (X KB)"
   - "This message contains encrypted content"
   - "[This message contains 2 attachments] <...Snippet text of  
initial body part...>"

rather than display actual textual content of any single message body.

In the interest of keeping the base specification as minimal as  
possible, I stuck with a single fuzzy algorithm.  In light of the  
above, maybe it makes sense to define an additional base algorithm to  
indicate to the server that it MAY try to create a dynamic snippet  
based on the entire content/structure of the message.

In other words, define both a FUZZYTEXT and a FUZZY algorithm where  
the former requests the server to limit snippet responses to message  
body text only while the latter allows the server to generate dynamic  
descriptions.  I would think that FUZZYTEXT would have to be the only  
MUST implement, as FUZZY starts becoming more involved when it comes  
to implementation details as it will require interaction with the  
LANGUAGE extension in order to allow for translatable snippets.

michael

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 06 May 2015 15:28:00 -0400

Hi Michael,

--On May 6, 2015 at 1:11:06 PM -0600 Michael M Slusarz 
<[email-masked]> wrote:

> As for your examples as when a server cannot extract a suitable snippet:
> those are actually great use cases as to why an extensible SNIPPET
> extension is useful.  As someone had mentioned in the initial
>    - "[This message contains 2 attachments] <...Snippet text of initial
> body part...>"
>

Well that gets tricky because really those strings would need to be 
localized. Perhaps the LANGUAGE extension (RFC5255) could be leveraged for 
that.

-- 
Cyrus Daboo

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 06 May 2015 13:31:55 -0600

This message never seemed to make it to my mailbox, so doing a  
copy/paste job to respond...  This is a response to Stu's message from  
several days ago:

----------

     7.  TODO

Assuming the latter, it would be good to cover the cases that demand a  
mutable snippet + offer recommendations for clients that don't use  
CONDSTORE/MODSEQ regarding snippet fetch frequency. I'd sure hate to  
see all the "FETCH 1:* (UID FLAGS)" polling turn into "FETCH 1:* (UID  
FLAGS SNIPPET)"

----------

This was sort of my way to punt this issue for future discussion about  
mutable/immutable.

My personal view (and what i almost wrote) is that a client SHOULD NOT  
care whether a SNIPPET is mutable or not.  Since the whole point of  
SNIPPET is to provide some sort of *quick* means of generating a  
preview, it should not matter as to the exact content of the snippet.   
Once a client has a snippet it should use that text going forward as  
long as it retains that data locally.  Trying to sync snippet data  
destroys the main benefit of using SNIPPET in the first place: a cheap  
way of generating a preview.

Additionally, given the nature of the FUZZY algorithm, inherently  
there is no way of enforcing that a server use the same algorithm to  
generate identical output text, at least for any given algorithm  
selection.

I would argue that a server SHOULD try to honor a given algorithm  
request, but I don't think it should be a requirement to rebuild the  
SNIPPET.  I would imagine many caching/indexing servers would  
automatically generate snippet data at the time it first parses the  
message (i.e. on initial delivery), so that by the time a user  
requests a FETCH on the message the data is already generated.

My focus on the SNIPPET draft was to ensure that it is as fast as  
possible.  If a client is looking for specific or reproducible  
behavior when generating a preview, it can obtain that the way it  
always has been able to: by doing the preview generation locally.   
Alternative: we have to implement at least one determinative algorithm  
that ensures that the snippet data is identical across any two  
different server implementations.

In short, I personally think that SNIPPET has no interaction with  
CONDSTORE/MODSEQs.  But would be glad to open the conversation and  
hear from all those who disagree.

michael

---

From: Michael M Slusarz <[email-masked]>
Date: Wed, 06 May 2015 13:35:43 -0600

Quoting Cyrus Daboo <[email-masked]>:

> localized. Perhaps the LANGUAGE extension (RFC5255) could be  
> leveraged for that.

Yup.  That's what I mentioned it later in that message :)

If that algorithm was to be implemented, I would think that it almost  
becomes a MUST requirement that dynamically generated text only be  
created if the client has explicitly set a language via the LANGUAGE  
command.  Otherwise the only sane response is to use the actual body  
content.

michael

---

From: Dave Cridland <[email-masked]>
Date: Wed, 6 May 2015 21:38:41 +0100

On 6 May 2015 20:35, "Michael M Slusarz" <[email-masked]> wrote:
>> --On May 6, 2015 at 1:11:06 PM -0600 Michael M Slusarz <
> If that algorithm was to be implemented, I would think that it almost
becomes a MUST requirement that dynamically generated text only be created
if the client has explicitly set a language via the LANGUAGE command.
Otherwise the only sane response is to use the actual body content.

A server could use preferences set externally. Doesn't seem like must
territory to me.

> michael
>
>
> _______________________________________________
> imapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/imapext

---

From: Chris Newman <[email-masked]>
Date: Mon, 11 May 2015 18:36:19 -0700

--On May 3, 2015 23:56:25 -0600 Michael M Slusarz <[email-masked]> wrote:
> A little late, but here's my efforts on an initial draft regarding SNIPPET so
> that it can be added to the WG.
> 
> Based on the conversation that occurred on this list a few months back, I
> made these editorial decisions:
> 
>    - Extend FETCH: several people said this would not be implemented if it
> used ANNOTATE/CONVERT
> I think the main point of discussion going forward is whether to explicitly
> define an algorithm for snippet generation (i.e. threading) or to allow the
> default algorithm to be entirely server-defined (i.e. FUZZY searching).  For
> now, I went with the latter, and even named it FUZZY.
> 
> Another issue regards the size of snippet return.  In this draft, this is not
> client configurable.  For the FUZZY algorithm, I used 150 octets as SHOULD
> and MUST NOT ever exceed 300 octets.  (Snippet data must be returned as
> UTF-8).
> 
> But hopefully this is a decent enough starting point where it can be used to
> continue the discussion going forward.

Good starting point. Thanks!

I strongly prefer an expert review process to standards track for new SNIPPET
algorithms. The expert review should simply make sure the algorithm is
deterministic if published and make sure private vendor algorithms have a
version number (e.g., VENDORv1, VENDORv2, ...). No reason to require standards
track processing, IMHO.

I also suggest a simpler syntax. Have the client use

>            C: A2 FETCH 1 (RFC822.SIZE SNIPPET=FUZZY)

or

and the server always include an algorithm name in the response such as:

>            S: * 1 FETCH (RFC822.SIZE 20000 SNIPPET=FUZZY {61}

or

>            S: * 1 FETCH (RFC822.SIZE 20000 SNIPPET=BLUDYBLOOP {61}

Don't use "X-" in examples per RFC 6648. If the client does not include an
algorithm name, then the server picks the algorithm (so it will provide the
deterministic name if there is one or use FUZZY). This is simpler syntax, and
it allows the client to determine the server's preferred deterministic
algorithm if there is one (which can be better for an offline client).

		- Chris