Subject: Re: [Extra] I-D Action: draft-ietf-extra-sieve-special-use-01.txt

From: Bron Gondwana <[email-masked]>
Date: Wed, 18 Jul 2018 07:39:16 +1000

Hi all, sorry about dropping the ball on following up on this.  I'm just
putting together the slides for Thursday now.
On Thu, Jun 14, 2018, at 01:40, Ned Freed wrote:
>>> This, incidentially, is one of those places where providing the
>>> actual IMAP>>> commands will be especially valuable: Pointing out that you should
>>> use the>>> NAMESPACE extension and if it isn't present, fallback to LIST ""
>>> would be very>>> helpful.
> 
>> Here's my initial attempt at addressing this point (separate
>> branch in>> Github):
> 
>> https://github.com/ietfextra/draft-ietf-extra-sieve-special-use/commit/035a1af6137a383089c8d54cd733238dc2dfebda> 
>> This adds an IMAP example for specialuse_exists.
> 
>> What do you think?
> 
> The first step seems fine:
> 
>   C: A01 NAMESPACE
>   S: * NAMESPACE (("INBOX/" "/")("Archive/" "/")) NIL (("Public/"
>      "/"))>   S: A01 OK NAMESPACE command completed
> 
> In the next step:
> 
>     Subsequently, when no particular mailbox is of interest (i.e., the>     "specialuse_exists" test has no mailbox argument), the client
>     lists>     all mailboxes with special-use flags in the two returned personal>     namespaces:
> 
>     C: A02 LIST (SPECIAL-USE) "" ("INBOX/*" "Archive/*") RETURN (SPECIAL-
>        USE)>     S: * LIST (\Drafts) "/" INBOX/Drafts
>     S: * LIST (\Trash) "/" INBOX/Trash
>     S: * LIST (\Sent) "/" INBOX/Sent
>     S: * LIST (\Archive) "/" Archive/Default
> 
> It should be noted that this depends on extended list being available.> 
> The final step is:
> 
>     C: A03 SELECT Archive/Default
>     S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
>     S: * OK [PERMANENTFLAGS (\Answered \Deleted \Seen \Draft \*)]
>        Flags permitted.>     S: * 11211 EXISTS
>     S: * 3 RECENT
>     S: * OK [UIDVALIDITY 1526903163] UIDs valid
>     S: * OK [UIDNEXT 11278] Predicted next UID
>     S: A03 OK [READ-WRITE] SELECT command completed
> 
> There are several issues here.
> 
> First, if you're going to use SELECT it's probably worth noting
> that since> inclusion of [READ-WRITE] for writable mailboxes is a SHOULD in RFC
> 3501 while> inclusion of [READ-ONLY] for non-writable is a MUST, a writability
> check really> needs to based on the absence of [READ-ONLY]. (RFC 5490 3.1.a appears
> incorrect> to me in this regard - should I file an errata?)
> 
> However, I think use of SELECT for this purpose is problematic: It's
> potentially expensive and worse, may cause undesireable state
> changes. Why> not use the MYRIGHTS command to obtain the access rights (and check
> for "p" or> "i") without having to select the mailbox?
> 
> And as a bonus, if it's available you could use the just-standardized> LIST-MYRIGHTS extension to piggyback getting the rights while getting
> the list> of special-use mailboxes.
> 
> More generally, this is why documenting the things necessary to
> implement the> necessary semantics of an extension is helpful - there may be
> optimizations and> concerns that should be called out lest implementors miss them.
> 
> It also helps clarify overall implementation difficulty. And
> while this> extension may be easy to implement in the Sieve-in-IMAP case, from the> perspective of an MTA/MDA implementation, even if all of the steps
> can be done> efficiently, there's still the question of IMAP server
> availability and the> handling of temporary errors.
> 
> All of which is roundabout way of saying I now think that we
> may want a> capability for specialuse_exists that's separate from
> "special-use"/:specialuse. Or better still, since we already have the
> "mailbox"> capability for "mailbox access stuff", I think we should require the
> presence> of both "mailbox" and "special-use" capabilities in order to use
> specialuse_exists.

Stephan, are you happy to revise based on Ned's feedback above?

> I also reviewed the rest of the revised draft, and it's looking
> good. My one> remaining concern is with the handling of multiple mailboxes
> having the> same special-use flag assigned. The draft currently says:
> 
>   More than one mailbox in the user's personal namespace can have a
>   particular special-use flag assigned.  In case of such
>   ambiguity, the>   mailbox that is chosen for delivery is implementation-defined.
>   However, while the set of mailboxes to which the involved special-
>   use>   flags are assigned remains unchanged, implementations MUST ensure
>   that the mailbox choice is made consistently, so that the same
>   mailbox is used every time.  Conversely, the chosen mailbox MAY
>   change once the special-use flag assignments that are relevant for
>   the mailbox choice are changed (usually by user interaction).
> 
> I don't see a way I can reasonably implement this MUST via IMAP
> unless the> IMAP server always returns the list of special-use mailboxes in the
> same order.> Sure, I can cache the user/special-use-attribute/actual-mailbox-
> list tuple> somewhere, but caches can always be lost. And mailbox metadata is (a)> Potentially much too expensive and (b) Subject to the
> vagarities of user> actions.
> 
> I'm not happy about it, but I think this needs to become a SHOULD.

I can live with SHOULD for cases where there are multiple mailboxes with
the same special-use.
> Another possibility is to use the default mailbox name as a means of
> disambiguating multiple mailboxes with the same special-use attribute:> If the default mailbox is one of these deliver to it preferentially,
> otherwise the behavior is implementation-defined. Not perfect, but it> provides a bit more consistent behavior that's easily achievable.
> 
> Aside from these issues, this draft looks good to go to me.

Thanks for the great feedback!

Cheers,

Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Stephan Bosch <[email-masked]>
Date: Wed, 18 Jul 2018 22:58:09 +0200

Hi Ned,

Op 13/06/2018 om 17:40 schreef Ned Freed:
>> actual IMAP
>> > commands will be especially valuable: Pointing out that you should 
>> use the
>> > NAMESPACE extension and if it isn't present, fallback to LIST "" 
>> would be very
>> > helpful.
>
>> Here's my initial attempt at addressing this point (separate branch in
>> Github):
>
>> https://github.com/ietfextra/draft-ietf-extra-sieve-special-use/commit/035a1af6137a383089c8d54cd733238dc2dfebda 
>>
>
>   S: * NAMESPACE (("INBOX/" "/")("Archive/" "/")) NIL (("Public/" "/"))
>   S: A01 OK NAMESPACE command completed
>
> In the next step:
>
>    Subsequently, when no particular mailbox is of interest (i.e., the
>    "specialuse_exists" test has no mailbox argument), the client lists
>    all mailboxes with special-use flags in the two returned personal
>    namespaces:
>
>    C: A02 LIST (SPECIAL-USE) "" ("INBOX/*" "Archive/*") RETURN 
> (SPECIAL-USE)
>    S: * LIST (\Drafts) "/" INBOX/Drafts
> It should be noted that this depends on extended list being available.

True. I thought LIST-EXTENDED is pretty much required for SPECIAL-USE, 
but the specification doesn't actually mention anything like that. Is 
there another way to obtain information on which mailboxes have a 
particular special-use flag assigned?

>    S: * OK [PERMANENTFLAGS (\Answered \Deleted \Seen \Draft \*)] Flags 
> permitted.
>    S: * 11211 EXISTS
>    S: * 3 RECENT
> First, if you're going to use SELECT it's probably worth noting that 
> since
> inclusion of [READ-WRITE] for writable mailboxes is a SHOULD in RFC 
> 3501 while
> inclusion of [READ-ONLY] for non-writable is a MUST, a writability 
> check really
> needs to based on the absence of [READ-ONLY]. (RFC 5490 3.1.a appears 
> incorrect
> to me in this regard - should I file an errata?)

Agreed.

> potentially expensive and worse, may cause undesireable state changes. 
> Why
> not use the MYRIGHTS command to obtain the access rights (and check 
> for "p" or
> "i") without having to select the mailbox?

Sure, if the ACL capability is available. I can mention that. If it is 
not available, there is really no alternative to using SELECT, right?

> And as a bonus, if it's available you could use the just-standardized
> LIST-MYRIGHTS extension to piggyback getting the rights while getting 
> the list
> of special-use mailboxes.

Sure.

> implement the
> necessary semantics of an extension is helpful - there may be 
> optimizations and
> concerns that should be called out lest implementors miss them.
>
> It also helps clarify overall implementation difficulty. And while this
> extension may be easy to implement in the Sieve-in-IMAP case, from the
> perspective of an MTA/MDA implementation, even if all of the steps can 
> be done
> efficiently, there's still the question of IMAP server availability 
> and the
> handling of temporary errors.
>

Yeah.

> All of which is roundabout way of saying I now think that we may want a
> capability for specialuse_exists that's separate from
> "mailbox"
> capability for "mailbox access stuff", I think we should require the 
> presence
> of both "mailbox" and "special-use" capabilities in order to use
> specialuse_exists.

Can you clarify this point? I don't see how this leads to requiring the 
mailbox extension.

> I also reviewed the rest of the revised draft, and it's looking good. 
> My one
> remaining concern is with the handling of multiple mailboxes having the
> same special-use flag assigned. The draft currently says:
>
>   particular special-use flag assigned.  In case of such ambiguity, the
>   mailbox that is chosen for delivery is implementation-defined.
>   However, while the set of mailboxes to which the involved special-use
>   flags are assigned remains unchanged, implementations MUST ensure
> I don't see a way I can reasonably implement this MUST via IMAP unless 
> the IMAP server always returns the list of special-use mailboxes in 
> the same order.
> Sure, I can cache the user/special-use-attribute/actual-mailbox-list 
> tuple
> somewhere, but caches can always be lost. And mailbox metadata is (a)
> Potentially much too expensive and (b) Subject to the vagarities of user
> actions.
>

Ok. I am not too attached to the MUST anyway.

> disambiguating multiple mailboxes with the same special-use attribute:
> If the default mailbox is one of these deliver to it preferentially,
> otherwise the behavior is implementation-defined. Not perfect, but it
> provides a bit more consistent behavior that's easily achievable.
>

Sounds like a good idea. I'll give this a closer look for the next revision.

> Aside from these issues, this draft looks good to go to me.

I still need to make an IMAP example for "fileinto :specialuse".

I should be able to work on this next week.

Regards,

STepham.

---

From: Ned Freed <[email-masked]>
Date: Thu, 19 Jul 2018 06:40:25 -0700 (PDT)

> particular special-use flag assigned?

Yes, according to RFC 6154 section 2 special use attributes MAY appear in
non-extended LIST responses. There's an example of this in section 5.1.

However, the fact that this is a MAY makes it effectively useless since you
can't tell the difference between a case where there are no special use
attributes assigned and one where the server has simply elected not to include
these attributes in its responses. And it's expected that the overwhelming
majority of mailboxes won't have any special use assigned.

In fact I'd go so far to say that this is a design error. It's one thing
to want to support existing implementations of special use attributes, it's
another to be so permissive that a compliant implementation - one that
supports SPECIAL-USE but not LIST-EXTENDED - can have no way to determine
what the special use mailboxes are actually present.

All that said, all I was asking for here was a note saying this depended on
LIST-EXTENDED. Nothing more. (Personally, I don't intend to even bother to
check if it's supported.)

> not available, there is really no alternative to using SELECT, right?

Not as far as I know. But again, the problem is that SELECTing a mailbox
on behalf of a user can change its state, with all that implies. I'd rather
have my implementation fail if ACL isn't present than have a sieve operation
cause a state change.

> Can you clarify this point? I don't see how this leads to requiring the

It doesn't unless we want it to, and I think we want it to. We have this
collection of stuff collected under the mailbox extension that lets sieve peek
at the state of the mailbox. And here we have an extension that consists of two
parts: One extending fileinto and the other adding to things you can peek at.

>From an MTA implementation perspective, these are very distinct things, and
the latter shares a ton of code with the mailbox extension. It therefore
makes sense to me to separate the two and link it to the mailbox extension.

It's really no different than the way SPECIAL-USE and LIST-EXTENDED
interact on the IMAP side of things: Both have to be present for the
RETURN (SPECIAL-USE) stuff to work.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 20 Jul 2018 00:14:47 +1000

On Thu, Jul 19, 2018, at 23:40, Ned Freed wrote:
>>> In the next step:
>>> 
>>>    Subsequently, when no particular mailbox is of interest (i.e.,
>>>    the>>>    "specialuse_exists" test has no mailbox argument), the client
>>>    lists>>>    all mailboxes with special-use flags in the two returned personal>>>    namespaces:
>>> 
>> True. I thought LIST-EXTENDED is pretty much required for SPECIAL-
>> USE,>> but the specification doesn't actually mention anything like that. Is>> there another way to obtain information on which mailboxes have a
> Yes, according to RFC 6154 section 2 special use attributes MAY
> appear in> non-extended LIST responses. There's an example of this in
> section 5.1.> 
> However, the fact that this is a MAY makes it effectively useless
> since you> can't tell the difference between a case where there are no
> special use> attributes assigned and one where the server has simply elected not
> to include> these attributes in its responses. And it's expected that the
> overwhelming> majority of mailboxes won't have any special use assigned.

MAY fricking shmay.  There are clients out there which flat out expect
that you return the SPECIAL-USE for an unadorned LIST, and customers
report your server as buggy if it doesn't do what their client expects.
> In fact I'd go so far to say that this is a design error. It's
> one thing> to want to support existing implementations of special use
> attributes, it's> another to be so permissive that a compliant implementation - one that> supports SPECIAL-USE but not LIST-EXTENDED - can have no way to
> determine> what the special use mailboxes are actually present.

I'm sure that RFC3501bis (IMAP4rev2) will fix this.

> All that said, all I was asking for here was a note saying this
> depended on> LIST-EXTENDED. Nothing more. (Personally, I don't intend to even
> bother to> check if it's supported.)

That seems very reasonable.

>>> However, I think use of SELECT for this purpose is problematic: It's>>> potentially expensive and worse, may cause undesireable state
>>> changes.>>> Why
>>> not use the MYRIGHTS command to obtain the access rights (and check>>> for "p" or
>> Sure, if the ACL capability is available. I can mention that.
>> If it is>> not available, there is really no alternative to using SELECT, right?> 
> Not as far as I know. But again, the problem is that SELECTing
> a mailbox> on behalf of a user can change its state, with all that implies.
> I'd rather> have my implementation fail if ACL isn't present than have a sieve
> operation> cause a state change.

Thanks \Recent.  But at least EXAMINE rather than SELECT should also
mean no change made (in theory).
>>> All of which is roundabout way of saying I now think that we may
>>> want a>>> capability for specialuse_exists that's separate from
>>> "special-use"/:specialuse. Or better still, since we already
>>> have the>>> "mailbox"
>>> capability for "mailbox access stuff", I think we should require the>>> presence
>> Can you clarify this point? I don't see how this leads to
>> requiring the>> mailbox extension.
> 
> It doesn't unless we want it to, and I think we want it to. We
> have this> collection of stuff collected under the mailbox extension that lets
> sieve peek> at the state of the mailbox. And here we have an extension that
> consists of two> parts: One extending fileinto and the other adding to things you can
> peek at.> 
> From an MTA implementation perspective, these are very distinct
> things, and> the latter shares a ton of code with the mailbox extension. It
> therefore> makes sense to me to separate the two and link it to the mailbox
> extension.>

---

From: Stephan Bosch <[email-masked]>
Date: Thu, 19 Jul 2018 16:30:12 +0200

Op 19-7-2018 om 16:14 schreef Bron Gondwana:
>>         changes.

The EXAMINE command will not provide information on the read-only status 
of the mailbox though:

RFC 3501, Section 6.3.2:

The text of the tagged OK response to the EXAMINE command MUST begin with the "[READ-ONLY]" response code.

Regards,

-- 

Stephan Bosch
Senior Developer

Phone: +49 2761 75252 00  Fax: +49 2761 75252 30
Email: [email-masked]

-------------------------------------------------------------------------------------
Open-Xchange AG,  Rollnerstr. 14, 90408 Nuremberg, District Court Nuremberg HRB 24738
Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Uwe Reumuth
Chairman of the Board: Richard Seibt

Dovecot Oy, Lars Sonckin Kaari 10, 02600 Espoo, Finland
Managing Director: Markku Kenttä
Chairman of the Board: Timo Sirainen
Board Member: Carsten Dirks

-------------------------------------------------------------------------------------

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 20 Jul 2018 00:38:26 +1000

On Fri, Jul 20, 2018, at 00:30, Stephan Bosch wrote:
>>>       potentially expensive and worse, may cause undesireable state>>>       changes.
>>>       Why
>>>       not use the MYRIGHTS command to obtain the access rights (and>>>       check
>>>       for "p" or
>>>   not available, there is really no alternative to using SELECT,
>>>   right?>>> 
>>> 
>>> Not as far as I know. But again, the problem is that SELECTing a
>>> mailbox>>> on behalf of a user can change its state, with all that implies. I'd>>> rather
>> Thanks \Recent.  But at least EXAMINE rather than SELECT should also>> mean no change made (in theory).
>> 
> 
> The EXAMINE command will not provide information on the read-
> only status> of the mailbox though:
> 
> RFC 3501, Section 6.3.2:
> 
Oh yeah, good point.  Right!  So MYRIGHTS seems the right way then
:)  (it's certainly what we're doing the moral equivalent of
inside our code)
Bron.

---

From: Ned Freed <[email-masked]>
Date: Thu, 19 Jul 2018 09:00:45 -0700 (PDT)

> MAY fricking shmay.

:)

> There are clients out there which flat out expect

Well, if that's the case, we have a problem independent of this Sieve
extension. Do we need to at least file an errata on this?

> I'm sure that RFC3501bis (IMAP4rev2) will fix this.

Which creates a dependency on yet another IMAP extension. Not sure if
this is sufficient.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 20 Jul 2018 07:45:17 +1000

On Fri, Jul 20, 2018, at 02:00, Ned Freed wrote:
>>> appear in> non-extended LIST responses. There's an example of
>>> this in>>> section 5.1.>
> 
>>> since you> can't tell the difference between a case where there
>>> are no>>> special use> attributes assigned and one where the server has simply
>>> elected not>>> to include> these attributes in its responses. And it's expected
>>> that the>>> overwhelming> majority of mailboxes won't have any special use
>>> overwhelming> assigned.> 
>> MAY fricking shmay.
> 
> :)
> 
>> that you return the SPECIAL-USE for an unadorned LIST, and customers>> report your server as buggy if it doesn't do what their client
>> expects.> 
> Well, if that's the case, we have a problem independent of this Sieve> extension. Do we need to at least file an errata on this?

Errata submitted, suggesting s/MAY/SHOULD/ in section 2.

>>> one thing> to want to support existing implementations of
>>> special use>>> attributes, it's> another to be so permissive that a compliant
>>> implementation - one that> supports SPECIAL-USE but not LIST-
>>> EXTENDED - can have no way to>>> determine> what the special use mailboxes are actually present.
> 
> Which creates a dependency on yet another IMAP extension. Not sure if> this is sufficient.

IMAP4rev2 is already planning to remove all the non-extended responses
to things.  Adding special-use and requiring that it be present in all
responses is totally reasonable within the scope of the other changes.
Bron.

---

From: "Chris Newman" <[email-masked]>
Date: Thu, 19 Jul 2018 18:47:41 -0400

On 19 Jul 2018, at 9:40, Ned Freed wrote:

>> the
> since you

I believe that's an incorrect interpretation of the spec. RFC 6154 
states:

    ...  If the client specifies the "SPECIAL-USE" selection option,
    the LIST command MUST return only those mailboxes that have a
    special-use attribute set.  If the client specifies the 
"SPECIAL-USE"
    return option, the LIST command MUST return the new special-use
    attributes on those mailboxes that have them set. ...

This spec applies to all servers that advertise SPECIAL-USE. Those two 
MUST clauses do not have an exception for servers that do not advertise 
LIST-EXTENDED. The example in section 5.2 supports this interpretation, 
since it does not include LIST-EXTENDED in the capabilities response.

Contrast that with section 4:

    If a server supports this extension and the METADATA extension
    [RFC5464], it SHOULD tie the special-use attributes for a mailbox to
    its metadata entry "/private/specialuse".

where there's an explicit exception to the SHOULD for servers that do 
not implement METADATA due to the "if" clause.

If I recall correctly, the difference in normative language between 
these two sections was deliberate.

		- Chris

> depended on
> mailbox
>> want a
>> things, and
> _______________________________________________
> Extra mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_extra&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=K_BObr5Kfkr3rxt1oBPF9KFiEU3xl9LcD2OOJG3TXfI&m=fqjptcmWVzu3waxuhECJKbXvzfSuBPDLN9t7KQMtjW8&s=G51gQpE9MxifdONliqnLO9agM4akhpX0R2j_SbJQvi8&e=

---

From: Ned Freed <[email-masked]>
Date: Thu, 19 Jul 2018 15:55:48 -0700 (PDT)

>     attributes on those mailboxes that have them set. ...

See the beginning of the paragraph you neglected to quote. It's quite clear
that  this is in the context of LIST-EXTENDED, not unextended LIST. This is
reiterated in the ABNF.

> since it does not include LIST-EXTENDED in the capabilities response.

So what you're saying is that despite the fact that the section in question is
specifically about using extended list - unextended list is covered in section
5.1 - the lack of the list-extended list in the example capabilities is
sufficient reason to believe that the presence of the special-use cability
magically enables the RETURN clause component of the special use extension and
nothing else?

If this is indeed how this is supposed to be interpreted it needs to be
clarified with an errata, because your reading is frankly pretty tortuous.

				Ned

---

From: "Chris Newman" <[email-masked]>
Date: Thu, 19 Jul 2018 19:26:39 -0400

> magically enables the RETURN clause component of the special use 

My reading matches our implementation. It would not surprise me if it 
matches most (perhaps all) implementations.

Our implementation implements LIST-EXTENDED except for the 
multiple-list-pattern feature. As a result, we can not advertise the 
LIST-EXTENDED extension. Implementing the multiple-list-pattern feature 
would require at least two solid weeks of development effort for our 
implementation; I have no plans to do that work.

We interpret and implement the STATUS-in-LIST extension the same way 
(RFC 5819). That one is a bit more straightforward since advertising 
that capability would be meaningless by itself absent this 
interpretation.

> clarified with an errata, because your reading is frankly pretty 

Since the specification does not match reality around the MAY/SHOULD 
issue, it needs a revision. Errata is not a good way to handle something 
like this.

		- Chris