Subject: [Extra] SAVEDATE updated required

From: Bron Gondwana <[email-masked]>
Date: Thu, 04 Jan 2018 11:54:55 +1100

HI Stephan,

I've also created a couple of tasks for SAVEDATE:

https://github.com/ietfextra/tracking/issues/12
https://github.com/ietfextra/tracking/issues/13

Of course, reading the standard that closely is really good, because it
also forced me to think about it!  I have some opinions!
1: if the mailbox doesn't support storing save date, the behaviour should be to return SAVEDATE NIL/
2: if the mailbox doesn't support storing save date, the SEARCH extensions should return an error if used, rather than a result, since there's no way to calculate a sensible answer to any of those three.

   SAVEDBEFORE <date>
      Messages whose save date (disregarding time and timezone) is
      earlier than the specified date.

   SAVEDON <date>
      Messages whose save date (disregarding time and timezone) is
      within the specified date.

   SAVEDSINCE <date>
      Messages whose save date (disregarding time and timezone) is
      within or later than the specified date.

The "disregarding time and timezone" is a horrorshow if you have users
in multiple timezones, but RFC3501 says that's how BEFORE and AFTER work
for INTERNALDATE commands, so I guess we're stuck with it.

3: Do we need to define SAVEDATE as a SORT key extending RFC5256 as well?  In theory it should always sort the same as UID.
Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Stephan Bosch <[email-masked]>
Date: Thu, 4 Jan 2018 11:10:36 +0100

Op 1/4/2018 om 1:54 AM schreef Bron Gondwana:
> Of course, reading the standard that closely is really good, because
> it also forced me to think about it!  I have some opinions!
>
> 2: if the mailbox doesn't support storing save date, the SEARCH
> extensions should return an error if used, rather than a result, since
> there's no way to calculate a sensible answer to any of those three.
>
>
>    SAVEDBEFORE <date>
> in multiple timezones, but RFC3501 says that's how BEFORE and AFTER
> work for INTERNALDATE commands, so I guess we're stuck with it.
>
>
> 3: Do we need to define SAVEDATE as a SORT key extending RFC5256 as
> well?  In theory it should always sort the same as UID.

OK, I will get back to you in the coming weekend.

Regards,

-- 
Stephan Bosch
Senior Developer 

Phone: +49 2761 75252 00  Fax: +49 2761 75252 30
Email: [email-masked]

-------------------------------------------------------------------------------------
Open-Xchange AG,  Rollnerstr. 14, 90408 Nuremberg, District Court Nuremberg HRB 24738
Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Michael Knapstein 
Chairman of the Board: Richard Seibt

Dovecot Oy, Lars Sonckin Kaari 10, 02600 Espoo, Finland
Managing Director: Markku Kentta
Chairman of the Board: Timo Sirainen
Board Member: Carsten Dirks

-------------------------------------------------------------------------------------

---

From: Stephan Bosch <[email-masked]>
Date: Sun, 7 Jan 2018 14:04:01 +0100

Hi Bron,

> https://github.com/ietfextra/tracking/issues/13

Fixed and closed one of these already. For the other one, read further..

> should be to return SAVEDATE NIL/

I agree. Timo also suggested this a while back. I changed this in the
last version.

>       within or later than the specified date.

Ok, for the sake of discussion, I added text with this effect in the
last version. However, I am not quite happy with it yet. Returning an
error seems a bit harsh. Do we need to provide some means for the client
to find out whether the mailbox supports the save date attribute before
trying to use it?

Also, there's the MULTISEARCH extension (RFC7377), which adds an ESEARCH
command that allows searching across a wide selection of mailboxes, some
of which may in fact lack support for the save date attribute. What is
to be done in that case? Fail the whole ESEARCH command? Ignore the
mailboxes that lack support? Make the SAVED* search items yield a false
result for that mailbox? None of these options seem very appealing.

> work for INTERNALDATE commands, so I guess we're stuck with it.

Yes, that is what this is based on.

> well?  In theory it should always sort the same as UID.

I don't think this is particularly useful, although it is not difficult
to add. I agree that sorting is normally equal to seq/UID. I checked:
Dovecot does not currently implement a sort key for this.

Still, this should be a point for discussion. Anyone else wish to comment?

I published my git repository and made a new draft submission.

Regards,

Stephan.

-- 

Stephan Bosch
Senior Developer

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 08 Jan 2018 09:04:26 +1100

On Mon, 8 Jan 2018, at 00:04, Stephan Bosch wrote:
>> extensions should return an error if used, rather than a
>> result, since>> there's no way to calculate a sensible answer to any of those three.>> 
>> 
>>    SAVEDBEFORE <date>
> last version. However, I am not quite happy with it yet. Returning an> error seems a bit harsh. Do we need to provide some means for
> the client> to find out whether the mailbox supports the save date
> attribute before> trying to use it?

That could be done:

TAG select "foo"
* 1 EXISTS
* 1 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1515362059] Ok
* OK [UIDNEXT 2] Ok
* OK [HIGHESTMODSEQ 7] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
TAG OK [READ-WRITE] Completed

Another line in there that said something like:

* OK [SAVEDATE] Ok

Would probably work fine.

> Also, there's the MULTISEARCH extension (RFC7377), which adds
> an ESEARCH> command that allows searching across a wide selection of
> mailboxes, some> of which may in fact lack support for the save date attribute. What is> to be done in that case? Fail the whole ESEARCH command? Ignore the
> mailboxes that lack support? Make the SAVED* search items yield a
> false> result for that mailbox? None of these options seem very appealing.

There are three choices:
1) reject the command if any folder doesn't support SAVEDATE
2) reject the command if none of the folders support SAVEDATE
3) succeed regardless of the support for SAVEDATE

And if there are folders that don't support SAVEDATE for case 2 or
3, you can:a) use the INTERNALDATE instead
b) don't match the message

I think (a) is clearly wrong here - if you're deleting everything that's
SAVEDBEFORE 1 month ago from a Trash folder, then INTERNALDATE will
purge things before you want to.
The alternative is to treat SAVEDATE as NULL in the same way SQL does,
and I think that's the right choice.  So you can say "SEARCH NOT
SAVEDAFTER {X} BEFORE {X}" if you want to get the behaviour of deleting
everything that was SAVEDBEFORE a time, or fallback to INTERNALDATE if
it's not supported.
> I don't think this is particularly useful, although it is not
> difficult> to add. I agree that sorting is normally equal to seq/UID. I checked:> Dovecot does not currently implement a sort key for this.

I'm fine with not having it.

By the way, I have plans for a JMAP extension which maps SAVEDATE into
JMAP.  Have been chatting with Neil about it.  It seems a good first
case for an example extension :)
Bron.

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 9 Jan 2018 20:37:28 +0100

Hi Bron,

I need to think about this a little more. I will reply this one a bit later.

Regards,

Stephan.

Op 1/7/2018 om 11:04 PM schreef Bron Gondwana:
> And if there are folders that don't support SAVEDATE for case 2 or 3,
> you can:
> a) use the INTERNALDATE instead
> I think (a) is clearly wrong here - if you're deleting everything
> that's SAVEDBEFORE 1 month ago from a Trash folder, then INTERNALDATE
> will purge things before you want to.
>
> SAVEDAFTER {X} BEFORE {X}" if you want to get the behaviour of
> deleting everything that was SAVEDBEFORE a time, or fallback to
> INTERNALDATE if it's not supported.
>
> _______________________________________________
> Extra mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/extra

-- 
Stephan Bosch
Senior Developer

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 10 Jan 2018 09:07:32 +1100

No problem at all!  I'm running a choir festival for the next 10 days,
so I'm going to have very limited availability anyway :)
Cheers,

Bron.

On Wed, 10 Jan 2018, at 06:37, Stephan Bosch wrote:
> Hi Bron,
> 
> I need to think about this a little more. I will reply this one a
> bit later.> 
> Regards,
> 
> Stephan.
> 
>>>   there's no way to calculate a sensible answer to any of those
>>>   three.>>> 
>>> 
>>>      SAVEDBEFORE <date>
>>>         Messages whose save date (disregarding time and timezone) is>>>         earlier than the specified date.
>>> 
>>>      SAVEDON <date>
>>>         Messages whose save date (disregarding time and timezone) is>>>         within the specified date.
>>> 
>>>      SAVEDSINCE <date>
>>>         Messages whose save date (disregarding time and timezone) is>>>         within or later than the specified date.
>>> 
>>> 
>>> Ok, for the sake of discussion, I added text with this effect in the>>> last version. However, I am not quite happy with it yet.
>>> Returning an>>> error seems a bit harsh. Do we need to provide some means for the
>>> client>>> to find out whether the mailbox supports the save date attribute
>>> before>>> trying to use it?
>> 
>> That could be done:
>> 
>> TAG select "foo"
>> * 1 EXISTS
>> * 1 RECENT
>>> Also, there's the MULTISEARCH extension (RFC7377), which adds an
>>> ESEARCH>>> command that allows searching across a wide selection of
>>> mailboxes, some>>> of which may in fact lack support for the save date attribute.
>>> What is>>> to be done in that case? Fail the whole ESEARCH command? Ignore the>>> mailboxes that lack support? Make the SAVED* search items yield
>>> a false>>> result for that mailbox? None of these options seem very appealing.>> 
>> There are three choices:
>> And if there are folders that don't support SAVEDATE for case 2 or 3,>> you can:
>> that's SAVEDBEFORE 1 month ago from a Trash folder, then INTERNALDATE>> will purge things before you want to.
>> 
>> The alternative is to treat SAVEDATE as NULL in the same way
>> SQL does,>> and I think that's the right choice.  So you can say "SEARCH NOT
>>>   3: Do we need to define SAVEDATE as a SORT key extending RFC5256
>>>      as>>>   well?  In theory it should always sort the same as UID.
>>> 
>>> 
>>> difficult>>> to add. I agree that sorting is normally equal to seq/UID. I
>>> checked:>>> Dovecot does not currently implement a sort key for this.
>> 
>> By the way, I have plans for a JMAP extension which maps
>> SAVEDATE into>> JMAP.  Have been chatting with Neil about it.  It seems a good first>> case for an example extension :)
>> 
>> Bron.
>> 
>> --
>> _________________________________________________
>> Extra mailing list
>> [email-masked]
> ----------------------------------------------------------------------
> ---------------> Open-Xchange AG,  Rollnerstr. 14, 90408 Nuremberg, District Court
> Nuremberg HRB 24738> Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Michael
> Knapstein> Chairman of the Board: Richard Seibt
> 
> ----------------------------------------------------------------------
> --------------->

---

From: Stephan Bosch <[email-masked]>
Date: Sun, 28 Jan 2018 15:51:07 +0100

> Would probably work fine.

Yes, that could work. Although, when the SQL NULL semantics (discussed
below) are implemented, this would likely no longer be necessary/useful.

> will purge things before you want to.

Yes, indeed.

> INTERNALDATE if it's not supported.

I see what you're trying to do here and I agree that something like that
would be a good idea. However, we need  to be very careful about the
semantics. With SQL 3-valued logic, I think (with my rather limited
SQL-fu) your example would evaluate to Unknown:

`NOT SAVEDAFTER {X} BEFORE {X}' logically translates to:

(not `SAVEDAFTER {X}') and `BEFORE {X}'
(not Unknown) and `BEFORE {X}'
Unknown and `BEFORE {X}'
Unknown

So, how did you expect this example to work? Or am I doing something
wrong in this inference?

> case for an example extension :)

Ok, good. :)

Regards,

-- 

Stephan Bosch
Senior Developer