Subject: Re: [Extra] SAVEDATE updated required

From: Stephan Bosch <[email-masked]>
Date: Thu, 15 Feb 2018 23:31:33 +0100

Hi Bron,

Any word on the questions I asked below?

Op 1/28/2018 om 3:51 PM schreef Stephan Bosch:
> Op 1/7/2018 om 11:04 PM schreef Bron Gondwana:
>> On Mon, 8 Jan 2018, at 00:04, Stephan Bosch wrote:
>>> Hi Bron,
>>>
>>> Op 1/4/2018 om 1:54 AM schreef Bron Gondwana:
>>>
>>>     2: if the mailbox doesn't support storing save date, the SEARCH
>>>     extensions should return an error if used, rather than a result,
>>>     since
>>>     there's no way to calculate a sensible answer to any of those three.
>>>
>>>
>>> Ok, for the sake of discussion, I added text with this effect in the
>>> last version. However, I am not quite happy with it yet. Returning an
>>> error seems a bit harsh. Do we need to provide some means for the client
>>> to find out whether the mailbox supports the save date attribute before
>>> trying to use it?
>> That could be done:
>>
>> TAG select "foo"
>> * 1 EXISTS
>> * 1 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Seen \*)] Ok
>> * OK [UNSEEN 1] Ok
>> * OK [UIDVALIDITY 1515362059] Ok
>> * OK [UIDNEXT 2] Ok
>> * OK [HIGHESTMODSEQ 7] Ok
>> * OK [URLMECH INTERNAL] Ok
>> * OK [ANNOTATIONS 65536] Ok
>> TAG OK [READ-WRITE] Completed
>>
>> Another line in there that said something like:
>>
>> * OK [SAVEDATE] Ok
>>
>> Would probably work fine.
> Yes, that could work. Although, when the SQL NULL semantics (discussed
> below) are implemented, this would likely no longer be necessary/useful.
>
>>> Also, there's the MULTISEARCH extension (RFC7377), which adds an ESEARCH
>>> command that allows searching across a wide selection of mailboxes, some
>>> of which may in fact lack support for the save date attribute. What is
>>> to be done in that case? Fail the whole ESEARCH command? Ignore the
>>> mailboxes that lack support? Make the SAVED* search items yield a false
>>> result for that mailbox? None of these options seem very appealing.
>> There are three choices:
>> 1) reject the command if any folder doesn't support SAVEDATE
>> 2) reject the command if none of the folders support SAVEDATE
>> 3) succeed regardless of the support for SAVEDATE
>>
>> And if there are folders that don't support SAVEDATE for case 2 or 3,
>> you can:
>> a) use the INTERNALDATE instead
>> b) don't match the message
>>
>> I think (a) is clearly wrong here - if you're deleting everything
>> that's SAVEDBEFORE 1 month ago from a Trash folder, then INTERNALDATE
>> will purge things before you want to.
> Yes, indeed.
>
>> The alternative is to treat SAVEDATE as NULL in the same way SQL does,
>> and I think that's the right choice.  So you can say "SEARCH NOT
>> SAVEDAFTER {X} BEFORE {X}" if you want to get the behaviour of
>> deleting everything that was SAVEDBEFORE a time, or fallback to
>> INTERNALDATE if it's not supported.
> I see what you're trying to do here and I agree that something like that
> would be a good idea. However, we need  to be very careful about the
> semantics. With SQL 3-valued logic, I think (with my rather limited
> SQL-fu) your example would evaluate to Unknown:
>
> `NOT SAVEDAFTER {X} BEFORE {X}' logically translates to:
>
> (not `SAVEDAFTER {X}') and `BEFORE {X}'
> (not Unknown) and `BEFORE {X}'
> Unknown and `BEFORE {X}'
> Unknown
>
> So, how did you expect this example to work? Or am I doing something
> wrong in this inference?

Regards,

-- 
Stephan Bosch
Senior Developer 

Phone: +49 2761 75252 00  Fax: +49 2761 75252 30
Email: [email-masked]

-------------------------------------------------------------------------------------
Open-Xchange AG,  Rollnerstr. 14, 90408 Nuremberg, District Court Nuremberg HRB 24738
Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Michael Knapstein 
Chairman of the Board: Richard Seibt

Dovecot Oy, Lars Sonckin Kaari 10, 02600 Espoo, Finland
Managing Director: Markku Kentta
Chairman of the Board: Timo Sirainen
Board Member: Carsten Dirks

-------------------------------------------------------------------------------------

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 18 Feb 2018 21:04:02 +1100

Hi Stephen, sorry about missing replying on this - it came through in
the week I was really sick and I didn't get back to it!
Let's look at it with SQL (sqlite in this case):

id|internaldate|savedate
1|2018-01-01|2018-01-01
2|2018-01-01|2018-02-01
3|2018-02-01|2018-02-01
4|2018-01-01|
5|2018-02-01|

table 'foo' - with rows 4 and 5 having NULL savedate.  We want to select
rows 1 and 4 without getting row 2.
Yeah, you're right:

sqlite> select * from foo where not savedate > '2018-01-30' and
sqlite> internaldate < '2018-01-30';id|internaldate|savedate
1|2018-01-01|2018-01-01
sqlite> select * from foo where (savedate is null or not savedate > '2018-01-
sqlite> 30') and internaldate < '2018-01-30';id|internaldate|savedate
1|2018-01-01|2018-01-01
4|2018-01-01|
sqlite>

Well, that's a pain.  I definitely want a way to say that second
expression - because that's what I would be working with if I wanted to
be able to select "old" messages regardless of whether SAVEDATE was
supported.
I guess in this particular case, the idea of "fall back to internaldate
only if savedate isn't defined" would be exactly what I wanted, but then
maybe I'd want SAVEDBEFORESTRICT and SAVEDBEFORERELAXED :p  Horrible
names though.
Mind you, with IMAP semantics you don't have Undefined, so having
SAVEDBEFORE and SAVEDAFTER both always evaluate false if you don't have
support would give the behaviour that my expression wants.
Bron.

On Fri, 16 Feb 2018, at 09:31, Stephan Bosch wrote:
>>>>   extensions should return an error if used, rather than a result,>>>>   since
>>>>   there's no way to calculate a sensible answer to any of those
>>>>   three.>>>> 
>>>> 
>>>> Ok, for the sake of discussion, I added text with this effect
>>>> in the>>>> last version. However, I am not quite happy with it yet.
>>>> Returning an>>>> error seems a bit harsh. Do we need to provide some means for the
>>>> client>>>> to find out whether the mailbox supports the save date attribute
>>>> before>>>> trying to use it?
>>> That could be done:
>>> 
>>> TAG select "foo"
>>> * 1 EXISTS
>>> * 1 RECENT
>> Yes, that could work. Although, when the SQL NULL semantics
>> (discussed>> below) are implemented, this would likely no longer be
>> necessary/useful.>> 
>>>> Also, there's the MULTISEARCH extension (RFC7377), which adds an
>>>> ESEARCH>>>> command that allows searching across a wide selection of
>>>> mailboxes, some>>>> of which may in fact lack support for the save date attribute.
>>>> What is>>>> to be done in that case? Fail the whole ESEARCH command? Ignore the>>>> mailboxes that lack support? Make the SAVED* search items yield a
>>>> false>>>> result for that mailbox? None of these options seem very appealing.>>> There are three choices:
>>> And if there are folders that don't support SAVEDATE for case
>>> 2 or 3,>>> you can:
>>> that's SAVEDBEFORE 1 month ago from a Trash folder, then
>>> INTERNALDATE>>> will purge things before you want to.
>> Yes, indeed.
>> 
>>> The alternative is to treat SAVEDATE as NULL in the same way
>>> SQL does,>>> and I think that's the right choice.  So you can say "SEARCH NOT
>> I see what you're trying to do here and I agree that something
>> like that>> would be a good idea. However, we need  to be very careful about the>> semantics. With SQL 3-valued logic, I think (with my rather limited
>> So, how did you expect this example to work? Or am I doing something>> wrong in this inference?
> 
> Regards,
> 
> --
> Stephan Bosch
> Senior Developer
> 
> 
> ----------------------------------------------------------------------
> ---------------> Open-Xchange AG,  Rollnerstr. 14, 90408 Nuremberg, District Court
> Nuremberg HRB 24738> Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Michael
> Knapstein> Chairman of the Board: Richard Seibt
> 
> ----------------------------------------------------------------------
> ---------------> 
> _________________________________________________
> Extra mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/extra

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Stephan Bosch <[email-masked]>
Date: Wed, 28 Feb 2018 09:03:30 +0100

Hi Bron,

> the week I was really sick and I didn't get back to it!

I haven't been having much time myself either lately.

> table 'foo' - with rows 4 and 5 having NULL savedate.  We want to
> select rows 1 and 4 without getting row 2.
>
> Yeah, you're right:
>
> internaldate < '2018-01-30';
> id|internaldate|savedate
> 1|2018-01-01|2018-01-01
> sqlite> select * from foo where (savedate is null or not savedate >
> '2018-01-30') and internaldate < '2018-01-30';
> id|internaldate|savedate
> 1|2018-01-01|2018-01-01
> 4|2018-01-01|
> sqlite>
>
> expression - because that's what I would be working with if I wanted
> to be able to select "old" messages regardless of whether SAVEDATE was
> supported.
>
> I guess in this particular case, the idea of "fall back to
> internaldate only if savedate isn't defined" would be exactly what I
> wanted, but then maybe I'd want SAVEDBEFORESTRICT and
> SAVEDBEFORERELAXED :p  Horrible names though.
>
> SAVEDBEFORE and SAVEDAFTER both always evaluate false if you don't
> have support would give the behaviour that my expression wants.
>

So, what do we do here?

In at least some storage implementations, Dovecot reverts to using the
current date instead, storing it in cache after that time, so that at
least something is remembered.

Regards,

Stephan.

>>     (discussed
>>         2 or 3,
>>         INTERNALDATE
>>         SQL does,
>>     like that
>> Email: [email-masked] <mailto:[email-masked]>
>>
>>
>> [email-masked] <mailto:[email-masked]>
> _______________________________________________
> Extra mailing list
> [email-masked]

-- 
Stephan Bosch
Senior Developer