Subject: [Extra] SEARCHRES / IMAP4rev2 clarifications

From: Timo Sirainen <[email-masked]>
Date: Fri, 13 Mar 2020 12:22:54 +0200

I was just fixing various bugs in my SEARCHRES implementation and realized that there's a few issues that aren't too clear:

1) SEARCH RETURN (SAVE) $

It's possible to use $ in the same search query that updates it, so the above search should preserve $. My code cleared $ before running the search, so the result was that $ was cleared.

2) The RFC text says:

   Upon successful completion of a SELECT or an EXAMINE command (after
   the tagged OK response), the current search result variable is reset
   to the empty sequence.

So I was going to test that $ is preserved if SEARCH/EXAMINE fails. Except selecting nonexistent mailbox closes the current mailbox. So then I thought testing with invalid SELECT options, like SELECT INBOX (blah) which returns BAD. But my current code still closes the current mailbox, and I don't see anything in RFCs whether this should or shouldn't happen. I suppose it's safer that the mailbox is still closed? So that means it's a bit strange/unnecessary to specify "tagged OK response" in the RFC text.

3) Interaction of CONTEXT and SEARCHRES: 

SEARCH RETURN (SAVE MIN PARTIAL 2:10) 1:100

Is the $ value going to be 1 (MIN), 2:10 (PARTIAL), 1:10 (MIN+PARTIAL) or 1:100 (ALL)? I'm thinking 1:100 is the right answer here.

SEARCH RETURN (SAVE MIN UPDATE) 1:100

I suppose UPDATE should be ignored here and $ is 1. RFC 5267 does say pretty clearly "There is no interaction between UPDATE and any other return options"

4) Interaction of FUZZY and SEARCHRES: SEARCH RETURN (SAVE MIN RELEVANCY)

Well, this is probably clear enough that RELEVANCY shouldn't affect $ results? But the FUZZY RFC doesn't specify very clearly how RELEVANCY should interact with MIN, MAX or COUNT options. Maybe it could use errata to clarify? I suppose with MIN/MAX only the one relevancy should be returned, and with COUNT all the relevancies?

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 13 Mar 2020 13:43:51 +0000

Hi Timo,

> It's possible to use $ in the same search query that updates it, so the above search should preserve $. My code cleared $ before running the search, so the result was that $ was cleared.
Right. So "SEARCH RETURN (SAVE) $ " does nothing to the $ value.
> So I was going to test that $ is preserved if SEARCH/EXAMINE fails. Except selecting nonexistent mailbox closes the current mailbox. So then I thought testing with invalid SELECT options, like SELE…
Let me think about this, need more coffee :-).
> Is the $ value going to be 1 (MIN), 2:10 (PARTIAL), 1:10 (MIN+PARTIAL) or 1:100 (ALL)? I'm thinking 1:100 is the right answer here.

I think you found a real issue here. You are right that none of RFCs say 
anything about this case.

I think my implementation is doing 2:10 (PARTIAL), but I need to double 
check. The intent of is for SAVE to use what is returned or what would 
have been returned. One can argue that "1:10 (MIN+PARTIAL)" is also a 
valid choice here.

I don't think setting $ to 1:100 (ALL) is right here, because this 
client only wants to know PARTIAL and MIN data. Data about ALL messages 
might not even be available in the server, depending on how PARTIAL is 
constructed.

> I suppose UPDATE should be ignored here and $ is 1. RFC 5267 does say pretty clearly "There is no interaction between UPDATE and any other return options"
Agreed.
> Well, this is probably clear enough that RELEVANCY shouldn't affect $ results? But the FUZZY RFC doesn't specify very clearly how RELEVANCY should interact with MIN, MAX or COUNT options. Maybe it…

We probably need to do an update at some point, but submitting errata is 
sensible in this case.

Best Regards,

Alexey

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 13 Mar 2020 16:09:47 +0200

On 13. Mar 2020, at 15.43, Alexey Melnikov <[email-masked]> wrote:
> I think you found a real issue here. You are right that none of RFCs say anything about this case.
> 
> I think my implementation is doing 2:10 (PARTIAL), but I need to double check. The intent of is for SAVE to use what is returned or what would have been returned. One can argue that "1:10 (MIN+PARTIAL)" is also a valid choice here.
> 
> I don't think setting $ to 1:100 (ALL) is right here, because this client only wants to know PARTIAL and MIN data. Data about ALL messages might not even be available in the server, depending on how PARTIAL is constructed.

I guess this same question also applies to what happens even if MIN isn't used. My current code is saving 1:100, but that's mainly because of how the implementation happens to work right now (PARTIAL isn't very optimized).

I had also been thinking that maybe a client could work like:

a SEARCH RETURN (SAVE PARTIAL 1:10) TEXT stuff
..
b SEARCH RETURN (PARTIAL 11:20) $
..
c SEARCH RETURN (PARTIAL 21:30) $

But I suppose that's not really something clients would do.

Then again, if you do:

a SEARCH RETURN (SAVE COUNT PARTIAL 1:10) 1:100

COUNT would override PARTIAL here in any case, so $ would have to be 1:100.

I wonder if we should just disallow SAVE+PARTIAL combination?

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 13 Mar 2020 15:38:42 +0000

Hi Timo,

On 13/03/2020 14:09, Timo Sirainen wrote:
> On 13. Mar 2020, at 15.43, Alexey Melnikov <[email-masked] 
> <mailto:[email-masked]>> wrote:
>>
>>> Is the $ value going to be 1 (MIN), 2:10 (PARTIAL), 1:10 
>>> (MIN+PARTIAL) or 1:100 (ALL)? I'm thinking 1:100 is the right answer 
>>> here.
>>
>> I think my implementation is doing 2:10 (PARTIAL), but I need to 
>> double check. The intent of is for SAVE to use what is returned or 
>> what would have been returned. One can argue that "1:10 
>> (MIN+PARTIAL)" is also a valid choice here.
>>
>> client only wants to know PARTIAL and MIN data. Data about ALL 
>> messages might not even be available in the server, depending on how 
>> PARTIAL is constructed.
>
> I guess this same question also applies to what happens even if MIN 
> isn't used. My current code is saving 1:100, but that's mainly because 
> of how the implementation happens to work right now (PARTIAL isn't 
> very optimized).
>
This means save the current partial segment into $, so I think it should 
be 1:10
> But I suppose that's not really something clients would do.
Probably not. And if they do, they better know what it means!
> COUNT would override PARTIAL here in any case, so $ would have to be 
Yes. If this is surprising, then I think it should be clarified 
somewhere. If a client wants to retrieve COUNT, but save PARTIAL, it 
would have to do 2 commands.
> I wonder if we should just disallow SAVE+PARTIAL combination?

I would rather not. I think combination of SAVE and PARTIAL has a well 
specified semantics.

Best Regards,

Alexey

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 16 Mar 2020 13:52:32 +0200

On 13. Mar 2020, at 15.43, Alexey Melnikov <[email-masked]> wrote:
> We probably need to do an update at some point, but submitting errata is sensible in this case.

This is actually getting a bit confusing as well. The simple cases:

RETURN (ALL RELEVANCY) : return ALL relevancies
RETURN (MIN RELEVANCY) : return only the MIN's relevancy
RETURN (MAX RELEVANCY) : return only the MAX's relevancy
RETURN (PARTIAL n:m RELEVANCY) : return only the PARTIAL result's relevancies

The more confusing ones:

RETURN (MIN MAX RELEVANCY) : return MIN and MAX relevancies. But what if MIN=MAX, i.e. only one result? Is there 1 or 2 relevancies returned?
RETURN (MIN MAX PARTIAL n:m RELEVANCY) : Similar to above - what if MIN/MAX overlaps the PARTIAL results?
RETURN (RELEVANCY) : Maybe this should be an error?
RETURN (COUNT RELEVANCY) : Does this even make sense either?
RETURN (MIN MAX PARTIAL n:m COUNT RELEVANCY) : Should COUNT change anything?

Maybe MIN and MAX should always prepend and append the min/max relevancies, regardless of whether they overlap each others or PARTIAL. And in COUNT case I'd make it a no-op regarding RELEVANCY.