Subject: [MORG] Initial set of documents to work on

From: Alexey Melnikov <[email-masked]>
Date: Sat, 18 Oct 2008 23:00:30 +0100

Hi,
 From the BOF in Dublin and discussions I had with various people after 
that it seems there is a desire to form a WG. In order to get the 
process started I would like to first discuss documents that would be 
worked on first. Speaking as an individual, I would like to propose the 
following set of documents (to be worked on in the specified order)

1). draft-karp-morg-sortdisplay-00.txt
2). draft-melnikov-imapext-status-in-list-00.txt and possibly status 
counts per message context (Peter Coates has a draft)
3). New collations - space ignore, numeric ignoring punctuation, etc. (me)
4). Sorting by relevancy (Timo)
5). draft-gulbrandsen-imap-inthread-02.txt
6). draft-melnikov-imapext-multimailbox-search-03.txt

Comments?

_______________________________________________
MORG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/morg
Note Well: http://www.ietf.org/maillist.html

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2008 12:46:55 +0200

Excuse my late reply; it was difficult to find something to say.

Alexey Melnikov writes:
> worked on first. Speaking as an individual, I would like to propose 
> the following set of documents (to be worked on in the specified 
> order)

And the only thing I can say is that I don't understand the list.

This list ought to be a set of tools to reach some vision, right? But I 
don't understand how these documents combine to work towards something, 
and what vision that something is.

My lack of understanding may be because I wasn't in Dublin. (I couldn't 
even join via jabber, I was blocked except at that exact time.) Still, 
some background for the list would help me to understand.

> 6). draft-melnikov-imapext-multimailbox-search-03.txt

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 21 Oct 2008 11:59:28 +0100

Arnt Gulbrandsen wrote:

>> From the BOF in Dublin and discussions I had with various people 
>> after that it seems there is a desire to form a WG. In order to get 
>> the process started I would like to first discuss documents that 
>> would be worked on first. Speaking as an individual, I would like to 
>> propose the following set of documents (to be worked on in the 
>> specified order)
>
> This list ought to be a set of tools to reach some vision, right? But 
> I don't understand how these documents combine to work towards 
> something, and what vision that something is.
>
> My lack of understanding may be because I wasn't in Dublin. (I 
> couldn't even join via jabber, I was blocked except at that exact 
> time.) Still, some background for the list would help me to understand.

 From my slides for the MORG BOF in Dublin:

Goals:
improve ability to find messages in an IMAP mailstore
1) Don't make things worse while doing that (think about extension 
interaction)
2) Should work for diverse clients
3) Mobile devices, web clients, desktop clients

And this is the updated list of drafts for the original charter I 
proposed <http://www.ietf.org/mail-archive/web/morg/current/msg00024.html>

Does this help?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2008 13:49:52 +0200

In view of Alexey's response just now.

> 1). draft-karp-morg-sortdisplay-00.txt

A fine draft and I'll commit to implementing it if Dan gets it to RFC, 
but it's not clear to me how it improves the ability to find messages.

> counts per message context (Peter Coates has a draft)

Again, not clear to me, etc.

> 3). New collations - space ignore, numeric ignoring punctuation, etc. (me)

Ditto.

> 4). Sorting by relevancy (Timo)

This is about returning an ordered list of search responses, rather than 
an unordered set? That sounds like a good idea.

> 5). draft-gulbrandsen-imap-inthread-02.txt

Hm, I have updated that, but not published, and it expired. Expect a revision.

I think this helps, in so far as it provides tools to perform some 
really common searches.

> 6). draft-melnikov-imapext-multimailbox-search-03.txt

Multimailbox search is obviously a great help with finding things. Of 
course it interacts with 4) above. Solvable.

Overall, I'd say that 4-6 could solve two of the four big findability 
problems. The other two are:

1. Searching for messages whose address fields contain a specific 
address or domain.

2. Searching for messages by word instead of substring. Substring 
serches are easy to implement, but IMO fairly useless. Prefix searches 
would be better, word searches even better, and IMO perfection would be 
word searches where the server may stem.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2008 17:04:20 +0200

Hi,

please post the appended internet-draft update.

The automatic upload tool has failed for me in the past hour or so; the 
upload suffers from Spinning Cursor Syndrome. I'll leave the web 
browser until it gives up, though. It seems to be remarkably patient.

Arnt

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 21 Oct 2008 18:22:58 +0300

> counts per message context (Peter Coates has a draft)

Sounds good. Should status counters also specify interaction with
NOTIFY?

> 4). Sorting by relevancy (Timo)

4) almost requires having a collation that is free to return search
results in pretty unspecified ways. I think people also want to just
type their search keys and expect the server to magically figure out
what the user really wants to see as results.

So if the server is allowed to perform magic, is there really a use for
3) collations? I doubt many users would change the collation themselves.
Perhaps a few power users with special power clients might, but I don't
really see it happening for "normal" people.

Or is there some use case I'm totally missing?

> 6). draft-melnikov-imapext-multimailbox-search-03.txt

I still think 6) is best to be done with some kind of virtual views, but
it needs more thinking. Good thing it's last on the list. :)
_______________________________________________
MORG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/morg
Note Well: http://www.ietf.org/maillist.html

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 21 Oct 2008 18:37:42 +0300

How is the thread algorithm selected with SORT? ESORT return option
probably?

THREADROOT should probably say clearly what to do when thread root is a
dummy. I guess that can be considered non-existing and instead all the
dummy's children are returned as roots.

Also I was hoping that THREAD=REFS would do the thread root sorting
differently for THREAD command. I wrote this about it before:

Replace step (4) with something like: Initialize each thread root's
sort-date to non-existing. For each (non-dummy) message get its
INTERNALDATE and find its thread root. If the thread root's sort-date is
non-existing or the INTERNALDATE is higher than sort-date, update
sort-date to INTERNALDATE. Finally sort all the thread roots by their
sort-date. (Because all threads have at least one non-dummy message,
there should be no non-existing sort-dates left when sorting.)

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Oct 2008 17:52:08 +0200

Timo Sirainen writes:
> How is the thread algorithm selected with SORT? ESORT return option probably?

Dave?

> THREADROOT should probably say clearly what to do when thread root is 
> a dummy. I guess that can be considered non-existing and instead all 
> the dummy's children are returned as roots.

Doesn't it say so? "Has no extant parents" or something?

> differently for THREAD command. I wrote this about it before:

I agree what THREAD=REFERENCES does isn't optimal. But I also like the 
situation where THREAD=REFS has no code of its own, it's 100% a subset 
of THREAD=REFERENCES.

A show of hands, please? What's best: Sorting a better way, or being 
able to reuse more code?

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 21 Oct 2008 17:04:57 +0100

>Sounds good. Should status counters also specify interaction with
Yes (but I don't think we need to mention this explicitly).

>what the user really wants to see as results.
After our discussion in Dublin I am not convinced that the final result 
of this work is going to be a collation, because relevancy scores are 
also of interest to users.
But we will see.

>3) collations?
I think there is, but not necessarily in IMAP.
However I think I have IMAP specific use cases for each one of them 
(e.g. telephone number search).

>I doubt many users would change the collation themselves.
This is perfectly fine, at this point I am more interested in picking 
workitems rather than specific solutions.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Oct 2008 18:21:21 +0100

> but it's not clear to me how it improves the ability to find messages.

It makes server side sort usable, so it helps clients to find messages 
(as in they need to do less work).

> Again, not clear to me, etc.

This reduces bandwidth and the number of roundtrips, so I think it is in 
scope.

>> 3). New collations - space ignore, numeric ignoring punctuation, etc. 

Better (or rather more suitable) collations improve ability to find 
messages, thus in scope.

> This is about returning an ordered list of search responses, rather 
> than an unordered set?

I think so, but details are yet to be determined.

> That sounds like a good idea.
>

Sure.

> 1. Searching for messages whose address fields contain a specific 

It might be late for me, but I am not sure what is missing. Can you give 
an example?

> would be better, word searches even better, and IMO perfection would 
> be word searches where the server may stem.

Speaking as an individual, I am all in favour of considering this for 
inclusion. But I think I need to see more details (not necessarily a draft).

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 22 Oct 2008 20:54:56 +0300

> >> 4). Sorting by relevancy (Timo)
..
> inclusion. But I think I need to see more details (not necessarily a draft).

I think this is highly relevant to 4). I'm mainly interested in getting
4) implemented using existing search engines and about none of them can
do substring searches. I also don't know if 4) is really even useful for
other purposes. I wasn't really planning on implementing my own
relevancy scoring algorithm.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 21 Oct 2008 17:08:31 +0100

>> How is the thread algorithm selected with SORT? ESORT return  
It could do, I suppose, although that means a normative dependency on  
the CONTEXT document, unless ESORT is extracted from it. Means that  
you'd emit ESEARCH responses, of course.

I'm easy, but I'd note that if we are to head this route, it seems  
odd that selection of thread algorithms is different in SORT and  
SEARCH than in THREAD itself, and moreover it's also different to  
comparator selection.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
MORG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/morg
Note Well: http://www.ietf.org/maillist.html

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Oct 2008 14:30:30 +0200

Alexey Melnikov writes:
>> A fine draft and I'll commit to implementing it if Dan gets it to 
>> RFC, but it's not clear to me how it improves the ability to find 
>> messages.
>

OK. Fine argument.

> This reduces bandwidth and the number of roundtrips, so I think it is 
> in scope.

Reducing bandwidth usage and number of roundtrips are surely fine goals, 
but they seem to be orthogonal to finding messages.

Unless you think _any_ performance improvement helps people find stuff faster?

> Better (or rather more suitable) collations improve ability to find 

I don't agree, but we don't have enough detail to argue usefully. Write 
something about these collations we can argue more usefully.

> It might be late for me, but I am not sure what is missing. Can you 
> give an example?

Compare with Sieve. Sieve has «address :localpart :is "arnt"». IMAP only 
has the equivalent of «header :contains "arnt"», «from arnt».

If you take a small risk, you can say «from arnt@», but that may or may 
not match messages «From: arnt (Arnt Gulbrandsen)». If you take a 
bigger risk, you can say «from <arnt@», but you can't trust that to 
match «From: [email-masked]» on all servers.

I can write a draft for this, if desired, based on my code. I would add 
two or three search keys. Exact address, exact domain and 
domain-or-subdomain. (Exact localpart would be a possibility too and 
perhaps easy to implement along with the rest, but there doesn't seem 
to be much demand for that.)

>> serches are easy to implement, but IMO fairly useless. Prefix 
>> searches would be better, word searches even better, and IMO 
>> perfection would be word searches where the server may stem.
>

I suggest that Timo write the introduction to a draft and post on the list.

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 26 Oct 2008 18:39:46 +0000

Arnt Gulbrandsen wrote:

>> It makes server side sort usable, so it helps clients to find 
>> messages (as in they need to do less work).
>
> OK. Fine argument.
>
>>>> 2). draft-melnikov-imapext-status-in-list-00.txt and possibly 
>>>> status counts per message context (Peter Coates has a draft)
>>>
> Reducing bandwidth usage and number of roundtrips are surely fine 
> goals, but they seem to be orthogonal to finding messages.
>

I meant improving "ability to find", which includes both new ways to 
limit search results, as well as to transfer them quicker ;-).

> I don't agree, but we don't have enough detail to argue usefully. 
> Write something about these collations we can argue more usefully.

Ok, after Minneapolis perhaps. Too busy at the moment.

>>> Overall, I'd say that 4-6 could solve two of the four big 
>>> findability problems. The other two are:
>>>
> Compare with Sieve. Sieve has «address :localpart :is "arnt"». IMAP 
> only has the equivalent of «header :contains "arnt"», «from arnt».
>
> If you take a small risk, you can say «from arnt@», but that may or 
> may not match messages «From: arnt (Arnt Gulbrandsen)». If you take a 
> I can write a draft for this, if desired, based on my code. I would 
> add two or three search keys. Exact address, exact domain and 

Sure.

> I suggest that Timo write the introduction to a draft and post on the 

I agree.

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 27 Oct 2008 00:03:00 +0200

On Oct 21, 2008, at 6:52 PM, Arnt Gulbrandsen wrote:

> I agree what THREAD=REFERENCES does isn't optimal. But I also like  
> the situation where THREAD=REFS has no code of its own, it's 100% a  
> subset of THREAD=REFERENCES.
>

Anyone?

I use two IMAP clients: Evolution and Apple Mail. Both sort the thread  
roots by their latest received message, not by the sent-date of the  
thread root. I'm guessing many (most? all?) other clients do the same.  
I've received bug reports about Dovecot's THREAD=REFERENCES  
implementation because it doesn't sort by the latest received message  
and users think they're not receiving new messages because they show  
up in the middle of the mailbox. So this sorting is something users  
really want and I'll have to do it one way or another, currently it's  
THREAD=X-REFERENCES2 but I'd prefer to make it a standard.

I also don't think the code reuse issue is all that big of a deal.  
It's replacing a single step in the threading algorithm. Dovecot  
needed less than 40 lines more to implement it (although that'll  
increase if I add some optimizations).