Subject: Re: [imapext] GMail's extensions: X-GM-THRID immutability

From: Bron Gondwana <[email-masked]>
Date: Sun, 07 Jul 2013 21:47:05 +1000

On Fri, May 10, 2013, at 07:53 AM, Brandon Long wrote:

- Any plans for support of the SORT extensions?

We have only the single SORT order in everything we do, so no.  Again,
we can't load the entire folder into memory just to SORT it on the
server, and we're not going to pre-compute and store each possible
order.  You'll note there is only the one order in all of the Gmail
interfaces.  Our marketing would say, don't sort, search!

I'm actually really surprised at this.  In Cyrus we worked on the
assumption that a mailbox might have a million messages.  We use this
data structure for the in-memory index state:

struct index_map {

    modseq_t modseq;

    modseq_t told_modseq;

    uint32_t uid;

    uint32_t recno;

    uint32_t system_flags;

    uint32_t user_flags[4];

    unsigned int isseen:1;

    unsigned int isrecent:1;

};

Which is 8 + 8 + 4 + 4 + 4 + 32 + (roundup) => 64 bytes per message.

So a million messages required 64 megabytes of RAM.

The more interesting issue of course is populating the new details
fast.  Offlineimap does:

FETCH 1:* (UID INTERNALDATE FLAGS)

So the only question is - do you store INTERNALDATE in an easily
fetchable location?

Cyrus USED to not store it in the index, so you had to fetch and parse
the date header from the cache.  That sucked.  Now it's in cyrus.index
which is 128 bytes per message, so reads in fast, even for "cold"
users.

Bron.

--
Bron Gondwana
[email-masked]

---

From: Brandon Long <[email-masked]>
Date: Mon, 8 Jul 2013 13:20:30 -0700

On Sun, Jul 7, 2013 at 4:47 AM, Bron Gondwana <[email-masked]> wrote:

> **
> We have only the single SORT order in everything we do, so no.  Again, we
> can't load the entire folder into memory just to SORT it on the server, and
> we're not going to pre-compute and store each possible order.  You'll note
> there is only the one order in all of the Gmail interfaces.  Our marketing
> would say, don't sort, search!
>
>
> assumption that a mailbox might have a million messages.  We use this data
> structure for the in-memory index state:
>
> struct index_map {
>     modseq_t modseq;
>     modseq_t told_modseq;
>     uint32_t uid;
>     uint32_t recno;
>     uint32_t system_flags;
>     uint32_t user_flags[4];
>     unsigned int isseen:1;
>     unsigned int isrecent:1;
> };
>

So, on a 64GB machine, you can only handle 1000 users.  That's not enough,
plus loading that 64MB off disk is expensive in IOPs , and we're not using
local disk, so transfer from one machine to another.

And we have accounts with 10M+ messages, if the user happens to try to sort
their All Mail folder, they could seriously impact the performance of other
users on that box.

>
> The more interesting issue of course is populating the new details fast.
> Offlineimap does:
>
> So the only question is - do you store INTERNALDATE in an easily fetchable
> location?
>

offlineimap hasn't done that in a couple years, since we asked them to stop
doing it (just confirmed that with the current github source). That data
for us requires loading the full meta-data for the message (ie, ENVELOPE
level response).  offlineimap doesn't actually use the information, so it
was just consuming resources for no reason.  No other client made that
request.

If more clients did request it, we'd need to make the data easier to access.

>
> Cyrus USED to not store it in the index, so you had to fetch and parse the
> date header from the cache.  That sucked.  Now it's in cyrus.index which is
> 128 bytes per message, so reads in fast, even for "cold" users.
>

I don't know if cyrus.index is a flat file or what, but re-writing large
files isn't cheap.  As you go up the ladder from a flat file to a btree to
a generic "db" like table where you need to deserialize each row... it adds
up.

Brandon