Subject: Re: [imapext] Mail submission over IMAP

From: Brandon Long <[email-masked]>
Date: Wed, 1 Aug 2012 17:13:52 -0700

On Mon, Jul 23, 2012 at 2:11 AM, Jan Kundrát <[email-masked]> wrote:
[snip]

>
> Please tell that to Google who still produces invalid BODYSTRUCTURE
> responses on e-mails with attached message/rfc822 parts (last checked a few
> months ago, anyway). As a client developer, I can assure you that it is NOT
> FUN to try to guess what five fields out of a sequence of twenty mandatory
> fields are missing today. I won't willingly inflict the same on any other
> programmer unless I'm in a really, really nasty mood.

I'm not sure its that complicated, but if you'd like the current
information on that...

I tried to fix this almost a year ago.  The bug was that our parser
considers message/rfc822 attachments with content-disposition: attachment
to be a blob, not a multipart container.

Simple problem, just have it always consider it a multipart container.

Except that doesn't work if the attachment is content-transfer-encoded.
 Note that its against spec to content-transfer-encode a message/rfc822
attachment, but some people do it anyways.  And there is some comment that
the future utf8 message headers would require you to cte a message/rfc822
attachment, but let's ignore that for now.

So, let's only treat it as a multipart container if its not encoded.  We
could also add a step somewhere else to "fix" messages to not be cte, but
that doesn't really happen at the parser level.  We could make a copy of
the attachment and decode it, but its not clear if that will work correctly
for body part handling in IMAP either (how do you ask for just the text of
a sub-part that's encoded at higher level?)

Anyways, turns out, this breaks a whole bunch of assumptions in our code,
but fix most of them.  Run into fact that during certain rebuilding steps,
we've now shifted what's considered an attachment in a major way on some
messages, making old parsed messages and new parsed messages be very
dissimilar.  Rollback change again.

Why I say its not complicated is because our "breakage" is in whether or
not the message/rfc822 is considered a container or not.  I don't like that
we're broken, but IMAP is essentially forcing us to handle messages in a
specific format for that protocol, which is in direct competition with the
more common use case for our customers (which is dragging and dropping an
".eml file" into an email, which they expect to be an attachment). I'd like
to fix it, but even if we correct our parser, I'm not sure when we'd get
around to re-parsing all of our existing messages to fix this case (we
often re-parse messages, but we've never had a change which requires
re-computing what's an attachment, doing this would require some immense
resources).

We could, theoretically, re-compose the full message and re-parse it in
order to correctly produce a BODYSTRUCTURE response, but the extra
resources necessary to always do that is completely infeasible for however
few clients out there haven't already worked around our brokenness.

As always, I'm sorry Gmail was not designed to satisfy IMAP from the get go
(no, really I am, since my team has to make the two mostly work together),
but it wasn't, so compromises are made.

In any case, the bug is still open, the fix is now a very known quantity,
and when the backend team gets the resources to work on it, it'll probably
get done... maybe next year.

Brandon

Brandon