Subject: [imapext] client support for RFC 3516 (BINARY)?

From: Jamie Nicolson (倪志明) <[email-masked]>
Date: Fri, 5 Dec 2014 14:02:23 -0800

Anyone know of clients that support RFC 3516? It seems like it could be
particularly useful for mobile clients, to reduce both download size and
CPU usage for base64-decoding.

---

From: Dave Cridland <[email-masked]>
Date: Fri, 5 Dec 2014 22:42:40 +0000

I have a vague recollection that it's in Inky, http://inky.com/ - it's
something of a no brainer to support in a client, in any case. Servers have
to handle binary append, such is somewhat harder.

Dave.

---

From: Michael M Slusarz <[email-masked]>
Date: Fri, 05 Dec 2014 16:56:20 -0700

> to handle binary append, such is somewhat harder.

Horde/IMP uses binary, if available, to FETCH binary data.  As Dave  
mentions, this is trivial to do in a client (if BINARY FETCH data is  
available, you just skip all the code client-side that does the  
transfer-decoding).

Might as well add some additional notes here, as they may possible be  
of interest to future implementers...

Current (or at least recent-ish versions) servers do several different  
things when they encounter bad encoded data.  See, e.g.:

http://markmail.org/message/anldcux5fhqz6m65

Dovecot used to issue NIL ... newer versions still issue NIL but with  
a UNKNOWN-CTE response.  Cyrus does the best it can to decode and  
returns that data.

This can be seen as a limitation of the current BINARY spec, since it  
isn't obvious which (if any) of these responses are correct.  Speaking  
as a client author, I just care about the UNKNOWN-CTE response though  
since that leaves open the possibility of fetching the unencoded part  
and dealing with broken transfer-encoding the way that I want to.

BINARY APPENDs are a different story though.  It is broken on multiple  
servers, and in several different ways.  At a minimum, versions of  
UW-IMAP, Cyrus, and Sun Java System Messaging have problems.  IIRC,  
one of the issues is that these servers have issues with the literal8  
format.  It's so widespread that we've disabled LITERAL+ entirely when  
appending BINARY data in order to prevent unneeded traffic.  (We  
normally use LITERAL+ for small APPENDs as a compromise between the  
need to catch errors before sending the data - RFC 4549 - and the  
benefits of pipelining).

My thought originally was, if BINARY was available, then always APPEND  
using that because it prevents us from having to scan the output data  
on the client side to determine whether the data really was binary  
(null, long-lines) or not.  But the above described server issues  
encountered ended that experiment fairly quickly.

michael

___________________________________
Michael Slusarz [[email-masked]]

---

From: Jamie Nicolson (倪志明) <[email-masked]>
Date: Fri, 5 Dec 2014 17:48:29 -0800

> to handle binary append, such is somewhat harder.

My reading of binary append is just that the server needs to accept the
<literal8> syntax and accept data that contains NULs. The spec allows for
modifying the CTE at append time, but discourages it in section 6 ("this
should only be done when it is absolutely necessary"). So it seems like
supporting binary append should be fairly straightforward.

Are you perhaps referring to the re-encoding of binary data with a
conforming CTE when it is FETCHed with a (non-binary) BODY or
BODYSTRUCTURE? This requirement puzzles me because it seems to be
retroactive, and independent of whether BINARY append is supported. We
already have binary data in our system, and we make no attempt to re-encode
it for IMAP download now, even without implementing BINARY. Is the spec
assuming that, prior to implementing BINARY, there was no way for binary
data to get into the system, and so only a server that supports BINARY
append needs to worry about re-encoding it for download?

---

From: Dave Cridland <[email-masked]>
Date: Sat, 6 Dec 2014 09:40:12 +0000

I think every implementation I knew of recodes binary mime to 8bit on
append. But yeah, I recall that Google lets you slip binary past it
sometimes, and it gives that back to clients. I don't think that's uncommon.

Dave.

---

From: Jan Kundrát <[email-masked]>
Date: Sat, 06 Dec 2014 11:38:35 +0100

On Friday, 5 December 2014 23:02:23 CEST, Jamie Nicolson (倪志明) wrote:
> Anyone know of clients that support RFC 3516?

Trojitá will happily use BINARY whenever it's available for all of its 
fetching operations where it's legal, and unless the client needs the raw, 
encoded data e.g. for signature validation.

Because the composer always produces 7bit messages, BINARY is not used for 
APPENDs.

With kind regards,
Jan

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: Jamie Nicolson (倪志明) <[email-masked]>
Date: Mon, 8 Dec 2014 17:15:24 -0800

Thanks for the responses. A couple of follow-up questions:

1. This thread (
http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-August/001803.html)
implies that fetching non-terminal MIME parts, while allowed by the spec,
is strongly discouraged because it may require rewriting CTE headers and
other on-the-fly modifications. Are people typically restricting the use of
BINARY to terminal parts, or are they issuing things like FETCH BINARY[] ?

2. Some folks indicated they use BINARY append. Does mean you're also using
a binary CTE in your messages, or that you're just using literal8 syntax to
upload messages that are 7bit or 8bit?

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 09 Dec 2014 14:44:07 +0100

On Tuesday, 9 December 2014 02:15:24 CEST, Jamie Nicolson (倪志明) wrote:
> Are people typically restricting the use of

In case of Trojita, we don't request a BINARY on a non-leaf part because I 
don't think doing that should be "legal". As I originally said in that 
e-mail thread, this is probably worth an errata update to the RFC, IMHO.

With kind regards,
Jan

---

From: Ken Murchison <[email-masked]>
Date: Tue, 09 Dec 2014 09:01:29 -0500

Hi Michael,

On 12/05/2014 06:56 PM, Michael M Slusarz wrote:
> UW-IMAP, Cyrus, and Sun Java System Messaging have problems.

I'm unaware of any issues with BINARY APPEND in Cyrus.  Have you opened 
a bug for this?

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Michael M Slusarz <[email-masked]>
Date: Tue, 09 Dec 2014 10:12:17 -0700

Quoting Ken Murchison <[email-masked]>:

>> BINARY APPENDs are a different story though.  It is broken on  
>> multiple servers, and in several different ways.  At a minimum,  
>> versions of UW-IMAP, Cyrus, and Sun Java System Messaging have  
>> problems.
>
> I'm unaware of any issues with BINARY APPEND in Cyrus.  Have you  
> opened a bug for this?

Hi Ken,

I can't reproduce BINARY APPEND issues in current Cyrus code, at least  
with the current Debian 7 version (v2.4.16-Debian-2.4.16-4+deb7u2).  I  
believe that's why I never filed a bug since the issue now appears  
fixed.

But I can confirm that old 2.2 versions of Cyrus have this issue.   
E.g., the current Debian 6 version shows this behavior:

* OK bernard Cyrus IMAP4 v2.2.13-Debian-2.2.13-19+squeeze3 server ready
[...]
B APPEND "INBOX" (\seen) ~{200+}
B NO [UNKNOWN-CTE] Can not process the binary data

My notes additionally indicate that certain versions of Cyrus, instead  
of responding with a NO + UNKNOWN-CTE, will simply issue a BYE and  
disconnect instead.

michael

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 9 Dec 2014 15:54:24 -0800

On Dec 9, 2014, at 5:44 AM, Jan Kundrát <[email-masked]> wrote:

> In case of Trojita, we don't request a BINARY on a non-leaf part because I don't think doing that should be "legal". As I originally said in that e-mail thread, this is probably worth an errata update to the RFC, IMHO.

Yes, 3516 was supposed to restrict binary fetches to terminal MIME parts.  I was sure the final version of the draft had that text, and it still baffles me how it got through last call without me noticing, but there you go.

I think the changes to add that restriction might be a bit too substantive for an errata.  (Barry, are you lurking  out there?)  Also, others have complained about parts of the document that are ambiguous or just not clear, so maybe it's time for an update.  

--lyndon

---

From: Dave Cridland <[email-masked]>
Date: Wed, 10 Dec 2014 08:54:05 +0000

On 9 Dec 2014 23:54, "Lyndon Nerenberg" <[email-masked]> wrote:
> Yes, 3516 was supposed to restrict binary fetches to terminal MIME
parts.  I was sure the final version of the draft had that text, and it
still baffles me how it got through last call without me noticing, but
there you go.

The RFC does seem to be written entirely with leaf nodes in mind, though it
never actually makes the restriction. On the basis that all implementations
reject noon leaf sections anyway, making it an errata seems the most
pragmatic solution, whatever the process says.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 11 Dec 2014 15:15:17 +0100

On Wednesday, December 10, 2014 12:54:24 AM CEST, Lyndon Nerenberg wrote:
> Yes, 3516 was supposed to restrict binary fetches to terminal 
> MIME parts.  I was sure the final version of the draft had that 
> text, and it still baffles me how it got through last call 
> without me noticing, but there you go.

Yes, I was the guilty reviewer. You told me you would or wanted to add a 
leaf-only rule and I then wrote unit tests only for leaf parts. I should 
have checked the wording again later but didn't, lazy slob that I am.

Things fall through cracks.

I don't think it really matters. We've survived eleven point something 
years without noticeable interop problems.

Arnt

---

From: Michael M Slusarz <[email-masked]>
Date: Fri, 19 Dec 2014 12:06:51 -0700

Quoting Lyndon Nerenberg <[email-masked]>:

> I think the changes to add that restriction might be a bit too  
> substantive for an errata.  (Barry, are you lurking  out there?)   
> Also, others have complained about parts of the document that are  
> ambiguous or just not clear, so maybe it's time for an update.

A real world BINARY issue, that was brought to my attention today, is  
the list behavior at vger.kernel.org.  I was told that incoming  
messages have mailing list information appended to them, regardless of  
the transfer-encoding of the content.

So a message with single base64 encoded part is sent like:

=====
Content-Type: text/plain
Content-Transfer-Encoding: base64

<base64 data>

<Plaintext Mailing List Trailer>
=====

Issuing a FETCH BINARY[] on this data produces inconsistent results.   
A NO [UNKNOWN-CTE] is possible, as well as trying to base64 decode as  
much as possible while stripping the rest, and finally base64 decoding  
and leaving the plaintext as is.

If 3516 is getting updated, might be useful to have a more consistent  
definition of what it means to be "unable to trasfer decode" so that  
this data is displayed consistently across servers.  i.e. Is it ok to  
do a best-guess decoding?

Additionally, it might be useful to add implementation note that a  
client should fallback to using FETCH BODY[] if the server returns NO.  
  The MUA that the original reporter is using (I don't know what it  
is) is simply showing a blank body for these kind of messages when a  
FETCH BINARY[] is unsuccessful.

michael

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 19 Dec 2014 19:15:47 +0000

As I recall, the base64 spec says to skip unknown input bytes, so if there are no padding bytes (=), that list adds well-defined data. If there is padding, I don't know what the spec days.

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Fri, 19 Dec 2014 11:58:03 -0800 (PST)

> Issuing a FETCH BINARY[] on this data produces inconsistent results.  A NO 
> [UNKNOWN-CTE] is possible, as well as trying to base64 decode as much as 
> possible while stripping the rest, and finally base64 decoding and leaving 
> the plaintext as is.

As a purist I would argue the IMAP server should never have accepted the 
message to begin with, but most implementations take it on faith the 
content matches what the metadata says is in there.

In this case, I'm not sure what to do.  The problem here is not a BINARY 
issue as such, but more of a general issue as to what a server should do 
when it discovers invalid data part way through a transaction.  The 
protocol in general doesn't provide a framework for dealing with this type 
of scenario.

My strong preference is for the server fully validate the message when 
it's being injected into the store, and reject these sorts of 
non-complaint messages at that time.  Most any IMAP store these days is 
already parsing the MIME structure at injection time, so it's really not a 
lot of extra work to validate the stated encodings match the body content 
as you're copying the message through.

> definition of what it means to be "unable to trasfer decode" so that this 
> data is displayed consistently across servers.  i.e. Is it ok to do a 
> best-guess decoding?

Absolutely not.  You must follow the directive in the MIME header (in this 
case, perform a bese64 decode).

RFC3501 section 7.4.2 talks about ENcoding binary content as base64, but 
since this transformation cannot fail there is no provision for a failure 
mode.  For BINARY, that's not the case, as your example points out.  And 
it was a situation I never anticipated when I wrote 3516 (most likely 
because I am quite fascist about rejecting invalid content up front before 
it becomes a problem).

At the moment, I'm not sure there is any way out of this other than for a 
server supporting BINARY to validate the CTEs at injection time, and 
reject any messages with content encoding violations.  I don't think there 
is any way for a server that discovers bad base64 in the middle of a 
BINARY FETCH to recover gracefully.

I *adamantly* oppose any notion of passing through anything that fails the 
CTE decoding, as this presents a wonderful opportunity for malicious code 
injection into a naive client implementation.  In light of that security 
concern alone, I don't think there is any alternative but to add a "server 
MUST validate the body content correctly decodes before transmitting the 
decoded content to the client" or words to that effect to the BINARY 
specification.  If this is done at message injection time, the overhead 
will be minimal, and would allow the server to refuse to accept the 
message.  If implementers are insistent about accepting such malformed 
content, then we will have to add something like an [INVALID-CTE-SYNTAX] 
respose code so the server can send back "foo NO [INVALID-CTE-SYNTAX]" 
when the runtime check fails.

> Additionally, it might be useful to add implementation note that a client 
> should fallback to using FETCH BODY[] if the server returns NO.  The MUA that 
> the original reporter is using (I don't know what it is) is simply showing a 
> blank body for these kind of messages when a FETCH BINARY[] is unsuccessful.

This is so stunningly obvious it doesn't deserve mentioning.

--lyndon

---

From: Jan Kundrát <[email-masked]>
Date: Sat, 20 Dec 2014 11:44:20 +0100

On Friday, 19 December 2014 20:58:03 CEST, Lyndon Nerenberg wrote:
> As a purist I would argue the IMAP server should never have 
> accepted the message to begin with

At the time you do it, your users won't be able to receive mail from the 
Linux Kernel Mailing Lists. (Actually, just a random portion of these 
e-mails when the original MUA decided to use the base64 CTE. That's gonna 
be fun.)

The vger's configuration is of course broken, and unfortunately I've been 
forwarded a message where they said that they aren't going to fix this 
after 20 years of operation. I believe Linus has a word for this attitude.

>> Additionally, it might be useful to add implementation note 
>> that a client should fallback to using FETCH BODY[] if the 
>> server returns NO.  The MUA that the original reporter is using 
>> (I don't know what it is) is simply showing a blank body for 
>> these kind of messages when a FETCH BINARY[] is unsuccessful.

I'm not sure whether you received that report from the same user as I did. 
If that is the case, the MUA is Trojita and there was a bug where the error 
in FETCH operation wasn't propagated to the GUI in a clear manner, but a 
spinner was kept running. It now at least shows an error message.

> This is so stunningly obvious it doesn't deserve mentioning.

Is it? There are just two widely used CTEs which require decoding, the 
base64 and the quoted-printable, and RFC2045 contains a pretty strong 
language against developing any additional CTEs.

Is there a realistic assumption that there's an IMAP server which:

- declares support for BINARY,
- cannot decode base64 or quoted-printable encoding,
- can handle that situation gracefully, without crashing,
- and client's local decoder would be able to cope with the malformed 
content?

By all means, sure, it will improve robustness to retry with a FETCH after 
a BINARY failed. I'm just saying that it was sufficiently non-obvious at 
the time I added BINARY support, and even now I have my doubts on whether 
my local decoder is going to be any better than e.g. Dovecot's.

On a quick look at its source, the decoder library which Trojita uses (Qt's 
QByteArray::fromBase64) appears to handle garbage in base64 data in 
accordance with RFC2045 which calls for ignoring any character that cannot 
be decoded. It doesn't appear to have any special handling of the == 
padding, which is in strong constrast to Python where the base64 module 
stops decoding when it hits the first padding. I wonder which approach is 
better.

With kind regards,
Jan

---

From: Michael M Slusarz <[email-masked]>
Date: Sat, 20 Dec 2014 10:26:24 -0700

Quoting Lyndon Nerenberg <[email-masked]>:

> At the moment, I'm not sure there is any way out of this other than  
> for a server supporting BINARY to validate the CTEs at injection  
> time, and reject any messages with content encoding violations.  I  
> don't think there is any way for a server that discovers bad base64  
> in the middle of a BINARY FETCH to recover gracefully.

For performance reasons, I would assume most servers are scanning for  
RFC 822 structure but not parsing the actual MIME data until a BINARY  
command is received.  So requiring rejection at injection time would  
be quite a performance hit, with no guarantee that the client will  
ever use BINARY.

> I *adamantly* oppose any notion of passing through anything that  
> fails the CTE decoding, as this presents a wonderful opportunity for  
> malicious code injection into a naive client implementation.  In  
> light of that security concern alone, I don't think there is any  
> alternative but to add a "server MUST validate the body content  
> correctly decodes before transmitting the decoded content to the  
> client" or words to that effect to the BINARY specification.  If  
> this is done at message injection time, the overhead will be  
> minimal, and would allow the server to refuse to accept the message.  
>  If implementers are insistent about accepting such malformed  
> content, then we will have to add something like an  
> [INVALID-CTE-SYNTAX] respose code so the server can send back "foo  
> NO [INVALID-CTE-SYNTAX]" when the runtime check fails.

I would think that many/most server implementers would take this  
approach instead, since it is much easier.  And I'm not sure I would  
disagree for performance reasons, not to mention that this appears to  
be the way that many of the BINARY servers currently operate.

As far as INVALID-CTE-SYNTAX ... do we need an additional response  
code?  I feel that UNKNOWN-CTE works here also (although the semantics  
might be a bit off).

 From a client perspective it just needs to know that data does exist  
in that part but it is due to some sort of decoding error, rather than  
the fact that the part doesn't exist for example, that is causing the  
data to not be displayed.  The actual reason the data can't be decoded  
is irrelevant for the client; it just needs to know that it may be  
able to display the content somehow using it's own algorithms to deal  
with broken data.

michael