Subject: [Extra] imap4rev2: RFC3516 BINARY

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 08 Nov 2018 06:34:38 -0500

This is one of the open issues from imap4rev2 - which extensions should be part of the mandatory baseline. 
 
This one might need some clarification from Alexey - I'm not sure why it's not trivially obviously a good thing to include. There was some stuff about "ability to fetch the decoded size" discussed, but I didn't capture good notes. 
 
Arguments for and against, or additional discussion, in response to this please. 
 
Cheers, 
 
Bron. 
 
-- 
 Bron Gondwana, CEO, FastMail Pty Ltd 
 [email-masked]

---

From: Brandon Long <[email-masked]>
Date: Fri, 9 Nov 2018 15:17:53 -0800

BINARYMIMEWe ran into some issues trying to implement BINARY, and also
didn't see a large number of clients which supported it (looking back, we
did this work mostly in 2014, so a while back).  My memory is a bit sketchy
of everything we ran into.

If I recall, we had some questions with the standard, ie what happens if
you request binary for non-leaf branches of the MIME tree (section-binary
implies that there's something special about what you can request, but it's
actually just the section-part).  Ie, if you ask for BINARY on a
message/rfc822, are we supposed to decode each individual part in that
multipart?  Again, we could have just responded NO with unknown-cte, I
guess.  There was also the question of what to do with data we'd munged, ie
we have heuristics for extracting base64 data when there are multiple
sections of it, or where a mailing list appended a non-encoded footer on
base64 data, etc, we probably would have just sent the data we'd extracted
instead of giving an unknown-cte response, for example.

We also had some issues with binary append which I don't recall, though
that you can just not support with some minor annoyance (why it didn't have
a separate CAPABILITY for APPEND instead of just allowing the server to say
NO, I dunno).  If I recall, we were worried about what version to store and
what to vend to non BINARY requests, and how that interoperated.  I guess
this wouldn't have mattered that much, we already have a "buggy" APPEND by
the standard, since the standard basically says you have to accept whatever
crap they APPEND, even if not an email message, and we have some base level
requirements for message formatting.

Overall, it was a part of our investigation into using binary more in
emali, ie cte binary and smtp (BDAT and BINARYMIME).  We implemented BDAT,
but our testing basically showed very few clients or servers that could
handle cte binary with embedded nulls.

Perhaps a cleaned up more strictly specified BINARY could be included,
otherwise I think you'll get a lot of unknown-cte responses from folks who
have to claim support but can't.
For one, its not clear that you can stream a base64 decode, at the least
you'd need to do two passes, one to verify correctness of the base64 data,
and the second to stream decode it.

Brandon

On Thu, Nov 8, 2018 at 3:34 AM Bron Gondwana <[email-masked]> wrote:

> This one might need some clarification from Alexey - I'm not sure why it's
> not trivially obviously a good thing to include.  There was some stuff
> about "ability to fetch the decoded size" discussed, but I didn't capture
> good notes.
>
> _______________________________________________
> Extra mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/extra
>

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 10 Nov 2018 03:12:51 +0200

I remember the BINARY specification wasn't very clear for all the troublesome cases. I mostly implemented it by thinking what made the most sense to me. Also I vaguely remember there being some talk about updating the RFC related to these? I couldn't find a new one now, so I guess that plan died.

I was also thinking whether COMPRESS (or TLS compression) might make BINARY irrelevant. I thought I remembered it compressing nicely enough, but then again now googling it the first hit is "Why does base64-encoded data compress so bad?"

All in all, I agree with Brandon. If BINARY is included, it needs to be defined better. Here are links to my old troubles with it:

http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-December/001908.html <http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-December/001908.html>
http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-August/001802.html <http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-August/001802.html>
http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-June/001788.html <http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-June/001788.html>
http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-June/001787.html <http://mailman13.u.washington.edu/pipermail/imap-protocol/2012-June/001787.html>

> On 10 Nov 2018, at 1.17, Brandon Long <blong=[email-masked]> wrote:
> 
> BINARYMIMEWe ran into some issues trying to implement BINARY, and also didn't see a large number of clients which supported it (looking back, we did this work mostly in 2014, so a while back).  My memory is a bit sketchy of everything we ran into.
> 
> If I recall, we had some questions with the standard, ie what happens if you request binary for non-leaf branches of the MIME tree (section-binary implies that there's something special about what you can request, but it's actually just the section-part).  Ie, if you ask for BINARY on a message/rfc822, are we supposed to decode each individual part in that multipart?  Again, we could have just responded NO with unknown-cte, I guess.  There was also the question of what to do with data we'd munged, ie we have heuristics for extracting base64 data when there are multiple sections of it, or where a mailing list appended a non-encoded footer on base64 data, etc, we probably would have just sent the data we'd extracted instead of giving an unknown-cte response, for example.
> 
> We also had some issues with binary append which I don't recall, though that you can just not support with some minor annoyance (why it didn't have a separate CAPABILITY for APPEND instead of just allowing the server to say NO, I dunno).  If I recall, we were worried about what version to store and what to vend to non BINARY requests, and how that interoperated.  I guess this wouldn't have mattered that much, we already have a "buggy" APPEND by the standard, since the standard basically says you have to accept whatever crap they APPEND, even if not an email message, and we have some base level requirements for message formatting.
> 
> Overall, it was a part of our investigation into using binary more in emali, ie cte binary and smtp (BDAT and BINARYMIME).  We implemented BDAT, but our testing basically showed very few clients or servers that could handle cte binary with embedded nulls.
> 
> Perhaps a cleaned up more strictly specified BINARY could be included, otherwise I think you'll get a lot of unknown-cte responses from folks who have to claim support but can't.
> For one, its not clear that you can stream a base64 decode, at the least you'd need to do two passes, one to verify correctness of the base64 data, and the second to stream decode it.
> 
> Brandon
> 
> On Thu, Nov 8, 2018 at 3:34 AM Bron Gondwana <[email-masked] <mailto:[email-masked]>> wrote:
>   [email-masked] <mailto:[email-masked]>
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/extra <https://www.ietf.org/mailman/listinfo/extra>

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 20 Nov 2018 14:15:44 -0800

I have no objections to adding BINARY fetch of leaf MIME parts, although 
it would be helpful to hear that client vendors found that useful prior 
to adoption.

I oppose adding BINARY fetch of non-leaf parts to IMAP4rev2 -- that gets 
complicated quickly. Having IMAP4rev2 require support for 
binary-content-transfer-encoding on APPEND would also be problematic as 
it creates several new cases with potential interoperability problems.

		- Chris

On 8 Nov 2018, at 3:34, Bron Gondwana wrote:
> This is one of the open issues from imap4rev2 - which extensions 
> should be part of the mandatory baseline.
>
> This one might need some clarification from Alexey - I'm not sure why 
> it's not trivially obviously a good thing to include. There was some 
> stuff about "ability to fetch the decoded size" discussed, but I 
> didn't capture good notes.
>