Subject: [lemonade] RFC 3516 (IMAP4 Binary Content Extension) Question

From: "Esh, Thomas D (Tom)" <[email-masked]>
Date: Mon, 13 Feb 2006 14:49:33 -0500

I have a question regarding RFC 3516 (IMAP4 Binary Content Extension).

What should be the value of a FETCH BINARY.SIZE on a multipart/mixed component?
The encoding types of the parts could vary.  Should the IMAP4 server return the size
as if all the components were decoded?

The same is true of a message/rfc822 component or any component that contains
other components.

Thanks
Tom

Thomas D. Esh               Email: [email-masked]
Lucent Technologies          Work: 614-367-4390
Room 4O05-1L               Mobile: 740-334-2319
6100 East Broad Street       Home: 740-321-1245
Columbus, OH  43213           MMS: [email-masked]

Our lives begin to end the day we become silent about 
things that matter. -- Martin Luther King, Jr.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Mon, 13 Feb 2006 20:43:27 +0000

> The same is true of a message/rfc822 component or any component 

This came up on the IMAP mailing list a little while ago - you may 
find the full discussion in the archives, but as I recall, the 
BINARY.SIZE for multipart/mixed is the same as its encoded size, 
because the multipart *itself* is encoded using an identity encoding.

It's the same for message/rfc822, etc. These all use 7BIT CTE, 
nothing else.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Mark Crispin <[email-masked]>
Date: Mon, 13 Feb 2006 12:57:50 -0800 (PST)

On Mon, 13 Feb 2006, Esh, Thomas D (Tom) wrote:
> What should be the value of a FETCH BINARY.SIZE on a multipart/mixed 
> component? The encoding types of the parts could vary.  Should the IMAP4 
> server return the size as if all the components were decoded?

Only terminal MIME parts (that is, those which are not MULTIPART or 
MESSAGE/RFC822) have an encoding other than 7BIT (in certain 
circumstances, a part of type MESSAGE/RFC822 may have an encoding of 
8BIT).  Thus, it is meaningless to FETCH BINARY.SIZE or BINARY on such 
parts.

I'm surprised that RFC 3516 does not forbid non-terminal nodes.  It 
should.

Note that BINARY does not take the same type of part specifier as BODY, In 
particular, you can not use any RFC 3501 section-text BODY parts (HEADER, 
TEXT, MIME, etc.) with BINARY.  Thus, there is no BINARY specifier for a 
MULTIPART directly under a message; the question only comes up with a 
nested MULTIPART.

So, in effect, there's a partial syntax prohibition of using BINARY on 
non-terminal parts.

Here is my advice to you:

1) In your client, do NOT attempt to FETCH BINARY or BINARY.SIZE for a 
non-terminal body part, because it may be forbidden in the future.  It 
does no good to use it; and in the case of BINARY.SIZE it's a complete 
waste of time since you already have that value from the BODYSTRUCTURE.

2) In your server, treat FETCH BINARY as equivalent to FETCH BODY, and 
FETCH BINARY.SIZE as a request for the number in the BODYSTRUCTURE, 
whenever the part specifier refers to a MULTIPART or MESSAGE/RFC822.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 13 Feb 2006 13:15:01 -0800

> I'm surprised that RFC 3516 does not forbid non-terminal nodes.  It should. 

And it will when the next revision gets published.  It's time to gather 
up an errata list and get going on this.  I need someone to document the 
issues that were raised in Vancouver as well.

--lyndon

---

From: Dave Cridland <[email-masked]>
Date: Sat, 18 Feb 2006 10:29:05 +0000

On Mon Feb 13 21:15:01 2006, Lyndon Nerenberg wrote:
> And it will when the next revision gets published.  It's time to 
> gather up an errata list and get going on this.  I need someone to 
> document the issues that were raised in Vancouver as well.

Looking through my INBOX:

1) LITERAL+ interaction. (I know it's generally felt that using 
LITERAL+ with APPEND is bad, but this isn't the case with 
CATENATE/MULTIAPPEND as well.)

2) Signalling whether uploads will always be rejected with 
UNKNOWN-CTE.

3) S/MIME and Binary mess. (Not an IMAP BINARY issue as such, but 
still needs addressing. I'm not clear on what the issues are, 
precisely, I assume S/MIME signs the representation and not a 
canonical form.)

4) Non-terminal MIME node handling.

5) When to issue UNKNOWN-CTE. (ie, before or after the literal 
transmission).

6) Section 4.4 ambiguity WRT content conversion.

I've a message dating from last May (2005-05-09 in fact) from you 
(Lyndon), subject "RFC3516", which might be useful - let me know if 
you need a copy, or if you'd like it forwarded to the list.

Is this a good enough start? Have I missed anything? (I've not looked 
through the Vancouver minutes, just added those items I could recall.)

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Sat, 18 Feb 2006 19:39:45 +0000

> 2) Signalling whether uploads will always be rejected with 

Arnt pointed out this is a little overbroad, which is quite true. 
This is not a situation we should encourage, but the fact remains 
that some IMAP BINARY implementations in the wild reject all attempts 
at an APPEND with a Binary CTE with UNKNOWN-CTE.

More specifically, some simply reject at the first use of a literal8 
in an APPEND command.

This is currently legal behaviour for RFC3516, so this point was 
merely suggesting that if a server only supports the BINARY FETCH 
items, there should be some mechanism to signal that to the client, 
so it doesn't attempt a binary upload.

For Lemonade, we can then choose to mandate "binary upload support" 
for Profile 2.

This may all be closely related to points 5 and 6, and indeed 
probably is - the points I've enumerated are merely those I could 
find in my INBOX and scatty memory, I've not attempted to sort them 
out.

Dave.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Sat, 18 Feb 2006 12:39:48 -0800

On Feb 18, 2006, at 2:29 AM, Dave Cridland wrote:

> 2) Signalling whether uploads will always be rejected with UNKNOWN- 
> CTE.

Why would the server not just do a dynamic conversion to a non-binary  
representation as it stores the message?  That's what sentence two of  
section 4.4 is all about.  I guess I don't understand why a server  
would be willing to translate in one direction (FETCH), but not the  
other (APPEND).

I realize that the second paragraph of §4.4 introduces an ambiguity.   
The intention was not to require the server to store the APPENDed  
data in a binary format.

--lyndon

---

From: Mark Crispin <[email-masked]>
Date: Sat, 18 Feb 2006 12:49:51 -0800 (PST)

On Sat, 18 Feb 2006, Lyndon Nerenberg wrote:
> I guess I don't understand why a server would be willing 
> to translate in one direction (FETCH), but not the other (APPEND).

FETCH is easy.  Just get the segment as normal, and if it has a CTE (which 
you get from the BODYSTRUCTURE) of BASE64 or QUOTED-PRINTABLE then apply 
the translation.

APPEND is hard.  The server actually has to parse the message, and 
transform it.

Unless, that is, the server is stupid and just accepts any old crap that 
is appended to it, and if binary content gets sent with a non-binary 
fetch, well the MUA/user deserved to lose.  That's the trouble with 
"unfunded mandates"; if there's a stupid thing to do that works for the 
98% case, it's all too easy to choose that instead of the right thing.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Feb 2006 15:18:52 +0100

Mark Crispin writes:
>> I guess I don't understand why a server would be willing to translate 
>> in one direction (FETCH), but not the other (APPEND).
>
> FETCH is easy.  Just get the segment as normal, and if it has a CTE 
> (which you get from the BODYSTRUCTURE) of BASE64 or QUOTED-PRINTABLE 
> then apply the translation.
>

I don't get that. IMAP servers contain enough parsing and transmormation 
smarts to handle FETCH BODYSTRUCTURE, FETCH BODY[1] and SEARCH BODY. 
APPEND with literal8 seems easy. It needs the same parsing code, plus 
b64 and/or q-p encoding functions, right?

Explain?

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Feb 2006 14:22:09 +0000

On Sun Feb 19 14:18:52 2006, Arnt Gulbrandsen wrote:
>>> I guess I don't understand why a server would be willing to 
>>> translate in one direction (FETCH), but not the other (APPEND).
>> 
>> FETCH is easy.  Just get the segment as normal, and if it has a 
>> CTE (which you get from the BODYSTRUCTURE) of BASE64 or 
>> QUOTED-PRINTABLE then apply the translation.
>> 
> I don't get that. IMAP servers contain enough parsing and 
> transmormation smarts to handle FETCH BODYSTRUCTURE, FETCH BODY[1] 
> and SEARCH BODY. APPEND with literal8 seems easy. It needs the same 
> parsing code, plus b64 and/or q-p encoding functions, right?
> 
> 
No. Handling 3516 for FETCH only means you can use the same MIME 
parser you've used for the past decade or so. Handling 3516 for 
APPEND means your MIME parser has to handle binary. Binary MIME is 
actually terribly easy to parse, but it has a fearsome reputation, 
and appears to have scared people for years - witness the statements 
at the top of RFC3030.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 19 Feb 2006 15:47:41 +0100

Hi Dave. I thought today is a Sunday?

Dave Cridland writes:
> APPEND means your MIME parser has to handle binary.

That's just-send-8 plus null bytes?

I wrote zero lines of code to handle that case ;) That's why I'm 
surprised when Mark says it's hard.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Sun, 19 Feb 2006 14:56:55 +0000

> Hi Dave. I thought today is a Sunday?
It is. :-/

> That's just-send-8 plus null bytes?
Not quite, it's infinite line length, too. Which isn't a problem, but 
"old" MIME parsers treat anything beyond 1000 octet line lengths as 
an error case.

> surprised when Mark says it's hard.

I don't think Mark's saying it *is* hard. Merely that it has an 
amount of effort, and intrusiveness, much higher than leaving out the 
APPEND side, hence some people have opted for skipping that.

Dave.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 19 Feb 2006 10:23:02 -0800

On Sun, 19 Feb 2006, Dave Cridland wrote:
> No. Handling 3516 for FETCH only means you can use the same MIME parser 
> you've used for the past decade or so. Handling 3516 for APPEND means your 
> MIME parser has to handle binary.

AND the message has to be transformed into a non-binary form when stored. 
All the counts in IMAP, starting with RFC822.SIZE, have to reflect this 
transformed form.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 19 Feb 2006 10:31:44 -0800

On Sun, 19 Feb 2006, Dave Cridland wrote:
> Not quite, it's infinite line length, too.

And tranformation to the non-binary form for default/ordinary IMAP 
fetches.  FETCH BINARY can be done on the fly from non-binary stored data. 
The reverse is much more costly (unacceptably costly) since it involves 
such things as RFC822.SIZE and BODYSTRUCTURE counts.

If your server returns BINARY as a C-T-E in BODYSTRUCTURE, or returns NUL 
in any non-BINARY FETCH, then it is non-compliant.  To do things right, 
you can't just accept any old text for APPEND and hope for the best; you 
have to validate the text and make certain that you can transform it into 
a non-BINARY form.

So, the delivery process has to do the work.  This is not a trivial task; 
but it's one that many people overlook.

-- Mark --