Subject: Factoring out Content-Disposition (i123), was: Content-Disposition (new issue?)

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Aug 2008 23:01:30 +0200

Hi,

we discussed this issue during the WG meeting in Dublin 
(<http://jabber.ietf.org/logs/httpbis/2008-07-29.txt>), and it seems 
there was some support for the idea of taking Content-Disposition out of 
HTTP/1.1.

My own proposal was to clearly specify how Content-Disposition works in 
existing HTTP clients, and to move that into a separate, 
standards-track, spec.

However, an essential part of supporting Content-Disposition is to 
understand a subset of RFC 2231, which defines  extensions to MIME 
header parameters (<http://greenbytes.de/tech/webdav/rfc2231.html>), for 
instance specifying how to transport non-ASCII characters in C-D's 
filename parameter. *That* mechanism is independent of C-D, and could be 
usable for other specs, such as the link header spec (which uses 
parameter notation as well).

So we actually need *two* documents:

1) A clarification of what parts of RFC 2231 apply to HTTP/1.1 
(essentially a profile), and

2) A specification of Content-Disposition in HTTP, built on top of RFC 
2183 and the profile above.

I've got a proposal ready for the first document -- it's available at 
<http://greenbytes.de/tech/webdav/draft-reschke-rfc2231-in-http-latest.html>. 
The plan is to submit a -00 draft soon, but I'll be more than happy to 
incorporate feedback that arrives before I do.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Aug 2008 18:43:25 +0200

Julian Reschke wrote:
> ...
> ...

FYI:

[email-masked] wrote:
 > A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
 >
 > 	Title           : Applicability of RFC 2231 Encoding to Hypertext 
Transfer Protocol (HTTP) Headers
 > 	Author(s)       : J. Reschke
 > 	Filename        : draft-reschke-rfc2231-in-http-00.txt
 > 	Pages           : 10
 > 	Date            : 2008-08-15
 >
 > By default, message header parameters in Hypertext Transfer Protocol
 > (HTTP) messages can not carry characters outside the ISO-8859-1
 > character set.  RFC 2231 defines an escaping mechanism for use in
 > Multipurpose Internet Mail Extensions (MIME) headers.  This document
 > specifies a profile of that encoding suitable for use in HTTP.
 >
 > A URL for this Internet-Draft is:
 > http://www.ietf.org/internet-drafts/draft-reschke-rfc2231-in-http-00.txt
 >
 > Internet-Drafts are also available by anonymous FTP at:
 > ftp://ftp.ietf.org/internet-drafts/
 >
 > Below is the data which will enable a MIME compliant mail reader
 > implementation to automatically retrieve the ASCII version of the
 > Internet-Draft.
 >
 >
 > ------------------------------------------------------------------------
 >
 > _______________________________________________
 > I-D-Announce mailing list
 > [email-masked]
 > https://www.ietf.org/mailman/listinfo/i-d-announce
 > Internet-Draft directories: http://www.ietf.org/shadow.html
 > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 15 Aug 2008 14:54:07 -0500

Julian Reschke wrote:
> By default, message header parameters in Hypertext 
> Transfer Protocol  (HTTP) messages can not carry characters 
> outside the ISO-8859-1 character set. RFC 2231 defines an 
> escaping mechanism for use in Multipurpose Internet Mail 
> Extensions (MIME) headers.  This document specifies a 
> profile of that encoding suitable for use in HTTP.

During the IETF meeting, what was the result of the discussions about
Unicode support in HTTP? Looking at the IRC log, it looked like the
discussion was leaning towards allowing UTF-8 in an otherwise-unencoded form
in headers (applications should start accepting unencoded UTF-8 but should
avoid sending it right now). If that is the way things are going to go, a
general RFC 2231 profile for HTTP seems counterproductive.

RFC 2231 + UTF-8 is an especially bad interchange format for text since it
requires over 9 bytes per letter for the vast majority of people's native
languages. Plus, there are no features for language tagging (needed for CJK
languages), BIDI (needed for middle-eastern languages), or accessibility
(for users of screen readers). IMO, the best thing to do is to keep
language-sensitive text out of HTTP as much as possible by recommending that
applications transfer language-sensitive text in entity bodies as much as
possible. Really, it is only suitable for short, language-neutral  strings
like (file and IRI) path fragments.

Nitpicks:

The draft references Unicode 4.0 indirectly through RFC3629. It would be
better to allow implementations to use any later versions, or at least the
current version, 5.1.

I don't see the point of requiring ISO-8859-1. ISO-8859-1 can only encode a
very small number of languages that are used by a small minority of people
(who just happen to be over-represented in standards committees). Advocating
ISO-8859-1 also seems to be the opposite of what was discussed at the IETF
meeting (AFAICT from the logs).

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Aug 2008 23:00:59 +0200

> general RFC 2231 profile for HTTP seems counterproductive.

As far as I recall, we played with the idea, but we were unsure whether 
this would be possible to do.

RFC 2231 already is implemented in two of the big four UAs, and it works 
around the whole issue, so I think making it easier to use (by clearly 
stating how it works in HTTP) is a good idea in any case.

> requires over 9 bytes per letter for the vast majority of people's native

Making UTF-16 support mandatory could help her, but I'm not sure how 
widespread support for that is (recall I'm trying to document what 
several UAs already do and have been doing for a long time). Will keep 
this in mind when writing test cases.

> (for users of screen readers). IMO, the best thing to do is to keep

RFC 2231 *does* include language tagging. WRT BIDI I'm no expert, but I 
thought Unicode has something to say here? And could you clarify the 
accessibility concern please?

> like (file and IRI) path fragments.

That's something I agree with. For instance, WebDAV doesn't suffer from 
these kinds of problems because anything that is text actually travels 
in entity bodies as XML.

That being said, you can't always avoid it, such as in 
Content-Disposition or Slug.

> better to allow implementations to use any later versions, or at least the

Yes, that's a nit, isn't it :-).

> meeting (AFAICT from the logs).

I originally want to mandate UTF-8 only, but people pointed out 
(rightfully), that any HTTP software already needs to understand 
ISO-8859-1, so it really doesn't make a difference.

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 15 Aug 2008 14:02:54 -0700

> general RFC 2231 profile for HTTP seems counterproductive.

That is not the way things are going.  More likely is reduction to
US-ASCII and reiterate that human-readable content does not belong
in the message envelope.

....Roy

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 15 Aug 2008 17:33:16 -0500

Julian Reschke wrote:
> Brian Smith wrote:
> > RFC 2231 + UTF-8 is an especially bad interchange format for text 
> > since it requires over 9 bytes per letter for the vast majority of 
> > people's native
> 
> Making UTF-16 support mandatory could help her, but I'm not 
> sure how widespread support for that is (recall I'm trying to 
> document what several UAs already do and have been doing for 
> a long time). Will keep this in mind when writing test cases.

I don't think UTF-16 support is worthwhile. I think it isn't an issue if the
intention is to support only very short, non-prose text like filenames (see
below). It seems there is some agreement that HTTP headers should not
contain human-oriented text. My concern is that having a separate standard
for RFC2231 in HTTP will promote the idea of human-oriented text in headers
instead of discouraging it.

> > language-sensitive text out of HTTP as much as possible by 
> > recommending that applications transfer language-sensitive text in 
> > entity bodies as much as possible. Really, it is only suitable for 
> > short, language-neutral  strings like (file and IRI) path fragments.
> 
> That's something I agree with. For instance, WebDAV doesn't 
> suffer from these kinds of problems because anything that is 
> text actually travels in entity bodies as XML.
> 

Since the primary (only?) use case for RFC2231 in HTTP is the
Content-Disposition header, why not just fold this into the spec. that you
are writing for Content-Disposition? URI references are already
ASCII-encoded IRIs, and Atom's Slug header field already has its own
mechanism for handling non-ASCII text.

> > languages. Plus, there are no features for language tagging (needed 
> > for CJK languages), BIDI (needed for middle-eastern languages), or 
> > accessibility (for users of screen readers). IMO, the best 
> > thing to do is to keep
> 
> RFC 2231 *does* include language tagging. WRT BIDI I'm no 
> expert, but I thought Unicode has something to say here?
> And could you clarify the accessibility concern please?

Language tagging, BIDI, and accessibility features are not really necessary
for the specific case of filenames. Those issues come into play when you try
to define a general-purpose mechanism for supporting human-oriented text.
For example, RFC 2231 only allows a language tag for the entire parameter
value, but doesn't provide a means of handling mixed-language text.

> 
> > Nitpicks:
> > 
> > The draft references Unicode 4.0 indirectly through 
> > RFC3629. It would be better to allow implementations to use
> > any later versions, or at least the current version, 5.1.
> 

Yes, but this issue seems to always come up when specifications reference
Unicode documents.

> > I don't see the point of requiring ISO-8859-1. ISO-8859-1 can only 
> > encode a very small number of languages that are used by a small 
> > minority of people (who just happen to be over-represented in 
> > standards committees). Advocating
> > ISO-8859-1 also seems to be the opposite of what was 
> > discussed at the IETF meeting (AFAICT from the logs).
> 
> I originally want to mandate UTF-8 only, but people pointed 
> out (rightfully), that any HTTP software already needs to 
> understand ISO-8859-1, so it really doesn't make a difference.

Judging from Roy's response, it looks like software won't have to understand
more than ASCII, though they will have to tolerate non-ASCII bytes
(presumably, regardless of whether those bytes can be decoded into valid
characters in any encoding). Historically, ISO-8859-1 seems to be very
difficult for implementers to get right since Windows-1252 and other similar
encodings are often sent as ISO-8859-1.

- Brian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 15 Aug 2008 17:45:19 -0500

> mechanism for handling non-ASCII text.

Or more to the point, TEXT* is defined as RFC2047 charset-encoded values,
so defining Content-Disposition filename as TEXT* solves the ascii/iso/uft8
puzzle.

The issue with filename is that it can (and often does) vary from the
resource name, e.g. download.aspx v.s. thatdocument.pdf.

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 16 Aug 2008 04:43:08 +0200

Brian Smith wrote:

> RFC 2231 + UTF-8 is an especially bad interchange format
> for text since it requires over 9 bytes per letter

The length is no obstacle for HTTP, and as you write later,
we are anyway talking about relatively short strings.  For
some languages legacy charsets will be "better" than UTF-8
wrt "compression", but I think interoperability is more
relevant for our purposes.

There is no way to jump from "raw Latin-1" to "raw UTF-8"
in HTTP/1.1 headers, any mixtures would be a horrible mess.

If the WG meeting had a coherent transition strategy, e.g.,
"stick to Latin-1 in HTTP/1.1, do UTF-8 later in HTTP/1.2",
or "deprecate Latin-1 in HTTP/1.1 now, introduce UTF-8 in
HTTP/1.1 later", I'd like to see precise minutes about it.

JFTR, again, I think we need a clear transition strategy.
But so far we don't have it.

> there are no features for language tagging

Of course there are.  Raw UTF-8 doesn't offer this, unless
you try the NOT RECOMMENDED obscure u+E00?? language tags.

> BIDI (needed for middle-eastern languages)

All charsets needing this, not limited to UTF-8, offer it.
The gibbous RFC 2231 percent-encoding doesn't change this.

> it is only suitable for short, language-neutral strings

Do you propose to remove the optional [language] element
in the draft ?  It's a possibility, but some lines above
you said language tagging is essential.

> RFC3629.

Strong NAK.  STD 63 is not, repeat NOT, bound to some
specific Unicode version.  In a parallel universe where
the Unicode Consortium tried to redefine UTF-8 they'd
be disappointed when STD 63 sticks to the definition 
as it was Unicode 4.  But that is bad science fiction.

And it doesn't affect the set of assigned code points,
UTF-8 can do anything up to u+10FFF as specified in 
STD 63.  Other non-IETF UTF-8 specifications are less
relevant for our purposes.  (As you see I believe in
bad science fiction, after ISO 29500.  Therefore it's
IMO perfect to reference STD 63).

> I don't see the point of requiring ISO-8859-1.

See above, so far all proposals to ditch Latin-1 didn't
make it.  As long as that doesn't change Latin-1 is the
only permitted form of any non-ASCII octets in HTTP/1.1
headers.

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 16 Aug 2008 05:12:57 +0200

Brian Smith wrote:

> URI references are already ASCII-encoded IRIs

That would be a downref in 2616bis, because 3987 and IDNA
are only at PS at the moment.  I think it's best to say that
URI references are what is specified in STD 68.  Not some
HTML5 disease, XML LEIRI, ooXML gobbledegook, or what else.

> Atom's Slug header field already has its own mechanism for
> handling non-ASCII text.

Never heard of that...  RFC 5023, some percent-encoded UTF-8
with (ugh, there it is again) RFC 2616 LWS.  That's fine, it
is for a completely new header field.  

BTW, RFC 5023 references RFC 3629, this does _*NOT*_ limit
slugtext to Unicode 4.0.

> For example, RFC 2231 only allows a language tag for the
> entire parameter value, but doesn't provide a means of
> handling mixed-language text.

AFAIK this is not the case in RFC 2231, each piece can have
its own charset and language, please check.  But admittedly
Julian's draft permits only one piece.

> Historically, ISO-8859-1 seems to be very difficult for 
> implementers to get right since Windows-1252 and other
> similar encodings are often sent as ISO-8859-1.

Yeah.  OTOH if you'd really believe in Latin-1, it has some
small advantages:  It's short (one code point => one octet),
any HTTP/1.0 or HTTP/1.1 software is supposed to be able to 
handle it, no matter how old it is, and the chances that all
interested parties support the 191 glyphs to display Latin-1
are excellent.  

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Aug 2008 09:45:27 +0200

>> mechanism for handling non-ASCII text.

(to Brian) The interoperability problems with Content-Disposition 
motivated me to work on this, right. But putting the RFC2231 profile for 
HTTP into the same spec as a discussion of Content-Disposition would 
mean conflating separate issues. Furthermore, RFC2231-style encoding 
could be used in other headers as well, such as in the title parameter 
for the Link header (which clearly should support I18N, but can't be 
sent in the entity body).

> so defining Content-Disposition filename as TEXT* solves the ascii/iso/uft8

Doesn't work for me. We *know* that RFC2231-encoding already is in use, 
and that 2 out of 4 UAs have been supporting it for a long time.

Why invent something new? How do you deploy it?

> resource name, e.g. download.aspx v.s. thatdocument.pdf.

Yes. That's one of the reasons Content-Disposition is useful.

BR, Julian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Sat, 16 Aug 2008 02:50:45 -0500

> only permitted form of any non-ASCII octets in HTTP/1.1

I'm becoming very confused.  RFC 2616 is very explicit;

    The TEXT rule is only used for descriptive field contents and values
    that are not intended to be interpreted by the message parser. Words
    of *TEXT MAY contain characters from character sets other than ISO-
    8859-1 [22] only when encoded according to the rules of RFC 2047
    [14].

        TEXT           = <any OCTET except CTLs,
                         but including LWS>

So, we have a clear definition of where and when and how non-8859-1
characters are permitted, in spite of your's and Julian's claims in his
recent draft.  I really wish people would be more accurate in their
assertions.

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Sat, 16 Aug 2008 02:56:33 -0500

> Why invent something new? How do you deploy it?

Ok, color me confused; RFC 2616 is quite a bit older than your draft.

> Yes. That's one of the reasons Content-Disposition is useful.

Ack.  I don't think that solving the encoding issue is tangential to the
problem of a persistent local name for a resource.  Although I would hope
we can kill C-D's assertion that a resource is inline or not, that has and
forever will be b/shit.  What a user agent chooses to render is it's own
prerogative.

Bill

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Aug 2008 09:57:21 +0200

> contain human-oriented text. My concern is that having a separate standard

I agree that it's a good thing to avoid human-readable test in headers. 
But I really don't see how to always follow that rule.

> instead of discouraging it.

RFC2231 encoding already *is* used in HTTP, and it's used in a situation 
where the data *can't* be in the entity body.

Clarifying RFC2231 for use in HTTP has the main purpose to get the two 
UA vendors who don't "get" it to finally implement it correctly.

> value, but doesn't provide a means of handling mixed-language text.

Yes, that's a restriction.

> Yes, but this issue seems to always come up when specifications reference

If you feel strongly about it, you may want to start working on getting 
RFC3629 revised.

> encodings are often sent as ISO-8859-1.

That's true. Not sure what point you're trying to make, though? Do not 
mention ISO-8859-1 because some implementors confuse it with different 
encodings?

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Aug 2008 10:00:35 +0200

> recent draft.  I really wish people would be more accurate in their

I recommend that you look at the mailing list discussions for recent 
discussions about which headers actually *do* use TEXT. See 
<http://tools.ietf.org/wg/httpbis/trac/ticket/111> and 
<http://greenbytes.de/tech/httpbis/issue-111.xhtml>.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Aug 2008 10:18:55 +0200

William A. Rowe, Jr. wrote:
>> Doesn't work for me. We *know* that RFC2231-encoding already is in 
>> use, and that 2 out of 4 UAs have been supporting it for a long time.
>>

But RFC2616 is not clear about whether it allows RFC2047-style encoding 
in parameter values. Again, see 
<http://tools.ietf.org/wg/httpbis/trac/ticket/111> and 
<http://greenbytes.de/tech/httpbis/issue-111.xhtml>.

Furthermore, you may have missed the fact that my draft doesn't define 
anything new; it just re-states a subset of RFC2231 (which BTW does 
predate RFC2616), which already is implemented in several UAs.

Turning it around: do you have any evidence of HTTP clients supporting 
RFC2047 encoding in parameter values? If yes, which of the encoding 
methods defined in <http://tools.ietf.org/html/rfc2047#section-4>? With 
what character sets?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 16 Aug 2008 10:21:14 +0200

William A. Rowe, Jr. wrote:

>> As long as that doesn't change Latin-1 is the only 
>> permitted form of any non-ASCII octets in HTTP/1.1
>> headers.
 
> I'm becoming very confused.

Sorry, I should have written that I'm talking about
raw / unencoded octets (bytes, decimal 128..255).

As soon as you encode charsets using these octets,
e.g., with RFC 2047 / 2231 techniques, you can of
course use other charsets.

In the case of the "slugtext" encoding you can only
use (raw) UTF-8 input arriving at percent-encoded
(ASCII) output in the HTTP header field.

The RFC 2047 / 2231 encodings also arrive at ASCII
output for input in any charset.  Output limited to
bytes decimal 32..126 is no problem wrt HTTP, that
is a proper subset of ASCII, and ASCII is a proper 
subset of Latin-1 => "good".

However raw unencoded UTF-8 is not a proper subset
of Latin-1 => "bad".  

Last but not least raw unencoded Latin-1 is "ugly",
especially when it is something else (windows-1252
or worse), as Brian said.

 Frank

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 17 Aug 2008 11:02:03 +0200

> <http://greenbytes.de/tech/httpbis/issue-111.xhtml>.

And also the rules in RFC2047 on when encoding MAY be applied, which
rules out RFC2047 encoding in most HTTP uses of TEXT, at least in the
2616 BNF..

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 Aug 2008 10:23:35 -0500

Julian Reschke wrote:
> Brian Smith wrote:
> > Judging from Roy's response, it looks like software won't have to 
> > understand more than ASCII, though they will have to tolerate 
> > non-ASCII bytes (presumably, regardless of whether those 
> > bytes can be decoded into valid characters in any encoding).
> > implementers to get right since Windows-1252 and other similar
> That's true. Not sure what point you're trying to make, 
> though? Do not mention ISO-8859-1 because some implementors 
> confuse it with different encodings?

Yes.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 Aug 2008 11:46:21 -0500

Julian Reschke wrote:
> >> Content-Disposition header, why not just fold this into the spec. 
> >> that you are writing for Content-Disposition? URI references are 
> >> already ASCII-encoded IRIs, and Atom's Slug header field 
> already has 
> >> its own mechanism for handling non-ASCII text.
> 
> (to Brian) The interoperability problems with 
> Content-Disposition motivated me to work on this, right. But 
> putting the RFC2231 profile for HTTP into the same spec as a 
> discussion of Content-Disposition would mean conflating 
> separate issues. Furthermore, RFC2231-style encoding could be 
> used in other headers as well, such as in the title parameter 
> for the Link header (which clearly should support I18N, but 
> can't be sent in the entity body).

Like I've said before, the Link header field is more trouble than it is
worth. The i18n problems with the title attribute are not even the biggest
problem with it. Links can be sent in entity bodies, even for media types
like image/png that do not have their own linking mechanism; see AtomPub's
media link entries and WebDAV's analogous property mechanism.

> Doesn't work for me. We *know* that RFC2231-encoding already 
> is in use, and that 2 out of 4 UAs have been supporting it 
> for a long time.

I know you are not trying to be misleading, but "2 out of 4" is definitely
an exaggeration. There are way more than four different web browsers in
common use and I'm not sure that Opera is even in the top four when you
count mobile browsers (especially Docomo's browser, Nokia's browser, and
NetFront).

I'm thinking of writing a submitting for Firefox that decodes the
Content-Disposition filename parameter as escaped UTF-8 if it is not
RFC2231-encoded. I am interested in your test cases that demonstrate IE's
behavior for escaped UTF-8 filenames. I am especially interested in the East
Asian test cases where IE decoded the filename incorrectly; I have not been
able to duplicate that problem in IE6 or IE7. I am also curious if you have
found any changes to IE8 beta 1's behavior.

Thanks,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 17 Aug 2008 19:00:08 +0200

> worth. The i18n problems with the title attribute are not even the biggest

I disagree. It's totally useful, and it seems lots of people agree with 
  Mark's current draft.

> media link entries and WebDAV's analogous property mechanism.

I don't see how that helps for the cases the Link header tries to 
handle. Unless you want to mandate PROPFIND or a similar metadata 
discovery mechanism.

> count mobile browsers (especially Docomo's browser, Nokia's browser, and

OK, two out of the four most widely used desktop browsers.

> RFC2231-encoded. I am interested in your test cases that demonstrate IE's

Well, that would make the parsing non-compliant with the specs. Are you 
aware of any pages that actually use that format?

> found any changes to IE8 beta 1's behavior.

I've never been able to reproduce this on my own systems, but I did have 
Korean customers complaining *and* Microsoft admitting there's no 
reliable way to do it in IE (because the behavior relies on local 
settings the server can't see).

WRT IE8beta1: I checked some time ago, and the behavior seemed to be the 
same as in previous releases.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 Aug 2008 23:09:02 -0500

Julian Reschke wrote:
> Brian Smith wrote:
> > I'm thinking of writing a [patch] for Firefox that decodes the 
> > Content-Disposition filename parameter as escaped UTF-8 if 
> > it is not RFC2231-encoded. I am interested in your test cases
> > that demonstrate IE's
> 
> Well, that would make the parsing non-compliant with the 
> specs. Are you aware of any pages that actually use that format?

When building test cases based on some KB articles I found today, I did
finally find some bad behavior of IE in the following situations:

* When the filename parameter is longer than 150 bytes (as few as 17+
encoded characters), the filename gets truncated and otherwise mangled. (In
your upcoming Content-Disposition draft, you might want to provide some
advice on minimum/maximum file name lengths.)

* When the filename has a non-ASCII extension, IE doesn't decode the
filename. 

Because of these two issues, I'm not sure it is worthwhile to try to get
Firefox to handle Content-Disposition in a way that is consistent with
Internet Explorer. I will wait to see if IE8 beta 2 removes the above two
limitations and/or supports RFC 2231.

I didn't reproduce the locale-sensitivity part of the problem but I found
some KB articles about it:

   http://support.microsoft.com/kb/436616/ja (Japanese)
   http://support.microsoft.com/kb/943570/ja (Japanese)
   http://support.microsoft.com/kb/895084/ko (Korean)
   http://support.microsoft.com/kb/832930    (English)
   http://support.microsoft.com/kb/816868    (English)
   http://support.microsoft.com/kb/933133    (English)

KB436616 is a support article for ASP.NET. It says should use RFC2231
encoding for Japanese filenames, but acknowledges that no version of
Internet Explorer supports RFC2231. Instead, you can try to encode the
filename using percent-encoded UTF-8; however, sometimes Internet Explorer
will decode the filename using Shift-JIS instead of UTF-8. You can configure
ASP.NET 2.0 to encode the filename in a different encoding (Shift-JIS in
this case) to work around this problem, but there is no mechanism provided
to detect what encoding IE will use to decode.

KB943570 is a support article for Internet Explorer 6.0 and 7.0 on Windows
XP/Vista/2003. It also acknowledges the problem but doesn't discuss RFC2231
at all. The suggested workarounds are: Do not use non-ASCII characters in
filenames; configure your (ASP.NET 2.0) server to encode filenames using
Shift-JIS instead of UTF-8. No other workarounds are presented, and no
advise is offered for determining what encoding Internet Explorer will use
for decoding.

It looks like IE always uses the same encoding for Content-Disposition that
it uses for query strings; when "Send UTF-8 URLs" is checked, it will use
UTF-8 for Content-Disposition and will encode URLs in a RFC3987-compliant
manner; otherwise, it will use the user's selected default encoding for
query strings and Content-Disposition, and it will not be compliant with
RFC3987 unless the user's default encoding is UTF-8. 

It looks like Content-Disposition, like Content-Location, is something that
applications should avoid whenever possible.

(The issues with the Link header field are best described in another
thread.)

Regards,
Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 Aug 2008 23:45:33 -0500

[ I sent this just to you because I think the important point was already
clarified on the list by Julian.]

Frank Ellermann wrote:
> That would be a downref in 2616bis, because 3987 and IDNA are 
> only at PS at the moment.  I think it's best to say that URI 
> references are what is specified in STD 68.  Not some

I meant simply that RFC2231 encoding isn't needed or desirable for putting
IRIs in headers, because the standard way of processing them is to encode
them as URIs.

> > For example, RFC 2231 only allows a language tag for the entire 
> > parameter value, but doesn't provide a means of handling 
> > mixed-language text.
> 

No, it is a limit explicitly stated in RFC 2231 in section 4.1: "Language
and character set information only appear at the beginning of a given
parameter value. Continuations do not provide a facility for using more than
one character set or language in the same parameter value."

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 18 Aug 2008 00:53:33 -0500

> But so far we don't have it.

Agreed. 

> you try the NOT RECOMMENDED obscure u+E00?? language tags.

Even with RFC2231 encoding the Unicode language tagging mechanism would
still be needed for mixed-language text. 

> Do you propose to remove the optional [language] element in 
> the draft ?  It's a possibility, but some lines above you 
> said language tagging is essential.

Language tagging is needed for a general-purpose mechanism. However, in the
specific case of filenames, the UA should and will display the filename as
it will be presented by the operating system. That means a Japanese user
will see a Chinese-language filename using Japanese glyphs if the operating
system's locale is Japanese. Similarly, AFAIK operating systems do not
preserve BIDI formatting codes in filenames, so the UA would have to strip
out any BIDI overrides in the filename before presenting it to the user.
Language tagging and BIDI override support for Content-Disposition's
filename parameter are not really necessary.

> > The draft references Unicode 4.0 indirectly through RFC3629.

> STD 63 is not, repeat NOT, bound to some
> specific Unicode version.  

OK.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Aug 2008 14:43:04 +0200

Hi Brian,

thanks a lot for doing that research!

> encoded characters), the filename gets truncated and otherwise mangled. (In

That's a problem I completely forgot about, thanks for the reminder.

> advice on minimum/maximum file name lengths.)

Any idea what that advice would be? Something like minimally 64 decoded 
characters, resulting in an encoded name of 64 * 9 = 576 characters?

> * When the filename has a non-ASCII extension, IE doesn't decode the

Interesting.

> limitations and/or supports RFC 2231.

I wouldn't hold my breath for beta 2 (which I think is due soon), but I 
certainly hope something can be done in a subsequent release.

> some KB articles about it:

I think I recall seeing KB pages about related problems with Sharepoint, 
but I currently can't find the article.

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 19 Aug 2008 02:08:57 +0200

Brian Smith wrote:
 
> I sent this just to you because

Fortunately you didn't, what I wrote was plain wrong:
 
>> AFAIK this is not the case in RFC 2231, each piece
>> can have its own charset and language, please check.
[...] 

> No, it is a limit explicitly stated in RFC 2231 in
> section 4.1: "Language and character set information
> only appear at the beginning of a given parameter
> value. Continuations do not provide a facility for
> using more than one character set or language in the
> same parameter value."

Indeed, thanks for checking.  IOW Julian's draft did
not add a new restriction wrt the number of languages
in a parameter value by limiting it to one piece.

For your other remark about the length of file names
in bytes (about 150):  You said this boils down to 17
code points.   150 / 3 / 17 is about 2.9, so far it's 
clear.  But the "3" in  150 / 3 is apparently for the
percent-encoding (?)

A file system supporting Unicode (UTF-8 or UTF-16)
should be able to store about 50 or 75 code points.

IMO file name length limits don't belong in 2616bis:

<joke> Everybody knows that the limit is 8+3, 8+8,
14, 99, 255, or something else in octets for EBCDIC,
ASCII, OEM, ANSI, UTF-8, UTF-16, or similar. </joke>

 Frank

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 18 Aug 2008 22:12:31 -0500

> ASCII, OEM, ANSI, UTF-8, UTF-16, or similar. </joke>

Most filesystems will take a filename between 248 and 255+
characters.  I'm not sure why folks are scared of long
headers, they've existed forever (cookies, et al), but
there's no reason to set arbitrary recommendations on
this point.  There are security holes of being unable to
represent a file path in one context that had been already
represented in another context.  The "system limit" is the
only gating factor, and sane authors will choose something
reasonable for most implementations.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Aug 2008 08:29:41 +0200

> in a parameter value by limiting it to one piece.

To be fair, I *thought* I added a restriction, when I wasn't. Thanks 
again to Brian for checking.

In the meantime I'm working on a test suite, which I hope a first 
version of which will be online later this week.

> clear.  But the "3" in  150 / 3 is apparently for the

Yes. In an UTF-8 octet sequence, any sequence of characters NOT 
representing an ASCII character will have the highest bit set, thus will 
require percent-escaping in a URL.

> should be able to store about 50 or 75 code points.

Not sure how you come to that conclusion. For NTFS, the limit depends on 
the API which is used and is either 64 or "unlimited" (I think).

> IMO file name length limits don't belong in 2616bis:

That's true.

But in a draft specific to Content-Disposition we may want to consider 
this. In particular, sending extraordinary long file names (after 
decoding) may be problematic just for the reason of usability in the 
UA's UI.

> ASCII, OEM, ANSI, UTF-8, UTF-16, or similar. </joke>

I recall a system where it was 8+0 (plus a single 8 character top level 
folder name).

BR, Julian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Tue, 19 Aug 2008 02:29:10 -0500

> the API which is used and is either 64 or "unlimited" (I think).

On NTFS it's most certainly 255 characters or 248 for directory element
names.  See http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx
On linux today, it's typically 255 characters (NAME_MAX).

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Aug 2008 10:04:17 +0200

William A. Rowe, Jr. wrote:
>> Not sure how you come to that conclusion. For NTFS, the limit depends 
>> on the API which is used and is either 64 or "unlimited" (I think).
> 

For Linux, that counts octets, not characters, right? So with UTF-8 
encoding in use (would that be the case in Asia?), this would mean ~85 
characters.

What about HFS+?

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 19 Aug 2008 09:18:04 -0500

> But in a draft specific to Content-Disposition we may want to 
> consider this. In particular, sending extraordinary long file 
> names (after decoding) may be problematic just for the reason
> of usability in the UA's UI.

I think it is just best to add a note that points out that there may be a
large disparity between the number of characters and the number of bytes to
encode those characters. For example, "มหาวิทยาลัยเชียงใหม่" (Thai for
"Chiang Mai University") becomes:

%E0%B8%A1%E0%B8%AB%E0%B8%B2%E0%B8%A7%E0%B8%B4%E0%B8%97%E0%B8%A2%E0%B8%B2%E0%
B8%A5%E0%B8%B1%E0%B8%A2%E0%B9%80%E0%B8%8A%E0%B8%B5%E0%B8%A2%E0%B8%87%E0%B9%8
3%E0%B8%AB%E0%B8%A1%E0%B9%88

20 characters * 3 octets per character for UTF-8 * 3 bytes for %-escaping =
180 bytes. In UTF-8, each URL-encoded codepoint requires up to 12 bytes, and
each character could require several (up to 18 for NFKD) such byte
sequences. 

Somewhere (in an appendix?) the current behavior of popular UAs should be
documented so that people understand what works now vs. what will
(hopefully) work in the future. In that appendix, it would be prudent to
mention IE's 150-byte limit. For example, "มหาวิทยาลัยเชียงใหม่" is already
way too long for IE (180 bytes), even though it accepts the longer (by
character count) "Chiang Mai University" just fine.

Julian: in your tests you should try out different normalization forms and
non-normalized forms. NTFS doesn't do any normalization at all and I imagine
Linux is the same, but Mac OS X's HFS+ file system converts all filenames to
NFD. Yet most software and specifications (W3C and RFC 3987 in particular)
uses NFC.

Regards,
Brian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 19 Aug 2008 21:30:55 +0200

William A. Rowe, Jr. wrote:

> The "system limit" is the only gating factor, and
> sane authors will choose something reasonable for
> most implementations.

+1  Before I posted my reply I tried to figure out
what a sane limit for pax might be, hoping that it
could be some kind of standard, which could deserve
an informative reference in 2616bis.

But asking Google I got no conclusive result, only
a vague "99" hint => not convincing for UTF-16. :-)

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 19 Aug 2008 21:42:57 +0200

Julian Reschke wrote:

> I recall a system where it was 8+0 (plus a
> single 8 character top level folder name).

What I had in mind when I wrote 8+8 was an
old IBM mainframe system, DOS 360 or MVS, I
forgot the details.  It was fun to (ab)use a
virtual "PUN" device for a kind of e-mail...

After getting over the shock that thousands
of very real PUNched cards were now history.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Aug 2008 13:24:12 +0200

> NFD. Yet most software and specifications (W3C and RFC 3987 in particular)

As always, it took a bit longer than expected, but I now have an initial 
set of tests available (yeah, with results!):

	<http://greenbytes.de/tech/rfc2231/>

I'll try to add more in the next few days.

BR, Julian