Subject: Content-Disposition: *sender* advice

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Feb 2011 22:33:34 +0100

Hi,

last week IE9RC came out, and sure enough, it has limited support for 
RFC 2231/5987 (limited in that it only supports UTF-8 (*)). See 
<http://greenbytes.de/tech/tc2231/>.

Which makes me wonder - should the spec include more advice for producers?

The current situation is:

1) Once IE9 is out, senders can rely on RFC5987/UTF-8 support for all 
"current" UAs, except for IE < 9 and Safari.

2) There is no fallback that would work both with Safari and legacy IE 
versions.

3) filename and filename* can safely be sent together (with filename 
acting as fallback), except that Firefox will still pick the wrong (see 
<https://bugzilla.mozilla.org/show_bug.cgi?id=588781> -- it would be 
great to see this finally fixed in FF5).

Given these constraints senders that really need to provide non-ASCII 
characters to "all" recipients have to do User Agent sniffing. That's 
bad, but I believe explaining the problem would still be better than 
being silent on it.

(*) Given that IE9RC only supports UTF-8, and the inclusion of 
ISO-8859-1 in RFC 5987 isn't essential anyway, should we note in the 
spec not ever to use ISO-8859-1?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Feb 2011 18:42:38 +0100

> spec not ever to use ISO-8859-1?

I got one "go ahead" off-list, so I tried to write down some advice.

I was determined not to mention User-Agent sniffing; but it's really the 
only choice today if you *need* I18N for "all" browsers.

To provide up-to-date information, I created a PURL URI for my test 
cases; maybe that's an acceptable compromise for linking to up-to-date 
information.

Feedback appreciated!

-- snip --
Appendix D.  Advice on Generating Content-Disposition Header Fields

    Unfortunately, the varying quality of user agent implementations
    makes it non-trivial to generate the filename parameter.

    The "token" and "quoted-string" forms are widely implemented, but
    senders need to be aware both of the constraints of these formats and
    several implementation quirks in user agents:

    o  The "token" form excludes characters that are likely to occur in
       filenames, such as the space character.

    o  Some implementations do not properly handle the escaping mechanism
       used in the "quoted-string" form; however, the escape character is
       "\", which isn't supposed to show up in a portable filename
       anyway.

    o  Some implementations misinterpret "percent" escapes, and therefore
       filenames containing a "%hh" sequences (where "h" is a "hex
       digit") can yield unexpected results (see Appendix C.2).

    o  Some implementations apply heuristics when encountering non-ASCII
       characters, and might unexpectedly switch to UTF-8 encoding (see
       Appendix C.3).

    The encoding used in "filename*" (see [RFC5987]) eliminates all these
    issues, but not all user agents support it.  Among those which do
    support it, the following issues are known:

    o  One implementation only supports UTF-8, although RFC 5987 requires
       support for ISO-8859-1 as well.  In practice, this is not a
       problem because UTF-8 is preferable anyway.

    o  One implementation picks the wrong parameter when both "filename"
       and "filename*" are specified, thus it's not always possible to
       specify "filename" as a fallback for old implementations.

    Consequently, the "best" approach depends on the use case.  In many
    cases, it might be acceptable to simply strip or replace certain
    characters that are known to be problematic.  In other cases, it
    might be ok to just ignore legacy user agents that do not support the
    "filename*" format.  Of course, using different formats based on the
    User-Agent request header field is also an option, but the drawbacks
    of this choice need be kept in mind (implementations improving,
    picking the right default, caching considerations).

    <http://purl.org/NET/http/content-disposition-tests> provides an
    overview of what implementations do support, and can assist in
    selecting an implementation approach.
-- snip --

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Feb 2011 13:09:37 +1100

> Appendix D.  Advice on Generating Content-Disposition Header Fields

I started to reply with revised wording for some points, and ended up with a rewrite, fwiw.

---8<---

To successfully interoperate with existing and future user agents, senders of the Content-Disposition header are advised to:

  - Inclue a 'filename' parameter on all Content-Disposition headers. 

  - Use the 'token' form of the filename parameter only when it does not contain disallowed characters (e.g., spaces); in these cases, the quoted-string form should be used.

 - Avoid including the percent character followed by two hexadecimal characters (e.g., %A9) in the filename parameter, since some existing implementations consider it to be an escape character, while others will pass it through unchanged.

 - Avoid including the "\" character in the quoted-string form of the filename parameter, as escaping is not implemented by some user-agents, and will be considered as an illegal path character. 

 - Avoid using non-ASCII characters in the filename parameter. Although some existing implementations will attempt to decode them, others have no interoperable way to fail gracefully.

  - Include a 'filename*' parameter on Content-Disposition headers where the desired filename contains characters outside the US-ASCII character encoding. Note that legacy user agents will not process this, and will fall back to using the 'filename' parameter's content.

 - Use UTF-8 as the encoding of the 'filename*' parameter, when present, because at least one existing implementation only implements that encoding.

Note that this advice may be superseded. <http://purl.org/NET/http/content-disposition-tests> provides an overview of current levels of support in various implementations.

--->8---

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Feb 2011 09:20:47 +0100

On 23.02.2011 03:09, Mark Nottingham wrote:
> ...

I'm happy with most of this, except for:

> ...
> ...
> ...

Depending on ordering, this advice either doesn't work for Firefox 
(including FF4), or for IE8 (and below). See 
<http://greenbytes.de/tech/tc2231/#attfnboth> and 
<http://greenbytes.de/tech/tc2231/#attfnboth2>.

That's why I ended up with the statement abour U-A sniffing. Nobody 
likes it. I don't like it. But like it or not, it's what you need to do 
as of today.

The alternative of course is not to have any advice here, and just link 
to something we can update more quickly.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Feb 2011 10:57:30 +0100

> <http://greenbytes.de/tech/tc2231/#attfnboth2>.

Expanding on this:

1) We can't recommend to unconditionally *only* use the "filename*" 
format, as it's not supported by Safari (a), nor by IE on Windows XP (b).

2) We can't recommend to unconditionally send both, with "filename*" 
first, because it fails with IE on Windows XP (b). See 
<http://greenbytes.de/tech/tc2231/#attfnboth2>.

3) We can't recommend to unconditionally send both, with "filename*" 
second, because it fails with Firefox, even in version 4 (c). See 
<http://greenbytes.de/tech/tc2231/#attfnboth>.

Further comments:

(a) Related Safari bug report is 
<https://bugs.webkit.org/show_bug.cgi?id=15287>, "rdar://4727992". My 
understanding is that this eventually will get fixed, but we won't know 
until it happens due to Apple's policy with respect to announcing products.

b) We could decide that this is not a problem because of Windows XP 
being phased out.

c) This has had a bug report and patch since September 2010 
(<https://bugzilla.mozilla.org/show_bug.cgi?id=588781>), but 
unfortunately it won't be fixed in FF4. It would be awesome if the 
Mozilla developers would reconsider this.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Feb 2011 21:43:53 +1100

(3) isn't ideal, but it does perform as advertised; i.e., the UA falls back to the filename parameter.

On 23/02/2011, at 8:57 PM, Julian Reschke wrote:

> 1) We can't recommend to unconditionally *only* use the "filename*" format, as it's not supported by Safari (a), nor by IE on Windows XP (b).
> 
> 2) We can't recommend to unconditionally send both, with "filename*" first, because it fails with IE on Windows XP (b). See <http://greenbytes.de/tech/tc2231/#attfnboth2>.
> 
> 3) We can't recommend to unconditionally send both, with "filename*" second, because it fails with Firefox, even in version 4 (c). See <http://greenbytes.de/tech/tc2231/#attfnboth>.
> 
> Further comments:
> 
> (a) Related Safari bug report is <https://bugs.webkit.org/show_bug.cgi?id=15287>, "rdar://4727992". My understanding is that this eventually will get fixed, but we won't know until it happens due to Apple's policy with respect to announcing products.
> 
> b) We could decide that this is not a problem because of Windows XP being phased out.
> 
> c) This has had a bug report and patch since September 2010 (<https://bugzilla.mozilla.org/show_bug.cgi?id=588781>), but unfortunately it won't be fixed in FF4. It would be awesome if the Mozilla developers would reconsider this.
> 
> Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Feb 2011 12:21:35 +0100

> (3) isn't ideal, but it does perform as advertised; i.e., the UA falls back to the filename parameter.

Well, Firefox does not "fall back", it "trips over".

This would mean that we recommend a format that breaks the UA which has 
been doing the "right thing" since "ever", just because some *other* UAs 
do not.

I don't think we should do that, at least not until 
<https://bugzilla.mozilla.org/show_bug.cgi?id=588781> is fixed. So I'd 
prefer no advice over bad advice here.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Feb 2011 16:39:56 +1100

I wouldn't say we break FireFox by recommending this ("breaking" means that C-D doesn't work at all, or produces garbage), and I wouldn't say they've been doing the right thing (because they implemented to prefer the non-internationalised filename).

Also, based on their bug traffic, they unfortunately don't seem too eager to fix this.

When you say "no advice" what do you mean? 

On 23/02/2011, at 10:21 PM, Julian Reschke wrote:

> This would mean that we recommend a format that breaks the UA which has been doing the "right thing" since "ever", just because some *other* UAs do not.
> 
> I don't think we should do that, at least not until <https://bugzilla.mozilla.org/show_bug.cgi?id=588781> is fixed. So I'd prefer no advice over bad advice here.
> 
> Best regards, Julian
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Feb 2011 08:54:41 +0100

> I wouldn't say we break FireFox by recommending this ("breaking" means that C-D doesn't work at all, or produces garbage), and I wouldn't say they've been doing the right thing (because they implem…

When I said "break" I mean that sending a I18Nized filename will not 
have the desired effect.

When I said "do the right thing" I mean that *they* implement the 
published specs properly (at least with respect to this), while other 
UAs do not (Safari), or just started to do so (Chrome 9, IE9).

The recommendation to pick "filename*" over "filename" is new (as of 
this spec), and it's only a SHOULD (we could change that, though).

> Also, based on their bug traffic, they unfortunately don't seem too eager to fix this.

Unfortunately. Hopefully the plan with respect to making major releases 
more frequent will happen; in which case having this in FF5 wouldn't be 
too bad.

> When you say "no advice" what do you mean?

I mean not having this appendix at all (after all, it appears it never 
was discussed seriously until last week, and we are months past WG LC), 
or alternatively to have a short statement that producing I18Nized field 
values is tricky because of the various implementation problems, and 
then have a pointer to something that's easier to maintain.

Maybe we could go to IETF LC and mark this question as something we're 
looking for feedback for?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Feb 2011 11:11:35 +0100

On 24.02.2011 08:54, Julian Reschke wrote:
>> I wouldn't say we break FireFox by recommending this ("breaking" means
>> that C-D doesn't work at all, or produces garbage), and I wouldn't say
>> they've been doing the right thing (because they implemented to prefer
>> the non-internationalised filename).
>

Looking at <https://bugzilla.mozilla.org/show_bug.cgi?id=588781#c21>, 
the fix will go into Firefox 5, which is planned to be 3 months behind 
Firefox 4.

Optimally, this would be ~July 2011.

On the other hand, to get to RFC we need

- to do the IETF LC -> two weeks
- get it through the IESG (I expect at least minimally one revision to 
be requested) -> ~ two months

Once it's in the RFC publication queue, it'll take at least another six 
weeks... Which would be around the same time.

Still, I'm very uncomfortable recommending something we're not sure will 
work when the RFC is published.

Related to this; I just updated the notes with respect to the 
filename*/filename fallback behavior; see 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1140>.

Best regards, Julian

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Fri, 25 Feb 2011 12:38:46 +0000

On 25 Feb 2011, at 10:11, Julian Reschke wrote:

> Looking at <https://bugzilla.mozilla.org/show_bug.cgi?id=588781#c21>, the fix will go into Firefox 5, which is planned to be 3 months behind Firefox 4.
> 
> - get it through the IESG (I expect at least minimally one revision to be requested) -> ~ two months
> 
> Once it's in the RFC publication queue, it'll take at least another six weeks... Which would be around the same time.
> 
> Still, I'm very uncomfortable recommending something we're not sure will work when the RFC is published.
> 

One alternative, if the WG/ADs are happy with the approach maybe to be proceed based on the current firefox implementation and if firefox change their code in time get some consensus on text from the WG that can be inserted as part of AUTH48?

Ben

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Feb 2011 14:29:58 +0100

> One alternative, if the WG/ADs are happy with the approach maybe to be proceed based on the current firefox implementation and if firefox change their code in time get some consensus on text from t…

Clarifying: "proceed based on the current firefox" essentially means 
recommending user-agent sniffing... I think if we have a choice we 
really really should avoid this.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 26 Feb 2011 12:12:23 +0100

> work when the RFC is published.

Ok, after lots of wordsmithing by Mark and myself we now have this:

-- snip --

    To successfully interoperate with existing and future user agents,
    senders of the Content-Disposition header field are advised to:

    o  Include a 'filename' parameter when US-ASCII is sufficiently
       expressive.

    o  Use the 'token' form of the filename parameter only when it does
       not contain disallowed characters (e.g., spaces); in such cases,
       the quoted-string form should be used.

    o  Avoid including the percent character followed by two hexadecimal
       characters (e.g., %A9) in the filename parameter, since some
       existing implementations consider it to be an escape character,
       while others will pass it through unchanged.

    o  Avoid including the "\" character in the quoted-string form of the
       filename parameter, as escaping is not implemented by some user
       agents, and can be considered as an illegal path character.

    o  Avoid using non-ASCII characters in the filename parameter.
       Although most existing implementations will decode them as ISO-
       8859-1, some will apply heuristics to detect UTF-8, and thus might
       fail on certain names.

    o  Include a 'filename*' parameter where the desired filename cannot
       be expressed faithfully using the 'filename' form.  Note that
       legacy user agents will not process this, and will fall back to
       using the 'filename' parameter's content.

    o  When a 'filename*' parameter is sent, to also generate a
       'filename' parameter as a fallback for user agents that do not
       support the 'filename*' form, if possible.  This can be done by
       substituting characters with US-ASCII sequences (e.g., Unicode
       character point U+00E4 (LATIN SMALL LETTER A WITH DIARESIS) by
       "ae").  Note that this may not be possible in some locales.

    o  When a 'filename' parameter is included as a fallback (as per
       above), 'filename' should occur first, due to parsing problems in
       some existing implementations. [[fallbackbug: Firefox is known to
       pick the wrong parameter; a bug fix is scheduled for Firefox 5.
       --jre]]

    o  Use UTF-8 as the encoding of the 'filename*' parameter, when
       present, because at least one existing implementation only
       implements that encoding.

    Note that this advice is based upon UA behaviour at the time of
    writing, and might be superseded.
    <http://purl.org/NET/http/content-disposition-tests> provides an
    overview of current levels of support in various implementations.
-- snip --

Where <http://purl.org/NET/http/content-disposition-tests> currently 
redirects to <http://greenbytes.de/tech/tc2231/>.

See <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1142>.

Best regards, Julian