Subject: Re: Content-Disposition next steps

From: Mark Nottingham <[email-masked]>
Date: Wed, 1 Dec 2010 22:12:16 +1100

Adam, do you have a proposal?

Regards,

On 09/11/2010, at 12:53 PM, Mark Nottingham wrote:

> Given that we currently have at least two vendors (probably more, I'm just going by public information) working on implementing the draft now (at least partly triggered by us going to WGLC), and it's possible we'll get feedback from them, I'm inclined to pause for a *small* number of weeks, to see if we can learn anything else. 
> 
> During that time, Adam can come up with a proposal for error handling, which we'll consider at the end of that period. 
> 
> Adam, you can do that either openly on the mailing list, or as a design team (i.e., work with other interested parties separately and bring a proposal forward when you think you're ready). I'd encourage you to do it openly on the list, to get constructive feedback from others as you go (thereby increasing your chances of delivering something people will accept). However, if Adam does this, I'd ask people not to question *why* it's being done. 
> 
> If it's done in time, and if we gather consensus on it, it can go into C-D as an optional appendix. If it's not ready in time, it can be published (by the WG or separately) on its own. Likewise, if the WG doesn't gain consensus on it, it can be published individually (in the IETF or elsewhere). 
> 
> Right now, I'm thinking three weeks -- November 30th (coincidentally, my birthday). This will give us time to re-publish, gain consensus (or not) and go to IETF Last Call by the end of the year.

--
Mark Nottingham   http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 11:50:23 -0800

> Adam, do you have a proposal?

Yeah.  Please find my proposal below.  It's certainly not beautiful,
and it likely needs more polish, but it should be a starting point.

I tried to be as "gramatical" as I could, but couldn't quite figure
out how avoid all the algorithmic aspects.  The proposal is based on
what Chrome does, but cleaned up slightly.  There's some sadness I
couldn't quite figure out how to avoid, but I'm certainly open to
talking about it more.

The rules for determining the disposition-type are particularly goofy.
 I wanted to do more homework to figure how if we can make those more
aesthetic, but I ran out of time.

One of the ground rules was that my proposal should only differ from
the current draft in error-handling cases.  I believe that's the case,
but I'm not 100% sure.  Please let me know if I've screwed that up.

Adam

== Extracting Parameter Values From Header Fields ==

To extract the value for a given parameter-name from an unparsed-string, parse
the unparsed-string using the following grammar:

  unparsed-string = *CHAR name *LWS "=" value [ ";" *CHAR ]
  value           = <CHAR, except ";">

where the name production is a gramatical production that is a case-insensitive
match for the given parameter-name.  If the unparsed-string can be parsed by
the grammar in multple ways, choose the one in which name appears as close to
the beginning of the string as possible.  If the unparsed-string cannot be
parsed by the grammar above, return the empty string.

== Decoding the File Name ==

To filename-decode an encoded-string, parse the encoded-string using the
following grammar:

  encoded-string = word *( 1*delimiter word )
  delimiter      = LWS
  word           = <CHAR, except delimiter>

Consider each gramatical element (either a delimiter or a word) in the order
they appear in the encoded-string:

  1) If the gramatical element is a delimiter, process the element as follows:

       a) If the previous gramatical element was an RFC2047-value, ignore this
          gramatical element.

       b) Otherwise, emit a SP character.

  2) If the gramatical element is a word, process the element as follows:

       a) If the word contains non-ASCII characters, process the element as
          follows:

            i)  If the word is a well-formed UTF-8 string, emit the word
                (decoded as UTF-8) and proceed to the next grammatical element.

            ii) Otherwise, *sadness*.  Apparently what we're supposed to do
                here is to use the "referrer" charset, if we have one.
                Otherwse, we fall back to the OS codepage.

        b) If the word is an RFC2047-value, emit the RFC2047 decoding of the
           word and proceed to the next grammatical element.

        c) Let the url-unescaped-word be the word %-unescaped.

        d) Emit the url-unescaped-word (decoded as UTF-8) and proceed to the
           next grammatical element.  (There's actually more sadness here if
           the url-unescaped-word isn't valid UTF-8.)

The emitted characters are the decoded file name.

== Determining the File Name ==

To determine the file name indicated by a Content-Disposition header field, use
the following algorithm:

  1) Let filename-star be the value extracted from the Content-Disposition
     header field for for the "filename*" parameter.

  2) If filename-star parses as a RFC5987-value, return the RFC5987-value of
     filename-star and abort these steps.

  3) Let filename be the value extracted from the Content-Disposition header
     field for the "filename" parameter.

  4) If filename is empty, instead let filename be the value extracted from the
     Content-Disposition header field for the "name" parameter.

  5) If filename is empty, return the empty string and abort these steps.

  6) Return the filename-decoding of filename.

== Determining the Disposition ==

To determine the disposition-type, parse the Content-Disposition
header field using
the following grammar:

  unparsed-string  = *LWS nominal-type *CHAR
  nominal-type = "inline" / "filename" / "name" / ";"

If the Content-Disposition header field parser fails to parse, then the
disposition type is "attachment".  Otherwise, the disposition-type is "inline".

== Processing the Content-Disposition Header Field ==

To process the Content-Disposition header field, use the following algorithm:

  1) Determine the disposition-type.

  2) If the disposition-type is "inline", then ...

  3) If the disposition-type is "attachment", then let filename be the file
     name indicated by the header field.  ...

---

From: Julian Reschke <[email-masked]>
Date: Wed, 01 Dec 2010 22:00:26 +0100

On 01.12.2010 20:50, Adam Barth wrote:
> On Wed, Dec 1, 2010 at 3:12 AM, Mark Nottingham<[email-masked]>  wrote:
> ...

Adam,

before I look at the details, I'd like to understand the intent.

You're saying this reflects roughly what Chrome is doing. Thanks for 
writing this down.

But exactly *why* would we consider this for the spec (I assume as 
recommended error recovery?)?

I note that you have handling of RFC2047-style encoding in there. That's 
something only Chrome and Firefox are doing, so I'd like to understand 
why you think it's needed, and whether you think 
Opera/Safari/Konqueror/IE should implement that (given the fact that 
changes the semantics of values that are valid).

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 13:13:54 -0800

> before I look at the details, I'd like to understand the intent.

Thanks for seeking to understand instead of jumping to conclusions.  :)

> You're saying this reflects roughly what Chrome is doing. Thanks for writing
> this down.
>

Oh, I thought we discussed this before.  From my perspective, the
current spec is optimized for server operators.  In particular, it's
helpful if you'd like to generate the Content-Disposition header.  I
raised the issue that user agent implementors might like information
about how to consume the Content-Disposition header.  We then had a
long discussion about why the optimal generation instructions and
consumption instructions might not be identical.

I think, then, Mark suggested that we could include that information
in an appendix that user agents could optionally implement, if they
were so inclined.  One important constraint is that the consumption
requirements agree with the rest of the spec on well-formed headers
(i.e., on those that meet the generation requirements).

If it's helpful for you to think about this information as recommended
error recovery, that's fine with me.  When consuming the
Content-Disposition header, it's not especially important to know
whether the header is well-formed (i.e., generated in accordance to
the generation requirements).

> something only Chrome and Firefox are doing, so I'd like to understand why
> you think it's needed, and whether you think Opera/Safari/Konqueror/IE
> should implement that (given the fact that changes the semantics of values
> that are valid).

Yeah, I wasn't sure whether to include the RFC2047 encoding.  I
certainly wouldn't recommend that servers generate Content-Disposition
headers using that encoding.  However, if I were writing a new user
agent today, I might well include RFC2047 support.  It boils down to a
cost/benefit analysis.  Some comments in the Chrome code indicate that
there are servers that do generate RFC2047-encoded Content-Disposition
headers, so there's at least some benefit.

One thing that might make sense is to demarcate those instructions as
again optional, that is an optional piece of the optional error
recovery, if you like.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 01 Dec 2010 22:44:36 +0100

> consumption instructions might not be identical.

I'm aware of that.

> (i.e., on those that meet the generation requirements).

Yes.

But your proposal is already in the grey area because it conflicts with 
the literal reading of the grammar, by applying certain decoding 
operations on something that is not *supposed* to be encoded, or by 
using a charset default which isn't backed by the specs.

This *could* be justified by claiming that the filename is advisory 
only, but it's really not satisfying, in particular when we have 
evidence that many UAs get away without it.

> error recovery, that's fine with me.  When consuming the

Error recovery implies a detectable error. A big part of what you 
propose changes the interpretation of valid fields.

> the generation requirements).

Unless you want to ignore it otherwise.

> headers, so there's at least some benefit.

But there's also damage, because there's a small risk to misinterpret a 
value that just happens to look like 2047-encoded.

>                 (decoded as UTF-8) and proceed to the next grammatical element.

According to RFC2616, the default is ISO-8859-1, and IE/Opera/Konqueror 
do exactly that (at least in my locale): 
<http://greenbytes.de/tech/tc2231/#attwithutf8fnplain>

>            the url-unescaped-word isn't valid UTF-8.)

That overloads the syntax of the parameter, and it is not done in 
FF/Opera/Safari/Konqueror: 
<http://greenbytes.de/tech/tc2231/#attwithfnrawpctenca>

Yes, the current landscape is a mess, but it's a *different* mess in 
each of the various UAs. Your recommendation appears to merge all the 
bad workarounds. My recommendation would be to try to get slowly rid of 
them.

Statistics on which of these workarounds are *really* used would be useful.

> again optional, that is an optional piece of the optional error

That would apply to all of RFC2047, UTF-8 defaulting, and 
percent-unescaping.

Are you willing to rephrase the proposal accordingly?

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 14:17:01 -0800

On Wed, Dec 1, 2010 at 1:44 PM, Julian Reschke <[email-masked]> wrote:
> But your proposal is already in the grey area because it conflicts with the
> literal reading of the grammar, by applying certain decoding operations on
> something that is not *supposed* to be encoded, or by using a charset
> default which isn't backed by the specs.
>
> This *could* be justified by claiming that the filename is advisory only,
> but it's really not satisfying, in particular when we have evidence that
> many UAs get away without it.

Hum...  There are two parts to this issue:

1) How we expect user agents to behave.
2) What we write in the document.

IMHO, the user agents who are interested in this information are not
going to stop performing this decoding (to issue 1), and the document
should tell the truth with respect to what these user agents are going
to do (to issue 2).

> Error recovery implies a detectable error. A big part of what you propose
> changes the interpretation of valid fields.

Perhaps the term "error recovery" isn't helpful?  Another option, of
course, is to adjust the definition of what is valid to exclude these
pieces of syntax.

> Unless you want to ignore it otherwise.

That is certainly one option for consumers.  However, in this
appendix, we're concerning ourselves with user agents who wish to
process ill-formed headers.

>>> why

I'd put that into the "cost" column in the cost/benefit analysis.

> According to RFC2616, the default is ISO-8859-1, and IE/Opera/Konqueror do
> exactly that (at least in my locale):

According to your tests Firefox, Chrome, and Safari use UTF-8.  Given
a free choice of UTF-8 or ISO-8859-1, I'd pick UTF-8, as I've done
here.

> <http://greenbytes.de/tech/tc2231/#attwithfnrawpctenca>

Indeed.  We've discussed this issue at length.  For senders, I suspect
that the optimal way of generating Content-Disposition headers is to
avoid using the % character because that character is interpreted
differently by different user agents.  For receivers, I suspect that
the optimal way of consuming Content-Disposition headers is to
%-decode them, as described above.

Perhaps the best outcome, then, is to forbid servers from generating
the % character?  That way the syntax won't be "overloaded."

> Yes, the current landscape is a mess, but it's a *different* mess in each of
> the various UAs. Your recommendation appears to merge all the bad
> workarounds. My recommendation would be to try to get slowly rid of them.

I suspect our different approaches reflect our different experiences.

> Statistics on which of these workarounds are *really* used would be useful.

Indeed.

> Are you willing to rephrase the proposal accordingly?

If rephrasing the proposal would be helpful, I'm happy to do that.
What specifically would you like rephrased?

Adam

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 01 Dec 2010 23:32:52 +0100

* Adam Barth wrote:
>what Chrome does, but cleaned up slightly.

I would have preferred, instead of dragging the working group through
philosophical debates about browser vendor's needs for two months, you'd
just have said you'd like the specifcation to say the header should be
processed as Chrome processes it, and detail any exceptions. I note that
you are not actually telling us what the differences should be even now.

>aesthetic, but I ran out of time.

You will excuse if I remind you of your response to concerns about the
lack of resources to work on this, "We have many problems in life, but
a lack of engineering resources isn't one of them". But let's see about

>disposition type is "attachment".  Otherwise, the disposition-type is "inline".

It is incorrect to specify the *LWS here as the draft uses RFC 2616
implied LWS rules. This also does not do what you intend as *CHAR
does not match what you think it matches. It also mishandles a common
case, the empty string as value, which fails to parse but is handled
pretty much universally as "inline", contrary to your proposal. Not
to mention that it's silly to treat `x=y; filename=example.txt` as if
it had an unrecognized disposition type and should thus be handled as
"attachment", which, say, Internet Explorer and Opera don't do, when
you treat plain `filename=example.txt` as having no disposition type.

>but I'm not 100% sure.  Please let me know if I've screwed that up.

I can't really make heads or tails of the rest of your proposal, for
instance, if you go by the processing rules already in the draft, you
would not need to discuss quote marks, but you seem to have your own
rules for processing parameters and parameter values, in which case
you would need to discuss quote marks, but your proposal does not.

But the impression I am getting is that you have provisions for both
RFC 2047 encoding and some url-escaping in quoted-string values, both
of which are non-standard interpretations of properly formed headers.
That does not belong in any proposal for error handling, as I said in
http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0395.html

Let me say again, if the idea behind the i18n behavior is to emulate
certain browsers, we need to know exactly how those browsers behave,
and that includes the configuration of those browsers which may vary
depending on things like the system locale, and better evidence for
specifying behavior that is inconsistent with them than Google being
unaware of bug reports; Chrome needs a much larger market share than
it has at the moment to make the lack of bug reports an argument.

>parsed by the grammar above, return the empty string.

This does not work as you intend as CHAR is US-ASCII 0x00 through 0x7F
so you would never get to filename parameter values that are not UTF-8.

Not to mention that this is utterly silly, if you have "x<name>=..."
this would be handled as if the value had a `name` parameter with the
empty string as value, as opposed to the semantically correct result,
which would be "there is no `name` parameter". And as you lack higher
level control logic to actually separate parameters, this results in
`example="filename=example.txt"` having the filename `example.txt"`,
as opposed to the correct result, namely that there is no filename.

Mark, please count me as opposed to delay submitting the draft to the
IESG to wait for Adam to come up with a proposal any longer; when it's
finished, it may just aswell be published as a separate specification;
we've been on this for two months now with essentially no progress.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 01 Dec 2010 23:47:41 +0100

On 01.12.2010 23:17, Adam Barth wrote:
> ....

I'm not convinced that all UAs are going to continue to keep 
workarounds; Mozilla, for instance, is considering to drop RFC2047 
decoding, see <https://bugzilla.mozilla.org/show_bug.cgi?id=610054>.

But even if it was the case: the UAs support *different* kinds of 
decoding. So there's no single way we could document that all of them do 
now.

What we *can* do is mention what's going on, and warn about what it 
means for interop. We already do that, but I'm open to add more text to 
that part.

> course, is to adjust the definition of what is valid to exclude these

That would mean disallowing to send "%" or "%hh" (where h is HEXDIG) in 
a filename. We can *warn* that some UAs misinterpret this, but unless we 
can recommend something else, that's not really helpful ("something 
else" would be RFC 5987 encoding, but Safari and IE do not understand that).

> process ill-formed headers.

We really need to separate the discussion about recovering from invalid 
headers from the other thing (applying un-escapings or encodings not 
supported by the specs).

> I'd put that into the "cost" column in the cost/benefit analysis.

Looking forward to that analysis.

> a free choice of UTF-8 or ISO-8859-1, I'd pick UTF-8, as I've done

Given a free choice, I'd pick it as well. But there is no free choice, 
as RFC 2616 says that it is ISO-8859-1, and a certain well-deployed 
browser actually does that.

> the % character?  That way the syntax won't be "overloaded."

"% can't occur in filenames"?

Me not happy.

> What specifically would you like rephrased?

I think it would be good to

- distinguish between the error-recovery parts and the filename decoding 
parts, and

- make every non-complaint part of the filename decoding (UTF-8 instead 
of ISO-8859-1, RFC2047 decoding, percent-unescaping) truly optional

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 14:53:58 -0800

Thanks for your feedback Bjoern.  Responses inline.  (Note I've
trimmed a bunch of non-technical content to try to focus on the
technical issues.)

On Wed, Dec 1, 2010 at 2:32 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> implied LWS rules.

Forgive me.  It's difficult to for me to keep track of which documents
use implied LWS.  Please read this text as if it were in a grammar
without implied LWS.

> This also does not do what you intend as *CHAR
> does not match what you think it matches.

Thanks.  I've replaced all instances of CHAR with OCTET.

> It also mishandles a common
> pretty much universally as "inline", contrary to your proposal.

Fixed.

> Not to mention that it's silly to treat `x=y; filename=example.txt` as if

Perhaps Julian would be willing to add this case to his test suite?
Silliness isn't one of the criteria I've applied.

> you would need to discuss quote marks, but your proposal does not.

It's possible I've screwed up handling quote marks.  Do you have a
specific test case you're worried about?  I was surprised as well that
I didn't need to mention quote marks.

> so you would never get to filename parameter values that are not UTF-8.

Fixed.

> which would be "there is no `name` parameter".

Perhaps this is another good test case to add to the suite.  I'm
willing to believe this behavior isn't necessary, but I'd like to look
at some more evidence before changing it.

> And as you lack higher

Sounds like another good test case.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 14:59:46 -0800

On Wed, Dec 1, 2010 at 2:47 PM, Julian Reschke <[email-masked]> wrote:
> I'm not convinced that all UAs are going to continue to keep workarounds;
> Mozilla, for instance, is considering to drop RFC2047 decoding, see
> <https://bugzilla.mozilla.org/show_bug.cgi?id=610054>.

RFC2047 support is particularly tempting to remove.  As I wrote
before, I debated whether to include it.

> But even if it was the case: the UAs support *different* kinds of decoding.
> So there's no single way we could document that all of them do now.

Indeed.  Fortunately we're not documenting all of them in this document.

> What we *can* do is mention what's going on, and warn about what it means
> for interop. We already do that, but I'm open to add more text to that part.

That's indeed useful information for servers but not particularly
helpful to user agent implementors.

> That would mean disallowing to send "%" or "%hh" (where h is HEXDIG) in a
> filename.

Yes.

> We can *warn* that some UAs misinterpret this, but unless we can
> recommend something else, that's not really helpful ("something else" would
> be RFC 5987 encoding, but Safari and IE do not understand that).

Well, we can warn or we can forbid.  The choice is ours.

> headers from the other thing (applying un-escapings or encodings not

They seem related.  The question at hand is how to consume the
Content-Disposition header.

> Given a free choice, I'd pick it as well. But there is no free choice, as
> RFC 2616 says that it is ISO-8859-1, and a certain well-deployed browser
> actually does that.

Fortunately neither of those things prevents us from making this choice here.

> - distinguish between the error-recovery parts and the filename decoding

Fortunately, this should be easy since they're textually contained in
different sections.

> - make every non-complaint part of the filename decoding (UTF-8 instead of
> ISO-8859-1, RFC2047 decoding, percent-unescaping) truly optional

Everything about this appendix is optional, so we should be fine on that score.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Dec 2010 00:15:26 +0100

> before, I debated whether to include it.

Ok, here's a +1 to exclude it.

> Indeed.  Fortunately we're not documenting all of them in this document.

Well, you're mentioning most.

> helpful to user agent implementors.

Helpful advice for UA implementers would be "get rid of spec violations 
if you can".

> Well, we can warn or we can forbid.  The choice is ours.

<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-content-disp-04.html#rfc.section.3.3>:

"Note: Many user agents do not properly handle escape characters when 
using the quoted-string form. Furthermore, some user agents erroneously 
try to perform unescaping of "percent" escapes (see Appendix C.2), and 
thus might misinterpret filenames containing the percent character 
followed by two hex digits."

> Content-Disposition header.

There's no simple answer to that, unless you're willing to be fully 
spec-compliant in which case the spec already tells you what to do.

> Fortunately neither of those things prevents us from making this choice here.

Sorry? The spec (2616) says it's ISO-8859-1. IE takes it as ISO-8859-1. 
Opera and Konqueror agree. I don't think there's room for choice here.

> Fortunately, this should be easy since they're textually contained in

Ack.

> Everything about this appendix is optional, so we should be fine on that score.

It would be helpful if you could provide text that is actually 
self-contained and ready to appear in the spec; in particular that means 
that this needs an explanation about who this is for, and what the 
conformance requirements are. If some parts are optional, these need to 
be marked.

If *all* of this is optional (you can pick whatever you like), then this 
is just another way to document the current state of implementations, in 
which case I'd recommend we just augment what's already in Appendix C.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Dec 2010 00:19:34 +0100

> Silliness isn't one of the criteria I've applied.

Will do tomorrow.

> I didn't need to mention quote marks.

That's probably because Chrome doesn't handle quoted-string properly: 
<http://greenbytes.de/tech/tc2231/#attwithasciifnescapedchar>

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 16:25:01 -0800

> Ok, here's a +1 to exclude it.

Done.

> "Note: Many user agents do not properly handle escape characters when using
> the quoted-string form. Furthermore, some user agents erroneously try to
> perform unescaping of "percent" escapes (see Appendix C.2), and thus might
> misinterpret filenames containing the percent character followed by two hex
> digits."

I'm not sure what point you're trying to make by quoting this part of
the document.  That is indeed a warning.

> spec-compliant in which case the spec already tells you what to do.

Hence my proposal.

> Opera and Konqueror agree. I don't think there's room for choice here.

I've changed this to ISO-8859-1.  One advantage of ISO-8859-1 is that
it can't fail to decode, so we can remove some of the sadness that
happens when UTF-8 decoding fails.

> conformance requirements are. If some parts are optional, these need to be
> marked.

I'm happy to do that once we've agreed on the substance.  Bjoern had
some test cases we should be sure we're happy with.  If you're willing
to update your page with the rest of the test cases, I can look at
them and adjust the text accordingly.

> If *all* of this is optional (you can pick whatever you like), then this is
> just another way to document the current state of implementations, in which
> case I'd recommend we just augment what's already in Appendix C.

I don't really understand what you mean by this paragraph.  The
framing we've been discussing is "if you'd like more information about
consuming the Content-Disposition header, here's a recommendation for
how you might like to do that."

>> Silliness isn't one of the criteria I've applied.

Great.  Let me know when it's posted and I'll take a look.  I really
appreciate your cross-testing a number of browsers.  That's
tremendously helpful in figuring out what to do in these cases.

> <http://greenbytes.de/tech/tc2231/#attwithasciifnescapedchar>

You write:

fail (saves "oo.html" (what's going on here?, see Chrome Issue 52577))

what's going on is that the "\" is being treated as a directory
separator and Chrome is giving you the "leaf" name of the path.
There's another level of platform-specific transformations that get
applied to the suggested filename, but I didn't include those in this
document.  It's a bit fuzzy where to draw the line between the two
transformations, but I tried to stick to segmenting the file name out
of the header and (roughly) character-by-character decoding.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Dec 2010 11:41:08 +1100

Hi Adam,

Thanks. 

I think this is a workable starting point for a definition of an optional algorithm for parsing the header + handling errors encountered in that process. In that way, it could be similar to other RFC appendices that give example code for parsing; the only difference here being that it's example psuedo-code.

A few things need to happen next:

* We need to assure that it doesn't conflict with the rest of the C-D spec, adjusting either it or the spec as necessary, and documenting where we don't have interop. Based on the discussion so far with Julian and Bjoern, it seems that's under way.

* We need to get other UA vendors on board; just having it reflect Chrome's behaviour isn't productive. Many are on-list, but I'll ping those I know to make sure they're aware. Please talk to those you know and make sure they know it's important that we have their input / buy-in here.

* One way or another, I'd like to get the C-D draft submitted for IETF LC by the holidays. If we can get this appendix hammered out by then, we can include it; if not, we can work on it as a separate document. 

To help move us along, it might be good to get the draft text somewhere where it can be collaboratively edited and viewed. How about on the WG Wiki?

Regards,

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 16:51:55 -0800

> * We need to assure that it doesn't conflict with the rest of the C-D spec, adjusting either it or the spec as necessary, and documenting where we don't have interop. Based on the discussion so far…

Yep.

> * We need to get other UA vendors on board; just having it reflect Chrome's behaviour isn't productive. Many are on-list, but I'll ping those I know to make sure they're aware. Please talk to those…

That's wise.

> * One way or another, I'd like to get the C-D draft submitted for IETF LC by the holidays. If we can get this appendix hammered out by then, we can include it; if not, we can work on it as a separa…

There don't seem to be that many technical differences.  Hopefully
we'll be able to make that deadline.

> To help move us along, it might be good to get the draft text somewhere where it can be collaboratively edited and viewed. How about on the WG Wiki?

Done:
http://trac.tools.ietf.org/wg/httpbis/trac/wiki/ContentDispositionErrorHandling

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Dec 2010 11:54:40 +1100

One personal comment -- I know we talked about making it as declarative as possible, but looking at this, I wonder if having a separate set of optional BNF really helps. Thoughts?

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 17:00:01 -0800

> One personal comment -- I know we talked about making it as declarative as possible, but looking at this, I wonder if having a separate set of optional BNF really helps. Thoughts?

I suspect someone more skilled in BNF would be able to construct more
beautiful grammars.

The "Extracting Parameter Values From Header Fields" grammar is
certainly more compact and understandable than the corresponding
algorithm.

In "Decoding the File Name", I'm hopeful we can get rid of the grammar
now that we don't need to handle the RFC2047 complexity and just apply
the operation to the whole string instead of word-by-word.

The grammar in "Determining the Disposition" seems pretty silly, but
is also easy to understand.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Dec 2010 10:54:31 +0100

> else" would be RFC 5987 encoding, but Safari and IE do not understand

Related to this: I realized yesterday that we don't have a test case for 
that. Now we do:

   <http://greenbytes.de/tech/tc2231/#attwithfn2231dpct>

Sadly, this fails with Chrome 9, which apparently post-processes the 
result of RFC5987-decoding in an undesirable way.

I believe this should be fixed before Chrome 9 ships, see

   <http://code.google.com/p/chromium/issues/detail?id=57830#c22>

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Dec 2010 13:26:46 +0100

> Silliness isn't one of the criteria I've applied.

I added

<http://greenbytes.de/tech/tc2231/#attmissingdisposition2>

which fails for FF3/Chrome/Chrome9 (I see shared bugs :-), and

<http://greenbytes.de/tech/tc2231/#emptydisposition>

which fails just for FF3.

> at some more evidence before changing it.

I'm not totally sure what exactly to test; please elaborate.

> Sounds like another good test case.

<http://greenbytes.de/tech/tc2231/#dispextbadfn>

failing in Chrome only.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Dec 2010 13:59:05 +0100

>> Ok, here's a +1 to exclude it.

That's good, but now I'm asking myself what the precise goal of this 
exercise is.

Writing down what Chrome does was a good idea in that it helped adding a 
few more test cases, of which many fail *only* with Chrome, and some 
fail for FF3 as well (not entirely surprising as the original code was 
written by the same author).

It would be cool to get the same information for closed-source 
implements like IE and Safari. (I exclude Opera here because it already 
works well in comparison).

Now we're moving away from documenting "what Chrome does" to something 
else. What exactly? What's the purpose? Do you want UAs to converge on 
that behavior? Even those who currently reject invalid header fields?

Note that we're targeting "Proposed Standard" here. It would be great to 
get this published, see how implementations improve (see both Chrome 9 
and Firefox post version 4), and *then* work on an implementation report 
for Draft Standard.

> the document.  That is indeed a warning.

The point is that we already have spec text, which is a warning. Do you 
want it to change?

>> spec-compliant in which case the spec already tells you what to do.

See above, I'm struggling to understand what the proposal actually is. 
(such as: placement, introduction, implication on conformance, ...).

Thanks.

> them and adjust the text accordingly.

I can't "agree" on anything unless I know what it is for, sorry.

> how you might like to do that."

I wouldn't want to "recommend" a behavior that makes existing 
implementations less compliant for valid messages.

I'm less concerned about processing invalid messages, but I'll say again 
that there's little interoperability for those messages, so I just don't 
see why we care.

> ...
> On Wed, Dec 1, 2010 at 3:19 PM, Julian Reschke<[email-masked]>  wrote:

See preceding mail.

> separator and Chrome is giving you the "leaf" name of the path.

OK, so it fails to do the unescaping on quoted-string. It would be great 
if this could be fixed.

Looking at 
<http://trac.tools.ietf.org/wg/httpbis/trac/wiki/ContentDispositionErrorHandling?version=7>:

> Determining the Disposition
>
> To determine the disposition-type, parse the Content-Disposition header field using the following grammar:
>
> unparsed-string = *LWS nominal-type *OCTET
> If the Content-Disposition header field is non-empty and fails to parse, then the disposition type is "attachment". Otherwise, the disposition-type is "inline".

Neither "filename" nor "name" are disposition types. It suggests that 
you can leave out the disposition type and get it treated as attachment; 
<http://greenbytes.de/tech/tc2231/#attmissingdisposition> indicates 
otherwise.

> Extracting Parameter Values From Header Fields ¶
>
> To extract the value for a given parameter-name from an unparsed-string, parse the unparsed-string using the following grammar:
>
> unparsed-string = *OCTET name *LWS "=" value [ ";" *OCTET ]
> value           = <OCTET, except ";">
>
> where the name production is a gramatical production that is a case-insensitive match for the given parameter-name. If the unparsed-string can be parsed by the grammar in multple ways, choose the one in which name appears as close to the beginning of the string as possible. If the unparsed-string cannot be parsed by the grammar above, return the empty string.

This doesn't handle quoted strings.

> Decoding the File Name ¶
>
> To filename-decode an encoded-string, use the following algorithm:
>
>    1. If the encoded-string contains non-ASCII characters, emit the encoded-string (decoded as ISO-8859-1) and abort these steps.

So by adding a non-ASCII character I can prevent percent-unescaping? Is 
this implemented anywhere?

>    2. Let the url-unescaped-string be the encoded-string %-unescaped.
>    3. Emit the url-unescaped-string (decoded as UTF-8). (There's actually more sadness here if the url-unescaped-string isn't valid UTF-8.)
>

<permathread>Why would we recommend something that only Chrome and IE do 
(and IE only does for some locals)</permathread>

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Fri, 3 Dec 2010 15:56:57 -0800

On Thu, Dec 2, 2010 at 4:26 AM, Julian Reschke <[email-masked]> wrote:
> which fails for FF3/Chrome/Chrome9 (I see shared bugs :-),

Hum...  This one sounds a bit tricky.  It's not clear to me which
option is better.

> and

Thanks.

> I'm not totally sure what exactly to test; please elaborate.

Content-Disposition: xfilename=foo.txt

> <http://greenbytes.de/tech/tc2231/#dispextbadfn>

Oh good.  I'll update the wiki.  With a more elaborate grammar.

On Thu, Dec 2, 2010 at 4:59 AM, Julian Reschke <[email-masked]> wrote:
> Writing down what Chrome does was a good idea in that it helped adding a few
> more test cases, of which many fail *only* with Chrome, and some fail for
> FF3 as well (not entirely surprising as the original code was written by the
> same author).
>
> It would be cool to get the same information for closed-source implements
> like IE and Safari. (I exclude Opera here because it already works well in
> comparison).
>
> Now we're moving away from documenting "what Chrome does" to something else.
> What exactly? What's the purpose?

>From my perspective, I'd like there to be a specification of how a
user agent should consume the Content-Disposition header.  I started
with Chrome's behavior because I'm most familiar with it and because
there's evidence that at least one implementor is willing to ship that
behavior.

Ideally, we'd get feedback from other user agent implementors about
what they'd like the specification to say.  We'd then have an easier
time polishing away the more exotic behaviors.  Instead, we're relying
on our collective judgement.

> Do you want UAs to converge on that behavior?

Yes.

> Even those who currently reject invalid header fields?

If a UA wants to reject invalid header fields, that sounds fine to me.
 What I'd like to avoid is there being N different ways of consuming
Content-Disposition, where N is the number of user agent
implementations.

> Note that we're targeting "Proposed Standard" here. It would be great to get
> this published, see how implementations improve (see both Chrome 9 and
> Firefox post version 4), and *then* work on an implementation report for
> Draft Standard.

In this discussion, you keep saying things that imply that we can't
write specs for user agents until all the user agents already have the
same behavior.  We're not mind readers.  It's quite helpful to have a
document that explains how you're supposed to consumer these headers.

> The point is that we already have spec text, which is a warning. Do you want
> it to change?

Personally, I don't feel that strongly about it.  However, I do feel
strongly about keeping the %-decoding in the UA Appendix.  If you're
fine with having both the warning and the %-decoding in the appendix,
that's a workable solution.  If you feel these are in conflict, then
I'd rather change the warning to an error and keep the %-decoding in
the appendix than remove the %-decoding from the appendix.

> See above, I'm struggling to understand what the proposal actually is. (such
> as: placement, introduction, implication on conformance, ...).

We've been talking about putting it in the appendix.  I'm not sure
whether you need to reference it from the introduction.  It doesn't
affect conformance for any conformance class.

>>> is
> that there's little interoperability for those messages, so I just don't see
> why we care.

We care because we want there to be more interoperability in the
future.  The goal of writing standards is to improve interoperability.

> OK, so it fails to do the unescaping on quoted-string. It would be great if
> this could be fixed.

I'm not sure what you mean by "fixed."  It's unclear whether user
agents want to do \-decoding on the file name, especially because \ is
a common directory separator on some operating systems.

>> If the Content-Disposition header field is non-empty and fails to parse,
>> then the disposition type is "attachment". Otherwise, the disposition-type
>> is "inline".
>
> Neither "filename" nor "name" are disposition types.

Indeed.

> It suggests that you
> can leave out the disposition type and get it treated as attachment;

I'm not sure I understand what you're saying.  The wiki text matches
UA behavior for
http://greenbytes.de/tech/tc2231/#attmissingdisposition.  Is there
another test case you're worried about?

>> Extracting Parameter Values From Header Fields
>>
>> To extract the value for a given parameter-name from an unparsed-string,
>> parse the unparsed-string using the following grammar:
>>
>> where the name production is a gramatical production that is a
>> case-insensitive match for the given parameter-name. If the unparsed-string
>> can be parsed by the grammar in multple ways, choose the one in which name
>> appears as close to the beginning of the string as possible. If the
>> unparsed-string cannot be parsed by the grammar above, return the empty
>> string.
>

How would you like quoted strings to be handled.  According to your
tests, what we should do is strip off matching leading and trailing "
characters and be careful to capture ; inside of ".  However, your
tests show that we should not \-decode the value.  I'm happy to make
that change.

>> Decoding the File Name
>>   1. If the encoded-string contains non-ASCII characters, emit the
>> encoded-string (decoded as ISO-8859-1) and abort these steps.
>
> So by adding a non-ASCII character I can prevent percent-unescaping? Is this
> implemented anywhere?

I'd encourage you to write a test and find out.  :)

>>   3. Emit the url-unescaped-string (decoded as UTF-8). (There's actually
>> more sadness here if the url-unescaped-string isn't valid UTF-8.)
>>

As you indicate, we've discussed this issue at length.  If you can
convince IE to remove this behavior, then we might be able to remove
it from this document.  Otherwise, we'd like to compete with IE in
this respect.

Adam

---

From: Adam Barth <[email-masked]>
Date: Sat, 4 Dec 2010 02:18:18 -0800

> Content-Disposition: xfilename=foo.txt

Note: I've removed support for this syntax on the wiki.  It seemed
particularly goofy.

> Oh good.  I'll update the wiki.  With a more elaborate grammar.

I've taken a stab at writing this requirement with a grammatical
presentation.  I'd appreciate some help with the grammar as it's not
my strong suit.  Here's what I've got (also on the wiki):

[[
unparsed-string = unbalanced-block / block * ( ";" block ) [ ";"
unbalanced-block ]
block = *run
unbalanced-block = *run unbalanced-run
run = unquoted-run / quoted-run
unquoted-run = non-quote *boring-octet
quoted-run = <"> *non-quote <">
unbalanced-run = <"> *non-quote
non-quote = <OCTET, except <"> >
boring-octet = <OCTET, except <"> and ";">
]]

I'm not overly confident in the first grammar.  What I'm trying to
encode is a parser with two states: initial and in-quote, together
with the following transition rules:

== State: initial ==
<"> => Buffer the current character, switch to the in-quote state, and
proceed to the next character.
";" => Emit the buffered characters, if any, and proceed to the next character.
EOF => Emit the buffered characters, if any, and terminate.
OTHER => Buffer the current character and proceed to the next character.

== State: in-quote ==
<"> => Buffer the current character, switch to the initial state, and
proceed to the next character.
EOF =>Emit the buffered characters, if any, and terminate.
OTHER => Buffer the current character and proceed to the next character.

My inclination is to write the state machine into the wiki, but I'm
hesitating because my understanding is that you all would prefer a
grammatical presentation.

Thanks,
Adam

---

From: Julian Reschke <[email-masked]>
Date: Sat, 04 Dec 2010 12:18:11 +0100

On 04.12.2010 00:56, Adam Barth wrote:
> On Thu, Dec 2, 2010 at 4:26 AM, Julian Reschke<[email-masked]>  wrote:
>> I added
>>

The specs are clear on it, and all UAs except FF and Chrome get this 
right. This seems to be a case where it's clear what's "better" (even if 
we may disagree on the metrics for "better").

>> which fails just for FF3.

Let's hope that this can get resolved in FF when the C-D related changes 
land after release of FF4.

> Content-Disposition: xfilename=foo.txt

I have added a test for

   Content-Disposition: attachment; xfilename=foo.txt

(the one you proposed would have been invalid anyway). This is

   <http://greenbytes.de/tech/tc2231/#attconfusedparam>

and it fails for Chrome only (reported as 
<http://code.google.com/p/chromium/issues/detail?id=65423>).

> Oh good.  I'll update the wiki.  With a more elaborate grammar.

See also <http://code.google.com/p/chromium/issues/detail?id=65276>.

> there's evidence that at least one implementor is willing to ship that

The specification says how to consume *valid* header instances (I know 
you know that, but I'm repeating it for people who might not have read 
all of this thread).

What's up for discussion is whether we want to talk about handling 
invalid headers, how to do that, and what kind of conformance comes with 
that.

> on our collective judgement.

As I said before, I'm not very interested in getting more interop for 
broken headers. What concerns me much more is interop for valid headers, 
and delaying the spec again and again doesn't help here.

So, is there any *hope* that we'll see that feedback from the other 
browser vendors? Maciej? Anne? Eric? Robert?

>> Do you want UAs to converge on that behavior?

Then I'd propose that you add an introduction to the Wiki clearly saying 
how you want it to appear in the spec.

Personally, I don't think it's a useful exercise for this Working Group, 
as the observed behavior for broken headers differs a lot (from proper 
parsing as in Konqueror to naive substring matching in some other UAs :-).

This is different, from, for instance, handling broken HTML (where we 
actually have evidence that UAs need to do this to stay in business).

> Content-Disposition, where N is the number of user agent

There should be one way to handle valid characters. I'm less interested 
in consistent behavior for invalid headers *unless* there's a related 
security risk.

As this whole discussion also applies to the actual set of HTTP specs we 
want to finish, I'd appreciated feedback from other WG members on 
whether they are interested in adding this type of information. And 
those who are please say whether you're willing to contribute test cases 
and report back from your code bases.

> document that explains how you're supposed to consumer these headers.

On the contrary. What I was trying to say is that this is for 
"Proposed". There's a next stage which, in the IETF process, is actually 
the first stage where we're looking at implementations. If this WG can't 
come up with a consensus on this issue, there will be another 
opportunity when we go to Draft Standard.

> the appendix than remove the %-decoding from the appendix.

I think they are in conflict, and wouldn't want to see any 
recommendation to do %-unescaping. It would make the spec incompatible 
with the previous spec (a conformance change), and it's only implemented 
in two out of six UAs I'm testing.

> affect conformance for any conformance class.

By all means please provide the complete text for the introduction of 
the Appendix. That's essential to understand what the expectations on 
implementations are.

If there aren't any, such as "we just think this is a good idea" I'd 
propose to have it in a separate document which may or not may be a WG 
work item.

> future.  The goal of writing standards is to improve interoperability.

Yes, but we usually draw a line between things we care about and things 
we don't. We happen to disagree on where to draw that line.

> a common directory separator on some operating systems.

"Fixing" means "changing things to work as specified".

So the question here is whether it would break things because there are 
servers sending unescaped backslashes. As far as I can tell, sending 
path separators in the filename indicates a bug in the sender, or an 
attempt to trick the user agent to do something it's not supposed to do.

So the "harm" of actually doing the unescaping would be that for a 
filename that needs to be postprocessed anyway, the problematic 
character would be filtered in a different way.

Starting with

   filename="a\bc"

the broken implementation sees "a" and "bc" separated by a path 
separator, and will prost-process this to "abc", "a_bc" or "bc" (where _ 
could be a different replacement character).

A correct implementation sees "abc".

I don't think there's a problem here.

>> Neither "filename" nor "name" are disposition types.

It's confusing and makes reviewing the text harder than it needs to be.

> another test case you're worried about?

No, sorry, I got this wrong.

>>> value           =<OCTET, except ";">
>>>

Quoted strings should be handled as specified, removing the quotes and 
performing \-unescaping. The tests show that indeed a majority of UAs 
get this wrong but that doesn't make it magically right.

I'd prefer that we invest our time to reduce the bugs in the UAs, 
instead of documenting them.

Note that fixing the quoted-string handling has already a proposed patch 
in Mozilla.

> I'd encourage you to write a test and find out.  :)

Writing tests doesn't come for free, and every test I add needs to be 
maintained and re-run. So I'd prefer to understand it's worth the time 
before.

> it from this document.  Otherwise, we'd like to compete with IE in

So can you convince Safari, Opera, Firefox, and Konqueror to adopt this 
handling as well? Otherwise I don't think we'll make progress. Two UAs 
do something funny, the other four do not. I don't want the 
specification to reflect those implementation bugs -- even if they can't 
be realistically removed from these UAs anytime soon.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 04 Dec 2010 12:25:41 +0100

On 04.12.2010 11:18, Adam Barth wrote:
> On Fri, Dec 3, 2010 at 3:56 PM, Adam Barth<[email-masked]>  wrote:

It is. Thanks.

> quoted-run =<">  *non-quote<">
> unbalanced-run =<">  *non-quote
> non-quote =<OCTET, except<">  >
> boring-octet =<OCTET, except<">  and ";">
> ]]
>

Before we invest too much time into this: do you have evidence that 
using a grammar other than the one already defines is really needed?

Maybe you could list certain invalid header instances you want to 
handle, and what the result should be?

If we want to keep handling valid header fields properly, we *probably* 
better start from the base grammar, and relax certain constructs.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Sat, 4 Dec 2010 10:16:19 -0800

On Sat, Dec 4, 2010 at 3:18 AM, Julian Reschke <[email-masked]> wrote:
> The specs are clear on it, and all UAs except FF and Chrome get this right.
> This seems to be a case where it's clear what's "better" (even if we may
> disagree on the metrics for "better").

I'm not sure it's so clear-cut.  I'll dig around a bit and see what
more information I can find.

> <http://code.google.com/p/chromium/issues/detail?id=65423>).

Great.  Fixed.

> See also <http://code.google.com/p/chromium/issues/detail?id=65276>.

Thanks for filing these bugs.  That's helpful.  My plan is to change
Chrome to match the spec once the spec has settled down.

> The specification says how to consume *valid* header instances (I know you
> know that, but I'm repeating it for people who might not have read all of
> this thread).
>
> What's up for discussion is whether we want to talk about handling invalid
> headers, how to do that, and what kind of conformance comes with that.

The path we're going down is to provide optional instructions for how
to consume the Content-Disposition header.  We have two competing sets
of contraints:

1) These instructions are limited to things that user agents are
willing to implement.
2) These instructions are limited to aligning with the requirements on
valid header fields.

Where we're having difficulty agreeing is where these two constraints
are in conflict.  There are three options:

A) Relax one of the constraints.
B) Change what user agents are willing to implement.
C) Change the definition of valid header fields to exclude syntax
where user agents aren't willing to implement the required behavior.

> As I said before, I'm not very interested in getting more interop for broken
> headers. What concerns me much more is interop for valid headers, and
> delaying the spec again and again doesn't help here.

Indeed.  We have different goals and priorities.

> So, is there any *hope* that we'll see that feedback from the other browser
> vendors? Maciej? Anne? Eric? Robert?

I suspect asking for their input in the middle of this giant thread
isn't the most likely way to get feedback.  :)

> Then I'd propose that you add an introduction to the Wiki clearly saying how
> you want it to appear in the spec.
>
> Personally, I don't think it's a useful exercise for this Working Group, as
> the observed behavior for broken headers differs a lot (from proper parsing
> as in Konqueror to naive substring matching in some other UAs :-).
>

Again, we have perspectives on what's useful.

> There should be one way to handle valid characters. I'm less interested in
> consistent behavior for invalid headers *unless* there's a related security
> risk.

Fortunately we control the definition of what's valid so we can make
this the case by picking option (C) above.

>>> want
> I think they are in conflict, and wouldn't want to see any recommendation to
> do %-unescaping. It would make the spec incompatible with the previous spec
> (a conformance change), and it's only implemented in two out of six UAs I'm
> testing.

Ok, then it sounds like we should make including a % syntactically
invalid.  Is there another solution?

>>> (such
> By all means please provide the complete text for the introduction of the
> Appendix. That's essential to understand what the expectations on
> implementations are.
>
> If there aren't any, such as "we just think this is a good idea" I'd propose
> to have it in a separate document which may or not may be a WG work item.

>From my perspective, the introductory text is less important than the
technical content.  If you'd like to propose introductory text that
makes you happy, that might be a more productive starting point than
me guess wildly.

>>> see
> Yes, but we usually draw a line between things we care about and things we
> don't. We happen to disagree on where to draw that line.

Indeed.

>>> if

Alternatively, we could change the specification to match what user
agents are willing to implement.

> servers sending unescaped backslashes. As far as I can tell, sending path
> separators in the filename indicates a bug in the sender, or an attempt to
> trick the user agent to do something it's not supposed to do.
>
> So the "harm" of actually doing the unescaping would be that for a filename
> that needs to be postprocessed anyway, the problematic character would be
> filtered in a different way.
>
> Starting with
>
>  filename="a\bc"
>
> the broken implementation sees "a" and "bc" separated by a path separator,
> and will prost-process this to "abc", "a_bc" or "bc" (where _ could be a
> different replacement character).
>

None of the user agents do \-decoding.  I don't see any value in them starting.

> It's confusing and makes reviewing the text harder than it needs to be.

Feel free to propose improvements.  I'm not sure part is confusing you.

> performing \-unescaping. The tests show that indeed a majority of UAs get
> this wrong but that doesn't make it magically right.

I disagree that we should do \-unescaping.

> I'd prefer that we invest our time to reduce the bugs in the UAs, instead of
> documenting them.

Again, we have different priorities.

> Note that fixing the quoted-string handling has already a proposed patch in
> Mozilla.

Bug number?

> maintained and re-run. So I'd prefer to understand it's worth the time

Well, then I can answer you question.  Yes.

> handling as well? Otherwise I don't think we'll make progress. Two UAs do
> something funny, the other four do not. I don't want the specification to
> reflect those implementation bugs -- even if they can't be realistically
> removed from these UAs anytime soon.

I think it's more likely that these user agents will adopt %-decoding
than that the other user agents will drop %-decoding.  Now, that's a
matter of opinion and reflects a judgement call.

> Before we invest too much time into this: do you have evidence that using a
> grammar other than the one already defines is really needed?

Yes.  The one currently in the document doesn't match every sequence
of input characters.  Therefore, it does not define a complete parser.
 We desire a complete specification of how to parse the header field.

> Maybe you could list certain invalid header instances you want to handle,
> and what the result should be?

Content-Disposition: attachment; filename="bar

Result => "bar

Content-Disposition: attachment; filename=foo"bar;baz"qux

Result => foo"bar;baz"qux

Content-Disposition: "foo; filename=bar;baz"; filename=qux

Result => qux

etc

> better start from the base grammar, and relax certain constructs.

Feel free to propose a modification to the base grammar that matches
every sequence of bytes and computes the needed segmentation.  It's
not at all obvious to me how to approach constructing the grammar from
that starting point.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Sat, 04 Dec 2010 20:59:29 +0100

> more information I can find.

What more information do you need?

>> See also<http://code.google.com/p/chromium/issues/detail?id=65276>.
>

The applicable spec hasn't really changed since it was published many 
years ago. The Internet Draft we're discussing is just clarifying and 
profiling.

> 1) These instructions are limited to things that user agents are

Not necessarily. I'm not aware of an IETF rule for that. As a matter of 
fact, what "UAs are willing to implement" may change over time.

> where user agents aren't willing to implement the required behavior.

D) Live with the fact that some user agents are doing the wrong thing, 
and document it. Warn about what can happen. Suggest alternatives when 
available.

> isn't the most likely way to get feedback.  :)

All of them (maybe except Robert whose name was the first Firefox person 
who came to mind who should be on this list) are aware of this discussion.

> invalid.  Is there another solution?

Yes, we can just re-state what has been true given the existing specs. 
It's a bug to do percent-unescaping here, but some UAs do it nevertheless.

> makes you happy, that might be a more productive starting point than

OK, here's my proposal:

"The User Agent behavior outlined below is derived from one specific 
implementation, with some aspects that cause invalid behavior for valid 
header fields subsequently removed. We observe that UAs differ in their 
handling of invalid headers, and that there's currently no uniform 
algorithm that could be specified."

> agents are willing to implement.

Alternatively, we could observe that some UAs get this right, and 
conclude that there's no agreement that breaking the specs is required.

> None of the user agents do \-decoding.  I don't see any value in them starting.

Konqueror does. Opera does. Mozilla has a proposed patch 
(<https://bugzilla.mozilla.org/show_bug.cgi?id=584418>).

> Feel free to propose improvements.  I'm not sure part is confusing you.

It would be less confusing if the grammar actually contained the legal 
values for "disposition".

>> Note that fixing the quoted-string handling has already a proposed patch in

<https://bugzilla.mozilla.org/show_bug.cgi?id=584418>

> Well, then I can answer you question.  Yes.

Added as <http://greenbytes.de/tech/tc2231/#attwithfilenamepctandiso>. 
Chrome behaves as described by you, IE does not (the others do not 
percent-unescape anyway).

> matter of opinion and reflects a judgement call.

Well, let's disagree on that.

The discussion has been going on for a very long time now, and I see no 
change.

> Content-Disposition: attachment; filename="bar

See <http://greenbytes.de/tech/tc2231/#attbrokenquotedfn2>.

Only Safari does what you say.

> Result =>  foo"bar;baz"qux

See <http://greenbytes.de/tech/tc2231/#attbrokenquotedfn3>.

Only Opera and Safari do what you say.

> Content-Disposition: "foo; filename=bar;baz"; filename=qux

See <http://greenbytes.de/tech/tc2231/#attmissingdisposition3>.

None of the UAs does what you say.

> etc

Yes, we can invent all kinds of new examples. What we see is that 
because the UAs all have different parsers, they fail in different ways 
(or in some cases, pass, thanks Konqueror and Opera).

There is no interop for this kind of garbage, so I see no incentive to 
define something other than rejecting those values.

> not at all obvious to me how to approach constructing the grammar from

It does not need to match any sequence. It's completely ok to state what 
it means not to match the grammar.

So let's start with:

   content-disposition = "Content-Disposition" ":"
                          disposition-type *( ";" disposition-parm )

   disposition-type    = "inline" | "attachment" | disp-ext-type
                       ; case-insensitive
   disp-ext-type       = token

   disposition-parm    = filename-parm | disp-ext-parm

   filename-parm       = "filename" "=" value
                       | "filename*" "=" ext-value

   disp-ext-parm       = token "=" value
                       | ext-token "=" ext-value
   ext-token           = <the characters in token, followed by "*">

We could relax requirements on ordering and cardinality of 
disposition-types:

   content-disposition = "Content-Disposition" ":"
                         ( disposition-parm / disposition-type )
                          *( ";" ( disposition-parm / disposition-type ))

and then relax the grammars for "token" and "quoted-string" to allow 
more characters that are not needed for detecting names and values. I 
think that would reduce the risk of defining something that breaks valid 
headers.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Dec 2010 11:04:24 +1100

This is an attractive approach, but I'm concerned that opening up the 'core' grammar to accommodate this might make some existent implementations too strict -- e.g., they'll reject characters that were previously disallowed but now in the ABNF.

I.e., the UA profile wouldn't be optional any more.

To address this, we'd probably need to discriminate between produce and consume ABNF, just as we've done elsewhere.

Regards,

On 05/12/2010, at 6:59 AM, Julian Reschke wrote:

> It does not need to match any sequence. It's completely ok to state what it means not to match the grammar.
> 
> So let's start with:
> 
> We could relax requirements on ordering and cardinality of disposition-types:
> 
> and then relax the grammars for "token" and "quoted-string" to allow more characters that are not needed for detecting names and values. I think that would reduce the risk of defining something that breaks valid headers.

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 13 Dec 2010 01:06:13 -0800

Here are some comments from my colleague Alexey Proskuryakov on your proposal. I know these may have been outpaced by the considerable discussion since that point, but they still seem like they could be useful.

> I only know about file name decoding - all parsing is of course in CFNetwork, and most logic is in Launch Services, I think.
> 
> Adam's proposal is a step forward in that it acknowledges the need to process raw non-ASCII bytes in filename, which is the only encoding style that matters. He also describes the proper algorithm, acknowledging that Chrome doesn't fully implement it. Unsurprisingly, that part was met with resistance from the "we always told you it was ISO-8859-1" crowd.
> 
> I agree that RFC2047 style encoding shouldn't be supported, and I'm ambivalent about RFC5987. RFC2231/5987 is a step in the wrong direction (opaque encoding for something that doesn't need it), but given that IETF won't cease pushing it, we might as well implement it and be more compatible with Firefox, if not the Web.
> 
> - WBR, Alexey Proskuryakov

>> Do the details of filename-decoding seem reasonable? They don't seem to quite match what we do, in particular I don't think they have Latin1 as the final fallback.
>> 
>>  - Maciej

> Yes, besides the lack of latin-1 fallback, the described sequence matches ours.
> 
>     // Always try UTF-8. If that fails, try frame encoding (if any) and then the default.
>     // For a newly opened frame with an empty URL, encoding() should not be used, because this methods asks decoder, which uses ISO-8859-1.
>     Settings* settings = m_frame->settings();
>     request.setResponseContentDispositionEncodingFallbackArray("UTF-8", writer()->deprecatedFrameEncoding(), settings ? settings->defaultTextEncodingName() : String());
> 
> I guess I should have mentioned the implied latin-1 fallback in the comment...

> 
> - WBR, Alexey Proskuryakov

In case anyone is wondering, the reason we use Latin1 as the final fallback is that it never fails to decode, so its guaranteed to give something, if our other attempts failed.

Regards,
Maciej

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Dec 2010 10:28:28 +0100

Hi Maciej,

thanks for forwarding.

On 13.12.2010 10:06, Maciej Stachowiak wrote:
> Here are some comments from my colleague Alexey Proskuryakov on your
> proposal. I know these may have been outpaced by the considerable
> discussion since that point, but they still seem like they could be useful.
>
>> I only know about file name decoding - all parsing is of course in
>> CFNetwork, and most logic is in Launch Services, I think.
>>
>> Adam's proposal is a step forward in that it acknowledges the need to
>> process raw non-ASCII bytes in filename, which is the only encoding

That's incorrect in that the base spec already says it's ISO-8859-1 
(although this may be hard to find in the published specs as opposed to 
the Internet Draft we're discussing).

(maybe this is a case where Alexey looked at an old proposal?)

>> style that matters. He also describes the proper algorithm,
>> acknowledging that Chrome doesn't fully implement it. Unsurprisingly,
>> that part was met with resistance from the "we always told you it was
>> ISO-8859-1" crowd.
>>
>> I agree that RFC2047 style encoding shouldn't be supported, and I'm
>> ambivalent about RFC5987. RFC2231/5987 is a step in the wrong
>> direction (opaque encoding for something that doesn't need it), but
>> given that IETF won't cease pushing it, we might as well implement it
>> and be more compatible with Firefox, if not the Web.
>>

In a perfect world we could declare that the HTTP header encoding is 
UTF-8. But it isn't.

If we *tried* to change the default encoding just for C-D/filename, we 
will still break existing code.

So I'm not sure what the "doesn't need it" refers to.

Best regards, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 13 Dec 2010 02:54:32 -0800

I wanted to add, for clarity, that we don't see any major problems in Adam's proposal as-is, though we'd suggest adding ISO-8859-1 as a final fallback.

If there is a test suite that matches the expectations of Adam's proposal and is easy to run, I'll try to get someone to run it. Or if this testing has already been done, I can comment on the ways it diverges from Safari behavior and whether we are likely to care.

On Dec 13, 2010, at 1:28 AM, Julian Reschke wrote:

> That's incorrect in that the base spec already says it's ISO-8859-1 (although this may be hard to find in the published specs as opposed to the Internet Draft we're discussing).
> 

I suspect he looked at the existing published RFC. In any case, treating everything as Latin1 is likely not acceptable to us. We came up with our (somewhat complicated) rule through a long process of trial and error based on bug reports and user requirements. 

> In a perfect world we could declare that the HTTP header encoding is UTF-8. But it isn't.
> 
> If we *tried* to change the default encoding just for C-D/filename, we will still break existing code.
> 

Treating filenames in Content-Disposition as Latin1 will in any case break existing code. I think the WG's only choices here are which code to break, and by how much.

Regards,
Maciej

---

From: Adam Barth <[email-masked]>
Date: Mon, 13 Dec 2010 03:00:50 -0800

Done.

> If there is a test suite that matches the expectations of Adam's proposal and is easy to run, I'll try to get someone to run it. Or if this testing has already been done, I can comment on the ways…

Julian has written a nice test suite.  We'll just need to set the expectations.

> I suspect he looked at the existing published RFC. In any case, treating everything as Latin1 is likely not acceptable to us. We came up with our (somewhat complicated) rule through a long process…

The rule in my proposal isn't quite as complicated as what Safari
implements.  In particular, the proposal doesn't take the current
frame encoding into account.

> Treating filenames in Content-Disposition as Latin1 will in any case break existing code. I think the WG's only choices here are which code to break, and by how much.

I've added back UTF-8 as the first encoding to try based on your and
Alexey's input.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Dec 2010 15:13:28 +0100

Hi there,

..trying to do some cherry-picking...:

>> and be more compatible with Firefox, if not the Web.

RFC 2047 encoding is currently only done in Firefox (for which I raised 
a bug report a few months ago) and Chrome. For the latter I just raised 
<http://code.google.com/p/chromium/issues/detail?id=66694>; maybe we can 
make some progress on getting rid of this.

Also, RFC 2231/5987 encoding has the interesting property that it can 
handle all the cases where currently interop isn't there (I18N, and 
handling of % and \). If the Safari team is willing to implement this as 
well, we may want to consider changing the RFC 5987 support requirement 
from "SHOULD" to "MUST", in which case we could recommend using it for 
these cases (where most of the remaining disagreement comes from).

That wouldn't help for IE, but at least for all non-IE browsers there'd 
be a reliable way to get these characters over the wire.

Best regards, Julian

---

From: Jungshik Shin (신정식, 申政湜) <[email-masked]>
Date: Wed, 15 Dec 2010 12:19:36 -0800

>>  I agree that RFC2047 style encoding shouldn't be supported, and I'm
IMHO, we have to base our decision on this issue with what web servers
actually emit these days. I meant to collect the stat on C-D header fields
 a long time ago, but haven't managed to do that.

>>> and be more compatible with Firefox, if not the Web.
That's why I had resisted RFC 5987 for rather long while, but 'Chrome' 'gave
in' and implemented it. :-)

>  > ...
>
> RFC 2047 encoding is currently only done in Firefox (for which I raised a
> bug report a few months ago) and Chrome. For the latter I just raised <
> http://code.google.com/p/chromium/issues/detail?id=66694>; maybe we can

Before rushing to remove it (as an optional 'fallback') , I'd like to have
some 'numbers' about what web servers do (FYI, some Google products emit RFC
2047 for Firefox and Chrome at the moment, but I guess Google has to switch
over to RFC 5987 for Firefox and Chrome). I'm not sure whether the cost of
supporting it is larger than the benefit.

 Jungshik

---

From: Julian Reschke <[email-masked]>
Date: Wed, 15 Dec 2010 21:42:36 +0100

Hi Jungshik,

On 15.12.2010 21:19, Jungshik Shin (신정식, 申政湜) wrote:
> ...
> Before rushing to remove it (as an optional 'fallback') , I'd like to
> have some 'numbers' about what web servers do (FYI, some Google products
> emit RFC 2047 for Firefox and Chrome at the moment, but I guess Google
> has to switch over to RFC 5987 for Firefox and Chrome). I'm not sure
> whether the cost of supporting it is larger than the benefit.
> ...

Well, there's an ongoing cost of having many ways to do the same thing, 
each of which not being interoperable.

I believe that of those the RFC 2047 is the worst.

But you are right of course -- data would be great.

Any idea on how long it would take to get this data?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 15 Dec 2010 23:40:06 +0100

> whether the cost of supporting it is larger than the benefit.

Indeed. GMail seems to use RFC2047-encoding (when saving an attachment 
with non-ASCII characters in the filename) for Firefox (and likely for 
Chrome as well).

So it's unlikely that UAs can remove this until this get fixed.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Dec 2010 14:36:51 +1100

Perhaps, but it depends on how Google sniffs the UAs. And I don't think it affects the spec.

On 16/12/2010, at 9:40 AM, Julian Reschke wrote:

> Indeed. GMail seems to use RFC2047-encoding (when saving an attachment with non-ASCII characters in the filename) for Firefox (and likely for Chrome as well).
>

---

From: Adam Barth <[email-masked]>
Date: Wed, 15 Dec 2010 22:27:31 -0800

Why doesn't it affect the spec?  It seems like implementations won't
want to remove support for a feature that's used on a site as popular
as Gmail.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Dec 2010 17:46:12 +1100

Because (if I read the original message correctly -- please correct me if I'm wrong) they're sniffing the UA to do it, and if they do that, they'll presumably adapt their sniffing based upon changes in the browser market (as anyone who sniffs and believes that they don't have to monitor the market tends to get bitten, hard).

Cheers,

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 15 Dec 2010 22:58:21 -0800

> Because (if I read the original message correctly -- please correct me if I'm wrong) they're sniffing the UA to do it, and if they do that, they'll presumably adapt their sniffing based upon change…

Typically in cases like this you want to get sites to change before breaking them. Often it takes surprisingly long for changes like this to get implemented and pushed in a large-scale site, even for a seemingly simple change.

Regards,
Maciej

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Dec 2010 18:01:25 +1100

Yes, I'm passingly familiar with the phenomenon. ;)

What I meant was that browsers can stage the changes over a few versions (as you suggest), e.g., bringing up a console warning and talking about it in their roadmap.

If desirable, we could reflect this in the spec by deprecating it and suggesting a warning now, and then disallowing it when the document progresses on the standards track.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Dec 2010 09:14:49 +0100

> If desirable, we could reflect this in the spec by deprecating it and suggesting a warning now, and then disallowing it when the document progresses on the standards track.

We can't deprecate something that was never allowed. We *can* warn that 
some UAs implement this.

Best regards, Julian

---

From: Jungshik Shin (신정식, 申政湜) <[email-masked]>
Date: Thu, 16 Dec 2010 10:19:02 -0800

On Mon, Dec 13, 2010 at 3:00 AM, Adam Barth <[email-masked]> wrote:

> > I wanted to add, for clarity, that we don't see any major problems in
> Adam's proposal as-is, though we'd suggest adding ISO-8859-1 as a final
> fallback.
>
> Done.
>
> > If there is a test suite that matches the expectations of Adam's proposal
> and is easy to run, I'll try to get someone to run it. Or if this testing
> has already been done, I can comment on the ways it diverges from Safari
> behavior and whether we are likely to care.
>
> (although this may be hard to find in the published specs as opposed to the
> Internet Draft we're discussing).
> >>
> > I suspect he looked at the existing published RFC. In any case, treating
> everything as Latin1 is likely not acceptable to us. We came up with our
> (somewhat complicated) rule through a long process of trial and error based
> on bug reports and user requirements.
>
>
Yes, just assuming ISO-8859-1 is rather Western-Europe-centric (it's like
assuming ISO-8859-1 for unlabelled HTML pages. ISO-8859-1 is supposed to be
assumed for unlabelled HTML pages, but none of major web browsers did that
because that simply does not work for non-Western-European web pages) and is
not acceptable for web compatibility. There are way too many web sites that
emit raw 8-byte sequences in other encodings (that are usually in the
encoding used by 'referring' pages, but not always).

> frame encoding into account.

Huh? In your initial proposal, you mentioned trying 'referrer charset' after
UTF-8. I've just checked your latest draft and that stop was dropped.

I think it does not reflect the reality. As I wrote above, there are way too
many web servers that just emit raw 8-bit byte sequences in 'legacy'
(non-UTF-8) encodings. Their numbers will get smaller as more pages switch
to UTF-8, but it'll be a long while before they're all gone (either
switching to UTF-8 and emitting raw UTF-8 bytes or becoming compliant to RFC
5987, the former of which is more likely for 'small' web sites).

Chrome's implementation has an argument for that and it works most of time.
Firefox has done the same for almost a decade and its implementation works
in more cases). 'referrer charset' is meant to be the character encoding of
a frame that refers to a file being downloaded. Even when that fails, one
can finally resorts to the legacy character encoding most widely used for
the UI language of a UA.

I saw, in Webkit tree that  Safari passes a list of fallback charsets
(including that of the frame charset) to try, but the last time I tried it
with my test pages, it didn't seem to work as expected. That may have
changed since.

In case of IE, depending on the default system code page of Windows where IE
is running, the interpretation of raw byte sequences  is different. A lot of
Chinese and Korean web sites can get away with emitting raw-byte sequences
in Windows-932 (GBK), Windows-950 (Big5), and Windows-949 (extended EUC-KR)
because IE is dominant in those markets (and falling back to the referer
charset supported by Firefox works most of time). Of course, this does not
work if IE runs on WIndows with its default legacy code page set to one
different from that assumed by web sites.  (e.g. IE on Windows with the
default codepage set to windows-1252). To test this, one can change the
default OS (legacy) codepage on Windows (in Control Panel - Regional and
language settings - Advanced - "languages to use for non-Unicode
application"), after which Windows has to be rebooted.

> > Treating filenames in Content-Disposition as Latin1 will in any case
> break existing code. I think the WG's only choices here are which code to
> break, and by how much.
>
I think the charset of a referer has to be tried after UTF-8.

Jungshik

---

From: Jungshik Shin (신정식, 申政湜) <[email-masked]>
Date: Thu, 16 Dec 2010 10:29:09 -0800

On Wed, Dec 15, 2010 at 10:58 PM, Maciej Stachowiak <[email-masked]> wrote:

> > Because (if I read the original message correctly -- please correct me if
> I'm wrong) they're sniffing the UA to do it, and if they do that, they'll
> presumably adapt their sniffing based upon changes in the browser market (as
> anyone who sniffs and believes that they don't have to monitor the market
> tends to get bitten, hard).
>

Yes, gmail sniffs the UA and emits RFC 2047 for Firefox and Chrome and RFC
5987 (RFC 2231) for Opera. The change to emit RFC 5987 for Opera was made
rather recently (before that, non-ASCII characters were just tunred to
question marks for Opera). Anyway, in case of gmail, it's relatively easy to
make (at least I used to know where the code is and I hope it's still there
for me to make a quick change).  Some other google products just turns
non-ASCII characters to question marks for all the UAs (e.g. Google Docs)
:-)   Obviously, it's a rather embarrasing bug to fix.

> Typically in cases like this you want to get sites to change before
> breaking them. Often it takes surprisingly long for changes like this to get
> implemented and pushed in a large-scale site, even for a seemingly simple
> change.
>

Yes, sites have to change before RFC 2047 support is dropped in Chrome and
Firefox.

Jungshik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Dec 2010 19:59:27 +0100

On 16.12.2010 19:29, Jungshik Shin (신정식, 申政湜) wrote:
>
>
> On Wed, Dec 15, 2010 at 10:58 PM, Maciej Stachowiak <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>      > Because (if I read the original message correctly -- please
>     correct me if I'm wrong) they're sniffing the UA to do it, and if
>     they do that, they'll presumably adapt their sniffing based upon
>     changes in the browser market (as anyone who sniffs and believes
>     that they don't have to monitor the market tends to get bitten, hard).
>
>
> Yes, gmail sniffs the UA and emits RFC 2047 for Firefox and Chrome and
> RFC 5987 (RFC 2231) for Opera. The change to emit RFC 5987 for Opera was
> made rather recently (before that, non-ASCII characters were just tunred
> to question marks for Opera). Anyway, in case of gmail, it's relatively
> easy to make (at least I used to know where the code is and I hope it's
> still there for me to make a quick change).  Some other google products
> just turns non-ASCII characters to question marks for all the UAs (e.g.
> Google Docs) :-)   Obviously, it's a rather embarrasing bug to fix.

That would be great.

Actually, I'd change to emit RFC5987 for all UAs *except* those which do 
not support it yet (IE, Chrome, Safari).

>     breaking them. Often it takes surprisingly long for changes like
>     this to get implemented and pushed in a large-scale site, even for a
>     seemingly simple change.
>
>
> Yes, sites have to change before RFC 2047 support is dropped in Chrome
> and Firefox.

Do we happen to know *which* other sites?

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Thu, 16 Dec 2010 12:10:04 -0800

2010/12/16 Jungshik Shin (신정식, 申政湜) <[email-masked]>:
>> > If there is a test suite that matches the expectations of Adam's
>> > proposal and is easy to run, I'll try to get someone to run it. Or if this
>> > testing has already been done, I can comment on the ways it diverges from
>> > Safari behavior and whether we are likely to care.
>>

Thanks Jungshik.  I'm very glad you're participating in this discussion.

I suspect adding the Referer charset to the charset array will make
various members of this working group sad, but I'd rather we write an
honest specification than optimize for the happiness of the working
group.

I'll make this change next time I edit the wiki.  Before Julian and
others bite my head off, I'm aware that this will likely mean you
won't find this text acceptable to include in the current document.
If that's the case, I can expand this text to a full-length
specification, and we can publish it either through this working group
or as an individual submission.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Fri, 17 Dec 2010 10:02:27 +0100

Indeed.

> honest specification than optimize for the happiness of the working

It depends on the goal. If you want to write down what *some* UAs are
currently doing, then you have to do that, agreed.

> or as an individual submission.

In which case it would be great if we could come to a conclusion about
whether we're good with

  <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-content-disp-04.html>

or whether we want to add something like the appendix I proposed in

  <http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0597.html>

Best regards, Julian

---

From: Jungshik Shin (신정식, 申政湜) <[email-masked]>
Date: Mon, 20 Dec 2010 16:27:28 -0800

Hi,

As I wrote below, I really appreciate Julian and Adam for working on this
long-standing issue at long last.  I should have joined the effort much
sooner. Anyway, whether it's a single RFC or two RFCs (one on standard-track
and the other being informational), I'm with Adam that it'd better reflect
the 'reality' of what web servers do in the wild.

2010/12/16 Julian Reschke <[email-masked]>

> Do we happen to know *which* other sites?

I found a few other Google products emit RFC 2047 for Firefox and Chrome. As
for Opera,  I must have been hallucinating !  I can almost swear that I saw
a change go in to make gmail  emit RFC 2231 (it's before RFC 5987) for
Opera, but it turned out that it's not the case, yet. I'm sorry for the
incorrect statement in the previous email.  On the other hand,  I found that
Google Sites emits 'RFC 2231'  for Firefox, but not for Opera (let alone
other browsers).

BTW, Google search turned up a 7-yr old  mail thread at www-international (
http://blog.gmane.org/gmane.org.w3c.internationalization.general/month=20031101
)
where I did advocate for RFC 2231 :-) but others (out of 'practical'
needs/concerns) recommended using RFC 2047. It's old and I have little idea
how wide-spread the practice is now, but it's an indication that using RFC
2047 (used to) has (have) some following. That may have been partly due to
HTML4's and RFC 2388's references to RFC 2047 (2045) although they're about
uploading a file as a part of multipart-form data.

Anyway, this issue should have been escalated a long long ago and I very
much appreciate Julian's and Adam's efforts to nail this down.

As for what's actually being emitted by web servers, my guess is that raw
byte sequences in various encodings (including UTF-8) are most common
followed by simple %-encoding (again in various encodings), which is in turn
followed by RFC 2047 (depending on UA web servers talk to) with RFC 5987
emitted by a small number of servers.

I wish I had some cycles to collect stats for C-D headers (over
'attachments' Google crawled). I'll ask around if anybody has done that
(when I did a year or so ago, I couldn't find any). However,  I'm
afraid  just tallying C-D headers of  'attachments' that were already
crawled (e.g. Google corpora) wouldn't work.  I guess we need to crawl a
good/representative sample of web sites again "masqurading as Firefox (or
other browsers)" because most web sites (that emit RFC 2047) are likely to
do that only for Firefox (and in some cases, Chrome).

As an alternative to the above, I wonder if browsers can collect some stats
on the format of C-D header.

Jungshik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 21 Dec 2010 11:05:47 +0100

On 21.12.2010 01:27, Jungshik Shin (신정식, 申政湜) wrote:
> Hi,
>
> As I wrote below, I really appreciate Julian and Adam for working on
> this long-standing issue at long last.  I should have joined the effort
> much sooner. Anyway, whether it's a single RFC or two RFCs (one on
> standard-track and the other being informational), I'm with Adam that
> it'd better reflect the 'reality' of what web servers do in the wild.
> ...

Well, one goal of writing a spec (in particular at "proposed standard") 
is to define a good way to do things.

Of course, documenting what's out there today is *also* useful. However, 
as my many tests show, there is currently no interoperability at all (at 
least for I18N), so I'm not sure what you're proposing beyond the fact 
that things are messed up (essentially giving instructions to do UA 
sniffing).

> ...
> I found a few other Google products emit RFC 2047 for Firefox and
> Chrome. As for Opera,  I must have been hallucinating !  I can almost
> swear that I saw a change go in to make gmail  emit RFC 2231 (it's
> before RFC 5987) for Opera, but it turned out that it's not the case,
> yet. I'm sorry for the incorrect statement in the previous email.  On
> the other hand,  I found that Google Sites emits 'RFC 2231'  for
> Firefox, but not for Opera (let alone other browsers).
> ...

At this point, servers can emit RFC2231 encoding for all browsers except 
IE, Safari, and Chrome (which will change with Chrome 9, right). I think 
this should be considered the best way to do it; it special-cases only 
those browsers who haven't implemented the recommended way of doing 
things yet.

> http://blog.gmane.org/gmane.org.w3c.internationalization.general/month=20031101 )
> needs/concerns) recommended using RFC 2047. It's old and I have little
> idea how wide-spread the practice is now, but it's an indication that
> using RFC 2047 (used to) has (have) some following. That may have been
> partly due to HTML4's and RFC 2388's references to RFC 2047 (2045)
> although they're about uploading a file as a part of multipart-form data.

Well, the only UA supporting RFC 2047 up until two years ago was 
Firefox, and at that point Firefox implemented RFC 2231 as well. So it's 
strange that we apparently have legacy servers actually *using* this.

It shows that the clarification re RFC 2047, both in the base specs and 
in the C-D spec are really needed.

> As for what's actually being emitted by web servers, my guess is that
> raw byte sequences in various encodings (including UTF-8) are most
> common followed by simple %-encoding (again in various encodings), which
> is in turn followed by RFC 2047 (depending on UA web servers talk to)
> with RFC 5987 emitted by a small number of servers.

I don't think it's true, but I can't prove it. Back when I researched 
this many years ago, the recommendation was to server %xx to IE, and 
sent RFC 2231 to everybody else (where everybody else back then 
essentially meant Firefox and Opera).

> good/representative sample of web sites again "masqurading as Firefox
> (or other browsers)" because most web sites (that emit RFC 2047) are
> likely to do that only for Firefox (and in some cases, Chrome).
>
> As an alternative to the above, I wonder if browsers can collect some
> stats on the format of C-D header.

That would be great.

So. How do we proceed now? Mark?

Best regards, Julian