Subject: \-decoding filename parameters [was: TICKET 259: 'treat as invalid' not defined]

From: Mark Nottingham <[email-masked]>
Date: Thu, 3 Feb 2011 14:49:58 +1100

Having had some time away from this, and re-reading the thread, I notice that HTTPbis already disallows the production of many \-encoded characters:

   Producers SHOULD NOT escape characters that do not require escaping
   (i.e., other than DQUOTE and the backslash character).

(p1, 1.2.2)

So, really, we're already talking about error handling when we talk about things like \b, etc.

Furthermore, AFAICT neither 2616 nor bis really talk about the semantics and handling of "unusual" \-encoded characters. According to the current definition, one could plausibly decode "\n" to be a newline, since all that's really said about it is 

   The backslash character ("\") MAY be used as a single-character
   quoting mechanism only within quoted-string and comment constructs.

(although I doubt that happens IRL).

At the least, then, we should continue to discourage the use of \-escaping for things other than "\" and <">. 

If we were to write error-handling advice for it, it seems that we could give *weak* advice to replace with "_" or "-" (based upon <http://greenbytes.de/tech/tc2231/#attwithasciifnescapedchar>). We should probably also consider that question for BIS, but that can wait for now.

To me the currently relevant question is whether implementers will eventually support escaping "\" and <">. Right now a few do (see <http://greenbytes.de/tech/tc2231/#attwithasciifnescapedquote>), but many don't. However, since this is a "soft" failure / interop problem (i.e., it affects how a file is named when saved on disk, but doesn't prevent it from being saved or named), I don't see that as a reason to not specify it.

Thoughts?

On 13/12/2010, at 7:36 PM, Julian Reschke wrote:

> On 13.12.2010 00:36, Adam Barth wrote:
>> ...
>> The only browsers in his test that \-decode the filename parameter are
>> Opera and Konquerer.  The other browsers, representing some 99% of the
>> market, do not \-decode.
>> 
>> Based on this data lone, I'd be quite hesitant to implement \-decoding
>> in my hypothetical user agent.  Worse, the \ character is actually
>> quite commonly used in file paths because it is the path separator on
>> Windows.  It seems entirely likely that some number of servers send
>> absolute paths rather than file names,
>> 
>> Content-Disposition: attachment; filename="C:\foo\newsheet.html"
>> 
>> or even just relative paths, like "foo\bar.html".
>> 
>> Now, I haven't gone out and measured the prevalence of the \ character
>> in the headers on the web.  If some would like to run that experiment,
>> I'd certainly be open to considering that data.  However, in the
>> absence of such data, I think it's unlikely that browsers will change
>> their behavior.
>> ...
> 
> We are going in circles, and have been for months now.
> 
> We discussed this just two 10 days ago. See <http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0536.html>, where I said:
> 
> -- snip --
> "Fixing" means "changing things to work as specified".
> 
> So the question here is whether it would break things because there are
> servers sending unescaped backslashes. As far as I can tell, sending
> path separators in the filename indicates a bug in the sender, or an
> attempt to trick the user agent to do something it's not supposed to do.
> 
> So the "harm" of actually doing the unescaping would be that for a
> filename that needs to be postprocessed anyway, the problematic
> character would be filtered in a different way.
> 
> Starting with
> 
>   filename="a\bc"
> 
> the broken implementation sees "a" and "bc" separated by a path
> separator, and will prost-process this to "abc", "a_bc" or "bc" (where _  could be a different replacement character).
> 
> A correct implementation sees "abc".
> 
> I don't think there's a problem here.
> -- snip --
> 
> You did reply to that in <http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0538.html> with
> 
> "None of the user agents do \-decoding.  I don't see any value in them starting."
> 
> ...which I quite frankly didn't find helpful. It appears that we have differing opinions on whether it's useful to have consistent handling of specific syntactical constructs.
> 
> I believe having different parsers for quoted-string depending on the context they appear in is both a bad idea in general, and also not *needed* here.
> 
>> Finally, I don't think we should require user agents to implement
>> behavior we have reason to believe will not be implemented by a number
>> of major implementations.  Every time we do that, we take one more
> 
> On the other hand, we also shouldn't require user agents to break the spec that has been around for ~10 years when there's no compelling reason to do so.
> 
>> step down the path to irrelevance.  I'd like to see more engagement
>> between the browser vendors and the IETF, not less.
> 
> Yes, so do I. That's why I'd like to hear from the other browser makers (not necessarily vendors), and also observe how far we'll get with fixing the C-D bugs in Firefox once they are past the FF4 release.
> 
>> ...
> 
> Best regards, Julian
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 3 Feb 2011 15:30:05 +1100

On 12/12/2010, at 6:42 AM, Adam Barth wrote:
>> 
>>   If the field value does not conform to the grammar (such as when not
>>   exactly one disposition type is specified), ignore the whole header
>>   field.
> 
> This doesn't cover cases like the following:
> 
> Content-Disposition: attachment; inline; filename=foo.exe
> 
> We want to treat those as an attachment.  Another grammer we could use
> might be the following:
> 
>     field-value = item *( ";" item )
>     item          = disp-type / param
>     disp-type   = <OCTET, except ";" and "=">
>     param       = param-name "=" param-value
>     param-name = <OCTET, except "=">
>     param-value = <OCTET, except ";">
> 
> We could then say that first disp-type and the first param are the
> ones that matter.  (I'm not sure this grammar handles <"> correctly,
> but I'm sure we can sort that out.)

As discussed earlier, that seems like pretty common behaviour:
  http://greenbytes.de/tech/tc2231/#attandinline

... although I'm not sure about the BNF change. 

What about loosening the definition of disposition-parm to make the '=' optional, so that anything after the first will be treated as a parameter (and therefore ignored)? Just thinking out loud.

>> D.3.  Checking Cardinality Constraints
>> 
>>   If the parameter sequence contains multiple instances of the same
>>   parameter name, ignore the whole header field.
> 
> We'd prefer to use the first one rather than ignore the header field.

That seems like pretty common behaviour:
  http://greenbytes.de/tech/tc2231/#attwith2filenames

>> D.5.  Extracting the Disposition Type
>> 
>>   The parsing step (Appendix D.2) has returned the disposition type (to
>>   be matched case-insensitively), which can be "attachment", "inline",
>>   or an extension type.  If the type is unknown, treat it like
>>   "attachment" (see Section 3.2).
> 
> What if there's no disposition type?
> 
> Content-Disposition: filename=foo.exe
> Content-Disposition: foo=bar
> 
> If I remember correctly, we're supposed to treat the former as inline
> and the later as attachment.

http://greenbytes.de/tech/tc2231/#attmissingdisposition

doesn't seem like there's much consensus among implementers here; unless one forms, I don't think we can say much.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Thu, 03 Feb 2011 19:52:43 +0100

> (although I doubt that happens IRL).

That has bugged me for some time, and I wasn't sure whether I was just 
too pedantic and the answer is obvious. We should treat this as 
something we need to fix.

That being said: the spec specifies quoted-pair in a way that other 
characters are allowed. If we don't change that, we need to say what it 
means to use this.

The grammar in HTTP was inspired by RFC 822 and successors, and RFC 5322 
says:

"Where any quoted-pair appears, it is to be interpreted as the character 
alone. That is to say, the "\" character that appears as part of a 
quoted-pair is semantically "invisible"." -- 
<http://greenbytes.de/tech/webdav/rfc5322.html#rfc.section.3.2.1>

This happens to match my intuitive understanding what escape characters 
are for, so I'd propose that we adopt that.

> At the least, then, we should continue to discourage the use of \-escaping for things other than "\" and<">.

Recommending not to escape things that do not need escaping is fine. I'm 
not sure whether this needs to be a SHOULD.

> If we were to write error-handling advice for it, it seems that we could give *weak* advice to replace with "_" or "-" (based upon<http://greenbytes.de/tech/tc2231/#attwithasciifnescapedchar>). We should probably also consider that question for BIS, but that can wait for now.

Disagreed.

First of all, this is only error handling if we actually forbid those 
sequences. We don't do that right now.

It would be bad if we ended up with different rules for processing 
quoted-string depending on where they occur, just to able that some 
broken implementations can claim that they are not.

> To me the currently relevant question is whether implementers will eventually support escaping "\" and<">. Right now a few do (see<http://greenbytes.de/tech/tc2231/#attwithasciifnescapedquote>), but many don't. However, since this is a "soft" failure / interop problem (i.e., it affects how a file is named when saved on disk, but doesn't prevent it from being saved or named), I don't see that as a reason to not specify it.

Sending a filename with a literal backslash character in it is likely an 
attempt by the sender to trick the recipient to overwrite files in 
another directory. The spec already recommends:

"When the value contains path separator characters, all but the last 
segment SHOULD be ignored. This prevents unintentional overwriting of 
well-known file system location (such as "/etc/passwd")." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-content-disp-04.html#rfc.section.3.3>

So it really doesn't matter a lot at what stage the \ disappears.

Escaped DQUOTEs 
(<http://greenbytes.de/tech/tc2231/#attwithasciifnescapedquote>) cover a 
use case that has no other solution (except for 5987-encoding 
everything). So I believe the right thing to do is to keep this 
specified, and potentially warn servers about the UAs that can't handle 
it (similar to the way we already warn about the "%" problem).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 03 Feb 2011 20:14:58 +0100

On 03.02.2011 05:30, Mark Nottingham wrote:
>>      disp-type   =<OCTET, except ";" and "=">
>>      param-name =<OCTET, except "=">
>>      param-value =<OCTET, except ";">
>>
>> ones that matter.  (I'm not sure this grammar handles<">  correctly,

Actually, *that* test doesn't provide any insight as most UAs *appear* 
to ignore the header field.

The next one, <http://greenbytes.de/tech/tc2231/#attandinline2>, 
reverses the disposition types, and is a bit more interesting.

Taken together, they show that most UAs take the first type, while IE 
takes the second.

To me this indicates that we don't have any interop here, and that it's 
unlikely that treating this as invalid would actually break existing 
content.

> What about loosening the definition of disposition-parm to make the '=' optional, so that anything after the first will be treated as a parameter (and therefore ignored)? Just thinking out loud.

There's some precedence here, RFC 5988 does exactly that (allowing 
value-less extension parameters, same in RFC 2068).

I'm not thrilled about that, and I note that this make this spec 
incompatible with Content-Disposition in MIME, which I was trying to avoid.

Are we discussing the Adam's "error-friendly" BNF, or the normative one?

>    http://greenbytes.de/tech/tc2231/#attwith2filenames

That may be true, but it doesn't give us any information about how 
frequently it's used. Which brings us back to the question whether we 
should talk about error handling when there's zero evidence that interop 
is relevant. Has anybody *seen* a header field instance like that in 
practice?

> doesn't seem like there's much consensus among implementers here; unless one forms, I don't think we can say much.

Right.

Going back to the archives -- in 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2010OctDec/0679.html>, 
Adam says:

> I'll make this change next time I edit the wiki.  Before Julian and
> others bite my head off, I'm aware that this will likely mean you
> won't find this text acceptable to include in the current document.
> If that's the case, I can expand this text to a full-length
> specification, and we can publish it either through this working group
> or as an individual submission.

That was six weeks ago -- Adam, do you still plan to make these changes?

We've been debating this since September, and I don't think we have made 
any progress towards even a rough consensus that we *want* to specify 
error handling. As this specification was only supposed to clarify 
something we already have (C-D in MIME, plus the text in 2616), I 
strongly suggest that we either give up on this and proceed to LC, or 
set a deadline.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Thu, 3 Feb 2011 12:37:13 -0800

> doesn't seem like there's much consensus among implementers here; unless one forms, I don't think we can say much.

As I've written before, this seems backwards.  How are implementations
to come to consensus if we don't write down a specification for how
they should behave?

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Feb 2011 16:01:57 +1100

On 04/02/2011, at 5:52 AM, Julian Reschke wrote:
> 
> That has bugged me for some time, and I wasn't sure whether I was just too pedantic and the answer is obvious. We should treat this as something we need to fix.
> 
> That being said: the spec specifies quoted-pair in a way that other characters are allowed. If we don't change that, we need to say what it means to use this.

Agreed. 

I think we need to at least consider changing it, because escaping them doesn't serve any useful purpose, and it's not well-specified or well-implemented. 

I've created <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/270> to track it.

> The grammar in HTTP was inspired by RFC 822 and successors, and RFC 5322 says:
> 
> "Where any quoted-pair appears, it is to be interpreted as the character alone. That is to say, the "\" character that appears as part of a quoted-pair is semantically "invisible"." -- <http://greenbytes.de/tech/webdav/rfc5322.html#rfc.section.3.2.1>
> 
> This happens to match my intuitive understanding what escape characters are for, so I'd propose that we adopt that.

How widely supported is it in MIME implementations? 

We've already strayed significantly from MIME (e.g., line folding), so I don't think we should be constrained by it (although of course, we shouldn't needlessly stray).

> Recommending not to escape things that do not need escaping is fine. I'm not sure whether this needs to be a SHOULD.

See above; it's not widely supported, and certainly not interoperably. Why wouldn't we steer senders away from it?

> First of all, this is only error handling if we actually forbid those sequences. We don't do that right now.
> 
> It would be bad if we ended up with different rules for processing quoted-string depending on where they occur, just to able that some broken implementations can claim that they are not.

It's pretty strong to call an implementation 'broken' because it doesn't implement something that is poorly specified and not useful. 

I agree, though, that it would be good to have a single answer for all of bis, rather than special-casing it.

> Sending a filename with a literal backslash character in it is likely an attempt by the sender to trick the recipient to overwrite files in another directory. The spec already recommends:
> 
> "When the value contains path separator characters, all but the last segment SHOULD be ignored. This prevents unintentional overwriting of well-known file system location (such as "/etc/passwd")." -- <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-content-disp-04.html#rfc.section.3.3>
> 

Your argument assumes that \ is recognised as a path separator; on some platforms, it's not. 

> Escaped DQUOTEs (<http://greenbytes.de/tech/tc2231/#attwithasciifnescapedquote>) cover a use case that has no other solution (except for 5987-encoding everything). So I believe the right thing to do is to keep this specified, and potentially warn servers about the UAs that can't handle it (similar to the way we already warn about the "%" problem).

Agreed.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Feb 2011 09:17:00 +0100

On 04.02.2011 06:01, Mark Nottingham wrote:
> I've created<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/270>  to track it.
>
>> "Where any quoted-pair appears, it is to be interpreted as the character alone. That is to say, the "\" character that appears as part of a quoted-pair is semantically "invisible"." --<http://greenbytes.de/tech/webdav/rfc5322.html#rfc.section.3.2.1>
>>

As I said before, I believe that the MIME rule describes common sense in 
escaping, and the fact that it wasn't stated in 2068/2616 was just some 
oversight.

What we should do is go through the spec, check for occurrences of 
quoted-string, and see what implementations do (and with that I do not 
only mean browsers). Yes, this will be time-consuming.

> See above; it's not widely supported, and certainly not interoperably. Why wouldn't we steer senders away from it?

I didn't say we shouldn't. I was questioning whether a RFC2119-SHOULD is 
the right way to do it.

>> "When the value contains path separator characters, all but the last segment SHOULD be ignored. This prevents unintentional overwriting of well-known file system location (such as "/etc/passwd")." --<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-content-disp-04.html#rfc.section.3.3>
>>

I'm not talking about OS behavior but UA behavior. For the purpose of 
Content-Disposition/filename, I would *hope* that UAs treat \ and / the 
same. My tests are <http://greenbytes.de/tech/tc2231/#attabspath> and 
<http://greenbytes.de/tech/tc2231/#attabspathwin> and my results reflect 
the Windows versions of UAs; it would be nice if somebody could try 
whether Safari/Chrome/Opera behave differently on MacOS...

Best regards, Julian

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Fri, 4 Feb 2011 10:40:00 +0000

On 4 Feb 2011, at 08:17, Julian Reschke wrote:
> I'm not talking about OS behavior but UA behavior. For the purpose of Content-Disposition/filename, I would *hope* that UAs treat \ and / the same. My tests are <http://greenbytes.de/tech/tc2231/#attabspath> and <http://greenbytes.de/tech/tc2231/#attabspathwin> and my results reflect the Windows versions of UAs; it would be nice if somebody could try whether Safari/Chrome/Opera behave differently on MacOS...

Under Mac OS X 10.6.5

http://greenbytes.de/tech/tc2231/#attabspath

Firefox 3.6.13 => _foo.html
Chrome 9.0.597.84 => foo.html
Safari 5.0.3 (6533.19.4) => -foo.html

http://greenbytes.de/tech/tc2231/#attabspathwin

Firefox => \foo.html
Chrome => \\foo.html
Safari => \\foo.html

HTH
Ben

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Feb 2011 12:13:41 +0100

On 04.02.2011 11:40, Ben Niven-Jenkins wrote:
>> I'm not talking about OS behavior but UA behavior. For the purpose of Content-Disposition/filename, I would *hope* that UAs treat \ and / the same. My tests are<http://greenbytes.de/tech/tc2231/#attabspath>  and<http://greenbytes.de/tech/tc2231/#attabspathwin>  and my results reflect the Windows versions of UAs; it would be nice if somebody could try whether Safari/Chrome/Opera behave differently on MacOS...
>
> Under Mac OS X 10.6.5
>

That's consistent with the Windows versions.

> http://greenbytes.de/tech/tc2231/#attabspathwin

But this is not; apparently those UAs consider "\" a legitimate 
character in a filename.

I believe that's a very bad idea.

Best regards, Julian

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Fri, 4 Feb 2011 11:46:42 +0000

On 4 Feb 2011, at 11:13, Julian Reschke wrote:
> But this is not; apparently those UAs consider "\" a legitimate character in a filename.
> 

The handling of file names with a "\" in on Mac OS X is somewhat erratic.

You can save a file from a "GUI" application with \ in the fileanme, and at least some applications will open them (I only tried with TextEdit and MS Word) but while you can see them with an ls in a shell you get a "no such file/directory" error if you try access one. If you create a file with a \ in the filename from the shell the \ gets stripped out.

Firefox has some inconsistent behaviour - it won't open a file that starts with a \ and just presents a blank page but will open a file with a filename of the form foo\bar

I didn't try any other browsers.

Ben

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 5 Feb 2011 10:39:37 +1100

> to come to consensus if we don't write down a specification for how

If people with implementations were actively participating in this discussion, we'd have a much better chance of doing so. 

I appreciate the effort you've put into this, Adam, but I don't see them coming to the table. I continue to be uncomfortable taking the specification too far down a road where we're speculatively adding text. 

Remember that we're chartered to *not* break existing implementations, except where doing so addresses security issues or materially improves interop. Error handling of cases like this <http://greenbytes.de/tech/tc2231/#attmissingdisposition3> are hard to see as a material improvement in interoperability, especially when there's only one person on the mailing list actively pushing for them.

What I'd suggest instead is for us to concentrate on making the current draft workable without defining error handling (e.g., making sure the BNF isn't too restrictive, checking the requirements), and simultaneously you can submit your error handling profile as an Internet-Draft. 

That decouples error handling from the constraints of our charter as well as the high bar of the standards track; even if all implementations aren't on board with it, it can still be published as an Informational RFC. And, if you do come up with something that is workable for everyone, I think it could be folded into C-D as it progresses down the standards track.

Make sense?

---

From: Adam Barth <[email-masked]>
Date: Fri, 4 Feb 2011 16:25:33 -0800

> That decouples error handling from the constraints of our charter as well as the high bar of the standards track; even if all implementations aren't on board with it, it can still be published as a…

That's fine with me.  I can see how this sort of work would fall
outside of the charter of this working group.  Thanks for the helpful
discussions.

Adam