Subject: draft-montenegro-httpbis-uri-encoding

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Mar 2014 15:36:47 +1100

One of the things we didn't get time to talk about in London was Gabriel's spec:
  http://tools.ietf.org/html/draft-montenegro-httpbis-uri-encoding-00

In a nutshell, this offers a way for a client to declare what character encoding was used prior to percent-encoding. 

I've heard hallway feedback about it that wonders if we just want to allow one value ("UTF-8"). Beyond that, folks seem generally neutral-to-positive about it, AFAICT.

What do people think about adopting this as a WG item, keeping in mind that we can change it in process if there's some particular aspect you don't like?

Cheers,

P.S. Just to be clear, this would be completely separate from the HTTP/2 work item.

--
Mark Nottingham   http://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 21 Mar 2014 15:15:30 +1000

On 21 March 2014 14:36, Mark Nottingham <[email-masked]> wrote:

> In a nutshell, this offers a way for a client to declare what character
> encoding was used prior to percent-encoding.
>
> I've heard hallway feedback about it that wonders if we just want to allow
> one value ("UTF-8"). Beyond that, folks seem generally neutral-to-positive
> about it, AFAICT.
>
> What do people think about adopting this as a WG item, keeping in mind
> that we can change it in process if there's some particular aspect you
> don't like?
>
> Cheers,
>

I'm interested, for what that's worth. It might even help inform some of my
work on the 'file' URI scheme.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 21 Mar 2014 18:43:53 +1300

> What do people think about adopting this as a WG item, keeping in mind that we can change it in process if there's some particular aspect you don't like?

Sounds good.

Regarding the draft contents:

 Is there any recommendation on what non-UA participants in the
transaction should do when receiving non-UTF-8 values for these headers?
(ie map the charset if they can? ignore it?)

 What should servers (and intermediaries) receiving non-UTF-8 values in
these headers do when they rely on receiving UTF-8 URLs?

 Also, Section 2 references RFC 2616 for charset definition. Which
declares it to be token with values defined in RFC1700. Which in turn
lists US-ASCII and various ISO-* as values. Nowhere in that definition
outside this draft is UTF-8 or alternative values mentioned.
 It would seem the better reference would be RFC2978 which defines ABNF
and values for constructino of tokens from the IANA mime-charset registry.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Mar 2014 17:18:16 +1100

> these headers do when they rely on receiving UTF-8 URLs?

I don't think it makes any difference to them; again, the URL on the wire is still ASCII, and that's still the cache key.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 08:19:14 +0100

> P.S. Just to be clear, this would be completely separate from the HTTP/2 work item.

My concerns are the same as when this was presented first: how does this 
help?

I hear that it makes security checks more reliable, but then, you can't 
rely on the header field being accurate (we do have experience with 
out-of-band encoding information, no?).

So before we do this, I'd like to see a *concrete* example use case.

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 08:43:02 +0100

Le Ven 21 mars 2014 05:36, Mark Nottingham a écrit :

Hi,

> one value ("UTF-8"). Beyond that, folks seem generally neutral-to-positive

The draft adds constrains to network nodes that try do do the right thing,
and gives bad nodes a free ticket. So the incentive is to continue to
behave non-deterministically.

I'd much rather have: anything that pretends to talk http/2 uses UTF-8
(perhaps not even % encoded) by default, unless it specifies another
encoding in one of the two optional headers. People who try to do anything
other than utf-8 without specifying it get broken connexions when
something fails in the path, if they don't want to make the effort to be
deterministic that's their problem.

The spec can add a reserved keyword for unknown encoding (probably with
the provision network nodes MAY refuse connexions with this keyword or any
encoding value not specified in the spec).

That avoids dragging two new headers on every connexion and does not make
bad encoding the problem of people who rightly use utf-8 but on the actual
bad encoding perpetrators.

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 08:59:45 +0100

Le Ven 21 mars 2014 08:19, Julian Reschke a écrit :
>> I've heard hallway feedback about it that wonders if we just want to
>> allow one value ("UTF-8"). Beyond that, folks seem generally
>> neutral-to-positive about it, AFAICT.
>>
> rely on the header field being accurate

There is a difference between working in heuristics mode all the time with
crossed fingers and rabbit legs and working in deterministic mode with
simple error handling (and error handling can be abort when what the other
node declares and what you receive are different – much more secure than
generalized guesswork)

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 09:13:28 +0100

On 2014-03-21 08:59, Nicolas Mailhot wrote:
> ...

So how *exactly* does the header field help you in deciding whether to 
be in heuristics mode or not?

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 21 Mar 2014 21:15:09 +1300

On 21/03/2014 7:18 p.m., Mark Nottingham wrote:
> 
> On 21 Mar 2014, at 4:43 pm, Amos Jeffries wrote:
> 

I have this nasty idea about client implementations starting to send
these headers with values indicating things like Unicode or Windows
charsets. That could lead to strange happenings in places like security
ACLs, background auth protocols, log processing software or TTY output
devices if the recipient intermediary was translating from url-encoding
to non-ASCII outputs.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 21 Mar 2014 21:17:48 +1300

> bad encoding perpetrators.

+1.

Amos

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 11:55:33 +0100

> be in heuristics mode or not?

If I know the encoding is supposed to be UTF-8 I can fail anything that
does not pass the usual UTF-8 sanity rules instead of iterating through
encodings hoping I find the right one before hitting a security bug

If I know my app is open to European users only I can block user agents
that try to use a CJK encoding before they manage to hit a CJK processing
bug

I don't have to log random ascii-ified percentage garbage in the hope that
the day it will need to be analysed and interpreted I'll have enough human
beings to review it line by line and assign the encoding necessary to
interpret it line by line.

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 12:01:51 +0100

> encodings hoping I find the right one before hitting a security bug

So what do you do once you actually get requests that claim to use UTF-8 
but don't? Investigate what's wrong?

> that try to use a CJK encoding before they manage to hit a CJK processing

I don't think any CJK-related is on the table.

> interpret it line by line.

That seems to be the same use case as #1.

Why don't you just try to UTF-8 decode, and if that works, assume that 
it indeed is UTF-8?

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 12:29:51 +0100

> Why don't you just try to UTF-8 decode, and if that works, assume that

Really, can't you read the abundant documentation that was written on the
massive FAIL duck typing is for encoding (for example, python-side)? Code
passing unit tests then failing right and left as soon as some new
encoding combo or text triggering encoding differences injected in the
system? Piles of piles of partial workarounds till there was complete loss
of understanding how they were all supposed to work in the first place?

That's the last thing you want to reinvent on security equipments (and
you'll reinvent it because the amount of non-ASCII urls is small now but
will only grow with time).

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 12:34:56 +0100

> of understanding how they were all supposed to work in the first place?

I understand the problems caused by not knowing what encoding something 
is in. What I don't understand is how an out-of-band signal helps if you 
really can't rely on it being accurate.

Practically, how is a UA supposed to *know* the encoding that was used 
for the URI *unless' it constructed it itself? (Which is not what 
browsers do; they only construct the query part).

Best regards, Julian

---

From: Michael Sweet <[email-masked]>
Date: Fri, 21 Mar 2014 08:30:11 -0400

+1 for working on this, although in its present form it'll still need some work...

> P.S. Just to be clear, this would be completely separate from the HTTP/2 work item.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 21 Mar 2014 13:54:04 +0100

>will only grow with time).

Julian asked for a concrete example use case. So far you have not given
one. It might help to assume the rest of us understands the subject at
hand at least as well as you do.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 14:05:21 +0100

> browsers do; they only construct the query part).

If the browser constructed the URL it knows damn well what is the encoding
of its address bar and how to convert to UTF-8

If the browser got the uRL in a web page or feed or whatever all those
documents are supposed to declare an encoding  so they can be interpreted
at all (and there is a default encoding in the spec if they don't) so it
can use that encoding and convert to utf-8 before sending

If the encoding declared in the document or in the http headers the web
site set is wrong things will fail but no more than if the web page author
made a typo in its link. And I want them to fail not propagate errors to
innocent bystanders.

The whole concept of attempting to silently fix problems with heuristics
till web site authors assume they can write garbage and it will be
autocorrected at the cost of security and reliability, can not work on a
large scale. There are too many people willing to exploit the holes the
autocorrection heuristics open right and left. People doing mistakes is
not an excuse to writing fuzzy specs to avoid laying responsibility and
then expect things to work out anyway. That's PHB thinking.

Anyone in 2014, who defines an URL container, and think he can avoid
specifying the encoding of this container, is in for a world of grief and
that won't change whether the http2 spec explicitly fixes this hole or
not. And I'd rather have http2 implementors avoid this particular pitfall
because the spec is clear on the subject.

-- 
Nicolas Mailhot

---

From: Mike Belshe <[email-masked]>
Date: Fri, 21 Mar 2014 06:10:50 -0700

> hand at least as well as you do.

I too am wondering this.

In general, this just sounds like "more stuff".

I'm not aware of anyone having real-world problems which this solves. It
would be very helpful if those real cases were enumerated.

Mike

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 14:18:58 +0100

> of its address bar and how to convert to UTF-8

OK. But that is true only if the URI was constructed by parsing the 
address bar. It's not the case when following links in documents (when 
try are already percent-escaped).

> can use that encoding and convert to utf-8 before sending

That's only helps when the link wasn't percent-escaped in the first place.

> then expect things to work out anyway. That's PHB thinking.

Oh please. I dislike heuristics as much as you do. But just because we 
dislike it we can't magically get rid of it.

What we discuss here is whether an out-of-band signal actually helps.

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 14:24:02 +0100

> hand at least as well as you do.

As I wrote last time he asked the same question, on some of our networks
accesses are controlled by regex-like checks on URL and not knowing the
encoding of processes URLs means this processing (and the processing of
security logs) is unreliable.

We already had several security incidents where carefully crafted urls
triggered security equipment bugs (so far, not using encoding tricks just
plain ascii but the writing is on the wall).

The first in-the-wild uses of punicode already triggered bugs in code that
assumed everything is ascii (and that's ok we can fix this case because it
is clearly defined – not fix every random encoding permutation people can
invent).

Not knowing encoding propagates encoding heuristics in all layers of the
software stack – from security appliances, to log handlers, to the apps
that process their output to inform users on their web usage, to the
stupid spreadsheet macros people use to simplify reporting/billing/quick
analysis tasks. The only sane way to limit the problem scope is to convert
everything to a single universal well managed encoding at entry point. No
different than what the python people did. No different from database
schema handling (if you don't use unicode and utc in your databases by
default today, you deserve the problems you get). Text is an early
decoding problem space, not lazy last-mile just-as-needed decoding problem
space.

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 14:39:14 +0100

> security logs) is unreliable.

I understand the pain caused by having to apply heuristics.

What I don't understand is how an out-of-band signal that can be 
incorrect helps. If this is about security-related checks, you can't 
trust it anyway, no?

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 14:47:30 +0100

> That's only helps when the link wasn't percent-escaped in the first place.

I'll give you a big secret: nobody writes in percent-escaped manually if
he can avoid it, just like nobody uses html entities.

The bulk of percent-escaped urls has been produced by automatons
converting human-written plain text that used the document main encoding,
so yes I do expect both encodings to match if the automaton was coded
properly.

Regards,

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 14:54:24 +0100

> so yes I do expect both encodings to match if the automaton was coded

That assumes that the "automaton" that did the URI-escaping actually 
knew the document encoding, and that the document the URI appears in 
never gets re-encoded.

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 14:58:35 +0100

> incorrect helps. If this is about security-related checks, you can't

In a security context if something is suspicious you block/fail/error out
and don't ask questions.

With undefined encoding everything is suspicious so you can't act because
it may be normal.

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 15:05:57 +0100

> knew the document encoding, and that the document the URI appears in

Yes, sure, there will always be border cases and people will continue to
invent convoluted ways to shoot themselves in the feet. But their number
won't go down unless the spec clearly states URLs need to be decoded
(stated this way this seems really stupid, but are we arguing about
anything else?)

If there were no encoding errors, there would be no need to eradicate
them, and all the examples you find are one more reason to clear the
swamp.

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 21 Mar 2014 14:26:46 +0000

I am of the opinion that HTTP/2's success will be determined by
what it eliminates from the HTTP-mess, rather than what it adds to it.

This topic/discussion is a very good example why I think so.

We can focus om making HTTP/2 simpler to implement correctly, by
choosing a simple and unambiguous rule, probably "URI's are UTF-8"
or we can add another tortured heuristic resolution flow-chart, which
attempts to guess what people though they understood when they read
the unclear description of all the point-less options in the standard.

What happened to the KISS principle ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 15:25:35 +0100

> With undefined encoding everything is suspicious so you can't act because

Again: what makes the out-of-band signal trustworthy?

If it is used for security-related checks, and it gets trusted, then 
attackers *will* forge it.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 15:36:28 +0100

> What happened to the KISS principle ?

It died when we agreed that we want to be able to gateway between 1.* 
and 2.*, and not ever break existing semantics.

Essentially, HTTP/2 is only a new wire format, and it inherits all 
legacy not related to the wire format from 1.1.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 21 Mar 2014 14:41:31 +0000

>legacy not related to the wire format from 1.1.

Then I'd like to offer two proposals:

1. Rename the current drafts "HTTP/1.2" to indicate that all the
   crap will be the same.

2. Start working on a real HTTP/2.0 protocol, which tries to solve
   the actual problems with HTTP/1.x

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 21 Mar 2014 15:43:32 +0100

> knew the document encoding, and that the document the URI appears in  

And this is where it gets really depressing, since the encoding the server  
expects may be different from the documents encoding. For form data you  
can at least have a hidden variable with a known unicode sequence, but for  
URLs, which often are processed in an environment outside of the control  
of the target server, you have to use ASCII until the world realize that  
UTF-8 always everywhere is the solution. We can stop percent encoding too  
at that point.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 15:43:15 +0100

> attackers *will* forge it.

The signal is not used for security checks. The URLs are used for security
check. Discrepancies between declared encoding and url content are one
cause for suspicion. Lack of clear encoding rules does not make the whole
thing more secure, it removes one parameter to check against, and prevents
writing of other security checks due to unknown encoding fog.

And, in theory, malware should repect 100% of all specs to avoid detection
when triggering conformance checks, but in the actual world it does not
because its aim is to work well enough for one campaign not work reliably
on all possible web sites like a full browser. And even if malware started
respecting 100% of all specs you would still win because you'd have
removed the corner cases it could try to exploit.

-- 
Nicolas Mailhot

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 21 Mar 2014 15:48:29 +0100

>    the actual problems with HTTP/1.x

Off-topic as well, but I'm sure that 2.0 will drive the adoption of
an easier version upgrade path for the future and that we'll be able
to propose 3.0 (or whatever we call it) later to break compatibility
again and clean the mess because people will be more careful about
the products they deploy after having been burnt by 2.0. So let 2.0
be marketing + new on-wire format (+ catch all post-release issues)
and fix them once everything is clearer in field.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 15:51:52 +0100

> 1. Rename the current drafts "HTTP/1.2" to indicate that all the

We can't do that as sticking with 1.* implies a compatible wire format.

>     the actual problems with HTTP/1.x

This definitively comes too late for 2.0, but I agree that future work 
should try to solve problems inherited from 1.*.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 15:54:19 +0100

> removed the corner cases it could try to exploit.

Nicolas,

I'll ask again: please present a *concrete* example where the 
out-of-band metadata helps. This would include a description of where 
the request comes from, what gets on the wire, what kind of checks your 
code does, and what it would do differently when it gets the encoding 
metadata.

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 21 Mar 2014 16:11:00 +0100

> code does, and what it would do differently when it gets the encoding

I've already given everything I can without exposing our internal
architecture which I won't do. There is nothing more complex that URL
logging, URL regex matching, processing of results in apps (embedded,
server or desktop side) and human checking that everything work well by
reading logs or reportings or whatever.

And I've already stated I don't want out of band metadata to declare if
URLs are in UTF-8, I want out of band metadata to declare when they are
not, and the processing in this case will be to kill connexions and avoid
encoding guesswork down the stack.

There is no hidden mystery use case. There is only the basic need to be
able to decode URLs.

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 21 Mar 2014 15:15:53 +0000

In message <[email-masked]>, Julian Reschke writes:

>This definitively comes too late for 2.0, [...]

I don't think it does.

In fact I think we can get a better protocol on the net faster by
not having to spend so much time on all the crinkly bits around the
fjords.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 21 Mar 2014 16:17:54 +0100

>There is no hidden mystery use case. There is only the basic need to be

Could you make a simple web page with a link where the browser, when the
user clicks the link, would tell the server the request address is not a
UTF-8 address?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 16:24:41 +0100

> reading logs or reportings or whatever.

Well, please allow me not to believe in use cases until a concrete one 
is brought up and explained.

> encoding guesswork down the stack.

That's not what Gabriel's draft proposes.

> There is no hidden mystery use case. There is only the basic need to be

An HTTP URI is a sequence of octets in the range of ASCII code points. 
It can contain percent-escapes, in which case these sequences can be 
decoded to raw octets. That's where the story ends from HTTP's point of 
view.

And yes, *most* of the time these octet sequences represent a character 
sequence that has been encoded using a character encoding scheme (such 
as ISO-8859-1, or UTF-8). I fully agree that it would be nice if you 
could rely on that always being the case, and always being UTF-8, but 
it's simply not true in practice.

I don't believe that adding out-of-band data is going to help. That 
being said, I'm interested in finding out *how* this is going to work in 
practice and how it's going to help, but so far I haven't seen any 
concrete example.

This header field adds a lot of noise to essentially every single HTTP 
request, thus I continue to be very very skeptical about this proposal.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 21 Mar 2014 15:36:22 +0000

>is brought up and explained.

Gettys rules #1 and #3:

1. Do not add new functionality unless an implementor cannot complete
   a real application without it.

3. The only thing worse than generalizing from one example is
   generalizing from no examples at all.

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Mar 2014 10:36:36 -0500

I don't think you need two headers; "URI-Path-Encoding" seems useless:

1. It's improbable that the origin server uses separate encoding
schemes for path and query. If the encoding scheme for the query part
is known, it can be assumed for the path part too.

2. If no html form is involved, and the browser simply sends a URI it
got from the server in a document, the browser has no idea of its
encoding scheme. It cannot be inferred from the document encoding - it
is not unlikely that a server app uses different encoding for document
generation and URI generation/parsing.

URI-Query-Encoding might be useful:

1. If a browser submits a GET form, it may declare the character
encoding used for the query part, in case that's useful to
intermediaries.

2. If a browser submits a POST form with
enctype=application/x-www-form-urlencoded, it may declare the
character encoding used for the request entity body. But we'll need a
better header name than "URI-Query-Encoding". Note that we cannot add
a "charset=" parameter to the Content-Type header for this purpose -
servers are not expecting it; they do not need it; they may be broken
because of it.

3. In any case, the server will NOT read nor honor this header. The
server is the sole authority of how to parse the request.

Zhong Yu

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 21 Mar 2014 16:42:19 +0100

>is known, it can be assumed for the path part too.

This is actually a common situation. One reason is that the server
software handles the path, and some independent script handles the
query, and you might well have a server system that uses UTF-8 in
paths, but a legacy script expects ISO-8859-1 query strings. If it
is necessary and possible to communicate the encoding, if any, then
these two components need separate labels for maximum compatibility.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Mar 2014 10:46:19 -0500

> these two components need separate labels for maximum compatibility.

Hmm, OK.

But the browser has no idea how the URI path was constructed, it would
be presumptuous for the browser to brave a guess and mislead the
intermediaries.

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 16:53:45 +0100

> servers are not expecting it; they do not need it; they may be broken

See 
<http://www.w3.org/TR/html5/forms.html#application/x-www-form-urlencoded-encoding-algorithm> 
and look for "_charset_".

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Mar 2014 11:00:59 -0500

> and look for "_charset_".

Yes. The issue addressed by Gabriel's draft seems to be in the scope
of HTML, not HTTP.

And even if some kind of mechanism of declaring query encoding becomes
official, any intermediary relying on it will have a bad time.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 21 Mar 2014 17:04:19 +0100

>be presumptuous for the browser to brave a guess and mislead the

The same is often also true for the query part if it doesn't come from a
form submission the browser controlled. Hence my question to Nicolas,
<http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1145.html>.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 22 Mar 2014 10:30:28 +1100

> official, any intermediary relying on it will have a bad time.

Reading this thread, I was starting to think the same thing.

In particular, this seems like something that needs to be coupled to *where* the link originates; e.g., a browsers’ behaviour for a link from an address bar is likely to be different than that from an ‘a’ tag, and even again different from a JavaScript-generated link.

Gabriel, have you brought it up over at the W3C or in the WHATWG?

Cheers,

---

From: Gabriel Montenegro <[email-masked]>
Date: Sat, 22 Mar 2014 05:40:50 +0000

> From: Mark Nottingham [mailto:[email-masked]] On 22 Mar 2014, at 3:00 
> am, Zhong Yu <[email-masked]> wrote:
> >
> > And even if some kind of mechanism of declaring query encoding 
> > becomes official, any intermediary relying on it will have a bad time.

Any intermediary that relies on it is broken. This is a header that is not at all guaranteed to be present. 

> In particular, this seems like something that needs to be coupled to 
> *where* the link originates; e.g., a browsers' behaviour for a link 
> from an address bar is likely to be different than that from an 'a' 
> tag, and even again different from a JavaScript-generated link.

I'm a bit confused by this. Determining whether to set the headers, and what to set them to (e.g., UTF-8) is certainly browser behavior. But once that decision has been taken, communicating that hint to the server is what the headers are for.

> Gabriel, have you brought it up over at the W3C or in the WHATWG?

No, I haven't.

BTW, here's some responses to the very long thread.

To respond to Nicolas Mailhot <[email-masked]> about fixing it for HTTP/2, on  http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1110.html: This is not an HTTP/2 issue any more. That issue was closed in the issue tracker, and instead the decision at the Zurich interim was to address this as a generic HTTP issue (for both HTTP 1.X and 2), hence the generic header proposal. 

There is some concern from Amos and others about non-UTF-8 values of the header and how to handle them. The current language in the draft says to treat that as the legacy case:

	"...invalid value or unrecognized charset, this is equivalent
	   to the legacy situation of non-determinism..."

So any intermediary or server that wishes to only accept UTF-8 can treat anything else as "unrecognized" which amounts to ignoring the header altogether. We could tighten the spec to only allow UTF-8, but given the above clause that appears to be an unnecessary constraint.  

>From Julian:

	Practically, how is a UA supposed to *know* the encoding that was used for the URI *unless' it constructed it itself? (Which is not what browsers do; they only construct the query part).
	
If you don't know for sure, then don't use the header. But if you know for sure, it's useful to indicate this fact by using the headers to tighten parsing at the other side. Notice that a malicious agent would have incentive to *not* use the header so as to continue exploiting the legacy situation. Using the header imposes constraints that make it harder to exploit the current situation of non-determinism. 

We happen to know often enough that these headers would be useful in our case (and this is why we're proposing these headers: we'd use them if available). Others have chimed in along those lines as well, but each one decides. Even if it's only useful in the query case for some situations, one can use the query header and not the path header. They are independent headers for that reason.

Gabriel

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Mar 2014 08:27:49 +0100

> If you don't know for sure, then don't use the header. But if you know for sure, it's useful to indicate this fact by using the headers to tighten parsing at the other side. Notice that a malicious…

I have the impression that some of the confusion is caused by different 
people making different claims about what recipients will do with the 
server. My understanding was that Nicolas sees this as a signal that 
allows him *not* to do certain checks, while you seem to say you want to 
do *stricter* checks (which makes sense).

It would be awesome if the spec was more clear about that.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Mar 2014 08:47:08 +0100

On 2014-03-22 08:27, Julian Reschke wrote:
>>     Practically, how is a UA supposed to *know* the encoding that was
>> used for the URI *unless' it constructed it itself? (Which is not what
>> If you don't know for sure, then don't use the header. But if you know
>> for sure, it's useful to indicate this fact by using the headers to
>> tighten parsing at the other side. Notice that a malicious agent would
>> have incentive to *not* use the header so as to continue exploiting
>> the legacy situation. Using the header imposes constraints that make
>> it harder to exploit the current situation of non-determinism.
>

Sorry. s/server/header field/

> ....

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Sat, 22 Mar 2014 08:16:04 -0500

On Sat, Mar 22, 2014 at 12:40 AM, Gabriel Montenegro
<[email-masked]> wrote:
> We happen to know often enough that these headers would be useful in

Can you name some of the cases? This is not only of interest to
clients; servers need to understand the circumstances when these
headers are generated too. Thanks.

our case (and this is why we're proposing these headers: we'd use them
if available). Others have chimed in along those lines as well, but
each one decides. Even if it's only useful in the query case for some
situations, one can use the query header and not the path header. They
are independent headers for that reason.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Mar 2014 15:45:25 +0900

I have had a quick look at the proposal slides as presented at the last 
IETF, and commented on the [email-masked] mailing list (W3C 
member only, but some of you might be able to have a look at it).

Below are somewhat similar comments.

>> one value ("UTF-8").

Only distinguishing UTF-8 and non-UTF-8 is definitely very well worth 
considering. But I think it would be better to turn things around: Only 
use 'Legacy' to express that UTF-8 is NOT used.
    URI-Path-Encoding: Legacy
    URI-Query-Encoding: Legacy
Why is that the better choice? Because the Web is moving more and more 
towards UTF-8. Explicitly marking UTF-8 therefore means we have to keep 
these headers for a long time (*).

Heuristics have also been mentioned in the followup discussion.
When it comes to heuristics for character encodings and UTF-8, there are 
essentially two levels:

1) UTF-8 byte pattern detection
As known since 1997 (see 
http://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf‎), this is highly 
discriminating and reliable even for very short strings such as URI/IRI 
components, although not 100%. (The biggest chunk of wrong positives is 
double-encoding, data encoded in UTF-8 interpreted as e.g. Latin-1 and 
converted to UTF-8 a second time.)

2) Anything else
This has to make use of many different kinds of information, such as the 
set of possible/probable encodings, their byte patterns, linguistic 
information, and so on. Given the necessary data, it's not too difficult 
for longer texts (e.g. one paragraph and up), but mostly not feasible 
for shorter texts such as e.g. URIs.

That 1) is quite reliable, i.e. that data that looks like UTF-8 usually 
is UTF-8, seems to provide further support for labeling legacy rather 
than UTF-8. In fact, one proposal may be to only label data that looks 
like UTF-8 but IS NOT. The advantage would be that the header field is 
necessary in only very few cases. The disadvantage would be that the 
header field is very rare, and therefore implementations wouldn't be 
tested seriously.

>> Beyond that, folks seem generally neutral-to-positive
>> about it, AFAICT.

The reason for this seems to be that everybody is happy when they see a 
solution to a character encoding problem. What the discussion on the 
list has shown is that it's not so clear where and when this proposal 
(in whatever form) helps, and where it doesn't help.

> behave non-deterministically.

There's definitely an incentive in that direction.

But if the proposal indeed has merit, and if we can get over the initial 
adoption chicken-and-egg problem (no servers support it so clients don't 
support it and vice versa), then there should also be incentives to do 
the right thing.

> deterministic that's their problem.

I agree that whatever we can do to speed up the (already strong) 
adoption of UTF-8 is a good thing.

> encoding value not specified in the spec).

That would be the above 'Legacy' value. If we go that route, it would 
probably be a good idea to register 'Legacy'.

> bad encoding perpetrators.

I definitely agree that we should try to make those pay who don't follow 
the specs, if possible.

Regards,   Martin.

(*) Essentially we first need to make sure there's nothing else than 
UTF-8 on the Web, then we have to make sure we change all the servers so 
that they don't assume not getting one of these headers means the data 
is in a legacy encoding. This means two full server/content deployment 
cycles, which can mean easily 30 to 40 years altogether :-(.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Mar 2014 16:53:55 +0900

> be presumptuous for the browser to brave a guess and mislead the

Björn described the situation from a server point of view. From a client 
point of view, the client just encodes the path part of an URI/IRI with 
UTF-8, and the query part with the encoding of the page.

This isn't exactly ideal to say the least, but it's what happens in 
practice (unless there's an accept-charset attribute, and that only 
works on forms, see 
http://www.w3.org/TR/html5/forms.html#attr-form-accept-charset).

Therefore indeed it makes sense to have two different header fields, or 
some other way to distinguish path encoding and query encoding (assuming 
it makes sense to have such header fields in the first place).

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 24 Mar 2014 09:01:04 +0100

> UTF-8, and the query part with the encoding of the page.

That's a bit misleading. The client doesn't encode the path at all, 
unless it *requires* encoding. That is, if I have "/path%A0" in a href 
attribute, this is what will get on the wire.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Mar 2014 17:03:24 +0900

> <http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1145.html>.

If the draft gets anywhere, it has to say very clearly that the header 
field(s) must only be used when the encoding is actually known. So input 
from the address field, form input, and in-browser conversion of IRIs to 
URIs are okay, but adding header fields for URIs that are already 
%-encoded is not.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Mar 2014 17:06:55 +0900

> In particular, this seems like something that needs to be coupled to *where* the link originates; e.g., a browsers’ behaviour for a link from an address bar is likely to be different than that from…

Mark, are you saying that we need to label URIs in HTTP requests with 
whether they originated from the address bar or where else? I think that 
would be going too far, but I'm not sure that's what you meant.

> Gabriel, have you brought it up over at the W3C or in the WHATWG?

The I18N Core WG has shortly looked at it last week. But I don't think 
that they have made up their mind.

Regards,   Martin.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 24 Mar 2014 10:20:15 +0100

Le Sam 22 mars 2014 00:30, Mark Nottingham a écrit :

> *where* the link originates; e.g., a browsers’ behaviour for a link from
> an address bar is likely to be different than that from an ‘a’ tag, and
> even again different from a JavaScript-generated link.

True.
What last decade unicode migration taught us, is that once you allow text
with undefined encoding in the pipeline, things are going to fail.
Encoding really wants to be end-to-end.

Still, this is little different from mime-types, where everyone relies on
mime-type hints for things to work well, but servers and web clients
sometimes have to infer them in less-than optimal ways to fill in the http
headers. Asking for url encoding to be defined at the http level (with a
fallback to a clear encoding default otherwise) is similar, with the
detection pushed to servers and web clients, which will presumably push
for better behaviour elsewhere to avoid being saddled with the heuristics
currently pushed on network nodes.

I'll add that we are living in an insecure world right now, that to
mitigate security problems everyone has been adding blacklists of
known-bad urls (from browsers to java to adobe reader to various security
extensions) but those blacklists rely on someone being able to review and
curate them. Which is obviously is a problem when you start to accept URLs
you're not able to decode or display in any reasonable manner.

Regards,

-- 
Nicolas Mailhot

---

From: Zhong Yu <[email-masked]>
Date: Mon, 24 Mar 2014 10:06:12 -0500

Hi, can you give a specific example? A real-world URL you want to
block, and how an adversary could circumvent it. The threat model may
be obvious to your industry, but not so much to the others.

On Mon, Mar 24, 2014 at 4:20 AM, Nicolas Mailhot
<[email-masked]> wrote:
>> *where* the link originates; e.g., a browsers' behaviour for a link from
>> an address bar is likely to be different than that from an 'a' tag, and

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 25 Mar 2014 12:04:35 +0100

>that we can change it in process if there's some particular aspect you 

Use of not otherwise encoded non-textual binary data in 'http' URIs is
vanishingly rare, and use of character encodings other than UTF-8 will
only go down over time. So, the ideal situation would be that around
100% of all requests come with these two additional headers, telling
servers something they practically already know. That does not make a
lot of sense to me. And so far nobody has explained how clients would
determine the encoding used in addresses, outside a narrowly confined
area. I think we would be better served with some "best practise" form
of document instead of the proposed headers.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/