Subject: Re: consensus on :query ?

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jul 2014 15:50:56 -0400

<https://github.com/http2/http2-spec/issues/556>

I’m not hearing anyone else express interest in this.

Comments? Unless there’s a clamour of support soon, I’m inclined to close with no action.

Cheers,

On 12 Jul 2014, at 3:19 am, Poul-Henning Kamp <[email-masked]> wrote:

> Some time ago I proposed and argued that :query should be split out
> from :path and as I perceived it, there were a lot of people nodding
> on that.
> 
> I don't want to inject it right now, we have plenty of balls in the
> air, but can I get you top open an issue for it, so we don't forget ?
> 
> (Some of) the arguments for :query:
> 
> 	Load balancers and other "triage" proxies seldom if ever
> 	look at the :query part to determine handling, splitting
> 	this field out of :path means they don't have as much data
> 	to run through the decompressor to make their decisions.
> 
> 	Semantically :path is much more static than :query is, which
> 	means that :path can be compressed by back-reference, whereas
> 	:query almost always will need huffman coding.
> 
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> [email-masked]         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe    
> Never attribute to malice what can adequately be explained by incompetence.

--
Mark Nottingham   http://www.mnot.net/

---

From: Phil Hunt <[email-masked]>
Date: Sun, 20 Jul 2014 13:31:15 -0700

There is a broader issue of how to handle queries in HTTP...

Do as a separate header against any http method as suggested?

Or...

Do as a new http method eg QUERY to reduce overloading of POST and other methods. 

A big security issue is how to keep sensitive information out of URLs. 

I am not sure about the appropriate solution but I am interested in discussion to find a better one. 

Phil

> On Jul 20, 2014, at 12:50, Mark Nottingham <[email-masked]> wrote:
>

---

From: Erik Nygren <[email-masked]>
Date: Sun, 20 Jul 2014 16:46:27 -0400

I'd think there would be plenty of sites where this could be a big win
HPACK-wise
where a small number of paths are used with a large number of different
query strings.
Do we have any data one way or the other?  It seems like evaluating this by
data would make sense.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 20 Jul 2014 13:49:49 -0700

It could make guessing things potentially easier.

In other words, this would need a security review.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 20 Jul 2014 20:59:29 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>I'm not hearing anyone else express interest in this.
>
>Comments? Unless there's a clamour of support soon, I'm inclined to 
>close with no action.

It would be interesting if somebody with a setup and data
could try to see how it affects compression.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 20 Jul 2014 21:03:23 +0000

>Do as a new http method eg QUERY to reduce overloading of POST

Uhm, I think you're misunderstanding.

My proposal is to transport the part of the url after the first '?'
in a separate pseudo-header (:query), just line we have already
split the url into :scheme, :authority, :method and :path.

The main argument is that this will improve compression, since
the :path part is much more constant than the :query part.

:query parts are and can be used with all :methods.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 20 Jul 2014 21:05:10 +0000

In message <CAP+FsNfy-3V_BRcqa1ATts7SgX=[email-masked]>

>It could make guessing things potentially easier.

Please explain ?

---

From: Roberto Peon <[email-masked]>
Date: Sun, 20 Jul 2014 18:33:01 -0700

One doesn't have to guess path + query, one only guess the query.
In some scenarios, this enhances the attacker's ability to probe.
The question is, does it do so enough for us to care.

-=R

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 21 Jul 2014 11:50:50 +1000

I'm +0 for this, in that I think it is a good idea, but not something that
is a must have.
We already imply the scheme, cut out the authority, so why not complete the
decomposition.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Phil Hunt <[email-masked]>
Date: Sun, 20 Jul 2014 23:26:29 -0400

Not sure why one would be worse than the other guess-ability wise. 

If a header aids compression too thats good. 

Phil

> On Jul 20, 2014, at 21:33, Roberto Peon <[email-masked]> wrote:
> 
>> On Sun, Jul 20, 2014 at 2:05 PM, Poul-Henning Kamp <[email-masked]> wrote:

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jul 2014 07:46:22 +0200

Hi Roberto,

> The question is, does it do so enough for us to care.

I don't see why it would be a trouble since the path is generally
well known and could be considered constant. It will be retrieved
from a link on a page, a location header, or will just be "/" or
something like this.

Willy

---

From: Roberto Peon <[email-masked]>
Date: Sun, 20 Jul 2014 23:51:37 -0700

Don't get me wrong-- I think it'd be mostly fine.
I also think, however, that this is a piece of information which is likely
to contain sensitive information, and as a result, if we want to do
something different than we do now, we should get it reviewed.

One simple example of how this makes stuff easier-- Since the path and
query are separated, an attack that attacks the local state by acting as a
malicious mitm of TCP packets may perform more attacks before the TCP recv
window runs out.

Do I think that is a particularly strong weakness? No. However, it still
needs review.
-=R

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jul 2014 09:04:34 +0200

> Do I think that is a particularly strong weakness? No. However, it still

OK so I think we're in line then. Sure it needs to be reviewed, and I'm
not worried either (I guess whatever can be found on it will also affect
current state of the spec).

Willy

---

From: Roberto Peon <[email-masked]>
Date: Mon, 21 Jul 2014 00:14:18 -0700

Assuming that query params get put into the compressor, splitting the path
off means that an attacker gets to test against all of those query-parts
with a query and *any* path.
This would be a big change in terms of security properties.

-=R

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jul 2014 09:53:12 +0200

> with a query and *any* path.

I'm not sure what you mean, we're speaking about having a single :query
for whatever follows the question mark, right ? If so, all the params
must be tried as a single block.

Willy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Jul 2014 06:20:27 -0700

> must be tried as a single block.

Yes, but there could be cases where the combination of path and query
contain sufficiently high entropy in combination, but one or other
contains insufficient entropy on its own to resist guessing attacks.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 21 Jul 2014 06:24:04 -0700

> contains insufficient entropy on its own to resist guessing attacks.

I concur with Martin's analysis

Consider the case where we have sensitive information split between the
path and the query. E.g.

https://login.example.com/ekr?<password>

If the username is unknown, this lets them be guessed independently.

-Ekr

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Jul 2014 15:29:44 +0200

> contains insufficient entropy on its own to resist guessing attacks.

...again, if we do things like that please do not couple it with "?". 
Just have two parts that get concatenated verbatim to reconstruct the 
full path+query.

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Jul 2014 06:37:34 -0700

On 21 July 2014 06:29, Julian Reschke <[email-masked]> wrote:
> ...again, if we do things like that please do not couple it with "?". Just
> have two parts that get concatenated verbatim to reconstruct the full
> path+query.

I don't know how to act on that.  Are you suggesting that a 1.1 to 2
gateway would be unable to perform any separation?

---

From: James M Snell <[email-masked]>
Date: Mon, 21 Jul 2014 06:37:07 -0700

Well, hopefully we'd be hard pressed to find many real world examples of
passwords in the query string, but there are many examples involving API
keys and authorization tokens. These, fortunately, tend to have a greater
entropy than passwords but are still quite dangerous. Whether separating
them out to a new :query field makes them more vulnerable, I have no idea.
Would be interesting to test.

---

From: Patrick McManus <[email-masked]>
Date: Mon, 21 Jul 2014 09:44:49 -0400

a quick survey of my open tabs shows 4 of them have non-query portions of
the URL that are things I would expect an attacker to have to guess - even
if she were familiar with the same application. So the current status of
things seems worthwhile, despite the compression hit. (That's the whole
basic tradeoff of hpack vs gzip anyhow.)

-P

On Mon, Jul 21, 2014 at 9:37 AM, James M Snell <[email-masked]> wrote:

>> On Mon, Jul 21, 2014 at 6:20 AM, Martin Thomson <[email-masked]
>> > wrote:
>>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jul 2014 09:46:32 -0400

On 21 Jul 2014, at 9:42 am, Willy Tarreau <[email-masked]> wrote:

> On Mon, Jul 21, 2014 at 06:24:04AM -0700, Eric Rescorla wrote:
> I didn't understand you were suggesting such a case, because for me "ekr"
> above would be well-known as it would typically be presented on the login
> page itself in the form of a link, so it would not be considered part of
> the secret.
> 
> Thanks for explaining your example case at least, even if I find it hard
> to find a real world case involing this and without "ekr" being already
> public.

The point is that this has a surprising interaction with HPACK. 

I know the security folks are already nervous about low-entropy secrets and HPACK, and this doesn’t help; while it’s true that in the common case the path is known, it creates yet another thing that people need to understand to secure the protocol properly.

At this point I’m concerned enough about getting through our security review that I’m inclined not to do this; the possible benefit (which the WG seems open to, if not enthusiastic about) doesn’t appear to be worth the risk (both that to the actual security of people using the protocol, and the risk of not making it through our security reviews).

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jul 2014 15:42:06 +0200

Willy

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 21 Jul 2014 06:53:16 -0700

On Mon, Jul 21, 2014 at 6:42 AM, Willy Tarreau <[email-masked]> wrote:

> I didn't understand you were suggesting such a case,

Well, this is the first time I have posted on this thread :)

> because for me "ekr"

I agree this is a bit contrived. I suspect there are non-contrived cases.

-Ekr

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 14:14:41 +0000

>Just have two parts that get concatenated verbatim to reconstruct the 

The definition (sorry RFC2616, I don't have the newer one here right
now):

     http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]]

So no, '?' it is if we do it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 14:16:57 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>At this point I'm concerned enough about getting through our security 
>review that I'm inclined not to do this; the possible benefit (which 
>the WG seems open to, if not enthusiastic about) doesn't appear to be 
>worth the risk (both that to the actual security of people using the 
>protocol, and the risk of not making it through our security reviews).

I would like to have some idea about the compression gain before
we make up our mind.

---

From: Jeroen de Borst <[email-masked]>
Date: Mon, 21 Jul 2014 14:23:54 +0000

Does adding :query imply that seeing a Œ?¹ in :path now requires error
handling?

Jeroen

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Jul 2014 16:28:22 +0200

> gateway would be unable to perform any separation?

No. It's about the fact that making a privacy-related decision based on 
what's in the path and what's in the query is misguided. Everything that 
could be a in a query can be in the path as well, and it would be bad if 
something we do here affects the way how people design their URIs.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 14:29:14 +0000

In message <[email-masked]>, Jeroen de Borst writes:

>Does adding :query imply that seeing a '?' in :path now requires error
>handling?

It be a good idea to make the :query optional to use.

That way people who care about the compression get it, and people
who worry about security impacts can avoid it.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Jul 2014 16:32:25 +0200

> So no, '?' it is if we do it.

I have no idea how that follows from the URI grammar.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 14:32:28 +0000

In message <[email-masked]>, Julian Reschke writes:
>what's in the path and what's in the query is misguided.

Uhm, the primary argument is compression ratio.

Privacy will only be relevant if we go further in HTTP/3.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jul 2014 10:32:48 -0400

> who worry about security impacts can avoid it.

That sounds like an interop nightmare… what do you do if there are both? Lots of edge cases...

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Jul 2014 07:36:04 -0700

> something we do here affects the way how people design their URIs.

Yes.  Absolutely.  I was taking that as read.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 14:52:17 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:
>
>On 21 Jul 2014, at 10:29 am, Poul-Henning Kamp <[email-masked]> =
>wrote:
>
>>=20
>>> Does adding :query imply that seeing a '?' in :path now requires =
>error
>>> handling?
>>=20
>>=20
>That sounds like an interop nightmare=85 what do you do if there are =
>both? Lots of edge cases...

You always append '?' and :query and leave people with the result
the asked for...

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jul 2014 17:02:38 +0200

> You always append '?' and :query and leave people with the result

Not exactly, I'd say you append '?' only if :query is present (eventhough
empty) then append :query.

Willy

---

From: James M Snell <[email-masked]>
Date: Mon, 21 Jul 2014 08:18:18 -0700

I'm -1 on having a separate :query. If only because it requires
implementers to look in two separate locations and perform additional
string manipulation for a task that currently (in h1) doesn't. I have seen
deployed implementations that split routing detail between the path and
query string. The relatively small theoretical compression gain is not
worth the additional cost.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 15:20:02 +0000

>empty) then append :query.

Sorry for being unclear:  That was what I meant.

---

From: Ted Hardie <[email-masked]>
Date: Mon, 21 Jul 2014 11:23:07 -0400

On Mon, Jul 21, 2014 at 11:02 AM, Willy Tarreau <[email-masked]> wrote:

> Will

​And in the case where there is both :path with a ? and a :query, you
then...?  Put in a %3F and the :query?
Omit the :query?  concatenate the query to the path with the ? or the %3f?

I agree with Mark; making this optional makes no sense.

Ted

---

From: Alvaro Lopez Ortega <[email-masked]>
Date: Mon, 21 Jul 2014 22:40:59 +0200

> I’m not hearing anyone else express interest in this.

It has pros and cons. On the one hand, it'll most likely introduce corner
cases that implementation will have to handle properly. On the other hand
though, I'd have loved this feature when I implemented Cherokee. It'd
definitely have made a certain parts of the code much more straightforward.

I'd say it's an interesting nice-to-have. +1.

Best,
Alvaro

-- 
All the best,
Alvaro

---

From: Jason Greene <[email-masked]>
Date: Mon, 21 Jul 2014 17:50:21 -0500

> I agree this is a bit contrived. I suspect there are non-contrived cases.

I think the reverse is a more likely vector. You rely on query getting stored in the header table, which contains authentication tokens, and then you alter the path to affect other resources.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 21 Jul 2014 23:21:44 +0000

I don't see how it makes any difference.  Splitting something in two 
(path?query vs. path, query) doesn't add or subtract information or 
alter entropy.  It's just a different way of parsing.

------ Original Message ------
From: "Martin Thomson" <[email-masked]>
To: "Willy Tarreau" <[email-masked]>
Cc: "Roberto Peon" <[email-masked]>; "Poul-Henning Kamp" 
<[email-masked]>; "Phil Hunt" <[email-masked]>; "Mark 
Nottingham" <[email-masked]>; "HTTP Working Group" <[email-masked]>
Sent: 22/07/2014 1:20:27 a.m.
Subject: Re: consensus on :query ?

---

From: Roberto Peon <[email-masked]>
Date: Mon, 21 Jul 2014 16:24:56 -0700

If the path contains:
/foo/RANDOM_NUMBER/bar

and the query contains:
q=foo&user=SOME_SECRET_ID

Then guessing:
/foo/RANDOM_NUMBER/bar?q=foo&user=SOME_SECRET_ID

is far, far FAR more difficult than guessing:
  q=foo&user=SOME_SECRET_ID
alone or
  /foo/RANDOM_NUMBER/bar
alone.

-=R

On Mon, Jul 21, 2014 at 4:21 PM, Adrien de Croy <[email-masked]> wrote:

> (path?query vs. path, query) doesn't add or subtract information or alter
> entropy.  It's just a different way of parsing.
>
>
>
> Cc: "Roberto Peon" <[email-masked]>; "Poul-Henning Kamp" <
> [email-masked]>; "Phil Hunt" <[email-masked]>; "Mark

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 21 Jul 2014 23:33:09 +0000

Sorry I still don't understand.

If the server needs both a correct path and correct query to provide the 
desired response, then surely you need to guess both.

Or are we suggesting that path can be guessed independently because 
there's a differernt status returned for invalid query vs invalid path?

In which case how does that differ from now?

------ Original Message ------
From: "Roberto Peon" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Martin Thomson" <[email-masked]>; "Willy Tarreau" 
<[email-masked]>; "Poul-Henning Kamp" <[email-masked]>; "Phil Hunt" 
<[email-masked]>; "Mark Nottingham" <[email-masked]>; "HTTP Working 
Group" <[email-masked]>
Sent: 22/07/2014 11:24:56 a.m.
Subject: Re: consensus on :query ?

---

From: Roberto Peon <[email-masked]>
Date: Mon, 21 Jul 2014 16:36:56 -0700

You're missing the nature of the attack.

The nature of the attack is to cause the client to emit packets, and to
look at the size of the packets.
If you've compressed something, then the packet (at least without padding)
is smaller.
This implies that a 3rd party can send links to the browser and, while
watching the output size, figure out when it 'hit' something in the
compression context.

The fact that the server is sending 4XXs or 5XXs, really doesn't come into
it (except that it is a hint to the server that there might be a malicious
attacker and it should set the compression context size to zero).
-=R

On Mon, Jul 21, 2014 at 4:33 PM, Adrien de Croy <[email-masked]> wrote:

> Cc: "Martin Thomson" <[email-masked]>; "Willy Tarreau" <[email-masked]>;
> "Poul-Henning Kamp" <[email-masked]>; "Phil Hunt" <
> [email-masked]>; "Mark Nottingham" <[email-masked]>; "HTTP Working

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 21 Jul 2014 23:52:13 +0000

OK thanks, I missed the context.

So many mails on this list now it's impossible to do any sort of day job 
if I want to keep up with them.

hence I'm well behind.

------ Original Message ------
From: "Roberto Peon" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Martin Thomson" <[email-masked]>; "Willy Tarreau" 
Sent: 22/07/2014 11:36:56 a.m.

>If you've compressed something, then the packet (at least without 
>padding) is smaller.
>The fact that the server is sending 4XXs or 5XXs, really doesn't come 
>into it (except that it is a hint to the server that there might be a 
>malicious attacker and it should set the compression context size to 
>zero).
>-=R
>
>
>
>>If the server needs both a correct path and correct query to provide 
>>the desired response, then surely you need to guess both.
>>
>><[email-masked]>; "Mark Nottingham" <[email-masked]>; "HTTP 
>>Working Group" <[email-masked]>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 22 Jul 2014 05:40:46 +0000

In message <CAP+FsNcaxeEhEpQCAteQUZGn03OXTv=[email-masked]>

>  q=foo&user=SOME_SECRET_ID

Only if you have an oracle to tell you that you got a hit.

Could you outline exactly how this attack would work ?

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 22 Jul 2014 07:53:53 +0200

> Could you outline exactly how this attack would work ?

You can for example share the same proxy as the victim, send requests
with your guesses there and observe the size of data on the encrypted
communication with the server to determine whether the proxy detected
the same path+query as the previous request and managed to reference
an indexed entry or had to send a literal.

Sometimes you can also control a page the victim displays, and reference
objects belonging to the site you want to steal the credentials for.
Using frames you can have the victim think he/she's on the correct site
and enter credentials (which are only sent to the real site), and in
another frame or using image links you can try to guess the contents
again by forging requests and observing the size on the link.

Regards,
Willy

---

From: Roberto Peon <[email-masked]>
Date: Mon, 21 Jul 2014 23:08:21 -0700

Like so: http://en.wikipedia.org/wiki/CRIME
-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 22 Jul 2014 06:15:11 +0000

In message <CAP+FsNfCO34LAr9gzetXDs5=UdS2zOh8CHRn=[email-masked]>

>Like so: http://en.wikipedia.org/wiki/CRIME

Apart from some pretty marginal difference in odds, I don't see
how splitting :query changes the equation.

To take CRIME seriously means that the URL should never be compressed
and always randomly padded, and we don't do that.

---

From: David Krauss <[email-masked]>
Date: Tue, 22 Jul 2014 14:44:41 +0800

> and always randomly padded, and we don't do that.

Well, actually that is the recommended practice for applications that do take CRIME seriously.

Which is why I’m not sure that there’s a security impact to splitting out :query. If :path is not secret, then it can be stored in the header table. If it is secret, then the same measures will be applied as to :query and the impact of splitting (vs not splitting) is just a few bits.

Anyway if :path remains constant then its contribution to the header block size will also be constant, regardless of compression or lack thereof.

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 23 Jul 2014 00:11:23 +0000

path isn't allowed to have '?'

I was really hoping moving to a binary protocol would help us avoid 
string parsing, which requires defining / escaping of in-line structural 
elements, and creates a lot of opportunities for the type of bugs that 
create security holes.  It's just like SQL - the difference between 
using parsed SQL (and all the injection exploits that opens up) 
containing commands interspersed with data vs SQLBindParameter where 
data can only be data, not a command.

I think the benefits of separating out path from query are huge.

------ Original Message ------
From: "Ted Hardie" <[email-masked]>
To: "Willy Tarreau" <[email-masked]>
Cc: "Poul-Henning Kamp" <[email-masked]>; "Mark Nottingham" 
<[email-masked]>; "Jeroen de Borst" <[email-masked]>; "Eric Rescorla" 
<[email-masked]>; "Martin Thomson" <[email-masked]>; "Roberto 
Peon" <[email-masked]>; "Phil Hunt" <[email-masked]>; "HTTP 
Working Group" <[email-masked]>
Sent: 22/07/2014 3:23:07 a.m.
Subject: Re: consensus on :query ?

>> > >> In message <mailto:[email-masked]>, Jeroen de 
>>Borst writes:
>> > >>=20

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Jul 2014 07:52:15 -0700

On 22 July 2014 17:11, Adrien de Croy <[email-masked]> wrote:
> I was really hoping moving to a binary protocol would help us avoid string
> parsing

What a quaint idea :)  Especially for URIs, for which denying their
string nature would be something of a surprise.

At this stage, we've made the framing binary, which helps.  But more
drastic changes (see Julian's -jfv draft) are needed to make the
"binary benefit" more pervasive.

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 23 Jul 2014 23:06:57 +0000

a URI is just a construction of several components glued together with 
delimiters, e.g.

://
@
:
/
?
&
#

etc.

this places constraints on the component values, since you can't use 
structural delimiters inside values.  This means if we do want to 
include such things, we have to escape them, and it snowballs from 
there.

Imagine if we just sent all individual parts of a URI in different 
fields, where we didn't need to parse them to distinguish the parts.  No 
more %20 vs +, no more string escape unicode exploits.

Sure we might need to aggregate things to create a cache key etc, but 
that's a safe operation.

------ Original Message ------
From: "Martin Thomson" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "HTTP Working Group" <[email-masked]>
Sent: 24/07/2014 2:52:15 a.m.
Subject: Re: consensus on :query ?

>>string

---

From: Zhong Yu <[email-masked]>
Date: Wed, 23 Jul 2014 19:35:19 -0500

On Wed, Jul 23, 2014 at 6:06 PM, Adrien de Croy <[email-masked]> wrote:
> structural delimiters inside values.  This means if we do want to include
> such things, we have to escape them, and it snowballs from there.
>
> Imagine if we just sent all individual parts of a URI in different fields,

We'll be facing a much more daunting task - define an official
structure for URIs. Is it a list? Is it a map? A multimap? How are
entries ordered? Should it be a tree instead? And "why don't you guys
just make it like json?!" etc. etc.

Zhong Yu
bayou.io

> where we didn't need to parse them to distinguish the parts.  No more %20 vs
> +, no more string escape unicode exploits.
>
> Sure we might need to aggregate things to create a cache key etc, but that's
> a safe operation.
>
>

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 24 Jul 2014 10:40:08 +1000

> a URI is just a construction of several components glued together with
Technically "&" isn't important in a query (or elsewhere in a http URI).
It's a common convention to make the query be a sequence of
key=value&key=value pairs, but there's nothing in the spec, so we can't
actually break down the query and have it still compliant with all possible
uses.

> such things, we have to escape them, and it snowballs from there.
The only ones we have to escape this way are the reserved characters, and
really, within a protocol like HTTP, the only ones we absolutely have to
escape are the ones that have meaning to the protocol and its applications
(e.g. "/" and "?" within the path). Although the gods know what sort of
random applications there are out that that absolutely depend on "$" being
distinguished from "%24", etc.

I could see some interest in splitting the path into either a single "*" or
a list of segments (removing "%2F" from the path string), but for one how
many URLs have %2F in the path? And for two the rules in RFC 3986 and a
couple of decades of experience already have that covered, I think.

I guess you could split the authority into userinfo/host/port if you were
desperate for such a thing.

> where we didn't need to parse them to distinguish the parts.  No more %20
> vs +, no more string escape unicode exploits.
>
>
The mapping of ​"+" to a space character is a de facto standard, but it's
not codified. We can't assume that all plus signs in a query string are
meant to represent space characters. And, as I said earlier, we can't
really make PHP's assumption about the query, and parse it down to a $_GET
assocarray, because there are more uses of HTTP than just PHP.

> that's a safe operation.
Not if you've converted both "%20" and "+" to spaces; that's non-reversible.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Zhong Yu <[email-masked]>
Date: Wed, 23 Jul 2014 19:57:09 -0500

The &key=value format and the encoding of SPACE as PLUS is from HTML
standard for form submission. It's not part of HTTP or URI standards.

On Wed, Jul 23, 2014 at 7:40 PM, Matthew Kerwin <[email-masked]> wrote:
> The mapping of "+" to a space character is a de facto standard, but it's not
> codified. We can't assume that all plus signs in a query string are meant to
> represent space characters. And, as I said earlier, we can't really make
> PHP's assumption about the query, and parse it down to a $_GET assocarray,
> because there are more uses of HTTP than just PHP.
>
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 24 Jul 2014 01:19:16 +0000

there already is a definition for all the parts of the URI, they are 
just glued together is all.

Actually though I wasn't proposing splitting path down to nodes.  That's 
file-system dependent and should be opaque to http.

But it shouldn't require encoding to fit into a URI.  It's not just / 
and ? (rare in a real path since it's usually a wild-card character 
anyway)... it's more often unicode and spaces that had to be encoded.

And there's probably little value in splitting out authority as well.

Adrien

------ Original Message ------
From: "Zhong Yu" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Martin Thomson" <[email-masked]>; "HTTP Working Group" 
<[email-masked]>
Sent: 24/07/2014 12:35:19 p.m.
Subject: Re: consensus on :query ?

>>include
>>fields,
>>that's

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 24 Jul 2014 12:04:13 +1000

On 24 July 2014 11:19, Adrien de Croy <[email-masked]> wrote:

>
> But it shouldn't require encoding to fit into a URI.  It's not just / and
> ? (rare in a real path since it's usually a wild-card character anyway)...
> it's more often unicode and spaces that had to be encoded.
>

HTTP defines URI schemes, which contains ASCII, not Unicode. We've never
mentioned HTTP IRIs, and I don't think it's reasonable at this stage to
unilaterally decide that all HTTP URIs must map to IRIs, via
percent-encoded UTF-8.

If you happen to map U+00E4 to "%C3%A4" when generating your URI that's
fine, but there's no mention in the HTTP spec (where it defines the URI
schemes) of un-percent-encoding such a URI, or that "%C3%A4" should map to
a UTF-8 "ä" (as opposed to, say, Windows-1252 "Ã¤", or EBCDIC "Cu"). RFC
3986 says "a new URI scheme ... should first be encoded as octets according
to the UTF-8 character encoding; then ... percent-encoded" but I don't
think that applies here -- the http URI scheme pre-dates RFC 3986, even if
RFC 7230 doesn't.

​Similarly, not all implementations equate "+" and "%20"; I recall even
seeing some where they are interpreted as different classes of space
delimiters.​

If there were separate :path and :query headers, they'd have to both
contain ASCII strings, including any percent-encoded reserved or non-ASCII
octets. There's not much more we could do to binarise them. (Maybe decoding
non-reserved "%XX" triplets into single octets, although I think that's
worse than just leaving them as they are, especially with our Huffman codes
that do good things with % and 0-9, A-F.)

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 24 Jul 2014 06:18:46 +0000

In message <CACweHNC9rP3i=[email-masked]>

>desperate for such a thing.

Uhmm, lets not go overboard here, please ?

I'm proposing splitting :query for :path, because it is a well known
distinction which through its established use-pattern gives us a
handy way to improve compression significantly in common scenarios.

We already split the HTTP URL into it's other major parts, :scheme,
:authority and :path, so splitting out the last major part is not
a big step and involves neither complex text nor introduces a lot
of error handling.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Jul 2014 13:30:52 -0400

As per our other decisions recently, this is too speculative to include in HTTP/2 at this late stage. Happy for the discussion to continue for the benefit of future work, but the underlying issue for HTTP/2 has been closed.

Cheers,

---

From: Roland Zink <[email-masked]>
Date: Thu, 24 Jul 2014 18:37:18 -0400

What if an attacker can get the path through an refer header?

> Am 22.07.2014 um 02:08 schrieb Roberto Peon <[email-masked]>:
> 
>> On Mon, Jul 21, 2014 at 10:40 PM, Poul-Henning Kamp <[email-masked]> wrote:

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Mon, 28 Jul 2014 09:03:25 +0000

select count(1) from <yesterday’s traffic via the CWS proxy> where query like '%&password=%';

130,740

So not very hard pressed.

Richard

From: James M Snell [mailto:[email-masked]]
Sent: 21 July 2014 14:37
To: Eric Rescorla
Cc: Mark Nottingham; Roberto Peon; Phil Hunt; Willy Tarreau; HTTP Working Group; Poul-Henning Kamp; Martin Thomson
Subject: Re: consensus on :query ?

Well, hopefully we'd be hard pressed to find many real world examples of passwords in the query string, but there are many examples involving API keys and authorization tokens. These, fortunately, tend to have a greater entropy than passwords but are still quite dangerous. Whether separating them out to a new :query field makes them more vulnerable, I have no idea. Would be interesting to test.
On Jul 21, 2014 6:28 AM, "Eric Rescorla" <[email-masked]<mailto:[email-masked]>> wrote:

On Mon, Jul 21, 2014 at 6:20 AM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:

-Ekr