Subject: Re: Do we kill the "Host:" header in HTTP/2 ?

From: Amos Jeffries <[email-masked]>
Date: Fri, 01 Feb 2013 22:47:50 +1300

On 1/02/2013 8:09 p.m., James M Snell wrote:
> One proposal on this particular topic...
>
> We can combine the :scheme, :method, :host and :path header fields 
> into a single :req Header with a compact binary encoding and require 
> that this single header always appear first in request header blocks.
>
> +------------------------------+
> |S|len(method)|method|len(host)|
> +-+-------+----+---------+-----+
> | host  | len(path) |  path  |
> +------------------------------+
>
> S = Single bit, when set, scheme = https, when not set, scheme = http
> len(method) = 7 bit length of method name
> method = name of method
> len(host) = uvarint(host)
> host = host
> len(path) = uvarint(path)
> path = path
>

This makes several assumptions which are false and will cause a lot of 
trouble:
  1) scheme of URI is always http(s)://.
  2) port is always the default scheme port.
  3) there is never a query string attached
  4) there is never a fragment attached

It also ignores that abomination otherwise known as the '*' URI.
IMHO, this is a good loss, we could require OPTIONS method on the '/' 
URI with the current machines hostname or IP do the equivalent and save 
some parser complexity + CPU checking the * cases on every request.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 01 Feb 2013 10:06:51 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------

>This makes several assumptions which are false and will cause a lot of 

I must admit that I'm not terribly happy about the lack of generality
in James' proposal either.

The performance and complexity difference between James proposal
and simply:

	[length][method][length][absolute_uri]

Is vanishingly small, but the latter would be much more general.

I might even be tempted to suggest:

	[length][method SP absolute_uri]

Because the most frequently used methods would take up less space
that way.

We could even decide to encode the HTTP "GET" method simply as "G",
"POST" as "P" etc. while still leaving room in the protocol for
somebody to implement a custom "FOOBAR" method of their own.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 01 Feb 2013 10:25:47 +0000

>From a compressibility / redundancy reduction standpoint, it makes sense 
to have the parts of the URI able to be individually addressed within 
the frame, in a similar way to the other fields.

There are many sites where every page is just index.php?id=xxxx with 
differing values for xxxx

Also why transmit the host every request when it hardly ever changes.

Therefore sending an aggregated URI, which needs to be parsed at the 
other end (or in the middle) is an idea I don't like.

Even if it makes it simpler to do URI-based routing decisions, well I'm 
not convinced that is a good enough reason.  Mostly I would expect 
switching to be done on the host part anyway.

It also doesn't make much sense to send methods as textual names.  A 
simple byte would do, reserving the value of 0xFF to indicate a textual 
name follows.  All other values being registered methods.  If 256 
methods aren't enough, maybe we need to re-evaluate our designs.  There 
could even be more information coded into method binary values, like 
reserving a bit in it for "safety" or "idempotency" or one for 
"invalidates cache" or "can have body" etc so that an intermediary need 
not necessarily understand the method to know what to do with it.

Adrien

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 01 Feb 2013 10:39:45 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <em21b5c00f-6f56-4671-b498-5a198dc090e9@bombed>, "Adrien W. de Croy"

>Also why transmit the host every request when it hardly ever changes.

Just because your installation doesn't see it changing, doesn't mean
that it "hardly ever changes" for everybody else :-)

A load-balancer in front of a web-hosting farm will often see
many different hosts requested on the same connection, in particular
if the connection comes from a proxy.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 1 Feb 2013 21:18:01 +0900

Most often of the same set of domains unless some interesting analytics are
being collected, in which one is always SOL :)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 1 Feb 2013 13:44:48 +0100

> Because the most frequently used methods would take up less space

Or something like :

   [host_len][uri_len][host][uri]

That way, host is always easy to look up (fixed position), uri is
easy to look up, and the concatenation of the two is always present.

> somebody to implement a custom "FOOBAR" method of their own.

That would be a good start, yes.

Willy

---

From: James M Snell <[email-masked]>
Date: Fri, 1 Feb 2013 07:43:27 -0800

On Feb 1, 2013 1:50 AM, "Amos Jeffries" <[email-masked]> wrote:
>> We can combine the :scheme, :method, :host and :path header fields into
a single :req Header with a compact binary encoding and require that this
single header always appear first in request header blocks.
>  1) scheme of URI is always http(s)://.

Yes, it does make this assumption. It seems, rather safe to me. What other
schemes do we need to support?

>  2) port is always the default scheme port.

No, I left this out of the examples but I assumed port would be specified
as part of host. A separate uvarint field could be used instead.

>  3) there is never a query string attached

No, I left this out of the examples but I assumed it would be sent as part
of the path.

>  4) there is never a fragment attached

Well, fragments currently are not passed so this is  a safe assumption, but
they would be included in the path if passed at all.

> IMHO, this is a good loss, we could require OPTIONS method on the '/' URI
with the current machines hostname or IP do the equivalent and save some
parser complexity + CPU checking the * cases on every request.

If we need it the * uri can be approximated by sending a zero length path.

- james

---

From: Nico Williams <[email-masked]>
Date: Fri, 1 Feb 2013 09:56:24 -0600

> schemes do we need to support?

I don't think that's a safe assumption at all.  I've heard of other
schemes used in production systems (in enterprises, granted, but so
what, the same might be useful in the Internet).

Nico
--

---

From: James M Snell <[email-masked]>
Date: Fri, 1 Feb 2013 08:12:45 -0800

On Feb 1, 2013 4:50 AM, "Willy Tarreau" <[email-masked]> wrote:
>[snip]

Given that we have at least three or four registered methods that start
with the letter P, that doesn't work. And given that methods are extensible
via ad hoc or registration, a numeric identifier is difficult (but not
impossible). I'm not opposed to it, but It would make deploying new
registered methods harder. Method names are short, there are higher
priority optimization targets.

- James

---

From: Karl Dubost <[email-masked]>
Date: Fri, 1 Feb 2013 11:14:30 -0500

> somebody to implement a custom "FOOBAR" method of their own.

Almost. The issue is POST and PUT with P. Two letters could do the job.
That said GET, POST and HEAD are massively used compared to the others, so maybe there's room to change PUT identifier as U, but that would confuse people and implementers.

C CO CONNECT
D DE DELETE
G GE GET
H HE HEAD
O OP OPTIONS
P PO POST
P PU PUT
T TR TRACE 

-- 
Karl Dubost, a Web opener to hire
http://www.la-grange.net/karl/

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 01 Feb 2013 08:32:24 -0800

Hi Karl,

--On February 1, 2013 11:14:30 AM -0500 Karl Dubost <[email-masked]> 
wrote:

> That said GET, POST and HEAD are massively used compared to the others,
> so maybe there's room to change PUT identifier as U, but that would
> confuse people and implementers.
>
> C CO CONNECT
> D DE DELETE
> G GE GET
> H HE HEAD
> O OP OPTIONS
> P PO POST
> P PU PUT
> T TR TRACE

There is a lot more than just those: 
<http://tools.ietf.org/html/draft-ietf-httpbis-method-registrations-10#appendix-A>.

-- 
Cyrus Daboo

---

From: Julian Reschke <[email-masked]>
Date: Fri, 01 Feb 2013 17:41:06 +0100

> That said GET, POST and HEAD are massively used compared to the others, so maybe there's room to change PUT identifier as U, but that would confuse people and implementers.

We could also use "?" for GET and "!" for POST.

Best regards, Julian

---

From: Karl Dubost <[email-masked]>
Date: Fri, 1 Feb 2013 11:42:49 -0500

Le 1 févr. 2013 à 11:32, Cyrus Daboo a écrit :
> There is a lot more than just those: <http://tools.ietf.org/html/draft-ietf-httpbis-method-registrations-10#appendix-A>.

oooh yes my bad. Had forgotten about those. Even two letters are not working.

-- 
Karl Dubost
Montréal, QC, Canada

---

From: James M Snell <[email-masked]>
Date: Fri, 1 Feb 2013 09:02:10 -0800

In such edge cases, the separate :scheme header can be used.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 01 Feb 2013 17:03:53 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>
, James M Snell writes:

>with the letter P, that doesn't work.

Sure does, just with less mnemonic encodings.

Remember:  These _are_ encodings, they do not have to have an
intrinsic meaning.

---

From: James M Snell <[email-masked]>
Date: Fri, 1 Feb 2013 09:22:38 -0800

Based on the feedback, we can change this to...

+------------------------------+
|S|len(method)|method|len(host)|
+-+----+----+-+-------+--------+
| host |port|len(path)|  path  |
+------------------------------+

The only change here really is the port field as a uvarint. The path would
contain the full query-string and path detail...

Example: GET https://example.com:443/foo?a=b

  [83,G,E,T,0B,e,x,a,m,p,l,e,.,c,o,m,BB,03,08,/,f,o,o,?,a,=,b]

If some other scheme needs to be specified, a separate :scheme header would
be specified identifying the scheme.

Method names are encoded as text because snumeric identifier or
abbreviation schemes are just going to add complexity for no demonstrated
benefit.

- James

On Thu, Jan 31, 2013 at 11:09 PM, James M Snell <[email-masked]> wrote:

> We can combine the :scheme, :method, :host and :path header fields into a
> single :req Header with a compact binary encoding and require that this
> Examples:
> GET request for http://example.net/foo would be:
>
>   [03,G,E,T,0B,e,x,a,m,p,l,e,.,c,o,m,04,/,f,o,o]
>
> POST request for https://example.net/foo would be:
>
>   [84,P,O,S,T,0B,e,x,a,m,p,l,e,.,c,o,m,04,/,f,o,o]
>
> Unregistered method 'FOO' request for https://example.net/foo would be:
>
>   [83,F,O,O,0B,e,x,a,m,p,l,e,.,c,o,m,04,/,f,o,o]
>
> If the client chooses not to send a host (because it's not needed or
> whatnot) they simply set len(host) to 0...
>
>   [03,G,E,T,00,04,/,f,o,o]
>
> If we do end up going with delta encoding for compression, we can require
> that the :req Header always be passed using the eref operation (ephemeral
> reference, meaning that the header is never stored in the compression
> state). No huffman-coding would be applied to the header, making it very
> quick and cheap to process.
>
> Thoughts?
>
> - James
>
>
>
> On Thu, Jan 31, 2013 at 8:04 AM, Ted Hardie <[email-masked]> wrote:
>
>> On Thu, Jan 31, 2013 at 5:15 AM, Roy T. Fielding about this
>>
>> >> We had no idea how early we were in the popularity curve of HTTP or
>> >> how dominant it would become, but it was clear even then that the
>> >> protocol would be very, very common on the network.  In retrospect, it
>> >> is clear that we shouldn't have looked at the current installed
>> >> base--we should have looked at what we expected eventual use would be.
>> >> That makes "the earlier the better" clear.
>> >
>> > I think your memory is a bit hazy there ... HTTP passed all
>> > application protocols other than email in 1995, and by that
>> > time (Mar 1996) was roughly double email traffic, IIRC (this was
>> > before email-based spam became common).  That's why the WG
>> > meeting contained a lot of people who had nothing to do
>> > with developing the Web protocols---there was panic in the air.
>> >
>>
>> I think we can both agree it only got more popular from there.  It may
>> well have surpassed email by that discussion, but the hockey stick had
>> a lot of run to go.
>>
>> > I find it amusing that you think we could have proceeded in any
>> > other way without relegating the IETF work to the garbage bin.
>> >
>>
>> It would have taken agreement from a lot of people, but the web
>> community of the time could have decided then to rev the major
>> protocol version for that change.  That suggestion was made.  The IETF
>> could not have mandated that.  The IETF *never* gets to dictate
>> protocol changes--it didn't have police then and it doesn't now, but
>> saying that the buy-in for that change didn't happen doesn't equate to
>> saying it could not.
>>
>> regards,
>>
>> Ted
>> >> For HTTP 2.0, where we can make non-backward compatible changes, I
>> >> personally think the right thing to do is to drop the Host: header
>> >> (that version shift is what we were waiting for 17 years ago, after
>> >> all).  If there are things folks are getting from side-effects of the
>> >> Host header (e.g. proxy targeting), we put them into the bin of
>> >> potential requirements for HTTP 2.0 *and create mechanisms to meet
>> >> those needs*.
>> >>
>> >> I think Adrien's proposal for extensions to the host header makes
>> >> clear that the need isn't perfectly met by the host header in any
>> >> case, so mapping out the real aim and meeting that seems like the best
>> >> notion to me.
>> >
>> > Oddly enough, waka separates the scheme+host routing information from
>> > the rest of the URI because that works better with multi-argument
>> > methods and message-based encryption. *shrug*
>> >
>> > ....Roy
>> >
>>
>>
>

---

From: Roland Zink <[email-masked]>
Date: Fri, 01 Feb 2013 18:26:15 +0100

> what, the same might be useful in the Internet).

The scheme which comes to my mind is ftp. As far as I know this is 
supported by browsers and there are proxies translating between HTTP and 
FTP.

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 2 Feb 2013 09:17:40 +1300

We see ftp:// all the time

As for using a character or 2 for the method. Why? What is wrong with numbers?

1 = GET
2 = HEAD
3 = POST
etc. 

Sent from my iPad

On 2/02/2013, at 6:26, Roland Zink <[email-masked]> wrote:

> The scheme which comes to my mind is ftp. As far as I know this is supported by browsers and there are proxies translating between HTTP and FTP.
> 
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 01 Feb 2013 21:28:58 +0100

On 2013-02-01 21:17, Adrien de Croy wrote:
> ...

Experimenting with new method names becomes really really hard.

Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 2 Feb 2013 09:32:01 +1300

Not really. 

Just reserve a number to indicate the method name is following in text

Or reserve a range for private use

Could even largely avoid sending method at all.  If it were treated like a field, default to GET you would only need to send if it changed

Sent from my iPad

---

From: James M Snell <[email-masked]>
Date: Fri, 1 Feb 2013 13:10:00 -0800

What does this buy us? Two or three bytes typically? Not worth it. Just
send the method name.

method just isn't a high priority for optimization.
On Feb 1, 2013 12:33 PM, "Adrien de Croy" <[email-masked]> wrote:

> Could even largely avoid sending method at all.  If it were treated like a
> field, default to GET you would only need to send if it changed
>
>
> Sent from my iPad
>

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 2 Feb 2013 12:53:37 +1300

It buys us also the ability to efficiently decipher the method. Currently we have to convert the string method name to an enum anyway. Why not just start out that way.

All the byte savings add up.  I think we should consider all the parts of the request line incl URI parts to be simply fields in the request, and be able to only send new values when they change.

Default values could be used for various parts as well such as scheme and method.

You never know how significant 2 or 3 bytes may be. It could be the difference between needing to split the buffer over more than one packet or not. 

Sent from my iPad

On 2/02/2013, at 10:10, James M Snell <[email-masked]> wrote:

> What does this buy us? Two or three bytes typically? Not worth it. Just send the method name.
>

---

From: Roberto Peon <[email-masked]>
Date: Sat, 2 Feb 2013 20:53:03 +0900

I'll point out that the delta compressor already deals these encoding
problems by using a static dictionary (i.e. O(1) state for program
execution) for key-values, some of which are certainly ":method", "GET",
etc...

Thus the various methods get assigned numeric IDs, and if one isn't
familiar, then you can either represent it as text every time, or if the
receiver allows for state storage, it can allocate a new numeric ID after
seeing the method once.

The only thing that isn't "solved" right now is putting the scheme, host,
and path components in whatever combination, at the beginning, mainly
because it makes the state machine for a(ny) compressor significantly more
annoying to manage as it can require that entries in the compression
context get expired.

If we're worried about the bit-width of these things, we can make some of
these "special" and only use a byte to index 'em (thusfar the static
dictionary has ~64 items, so this should be quite safe). Personally, I hate
that idea, and think that 2-bytes to encode this is perfectly fine. :)

-=R

On Sat, Feb 2, 2013 at 8:53 AM, Adrien de Croy <[email-masked]> wrote:

> It buys us also the ability to efficiently decipher the method. Currently
> we have to convert the string method name to an enum anyway. Why not just
> start out that way.
>
> All the byte savings add up.  I think we should consider all the parts of
> the request line incl URI parts to be simply fields in the request, and be
> able to only send new values when they change.
>
> You never know how significant 2 or 3 bytes may be. It could be the
> difference between needing to split the buffer over more than one packet or
> not.
>
>
>
> Sent from my iPad
>
>> Could even largely avoid sending method at all.  If it were treated like
>> a field, default to GET you would only need to send if it changed
>>
>>
>> Sent from my iPad
>>

---

From: Mark Baker <[email-masked]>
Date: Sat, 2 Feb 2013 09:34:52 -0500

> schemes do we need to support?

All of them, potentially. Anomalies aside (thanks, DAV :), HTTP
methods are uniform so apply to all resources independent of the
scheme used by any particular identifying URI.

"ftp" is the only one I've seen used in practice though, as others
have pointed out.

$ nc ftp.debian.org 80
GET ftp://ftp.debian.org/debian/README HTTP/1.0

Mark.

---

From: Eliezer Croitoru <[email-masked]>
Date: Sun, 03 Feb 2013 21:47:00 +0200

On 2/2/2013 1:53 PM, Roberto Peon wrote:
> receiver allows for state storage, it can allocate a new numeric ID
> after seeing the method once.
>
> The only thing that isn't "solved" right now is putting the scheme,
> host, and path components in whatever combination, at the beginning,
> mainly because it makes the state machine for a(ny) compressor
> significantly more annoying to manage as it can require that entries in
> the compression context get expired.
>
> If we're worried about the bit-width of these things, we can make some
> of these "special" and only use a byte to index 'em (thusfar the static
> dictionary has ~64 items, so this should be quite safe). Personally, I
> hate that idea, and think that 2-bytes to encode this is perfectly fine. :)
>
> -=R
>

I was just wondering about what is the proposal for now?

-- 
Eliezer Croitoru
http://www1.ngtech.co.il

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 5 Feb 2013 16:16:47 +1100

Thanks for making concrete proposals, James -- that's helpful.

We had a brief conversation at the F2F about requiring "special" headers (e.g., :scheme :method :host :path) to be at the beginning of the set of headers.

That's effectively a different serialisation of the information here (ignoring the separation of the port). Each approach has advantages and disadvantages, but what might help us move forward here is first figuring out *what* information needs to be separated out, before we talk about the specific format of the bits on the wire.

A few points to consider (trying to move the conversation forward, more than stating a position):

* HTTP/1.1 has two ways of serialising what we call the Effective Request URI in HTTPbis, and I don't think it's too controversial to say that this is bad, and in /2 we should just have one way to do it.

* One of the HTTP/1.1 forms omits the scheme in use. Discussion so far seems to imply that people want the scheme to be explicit in /2. Anyone have any argument as to why not?

* If we do make the scheme explicit, I'd note that HTTPbis allows use of schemes other than HTTP / HTTPS, so we'd need to accommodate that. I.e., a single bit is out.

* Most people seem to see the value in separating the authority portion of the URI into a separate header, because that's routed upon (and it could also benefit from delta-based compression). Anyone disagree?

* Separating the query string from the path would save the origin server a bit of parsing. I see arguments on both sides; who wants to make them?

* Request routing is generally done on the host/port tuple; i.e., the port doesn't have informational value *in the HTTP message* when it's separate from the port. So, I'm not sure about the value proposition of separating it out here; can you illustrate?

* We'll need to do all of this for the response status code as well. Maybe not the phrase; we touched on this briefly at the F2F, and I put forth the opinion that since it's human-readable, and our message format isn't really any more, it doesn't have much utility to actually include in the message. Anyone think it's useful enough to justify the bits?

* We also talked about :version at the F2F, both in requests and responses. I don't think it's necessary, as it's effectively hop-by-hop information, and the connection negotiation + magic takes care of that. Discuss.

Cheers,

On 02/02/2013, at 4:22 AM, James M Snell <[email-masked]> wrote:

> The only change here really is the port field as a uvarint. The path would contain the full query-string and path detail...
> 
> If some other scheme needs to be specified, a separate :scheme header would be specified identifying the scheme. 
> 
> Method names are encoded as text because snumeric identifier or abbreviation schemes are just going to add complexity for no demonstrated benefit.
> 
> - James
> 
> 
> 
> We can combine the :scheme, :method, :host and :path header fields into a single :req Header with a compact binary encoding and require that this single header always appear first in request header blocks.
> 
> If the client chooses not to send a host (because it's not needed or whatnot) they simply set len(host) to 0...
> 
>   [03,G,E,T,00,04,/,f,o,o]
> 
> If we do end up going with delta encoding for compression, we can require that the :req Header always be passed using the eref operation (ephemeral reference, meaning that the header is never stored in the compression state). No huffman-coding would be applied to the header, making it very quick and cheap to process.
> 
> Thoughts?
> 
> - James
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Mon, 4 Feb 2013 21:34:40 -0800

> Thanks for making concrete proposals, James -- that's helpful.
No problem.. mainly just wanted to get something concrete to kick around...

> [snip]
>
> * HTTP/1.1 has two ways of serialising what we call the Effective Request
> URI in HTTPbis, and I don't think it's too controversial to say that this
> is bad, and in /2 we should just have one way to do it.
>
> * One of the HTTP/1.1 forms omits the scheme in use. Discussion so far
> seems to imply that people want the scheme to be explicit in /2. Anyone
> have any argument as to why not?
>

So long as we're able to minimize any redundancy caused by making this
explicit (e.g. delta) I have no problem with this. If we can identify the
most commonly used schemes (i'm thinking http, https and ftp really) and
include those in the default delta dictionary, then I think we're fine. It
becomes nothing more than just a toggl operation.

>
> * If we do make the scheme explicit, I'd note that HTTPbis allows use of
> schemes other than HTTP / HTTPS, so we'd need to accommodate that. I.e., a
> single bit is out.
>
> * Most people seem to see the value in separating the authority portion of
> the URI into a separate header, because that's routed upon (and it could
> also benefit from delta-based compression). Anyone disagree?
>
>
Definitely no disagreement from me on this one.

> * Separating the query string from the path would save the origin server a
> bit of parsing. I see arguments on both sides; who wants to make them?
>
>
Mixed feelings on this. Part of me says it's ok, another part is screaming
at me No! Applications on the server seem to be all over the map when it
comes to parsing and handling of request URIs. I guess I would have to just
see what the impact of separating these would be and weigh those against
whatever the justification is for splitting them.

> * Request routing is generally done on the host/port tuple; i.e., the port
> doesn't have informational value *in the HTTP message* when it's separate
> from the port. So, I'm not sure about the value proposition of separating
> it out here; can you illustrate?
>

I split them out separately in my strawman solely to provide a more compact
encoding (encoding the port as a uvarint rather than a string of ascii
symbols).

>
> * We'll need to do all of this for the response status code as well. Maybe
> not the phrase; we touched on this briefly at the F2F, and I put forth the
> opinion that since it's human-readable, and our message format isn't really
> any more, it doesn't have much utility to actually include in the message.
> Anyone think it's useful enough to justify the bits?
>
> * We also talked about :version at the F2F, both in requests and
> responses. I don't think it's necessary, as it's effectively hop-by-hop
> information, and the connection negotiation + magic takes care of that.
> Discuss.
>
>
I'm all for dumping status-text and :version headers. The more redundant
useless data we can dump from headers the better.

- James

> > The only change here really is the port field as a uvarint. The path
> would contain the full query-string and path detail...
> >
> > If some other scheme needs to be specified, a separate :scheme header
> would be specified identifying the scheme.
> >
> > If we do end up going with delta encoding for compression, we can
> require that the :req Header always be passed using the eref operation
> (ephemeral reference, meaning that the header is never stored in the
> compression state). No huffman-coding would be applied to the header,
> making it very quick and cheap to process.
> >
> > Thoughts?
> >
> > - James
> >
> >
> >

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 05 Feb 2013 08:57:37 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>* Most people seem to see the value in separating the authority portion
>of the URI into a separate header, because that's routed upon (and it
>could also benefit from delta-based compression). Anyone disagree?

As long as there is only one place I need to look for it, and as long
as finding that place, preferably in the first 1400-ish bytes, is cheap,
I'm happy.

As for separating out the query string, there may be sound security
in doing so, if we thereby can increase privacy for the query-string.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 5 Feb 2013 05:46:12 -0500

I think that you have most of it right.

> bit of parsing. I see arguments on both sides; who wants to make them?

I tend to prefer the approach that says that path and query are just
opaque strings.  Splitting them would add another bucket for
query-stuff to go into, which makes it harder all 'round to do
matching and all the other sorts of processing that doesn't rely on
breaking the URI down into constituent parts.  Besides, the breaking
down thing is done very differently by different software, so it would
be presumptuous to stipulate just one method.

> * We also talked about :version at the F2F, both in requests and responses.
> I don't think it's necessary, as it's effectively hop-by-hop information,
> and the connection negotiation + magic takes care of that. Discuss.

Version and phrase are - at best - byte-wasters.  Of course, they both
have great untapped potential for harm, if that is your goal.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 5 Feb 2013 05:44:40 -0500

I tend to prefer the approach that says that path and query are just
opaque strings.  Splitting them would add another bucket for
query-stuff to go into, which makes it harder all 'round to do
matching and all the other sorts of processing that doesn't rely on
breaking the URI down into constituent parts.  Besides, the breaking
down thing is done very differently by different software, it would be
presumptuous to stipulate just one method.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 05 Feb 2013 14:27:07 +0100

* Mark Nottingham wrote:
>* Separating the query string from the path would save the origin server 
>a bit of parsing. I see arguments on both sides; who wants to make them?

Whether that is more, or less, work for the origin server depends quite
a bit on the origin server. For instance, the origin server may have to
concatenate the two prior to parsing because it can parse it only as one
single string (as if often the case with simplistic ad-hoc parsers). I
also see some potential for bugs and security vulnerabilities there, say
how to handle a resource identifier with two question marks (a component
might treat anything after the first question mark as query string, some
middle component puts the parts back together, and then a legacy compo-
nent only uses the part after the final question mark as query string).

>* We'll need to do all of this for the response status code as well. 
>Maybe not the phrase; we touched on this briefly at the F2F, and I put 
>forth the opinion that since it's human-readable, and our message format 
>isn't really any more, it doesn't have much utility to actually include 
>in the message. Anyone think it's useful enough to justify the bits?

I don't think they are useful on the wire, but standard names are quite
useful in discussions and searches using your favourite search engine,
so definitions for new codes, even when they are "HTTP/2-only", if there
will be such codes at all, should still provide them.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Zhong Yu <[email-masked]>
Date: Tue, 5 Feb 2013 11:25:58 -0600

> * Separating the query string from the path would save the origin server a bit of parsing. I see arguments on both sides; who wants to make them?

The saving would be minuscule in the best case.

The server must validate them anyway, you can't trust a client won't
insert a "?" in the supposed path component.

With a single request uri, one validation process is enough; with
separated path and query, we need 2 validation processes, slightly
different, which can't be good for performance.

Zhong Yu

---

From: Roberto Peon <[email-masked]>
Date: Tue, 5 Feb 2013 13:00:18 -0800

On Mon, Feb 4, 2013 at 9:34 PM, James M Snell <[email-masked]> wrote:

>> of the URI into a separate header, because that's routed upon (and it could

I'm of two minds about this one as well. On one hand, it is easier for
parsing. On the other hand, I haven't done any analysis to show or disprove
that this will be better for bytes on the wire, so, while I understand some
of the cost, I don't understand all of the benfit..

-=R

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 06 Feb 2013 22:44:09 +0000

------ Original Message ------
From: "Mark Nottingham" <[email-masked]>
To: "James M Snell" <[email-masked]>
Cc: "HTTP Working Group" <[email-masked]>
Sent: 5/02/2013 6:16:47 p.m.
Subject: Request Routing Information [was: Do we kill the "Host:" header 
in HTTP/2 ?]
>We had a brief conversation at the F2F about requiring "special" 
>headers (e.g., :scheme :method :host :path) to be at the beginning of 
>the set of headers.
>
>That's effectively a different serialisation of the information here 
>(ignoring the separation of the port). Each approach has advantages and 
>disadvantages, but what might help us move forward here is first 
>figuring out *what* information needs to be separated out, before we 
>talk about the specific format of the bits on the wire.
>
>* HTTP/1.1 has two ways of serialising what we call the Effective 
>Request URI in HTTPbis, and I don't think it's too controversial to say 
>that this is bad, and in /2 we should just have one way to do it.

fine with that as long as it's clear whether a client is talking 
proxy-ese or not.  Esp if you consider intercepted connections may be in 
the mix.  In fact explicit support for clients / servers / proxies to 
know the connection is intercepted would be good.

>have any argument as to why not?
Apart from proxy requests, the scheme has always been http.  You can 
only make a ftp:// request to a proxy.  https:// was never used.  So 
there was only http when talking directly to a server.  Personally 
though I would propose putting the scheme in always, to enable things 
like semantic equivalent of GET https://some-secure-site.com/whatever

>
>* If we do make the scheme explicit, I'd note that HTTPbis allows use 
>of schemes other than HTTP / HTTPS, so we'd need to accommodate that. 
>I.e., a single bit is out.
>
nope, I'm in favour of that.  I would also split out port.

>
>* Separating the query string from the path would save the origin 
>server a bit of parsing. I see arguments on both sides; who wants to 
>make them?
I would be in favour.  Lots of sites (e.g. sites running on a CMS 
without mod_rewrite) are all calls to index.php with the only thing 
changing being the query string.  So splitting them out would enable us 
to save re-transmitting the path if it didn't change.

>
>* Request routing is generally done on the host/port tuple; i.e., the 
>port doesn't have informational value *in the HTTP message* when it's 
>separate from the port. So, I'm not sure about the value proposition of 
>separating it out here; can you illustrate?
proxies always need to parse this.  Is the proposal that we'd still need 
to string parse server:port, or would there be some binary encoding of 
server and port parts?

I am assuming we plan to adopt default values for various fields, which 
therefore never need to be transmitted unless the value differs from the 
default.

E.g.

default values for

scheme = http
method = GET
port = 80

there could also be a case to allow a client to push fields to a peer 
without there being a request.  for instance the browser on loading 
could push things like user-agent, accept-encoding etc to a proxy before 
issuing the first request.  Basically allow discovery and advertisement 
of support for protocol functions at startup.
>forth the opinion that since it's human-readable, and our message 
>format isn't really any more, it doesn't have much utility to actually 
>include in the message. Anyone think it's useful enough to justify the 
>bits?
could be optional.  You can still see the text in packet captures, and 
logs so it can still be useful.

>information, and the connection negotiation + magic takes care of that. 

May be more useful to have a field to indicate the version that the 
request came from.  E.g. if a HTTP/1.1 request was made to a proxy which 
up-graded it to a 2.0 server, the server may need the original client 
version to decide what functionality to enable.

Via doesn't quite cut this, since it omits the original version.

Again, it could default to 2.0, and only be present if it's different.

Adrien

>>abbreviation schemes are just going to add complexity for no 
>>demonstrated benefit.
>>
>>  - James
>>
>>
>>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 14:53:44 +1300

On 7/02/2013 11:44 a.m., Adrien W. de Croy wrote:
> Subject: Request Routing Information [was: Do we kill the "Host:" 
> header in HTTP/2 ?]
>> (ignoring the separation of the port). Each approach has advantages 
>> and disadvantages, but what might help us move forward here is first 
>> Request URI in HTTPbis, and I don't think it's too controversial to 
>> say that this is bad, and in /2 we should just have one way to do it.
>
> proxy-ese or not.  Esp if you consider intercepted connections may be 
> in the mix.  In fact explicit support for clients / servers / proxies 
> to know the connection is intercepted would be good.
>
>>
>> * One of the HTTP/1.1 forms omits the scheme in use. Discussion so 
>> far seems to imply that people want the scheme to be explicit in /2. 
>> Anyone have any argument as to why not?

Exactly. If we are to multiplex HTTP, HTTPS, WebSockts and possibly 
other future protcols using the HTTP frame base format. Which seems like 
a reasonable ability. Then we are going to need to specify those schemes 
in the URL, to indicate for example what protocol frame format the 
client expects used on the response. https:// to indicate encrypted 
payload is just the current testable variant on http://.

In fact HTTP/2 frames with encrypted payload opens us to that wonderful 
world of shttp:// scheme. With all the security available in https:// 
but routing flexibility and proxy handling available in http://

>> * Most people seem to see the value in separating the authority 
>> portion of the URI into a separate header, because that's routed upon 
>> (and it could also benefit from delta-based compression). Anyone 
>> disagree?
> changing being the query string.  So splitting them out would enable 
> us to save re-transmitting the path if it didn't change.

That would be a problem for the origin server caused by itself. Each of 
these URLs is a different object, meaning you would have to re-combine 
the two fields to perform things like cache lookups and validation.
Splitting them only benefits when compressing without prefix matching 
the field against earlier versions of itself - which if I am 
understanding CRME right is not a vulnerability (the problem being 
prefix matching field A with field B when one has secure data and the 
other is attacker modifiable).

Client, middlware, and routing infrastructure do not need to care about 
the path+query portion for their operations other than as an opaque 
blob. Splitting them just makes two opaque blobs - useless waste of 
processing one length-skip operation. And adds the '?' embedding 
vulnerabiities mentioned earlier. Not to mention some schemes like urn: 
where '?' does *not* signal a query string.

NP: I do know of situations where the proxies "need" to split path and 
query string. But, all of those cases are where it is performing 
origin-server operations on the request. Which just supports the idea of 
leaving these two split up to the origin server.

>> separate from the port. So, I'm not sure about the value proposition 
>> of separating it out here; can you illustrate?
> proxies always need to parse this.  Is the proposal that we'd still 
> need to string parse server:port, or would there be some binary 
> encoding of server and port parts?
>
> I am assuming we plan to adopt default values for various fields, 
> which therefore never need to be transmitted unless the value differs 
> from the default.
>
> E.g.
>
> default values for
>
> scheme = http
> method = GET
> port = 80
>
> could push things like user-agent, accept-encoding etc to a proxy 
> before issuing the first request.  Basically allow discovery and 
> advertisement of support for protocol functions at startup.

Assuming that we can agree on suitable enumerations for these details, 
they can be fixed-length field and there is no extra cost of them being 
always present, even when 'default'. Defaults become a property inferred 
by the clients, not by any middleware or servers - which themselves may 
not be coded up to handle the hard-coded future scheme foo:// default 
port assignment.

>> Maybe not the phrase; we touched on this briefly at the F2F, and I 
>> put forth the opinion that since it's human-readable, and our message 
>> format isn't really any more, it doesn't have much utility to 
>> actually include in the message. Anyone think it's useful enough to 
>> justify the bits?

Yes +1 on optional phrase. It can be a header as easily as anything else 
and saves a lot of bytes by sending just status code even before appying 
a compact binary format on status.

>> responses. I don't think it's necessary, as it's effectively 
>> hop-by-hop information, and the connection negotiation + magic takes 
>> care of that. Discuss.
>
> request came from.  E.g. if a HTTP/1.1 request was made to a proxy 
> which up-graded it to a 2.0 server, the server may need the original 
> client version to decide what functionality to enable.
>

What about an upgraded version of Via which:
  * omits the () comments portion
  * only added/altered *if* the version has changed since the last hop.

+ backward compatible with HTTP/1.1 for 1.1<->2.0 gateway which 
permitted any hop to 'compact' multiple entries if they shared a server 
name and treated it as optional on client->origin traffic.
+ caters to user security better by only giving indication of protocol 
gateways, nothing deterministic about the end-user or end-server.

Amos

---

From: Nicolas Mailhot <[email-masked]>
Date: Fri, 22 Feb 2013 18:20:12 +0000 (UTC)

Nico Williams <nico@...> writes:

> 
> On Wed, Jan 30, 2013 at 3:31 PM, Adrien W. de Croy <adrien@...> wrote:
> > from a proxy POV, it's very useful, nay vital that we can tell the
> > difference between a request that a client thinks it is sending to a proxy,
> > vs a request the client thinks it is sending to a server.
> >
> > [...]
> > In fact for authentication, I would extend it to allow for the definition of
> > the target of the auth.  If you have a request going through a chain, and
> > several links require auth from the client, there's currently no way to do
> > it safely.
> 
> There are separate headers for authentication to proxies.

The current separate headers for authentication to proxies are a miserable
failure which is impossible to use in all but the simplest cases:

1. they only work with a single intermediary. Want to chain networks with
authentification gateways?  No can do. Someone forgot that the Internet was
built by interconnecting distinct networks, and that those networks are not
freely accessible except for entertainment junkies at home.

2. they are insecure: you can't send them securely to the intermediary without
piggy-backing on the main transport crypto. Want to authenticate to the proxy to
access an http site ? Best hope every link between the user and the intermediary
is secure, because everyone will see your credentials. Want to access an https
web site? Need MITM by the intermediary to be able to read your damn proxy
credentials

3. since web clients do not really have a way to send them to the right network
node at the right time many just spread-fire proxy auth on every request in the
hope they'll get picked up whenever necessary. Including when they are not using
a proxified link (so it works all the time, at the expense of giving proxy
secrets to every random web site).

HTTP1 proxy auth is too broken to be rescued. It should be taken in the backyard
and quietly shot in the head. I do hope HTTP2 will make things better.

(HTTP 511 is a bit better, but relies on js support in the web clients)

-- 
Nicolas Mailhot

---

From: Nicolas Mailhot <[email-masked]>
Date: Fri, 22 Feb 2013 18:49:12 +0000 (UTC)

Amos Jeffries <squid3@...> writes:

> blob. 

Unfortunately not true. We had cases where misbehaving users (that *knew* they
were misbehaving) changed dynamically the name of the accessed host, and the
only way to stop the damage was a path match (which fortunately was
discriminating).

And a lot of botnet attacks can be identified by the access to a special path,
which is the same on all infected servers users access to.

In all those cases the query portion is just garbage to be ignored, the path –
not.

'Do not need to care' is another word for 'no creative users'

--  
Nicolas Mailhot

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 23 Feb 2013 23:52:08 +1300

> freely accessible except for entertainment junkies at home.

That someone seems to be you...

Case 1) End-user logging into an origin HTTP server.
    --> www-authenticate header

Case 2) End user logging into their ISP proxy
   -> proxy-authenticate header from end-user to ISP proxy.

Case 3) ISP proxy logging into their upstream ISP proxy
   -> proxy-authenticate header from ISP proxy to upstream.

No I am not forgetting interception proxies. They are forbidden from 
dropping or performing authentication due to the credentials they 
receive *always* being destined to some upstream location.

*Never* does end-user need to explicitly log into the Tier-1 proxy farm 
directly, nor does middleware of any type need to play with the origin 
server account credentials. Proxy-authenticate is hop-by-hop header and 
www-authenticate is end-to-end for these reasons and separation is both 
required for correct operation, security and works *perfectly well* for 
all of these purposes.

If you were to propose a replacement architecture where these 
credentials were shareable to the security WG they would turn around and 
require you to re-implement this separation of scope in some form or 
another. Look at what happend to the OAuth 2.0 efforts to unify user 
credentials into a single item accepted around the world in all OAuth 
devices. We now have a multitude of alternative token types (Bearer, 
MAC, ...) split over several relay locations (headers, URLs, many entity 
formats, Cookies) multiplied by the separation of credentials from 
services, controllers, assignation verifiers. End result: a massive mess 
of confusion about interoperability, complexity and which portion of the 
system is using up-to-date token values when two parts contradict. 
Compare the thousands of lines of OAuth text needed to describe their 
"simple" system to the above HTTP architecture description in three cases.

> is secure, because everyone will see your credentials.

If your public credential Proxy-Auth token is subject to decryption or 
replay there are worse problems you face than others being aware of what 
your token looks like. Use a better security token type is the 
counter-argument for this.

>   Want to access an https

This is what the 'S' in HTTPS is *for* and *why* it is on the end. 
Everybody on the link could see what URL you are requesting as well if 
the HTTP portion was unencrypted. The URL is just as much sensitive data 
as the user credentials in the traffic HTTPS was intended for.
If the HTTP transport/transfer details were not security critical we 
would be talking about shttp:// . Which it appears nobody in the browser 
arena is willing to implement despite years of lobbying by middleware 
authors.

Oh and the CONNECT wrapper headers is where the Proxy-Auth credentials 
are supposed to be sent on proxied HTTPS traffic.

> secrets to every random web site).

You mean clients are not aware of which service they are talking to at 
the other end of any TCP connection? that is plain wrong.

Notice how I wrote "service" above instead of "node". One node can be 
multiple services, or one service on multiple nodes.

There is no need for HTTP proxy credentials to be sent to a TCP router 
node for example. If login to routers were required to use them TCP 
would contain an authentication model.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 23 Feb 2013 23:50:40 +1300

> only way to stop the damage was a path match (which fortunately was

Please explain in more detail. How did they dynamically change the 
accessed host?

And why did your HTTP middleware allow the change?

If you are talking about fiddling the Host: header versus absolute-URL 
versus TCP destination and similar vulnerabilities in the middleware 
there is no excuse for it being vulnerable. I point you at Squid-3.2 and 
the way we prohibit Host and TCP address differing - to the point or 
marking players like Google and Akamai regularly as "forgers".

> which is the same on all infected servers users access to.

You seem to be misunderstanding the meaning of "opaque". It has nothing 
to do with obscuring anything.

Botnet requests with a consistent ETag prefix for example would be 
equally detectable and preventable using also the same method: a pattern 
match against the relevant field.

I posit that what you are doing there is that _you_ (the human) are 
reading the blob following a URL hostname, _you_ are understanding it, 
and writing a tool that detects a pattern in that field-value. The tool 
itself only needs to determine if the field as a whole matches the 
pattern you gave it.
  If those same botnets were sending urn: with hostname and a path 
segment you would just as easily identify the pattern and have tools 
matching it - even though the "path" segment of URN is an opaque blob 
everywhere except the origin.

There are cases where middleware does need to manipulate the path. But 
these are also the cases where you would be parsing it completely 
anyway, to gain full understanding of all the pieces inside it right 
down to the byte level. That would always be done with a parser which 
re-assembled the pieces and assigned specific meaning to each byte - 
including the query portion.

> In all those cases the query portion is just garbage to be ignored, the path –

This tells me you have not encountered (or noticed) the Spam attacks 
involving query-injection last decade. Lucky you.

/history/
Now dead versions of Outlook used to make magic hyperlinks links on any 
http:// text it detected in plain text by hiding all the text it decided 
was URL and showing only the domain name. The attacker would carefully 
craft login credentials containing encoded @, / and ? in ways which 
outlook would mistake as delimiters but the browser would decode before 
parsing the URL. User and anyone not clued up enough to notice would see 
as a link to a victum website, example.com (or in some cases localhost!) 
which when clicked would go completely sidways to a Phishing or viral 
infected URL on a host somewhere else.

> 'Do not need to care' is another word for 'no creative users'

No. 'Do not need to care' is another word for 'I already have a better 
way to detect those creative users'.

In particular, As has been mentioned already. Splitting those two fields 
will simply give those creative users another tool to play with while 
making the middleware do more work to prevent them using it, um, creatively.

Amos