Subject: SPDY Header Frames

From: James M Snell <[email-masked]>
Date: Fri, 13 Jul 2012 15:16:35 -0700

This note is intended to provide some additional thoughts for discussion
around the design and use of SPDY as the possible basis for HTTP/2.0. The
intent is to provide fuel for discussion... comments are definitely welcome.

As discussed within draft-tarreau-httpbis-network-friendly-00, and as has
been mentioned several times in discussion on list, handling of headers
within the current SPDY framing, and in particular the layering of HTTP/1.1
messages into SPDY frames is less than optimal. There is significant wasted
space, duplication, etc that -- strictly speaking -- really isn't
necessary. While I recognize that the following increases the basic
complexity of the protocol, it allows fairly significant optimization
following the same basic lines of reasoning expressed in
draft-tarreau-httpbis-network-friendly-00.

Section 2.6.1 of the SPDY draft defines header blocks using the following
format:

   +------------------------------------+
   | Number of Name/Value pairs (int32) |
   +------------------------------------+
   |     Length of name (int32)         |
   +------------------------------------+
   |           Name (string)            |
   +------------------------------------+
   |     Length of value  (int32)       |
   +------------------------------------+
   |          Value   (string)          |
   +------------------------------------+
   |           (repeats)                |

This structure is used within SYN_STREAM and HEADERS frames.

What I propose is the following revised structure:

   +------------------------------------+
   |     Number of Headers (int32)      |
   +------------------------------------+
   |T| Flags (7) |     Length (24)      |
   +------------------------------------+
   |              Data                  |
   +------------------------------------+
   |T| Flags (7) |     Length (24)      |
   +-------------------------------------
   |              Data                  |
   +-------------------------------------
   |             (repeats)              |

T is a single bit identifying the Header Type. There are two types..
REGISTERED (0) and EXTENSION (1)

Flags provides flags for the specific header field. The flag 0x1 indicates
that the header value contains Character Data. If not set, the value is
assumed to consist of raw octets. 0x2 indicates that the value is
compressed.

Length is an unsigned 24-bit value specifying the number of octets after
the length field.

When the T bit is NOT set, the Header field is a REGISTERED Header, the
structure of which is:

   +------------------------------------+
   |0| Flags (7) |     Length (24)      |
   +------------------------------------+
   | ID | Value Length (int32) |Value...|
   +------------------------------------+

The ID is a 32-bit number uniquely identifying the registered field. Each
is assigned by the registrar. For instance, the "Host" field could have a
registered value of "1", the "Accept-Lang" field could have a registered
value of "6", and so forth.

The Value Length is a 32-bit value indicating the length of the value.

If Flag 0x1 is set, the value is assumed to contain character data. When
set, the value MUST be preceded by a single unsigned 8-bit integer
identifying the character encoding utilized. The values are assigned by the
registrar. For instance, US-ASCII could have a registered value of "1",
while "UTF-8" could have a registered value of "2".

For example:

   +------------------------------------+
   |0| 0000001 |     24                 |
   +------------------------------------+
   | 1 | 16 | 1 |    www.example.org    |
   +------------------------------------+

This Header record indicates a REGISTERED header containing character
content, the header ID = 1, the charset used is US-ASCII and the value is "
www.example.org". The header is expressed with a total of 28 bytes.

When the T bit IS set, the Header field is an EXTENSION Header, the
structure of which is:

   +------------------------------------+
   |0| Flags (7) |     Length (24)      |
   +------------------------------------+
   |      Length of name (int32)        |
   +------------------------------------+
   |           Name (string)            |
   +------------------------------------+
   |      Length of value (int32)       |
   +------------------------------------+
   |              Value                 |
   +------------------------------------+

For example.. an extension header that contains raw binary data...

   +------------------------------------+
   |0| 0000000 |       Length (24)      |
   +------------------------------------+
   |                5                   |
   +------------------------------------+
   |              x-foo                 |
   +------------------------------------+
   |                4                   |
   +------------------------------------+
   |           {raw bytes}              |
   +------------------------------------+

The header is expressed with a total of 21 bytes.

The same flags apply. 0x1 indicates that the value is character data. If
0x1 is not set, the value contains raw octets. The key difference is that
there is a 32-bit name length and variable length name field in place of
the 32-bit ID field in the REGISTERED header. All other details remain the
same.

As is currently the case in SPDY, if a single header value contains
multiple values, each can be separated using a single NUL (0) byte.

There are several advantages to this approach:

1. Commonly used header names are omitted in favor of registered, known
numeric IDs, saving space and making it more efficient to scan over
commonly used headers. For instance, intermediaries that route requests
based on common headers such as Host etc could choose to ignore EXTENSION
header fields entirely, and scan only for the ID's of the fields they are
interested in, rather than having to parse the entire bag of header names.

2. Header values can be expressed as raw octets or character data.
Currently, mechanisms within HTTP require developers to muck around with
Base64 encoding or other encodings when including detail within a header.
This approach would eliminate that extra step. For instance, if I wanted to
have a Content-Integrity header whose value is an hmac digest, I would be
able to drop the raw bytes of the digest into the header value rather than
base64 or hex encoding it into an ASCII string, saving CPU cycles and
reducing the amount of data that must be transmitted.

3. Header values that contain character data would not be limited to
US-ASCII. Multiple charset encodings would be allowed... obviously this has
a whole slew of issues associated with it that need to be carefully
considered. The charset encoding flag could be dropped, if necessary, from
this proposal.

For HTTP/1.1 Compatibility, each REGISTERED Header would be mapped to a
known, registered HTTP/1.1 header, allowing one to one translation from the
optimized form to the HTTP/1.1 form. Binary values would be base64-encoded.
If a particular header does not allow for Base64 encoded values under
HTTP/1.1, the down-level recipient would have the option of responding with
an appropriate 404 response.

That's it for now. There are additional considerations to be given to the
specific selection of header fields to include within the SYN_STREAM vs.
follow-on HEADERS frames but that's a separate conversation. As always,
feedback is welcome...

- James

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 13 Jul 2012 22:27:22 +0000

In message <CABP7RbepWH4ahSPHDU_M_w0tRVz_RRm1FV-jM_Y72=[email-masked]>
, James M Snell writes:

>messages into SPDY frames is less than optimal. [...]

While I appreciate your willingness to keep trying to modify SPDY,
I think it is premature to do so, certainly until we know what
HTTP/2.0 should actually do and how it should do it.

For instance if we decide to do the envelope/substance split,
(probably envelope/metadata/substance actually), that has pretty big
implications for the requirements to the actual protocol.

Such fundamental questions should be resolved before we waste time
trying to fine-tune representations.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 14 Jul 2012 00:51:04 +0200

Hi Poul-Henning,

> trying to fine-tune representations.

I think such contributions here on the WG are useful. As you probably
have noticed when we spent countless hours rotating bits in every
possible combinations, starting from a blank page is extremely difficult,
and most of the progress comes from contributors who are able to comment
on what they like or don't like in the design.

Whatever will be retained as a basis for HTTP/2.0, this exercise is
useful and may incite other users to provide very valuable feedback.

Best regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 13 Jul 2012 23:01:15 +0000

>useful and may incite other users to provide very valuable feedback.

I think it is premature, because it obscures and prevents the much
needed high-level design of HTTP/2.0.

And that is exactly why I think the current approach and timeline
is a road to nowhere fast.

The fact that we just saw Google say they would get behind any
improvement, as long as it is SPDY pretty much dooms the HTTP/2.0
effort right there and then:  All that's on the table is minor
rearrangements of the deckchairs, there is no opening for
changing the course.

And with that, over and out.

---

From: Mike Belshe <[email-masked]>
Date: Fri, 13 Jul 2012 16:10:18 -0700

> improvement, as long as it is SPDY pretty much dooms the HTTP/2.0

For the record, Google did not say that.

Such negative biases really don't help the discussion.

Mike

---

From: Jonathan Ballard <[email-masked]>
Date: Fri, 13 Jul 2012 16:27:03 -0700

I remembered when you called someone "bot".

No, it did not help at all. We need not enumerate reactions.

---

From: Roland Zink <[email-masked]>
Date: Sat, 14 Jul 2012 01:36:05 +0200

A similar binary encoding for HTTP headers is used in the WAP (Wireless 
Application Protocol). The goal there was to reduce the number of bytes 
so there are more optimization on size but skipping of values is a bit 
more difficult. If anybody is interested the full specification can be 
found at 
http://www.openmobilealliance.org/tech/affiliates/wap/wap-230-wsp-20010705-a.pdf. 
The header encoding is in section 8.4.

Regards,
Roland

On 14.07.2012 00:16, James M Snell wrote:
> This note is intended to provide some additional thoughts for 
> discussion around the design and use of SPDY as the possible basis for 
> HTTP/2.0. The intent is to provide fuel for discussion... comments are 
> definitely welcome.
>
> As discussed within draft-tarreau-httpbis-network-friendly-00, and as 
> has been mentioned several times in discussion on list, handling of 
> headers within the current SPDY framing, and in particular the 
> layering of HTTP/1.1 messages into SPDY frames is less than optimal. 
> There is significant wasted space, duplication, etc that -- strictly 
> speaking -- really isn't necessary. While I recognize that the 
> following increases the basic complexity of the protocol, it allows 
> fairly significant optimization following the same basic lines of 
> reasoning expressed in draft-tarreau-httpbis-network-friendly-00.
>
> Section 2.6.1 of the SPDY draft defines header blocks using the 
> following format:
>
> Flags provides flags for the specific header field. The flag 0x1 
> indicates that the header value contains Character Data. If not set, 
> the value is assumed to consist of raw octets. 0x2 indicates that the 
> value is compressed.
>
> Length is an unsigned 24-bit value specifying the number of octets 
> after the length field.
>
> When the T bit is NOT set, the Header field is a REGISTERED Header, 
> the structure of which is:
>
> The ID is a 32-bit number uniquely identifying the registered field. 
> Each is assigned by the registrar. For instance, the "Host" field 
> could have a registered value of "1", the "Accept-Lang" field could 
> have a registered value of "6", and so forth.
>
> If Flag 0x1 is set, the value is assumed to contain character data. 
> When set, the value MUST be preceded by a single unsigned 8-bit 
> integer identifying the character encoding utilized. The values are 
> assigned by the registrar. For instance, US-ASCII could have a 
> registered value of "1", while "UTF-8" could have a registered value 
> of "2".
>
> For example:
>
>    | 1 | 16 | 1 | www.example.org <http://www.example.org>    |
> content, the header ID = 1, the charset used is US-ASCII and the value 
> is "www.example.org <http://www.example.org>". The header is expressed 
> with a total of 28 bytes.
>
> The same flags apply. 0x1 indicates that the value is character data. 
> If 0x1 is not set, the value contains raw octets. The key difference 
> is that there is a 32-bit name length and variable length name field 
> in place of the 32-bit ID field in the REGISTERED header. All other 
> details remain the same.
>
> 1. Commonly used header names are omitted in favor of registered, 
> known numeric IDs, saving space and making it more efficient to scan 
> over commonly used headers. For instance, intermediaries that route 
> requests based on common headers such as Host etc could choose to 
> ignore EXTENSION header fields entirely, and scan only for the ID's of 
> the fields they are interested in, rather than having to parse the 
> entire bag of header names.
>
> Currently, mechanisms within HTTP require developers to muck around 
> with Base64 encoding or other encodings when including detail within a 
> header. This approach would eliminate that extra step. For instance, 
> if I wanted to have a Content-Integrity header whose value is an hmac 
> digest, I would be able to drop the raw bytes of the digest into the 
> header value rather than base64 or hex encoding it into an ASCII 
> string, saving CPU cycles and reducing the amount of data that must be 
> transmitted.
>
> US-ASCII. Multiple charset encodings would be allowed... obviously 
> this has a whole slew of issues associated with it that need to be 
> carefully considered. The charset encoding flag could be dropped, if 
> necessary, from this proposal.
>
> For HTTP/1.1 Compatibility, each REGISTERED Header would be mapped to 
> a known, registered HTTP/1.1 header, allowing one to one translation 
> from the optimized form to the HTTP/1.1 form. Binary values would be 
> base64-encoded. If a particular header does not allow for Base64 
> encoded values under HTTP/1.1, the down-level recipient would have the 
> option of responding with an appropriate 404 response.
>
> That's it for now. There are additional considerations to be given to 
> the specific selection of header fields to include within the 
> SYN_STREAM vs. follow-on HEADERS frames but that's a separate 
> conversation. As always, feedback is welcome...
>
> - James
>

---

From: Jonathan Ballard <[email-masked]>
Date: Fri, 13 Jul 2012 17:12:09 -0700

Maybe it's not quite intuitive enough, especially when we cross-reference
the similarities of that WAP binary encode with JPEG2000's binary encode,
the MIME types specifically: http://www.faqs.org/rfcs/rfc3745.html (and
more)

I haven't seen any patent that extends those MIME types with either the
static index or dynamic index value, which accomplishes the compressed
header desired in HTTP2. Sorry, I have no sample for this post.

On Fri, Jul 13, 2012 at 4:36 PM, Roland Zink <[email-masked]> wrote:

> Application Protocol). The goal there was to reduce the number of bytes so
> there are more optimization on size but skipping of values is a bit more
> difficult. If anybody is interested the full specification can be found at
> indicates that the header value contains Character Data. If not set, the
> value is assumed to consist of raw octets. 0x2 indicates that the value is
> compressed.
>
> Each is assigned by the registrar. For instance, the "Host" field could
> have a registered value of "1", the "Accept-Lang" field could have a
> registered value of "6", and so forth.
>
> the specific selection of header fields to include within the SYN_STREAM
> vs. follow-on HEADERS frames but that's a separate conversation. As always,
> feedback is welcome...
>
>  - James
>
>
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 14 Jul 2012 07:29:33 +0200

> is a road to nowhere fast.

On a personal taste, I find it fast too. 4 months to provide proposals
to replace the 15-year old HTTP/1, and 4 others to review them is short
in my opinion. Roy did not even have the time to publish the Waka spec
which could have brought a lot of fuel to the discussion !

> rearrangements of the deckchairs, there is no opening for

If you're talking about Roberto's mail, I read quite the opposite in
fact. It was said that Google was open to have anything provided that
the concepts raised in SPDY were preserved. By this I understand that
they want something that offers similar end user experience. I find
it much more open than how you read it.

I have talked long hours with the SPDY team at IETF83. They have running
code, users and data. Some of us (including me) don't like the way they
addressed certain things, especially in the context of HTTP/2. They
clearly said they were open to changes. What else do you want ? That's
why we're working on selected points that we think need to be addressed
instead of reinventing the wheel from scratch.

With more time, it would probably make sense to discuss every single point
here on the WG as was done for WebSocket with nothing stabilizing for 1
year, because a group-designed protocol should be much better than the one
designed by a small team to address a specific issue. But I think it still
makes sense to use the proposals as a basis for new work, eventhough some
concepts are missing from all of them (eg: user session).

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 14 Jul 2012 07:44:41 +0200

> The header encoding is in section 8.4.

Wow, very instructive, thanks Roland! Much more work has been done there
than what we proposed in our draft. And this one was implemented in many
ridiculously powered devices, which means that it's not that complicated
to implement. With a bit of refreshing, that could be used as a basis for
future encoding works.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 14 Jul 2012 06:27:52 +0000

In message <[email-masked]>

>For the record, Google did not say that.

Yes, Google did in fact say that, in Roberto Peon's email, which very
clearly and unmistakably didn't even acknowledge the existence of
any other proposal but SPDY.

You don't have to be a rocket scientist to figure out that the 500
pound gorilla is going to get what it wants, no matter what this
WG decides or not, as evidenced by the fact that they have rolled
out SPDY to the extent it is already.

And let me acknowledge that Google would be foolish if they just
dropped three years of work on SPDY, if nothing else it would
negatively impact their image as the web technology leader to be
told by some random dude in Denmark that their amazing protocol
isn't.

But it doesn't change the fact that SPDY will not be the simple and
efficient protocol we can live with and build on for the next 20
years.

HTTP2.0 should require less code to implement than HTTP/1.1, and
less pages of RFC to define.

Anything else is a fiasco.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 14 Jul 2012 06:42:10 +0000

In message <[email-masked]>, Willy Tarreau writes:

>the concepts raised in SPDY were preserved. 

Well, which way is it Roberto ?

Care to clarify ?

>I have talked long hours with the SPDY team at IETF83. [...] They
>clearly said they were open to changes. What else do you want ?

I want a sensible and ethical engineering process, one that doesn't
dive headlong into serialization of HTTP headers until we have
established what problems HTTP/2.0 should solve.

Sneaking things like server-push through the backdoor is simply not
acceptable, without structured decision to give up on HTTP's strict
request-response model.

It may be that we want to loose the strict R-R model, it may be
that we do not.

But making the HTTP/2.0 process a matter of "who already has a ready
ID" is not the way to decide a question like that.

---

From: Roberto Peon <[email-masked]>
Date: Sat, 14 Jul 2012 00:14:23 -0700

> Well, which way is it Roberto ?

Willy has it right.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 14 Jul 2012 07:17:30 +0000

In message <[email-masked]>

>> Well, which way is it Roberto ?

Thanks.

Does that include server-push ?

---

From: "HAYASHI, Tatsuya" <[email-masked]>
Date: Sun, 15 Jul 2012 03:56:11 +0900

Response inline.

> which could have brought a lot of fuel to the discussion !

+1.
I think that it is an important point.
Should we make what is replaced with HTTP1?
(Of course, compatibility is absolutely required.)
Or is something added on HTTP1?

It may be the same.
However, the goal may change.

--
HAYASHI, Tatsuya
Lepidum Co. Ltd.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 14 Jul 2012 21:22:15 +0200

> Should we make what is replaced with HTTP1?

I'm not sure what you mean.

> (Of course, compatibility is absolutely required.)

Yes it is, but not necessarily on the wire.

> Or is something added on HTTP1?

We're not adding on top of HTTP1 but redesigning the on-wire protocol
which is why it will not be called 1.x.

Regards,
Willy

---

From: Roland Zink <[email-masked]>
Date: Sat, 14 Jul 2012 22:16:55 +0200

You are welcome. From implementing a WAP gateway the server part seem to 
work as well, although the traffic was not as high as today's requirements.

Interestingly there was also a multiplexing implemented in the layer 
below (Wireless Transaction Protocol working over UDP) and the gateway 
had to translate this to HTTP.

Regards,
Roland

---

From: Roberto Peon <[email-masked]>
Date: Sat, 14 Jul 2012 13:54:20 -0700

It is difficult for me to make assertions about something that people have
not yet had time to play with. I'm of the school that believes that good
design is a byproduct of experience and refinement, and rarely comes from
vacuum or achieves perfection on the first imagining.

So, I don't know yet.
-=R
On Jul 14, 2012 12:17 AM, "Poul-Henning Kamp" <[email-masked]> wrote:

> In message <
> [email-masked]>
> , Roberto Peon writes:
>

---

From: "HAYASHI, Tatsuya" <[email-masked]>
Date: Sun, 15 Jul 2012 07:05:30 +0900

Dear Willy,

Response inline.

> I'm not sure what you mean.

Sorry.
"HTTP/2.0" are the marks of a big change for me.
Probably it is the same also for many Web programmers.
For example, big change excited like "Waka".
I had imagined "Version Number" such.
So, My "replaced" was putting it.
In this short time, I think that it is difficult.

> Yes it is, but not necessarily on the wire.

I see.

> which is why it will not be called 1.x.

I understood.
Your opinion is also right.
My viewpoint was partial.

Neither SPDY nor other proposals are big change for many Web programmers.
They do not need to be conscious although it is important, of course.
But, Surely it is not called 1.x.

Hummmm...

Thanks!
Sincerely,
--
HAYASHI, Tatsuya
Lepidum Co. Ltd.

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 15 Jul 2012 18:37:39 +1200

> In this short time, I think that it is difficult.

"Big change", but in what way? ....

If the next protocol were to be called HTTP/1.2 we would be forced to:
- use plain text lines limited toASCII charset ending in CRLF
- use full-name headers in human readable format
- support all the human-readable date formats, mime type formats, 
options names etc.
- use strict one request message followed by one response body stateless 
sequence

In short, we would be forced to NOT fix any of the real performance 
problems visible in HTTP/1.

"2.0" is a BIG change for the incoming stream parsers and outgoing byte 
packers. Flow control, input validation, messaging and potential future 
extensions are also affected in a big way. With just binary framing 
change, no new features, no new headers, no new anything else but framing.

Hopefully the result will be a BIG change towards simplicity. But only 
time will tell about that.

AYJ

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 15 Jul 2012 07:07:10 +0000

>time will tell about that.

I don't see any signs of that anywhere, care to elaborate what you
base your hope on ?

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 15 Jul 2012 20:58:52 +1200

> I don't see any signs of that anywhere, care to elaborate what you

"Hope is hope. If we had reasons, it'd be called surety" - Anonymous.

We have the opportunity for reducing the HTTPbis specs into one spec 
which describeds the framing for HTTP/2 and flor controls. Leaving the 
rest of the protocol features out as separate optional extensions. There 
is hope, slim maybe, but hope.

AYJ

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 15 Jul 2012 20:59:13 +1200

We have the opportunity for reducing the HTTPbis specs into one spec 
which describes the framing for HTTP/2 and flow controls. Leaving the 
rest of the protocol features out as separate optional extensions. There 
is hope, slim maybe, but hope.

AYJ

---

From: "HAYASHI, Tatsuya" <[email-masked]>
Date: Mon, 16 Jul 2012 09:24:07 +0900

Responses inline.

On Sun, Jul 15, 2012 at 3:37 PM, Amos Jeffries <[email-masked]> wrote:
> - support all the human-readable date formats, mime type formats, options
> names etc.
> In short, we would be forced to NOT fix any of the real performance problems
> visible in HTTP/1.

Sure. I agree.

I think that these parts were the power of HTTP.
And I understand that it is a part of problem of the now and the future.
I am interested in these two balance.

> extensions are also affected in a big way. With just binary framing change,
> no new features, no new headers, no new anything else but framing.

Are these already determination?
If I become so, I will think that I am glad.
I thought that I was discussion for determination.
(Please put me right if I make a mistake...)

> Hopefully the result will be a BIG change towards simplicity. But only time
> will tell about that.

I hope too.

Thanks.
Sincerely,

--
HAYASHI, Tatsuya
Lepidum Co. Ltd.

---

From: James M Snell <[email-masked]>
Date: Sun, 15 Jul 2012 17:52:37 -0700

On Sun, Jul 15, 2012 at 1:58 AM, Amos Jeffries <[email-masked]> wrote:

>> In message <[email-masked].**nz<[email-masked]>>,
>> Amos Jeffries writes:
>>
> We have the opportunity for reducing the HTTPbis specs into one spec which
> describeds the framing for HTTP/2 and flor controls. Leaving the rest of
> the protocol features out as separate optional extensions. There is hope,
> slim maybe, but hope.
>
>
While "leaving the rest ... out as separate optional extensions" sounds
great if you're in the business of providing the fundamental architecture,
it's not so great if you're building applications on top that rely on those
"optional" pieces. Simplification is a great thing, but HTTP/2 can't be
ONLY about the framing... it must also address very real issues that exist
elsewhere in the specification.

- James

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 16 Jul 2012 13:43:43 +1200

On 16.07.2012 12:52, James M Snell wrote:
> On Sun, Jul 15, 2012 at 1:58 AM, Amos Jeffries wrote:
>
>> which

This sounds like you are arguing that we need to force mandatory Range: 
request support, mandatory caching, mandatory authentication, etc, etc. 
all the way down the stack from the web application which needs it, to 
the simple load balancers or "HTTP router" pushing HTTP/2 around a CDN 
hierarchy.

I agree that application level implementations have a whole bunch of 
things they need to support. Browsers being probably the best example 
here, with have a far wider range of RFCs to support today than even 
just HTTP/1.1. CGI interface libraries not being far behind.

I'm questioning why implementers have to be faced with reading and 
verifying compliance of all this detail as "The HTTP/2.0 Document" when 
the request details are controlled by an application and the reply is 
generated by one. Consider an HTTP server which runs a simple CGI 
library. The server can decode and multiplex the framing, but has no 
need to handle or even implement most of the finer details for features 
the CGI library can cope with. It just needs to pass on the request and 
assemble the response bits (CGI could encoded its specific 
application-level headers easily enough).
  -> the server needs compliance with HTTP/2
  -> the CGI needs compliance with some application feature spec(s) it 
want to implement and HTTP/2

To take this to the extreme case;  Should we also borg the entirety of 
WebDAV (an application feature definition) into the HTTP/2.0 document 
just because web application layer needs it? I think not.
  Neither do I think www-auth should be part of the core HTTP/2 
specification. As opposed to proxy-auth in its role as connection 
authentication which *is* relevant to the transport level for even 
simple load balancers to handle properly.
  The SPDY draft is a good example of creeping the scope out beyond 
transport and the disagreements have *all* been focused around different 
"groups" needs being unmet by that all-in-one spec.

If you read PHKs' arguments about spec page length with a grain of 
salt, these same points are what he is arguing about.

A lot of the features we will be wanting to keep from HTTP/1.x 
(www-auth, ranges, caching etc being good ones). Describing their 
semantics in a separate document pointed at by both HTTP/2 and HTTP/1 
would be a great way to simplify the texts and ensure semantic 
compatibility. New features may well be needing to define HTTP/1 
backward compatibility and this style of specification would make it 
easier to extend 1.1+

Come to think of it, in our network-friendly draft we only specify 
version as far as "HTTP/2", the .x part is left unstated in the frame 
headers and could be added as a capabilities header minor version hint, 
with a finer grain of capability advertisement than then 1.x style of 
1/2/3/4/etc offers.

A lot of the things we want to do are use-case specific wants/needs. I 
simply want the thing calling itself HTTP/2.0 to state "X,Y,Z are 
REQUIRED". That is all. When we have a choice to specify something 
optional elsewhere lets do so.

AYJ

Afterword: WG editors; that opinion goes for the HTTPbis drafts final 
arrangement choices as well.

---

From: James M Snell <[email-masked]>
Date: Mon, 16 Jul 2012 23:51:30 -0700

Ok... spent some time this evening playing around with the header frame
syntax a bit more to see what further optimizations could be made and to
see if the binary encoded header id's made any noticeable difference in
size and ease of processing.

Here's the revised structure I played around with...

1. Within a HEADER block, I assume two possible types of headers,
REGISTERED and EXTENSION. A REGISTERED header is one that would be known to
the registrar and assigned a numeric id and a codepage. If the codepage is
0, the implication is that the header is MUST UNDERSTAND and is considered
one of the core headers for the basic operation of the protocol. Codepages
1-9 are MUST-IGNORE... that is, if a user-agent or server comes across a
header on these code pages that is not understood, the header can simply be
ignored. Codepages 10-14 are PRIVATE USE, with Codepage 10 being reserved
for MUST UNDERSTAND PRIVATE USE headers. EXTENSION headers are simple name
value pairs essentially as they exist today.

2. Within extension headers, the name portion MUST be ASCII and MUST NOT be
longer than 255 bytes (quite generous really).

3. Values may be binary or character based, as indicated by a flags field.
Values may be up to max(int32) in length.

4. Registered HTTP Methods can be identified by numeric value. Extension
Methods can be identified by character value.

5. The structure for REGISTERED HEADERS is...

  +------------------------------+
  |0| id (15-bit)| flags(8-bit)  |
  +------------------------------+
  | len (32-bit) |     value     |
  +------------------------------+

6. The structure for EXTENSION HEADERS is...

  +------------------------------+
  |1| flags(7-bit) | namelen (8) |
  +------------------------------+
  | name | val len (32) | value  |
  +------------------------------+

Assuming the following registered headers...

  public static final short VERSION = 1;
  public static final short METHOD = 2;
  public static final short HOST = 3;
  public static final short SESSION = 4;
  public static final short CHARSET = 5;
  public static final short REQUEST_URI = 6;
  public static final short ACCEPT_LANG = 4097;

And the following registered methods...

  public static final byte GET = 1;
  public static final byte POST = 2;
  public static final byte PUT = 3;
  public static final byte DELETE = 4;
  public static final byte PATCH = 5;
  public static final byte HEAD = 6;
  public static final byte OPTIONS = 7;

Let's assume that what we want to to encode a HTTP GET for resource:
  http://www.example.org/this/is/the/request?is=it&not=beautiful

With a session identifier of "session_key", ACCEPT_LANG = en-US and default
charset encoding for all character based header values is "US-ASCII"...
Let's also add an extension header "ext" with value "foo"...

The Version header can be encoded as:
  {0, 1, 0, 0, 0, 0, 2, 2, 0}

The GET Method header can be encoded as:
  {0, 2, 0, 0, 0, 0, 1, 1}

The Host header would be encoded as:
  {  0,   3,   1,  0,   0,   0,  15, 119, 119, 119,
    46, 101, 120, 97, 109, 112, 108, 101,  46, 111,
   114, 103}

The Accept-Lang header would be encoded as:
  {16, 1, 1, 0, 0, 0, 5, 'e', 'n', '-', 'U', 'S'}

The Extension header ext: foo would be encoded as:
  {-128, 1, 3, 101, 120, 116, 0, 0, 0, 3, 102, 111, 111}

The entire header block is encoded into a structure of 145 bytes in length;

[8, 0, 1, 0, 0, 0, 0, 2, 2, 0, 0, 2, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 0,
15, 119, 119, 119, 46, 101, 120, 97, 109, 112, 108, 101, 46, 111, 114, 103,
0, 6, 1, 0, 0, 0, 40, 47, 116, 104, 105, 115, 47, 105, 115, 47, 116, 104,
101, 47, 114, 101, 113, 117, 101, 115, 116, 63, 105, 115, 61, 105, 116, 38,
110, 111, 116, 61, 98, 101, 97, 117, 116, 105, 102, 117, 108, 0, 5, 1, 0,
0, 0, 8, 117, 115, 45, 97, 115, 99, 105, 105, 0, 4, 1, 0, 0, 0, 11, 115,
101, 115, 115, 105, 111, 110, 95, 107, 101, 121, 16, 1, 1, 0, 0, 0, 5, 101,
110, 45, 85, 83, -128, 1, 3, 101, 120, 116, 0, 0, 0, 3, 102, 111, 111]

By comparison, the same structure encoded using the existing SPDY HEADER
block would require 208 bytes sans compression.

After applying compression of the block using the SPDY dictionary, the
block compresses into 6 compact bytes.

[120, 63, -29, -58, -89, -62]
Assuming this structure was used within a SPDY_STREAM message, unencrypted,
a proxy/router that is scanning the headers to determine where to route the
SYN_STREAM too would need only to look at the first two bytes of each
header to determine if the header is either the HOST, METHOD, REQUEST_URI,
VERSION or SESSION identifier. This scheme should prove to be significantly
faster to scan and perform operations on than the current all-text-key-pair
model. As always, tho, your mileage may vary.

/end-experiment

- James

---

From: Mike Belshe <[email-masked]>
Date: Tue, 17 Jul 2012 00:18:03 -0700

i like the direction of this.  a good blend of a registry and extension
headers.

i wasn't quite sure how you got to the 6 byte compressed header, tho.

mike

On Mon, Jul 16, 2012 at 11:51 PM, James M Snell <[email-masked]> wrote:

> 2. Within extension headers, the name portion MUST be ASCII and MUST NOT
> be longer than 255 bytes (quite generous really).
>
> With a session identifier of "session_key", ACCEPT_LANG = en-US and
> default charset encoding for all character based header values is
> "US-ASCII"... Let's also add an extension header "ext" with value "foo"...
>
> Assuming this structure was used within a SPDY_STREAM message,
> unencrypted, a proxy/router that is scanning the headers to determine where
> to route the SYN_STREAM too would need only to look at the first two bytes
> of each header to determine if the header is either the HOST, METHOD,
> REQUEST_URI, VERSION or SESSION identifier. This scheme should prove to be
> significantly faster to scan and perform operations on than the current
> all-text-key-pair model. As always, tho, your mileage may vary.
>
> /end-experiment
>
> - James
>

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 00:19:51 -0700

Sigh... I should know better than to write code without coffee or beer in
hand... the actual compressed size of the header block is
128 bytes...

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 00:21:11 -0700

> i wasn't quite sure how you got to the 6 byte compressed header, tho.
Heh.. chalk that up to temporary idiocy on my part... and a complete lack
of either caffeine or alcohol in my system.

>> 3. Values may be binary or character based, as indicated by a flags
>> field. Values may be up to max(int32) in length.
>>
>>> has been mentioned several times in discussion on list, handling of headers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 07:27:15 +0000

In message <CABP7RbdZPYYqtDFWUprrAmGQ8GehOTh-LvXq9gizXGUJg4=[email-masked]>

>Values may be up to max(int32) in length.

Allowing such large fields is counterproductive in my opinion.

No implementation can have a limit that large without becoming
a sitting duck for DoS attacks.

It would be far better to put a realistic, but still generous size
restriction.  Not sure if 16 bits is always enough, but the advantage
will likely outweigh any disadvantages.

>The entire header block is encoded into a structure of 145 bytes in length;
[...]
>block compresses into 6 compact bytes.

This is where I think we should be careful about what we measure
and optimize for.

Those six bytes will take a lot more CPU effort to unpack than
your 145 bytes (I think your proposed encoding is quite bloated
btw, but that's not an argument I need to carry for this point).

The really important metric is that most requests fit in a single
MTU.  Inside that limit, expending bytes to gain parsing speed can
be both a good and necessary (for HTTP-routers) tradeoff.

In that view, the difference between six and 208 bytes is not
terribly significant on its own, whereas a difference between
1400 and 1602 bytes would be.

The main and almost only reason requests don't fit in one MTU is
cookies, with a minor side-order of URI and User-Agent,
so if we can design away the need for cookies, we (almost)
don't need the computational complexity of deflate.

And there is another aspect which has me very worried about
deflate at this point in the protocol:  Has anybody tried
to see how big a deflate output they can produce with
a single MTU of input ?

My guestimate is well north of a 1MB.

That's a lot of DoS leverage for a single packet.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 17 Jul 2012 16:44:39 +0900

Hello James,

I agree with Poul-Henning that this is way too long.

> while "UTF-8" could have a registered value of "2".

The IANA charset registry already has MIBenum numbers. It would reduce 
registry effort a lot if these could be reused. More than one number for 
one and the same thing will create confusion. Unfortunately, theese 
numbers are in the range of 0-3000 or so, which doesn't fit into 8 bits.

But a much, much better solution in this day and age is to only allow 
one encoding, UTF-8. That by definition includes US-ASCII, covers all 
the world's characters, and is what HTML is moving towards (with quite 
surprising speed these days). And while in HTML (and other content 
formats), non-ASCII is extremely widespread, in HTTP, it is not, and 
having more than one encoding is needlessly complicated.

Regards,    Martin.

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 00:44:23 -0700

> will likely outweigh any disadvantages.

i tend to agree but I think we need to test out a variety of scenarios
before we limit too much. Instinct tells me that 16-bits is likely going to
be more than enough.

> btw, but that's not an argument I need to carry for this point).
Yeah.. um.. ignore that compression ratio, lol... that was an oopsie on my
part. The actual value is 128-bytes compressed using the SPDY dictionary
and optimizing for size. And yes, looking at the ratio (the..um.. real
one), i'm not convinced that the compression is completely worth the
additional cost. As the SPDY spec itself acknowledges, it is counter
productive to compress already compressed data; so if our header encoding
is compact enough without it, why spend the additional cycles to save 20
bytes or so.

I am interested in hearing what further optimizations could be made to the
structure. At this point I would definitely hesitate to call this a
"proposal" in the formal sense. It's an exercise to experiment with
alternative header encodings to see what benefits we can reap, if any. If
the exercise proves valuable and bears fruit, I'll draft up a proposal.

> don't need the computational complexity of deflate.

I'm all for eliminating or reducing the use of cookies. A Session header is
an excellent suggestion, IMHO, and something that reasonably should have
been done a long time ago. The more I look at it (assuming just for the
sake of discussion we're using SPDY)... the more I want to limit the number
of HTTP Request headers we drop into SYN_STREAM. SYN_STREAM should be kept
minimal, consisting of the version, method, host, request-uri and session.
All additional request headers can be pushed into one or more HEADER frames
following the SYN_STREAM and sized so that things easily fit into one MTU.

- James

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 09:56:01 +0200

On 2012-07-17 09:44, "Martin J. Dürst" wrote:
> ...
> ...

*If* we make a breaking change with respect to character encoding 
schemes, this is indeed the change to make.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 07:59:49 +0000

In message <[email-masked]>, Julian Reschke writes:
>On 2012-07-17 09:44, "Martin J. DÃ¼rst" wrote:
>> ...

Indeed, and a change I think HTTP/2.0 should make, in light of a
20 year design lifetime.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 17 Jul 2012 01:00:16 -0700

The max theoretical compression with a perfect deflate implementation is
1024 to one, according to documentation.
Thankfully, the zlib reference implementation allows stream decompression
and thus allows you to specify the max amount of bytes one allows the
incoming stream to inflate into. In other words, memory usage is bounded
and you can reject stuff that is overly large before it causes any problems.

I'm not particularly enamored of the deflate stuff as a whole solution. I
think it is just part of a whole solution in a solution space than needs
more exploration.
I'll be putting something out about this shortly, actually... just have to
stop responding here so I have time to finish writing it up (and then get
to work simulating and experimenting)!

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 08:03:48 +0000

In message <[email-masked]>

>think it is just part of a whole solution in a solution space than needs

To be honest:  If I were trying to see and measure benefits from smaller
HTTP requests, I would probably also start out with deflate, just to get
some numbers in, but as a permanent solution it's clearly a no-go.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 10:08:13 +0200

> Indeed, and a change I think HTTP/2.0 should make, in light of a

As far as I can tell, the only thing that makes this hard is the desire 
to be able to tunnel arbitrary HTTP/1.1 through HTTP/2.0.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 09:13:51 +0000

In message <CABP7Rbfg6nf=[email-masked]>
, James M Snell writes:

>Yeah.. um.. ignore that compression ratio, lol...

I would have checked it but I have become allergic to dissecting
gzip files after implementing ESI:includes on gzip'ed files in
Varnish :-)

>[...]so if our header encoding is compact enough without it, why spend
>the additional cycles to save 20 bytes or so.

Indeed.

Speaking of which:  I was surprised to find that the SPDY dictionary
didn't contain any fodder for compressing User-Agent ?

Was this future-proofing/neutrality, to avoid trademark issues or
because there were no reasonable general benefit to be had ?

(And what DO we do with User-Agent in HTTP/2.0, it must be possible
to come up with something actually usable... ?

>structure.

I take it that you have read draft-tarreau-httpbis-network-friendly-00 ?

It has a lot of good ideas...

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 17 Jul 2012 22:19:49 +1200

On 17/07/2012 8:08 p.m., Julian Reschke wrote:
> As far as I can tell, the only thing that makes this hard is the 
> desire to be able to tunnel arbitrary HTTP/1.1 through HTTP/2.0.

There is hope though. Clients will be forced to begin new connections 
with HTTP/1.1 syntax Upgrade request (hopefully with zero-cost as in 
network-friendly) for the next 5-15 years or so until HTTP/1 dies out.

This means there is a server response along with path details known to 
both ends of the connection before any HTTP/2 non-ASCII character 
encoding can come into effect. Via header is already mandatory, and from 
this datum we can gather the path software versions (good luck to those 
who hide their HTTP compliance level). Which tells both server and 
client whether they can assume safe use of non-ASCII characters or if 
they have to mangle things down to the HTTP/1 compatible headers.

This is only relevnt to endpoints which want/need to use non-ASCII. We 
have two aternatives:
  1) mandate receivers accept UTF-8 HTTP/2, mandate senders generate 
ASCII in the 2.0 spec for later opening up in 2.1 or so.
  2) advise senders to always use ASCII characters within UTF-8 possible 
range, but not forbid non-ASCII.

Either way mandating receivers to accept UTF-8 is a long overdue MUST in 
my books for HTTP/2.

AYJ

---

From: Roberto Peon <[email-masked]>
Date: Tue, 17 Jul 2012 06:24:24 -0700

On Jul 17, 2012 2:15 AM, "Poul-Henning Kamp" <[email-masked]> wrote:
>
> In message <CABP7Rbfg6nf=
[email-masked]>
> because there were no reasonable general benefit to be had ?

A combination of future proofing and neutrality, as you've guessed.

We didn't want to provide any motivation for people to lie about user-agent
any more than they do today, and having something in the dictionary would
have provided such a motivation.
We also didn't want the protocol to "choose" any favorites, i.e. we wished
it to be neutral.
-=R

 On Jul 17, 2012 2:15 AM, "Poul-Henning Kamp" <[email-masked]> wrote:

> In message <CABP7Rbfg6nf=
> [email-masked]>
> , James M Snell writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 13:38:02 +0000

In message <[email-masked]>
, Roberto Peon writes:

[lack of dictionary support for user-agent]

>A combination of future proofing and neutrality, as you've guessed.

Yeah, I can understand that.

Still sucks though.

The WURFL database contains what ?  17.000 entires ?

and that's just mobile devices.

There must be a smarter way than "User-Agent:"...

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 17 Jul 2012 09:48:19 -0400

Could I just point out that when you have a problem with an almost
infinite scope for minor variations, having a strawman on the table is
an essential pre-condition for success?

SPDY is not the first attempt to do multiplexing. Jim Gettys and
Henryk Nielsen worked on a similar scheme in '95. Before that Simon
Spero developed HTTP-NG. What is different this time round is that
there is at least one 500 pound gorilla who wants the banana.

Anyone can bikeshed the names, semantics of new headers. What really
interests me is questions such as:

* How will the new proposal be layered onto the Transport and
Application layers?
* Will the new scheme take advantage of multicast?
* What security will be provided?
* How robust will the new scheme be?

There are design issues in SPDY that I find a little troubling. I much
prefer using a compact encoding for headers over compression. A
compact encoding is easier to implement and much easier to debug. It
is also more robust as single bit errors are less likely to lead to
catastrophe.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 15:48:50 +0200

> There must be a smarter way than "User-Agent:"...

Actually one nice potential optimization is if the server can declare 
that it's not interested in the User-Agent at all; see 
<http://tools.ietf.org/html/draft-nottingham-http-browser-hints-03#section-5.7>

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 07:36:02 -0700

On Tue, Jul 17, 2012 at 6:48 AM, Phillip Hallam-Baker <[email-masked]>wrote:

> [snip]

I definitely have to agree with preferring the compact encoding over
compression. At the moment, I'm just not seeing the benefit of applying the
compression.

FWIW, I made a few additional tweaks to my test.. namely, reducing the
header value size to max(int16), and eliminating the charset header with
the assumption everything would be UTF-8 and the header block becomes a
simple 116 bytes in length. The compressed size (done properly this time,
albeit using the existing SPDY dictionary that obviously would need to be
tweaked to account for the new structure) comes out to only 114 bytes...

Just for fun, I added in my chrome browser's user-agent string, swelling
the encoded size to 237 bytes... compressed at 220. Again, Not enough of a
savings to justify the additional cost.... and yeah, let's definitely see
what we can do about fixing the whole user-agent crap-fest.

- James

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 07:48:13 -0700

Tunneling 1.1 traffic via 2.0 would likely be the easy part; it's the
possibility of having to downgrade from 2.0 to 1.1 mid stream that would
make this difficult. Declaring that all header values must be UTF-8 would
essentially make it impossible to reliably bridge 2.0 traffic to 1.1
because whichever component is doing that bridging would have to know
exactly how to translate every header into it's appropriate us-ascii
equivalent. For some of the well-known "registered" headers, that's not a
major problem but it obviously falls down completely when we start talking
about extension headers.

Of course, we face the exact same problem if we allow for purely binary
header values (as I have also suggested).

The one thing we need to determine is: how critical is the ability to
support seamless down-level conversion from 2.0 to 1.1 within a request? Is
it acceptable for us to say that while 2.0 can be used to transport 1.1
messages, the reverse is not possible.

- James

On Tue, Jul 17, 2012 at 3:19 AM, Amos Jeffries <[email-masked]> wrote:

> There is hope though. Clients will be forced to begin new connections with
> HTTP/1.1 syntax Upgrade request (hopefully with zero-cost as in
> both ends of the connection before any HTTP/2 non-ASCII character encoding
> can come into effect. Via header is already mandatory, and from this datum
> we can gather the path software versions (good luck to those who hide their
> HTTP compliance level). Which tells both server and client whether they can
> assume safe use of non-ASCII characters or if they have to mangle things
> down to the HTTP/1 compatible headers.
>
>  1) mandate receivers accept UTF-8 HTTP/2, mandate senders generate ASCII
> in the 2.0 spec for later opening up in 2.1 or so.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 16:59:17 +0200

> Tunneling 1.1 traffic via 2.0 would likely be the easy part; it's the

Not even that. Given an HTTP/1.1 message containing non-ASCII octets in 
header field value, you simply don't know what unicode characters to map 
them to.

This is not theoretical; some UAs process UTF-8 in Content-Disposition, 
some use the installation's locale character set.

Yes, this is a mess, but it's not clear to me how to break out of it 
without breaking *some* setups that currently "work".

> support seamless down-level conversion from 2.0 to 1.1 within a request?
> Is it acceptable for us to say that while 2.0 can be used to transport
> 1.1 messages, the reverse is not possible.
> ...

So how do you transport a 1.1 message inside 2.0 if it contains 
non-ASCII? Treat the header field value as binary?

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 08:05:27 -0700

> without breaking *some* setups that currently "work".
+1 ... there are quite a few issues that I would like to see HTTP/2.0
resolve that fall into the same issue. Some breakage will occur and I'm
good with that.

> So how do you transport a 1.1 message inside 2.0 if it contains non-ASCII?
> Treat the header field value as binary?
>
>
That's my initial thought but still noodling over it. I'm open to
suggestions but really want to keep the header encoding as compact as
possible.

- James

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 15:47:16 +0000

In message <CABP7RbfSMfNJBOkqRzsqdCOJ9Yi=[email-masked]>

>> Treat the header field value as binary?

I think we should try to avoid mixing 1.1 and 2.0 that way.

It will require amazing amounts of verbiage to specify, and the
requirement of 2.0->1.1 downgrade will serverely curtail any attempt
at improvement in User-Agent, Cookie and other hacks.

If we take an approach where you cannot send 1.1 messages once you
upgrade a connection to 2.0 and cannot send 2.0 messages over
a 1.1 connection, what are the consequences ?

For direct client-server connections ?  No trouble.

A 1.1+2.0 dualmode server with 1.1+2.0 dualmode proxies/gateways/routers
in front of it ?  No trouble.

A 1.1+2.0 dualmode client with a 1.1+2.0 dual mode proxy and an
unknown server ?  Problem: The client cannot know if it can send
2.0 the proxy, because if the server can only do 1.1, it's stuck.

Solution 1:  Require a WKS record for HTTP/2.0, then the client
will know.  Works.  Some Well Known Downsides.  Not pretty.

Solution 2:  Make first request always be 1.1, tag reponse to
indicate tha 2.0 end-to-end is availble.  Works, but backwards
relative to end-goal, and prevents 1.1 from being killed off.

Solution 3:  Have the proxy return 2.0 status-code that says non
2.0-end-to-end-path available, client reissues request as 1.1 (on
diffenent connection) to proxy.  Works, but cost of (local) RTT's
first time (client can cache "no-2.0" bit for Host: value.)

Solution 4:  Always send first request in "pidgin-2.0" which
is a subset for which a trivial conversion to 1.1 can be defined.

Solution 5:  Give 2.0 a special verb "Encapsulated 1.1 request",
use this to send 1.1 request to proxy as binary blob.  If 2.0-e2e
not available, response will be (similarly encapsulated) 1.1 reponse.
If 2.0-e2e is available, reponse will be 2.0 plain reponse, and
client can send plain 2.0 thereafter.

I think I prefer solution 5.  It's easy to specify and implement,
and leaves us full freedom to make 2.0 better than 1.1.

---

From: "Robert Brewer" <[email-masked]>
Date: Tue, 17 Jul 2012 08:57:55 -0700

> non-ASCII? Treat the header field value as binary?

Just to share a field note: The Python web community dealt with this exact problem recently with the advent of Python 3, which elevated Unicode quite a bit and exposed this problem more clearly to many. The chosen solution was to take the bytes-of-unknown-encoding and decode them as ISO-8859-1 (which at least won't error on any byte sequence), and leave that mess for a higher layer (which presumably would have more context) to re-encode/decode if they liked. Not a perfect solution but better than nothing.

Robert Brewer
[email-masked]

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 09:18:04 -0700

On Tue, Jul 17, 2012 at 8:57 AM, Robert Brewer <[email-masked]> wrote:

> Just to share a field note: The Python web community dealt with this exact
> problem recently with the advent of Python 3, which elevated Unicode quite
> a bit and exposed this problem more clearly to many. The chosen solution
> was to take the bytes-of-unknown-encoding and decode them as ISO-8859-1
> (which at least won't error on any byte sequence), and leave that mess for
> a higher layer (which presumably would have more context) to
> re-encode/decode if they liked. Not a perfect solution but better than
> nothing.
>
>
>
I suspect that we'll simply have to take the same approach here. I think I
would lean more towards not doing any decoding at all. Simply treat the
header value as a bag of octets and let the application figure it out from
there.

The one mechanism we should key off of, of course, is the Version header.
If we specify version 1.1, then the assumption is that the rather undefined
semantics of 1.1 are at play with regards to header values. Binary header
values would be disallowed for anything other than the core headers (host,
version, method, etc) and it would largely be up to the application to
figure things out. If we specify version 2.0, then the assumption is that
binary header values are allowed and all character-based header values are
UTF-8. This, at least, allows us to be no worse off than we are currently
with 1.1 but still allow improvement in 2.0.

It would just need to be understood that a down-level conversion from 2.0
to 1.1 (e.g. a 2.0-enabled client making a request against a 1.1-enabled
server) would be a necessarily lossy conversion.

- James

---

From: Patrick McManus <[email-masked]>
Date: Tue, 17 Jul 2012 12:23:02 -0400

when comparing against deflate you need to consider all the transactions
after the first one too. They compress much more effectively than the
first. (generally > 90% ratios). In a stream of ~80 requests, the first
one is only part of the story[*].

-P

[*] I didn't say it was an unimportant part :)

On Tue, 2012-07-17 at 07:36 -0700, James M Snell wrote:
> 
> 
> On Tue, Jul 17, 2012 at 6:48 AM, Phillip Hallam-Baker
> <[email-masked]> wrote:
>         [snip]
> compression. At the moment, I'm just not seeing the benefit of
> applying the compression.
> 
> 
> header value size to max(int16), and eliminating the charset header
> with the assumption everything would be UTF-8 and the header block
> becomes a simple 116 bytes in length. The compressed size (done
> properly this time, albeit using the existing SPDY dictionary that
> obviously would need to be tweaked to account for the new structure)
> comes out to only 114 bytes... 
> 
> 
> Just for fun, I added in my chrome browser's user-agent string,
> swelling the encoded size to 237 bytes... compressed at 220. Again,
> Not enough of a savings to justify the additional cost.... and yeah,
> let's definitely see what we can do about fixing the whole user-agent
> crap-fest.
> 
> 
> - James

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 18:38:05 +0200

> Just to share a field note: The Python web community dealt with this exact problem recently with the advent of Python 3, which elevated Unicode quite a bit and exposed this problem more clearly to…

That's also what APIs like XMLHTTPRequest and the servlet API are doing.

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 09:59:27 -0700

Noted and certainly understood. I'm still not convinced... assuming we can
come up with optimized binary encodings for the overwhelming majority of
the most common headers, eliminate redundant headers across requests, and
deal properly with the Crap that is Cookies and User-Agent headers, the
header block can be made to be very compact without applying any
compression.

If anything, I think I'm leaning more towards limiting the headers included
in the SYN_STREAM and SYN_REPLY frames to only a core set of uncompressed,
optimized headers but allowing HEADERS frames to optionally apply
compression. Doing so provides a reasonable balance between the two
approaches while reducing the amount of work routing intermediaries have to
do.

- james