Subject: FYI... Binary Optimized Header Encoding for SPDY

From: James M Snell <[email-masked]>
Date: Wed, 1 Aug 2012 17:37:57 -0700

I have submitted an I-D describing the alternative header encoding for SPDY
that I discussed previously. Should be pretty self-explanatory and there
are plenty of examples given throughout. I know it still has yet to be
decided whether SPDY will be the starting point for the HTTP/2.0 effort,
but I wanted to make this available for discussion.

http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt
http://tools.ietf.org/html/draft-snell-httpbis-bohe-00

- James

---

From: Jonathan Ballard <[email-masked]>
Date: Wed, 1 Aug 2012 18:41:20 -0700

Section 1.4 p7: http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt

That would be, after 89 bytes and SPDY, 150 bytes and 150 bits, 28 bytes
and 28 bits, 102 bytes minus 102 bits. One SPDY version uses UTF8, another
uses int32, and the one with mobility uses only 16bit words (network
friendly).

On Wednesday, August 1, 2012, James M Snell wrote:

> I have submitted an I-D describing the alternative header encoding for
> SPDY that I discussed previously. Should be pretty self-explanatory and
> there are plenty of examples given throughout. I know it still has yet to
> be decided whether SPDY will be the starting point for the HTTP/2.0 effort,

---

From: Mike Belshe <[email-masked]>
Date: Wed, 1 Aug 2012 20:30:48 -0700

A couple of thoughts:

* Thanks for writing up!

* I don't think we need utf-8 encoded headers.  Not sure how you'd pass
them off to HTTP anyway?

* The codepages seem like complexity, but I'm not sure the benefit.  I
would remove them.

* I would remove the flags too - per header flags - do we really need it?
 I'd remove it without a very clear use case.

* I know that 32bits seems like a lot.  Defining length fields has two
routes:  fixed length or variable length.  I like the fixed length because
I believe they are simpler.  However, the price of that simplicity is that
you've got limits.  Everyone hates limits :-)  In your proposal you whacked
the number of headers to 8 bits, or 256 headers.   While I agree this is an
edge, I don't see a reason why it should be against the rules to have more.
 Same for the length of a header value - you've used 16 bits (64KB).  While
this seems massive by today's standards, in 10 years maybe 1MB cookies are
the norm.  I don't know, but I'd hate to have the limit.  So.... this
leaves us thinking that maybe we should use variable length encoding.
 Personally, I think the fixed length simplicity is worth it.  But this is
subjective, of course.   Just use 32bits everywhere - it works well and you
won't notice the perf difference at all (I measured :-)

---

From: Yoav Nir <[email-masked]>
Date: Thu, 2 Aug 2012 07:25:01 +0300

On Aug 1, 2012, at 8:30 PM, Mike Belshe wrote:

> A couple of thoughts: 
> 
> * Thanks for writing up!
> 
> * I don't think we need utf-8 encoded headers.  Not sure how you'd pass them off to HTTP anyway?
> 
> * The codepages seem like complexity, but I'm not sure the benefit.  I would remove them.
> 
> * I would remove the flags too - per header flags - do we really need it?  I'd remove it without a very clear use case.
> 
> * I know that 32bits seems like a lot.  Defining length fields has two routes:  fixed length or variable length.  I like the fixed length because I believe they are simpler.  However, the price of that simplicity is that you've got limits.  Everyone hates limits :-)  In your proposal you whacked the number of headers to 8 bits, or 256 headers.   While I agree this is an edge, I don't see a reason why it should be against the rules to have more.  Same for the length of a header value - you've used 16 bits (64KB).  While this seems massive by today's standards, in 10 years maybe 1MB cookies are the norm.  I don't know, but I'd hate to have the limit.  So.... this leaves us thinking that maybe we should use variable length encoding.  Personally, I think the fixed length simplicity is worth it.  But this is subjective, of course.   Just use 32bits everywhere - it works well and you won't notice the perf difference at all (I measured :-)

Hi

I agree. I haven't written a draft like James, but I would like to suggest an alternative encoding, that I think has the advantage of being easier to parse:

Every header begins with a 48-bit descriptor (I think 32-bit would be enough with a 16-bit length/value, but this is safer)

  L0cRiiiiiiiiiiiiVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
or
  L1cRiiiiiiiiiiiillllllllllllllllllllllllllllllll

The first bit is the "Last" bit - 1 for the last header and 0 for all the previous ones
The second bit is 0 for TV headers (type-value) and 1 for TLV headers (type-length-value)
The third bit is 1 for critical (do not parse the request if you don't understand this) or zero for not
The fourth bit is reserved.

The next 12 bits are the type of header. There are actually 2^13 possible headers, because the same type can be used for both a TV and a TLV header.

For TV, the next 32 bits hold the actual value - a number from 0 to 4,294,967,295 - should be enough for lots of headers, including those denoting time
For TLV, the next 32 bits hold the length, and are followed by the value.

Using your registries from section 1.4 as an example, we get this:

20 01 00 02 00 00 (version=2.0)
20 02 00 00 00 01 (method=GET)
60 04 00 00 00 01 2F (path="/")
C0 03 00 00 00 0F 77 77 77 2E 65 78 61 6d 70 6C 65 2E 63 6F 6D (host="www.example.com")

That's 40 bytes. The HTTP/1.1 version is 39 bytes:

GET / HTTP/1.1
Host: www.example.com

Of course, you save more on headers that have longer names, like "strict-transport-security" or "Original-Encoded-Information-Types" or "Downgraded-Disposition-Notification-To"
You also save a lot on those that can be TV, and the value doesn't have to be numeric. It could be a bit string of whatever, as long as it fits in 32 bits.

Yoav

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 02 Aug 2012 05:21:53 +0000

need to be able to handle / transport private/unknown/extension headers.

I'd propose just flagging the header as such, and making the payload 
contain the header name followed by a single 00 byte then the value.  
Since header names have to be ASCII, this should be no problem.

Also, since this gives you an option to transport ANY header.

Also, length of size fields could be a frame attribute, rather than 
just 32 bits per-header.  So, if the frame wants to, it could say 
lengths are in 16 bit words, rather than 32, or whatever.  If you have 
a header that won't fit, you'd mark it in the frame.  This would give 
an upgrade path for lengths without the cost now.

I don't really like using 32 bit for TV headers though.  That won't 
work for dates.  I'd prefer a byte for value TYPE, e.g. 

00 = 32 bit integer
01 = 64 bit integer
02 = datetime (possibly just 64 bit ns since epoch)
03 = text
04 = binary/opaque
05 - 7F extension types (length prefixed value)

You'd potentially use this instead of differentiating between TV and 
TLV headers, since anything with variable length (text or binary) would 
have a length field in there as well, fixed-length types wouldn't need 
it.

New types could be overloaded within binary, and the length would take 
care of it.

Or could just use the bit for whether there's a length field before 
payload.  But I think we need to cater for more than just 32bit integer 
when it comes to fixed-length value types (if only date/time).

Actually date time would benefit from future-proofing as well, since ns 
may be great resolution now, but hopeless in 10yrs.  Which would 
require either an extension type, or variable length type for datetime 
encoding (e.g. maybe just for fractional second part).

Adrien

------ Original Message ------
From: "Yoav Nir" <[email-masked]>
To: "Mike Belshe" <[email-masked]>
Cc: "James M Snell" <[email-masked]>;"[email-masked]" 
<[email-masked]>
Sent: 2/08/2012 4:25:01 p.m.
Subject: Re: FYI... Binary Optimized Header Encoding for SPDY

---

From: James M Snell <[email-masked]>
Date: Wed, 1 Aug 2012 22:51:47 -0700

On Aug 1, 2012 8:30 PM, "Mike Belshe" <mike@
<[email-masked]>belshe.com<[email-masked]>>
them off to HTTP anyway?

The hand off to http 1.1 could be problematic, yes and I'm not 100% sold on
the utf-8 thing yet either. It's not an impossible problem tho and
shouldn't be too much of a challenge since most of the existing header
value definitions would likely not be modified... that is, just because
utf-8 is allowed in general doesn't mean that the existing value
definitions would all automatically be changed to allow for all possible
utf-8 characters.

would remove them.

The code pages are specifically *because* of the http 1.1 handoff and the
need to continue supporting extensibility of headers. If an intermediary
happens across a code page zero header that it does not understand, and
therefore cannot translate to an appropriate http 1.1 style header, a
protocol error can be returned.

The immediate use cases are documented within the draft already: binary
header values and multiple values.

If you need more than 256 headers, use multiple HEADERS frames. There is no
limit to the number of HEADERS frames you can send so that should cover
your edge case.

And I would argue that anyone who is using header values longer than
max(int16) Is Doing It Wrong.

Variable length encoding would be a mistake because it's simply not
necessary.

- James

>
>
>
> On Wed, Aug 1, 2012 at 5:37 PM, James M Snell <jasnell <[email-masked]>
>>
>> http <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>://<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
www.ietf.org <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>/<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
id <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>/draft-<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
snell <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>-<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
httpbis <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>-<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
bohe <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>-00.<http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
txt <http://www.ietf.org/id/draft-snell-httpbis-bohe-00.txt>
>> http <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>://<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>
tools.ietf.org <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>/<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>
html <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>/draft-<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>
snell <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>-<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>
httpbis <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>-<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>
bohe <http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>-00<http://tools.ietf.org/html/draft-snell-httpbis-bohe-00>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 Aug 2012 06:16:49 +0000

In message <emc1f2400c-3d0f-48eb-8eea-cb91edab99b3@bombed>, "Adrien W. de Croy"

>02 = datetime (possibly just 64 bit ns since epoch)

No, no, no, no!

If if you want fractional seconds in binary representation, you make
it fractional seconds in binary representation:

	32.32 bits since epoch.

(or if you prefer: 64 bit count of 1/2^32 seconds since epoch)

(See also: NTP)

Your suggested format would pointlessly require a 64bit division
and mulitiply operation to pull out the integer seconds field, or
to turn a integer second field into your format.

With a true binary format, it is just a matter of pulling 32 bits
in/out of 64 bits.

Should we later need higher resolution, a true binary format can
trivially be extended to 48.48 bits since epoch (or 37.41 bits
if aliens take over our computer architecture) without costing
an expensive conversion routine.

Mixed radix, and alien-radix representations of time is _always_
wrong, no matter who blesses the format (see timeval, timespec and
many other mistakes.)

In reality, it is dubious if we need more than millisecond resolution
for HTTP timestamps any time soon, and therefore the best current
compromise format would probably be 48.16.

(64 bit count of 1/65536ths of a second)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mike Belshe <[email-masked]>
Date: Thu, 2 Aug 2012 00:59:08 -0700

On Wed, Aug 1, 2012 at 10:51 PM, James M Snell <[email-masked]> wrote:

>
> On Aug 1, 2012 8:30 PM, "Mike Belshe" <mike@ <[email-masked]>belshe.com<[email-masked]>>
> wrote:
> >
> > A couple of thoughts:
> >
> > * Thanks for writing up!
> >
> The hand off to http 1.1 could be problematic, yes and I'm not 100% sold
> on the utf-8 thing yet either. It's not an impossible problem tho and

I just don't see any problem being solved by adding this?  If there is no
benefit, we should not do it, right?

> protocol error can be returned.
Why would there be a code page zero header that it doesn't understand?  We
don't want a registry here which changes over time - those are nightmares
to maintain.  However, a static translation could work, which is what I
thought you had with the known headers and the extensible headers. The code
pages aren't necessary to make this work - and seem to just add unnecessary
partitioning.

>  > * I would remove the flags too - per header flags - do we really need
> it?  I'd remove it without a very clear use case.
> >
>
One was for utf8, which I think we could nix.  I'm not sure why we need the
binary/multivalue flags - what feature is this for?   I don't see why we'd
want this - its not like you can pass binary data up to HTTP anyway?

> If you need more than 256 headers, use multiple HEADERS frames. There is
> no limit to the number of HEADERS frames you can send so that should cover
> your edge case.
>
> I can live with that; there might be a sentinel issue here - when did you
get the last set of headers.

> max(int16) Is Doing It Wrong.
I disagree.  If you had a 1Gbps line to your phone and a 1Tbps line to your
house, a 100KB cookie might be just great...  we have to design this for
the future.

>  Variable length encoding would be a mistake because it's simply not
I don't like it either.  But compressing the 32bits to 16bits is a poor
trade.  Just make it work.  This level of compression won't have an impact
on performance, but will have an impact on how the protocol evolves.

Mike

- James
>
> >
> >
> >
> > On Wed, Aug 1, 2012 at 5:37 PM, James M Snell <jasnell<[email-masked]>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 Aug 2012 08:27:35 +0000

In message <[email-masked]>

>benefit, we should not do it, right?

If this would solve any major problems inside a 20 year horizon, we
should do it.

That being said, I am not a big fan of UTF8 in high-performance
protocol context:  It is much slower to process than 8bit string
formats.

UTF8 also gives rise to a number of interesting security aspects,
primarily where humans eyeball for security and don't detect minor
differences between glyphs, particularly in FQDNs, but I can't see
how we can do anything about that in HTTP/2.0.

It's not obvious to me, that we can evade the UTF8 requirement,
so it might be worthwhile to consider what we can gain by embracing
it.

For instance, could we get rid of the %-encoding of URIs by allowing UTF8 ?

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Aug 2012 12:20:31 +0200

On 02.08.2012 07:51, James M Snell wrote:
> <mailto:[email-masked]>belshe.com <mailto:[email-masked]>> wrote:
>  >
>  > A couple of thoughts:
>  >
>  > * Thanks for writing up!
>  >
 > ...

I believe solving the header field encoding problem would be great; in 
particular with some people trying to move all metadata into header fields.

As far as I can tell, the only widely used header field that *does* 
require non-ASCII is Content-Disposition, for which a non-lossy 
transformation is available (see 5987/6266), although some older UAs do 
not get that.

Best regards, Julian

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 02 Aug 2012 12:09:42 +0000

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
>to turn a integer second field into your format.
I guess the point I was trying to make is maybe we should consider 
value types other than only 32 bit integers.

Best representation of date / time depends on what it will be used for 
in the end.  Any sort of epoch-based representation requires a bunch of 
divisions to convert to human-readable form, whereas epoch-based is 
obviously better for comparisons.

So maybe even several date/time types would be appropriate (I can hear 
the groans).  But that seems way OTT.

In the end though, when I get a file attributes from windows, the 
filetime members are in 64 bit ns since epoch.  So either I'd convert 
to something before transmission, or push that task through to the 
client side.

I'd have a bunch of 64 bit division to do to convert that to 32.32 or 
whatever other format.

I don't know what sort of timestamps unix serves up.  I'd expect it 
mainly to be determined by underlying clock hardware on motherboards.  

Adrien

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 Aug 2012 12:28:54 +0000

In message <em57fc159f-7eb0-4506-89a6-a55d04ed64ed@reboist>, "Adrien de Croy" w

>to something before transmission, or push that task through to the 

Just because somebody else did it wrong, doesn't mean we should
emulate them.

>mainly to be determined by underlying clock hardware on motherboards.  

No, it's been standardized (wrong!) for 20+ years: struct timeval
and struct timespec, both of which are also mixed-radix formats.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 02 Aug 2012 09:39:59 -0700

> 05 - 7F extension types (length prefixed value)

I have a nagging concern that we not reinvent ASN.1 in this process.

Eliot

---

From: Mike Belshe <[email-masked]>
Date: Thu, 2 Aug 2012 09:42:34 -0700

> I have a nagging concern that we not reinvent ASN.1 in this process.

+1!

"Better is the enemy of good".

Mike

---

From: Martin Thomson <[email-masked]>
Date: Thu, 2 Aug 2012 09:44:07 -0700

> I have a nagging concern that we not reinvent ASN.1 in this process.

We could always USE ASN.1 (and PER).

---

From: Eliot Lear <[email-masked]>
Date: Thu, 02 Aug 2012 09:58:36 -0700

Martin,

> We could always USE ASN.1 (and PER). 

Thank you for bringing home the point that there are complexity
tradeoffs for doing all of this.

Eliot

---

From: James M Snell <[email-masked]>
Date: Thu, 2 Aug 2012 10:08:53 -0700

> benefit, we should not do it, right?

The ability to define header values as UTF-8 would help application
developers in a number of ways... just look at the various inconsistencies
that currently exist in the definitions of standardized headers like
Authorization, Content-Disposition, Link, etc.

> pages aren't necessary to make this work - and seem to just add unnecessary

Um... HTTP headers are extensible. There are new headers popping up all the
time. A registry that changes over time would be ideal and the codepages
mechanism (or other similar must-understand type system) would be essential
to enabling translation back to 1.1. We have to design for the future,
right? ;-)

> One was for utf8, which I think we could nix.  I'm not sure why we need
> the binary/multivalue flags - what feature is this for?   I don't see why
> we'd want this - its not like you can pass binary data up to HTTP anyway?
>
>
Again, if there's anything in an HTTP 2.0 message that the intermediary
cannot reliably translate to 1.1 when it needs to, then the intermediary is
free to return a protocol error back to the client. The rational behind
allowing binary values is to eliminate the necessity of applying
ascii-transformations to values and allow for more efficient use of the
transport.

>
>
>>  > * I know that 32bits seems like a lot.  Defining length fields has
>> two routes:  fixed length or variable length.  I like the fixed length
>> because I believe they are simpler.  However, the price of that simplicity
>> is that you've got limits.  Everyone hates limits :-)  In your proposal you
>> whacked the number of headers to 8 bits, or 256 headers.   While I agree
>> this is an edge, I don't see a reason why it should be against the rules to
>> have more.  Same for the length of a header value - you've used 16 bits
>> (64KB).  While this seems massive by today's standards, in 10 years maybe
>> 1MB cookies are the norm.  I don't know, but I'd hate to have the limit.
>>  So.... this leaves us thinking that maybe we should use variable length
>> encoding.  Personally, I think the fixed length simplicity is worth it.
>>  But this is subjective, of course.   Just use 32bits everywhere - it works
>> well and you won't notice the perf difference at all (I measured :-)
>> >
>> >
>>

I'm not convinced there needs to be a signal for the "last set of
headers"... I've been kicking around a variety of cases where it could make
sense to interleave HEADERS frames throughout the data stream right up
until the FIN is sent. Going to be thinking through it a bit more.

> I disagree.  If you had a 1Gbps line to your phone and a 1Tbps line to
> your house, a 100KB cookie might be just great...  we have to design this
> for the future.
>
>

"Designing for the future" also means guiding and encouraging developers to
make better choices about what kind of data they go about sticking in
things like Cookies.

- James

---

From: Yoav Nir <[email-masked]>
Date: Thu, 2 Aug 2012 20:27:58 +0300

> tradeoffs for doing all of this.

Why?  It makes things so simple:

  HTTPHeaders  ::=  SEQUENCE OF HTTPHeader

  HTTPHeader   ::=  SEQUENCE {
        type     OBJECT
	critical BOOLEAN
        value    OCTETSTRING
   }

All you need is a 

No, I'm not seriously proposing this. But a simple TV/TLV scheme could work.

---

From: James M Snell <[email-masked]>
Date: Thu, 2 Aug 2012 10:48:52 -0700

On Thu, Aug 2, 2012 at 1:27 AM, Poul-Henning Kamp <[email-masked]>wrote:

> In message <
> [email-masked]>
> , Mike Belshe writes:
>
This is precisely why I favor the introduction of a binary value option and
the definition of highly-optimized binary encodings for the most commonly
used protocol headers (like method, version, etc). Things like Host and
Request URI need to be looked at tho. I suspect that, for a variety of
reason, we'll want to keep limiting those values to ASCII only.  (just
because the value COULD be UTF-8, doesn't mean a specific header definition
cannot limit the actual value to some reasonable subset).

> For instance, could we get rid of the %-encoding of URIs by allowing UTF8 ?

It would be possible, for instance, to begin using IRI's directly without
translating those to URI's first. Doing so, however, does not eliminate the
need for %-encoding, and there are a range of possible issues that could
make this problematic.

- James

---

From: Timothy Knox <[email-masked]>
Date: Thu, 2 Aug 2012 12:29:47 -0700

Somewhere on Shadow Earth, at Thu, Aug 02, 2012 at 10:48:52AM -0700, James M Snell wrote:
<snip>
> cannot limit the actual value to some reasonable subset).

For the Host header, I have just three letters to say to you: I-D-N. :-)
-- 
Timothy Knox <mailto:[email-masked]>
I like to say that in the old days, if you reinvented the wheel,
you would get your wrist slapped for not reading. But nowadays
people are reinventing the flat tire. I'd personally be happy if
they reinvented the wheel, because at least we'd be moving forward.
    -- Alan Kay

---

From: Eliot Lear <[email-masked]>
Date: Thu, 02 Aug 2012 13:22:14 -0700

> Why?  It makes things so simple:

Let's just see if it stays simple by the time it's done.

Eliot

---

From: Yoav Nir <[email-masked]>
Date: Thu, 2 Aug 2012 23:24:45 +0300

> Let's just see if it stays simple by the time it's done.

Please, let's not see that. Besides, if the PKIX people can be kept away, we'll be fine.

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 03 Aug 2012 01:04:32 +0200

On Thu, 02 Aug 2012 05:30:48 +0200, Mike Belshe <[email-masked]> wrote:

> In your proposal you whacked the number of headers to 8 bits, or 256  
> headers.

Given that the frame itself has a size field, we don't need to state the  
number of headers at all.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 03 Aug 2012 01:25:08 +0200

> protocol context:  It is much slower to process than 8bit string

I would like to know more on what operations you need. I imagine that most  
relevant operations (splitting, joining, comparing, strlen) can be  
performed directly on the encoded UTF8 string as efficient as on ASCII.  
Normalization and upper/lowercasing is trickier, but mostly because of all  
the Unicode rules, not UTF8 itself(though it doesn't help).

> how we can do anything about that in HTTP/2.0.

Defining legal character ranges and what character encodings to use are  
two different problems. Similar looking characters are indeed a problem  
already today, and it is known and worked on on the browser side.

/Martin Nilsson

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 03 Aug 2012 13:24:52 +0900

> For the Host header, I have just three letters to say to you: I-D-N. :-)

You mean Internationalized Domain Names, and specifically U-Labels (the 
thing that can actually be read the way it's intended to read, rather 
than some useless salad of letters after xn--)? I fully agree. The 
reason these are in punycode in HTTP/1.1 is that HTTP/1.1 is way older 
than IDNs, but for HTTP/2.0, that's not the case.

Regards,   Martin.

---

From: James M Snell <[email-masked]>
Date: Thu, 2 Aug 2012 21:33:57 -0700

> You mean Internationalized Domain Names, and specifically U-Labels (the
thing that can actually be read the way it's intended to read, rather than
some useless salad of letters after xn--)? I fully agree. The reason these
are in punycode in HTTP/1.1 is that HTTP/1.1 is way older than IDNs, but
for HTTP/2.0, that's not the case.

+1... while I recognize there are a list of concerns with their use, it
would be excellent to avoid having to punycode those or apply other
different encoding mechanisms to other headers as well.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 03 Aug 2012 13:35:00 +0900

On 2012/08/03 2:48, James M Snell wrote:
> On Thu, Aug 2, 2012 at 1:27 AM, Poul-Henning Kamp<[email-masked]>wrote:

> translating those to URI's first.

Great idea. Please note that that will also save a few bytes (but that's 
definitely not the main reason for doing it).

> Doing so, however, does not eliminate the
> need for %-encoding,

Yes, a '#' or '?' in a path segment and similar stuff still have to be 
%-encoded.

> and there are a range of possible issues that could
> make this problematic.

Could you list up the issues you're thinking about? (I don't want to say 
there are none, but I can't at the moment come up with something that 
wouldn't already be around currently.)

Regards,    Martin.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 03 Aug 2012 05:15:04 +0000

------ Original Message ------
From: "Martin J. Dürst" <[email-masked]>
To: "James M Snell" <[email-masked]>
Cc: "Poul-Henning Kamp" <[email-masked]>;"Mike Belshe" 
<[email-masked]>;"[email-masked]" <[email-masked]>
Sent: 3/08/2012 4:35:00 p.m.
Subject: Re: FYI... Binary Optimized Header Encoding for SPDY
>Great idea. Please note that that will also save a few bytes (but 
>that's definitely not the main reason for doing it). 
>

if we're defining a new binary-safe transport for header values, 
shouldn't we try to avoid all multiplexing / escaping and parsing of 
strings?

e.g. just put querystring in another "header" instead.  Then anything 
can contain '?'

same with fragments (#) although I thought these weren't allowed on the 
wire...

In fact the concept of a single string which is a URI could be 
deprecated for 2.0 and just be sent as individual fields in a request.

gatewaying back to 1.1 would require assembling a URI from the pieces, 
but that should be easy.  

Seems a bit nuts to go binary and leave some parts as overloaded string 
fields requiring string parsing and escaping.

Adrien

>Could you list up the issues you're thinking about? (I don't want to 
>say there are none, but I can't at the moment come up with something 
>that wouldn't already be around currently.) 
>
>Regards, Martin. 
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Aug 2012 05:39:14 +0000

In message <[email-masked]>, "Martin Nilsson" 
writes:
>> [Not thrilled about UTF8 due to performance]
>

It is indeed the complex Unicode tables that worries me, but as I said,
there may also be advantages to UTF8, so I'm not decided either way.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Aug 2012 05:56:27 +0000

>e.g. just put querystring in another "header" instead.  Then anything 

I have 100% assumed this would be the case all along, if nothing else
because the URI obviously belongs in the envelope, but the query string,
for reasons of privacy, probably does not.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 03 Aug 2012 16:59:07 +0900

> e.g. just put querystring in another "header" instead. Then anything can
> contain '?'

Well, yes, but then you might have a '=' or a '&' in an parameter value 
(hopefully not in a parameter name), and you will have to escape that. 
And then you could create separate headers (or what not) for each 
parameter name/value pair, but that would be in conflict with the 
understanding that the syntax details of query parts are essentially 
between the client application and the server.

> same with fragments (#) although I thought these weren't allowed on the

Yes, the aren't sent.

> deprecated for 2.0 and just be sent as individual fields in a request.

For some cases, such as chopping off the query part, that might work and 
be a good idea (e.g. for privacy reasons as Poul-Henning notes in a 
followup), but in general, it's a slippery slope, and it may be really 
difficult to get rid of escaping altogether.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 03 Aug 2012 19:27:14 +0900

On 2012/08/02 17:27, Poul-Henning Kamp wrote:
> In message<[email-masked]>
> , Mike Belshe writes:
>

It will solve quite a few problems, some of them major, maybe not for 
HTTP itself, but for the applications on top. It will actually solve 
some problems that have been around for at least the last 15 years.

HTML and HTTP were created when the breakthrough of iso-8859-1 (Latin-1) 
in Western Europe was predictable (the nascent Web helped to unify the 
Western Europe 'national' 7-bit and 8-bit encodings quite a bit).

At least as early as 1995 (RFC 2070) or 1996 (RFC 2130, RFC 2277), it 
was clear to those concerned that Unicode and UTF-8 was the way of the 
future. As everybody should be able to confirm when thinking about 
US-ASCII, using a single character encoding (rather than e.g. ASCII and 
EBCDIC or some such alternatively) brings HUGE benefits. The same is 
true when streamlining from a zoo of character encodings to UTF-8.

These days, over 60% of the Web is already in UTF-8, and if you add in 
the 20% of pure ASCII which is trivially also UTF-8, it's 80%. All other 
encodings are in serious decline. (see p. 52 of the July IEEE Spectrum). 
And efforts such as HTML5 are strongly pushing to get more UTF-8. I 
think lots of HTTP users would appreciate a better commitment from HTTP 
with respect to character encoding in headers and the like. What's 
currently there is really just a mess, and should be cleaned up.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 03 Aug 2012 19:27:59 +0900

On 2012/08/03 8:25, Martin Nilsson wrote:
> On Thu, 02 Aug 2012 10:27:35 +0200, Poul-Henning Kamp
> <[email-masked]> wrote:
>
> I would like to know more on what operations you need. I imagine that
> most relevant operations (splitting, joining, comparing, strlen) can be
> Normalization and upper/lowercasing is trickier, but mostly because of
> all the Unicode rules, not UTF8 itself(though it doesn't help).

I very much agree. I'm also not sure what "8bit string" means. If it 
means binary data, then I agree that's the way to go for things like 
dates, but it doesn't help at all for text. If it's actual text, then it 
doesn't make sense because one needs more than 256 characters to write 
the world's languages.

>> how we can do anything about that in HTTP/2.0.

Yes. Human eyballs can also easily be confused by changing between 0/O 
and 1/l/I and such, so that's not a new problem.

> already today, and it is known and worked on on the browser side.

Yes indeed!

Regards,   Martin.

---

From: Jonathan Ballard <[email-masked]>
Date: Fri, 3 Aug 2012 08:30:37 -0700

ASCII is not "trivially UTF8." UTF8 lacks the available flow control of
ASCII. Any conversion between ASCII and EBCDIC is best done in hardware. We
already know the security issue of conversions from unicode to EBCDIC, and
I doubt that is something we can scheme here on on-topic.

On Friday, August 3, 2012, "Martin J. Dürst" wrote:

>> In message<
> It will solve quite a few problems, some of them major, maybe not for HTTP
> itself, but for the applications on top. It will actually solve some
> problems that have been around for at least the last 15 years.
>
> At least as early as 1995 (RFC 2070) or 1996 (RFC 2130, RFC 2277), it was
> clear to those concerned that Unicode and UTF-8 was the way of the future.
> As everybody should be able to confirm when thinking about US-ASCII, using
> a single character encoding (rather than e.g. ASCII and EBCDIC or some such
> alternatively) brings HUGE benefits. The same is true when streamlining
> from a zoo of character encodings to UTF-8.
>
> These days, over 60% of the Web is already in UTF-8, and if you add in the
> 20% of pure ASCII which is trivially also UTF-8, it's 80%. All other
> And efforts such as HTML5 are strongly pushing to get more UTF-8. I think
> lots of HTTP users would appreciate a better commitment from HTTP with
> respect to character encoding in headers and the like. What's currently
> there is really just a mess, and should be cleaned up.
>
>
> Regards,    Martin.
>
>

---

From: Mike Belshe <[email-masked]>
Date: Fri, 3 Aug 2012 08:33:58 -0700

One of the charter requirements of HTTP/2, I thought, was interop to
HTTP/1.1 servers.

If so, how would we pass UTF8 headers to HTTP/1.1 servers?

If we can't then we don't need to support them, right?

Mike

On Fri, Aug 3, 2012 at 8:30 AM, Jonathan Ballard <[email-masked]> wrote:

>> HTTP itself, but for the applications on top. It will actually solve some

---

From: Jonathan Ballard <[email-masked]>
Date: Fri, 3 Aug 2012 08:52:05 -0700

Besides entities, there is the pickle goal between HTTP/1.1 and HTTP/2.

On Friday, August 3, 2012, Mike Belshe wrote:

> On Fri, Aug 3, 2012 at 8:30 AM, Jonathan Ballard <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
>>> EBCDIC or some such alternatively) brings HUGE benefits. The same is true
>>> when streamlining from a zoo of character encodings to UTF-8.
>>>

---

From: James M Snell <[email-masked]>
Date: Fri, 3 Aug 2012 08:52:30 -0700

I'll say it again: simply allowing header values to contain UTF-8
characters does not break compatibility with 1.1 because the existing
header definitions for the existing headers would not be changed. The
change would impact new header definitions or applications that are
specifically targeted for 2.0 implementations.

For example, suppose we define a new binary optimized encoding for the host
header that accepts IDN's. That new encoding does not change the details of
the existing Host header in HTTP/1.1. When an intermediary translates the
2.0 message into a 1.1 message, it would convert the IDN into a proper
punycode value within the 1.1 Host header. If the intermediary happens
across some other arbitrary new header using UTF-8 that it does not
understand or know how to translate, it can either ignore it or return a
protocol error. Interoperability is not affected at all.

Limits are a bad thing... aren't they? ;-)

- James

On Fri, Aug 3, 2012 at 8:33 AM, Mike Belshe <[email-masked]> wrote:

> On Fri, Aug 3, 2012 at 8:30 AM, Jonathan Ballard <[email-masked]>wrote:
>

---

From: Zhong Yu <[email-masked]>
Date: Fri, 3 Aug 2012 12:04:05 -0500

Wait... why should HTTP care about the internal structure of a URI? As
far as HTTP is concerned, a URI is an opaque string(though
canonicalization needs to be defined)

The internal structure of a URI is agreed upon by the client app and
the server app. For example, the common ?n1=v1&n2=v2 form is actually
defined by the HTML standard; the way n/v's are escaped is also
defined by HTML. HTTP has no business in it.

http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13.4.1

Zhong YU

On Fri, Aug 3, 2012 at 12:15 AM, Adrien W. de Croy <[email-masked]> wrote:
>>> Doing so, however, does not eliminate the need for %-encoding,
>>
>>
> if we're defining a new binary-safe transport for header values, shouldn't
> we try to avoid all multiplexing / escaping and parsing of strings?
>
> In fact the concept of a single string which is a URI could be deprecated
> for 2.0 and just be sent as individual fields in a request.
>
> gatewaying back to 1.1 would require assembling a URI from the pieces, but
> that should be easy.
>>> and there are a range of possible issues that could make this
>>> problematic.
>>
>>

---

From: James M Snell <[email-masked]>
Date: Fri, 3 Aug 2012 10:11:53 -0700

+1... at the HTTP level, we should not be trying to split up the request
URI like this. Treat it like an opaque string...

Besides, there are MANY applications that mix application specific
parameters into the path as well as the query string, so splitting those up
to protect privacy doesn't really make any sense at all.

- James

On Fri, Aug 3, 2012 at 10:04 AM, Zhong Yu <[email-masked]> wrote:

> shouldn't
> but

---

From: Mike Belshe <[email-masked]>
Date: Fri, 3 Aug 2012 10:15:00 -0700

OK.  More concrete use cases for why we need this would help me understand
better.

>From the User Agent's perspective - how would it know whether a given
HTTP/2 server would be able to grok its UTF8 headers?  Just try and fail?

If you send a UTF8 header, and the origin server can't handle it, does it
cause a failure of the entire request?  Or is this a partial failure?  I
see you classified some as ignorable and others not.  Is the error code
specifically a "couldn't pass UTF8 header" error?  If not, how would the UA
differentiate an error that the server didn't grok the UTF8 vs other server
errors?

These are the new edge cases I was referring to earlier.

Mike

On Fri, Aug 3, 2012 at 8:52 AM, James M Snell <[email-masked]> wrote:

> For example, suppose we define a new binary optimized encoding for the
> host header that accepts IDN's. That new encoding does not change the
> details of the existing Host header in HTTP/1.1. When an intermediary
> translates the 2.0 message into a 1.1 message, it would convert the IDN
> into a proper punycode value within the 1.1 Host header. If the
> intermediary happens across some other arbitrary new header using UTF-8
> that it does not understand or know how to translate, it can either ignore
> it or return a protocol error. Interoperability is not affected at all.
>
>>>> HTML and HTTP were created when the breakthrough of iso-8859-1
>>>> (Latin-1) in Western Europe was predictable (the nascent Web helped to
>>>> unify the Western Europe 'national' 7-bit and 8-bit encodings quite a bit).
>>>>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 3 Aug 2012 10:33:07 -0700

I'm biased against utf-8, because it is trivial at the application layer to
write a function which encodes and/or decodes it.
I see that handling utf-8 adds complexity to the protocol but buys the
protocol nothing. It adds minimal advantage for the entities using the
protocol, and makes intermediaries lives more difficult since they'll have
to do more verification.

Saying that the protocol handles sending a length-delimited string or a
string guaranteed not to include '\n' would be fine, however, as at that
point whatever encoding is used in any particular header value is matter of
the client-app and server, as it should be for things that the protocol
doesn't need to know about.

I'd need a bunch of examples of where this is essential or useful before
I'd be happy supporting it.
-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 3 Aug 2012 10:33:11 -0700

> HTTP/2 server would be able to grok its UTF8 headers?  Just try and fail?
No different than what the user-agent does today: attempt to send the
message and be prepared to deal with failure. Currently, there's no
guarantee that an origin server supports the various Allow-* headers...
some do, some don't. That's ok.

> differentiate an error that the server didn't grok the UTF8 vs other server
The "must-understand" and "must-ignore" aspects of the codepage mechanism
have more to do with translation of headers that use numeric identifiers
into http/1.1 named headers than it does the translation of UTF-8 values.
However, more to the point.. a 400 Bad Request response would be fine for
the most part but a more specific error status could be provided. We
already have response codes like 431 Request Header Too Large, it would not
be unreasonable to have a similar 4xx response that indicates that the
message could not be translated successfully to a downlevel protocol.

---

From: Mike Belshe <[email-masked]>
Date: Fri, 3 Aug 2012 12:43:53 -0700

On Fri, Aug 3, 2012 at 10:33 AM, James M Snell <[email-masked]> wrote:

>> OK.  More concrete use cases for why we need this would help me
>> understand better.
>>

OK - so this means browsers won't use it :-)

So now we need to know which use cases *would* use it :-)

Overall, its a fine idea, I just don't think we need it.

Mike

>>>>> ASCII is not "trivially UTF8." UTF8 lacks the available flow control
>>>>> of ASCII. Any conversion between ASCII and EBCDIC is best done in hardware.
>>>>> We already know the security issue of conversions from unicode to EBCDIC,
>>>>> and I doubt that is something we can scheme here on on-topic.
>>>>>
>>>>>
>>>>>> These days, over 60% of the Web is already in UTF-8, and if you add
>>>>>> in the 20% of pure ASCII which is trivially also UTF-8, it's 80%. All other

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 03 Aug 2012 21:04:24 +0000

------ Original Message ------
From: "Zhong Yu" <[email-masked]>
>Wait... why should HTTP care about the internal structure of a URI?
>

shared caches

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 03 Aug 2012 21:05:53 +0000

------ Original Message ------
From: "James M Snell" <[email-masked]>
To: "Zhong Yu" <[email-masked]>
Cc: "Adrien W. de Croy" <[email-masked]>;"Martin J. Dürst" 
<[email-masked]>;"Poul-Henning Kamp" <[email-masked]>;"Mike 
Belshe" <[email-masked]>;"[email-masked]" <[email-masked]>
Sent: 4/08/2012 5:11:53 a.m.
Subject: Re: FYI... Binary Optimized Header Encoding for SPDY
>+1... at the HTTP level, we should not be trying to split up the 
>request URI like this. Treat it like an opaque string...
>parameters into the path as well as the query string, so splitting 
>those up to protect privacy doesn't really make any sense at all.

I wasn't suggesting it as a privacy thing.

Merely as a way to avoid string parsing, which necessitates escaping, 
which reduces performance, and introduces other problems.

Adrien

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 05 Aug 2012 23:35:05 +1200

>> Wait... why should HTTP care about the internal structure of a URI?

Only if you try to cache along the assumed filesystem hierarchy implicit 
in URLs. Using the absolute URL as an opaque hash key (as Squid does) 
instead of reading any meaning in its syntax avoids all these issues 
completely.

Amo

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 05 Aug 2012 12:31:55 +0000

>instead of reading any meaning in its syntax avoids all these issues 

But opens you up to DoS attacks along the lines of:

	GET /ABCDEF.html
	GET /%41BCDEF.html
	GET /A%42CDEF.html
	...

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 5 Aug 2012 12:23:42 -0400

> But opens you up to DoS attacks along the lines of:

Those are actually the same URL. Just different encodings.

-- 
Website: http://hallambaker.com/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 05 Aug 2012 16:39:46 +0000

In message <CAMm+Lwj_MqNJRkXLVUbwCZdqFru_GwFs9Pe8AB+jYSQNO8jy=[email-masked]>

>Those are actually the same URL. Just different encodings.

That's exactly the point.

Intermediaries need to decode URI and therefore the question of ASCII
vs. UTF8 performance is relevant.

But as I said earlier: I'm not sure if the advantage goes to ASCII
with the need for further encoding, or to UTF8 with no further encoding
needed.

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 06 Aug 2012 11:03:35 +1200

On 06.08.2012 04:39, Poul-Henning Kamp wrote:
> Phillip Hallam-Baker writes:
>>On Sun, Aug 5, 2012 at 8:31 AM, Poul-Henning Kamp wrote:
>

BUT, they don't need to know that A or B expanded to certain special 
directory path macros by the involved apps.

Encoding != Interpretation != Character set.

  * We already have to decode malicious (and just plain stupid 
over-encodings) for ASCII characters.
  * We already have to encode UTF-8 characters passed by some 
clients/servers using UTF-8 internally.
  => There is no new problem created here.

The only relevance this has is that 2.0 native middleware will no 
longer have to *encode* (memory allocate + copy) the raw-UTF emitted by 
dumb sources until it takes a 1.1 hop. Decoding is only a copy with no 
memory allocation - so not optimizing that is sad but does not make 
anything worse.

In short we gain a needed optimization for free, which improves its 
benefit the further 2.0 rolls out.

AYJ

---

From: "Adrien de Croy" <[email-masked]>
Date: Sun, 05 Aug 2012 23:22:57 +0000

personally I see little value in allowing a "safe" character to be 
encoded with %, and specify it has no semantic meaning (that it is 
encoded rather than "native")

Why not simply deprecate such things for 2.0, and when it comes to 
putting together a 1.1 message from a 2.0 message, it needs encoding at 
that stage, and at that stage, there's only 1 allowed way to do it, 
e.g. safe chars MUST NOT be %-encoded etc.

Adrien

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Phillip Hallam-Baker" <[email-masked]>
Cc: "Amos Jeffries" <[email-masked]>;"[email-masked]" 
<[email-masked]>
Sent: 6/08/2012 4:39:46 a.m.
Subject: Re: FYI... Binary Optimized Header Encoding for SPDY

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 06 Aug 2012 11:27:48 +0900

On 2012/08/06 8:22, Adrien de Croy wrote:
> that stage, and at that stage, there's only 1 allowed way to do it, e.g.
> safe chars MUST NOT be %-encoded etc.

It's possible for HTTP to forbid such useless %-encoding. Please note 
however that URIs (and IRIs) allow it, and HTTP can't change that.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 06 Aug 2012 11:41:50 +0900

On 2012/08/04 4:43, Mike Belshe wrote:
> On Fri, Aug 3, 2012 at 10:33 AM, James M Snell<[email-masked]>  wrote:

>> On Fri, Aug 3, 2012 at 10:15 AM, Mike Belshe<[email-masked]>  wrote:
>>

Well, actually, if HTTP/2 says headers can contain UTF-8, then every 
HTTP/2 server must grok them, or not? Of course that doesn't mean that 
every protocol element in every header can use UTF-8, or that where 
UTF-8 is allowed by the definition of the header syntax, every value 
will be accepted by the server. But that's the same for ASCII now.

> OK - so this means browsers won't use it :-)

I'm rather sure that browser makers would prefer to use UTF-8 rather 
than some obscure encoding if HTTP allowed that. Actually, that would 
have been the case 10 and more years ago.

> So now we need to know which use cases *would* use it :-)

Look e.g. at http://tools.ietf.org/html/rfc6266#section-6 and
http://tools.ietf.org/html/rfc5987, which are just a hopeless mess.

> Overall, its a fine idea, I just don't think we need it.

You probably don't think you need it because you are not an user of HTTP 
who has to deal with non-ASCII characters day-in-day-out.

If HTTP used something like Baudot code, and occasionally a bit of ASCII 
or EBCDIC (potentially identified as such, but potentially not) 
depending on various circumstances, I guess you'd have an easier time 
understanding the need.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 06 Aug 2012 16:21:03 +0900

> write a function which encodes and/or decodes it.

It's maybe trivial to write such functions once, but it's a total waste 
of time to write them over and over.

> I see that handling utf-8 adds complexity

What complexity?

> to the protocol but buys the
> protocol nothing.

It doesn't buy the protocol itself much. But it buys the users of the 
protocol a lot.

> It adds minimal advantage for the entities using the

No, it is not fine. First, for most headers, interoperability should be 
between all clients and all servers. Second, it is absolutely no fun for 
client apps developers to solve the same character encoding problem 
again and again. It's just useless work, prone to errors. If you got 
told today that the host header can be in ASCII or EBCDIC, it's just 
between your client and your server, what would you say?

Regards,    Martin.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 6 Aug 2012 09:28:48 -0700

But they don't... it is almost always a single function call where the
function is provided to them.

>> I see that handling utf-8 adds complexity

Reencoding to ASCII for http/1.1, checking that all the characters are
actually displayable, parsing the dang strings in the cases where it does
wish to encode a multi byte character.

I don't see why proxies should have to do this. I don't care, however, so
long as a distinction is made for opaque (user set) headers, at which point
you could use an xor encoding for all I care.
protocol a lot.

Which users? I'm having a hard time imagining why metadata has to be utf-8.

> No, it is not fine. First, for most headers, interoperability should be
between all clients and all servers.

The person who wrote the application also controls the server. They will
interpret the byte stream how they see fit.
It is the other parties to the exchange that won't-- forward and reverse
proxies, for instance.

> Second, it is absolutely no fun for client apps developers to solve the
same character encoding problem again and again. It's just useless work,
prone to errors.

No disagreement there. Am I wrong about such functions already being
provided to such client app writers?

> If you got told today that the host header can be in ASCII or EBCDIC,
it's just between your client and your server, what would you say?

I'd say to ignore EBCDIC in more colorful words :)
-=R

---

From: James M Snell <[email-masked]>
Date: Mon, 6 Aug 2012 10:00:42 -0700

> function is provided to them.
Except when it's not... which happens much more frequently than you may
imagine. There is actually quite a bit of inconsistency in the various,
more complex headers such as Authorization, Content-Disposition, Link, etc.
Developers don't always know when they need to be using Base64, RFC5987,
B-Coding, Q-Coding, %-encoding, or no encoding allowed at all and the
existing specification does not provide any help at all.

> you could use an xor encoding for all I care.
But, I thought writing encoders was a trivial exercise? And the proxy
really shouldn't have to care if the characters are displayable.. in fact,
for the vast overwhelming majority of cases, proxies will simply treat the
headers as opaque and pass them along. It would be excellent if HTTP/2.0
could make things a bit easier for application developers too.

> Which users? I'm having a hard time imagining why metadata has to be utf-8.

No one said it *HAS* to be. Again, simply saying that a particular value
could contain UTF-8 characters doesn't mean we change the specific value
definitions of individual headers. Allowing for UTF-8, however, would
provide benefits to non-english speaking users who are making use of things
like IDN's, Content-Disposition, Link header titles, etc. The typical every
day browser user likely would not see a difference, but developers and
users of HTTP API's would likely see significant advantage.

> It is the other parties to the exchange that won't-- forward and reverse
It is certainly not always true that the person writing the application
controls the server. In fact, I would say quite the opposite in countless
cases.

> provided to such client app writers?
Spotty, at best. Oh sure, Base64 encoders and decoders are a dime a dozen
and there are implementations of other encoders out there. *That* is the
easy part. It's knowing when encoding is required, and which particular
scheme to use in which particular case, and dealing with inconsistent
implementation across servers and clients, and dealing with intermediaries
in the middle that potentially screw things up further.

- James

---

From: Zhong Yu <[email-masked]>
Date: Mon, 6 Aug 2012 12:03:41 -0500

Equivalency between URIs is ultimately decided by the application. For
example majority of apps ignore unknown query parameters, therefore
the attacker can simply append some garbage to expand the key space.
The cache needs some other anti-abuse detection mechanism; trying to
normalize maliciously crafted URIs won't go very far.

Imagine URI was defined to be just an opaque sequence of allowed
chars; there's no standard delimiters and escaping rules, it's all up
to apps. Would that have caused any serious problems? Probably not.

Now URI does define standard delimiters, and the escaping rules that's
then necessary, so we can somewhat recognize components of a URI.
That's good for anyone who finds it useful. But it's only one flat
level. If an app needs deeper structure in URIs, it'll invent its own
delimiting and escaping mechanisms.

Zhong Yu

---

From: Jonathan Ballard <[email-masked]>
Date: Mon, 6 Aug 2012 10:50:58 -0700

The %-encoding is literal compared to entities. If there are no such
literals in the URI then, we can assume the message is ASCII. That is true
to the point we don't see any registered UTF8 headers here unless I skipped
something in HTTP/2.0.

On Monday, August 6, 2012, James M Snell wrote:

>
> On Mon, Aug 6, 2012 at 9:28 AM, Roberto Peon <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
>>
>> On Aug 6, 2012 12:21 AM, Martin J. Dürst <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>>
>> wrote:
>> >

---

From: "Martin Nilsson" <[email-masked]>
Date: Mon, 06 Aug 2012 20:53:17 +0200

The HTTP proxy syntax is an ongoing source for security issues, due to too  
relaxed pattern matching.

GET http://random.com/?facebook.com HTTP/1.1
GET http://[email-masked]/ HTTP/1.1
GET http://facebook.com.random.com/ HTTP/1.1

etc.

/Martin Nilsson

On Fri, 03 Aug 2012 19:04:05 +0200, Zhong Yu <[email-masked]> wrote:

>>> that's
>> can
>> deprecated
>>> say

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 07 Aug 2012 11:08:35 +1200

On 07.08.2012 04:28, Roberto Peon wrote:
> On Aug 6, 2012 12:21 AM, Martin J. Dürst wrote:
>>
>>> I'm biased against utf-8, because it is trivial at the application 
>>> layer
> to

names: File names, user names, country names, domain names, protocol 
names, User-Agent: names, Server: names, Accept-* names, type names ... 
metadata is chock full of names when you start looking at it closely. 
And "for some strange reason" people around the world insist on being 
able to send/receive them in different languages and non-American 
spellings nowdays.

>>> protocol, and makes intermediaries lives more difficult since 
>>> they'll
> have
>>> to do more verification.
>>>
>>> point whatever encoding is used in any particular header value is 
>>> matter
> of

+1000.

> provided to such client app writers?

They are. But the problem I imagine Martin is speaking of is also 
present at that same abstract level. App developers remembering 
over-and-over whether it was serialize before X-encode, then Y-encode, 
or X-encode, de-serialize, Y-encode on the sending handler or receiving 
handler - for which header the data is being appended. Its work to 
remember, its work to lookup, its work to even remember where to lookup.

  In the popular dynamic content frameworks these decisions and 
orderings have to be done by hand in almost every page generator script. 
It is tiring and mistake prone - I just spent most of yesterday 
debugging why one graph out of a whole set would not display - due to 
serlialize data being passed to url-decode first (result being empty 
dataset), encodings only used because the header cant contain a raw 
semi-colon or comma.

Removing the need for developers to do the encoding at all would be a 
great thing. Partially this is a framework problem, partially a protocol 
one. The frameworks allow developers to write custom headers directly 
into the protocol - including customization of the standard headers. I 
don't see that changing. However,...

  For 2.0 they will have to be providing a translation layer in the app 
either changing familiar HTTP/1.1 labels to 2.0 binary ones or providing 
a better header API which enables the framework to take up the encoding 
workload a lot easier. That opens the door to UTF-8 in a safe way.

> I'd say to ignore EBCDIC in more colorful words :)

Then your coding partner decides to use *one* header which offers only 
EBCDIC vs UTF-8... insert identical colour full words here.

AYJ

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 07 Aug 2012 11:40:22 +1200

On 07.08.2012 06:53, Martin Nilsson wrote:
> The HTTP proxy syntax is an ongoing source for security issues, due
> to too  relaxed pattern matching.
>

What "proxy syntax"? All I see there is a bunch of absolute-URI for 
random.com.

Splitting this into pieces on-wire and then re-assembling them into the 
same canonical URL before pattern matching will not result in admin 
people suddenly knowing safer patterns. The ones having trouble now 
already fail to use the tools provided correctly...

This would require re-writing most RFCs to handle new URL syntax, and 
we would have to maintain backward-compatibility and accept these forms 
anyway - which means no gain. Abolishing the second form would be nice 
to avoid credentials leakage in HTTP when its used as a Basic-auth 
substitute. But then again its used for a lot more than basic auth these 
days. Think salting parameter, three-legged auth algorithm name(s), 
domain realm, session ID, etc.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Tue, 7 Aug 2012 17:06:35 -0700

> metadata is chock full of names when you start looking at it closely. And
> "for some strange reason" people around the world insist on being able to
> send/receive them in different languages and non-American spellings nowdays.

I think that allowing something used for names where the character set was
not constrained and whose visual representation is not always defined and
often ambiguous for names where being able to tell uniqueness is an
important security property was a royally stupid decision. Frankly, I could
care less if names were represented in chinese characters, egyption
pictographs, or what have you so long as they're unambiguous.

In any case, that ship has sailed.
I think you have a convincing argument... it still "smells" wrong to me,
and I'll attempt to ponder why. If I don't come up with anything, I won't
argue against it :)

-=R

---

From: Roberto Peon <[email-masked]>
Date: Fri, 10 Aug 2012 15:10:29 -0700

Aha. I had a discussion that reminded me of one of the reasons to dislike
utf-8

If we use ascii, we KNOW we can drop the first bit, which means we get a
possible 12.5% compression trivially.
I'd rather have the complexity in encoding and reduce complexity on
compression. Of course, if we're willing to all implement huffman coding,
this isn't a problem, but that is substantially more complexity than
dropping a bit off every character.

Are we willing to implement huffman coding? It'd certainly make me happy.

-=R