Subject: Type codecs within hpack

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 09:00:00 -0700

With the assumption that hpack is what we'll ultimately end up
sticking with for header encoding, I wanted to take a moment to
illustrate how the binary type codecs would look within the hpack
encoding... (note, this email is **NOT** discussing the alternative
header compression I describe in my separate I-D.. this is talking
about applying the value type encodings to hpack).

It would be very helpful if the various implementers would give some
kind of indication about whether they'd be willing to implement these
encodings and, if so, when.

First, a quick note: The type codecs would be defined independently of
the compression mechanism itself... that is, just as Roberto's been
wanting, we can separate header value type details out of hpack
entirely so that hpack can deal specifically with header compression.

Specifically, hpack would be updated such that it would not define a
specific encoding for header field values. As far as the basic hpack
mechanism is concerned, the value encoding would be an opaque octet
sequence.

Literal Header without Indexing (Indexed Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+-------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

Literal Header without Indexing (New Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +-------------------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

Literal Header with Incremental Indexing (Indexed Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+-------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

Literal Header with Incremental Indexing (New Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +-------------------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

Literal Header with Substitution Indexing (Indexed Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +-------------------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

Literal Header with Substitution Indexing (New Name):

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+-----------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +-------------------------------+
   |    Substituted Index (8+)     |
   +-------------------------------+
   |      Value Encoding (8+)      |
   +-------------------------------+

There are five possible value encodings which manifest two basic
patterns... The value types are:

1. UTF-8
2. Legacy
3. Opaque
4. Integer
5. Timestamp

On the wire these look like:

UTF-8
     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Integer

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Timestamp

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Legacy
     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +-------------------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Opaque
     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +-------------------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Notice that the UTF-8, Legacy and Opaque type encodings are identical
other than the leading three-bits. Likewise, the Integer and Timestamp
encodings are identical other than the leading three bits.

For UTF-8, Legacy and Opaque, the Value Length is encoding as an
integer with a 5-bit prefix.

For Integer, the Value is encoded as an integer with a 5-bit prefix.
Negative or fractional values cannot be represented. Theoretically
there is no upper value limit to this encoding, however, it would
likely be good to recommend that only values up to 64-bits are
encoded.

For Timestamp, the number of milliseconds since the epoch is encoded
as an integer with a 5-bit prefix. Dates before the epoch cannot be
represented.

If we put these types together with hpack, this is what we end up with:

Literal Header without Indexing (Indexed Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header without Indexing (Indexed Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header without Indexing (Indexed Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header without Indexing (Indexed Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header without Indexing (Indexed Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Literal Header without Indexing (New Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header without Indexing (New Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header without Indexing (New Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header without Indexing (New Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header without Indexing (New Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 1 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (Indexed Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (Indexed Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Incremental Indexing (Indexed Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Incremental Indexing (Indexed Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (Indexed Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Index (5+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (New Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (New Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Incremental Indexing (New Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Incremental Indexing (New Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Incremental Indexing (Indexed Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |         0         |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (Indexed Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (Indexed Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Substitution Indexing (Indexed Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Substitution Indexing (Indexed Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (Indexed Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |      Index (6+)       |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (New Name), UTF-8 Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (New Name), Integer Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 0 | 1 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Substitution Indexing (New Name), Timestamp Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 0 | 1 | 0 |    Value (5+)     |
   +---+---+---+---+---+---+---+---+

Literal Header with Substitution Indexing (New Name), Legacy Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 0 | 0 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value String (Length Octets)  |
   +-------------------------------+

Literal Header with Substitution Indexing (New Name), Opaque Value

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 0 | 0 |           0           |
   +---+---+---+-------------------+
   |       Name Length (8+)        |
   +-------------------------------+
   |  Name String (Length octets)  |
   +---+---+-----------------------+
   |    Substituted Index (8+)     |
   +---+---+---+---+---+---+---+---+
   | 1 | 1 | 1 | Value Length (5+) |
   +---+---+---+-------------------+
   | Value Octets (Length Octets)  |
   +-------------------------------+

For backwards compatibility with HTTP/1, the following conversion
rules would apply when translating an HTTP/2 request into HTTP/1

1. UTF-8 value encodings is converted to ASCII, with all non-ASCII
codepoints pct-encoded.
2. Legacy value encodings are passed through without any translation
3. Opaque values are Base64 encoded
4. Integer values are converted into the ASCII representation of the
number (e.g. 1234 = "1234")
5. Timestamp values are converted to the HTTP-date equivalent string,
with millisecond precision lost.

There is no normative translation from HTTP/1 to HTTP/2. All HTTP/1
header field values would be passed without translation using the
Legacy Value Encoding.

Header fields would have to be specifically defined at the http layer
to use the new value encodings. The definitions of existing HTTP/1
header fields remain the same unless those headers are specifically
redefined.

Some of the existing standard headers that we know can be redefined include:

* :status (integer)
* :host (legacy or utf-8)
* :path (legacy or utf-8)
* content-length (legacy or integer)
* date (legacy or timestamp)
* max-forwards (legacy or integer)
* retry-after (legacy, integer or timestamp)
* if-modified-since (legacy or timestamp)
* if-unmodified-since (legacy or timestamp)
* last-modified (legacy or timestamp)
* age (legacy or integer)
* expires (legacy, integer or timestamp)
* etag (legacy or opaque... an opaque etag is always "strong", never "weak")

There is a question about whether we really need to have any
distinction between UTF-8, Legacy and Opaque. First, it's important to
note that, with the exception of the first three bits, these three
types serialize exactly the same on the wire and are handled in
exactly the same way by the header compression mechanism. The type
distinction comes into play only when we want to use the value.

Allowing for UTF-8 gives developers the ability to use extended
characters.. for instance, we could use IRI's directly within our
requests without any translation to ASCII URI's. That means, passing
:host with an IDN rather than punycode equivalent; and passing :path
with extended characters without being required to do additional
pct-encoding.

Allowing for Legacy gives us a fallback for HTTP/1 compatibility.
There are very few enforced rules in HTTP/1 for values and we need to
have a way of indicating that the value being passed through is
untouched relative to HTTP/1 and follows all the same rules at HTTP/1.

Allowing for Opaque values gives us the opportunity to achieve highly
efficient encodings for things that, in the past, haven't been encoded
very efficiently. It allows us to pass raw binary sequences that ought
not to be interpreted as text necessarily.. avoiding the need to apply
additional base64 or hex encoding.

Lastly, I know there have been some discussions around structured
alternatives for timestamp encoding. Honestly, I've played around with
a number of variations and kept coming to the same conclusion:
milliseconds since the epoch encoded as a variable length integer
really is good enough. It keeps things simple while delivering a
feature many appdevs have been asking for for a very long time.

---

From: Mike Bishop <[email-masked]>
Date: Fri, 23 Aug 2013 21:32:35 +0000

Integer and Timestamp are defined with the same leading three bits as well.  I assume that's a typo.

-----Original Message-----
From: James M Snell [mailto:[email-masked]] 
Sent: Friday, August 23, 2013 9:00 AM
To: [email-masked]
Subject: Type codecs within hpack

With the assumption that hpack is what we'll ultimately end up sticking with for header encoding, I wanted to take a moment to illustrate how the binary type codecs would look within the hpack encoding... (note, this email is **NOT** discussing the alternative header compression I describe in my separate I-D.. this is talking about applying the value type encodings to hpack).

It would be very helpful if the various implementers would give some kind of indication about whether they'd be willing to implement these encodings and, if so, when.

First, a quick note: The type codecs would be defined independently of the compression mechanism itself... that is, just as Roberto's been wanting, we can separate header value type details out of hpack entirely so that hpack can deal specifically with header compression.

Specifically, hpack would be updated such that it would not define a specific encoding for header field values. As far as the basic hpack mechanism is concerned, the value encoding would be an opaque octet sequence.

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

There are five possible value encodings which manifest two basic patterns... The value types are:

1. UTF-8
2. Legacy
3. Opaque
4. Integer
5. Timestamp

UTF-8
     0   1   2   3   4   5   6   7

Integer

     0   1   2   3   4   5   6   7

Timestamp

     0   1   2   3   4   5   6   7

Legacy
     0   1   2   3   4   5   6   7

Opaque
     0   1   2   3   4   5   6   7

Notice that the UTF-8, Legacy and Opaque type encodings are identical other than the leading three-bits. Likewise, the Integer and Timestamp encodings are identical other than the leading three bits.

For UTF-8, Legacy and Opaque, the Value Length is encoding as an integer with a 5-bit prefix.

For Integer, the Value is encoded as an integer with a 5-bit prefix.
Negative or fractional values cannot be represented. Theoretically there is no upper value limit to this encoding, however, it would likely be good to recommend that only values up to 64-bits are encoded.

For Timestamp, the number of milliseconds since the epoch is encoded as an integer with a 5-bit prefix. Dates before the epoch cannot be represented.

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

     0   1   2   3   4   5   6   7

For backwards compatibility with HTTP/1, the following conversion rules would apply when translating an HTTP/2 request into HTTP/1

1. UTF-8 value encodings is converted to ASCII, with all non-ASCII codepoints pct-encoded.
2. Legacy value encodings are passed through without any translation 3. Opaque values are Base64 encoded 4. Integer values are converted into the ASCII representation of the number (e.g. 1234 = "1234") 5. Timestamp values are converted to the HTTP-date equivalent string, with millisecond precision lost.

There is no normative translation from HTTP/1 to HTTP/2. All HTTP/1 header field values would be passed without translation using the Legacy Value Encoding.

Header fields would have to be specifically defined at the http layer to use the new value encodings. The definitions of existing HTTP/1 header fields remain the same unless those headers are specifically redefined.

* :status (integer)

There is a question about whether we really need to have any distinction between UTF-8, Legacy and Opaque. First, it's important to note that, with the exception of the first three bits, these three types serialize exactly the same on the wire and are handled in exactly the same way by the header compression mechanism. The type distinction comes into play only when we want to use the value.

Allowing for UTF-8 gives developers the ability to use extended characters.. for instance, we could use IRI's directly within our requests without any translation to ASCII URI's. That means, passing :host with an IDN rather than punycode equivalent; and passing :path with extended characters without being required to do additional pct-encoding.

Allowing for Legacy gives us a fallback for HTTP/1 compatibility.
There are very few enforced rules in HTTP/1 for values and we need to have a way of indicating that the value being passed through is untouched relative to HTTP/1 and follows all the same rules at HTTP/1.

Allowing for Opaque values gives us the opportunity to achieve highly efficient encodings for things that, in the past, haven't been encoded very efficiently. It allows us to pass raw binary sequences that ought not to be interpreted as text necessarily.. avoiding the need to apply additional base64 or hex encoding.

Lastly, I know there have been some discussions around structured alternatives for timestamp encoding. Honestly, I've played around with a number of variations and kept coming to the same conclusion:
milliseconds since the epoch encoded as a variable length integer really is good enough. It keeps things simple while delivering a feature many appdevs have been asking for for a very long time.

---

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 14:36:36 -0700

Yes, cut n paste error on my part.
On Aug 23, 2013 2:32 PM, "Mike Bishop" <[email-masked]> wrote:

> With the assumption that hpack is what we'll ultimately end up sticking
> with for header encoding, I wanted to take a moment to illustrate how the
> binary type codecs would look within the hpack encoding... (note, this
> email is **NOT** discussing the alternative header compression I describe
> in my separate I-D.. this is talking about applying the value type
> encodings to hpack).
>
> It would be very helpful if the various implementers would give some kind
> of indication about whether they'd be willing to implement these encodings
> and, if so, when.
>
> First, a quick note: The type codecs would be defined independently of the
> compression mechanism itself... that is, just as Roberto's been wanting, we
> can separate header value type details out of hpack entirely so that hpack
> can deal specifically with header compression.
>
> Negative or fractional values cannot be represented. Theoretically there
> is no upper value limit to this encoding, however, it would likely be good
> to recommend that only values up to 64-bits are encoded.
>
> For Timestamp, the number of milliseconds since the epoch is encoded as an
> integer with a 5-bit prefix. Dates before the epoch cannot be represented.
>
> For backwards compatibility with HTTP/1, the following conversion rules
> would apply when translating an HTTP/2 request into HTTP/1
>
> 2. Legacy value encodings are passed through without any translation 3.
> Opaque values are Base64 encoded 4. Integer values are converted into the
> ASCII representation of the number (e.g. 1234 = "1234") 5. Timestamp values
> are converted to the HTTP-date equivalent string, with millisecond
> precision lost.
>
> There is no normative translation from HTTP/1 to HTTP/2. All HTTP/1 header
> field values would be passed without translation using the Legacy Value
> Encoding.
>
> Header fields would have to be specifically defined at the http layer to
> use the new value encodings. The definitions of existing HTTP/1 header
> fields remain the same unless those headers are specifically redefined.
>
> There is a question about whether we really need to have any distinction
> between UTF-8, Legacy and Opaque. First, it's important to note that, with
> the exception of the first three bits, these three types serialize exactly
> the same on the wire and are handled in exactly the same way by the header
> compression mechanism. The type distinction comes into play only when we
> want to use the value.
>
> characters.. for instance, we could use IRI's directly within our requests
> without any translation to ASCII URI's. That means, passing :host with an
> IDN rather than punycode equivalent; and passing :path with extended
> characters without being required to do additional pct-encoding.
>
> There are very few enforced rules in HTTP/1 for values and we need to have
> a way of indicating that the value being passed through is untouched
> relative to HTTP/1 and follows all the same rules at HTTP/1.
>
> efficient encodings for things that, in the past, haven't been encoded very
> efficiently. It allows us to pass raw binary sequences that ought not to be
> interpreted as text necessarily.. avoiding the need to apply additional
> base64 or hex encoding.
>
> alternatives for timestamp encoding. Honestly, I've played around with a
> number of variations and kept coming to the same conclusion:
> milliseconds since the epoch encoded as a variable length integer really
> is good enough. It keeps things simple while delivering a feature many
> appdevs have been asking for for a very long time.
>
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 23 Aug 2013 14:46:00 -0700

> 5. Timestamp

With only 5 values, you could regain 1 bit on three of those values,
or 2 bits on one.

0, 100, 101, 110, 111 or 00, 01, 10, 110, 111

The first option might work well if you think that Legacy is going to
be very common.  Otherwise you could split UTF-8/Legacy/Opaque three
ways.

---

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 14:48:40 -0700

I went with this so we could reserve a few additional types but I'm not
that particular about it. Whichever works.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 23 Aug 2013 14:53:47 -0700

On 23 August 2013 14:48, James M Snell <[email-masked]> wrote:
> I went with this so we could reserve a few additional types but I'm not that
> particular about it. Whichever works.

Extensibility is overrated.  You only need a few points of articulation.

I wonder if you could possibly drop Integer from the list (or even
merge it with Timestamp).

---

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 14:58:55 -0700

Integer is helpful for things like content-length. I'd rather keep that
one. The only difference between it and timestamp is one bit so merging
them wouldn't do much, but we could just dump timestamp and just say that
timestamp headers use integer. But that makes translation to http/1
harder.  You'd have to know in advance that any given header is a http-date
or number.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 23 Aug 2013 15:11:05 -0700

On 23 August 2013 14:58, James M Snell <[email-masked]> wrote:
> But that makes translation to http/1 harder.  You'd have to know in advance
> that any given header is a http-date or number.

At the point you move to typed headers, I think that you need to
accept this possibility.  Solving the problem for dates specifically
still leaves the opaque type.

---

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 15:19:42 -0700

The opaque type does not have an existing standard analog in http/1, the
translation path is to simply base64 encode those.

---

From: Robert Collins <[email-masked]>
Date: Sat, 24 Aug 2013 10:21:36 +1200

On 24 Aug 2013 04:02, "James M Snell"
> header field values would be passed without translation using the

Surely PCT encoding has to be undone or 1 to 2 to 1 will double encode.

-rob

---

From: James M Snell <[email-masked]>
Date: Fri, 23 Aug 2013 16:01:16 -0700

1 to 2 would cause the header values to encode as legacy encodings without
any transformation. When those go from 2 to 1, legacy octets are encoded as
is without transformation. So there is no risk of double encoding in the
common case. Impls that choose to transform when they decide it's possible
will have to understand the possible issues that come along with that.