Subject: Set-Cookie and Cookie Optimized Binary Encoding

From: James M Snell <[email-masked]>
Date: Fri, 18 Jan 2013 09:49:15 -0800

Just continuing on with my exploration of improvements we could possibly
make for the binary encoding of various headers. Currently I'm looking at
Cookie and Set-Cookie. Here's what I've come up with so far...

Here's an random example of a simple Set-Cookie taken from the har file
samples Mark has collected:

  Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Sat,
03-Nov-2012 13:04:26 GMT

Looking at this, ignoring the actual key and value for a minute, there is
quite a bit of wasted overhead in this format (Set-Cookie2 is even worse,
but I'll get to that in a bit). The main question is, can we do better and
still provide an easy 1.1 <-> 2.0 transform for Cookies. I definitely think
we can. Here's what I have in mind:

+-----------------------------------------+
|H|S|B|P|M|X|X|X|len(key)|key|len(val)|val|
+-----------------------------------------+
|len(path)|path|len(domain)|domain|expires|
+-----------------------------------------+
|num_params|... repeating param key block |
+-----------------------------------------+

  H = Http-Only Bit
  S = Secure-Only Bit
  B = Binary-Value Bit
  P = Optional-Param List Bit (indicates whether the Set-Cookie includes
the optional parameters block (third row in the record above)
  M = Max-Age Bit - If set, the expires field specifies the Maximum-Age of
the Cookie, otherwise expires is a compact encoded date time (as we've
already discussed). Either way, the expires field is always 4 bytes.
  X = Reserved Flag Bit

  len(key)    -> 1 byte
  len(val)    -> 4 bytes
  len(path)   -> 2 bytes
  len(domain) -> 2 bytes
  expires     -> 4 bytes

For typical Set-Cookies, with no optional parameters (I'll explain this
later) this gives us 14-bytes of overhead. For the example cookie above, we
go from 78 bytes down to 36 bytes. In the Http/1 format, adding HttpOnly
and Secure flags adds at least 18 bytes of additional data. In this
optimized form those require no additional space.

When set, the Binary Value bit indicates that the val field contains
arbitrary binary data as opposed to encoded-text. As Mark and others have
pointed out, allowing for Binary cookie values can lead to trouble with
backwards compatibility with 1.1 applications. To address this I propose
the following: When translating an optimized Http/2 Set-Cookie to Http/1,
if the Binary Value Bit is set, the val is Base64 encoded and a new
"Binary" flag is added to the Http/1 Set-Cookie. For example:

  Set-Cookie: data={base64}; path=/; domain=.example.net; expires=...;
HttpOnly; Secure; Binary

When translating from Http/1 to Http/2, if the Set-Cookie includes the
Binary flag, we can attempt to base64 decode the value and include it
directly in the optimized Http/2 record syntax directly, setting the Binary
Value Bit, and hopefully saving significant amount of space.

For the Cookie Header itself, we would also have a binary syntax..

+-----------------------------------+
|B|XXXXXXX|len(key)|key|len(val)|val|
+-----------------------------------+

  B = Binary-Value Bit

When value is simple text, this record format actually comes out longer
than the existing format, but that's OK, I think. For binary cookie values,
however, this format gives us the option of reducing the record size
significantly. So it's a tradeoff. Here, however, the translation to and
from Http/1 gets a bit more difficult because the Cookie header does not
allow for optional flags the way Set-Cookie does, so we have to get a bit
creative. What I propose is that we use a convention similar to that used
in RFC5987 and append an asterisk * to the key name..

For example, when translating a Http/2 Binary Cookie header to Http/1, it
would be encoded as:

  Cookie: key*={base64}

When translating back to Http/2, if we see the * in the key name, we can
attempt to base64 decode the value and set the Binary Value Flag in the
Http/2 Binary Cookie header. It's not perfect, but it ought to work.

Getting back to Set-Cookie, there's the part about the optional parameters.
While the current Set-Cookie format is pretty stable, there have been times
in the past (Set-Cookie2) where additional parameters were specified.
Set-Cookie is one of those things where it may make sense to support some
long term extensibility. For that purpose, I have included 3 additional
reserved flag bits and an optional block of extension key=value pairs that
can be tacked on at the end. If the Optional Parameter Bit is set, we know
to go looking for those additional parameters at the end. The value of
those parameters must be text, keys are text. len(key) = 1 byte, len(val) =
4 bytes. This gives a significant amount of latitude for extension and
translates fine to the existing Http/1 Set-Cookie syntax.

With these changes, I estimate that we can reduce the overall overhead of
transmitting Set-Cookie and Cookie values by around 50% on average, while
maintaining compatibility with Http/1.

Thoughts?

---

From: Martin Thomson <[email-masked]>
Date: Fri, 18 Jan 2013 10:04:36 -0800

On 18 January 2013 09:49, James M Snell <[email-masked]> wrote:
> When translating an optimized Http/2 Set-Cookie to Http/1, if the Binary
> Value Bit is set, the val is Base64 encoded and a new "Binary" flag is added
> to the Http/1 Set-Cookie.

This, I have no problem with, it merely indicates that the header

>  Cookie: key*={base64}

Having the name change is surprising.  Why is it so critical for an
HTTP/1.1 encoded value to be encoded in the binary form in HTTP/2.0?

---

From: James M Snell <[email-masked]>
Date: Fri, 18 Jan 2013 10:34:04 -0800

I'm not convinced it is critical really. I included that basically in case
others think lossless bidirectional transformation between 1.1 and 2.0 it's
essential.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 18 Jan 2013 11:59:11 -0800

> others think lossless bidirectional transformation between 1.1 and 2.0 it's

Lossless, yes.  Optimized for compression, maybe not.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 10:55:00 +1100

Keep in mind that Set-Cookie is an extensible field, and we need to leave that gate open. Also, the BNF in rfc6265 is very loose, which is going to complicate things a bit...

On 19/01/2013, at 4:49 AM, James M Snell <[email-masked]> wrote:

> Just continuing on with my exploration of improvements we could possibly make for the binary encoding of various headers. Currently I'm looking at Cookie and Set-Cookie. Here's what I've come up with so far...
> 
> Here's an random example of a simple Set-Cookie taken from the har file samples Mark has collected:
> 
>   Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Sat, 03-Nov-2012 13:04:26 GMT
> 
> Looking at this, ignoring the actual key and value for a minute, there is quite a bit of wasted overhead in this format (Set-Cookie2 is even worse, but I'll get to that in a bit). The main question is, can we do better and still provide an easy 1.1 <-> 2.0 transform for Cookies. I definitely think we can. Here's what I have in mind:
> 
>   P = Optional-Param List Bit (indicates whether the Set-Cookie includes the optional parameters block (third row in the record above)
>   M = Max-Age Bit - If set, the expires field specifies the Maximum-Age of the Cookie, otherwise expires is a compact encoded date time (as we've already discussed). Either way, the expires field is always 4 bytes.
>   X = Reserved Flag Bit
> 
>   len(key)    -> 1 byte
>   len(val)    -> 4 bytes
>   len(path)   -> 2 bytes
>   len(domain) -> 2 bytes
>   expires     -> 4 bytes
> 
> For typical Set-Cookies, with no optional parameters (I'll explain this later) this gives us 14-bytes of overhead. For the example cookie above, we go from 78 bytes down to 36 bytes. In the Http/1 format, adding HttpOnly and Secure flags adds at least 18 bytes of additional data. In this optimized form those require no additional space. 
> 
> When set, the Binary Value bit indicates that the val field contains arbitrary binary data as opposed to encoded-text. As Mark and others have pointed out, allowing for Binary cookie values can lead to trouble with backwards compatibility with 1.1 applications. To address this I propose the following: When translating an optimized Http/2 Set-Cookie to Http/1, if the Binary Value Bit is set, the val is Base64 encoded and a new "Binary" flag is added to the Http/1 Set-Cookie. For example:
> 
>   Set-Cookie: data={base64}; path=/; domain=.example.net; expires=...; HttpOnly; Secure; Binary
> 
> When translating from Http/1 to Http/2, if the Set-Cookie includes the Binary flag, we can attempt to base64 decode the value and include it directly in the optimized Http/2 record syntax directly, setting the Binary Value Bit, and hopefully saving significant amount of space. 
> 
> When value is simple text, this record format actually comes out longer than the existing format, but that's OK, I think. For binary cookie values, however, this format gives us the option of reducing the record size significantly. So it's a tradeoff. Here, however, the translation to and from Http/1 gets a bit more difficult because the Cookie header does not allow for optional flags the way Set-Cookie does, so we have to get a bit creative. What I propose is that we use a convention similar to that used in RFC5987 and append an asterisk * to the key name..
> 
> For example, when translating a Http/2 Binary Cookie header to Http/1, it would be encoded as:
> 
>   Cookie: key*={base64}
> 
> When translating back to Http/2, if we see the * in the key name, we can attempt to base64 decode the value and set the Binary Value Flag in the Http/2 Binary Cookie header. It's not perfect, but it ought to work.
> 
> Getting back to Set-Cookie, there's the part about the optional parameters. While the current Set-Cookie format is pretty stable, there have been times in the past (Set-Cookie2) where additional parameters were specified. Set-Cookie is one of those things where it may make sense to support some long term extensibility. For that purpose, I have included 3 additional reserved flag bits and an optional block of extension key=value pairs that can be tacked on at the end. If the Optional Parameter Bit is set, we know to go looking for those additional parameters at the end. The value of those parameters must be text, keys are text. len(key) = 1 byte, len(val) = 4 bytes. This gives a significant amount of latitude for extension and translates fine to the existing Http/1 Set-Cookie syntax. 
> 
> With these changes, I estimate that we can reduce the overall overhead of transmitting Set-Cookie and Cookie values by around 50% on average, while maintaining compatibility with Http/1. 
> 
> Thoughts?
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Sat, 19 Jan 2013 16:11:09 -0800

Yes, indeed. I'm wondering if there's a reasonable balance we can support
here... That is, provide for backwards compatibility for most but not all
1.1 features and optimize for the most commonly used bits. Restricting set
cookie syntax could be one of those compromises.
On Jan 19, 2013 3:55 PM, "Mark Nottingham" <[email-masked]> wrote:

> Keep in mind that Set-Cookie is an extensible field, and we need to leave
> that gate open. Also, the BNF in rfc6265 is very loose, which is going to
> complicate things a bit...
>
>
> > Looking at this, ignoring the actual key and value for a minute, there
> is quite a bit of wasted overhead in this format (Set-Cookie2 is even
> worse, but I'll get to that in a bit). The main question is, can we do
> better and still provide an easy 1.1 <-> 2.0 transform for Cookies. I
> definitely think we can. Here's what I have in mind:
> >
> >   M = Max-Age Bit - If set, the expires field specifies the Maximum-Age
> of the Cookie, otherwise expires is a compact encoded date time (as we've
> > Getting back to Set-Cookie, there's the part about the optional
> parameters. While the current Set-Cookie format is pretty stable, there
> have been times in the past (Set-Cookie2) where additional parameters were
> specified. Set-Cookie is one of those things where it may make sense to
> support some long term extensibility. For that purpose, I have included 3
> additional reserved flag bits and an optional block of extension key=value
> pairs that can be tacked on at the end. If the Optional Parameter Bit is
> set, we know to go looking for those additional parameters at the end. The
> value of those parameters must be text, keys are text. len(key) = 1 byte,
> len(val) = 4 bytes. This gives a significant amount of latitude for
> extension and translates fine to the existing Http/1 Set-Cookie syntax.
> >
> > With these changes, I estimate that we can reduce the overall overhead
> of transmitting Set-Cookie and Cookie values by around 50% on average,
> while maintaining compatibility with Http/1.
> >
> > Thoughts?
> >
>
> --

---

From: James M Snell <[email-masked]>
Date: Sat, 19 Jan 2013 17:31:57 -0800

Part of that escape hatch ought to be a bit in the encoded header that
indicates a binary or text value. If it's text, then the 1.1 format is
used. If it's binary, the optimized format is used and we get a closest we
can get compromise on the translation from one to the other.
On Jan 19, 2013 5:28 PM, "Nico Williams" <[email-masked]> wrote:

> On Sat, Jan 19, 2013 at 6:11 PM, James M Snell <[email-masked]> wrote:
> We should try to optimize bridging between HTTP/1.* and HTTP/2.0, but
> we should not penalize a pure HTTP/2.0 future in the process.  The
> obvious thing to do is to provide an escape hatch for contents we
> don't optimize and not try to optimize absolutely everything.
>
> Nico
> --
>

---

From: Nico Williams <[email-masked]>
Date: Sat, 19 Jan 2013 19:42:30 -0600

On Sat, Jan 19, 2013 at 7:31 PM, James M Snell <[email-masked]> wrote:
> indicates a binary or text value. If it's text, then the 1.1 format is used.
> If it's binary, the optimized format is used and we get a closest we can get
> compromise on the translation from one to the other.

Exactly.

---

From: Nico Williams <[email-masked]>
Date: Sat, 19 Jan 2013 19:28:10 -0600

Nico
--