Subject: YAC Proposal

From: <[email-masked]>
Date: Wed, 2 Jul 2014 19:51:56 +0000

I'd like to propose yet another option to Mark's list of options for dealing with the "CONTINUATION issue".

Make it an extension.

In NYC several people mentioned that if we add extensibility, we should have an extension(s) right from the start that are used.  CONTINUATION IMO is a good option for an extension.

Here is the CONTINUATION extension draft:
http://www.ietf.org/id/draft-johndoe-http2-large-header-blocks-00.txt

Here is the pull request to remove CONTINUATION from the core h2 draft:
https://github.com/http2/http2-spec/pull/547

-keith

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 15:06:08 -0500

This proposal addresses my concerns, so would of course be in favor. 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 02 Jul 2014 20:45:50 +0000

>with the "CONTINUATION issue".

I can certainly live with that.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Adrian Cole <[email-masked]>
Date: Wed, 2 Jul 2014 15:21:33 -0700

sounds good

---

From: Nicholas Hurley <[email-masked]>
Date: Wed, 2 Jul 2014 16:22:01 -0700

This sounds like a non-starter, to me. As mentioned in the other thread, I
prefer either the status quo, or failing that a stateless CONTINUATION.
This seems worse to me than either of those options. With a stateless
CONTINUATION, in the rare occasions when I'll encounter a compressed header
block > 16k, I can roll back just one step in my HPACK compression, and
then continue on sending an uncompressed form of the headers. With
CONTINUATION as an extension, when the other side doesn't advertise
support, I would either have to either roll back the entire HPACK state
that I changed when trying to compress the headers and generate a
RST_STREAM (likely synthesizing an 413 or some other error in the process
for my own side), or I'd have to send a GOAWAY, and dial back. Neither of
those options is in any way more appealing than stateless CONTINUATION
(which I'm still not entirely sold on to begin with!)

--
Peace,
  -Nick

On Wed, Jul 2, 2014 at 12:51 PM, <[email-masked]> wrote:

> In NYC several people mentioned that if we add extensibility, we should
> have an extension(s) right from the start that are used.  CONTINUATION IMO
> is a good option for an extension.
>
> This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.
>
>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 2 Jul 2014 17:21:48 -0700

One cannot remove continuation without addressing backwards compatibility
with HTTP/1.1 deployments sending large headers.

The status quo, or a variation which improves the state machine around
continuation (e.g. by moving the flags to the last frame of the headers
block) makes the most sense to me
-=R

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 19:32:22 -0500

Compatibility isn’t really affected here. You simply use the extension if you wish to accept > 16KB of compressed header data. HTTP/1.1 servers were not required to support it any more than they are today.

On Jul 2, 2014, at 7:21 PM, Roberto Peon <[email-masked]> wrote:

> One cannot remove continuation without addressing backwards compatibility with HTTP/1.1 deployments sending large headers.
> 
> The status quo, or a variation which improves the state machine around continuation (e.g. by moving the flags to the last frame of the headers block) makes the most sense to me
> -=R
> 
> 
> On Wed, Jul 2, 2014 at 4:22 PM, Nicholas Hurley <[email-masked]> wrote:
> This sounds like a non-starter, to me. As mentioned in the other thread, I prefer either the status quo, or failing that a stateless CONTINUATION. This seems worse to me than either of those options. With a stateless CONTINUATION, in the rare occasions when I'll encounter a compressed header block > 16k, I can roll back just one step in my HPACK compression, and then continue on sending an uncompressed form of the headers. With CONTINUATION as an extension, when the other side doesn't advertise support, I would either have to either roll back the entire HPACK state that I changed when trying to compress the headers and generate a RST_STREAM (likely synthesizing an 413 or some other error in the process for my own side), or I'd have to send a GOAWAY, and dial back. Neither of those options is in any way more appealing than stateless CONTINUATION (which I'm still not entirely sold on to begin with!)
> 
> --
> Peace,
>   -Nick
> 
>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 2 Jul 2014 17:33:09 -0700

Is the extension guaranteed to be supported everywhere?
If not, then HTTP2 doesn't offer HTTP1's semantics.
-=R

On Wed, Jul 2, 2014 at 5:32 PM, Jason Greene <[email-masked]>
wrote:

> Compatibility isn’t really affected here. You simply use the extension if
> you wish to accept > 16KB of compressed header data. HTTP/1.1 servers were
> not required to support it any more than they are today.
>
> > One cannot remove continuation without addressing backwards
> compatibility with HTTP/1.1 deployments sending large headers.
> >
> > This sounds like a non-starter, to me. As mentioned in the other thread,
> I prefer either the status quo, or failing that a stateless CONTINUATION.

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 19:38:45 -0500

Sure every server that supports > 16KB of compressed header data will implement the extension, or perhaps another extension :) Either way the end result is the same.

On Jul 2, 2014, at 7:33 PM, Roberto Peon <[email-masked]> wrote:

> On Wed, Jul 2, 2014 at 5:32 PM, Jason Greene <[email-masked]> wrote:

---

From: Roberto Peon <[email-masked]>
Date: Wed, 2 Jul 2014 17:42:36 -0700

I wonder how many servers don't. I don't know of any major servers with
such a cap, but I'm surely in the dark.
Do we know how many place a cap on the size of response headers?

If just about everything supports it, why would it be an extension?
-=R

On Wed, Jul 2, 2014 at 5:38 PM, Jason Greene <[email-masked]>
wrote:

> Sure every server that supports > 16KB of compressed header data will
> implement the extension, or perhaps another extension :) Either way the end
> result is the same.
>
> > Compatibility isn’t really affected here. You simply use the extension
> if you wish to accept > 16KB of compressed header data. HTTP/1.1 servers
> were not required to support it any more than they are today.
> >
> > > This sounds like a non-starter, to me. As mentioned in the other
> thread, I prefer either the status quo, or failing that a stateless
> CONTINUATION. This seems worse to me than either of those options. With a
> stateless CONTINUATION, in the rare occasions when I'll encounter a
> compressed header block > 16k, I can roll back just one step in my HPACK
> compression, and then continue on sending an uncompressed form of the
> headers. With CONTINUATION as an extension, when the other side doesn't
> advertise support, I would either have to either roll back the entire HPACK
> state that I changed when trying to compress the headers and generate a
> > > In NYC several people mentioned that if we add extensibility, we
> should have an extension(s) right from the start that are used.
>  CONTINUATION IMO is a good option for an extension.
> > >
> > > This email message is intended only for the use of the named
> recipient. Information contained in this email message and its attachments
> may be privileged, confidential and protected from disclosure. If you are
> not the intended recipient, please do not read, copy, use or disclose this

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 20:06:20 -0500

IIRC several implementations spoke up with having an 8K limit. Now in my case I have a 50k limit (don’t ask!), however, since the continuations feature doesn’t let me set a limit our http/2 implementation would likely default to 16KB (at least when proxying), and enable the extension when raised. Then at least the 99.8% is handled without HOL problems.

Now, my preference would still be to fix continuations in some way that the HOL issues are mitigated, but I’ll take what I can get.

No idea on response limit. I can tell you that my implementation has no limit on response size. However when used as a proxy the request size takes over. Things are a bit different in HTTP/1.1 though because we can efficiently copy between streams. Can’t really do that in HTTP/2.

On Jul 2, 2014, at 7:42 PM, Roberto Peon <[email-masked]> wrote:

> I wonder how many servers don't. I don't know of any major servers with such a cap, but I'm surely in the dark.
> On Wed, Jul 2, 2014 at 5:38 PM, Jason Greene <[email-masked]> wrote:

---

From: Mike Bishop <[email-masked]>
Date: Thu, 3 Jul 2014 01:16:29 +0000

When I asked Osama about this, his immediate concern was about response headers as well.  We cap incoming headers at 16KB (both individual size and total header set) by default, with config options to increase those as far as 64KB and 16MB if the administrator wants.  But we have no code that checks the size of response headers – we let the client discard the response if they don’t like it.

That led to how we would deal with an app that tried to send something that used to be valid and now isn’t.

From: Roberto Peon [mailto:[email-masked]]
Sent: Wednesday, July 2, 2014 5:43 PM
To: Jason Greene
Cc: Nicholas Hurley; [email-masked]; IETF HTTP WG
Subject: Re: YAC Proposal

On Wed, Jul 2, 2014 at 5:38 PM, Jason Greene <[email-masked]<mailto:[email-masked]>> wrote:
Sure every server that supports > 16KB of compressed header data will implement the extension, or perhaps another extension :) Either way the end result is the same.

On Jul 2, 2014, at 7:33 PM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:

> On Wed, Jul 2, 2014 at 5:32 PM, Jason Greene <[email-masked]<mailto:[email-masked]>> wrote:
> On Jul 2, 2014, at 7:21 PM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
>
> > On Wed, Jul 2, 2014 at 4:22 PM, Nicholas Hurley <[email-masked]<mailto:[email-masked]>> wrote:
> > On Wed, Jul 2, 2014 at 12:51 PM, <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 03 Jul 2014 13:42:25 +1200

If we are required to deal with CONTINUATION instead of jumbo frames I 
would much rather deal with this than the status quo definition.

Some nits on this proposal:

* IMHO, the END_STREAM flag should be allowed/moved to the last 
CONTINUATION. Both endpoints have explicitly negotiated this feature, so 
moving the END_STREAM flag is already agreed between them. There is no 
reason for it to be set on the HEADERS at all and it just adds confusion 
over where the stream actually ends from a naive framing viewpoint.

* Would it make sense having the SETTINGS_ENABLE_CONTINUATION value 
indicate the count of CONTINUATION frames that will be accepted? That 
would allow a rough 16K resolution on acceptible compressed size limit 
to be negotiated.

PS. reminders to everyone objecting so far:

1) the primary use case presented for large headers is Kerberos tickets 
in Negotiate connection-based authentication. That already requires 
dedicated end-to-end "connection pinning" implementation support in 
HTTP/2 (HTTP/1 gets away with closing connections after each request - 
which is not an option now). In light of that, making CONTINUATION 
require *negotiated* end-to-end implementation support is actually an 
overall gain as we can identify more cases of failure in advance - along 
with a clear reason for failure.

2) the secondary use case; large sets of Set-Cookie to clear cookies 
seems to be better served with an extension to create a end-to-end 
"clear cookies" frame type or an application level object inside the 
DATA payload.

Do we actually have any other real-world use cases for large headers 
that need to be considered?

Amos

On 2014-07-03 12:21, Roberto Peon wrote:
> compatibility
>> thread, I
>>> should
>>> recipient.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 03 Jul 2014 13:48:44 +1200

On 2014-07-03 13:16, Mike Bishop wrote:
> When I asked Osama about this, his immediate concern was about
> response headers as well.  We cap incoming headers at 16KB (both
> individual size and total header set) by default, with config options
> to increase those as far as 64KB and 16MB if the administrator wants.
> But we have no code that checks the size of response headers – we let
> the client discard the response if they don’t like it.
> 

Used to be valid in HTTP/2 or SPDY ?
  ALPN and Upgrade tokens are (or should be) ensuring that both endpoints 
have the same spec version implemented. All this requires is a bump in 
the draft number.

Used to be valid in HTTP/1 ?
  My last post to this thread reiterated the two know use-cases. We need 
to know of any other legitimate uses ASAP in order to determine how (or 
why not) they fit within the h2 protocol overall.

Amos

> I wonder how many servers don't. I don't know of any major servers
> with such a cap, but I'm surely in the dark.
> On Wed, Jul 2, 2014 at 5:38 PM, Jason Greene
> <[email-masked]<mailto:[email-masked]>> wrote:
> implement the extension, or perhaps another extension :) Either way
> the end result is the same.
> 
> On Jul 2, 2014, at 7:33 PM, Roberto Peon
> <[email-masked]<mailto:[email-masked]>> wrote:
> 
>> On Wed, Jul 2, 2014 at 5:32 PM, Jason Greene 
>> <[email-masked]<mailto:[email-masked]>> wrote:
>> if you wish to accept > 16KB of compressed header data. HTTP/1.1 
>> servers were not required to support it any more than they are today.
>> 
>> On Jul 2, 2014, at 7:21 PM, Roberto Peon 
>> <[email-masked]<mailto:[email-masked]>> wrote:
>>

---

From: Michael Sweet <[email-masked]>
Date: Wed, 02 Jul 2014 23:12:02 -0400

Speaking only for CUPS, we limit most header values (all but Authorization, Cookie, and Set-Cookie) to 255 bytes apiece.  The others we allow (currently) to a maximum of ~87k for authorization data (Kerberos) and 32k for all cookies.  So figure a maximum of 128k worth of headers on the wire.  In practice we send/receive less than 1k, even for the web-based UI.

> > > This sounds like a non-starter, to me. As mentioned in the other thread, I prefer either the status quo, or failing that a stateless CONTINUATION. This seems worse to me than either of those op…

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Jul 2014 07:02:08 +0200

> Do we know how many place a cap on the size of response headers?

Any request or response which flows through haproxy in its default settings
will be limited to 8kB by default, and most sites using it default to 15kB
as they start from common examples using this setting. This is for both
directions, request and response. I'm still very surprized that some
intermediaries have different limits for requests and responses, as if
they need to process headers in one direction they surely have to process
them in the other direction as well, at least to correctly parse the
response and content-length to keep the framing alright.

> If just about everything supports it, why would it be an extension?

The purpose clearly is that all products which do not need to support more
than 16kB do not *need* to implement them.

Willy

---

From: Mike Bishop <[email-masked]>
Date: Thu, 3 Jul 2014 05:15:23 +0000

Used to be valid because the same APIs we have been using for HTTP/1 will now transparently support HTTP/2 without change from the application above us.  We have some extreme corner cases for which we've discussed having to abort the response and turn it into a 505 sending the request back to HTTP/1.1; I'd really rather not add another one, especially based on something we currently don't even check.

And again, the problem is simply that we're a general purpose server -- we *don't know* all scenarios where the application above us might be returning large headers.  We protect ourselves from large request headers as a DOS issue; large response headers, we don't block.

In your other mail, you suggested replacing clearing headers via Set-Cookie with an "application-layer construct" in the DATA frame, or with an extension that would presumably recreate all the required Set-Cookie commands at the 2-1 transition point.  Without changing HTTP semantics, I'm not really clear on how that would work.

-----Original Message-----
From: Amos Jeffries [mailto:[email-masked]] 
Sent: Wednesday, July 2, 2014 6:49 PM
To: [email-masked]
Subject: RE: YAC Proposal

Used to be valid in HTTP/2 or SPDY ?
  ALPN and Upgrade tokens are (or should be) ensuring that both endpoints have the same spec version implemented. All this requires is a bump in the draft number.

Used to be valid in HTTP/1 ?
  My last post to this thread reiterated the two know use-cases. We need to know of any other legitimate uses ASAP in order to determine how (or why not) they fit within the h2 protocol overall.

Amos

>> Compatibility isn’t really affected here. You simply use the 
>> extension if you wish to accept > 16KB of compressed header data. 
>> HTTP/1.1 servers were not required to support it any more than they are today.
>> 
>> > The status quo, or a variation which improves the state machine 
>> > around continuation (e.g. by moving the flags to the last frame of 
>> > the headers block) makes the most sense to me -=R
>> >
>> >
>> > CONTINUATION. This seems worse to me than either of those options. 
>> > With a stateless CONTINUATION, in the rare occasions when I'll 
>> > encounter a compressed header block > 16k, I can roll back just one 
>> > step in my HPACK compression, and then continue on sending an 
>> > uncompressed form of the headers. With CONTINUATION as an 
>> > extension, when the other side doesn't advertise support, I would 
>> > either have to either roll back the entire HPACK state that I 
>> > changed when trying to compress the headers and generate a 
>> > RST_STREAM (likely synthesizing an 413 or some other error in the 
>> > process for my own side), or I'd have to send a GOAWAY, and dial 
>> > back. Neither of those options is in any way more appealing than 
>> > stateless CONTINUATION (which I'm still not entirely sold on to 
>> > begin with!)
>> >
>> > --
>> > Peace,
>> >   -Nick
>> >
>> >
>> WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red 
>> Hat
>> 
>> 
> 
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red 
> Hat

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 03 Jul 2014 05:21:41 +0000

>Do we know how many place a cap on the size of response headers?

Do you mean a hard cap or a configurable cap ?

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 3 Jul 2014 16:43:42 +1000

> Do we know how many place a cap on the size of response headers?

Jetty currently has a setting for max request and response header size,
which both default to 8k.   Users can set it as large as they like, but it
does use more memory and we are probably not as efficient as we could be.

We'd love to keep the same behaviour, but we simply do not trust a
mechanism that will only be activated for tine fraction of users that set a
max header size > 20-something KB.   The benefit of Jumbo frames is that
they allow the same code path to be used for small and large headers - thus
it becomes just a matter of configuration.

If we can drop CONTINUATIONS in favour of jumbo frames, then I do not think
this needs to be an extension.   If we don't then I do.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 04 Jul 2014 00:57:22 +1200

On 2014-07-03 17:15, Mike Bishop wrote:
> Used to be valid because the same APIs we have been using for HTTP/1
> will now transparently support HTTP/2 without change from the
> application above us.  We have some extreme corner cases for which
> we've discussed having to abort the response and turn it into a 505
> sending the request back to HTTP/1.1; I'd really rather not add
> another one, especially based on something we currently don't even
> check.
> 
> And again, the problem is simply that we're a general purpose server
> -- we *don't know* all scenarios where the application above us might
> be returning large headers.  We protect ourselves from large request
> headers as a DOS issue; large response headers, we don't block.
> 
> In your other mail, you suggested replacing clearing headers via
> Set-Cookie with an "application-layer construct" in the DATA frame, or
> with an extension that would presumably recreate all the required
> Set-Cookie commands at the 2-1 transition point.  Without changing
> HTTP semantics, I'm not really clear on how that would work.
> 

Thank you for the view into server design :-).

The design you have described is (almost) incompatible with HTTP/2 
requirement to treat HEADERS+CONTINUATION as a single "atomic" block. 
And leaves both you and the application at the mercy of upstream network 
hops.

Right now you are consuming network bandwidth, a TCP connection and RTT 
delays in order to punt generation of 400/414/431/500/502 error off to 
the remote end of that connection.

A small design change to buffer the generated headers from the 
application and send a single jumbo frame with known length will benefit 
your users greatly as you can then emit a 4xx/5xx error directly back to 
them immediately informing about the known jumbo-frame limit negotiated 
with the remote end of the connection. You will also be able to pass the 
entire header set to HPACK as a block and emit the resulting buffer as a 
single write, saving yourself a HOL-blocked outgoing connection while 
the application builds its header set.

Amos

>   ALPN and Upgrade tokens are (or should be) ensuring that both
> endpoints have the same spec version implemented. All this requires is
> a bump in the draft number.
> 
>   My last post to this thread reiterated the two know use-cases. We
> need to know of any other legitimate uses ASAP in order to determine
> how (or why not) they fit within the h2 protocol overall.
> 
> Amos
> 
>>