Subject: #540: "jumbo" frames

From: Mark Nottingham <[email-masked]>
Date: Fri, 27 Jun 2014 17:07:52 +1000

I think this discussion has converged upon not making any changes to the HTTP/2 spec, but allowing experimentation to take place in a "jumbo" extension.

As such, I'm going to close the issue. If implementation and deployment experience in the next round leads us to think differently, we can revisit the question, of course.

Cheers,

P.S. If anyone wants to launch an extension draft, please say so; I'm happy to coordinate that through the WG.

--
Mark Nottingham   https://www.mnot.net/

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Jun 2014 09:17:44 +0200

I accept that the WG probably has to move on and that we can't iterate on
this argument forever.

But please don't say there has been convergence.   Several implementors
have spoken here saying that they will not support continuations, so the
current draft rather than supporting arbitrary large headers has actually
enforced even worse support for them in deployed implementations.

On 27 June 2014 09:07, Mark Nottingham <[email-masked]> wrote:

> I think this discussion has converged upon not making any changes to the
> HTTP/2 spec, but allowing experimentation to take place in a "jumbo"
> extension.
>
> As such, I'm going to close the issue. If implementation and deployment
> experience in the next round leads us to think differently, we can revisit
> the question, of course.
>
> Cheers,
>

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 27 Jun 2014 17:19:35 +1000

On 27 Jun 2014, at 5:17 pm, Greg Wilkins <[email-masked]> wrote:

> I accept that the WG probably has to move on and that we can't iterate on this argument forever.   

Thanks.

> But please don't say there has been convergence.   Several implementors have spoken here saying that they will not support continuations, so the current draft rather than supporting arbitrary large headers has actually enforced even worse support for them in deployed implementations.

That's a wrinkle that needs to play out; I'll be interested to see how interop happens, and whether we need to change text or revisit decisions as a result. 

My biggest concern right now is that talking doesn't make the situation any clearer at this point, but it does block us from implementing and deploying it to get the experience we need.

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 27 Jun 2014 07:21:23 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>I think this discussion has converged upon not making any changes to the =
>HTTP/2 spec, but allowing experimentation to take place in a "jumbo" =
>extension.
>
>As such, I'm going to close the issue. If implementation and deployment =
>experience in the next round leads us to think differently, we can =
>revisit the question, of course.

I would like to propose that CONTINUATION be removed to an extension
as well then.

Several people here have already said that they are not going to
implement CONTINUATION and there is presently no way to negotiate
if CONTINUATION will be available or not end-to-end.

CONTINUATION seriously inflates and complicates the draft-RFC
with its many mysterious and counter-intuitive rules (Such as
"END" flags which suddenly don't mean "END" and so on.)

There is also no published implementation experience with CONTINUATION
that shows it to be a particular good solution to anything in particular
or for that matter at all.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 27 Jun 2014 17:35:42 +1000

> I would like to propose that CONTINUATION be removed to an extension

Can someone please provide a proposal? Some features of the protocol aren't amenable to being shifted to extensions, and I very much suspect that CONTINUATION is one of them...

> if CONTINUATION will be available or not end-to-end.

There's a slight difference here -- people have said that they'll refuse requests with CONTINUATION, which can be seen as part of the choices they make when defining the protocol elements (and sizes thereof) that their implementations will accept.

> or for that matter at all.

I'm just finishing drafting the announcement of the next implementation draft and WGLC, the plan being to allow the spec to stabilise for a longer period of time, so as to allow both implementation and deployment. That should give better evidence one way or another.

Cheers,

---

From: David Krauss <[email-masked]>
Date: Fri, 27 Jun 2014 18:01:44 +0800

> if CONTINUATION will be available or not end-to-end.

Perhaps I missed something, but who is implementing CONTINUATION? I only recall hearing from folks who were not. Are the results from interoperability testing public knowledge? It might be a good idea to give credit to those who’ve demonstrated good support.

To my architecture, multiple frame blocks look a lot like jumbo frames. It could go either way. But implementing both is definitely the worst case, and the more popular alternative is going to happen at least first, if not ever.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 27 Jun 2014 09:43:32 -0700

On 27 June 2014 03:01, David Krauss <[email-masked]> wrote:
> Perhaps I missed something, but who is implementing CONTINUATION? I only recall hearing from folks who were not.

I don't know how many of these [1] implement CONTINUATION, but I'd
imagine that it's close to 100%.

I know that it makes some people sad, where perhaps a larger frame
size would not.  But of those that have HTTP/2 implementations, the
number of complaints we've received is fairly low, if vocal.

The more general issue of frame size and forwarding performance is
probably less well-tested, probably because the focus thus far has
been on functional interoperability.  Performance on the scale that
Willy and PHK concern themselves with takes a little more time and
effort to reach.  On the other hand, I hear that Google do test - and
have tested - at scale and aren't concerned about performance, but
it's not perfectly clear cut.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 27 Jun 2014 12:45:23 -0400

FWIW, my current HTTP/2 implementation does not support continuation headers, for the simple reason that 99.9999% of our users do not need extremely large header support (for ActiveDirectory/RFC 4559 deployments).

> P.S. If anyone wants to launch an extension draft, please say so; I'm happy to coordinate that through the WG.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Martin Thomson <[email-masked]>
Date: Fri, 27 Jun 2014 10:05:41 -0700

On 27 June 2014 09:45, Michael Sweet <[email-masked]> wrote:
> FWIW, my current HTTP/2 implementation does not support continuation
> headers, for the simple reason that 99.9999% of our users do not need
> extremely large header support (for ActiveDirectory/RFC 4559 deployments).

Just to clarify behaviour: what do you do when:
a) you are asked to transmit more headers than will fit in a single
frame through your API?
b) you receive a CONTINUATION frame?

---

From: Mike Bishop <[email-masked]>
Date: Fri, 27 Jun 2014 17:12:45 +0000

Microsoft will be supporting headers larger than 16k in both our clients (IE and other apps) and server (IIS).  One reason is that a large number of our enterprise customer have Active Directory integrated solutions for authentication and authorization.  AD uses Kerberos as the preferred authentication mechanism and the authentication tokens can grow quite large.  IIS used to have a default maximum header size of 16k, but we have had to increase this to 64k in recent releases.  This is just one reason for a need to implement large headers.

We have been following the debate here in the WG and as implementers of a browser, an application platform, a web server, and a forward proxy, we do not see a tangible difference between the proposals.  Therefore, our opinion is that there is no reason to delay WGLC to make a major design change at this stage.

From: Michael Sweet [mailto:[email-masked]]
Sent: Friday, June 27, 2014 9:45 AM
To: Greg Wilkins
Cc: Mark Nottingham; HTTP Working Group
Subject: Re: #540: "jumbo" frames

On Jun 27, 2014, at 3:17 AM, Greg Wilkins <[email-masked]<mailto:[email-masked]>> wrote:

On 27 June 2014 09:07, Mark Nottingham <[email-masked]<mailto:[email-masked]>> wrote:
I think this discussion has converged upon not making any changes to the HTTP/2 spec, but allowing experimentation to take place in a "jumbo" extension.

Cheers,

--
Greg Wilkins <[email-masked]<mailto:[email-masked]>>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com<http://www.webtide.com/>  advice and support for jetty and cometd.

---

From: Adrian Cole <[email-masked]>
Date: Fri, 27 Jun 2014 11:22:34 -0600

Conflating implementation with being in favor of continuations is
probably not accurate. Implementing isn't a vote for the idea, more
just being compliant.

In okhttp we had to implement continuations in order to be spec
compliant.  We deferred writing continuations until we got a bug
report as the user overflowed 16k. Latest codebase is on -13 and
supports continuation. In any codebase, I would love an opportunity to
rip out complicated code for something simpler. I personally would
love to rip out code I wrote in favor of jumbo frames. I loved ripping
out code to support -13.

Love isn't a requirement though, so if folks believe that continuation
is a compromise worth continuing, I'm not going to make my own
protocol :)

-A

---

From: <[email-masked]>
Date: Fri, 27 Jun 2014 17:40:25 +0000

On Friday,27 June 2014 09:36, [email-masked] wrote:
>> In message <[email-masked]>, Mark Nottingham writes:
>>>
>>> I think this discussion has converged upon not making any changes to
>>> the HTTP/2 spec, but allowing experimentation to take place in a "jumbo" extension.
>>>
>>> As such, I'm going to close the issue. If implementation and
>>> deployment = experience in the next round leads us to think
>>> differently, we can = revisit the question, of course.
> >
> Can someone please provide a proposal? Some features of the protocol aren't
> amenable to being shifted to extensions, and I very much suspect that  CONTINUATION
> is one of them...

I am willing to take a shot and see if it is possible.

While discussing extensibility in NYC [1], there was some discussion regarding taking some of the current spec and moving it to an extension as a means of keeping extensibility from getting too messy.  CONTINUATION seems like a good candidate.

[1] https://github.com/httpwg/wg-materials/blob/master/interim-14-06/minutes.md starting with the line "Frame type extensibility"

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 27 Jun 2014 14:05:57 -0400

Martin,

> a) you are asked to transmit more headers than will fit in a single

The API will return an error.

For IPP requests the likelihood of going over 16k-1 is basically 0 unless Kerberos authentication (RFC 4559) is being used.  If it becomes an issue I can always force HTTP/1.1 operation when Kerberos authentication is used.  Of I could decide to support CONTINUATION frames in the future.  But since 99.99% of our users do not use Kerberos for printing (and those that do are generally using SMB or IPP to a Windows server where I don't expect IPP over HTTP/2 support), I have little incentive to implement support for something I'll never use.

> b) you receive a CONTINUATION frame?

413

---

From: Martin Thomson <[email-masked]>
Date: Fri, 27 Jun 2014 11:08:26 -0700

>> b) you receive a CONTINUATION frame?

And the header compression state changes it might contain?  Or are you
setting the header table to size 0?

---

From: Jason Greene <[email-masked]>
Date: Fri, 27 Jun 2014 13:31:41 -0500

On Jun 27, 2014, at 1:08 PM, Martin Thomson <[email-masked]> wrote:

> setting the header table to size 0

Even if you set table size to 0 you still have to process them before you can return a 413, since the setting is delayed. and you might get them before it takes effect. Only way to not process them is GOAWAY + close.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Martin Thomson <[email-masked]>
Date: Fri, 27 Jun 2014 11:33:12 -0700

> Even if you set table size to 0 you still have to process them before you can return a 413, since the setting is delayed. and you might get them before it takes effect. Only way to not process them…

A table of size zero means that you don't carry any state forward, so
it's safe to discard the frames.

---

From: Jason Greene <[email-masked]>
Date: Fri, 27 Jun 2014 13:41:55 -0500

> it's safe to discard the frames.

Only when it is in effect. A client can connect to the origin and send a ton of continuation frames, followed by additional requests at the same time as the server is sending the setting to set the table size to 0.Thos additional requests can’t be handled because they contain state relative to the CONTINUATION. I suppose the server could reply with a 503 or something.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 27 Jun 2014 14:49:34 -0400

Martin,

> setting the header table to size 0?

Currently that is the case, yes.

If I do decide to support a non-zero header table size, I may have to bite the bullet and support CONTINUATION frames, but for CUPS the size of the headers is dwarfed by the size of the message body so header compression isn't crucial for me.  The only reason I am interested in HTTP/2 is for the multiplexing/framing layer.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 27 Jun 2014 14:53:12 -0400

Jason,

> Even if you set table size to 0 you still have to process them before you can return a 413, since the setting is delayed. and you might get them before it takes effect. Only way to not process them…

How does a client know not to reconnect and try the same request again after getting a GOAWAY?

---

From: Jason Greene <[email-masked]>
Date: Fri, 27 Jun 2014 13:57:00 -0500

> How does a client know not to reconnect and try the same request again after getting a GOAWAY?

Sorry I was trying to say “just a 413”. you can of course send the 413 before you do the GOAWAY.

Slight tangent - 413 might not match the situation if the client is sending continuations who’s total is < 16 KB or whatever your limit is.
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Jason Greene <[email-masked]>
Date: Fri, 27 Jun 2014 15:11:48 -0500

> We have been following the debate here in the WG and as implementers of a browser, an application platform, a web server, and a forward proxy, we do not see a tangible difference between the propos…

Hi Mike, 

I hope you don’t mind a follow-up, since you plan to use this capability. Does IIS have plans to coalesce connections when acting as a proxy, and if so are you not concerned with the HOL blocking issues that will occur with the current behavior of CONTINUATIONS?

---

From: Mike Bishop <[email-masked]>
Date: Fri, 27 Jun 2014 23:05:42 +0000

Yes, we are planning to coalesce.  There are two failure scenarios we consider here:

One is the stalling of the entire connection by sending a HEADERS, then *not* sending the CONTINUATION.  This is a case where a poor proxy implementation could break things if they're streaming out to the other connection.  That's actually no different with a jumbo frame, since a poorly-implemented sender could stop sending mid-frame just as easily.  Either way, if it happens on the client side, the only person affected is the sender because we won't start relaying the headers until we have the full set, and if the client wants to shoot themselves in the foot, they may.  If this happens from the back-end server, that's more painful, because the effect spreads - but if it lasts too long, our usual timeouts will cancel the blocked requests and clients will retry.  And again, the impact is localized to clients of the misbehaving server.  If servers are implemented poorly, their clients will suffer regardless of how the intermediary handles it.

The second is the fact that HEADERS can take time/throughput from DATA.  Again, this is a property of large headers, not a property of how it gets framed.  Allowing interleaving only trades out problems here -- we can't act on the headers until we have the full header set, and keeping them as a single, non-interleaved unit reduces the need to buffer and wait for the rest to show up at some point in the future.  We'd rather deal with brief HOLB while the full set arrives (in the rare case that it's large) than have a partial header set in memory that we can't deliver to the app yet.

In either case, we have to support large headers, and the drawbacks between a HEADERS+CONTINUATION chain and a single larger HEADERS frame are the same.  At the same time, there is deployment data already discussed which argues against spreading the larger frame size to DATA.  The churn doesn't appear to buy anything, so we prefer stability.

I understand the desire for more predictability, though -- defining settings (in spec or via an extension) that allow a server to inform clients what size individual or total header will result in rejection isn't a bad idea.  Others have expressed a desire not to publish their limits to misbehaving clients, however.

-----Original Message-----
From: Jason Greene [mailto:[email-masked]] 
Sent: Friday, June 27, 2014 1:12 PM
To: Mike Bishop
Cc: Michael Sweet; Greg Wilkins; Mark Nottingham; HTTP Working Group
Subject: Re: #540: "jumbo" frames

Hi Mike, 

I hope you don't mind a follow-up, since you plan to use this capability. Does IIS have plans to coalesce connections when acting as a proxy, and if so are you not concerned with the HOL blocking issues that will occur with the current behavior of CONTINUATIONS? 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat