Subject: Interleaving #481 (was Re: Limiting header block size)

From: Martin Thomson <[email-masked]>
Date: Mon, 2 Jun 2014 16:02:45 -0700

On 22 May 2014 06:13, Michael Sweet <[email-masked]> wrote:
> https://github.com/http2/http2-spec/issues/481

Quoting:
> Currently the HTTP2 draft requires that HEADER frames be contiguous. Since a header block can be arbitrarily large, this presents both an obvious DoS vector and a practical issue with streaming performance and preservation of the priority scheme that HTTP/2 provides.
>
> A simple solution is to allow intervening DATA frames on other, established streams. That will allow high-priority data through without major interruptions.

As a DoS vector, the only parties being denied service are the parties
engaging in the HTTP/2 connection, which is not an issue.  Those
parties have far better means of denying each other service than this.

The impact on multiplexing is largely congruent with the above.  The
best response here is "don't do that" with respect to large header
blocks.  In the worst cases (those in the tail of Richard's stats),
this is ~64k, which will have those parties adversely affected.

The only place this is an actual problem is where messages are merged
from multiple clients (or origin servers) onto a single connection by
an intermediary.  In those cases, the intermediary is basically
responsible.

An intermediary can apply policy to prevent this from happening.  It
seems like many already do this, by setting a limit (8k being fairly
common).  Other options include putting bad actors into TIME_WAIT,
putting all bad actors into a separate low bitrate connection.

The main reason I'm against this change is that it prevents the sort
of clean abstraction that we currently have, which states that HEADERS
+ *CONTINUATION or PUSH_PROMISE + *CONTINUATION can be treated as a
single contiguous unit.  An implementation that fails to apply that
abstraction ends up with the state machine Greg shared with us in #484
(which is incomplete, BTW).

---

From: Roland Zink <[email-masked]>
Date: Tue, 03 Jun 2014 10:48:31 +0200

The simple abstraction that headers + continuation can be treated as a 
single unit is similar to a single frame. Wouldn't it be a cleaner 
abstraction to merge this into one frame?

Regards,
Roland

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Jun 2014 10:56:13 +0200

Martin,

The DoS vector is also against the server.  Allowing arbitrary large
headers is an unconstrained memory commitment that each server has to make
when accepting a new stream.

Sure, servers can apply an arbitrary limit, but that makes the clients
uncertain as to what can or can't be sent.   The servers are also
vulnerable to market forces - as client vendors may choose to
allow/encourage large headers, as happened with the two connection
limit.    I truly do think this is a network neutrality issue as growth in
acceptable header size will effectively prevent resource constrained
operators from terminating large numbers of connections.

I also do not think you can so easily dismiss the intermediary issue as
simple a problem for intermediaries to solve.  It may be very difficult to
detect a bad actor before a DoS occurs.     Consider a load balancer in
front of an auction site, which will typically receive a burst of bids just
before the closing time.  A bad actor can be a perfectly reasonable client
until a few seconds before the auction closes, at which time they suddenly
send several incomplete header blocks.  Such an attack will be able to
effectively lock out a substantial number of clients over a few vital
seconds - even those with established connections.  The intermediary will
not be able to identify the bad actors until it is too late.   Note that
this might not even be malicious, but can be caused by network failure
(assuming that continuations are actually ever routinely needed).

One solution would be for load balancers to always reject streams with
continuation frames.  This would work perfectly well for 99.9% of traffic.
So one may ask why are continuation frames in the spec?

I believe that we should be able to apply a known limit to header sizes so
that applications can be written in such a way that they know they will
pass intermediaries and be acceptable to servers.
An ecosystem where we say that unlimited headers are allowed, but then
arbitrary undocumented and undiscovereable limits are applied, is a
difficult space to effectively use meta data in.

But failing that, interleaving data frames from existing streams would at
least somewhat address the attack I described above.

Finally, I understand that it may be simple to conceptually consider a
HEADER + CONTINUATION* sequence as a single unit, but the reality of it is
that sending/receiving such a sequence cannot be even approximately
considered an atomic state transition, so the state machine I drafted in
#484 is certainly closer to reality that the current state machine in the
document (and as we are approaching our actual implementation of it, I'm
sure I will discover the incomplete parts :)

cheers

On 3 June 2014 01:02, Martin Thomson <[email-masked]> wrote:

> > Currently the HTTP2 draft requires that HEADER frames be contiguous.
> Since a header block can be arbitrarily large, this presents both an
> obvious DoS vector and a practical issue with streaming performance and
> preservation of the priority scheme that HTTP/2 provides.
> >
> > A simple solution is to allow intervening DATA frames on other,
> established streams. That will allow high-priority data through without
> major interruptions.
>

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Jun 2014 02:01:37 -0700

If one does so, then different frames have different max sizes, and are
parsed differently.
This would imply that parsing takes more steps than it does today, as it
would need to examine the frametype, then know the length of the
length-field, then parse the length field.
This is more painful, more codepaths and slower in the common case/common
path.

-=R

---

From: Roland Zink <[email-masked]>
Date: Tue, 03 Jun 2014 11:19:20 +0200

On 03.06.2014 11:01, Roberto Peon wrote:
> If one does so, then different frames have different max sizes, and 
> are parsed differently.
You seem to assume that the length field needs to have a different 
length, although I think this isn't really necessary.
> This would imply that parsing takes more steps than it does today, as 
> it would need to examine the frametype, then know the length of the 
The examination of the frametype is necessary today to detect if it is 
header / continuation frame and to handle it differently.

> This is more painful, more codepaths and slower in the common 
> case/common path.
Even better would be if header frames could be handled like other frames 
and forwarded without much modification.

>
> -=R
>
>
> On Tue, Jun 3, 2014 at 1:48 AM, Roland Zink <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     The simple abstraction that headers + continuation can be treated
>     as a single unit is similar to a single frame. Wouldn't it be a
>     cleaner abstraction to merge this into one frame?
>
>     Regards,
>     Roland
>
>
>
>         On 22 May 2014 06:13, Michael Sweet <[email-masked]
>         <mailto:[email-masked]>> wrote:
>
>             Currently the HTTP2 draft requires that HEADER frames be
>             contiguous. Since a header block can be arbitrarily large,
>             this presents both an obvious DoS vector and a practical
>             issue with streaming performance and preservation of the
>             priority scheme that HTTP/2 provides.
>
>             A simple solution is to allow intervening DATA frames on
>             other, established streams. That will allow high-priority
>             data through without major interruptions.
>

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Jun 2014 02:35:11 -0700

On Tue, Jun 3, 2014 at 2:19 AM, Roland Zink <[email-masked]> wrote:

> You seem to assume that the length field needs to have a different length,
> although I think this isn't really necessary.
>

Welp, I can tell you that we do see headers, especially for responses that
are > 64k.

> header / continuation frame and to handle it differently.

Not for the basic framing, though-- that is common across all frames. The
interpretation of the basic framing does require examination of the fields,
but that is a separate (and currently separable) step.
Anything that eliminates the commonality between frames requires conflating
the complexity of interpretation with that of framing.

I'll point out that a non-remuxing proxy can forward frames without
modification given the spec today.

> and forwarded without much modification.

If you're arguing for a 64k limit, then you're saying this will never
happen enough that we'd care, and thus almost all the time header frames
could be forwarded without modification for a 1:1 proxy.
If you're arguing that 64k is too little, then you either want every
frame's size to be increased, which would likely (as experience has shown)
increase muxing latency, or you want to string sequences of frames together
as is done with the spec today, or you have different sizes for each frame,
which breaks the ability to do basic parsing or dumb inspection.

-=R

---

From: Roland Zink <[email-masked]>
Date: Tue, 03 Jun 2014 12:43:33 +0200

My code checking if only a continuation frame is allowed looks already 
ugly to me but can be extended to allow data frames.

On 03.06.2014 11:35, Roberto Peon wrote:
>
>
>
> On Tue, Jun 3, 2014 at 2:19 AM, Roland Zink <[email-masked] 
>>     If one does so, then different frames have different max sizes,
>>     and are parsed differently.
> Welp, I can tell you that we do see headers, especially for responses 
> that are > 64k.
I can tell you that we do see headers that are > 64k, but not many. I 
even have seen a URL > 90 k.
>
>>     This would imply that parsing takes more steps than it does
>>     today, as it would need to examine the frametype, then know the
>>     length of the length-field, then parse the length field.
>     The examination of the frametype is necessary today to detect if
>     it is header / continuation frame and to handle it differently.
>
>
> Not for the basic framing, though-- that is common across all frames. 
> The interpretation of the basic framing does require examination of 
> the fields, but that is a separate (and currently separable) step.
> Anything that eliminates the commonality between frames requires 
> conflating the complexity of interpretation with that of framing.
>
It can unless it did header modifications previously.
>     Even better would be if header frames could be handled like other
>     frames and forwarded without much modification.
>
>
> happen enough that we'd care, and thus almost all the time header 
> frames could be forwarded without modification for a 1:1 proxy.
> frame's size to be increased, which would likely (as experience has 
> shown) increase muxing latency, or you want to string sequences of 
> frames together as is done with the spec today, or you have different 
> sizes for each frame, which breaks the ability to do basic parsing or 
> dumb inspection.
>
> -=R
>
>
>
>>
>>     -=R
>>
>>
>>         The simple abstraction that headers + continuation can be
>>         treated as a single unit is similar to a single frame.
>>         Wouldn't it be a cleaner abstraction to merge this into one
>>         frame?
>>
>>         Regards,
>>         Roland
>>
>>
>>
>>                 Currently the HTTP2 draft requires that HEADER frames
>>                 be contiguous. Since a header block can be
>>                 arbitrarily large, this presents both an obvious DoS
>>                 vector and a practical issue with streaming
>>                 performance and preservation of the priority scheme
>>                 that HTTP/2 provides.
>>
>>                 A simple solution is to allow intervening DATA frames
>>                 on other, established streams. That will allow
>>                 high-priority data through without major interruptions.
>>

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Jun 2014 15:26:10 +0200

On 3 June 2014 12:43, Roland Zink <[email-masked]> wrote:

> I can tell you that we do see headers that are > 64k, but not many. I even
> have seen a URL > 90 k.

So already with HTTP/1.1 applications are putting a lot of meta data into
the transport meta-data channel.

Now imagine what will happen when we make that meta data channel have
favourable transport characteristics above and beyond normal data
frames!    I predict it's use will explode and there will be lots of
pressure on intermediaries and servers to increase any limits they place on
header size.

So far, the only response I have seen to these concerns is to say that a
server/intermediary is free to unilaterally apply its own limit.  But I
fail to see how applying an unknown undiscoverable limit is in any way
better than a specification mandated (or negotiated) limit.

I think that the unknown resource limitation that HTTP/1.1 applications
face today is something that really should be solved in http2

Applying flow control and interleaving to headers would at least remove the
incentive to use meta data instead of data and would essentially put us in
the same situation as HTTP/1.1 is today.  This can be done with hpack by
restricting header sets to 1 frame and then aggregating sets until end of
headers.    This is a slight complication in the encoder, but not much.

Having a 1 frame limit on the size of HTTP headers I think is acceptable,
although it would be disruptive to a small number of applications.
However I'm certainly will to consider making an application specific meta
data channel available to such applications.  In fact we already allow meta
data frames to be interleaved with data frames, so surely it is just a
matter of migrating the few applications that need large meta data to use
this new mechanism on HTTP/2 rather than to encourage further flooding of
the transport meta data channel with application data.

In short I am proposing:

   - All hpack header blocks be limited in size to 1 frame, thus allowing
   arbitrary interleaving of frames.
   - Header sets from continuation frames be aggregated until
   END_OF_HEADERS is received, so arbitrarily large header sets can be
   transported as subsets.
   - Headers, continuations and push promises frames will be flow
   controlled.
   - The initial headers for a HTTP request is limited to 1 frame, this
   limits the memory commitment of the server accepting a stream.
   - Response headers, trailers and embedded meta data may be unlimited
   number of frames, so arbitrary large meta data can still be transported,
   but not as part of the HTTP request meta data.
   - Only the initial header frame will be made available to request
   handlers via the normal HTTP meta data semantics. This limits the servers
   memory commitments.
   - Applications that require large metadata associated with a
   request/response are encourage to developer/propose/standardize APIs to
   access the HTTP/2 capability for trailers and embedded meta data.

cheers

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 10:08:39 -0700

On 3 June 2014 01:56, Greg Wilkins <[email-masked]> wrote:
> The DoS vector is also against the server.  Allowing arbitrary large headers
> is an unconstrained memory commitment that each server has to make when
> accepting a new stream.

Well, if you stipulate that servers have to accept the request, then
they have to accept the request and all that it entails.

As I've pointed out before, a limit won't prevent effectively
unbounded commitment.  HPACK can be used to generate a much larger
state commitment than it's wire format requires.

Servers do have the option of rejecting the string and avoiding the
memory commitment.  Market forces notwithstanding.

> I truly do think this
> is a network neutrality issue

In that case, I recommend that you direct your comment to
http://www.fcc.gov/comments

> send several incomplete header blocks.

You had me until this point.  An intermediary shouldn't commit
incomplete header blocks to a multiplexed connection.  In most cases,
they can't.  This is part of what I'm talking about when I say that
the intermediary has a responsibility to multiplex properly.

> So one may ask why are continuation frames in the spec?

This would be a bad idea.  The existence of continuations is largely
orthogonal to header size.  You might as well reject requests based on
whether the current date when converted to RGB is a colour you don't
like.  Such behaviour leads to the sorts of perversions clients are
forced to do to get HTTP/1.1 requests working today.

> that applications can be written in such a way that they know they will pass
> intermediaries and be acceptable to servers.

I'm inclined to agree with the general principle, but am unable to
effectively turn this into practice without breaking things.  We've
already established (several times over now) that a hard limit is
unacceptable.  It's also been established that a hop-by-hop
negotiation for size doesn't work for something that is essentially
end-to-end.  Thus, we arrive at where we are today.

> least somewhat address the attack I described above.

I think that it would enable it more than it would mitigate it.

---

From: Simone Bordet <[email-masked]>
Date: Tue, 3 Jun 2014 20:06:52 +0200

Hi,

> forced to do to get HTTP/1.1 requests working today.

Sorry, I don't understand the date/RGB simile.

Can you please expand in a more technical way the arguments of why it
is a bad idea, and how the existence of continuations is orthogonal to
header size ?
Making examples would help.

Thanks !

-- 
Simone Bordet
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 11:16:44 -0700

> Making examples would help.

A header block can contain any amount of actual data.  Anywhere from
absolutely nothing (because it's all padding, or it's really short) to
really ---ing gigantic (because it uses HPACK).

Deciding that you want to reject a frame based on a signal that is so
abstractly connected to the actual thing you are concerned is a bad
idea.  It is essentially arbitrary (hence the date/RGB comment).
Arbitrary rejections lead to all sorts of bad behaviour from clients
trying to avoid arbitrary behaviour, up to and including cargo
cult-type actions.

(And yes, I'm aware of how this is an argument for having a known,
deterministic way to know whether a request is acceptable before
sending it, but, as I explained, I don't think that this is feasible.)

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Jun 2014 21:43:11 +0200

Martin,

I don't understand how a single frame can be of arbitrary size or even
significantly larger than the negotiated max header table size?

A received header block may contain either literal headers or indexes into
the header table.  If the header block is all huffman, than the max size is
about twice the frame size.  If the header block is all indexes, then there
is no additional memory commitment at all other than the strings already in
the header table and the points to them in the buffer that the reader has
already committed to read.    An efficient server is probably going to make
a random access structure from the buffer contents, but that too will have
a limited size as it will just contain references to the header table,
which is a memory commitment already made.

I'm glad that you agree with the fundamental idea that limits should not be
hidden and undiscovereable.    I also agree that hop by hop negotiation is
complex and of little use - because it just tells an application that it
can't work.  Instead it should be the other way around that the transport
should tell the application what it should do to always work over a
compliant http2 connection.

I get it that it is very frustrating when people like me come along and try
to over turn decisions that have been put to consensus before.  However I
have a responsibility to raise the concerns as I see them and have already
apologised for my late reentry in the process.   If having raised my
concerns, I fail to make my case, then I will hopefully not turn into a
chronic malcontent and accept the consensus and move on.

But I would also point out that the proposal I made above does support
arbitrary sized headers.  Just not for the initial headers of a HTTP
request.  So it is not totally at odds with the previous consensus.     It
is trying to both protect the intermediaries and servers from unlimited
commitments while allowing applications to send arbitrary larged meta data
- just separately from the transport meta data.

Finally, if proxies are really required to buffer an unknown number of
frames before forwarding any of them, then I expect a lot will just opt for
the easy path of closing any connection that attempts to send a
continuation frame.     Are there any server / intermediary developers here
who plan to allow headers significantly larger than the existing defacto
standard of 8k?

regards

PS.  it would also be good to know if the small percentage of requests with
headers larger that 16k are actually valid requests.  For all we know they
are application errors sending stupidly large repeated cookies/headers that
are already rejected by the server as being too large.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 13:49:15 -0700

On 3 June 2014 12:43, Greg Wilkins <[email-masked]> wrote:
> already committed to read.

Not really.  Since referenced entries can be evicted after they are
emitted, the actual size can be much larger (though perhaps with
careful reference counting, only a smidgeon over 32K + header table
size).

> compliant http2 connection.

That is quite hard because you are potentially talking to an HTTP/1.1
hop, unaware of the limits imposed by other hops.

> to over turn decisions that have been put to consensus before.

Sometimes.  But only occasionally, when the tone is off, and that sort of thing.

> arbitrary sized headers.

Not quite.  Arbitrary sized blocks, but not individual fields.  Long
URLs and cookies are a part of the landscape unfortunately.

> the easy path of closing any connection that attempts to send a continuation
> frame.

As I said, I hope that they don't.  I hope that they look at the size
of the header fields (before or after compression, depending on how
they are built).  After all, post-decompression size is potentially
much larger than you think.

But I do agree: I expect that a limit like 8k is in the right ballpark
here - for most users and use cases.  I'm sure that you would be
entirely justified in applying that limit.  What I object to here is
the implication that that is enough for everybody, and for every use
case.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 3 Jun 2014 21:02:16 +0000

In particular, we have cases where the headers exceeded 16KB, and we had to expand our maximum to support them.  Kerberos tickets can be insanely large.

We now cap at 64KB, and haven't yet encountered a use case which required exceeding that.  While you may be comfortable with an 8KB max through your implementation, you'd be breaking real-world scenarios to write that into the protocol.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Tuesday, June 3, 2014 1:49 PM
To: Greg Wilkins
Cc: Simone Bordet; Michael Sweet; HTTP Working Group
Subject: Re: Interleaving #481 (was Re: Limiting header block size)

On 3 June 2014 12:43, Greg Wilkins <[email-masked]> wrote:
> A received header block may contain either literal headers or indexes 
> into the header table.  If the header block is all huffman, than the 
> max size is about twice the frame size.  If the header block is all 
> indexes, then there is no additional memory commitment at all other 
> than the strings already in the header table and the points to them in 
> the buffer that the reader has already committed to read.

Not really.  Since referenced entries can be evicted after they are emitted, the actual size can be much larger (though perhaps with careful reference counting, only a smidgeon over 32K + header table size).

> complex and of little use - because it just tells an application that 
> it can't work.  Instead it should be the other way around that the 
> transport should tell the application what it should do to always work 
> over a compliant http2 connection.

That is quite hard because you are potentially talking to an HTTP/1.1 hop, unaware of the limits imposed by other hops.

> I get it that it is very frustrating when people like me come along 
> and try to over turn decisions that have been put to consensus before.

> But I would also point out that the proposal I made above does support 

Not quite.  Arbitrary sized blocks, but not individual fields.  Long URLs and cookies are a part of the landscape unfortunately.

> frames before forwarding any of them, then I expect a lot will just 
> opt for the easy path of closing any connection that attempts to send 
> a continuation frame.

As I said, I hope that they don't.  I hope that they look at the size of the header fields (before or after compression, depending on how they are built).  After all, post-decompression size is potentially much larger than you think.

But I do agree: I expect that a limit like 8k is in the right ballpark here - for most users and use cases.  I'm sure that you would be entirely justified in applying that limit.  What I object to here is the implication that that is enough for everybody, and for every use case.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Jun 2014 23:13:15 +0200

> URLs and cookies are a part of the landscape unfortunately.

Well it should be pretty simple to either modify hpack to support field
aggregation or come up with a field naming convention to support
aggregation of fields from different sets.

My proposal would then still allow arbitrarily large meta data sets and
fields - it would just keep them out of the transport meta data channel
which intermediaries and servers have to handle specially.

Yes there are current examples of some fields approaching 64KB (thanks
Mike), but their existence is not an argument to make the transport meta
channel unlimited.  Those near 64KB use-cases are examples of exactly the
kind of usage creep that will consume more and more resources from the
intermediaries and servers unless we put a stop to it and offer them a more
suitable channel for this application meta data.

cheers

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 14:47:50 -0700

On 3 June 2014 14:13, Greg Wilkins <[email-masked]> wrote:
> channel unlimited.

Actually, it is.  We've been chartered to produce a protocol that
provides "... a new expression of HTTP's current semantics ..."  I
understand that's it's a bit of weak-sauce to use appeal from
authority, but I'd interpret any attempt to constrain size as a change
to the protocol semantics.

Now the ideas you have described make different trade-offs between
implementation costs, protocol complexity, and the various
externalities that they cause.  If you do manage to find a
less-objectionable way of reaching a solution to the same problem, I
encourage you to work through your proposal in more detail.  There are
fairly far-reaching ramifications of any change of this sort and we
need to view this a little more holistically if we are even going to
consider changes.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Jun 2014 00:47:08 +0200

Martin,

I must be making my case if you are resorting to such weak sauce:)

I think it is very weak sauce as:

   - HTTP/1 includes 413 and 414 error codes, so its semantics do support a
   limit on request entity size.  I'm only proposing to define what that limit
   is.
   - The charter calls out "misuse of the underlying transport" as a
   motivation for HTTP/2 and HTTP headers were never intended to send
   arbitrarily large amounts of application meta data.
   - The charter calls out that the specification should  "Address the
   "head of line blocking" problem in HTTP", and the current header design can
   cause head of line blocking.
   - The charter only says it is "expected to meet these goals for common
   existing deployments of HTTP", which appears to be 8KB, so that does
   give a lot of wiggle room for excluding >16KB compressed.
   - The WG has already used some wiggle room to drop existing HTTP
   semantics such as 100 and 102 responses.  I have encountered 100 responses
   many times in the wild, but very rarely have I had apps hit the 8k limit.
   - The charter allows for extensions and I am proposing an extension that
   will support the use case of arbitrary large headers/trailers etc.

> If you do manage to find a
> encourage you to work through your proposal in more detail.
>

Will do and thanks for the feedback to date as I can see some of your
objections that need to be addressed (Eg big fields as well as big sets).
   Thanks also acknowledging that there is a problem here worth trying to
solve.

cheers

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Jun 2014 15:59:11 -0700

The presence of 413, etc. does not impose a limit, actually. It simply
imposes a limit for that request.
This is quite different than an overall limit.

How would one express such a limit through a proxy, even if it were known
(it could be changing from second to second)?
Client C talks via proxy P to servers A and B. What limit would proxy P
advertise when the limits of A and B are different?

Solving these problems requires far more complexity than what exists right
now.

Common, existing deployments includes Google's properties, I'd wager to
guess. I've said a number of times that I see large responses, and that
those responses exist because of implementation deficiencies in the web
platform that are not necessarily related to HTTP.
Please don't ignore this! It would be rather difficult to attempt to
explain to the developers that *users* now have to change their behaviors
because of a protocol change that should have been transparent.

-=R

On Tue, Jun 3, 2014 at 3:47 PM, Greg Wilkins <[email-masked]> wrote:

>    - HTTP/1 includes 413 and 414 error codes, so its semantics do support
>    a limit on request entity size.  I'm only proposing to define what that
>    limit is.
>    - The charter allows for extensions and I am proposing an extension
>    that will support the use case of arbitrary large headers/trailers etc.
>
>
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 16:50:08 -0700

I think that you need to find some stronger arguments than these.

On 3 June 2014 15:47, Greg Wilkins <[email-masked]> wrote:
> HTTP/1 includes 413 and 414 error codes, so its semantics do support a limit
> on request entity size.  I'm only proposing to define what that limit is.

I think that Roberto addressed this.  Those are per-resource or
per-request limits.  It's a very different proposition to change the
entire protocol.

> The charter calls out "misuse of the underlying transport" as a motivation
> for HTTP/2

Yeah, I'm sure that people can misuse what we provide them.  I think
that the point is that in HTTP/1.1 you have to misuse TCP to get
anything like a good result.  Misuse of in HTTP/2 - of the sort that
you describe, at least - merely results in worse results.  That, for
me, provides sufficient incentive to have users do the right thing.

> and HTTP headers were never intended to send arbitrarily large
> amounts of application meta data.

That's speculative.  Though I'll concede that here we leak our
abstraction, meaning that yes header fields are a poor choice.  But I
couldn't categorically say that they shouldn't be used.

> The charter calls out that the specification should  "Address the "head of
> line blocking" problem in HTTP", and the current header design can cause
> head of line blocking.

I address this one above.  But I'll point out that HOL blocking is
still possible: we're still using TCP.  But a good HTTP/2
implementation on both client and server can avoid HOL blocking.  We
don't have to make it impossible, because that is putting the perfect
up as the enemy of the good.

> The charter only says it is "expected to meet these goals for common
> existing deployments of HTTP", which appears to be 8KB, so that does give a
> lot of wiggle room for excluding >16KB compressed.

That's not a license to cut out all the features that only the most
common implementations/deployments use.  HTTP has an extremely diverse
deployment base.

> The WG has already used some wiggle room to drop existing HTTP semantics
> such as 100 and 102 responses.  I have encountered 100 responses many times
> in the wild, but very rarely have I had apps hit the 8k limit.

We removed 1xx series responses because we provided better
alternatives for each (i.e., flow control).

> The charter allows for extensions and I am proposing an extension that will
> support the use case of arbitrary large headers/trailers etc.

Hmm.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Jun 2014 15:01:07 +0200

OK, so I think we've descended into semantic analysis of the charter now.

I really can't see the difference between your removal of 1xx and replacing
it with a better alternative  vs my proposal of removing large application
headers from the initial request header frame and replacing it with a
better alternative (a flow controlled as infinite as you like in stream
meta data mechanism).

I also disagree that a well written HTTP/2 impl can avoid HOL blocking,
precisely because we allow unlimited non fragmentable headers.   Thus to
send a header, HTTP/2 implementations will always be at risk of hitting the
TCP/IP flow control windows and thus holding up all other streams.

Finally I don't think you have fully understood my proposal - but I guess
that is my responsibility to write it up in some more detail.   I've not
proposed disallowing arbitrarily large headers.  Large response headers can
still be transmitted with exactly HTTP semantics.   Large request headers
can still be transmitted, but with the small change that only the first set
will be considered to be part of the transport request meta data.  The
other larger headers will be available via alternate, but just not as part
of the transport request abstraction.

Now nothing is stopping the next layer up - say a servlet container,
deciding to wait until all the header sets have been received before
dispatching the request to the servlet - thus preserving the existing
semantics - it is just not mandated by the protocol that they must do this
and alternate implementation can be developed that have less memory impact
on servers and a better chance of avoiding HOL blocking.

Anyway,  I'll leave this be for a while and contemplate writing up a more
detailed proposal, but that may have to wait until we've implemented a bit
more of the existing draft.

cheers

On 4 June 2014 01:50, Martin Thomson <[email-masked]> wrote:

> give a