Subject: Making extensibility cheap

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jun 2014 17:00:30 -0700

I've been somewhat convinced by Mike Bishop's arguments for restoring
extensibility[1].

What I find hard to swallow is the associated cost.  I think that
Mike's proposal could be trimmed further.  So I'm going to take up the
challenge.

Here's what I think we absolutely need:

1. A way to negotiate the use of hop-by-hop extensions.
2. A way to carry end-to-end extensions.
3. Extensibility for settings, frames and error codes.

To that end, here's my proposed reduction, which I think is largely
keeping with the spirit of Mike's draft:

Extensibility

As Mike suggests, we can open a few IANA registries for business.
That's text we can restore from old drafts.  Easy.

I agree that we need more space for settings than 8 bits, but I'm
going to be aggressive and suggest that 16 bits is enough.  We can
reserve some portion of that space for mucking around (a quarter is
what I'd suggest, but I don't care).

End-to-end

Here I'm going to suggest something far more limited than what Mike
does.  I think that we can get away with an end-to-end, flow
controlled, ordered frame.  A new frame type, modelled on Mike's
should work here.  The new frame type would include a 32-bit extension
ID, for which we can open a registry; we could piggyback on the PEN
registry (private enterprise number); or, we could recommend random
selection.  Again, I care not about these details and will go with
what people seem to like most.

We could do something more with optional flow control, optional
end-to-end and optional ordering, but I think that is altogether too
much optionality.

I'm aware that forcing flow control here might be controversial, but I
think that if we require intermediaries to forward this - and I think
we have to - this is the only good option.

Negotiation

I think that this doesn't need a new "EXTENSIONS" frame, I think that
we can use settings.  Each peer can set a setting to indicate that
they support feature X and if both support the feature, then the
state-affecting components of that feature can be activated.

Otherwise, all error codes map to INTERNAL_ERROR (or some new error
code we define that has equivalent semantics, i.e., none); all unknown
frame types are dropped; and all settings are ignored.

Note that this is important.  Unless you negotiate, only the special
"EXTENSION" frame will traverse intermediaries.

Advice Column

Mike's draft offers good guidance for extension designers.  I think we
need to crib some of that text.  Suggesting that an extension deal
with translating to and from HTTP/1.1, or dealing with peers that
don't support the extension is motherhood and apple pie.  More
important though is establishing the parameters for extension.  Much
of the above will need to reside in this section.

I am going to try to turn out some proposed text for this on the plane
tomorrow, in case this is what we decide to pursue.

--Martin

[1] in draft-bishop-http2-extension-frames

---

From: James M Snell <[email-masked]>
Date: Tue, 3 Jun 2014 17:34:24 -0700

+1. Happy to see this finally being properly addressed.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 4 Jun 2014 12:06:13 +0900

> we have to - this is the only good option.

If flow control is forced, payload of this new frame must be able to be
fragmented in arbitrary size to avoid deadlock just like DATA frame, which
Johnny and I wrote in the other thread.  If both ends started to send new
frames and blocked in the middle of their payload due to exhaustion of
window, deadlock will happen.

Best regatds,
Tatsuhiro Tsujikawa

---

From: Daniel Sommermann <[email-masked]>
Date: Tue, 3 Jun 2014 23:31:53 -0400

+1. Overall this reduction and the direction of the original proposal 
are looking great.

Martin, to just use SETTINGS to negotiate, are you implying that the 
extension takes effect for all frames sent after the SETTINGS-ack, as 
long as both the previously sent SETTINGS and the received SETTINGS 
contain the extension?

I am curious to first see some use cases for end-to-end extensions though.

I agree with the original proposal that ALTSVC and DATA gzip compression 
would be good candidates to move to extensions.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 4 Jun 2014 14:26:33 +1000

I'm also becoming pretty warm to the idea; as I said in the other thread,
I'd like to emphasise noisy errors over silent failure wherever possible.

> would be good candidates to move to extensions.
​If DATA compression was made an extension, there are two ways forward that
I can see:

1. Give hop-by-hop extensions the ability to modify existing frames (add
flags, maybe add fields, etc.) -- note that this pretty much leaves DATA
compression exactly where it is, but with different name for the setting.

2. Mint an extension frame that looks almost exactly like DATA, but
includes the compression stuff. If this was a hop-by-hop extension I think
there'd be very little disruption to the current spec; however it could
also become an end-to-end frame. That idea has much more far-reaching
consequences (e.g. does it cross a semantics boundary to move data outside
of DATA frames? What do non-extension caching proxies do? Etc.) and I've
put no more thought into it than what I've just written, but it's a
possibility.

​I'm keen to see Martin's proposed words.​

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Mike Bishop <[email-masked]>
Date: Wed, 4 Jun 2014 05:32:21 +0000

The second is what's in my draft -- I just copied your proposed version of DATA, added some extra flags, and called it a COMPRESSED_DATA frame.  I think it's hard to sell it as an end-to-end frame for a couple reasons, but the biggest is that you don't have the opportunity for negotiation with the far side.  That's why I keep coming back to an end-to-end frame that can be down-converted to a DATA frame if it hits a hop that doesn't support the extension.

Sent from Windows Mail

From: Matthew Kerwin<mailto:[email-masked]>
Sent: ?Tuesday?, ?June? ?3?, ?2014 ?9?:?30? ?PM
To: Daniel Sommermann<mailto:[email-masked]>
Cc: Martin Thomson<mailto:[email-masked]>, HTTP Working Group<mailto:[email-masked]>

I'm also becoming pretty warm to the idea; as I said in the other thread, I'd like to emphasise noisy errors over silent failure wherever possible.

On 4 June 2014 13:31, Daniel Sommermann <[email-masked]<mailto:[email-masked]>> wrote:

I agree with the original proposal that ALTSVC and DATA gzip compression would be good candidates to move to extensions.

?If DATA compression was made an extension, there are two ways forward that I can see:

1. Give hop-by-hop extensions the ability to modify existing frames (add flags, maybe add fields, etc.) -- note that this pretty much leaves DATA compression exactly where it is, but with different name for the setting.

2. Mint an extension frame that looks almost exactly like DATA, but includes the compression stuff. If this was a hop-by-hop extension I think there'd be very little disruption to the current spec; however it could also become an end-to-end frame. That idea has much more far-reaching consequences (e.g. does it cross a semantics boundary to move data outside of DATA frames? What do non-extension caching proxies do? Etc.) and I've put no more thought into it than what I've just written, but it's a possibility.

?I'm keen to see Martin's proposed words.?

---

From: Mike Bishop <[email-masked]>
Date: Wed, 4 Jun 2014 05:34:41 +0000

The deadlock is an interesting point.  If we introduce fragmentation in HTTP/2, the transport folks may renew their complaints that we're reimplementing their layer, but I understand the concern.  I look forward to discussing possible solutions, though I don't have one off the top of my head.

Sent from Windows Mail

From: Tatsuhiro Tsujikawa<mailto:[email-masked]>
Sent: ?Tuesday?, ?June? ?3?, ?2014 ?8?:?10? ?PM
To: Martin Thomson<mailto:[email-masked]>
Cc: HTTP Working Group<mailto:[email-masked]>

> we have to - this is the only good option.

If flow control is forced, payload of this new frame must be able to be fragmented in arbitrary size to avoid deadlock just like DATA frame, which Johnny and I wrote in the other thread.  If both ends started to send new frames and blocked in the middle of their payload due to exhaustion of window, deadlock will happen.

Best regatds,
Tatsuhiro Tsujikawa

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Jun 2014 10:25:43 +0200

> window, deadlock will happen.

Sorry, but just as for headers, I don't get this one.  If h2 doesn't flow
control it, then TCP will and you can still fill your windows and
deadlock.  You can deadlock two UTF-8 decoders talking over a raw socket if
you try hard enough.

You get this in naive HTTP client/proxy implementations all the time, where
they try to write the entire request with blocking IO and then get the same
thread to read the response.   The answer is just don't write code like
that - got async or multi-threaded, don't try to hold the entire
request/response body at once.

The solution for HTTP has not been to seek an exemption from TCP flow
control, so I don't see why h2 headers/extensions should seek such an
exemption from their framing layer.
If an extension really must read 1GB of data before it is prepared to
continue, then it just needs to commit to reading that 1GB of data - either
decoding it as a stream of putting it in a buffer, and to not block it's
writing on it's reading and vice versa

regards

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 04 Jun 2014 23:01:06 +1200

> would be good candidates to move to extensions.

As would an extension permitting the interleave of frames from other
earlier streams (0, X, Y) between HEADER and CONTINUATION of stream Z.
Provided those interleaved frames do not touch HPACK state.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 04 Jun 2014 23:05:04 +1200

> The second is what's in my draft -- I just copied your proposed version of DATA, added some extra flags, and called it a COMPRESSED_DATA frame.  I think it's hard to sell it as an end-to-end frame…

I agree. The end-to-end is far better serviced by content-encoding
already. A hop-by-hop replacement for transfer-encoding in an extension
RFC would be great and does not need to hold up the core spec document.

Amos

---

From: Daniel Sommermann <[email-masked]>
Date: Wed, 4 Jun 2014 10:06:44 -0400

On 06/04/2014 12:26 AM, Matthew Kerwin wrote:
>
> If DATA compression was made an extension, there are two ways forward 
> that I can see:
>
> 1. Give hop-by-hop extensions the ability to modify existing frames 
> (add flags, maybe add fields, etc.) -- note that this pretty much 
> leaves DATA compression exactly where it is, but with different name 
> for the setting.
>
> includes the compression stuff. If this was a hop-by-hop extension I 
> think there'd be very little disruption to the current spec; however 
> it could also become an end-to-end frame. That idea has much more 
> far-reaching consequences (e.g. does it cross a semantics boundary to 
> move data outside of DATA frames? What do non-extension caching 
> proxies do? Etc.) and I've put no more thought into it than what I've 
> just written, but it's a possibility.
>

I think it would be nice to allow extensions to add flags to existing 
frames, but in the proposals there is no segmentation between "reserved 
for RFC" and "experimental" flags, so that could potentially lead to 
conflicts down the road. We may need to go with #2.

I'm still not sold on the usefulness of end-to-end frames. If the goal 
is to avoid unnecessary decompress/recompress in proxies for 
COMPRESSED_DATA frames on each hop, I think that could be avoided just 
as well with a hop-by-hop negotiated compression. If both sides of a 
proxy negotiate the same COMPRESSED_DATA extension, then the frame can 
be forwarded as-is, just as with the end-to-end extension.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 4 Jun 2014 23:30:49 +0900

On Wed, Jun 4, 2014 at 5:25 PM, Greg Wilkins <[email-masked]> wrote:

> You get this in naive HTTP client/proxy implementations all the time,
> where they try to write the entire request with blocking IO and then get
> the same thread to read the response.   The answer is just don't write code
> like that - got async or multi-threaded, don't try to hold the entire
Well written clients, utilizing async, multithreading
​, evented, whatever, can deal with TCP flow control quite well without
problem.
But they cannot avoid HTTP/2 flow control dead lock I am discussing.

The big difference between HTTP/2 flow control and TCP flow control is that
in HTTP/2 flow control receiver has to send WINDOW_UPDATE frame as
application data to notify the sender the consumed/processed size.

We cannot send WINDOW_UPDATE when transmission is blocked in the middle of
the other frame's payload due to flow control.  In DATA frame, we avoid
dead lock by slicing data size according to the remaining window size.  If
new extension frame is going to be subject to flow control, we need similar
way or something rather to avoid the dead lock.

Best regards,
Tatsuhiro Tsujikawa

---

From: Martin Thomson <[email-masked]>
Date: Wed, 4 Jun 2014 09:37:40 -0700

That is an excellent point. One that definitely needs to be covered.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Jun 2014 20:28:17 +0200

On 4 June 2014 16:30, Tatsuhiro Tsujikawa <[email-masked]> wrote:

> the other frame's payload due to flow control.

Ah, you are considering sending partial frames?   I don't think that is a
good idea.  If you only have a 4KB window and an 8KB frame, then you are
asking for trouble if you send half the frame and hope a window update will
arrive to let you send the rest of it.

If you don't have sufficient window to send a complete frame or you just
don't send it and that stream is stalled.    It's good that data frames can
be fragmented to smaller frames so that small windows can be used, but it
is not a necessary feature to make flow control work, so long as the window
size will eventually be made larger than max frame size.

That said, having fragmentable extension frames is probably desirable.
Isn't this just a matter of providing an END_SEGMENT flag on the extension
frame for extension that need semantic boundaries can reassemble?  Judging
from Mikes email above this appears to have been previously raised,
discussed and rejected?    But I can't see anything about objecting to
extension fragmentation in the original thread:
http://lists.w3.org/Archives/Public/ietf-http-wg/2013OctDec/0601.html
Anybody got any pointers to any discussion on why this was rejected?

cheers