Subject: Re: HTTP/2 and Websockets

From: Robert Collins <[email-masked]>
Date: Fri, 21 Nov 2014 09:11:53 +1300

On 15 October 2014 00:00, Amos Jeffries <[email-masked]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 14/10/2014 11:01 p.m., Robert Collins wrote:
>> On 1 October 2014 23:37, Amos Jeffries wrote:
>>
>>>> All the implementor discussion I've seen during the
>>>>> HTTP/2 discussions has focused on how intermediaries want to
>>>>> be scalable: and buffering is anti-scaling. So - is it a
>>>>> pragmatic concern, or do we expect DATA stream buffering to
>>>>> take place [outside of protocol gateways converting to
>>>>> HTTP/1.1 where non upload can require buffering - and note
>>>>> that such a gateway can't carry ws anyway unless its aware of
>>>>> it, and if its aware of it, it can make sure it does not
>>>>> buffer].
>>>
>>>
>>> I think the problem is not buffering in HTTP/2 per-se but the
>>> DATA frame (de-)aggregation that can happen if the frames are
>>> buffered by general network conditions (ie in TCP bottlenecks).
>>> This would not be good for a 1:1 relationship between DATA and ws
>>> frames.
>>>
>>> Amos
>>
>> So hang on a second here. If we say that ws frames can't be split
>> over multiple HTTP/2 frames that implies that we have to buffer
>> them until there is enough in the window to transmit a potentially
>> very large packet all at once. It also conflicts with RFC6455 - the
>> specific intent there is to not be a stream based system.
>
> If a ws frame *has* to be that long, not doing so would block the
> entire HTTP/2 connection until all bytes of that frame were delivered
> anyway. So you trade off buffering that single frame at the sender,
> versus blocking all HTTP/2 traffic end-to-end.
>
> If the ws data is so critical to get transmitted fast why is that
> single ws frame so large to begin with? surely it would be transmitted
> faster as a sequence of WS + *WSDATA frames emited as the payload was
> available to send.

I agree that its inconsistent which is why I don't think it matters
and mapping down to h2 frames as a sequence of octets would be fine.
But you seem to both agree with my reasoning and disagree with my
conclusion. This is confusing.

>> I was suggesting that we just treat the HTTP/2 stream like the TCP
>> connection in RFC 6455 - the conversation from stream to message
>> based semantics and so on can take place above that in the ws
>> implementation - and that we should still apply the transmission
>> windows etc to ws streams.
>
> If you do that you loose any and all benefits from HTTP/2 frames.
> Everything from ws frame headers to data content becomes semantically
> identical to the opaque payload of a DATA frame on an HTTP/2 CONNECT
> request. I believe Yutaka is seeking to get away from that situation
> where DATA frames may be split, joined or buffered at any point.

Sorry, I just don't follow that. We have a primitive which appears to
fit ws entirely, with the only caveat being that we haven't defined
the mapping from the high level frames to the h2 primitives. If the
spec identifies how ws is negotiated and framed within h2, its not
opaque at all. And ws implementations that support raw ws (which
they'll do for quite some time...) have to deal with tcp which offers
no better semantics than this.

-Rob

-- 
Robert Collins <[email-masked]>
Distinguished Technologist
HP Converged Cloud

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 09:47:04 +0800

>I agree that its inconsistent which is why I don't think it matters

I am the author of libwebsockets, we are adding http2 support at the moment.  The basic http2 serving is done and works for http, but we're all dressed up and nowhere to go in terms of treating websocket connections as just another kind of http2, since the framing is "TBD".  I am sorry I am a bit late to the party.

I think you have to take the atomic large ws frame thing as a genuine problem because http2 has the transmit credit concept.  So even if you buffered one ws frame, you can't sit there spewing as much http2 DATA frame as it needs to atomically encapsulate it, without sizing your http2 frame to suit the tx credit.

But it's OK because the implementation can transparently fragment the ws data using the ws message semantics... I think there's no choice but to take that approach.

Otherwise you get into being able to DoS even an http2 "big pipe aggregation" by just one mux element spewing an endless ws frame and blocking every other mux'd connection... it cannot be right.

>>> windows etc to ws streams.

Yes ---^ this is how it has to be I think.

>the mapping from the high level frames to the h2 primitives. If the

Yeah.

>no better semantics than this.

Right now if I understood it the ws connections can still negotiate themselves transparently inside http2 mux connections, using the RFC6455 upgrade on their individual session ID, do the extra RTT and tx data masking.

Formalizing how to encapsulate the same thing in http2 doesn't buy much above that... the benefit we can get is map the RFC6455 framing on to http2 native framing and get rid of the duplication simple encapsulation has (for many small frames, it would be really painful overhead actually).  So if we will do anything, it should indeed be define how to map RFC6455 framing on http2 framing.

I guess it can be done in parallel with http2 coming to an end rather than trying to block it, just by defining some new optional frame types...

-Andy

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 11:02:28 +0900

>
> I think you have to take the atomic large ws frame thing as a genuine
> problem because http2 has the transmit credit concept.  So even if you
> buffered one ws frame, you can't sit there spewing as much http2 DATA frame
> as it needs to atomically encapsulate it, without sizing your http2 frame
> to suit the tx credit.
> But it's OK because the implementation can transparently fragment the ws
> data using the ws message semantics... I think there's no choice but to
> take that approach.

Sorry I don't understand what you are proposing. Can you explain?

I guess it can be done in parallel with http2 coming to an end rather than
> trying to block it, just by defining some new optional frame types...

I agree to define a ws-dedicated frame type and use it.

On Fri, Nov 21, 2014 at 10:47 AM, Andy Green <[email-masked]> wrote:

> I am the author of libwebsockets, we are adding http2 support at the
> moment.  The basic http2 serving is done and works for http, but we're all
> dressed up and nowhere to go in terms of treating websocket connections as
> just another kind of http2, since the framing is "TBD".  I am sorry I am a
> bit late to the party.
>
> I think you have to take the atomic large ws frame thing as a genuine
> problem because http2 has the transmit credit concept.  So even if you
> buffered one ws frame, you can't sit there spewing as much http2 DATA frame
> as it needs to atomically encapsulate it, without sizing your http2 frame
> to suit the tx credit.
>
> But it's OK because the implementation can transparently fragment the ws
> data using the ws message semantics... I think there's no choice but to
> take that approach.
>
> Otherwise you get into being able to DoS even an http2 "big pipe
> aggregation" by just one mux element spewing an endless ws frame and
> blocking every other mux'd connection... it cannot be right.
>
> Right now if I understood it the ws connections can still negotiate
> themselves transparently inside http2 mux connections, using the RFC6455
> upgrade on their individual session ID, do the extra RTT and tx data
> masking.
>
> Formalizing how to encapsulate the same thing in http2 doesn't buy much
> above that... the benefit we can get is map the RFC6455 framing on to http2
> native framing and get rid of the duplication simple encapsulation has (for
> many small frames, it would be really painful overhead actually).  So if we
> will do anything, it should indeed be define how to map RFC6455 framing on
> http2 framing.
>
> I guess it can be done in parallel with http2 coming to an end rather than
> trying to block it, just by defining some new optional frame types...
>
> -Andy
>
> >-Rob
>
>
>

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 10:42:11 +0800

>Sorry I don't understand what you are proposing. Can you explain?

I'm agreeing with what was already written by someone else on the thread.

Talking about buffering huge ws frames until you have enough to issue it all in one big http2 DATA frame will not fly.

If you're using this putative ws-over-http2 scheme, and you get given a huge ws frame to transmit, you should fragment it using RFC6455 message semantics to some implementation-defined limit that is friendly for mux'd http2 transport.

>I agree to define a ws-dedicated frame type and use it.

Super... has anyone proposed how to map RFC6455 to http2 framing in detail yet?

-Andy

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 12:30:46 +0900

On Fri, Nov 21, 2014 at 11:42 AM, Andy Green <[email-masked]> wrote:

> Talking about buffering huge ws frames until you have enough to issue it
> all in one big http2 DATA frame will not fly.
>
> If you're using this putative ws-over-http2 scheme, and you get given a
> huge ws frame to transmit, you should fragment it using RFC6455 message
> semantics to some implementation-defined limit that is friendly for mux'd
> http2 transport.
>
Thanks.

Strictly speaking, RFC6455 allows an extension to give meaning to WebSocket
frames, so merging / fragmenting frames breaks such extensions.
We discussed this problem in HyBi and many of us said "don't care".
In any case, an http/2 frame cannot be bigger than 2^24 (or 2^14 without an
explicit permission), so I think we don't have to worry about DoS.

> Super... has anyone proposed how to map RFC6455 to http2 framing in detail
I think not.
I did list several ways at [1], but I deleted it from the next version[2]
because HTTP/2 situation had changed.

1: http://tools.ietf.org/html/draft-hirano-httpbis-websocket-over-http2-00
2: http://tools.ietf.org/html/draft-hirano-httpbis-websocket-over-http2-01

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 11:38:26 +0800

On 21 November 2014 11:30:46 GMT+08:00, Yutaka Hirano <[email-masked]> wrote:
>> On 21 November 2014 10:02:28 GMT+08:00, Yutaka Hirano
><[email-masked]>
>> wrote:
>> >>
>> >> buffered one ws frame, you can't sit there spewing as much http2
>DATA
>> >frame
>> >> as it needs to atomically encapsulate it, without sizing your
>http2
>> >frame
>> >> to suit the tx credit.
>> >> But it's OK because the implementation can transparently fragment
>the
>> >ws
>> >> data using the ws message semantics... I think there's no choice
>but
>> >to
>> >> take that approach.
>> >

Yeah extensions except for compression have just not come into existence.

So I also see it as don't care.  I'm not even sure it's true since the intention during ws discussion was an intermediary can fragment frames same as how tcp packets may be fragmented.

>explicit permission), so I think we don't have to worry about DoS.

... I don't see that.  If I keep spamming 16MB frames even on one stream on a consumer link your latency goes to pieces and if something is doing multiple instances of it even a big pipe will feel pain.

>> >I guess it can be done in parallel with http2 coming to an end
>rather
>> >than
>> Super... has anyone proposed how to map RFC6455 to http2 framing in
>detail
>> yet?
>>
>I think not.
>I did list several ways at [1], but I deleted it from the next

Alright I will go look at these.

The discussion is moribund somebody should probably issue a 'stalking horse' since it's easier for people to jump in and tell you you're doing it wrong ^^

-Andy

>version[2]
>1:
>http://tools.ietf.org/html/draft-hirano-httpbis-websocket-over-http2-00
>2:
>http://tools.ietf.org/html/draft-hirano-httpbis-websocket-over-http2-01
>
>
>>
>> -Andy
>>
>> >
>> >> >>>> This would not be good for a 1:1 relationship between DATA
>and
>> >ws
>> >> >>>> frames.
>> >> >>>>
>> >> >>>> Amos
>> >> >>>
>> >> >>> very large packet all at once. It also conflicts with RFC6455
>-
>> >the
>> >> >> faster as a sequence of WS + *WSDATA frames emited as the
>payload
>> >was
>> >> >> available to send.
>> >> >
>> >> just another kind of http2, since the framing is "TBD".  I am
>sorry I
>> >am a
>> >> bit late to the party.
>> >>
>> >> buffered one ws frame, you can't sit there spewing as much http2
>DATA
>> >frame
>> >> as it needs to atomically encapsulate it, without sizing your
>http2
>> >frame
>> >> to suit the tx credit.
>> >>
>> >> But it's OK because the implementation can transparently fragment
>the
>> >ws
>> >> data using the ws message semantics... I think there's no choice
>but
>> >to
>> >> take that approach.
>> >>
>> >> >Sorry, I just don't follow that. We have a primitive which
>appears
>> >to
>> >> above that... the benefit we can get is map the RFC6455 framing on
>to
>> >http2
>> >> many small frames, it would be really painful overhead actually). 
>So
>> >if we
>> >> I guess it can be done in parallel with http2 coming to an end
>rather
>> >than

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 12:13:28 +0800

On 21 November 2014 11:38:26 GMT+08:00, Andy Green <[email-masked]> wrote:
>>> >> problem because http2 has the transmit credit concept.  So even
>if
>>> >you
>>> Talking about buffering huge ws frames until you have enough to
>issue
>>it
>>> If you're using this putative ws-over-http2 scheme, and you get
>given
>>a
>So I also see it as don't care.  I'm not even sure it's true since the
>intention during ws discussion was an intermediary can fragment frames
>same as how tcp packets may be fragmented.
>
>... I don't see that.  If I keep spamming 16MB frames even on one
>stream on a consumer link your latency goes to pieces and if something
>is doing multiple instances of it even a big pipe will feel pain.
>

5.2.1 is like pure tunneling, it's simple alright but it's bloated for small frames.

5.2.2 has a needless flushing thing on top... in a real ws setup pieces of data can arrive any old way they should be passed on as they come or coalsced according to intermediary policy, I don't see the point.

5.3.3 headers + data per websocket frame... super inefficient

Has it already been discussed to just merge http2 frame with ws frame?

 - use the headers scheme you defined to negotiate the connection + ws protocol

 - There's an http2 frame type called like WSDATA

 - WSDATA's 8-bit flags field, is the ws flags [fin][rsv1-3][opcode]

 - WSDATA's payload, is the ws frame payload... the ws framing disappears completely into the http2 framing.

... what problems does this create?

-Andy

>The discussion is moribund somebody should probably issue a 'stalking
>horse' since it's easier for people to jump in and tell you you're
>doing it wrong ^^
>
>-Andy
>
>>version[2]
>>> >> problem because http2 has the transmit credit concept.  So even
>if
>>> >you
>>> >> >>> I was suggesting that we just treat the HTTP/2 stream like
>the
>>> >TCP
>>> >> Formalizing how to encapsulate the same thing in http2 doesn't
>buy
>>> >much
>>> >> native framing and get rid of the duplication simple
>encapsulation
>>> >has (for

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 13:33:45 +0900

>  - WSDATA's 8-bit flags field, is the ws flags [fin][rsv1-3][opcode]

We may want END_STREAM as well.
Corresponding http2 frames can substitue ping / pong / close (note:
WebSocket ones have payload though h2 ones don't). Shoud we have opcodes
for them?

On Fri, Nov 21, 2014 at 1:13 PM, Andy Green <[email-masked]> wrote:

> 5.2.2 has a needless flushing thing on top... in a real ws setup pieces of
> data can arrive any old way they should be passed on as they come or
> coalsced according to intermediary policy, I don't see the point.
>

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 13:13:02 +0800

>We may want END_STREAM as well.

Good point... and to be friendly with http2's obfuscation goals, PADDING.

It means we need an extra byte to hold all the flags... well it's not too bad per-frame.

If so, we could also do everything the same reusing using DATA with a flag "WS", indicating the payload is [WS_FLAGS byte + ws payload].  That has an advantage --->

>WebSocket ones have payload though h2 ones don't). Shoud we have

We should probably keep the ws ones too.  Things like ws echo services do use the payload.  Actually endpoints can prefer the http2 ones if the see they can use them and know how to.  But supporting both everyone's happy.

Ws has his own close codes and payload, if we try mapping those on http2 codes etc it might be a step too far.  So keeping ws close frame will be good.

Close is a bit tricky if we try to say that a WSDATA close is an implicit http2 stream close everybody has to understand that, even if they were intermediaries that didn't really understand WSDATA and were just passing it through.  That's probably too much to ask.

If we re-use DATA with a WS flag, the extra advantage is all http2 intermediaries and endpoints already know the meaning of DATA frame with END_STREAM, so even if they do not follow what WS flag on DATA means, they can still follow the stream lifecycle without extra work.

How realistic is it to get a DATA flag bit assigned now?

-Andy

>> >>> semantics to some implementation-defined limit that is friendly
>for
>> >>mux'd
>> >>> http2 transport.
>> >>>
>> >>Thanks.
>> >>
>> 5.2.1 is like pure tunneling, it's simple alright but it's bloated
>for
>> small frames.
>>
>>  - use the headers scheme you defined to negotiate the connection +
>ws
>> protocol
>>
>>  - WSDATA's payload, is the ws frame payload... the ws framing
>disappears
>> completely into the http2 framing.
>>
>> >>> >> >On 15 October 2014 00:00, Amos Jeffries
><[email-masked]>
>> >>> >wrote:
>> >>> >> >>>>>> HTTP/2 discussions has focused on how intermediaries
>want
>> >>to
>> >>> >> >>>>>> pragmatic concern, or do we expect DATA stream
>buffering
>> >to
>> >>> >> >>>>>> that such a gateway can't carry ws anyway unless its
>aware
>> >>of
>> >>> >> >>> So hang on a second here. If we say that ws frames can't
>be
>> >>split
>> >>> >> >> If a ws frame *has* to be that long, not doing so would
>block
>> >>the
>> >>> >> >> If the ws data is so critical to get transmitted fast why
>is
>> >>that
>> >>> >> I am the author of libwebsockets, we are adding http2 support
>at
>> >>the
>> >>> >> moment.  The basic http2 serving is done and works for http,
>but
>> >>> >we're all
>> >>> >> aggregation" by just one mux element spewing an endless ws
>frame
>> >>and
>> >>> >> >and mapping down to h2 frames as a sequence of octets would
>be
>> >>fine.
>> >>> >> >But you seem to both agree with my reasoning and disagree
>with
>> >my
>> >>> >> >> identical to the opaque payload of a DATA frame on an
>HTTP/2
>> >>> >CONNECT
>> >>> >> >the mapping from the high level frames to the h2 primitives.
>If
>> >>the
>> >>> >>
>> >>> >> Yeah.
>> >>> >>
>> >>> >> >spec identifies how ws is negotiated and framed within h2,
>its
>> >>not
>> >>> >> >they'll do for quite some time...) have to deal with tcp
>which
>> >>> >offers
>> >>> >> themselves transparently inside http2 mux connections, using
>the
>> >>> >RFC6455
>> >>> >> upgrade on their individual session ID, do the extra RTT and
>tx
>> >>data
>> >>> >> masking.
>> >>> >>
>> >>> >> above that... the benefit we can get is map the RFC6455
>framing
>> >on
>> >>to
>> >>> >http2
>> >>> >> will do anything, it should indeed be define how to map
>RFC6455
>> >>> >framing on
>> >>> >> http2 framing.
>> >>> >>

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 14:15:00 +0800

On 21 November 2014 13:13:02 GMT+08:00, Andy Green <[email-masked]> wrote:
>If so, we could also do everything the same reusing using DATA with a
>flag "WS", indicating the payload is [WS_FLAGS byte + ws payload]. 
>That has an advantage --->
>
>We should probably keep the ws ones too.  Things like ws echo services
>do use the payload.  Actually endpoints can prefer the http2 ones if
>the see they can use them and know how to.  But supporting both
>everyone's happy.
>
>Ws has his own close codes and payload, if we try mapping those on
>http2 codes etc it might be a step too far.  So keeping ws close frame
>will be good.
>
>Close is a bit tricky if we try to say that a WSDATA close is an
>implicit http2 stream close everybody has to understand that, even if
>they were intermediaries that didn't really understand WSDATA and were
>just passing it through.  That's probably too much to ask.
>
>If we re-use DATA with a WS flag, the extra advantage is all http2
>intermediaries and endpoints already know the meaning of DATA frame
>with END_STREAM, so even if they do not follow what WS flag on DATA
>means, they can still follow the stream lifecycle without extra work.
>

Ah since the endpoints negotiated ws using HEADERS, actually we could do without a WS flag on DATA.

The fact that it's DATA for stream ID xxx and that ID went through a ws handshake means both sides and all intermediaries present during the ws handshake can know it is ws protocol in the DATA without an explicit flag.

So we could just use DATA for this without any modification.  Peers that understand websockets can know in some streams there's a WS_FLAGS byte at the start of the DATA payload.  Peers that don't understand websockets just move opaque DATA payloads around as usual and only see normal DATA semantics going on.

-Andy

>>> On 21 November 2014 11:38:26 GMT+08:00, Andy Green
><[email-masked]>
>>> wrote:
>>> >
>>> >
>>> >>> I'm agreeing with what was already written by someone else on
>the
>>> >>thread.
>>> >>>
>>> >>> >> >>>> I think the problem is not buffering in HTTP/2 per-se
>but
>>> >the
>>> >>> >> >>>> DATA frame (de-)aggregation that can happen if the
>frames
>>> >are
>>> >>> >> >> entire HTTP/2 connection until all bytes of that frame
>were
>>> >>> >delivered
>>> >>> >> >> anyway. So you trade off buffering that single frame at
>the
>>> >>> >sender,
>>> >>> >> >>> based semantics and so on can take place above that in
>the
>>ws

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 15:41:19 +0900

On Fri, Nov 21, 2014 at 12:38 PM, Andy Green <[email-masked]> wrote:

> intention during ws discussion was an intermediary can fragment frames same
> as how tcp packets may be fragmented.
>
> ... I don't see that.  If I keep spamming 16MB frames even on one stream
> on a consumer link your latency goes to pieces and if something is doing
> multiple instances of it even a big pipe will feel pain.
>
>
The spec says:
  Length:
  The length of the frame payload expressed as an unsigned 24-bit integer.
Values greater than 2^14 (16,384) MUST NOT be sent unless the receiver has
set a larger value for SETTINGS_MAX_FRAME_SIZE.
So by default 2^14 is the upper limit. Larger frames will be used only if
the receiver wants.

> horse' since it's easier for people to jump in and tell you you're doing it
> wrong ^^
>
> -Andy
>
> >version[2]

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 14:58:55 +0800

On 21 November 2014 14:41:19 GMT+08:00, Yutaka Hirano <[email-masked]> wrote:
>> On 21 November 2014 11:30:46 GMT+08:00, Yutaka Hirano
><[email-masked]>
>> wrote:
>stream

Okay.  But if buffering whole ws frames no matter how big was the plan, and the one end wanted to send a 16MB ws frame, the other end would say, "sure" you can send me >16K and he'd spam his 16MB frames.  Default 16K limit doesn't help much because the server is complicit in thinking sending huge ws frames is good.

I think we're agreeing, that is not what anybody wants to see.

-Andy

>doing it

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 16:03:03 +0900

On Fri, Nov 21, 2014 at 3:15 PM, Andy Green <[email-masked]> wrote:

> The fact that it's DATA for stream ID xxx and that ID went through a ws
> handshake means both sides and all intermediaries present during the ws
> handshake can know it is ws protocol in the DATA without an explicit flag.
>
> So we could just use DATA for this without any modification.  Peers that
> understand websockets can know in some streams there's a WS_FLAGS byte at
> the start of the DATA payload.  Peers that don't understand websockets just
> move opaque DATA payloads around as usual and only see normal DATA
> semantics going on.
>
> I don't like the idea of using the existing DATA frame.

1. An intermediary must flush data when it sees an end of message (FIN).
IIUC we don't have such flag in the plain http2 (we had END_SEGMENT but it
is gone). So "just move opaque DATA payloads around" is not enough.
2. With explicit negotiation, I know that all peers in the communication
path understand ws-over-http2 perfectly and there is no chance to
misunderstand DATA in http2 and DATA in ws2, in theoretically. In reality,
that is not true. The history of WebSocket is the history of battles with
mis-implemented proxies (Note that the native WebSocket has the explicit
negotiation but proxies modify payloads without understanding it). I would
like to reduce the possibility of such confusion.

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 15:37:49 +0800

On 21 November 2014 15:03:03 GMT+08:00, Yutaka Hirano <[email-masked]> wrote:
>> >It means we need an extra byte to hold all the flags... well it's
>not
>> >too bad per-frame.
>> >
>> Ah since the endpoints negotiated ws using HEADERS, actually we could
>do
>> without a WS flag on DATA.
>>

Hm he can only flush what he can send due to tx credit and priority.  But okay.

Maybe he should use priority to make ws connections more realtimey... no intermediary is going to want to hang on to stuff for long because the memory costs money.

And any peer in the path that's smart enough to note that stream did a ws upgrade can snoop the flags byte and follow this explicitly.

Yeah.

I suggest it because it reduces the demands on http2 itself to handle this to nothing.... no new flags and no new frame type.

>like to reduce the possibility of such confusion.

What I witnessed in Hybi is "the history of Websockets is the history of battles" to get agreement about any specific implementation that can go forward ^^

Well there are two other choices discussed today then

1) DATA with a WS flag

2) WSDATA

If it's going to be WSDATA, interoperability might be much worse if it's not widely understood since it's not part of http2, and peers just drop what they don't understand.

If there was a WS flag on DATA, is it reasonable to think we can still get than in as part of http2 or has that boat sailed?

-Andy

---

From: Robert Collins <[email-masked]>
Date: Fri, 21 Nov 2014 21:33:03 +1300

> So we could just use DATA for this without any modification.  Peers that understand websockets can know in some streams there's a WS_FLAGS byte at the start of the DATA payload.  Peers that don't u…

Right, that was my point. As long as intermediaries that don't
understand its websockets won't /break/ things, we don't need anything
above the base framing (at least so far as I can tell).

-Rob

---

From: Robert Collins <[email-masked]>
Date: Fri, 21 Nov 2014 21:44:47 +1300

On 21 November 2014 20:03, Yutaka Hirano <[email-masked]> wrote:

> 1. An intermediary must flush data when it sees an end of message (FIN).

But intermediaries won't be buffering: they'll be forwarding as soon
as they can, since h2 is being established from the get-go as an
interactive protocol.

> is gone). So "just move opaque DATA payloads around" is not enough.

Why not? Any ws ignorant h2 proxy that holds buffers for indeterminate
periods because e.g. it wants to virus scan uploads/downloads is just
as likely to a) reject unknown frames or b) inspect and scan ws
frames. I don't see how we prevent that by using a custom frame.

> path understand ws-over-http2 perfectly and there is no chance to

The proposed negotiation in the draft I saw didn't guarantee that (see
my comments in this thread) because options isn't graph-capable: an
intermediary that is capable advertises its capability, not the
capability of peers on the other side of it, and no replacement
negotiation has been proposed that I've seen.

> like to reduce the possibility of such confusion.

WebSockets' HTTP/1.1 upgrade hack was a clever way to avoid a new port
for a new protocol layered on top of 20 years of proxy evolution.
We're not *at all* in the same situation now, and I think we should
bear that in mind.

-Rob

---

From: Yutaka Hirano <[email-masked]>
Date: Fri, 21 Nov 2014 18:08:00 +0900

On Fri, Nov 21, 2014 at 5:44 PM, Robert Collins <[email-masked]>
wrote:

> Hmm... Is that specified?
Are such buffering proxies prohibited explicitly?

> We can prevent the situation with WS_DATA, because a ws ignorant h2 proxy
must not transfer the custom frame.
The spec says:
  Implementations MUST discard frames that have unknown or unsupported
types.

Unable to communicate is better than relying on intermediaries that do not
understand what they are doing, I think.

> negotiation has been proposed that I've seen.
I will think more on this point.

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 17:39:46 +0800

>understand what they are doing, I think.

I think this approach will continue the tradition of there being no workable way to do this on http2.

Eg, adding a flag on http2 DATA for WS payload type seems to answer your objections... you reckon that's no longer possible at this point?

-Andy

>> > negotiation but proxies modify payloads without understanding it).
>I
>> would

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 21 Nov 2014 23:57:13 +1300

On 21/11/2014 7:58 p.m., Andy Green wrote:
>>>> On Fri, Nov 21, 2014 at 11:42 AM, Andy Green
>>>> <[email-masked]>
>> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>>>> I think you have to take the atomic large ws frame
>>>>>>> thing as a
>>>> genuine
>>>>>>> buffered one ws frame, you can't sit there spewing as
>>>>>>> much
>> http2
>>>> DATA
>>>>>> frame
>>>>>>> to suit the tx credit. But it's OK because the
>>>>>>> implementation can transparently
>> fragment
>>>> the
>>>>>> ws
>>>>>>> data using the ws message semantics... I think there's
>>>>>>> no
>> choice
>>>> but
>>>>>> to
>>>>>>> take that approach.
>>>>>> 
>>>>> I'm agreeing with what was already written by someone else
>>>>> on the
>>>> thread.
>>>>> 
>>>>> huge ws frame to transmit, you should fragment it using
>>>>> RFC6455
>>>> message
>>>> Strictly speaking, RFC6455 allows an extension to give
>>>> meaning to WebSocket frames, so merging / fragmenting frames
>>>> breaks such extensions. We discussed this problem in HyBi and
>>>> many of us said "don't care".
>>> 
>>> So I also see it as don't care.  I'm not even sure it's true
>>> since
>> the
>>> intention during ws discussion was an intermediary can
>>> fragment
>> frames same
>>>> In any case, an http/2 frame cannot be bigger than 2^24 (or
>>>> 2^14 without an explicit permission), so I think we don't
>>>> have to worry about DoS.
>>> 
>>> on a consumer link your latency goes to pieces and if something
>>> is
>> doing
>> The spec says: Length: The length of the frame payload expressed
>> as an unsigned 24-bit integer. Values greater than 2^14 (16,384)
>> MUST NOT be sent unless the receiver has set a larger value for
>> SETTINGS_MAX_FRAME_SIZE. So by default 2^14 is the upper limit.
>> Larger frames will be used only if the receiver wants.
> 
> Okay.  But if buffering whole ws frames no matter how big was the
> plan, and the one end wanted to send a 16MB ws frame, the other end
> would say, "sure" you can send me >16K and he'd spam his 16MB
> frames.  Default 16K limit doesn't help much because the server is
> complicit in thinking sending huge ws frames is good.
> 

You switched between "receiver" and "server" there in your concepts.

Overlooking the fact that intermediaries in HTTP can cut DATA up into
smaller frames and stitch them together into larger ones as needed
hop-by-hop. So what your sender gets told it can send on the
client->proxy connection may not be what gets delivered to the server.

That was the major reason for WSDATA not being HTTP/2 DATA. That flags
byte being mentioned at the beginning of DATA payload will get screwed
over and moved around too much. Unless it is at fixed positions of the
total WS stream inside the DATA payloads.

But then, as has been mentioned since HEADERS initiated a WS stream,
why is DATA needing to be re-flagged specially? it is opaque payload
bytes of the stream already flagged as being a WS transaction.

Amos

---

From: Andy Green <[email-masked]>
Date: Fri, 21 Nov 2014 19:53:40 +0800

>You switched between "receiver" and "server" there in your concepts.

Well okay, it's not my day job.

>total WS stream inside the DATA payloads.

What does 'too much' mean... it should not attempt to preserve the original ws frame sizes, just send http2 frames of a size that suit it when an amount of ws payload data that suits it has appeared.  So the bit of tx logic that chops up the ws payload data, takes care of deferring FIN until the last chunk.  And the receiver doesn't have to do anything just pass up the frames it got with the ws flags it got.

What's the problem?

>bytes of the stream already flagged as being a WS transaction.

I think it's alright being an implicit attribute of the stream.

But I understood the general objection from Hirano-san, it's that there might be dumb proxies who didn't follow the update activity on that stream and might parse what they see there as... http2 data.

'Magic bad proxies' was a stick that extinguished a lot of discussion on hybi.... he's not going to get fired for shaking that stick at us.  It doesn't damage the argument to point out there's no evidence the bad proxy exists.  But if it's WSDATA and http2 wraps up without containing WSDATA this stuff will never get deployed, because if anybody doesn't understand WSDATA the connection won't work even if the handshake does, lol

-Andy

>Amos
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2.0.22 (MingW32)
>
>iQEcBAEBAgAGBQJUbxqJAAoJELJo5wb/XPRjdp8H/1s4aNBEEwSItqQSOYilrCZC
>+OHb1mTSTHHW5np5zPdbk3qOlyoz0jZJw1l5RAivq5xFYBsR2uyoaTBfgePAaYND
>j3rX7oA57h3smbq7mKeYi+fwJNOtqYwL6sJgDwd3dho1TFbL9+EfhOuWOpidl9SG
>FgpiK1k1KzZ/67pnZx9bcsI+yyOghYvU1l14O2Wx5nKA6hudo8HJgAZDQok1OMin
>j/z8QRZcuS+zUq8skSIpv4TUDYXoUUG8ztz4VDwmXt+MglEahLkaw6nucO3Rx3l8
>RCKDz+MZD+Q1lGOeSaf9XC/Y+XUUcEaYyvBnnhs8omr3yLyTFid8946qVvz/u4Q=
>=Lxws
>-----END PGP SIGNATURE-----

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 22 Nov 2014 05:01:49 +1300

On 22/11/2014 12:53 a.m., Andy Green wrote:
> 
> 
> On 21 November 2014 18:57:13 GMT+08:00, Amos Jeffries 
> <[email-masked]> wrote: On 21/11/2014 7:58 p.m., Andy Green 
> wrote:
>>>> 
>>>> 
>>>>>>>>>> I think you have to take the atomic large ws
>>>>>>>>>> frame thing as a
>>>>>>> genuine
>>>>>>>>>> buffered one ws frame, you can't sit there
>>>>>>>>>> spewing as much
>>>>> http2
>>>>>>> DATA
>>>>>>>>> frame
>>>>>>>>>> data using the ws message semantics... I think 
>>>>>>>>>> there's no
>>>>> choice
>>>>>>> but
>>>>>>>>> to
>>>>>>>>>> take that approach.
>>>>>>>>> 
>>>>>>>> I'm agreeing with what was already written by
>>>>>>>> someone else on the
>>>>>>> thread.
>>>>>>>> 
>>>>>>>> huge ws frame to transmit, you should fragment it 
>>>>>>>> using RFC6455
>>>>>>> message
>>>>>>>  meaning to WebSocket frames, so merging / fragmenting 
>>>>>>> frames breaks such extensions. We discussed this
>>>>>>> problem in HyBi and many of us said "don't care".
>>>>>> 
>>>>>> Yeah extensions except for compression have just not
>>>>>> come into
>>>>> existence.
>>>>>> 
>>>>>> So I also see it as don't care.  I'm not even sure it's 
>>>>>> true since
>>>>> the
>>>>>>> In any case, an http/2 frame cannot be bigger than
>>>>>>> 2^24 (or 2^14 without an explicit permission), so I
>>>>>>> think we don't have to worry about DoS.
>>>>>> 
>>>>>> on a consumer link your latency goes to pieces and if 
>>>>>> something is
>>>>> doing
>>>>> The spec says: Length: The length of the frame payload 
>>>>> expressed as an unsigned 24-bit integer. Values greater
>>>>> than 2^14 (16,384) MUST NOT be sent unless the receiver has
>>>>> set a larger value for SETTINGS_MAX_FRAME_SIZE. So by
>>>>> default 2^14 is the upper limit. Larger frames will be used
>>>>> only if the receiver wants.
>>>> 
>>>> Okay.  But if buffering whole ws frames no matter how big
>>>> was the plan, and the one end wanted to send a 16MB ws frame,
>>>> the other end would say, "sure" you can send me >16K and he'd
>>>> spam his 16MB frames.  Default 16K limit doesn't help much
>>>> because the server is complicit in thinking sending huge ws
>>>> frames is good.
>>>> 
> 
> Overlooking the fact that intermediaries in HTTP can cut DATA up 
> into smaller frames and stitch them together into larger ones as 
> needed hop-by-hop. So what your sender gets told it can send on the
> That was the major reason for WSDATA not being HTTP/2 DATA. That 
> flags byte being mentioned at the beginning of DATA payload will
> get screwed over and moved around too much. Unless it is at fixed 
> positions of the total WS stream inside the DATA payloads.
> 
>> What does 'too much' mean... it should not attempt to preserve
>> the original ws frame sizes, just send http2 frames of a size
>> that suit it when an amount of ws payload data that suits it has
>> appeared. So the bit of tx logic that chops up the ws payload
>> data, takes care of deferring FIN until the last chunk.  And the
>> receiver doesn't have to do anything just pass up the frames it
>> got with the ws flags it got.
> 
>> What's the problem?

"with the ws flags it got" is the problem. HTTP/2 recipients know
nothing about any syntax embeded in the opaque DATA payload area.

> 
> But then, as has been mentioned since HEADERS initiated a WS
> stream, why is DATA needing to be re-flagged specially? it is
> opaque payload bytes of the stream already flagged as being a WS
> transaction.
> 
>> But I understood the general objection from Hirano-san, it's
>> that there might be dumb proxies who didn't follow the update
>> activity on that stream and might parse what they see there as...
>> http2 data.
> 
>> 'Magic bad proxies' was a stick that extinguished a lot of 
>> discussion on hybi.... he's not going to get fired for shaking
>> that stick at us.  It doesn't damage the argument to point out
>> there's no evidence the bad proxy exists.  But if it's WSDATA and
>> http2 wraps up without containing WSDATA this stuff will never
>> get deployed, because if anybody doesn't understand WSDATA the 
>> connection won't work even if the handshake does, lol

Nod. One of the flaws behind the basic concept of WebSockets is that
it imposes knowledge upon the underlying routing infrastructure - or dies.

Imagine how useful (or not) TCP would be if you could successfully
SYN/SYN+ACK a stream setup, then had to rely on every IP-router along
the pathway decoding and processing the layer-7 protocols inside the
TCP stream in order to pass on each packet - or would terminate the
connection. The Internet would be a very different and unfriendly place.

If its a DATA frame then recipients should be handling it as per the
HTTP/2 spec handling regardless of WS knowledge level. The DATA
payload is opaque and may be sliced and diced on any 8-bit boundary
within, so long as the end recipient gets it all in proper order.
(Think DATA approximates TCP DATA packet with HEADERS as the SYN
handshake preceeding it all).

So it comes down to a question of what WSDATA frames do exactly. Are
they there to signal length+etc information about formatted chunks of
the data stream? or are they there to transfer opaque payload blobs
around?
 - if they are there for passing opaque payload, then that overlaps
directly with DATA and the WS specific bits seem needless.
 - if they are there to signal lengths or type of pieces inside the
payload, then a DATA sequence with all the formatting meta data
opaquely inside it would be the right mapping IMHO. WS just needs to
be able to receive and parse it separately to the DATA frames which
deliver the bytes.

Amos

---

From: Jason Greene <[email-masked]>
Date: Fri, 21 Nov 2014 10:32:31 -0600

> On Nov 21, 2014, at 3:08 AM, Yutaka Hirano <[email-masked]> wrote:
> 
> On Fri, Nov 21, 2014 at 5:44 PM, Robert Collins <[email-masked]> wrote:

Intermediaries are likely to do some form of buffering because:

- Each hop can have its own frame size and flow control limits
- Environmental factors including capacity of the wire and the server vary how flow control will behave and how large a frame can/will be at each hop
- Intermediaries can coalesce multiple clients on the same connection forcing some massaging of the individual streams

On the other hand, what you say above is true, that it is in the intermediary’s interest to get rid of the data as soon as possible.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Andy Green <[email-masked]>
Date: Sat, 22 Nov 2014 00:42:09 +0800

>nothing about any syntax embeded in the opaque DATA payload area.

Well I think this boils down only to the "intermediary who doesn't know about tracking stream upgrade state corrupting DATA" thing.  Otherwise there's no problem chopping up ws frames.  If that's not it please help me with an example corruption sequence.

>handshake preceeding it all).

Yeah... libwebsockets anyway has a bytewise parser so he doesn't care about as much fragmentation as you like.

>the data stream? or are they there to transfer opaque payload blobs

Literally they are there to hide the payload from Hirano-san's theoretical intermediary that treats all DATA as something to corrupt, otherwise we can put it in DATA neatly.

>be able to receive and parse it separately to the DATA frames which

Hirano-san will throw his DATA-corrupting proxy at any solution involving non http2 data inside DATA, which applies to what you're saying there.  So it can't go forward unless Hirano-san recants.

At some point the protocol tunelling with full frames doesn't buy anything over the individual streams doing a full ws upgrade themselves (they can do that today on http2, right?) and then the DATA frames... will contain ws frames... the thing Hirano-san says is verboten due to the magic corruption proxies...

-Andy

>iQEcBAEBAgAGBQJUb2HsAAoJELJo5wb/XPRjfq0IAInV42knM225y7NprbTxEHdG
>fqIAyfGq7B6yZzsgVlPkG0v88CVLOqrzdmlqT8BVuBqna0Q6Fe97HTVjDMMoYyW0
>5bcpcNukq7AbdCGS0gcy2aGwiHb1NGLUr5N4/ZgNCQHxZmKIsk3r1ffgRh2friXh
>mzLDdAbXKOrt3TNDBzAib2sIQENxv/2vH3Fhc4NBesbWHhsDXqhdWlO9DVi6iALC
>YU0LrjZDptCEvrEZudOzlJP4UQ4cx0jUoFS31plZHuM8pVoPYt2nb4rnNzMieiTv
>JkKvO3+4cd2g6zd1rfmTDgr1yulKwj4D8EUzF+0HyARghNA6bnenlgi0GTdqTVg=
>=c2/+

---

From: Jason Greene <[email-masked]>
Date: Fri, 21 Nov 2014 10:54:47 -0600

> On Nov 21, 2014, at 5:53 AM, Andy Green <[email-masked]> wrote:
> 
> 
> 

A bad proxy that corrupts DATA frames is going to break more than web sockets, it will also break common HTTP use cases.

---

From: Jason Greene <[email-masked]>
Date: Fri, 21 Nov 2014 11:07:01 -0600

> On Nov 21, 2014, at 10:01 AM, Amos Jeffries <[email-masked]> wrote:
> 

I agree. The individual hop has the best understanding of how to optimally format its messages. I could, however, understand a desire to pass a hint as a flag that a particular frame is urgent, and should favor flushing as soon as possible.

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Nov 2014 07:34:34 +0800

On 22 November 2014 00:54:47 GMT+08:00, Jason Greene <[email-masked]> wrote:
><[email-masked]> wrote:
>into
>flags
>> What does 'too much' mean... it should not attempt to preserve the
>original ws frame sizes, just send http2 frames of a size that suit it
>when an amount of ws payload data that suits it has appeared.  So the
>bit of tx logic that chops up the ws payload data, takes care of
>deferring FIN until the last chunk.  And the receiver doesn't have to
>do anything just pass up the frames it got with the ws flags it got.
>> 
>> What's the problem?
>> 
>> But I understood the general objection from Hirano-san, it's that
>there might be dumb proxies who didn't follow the update activity on
>that stream and might parse what they see there as... http2 data.
>> 
>> 'Magic bad proxies' was a stick that extinguished a lot of discussion
>on hybi.... he's not going to get fired for shaking that stick at us. 
>It doesn't damage the argument to point out there's no evidence the bad
>proxy exists.  But if it's WSDATA and http2 wraps up without containing
>WSDATA this stuff will never get deployed, because if anybody doesn't
>understand WSDATA the connection won't work even if the handshake does,
>lol
>
>A bad proxy that corrupts DATA frames is going to break more than web
>sockets, it will also break common HTTP use cases. 

Yeah.

So what's the plan to get websockets working on HTTP2?  This is the place where we're supposed to be sorting that out.  But it's looking stalled.

Since there is no active plan right now, the default situation is a big nothing and perhaps a big crappy rollout of WSDATA frames that nobody supports leading to negotiated and then dead WS connections.

Is that an OK result for everybody?

As discussed - nobody disagreed with the logical workability of this - it can be done concisely over DATA today with zero change or delay to HTTP2.

The objection from Hirano-san is if we put [1 byte WS flags] + [ WS payload ] in HTTP2 DATA frames, "a bad proxy / intermediary" may think it's OK to mangle the content of DATA frames as if it was HTTP content.

Personally I don't think it's realistic, but I saw this argument block and destroy things on hybi, so we have to shrug and pretend it is a serious issue.  One of the reasons this argument is so damaging and unhelpful is when you ask, "how broken is that proxy exactly", there is never any limit to how deranged it is in theory, because it can be as deranged as you like, and the guy proposing it does not want to give up any ground by saying, "well, I guess it might not flip bits randomly then", or "yes it must understand http headers".  So we get into arguments about the characteristics of the nonexistant magic broken proxy rather than do engineering.

How about

1) Exchange fake HTTP headers after the ws upgrade completes, which is like GET "endless-crap" replied to by 200 mime-type: binary no content-length, the receiver of each fake header frame knows it's coming and swallows it.  So we help the magic broken proxy to get to a state where he's not going to look at the connection content.  Unless the proxy munges HTTP2 binary files... do we really believe that?

2) Add a CRC16/32 or so on the frame or message.... this is expensive to compute and check, though probably about on a par with the ws masking.

... any other ideas?

-Andy

---

From: Jason Greene <[email-masked]>
Date: Mon, 24 Nov 2014 19:01:23 -0600

> On Nov 24, 2014, at 5:34 PM, Andy Green <[email-masked]> wrote:
> 
> 
> 

My argument to that is that anything which would break web sockets also breaks HTTP/2. If web sockets uses HTTP/2 as the transport, then you should be able to rely on its semantics. That, I could understand the desire for a per-hop negotiation to ensure that specific features or behavioral characteristics are available. Ideally this would just be negotiating an extension.

> 2) Add a CRC16/32 or so on the frame or message.... this is expensive to compute and check, though probably about on a par with the ws masking.

Those options sound pretty bad. An extension would be better, and it doesn’t have to block the H2 spec.

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Nov 2014 09:14:13 +0800

On 25 November 2014 09:01:23 GMT+08:00, Jason Greene <[email-masked]> wrote:
>stream,
>discussion
>> So what's the plan to get websockets working on HTTP2?  This is the
>place where we're supposed to be sorting that out.  But it's looking
>stalled.
>> 
>> Since there is no active plan right now, the default situation is a
>big nothing and perhaps a big crappy rollout of WSDATA frames that
>nobody supports leading to negotiated and then dead WS connections.
>> 
>> As discussed - nobody disagreed with the logical workability of this
>- it can be done concisely over DATA today with zero change or delay to
>HTTP2.
>> 
>> The objection from Hirano-san is if we put [1 byte WS flags] + [ WS
>payload ] in HTTP2 DATA frames, "a bad proxy / intermediary" may think
>it's OK to mangle the content of DATA frames as if it was HTTP content.
>> 
>> Personally I don't think it's realistic, but I saw this argument
>block and destroy things on hybi, so we have to shrug and pretend it is
>a serious issue.  One of the reasons this argument is so damaging and
>unhelpful is when you ask, "how broken is that proxy exactly", there is
>never any limit to how deranged it is in theory, because it can be as
>deranged as you like, and the guy proposing it does not want to give up
>any ground by saying, "well, I guess it might not flip bits randomly
>then", or "yes it must understand http headers".  So we get into
>arguments about the characteristics of the nonexistant magic broken
>proxy rather than do engineering.
>
>My argument to that is that anything which would break web sockets also
>breaks HTTP/2. If web sockets uses HTTP/2 as the transport, then you

That's a good argument.

>should be able to rely on its semantics. That, I could understand the
>desire for a per-hop negotiation to ensure that specific features or
>behavioral characteristics are available. Ideally this would just be
>negotiating an extension.

Yes, or you can just design it to use what's in the base spec by using DATA and be sure you get a working link.

>> How about
>> 1) Exchange fake HTTP headers after the ws upgrade completes, which
>is like GET "endless-crap" replied to by 200 mime-type: binary no
>content-length, the receiver of each fake header frame knows it's
>coming and swallows it.  So we help the magic broken proxy to get to a
>state where he's not going to look at the connection content.  Unless
>the proxy munges HTTP2 binary files... do we really believe that?
>> 
>> 2) Add a CRC16/32 or so on the frame or message.... this is expensive
>to compute and check, though probably about on a par with the ws
>masking.
>> 
>

The problem is an extension needs the endpoints and everybody inbetween to support it, to get a websocket connection.  Which means for the foreseeable future, no ws on http2.  (Honestly I am pretty surprised http2 is nearly done and this has just withered on the vine.)

However every HTTP2 implementation supports HEADERS and DATA, so passing ws traffic using those will work from day 1 on all HTTP2 links.

-Andy

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 25 Nov 2014 14:18:25 +1300

On 25/11/2014 12:34 p.m., Andy Green wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>> 
>>>>>>>> Strictly speaking, RFC6455 allows an extension to
>>>>>>>> give meaning to WebSocket frames, so merging /
>>>>>>>> fragmenting frames breaks such extensions. We
>>>>>>>> discussed this problem in HyBi and many of us said
>>>>>>>> "don't care".
>>>>>>> 
>>>>>>> So I also see it as don't care.  I'm not even sure
>>>>>>> it's true since
>>>>>> the
>>>>>> than 2^14 (16,384) MUST NOT be sent unless the receiver
>>>>>> has set a larger value for SETTINGS_MAX_FRAME_SIZE. So
>>>>>> by default 2^14 is the upper limit. Larger frames will be
>>>>>> used only if the receiver wants.
>>>>> 
>>>>> was the plan, and the one end wanted to send a 16MB ws
>>>>> frame, the other end would say, "sure" you can send me >16K
>>>>> and he'd spam his 16MB frames.  Default 16K limit doesn't
>>>>> help much because the server is complicit in thinking
>>>>> sending huge ws frames is good.
>>>>> 
>>>> 
>>>> smaller frames and stitch them together into larger ones as 
>>>> needed hop-by-hop. So what your sender gets told it can send
>>>> on the client->proxy connection may not be what gets
>>>> delivered to the
>> server.
>>>> 
>>>> byte being mentioned at the beginning of DATA payload will
>>>> get
>> screwed
>>>> over and moved around too much. Unless it is at fixed
>>>> positions of
>> the
>> original ws frame sizes, just send http2 frames of a size that
>> suit it when an amount of ws payload data that suits it has
>>>> stream, why is DATA needing to be re-flagged specially? it
>>>> is opaque payload bytes of the stream already flagged as
>>>> being a WS transaction.
>>> 
>> there might be dumb proxies who didn't follow the update
>> on hybi.... he's not going to get fired for shaking that stick
>> at us. It doesn't damage the argument to point out there's no
>> evidence the bad proxy exists.  But if it's WSDATA and http2
>> wraps up without containing WSDATA this stuff will never get
>> deployed, because if anybody doesn't understand WSDATA the
>> A bad proxy that corrupts DATA frames is going to break more
>> than web sockets, it will also break common HTTP use cases.
> 
> Yeah.
> 
> So what's the plan to get websockets working on HTTP2?  This is
> the place where we're supposed to be sorting that out.  But it's
> looking stalled.
> 
> Since there is no active plan right now, the default situation is
> a big nothing and perhaps a big crappy rollout of WSDATA frames
> that nobody supports leading to negotiated and then dead WS
> connections.
> 
> As discussed - nobody disagreed with the logical workability of
> this - it can be done concisely over DATA today with zero change or
> delay to HTTP2.
> 
> The objection from Hirano-san is if we put [1 byte WS flags] + [
> WS payload ] in HTTP2 DATA frames, "a bad proxy / intermediary"
> may think it's OK to mangle the content of DATA frames as if it was
> HTTP content.
> 
> block and destroy things on hybi, so we have to shrug and pretend
> it is a serious issue.  One of the reasons this argument is so
> damaging and unhelpful is when you ask, "how broken is that proxy
> exactly", there is never any limit to how deranged it is in theory,
> because it can be as deranged as you like, and the guy proposing it
> does not want to give up any ground by saying, "well, I guess it
> might not flip bits randomly then", or "yes it must understand http
> headers". So we get into arguments about the characteristics of the
> nonexistant magic broken proxy rather than do engineering.

I think Hirano-san's proxy case is a good one to work with. But limit
it to being a HTTP/2 compliant proxy. That limits it to
slicing/aggregating DATA frames any way it likes but not re-ordering
or changing payload bytes unless there are transformations which the
HTTP/2 headers permit (the TCP analogy).

A truely broken magic proxy does not exist yet in any HTTP/2
implementations and HTTP/2 itself is taking the "hard line" of
defining RST_STREAM or GOAWAY to be used when any kind of binary level
broken proxy is detected.

So the magic proxy can take your 1-byte flags + 6 byte WS payload DATA
frame and slice it. Delivering 1 byte DATA frame followed by 6 byte
DATA frame to the server. This is particularly likely to happen if
your WS client writes the flags and WS payload in separate operations
to the HTTP/2 library/backend.
 - all you need to resolve this is a length+flags+payload structure to
the WS data. You can do that at the HTTP//2 stream level, or inside
the DATA frame payload.
 - if you dont know the length in advance then stream level is best
for HTTP/2 compatibility.

Amos

---

From: Robert Collins <[email-masked]>
Date: Tue, 25 Nov 2014 14:30:31 +1300

> broken proxy is detected.

Right.

> your WS client writes the flags and WS payload in separate operations

Or it can do 7 x 1 byte frames, or one 8 byte frame. Shrug. Its a
series of octets.

> the WS data. You can do that at the HTTP//2 stream level, or inside

The WS internal framing already has this. Its how it works on TCP, and
its trivial.

> for HTTP/2 compatibility.

ws does, always, because the api is messages, not octets.

-Rob

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Nov 2014 09:42:53 +0800

>HTTP/2 headers permit (the TCP analogy).

Well I agree, you're basically saying it has to survive working over an http2 proxy, that's clearly required.

Hinaro-san is saying something further than that, which is, to his mind, every proxy must be able to look inside every DATA packet and understand it according to an HTTP state (post-upgrade to ws) that it may be incapable of understanding.

That magic broken proxy is in trouble anyway because nothing stops me opening an http2 stream and negotiate an upgrade on it to xxx-123 protocol and then all the http2 DATA packets are full of xxx-123 protocol data, without needing any blessing from anyone.

It's already legal in http2 so his objection seems to shine no light on anything.

>to the HTTP/2 library/backend.

Yes... the parsing code must be immune to fragmentation attacks anyway, this is no extra burden than that.

>for HTTP/2 compatibility.

I didn't really understand what you meant here.

>From what was discussed, the frame length is always known since it's in the http2 frame.  The message length can be indeterminite (or infinite) in ws.

-Andy

>iQEcBAEBAgAGBQJUc9jhAAoJELJo5wb/XPRjDsAH/21qGwDJVC2iOEYDeBEb5rBG
>kgYAeMri9yLrRis/4Pv0QtEQgxxMNVkWxDP+NNKmwCkhz3dF3X/y0v8ryXyr40aE
>Kw6+m5vsLEtQ1/d7Xxbkz8iqJBZsM/1fklRGJ2G0YKLUED4KDf2hKmnkQwbDi7la
>LUV1t6ij50gT9VmfW6r0dUwnLOt2M9qYRbTbGmgQbuc/6Zsqqp1VNtSOTpG/HC3+
>xl3iZCDTJYyi01lNXZNsqzlXrx8zLT0i7KdE+QM+Vg/v708rUAp8sF26jxzsB/M2
>d9oL2jXBJQ4ydh16Od5lwJrKlCqK7QHg3QHxRe8wmP9dabzHUVef3QmqPtoXJwg=
>=9I96

---

From: Yutaka Hirano <[email-masked]>
Date: Tue, 25 Nov 2014 17:16:38 +0900

Hmm, OK, I agree that "magic bad proxy" is a magic word.

Let me confirm:
 1 We use http2 DATA frames to convey WebSocket data.
  1.1 Each http2 frame payload consists of one-byte WebSocket header and
WebSocket data payload.
 2. All intermediaries MUST understand WebSocket (i.e. If there is an
intermediary that doesn't understand WebSocket, the opening handshake must
fail).

Do you agree with these?

On Tue, Nov 25, 2014 at 10:42 AM, Andy Green <[email-masked]> wrote:

> Hinaro-san is saying something further than that, which is, to his mind,
> every proxy must be able to look inside every DATA packet and understand it
> according to an HTTP state (post-upgrade to ws) that it may be incapable of
> understanding.
>
> That magic broken proxy is in trouble anyway because nothing stops me
> opening an http2 stream and negotiate an upgrade on it to xxx-123 protocol
> and then all the http2 DATA packets are full of xxx-123 protocol data,
> without needing any blessing from anyone.
>
> >From what was discussed, the frame length is always known since it's in
> the http2 frame.  The message length can be indeterminite (or infinite) in
> ws.
>
> -Andy
>
> >Amos

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Nov 2014 18:36:42 +0800

>Hmm, OK, I agree that "magic bad proxy" is a magic word.

Thanks, I appreciate you gave it some thought.

> 1 We use http2 DATA frames to convey WebSocket data.

Yes.

> 1.1 Each http2 frame payload consists of one-byte WebSocket header and

Yes.  Basically there's nowhere to put the ws flags, we discussed and they won't fit in the http2 header without dropping some reserved bits, which is too restrictive.  So the idea at the moment is stick them as the first byte of every ws-mode http2 DATA frame.

There's an implied 1.2

1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any original ws framing is not respected.  Refragmenting will not be a theoretical corner-case, http2 will make it desirable to cut the ws-carrying DATA frame to fit the tx credit on the stream, so it will be normal the frames are refragmented to make the best use of the current tx credit for minimum latency.  The only heuristic is the last http2 frame-fragment created an incoming original ws frame must show the real FIN state of the original incoming ws frame, and the others all have FIN=0.

>intermediary that doesn't understand WebSocket, the opening handshake
>must fail).

Can you live with restricting it to "any intermediary capable of rewriting DATA" that does not understand ws-over-http2 must kill the opening handshake?  Because if there's an intermediary like a load balancer, we don't really care do we?  It's not going to screw anything up just move frames around.

How about in http2 spec, ask for

a) rewriting intermediaries MUST read HEADERS

b) if they see an UPGRADE header they do not know how to deal with, they must kill the stream

This also covers "upgrade xxx-123" case for http2.

>Do you agree with these?

I would like to see if we can find a less exclusive solution for the corruption problem.

Original ws did not give each intermediary a veto.  It's easy to deploy and from my experience reliable to use.  There's a big danger with the per-intermediary, default-no, veto, when it's just an adjunct to http2 and not part of it, we would define something that will end up undeployable.

-Andy

>> Well I agree, you're basically saying it has to survive working over
>an
>> http2 proxy, that's clearly required.
>>
>> It's already legal in http2 so his objection seems to shine no light
>on
>> anything.
>>
>> Yes... the parsing code must be immune to fragmentation attacks
>anyway,
>> this is no extra burden than that.
>>

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 25 Nov 2014 23:39:41 +1300

On 25/11/2014 9:16 p.m., Yutaka Hirano wrote:
> Let me confirm: 1 We use http2 DATA frames to convey WebSocket
> data. 1.1 Each http2 frame payload consists of one-byte WebSocket
> header and WebSocket data payload. 2. All intermediaries MUST
> understand WebSocket (i.e. If there is an intermediary that doesn't
> understand WebSocket, the opening handshake must fail).
> 
> Do you agree with these?
> 

(2) "all intermediaries" does not seem possible. To achieve it an
intermediary receiving a SETTINGS negotiating WS extansion must
already know the destination servers and upstream contact it will have
for all possible WS requests/connections sent by that client.

IMHO (2) should be phrased along the lines of intermediaries
advertising WS extension to HTTP/2 MUST be capable of fallback to WS
over TCP (or other such as HTTP/1.1) if any of their upstream peers do
not negotiate WS extension as well.

That way if anyone offers WS support they are guaranteeing that the WS
connections sent to them will be possible one way or another. It is
not the end clients business how they achieve WS, so long as they do.

Amos

---

From: Yutaka Hirano <[email-masked]>
Date: Tue, 25 Nov 2014 21:28:13 +0900

> Yes.  Basically there's nowhere to put the ws flags, we discussed and they
> won't fit in the http2 header without dropping some reserved bits, which is
> too restrictive.  So the idea at the moment is stick them as the first byte
> of every ws-mode http2 DATA frame.
> There's an implied 1.2
> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any original
> ws framing is not respected.  Refragmenting will not be a theoretical
> corner-case, http2 will make it desirable to cut the ws-carrying DATA frame
> to fit the tx credit on the stream, so it will be normal the frames are
> refragmented to make the best use of the current tx credit for minimum
> latency.  The only heuristic is the last http2 frame-fragment created an
> incoming original ws frame must show the real FIN state of the original
> incoming ws frame, and the others all have FIN=0.

I have no problem with that.

> Can you live with restricting it to "any intermediary capable of rewriting
> DATA" that does not understand ws-over-http2 must kill the opening
> handshake?  Because if there's an intermediary like a load balancer, we
> don't really care do we?  It's not going to screw anything up just move
> frames around.

We could ask for it, but I'm not sure if giving a special meaning for
"rewriting" is suitable from http/2 point of view and it is suitable to
request at this moment, given that http/2 is almost stabilized (not?).
cc: Martin.
We had a discussion about the capability check and agreed to use SETTINGS
frames.
http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1114.html

> Original ws did not give each intermediary a veto.  It's easy to deploy
> and from my experience reliable to use.  There's a big danger with the
> per-intermediary, default-no, veto, when it's just an adjunct to http2 and
> not part of it, we would define something that will end up undeployable.

Originally, I wanted ws-over-http to be on top of http/2 framing layer
without extending it.
IIUC that was rejected when END_MESSAGE was removed from the http/2 spec.

On Tue, Nov 25, 2014 at 7:36 PM, Andy Green <[email-masked]> wrote:

> Yes.  Basically there's nowhere to put the ws flags, we discussed and they
> won't fit in the http2 header without dropping some reserved bits, which is
> too restrictive.  So the idea at the moment is stick them as the first byte
> of every ws-mode http2 DATA frame.
>
> There's an implied 1.2
>
> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any original
> ws framing is not respected.  Refragmenting will not be a theoretical
> corner-case, http2 will make it desirable to cut the ws-carrying DATA frame
> to fit the tx credit on the stream, so it will be normal the frames are
> refragmented to make the best use of the current tx credit for minimum
> latency.  The only heuristic is the last http2 frame-fragment created an
> incoming original ws frame must show the real FIN state of the original
> incoming ws frame, and the others all have FIN=0.
>
> Can you live with restricting it to "any intermediary capable of rewriting
> DATA" that does not understand ws-over-http2 must kill the opening
> handshake?  Because if there's an intermediary like a load balancer, we
> don't really care do we?  It's not going to screw anything up just move
> frames around.
>
> Original ws did not give each intermediary a veto.  It's easy to deploy
> and from my experience reliable to use.  There's a big danger with the
> per-intermediary, default-no, veto, when it's just an adjunct to http2 and
> not part of it, we would define something that will end up undeployable.
>
> -Andy
>
> >

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Nov 2014 21:23:09 +0800

On 25 November 2014 20:28:13 GMT+08:00, Yutaka Hirano <[email-masked]> wrote:
>> b) if they see an UPGRADE header they do not know how to deal with,
>they
>> must kill the stream

Thanks.

The problem about making sure the stream upgrade fails if the intermediaries that might corrupt it don't understand the protocol upgraded to isn't ws specific, it's an http2-level issue for any kind of stream upgrades in http2.

Because stream upgrades generally are not forbidden in http2 they're already legit and will place non-http payload in DATA by default.

There's another general problem if the upgrade implies new frame types.  Because without some filtering by intermediaries at upgrade-time, the upgrade handshake will succeed but http2 demands unknown frame types be dropped.  So the end result is a connected but unusable upgrade if any intermediary does not understand the new frame types.  Instead any intermediary who objects to the upgrade should definitively kill it at upgrade handshake time.

Intermediaries that operate at frame level without interpreting or rewriting the payload have no reason to object to a stream upgrade.

Intermediaries that interpret or rewrite the payload

 - must recognize the upgrade header, and

 - either disable their interpretation + rewrite activity on the stream after upgrade to something they don't understand, or disallow the upgrade by killing the stream

Does it sound reasonable?

-Andy

>> On 25 November 2014 16:16:38 GMT+08:00, Yutaka Hirano
><[email-masked]>
>> wrote:
>> b) if they see an UPGRADE header they do not know how to deal with,
>they
>> must kill the stream
>>
>> >> >> this - it can be done concisely over DATA today with zero
>change
>> >or
>> >> >> delay to HTTP2.
>> >> >>
>> >> >> because it can be as deranged as you like, and the guy
>proposing
>> >it
>> >> > - all you need to resolve this is a length+flags+payload
>structure
>> >to
>> >> >From what was discussed, the frame length is always known since
>it's
>> >in

---

From: Martin Thomson <[email-masked]>
Date: Tue, 25 Nov 2014 11:05:40 -0800

> http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1114.html

I don't see how Andy's request is feasible.  The way HTTP/2 is
structured, intermediaries are permitted to collapse and re-assemble
DATA as they choose.  You cannot rely on frame boundaries to remain.

To that end, you either do as suggested up-thread and force support by
all intermediaries, or you add your own framing.

I don't know if you have considered this...
You could use CONNECT and only use the multiplexing features of the
protocol.  A special CONNECT would be rejected by all but those that
recognize it.  More so if you made the CONNECT pseudo-header fields
include a ":scheme" of "wss", which is illegal.  I note that we don't
identify what error/status is generated if you break that rule.

---

From: Jason Greene <[email-masked]>
Date: Tue, 25 Nov 2014 13:55:05 -0600

> On Nov 25, 2014, at 1:05 PM, Martin Thomson <[email-masked]> wrote:
> 

What is the benefit to preventing reframing? The negative of such a restriction is that intermediaries will be unable to utilize their knowledge of their network topology to improve performance.

Is it to prevent delays from overly coalescing messages? If so a simple urgency flag or could accomplish the same thing.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 25 Nov 2014 13:17:41 -0800

> What is the benefit to preventing reframing? The negative of such a restriction is that intermediaries will be unable to utilize their knowledge of their network topology to improve performance.

It would appear that Yukata and Andy want to use HTTP/2 frame
boundaries as the basis for WS frame boundaries.  That implies that
you need to prevent blind reframing.  Since reframing is an inherent
part of HTTP/2, I note that this would be unwise.  (And for more than
just the reasons you describe.)

---

From: Roberto Peon <[email-masked]>
Date: Tue, 25 Nov 2014 13:25:57 -0800

Sadly, since it was something that a few of us in the WG were pushing for,
as it would have allowed intermediaries without knowledge of protocols to
do what was right for them.
That is why I truly hope that the WS extension happens (and quickly)-- I'd
love to see loadbalancers relieved of having to introspect into the data
further than the HTTP2 frame layer in order to figure out how to deal with
it reasonably.
-=R

On Tue, Nov 25, 2014 at 1:17 PM, Martin Thomson <[email-masked]>
wrote:

> > What is the benefit to preventing reframing? The negative of such a
> restriction is that intermediaries will be unable to utilize their
> knowledge of their network topology to improve performance.
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 25 Nov 2014 22:27:10 +0100

> just the reasons you describe.)

I agree. Let's not reinvent application-aware chunking...
Also I note that we can already have tunnels over HTTP/2, so there's
nothing which prevents one form passing the whole WS stream inside.
Sure it's a double encoding, but that's really WS/1 over HTTP, regardless
of the HTTP version. We could later work on WS/2 which would benefit more
efficiently from H2 framing if that makes sense at all, but let's not try
to optimize first then fix...

Willy

---

From: Jason Greene <[email-masked]>
Date: Tue, 25 Nov 2014 15:28:15 -0600

> On Nov 25, 2014, at 3:17 PM, Martin Thomson <[email-masked]> wrote:
> 

Could you elaborate?

---

From: Jason Greene <[email-masked]>
Date: Tue, 25 Nov 2014 15:32:36 -0600

> On Nov 25, 2014, at 3:27 PM, Willy Tarreau <[email-masked]> wrote:
> 
> to optimize first then fix…

It seems there is a great deal of overlap between Web Sockets and HTTP/2, so I am questioning whether it makes sense to try to simply map what made since for HTTP/1 on top of HTTP/2.

---

From: Andy Green <[email-masked]>
Date: Wed, 26 Nov 2014 09:00:42 +0800

>DATA as they choose.  You cannot rely on frame boundaries to remain.

It does not prevent http2 reframing nor does it rely on frame boundaries to remain.

But the approach of sticking a byte in front of each DATA after ws upgrade would mean the reframer would need to understand that stream had upgraded and reframing would mean sticking a flags byte at the front of the split frame, and eliding the then-superflous flags byte when combining them.

That's asking a lot of what might be a dumb http2 router or whatever I agree.

>all intermediaries, or you add your own framing.

Yeah, although 'add your own framing' may be out of band by

 - using a different http2 frame type (which kills deployment), or

 - hiding the ws flags inside the DATA frame header (which needs http2 spec buyin),

or in-band by

 - tunelling ws completely as Willy says (which is inefficient bandwidth-wise and means nasty crap like masking lives on).

>identify what error/status is generated if you break that rule.

It sounds reasonable.  It was a regular suggestion on hybi to replace upgrade.  Honestly, I don't really know enough about the merits of either approach, although upgrade is what ws1 uses.

But I think the problem we identified about upgraded connections in http2 generally being negotiable, but that being divorced from the ability to pass data over them is a separate issue that needs addressing.  If native http2 has a way for intermediaries to veto upgrade activity they know they will make trouble on, that will be a better way to bound the problem than explicit buyin per-protocol.

-Andy

---

From: Andy Green <[email-masked]>
Date: Wed, 26 Nov 2014 09:28:26 +0800

>just the reasons you describe.)

Nah.

Like http2 framing, ws framing is perfectly fungible with no guarantees that what was sent is what is received, other than for the payload to be the right total amount eventually and in the right order.  Ws is also fine with frame fragmentation during transport as much as you like.

Since in the scheme described the DATA frame is the ws frame, fragmenting the DATA at the same time fragments the ws frame which is fine.

The only problem is we stuck the ws frame flags byte at the start of the DATA / ws frame.  If that's how it is, then the fragmentation action has to understand the stream upgraded to ws and manage adding / removing that extra byte when splitting or combining DATA.

If you have a suggestion for getting rid of the extra byte and just having pure ws payload in the DATA, that would make it completely transparent for http2 fragmentation.

-Andy

---

From: Yutaka Hirano <[email-masked]>
Date: Wed, 26 Nov 2014 10:32:48 +0900

That is what you called "super-inefficient" (HEADERS + DATAs).

On Wed, Nov 26, 2014 at 10:28 AM, Andy Green <[email-masked]> wrote:

> Like http2 framing, ws framing is perfectly fungible with no guarantees
> that what was sent is what is received, other than for the payload to be
> the right total amount eventually and in the right order.  Ws is also fine
> with frame fragmentation during transport as much as you like.
>
> Since in the scheme described the DATA frame is the ws frame, fragmenting
> the DATA at the same time fragments the ws frame which is fine.
>
> The only problem is we stuck the ws frame flags byte at the start of the
> DATA / ws frame.  If that's how it is, then the fragmentation action has to
> understand the stream upgraded to ws and manage adding / removing that
> extra byte when splitting or combining DATA.
>
> If you have a suggestion for getting rid of the extra byte and just having
> pure ws payload in the DATA, that would make it completely transparent for
> http2 fragmentation.
>
> -Andy
>

---

From: Andy Green <[email-masked]>
Date: Wed, 26 Nov 2014 09:34:02 +0800

>nothing which prevents one form passing the whole WS stream inside.

Well, there is this whole general, not-only-ws, subject of upgrade veto by intermediaries to ensure the link is happy with whatever the upgrade shovels into the DATA.

>to optimize first then fix...

Tunnelled websockets is a bit nasty though.

In addition to the bloat from adding another framing which is going to be painful for short ws frames, ws has this ugly masking stuff if you're not going to make any effort to redefine ws transport.

Overall though, if it's this approach or nothing, this approach will work (if the upgrade thing is looked at).  But we should maybe strive for a better way especially if it doesn't need any blessing by http2.

-Andy

---

From: Yutaka Hirano <[email-masked]>
Date: Wed, 26 Nov 2014 10:36:11 +0900

> not negotiate WS extension as well.

You are right.
What I wanted to say was that WS-unaware intermediaries should be rejected.

On Tue, Nov 25, 2014 at 7:39 PM, Amos Jeffries <[email-masked]> wrote:

> iQEcBAEBAgAGBQJUdFxtAAoJELJo5wb/XPRjjSEIALUD+ZERvbKsKqsN3XZqZ+Fn
> ogFDANbTeKcMjAH/0FNKLDF2+SlSeGFIvMT+cVtUBg7Cr8JSglPYpFea/ylP/K1n
> rKzDoN+nrM8zvLTf10+sjirlojkH2gtKc+QDKKDJnYAFyQhw75u2DiN5b31YmqZ+
> ZU9pQB8eyMF9vBVTtsle7wP4gujub8kXIDgDZtgGUnxYiAi1pAJlKIzMI3lfrogd
> +QMLt5nxZZe8EMUcLiUe6NC8xfORGQkHPq4rvDnz6Zz+62k1wkzjzzsoJfbJDspr
> 9J0cPleAzwTjryz80HUSca2o4amSTGdxeux+hB79szD0BxA6fRBRxDrEjf8hRcU=
> =0xoT

---

From: Andy Green <[email-masked]>
Date: Wed, 26 Nov 2014 09:35:44 +0800

>That is what you called "super-inefficient" (HEADERS + DATAs).

Sorry maybe I misunderstood your writeup.  It sounded like it wanted to send HEADERS + DATA per frame.

-Andy

---

From: Yutaka Hirano <[email-masked]>
Date: Wed, 26 Nov 2014 10:39:37 +0900

Sorry that the proposal was updated in another place:
https://github.com/yutakahirano/ws-over-http2/blob/master/ws-over-http2-message-mapping.md
I forgot to link the document.

In the github document, a WS message is encoded to HEADERS + DATAs.

---

From: Andy Green <[email-masked]>
Date: Wed, 26 Nov 2014 09:43:52 +0800

>In the github document, a WS message is encoded to HEADERS + DATAs.

OK... then I guess we could agree, calling it super inefficient was being kind ^^

What we're talking about now, HEADERS to make the upgrade one time and then pure DATA representing one ws frame, would be perfect.  But I don't know how to get rid of the WS_FLAGS byte in the payload right now to get there.

-Andy

>> On 26 November 2014 09:32:48 GMT+08:00, Yutaka Hirano
><[email-masked]>
>> wrote:
>> Sorry maybe I misunderstood your writeup.  It sounded like it wanted
>to
>> send HEADERS + DATA per frame.
>>
>> -Andy
>>
>> >> >On 25 November 2014 at 11:55, Jason Greene
><[email-masked]>
>> >> >wrote:
>> >> >>
>> >> >> What is the benefit to preventing reframing? The negative of
>such
>> >a
>> >> that what was sent is what is received, other than for the payload
>to
>> >be
>> >> the right total amount eventually and in the right order.  Ws is
>also
>> >fine
>> >> The only problem is we stuck the ws frame flags byte at the start
>of
>> >the
>> >> DATA / ws frame.  If that's how it is, then the fragmentation
>action
>> >has to
>> >> If you have a suggestion for getting rid of the extra byte and
>just
>> >having

---

From: Martin Thomson <[email-masked]>
Date: Wed, 26 Nov 2014 04:14:41 +0000

I'm not entirely sure yet, but it could be as simple as using CONNECT with
a ws[s]:// URI. That might seem crazy, but acceptance of that would
indicate that the intermediary supported the protocol, it would therefore
be hop by hop with an option at each hop to fall back to a non-multiplexed
form (i.e., CONNECT in the standard form with all the baggage of the two
layers), and you could even steal flags for the TBC and bin/text bits.

On Tue, Nov 25, 2014, 17:44 Andy Green <[email-masked]> wrote:

> What we're talking about now, HEADERS to make the upgrade one time and
> then pure DATA representing one ws frame, would be perfect.  But I don't
> know how to get rid of the WS_FLAGS byte in the payload right now to get
> there.
>
> -Andy
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 26 Nov 2014 08:04:58 +0100

Hi Andy,

On Wed, Nov 26, 2014 at 09:34:02AM +0800, Andy Green wrote:
> Well, there is this whole general, not-only-ws, subject of upgrade veto by
> intermediaries to ensure the link is happy with whatever the upgrade shovels
> into the DATA.

No, it's avoiding conflating layers. A long time ago some LB vendors used
to rely on packet matching to extract HTTP cookies. *packet matching*. They
would work on most applications, except the few who used to perform multiple
calls to the write() syscall to build the response and headers.

Here it's the same. The application wants to send whatever length is possible
and it does not care about the NIC's MTU, nor the HTTP chunk/frame size because
these are transport issues. What matters is that application level *messages*
are not molested. That's what allows an application to transport 1-MB messages
over a link with MTU=1500 and an HTTP server with 16kB buffers. Please note
that WS already supports fragmentation explicitly to ensure that messages are
fragmented and reassembled on framing boundaries.

> Tunnelled websockets is a bit nasty though.

I know but the underlying protocol will definitely work. From this point
we can try to improve it.

> In addition to the bloat from adding another framing which is going to be
> painful for short ws frames, ws has this ugly masking stuff if you're not
> going to make any effort to redefine ws transport.

One more reason for starting from this point : it's much easier to say "ok
we can do this and it works, now how can we improve it". I'm seeing several
first steps :
  - replace the handshake (not needed anymore)
  - remove masking (not needed anymore)
  - map framing on H2 framing (fragmentation, not messages size)

> Overall though, if it's this approach or nothing, this approach will work (if
> the upgrade thing is looked at).  But we should maybe strive for a better way
> especially if it doesn't need any blessing by http2.

By doing it the correct way, we should gain the benefits of H2. One of them
is the multiplexing that was being worked on 2 years ago already and which
was not trivial in standalone WS. We certainly can do much better but things
need to be done the proper way (ie: turn something working non-optimally into
something working optimally).

Regards,
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 26 Nov 2014 22:03:55 +1300

On 26/11/2014 2:43 p.m., Andy Green wrote:
> 
> 
> On 26 November 2014 09:39:37 GMT+08:00, Yutaka Hirano wrote:
> What we're talking about now, HEADERS to make the upgrade one time
> and then pure DATA representing one ws frame, would be perfect.
> But I don't know how to get rid of the WS_FLAGS byte in the payload
> right now to get there.
> 

andy, why are you still talking about "upgrade" ?
 moving the entire underlying protocol connection from HTTP/1 to WS
was a HTTP/1 artifact that should no longer be necessary at all in
HTTP/2. Just open a new WS-type stream to a peer advertising WS
support and presto it contains WS protocol.

Amos

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 27 Nov 2014 10:53:01 +1100

> further than the HTTP2 frame layer in order to figure out how to deal with

+1

Non-constructive comments from me are that this is simply way too late in
the process for this to be considered and I suspect that with the framing
layer we have given ourselves, all possible solutions will now be pretty
ugly.

It comes down to a choice between:

a) defining new frame types for WS.   Which is going to require very
explicit support from intermediaries (imagine if we had to do that in TCP
for every new protocol!).

b) reusing the existing HEADER/DATA frames, but they miss some useful
semantics and have been so conflated with HTTP semantics that we are back
to websockets pretending to be HTTP - which is exactly the sort of protocol
mis-use that we were chartered to resolve!

Currently I can't pick between these two ugly ducklings.       We long ago
missed the opportunity of coming up with a truly multiple protocol web
framing layer that would support diverse semantics such as HTTP, websocket
and whatever the future may bring.    So failing that, our best solution is
to pick whatever hack we have to sooner rather than later so that we can at
least hammer in special case handling for websockets into intermediaries
and hope that no new semantics emerge.

Will try to find the time in the next week to experiment with some of the
options, so we can be a bit more constructive about which we prefer.

regards

-- 
Greg Wilkins <[email-masked]>  @  Webtide - *an Intalio subsidiary*
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Andy Green <[email-masked]>
Date: Thu, 27 Nov 2014 08:02:47 +0800

>the process for this to be considered and I suspect that with the

Hmmmm it's late in the process to consider it because everybody put off working on it!

It's not a new observation this http2 stuff will be carrying ws traffic and clearly Hirano-san has been trying to move it on for a long while.

>mis-use that we were chartered to resolve!

No... for http2 it's helpful to think about this as a generic stream upgrade to opaque data in DATA issue, not particularly websockets.

-Andy

---

From: Andy Green <[email-masked]>
Date: Thu, 27 Nov 2014 08:05:32 +0800

>andy, why are you still talking about "upgrade" ?

Because we will indeed be upgrading the default http2 protocol for the stream to ws.

So we need to signal to intermediaries that we're changing the rules for that stream.

If the intermediary thinks... "oho... I don't think so" then he can NAK the upgrade right there and avoid a nasty situation the endpoints wrote a cheque that the intermediaries cannot cash.

-Andy

>iQEcBAEBAgAGBQJUdZd6AAoJELJo5wb/XPRjyIsIAKeF/vNn/i3bzWSCtnbUu5Rb
>wx7RuzWsbYfRaXWXjjPxtdfNAT1uNHuL10gaoznD+GVBatJtQp2AeaBo0zqpLHob
>oNIFqHNMQ1AJOdFIWfVv4LRcvefMTFDGXk1WOjtV7sw0DrQFiKTJY47/cKTYwzCO
>IgTpXqjjTsNcHbRiSh2UXOVsn7q9aoQsdHy6bIfTbc9+8OATjK62VauDtczKKJtt
>dL3pjs7dFePpCd3taw9apf9QoQynNh+DoIPHXXCDo1IkPdyXnHq+wcUF7iA9q+9T
>/vetWnSEEAjkmmuobxjO5rjTEMGhJDQ6ARV6PjfFuH+P5l9Gsb9pVdTsuiEHjc8=
>=oK+7

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 29 Nov 2014 17:26:34 +1300

On 27/11/2014 1:05 p.m., Andy Green wrote:
> 
> 
> On 26 November 2014 17:03:55 GMT+08:00, Amos Jeffries wrote: On
> 26/11/2014 2:43 p.m., Andy Green wrote:
>>>> 
>>>> 
>>>> What we're talking about now, HEADERS to make the upgrade one
>>>> time and then pure DATA representing one ws frame, would be
>>>> perfect. But I don't know how to get rid of the WS_FLAGS byte
>>>> in the payload right now to get there.
>>>> 
> 

But the stream does not exist as a HTTP/2 stream until its started as
a WS stream. So I think there is a paradigm shift needed in the thinking.

>> So we need to signal to intermediaries that we're changing the

Opening a WS stream does that.

> 
>> If the intermediary thinks... "oho... I don't think so" then he
>> can NAK the upgrade right there and avoid a nasty situation the
>> endpoints wrote a cheque that the intermediaries cannot cash.
> 

If the intermediary advertised WS support in its SETTINGS previously
then it has already guaranteed that it will understand and properly
handle the WS streams.

So...

> support and presto it contains WS protocol.

Overall I am thinking that WS just needs 1) a WSHEADERS frame type to
initiate WS streams, and 2) a SETTINGS parameter to advertise support
for WSHEADERS streams.
 DATA contains payload for the stream either WS or h2. Both handled in
the same way by intermediaries.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Fri, 28 Nov 2014 20:51:45 -0800

All it needs is a new DATA frame which allows the signaling of
end-of-message (with a single bit), and a mechanism (HEADERS would do if it
can be flow controlled on the one stream) to modify metadata

Everything else is there in the protocol already.

-=R

On Fri, Nov 28, 2014 at 8:26 PM, Amos Jeffries <[email-masked]> wrote:

> iQEcBAEBAgAGBQJUeUr6AAoJELJo5wb/XPRjvxoH/2VG9f7uN+VZM2LJAghbEDK3
> nEUoptPw4TIG/nt2LgBWU8qCnL+kNfVzfSWSEdw2FFI9XdS3T99eAAWXBEJcHsE6
> Vf2AxpoassRYXr13p2Tmjo6Lz0/uLhoFgNYePGdSEyKS75tULaoQP0rM05H3w3Ym
> EWPvn4vtFkmWgRm/39lMvGBZjvnTlaKBEIl8rrSRE2BUDAbfSBzcr/2V8ZOpPHm+
> DnetSTFP2UpLRu1Ufsir4bnTh3vhf6810HBTz5ylO8rUYIuqkGJwLuGmjSIE2V3z
> uB/tVxr7903nw6o4/FtwGwvJKXt8QJujEswdEan2SoQhBkJaO+tfBaq4hyO3Wc0=
> =WW/v