Subject: Re: [hybi] Multiplexing extension spec draft 03

From: "Arman Djusupov" <[email-masked]>
Date: Mon, 19 Mar 2012 16:37:10 +0200

The procedure of closing the logical channel is not detailed enough. Closing
the logical channel should be performed in a way similar to closing the
websocket connection when no mux extension is used. We need a close control
frame to roundtrip in order to ensure the mutual agreement between the two
sides when a logical channel is closed. Currently the specification does not
require the side that receives a DropChannel control frame to reply to it
(unless I missed something), which does not ensure the graceful closure of
logical channels.

"When an endpoint received DropChannel, the endpoint MUST remove the logical
channel and the application instance that used the logical channel MUST
treat this as closure of underlying transport."
 
One side could be in the process of sending a message when it receives a
DropChannel frame, so it is important to ensure that the logical channel is
closed gracefully without dead channel frames left on the wire. The best way
to do it is to let the DropChannel frame roundtrip back to its sender. When
both sides have received the DropChannel frame they are in mutual agreement
to release the channel ID, being sure that no more frames with the same
channel ID are expected to arrive from the remote side.
 
With best regards,
Arman

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Takeshi Yoshino
Sent: Tuesday, February 28, 2012 6:47 PM
To: [email-masked]
Subject: [hybi] Multiplexing extension spec draft 03

http://tools.ietf.org/html/draft-tamplin-hybi-google-mux-03 

Diff http://tools.ietf.org/rfcdiff?url2=draft-tamplin-hybi-google-mux-03.txt

- Added a section to describe terms "physical channel" and "logical channel"

- Now the order of extensions in Sec-WebSocket-Extensions determines where
those extensions are applied to frames (before or after (de)multiplexing)

-- Also added explanation of the reason why such a mechanism is needed

-- based on http://www.ietf.org/mail-archive/web/hybi/current/msg09410.html

- Added note discussing how implementors should design quota assignment to
avoid starvation, etc.

-- based on http://www.ietf.org/mail-archive/web/hybi/current/msg09402.html

- Defined operation identifiers such as _Fail the Physical Channel_

- Clarified how events, operations on logical/physical channels must be
handled

- Revised layout of section 7 for readability

- Added DropChannel command to handle logical channel level errors

- Added "Timeout" section

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 23 Mar 2012 15:22:08 +0900

The idea in my mind is that the DropChannel command works similar to TCP
FIN/RST when no mux is used. I.e., each logical channel still exchanges
close frames (with their channel ID in extension data portion) and uses
status code in them (e.g. CloseEvent's code attribute), and after handling
them, each side issues the DropChannel command as well as TCP shutdown.

So, there's no such problem with graceful shutdown for logical channels'
traffic. In terms of control traffic, we may receive some mux command for
logical channel X after sending DropChannel command for X, but for now we
can just ignore any of control commands safely:
- FlowControl X -> safe to ignore
- DropChannel X -> no problem. it's totally fine that both sides drop the
same channel at the same time.
- AddChannel X -> if we receive this, it just means the other peer is
misbehaving

That said, we can't distinguish this from invalid commands sent by broken
peers (e.g. logical channel Y never existed but received command for Y).
Maybe it's better to have endpoints respond to DropChannel as well as close
frame to sync active channel list.

Thanks
Takeshi

On Mon, Mar 19, 2012 at 23:37, Arman Djusupov <[email-masked]> wrote:

> The procedure of closing the logical channel is not detailed enough.
> Closing the logical channel should be performed in a way similar to closing
> the websocket connection when no mux extension is used. We need a close
> control frame to roundtrip in order to ensure the mutual agreement between
> the two sides when a logical channel is closed. Currently the specification
> does not require the side that receives a DropChannel control frame to
> reply to it (unless I missed something), which does not ensure the graceful
> closure of logical channels.****
>
> ** **
>
> “When an endpoint received DropChannel, the endpoint MUST remove the logical channel and the application instance that used the logical channel MUST treat this as closure of underlying transport.“****
>
> ** **
>
> One side could be in the process of sending a message when it receives a DropChannel frame, so it is important to ensure that the logical channel is closed gracefully without dead channel frames left on the wire. The best way to do it is to let the DropChannel frame roundtrip back to its sender. When both sides have received the DropChannel frame they are in mutual agreement to release the channel ID, being sure that no more frames with the same channel ID are expected to arrive from the remote side.****
>
> ** **
>
> With best regards,****
>
> Arman
>
>

---

From: "Arman Djusupov" <[email-masked]>
Date: Fri, 23 Mar 2012 10:14:33 +0200

Hello Takeshi,

Using both a DropChannel and a close frame to close a logical channel seems
like overkill. Exchanging DropChannel control messages should be enough for
mutual shutdown of a logical channel. A more important issue is that if a
close frame is sent through an intermediary that does not understand the mux
extension, the intermediary may simply drop the physical channel /
connection. It's easy to imagine a frame compression intermediary that would
simply stop reading from the network connection once it encounters a close
frame (as per WS spec no data is expected to follow the close frame). So
probably the sending of a close frame should be avoided (i.e. prohibited)
for logical channels.

With best regards,

Arman

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Friday, March 23, 2012 8:22 AM
To: Arman Djusupov
Cc: [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

Thanks
Takeshi

 
 
With best regards,
Arman

---

From: "Arman Djusupov" <[email-masked]>
Date: Fri, 23 Mar 2012 11:38:18 +0200

Hello Takeshi,

Currently the specification does not make it clear which side assigns a
channel ID when an AddChannel control message is sent. I think this could be
made flexible; for example if Objective Channel ID is set to 0 then this
means that the client prefers the server to select the channel ID, if it is
greater than 1 then the client requests a specific channel ID.

Giving the client the option to select the channel ID can facilitate the use
of application-specific channel IDs, where the predefined logical channel ID
is reserved for a specific role, e.g. 1 - Video channel, 2 - Voice channel,
3 - Control channel, 4 - Subtitles channel (optional). 

It is also worth mentioning in the spec that the protocol specified by the
Sec-WebSocket-Protocol header MAY define a default set of logical channels
that are considered as being implicitly established when the mux extension
is agreed. Some protocols, as mentioned above, may require a fixed set of
channels with predefined IDs. Negotiating the same set of channel IDs using
an AddChannel request<->response every time a new connection is established
would not be efficient. It should be permitted for the "protocol" to imply a
default set of channels that don't need to be negotiated (at least this
should not be against the specification).

As an example, when Sec-WebSocket-Protocol set to "smart-playback" would
mean then once the mux is negotiated the following logical channels are
considered already established: 1 - Video, 2 - Voice, 3 - Control,  4 -
Subtitles.

I think this would make the specification much more versatile.

With best regards,

Arman

Thanks
Takeshi

 
 
With best regards,
Arman

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 23 Mar 2012 18:45:13 +0900

On Fri, Mar 23, 2012 at 17:14, Arman Djusupov <[email-masked]> wrote:

> Hello Takeshi,****
>
> ** **
>
> Using both a DropChannel and a close frame to close a logical channel
> seems like overkill.
>
The original aim of addition of DropChannel was only to provide a way to
terminate logical channel which is behaving abnormally by some command out
of that logical channel.

To make abstraction simple, I gave it a role to terminate a logical channel
normally, too. But I'm ok with getting normal close done by only one
command (one close frame, one DropChannel command, etc.)

> Exchanging DropChannel control messages should be enough for mutual
> shutdown of a logical channel. A more important issue is that if a close
> frame is sent through an intermediary that does not understand the mux
> connection.
I understand your concern. Actually, IIRC it was one of the motivation to
introduce close frame that a close frame tells intermediaries the end of
session instead of unreliable TCP FIN so that they can release resource for
the session more quickly.

John's original proposal was adding a new framing onto WebSocket framing to
implement multiplexing. There was some discussion about this and some
suggested  reuse of opcode in base framing. See this thread for example
http://www.ietf.org/mail-archive/web/hybi/current/msg07240.html.

> It’s easy to imagine a frame compression intermediary that would simply
> stop reading from the network connection once it encounters a close frame
> (as per WS spec no data is expected to follow the close frame). So probably
> the sending of a close frame should be avoided (i.e. prohibited) for
> logical channels.
>
I'd like to disallow any control frame with channel ID != 0 and have
corresponding mux commands for each of them if we take this approach rather
than just converting close frame into DropChannel ad-hoc.

Also we need to give DropChannel some field to hold the information close
frame has (status and reason).

---

From: "Arman Djusupov" <[email-masked]>
Date: Fri, 23 Mar 2012 13:51:48 +0200

I think it's OK to disallow control frames with a channel ID other than 0.
Since WS control frames apply to a physical channel,  I'm not sure they
should apply to mux logical channels too (it will always confuse
intermediaries that are not mux aware). We could actually create a single
mux command for delivering all types of logical channel control frames,
including custom opCodes that might be assigned in future. In such a case we
need a general purpose mux control command to move control frames of logical
channels with their opCode, RSV bits and payload. 

The more I think about it the fewer other options I see. A mux-aware
intermediary will have to forward close frames and their payload, while a
mux-unaware intermediary might kill the connection when encountering such a
frame. This means that the control frames of a logical channel need to be
enveloped into a mux command message, unless we figure out a better
solution.

With best regards,

Arman

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Friday, March 23, 2012 11:45 AM
To: Arman Djusupov
Cc: [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

Hello Takeshi,

like overkill.

To make abstraction simple, I gave it a role to terminate a logical channel
normally, too. But I'm ok with getting normal close done by only one command
(one close frame, one DropChannel command, etc.)

Exchanging DropChannel control messages should be enough for mutual shutdown
of a logical channel. A more important issue is that if a close frame is
sent through an intermediary that does not understand the mux extension, the
intermediary may simply drop the physical channel / connection.

It's easy to imagine a frame compression intermediary that would simply stop
reading from the network connection once it encounters a close frame (as per
WS spec no data is expected to follow the close frame). So probably the
sending of a close frame should be avoided (i.e. prohibited) for logical
channels.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 24 Mar 2012 03:57:36 +0900

Hello Arman,

On Fri, Mar 23, 2012 at 18:38, Arman Djusupov <[email-masked]> wrote:

> channel ID when an AddChannel control message is sent. I think this could
> be made flexible; for example if Objective Channel ID is set to 0 then this
> **
> Giving the client the option to select the channel ID can facilitate the
> use of application-specific channel IDs, where the predefined logical
> channel ID is reserved for a specific role, e.g. 1 - Video channel, 2 -
> Voice channel, 3 - Control channel, 4 - Subtitles channel (optional).
>
Some people were discussing intermediaries with mux capacity which
bundle/unbundle logical channels. Here, your proposal seems to try to use
channel ID values as service identifier (like port number of TCP/UDP). It
doesn't go together with such intermediaries. To make this work, we need to
ensure in the spec that channel ID is preserved between endpoints.

IMO, channel IDs should be hidden to application level and resource or
Sec-WebSocket-Protocol should be used for such mapping instead as well as
WebSocket w/o mux.

Are you also going to use mux to provide multiple channels (e.g. video,
voice, ...) but keep them grouped for one session (e.g. video conference
session consists of them)? If so, we need to ensure in the spec that
logical channels must not be unbundled into separate physical channels.

These new restrictions enables the usage as you described, but also
complicates the spec. For me as one of browser developer, they're
unnecessary. The current mux extension's goal is to provide logical
equivalent of normal WebSocket connections in one TCP connection. Your
proposal is beyond that. I'd like to get more opinions on these points.

> ****
> would not be efficient.
Providing a way to open multiple logical connections at once is
interesting. As I said above, basically I prefer to realizing it by adding
some way to issue AddChannel commands (different resources may be used to
tell which service each logical channel wants to access) on handshake.

How about this, for example?

C->S Sec-WebSocket-Extensions: mux; preopen="vc-video, vc-voice,
vc-control, vc-subtitle"
(each comma-separated element in preopen parameter's value means value of
Sec-WebSocket-Protocol)

S->C Sec-WebSocket-Extensions: mux; preopen="vc-video=1, vc-voice=2,
vc-control=3, vc-subtitle=4"
and the server sends back 4 AddChannel responses.

We don't have to pay additional RTT though change on request headers is
limited to Sec-WebSocket-Protocol.

We may also add some grouping annotation.

(OT. I don't want core specs to say that Sec-WebSocket-Protocol has
capability to change WebSocket protocol's behavior. Such modification
should be made using extension framework. Also, for now,
Sec-WebSocket-Protocol is an attribute for each logical channel. I'm
disinclined to invite headers other than Sec-WebSocket-Extensions to
determine physical channel (mux extension)'s behavior.)

> It should be permitted for the “protocol” to imply a default set of
> channels that don’t need to be negotiated (at least this should not be
> against the specification).****
>
> **
>
I don't object to make it able to define/use such preset. Maybe useful.

> **
>
> As an example, when Sec-WebSocket-Protocol set to “smart-playback” would
> Subtitles.****

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 24 Mar 2012 11:56:27 +0900

> and the server sends back 4 AddChannel responses.

Sorry. This example was wrong.

C->S
- Sec-WebSocket-Extensions: mux; add="vc-video=1, vc-voice=2, vc-control=3,
vc-subtitle=4"
(Format: <subprotocol>=<channel ID>)

S->C
- Sec-WebSocket-Extensions: mux; add="vc-video, vc-voice, vc-control,
vc-subtitle"
- 4 AddChannel responses

---

From: "Arman Djusupov" <[email-masked]>
Date: Tue, 27 Mar 2012 13:41:54 +0300

The sample is OK. But it would add complexity to the handshake. On the
positive side what we get as a result of such communication is actually
named-channels.

So the API can specify a channel by name e.g. socket["voice"].send(data) .

I would also like to read more opinions on this.

With best regards,

Arman

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Saturday, March 24, 2012 4:56 AM
To: Arman Djusupov
Cc: [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

C->S Sec-WebSocket-Extensions: mux; preopen="vc-video, vc-voice, vc-control,
vc-subtitle"

C->S

S->C

- 4 AddChannel responses

---

From: "Arman Djusupov" <[email-masked]>
Date: Tue, 27 Mar 2012 13:41:54 +0300

Hello Takeshi,

We are not implementing mux for providing video streams specifically, but
for general purpose mux support. I see the mux extension being useful not
only to the browser, but certainly to other types of software too which need
multiple channels in a single connection. It is applicable to all cases when
the client has to reduce the number of concurrent connections to the server
pool.

Client-mux intermediary-servers is just one specific use case, once which is
required by browsers. Consider for example a MMO game engine that also uses
mux between the client application and server poo, where the mux
intermediary does not simply bundle and unbundle channels but is designed
specifically for the game requirements and has specific logic on how to
forward each channel based on the channel ID or other criteria. We can
assume that in such cases an incompatible intermediary is not expected to
occur on the wire between the client and server. So the specification is not
required to prohibit the bundling/unbundling of the logical channel or the
change of channel ID.

BTW it still unclear which of the communicating sides assigns a channel ID
when an AddChannel request is sent. Since the Objective Channel ID can be
sent both ways, what should be the Objective Channel ID for an AddChannel
request message?

It's ok with me if the channel ID cannot be assigned by the client side,
since the application may identify the type of the channel by the first
message sent through the channel. This may cause an extra roundtrip, but it
is not a big deal if we implement some way of opening multiple logical
connections at once.

With best regards,

Arman

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Friday, March 23, 2012 8:58 PM
To: Arman Djusupov
Cc: [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

Hello Arman,

Are you also going to use mux to provide multiple channels (e.g. video,
voice, ...) but keep them grouped for one session (e.g. video conference
session consists of them)? If so, we need to ensure in the spec that logical
channels must not be unbundled into separate physical channels.

Providing a way to open multiple logical connections at once is interesting.
As I said above, basically I prefer to realizing it by adding some way to
issue AddChannel commands (different resources may be used to tell which
service each logical channel wants to access) on handshake.

(OT. I don't want core specs to say that Sec-WebSocket-Protocol has
capability to change WebSocket protocol's behavior. Such modification should
be made using extension framework. Also, for now, Sec-WebSocket-Protocol is
an attribute for each logical channel. I'm disinclined to invite headers
other than Sec-WebSocket-Extensions to determine physical channel (mux
extension)'s behavior.)

It should be permitted for the "protocol" to imply a default set of channels
that don't need to be negotiated (at least this should not be against the
specification).

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 27 Mar 2012 14:34:51 +0200

Hi,

On Tue, Mar 27, 2012 at 12:41, Arman Djusupov <[email-masked]> wrote:

> only to the browser, but certainly to other types of software too which
> need multiple channels in a single connection.
>
Yes, that's why I didn't object to use of mux ID but just showed concern
about intermediaries' bundle/unbundle.

>  It is applicable to all cases when the client has to reduce the number of
> concurrent connections to the server pool.****
>
> ** **
>
> Client-mux intermediary-servers is just one specific use case, once which
> is required by browsers. Consider for example a MMO game engine that also
> uses mux between the client application and server poo, where the mux
> occur on the wire between the client and server. So the specification is
> not required to prohibit the bundling/unbundling of the logical channel or
> the change of channel ID.****
>
> **
>
So, ... your point is that we should clarify which side assigns a channel
ID, and then you can divert WebSocket for such an application which gives
special meaning for each channel ID. Right?

My question is whether you're suggesting that we explicitly mention/allow
the use of channel IDs from application level or not.

> request message?****
Sorry. Yes, I'll do this clarification. My understanding is that the
AddChannel request specifies the channel ID for the new logical channel by
putting the ID in the Objective Channel ID part.

> connections at once
We may allow clients to send some value meaning "undefined" in the
Objective Channel ID part to get ID assigned by the server with AddChannel
response. For now, I'm not going to put this in the spec because of
additional complexity.

---

From: "Arman Djusupov" <[email-masked]>
Date: Tue, 27 Mar 2012 17:06:48 +0300

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Tuesday, March 27, 2012 3:35 PM
To: Arman Djusupov
Cc: [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

Hi,

think we should and if there is an intermediary that messes up with channel
IDs or completely strips off the mux, it would be normal for such an
application to fail (so such an application would require that there is no
de-multiplexing intermediary in between). Using fixed channel IDs like
TCP/IP port numbers is optional, there are alternative ways to negotiate the
protocol used by the logical channel using the AddChannel request handshake.
But the general idea is that Yes the application layer should be permitted
to access a specific logical channel by ID, protocol or other criteria, at
least the specification should not require that logical channels are
completely abstracted from the application layer. 

This is fine with me J

We may allow clients to send some value meaning "undefined" in the Objective
Channel ID part to get ID assigned by the server with AddChannel response.
For now, I'm not going to put this in the spec because of additional
complexity.

Both options are OK with me.

---

From: John Tamplin <[email-masked]>
Date: Tue, 27 Mar 2012 10:16:27 -0400

I really don't think exposing channels to JS is the proper thing to do.
 For one, that means that the application must be prepared to deal with
browsers which support and don't support MUX (and therefore don't even
provide that API), and connections that go through an intermediary that
doesn't support it.

Instead, I think applications that want multiple channels should just open
them as they do today.  If the browser and server negotiate MUX, then the
browser will aggregate those connections, and if not it will continue to
work.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 27 Mar 2012 17:48:28 +0200

OK. I will
- clarify that channel IDs are assigned by clients
- leave assignment algorithm up to implementations but warn that channel
IDs may be changed by intermediaries
but I will NOT
- add any constraint for intermediaries (so, intermediaries may reassign
channel IDs)
- explicitly expose channel ID to application layer (i.e. I won't add any
section like RFC 6455's section 12)

Are you fine with this, Arman?

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 27 Mar 2012 17:56:41 +0200

On Fri, Mar 23, 2012 at 12:51, Arman Djusupov <[email-masked]> wrote:

> I think it’s OK to disallow control frames with a channel ID other than 0.
> Since WS control frames apply to a physical channel,  I’m not sure they
> including custom opCodes that might be assigned in future.
>

Anyone objects to this? I heard from John that some people objected to put
everything into application data portion. Here, instead, we're going to
- use WebSocket frames' opcode to convey data frames' opcode of logical
channels
- use application data portion to convey control frames' opcode of logical
channels

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 27 Mar 2012 18:16:14 +0200

BTW, it looks like we need to clarify what to set to _Extensions In Use_.

I can come up with two options
- cut off mux token and everything after mux token that are applied to the
physical channel
- as-is

IIRC, the motivation to expose the list of extensions in use to JavaScripts
is to allow them to switch their behavior based on availability of
optimization (compress, mux), e.g. fallback to HTTP when compression is not
available. This can be done by server side too (We can reject the
connection if mandatory extensions are not available to ask the client side
to do something else. We also have close status 1010 to do this now), so
not critical, but respecting the motivation, we should expose extensions in
use "as-is".

---

From: John Tamplin <[email-masked]>
Date: Tue, 27 Mar 2012 12:38:16 -0400

On Tue, Mar 27, 2012 at 12:16 PM, Takeshi Yoshino <[email-masked]>wrote:

> IIRC, the motivation to expose the list of extensions in use to
> JavaScripts is to allow them to switch their behavior based on availability
> of optimization (compress, mux), e.g. fallback to HTTP when compression is
> not available. This can be done by server side too (We can reject the

hybi has no control over the JS API -- if you want to change the API
exposed to JS, that has to go through WHATWG.  This also doesn't seem
specific to any of the extensions we are discussing here, but rather a
general facility for exposing which extensions are in use.  AFAIK, there is
no standard API for the server side, so this would have to be done by each
implementation.

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 28 Mar 2012 09:48:34 +0300

Yes, Thanks!  That is fine with me.

With best regards,

Arman

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Tuesday, March 27, 2012 6:48 PM
To: John Tamplin
Cc: Arman Djusupov; [email-masked]
Subject: Re: [hybi] Multiplexing extension spec draft 03

OK. I will

- leave assignment algorithm up to implementations but warn that channel IDs
may be changed by intermediaries

but I will NOT