Subject: [hybi] Multiplexing extension spec draft 03

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 28 Feb 2012 11:46:40 -0500

http://tools.ietf.org/html/draft-tamplin-hybi-google-mux-03
Diff http://tools.ietf.org/rfcdiff?url2=draft-tamplin-hybi-google-mux-03.txt

- Added a section to describe terms "physical channel" and "logical channel"
- Now the order of extensions in Sec-WebSocket-Extensions determines where
those extensions are applied to frames (before or after (de)multiplexing)
-- Also added explanation of the reason why such a mechanism is needed
-- based on http://www.ietf.org/mail-archive/web/hybi/current/msg09410.html
- Added note discussing how implementors should design quota assignment to
avoid starvation, etc.
-- based on http://www.ietf.org/mail-archive/web/hybi/current/msg09402.html
- Defined operation identifiers such as _Fail the Physical Channel_
- Clarified how events, operations on logical/physical channels must be
handled
- Revised layout of section 7 for readability
- Added DropChannel command to handle logical channel level errors
- Added "Timeout" section

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 28 Feb 2012 09:41:05 -0800

>>- Now the order of extensions in Sec-WebSocket-Extensions determines where those extensions are applied to frames (before or after (de)multiplexing)

Huh. _Order is significant_ ??

Is that even supported by the WS base spec? If so, is there precedence with other HTTP headers introducing order sign.?

Somehow I feel uncomfortable with introducing order sign. in a header .. 

And what if Sec-WebSocket-Extensions header appears more than once (don't remember right now
if thats allowed for that specific header)? If so, aren't intermediaries allowed to reorder headers?

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 28 Feb 2012 14:00:58 -0500

On Tue, Feb 28, 2012 at 12:41, Tobias Oberstein <[email-masked]
> wrote:

> >>- Now the order of extensions in Sec-WebSocket-Extensions determines
> where those extensions are applied to frames (before or after
> (de)multiplexing)
>
> Is that even supported by the WS base spec?

Yes, as noted in http://tools.ietf.org/html/rfc6455#section-9.1

> If so, is there precedence with other HTTP headers introducing order sign.?
>

What kind of HTTP headers are you worrying about?

> allowed to reorder headers?
As described in RFC 6455 p48.

HTTP (RFC 2616) also prohibits reordering such multiple header entries. See
the section 4.2 Message Headers.

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 28 Feb 2012 11:52:35 -0800

>Yes, as noted in http://tools.ietf.org/html/rfc6455#section-9.1 

Ok. Sorry. Thanks for pointing out!
 
>What kind of HTTP headers are you worrying about? 

I was wondering whether _outside_ of WS, there is a case with HTTP headers
where order is significant.

Or is this order significance of that particular header a exotic "speciality" of WS?

---

From: John Tamplin <[email-masked]>
Date: Tue, 28 Feb 2012 15:00:28 -0500

> where order is significant.

Isn't Accept-Encoding/etc sensitive to the order (in the case of equal q
values)?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 28 Feb 2012 14:58:39 -0800

>Isn't Accept-Encoding/etc sensitive to the order (in the case of equal q values)?

Don't know for sure, but that would make sense. If so, then there is a wide-spread
"precedence case" for order significance in HTTP header. So I retract this point --

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Feb 2012 00:11:24 +0100

> if thats allowed for that specific header)? If so, aren't intermediaries allowed to reorder headers?

<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.2p.5>

"Multiple message-header fields with the same field-name MAY be present 
in a message if and only if the entire field-value for that header field 
is defined as a comma-separated list [i.e., #(values)]. It MUST be 
possible to combine the multiple header fields into one "field-name: 
field-value" pair, without changing the semantics of the message, by 
appending each subsequent field-value to the first, each separated by a 
comma. The order in which header fields with the same field-name are 
received is therefore significant to the interpretation of the combined 
field value, and thus a proxy MUST NOT change the order of these field 
values when a message is forwarded."

But that doesn't necessarily mean it's a good idea.

Best regards, Julian

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 29 Feb 2012 11:03:20 +1100

Some comments

3.

   If WebSocket payload data is masked by a per-frame key, such masking
   is applied to frames for each logical channel separately.

I'm not sure what it means to apply masking to a logical channel
separately?  Surely a mask is applied to the frame that carries it before
the MUX extension interprets the payload?

5.  Flow Control   Each logical channel, including the implicitly
created channel 1, is
   initially given a quota of bytes that may be transmitted in each
   direction without acknowledgement.  It is illegal to send more bytes
   than the remaining send quota, and the receiver MUST _Fail the
   Logical Channel_ for any sender that does so.

Should this not be a failure of the physical channel as it indicates a
failure of the other ends MUX implementation?

   This send quota is
   replenished via control frames as the receiver processes the data.

s/This/The/

7.  Multiplex Control Frames

nice clarifications in this section! - specifically "Objectve channel ID"

Is the text diagram a standard?  I find the + - - - - - + lines confusing
and would prefer to see:

      0 1 2 3 4 5 6 7
     +---------------+
     | Objective     |
     : channel ID    :
     | (8-32)        |
     +-----+---------+
     | Opc | Opcdata |
     +-----+---------+
     | Additional    |
     : data          :
     | (0-?          |
     +---------------+

7.1.  Multiplex Control Opcodes   0 - AddChannel request (only from client)
      Create a new logical channel, exactly as if a new connection were
      received on a separate transport connection,

s/exactly //
s/a new logical channel/the objective channel/

probably need to state somewhere that it is an error to add and already
added channel.

      0 - uncompressed

This is a bad name for this encoding scheme as it implies there is a
compressed scheme (OK delta encoding is a compression... but you know what
I mean).   How about calling this "absolute" or "complete" or "plain" ?

      The initial quota for the new logical channel is 0, so the client
      may not send any data for this connection until the AddChannel
      response is received.

Does this mean that control frames can be sent for a channel before the add
channel response is received?

   3 - DropChannel

Can a dropped channel be readded later?

11.  Fairness

   A multiplexing implementation MUST ensure reasonable fairness among
   the logical channels.  This is accomplished in several ways:

   Receiver side

   o  The receiver MAY limit the other peer's send quota of a logical
      channel by not replenishing the send quota to make sure that any
      logical channel cannot dominate its buffer space on the sender.

s/on the sender//   as it is the buffer space on the receiver that is of
concern

Also we should say that the receiver does not need to process frames from
different logical channels serially.  ie the processing of a message from a
logical channel should not defer the processing of messages on other
logical channels.

   o  The sender MUST fragment a message into smaller frames when it's
      too big so that that logical channel will occupy the connection
      and the other logical channels get stuck for long time.

Cumbersome.  How about: The sender MUST fragment a large message into
smaller frames to prevent a large message in a logical channel occupying
the physical channel and thus delaying messages in other logical channels.

   o  Logical channel frames that are sent SHOULD be limited in size
      (such as by refragmenting) when there is contention for the
      physical channel to minimize head-of-line blocking

How is this different from the previous?

13.  Nesting

I think we should disallow nesting.  Any intermediary that wants to MUX
channels understands MUX, so it can flatten.

cheers

---

From: Joakim Erdfelt <[email-masked]>
Date: Wed, 29 Feb 2012 11:44:48 -0700

Would it be appropriate to reference RFC6455 Sec 1.3 Opening Handshake in
the section explaining the mux AddChannel request (mux Sec 7.1) headers?

Similar in the way that RFC6455 Sec 1.3 references RFC2616?

--
Joakim Erdfelt
[email-masked]

---

From: John Tamplin <[email-masked]>
Date: Wed, 29 Feb 2012 15:03:09 -0500

> Similar in the way that RFC6455 Sec 1.3 references RFC2616?

Sounds good.

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 29 Feb 2012 15:11:23 -0500

Hi Greg,

Thanks for commenting throughout.

> the MUX extension interprets the payload?
John, may I confirm what this text is intended to mean?

> 5.  Flow Control   Each logical channel, including the implicitly created channel 1, is
I wondered if it's useful if there's a choice to drop only one logical
channel. As quota is independent among logical channels, we don't have to
close the physical channel.

But yes, as you said, this means there's bug in the sender's multiplexing.
If no one opposes, I'll change this to _Fail the Physical Channel_ from the
next version.

>    replenished via control frames as the receiver processes the data.

Thanks

> Is the text diagram a standard?  I find the + - - - - - + lines confusing
Just used the same formatting as RFC 6455. But yes... for this case, it's
just making it ugly. I'll remove them.

ok

> s/a new logical channel/the objective channel/

ok

> probably need to state somewhere that it is an error to add and already
good point.

> I mean).   How about calling this "absolute" or "complete" or "plain" ?
"plain" sounds good. how about identity as like content-encoding of HTTP?

> Does this mean that control frames can be sent for a channel before the
> add channel response is received?
>
>
ok. it might be misleading to say this within the context of quota. We
should just say "any frame MUST not be sent over the logical channel until
AddChannel response is received" somewhere else. I'll try to fix this.

> Can a dropped channel be readded later?
Hmm. One case we need to consider is:

- S sends a frame(ID=1, body="blah")
- C issues a DropChannel(ID=1)
- C issues a AddChannelReq(ID=1)

We need to be able to determine if the data frame should be dropped or not
when these three happen at almost same time.

But I think this is not a problem. C should drop any frame received after
issue of DropChannel but before AddChannel response.

11.  Fairness
> s/on the sender//   as it is the buffer space on the receiver that is of

This text is intended to note that a receiver can try to maintain fairness
of inbound traffic between logical channels even if the sender doesn't
employ good algorithm to assign transmission slot to each logical channel.
That's why there's "on the sender".

Hmm, but maybe I should revise this like...

The receiver MAY limit the send quota of a logical channel by not
replenishing it to make sure that any logical channel doesn't dominate the
connection.

> logical channel should not defer the processing of messages on other
OK. Let's add some text to note that point.

> the physical channel and thus delaying messages in other logical channels.

Great. I'll take it.

> How is this different from the previous?
Ah,... maybe I've duplicated.

> channels understands MUX, so it can flatten.
Thanks for opinion.

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 29 Feb 2012 15:13:17 -0500

Thanks. I'll do so in the next version.

On Wed, Feb 29, 2012 at 15:03, John Tamplin <[email-masked]> wrote:

> On Wed, Feb 29, 2012 at 1:44 PM, Joakim Erdfelt <[email-masked]>wrote:
>

---

From: John Tamplin <[email-masked]>
Date: Wed, 29 Feb 2012 15:33:46 -0500

> John, may I confirm what this text is intended to mean?

This is similar to the idea of compressing logical channels vs. the
physical channel.  If masking is only applied to the physical channel, then
an intermediary mux will have to unmask the input stream and remask it as
part of the output stream.  Also, it keeps separate of layers cleaner.

I am open to changing this, but I think there are issues to consider both
ways.

> 5.  Flow Control
>>    Each logical channel, including the implicitly created channel 1, is

The physical channel may bundle unrelated connections, and dropping them
all because one sender failed seems excessive.  I could imagine a WS mux
that uses some non-WS protocol with flow control support, and simply relied
on the sender respecting the flow control.

Still, I won't object strongly.

> issue of DropChannel but before AddChannel response.

The intent was that the client could not reuse the channel ID until it
receives a response to the DropChannel request (which isn't actually
present in the doc).

>> channels understands MUX, so it can flatten.

Again, the point of this is to allow an intermediary to pass frames along
with minimal effort, rather than having to demux and remux.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 1 Mar 2012 09:03:46 +1100

> I am open to changing this, but I think there are issues to consider both

John,

Sorry, but I'm not understanding either choice you are implying exists.
Can you expand with examples?

Masking is a per frame operation, so I don't understand how it can be
applied to a logical channel?

>> But yes, as you said, this means there's bug in the sender's
>> multiplexing. If no one opposes, I'll change this to _Fail the Physical
>> Channel_ from the next version.
>>
>

I see a flow control violation not as a failure of one sender, but of the
mux implementation of the sender. But hopefully this violation will rarely
occur, so I guess it does not really matter what the error response is, so
long as there is one.

> with minimal effort, rather than having to demux and remux.

True, but the cost is passing on extra complexity and work to the server
and tools.  One of the purposes of having intermediaries is to offload work
from the servers, so if they can do the extra work to flatten multiple
multiplexed layers into a single channel space, then that will be of great
assistance to the servers.   Requiring muxing multiplexers to demux any
incoming muxed streams feels like a lot less work and complexity rather
than requiring all servers and tools to support an arbitrary depth of
nesting. (So says the server developer....   shat say the intermediary
developers?).

cheers

---

From: John Tamplin <[email-masked]>
Date: Wed, 29 Feb 2012 17:34:43 -0500

> applied to a logical channel?

You're right -- that was left over from when masking had a per-connection
key and maintained state.

> nesting. (So says the server developer....   shat say the intermediary

I think of a router or switch where you want to minimize the processing as
you pass bits from an input to an output.  A WS mux intermediary is likely
to be higher level than that, probably about the same as a load balancer so
it doesn't necessarily apply at that level.  I can see the argument both
ways, and I think we need to discuss the relative merits of both before
deciding.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 1 Mar 2012 09:42:39 +1100

> ways, and I think we need to discuss the relative merits of both before

Sure - just advocating one side of it, but can live with nested if we
determine an advantage.

Note that an intermediary that is demuxing/remuxing will have little work
to do on the average data packet - it will simply need to maintain a
mapping of inbound channelId to outbound channelID and update the channel
ID in the packets as it forwards them (this may be an argument for fixed
length channelID encoding).  This like flow control wont need to be fully
implemented, as the intermediary can just pass back quotas to the source.

cheers