Subject: [hybi] Interleaved frames and x-google-mux

From: Greg Wilkins <[email-masked]>
Date: Mon, 15 Aug 2011 10:44:44 +1000

Moving this discuss here from google doc comments on

https://docs.google.com/document/d/14DYRxmRTPVhV1GbVXlxg8KVrS1SEq4UhCNq6q8iNOws/edit?hl=en&ndplr=1&pli=1

The comments said:

> Now that the websocket spec allows fragments of one message to be
> interleaved with fragments of another, are MUX blocks required?
> Can’t each ws frame be a mux block?   We already have the restriction that
> fragmentation can only be done if the extensions are known and understood.
> —gregory.j.wilkins

> Where does the WS spec allow this?  How do you distinguish which
> message a fragment is for if you allowed interleaving fragments from
> different messages?
> --jat

The specification now allows extensions to interleave messages - in
section 4.4 it says:

   o  The fragments of one message MUST NOT be interleaved between the
      fragments of another message unless an extension has been
      negotiated that can interpret the interleaving.

Thus I think it would be a significant simplification of the mux
proposal to remove the MUX block layer and directly use websocket
frames.

The current x-google-mux proposal has websocket frames (opcode 0x7)
wrapping MUX block frames which then wrap more websocket frames (which
may themselves contain more MUX blocks wrapping more websocket
frames).

With frame interleaving, we can avoid this nested/recursive framing
and instead just use extension data in the normal framing to add a mux
opcode and mux channel id.  I think the rest of the semantics can then
follow more or less as outlined in the proposal.

Are there any other reasons to maintain MUX blocks?

regards

---

From: John Tamplin <[email-masked]>
Date: Sun, 14 Aug 2011 21:01:36 -0400

Hmm, I didn't notice that going into the spec, and I am not sure that is a
good idea.  Now, an intermediary that doesn't understand all the negotiated
extensions can't even identify message boundaries.

I thought this was one of the reasons you were so unhappy with
deflate-stream, that it obscured message boundaries to those who didn't
understand the extension.

> Are there any other reasons to maintain MUX blocks?

You would have to do more than that, as there are other things being sent,
for example flow control information and add/drop channel messages.  Using a
separate frame to transfer that information makes it significantly more
expensive, since you are paying the header (including mask) cost for those.
 You would still need a sub-opcode, since we didn't put that into the base
spec we can't afford to use multiple opcodes for MUX.  So, I am not sure the
value of simplifying the structure justifies the cost.

Other than that, I agree that it could be changed as you suggest.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 15 Aug 2011 12:48:06 +1000

> understand the extension.

An intermediary that does not understand the extensions negotiated is
unlikely to be able to do anything meaningful with messages or
messages boundaries.   It certainly is not able to aggregate frames or
fragment, as this is explicitly prohibited by the spec.

It will still be able to identify frame boundaries and control frames,
which would be useful for buffering/flushing purposes.

> for example flow control information and add/drop channel messages.
> Using a separate frame to transfer that information makes it significantly
> more expensive, since you are paying the header (including mask) cost
> for those.  You would still need a sub-opcode, since we didn't put that into
> the base spec we can't afford to use multiple opcodes for MUX.

I'm not sure I'd describe a 2 or 6 bytes frame overhead as a
significant cost.   The most frequent control messages will be the
flow control ones.   If traffic is mostly 1 way then the back channel
is mostly idle and sending any data < the networks MTU is
approximately equal cost.    If the traffic is two way, then there is
lots of data in the back channel and 2/6 bytes extra will represent a
very small fraction of that.     OK this is a simplistic analysis, but
is there really concern about 2/6 byte frame overhead on control
frames?

> So, I am  not sure the value of simplifying the structure justifies the cost.

For my implementations, the cost is definitely justified.   The WS
frame inside a MUX block inside a WS frame is a significant
complication in the infrastructure.  It would not be so bad if it was
just framing, as I could simply reuse WS parsers/generators.  For me
the issue is that the nested WS framing is a full WS infrastructure
that can negotiate its own extensions, including another layer of MUX.
    Because MUX is achieved by wrapping frames rather than augmenting
them, I would think that MUX inside MUX is highly likely to occur if
there are multiple layers of aggregating devices (eg MUX in browser
then MUX in proxy).   So implementations and all the state associated
with them would have to be fully recursive, which can be needlessly
complicated, have lots of data copies and buffering and will make
resource allocation/management very difficult.

Consider the question - how many open channels are there in a server?
To answer this is difficult because there would not be a single
channel map of ID to channel. Instead there is a map of maps (of maps
...) with the leaves being channel IDs.

Without the frame wrapping, MUX would have to augment existing frames.
     This would put a bit more responsibility onto MUXing
intermediaries, as they would need to resolve streams that were
already MUXed (eg in browser) and map incoming channel IDs to outgoing
channel IDs.   A server implementation would have a single map of
channelID to channel, making resource accounting much simpler.

I'd pay 2/6 bytes extra on a few control messages for this simplicity any day!

cheers

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Mon, 15 Aug 2011 10:45:55 +0100

On 08/15/2011 03:48 AM, Somebody in the thread at some point said:
> On 15 August 2011 11:01, John Tamplin<[email-masked]>  wrote:
>> On Sun, Aug 14, 2011 at 8:44 PM, Greg Wilkins<[email-masked]>  wrote:
>>>

To be fair that is effectively what x-google-mux is doing anyway, it's 
an extension which allows bits of different messages to be interleaved 
with each other.

> which would be useful for buffering/flushing purposes.

I can't really imagine what an intermediary faced with a "roughly 
chopped message pate" it was unable to follow could really usefully do. 
  If the control frames are in a subframe context, actually it can't 
follow those either since in the case it doesn't know the extension it 
doesn't know the subframe coding scheme of the extension either.

Either it keeps seeing subframe close control frames coming and 
interprets them as main channel close or they're coded with a different 
control opcode or are not control opcodes then it doesn't even know any 
closing was happening.

So it could only really pass stuff through AFAICS without even 
understanding subchannel lifecycle; the best you can say is it would be 
able to be transparent same as current scheme.

> is mostly idle and sending any data<  the networks MTU is

I have to agree with Greg (and I believe Bob Gezelter was suggesting a 
similar simplified single header scheme months ago), as it is defined 
mux is a very complex to implement.  This would simplify it a bit.

However there are two separate issues suggested here:

1) Migrate mux block info inside a standard frame

2) 'Flatten' the mux so we just have frames in one dimension, not frames 
inside frames as currently

> just framing, as I could simply reuse WS parsers/generators.  For me

You can reuse parsers and generators with John's existing scheme; I did 
it, they're just wrapped in another layer.  You will lose the ability to 
mux mux parents aka "turtles all the way down".

> resource allocation/management very difficult.

That's not the problem I found implementing this.  The problem was it 
became so intricately nested it was very hard to debug and quickly 
packet dumps became effectively white noise.  When it doesn't "just 
work", you are quickly lost in a vast spew of logging.

Removing the nesting concept (I recall an argument about if it is nested 
or not, so read it as "prepending with mux headers" if you like) and 
just using one dimension of framing should simplify that.

Whether it's done with subopcode and subchannel tacked on to existing 
framing, or done with a "mux opcode" that has that is less critical.

> ...) with the leaves being channel IDs.

Yeah ie, flatten the mux structure and disallow nesting.

> I'd pay 2/6 bytes extra on a few control messages for this simplicity any day!

As I say I think you find where to put the subchannel and subopcode info 
is providing less simplification than flattening out the mux into a 
single flat stream of frames belonging to one subchannel or another.

If getting rid of "the mux opcode" is worthwhile should be something we 
can figure out based on "bang for the byte".

-Andy