Subject: [hybi] Intermedaries: coalescing with interleaved control messages

From: Tobias Oberstein <[email-masked]>
Date: Tue, 16 Aug 2011 01:34:28 -0700

In the absence of a negotiated extension / use of any reserved bits, an intermediary MAY modify
the fragmentation a of data message: coalesce fragments or fragment further.

How does that play with interleaved control messages?

Do control messages act as "barries" across which intermedaries must not coalesce?

For example, when an intermediary receives:

[MSG1-fragment1] [MSG1-fragment2] [CONTROL-message] [MSG1-fragment3]

Which of the following is allowed to the intermediary?

Coalescing

C1.
[MSG1-fragment12] [CONTROL-message] [MSG1-fragment3]

C2.
[MSG1-fragment123] [CONTROL-message]
 
C3.
[CONTROL-message] [MSG1-fragment123]

C4.
[MSG1-fragment1] [CONTROL-message] [MSG1-fragment23]

Fragmenting

F1.
[MSG1-fragment1] [MSG1-fragment2a] [MSG1-fragment2b]  [CONTROL-message] [MSG1-fragment3]

F2.
[MSG1-fragment1] [MSG1-fragment2a] [CONTROL-message] [MSG1-fragment2b]  [MSG1-fragment3]

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 16 Aug 2011 18:56:39 +1000

> Do control messages act as "barries" across which intermedaries must not coalesce?

It would be nice if this was the case, as that would allow a ping/pong
to act as a kind of acknowledgement.  ie when you receive the pong you
will know that all frames sent prior to the ping have been delivered
(at least to the destination host, but not the application).
Measuring latency between the pong and an application response to a
message prior to the ping would be a good way to remotely measure
application latency vs network latency.

cheers

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 16 Aug 2011 10:01:02 +0100

On 08/16/2011 09:34 AM, Somebody in the thread at some point said:

Hi -

> In the absence of a negotiated extension / use of any reserved bits,

I think this is "a negotiated extension that the intermediary does not 
understand".

> an intermediary MAY modify the fragmentation a of data message:
> coalesce fragments or fragment further.
>

You mean control *frames*.

> C1. [MSG1-fragment12] [CONTROL-message] [MSG1-fragment3]
>
> C2. [MSG1-fragment123] [CONTROL-message]
>
> C3. [CONTROL-message] [MSG1-fragment123]
>
> C4. [MSG1-fragment1] [CONTROL-message] [MSG1-fragment23]
>
> Fragmenting
>
> F1. [MSG1-fragment1] [MSG1-fragment2a] [MSG1-fragment2b]
> [CONTROL-message] [MSG1-fragment3]
>
> F2. [MSG1-fragment1] [MSG1-fragment2a] [CONTROL-message]
> [MSG1-fragment2b]  [MSG1-fragment3]

I think it should not bind together frames either side of the 
non-payload frame, but send the control frame out and do what it likes 
with contiguous frames of the same message.

But... what's the advantage to binding the frame content together in 
fact?  It implies it is willing to buffer multiple frame payloads (in 
order to find out the aggregate lengths).  That sounds kind of prone to 
not scaling.

Unless there's something the intermediary was planning to do that likes 
getting a lot of payload, eg compression, it sounds like a better plan 
the intermediary would just spill what was coming in as it came in 
packet by packet and not bind frames together at all.

-Andy

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 16 Aug 2011 02:19:29 -0700

> It would be nice if this was the case, as that would allow a ping/pong to act as
> a kind of acknowledgement.  ie when you receive the pong you will know
> that all frames sent prior to the ping have been delivered (at least to the
> destination host, but not the application).

Interesting usage .. didn't thought about that kind.

Can this be generalized?

I mean:

With base WS we have 1 ordered channel for data messages. Intermediaries must obviously not change ordering.

But we also have a 2nd channel: control. Within that channel, ordering must also be preserved.

Are there rules which introduce ordering constraints between the 2?

Having a message on the control channel act as a barrier for the data channel is one conceivable constraint. Or is it even
already implied (or meant) by the spec?

In the presence of some MUX with more channels: what about ordering constraints?

When channels had priorities, a message on channel A could act as reordering barrier for messages on channels with lower prio.

The control channel is then a highest prio channel .. it introduces barriers for all data channels ..

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 16 Aug 2011 02:33:01 -0700

> I think this is "a negotiated extension that the intermediary does not

I meant no extension has been negotiated .. that is both peers agreed
to speak the base WS protocol without any extension.

> You mean control *frames*.

Yep.  I use that synonymously for control, since control "messages" can't
be fragmented. Shouldn't do that probably;)

> I think it should not bind together frames either side of the non-payload
> frame, but send the control frame out and do what it likes with contiguous
> frames of the same message.
> 
> But... what's the advantage to binding the frame content together in fact?  It
> implies it is willing to buffer multiple frame payloads (in order to find out the
> aggregate lengths).  That sounds kind of prone to not scaling.
> 
> getting a lot of payload, eg compression, it sounds like a better plan the
> intermediary would just spill what was coming in as it came in packet by
> packet and not bind frames together at all.

I have no clue what intermediaries want to do and for what reasons. What I
was trying to find out is what they are _allowed_ to do by the spec.

My biggest worry is: some insance content inspection systems wants to
inspect on a per-message basis. To do that, it buffers all frames of a message,
does it's inspection thingy, and only then forward the message in one frame
(thus, coalescing all frames of the fragmented message into one).

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 16 Aug 2011 10:52:08 +0100

> to speak the base WS protocol without any extension.

Right.. but actually AIUI you are additionally allowed to mess with 
framing in the case extensions are negotiated, and the intermediary 
understands the extensions.

> (thus, coalescing all frames of the fragmented message into one).

Hm well messages can be of infinite length in fact so it's on a bit of a 
loser with that approach.  I know what you mean though with the case of 
buffer boundaries making the inspection parsers blind.

However it'll have to find a smarter (eg, bytewise state machine) scheme 
if it isn't going to be prone to simple attacks prepended with enough 
whitespace it has to give up on scanning the message at all.

There's something else about aggregating frames that's not good, if the 
thing is going to buffer n frames then it cannot begin to issue until it 
has the header of the nth frame, since it must first emit the aggregated 
length.  If the traffic is realtime latency sensitive like voice and 
issued from the endpoint at a fixed rate again like many audio 
implementations, that introduces really disproportionate latency.

For example, if you have 8kHz uLaw coming (1 byte per sample) originally 
is 1kByte packets every 125ms, if the intermediary decides to buffer and 
aggregate at 4kByte buffer size, you introduced half a second latency in 
the audio connection for no gain other than a few bytes less from not 
having to give the three frame headers that got merged.

-Andy

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 16 Aug 2011 03:30:45 -0700

> >> I think this is "a negotiated extension that the intermediary does
> >> not understand".
> >
> Right.. but actually AIUI you are additionally allowed to mess with framing in
> the case extensions are negotiated, and the intermediary understands the
> extensions.

Yes, my question 

"What exactly are the rules in the base WS spec that restrict how intermediaries may
change fragmentation? - In particular in the presence of interleaved control frames."

applies to a) and c):

a)
- no extension active
=> intermediary MAY change fragmentation according to rules of the base WS spec

b)
- extension active
- intermediary does not understand extension
=> intermediary MUST NOT change fragmentation in any way

c)
- extension active
- extension introduces no own fragmentation rules
- intermediary does understand extension
=> intermediary MAY change fragmentation according to rules of the base WS spec

d)
- extension active
- extension introduces own fragmentation rules
- intermediary does understand extension
=> intermediary MAY change fragmentation ONLY according to rules of the extension

and the extension in d) can of course just require: don't touch fragmentation in any way.
 
> There's something else about aggregating frames that's not good, if the thing
> is going to buffer n frames then it cannot begin to issue until it has the
> header of the nth frame, since it must first emit the aggregated length.  If the
> traffic is realtime latency sensitive like voice and issued from the endpoint at
> a fixed rate again like many audio implementations, that introduces really
> disproportionate latency.

Yes, bad. Intermedaries should not get into the way.

I wonder why the base WS spec allows coalescing by intermediaries at all.