Subject: [hybi] Multiplexing extension spec draft 02

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 27 Jan 2012 13:54:38 +0900

http://tools.ietf.org/html/draft-tamplin-hybi-google-mux-02
Diff http://tools.ietf.org/rfcdiff?url2=draft-tamplin-hybi-google-mux-02.txt

No semantic change.

- some typo fixes
- named "control channel" explicitly
- updated reference to point RFC
- added some vertical space for readability

---

From: Martin Sustrik <[email-masked]>
Date: Fri, 27 Jan 2012 15:57:15 +0900

> http://tools.ietf.org/html/draft-tamplin-hybi-google-mux-02

I would say it's worth explicitly stating that an endpoint MUST NOT 
announce larger quota than it can immediately read from the network 
(i.e. the amount of empty space in channel's rx buffer).

Failure to do so can result in all the channels being stuck because of 
the single offending channel.

Martin

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 30 Jan 2012 14:08:55 +0900

Hi,

On Fri, Jan 27, 2012 at 15:57, Martin Sustrik <[email-masked]> wrote:

>> http://tools.ietf.org/html/**draft-tamplin-hybi-google-mux-**02<http://tools.ietf.org/html/draft-tamplin-hybi-google-mux-02>
>>
>
> announce larger quota than it can immediately read from the network (i.e.
> the amount of empty space in channel's rx buffer).
>

I agree it's a caveat good to note, but we shouldn't limit flexibility
of scheduling granularity by putting such MUST NOT in the protocol spec, I
think. There might be some case where one wants to give more slot to some
particular logical channel greater than size of that channel's buffer (any
demand of large time slot for a particular channel, or any special QoS
algorithm).

> Failure to do so can result in all the channels being stuck because of the
> single offending channel.

I'd add this generalized implementation note if necessary.

Be careful when you assign big quota for some logical channel. There is
risk that the quota exceeds the processing capacity (buffer size,
processing power, bandwidth, etc.) for that channel, and the endpoint can
get stuck by getting excess data for that single offending channel
occupying one shared TCP connection while there's room of processing
capacity for the other channels.

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 30 Jan 2012 15:05:35 +0900

On 01/30/2012 02:08 PM, Takeshi Yoshino wrote:

> of scheduling granularity by putting such MUST NOT in the protocol spec,
> I think. There might be some case where one wants to give more slot to
> some particular logical channel greater than size of that channel's
> buffer (any demand of large time slot for a particular channel, or any
> special QoS algorithm).

The scenario that I was referring to looks like this:

1. Receiver on channel 1 has 100 bytes in rx buffer free.
2. Despite of that it announces quota of 200 bytes.
3. Sender on channel 1 sends two 100-byte frames.
4. Sender on channel 2 sends another 100-byte frame.
5. Receiver on channel 1 receives the first 100-byte frame and stores it 
in the rx buffer.
6. Receiver on channel 1 can't receive the next 100-byte frame because 
its rx buffer is full.
7. Receiver on channel 2 has an empty rx buffer but cannot receive the 
frame send on the channel 2 because there's the pending channel 1 frame 
still lurking in the TCP rx buffer in front of it.

This way overcommiting the quota on channel 1 caused channel 2 to be 
stuck (as well as channel 3 or any other channel on the same TCP 
connection).

My original proposition was meant to explicitly prevent this kind of 
situation.

Martin

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 30 Jan 2012 07:10:07 +0100

Hi,

> demand of large time slot for a particular channel, or any special QoS

In fact it is exactly similar to how TCP announces its receive window size,
helping the sender not to overflow it.

(...)
> capacity for the other channels.

This means the endpoint should be smart enough not to announce the whole
buffer for a single channel, and always announce less than what remains
divided by the number of channels, in order to be able to infinitely divide
it. That's suboptimal but offers equal space to everyone. On top of that,
congestion control can be added to scale the advertised buffer size depending
on how it's used. Multiplexing always comes with congestion control
unfortunately, so it's not surprizing we're getting back to the same issues
as lower layer protocols.

I've already seen some papers about receive-side algorithms to modulate the
advertised window but I can't find them right now.

Regards,
Willy

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 30 Jan 2012 15:42:15 +0900

On 01/30/2012 03:10 PM, Willy Tarreau wrote:

> it. That's suboptimal but offers equal space to everyone.

This way one overloaded channel can starve out other channels, although 
it'll happen in multiple iterations.

In case of 2 channels, one idle, the other one overloaded the overloaded 
channel would first advertise 1/2 of the rx buffer, then 1/2 of the 
remaining part of the buffer (1/4 of the whole buffer) then 1/2 of that etc.

The whole algorithm asymptotically approaches the entire buffer size. 
Once it is full, both channels are stuck, even the innocent one that was 
sending no data.

The only way to ensure that one channel won't starve another one is to 
allocate separate buffer for each channel.

That in turn means that you can DoS your peer by simply asking for 
inordinate number of channels and running it out of memory that way.

SCTP which is another protocol with multiplexing built in solves this 
problem by asking user to specify the max number of channels when the 
endpoint is established.

Martin

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 30 Jan 2012 16:00:16 +0900

On Mon, Jan 30, 2012 at 15:05, Martin Sustrik <[email-masked]> wrote:

> ...snip...

> My original proposition was meant to explicitly prevent this kind of

Yes, I imagined similar example from your original post, and proposed the
generalized text.