Subject: [hybi] Send quota value before receiving AddChannelResponse (was: Re: MUC: channel ID semantics)

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 18 Apr 2012 17:15:06 +0900

On Fri, Apr 13, 2012 at 20:09, Arman Djusupov <[email-masked]> wrote:

> BTW:
>
> “Different from non-multiplexed WebSocket connection, data frames of multiplexed channel except for the channel 1 opened by default MAY be sent before receiving AddChannelResponse as far as there's sufficient****
>
> send quota.  In case the AddChannelRequest fails, those frames are ignored by the other peer.”****
>
> ** **
>
> As far as I had previously understood, until a new logical channel
> receives the AddChannelResponse its flow control quota is 0. The initial
> quota is assumed to be 64K but this quota applies only once the channel is
> accepted and the handshake did not include/specify a quota that overrides
> this value. Wouldn’t this mean that the logical channel cannot send
> anything prior it was accepted due to the 0 quota even if the specification
> permits it to send messages, or did I miss something?
>

You're right. In the latest spec, it's not explicitly specified what the
value of send quota is until AddChannelResponse is received.

There're several options we can take.

a) specify the fixed value of send quota before AddChannelResponse in the
spec
b) add a new parameter "pre_handshake_quota" to multiplexing extension to
negotiate the value of send quota before AddChannelResponse
c) drop the feature to configure initial quota for each logical channel,
and redefine the quota parameter as pre-handshake send quota for all
logical channels

I'm not sure how many people really want to configure initial quota for
each logical channel independently.

Comment please.

---

From: John Tamplin <[email-masked]>
Date: Wed, 18 Apr 2012 08:01:24 -0400

> and redefine the quota parameter as pre-handshake send quota for all

Until we know the AddChannel was accepted, how can we send data anyway?
 This seems just like the initial handshake on a new connection -- you
wouldn't consider sending that data until you knew the connection was
accepted.  Since we are avoiding TCP setup requirements, this will be much
less latency than if the connection were created without multiplexing.

I suppose it would be possible to send the data while keeping it buffered
as an optimization, but since adding a channel should be infrequent
compared to sending data on it, I don't see it as that important to
optimize for.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 18 Apr 2012 16:18:36 +0300

I also feel it is unnecessary to let the client send some payload that would need to be discarded if the channel is not accepted. 

But if this feature is needed for some use cases then I would opt for using the quota negotiated during the initial handshake as the initial quota for all logical channels (and wouldn’t negotiate an initial quota for logical channels since they already have it granted).

With best regards,

Arman

From: John Tamplin [mailto:[email-masked]] 
Sent: Wednesday, April 18, 2012 3:01 PM
To: Takeshi Yoshino
Cc: Arman Djusupov; [email-masked]
Subject: Re: Send quota value before receiving AddChannelResponse (was: Re: [hybi] MUC: channel ID semantics)

On Wed, Apr 18, 2012 at 4:15 AM, Takeshi Yoshino <[email-masked]> wrote:

You're right. In the latest spec, it's not explicitly specified what the value of send quota is until AddChannelResponse is received.

a) specify the fixed value of send quota before AddChannelResponse in the spec

b) add a new parameter "pre_handshake_quota" to multiplexing extension to negotiate the value of send quota before AddChannelResponse

c) drop the feature to configure initial quota for each logical channel, and redefine the quota parameter as pre-handshake send quota for all logical channels

Until we know the AddChannel was accepted, how can we send data anyway?  This seems just like the initial handshake on a new connection -- you wouldn't consider sending that data until you knew the connection was accepted.  Since we are avoiding TCP setup requirements, this will be much less latency than if the connection were created without multiplexing.

I suppose it would be possible to send the data while keeping it buffered as an optimization, but since adding a channel should be infrequent compared to sending data on it, I don't see it as that important to optimize for.

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 19 Apr 2012 01:49:53 +0900

> compared to sending data on it, I don't see it as that important to

Yes. It's infrequent.

If the information the impatient client want to send is small, it can also
be just embeded into URL or subprotocol to avoid the additional RTT.

If no one really needs this feature and has justification, I'm fine with
keeping the same restriction (no transmission until AddChannelResponse) as
no-mux connection.

IIRC, Gabriel was interested in this optimization. Gabriel, do you have any
comment about this?

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 19 Apr 2012 05:41:00 +0000

In all previous discussions in this WG there has been relentless push to optimize any RTTs out of the protocol. Consistent with this is being able to use the lightweight channel setup (as is possible in SPDY) in which data transmission is possible without requiring the ack. If I'm the only one asking about this, perhaps the WG is no longer interested in such optimizations, but I'd still be interested in understanding the change of heart.

From: Takeshi Yoshino [mailto:[email-masked]]
Sent: Wednesday, 18 April, 2012 9:50
To: John Tamplin; Gabriel Montenegro
Cc: Arman Djusupov; [email-masked]
Subject: Re: Send quota value before receiving AddChannelResponse (was: Re: [hybi] MUC: channel ID semantics)

On Wed, Apr 18, 2012 at 21:01, John Tamplin <[email-masked]<mailto:[email-masked]>> wrote:
On Wed, Apr 18, 2012 at 4:15 AM, Takeshi Yoshino <[email-masked]<mailto:[email-masked]>> wrote:
You're right. In the latest spec, it's not explicitly specified what the value of send quota is until AddChannelResponse is received.

Yes. It's infrequent.

If the information the impatient client want to send is small, it can also be just embeded into URL or subprotocol to avoid the additional RTT.

If no one really needs this feature and has justification, I'm fine with keeping the same restriction (no transmission until AddChannelResponse) as no-mux connection.

IIRC, Gabriel was interested in this optimization. Gabriel, do you have any comment about this?

---

From: "Arman Djusupov" <[email-masked]>
Date: Thu, 19 Apr 2012 13:13:14 +0300

We are simply considering whether letting the client to send frames before
the logical channel is accepted is worth it in order to remove this short
RTT. This feature would also require that the sever buffers the frames of
the logical channel until it is accepted. On the server side the procedure
of adding logical channels is not certain to be fast. It may involve
querying the database or establishing a new physical connection. For the
duration of this operation the server would have to spend some amount of its
resources despite it would be unclear whether the new logical channel is
accepted or not.

The best solution would be to have the server control this by using a
default value for the initial quota during the physical channel handshake.
For example, the server could set the quota explicitly to "0" during the
physical channel handshake, in which case all logical channels (including
first one) would be prohibited from sending any data until the channel is
accepted and so all clients would have to wait for the first flow control
message to arrive from the server side. Note that this flow control message
may arrive in the same frame as the AddChannelResponse without any
additional RTT.

Practically we don't need to change much in the spec, we just need to be
able to negotiate the default quotas during the physical channel handshake.
Any additional quotas can be easily granted using the flow control messages
that may arrive in the same frame as AddChannelRequest or
AddChannelResponse.

With best regards,

Arman

From: Gabriel Montenegro [mailto:[email-masked]] 
Sent: Thursday, April 19, 2012 8:41 AM
To: Takeshi Yoshino; John Tamplin
Cc: Arman Djusupov; [email-masked]
Subject: RE: Send quota value before receiving AddChannelResponse (was: Re:
[hybi] MUC: channel ID semantics)

In all previous discussions in this WG there has been relentless push to
optimize any RTTs out of the protocol. Consistent with this is being able to
use the lightweight channel setup (as is possible in SPDY) in which data
transmission is possible without requiring the ack. If I'm the only one
asking about this, perhaps the WG is no longer interested in such
optimizations, but I'd still be interested in understanding the change of
heart.

From: Takeshi Yoshino [mailto:[email-masked]] 
Sent: Wednesday, 18 April, 2012 9:50
To: John Tamplin; Gabriel Montenegro
Cc: Arman Djusupov; [email-masked]
Subject: Re: Send quota value before receiving AddChannelResponse (was: Re:
[hybi] MUC: channel ID semantics)

On Wed, Apr 18, 2012 at 4:15 AM, Takeshi Yoshino <[email-masked]>
wrote:

c) drop the feature to configure initial quota for each logical channel, and
redefine the quota parameter as pre-handshake send quota for all logical
channels

I suppose it would be possible to send the data while keeping it buffered as
an optimization, but since adding a channel should be infrequent compared to
sending data on it, I don't see it as that important to optimize for.

Yes. It's infrequent.

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 19 Apr 2012 15:13:50 +0000

I'm not sure "short RTT" is a valid expression any more. At any rate, I'm fine with adding safeguards, but ultimately, it would be a shame to preclude data from flowing along with the AddChannel request.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Arman Djusupov
Sent: Thursday, 19 April, 2012 3:13
To: Gabriel Montenegro; 'Takeshi Yoshino'; 'John Tamplin'
Cc: [email-masked]
Subject: Re: [hybi] Send quota value before receiving AddChannelResponse (was: Re: MUC: channel ID semantics)

We are simply considering whether letting the client to send frames before the logical channel is accepted is worth it in order to remove this short RTT. This feature would also require that the sever buffers the frames of the logical channel until it is accepted. On the server side the procedure of adding logical channels is not certain to be fast. It may involve querying the database or establishing a new physical connection. For the duration of this operation the server would have to spend some amount of its resources despite it would be unclear whether the new logical channel is accepted or not.

The best solution would be to have the server control this by using a default value for the initial quota during the physical channel handshake. For example, the server could set the quota explicitly to "0" during the physical channel handshake, in which case all logical channels (including first one) would be prohibited from sending any data until the channel is accepted and so all clients would have to wait for the first flow control message to arrive from the server side. Note that this flow control message may arrive in the same frame as the AddChannelResponse without any additional RTT.

Practically we don't need to change much in the spec, we just need to be able to negotiate the default quotas during the physical channel handshake. Any additional quotas can be easily granted using the flow control messages that may arrive in the same frame as AddChannelRequest or AddChannelResponse.

With best regards,
Arman

From: Gabriel Montenegro [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, April 19, 2012 8:41 AM
To: Takeshi Yoshino; John Tamplin
Cc: Arman Djusupov; [email-masked]<mailto:[email-masked]>
Subject: RE: Send quota value before receiving AddChannelResponse (was: Re: [hybi] MUC: channel ID semantics)

From: Takeshi Yoshino [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Wednesday, 18 April, 2012 9:50
To: John Tamplin; Gabriel Montenegro
Cc: Arman Djusupov; [email-masked]<mailto:[email-masked]>
Subject: Re: Send quota value before receiving AddChannelResponse (was: Re: [hybi] MUC: channel ID semantics)

Yes. It's infrequent.

---

From: Patrick McManus <[email-masked]>
Date: Sun, 22 Apr 2012 19:32:13 -0400

On 4/19/2012 11:13 AM, Gabriel Montenegro wrote:
>
> I'm not sure "short RTT" is a valid expression any more. At any rate, 
> I'm fine with adding safeguards, but ultimately, it would be a shame 
> to preclude data from flowing along with the AddChannel request.
>

I agree with Gabriel. TCP has been trying to get data-with-the-syn on 
and off for decades (fast-open is the current effort), and sending the 
request (subject to some reasonable but non-trivial flow control) with 
the channel open is a large reason for spdy's performance boost.. 
websockets mux would be pretty broken if it didn't learn that lesson.

-P

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 24 Apr 2012 16:04:31 +1000

In the case of MUX, the server has already accepted the main
connection from the client and agreed to the the MUX extensions.
Thus I would think that rejecting addChannel would be an
extraordinarily rare occurrence.    Perhaps it might be provoked by a
different channel extension, but if we don't expose the extension
choice to the application then I would expect the same extension and
settings for all channels anyway (hence the worth of the delta header
design).

So I think that a quick start for sending data can be another benefit
of using MUX.

cheers

---

From: "Arman Djusupov" <[email-masked]>
Date: Tue, 24 Apr 2012 11:16:25 +0300

The server might refuse the channel due to reaching the maximum number of
mux logical channels on the same connection. The issue is not only with the
cases when the channel is refused but also when the server can't allocate a
new channel fast, which would force the server to buffer frames of pending
logical channels until they get accepted in order to read from the wire the
subsequent frames belonging to already established channels. For this reason
the server needs a way to control a default quota. My suggestion was to let
the server define the default quota for the logical channel during the
physical channel handshake rather than hardcoding it into the spec.

As I already mentioned in a previous message, according to the current
specification (a) the client may send data prior to receiving an
AddChannelResponse, but (b) it doesn't have any flow control quota until an
AddChannelResponse with or without quota parameter is received. These two
obviously contradict each other. So if we want the client to be able to send
some data right after the AddChannelResponse is sent then we need to define
how the default quota is negotiated. In such a case we actually don't need
any quota parameter in the AddChannelResponse handshake. In other words the
part of the spec that describes the initial flow control quota negotiation
has to change anyway.

PS: I totally agree on the evilness of 'RTT', but there has to be a clean
way to control the amount of data that the client can send before the
channel is accepted, preferably not just a hardcoded default quota for all
clients in all cases.

With best regards,
Arman

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg
Wilkins
Sent: Tuesday, April 24, 2012 9:05 AM
To: Patrick McManus
Cc: [email-masked]
Subject: Re: [hybi] Send quota value before receiving AddChannelResponse
(was: Re: MUC: channel ID semantics)

cheers
_______________________________________________
hybi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/hybi

---

From: John Tamplin <[email-masked]>
Date: Tue, 24 Apr 2012 09:18:52 -0400

> physical channel handshake rather than hardcoding it into the spec.

The problem is that you are using up buffer space in the client to buffer
any frames which might need to be re-sent if the logical connection is not
successful.  So, having the server specify that seems backwards to me.
 Since frames can be nearly arbitrarily large, that means that the client
might have to buffer an arbitrarily large amount of data (even if it
refragments it and sends only part of it, it has to hold onto the rest).

So, the only way I think it makes sense to send data with the connection
are when we know it can't fail or when we pass off responsibility for
buffering the data to the client.  Ie, in a hypothetical API where you can
send data with the open, you explicitly say that data will be dropped if
the connection is not successful, so the code calling WebSocket.open is
responsible for re-sending the data on a later connection attempt.

> obviously contradict each other.

No, there is no conflict - a client is always free to send up to its quota,
and the quota is held to zero until you get a connection.  That means there
are fewer special-cases to consider, both in validating the spec and
implementing it

> channel is accepted, preferably not just a hardcoded default quota for all

Ok, but who decides, and who bears the buffering cost of that decision?

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 25 Apr 2012 16:33:02 +0300

The problem is that you are using up buffer space in the client to buffer any frames which might need to be re-sent if the logical connection is not successful.  So, having the server specify that seems backwards to me.  Since frames can be nearly arbitrarily large, that means that the client might have to buffer an arbitrarily large amount of data (even if it refragments it and sends only part of it, it has to hold onto the rest).

If the API permits a message to be sent while the socket is not yet connected, the implementation would have to buffer any amount of data being pushed into the Socket unless this amount is less than the default quota. Currently the default quota is 0, so it will have to buffer everything.  As you have pointed out, even if a quota is available, all data sent before the logical channel is established will have to be buffered in order to be resent if the connection fails. Note that if the API permits this than it would also have to buffer in cases when mux is not used since the API has no knowledge whether mux is present or not.

So far I can imagine these options to determine the initial quota:

1)      Change spec so that, by default, 64k are granted as initial quota to all logical channels before receiving the AddChannelResponse. Spec decides, both bear cost, w/o RTT.

2)      Leave the default initial quota as is at zero. But then why permit the client to send data before the AddChannelResponse when we know that it doesn’t have any available quota. Nobody decides, nobody bears cost, w/ RTT.

3)      Let the server set the default quota for all logical channels during the first handshake. Server decides, cost depends on decision, w/o RTT.

4)      Let the client set a “self-granted” quota for every channel, so that the server either accepts it or refuses the logical channel. Client decides, cost depends on decision, w/o RTT.

5)      Let the client request a “desired” quota for all logical channels, and the server grant an equal or lower quota during the first handshake. Server decides, cost depends on decision, w/o RTT.

With best regards,

Arman