Subject: [hybi] Are RSV1-3 bits really useful?

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 23 Aug 2011 13:01:46 +0200

>From the spec:

   RSV1, RSV2, RSV3:  1 bit each

      MUST be 0 unless an extension is negotiated which defines meanings
      for non-zero values.  If a nonzero value is received and none of
      the negotiated extensions defines the meaning of such a nonzero
      value, the receiving endpoint MUST _Fail the WebSocket
      Connection_.

So what about if a WS handshake requires two extensions ext1 and ext2,
and both extensions want to make usage of the same RSV1 bit?

This is:

- For ext1 bit RSV1 means: 1 (do something) / 0 (do other thing)
- For ext2 bit RSV1 means: 1 (do something) / 0 (do other thing)

Who decides the value of RSV1? the last extension requested? there
would be an obvious conflict.

Is the usage of these RSV bits really well designed? or is it more
like "let's think about it in a future"?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 23 Aug 2011 20:34:49 +0900

In June, bit allocation method was actively discussed.
Here's chairs' conclusion then.
http://www.ietf.org/mail-archive/web/hybi/current/msg07455.html

The spec is requesting registry for RSV* bits.
http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-11#section-11.9

> like "let's think about it in a future"?

Kind of.

---

From: John Tamplin <[email-masked]>
Date: Tue, 23 Aug 2011 10:23:43 -0400

> and both extensions want to make usage of the same RSV1 bit?

Clearly if two extensions require use of the same fixed reserved bit, then
they cannot both be used at the same time.  If a client were to request
both, the server would have to accept no more than one of them.

At the point we need to allocate them, we can always define a standard for
dynamically allocating them (which has been discussed here multiple times)
if desired, and have an extension say that it uses that method.  Given how
hard it is to get agreement on anything here (witness the current debate of
rolling the clock back a year and a half to remove text frames), it seems
prudent to defer standardizing what we don't know we even need until we
actually need it.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 24 Aug 2011 09:51:45 +1000

Inaki,

the ability to negotiate dynamic allocation of opcodes and reserve
bits to extensions was deemed to be beyond the capabilities of the
developers implementing websockets.
So we are instead left with either an implied free for all (if you
read 4.8) or a process of static allocation by request (11.8 /11.9).

As static allocation will be untenable for all but a few hi profile
extensions,  most ad-hoc extensions/protocols will need to put their
own opcodes/reserved bits into the payload.

cheers

On 23 August 2011 21:01, Iñaki Baz Castillo <[email-masked]> wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: John Tamplin <[email-masked]>
Date: Tue, 23 Aug 2011 20:01:45 -0400

> developers implementing websockets.

What was deemed was that there was no need to define a method for allocating
the bits/opcodes before there was an extension that needed them.  The first
extension that needs to allocate a bit/opcode can define such a mechanism if
it is justified.

Until then, delaying the base protocol to achieve consensus on something
that isn't even needed yet seems foolhardy.

> own opcodes/reserved bits into the payload.

You objected at the time, and there was wording put into the spec at your
request that says only registered extensions can use them, so I don't know
why you keep beating the same horse that hi profile companies can force use
of them but others can't.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 24 Aug 2011 10:20:31 +1000

> extension that needs to allocate a bit/opcode can define such a mechanism if

sorry for my sarcasm about the reason - not helpful.

> that isn't even needed yet seems foolhardy.

I agree we should not further delay the protocol.   But I still
believe lack of negotiated use of opcodes and reserved bits is a
significant deficiency that would have been very simple to have solved
without delay - had we addressed it at the time.     I only bring it
up now because others are asking about this obvious deficiency.

> of them but others can't.

I'm not beating that horse anymore.    In the absence of dynamic
allocation, IANA request is the only sane thing to do.    By high
profile extension, I am NOT meaning high profile company using market
share - I'm meaning an extension that has sufficient wide appeal that
it will be standardised through the IETF and be able to make a good
case to IANA for allocation of these scarce resources.      This is
fine and fair and the best we can do for now.    But it does mean that
reserve bits and opcodes are out of bounds for proprietary extensions
(eg what we might see between an intermediary and an app server).
I did mention google-mux's use of 0x7 in the other thread - but such
usage would now have to be approved by IANA - so my apologies for
alluding to that dead horse.

regards

---

From: John Tamplin <[email-masked]>
Date: Tue, 23 Aug 2011 20:59:04 -0400

> up now because others are asking about this obvious deficiency.

I still don't believe it would be added without delay -- currently we are
re-debating the framing that was decided over a year ago, and there were
significant objections to your proposal.  So, I think it would take a while
to achieve something that people can agree on.

> alluding to that dead horse.

If you own the network the connection runs over, you can allocate reserved
bits to your heart's content -- you just can't use them on the public
network with endpoints you don't control.

In reality, the endpoints that you don't control are going to be browsers,
and it seems a high enough bar to get them to implement some extension that
the extra friction of having to get a bit/opcode allocated seems to be in
the noise.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 24 Aug 2011 14:39:02 +1000

> to achieve something that people can agree on.

What proposal?  I'm not proposing we change framing and I'm not
proposing that we delay now to add negotiation for reserve bits and/or
opcodes.  I think we are good to go.      I was just explaining to
Inaki why we don't have a mechanism to negotiate reserve bit/codes and
that we must statically allocate them via IANA.   I think this is
probably sufficient for the immediate future, but in the mid/long term
is going to stifle innovation.

OK I mixed in some sour grapes with that explanation because I do
think it was a failing of this WG not to address what is essentially a
pretty simple issue.

> bits to your heart's content

Not really, as you still need to deal with future allocation by end to
end extensions.   Say for example if I allocated RSV bit 3 to talk
to/from my proprietary WS load balancer, but then IANA allocates that
bit to a compression extension, my load balancer would then not be
able to pass standard compressed frames.  Currently it will be very
difficult for intermediaries to do any in-band signalling that will be
future proof, since any indication you pick to mark out a signal may
subsequently be allocated to a standard extension.

> the extra friction of having to get a bit/opcode allocated seems to be in

True that is going to present significant inertia, and for good
reasons I don't think browsers are going to be in a rush to add a huge
number of extensions any time soon.   So I think we are good to go and
don't need to delay the process for this issue.  But it remains an
issue that I believe will need to be addressed eventually.

regards

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Wed, 24 Aug 2011 06:13:25 +0100

> able to pass standard compressed frames.  Currently it will be very

This is correct... as it is to be sure you have to add framing bytes or 
a pre-opcode or similar for your "load balancer", wrapping the existing 
framing and be unable to leverage the reserved bits.  It's not deadly 
just a tiny bit less optimal than it could be.

In return though, dynamic reserved bit management can be left out of the 
base spec, which is its own different kind of win; and your load 
balancer is proof against whatever is done with the reserved bits 
eventually, if anything.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Wed, 24 Aug 2011 01:13:16 -0400

On Wed, Aug 24, 2011 at 12:39 AM, Greg Wilkins <[email-masked]> wrote:

> What proposal?

The one you made a while back - I'm not saying you were making one now,
though you were complaining that your proposal was not included in the base
spec.

> subsequently be allocated to a standard extension.

The gateway between the part you control and the outside world is free to
refuse to accept the requested extension that conflicts with the internal
private extension.

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 24 Aug 2011 10:38:30 +0100

John Tamplin wrote:
>    On Wed, Aug 24, 2011 at 12:39 AM, Greg Wilkins <[1][email-masked]>
>    wrote:
> 
>    What proposal?
> 
>    though you were complaining that your proposal was not included in the
>    base spec.
> 
>      end extensions. Â  Say for example if I allocated RSV bit 3 to talk
>      able to pass standard compressed frames. Â Currently it will be very
>    The gateway between the part you control and the outside world is free
>    to refuse to accept the requested extension that conflicts with the
>    internal private extension.

If it blocks the outside world's end-to-end extension, something the
load balancer should not care about (because it's end-to-end), the
load balancer will be quite right regarded as broken.

It is not even a case of refusing during negotiation, but rather
dropping connections spuriously as far as other components see it.

Since *end-to-end* negotiation is not something a load balancer should
be participating in, it would not know what to reject at negotiation
time, and would have to abort the connection when it sees extra bits
that it knows it cannot forward.

Luckily, there is a solution: As private extensions require both sides
to agree before using them, they can also agree to encode conflicting,
unrecognised bits from the outside world on entry to that hop, and
decode them on exit.  I.e. tunnel what they don't understand.

(The same applies to mux encodings, which is why I have argued before
that any frame bits and semantics that are present just to make room
for future mux encodings, are unnecessary.)

Unluckily, use of standardised bits *when automatically negotiated*
for private purposes likely to be badly mangled by filtering
("censoring") intermediaries at some point, where components think
they have hop-to-hop agreement but something in between thinks it is
interfering with a standard WebSocket stream.  But that's just
something we'll have to deal with in the same ad-hoc way as with HTTP.

Almost all of this comes down to whether negotiation is robust and
reliable, and respected by every component in the path.

-- Jamie

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 24 Aug 2011 12:18:34 +0200

2011/8/24 Greg Wilkins <[email-masked]>:
> Say for example if I allocated RSV bit 3 to talk
> able to pass standard compressed frames.

Wow, WebSocket is not yet a standard and people here is already
thinking about using WS frame features for a propietary load balancing
system.
This must be the famous HTTP world...

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Philipp Serafin <[email-masked]>
Date: Wed, 24 Aug 2011 12:20:55 +0200

> This must be the famous HTTP world...
Hey, if people use HTTP as an internal database API (see couchdb),
they'll use WS for load balancing, too.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 24 Aug 2011 12:56:02 +0200

> they'll use WS for load balancing, too.

Yes, the new OSI model:

  4      - Any kind of stuf Layer
  3      - HTTP / WebSocket Layer
  2.5   - DNS just type A Layer
  2      - Data Link Layer
  1      - Physical Layer

- There is no need for a Transport Layer because applications just use
TCP port 80 or 443 (TLS).

- Network layer is neither needed as WebSocket Layer already replaces it.

- No need for DNS queries other than A (AAAA is too modern and MX is
not needed with current webmails).

- There is no Application Layer (as any application must be built
within a web page). HTML5 already provides audio/video (waiting for
web-rtc) so SIP and XMPP are dropped.

- JavaScript is the only language in the world for client
applications, and any new specification must be designed to fit well
with JavaScript. Also PHP is allowed in server side (the best and the
only server side language).

- Finally desktop operating systems (Windows, Linux/Ubuntu, MacOSX)
must be replaced with single web browsers. Personal data must be
stored in the cloud.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 24 Aug 2011 14:59:02 +0100

> thinking about using WS frame features for a propietary load balancing

I think you misunderstood.

That WS load balancer is distributing *WS* connections among a backend
of *normal WS* server applications.

It's completely reasonably to at least consider WS, or modified WS,
for the backends in that case, instead of inventing a different
protocol which just carries WS traffic anyway.

Imagine if you have 100,000,000 users connected by WS to your server.

Obviously you cannot do it without a WS-aware load balancer.

(I think 1,000,000 is roughly the maximum order of magnitude for a
single, very well tuned server at the moment.)

I expect even 10,000 connections will stress many WS applications, so
WS-aware load balancers will be popular, I expect.

> This must be the famous HTTP world...

If you mean it needs to scale, yes of course...

-- Jamie

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 24 Aug 2011 16:05:17 +0200

> Imagine if you have 100,000,000 users connected by WS to your server.

I would never do that. Instead I would try to implement DNS SRV
mechanism within the protocol so I would never require a single
server/point receiving all the clients connection, but a distributed
cluster and each client connecting to a random server (using DNS SRV
procedures).

But better not to restart such discussion again :)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 24 Aug 2011 07:23:52 -0700

> That WS load balancer is distributing *WS* connections among a backend of
> *normal WS* server applications.
> 
> It's completely reasonably to at least consider WS, or modified WS, for the
> backends in that case, instead of inventing a different protocol which just
> carries WS traffic anyway.
> 

I have zero experience with a scale like this, but why do you need the
load-balancer have any knowledge of WS?

There is stuff like OpenBSD relayd which can do stateful Layer-3 (not 7) load balancing.
An incoming TCP connection is routed to a backend node, that rule is then dynamically
inserted into the OpenBSD pf firewall (which is stateful). Subsequent traffic on layer 3
(IP) is then routed to that backend node. It can hearbeat/monitor backend cluster,
and pf can do failover/scale itself using CARP.

http://www.openbsd.org/cgi-bin/man.cgi?query=relayd&sektion=8&format=html

So, why does one need the load-balancer be WS aware? Just interested ..

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 24 Aug 2011 16:03:40 +0100

> So, why does one need the load-balancer be WS aware? Just interested .. 

The simplest is probably using mux of some kind, even if it's only
to the backend servers.

Also, connection transfers.  As WS depends on a continuous connection,
unlike HTTP, if you have servers going up and down, or their
availability varying with individual load, rerouting connections
without the client seeing a drop may be desirable in some
applications.  (All the more if some nice programming model hides it.)

It goes deeper when you route individual requests in the WS stream
based on request type, object or whatever (like HTTP relays do based
on the URL or Cookie), or aggregate pub-sub subscriptions or multicast
updates.

Since WS conversations are likely to be used, in some applications,
for continuous real-time state which may well depend on real-time
activity of any of the other users, there's potentially more internode
traffic than HTTP where the coordination tends to be isolated in the
database and memcached.  So the message topology matters more.

I will readily agree at that point you're using another protocol,
though you may well want it to be an WS application protocol or WS
extension, so the backend servers are not much different than serving
a smaller version of the same site.  That's what's attractive about
"horizontal scaling" after all - no need to rewrite everything as you
scale up.

All the best,
-- Jamie

---

From: Philipp Serafin <[email-masked]>
Date: Wed, 24 Aug 2011 17:30:53 +0200

> must be replaced with single web browsers. Personal data must be
I'd like to pass this on to randall munroe, please.

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 24 Aug 2011 08:30:37 -0700

> The simplest is probably using mux of some kind, even if it's only to the
> backend servers.

Ok, I see, when muxed complete WS connections (like with x-google-mux) or
individual message channels (like with a yet to be defined flat-mux) should
be targeted at different backend nodes, then yes, that can't be done
with stateful layer-3 or layer-7 (TCP) load-balancing.

> unlike HTTP, if you have servers going up and down, or their availability
> varying with individual load, rerouting connections without the client seeing a
> drop may be desirable in some applications.  (All the more if some nice
> programming model hides it.)

That will get quite tricky. I.e. the WS connection that was previously
authenticated on backend node 1 during WS handshake now needs to
be handed over to node 2, while telling node 2 that the WS is already
established/authenticated.

But yes, as far as I know, relayd can transparently failover itself (by pf/carp),
but not mediated, established TCP connections when the target node fails.
I might be wrong here.

> It goes deeper when you route individual requests in the WS stream based
> on request type, object or whatever (like HTTP relays do based on the URL or
> Cookie), or aggregate pub-sub subscriptions or multicast updates.
> 
> Since WS conversations are likely to be used, in some applications, for
> continuous real-time state which may well depend on real-time activity of
> any of the other users, there's potentially more internode traffic than HTTP
> where the coordination tends to be isolated in the database and
> memcached.  So the message topology matters more.
> 
> I will readily agree at that point you're using another protocol, though you
> may well want it to be an WS application protocol or WS extension, so the
> backend servers are not much different than serving a smaller version of the
> same site.  That's what's attractive about "horizontal scaling" after all - no
> need to rewrite everything as you scale up.

What you then do with above is put deep application logic into the
load-balancing layer. Whether thats good is open for debate. I for at least,
would be afraid of doing that. I'd prefer having the LB layer only know
who's up and maybe what's the load on each backend node.

Also, with all of above I'd guess we are talking about really large deployments.
I mean, probably 10 world-wide. A couple of OBSD boxes with sane NICs and
low end CPUs can saturate multiple Gb/s links. pf can maintain statefully
really large numbers of connections. That should take one quite far, without
the complexity of WS aware balancing .. just my 2cts.

Anyway, thanks for your insights!

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 24 Aug 2011 17:54:28 +0200

> I'd like to pass this on to randall munroe, please.

Oh, it could fit well:

  http://xkcd.com/934/

XD

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 24 Aug 2011 17:19:57 +0100

> established/authenticated.

With WS there's application state, but there's no reason it can't be a
generic blob as far as failover components are concerned.  You still
want the client to be able to resync (networks cut off TCP randomly),
but state transfer reduces the incidence of delays due to that, and
frees up the engineers who can bring up and take down servers more freely.

> who's up and maybe what's the load on each backend node.

I agree, avoid deep application logic in the LB if you can.  Keep the
equipment / software generic at each layer.

However certain useful structures like publish-subcribe and
pseudo-multicasting are generic and useful enough to build into LBs.

That the same structures also simplify many applications on a single
server, and then scale well, is another motivation.

We're talking here about LBs and backends, but the same applies
elsewhere, though it's realistically a lot harder to deploy elsewhere.

> Also, with all of above I'd guess we are talking about really large
> deployments.  I mean, probably 10 world-wide. A couple of OBSD boxes
> with sane NICs and low end CPUs can saturate multiple Gb/s links.

Everything depends on the amount of CPU and memory you need to work
out *what* to send, and the size of individual messages if you are
talking about saturating links.

> pf can maintain statefully really large numbers of connections. That
> should take one quite far, without the complexity of WS aware
> balancing .. just my 2cts.

Oh I agree.  It remains to be seen what will really work & there's not
much point optimising what isn't needed.

We shouldn't design much *in* to WS, just avoid over-constraining it.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 25 Aug 2011 13:48:16 +1000

On 25 August 2011 00:23, Tobias Oberstein <[email-masked]> wrote:

> but why do you need the

My experience of HTTP intermediaries is that there is a vast and
unexpected range of value add features and I see no reason to expect
that similar value add features wont be invented for WS. Even if such
features are not strictly necessary or best practise, commercial
realities often drive the need to differentiate.

I can image that an intermediary may need  to speak WS as it may take
on the responsibility of sending/responding to pings.   It may
aggregate fragments so that app servers only deal with whole messages.
 It may do authentication and authorisation, SSL offload  etc.      It
may offer broadcasting of one message to a large number of
connections.

Many of these features will need either configuration or some kind of
signalling.  For example offloaded SSL may need to communicate a
client certificate, cipher strength and or session id to the
application server.   Aggregation may need buffer sizes configured.
Intermediaries may offer different quality of service settings
depending on authentication and/or application status etc. etc.

Some of this configuration/signalling may well be able to be done out
of band, but much of it is more obvious to do inband, ie on the actual
connection that is being configured/signalled about.

cheers

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 25 Aug 2011 05:38:21 -0700

> My experience of HTTP intermediaries is that there is a vast and unexpected
> range of value add features and I see no reason to expect that similar value
> add features wont be invented for WS. Even if such features are not strictly
> necessary or best practise, commercial realities often drive the need to
> differentiate.

I see.

> on the responsibility of sending/responding to pings.   It may

[cut interesting examples]

As you mention this example, it brings back a question IMHO still open.

With regard to message fragment ordering, the spec is explicit:

"""
5.4.  Fragmentation

Message fragments MUST be delivered to the recipient in the order
      sent by the sender."
http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-12#page-32
"""

I assume that message ordering must also be preserved.

However, what about the ordering guarantees wrt to

* control frames
* relation between control frames and messages/message frames

?

I was reminded of that question I have, because in your example, when
an intermediary replies to pings himself, this might imply that the control
frames are "an independent channel" (independent from the "message
channel").

So:
* message/frame ordering is guaranteed within the message channel
* no guarantees wrt to ordering even within the control channel
* are control/message really two channels unrelated in ordering

?

Whatever the answer is, I really think it should be made explicit in the spec.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 25 Aug 2011 14:55:31 +0200

> * relation between control frames and messages/message frames?

There is no such relation. Control frames can be sent between
text/binary frames belonging to the same WS message, but there is no
relation at all.

-- 
Iñaki Baz Castillo
<[email-masked]>