Subject: [hybi] Robert Gezelter on the current spec

From: Julian Reschke <[email-masked]>
Date: Mon, 04 Apr 2011 18:21:20 +0200

FYI: 
http://www.rlgsc.com/blog/ruminations/websocket-rediscovered-travails.html

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 4 Apr 2011 20:26:25 +0200

> http://www.rlgsc.com/blog/ruminations/websocket-rediscovered-travails.html

Interesting. The guy did not understand all the goals which led to the
current draft, reason why I'm regularly repeating that points that look
strange must be at least argumented a bit, otherwise people will poorly
deal with them instead of getting the right thing.

It seems the guy did not understand the goal of the closing handshake. He
understood that it was in order to replace the TCP closing handshake while
the real goal was to let the remote side know if the close was expected or
not (TCP close without WS close indicates it comes from an intermediary,
and automatic reconnection might be desirable in some cases).

He makes some points about the text/binary distinction. I remember we had
valid reasons for this, which apparently became unclear to some people with
time since I don't clearly remember all the use cases now.

He came to the conclusion that we need multiplexing, fortunately we left
provisions for that, too bad we never reached the point we can implement it,
as I think it will appear by itself and we'll have to retro-document it later.

His pause/resume mechanism is what I normally call flow control. It might
be nice to study the idea, though it can have huge security impacts as it
has on Ethernet and TCP (easy DoSes).

The guy is apparently not used to TCP, as he believes that it's not possible
to open more than 64k connections on a machine due to the 16-bit port
numbering, as he assumes a 1:1 socket-port relation. Fortunately this is not
true, or the internet would be much more congested ! Not only does the server
run all its incoming sockets on one port, but even the clients can initiate
more than 64k sockets from less than 64k ports! FTP servers open all their
data connections from the same port 20 ;-)

Nevertheless, the fact that people start reviewing the whole draft and
publicly post their comments and feelings is a sign that we're on the last
mile and that the protocol is starting to drain interest.

Regards,
Willy

---

From: John Tamplin <[email-masked]>
Date: Mon, 4 Apr 2011 14:49:39 -0400

> as I think it will appear by itself and we'll have to retro-document it

There seemed to be plenty of people that didn't want the complication of
multiplexing in the base spec, there were disagreements over exactly how it
should be implemented, and we made sure the extensions mechanisms was
sufficient to implement multiplexing as an extension.

I am working on a multiplexing extension that I hope to have a sample
implementation for in the next week or two -- email me offlist if you are
interested in participating.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Andy Green <[email-masked]>
Date: Mon, 04 Apr 2011 20:09:41 +0100

On 04/04/2011 07:49 PM, Somebody in the thread at some point said:
> On Mon, Apr 4, 2011 at 2:26 PM, Willy Tarreau <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> multiplexing in the base spec, there were disagreements over exactly how
> it should be implemented, and we made sure the extensions mechanisms was
> implementation for in the next week or two -- email me offlist if you
> are interested in participating.

FWIW I am halfway through a libwebsockets implementation too based on 
John's draft.

-Andy

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 4 Apr 2011 21:29:00 +0200

Hi guys,

> FWIW I am halfway through a libwebsockets implementation too based on 

That's excellent news! I really don't have time to spend on this
unfortunately, still I'm interested in putting my nose on whatever you'll
publish !

Cheers,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Mon, 04 Apr 2011 21:31:37 +0100

> time since I don't clearly remember all the use cases now.
The basic reasoning behind the distinction is reliant on the  
understanding of the WebSocket Protocol as part of your complete  
breakfast, ie, WebSockets.

In particular, WebSockets consists of a JavaScript API, which  
currently passes String objects to an event handler on each new  
message. Very early on, provision was made to have binary frames  
within the protocol; what we did here was unify the two frame types,  
but leave an indicator as to which they were.

This allows - ultimately - the event handler in the JavaScript to be  
passed either a String or a ByteArray object, and avoids any UTF-8  
decoding within the JavaScript.

> has on Ethernet and TCP (easy DoSes).

No, his pause/resume involved disconnect/reconnect; it's akin to  
session resumption. Pasi Eironen, as I recall, wrote a lovely paper  
on how to do this generically with TLS. Similar options exist in  
XMPP, with XEP-0198.

> data connections from the same port 20 ;-)
You can't open more than 64k connections from one IP address to  
another IP address and port. His concern is that this may occur:

1) Outbound, on enterprise NATs. This would involve 64k connections  
from within the enterprise - I don't think this is too likely through  
a single NAT, but his argument is that you need fewer clients because  
of people running multiple scripts with websocket connections to the  
same host.

2) In server farms, where you have reverse NAT. I've not seen this  
deployed, actually, in the manner he describes.

I don't want to rubbish his arguments entirely, but I think they're  
limited to case (1), and I suspect that it would be mitigated in  
various ways.

Firstly, if this genuinely became a problem, then this would drive  
IPv6, and I'm all in favour of that. Secondly, multihoming either end  
of the session increases the number of potential connections, and if  
both ends are multihomed across two IP addresses, it's already up to  
256k.

I would note that SRV support would allow usage of multiple ports on  
a single machine for this, although frankly, once you reach 64k  
sessions you probably want to consider a second machine anyway.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 5 Apr 2011 07:26:52 +0200

Hi Dave,

On Mon, Apr 04, 2011 at 09:31:37PM +0100, Dave Cridland wrote:
(...)
> decoding within the JavaScript.

OK, thanks for the details.

> on how to do this generically with TLS. Similar options exist in  

In this case, it can indeed be nice for some uses.

(...)
> another IP address and port.

I agree with that statement, though it's not what he was saying.

> His concern is that this may occur:
> 

You already have the same issue with outbound proxies with HTTP keep-alive,
and enterprises who need more than 64k concurrent connections generally
have a number of users who are a large part of this, and will not restrict
themselves to a single IP address, at least because they'll do some load
balancing between multiple outbound proxies. And clearly, 64k conns to the
same target IP:port is not something we'll see soon from enterprises,
because it will probably be social networking and an employer who sees
64k of his employees chat all the day on social networks will surely take
some actions :-)

Last, many sites already run with multiple IP addresses for various reasons,
there is no reason this will not be done with WS.

> deployed, actually, in the manner he describes.

On the server side, the point does not exist because you're not limited
by the number of ports, you can accept up to 64k connections per source
IP address and per destination address.

> limited to case (1), and I suspect that it would be mitigated in  

Exactly.

> sessions you probably want to consider a second machine anyway.

Indeed.

Cheers,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Tue, 05 Apr 2011 09:10:13 +0100

> In this case, it can indeed be nice for some uses.
Can be, but I'd note that it's reliant on storing the state for the  
session on the server cluster in such a way that it can be fully  
recovered. That involves a lot of state, and to implement it  
perfectly, relies on a lot of shared state within a cluster. This  
tradeoff makes sense in XMPP, where there is substantial state in a  
session which takes time to build up. Equally, though, XMPP is a  
clear demonstration that it's possible to achieve this purely in the  
application layer. (And I note in passing that this statement  
characterises WebSockets as a transport layer by inference, but I've  
never been big on trying to shoehorn protocols onto explicitly named  
layers within a theoretical model).

> I agree with that statement, though it's not what he was saying.
It's how I read it; it doesn't make sense otherwise. I have to admit  
I found the text somewhat impenetrable in parts.

> 64k of his employees chat all the day on social networks will  

We do see this for IM, but I'd note the mitigation can be very  
different - they can just deploy their own XMPP service which  
drastically reduces outbound connections.

> IP address and per destination address.
Right. As I say, he posits a case where there is a SNAT at the server  
farm on inbound connections. Or, possibly, a DNAT performing IP-level  
load balancing. I have to admit I found this text confusing as well -  
the latter seems much more likely to exist, so I can only assume  
that's what he meant.

The overall synopsis of much of his text might read "different  
network protocols have different effects on the network, and some of  
these effects may be bad", which I note is similar to what Roy  
Fielding comments. I don't dispute that there will be unforseen  
complications with running the WebSocket protocol over a collection  
of (often "interesting") network equipment that expects only, but the  
ones that Gezelter lists are, I feel, not the ones to be concerned  
with.

Besides which, they're not unforseen anymore. ;-)

Dave.

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 05 Apr 2011 13:22:19 -0600

<hat type='AD'/>

On 4/5/11 2:10 AM, Dave Cridland wrote:

> (And I
> note in passing that this statement characterises WebSockets as a
> transport layer by inference, but I've never been big on trying to
> shoehorn protocols onto explicitly named layers within a theoretical
> model).

This was noted during the HYBI WG session last week. Your ADs will be
pinging the Transport ADs about getting an early review from a relevant
expert in the Transport Area...

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 6 Apr 2011 19:19:16 +0100

Dave Cridland wrote:
> >On the server side, the point does not exist because you're not
> >limited by the number of ports, you can accept up to 64k
> >connections per source IP address and per destination address.
> >

SNAT is needed when you have multiple entry points, typically for
geographic or link diversity, such as 'anycasting' or multi-homing
over more than one router, forwarding to shared server(s) behind.
SNAT is needed so the outgoing reply packets go back through the same
router, otherwise they're blocked by router-local NAT state.

The 64k port limit can easily hit in that scenario, even with diverse
web clients, but you can overcome it by SNATing from per-router pools
of internal IP addressses.  You may need rather large IP pools
(hundreds/thousands) to handle large client numbers.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 6 Apr 2011 20:46:03 +0100

> and automatic reconnection might be desirable in some cases).

As I recall, your reason isn't the real goal either!  I recall it
being to prevent the TCP reset-on-close hazard, not to distinguish
endpoint vs. intermediary closes.

TCP implementations drop all buffered data, in _both_ directions, on
receiving TCP RST, and TCP RST is sent when data is received to a
closed socket.

The WS close is to prevent that during normal error-free operation
when one side closes an idle or finished connection.  For example,
when sending a message followed by WS close, the other end gets that
message.  With TCP close, there is a chance the message (or even
multiple ones) before close will be lost or truncated before the other
end sees them.  And messages sent in the opposite direction at the
wrong moment can also be lost or truncated, unexpected to the sender,
and without it knowing how much the receiver got.

Some HTTP servers use TCP half-close for a similar reason (even though
it's not in the spec - see Apache code).  That was considered for WS
and discarded, as it's not reliable with HTTP intermediaries and
symmetric protocols.

The WS close also allows idle/finished close (normal operation) to be
distinguished from network error (should be rare & referred upwards),
so automatic retry decisions are easier - you would safely, quietly &
*immediately* retry a WS message on idle close, and be confident the
message won't be seen twice, but on network error you would be more
cautious, retrying after a delay, trying a different server, checking
for duplicates etc.  Again this is because of TCP RST, which looks
like a network error, but could arise from idle/finished close if
there was no close handshake.

The problem of detecting lost messages on idle receiver close also
happens in HTTP, and is why HTTP clients generally avoid sending POST
requests on a persistent or pipelined connection.

It remains to be seen if the WS close mechanism proves dependable,
especially if intermediaries cause the same TCP RST problem by
randomly closing connections themselves anyway.  If WS close works,
great, if not, robust WS applications will need their own reliable
messaging layer on top of WS anyway, and WS close will only help the
less robust ones be a little bit better.

> has on Ethernet and TCP (easy DoSes).

Note that per-channel flow control is essential for safe & secure
multiplexing.  If it is omitted, either the flows aren't independent
(head of line blocking), or a memory DOS is possible (demuxer has to
read & buffer eagerly without bound).  Either way makes
application-independent multiplexing unsafe, and the former can lead
to deadlocks; flow control solves both problems.  This is especially
important for "invisible" multiplexing inside the browser and/or
intermediaries.

_Not_ having flow control when you have multiplexing can have huge
security impacts including DOSes and deadlocks...

-- Jamie

---

From: John Tamplin <[email-masked]>
Date: Wed, 6 Apr 2011 15:51:02 -0400

> security impacts including DOSes and deadlocks...

Right, but it isn't clear you have to have that in the base protocol, rather
than just in the multiplexing extension.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 6 Apr 2011 22:34:23 +0200

Hi Jamie,

> receiving TCP RST, and TCP RST is sent when data is received to a

Ah yes, I know that issue too well. It forces some servers to infinitely
read data, hoping what they sent was received, just because there is no
way for the application to be notified of the ACK of a shutdown(). Indeed,
a close handshake then makes sense for that case.

(...)
> important for "invisible" multiplexing inside the browser and/or

Multiplexing can be made safe using fair and simple algorithms. Some
of them involve limiting the fragment size (I mean refusing fragments
larger than negociated). After all, that's exactly what happens in an
event-based proxy : it reads chunks up to a certain size from many
sockets and processes them in turns so that everyone gets served. It's
the implementer's responsibility not to stick on the same flow forever
(on the multiplexer size) and the demux must check fragment sizes.

> security impacts including DOSes and deadlocks...

I see, but having it is also source of the same issues if improperly
used. The attack (I forgot its name) which leaves server sockets in
FIN_WAIT1 state by sending a zero TCP window after an HTTP request
clearly is the type of DOS which is caused by flow control. A similar
one exists on Ethernet where you can send pause frames to your victim
instead of 01:80:C2 and see it totally stop talking with just 30 pps
of spoofed traffic.

I agree flow control has its uses, but I mean they are sometimes very
dangerous and a lot of care needs to be taken when designing such a
system.

Regards,
Willy

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 7 Apr 2011 00:35:14 +0100

> (on the multiplexer size) and the demux must check fragment sizes.

That doesn't work.  The failure case is what I mean by "the flows
aren't independent".

Think about a proxy, which is receiving mux'd WebSocket (for example),
demuxing and forwarding fragments to individual server processes -
perhaps on different servers at a different location.

Now if any of those server processes runs slower than the others, it
will cause _all_ the processes to slow down to that speed, _or_ the
proxy will have to buffer up fragments for that slow server, while
it forwards received fragments to the others.

If one of the servers runs very slowly, it will effectively block all
the others.

You can't avoid this, unless the client sends fragments for the slower
endpoint less often than for the other endpoints.  That's flow
control!  Positive window tokens are the simplest way to do this;
there are others.

In the event-based processing you are thinking of, the processes
aren't independent.  That's ok: You want fair processing, and you
accept that processes affect the rate of each other's progress, and
you don't allow any of them to stop.  You control them, and You
probably write them with that in mind.

But it doesn't work if you need to demux to independent receiving
processes that you don't control, without rogue or just slow behaviour
of one receiver blocking or unduly slowing the others.  This
characteristic is needed by "invisible" multiplexing of WebSocket
streams in browser clients and in proxies (at the client/ISP).
Otherwise, it is not safe to do.

It's not important when you control all the processes yourself, only
when you are imposing muxing on processes that you don't control, and
should not cause undue interference between them.  Independence is key.

I only realised this issue exists when, by chance, I noticed flow
control mentioned succinctly in the BEEP-over-TCP spec.  Then I
realised it's critical to "invisible" muxing, that per-channel
flow control solves it, and none of this is obvious.

It has to be done with some care - new channel creation has to be
flow-controlled too[*], and initial windows have be pre-negotiated if
you want cheap channels without setup RTTs.

[*] (TCP provides flow control for data, and for new connections.  It
doesn't look like it for new connections, but that's because TCP SYN
backoff provides it in a different way.)

This sort of blocking has been given (by others) as an objection to
multiplexing WS, compared with TCP where application flows are
somewhat independent.  (I.e. they compete, but don't block each
other).  Flow control sounds unnecessarily complex (especially if you
don't know why), and it doesn't seem very elegant, but it solves the
independence problem.  I'll be delighted if something better turns up.

> clearly is the type of DOS which is caused by flow control.

I agree.  This one shows the importance of controlling where data is
buffered, so it doesn't block resources by buffering in the wrong
place.  It is not an easy problem to solve in general, because there
are legitimate slow data consumers too.

Btw, can you DOS a WebSocket server in the same way? :-)

> A similar one exists on Ethernet where you can send pause frames to
> your victim instead of 01:80:C2 and see it totally stop talking with
> just 30 pps of spoofed traffic.
> 

Quite right.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 7 Apr 2011 01:01:57 +0100

John Tamplin wrote:
> 
>    On Wed, Apr 6, 2011 at 3:46 PM, Jamie Lokier <[1][email-masked]>
>    wrote:
> 
>      multiplexing. Â If it is omitted, either the flows aren't
>      independent
>      read & buffer eagerly without bound). Â Either way makes
>      to deadlocks; flow control solves both problems. Â This is
>      especially
>    Right, but it isn't clear you have to have that in the base protocol,
>    rather than just in the multiplexing extension.

I agree.  WebSocket already has flow control.  It's called TCP.

Though it can be argued that it _already_ muxes two things: Normal
messages, and control frames, such as the Close frame.

An endpoint can't send the Close frame if the socket is full of data
being read by a slow process.  It can only queue it for sending later.
This may or may not impact the reliability of graceful close; I
haven't given it much thought.  It would undermine
Close-With-Reason-String, if there were such a thing in the base
protocol - or even if that were an extension added later.  You need
data channel flow control to decouple the WS protocol handler's
behaviour from the data consumer's behaviour.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 7 Apr 2011 07:40:19 +0200

Hi Jamie,

> it forwards received fragments to the others.

OK got it! We were both talking about the opposite side. I thought
you were suggesting that a client could make a multiplexer fill its
channel using too large data, but in fact it's the opposite, it's
the fact that the muxed channel stalls due to one stalled flow. It's
sometimes required to implement window management on top of this but
this becomes complex and sometimes inefficient (eg: tcp over tcp).

Thanks for the explanation Jamie!
Willy