Subject: Re: [hybi] On Fragmentation of Frames

From: Bob Gezelter <[email-masked]>
Date: Sun, 10 Apr 2011 09:03:48 -0500

As everyone is well aware, I am most definitely of the opinion that
multiplexing support is necessary for the WebSocket protocol to succeed in
mass use. Reducing the order of the scaling curve requires it.

That said, the recent thread "On Fragmentation of Frames" brings up a number
of important points.

Error handling is one of, if not the most, important features of a protocol.
Simply dropping a connection is not an acceptable result in all but the most
serious situations. For this reason, maximum frame size should absolutely be
negotiated during connection establishment. If a longer frame is sent, the
connection may be terminated; however, the termination must include the
reason for the termination (e.g., frame of 1000000 received in excess of
65536 negotiated). Simple unexplained drops of connections creates a
regression to the "dark ages" of network troubleshooting. Troubleshooting
a WebSocket protocol connection should not require a wire-level trace of
the underlying TCP link.

Maximum frame size affects latency and resource allocation throughout the
data path (e.g., transmission times, buffer sizes). I see no reason why
intermediaries (e.g., traffic directors) could not re-frame traffic for
reasons not apparent to either endpoint. Of course, such an intermediary is
responsible for managing both the client and server relationships.

Seen from this perspective, IMHO, multiplexing becomes less of a problem.
Each subchannel (my term) is created with a specified quota, and the flow
within that subchannel is managed using a debit/credit approach. This is
fully pipelined, so it need not be a bottleneck provided the quota is
appropriate for the connection.

This is emphatically not the same as TCP-level flow control. TCP-level
flow control is also not a substitute. One can easily imagine a client
with 20 different WebSocket protocol connection under the present non-
multiplexed specification creating resource problems by attempting to
transmit/receive huge volumes of traffic. This approach is also a case
of "beggar my neighbor", as it will tie up large numbers of TCP buffers,
both at endpoints and intermediaries. TCP buffers, one should note, are
often a finite resource, being allocated out of kernel space. One of the
comments in this thread spoke of a 3,000,000 connection application,
consider the size of the buffer pool to deal with reassembly of IP packets
received out of sequence.

Lastly, on the subject of sendfile and large transmissions. The simple
fact is that the overhead of packetizing is there, regardless of
whether sendfile is aware of it or not. Actual calls to sendfile may
or may not be needed, but I suggest careful analysis of what will
happen when one attempts to send a 4GB file in one piece. There are
paging and a range of other issues. As economists often say, "There
is no such thing as a free lunch". Having implemented a wide range of
device and network interfaces, there are ways to improve the efficiency
of such situations. The most time-honored example is some form of
command chaining, which was very useful in the IBM System/360 IO
architecture for achieving high performance. It also appeared as a
way to reportedly read and de-serialize analog seismic data tapes,
which were characterized by a lack of inter-record gaps. Similar
approaches are used by network controllers and device drivers, to
avoid excessive call overhead.

- Bob
+--------------------------------------------------------------------------+
| Robert "Bob" Gezelter                       E-Mail:  [email-masked]  |
| Robert Gezelter Software Consultant                                      |
| 35-20 167th Street, Suite 215               Fax:       (on Request)      |
| Flushing, New York  11358-1731                                           |
| United States of America                    http://www.rlgsc.com         |
+--------------------------------------------------------------------------+

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 18:59:32 +0100

On 04/10/2011 03:03 PM, Somebody in the thread at some point said:

Hi -

> mass use. Reducing the order of the scaling curve requires it.

AFAIK everyone agrees with that.  But practically we need to do it in 
two steps since there is surely going to be a lot of discussion about 
mux and plain websockets is needed right away.  FWIW the plan is mux 
will be an extension to the current plain standard without changing it, 
and the subchannels are to use the plain standard protocols inside.  So 
it has a reasonable story that doing it in two pieces won't be a disaster.

> a WebSocket protocol connection should not require a wire-level trace of

I would suggest focus on frame size isn't really important in websockets.

As I understand it, super-big frames are in fact useless since if you 
want to send a 1MByte frame as in your example, you could as easily have 
sent it on the wire as 16 x 64KByte frames and the client can't tell the 
difference.  If control frames were coming inbetweentime, they would 
also arrive with less latency.  Considering someone is allowed to block 
the wire for 2^63 size frame, that's unreasonable from control frame 
latency POV so why is it allowed?

Also, messages are defined to be atomic for the Javascript case, you 
have to sit and buffer all frame content until one comes with FIN, then 
you can pass it all up as one action.

But for frames, there is no requirement in the standard about their 
needing to be buffered and managed as full frames.  In fact the state 
machine that handles packets can pass up chunks of frames to the upper 
layer without aggregating the packets into a full frame.  So I don't see 
any reason to introduce a frame MTU concept since frames themselves are 
not being reassembled.

It does make sense to not allow the protocol to talk about antisocial or 
deadly (given they will block the wire beyond ping-pong keepalive 
limits) frame lengths.

> appropriate for the connection.

Well, notice first that the frame issue is not bound to mux usage, it is 
an issue in non-muxed connections the same.

Otherwise, yeah that is the approach of x-google-mux from John Tamplin.

> received out of sequence.

Well, 3M connections is not a problem on my plate, if it was I would be 
solving it in a scalable way with a bunch of cheap servers.

I think mux implementation is coming really quite soon due to the 
pressures you are talking about.  But politically, websockets needs to 
lay a 1.0 egg, despite it's not technical that isn't a consideration 
that can be ignored.

> happen when one attempts to send a 4GB file in one piece. There are

Well AFAIK there are tricks like mmap the file and then the kernel side 
is taking care of everything.  It's still paddling hard under the water, 
right.

But as I said if we consider the muxing action, sendfile is pointless to 
think about since the payload is going to be chunked up in mux blocks 
with their ownoverhead on the wire that must be added too.  And sendfile 
was the only reason for allowing supersized frame lengths at all.

-Andy