Subject: [hybi] Max frame size

From: Francis Brosnan Blázquez <[email-masked]>
Date: Wed, 22 Jun 2011 07:34:20 +0200

Hi,

Current frame design allows to send a very big single frame size (63
bits). In terms of memory a websocket server will have problems with
this because he must buffer all content until he can deliver the entire
message to the app level (assuming message based API).

I think we need a way for servers (and possibly clients) to notify each
other which is the max frame size each part is willing to accept so each
site administrator can configure this according to its resources,
application type, number of connections, etc.

I think a simple header notified at the websocket handshake will provide
this valuable control to each party:

   Max-Frame-Size: 4096

This will cause each party to fragment application level messages if it
is found bigger than Max-Frame-Size.

In the same direction, if not provided Max-Frame-Size, it looks
reasonable to assume a default value for this header. Again, 4096 bytes
looks like a good value.

What do you think?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 22 Jun 2011 08:05:14 +0200

Hi,

> reasonable to assume a default value for this header. Again, 4096 bytes

There is nothing in the protocol which makes whole frame buffering mandatory
and in fact it's designed so that it can work as a stream instead. While the
app server may decide to buffer some frames before delivering them, it should
never do so for large ones, because 1) it will not be able to do so past one
point, and 2) it will only add important latency. It's exactly the same as
with large POST requests : for large posts, it's common to call the app and
pass it the fd and say "read the data from that fd and for XXX bytes". That's
what makes file upload possible, otherwise you would not be able to use web
interfaces to send mails.

Regards,
Willy

---

From: Norio Kobota <[email-masked]>
Date: Wed, 22 Jun 2011 17:55:03 +0900

Hi,

I think that 63 bits-length is too big for single frame, too.
But there is no need to provide any header in handshake.

The websocket frame format has a FIN bit for fragment message.
So we can use this bit when we want to send very long message.

I think that the websocket frame format can be more simple.

+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len |    Extended payload length    |
|I|S|S|S|  (4)  |A|     (7)     |             (16)              |
|N|V|V|V|       |S|             |   (if payload len==126)       |
| |1|2|3|       |K|             |                               |
+-+-+-+-+-------+-+-------------+-------------------------------+
| Masking-key, if MASK set to 1 | Masking-key (continued)       |
+-------------------------------+-------------------------------+
|          Payload Data...                                      |
+---------------------------------------------------------------+
|          Payload Data (continued)                             |
+---------------------------------------------------------------+
or
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| reserve     |    Payload length             |
|I|S|S|S|  (4)  |A| (7)         |    (16)                       |
|N|V|V|V|       |S|             |                               |
| |1|2|3|       |K|             |                               |
+-+-+-+-+-------+-+-------------+-------------------------------+
| Masking-key, if MASK set to 1 | Masking-key (continued)       |
+-------------------------------+-------------------------------+
|          Payload Data                                         |
+---------------------------------------------------------------+
|          Payload Data (continued)                             |
+---------------------------------------------------------------+

Regards, nori

(2011/06/22 14:34), Francis Brosnan Blázquez wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>
>

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 22 Jun 2011 02:03:47 -0700

We spent almost a year before we got to the point on agreeing on framing.
We're now in last call. It's not the time to re-open such lengthy
conversations based on individual preferences (e.g. you're not saying
anything the group hasn't already considered).

As for negotiation, that's a very complex topic, which, while interesting,
is not necessary for the base protocol to function. I would suggest that if
you're interested you propose text for an extension that would handle
negotiation of such parameters as optimal/max frame sizes etc.

-Ian

On Wed, Jun 22, 2011 at 1:55 AM, Norio Kobota <[email-masked]>wrote:

> +-+-+-+-+-------+-+-----------**--+---------------------------**----+
> +-+-+-+-+-------+-+-----------**--+---------------------------**----+
> +-----------------------------**--+---------------------------**----+
> |          Payload Data...                                      |
> +-----------------------------**------------------------------**----+
> +-----------------------------**------------------------------**----+
> or
> +-+-+-+-+-------+-+-----------**--+---------------------------**----+
> +-+-+-+-+-------+-+-----------**--+---------------------------**----+
> +-----------------------------**--+---------------------------**----+
> |          Payload Data                                         |
> +-----------------------------**------------------------------**----+
> +-----------------------------**------------------------------**----+
>
> Regards, nori
>
>
>> ______________________________**_________________
>> hybi mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/**listinfo/hybi<https://www.ietf.org/mailman/listinfo/hybi>
>>
>>
>>
> ______________________________**_________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/hybi<https://www.ietf.org/mailman/listinfo/hybi>
>

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Wed, 22 Jun 2011 11:52:33 +0200

Hi Norio,

I'm talking about a very different issue: if the server that is going to
receive a big message that can't buffer, and assuming the absence of a
max frame size notification, he only can pass incomplete parts of an
unfinished frame to the app level...that is, if the websocket header
tells you are going to receive 200MB, but you can only buffer 64k, the
server will be forced to pass to the app level this frame fragments...

Is this the pattern expected by the list?

-- 
Francis Brosnan Blázquez <[email-masked]>
ASPL
91 134 14 22 - 91 134 14 45 - 91 116 07 57

AVISO LEGAL

Este mensaje se dirige exclusivamente a su destinatario. Los datos
incluidos en el presente correo son confidenciales y sometidos a secreto
profesional, se prohíbe divulgarlos, en virtud de las leyes vigentes. Si
usted no lo es y lo ha recibido por error o tiene conocimiento del mismo
por cualquier motivo, le rogamos que nos lo comunique por este medio y
proceda a destruirlo o borrarlo.

En virtud de lo dispuesto en la Ley Orgánica 15/1999, de 13 de
diciembre, de Protección de Datos de Carácter Personal, le informamos de
que sus datos de carácter personal, recogidos de fuentes accesibles al
público o datos que usted nos ha facilitado previamente, proceden de
bases de datos propiedad de Advanced Software Production Line, S.L.
(ASPL). No obstante, usted puede ejercitar sus derechos de acceso,
rectificación, cancelación y oposición dispuestos en la mencionada Ley
Orgánica, notificándolo por escrito a:
ASPL - Protección Datos, C/Antonio Suárez 10 A-102, 28802, Alcalá de
Henares (Madrid).

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Wed, 22 Jun 2011 12:11:43 +0200

Hi Ian,

> We spent almost a year before we got to the point on agreeing on
> framing. We're now in last call. It's not the time to re-open such
> lengthy conversations based on individual preferences 

I'm not trying to reconsider framing; that's ok. I'm talking about
providing a way to each party to advice is buffering capabilities which
has nothing to do with the framing format. 

> (e.g. you're not saying anything the group hasn't already considered).

Possibly but the problem is yet not addressed. I think this will cause
lot of wrong patterns and a security problem at the server side like
passing to the app level not completed frame (unfragemented or not) but
just octects because the server was unable to buffer.

That is, assuming a 2MB message, with a 64KB buffer limit, the app level
may receive a first indication that a frame was received, only 64kb, and
subsequent indications of pending content without header?

If this is the pattern accepted, this will make easy do stream injection
for those servers that won't buffer 20MB content until pass a complete
frame to the app level..

> As for negotiation, that's a very complex topic, which, while
> interesting, is not necessary for the base protocol to function. I
> would suggest that if you're interested you propose text for an
> extension that would handle negotiation of such parameters as
> optimal/max frame sizes etc.

Ok, I'll wait for your comments since I believe this is not an
individual preference but a risk at the server side. The idea is to
ensure server side delivers complete frames to the app level, not
octects without header..

> -Ian
> 
> On Wed, Jun 22, 2011 at 1:55 AM, Norio Kobota
> <[email-masked]> wrote:
>         Hi,
>         
>         | |1|2|3|       |K|             |
>         |
>         |          Payload Data...
>            |
>         |N|V|V|V|       |S|             |
>         |
>         | |1|2|3|       |K|             |
>         |
>         |          Payload Data
>         |

AVISO LEGAL

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 22 Jun 2011 03:16:05 -0700

If a frame is sent that a server is unable to buffer for whatever reason, I
would expect the server to close the connection with error code 1004

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Wed, 22 Jun 2011 12:33:31 +0200

Hi Willy,

> There is nothing in the protocol which makes whole frame buffering mandatory

I know, but I'm not talking about buffering, I'm talking about what to
do when you receive more content than you can buffer. 

With the current design, the server have the following options: close
the connection, or keep buffering or pass octects directly to the app
level without having a frame header (with previous frame closed)...all 3
options are really bad idea.

Assuming this escenario, it clear the sending part can send the content
already splitted so that the server can deliver complete frames to the
app level (not just bytes in the middle) and close the connection when
the advised buffer is exceeded...

Regards,

AVISO LEGAL

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Wed, 22 Jun 2011 12:55:51 +0200

> If a frame is sent that a server is unable to buffer for whatever
> reason, I would expect the server to close the connection with error
> code 1004 

Sorry Ian, but I don't see how this will solve the problem. 

Given the 1004 solution, the websocket server developer will have to
"assume" a default max frame size (for the reasons pointed).

He can even provide a config so every site administrator can tweak this,
however, any value selected will be easily exceeded soon or
later...causing a 1004?

At the same time current websocket javascript API do not provide a way
to send fragmented messages (which is right, this is message oriented).
However, this leads the browser to decide how to fragment content sent
to the server. 

In this context I think the browser needs the right value to avoid the
1004 error by splitting the message into frames accepted by the server.

AVISO LEGAL

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 22 Jun 2011 15:23:39 +0300

> If a frame is sent that a server is unable to buffer for whatever reason, I would expect the server to close the connection with error code 1004.

If a sever cannot buffer an entire frame for any reason then it can still preform local frame fragmentation just as an intermediary would do.

With best regards,
Arman

On Wed, Jun 22, 2011 at 3:11 AM, Francis Brosnan Blazquez <[email-masked]> wrote:
Hi Ian,

AVISO LEGAL

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 22 Jun 2011 14:25:21 +0200

> options are really bad idea.

But why would it strip the frame header ? It should simply pass the frame
header and stream the data.

The server should proceed exactly as it would proceed if receiving a large
POST. All file transfer applications which support large POSTs (webmails
etc...) don't buffer everything, they pass it as a stream. Do you imagine
if the server had to buffer a DVD image or something like this before passing
it to the application ?

I don't see the difference with frames. You can call the application saying
"hey, here come 5 GB of data, please read them from this connection handle".

Regards,
Willy

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Wed, 22 Jun 2011 17:35:13 +0200

Hi Willy,

> header and stream the data.

I'm trying to figure how the use case for this would be. 

So, if the frame received is less than 127 or 65535 bytes the server
reads everything, checks header correctness and that the header
represents bits advised, etc, and then pass the data through a handler
to the user level code...but if the data is bigger, you pass a file
descriptor??

> if the server had to buffer a DVD image or something like this before passing

I see your example but this is not the same. In the POST case you have a
single app handling its data (HTTP). 

In the websocket case you have a webserver + an application level code
that implements an application protocol that is unknown to the webserver
so it is not a nice pattern to just pass a file descriptor to the app
level...

> "hey, here come 5 GB of data, please read them from this connection handle".

Ok, the way it should be done, assuming we have a message/frame oriented
protocol, is that the app level register a handler which is called every
time a single frame is received. 

This way, the handler is called, for example, each 4k, so the handler
can receive those 5GB with minimum memory allocation and with the
security it handles messages not raw access to a file descriptor (with
the security and code complexity implications it has).

AVISO LEGAL

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Wed, 22 Jun 2011 16:48:43 +0100

On 06/22/2011 04:35 PM, Somebody in the thread at some point said:

Hi -

> to the user level code...but if the data is bigger, you pass a file

There's more than one way to write the code to deal with this, but the 
way that is immune to running out of buffer space or funny boundary 
conditions / packet fragmentation fragility is to pass everything 
through a bytewise state machine, as Willy says treat it strictly as a 
stream of bytes.

Then you really don't care about what size packets it comes in, how big 
a frame can be, you interpret the frame header as each byte of it comes 
in and just buffer payload locally using any size buffer or none at all, 
and spill those up to the guy who takes payload for that kind of frame 
as it fills up or you see it was the last byte.

> the security and code complexity implications it has).

I guess you could write it like that but it has other problems, like 
blocking and timeout management that are handled neatly by having the 
state machine as the one place for everything.

-Andy

---

From: Scott Ferguson <[email-masked]>
Date: Wed, 22 Jun 2011 08:58:25 -0700

> time a single frame is received.

No. It's like a HTTP chunked POST, where the application receives a 
stream of data, not the underlying HTTP chunks.

A server (or client) which exposes the frames as its primary API is 
doing it wrong.

> the security and code complexity implications it has).

See stdio.h, specifically the FILE methods like fdopen, fread, fgets, 
etc. That's the level of abstraction the application needs to see.

-- Scott

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 22 Jun 2011 18:05:40 +0200

2011/6/22 Scott Ferguson <[email-masked]>:
> No. It's like a HTTP chunked POST, where the application receives a stream
> of data, not the underlying HTTP chunks.
>
> A server (or client) which exposes the frames as its primary API is doing it
> wrong.

Completely right. There is something in OSI and Internet model called
"layers" (even if here we are already in application layer).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Francis Brosnan Blázquez <[email-masked]>
Date: Thu, 23 Jun 2011 13:01:01 +0200

Hi Scott,

> A server (or client) which exposes the frames as its primary API is 

I think you are getting my wrong maybe because I mentioned "...which is
called every single frame is received", but obviously my point is
focused on the concept of having a handler called when a frame "with
useful app level data (message)", not control frames directed to the
engine.

>From websocket draft we can read:

   The WebSocket protocol uses this framing so that specifications that
   use the WebSocket protocol can expose such connections using an
   event-based mechanism instead of requiring users of those
   specifications to implement buffering and piecing together of
   messages manually.

This definition matches the handler concept I'm exposing and...current
websocket javascript API (on message event).

> etc. That's the level of abstraction the application needs to see.

..and you really think this API is suitable for a message oriented
protocol? (I assuming your fread concept will return bytes as it comes
not complete messages)

While I see this API makes the buffering (max frame size) problem for
the websocket developer to be not an issue, is in fact not solving but
moving the problem to the app level where, again, the application will
have to buffer the content until a message is completed.

I still think we need a way to each party to notify its buffering or
message size limit capabilities, *mainly*, because we have a framing
designed to notify messages, defined as *units* by the draft, not
bytes...

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Thu, 23 Jun 2011 12:12:36 +0100

> designed to notify messages, defined as *units* by the draft, not

The way I think of it is that frames are directly analogous to TCP/IP 
packets, one level up.

Everyone accepts application code has to act defensively and not make 
assumptions about tcp / ip packet lengths and if they got merged or 
fragmented along the way.  How much turns up at any given time when 
poll() or select() tells you something came must be treated like a 
random number you find out when you read from the socket and it says how 
much was ready.  You can't be expecting 32 bytes every time because you 
think that the sender always sends 32 byte packets.

It's the same with frames, reliance on assumptions about when they turn 
up, in how many pieces of what size needs to be defended against in the 
code so it's not vulnerable to problems.  If you take that approach it 
naturally leads you away from this idea you need to buffer whole frames.

-Andy

---

From: Brodie Thiesfield <[email-masked]>
Date: Thu, 23 Jun 2011 20:26:18 +0900

> called every single frame is received", but obviously my point is
[snip]
> protocol? (I assuming your fread concept will return bytes as it comes

Scott understood you and was only talking about the data, however the
file stream API metaphor might be distracting.

Websocket is a message based protocol, but the message contents are
byte streams that might be split into any number of frames. So your
websocket layer notifies your upper layer only of the messages and
bytes. The frames are a websocket layer concept that the next layer
has no need to know of.

The websocket layer will tell the upper layer about the start of a
message, then feed it bytes until the message is complete, then tell
it that the message is complete. The bytes may come from a single
frame, or multiple frames. The upper layer has no need to know.

Think of the processing as something like the following (note that
control frames, compression, etc is ignored):

while have connection:
    receive frame header
    if first frame in message:
        notify application layer of the start of a message
    receive bytes up to max of frame length:
        pass bytes up to the application layer as message data
    if last frame in message:
        notify application layer that the message is complete

The frames are never notified to the application layer. It doesn't need to know.

Brodie

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 23 Jun 2011 09:37:21 -0700

> bytes. The frames are a websocket layer concept that the next layer

+1

Yes, that's exactly what I'd meant.

If you use the stream model (assuming it's not too distracting), the 
FILE/fread ends when the message ends. Each new message produces a new 
FILE as the new stream which ends when the new message is fully read.

The application sees messages as distinct byte/char-streams but doesn't 
see the frames.

For example, the API could have a callback to the application for each 
message:

class WebSocketHandler {
   void onBinaryMessage(InputStream is);
   void onTextMessage(Reader in);
}

The InputStream/Reader produces bytes/chars until the message is done 
and then returns EOF.

-- Scott