Subject: [hybi] Value of "superframes"

From: Andy Green <[email-masked]>
Date: Tue, 25 Jan 2011 09:34:25 +0000

Hi -

What's the thinking behind the FIN bit and frames with a continuation 
opcode?

It seems to create a concept of "superframes", each consisting of an 
unbounded number of frames which identify their own frame length in advance.

What is the value of that system compared to just having atomic frames 
with their length identified in advance?

-Andy

---

From: Patrick McManus <[email-masked]>
Date: Tue, 25 Jan 2011 10:10:57 -0500

> What's the thinking behind the FIN bit and frames with a continuation 

allowing the sender to arbitrarily break a message into chunks enables
low latency and low buffering streaming. Clients bound by the JS API
aren't helped, but servers are as well as other potential classes of
clients. It also preserves the flexibility to build a multiplexing
extension later on which will be good for everyone.

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Jan 2011 15:17:50 +0000

> extension later on which will be good for everyone.

OK so instead of being some "superframe" concept it's actually a 
"microframe" for splitting up the sized frames.  Fine.

It seems it's enough to just always send FIN set and a TEXT or BINARY 
opcode to ignore it then, which is what I am doing already.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Tue, 25 Jan 2011 10:31:24 -0500

> "microframe" for splitting up the sized frames.  Fine.

In addition, it is necessary when the sender doesn't know the size of the
frame up front, such as when it is dynamically generated or compressed
frames are being sent.  Without fragmentation, you would have to buffer the
entire message in memory before starting to send any of it.  With
fragmentation, you choose an appropriate buffer size and send it when it is
full.

The large maximum frame size was a compromise for those that wanted to be
able to send large messages in a single fragment (perhaps using sendfile),
and likely would never be used on a fragmented message.

> opcode to ignore it then, which is what I am doing already.

A sender is never required to fragment, but a receiver has to be prepared to
receive fragmented messages even if the message was sent in a single frame.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Jan 2011 15:42:10 +0000

On 01/25/11 15:31, Somebody in the thread at some point said:
> On Tue, Jan 25, 2011 at 10:17 AM, Andy Green <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> In addition, it is necessary when the sender doesn't know the size of
> the frame up front, such as when it is dynamically generated or
> compressed frames are being sent.  Without fragmentation, you would have
> to buffer the entire message in memory before starting to send any of
> it.  With fragmentation, you choose an appropriate buffer size and send
> it when it is full.
>
> The large maximum frame size was a compromise for those that wanted to
> be able to send large messages in a single fragment (perhaps using
> sendfile), and likely would never be used on a fragmented message.

What would the problem be just sending lots of small arbitrary-sized 
frames whenever you wanted to spill your buffer, without making any 
statement about overall length?  (I mean now with munging in the browser 
-> server direction, I can see what the problem would be in terms of 
more SHA-1s, but otherwise).

>     It seems it's enough to just always send FIN set and a TEXT or
>     BINARY opcode to ignore it then, which is what I am doing already.
>
>
> A sender is never required to fragment, but a receiver has to be
> prepared to receive fragmented messages even if the message was sent in
> a single frame.

OK, makes sense thanks.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Tue, 25 Jan 2011 10:53:05 -0500

On Tue, Jan 25, 2011 at 10:42 AM, Andy Green <[email-masked]> wrote:

> What would the problem be just sending lots of small arbitrary-sized frames
> whenever you wanted to spill your buffer, without making any statement about
> overall length?  (I mean now with munging in the browser -> server
> direction, I can see what the problem would be in terms of more SHA-1s, but
> otherwise).
>

That is exactly the current model -- there is no notion of overall message
length (though there was talk of adding it back as an extension to ease
receiver buffer management in the case where the size is known up front by
the sender), only the lengths of individual frames.  So, if I am dynamically
generating a message, it looks something like this:

opcode = BINARY;
while ((n = readStream(buf, BUF_SIZE)) == BUF_SIZE) {
   writeFrame(buf, n, opcode);
   opcode = CONTINUATION;
}
writeFrame(buf, n, opcode | FIN);

where readStream could be dynamically generating content, reading from a
network connection, or simply compressing the real content.

---

From: Andy Green <[email-masked]>
Date: Tue, 25 Jan 2011 17:49:23 +0000

> network connection, or simply compressing the real content.

Thanks.  I think I can work with all that just by adding a NO_FIN flag 
that the caller can set when flushing a buffer; the caller can already 
control the opcode.  If he doesn't give the flag (because he don't 
understand all this CONTINUATION stuff) everything goes out with FIN | 
BINARY or FIN | TEXT and that is already OK; if he does give the flag 
it's his job to give it and set the CONTINUATION opcode on the right 
packets.

-Andy