Subject: [hybi] On Fragmentation of Frames

From: Micheil Smith <[email-masked]>
Date: Sun, 10 Apr 2011 02:23:56 +1000

I'm currently working further still on my websocket protocol implementation for node.js, and 
I'm trying to figure out how to support the fragmentation of frames and whether I should have 
a automatic fragmentation of frames greater than a certain length.

Can anyone answer the following:
- Should messages be automatically fragmented?
- What are recommended fragmentation sizes?
- Should the complete headers be repeated each fragment?
- Is the payload length the length of just that fragments payload, or is it the total payload length?

Regards,
Micheil Smith
--
BrandedCode.com

---

From: Andy Green <[email-masked]>
Date: Sat, 09 Apr 2011 17:33:53 +0100

> - Is the payload length the length of just that fragments payload, or is it the total payload length?

Not sure if this is what you are asking overall, but it was the 
understanding I was initially missing:

  - length on frames is bogus in terms of overall message length, 
totally meaningless

  - Nothing actually describes overall message length unless it's the 
degenerate case the message is one frame which has FIN set; the frame 
length is the message length then.

  - when you're in a frame of the length that was told, nothing else can 
come on the wire until you sent enough packets to deliver the amount of 
content promised

  - therefore to allow PING and PONG control frames to happen with 
reasonable latency, you should restrict the max frame length you ever issue

  - message completion is only shown by a frame with FIN bit set.  What 
went before in terms of frames without FIN bit set is meaningless except 
for the total amount of payload it represented.  Only the end of the 
frame who had FIN bit set triggers the whole message, which can be many 
frames each of which may be many packets, being completed

  - In browser case, Javascript only gets told about one event when the 
FIN frame is completed and gets the whole aggregated payload.

Actually 64 bit frame length is kind of worthless AFAIK.  I don't care 
either way but it'd be fine restricting it to 16 or 32 bits given the 
need to break up messages into short frames for control frame latency.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Sat, 9 Apr 2011 12:44:38 -0400

> - Should messages be automatically fragmented?

If it is convenient for you, you are free to fragment.  The two primary
reasons for supporting fragmentation are (from my recollection anyway):
 - to allow sending dynamically generated content without requiring the
sender to buffer the entire message first
 - to allow a multiplexor to fragment messages from logical channels to
allow fair use of the shared channel

> - What are recommended fragmentation sizes?

None have been recommended.  For the dynamic content use case above, it
would likely be a reasonable buffer size, such as 64k.  For the mux case, it
is likely to be dynamic based on the utilization of the shared channel or
the characteristics of the logical channel flows, and in any case would be
defined in the mux extension.

> - Should the complete headers be repeated each fragment?

What complete headers are you referring to?

> the total payload length?

A frame's payload length is the length of that frame, including any
extension data.  There is no length for the unfragmented message.  During
the framing discussions, it was suggested that could be provided in an
extension if it proved useful (to allow the receiver to allocate the entire
buffer for the message).

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: John Tamplin <[email-masked]>
Date: Sat, 9 Apr 2011 12:47:17 -0400

On Sat, Apr 9, 2011 at 12:33 PM, Andy Green <[email-masked]> wrote:

> either way but it'd be fine restricting it to 16 or 32 bits given the need
> to break up messages into short frames for control frame latency.
>

During the framing discussions, there were people that wanted to reduce load
on the server by allowing an entire file in a single frame using sendfile
and equivalent.  Given that available bandwidth and the sizes of media
files, it was argued that 4G was insufficient, and the cost to support it
seems small.

---

From: Andy Green <[email-masked]>
Date: Sat, 09 Apr 2011 18:32:01 +0100

On 04/09/2011 05:47 PM, Somebody in the thread at some point said:
> On Sat, Apr 9, 2011 at 12:33 PM, Andy Green <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Actually 64 bit frame length is kind of worthless AFAIK.  I don't
>     care either way but it'd be fine restricting it to 16 or 32 bits
>     given the need to break up messages into short frames for control
>     frame latency.
>
>
> During the framing discussions, there were people that wanted to reduce
> load on the server by allowing an entire file in a single frame using
> sendfile and equivalent.  Given that available bandwidth and the sizes
> of media files, it was argued that 4G was insufficient, and the cost to
> support it seems small.

Sure it's not any big problem.

It does make people reading the spec assume it must be message size 
though because of the bother taken to allow the headroom, and I would 
have gotten the bit of my life back I wasted with people arguing about 
The Horror of up to 8 bytes controllable on the wire if it was just 
defined to be 2 bytes max.

I don't believe anyone will implement systems using 63 bit frame lengths 
and it's almost zero cost to break a large message into multiple frames 
at some length ceiling.  Does anyone actually plan to use 8-byte frame 
(nothing to do with message) lengths?

-Andy

---

From: Brian McKelvey <[email-masked]>
Date: Sat, 9 Apr 2011 12:15:29 -0700

Nope.  In my as3 and JavaScript (Node.JS) implementations I actually throw an error if the most significant four bytes of the 64-bit length field are anything other than zero.  At least for right now, there's no chance in hell that I'll need to handle a frame that large.

Brian

Sent from my iPhone

On Apr 9, 2011, at 10:32 AM, Andy Green <[email-masked]> wrote:

> I don't believe anyone will implement systems using 63 bit frame lengths and it's almost zero cost to break a large message into multiple frames at some length ceiling.  Does anyone actually plan to use 8-byte frame (nothing to do with message) lengths?

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 10 Apr 2011 07:49:48 +1000

Currently the jetty server has a configurable buffer size.  If a frame
arrives that is larger than the buffer size, we close the connection
with a message too large status code.

This suggests to me that we should be able to declare the maximal
frame size we are prepared to handle, so that a client does not need
to guess, reduce, guess, reduce - or just pick a low value.

Of course the server could handle large frames by streaming to disk -
but then we often run in situation that have no disk, or that are disk
constrained.  Plus 64b frames would pretty soon be able to DOS attack
most disks.

cheers

On 10 April 2011 02:47, John Tamplin <[email-masked]> wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 10 Apr 2011 08:23:37 +0200

> to guess, reduce, guess, reduce - or just pick a low value.

Indeed. Announcing that makes a lot of sense and is comparable to what
TCP does with MSS. Both the client and the server should announce what
they support, because WS is bidirectional. We could use control frames
for that, and specify that any implementation should support frames of
at least XXX bytes until advertised otherwise (something like 1kB seems
fine, as it helps remaining below one typical MSS).

That would allow both sides to start sending and to advertise their
receive capabilities in the same round trip as the data. For instance :

Client                                   Server
        GET + Upgrade ----------------->
        <----------------- 101 + Upgrade + MFS + data
        MFS + data -------------------->

MFS = Max Frame Size. Control frame to be defined. We should also
      mention that all sizes below the default one are forbidden.
      Alternatively we can decide that what is announced is the
      difference between the real size and the default size. The
      frame size should cover the mask too, it should really be
      the buffer size.

> Of course the server could handle large frames by streaming to disk -

Buffering to disk is always bad in networked environments anyway. It
slows down everyone and prevents scaling. So we should do anything we
can to ensure this can be avoided.

Willy

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 08:17:42 +0100

>> with a message too large status code.

libwebsockets takes the approach the user code can decide the policy and 
does no aggregation buffering.  It passes up every packet in a frame as 
it comes and the user code can see if it is the last (FIN) one.  That 
way the user code can choose to have it as it comes or do its own 
aggregation buffering, but either way the library is being as 
lightweight as possible.

> TCP does with MSS. Both the client and the server should announce what

A better way to come at it is just define a sane maximum frame size in 
the protocol, say 64Kbytes, and require that the worst case is 
supported, rather than add more control messages and code and stuff to 
go wrong.  I guess 63-bit length started off as being message size, got 
refined into the FIN scheme and until now did not get reassessed for 
still being useful for frame length job it finds itself changed to.

There's another limit which is maximum message size that the client is 
willing to cope with.  It's probably more useful to tell that in headers 
if telling anything.

-Andy

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 10 Apr 2011 09:58:23 +0200

Hi Andy,

On Sun, Apr 10, 2011 at 08:17:42AM +0100, Andy Green wrote:
> supported,

We'll never be able to settle on a one-size-fits-all, because many people
have different requirements. 64k is already too large for large MUXes which
will have to deal with 1 million concurrent connections, that's 64GB of
buffers in each direction ! And this week I've been in touch with someone
trying to size a system for 3 million connections.

In contrast, a 64k limit prevents any server from efficiently making use of
sendfile().

> rather than add more control messages and code and stuff to 
> go wrong.

I'm not speaking about something complicated. It basically means "For your
convenience, I have allocated a receive buffer of XXX bytes, feel free to
send frames that large if you want, or feel free to ignore this message and
keep the default value instead".

> I guess 63-bit length started off as being message size, got 

IIRC, the 63-bit came from the requirement by some to support more than
32-bit in one sendfile().

> willing to cope with.  It's probably more useful to tell that in headers 

The problem with headers is that the more info we put there, the harder it
will be to support WS on a dedicated port. There was a very good proposal
here on the list by someone I forgot the name, who suggested to support a
preamble header in each direction in order to save one RTT. This preamble
could very well contain the control frames which indicate the max frame
and message sizes each side can cope with.

Willy

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 09:56:22 +0100

On 04/10/2011 08:58 AM, Somebody in the thread at some point said:

Hi -

> trying to size a system for 3 million connections.

Is anything stopping a mux rewriting frames with the length of what it 
has received at any time?  The frame length thing seems completely 
bogus, directly analogous to people attending to packet boundaries in a 
streamed protocol and the frame length is just as amorphous as 
coalescing packets or fragmenting them further.

Otherwise I do see the point there is an implicit MTU for frames.

Anyway in the x-google-mux case there are other flow control 
arrangements above all this that render simple max frame buffer case not 
relevant even for 'back of the envelope' calculations as you are doing 
above.

> In contrast, a 64k limit prevents any server from efficiently making use of

I do not understand that at all, maybe I am missing the point.  If 
someone has a big array or file in Javascript and wants to send it as 
one message, why does he care about frame fragmentation detail?  It's 
directly analogous to packetization one level below.  If you mean by 
"efficiently" that adding a few bytes every 64KByte is inefficient then 
I think you find the math disagrees with that.

> keep the default value instead".

Intermediaries will need to rewrite this stuff according to the floor() 
of their buffer arrangements.

> 32-bit in one sendfile().

As I say I don't understand the use of sendfile() to demonstrate 
anything about framing.  Message size I can see must consider it but 
actually there is no limit about message size in the protocol.  It 
sounds like you're saying TCP/IP must allow 63-bit length packets 
because otherwise it's 'not efficient' to send large files.

> and message sizes each side can cope with.

Maybe I am just stupid today (or every day) but that logic makes no 
sense either.  Today the protocol mandates http header based startup and 
I didn't understand why putting the same stuff in binary frames is 
better.  That idea certainly isn't matured to the point that we use it 
to argue against other things just because it conflicts.

-Andy

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 10 Apr 2011 11:23:45 +0200

On Sun, Apr 10, 2011 at 09:56:22AM +0100, Andy Green wrote:
> has received at any time?

Probably not, you might be right.

> The frame length thing seems completely bogus, directly analogous to people
> attending to packet boundaries in a streamed protocol and the frame length
> is just as amorphous as coalescing packets or fragmenting them further.

I disagree on this point. In order to support multiplexing, we need frame
length. And even on a single stream it makes sense when you send compressed
data and don't know the length in advance. It's akin to a chunk size in HTTP.

> I think you find the math disagrees with that.

No I don't mean that. I mean that having an application wake up every
64kB to call sendfile() to send the next chunk is expensive. That's
basically 20k wakeups+sendfile() calls per second at 10 Gbps while
the application could get rid of this when it sends, say, a video from
disk.

(...)
> better.

Right now we have two things :
  - an HTTP Upgrade handshake
  - a bidirectional WS stream

There will be some situations where the HTTP Upgrade will not be needed
nor desirable. WS over TCP on a dedicated port will not rely on Upgrade.
Similarly when we can make use of TLS NPN, we'll be able to talk WS over
TLS. The more WS-specific info we put in HTTP headers, the more we'll
have to find a different way to transport when switching to a different
transport protocol. As long as we can have the WS stream independant of
the HTTP hanshake, it'll be easier to provide alternative transport
schemes and get rid of the masking.

Therefore, I think we should be very careful not to put WS-specific
information into the HTTP handshake, and instead make provisions in
the WS protocol to carry these information.

Regards,
Willy

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 10:56:22 +0100

> data and don't know the length in advance. It's akin to a chunk size in HTTP.

I'm not suggesting getting rid of frame length, just understanding it's 
about as meaningful as TCP/IP packet length.  We have to track it but 
it's otherwise meaningless and we have to actively write code to not 
care about it.

Anyway to support muxing you do not need any frame length actually, it 
could be done totally blind to the semantic content of what it is muxing 
with arbitrary mux-decided packetization of the subchannel content.

In x-google-mux it does require to parse the protocol it is carrying to 
determine length of the content, it doesn't manage it at the mux layer. 
  That trades architectural simplicity for saving a few bytes per mux block.

> the application could get rid of this when it sends, say, a video from

Hum.  I think that ship has sailed, we must fragment into smaller frames 
to allow control frames to be seen and responded to with reasonable latency.

In the mux case you're talking about above, the mux layer is going to be 
all over everything like a rash having to prepend mux magic to packets 
and so on, you can forget about some optimized single channel blasting 
10Gps of video with 99% idle CPU.  Whatever you do, sendfile to a local 
socket read by the mux layer, the mux layer must be doing the waking up 
per subframe and dealing with mux overhead anyway so you didn't get away 
from it.

> the WS protocol to carry these information.

It's a different argument about a different protocol really.  Obviously 
we are putting "WS-specific information in the HTTP handshake" and that 
is how it is specified, the time to be "very careful" to not do that has 
gone.

-Andy

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2011 06:34:06 -0400

> I'm not suggesting getting rid of frame length, just understanding it's
about as meaningful as TCP/IP packet length.  We have to track it but it's
otherwise meaningless and we have to actively write code to not care about
it.
> In the mux case you're talking about above, the mux layer is going to be
all over everything like a rash having to prepend mux magic to packets and
so on, you can forget about some optimized single channel blasting 10Gps of
video with 99% idle CPU.  Whatever you do, sendfile to a local socket read
by the mux layer, the mux layer must be doing the waking up per subframe and
dealing with mux overhead anyway so you didn't get away from it.
> It's a different argument about a different protocol really.  Obviously we
are putting "WS-specific information in the HTTP handshake" and that is how
it is specified, the time to be "very careful" to not do that has gone.
> -Andy

I'm a little confused on this thread. There is a 1kb minimum frame size?
That makes no sense for a chat client or other basic use of WS in the
browser.

As for video, even if WS was the appropriate transport, the video is likely
to be a TS stream which can packetized as required, or will be converted to
TS before being streamed, in which case sendfile is unusuable anyway.

I think the WG should consider the prime use case for WS to be browser based
real time communication and should not damage the utility of that usage to
accommodate other uses. When used in the backend or from dedicated clients
the use should resemble a browser client (with or without HTTP initially).
Muxing should be seen as a way of efficiently serving browser clients, not
as an end unto itself.

Thanks,
Timothy Meade
Real time web developer

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 10 Apr 2011 13:51:16 +0200

> I'm a little confused on this thread. There is a 1kb minimum frame size?

Fortunately not! I was suggesting a minimum size that each end should
be able to receive. Right now, if I send you a 4GB frame, are you able
to receive it and to process it ? Maybe, maybe not. It may depend on
the application. If we start seeing WS agents which arbitrarily decide
that they can't support frames larger than XXX, then we'd better provide
them a means for announcing that instead of letting the transfer fail.
My suggestion was that once we've settled on a way to announce this
frame size limit, we should fix the minimum limit that anyone should
support (eg: 1 kB).

> That makes no sense for a chat client or other basic use of WS in the

Video conferencing certainly is one valid use. Video streaming/playing
may also be one if it enables smoother controls from the client (stop/
start/skip/rewind...).

> TS before being streamed, in which case sendfile is unusuable anyway.

Unless you store the TS stream. Anyway, even if this might not be the best
example, there were concerns several months ago about the protocol's
efficiency when dealing with large files and sendfile certainly was one
this to care about.

> accommodate other uses.

I 100% agree with you and I really don't think we're damaging it, quite the
opposite. I think you got that feeling because you thought we were suggesting
a 1kB-min frame.

> When used in the backend or from dedicated clients
> as an end untoo itself.

Muxing has two uses :
  - better serve clients (eg: support both chat and video conferencing in a
    single channel)

  - make servers scale better. Despite how much you find it a pity, many
    servers nowadays can't process more than one connection per thread. If
    you want your server to process 1 million concurrent connections, you
    have to install a mux in front of it so that you run on a few hundreds
    or thoudands threads and process all the streams there. It will also
    save them a lot of network traffic by aggregating multiple streams in
    few TCP segments.

Regards,
Willy

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 18:38:34 +0100

> be able to receive. Right now, if I send you a 4GB frame, are you able

Yeah, as Willy says you are able to send any size frame you like whose 
length can fit in 63 bits.  But nobody can actually receive that max 
frame atomically, buffering it all up and passing it on at the end. 
There's nothing about a 1KByte minimum.

> frame size limit, we should fix the minimum limit that anyone should

The alternative is to understand that in websockets clients, only 
messages matter, the detail of how they're fragmented into frames in 
unavailable to the client.  It's like this

Javascript:  whole message in one hit when it has all come

    Websocket:  message may be fragmented into as many frames as you 
like each of whatever size

      Network:   websocket frames may be fragmented into as many packets 
as you like each of whatever size

Nobody is suggesting websocket layer must particularly care about how 
many packets came of what size at Network layer.  Similarly, I don't 
think we should care about how many frames came of what size at 
Websocket layer, not treat them espectially atomically more than we have to.

> opposite. I think you got that feeling because you thought we were suggesting

Yeah this is a red herring.  Willy and I are just talking about whether 
it's better to have an explicit frame MTU negotiated like TCP/IP MTU or 
if it's enough to reduce the max frame size from 2^63 to something saner 
like 2^16 and demand everyone can support that 64KByte max.  Actually we 
do agree there can be a problem as it stands and someone decides to send 
2GByte frame to a client who likes to buffer frames but cuts off at 
1MByte, say; basically those guys can't communicate but are compliant.

>      save them a lot of network traffic by aggregating multiple streams in

Right also I can imagine one script on one page might have several 
widgets or other dynamic content that each want a websocket.  Muxing 
will make a nice solution where all that is one socket if it's the same 
server and URL serving those websockets.  The notification stuff in 
Javascript will make that script architecture clean and simple.

-Andy

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2011 14:37:17 -0400

> Yeah, as Willy says you are able to send any size frame you like whose
length can fit in 63 bits.  But nobody can actually receive that max frame
atomically, buffering it all up and passing it on at the end. There's
nothing about a 1KByte minimum.
> The alternative is to understand that in websockets clients, only messages
matter, the detail of how they're fragmented into frames in unavailable to
the client.  It's like this
>   Websocket:  message may be fragmented into as many frames as you like
each of whatever size
>
>     Network:   websocket frames may be fragmented into as many packets as
you like each of whatever size
>
> Nobody is suggesting websocket layer must particularly care about how many
packets came of what size at Network layer.  Similarly, I don't think we
should care about how many frames came of what size at Websocket layer, not
treat them espectially atomically more than we have to.
> Yeah this is a red herring.  Willy and I are just talking about whether
it's better to have an explicit frame MTU negotiated like TCP/IP MTU or if
it's enough to reduce the max frame size from 2^63 to something saner like
2^16 and demand everyone can support that 64KByte max.  Actually we do agree
there can be a problem as it stands and someone decides to send 2GByte frame
to a client who likes to buffer frames but cuts off at 1MByte, say;
basically those guys can't communicate but are compliant.

Me again, why is the frame size in bytes instead of some other unit (like
octets?)

>>     save them a lot of network traffic by aggregating multiple streams in

Another case were assumptions can be a problem, evented servers like nginx
and node.js exist. It does makes sense to accommodate behind the scenes
muxing of course, not saying it doesn't.

>
> Right also I can imagine one script on one page might have several widgets
or other dynamic content that each want a websocket.  Muxing will make a
nice solution where all that is one socket if it's the same server and URL
serving those websockets.  The notification stuff in Javascript will make
that script architecture clean and simple.

Why same url? Are you saying that every script in the page must cooporate on
a muxing scheme or require more than one TCP socket? This defeats the
scaling and makes little sense in a world with multiple toolkits and widgets
in the same page. Having MUXing as an extension is fine, but WS JS API
should require such an extension be supported and allow multiple scripts in
the same page to subscribe to channels. To not do so would be a regression
over pubsub protocols like bayeux.

MUXing should make a distinction between browser-server MUXing and utility
MUXing performed transparently on the backend. All the discussion seems to
be on the latter.

Timothy Meade
Real time web developer

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 Apr 2011 14:40:21 -0400

Sorry, not octets but 8 or 16 byte groups

---

From: John Tamplin <[email-masked]>
Date: Sun, 10 Apr 2011 14:45:39 -0400

On Sun, Apr 10, 2011 at 2:37 PM, [email-masked] <[email-masked]>wrote:

> Why same url? Are you saying that every script in the page must cooporate
> on a muxing scheme or require more than one TCP socket? This defeats the
The idea would be that the JS API is totally unchanged for MUX.  If there
are multiple connections to the same server URL, the browser would
transparently MUX them if the server accepted the MUX extension on the first
connection.  The server has to be the same in order to safely combine the
logical channels into one physical channel.

Requiring an extension to be supported is equivalent to bundling it into the
base specification, and when we first discussed multiplexing it was left out
of the base spec because we could not get everyone to agree.  Also, some
simple servers don't care about scalability and those shouldn't have to deal
with the complexity.  If all the user agents support MUX, then the servers
that care about scalability will implement it and everything is fine, so I
am not sure I see the value in making MUX support required.

---

From: Andy Green <[email-masked]>
Date: Sun, 10 Apr 2011 19:51:24 +0100

> Sorry, not octets but 8 or 16 byte groups

No idea where "8 or 16 byte groups" is coming from.  The protocol counts 
bytes, it's byte oriented, so I talk about bytes.

I guess I am too young or lucky to ever work on anything other than an 
8-bit byte, so I don't use 'octet' much either.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 11 Apr 2011 08:04:38 +1000

On 10 April 2011 17:17, Andy Green <[email-masked]> wrote:
> does no aggregation buffering.  It passes up every packet in a frame as it
> comes and the user code can see if it is the last (FIN) one.  That way the
> user code can choose to have it as it comes or do its own aggregation
> buffering, but either way the library is being as lightweight as possible.

Andy,

I agree that this is a good way to handle maximal message size.
Jetty also gives the use the option to receive messages frame by frame.

I'm more concerned by maximal frame size, as these can be 64bit and I
don't really want to get into passing an application data that is only
part of a frame, as it is much more likely that it will be on an
unusable boundary (eg 2 bytes of a 3 bytes UTF-8 character).

> A better way to come at it is just define a sane maximum frame size in the
> protocol, say 64Kbytes, and require that the worst case is supported, rather
> than add more control messages and code and stuff to go wrong.  I guess
> 63-bit length started off as being message size, got refined into the FIN
> scheme and until now did not get reassessed for still being useful for frame
> length job it finds itself changed to.

There was certainly a few that argued for small fixed max frame size,
but unlimited message size.   However I think that a few were able to
make their argument for effectively infinite frames as well, so we
have both effectively infinite frames and infinite messages.

So I don't wont to reopen those debates - just say that it would be
useful to have some default minimal frame sizes that must be supported
and a way to declare frame sizes larger than that.

> willing to cope with.  It's probably more useful to tell that in headers if
> telling anything.

I find that less useful, as often for large messages the size will not
be known, either because it is not yet generated or perhaps because
the utf-8 encoding size has not been calculated.

Maximal frame sizes is something that a WS implementation must deal
with directly and effects buffers, APIs etc.   Message size is not
something that an impl ever need calculate, let alone limit.

cheers

---

From: Scott Ferguson <[email-masked]>
Date: Mon, 11 Apr 2011 09:02:50 -0700

On 04/10/2011 03:04 PM, Greg Wilkins wrote:
> On 10 April 2011 17:17, Andy Green<[email-masked]>  wrote:

You shouldn't be passing frames to the application. The application 
should only see messages.

The receiver can always support any frame length, even with a fixed 
receiving buffer. The frame length is only required for the sender.

-- Scott

---

From: Andy Green <[email-masked]>
Date: Tue, 12 Apr 2011 09:29:56 +0100

On 04/10/2011 11:04 PM, Somebody in the thread at some point said:

Hi -

> unusable boundary (eg 2 bytes of a 3 bytes UTF-8 character).

This is a bit wimpy to say "much more likely".  Either you can't deal 
with hanging coding it and it blows up sooner or later, or you can deal 
with it.  If you can deal with it, you don't care any more about frame 
reassembly.

If you choose to implement in a way that breaks with legal, large 
frames, that's a detail about your implementation.  You can adapt your 
implementation to not care rather than introduce frame MTU concept into 
the spec.

> something that an impl ever need calculate, let alone limit.

That is wrong.  If you write an implementation that aggregates frames 
you introduce a fragility to legal frame length and your implementation 
has a problem.  If you implement it in a better way, it is not sensitive 
to frame size.

Separately, the maximum legal frame length is meaningless in the real 
world.  It can stay like it is without direct problem but it just gives 
ammo to the masking-fudsters.  It'd be better to limit it to 16-bit or 
32-bit length and make masking subordinate to framing.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 13 Apr 2011 09:57:15 +1000

On 12 April 2011 18:29, Andy Green <[email-masked]> wrote:

> This is a bit wimpy to say "much more likely".  Either you can't deal with
> hanging coding it and it blows up sooner or later, or you can deal with it.

true.

Hmmm I guess if we allow intermediaries to fragment arbitrarily - even
without knowledge of the extensions being applied, then any limits on
handling atomic chunks needs to be based on message size and not frame
size.

Eitherway, I think there are going to be implementations that have at
least message size limits (if not frame size as well), and unless we
have a way to declare that, then we will have interoperability
problems.

cheers

---

From: Andy Green <[email-masked]>
Date: Wed, 13 Apr 2011 08:45:14 +0100

On 04/13/2011 12:57 AM, Somebody in the thread at some point said:

Hi -

> have a way to declare that, then we will have interoperability

Yeah.  It only applies to cases where the peer wants to hold atomic 
messages.

But I'm not sure if telling a magic limit reflects reality at the client 
if there is a shared pool of memory like malloc, many websocket 
connections are opened saying 10MB is the limit, potentially 
overcommitting the pool and then suddenly all of them get a max size 
message exhausting the pool or something else that uses the pool commits 
most of it and big messages come, etc.  So it can still fail unless you 
really preallocated 10MB per conn just in case which is unrealistic.

Or, it might be a file transfer protocol, the message is a file coming, 
but you are out of disk space partway.

So it doesn't solve the issue of messages coming that can't be handled 
for whatever reason.

If the peers tell their in headers, it does genuinely tell you that you 
can't send a message above that limit, but it does not tell you that you 
can succeed to send a message below.  It might be worth having that 
optional "Don't send me messages bigger than X" header anyway and if 
it's not there it defaults to accepting any size message because it 
doesn't buffer them.

I think if the other peer, dynamically, finds it can't cope with what 
ends up coming all being called one message, it's OK if the websocket 
connection closes due to that.  It can't actually cope.

-Andy

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 13 Apr 2011 13:47:50 +0300

As far as I remember the huge frame length header was introduced in order to
support TransferFile or similar routines that allow the kernel to transfer
data directly from the file IO handle into the socket. In such cases the
file would be sent as one huge frame.  But since the introduction of payload
masking I'm not sure if this approach in sending data is still applicable.

I think we should reconsider the frame length field again. I don't have any
specific preference but I agree than a 32 bit frame length field would be
sufficient.

I also agree that it would a good idea to have the communicating sides
negotiate the maximum allowed message size (i.e. the total size of all
frames for a message).

Not negotiating this option will result in the communicating sides being
forced to drop the connection when their limit is reached.
Sending some error code is not always an option since in many cases simple
applications do not read when sending data so they would not receive the
error until the connection is aborted by the remote side.

Maybe the two sides could exchange such properties during the WS handshake?

With best regards,
Arman

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy
Green
Sent: Tuesday, April 12, 2011 11:30 AM
To: Greg Wilkins
Cc: Hybi
Subject: Re: [hybi] On Fragmentation of Frames

Hi -

This is a bit wimpy to say "much more likely".  Either you can't deal with
hanging coding it and it blows up sooner or later, or you can deal with it.
If you can deal with it, you don't care any more about frame reassembly.

If you choose to implement in a way that breaks with legal, large frames,
that's a detail about your implementation.  You can adapt your
implementation to not care rather than introduce frame MTU concept into the
spec.

> something that an impl ever need calculate, let alone limit.

That is wrong.  If you write an implementation that aggregates frames you
introduce a fragility to legal frame length and your implementation has a
problem.  If you implement it in a better way, it is not sensitive to frame
size.

Separately, the maximum legal frame length is meaningless in the real world.
It can stay like it is without direct problem but it just gives ammo to the
masking-fudsters.  It'd be better to limit it to 16-bit or 32-bit length and
make masking subordinate to framing.

-Andy

---

From: Andy Green <[email-masked]>
Date: Wed, 13 Apr 2011 12:00:19 +0100

> specific preference but I agree than a 32 bit frame length field would be

Yeah I don't mind either 16-bit or 32-bit either, although I don't 
really see what I will do with the extra 16 bits when I can just spam 
extra frames without limit and other pressures make me not want to have 
particularly long frames.

> Maybe the two sides could exchange such properties during the WS handshake?

That also sounds fine, but we need to be clear just telling the maximum 
doesn't mean the peer can always actually take a message below the 
maximum, ie, it doesn't get us out of handing the same error conditions 
coming randomly because the peer has exhausted some resource it depended 
on to take the max message on that connection at that moment.

It will help that at least the other peer will presumably never issue a 
message larger than the maximum his partner could even imagine to handle 
at any time.

-Andy

---

From: "Simon Pieters" <[email-masked]>
Date: Wed, 13 Apr 2011 13:28:48 +0200

> masking I'm not sure if this approach in sending data is still  

The masking is only from client to server.

-- 
Simon Pieters
Opera Software

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 13 Apr 2011 18:35:41 +0300

>Yeah I don't mind either 16-bit or 32-bit either, although I don't
really see what I will do with the extra 16 bits when I can just spam extra
frames without limit and other pressures make me not want to have
particularly long frame

64KB (16 bits) is too small for a single frame. There is often a need to
send an already buffered message that exceeds 64KB, in such cases the
message that is otherwise prepared for sending would need to be chunked.
This would require spending additional processing on splitting it. Receiving
such a message would also require additional processing for reading each
frame's header and concatenating the payload. In my opinion this continuous
splitting/concatenating is unnecessary just for saving 16 bits in the frame
header. Note that my background is in non-browser communications where
medium/large/super large messages are the norm.

A 32 bit header would permit the sending of 2GB as a single-frame message
which is more than enough for today and is probably going to be a reasonable
frame size for the foreseeable future.

This not a big deal, if the WG doesn't feel like changing the frame header I
can live with what we have now.

> That also sounds fine, but we need to be clear just telling the
maximum doesn't mean the peer can always actually take a message below the
maximum, ie, it doesn't get us out of handing the same error conditions
coming randomly because the peer has exhausted some resource it depended on
to take the max message on that connection at that moment.
> 
> It will help that at least the other peer will presumably never
issue a message larger than the maximum his partner could even imagine to
handle at any time.

If the message is below the maximum limit, the peer can stop receiving and
wait for either a timeout or some resources to become available.
However if the message is above the maximum the receiver will have to either
drop the connection or waste a lot of time by receiving and discarding the
frames of the message that it can't process due to its too large size. This
is presently a problem with some HTTP services where time is wasted
uploading a file only to get an HTTP error reporting that the file is too
big.

I would say that negotiating the max message size is not a requirement, but
it would definitely be a good-to-have feature.
Support for this feature could be optional, but the feature itself should be
in the standard so that everyone who supports it does so in an interoperable
manner.

With best regards,
Arman

---

From: Andy Green <[email-masked]>
Date: Wed, 13 Apr 2011 17:15:21 +0100

> 64KB (16 bits) is too small for a single frame. There is often a need to

I don't see why it is "too small", too small for what?  Obviously it 
works fine with 16 bits.  Anyway I don't mind either way.

> medium/large/super large messages are the norm.

I don't mind much if it's 32 bits but don't forget there are pressures 
to keep frames small.  In the mux case, there is going to be all kind of 
chopping and tagging and management overhead going on so cutting at 
64Kbytes is irrelevant there.

> frame size for the foreseeable future.

I didn't really understand what bad thing happens with smaller maximum 
frame size.  It's not like it stops working or there's massive overhead, 
nor is there any real value in getting 2GByte in one blob blocking 
control messages for minutes on a typical connection.  Anyway it doesn't 
matter either way.

> wait for either a timeout or some resources to become available.

I guess, although then the standard might need to talk about timeouts 
being mandatory if it is expected one end will use stalling forever as 
flow control.

> in the standard so that everyone who supports it does so in an interoperable

Yeah it makes sense to define it, unlike frame MTU case.

-Andy

---

From: Scott Ferguson <[email-masked]>
Date: Wed, 13 Apr 2011 09:17:00 -0700

> in the standard so that everyone who supports it does so in an interoperable

Postponing until after mux might be a good idea. mux needs frame/channel 
limits intrinsically as part of its basic requirements, and having two 
separate max size mechanisms may be a bit of a mess.

-- Scott

---

From: Andy Green <[email-masked]>
Date: Wed, 13 Apr 2011 17:42:02 +0100

> separate max size mechanisms may be a bit of a mess.

FWIW x-google-mux currently has a scheme where differential headers are 
issued to override the ones that differ from the original connection 
set.  In that way, it can use exactly the same header-based scheme for 
singletons and muxed child connections for this any anything else 
header-based.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Wed, 13 Apr 2011 14:32:49 -0400

> Yeah it makes sense to define it, unlike frame MTU case.

It seems hard to imagine how an endpoint would determine this value, and it
goes back to when we were discussing sending the total message length in the
initial frame.

If I am going to convert text messages to UTF16, how do I know what to
advertise as my maximum message length?  Do I play it conservative, assuming
that all text messages are US-ASCII, so I send floor(bufsize/2)?  Do I
assume the best case of floor(bufsize*3/2) [fitting 3-byte UTF8 characters
in one UTF16 character] so a message that might fit won't be rejected?  Do I
assume some probabilistic value based on the expected distribution of
characters in received messages?

If I just allocate buffers from the heap, how can I get a reasonable value
for this maximum message size?  Even if I can, how can one value be correct
for the life of the connection, as other things will be using/freeing heap
unrelated to the WebSocket message stream.  If I am a busy server handling
lots of connections,

It seems likely that if such a message size limit were required, endpoints
would simply pick some arbitrary value and advertise that, and I don't think
that is any better than not advertising any value.

---

From: Andy Green <[email-masked]>
Date: Wed, 13 Apr 2011 20:00:03 +0100

On 04/13/2011 07:32 PM, Somebody in the thread at some point said:

Hi -

> It seems hard to imagine how an endpoint would determine this value, and
> it goes back to when we were discussing sending the total message length
> in the initial frame.
>
> advertise as my maximum message length?  Do I play it conservative,

If a peer has already 'picked a static number' for what they're willing 
to malloc on one connection, since no peer doing buffering as in browser 
case can live up to infinite message length, they might as well tell it 
to the peer.

> If I just allocate buffers from the heap, how can I get a reasonable
> value for this maximum message size?  Even if I can, how can one value
> be correct for the life of the connection, as other things will be
> using/freeing heap unrelated to the WebSocket message stream.  If I am a
> busy server handling lots of connections,

Right but I already made this point twice, that telling your maximum is 
not the same as guaranteeing that anything below will pass.  Yet still 
telling the maximum could have value if the sender can adapt to the 
knowledge, since the alternative is you waste time sending up to the 
limit and then your connection is dropped.

> It seems likely that if such a message size limit were required,
> endpoints would simply pick some arbitrary value and advertise that, and
> I don't think that is any better than not advertising any value.

I think the missing ingredient is peers that are buffering the message 
locally absolutely cannot tolerate infinite message length the protocol 
allows since it lets a remote peer kill its partner.  So they will take 
action at some point to close the connection wasting what was sent.  So 
you may as well tell the other side, at best, you can never send me a 
message of more than that amount.

Trying to start the message with length indication is no help either 
since it may be a stream the sender does not know the length of.

In the end the receiving peer is always the guy that decides if and when 
he has had enough in part with a possible static sanity limit and in 
part dynamically because his malloc for the next bit failed.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 14 Apr 2011 11:03:25 +1000

> frame size for the foreseeable future.

Currently the way that jetty is implemented on the server side is that
I have a  fixed pre allocated buffer size into which frames are
received.
Applications can then individually set the max size of an aggregation
buffer per connection, which is used to aggregate multiple frames into
a single message.  Applications may also choose to consume the frames
directly without any aggregation.

So for sending a 2GB file to Jetty, you will have to fragment it,
because nobody is going to set the pre allocated frame buffer at
anything large enough to accept 2GB.   The default is 64MB.

Now this is not the only way to implement a server and it may not even
be close to the best way.  But it is certainly not an unusual way to
handle this kind of thing and I think Jetty should be totally within
it's rights to say to a client - if you want to send me a 2GB message,
please send it to me in fragments < 64MB.

Currently if a client wants to send 2GB frames, then it wont be able
to talk to Jetty. Having a negotiated max frame size would allow such
a client to talk to Jetty.

cheers