Subject: [hybi] "Frame too large" error with and without max frame size

From: Bruce Atherton <[email-masked]>
Date: Fri, 05 Aug 2011 18:28:31 -0700

I don't seem to be getting anywhere with my appeal about why I need an 
optional max-frame-size header, so I thought I would try another approach.

There is a "Frame too large" error code defined in the spec, which means 
that any compliant Websocket endpoint needs to be able to handle it. 
Note that there is no "Message too large" error code, so compliant 
Websocket endpoints do not need to deal with that.

You probably want to handle a "Frame too large" error in one of two ways:

1) Your frame size is FIXED, so you close a connection if the other end 
can't handle it.

2) Your frame size is DYNAMIC, so you adjust it until you stop getting 
"Frame too large" errors.

Let's see what happens with and without information from the other end 
about the frame size it can handle.

A. Other end sends you the maximum frame size it can handle:

   FIXED) As soon as you get the Max-Frame-Size header when upgrading 
the connection, you see it is less than your fixed frame size and so you 
fail fast by closing the connection.

   DYNAMIC) As soon as you get the Max-Frame-Size header when upgrading 
the connection, if it is less than your current frame size you adjust 
your frame size to match the size the other end can handle. Your first 
and all subsequent frames are sent with that size.

B. The other end has no way to indicate the maximum frame size it can 
handle:

   FIXED) You upgrade the connection and it appears to work. You may 
receive a few messages from the other end. But when you send out your 
first frame you get back a "Frame too large" error. At that point you 
close the connection.

   DYNAMIC) You upgrade the connection and it appears to work. You may 
receive a few messages from the other end. But when you send out your 
first frame you get back a "Frame too large" error. You cut your frame 
size (perhaps in half) and try sending your frame again. You may get 
another "Frame too large" error. Your code must continue sending out 
frames with progressively smaller sizes until it stops getting that 
error. You then use the resulting frame size for the duration of the 
connection.

So there you have it. You have to choose A or B to implement if you want 
to be compliant. Note that you have to implement this no matter how you 
personally deal with large frame sizes, whether you do streaming or not, 
because you can't control how the other end is implemented.

Now honestly, which is more sensible, A or B?

I choose A every time.

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 5 Aug 2011 22:17:45 -0700

> that any compliant Websocket endpoint needs to be able to handle it.

A status code can only appear in a CLOSE frame, upon receiving the draft
says what to do: send back a CLOSE (unless you sent one before),
after which the original sender of CLOSE is supposed to close the TCP
connection.

In the absence of a protocol implied or announced/negotiated max. frame
size, IMHO it would be consequent to remove the predefined status code
"frame too large" from the draft.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 6 Aug 2011 07:31:58 +0200

> Now honestly, which is more sensible, A or B?

I agree with you Bruce. In fact, we've had a comparable issue with TCP
years ago : the MSS was exchanged between both endpoints with no way
for intermediaries (routers) to indicate whay they support, so in practice
a path MTU discovery protocol was invented for that, consisting in attempts
to send large packets and expect an ICMP message in return. The problem is,
ICMP does not pass everywhere so the PMTU mechanism is not reliable over
the net these days. This has resulted in everyone assuming that an MTU or
1500 was necessarily fine over the net since many routers have ethernet
connectivity, which led to many issues with ADSL lines using PPPoE for
several years because of the PPPoE encapsulation header. In the end, since
nobody was able to tell if 1500 bytes frames would pass or not, many sites
have deliberately reduced their MSS in the hope that ADSL users will not
have connectivity issues. And the Net is now supposed to have a 1500 MTU.

If the IP protocol had supported an MTU field from start, or if TCP headers
had been mangled by routers so that each one would adjust the advertised
MSS to set its supported value, there would never have been such an issue
and right now it would still be possible to work beyond 1500 between capable
endpoints.

Now back to WebSocket, what I fear without a max-frame-size parameter is
that once we discover *at least* one very common product with limitations,
whether it's a server, browser, or worse and intermediary, we'll quickly
have a static max-frame-size tunable in various agents in order to fit the
limited component's capabilities. And this value will become the "well known"
frame size, with nobody daring to send anything larger due to lack of
reliability.

And my suspicions go directly to the content inspection gateways which will
inevitably appear. Most likely they will work on whole frames first because
it's easier for them, just as pattern lookups in TCP is often performed on
packets within many network components, with no consideration for packet
boundaries, ordering nor fragmentation. If such stupid components have no
way to advertise a size limit, surely they'll enforce one, resulting in bad
experience for users and/or static setting in many agents.

For this reason I think that the max-frame-size is a safe long-term choice now.

Regards,
Willy

---

From: Paul Colomiets <[email-masked]>
Date: Sat, 6 Aug 2011 19:25:16 +0300

Hi Willy,

> experience for users and/or static setting in many agents.
I have a similar thoughts. But inspection of content by frames means that
malicious sites will try to split frames arbitrarily to skip those gateways.
Also if new gateways will need to reconstruct tcp stream to decode frames,
they probably can be smart enought to join frames into a message.

-- 
Paul

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 6 Aug 2011 18:32:02 +0200

Hi Paul,

> malicious sites will try to split frames arbitrarily to skip those gateways.

Indeed, but that is not our problem. We're aiming for interoperability,
not for helping lazy developers make their products better.

> they probably can be smart enought to join frames into a message.

Reconstructing a TCP stream is trivial, you just need to transparently
bind to an IP:port and you receive it. Inspecting a stream is slightly
more complex, especially when it comes to pattern matching across buffer
boundaries. IDS/IPS have been fooled that way for a long time, and
history repeats itself. And my guess is that in order to limit issues,
such products will simply enforce limits and move the problem somewhere
else.

Regards,
Willy

---

From: Paul Colomiets <[email-masked]>
Date: Sat, 6 Aug 2011 21:59:53 +0300

Hi Willy,

> not for helping lazy developers make their products better.
Wow! So why we have that stupid masking and handshake? Ok, thats
rhetorical question :)

> such products will simply enforce limits and move the problem somewhere

That's roughly my point. I just think that limit will be for message size.
Like with http: there could be a limit for file size in multipart
body. But usually
servers limit maximum body size. And there are several methods to signal
that to browser, e.g. Expect header or MAX_POST_SIZE (don't exacly
remember spelling) form field.

Since I expect more users will use websockets with small messages (and
use http for big down/uploads), this kind of limit might be very useful.

-- 
Paul

---

From: Bruce Atherton <[email-masked]>
Date: Sat, 06 Aug 2011 12:39:03 -0700

> after which the original sender of CLOSE is supposed to close the TCP

Right, but what you do after is the important thing. Do you just leave 
things at that, or do you retry the connection with a smaller frame 
size? Those were the two scenarios I was illustrating.

> "frame too large" from the draft.

Exactly. And since "frame too large" is a sensible reason that an 
implementation may want to close the connection, and since the knowledge 
about that reason can enable the other end to take corrective action, it 
makes no sense to drop it. There is no way people are going to agree to 
a protocol specific maximum (see the payload length field discussion) so 
an announced/negotiated maximum is the only one that makes sense.

This shows why announcing maximum message size makes no sense at the 
Websockets level (though it may well at the application level). There is 
nothing that the Websockets implementation can do to retry a message 
that is too large.

---

From: Tobias Oberstein <[email-masked]>
Date: Sun, 7 Aug 2011 01:04:00 -0700

> implementation may want to close the connection, and since the knowledge

I don't agree. An implementation incapable of processing frames above
some size is bogus:

The spec allows intermediaries to further fragment or coalesce (data) frames
as it likes.

Thus, endpoints must be capable of processing frames of any size.

A further consequence is that frame boundaries can't carry any semantics.

Thus it is always valid for endpoints to fragment/coalesce frames
themselves.

==

So, if for any reason you don't want to process frames above some size, just
fragment them as you receive them, and always send frames small enough.

Given that, there should be no close reason "frame too large", and a peer
will not want to retry, since there is no corrective action it can take. The
corrective action is fix the bogus implementation.

-1: announcing/negotiating max. frame size in core protocol
+1: removing close frame status code "frame too large" from spec

> nothing that the Websockets implementation can do to retry a message

Agreed. In general, when an endpoint is operating under constraints
originating outside the WebSockets protocol, those constraints must be dealt
with above the WebSocket protocol level.

However, even then, "dealt with" may not always involve "announce" or
"negotiate".

For example, the current JavaScript API for WebSockets in browsers is
message based, not frame-based, and not streaming.

This is a constraint external to the WebSockets protocol. A consequence of
this constraint is that a browser will have to buffer received frames that
make up a received message.

A browser could:

a) Just go ahead. It'll allocate more and more memory, and some point
swapping will kick in, and at some point the system will halt. The TCP will
be dropped.
b) Have some hard-coded, implementation defined, or user setting for max.
message size.
c) Determine a sane max. message size based on system RAM.
d) Buffer the message to disk, and fire an event not with ArrayBuffer but
some BFile

d) could only kick in, when a defined threshold for buffered frames for a
message is crossed. i.e. buffer data frames up to 1M in memory, then flush
to disk and subsequently buffer to disk. when message is complete, fire
appropriate event.

d) could also work by memory-mapping the temp file buffering the message,
which might allow to create an ArrayBuffer based event from the mmap'ed file

I'd probably prefer d). No need to announce/negotiate max. message size ..

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 7 Aug 2011 11:20:07 +0200

> Thus it is always valid for endpoints to fragment/coalesce frames

I think you did not read the examples that were given about frame-based
compression, signature or any other extension that intermediaries are
not allowed to refragment, and which would require to buffer a whole
frame. Then it makes sense to be able to advertise a limit.

Regards,
Willy

---

From: Tobias Oberstein <[email-masked]>
Date: Sun, 7 Aug 2011 02:39:13 -0700

> > The spec allows intermediaries to further fragment or coalesce (data)
> > frames as it likes.
> >
> compression, signature or any other extension that intermediaries are not
> allowed to refragment, and which would require to buffer a whole frame.
> Then it makes sense to be able to advertise a limit.

Well, I did.

I was referring to the base WebSockets protocol, the case when no extension is active and RSV = 0.

The draft spec says:

"""
A sender MAY create fragments of any size for non-control
      messages.

An intermediary MUST NOT change the fragmentation of a message if
      any reserved bit values are used and the meaning of these values
      is not known to the intermediary.

An intermediary MUST NOT change the fragmentation of any message
      in the context of a connection where extensions have been
      negotiated and the intermediary is not aware of the semantics of
      the negotiated extensions.
"""

So everything I said above I still hold on to.

What is done outside the base WebSockets protocol, be it with extensions or
application level, should not creap into the base protocol spec. It's their business.

---

From: Paul Colomiets <[email-masked]>
Date: Sun, 7 Aug 2011 19:24:14 +0300

Hi Bruce,

On Sat, Aug 6, 2011 at 10:39 PM, Bruce Atherton <[email-masked]> wrote:
> nothing that the Websockets implementation can do to retry a message that is
> too large.
>

You are partially right. But you don't learning from HTTP. In HTTP there usually
is a frontend server which buffers the whole HTTP request. Then forwards it
throught another protocol to a backend (so it's not exactly intermediary).
Frontend has a limit for sure (or otherwise it is very vulnerable to
DOS attacks).
But backend has no reliable way to determine that limit.

So the problem of handling message size is an application level problem. But
the only thing that frontend server can do is close connection. (And Yes, I
don't think that streaming big messages to backend is a viable solution. In
HTTP world streaming is going only for big file uploads, and I believe that
big file uploads will stay out of websockets in future for lots of reasons).
So it would be better if javascript could just have an exception when trying
to send something too big. Then application can handle that.

All that means, that some bigger companies will have no troubles (they
control all ifrastructure). And most smaller projects will appear unreliable
with websockets (while tiny bit of them will use some strange heuristics
to determine message size or will try to always use very small ones).

-- 
Paul

---

From: Bruce Atherton <[email-masked]>
Date: Sun, 07 Aug 2011 13:10:49 -0700

> I don't agree. An implementation incapable of processing frames above

Prepare for a number of bogus implementations, then, mine among them. 
Because I think you are just plain wrong. It is perfectly reasonable to 
have a maximum frame size, particularly when extensions may need to 
operate on an entire frame of data at once.

> Thus, endpoints must be capable of processing frames of any size.

Intermediaries can fragment and coalesce frames, you are right. But that 
doesn't mean the other end has to accept whatever they coalesce the 
frames in to. Whether the other end can close the connection with a 
"Frame too large error" or must use a generic error code, it still gets 
to close the connection.

Losing the specific error code just means that the sending side cannot 
recover.

> Thus it is always valid for endpoints to fragment/coalesce frames

Frame boundaries can contain semantics for extensions. That is why 
intermediaries are required to understand all extensions before they can 
alter frame boundaries.

> However, even then, "dealt with" may not always involve "announce" or

Right. Websockets has no business defining how an application with a 
message size that is too large deals with that issue. It is application 
specific. As you say, the implementation can decide how it communicates 
the problem to the application. It may just close the connection, It may 
call a specific callback, it may return an error code, it may throw an 
exception up to the application, it may consume memory until it fails, 
whatever.

---

From: Bruce Atherton <[email-masked]>
Date: Mon, 08 Aug 2011 17:06:54 -0700

On 07/08/2011 9:24 AM, Paul Colomiets wrote:
> So it would be better if javascript could just have an exception when 
> trying to send something too big. Then application can handle that.

I believe I understand why people want a maximum message size header. 
While messages are an application layer concern, there is a layer of the 
Websockets implementation that may have to concern itself with how much 
buffering is done by the message if the Websockets implementation 
actually provides the message implementation and not just the interface. 
That means that a message that is a reasonable size for both sides of 
the application as well as one end of the Websockets channel may be 
rejected by the other end due to limitations in the WS layer.

In such cases, the application cannot know what limits the underlying 
Websockets implementation places on message size unless the 
implementation signals that limit to the application, and it has to do 
so at both client and server ends to prevent data from being sent 
needlessly. Provided the API has some mechanism to signal to the 
application that the message will fail to be delivered because it is too 
large, the application can be given a chance to rejig a message before 
it is actually sent on the wire.

I don't find the argument as compelling as the one for max-frame-size, 
but I get that there is an argument to be made.

As for our apparent impasse on max-frame-size, perhaps we could all 
reach a compromise. How about we allow that an endpoint MAY send a 
max-frame-size and the other end SHOULD adjust its maximum sending frame 
size downward. That way if you find this adds too much complexity to 
your implementation, you can completely ignore it. Your implementation 
can act as if the max-frame-size header does not exist in the spec and 
is never sent, and still remain completely compatible.

Would that move us forward?

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 9 Aug 2011 11:22:40 +1000

Bruce,

I think your post is a good way to look at it.   Since we have a frame
too large error, we have to deal with it.  If a sender is not prepared
to adjust the sizes they send in response, then we will have
interoperability failures with some implementations unable to to
communicate.    So senders will have to adjust the size of frames they
send.    Which leaves us with 2 choices - guessing the size or using
passed value.     I think guessing is a poor choice.

The only alternative is to not have a frame too large error - which
will force all implementations to have to handle indefinite lengths of
both frames and messages.    In reality, few implementations will
actually be written to handle both infinite frame and message sizes
and almost all will actually have a size limit of some kind.  They
will just not have a way to communicate this limit.

Let's also consider the flip side.   What is the harm of declaring a
max-frame-size?    If there are implementations that really can handle
2^64 frames, then they can declare that size and we've wasted a few
bytes in the handshake - or they can just not declare and no bytes are
wasted.   I can see why some with good implementations might see that
a declaration is not useful, but I can't see why anybody could
consider it harmful.

cheers

On 6 August 2011 11:28, Bruce Atherton <[email-masked]> wrote:
> that any compliant Websocket endpoint needs to be able to handle it. Note
> that there is no "Message too large" error code, so compliant Websocket
> endpoints do not need to deal with that.
>
> Let's see what happens with and without information from the other end about
> the frame size it can handle.
>
>  FIXED) As soon as you get the Max-Frame-Size header when upgrading the
> connection, you see it is less than your fixed frame size and so you fail
> fast by closing the connection.
>
>  DYNAMIC) As soon as you get the Max-Frame-Size header when upgrading the
> connection, if it is less than your current frame size you adjust your frame
> size to match the size the other end can handle. Your first and all
> subsequent frames are sent with that size.
>
>  FIXED) You upgrade the connection and it appears to work. You may receive a
> few messages from the other end. But when you send out your first frame you
> get back a "Frame too large" error. At that point you close the connection.
>
>  DYNAMIC) You upgrade the connection and it appears to work. You may receive
> a few messages from the other end. But when you send out your first frame
> you get back a "Frame too large" error. You cut your frame size (perhaps in
> half) and try sending your frame again. You may get another "Frame too
> large" error. Your code must continue sending out frames with progressively
> smaller sizes until it stops getting that error. You then use the resulting
> frame size for the duration of the connection.
>
> So there you have it. You have to choose A or B to implement if you want to
> be compliant. Note that you have to implement this no matter how you
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: John Tamplin <[email-masked]>
Date: Mon, 8 Aug 2011 21:25:36 -0400

On Mon, Aug 8, 2011 at 8:06 PM, Bruce Atherton <[email-masked]> wrote:

> As for our apparent impasse on max-frame-size, perhaps we could all reach a
> compromise. How about we allow that an endpoint MAY send a max-frame-size
> and the other end SHOULD adjust its maximum sending frame size downward.
> That way if you find this adds too much complexity to your implementation,
> you can completely ignore it. Your implementation can act as if the
> max-frame-size header does not exist in the spec and is never sent, and
> still remain completely compatible.
>

+1

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Scott Ferguson <[email-masked]>
Date: Mon, 08 Aug 2011 19:33:04 -0700

On 08/08/2011 06:22 PM, Greg Wilkins wrote:
> I can see why some with good implementations might see that

Unnecessary complexity is always harmful.

It's a tax on every implementation and it's an additional source of 
interoperability failures.

A simpler spec allows for simpler implementations. That's always a win.

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 9 Aug 2011 13:11:18 +1000

> interoperability failures.

I don't understand how this can be seen as a tax on every
implementation because I have not seen a suggestion of an alternative
that results in a less complex or more interoperable solution.

If we have no max frame size, then we could pretend that all
implementations will handle 2^64 frame sizes.   But in reality, few
implementations will be able or allowed to handle such huge frames.
Even spooled to disk such frames represent significant resources that
many implementations might not have or be prepared to allocate to a
single connection.    So either implementations will have to be
complex enough to deal with effectively infinite frames, or there is a
secret unknown interoperability failure point.

If we just allow implementations to have secret max frame sizes, that
just shifts the complexity to the other end, which either has to guess
the acceptable frame size or will just be non interoperable.

If an implementation wants to assume acceptable frame sizes, then it
is free to ignore max frame sizes and not declare itself - there is no
tax.

> ... and it's an additional source of  interoperability failures.

What interoperability failures are created with a declared max frame
size?    Or are you saying that the problem comes from a max frame
size (even if not declared).    If so, do you really think all
implementations MUST accept 2^64 frames?

regards

---

From: Diogo Pereira <[email-masked]>
Date: Tue, 9 Aug 2011 06:47:26 +0100

> secret unknown interoperability failure point.

Browsers will have to buffer the whole message, so I don't think
anyone expects them to handle 2^64 byte messages. As a consequence,
they won't be expected to handle 2^64 byte frames.
If an implementation can handle 2^64 byte messages, however, I would
certainly expect it to be able to handle a 2^64 byte frame. This is
not complex at all, and it can be done with only a few hundred bytes
of memory.

--
Diogo

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 9 Aug 2011 16:44:53 +1000

> not complex at all, and it can be done with only a few hundred bytes

This is not an issue about the complexity of handling large frames.

This is an issue not handling large frames and what to do if a sender
is talking to a receiver that has a limit.

It is perfectly valid for implementations to have a limit on the size
of frame they are prepared to handle and almost all implementations
will have a limit < 2^64.      When a sender wants to send a frame to
a receiver that has a frame size limit, then the choices are:

 a) not deal with it.  The sender and receiver are just not
interoperable above a certain sizes.   This will mostly work as many
applications will exchange only small messages and I'd expect to see
lots of  implementation get away with default limits of 8k.  But as
applications that need large messages are developed, they will then
start hitting problems and the result will either be applications
doing their own fragmentation or implementations assuming some defacto
max frame size limit.
 b) deal with it by the sender guessing a smaller and smaller fragment
size until the connection works.
 c) deal with it by having a declared size.

Is there an option that I'm missing.

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 09 Aug 2011 09:00:27 +0100

On 08/09/2011 07:44 AM, Somebody in the thread at some point said:

Hi -

> of frame they are prepared to handle and almost all implementations

A peer with a frame size limit is BROKEN.

Adding a "Frame MTU" and all the business around that is just trying to 
help the irretrievably broken by adding more complexity and useless 
stuff to implement.

Since nobody can mandate correctness in the real world, broken 
implementations should just defiantly glory in their shame of not 
working properly until they're fixed or deprecated for stuff that is 
written better and actually works.

The spec itself is deranged in this and encouraging this kind of 
breakage by pretending huge frame sizes have any meaning or use.

-Andy

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 9 Aug 2011 01:18:35 -0700

+1

And consequently, the "frame too large" close error code 1004 should be removed
from the spec, since it suggests the wrong, namely that it would be acceptable
for a peer to deny processing of frames above some size, and by that impose the
burden of it's brokenness onto the others.

Rather the spec should make it explicit:
a peer MUST be capable of processing frames of any size.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 09 Aug 2011 12:17:10 +0100

> will just not have a way to communicate this limit.
Agreed.
+1

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 09 Aug 2011 13:42:53 +0100

>> a declaration is not useful, but I can't see why anybody could

This is already on top of a reliable transport protocol with ordering of 
out-of-order packets and workable streaming in the form of TCP/IP.  It 
already has stuff like path mtu discovery at the lower level.

There is no benefit in recreating in a higher level protocol the same 
mtu restrictions and then workarounds that's already handled to give us 
the ability to stream endlessly how we like.

It actually is harmful to write up this in the spec because it 
encourages people to understand they should be writing broken peers 
buffering frames instead of peers dealing with streams of bytes and the 
frame sizes only useful to find where the next frame header is coming.

-Andy

---

From: Diogo Pereira <[email-masked]>
Date: Tue, 9 Aug 2011 14:54:45 +0100

> This is not an issue about the complexity of handling large frames.

Well, if handling large frames is as easy as handling small frames,
then what is the reason for having a frame size limit?

> will have a limit < 2^64.

It is only valid in the sense that it is allowed. There is no valid
reason for it, so it should not be encouraged.

--
Diogo

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 09 Aug 2011 14:56:43 +0100

> A peer with a frame size limit is BROKEN.
No. It can be running on a device with very little memory. Not every 
client is a web browser running on a desktop.

As a side note, this feature is already present in other places such as 
SASL (RFC 4422).

But anyway, I think we should agree to disagree and move on. I don't see 
many people changing opinions after discussing this thread.

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 09 Aug 2011 16:43:52 +0100

> client is a web browser running on a desktop.

Right, I wrote libwebsockets for exactly that case.

> many people changing opinions after discussing this thread.

Maybe so, but in fact there was a thread about exactly this some months 
ago which did usefully change some peoples' opinions about how to 
architect a websocket parser.

If you treat it as a byte stream and pass on whatever payload you can 
buffer to upper layers as it comes in, you do not need to get into the 
mindset of "buffering frames" and then your code is freed from limits 
around frame size.

-Andy

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 09 Aug 2011 08:45:25 -0700

+1

Specifications shouldn't add complexity to work around broken 
implementations.

-- Scott

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 09 Aug 2011 08:49:17 -0700

> client is a web browser running on a desktop.

No. A small fixed-size buffer is sufficient to handle arbitrary frame 
and message sizes in the WebSocket layer. You're assuming a broken 
WebSocket layer :)

Application-layer limitations are outside the scope of the WebSocket spec.

>
> But anyway, I think we should agree to disagree and move on. I don't 
> see many people changing opinions after discussing this thread.

Since we're wrapping up, "moving on" would mean dropping the proposed 
spec addition.

-- Scott

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 09 Aug 2011 08:59:52 -0700

On 08/08/2011 08:11 PM, Greg Wilkins wrote:
> On 9 August 2011 12:33, Scott Ferguson<[email-masked]>  wrote:

Nope.

Since it's easy to support arbitrary frame sizes, most implementations 
will do so. The ones which don't are broken.

In HTTP, if a big chunk breaks POST handling, the web server is broken.

In WebSockets, if a big frame breaks message handling, the WebSockets 
implementation is broken.

> secret unknown interoperability failure point.

What are you talking about?

Streams aren't "complex". They don't require disk spooling. They only 
need a small, fixed buffer. I know you know how to implement streams, 
because you've implemented POST chunk reading, and I can't imagine that 
you buffer each chunk.

> is free to ignore max frame sizes and not declare itself - there is no

Of course there's a tax. Every implementation would need to limit frames 
to the max-frame-size, plus the code for the negotiation, plus the QA to 
test the requirement.

> size? Or are you saying that the problem comes from a max frame size (even if not declared).    If so, do you really think all implementations MUST accept 2^64 frames?

Yes. All implementations MUST accept 2^64 frames.

It's not hard to implement.

-- Scott

---

From: Piotr Kulaga <[email-masked]>
Date: Tue, 9 Aug 2011 16:22:47 +0000

-1

If there are no extensions, handling 2^64 frames is easy.
If there are extensions that cannot be handled in a streamed way, than the extension itself is broken. If it can be handled in a streamed way, handling 2^64 frames should be fairly easy.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alexey Melnikov
Sent: Tuesday, August 09, 2011 4:17 AM
To: Greg Wilkins
Cc: [email-masked]
Subject: Re: [hybi] "Frame too large" error with and without max frame size

Agreed.
+1

---

From: Diogo Pereira <[email-masked]>
Date: Tue, 9 Aug 2011 17:31:44 +0100

> Yes. All implementations MUST accept 2^64 frames.

Browsers need to buffer the entire message, so that doesn't really work.

Whether an implementation accepts a message or not should not depend
on the fragmentation, though.

--
Diogo

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 9 Aug 2011 10:21:51 -0700

> Browsers need to buffer the entire message, so that doesn't really work.

that is consequence of the limited API offered in the current WS JS browser object

there only is

WebSocket.onmessage

would there be additionally

WebSocket.onmessage_begin
WebSocket.onmessage_data
WebSocket.onmessage_end

to override, one could use streaming in JS.

the browser could still provide default implementations of those, which just do internal
buffering, in which case you would override the

WebSocket.onmessage

as today.

when you override one of the streaming hooks, the browser internal buffering is disabled.

you can have it both.

---

From: Bruce Atherton <[email-masked]>
Date: Tue, 09 Aug 2011 10:41:27 -0700

All three of you are wrong. Plain and simple. But I understand the idea 
that is leading you astray.

The benefit to making the interfaces between layers streamy is that you 
can handle extremely large frames. That's great if you actually need it. 
If you don't, though, it is a complete waste of time.

Yes, the protocol allows you to define an extremely large frame size. 
That is because a few people had some edge cases where they thought it 
would be useful. Nobody ever said that every implementation had to 
support frame sizes up to 2^63. If they had, I imagine the outcry 
against allowing that size would have been huge.

There are benefits to NOT using a streaming interface. It allows you to 
match your implementation to the protocol spec in a much cleaner way. 
This makes it easier to initially implement, find bugs, and upgrade to 
new versions of the protocol.

It also allows you to use extensions that work with a full frame of 
data. I've seen claims that if an extension requires that then it is 
broken, but that is nonsense. This is circular logic based on the 
mistaken notion that all implementations are required to handle enormous 
frame sizes.

The bottom line is you don't get to decide what tradeoffs I make in my 
design, I do, so long as I remain within the spec. If you want to 
require that my implementation can handle 2^63 sized frames then get 
that requirement explicit in the spec. And no, the current payload 
length field does not make that requirement.

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Tue, 09 Aug 2011 19:56:24 +0100

> that is leading you astray.

Well, it's nice that things are simple for you ^^

> If you don't, though, it is a complete waste of time.

Frames in websockets are pretty much the same deal as packets in regular 
TCP/IP traffic.

They contain payload but how big they are, how often they come, how 
fragmented they got needs to be immaterial to the content they're 
containing.  In particular the information about the packet, other than 
telling you how much more payload just arrived to add to the pile, is 
not only meaningless but dangerous to depend on.

> would be useful. Nobody ever said that every implementation had to

Months ago there was a thread examining this from the angle of two or 
three generations of best Ethernet bandwidth possible and 2^63 was still 
taking many years.  The best explanation I was given was that 2^63 was 
wanted as a practically unreachable goal to enforce the code to support 
in fact arbitrary frame lengths!  That's exactly what's being violated 
by wanting to introduce frame mtu concept - 2^63 is then perfectly 
pointless.

> against allowing that size would have been huge.

Hm.  Well, believe me you shouldn't take from the fact it's in this spec 
that there was no huge outcry against it.

> design, I do, so long as I remain within the spec. If you want to

This is not about chest-beating but arriving at agreement about what 
should be in the spec.

In particular the two camps about this issue seem to fall into people 
who think a frame is meaningful in itself and those that understand it 
as am immediately discarded wrapper for "some more payload" which must 
then be parsed according to the payload protocol and without expectation 
of clean boundaries.

-Andy

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 09 Aug 2011 13:11:59 -0700

On 08/09/2011 09:31 AM, Diogo Pereira wrote:
> On Tue, Aug 9, 2011 at 16:59, Scott Ferguson<[email-masked]>  wrote:

Ok, more precisely, if an implementation can accept a message of size N, 
it must accept that message as a single frame of size N (where N < 2^63).

> on the fragmentation, though.

Exactly.

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 10 Aug 2011 10:09:24 +1000

On 10 August 2011 01:59, Scott Ferguson <[email-masked]> wrote:
> Since it's easy to support arbitrary frame sizes, most implementations will
> do so. The ones which don't are broken.

There are many reasons to not support 2^64 frames is not because of
simple/broken implementations.     Not the least to avoid DOS attacks.

In HTTP there is not protocol limit to the length of the URL, header
fields or form content body, but in practise all servers implement
size limits on all of these.   Unfortunately these limits are not
disclosed so applications have to guess what they are - typically
settling for lowest commonly know values.   Worse still, many
applications are deployed without any knowledge that such limits exist
and then fail in production as user data gets bigger and blows out one
limit or the other.

Implementations are not going to handle 2^64 frames not because they
don't know how, or because the implementers are lazy, stupid or both.
 Frame sizes will be limited for good resource  allocation reasons.
This will also apply to implementations that do support dynamic
growing buffers for frames.

Are there any implementations out there today that can really accept a
2^64 frame without blowing memory or filling up their hard disk?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 10 Aug 2011 10:37:21 +1000

On 10 August 2011 04:56, "Andy Green (林安廸)" <[email-masked]> wrote:
> In particular the two camps about this issue seem to fall into people who
> think a frame is meaningful in itself and those that understand it as am
> immediately discarded wrapper for "some more payload" which must then be
> parsed according to the payload protocol and without expectation of clean
> boundaries.

Andy,

you have misrepresented the case for a declared frame size limit, so
I'm guessing you are not understanding it. So last attempt to express
it then I'll rest on this one....

ALL implementations will have a frame size limit.   For some that
might be imposed by a fixed buffer size, for others that might be from
a memory allocation policy; yet some might be constrained by available
disk space; and a few cases it might even be able to handle 2^64
frames.

But it is an indisputable fact that frames are of limited size -
either by protocol,  by policy, by deployment or by implementation.

Thus give that limits exist, this is only a question of cost/benefit
in providing a standard mechanism of disclosing what that limit is.

There have been numerous benefits described of a disclosed limit.  I
accept that these benefits are not universal, but I do believe they
exist in a significant number of cases.

I have seen no substantial costs.    Perhaps one could argue that a
disclosed limit will force more implementation to support
fragmentation of sends - but lack of disclosed limits will only make
implementations without fragmented sends inoperable with
implementations that have frame limits.   A declared limit will at
least give an implementation the option of avoiding such a failure.

regards

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 09 Aug 2011 18:08:12 -0700

> 2^64 frame without blowing memory or filling up their hard disk?

Of course there are.

You don't even need a buffer; you can pass-through data reads to 
underlying socket stream and use the caller's buffer.

In fact, if the application was a /dev/null or a hash generator or an 
echo service, the application could handle that message without needing 
any extra memory. (Granted, the Sun would burn out first, but it 
wouldn't require any memory.)

There's no need for infinite memory or disk space. A small fixed buffer 
on top of the raw socket can be useful for efficiency, but even that 
isn't strictly necessary.

  -- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 10 Aug 2011 12:18:52 +1000

On 10 August 2011 11:08, Scott Ferguson <[email-masked]> wrote:
> You don't even need a buffer; you can pass-through data reads to underlying
> socket stream and use the caller's buffer.
>
> In fact, if the application was a /dev/null or a hash generator or an echo
> service, the application could handle that message without needing any extra
> memory. (Granted, the Sun would burn out first, but it wouldn't require any
> memory.)

So you say that there are implementations out there (or a likely to be
written) that provide a streaming interface and are running
application that are /dev/null, hash generator or echo.

Let's assume that such beasties do actually exist - they still have a
max frame size!  it is 2^64.

How are such implementations adversely impacted by the ability to
declare that they can handle full sized frames?  I would posit that
such implementations are going to be SO rare that they would indeed
benefit from being able to declare their ability to handle full size
frames:

  WebSocket-Max-Frame-Size: 18446744070000000000

So that any other implementation that could possibly send such a frame
would know that it had connected with an implementation that could
handle it.

Without the ability to declare the ability to handle such huge frames,
no sane sender will ever attempt to actually send such a frame.

regards

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Wed, 10 Aug 2011 07:34:52 +0100

On 08/10/2011 01:37 AM, Somebody in the thread at some point said:
> On 10 August 2011 04:56, "Andy Green (林安廸)"<[email-masked]>  wrote:

Actually I believe I have captured the underlying problem in the quote 
above.

> it then I'll rest on this one....

Having implemented this, and now reading what you have written, I 
believe I understand it better than your good self.

> disk space; and a few cases it might even be able to handle 2^64

Above you first say "ALL" implementations have a frame size limit, then 
admit that the limit may be 2^64 (you mean 2^63) which is "no limit" 
since it's the biggest the protocol can deploy.  What are you even 
trying to say?  You are confusing message size limit with frame size limit?

libwebsockets (and it's nothing special about that, just the parsing 
architecture it has) has no frame size limit except the biggest size 
that can be told by websocket itself.  Neither "fixed buffer size", nor 
"memory allocation policy" nor "available disk space".  What you have 
written above is wrong.

If you view a "frame" as nothing more important than an "immediately 
discarded wrapper for some more payload", just there to tell you how 
many bytes to skip to find the next frame header, your mind is freed 
from this focus on dealing with frames as atomic entities.  You can take 
in as much of the payload as convenient (1 byte at a time if you like, 
or 256 bytes or whatever) and pass it up to whatever will consume it, 
and keep doing that until all the frame payload is all used.  The frame 
comes to you in TCP/IP packets, so you can play that game the moment the 
first packet comes without waiting for any more let alone the whole frame.

> either by protocol,  by policy, by deployment or by implementation.

I not only dispute it but have code showing that is wrong.

> in providing a standard mechanism of disclosing what that limit is.

MESSAGES may have to be held atomically, as in the browser / javascript 
API case.  FRAME size limit does nothing for that.  I can burst my 
message buffer by setting the frame limit at 2/3 the message buffer and 
sending two frames.  What have you achieved with a "frame limit"?

> There have been numerous benefits described of a disclosed limit.  I

The is one legit pressure to limit size of frames, and that is latency 
for the next, possibly control message.  So somebody sends a PING, you 
decided to issue a gigabyte frame and before there is no chance to issue 
the PONG frame before the other side timed out.  But that's different 
from frame size negotiation which I did not hear one good reason for yet 
-- feel free to enumerate these "numerous benefits".

For the latency reason, 2^63 (a frame takes many years to transmit even 
at Ethernet two generations ahead) frames are not useful.

> I have seen no substantial costs.    Perhaps one could argue that a

Come on man, that's not an argument it's just blather.

-Andy

---

From: Diogo Pereira <[email-masked]>
Date: Wed, 10 Aug 2011 11:47:54 +0100

> simple/broken implementations.     Not the least to avoid DOS attacks.

How is sending a 2^63 frame worse than sending an infinite number of
125 byte frames?

> and then fail in production as user data gets bigger and blows out one

The reason those limits exist is that the HTTP server needs to parse
those fields. A WebSocket server only needs to parse the frame header
(and there is already a limit for that: 14 bytes), and pass the
payload to the application. The application might have a limit, but
that will be a message size limit, not a frame size limit. Frames
don't matter.

> growing buffers for frames.

For the nth time, large frames don't use more resources. In fact, they
help you use less.

When you receive a frame with FIN=0, you have no way of knowing how
large the message will be. If you need to buffer the message (as
browsers do), then you can:

a) Allocate just enough memory for the frame;
b) Allocate enough memory for your maximum message size;
c) Guess.

Even if you're very good at guessing, you'll probably never do as well
as if you knew from the start how large a buffer you were going to
need.

Another related problem is if you send a large message in small
frames, the (message-buffering) receiver cannot close the connection
immediately if the message is larger than the maximum it can handle.
For example, if you try to send a 20 MB message to Firefox in 8 KB
fragments, Firefox will close the connection after it receives 16 MB.
If instead you send a single 20 MB frame, it will close immediately.

Large frames let you plan ahead, resulting in fewer wasted resources.

--
Diogo

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 11 Aug 2011 09:26:56 +1000

> For the nth time, large frames don't use more resources. In fact, they

And for the Nth time I'm saying that resource limits may be applied
for good reasons other than lazy programmers who don't know how to
parse large frames with small buffers.

> Large frames let you plan ahead, resulting in fewer wasted resources.

I am not arguing against sending large frames.  Indeed I believe that
declaring a frame size limit will facilitate large frames being sent.

Without knowledge of any limit that may apply, implementations may
assume lowest common denominator frame sizes and needlessly fragment.

regards