Subject: [hybi] Binary/Text Distinction

From: "Bob Gezelter" <[email-masked]>
Date: Mon, 15 Aug 2011 11:18:40 -0700

I agree with I?aki that the distinction between text/binary data is
unnecessary. Similarly, I agree with Dave Endicott's and I?aki's
comments about the mixing of different classes of functionality in the
WebSocket protocol.

As has been observed, the WebSocket protocol is payload-agnostic.
Regardless of the politics, it is thus emphatically NOT an applications
protocol. It is a protocol that provides a foundational service layered
on top of another transport protocol (e.g. TCP).

The protocol definition and usage would be far clearer if the areas
where mixing is occurring are edited. Examples include:

- The text/binary dichotomy in the opcodes and payloads
- The dependency on TCP (which is actually a dependency on a "serial,
guaranteed delivery transport"
- The interrelationship with HTTP (which is limited to session
establishment, and the specification of
  the WS and WSS schema

Additionally, as I have observed in the past, IMHO multiplexing is
necessary for effective use of the WebSocket protocol in the wild, it
should be part of the base specification, not an add on. However, I
freely admit that I do not see a compelling use case for a multiplexing
protocol with nesting, the aforementioned removal of the text/binary
distinction (to a separate document), and the non-excluded potential for
running a VPN-like tunnel within a WebSocket protocol sub-channel seems
more than sufficient to allow nesting without specifically including it.

Similarly, intermediate proxies could split a multiplexed stream into
separate streams as needed. IMHO, nesting makes that task more complex.

- Bob Gezelter, http://www.rlgsc.com

---

From: John Tamplin <[email-masked]>
Date: Mon, 15 Aug 2011 14:24:23 -0400

On Mon, Aug 15, 2011 at 2:18 PM, Bob Gezelter <[email-masked]> wrote:

> unnecessary.

There were months of discussions about this point during the framing
discussion.  Revisiting them at this point seems suboptimal.

Basically, the browsers need to be able to call JS code delivering the
message, and they want a simple API for JS code to access the message.  If
you left it up to the JS code to interpret it, then you would have to have a
much more complicated API and you would also leave open accessing binary
data as if it were text.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Stephen Williams <[email-masked]>
Date: Mon, 15 Aug 2011 11:27:00 -0700

I strongly agree that the protocol should be binary payload based.
Nesting will be needed, and tunneling should be an examined case for 
that.  However, there should be a lighter weight way to do something 
similar.  Channel naming might be an easy way to accomplish that: A 
subchannel could just be a normal peer channel "named" in a way that 
all of the peer channels are clearly together, for anything that needs 
to know that.  Then the protocol only need to work with a flat set of 
peer channels in most or all respects.

On Mon Aug 15 11:18:40 2011, Bob Gezelter wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

-- 
Stephen D. Williams [email-masked] [email-masked] LinkedIn: 
http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Yahoo:sdwlignet Resume: 
http://sdw.st/gres
Personal: http://sdw.st facebook.com/sdwlig twitter.com/scienteer

---

From: David Endicott <[email-masked]>
Date: Mon, 15 Aug 2011 15:15:22 -0400

If consensus has not been reached then further discussion is the optimal
choice.

To recap my understanding of why we ended up with binary and text frame
types:  since JS (currently) has limited ability to support binary content
(byte sequences) a concession to the de-facto reality is what the text type
was designed to support.   It would allow a "just-work" arrangement for JS
clients.      For those non JS client-agents that can support "pure" binary,
we have the binary frame type.

My choice, if it had been solely up to me, would have only an opaque frame
and let the application layer decide what the content-type was.   I can
envision several JS client implementations that would handle this without
much upset.   (ie. you could include an .as_text() method on the
MessageEvent or even .as_content(content-type) for a more general solution.
  The existing Blob/arraybuffer interface would remain as the default.)

I continue to fail to see the true advantage of a text frame type - it
brings content-type awareness into the WS protocol, but in a limited, kludgy
manner.    If a WS agent specifies a text frame, but then includes data that
isn't UTF-8 ... what are the failure modes?  Is it kept, but mangled?  Is it
discarded?  Is the connection dropped?   Error response?

If there was only an untyped binary frame, the decision would then be left
(properly) to the application layer.

My opinion is that this was a bad design choice.

My position is that I don't really care as it's not a big headache to work
with or around.

---

From: Stephen Williams <[email-masked]>
Date: Mon, 15 Aug 2011 16:18:08 -0700

I have a lot of experience with implementing and designing text, structured data, media, binary XML (i.e. W3C EXI), and otherwise 
binary protocols in C++, Java, and some other environments.  Based on that experience, I strongly suggest to make any text-only mode 
a subset of the default binary case, or ideally just a convention.  I would vote for UTF-8, no BOM as the de facto "binary" text 
representation.

I'm not very well versed on traditional limitations and headaches of Javascript in browsers (although I'm doing js development now 
and getting there), however I tend to think that the following demonstrates that binary can't really be much of a problem for modern 
Javascript in a modern browser:

http://jslinux.org/

I would suggest that minor issues with ease of programming today for text vs. binary protocol / application payloads can likely be 
solved by small libraries and/or language improvements.

The reality is that this protocol, if done right, will be used for all kinds of things in all types of languages.  It should be 
designed properly for that endpoint.

Stephen

On 8/15/11 12:15 PM, David Endicott wrote:
> If consensus has not been reached then further discussion is the optimal choice.
>
> To recap my understanding of why we ended up with binary and text frame types:  since JS (currently) has limited ability to 
> support binary content (byte sequences) a concession to the de-facto reality is what the text type was designed to support.   It 
> would allow a "just-work" arrangement for JS clients.      For those non JS client-agents that can support "pure" binary, we have 
> the binary frame type.
>
> My choice, if it had been solely up to me, would have only an opaque frame and let the application layer decide what the 
> content-type was.   I can envision several JS client implementations that would handle this without much upset.   (ie. you could 
> include an .as_text() method on the MessageEvent or even .as_content(content-type) for a more general solution.   The existing 
> Blob/arraybuffer interface would remain as the default.)
>
> I continue to fail to see the true advantage of a text frame type - it brings content-type awareness into the WS protocol, but in 
> a limited, kludgy manner.    If a WS agent specifies a text frame, but then includes data that isn't UTF-8 ... what are the 
> failure modes?  Is it kept, but mangled?  Is it discarded?  Is the connection dropped?   Error response?
>
> If there was only an untyped binary frame, the decision would then be left (properly) to the application layer.
>
> My position is that I don't really care as it's not a big headache to work with or around.
>
>
>
> On Mon, Aug 15, 2011 at 2:24 PM, John Tamplin <[email-masked] <mailto:[email-masked]>> wrote:
>
>     On Mon, Aug 15, 2011 at 2:18 PM, Bob Gezelter <[email-masked] <mailto:[email-masked]>> wrote:
>
>     There were months of discussions about this point during the framing discussion.  Revisiting them at this point seems suboptimal.
>     Basically, the browsers need to be able to call JS code delivering the message, and they want a simple API for JS code to
>     access the message.  If you left it up to the JS code to interpret it, then you would have to have a much more complicated API
>     and you would also leave open accessing binary data as if it were text.
>
>     -- 
>     John A. Tamplin
>     [email-masked] <mailto:[email-masked]>

-- 
Stephen D. Williams [email-masked] [email-masked] LinkedIn: http://sdw.st/in V:650-450-UNIX (8649) V:866.SDW.UNIX 
V:703.371.9362 F:703.995.0407 AIM:sdw Skype:StephenDWilliams Yahoo:sdwlignet Resume: http://sdw.st/gres Personal: http://sdw.st 
facebook.com/sdwlig twitter.com/scienteer

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 16 Aug 2011 10:10:39 +1000

> My position is that I don't really care as it's not a big headache to work

+1

It is a poor design, but it was a necessary compromise to lose the two
different framings we had for text and binary messages (YES, for those
new to the WG, we really did have totally different message framing
depending on the content type of the message!!!!).

The cost now is just the allocation of an opcode and a bit of split
personality in the APIs - which is a price that I'm prepared to pay to
avoid another round of design by war of attrition.

I can say the same for the 2^64 max frame size.   I can't justify it -
I expect few implementations will handle it - I expect even fewer will
ever send frames larger that 2^16, but I'm prepared to accept it in
the spec just so we can make progress.

cheers

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 16 Aug 2011 02:51:48 +0200

>comments about the mixing of different classes of functionality in the

Not having this distinction in the protocol would mean the information
is lost in transit and has to be coordinated out of protocol. I know of
no notable system where this has not caused problems. On the contrary,
it is common practise now to design systems to maintain the distinction,
new programming languages for instance typically have dedicated types
for binary data and separate types for text data.

If you want this to be reconsidered, show us systems where the need to
convey this meta data out of band has not lead to problems and has not
lead to complaints about how unwiedly use of the system is; or systems
where rates of both are very low, or systems where this distinction is
maintained where that has lead to something worse.

Note that the general expectation is that WebSocket-based protocols do
not enjoy the level of coordination, documentation, standardization, we
would expect of TCP-based protocols. With TCP you might say "Oh, HTTP
will handle that" (and HTTP might say media types handle that) but the
asumptions are do not involve standardized lower levels in the same way.

JavaScript applications are a primary target for the protocol. In JS,
the distinction between text and binary data is maintained through its
data types. Your argument is that instead of letting computers maintain
the distinction, humans should expend extra effort to re-establish it.
I understand not liking the distinction, but that doesn't make it wrong.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 16 Aug 2011 09:00:50 +0800

On 2011-08-16 at 10:51:48, Bjoern Hoehrmann wrote:
> is lost in transit and has to be coordinated out of protocol. 

The defining characteristic of thewebsocketprotocol is it's deployment model, which provides a perfect channel for this sort of coordination.  The server is effectively talking to itself.  Thus, no need for any form of negotiation...the server is no longer interacting with the browser, it's interacting with itself.

If you deploy a page with some JS that talks back to your server using XmlHTTPRequest, do you demand standardization or negotiation of the JSON encoding [1]?  I'll bet that most JS doesn't even check Content-Type headers before calling JSON.parse.  This is no different.

Negotiating text and binary is an artefact of the process that got us to this point, nothing more.  It provides no significant utility.

--Martin

---

From: David Endicott <[email-masked]>
Date: Mon, 15 Aug 2011 23:03:14 -0400

On Mon, Aug 15, 2011 at 9:00 PM, Thomson, Martin <
[email-masked]> wrote:

> The defining characteristic of thewebsocketprotocol is it's deployment
> model, which provides a perfect channel for this sort of coordination.  The
> server is effectively talking to itself.  Thus, no need for any form of
> negotiation...the server is no longer interacting with the browser, it's
> interacting with itself.
>

Please clarify.   If you're saying what I think you're saying, I'm must
disagree.

I would say the defining characteristics (there are several) of WS is that
it is an asynchronous framing transport protocol intended to be used
primarily by browser client-agents.

I don't understand what you mean by "server is effectively talking to
itself" - that seems nonsensical.

Further to Bjoern's comments about, what I believe he's saying about
including content-type information in the protocol: I would point out that
WS is content agnostic - the payload of the frame is not examined or
considered by WS.   This is because WS is a transport protocol.

TCP does not care what goes thru it, be it HTTP, SMTP, SSH or whatever, it's
job is only to transport the bytes to the endpoint.  Similarly, WS does not
care what is in it's frames.    It is the application using WS that imposes
the concept of content-type on the data communicated.

The text frame hack is simply a nod to common use convenience.   (And an
unnecessary one at that, IMHO)

> If you deploy a page with some JS that talks back to your server using
> XmlHTTPRequest, do you demand standardization or negotiation of the JSON
> encoding [1]?  I'll bet that most JS doesn't even check Content-Type headers
> before calling JSON.parse.  This is no different.
>
> Negotiating text and binary is an artefact of the process that got us to
> this point, nothing more.  It provides no significant utility.
>
> --Martin

---

From: Stephen Williams <[email-masked]>
Date: Mon, 15 Aug 2011 20:12:22 -0700

On 8/15/11 8:03 PM, David Endicott wrote:
>
>
> On Mon, Aug 15, 2011 at 9:00 PM, Thomson, Martin <[email-masked] <mailto:[email-masked]>> wrote:
>
>     The defining characteristic of thewebsocketprotocol is it's deployment model, which provides a perfect channel for this sort
>     of coordination.  The server is effectively talking to itself.  Thus, no need for any form of negotiation...the server is no
>     longer interacting with the browser, it's interacting with itself.
>
>
> Please clarify.   If you're saying what I think you're saying, I'm must disagree.
>
> I would say the defining characteristics (there are several) of WS is that it is an asynchronous framing transport protocol 
> intended to be used primarily by browser client-agents.
>
> I don't understand what you mean by "server is effectively talking to itself" - that seems nonsensical.

I took that to mean: The typical use case for WS will be for a web server app that serves a Javascript-enabled web page to a 
browser.  Since the Javascript code is coming from the server it will be talking to, it shouldn't typically be very hard to keep 
them in sync.  This is opposed to use as a Web API for general applications that might involve code from multiple unrelated 
parties.  Because this may be the first widely available web (server/protocol/security model) protocol for applications, it would be 
good to be friendly to the JS page and general cases.

>
> Further to Bjoern's comments about, what I believe he's saying about including content-type information in the protocol: I would 
> point out that WS is content agnostic - the payload of the frame is not examined or considered by WS.   This is because WS is a 
> transport protocol.
>
> TCP does not care what goes thru it, be it HTTP, SMTP, SSH or whatever, it's job is only to transport the bytes to the endpoint. 
>  Similarly, WS does not care what is in it's frames.    It is the application using WS that imposes the concept of content-type on 
> the data communicated.
>
> The text frame hack is simply a nod to common use convenience.   (And an unnecessary one at that, IMHO)

A flag to indicate something like "Content-Type: application/UTF-8" is OK.  But it shouldn't be necessary.

>
>
>
>
>     If you deploy a page with some JS that talks back to your server using XmlHTTPRequest, do you demand standardization or
>     negotiation of the JSON encoding [1]?  I'll bet that most JS doesn't even check Content-Type headers before calling
>     JSON.parse.  This is no different.
>

sdw

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 16 Aug 2011 11:35:07 +0800

> itself" - that seems nonsensical.

It's how thewebsocketprotocol works.  In the most hand-wavey fashion: the server deploys some of its code.  The server exchanges messages with that code over thewebsocketprotocol.
 
> an unnecessary one at that, IMHO)

I agree.

--Martin

---

From: David Endicott <[email-masked]>
Date: Tue, 16 Aug 2011 10:35:34 -0400

> I took that to mean: The typical use case for WS will be for a web server
> app that serves a Javascript-enabled web page to a browser.  Since the
> Javascript code is coming from the server it will be talking to, it
> shouldn't typically be very hard to keep them in sync.  This is opposed to
> use as a Web API for general applications that might involve code from
> multiple unrelated parties.  Because this may be the first widely available
> web (server/protocol/security model) protocol for applications, it would be

Yep, I completely missed the point.    I agree with your explanation.    The
server extrudes part of the code onto the browser.   A rendering front-end.
   This will be a common use case.

Another use case I suspect will be equally common is where Websockets
becomes a conduit for messages.    The client-agents will process these
messages in their own fashion.    The server then becomes a resource api
(ie. service).

> But it shouldn't be necessary.

I remain firmly in the position that there should be NO content type
specification of any time.   I believe strongly that WS should be (is) a
transport protocol and must by content agnostic.

I'm willing to trade a little purity for a little convenience, but I think
it would have been better handled at the end-points (WS clients/servers)
rather than as part of the protocol.

I would drop the text/binary distinction, have all frames be opaque (and
presumably "binary") and expect the WS using applications to know/determine
content type on their own.

---

From: Bruce Atherton <[email-masked]>
Date: Tue, 16 Aug 2011 10:50:54 -0700

> on top of another transport protocol (e.g. TCP).

Agreed. However, it does provide the ability to specify an application 
protocol through the subprotocol header. To my mind, this is where any 
distinction between a binary or text message or stream belongs: on the 
connection itself, not on individual frame/message opcodes.

Having said that, I don't think this nit is large enough to hold up 
getting the protocol out the door, warts and all.