Subject: Re: [hybi] Binary/Text Distinction

From: "Bob Gezelter" <[email-masked]>
Date: Mon, 22 Aug 2011 06:19:30 -0700

Tobias,

I have not looked into how JSON parse/marshaling is handling the typed
arrays, particularly the question of uniformity across different
browsers (or different versions of the same browser). However, IMHO,
that is at best a browser-related WebSockets API question, there remains
no need to push that encoding/decoding issue into the WebSocket protocol
itself.

As I have noted in the past, one of the few places that such a dichotomy
is associated with an IETF protocol is FTP. The reason for this is
historical, and still relevant in cross-platform contexts. When text
files are transmitted between hosts, they are subject to transformations
designed to "preserve" the textual content (e.g., character codes,
formatting, line breaks) across the transmission. By contrast, binary
files are transmitted without change. 

The transformations of "text" mode, while often useful, have been the
source of innumerable accidents and problems since their advent. They
were, and are necessary, but they should be viewed with caution.

In the case of the WebSocket protocol, there is little to be gained by
the text/binary dichotomy (as I noted previously, this is
straightforward to implement as an "applications protocol"/understanding
at the level of the WebSocket API).

- Bob Gezelter, http://www.rlgsc.com

Tobias Oberstein wrote:

>Bob,

>> It has been mentioned that "most applications will not need to exchange anything but text messages". 
>>This is an unproven (and unprovable) assertion. It is also true that what is true in the aggregate 
>>will most likely not apply to individual application implementers.
>
>It's indeed pointless to talk about proofs in this context, since the contrary (that most applications will >want to only exchange binary messages) is also unprovable.
>
>Here is my view:
>
>Assumption 1: Most WS clients will be browsers.
>Assumption 2: Most (JavaScript) WS apps running in the browser will exchange small, JSON encoded messages.
>
>JSON is UTF-8-based.
>
>IMHO we should avoid anything which makes that use case less workable/convenient.
>
>You can of course question those assumptions ..
>
>>A small mezzanine protocol (layered on top of the WebSockets API, using the WebSocket protocol 
>>for transport) can easily serve those implementers that wish to have a text-only connection. 
>>It is thus unnecessary to place that support within the WebSocket protocol.
>>
>>That I am suggesting a change in the WebSockets API is unclear. There is already a need to 
>>distinguish Text from Binary at the level of the API, I merely suggest that the distinction be 
>>handled somewhat differently than is currently envisioned.
>
>It works today .. it is good enough .. why change?
>
>There is a different issue (unrelated to WS protocol .. see my other post):
>
>There is upcoming support for typed array (ArrayBuffer and typed array views over that) in JS.
>
>How is that transported in JSON payload and made transparent in the JSON.stringify/parse 
>marshaling/demarshaling API?
>
>Tobias

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:32:23 +0200

2011/8/22 Bob Gezelter <[email-masked]>:
> source of innumerable accidents and problems since their advent.

So ***great*** real case.

WS authors, please, learn from the History (or you will repeat it).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Mon, 22 Aug 2011 09:35:32 -0400

> WS authors, please, learn from the History (or you will repeat it).

I don't think this proves what you think it does -- in fact, to me it argues
for exactly what we have now, which is defining exactly how text messages
are transported and represented, rather than leaving it up to each
applcation.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 22 Aug 2011 07:16:38 -0700

>I have not looked into how JSON parse/marshaling is handling the typed arrays, particularly the question of uniformity across different browsers (or different versions of the same browser). However, IMHO, that is at best a browser-related WebSockets API question, >there remains no need to push that encoding/decoding issue into the WebSocket protocol itself.

This wasn't a suggestion to discuss that issue on this list. It was a wink to take the bigger picture into account.

>
>As I have noted in the past, one of the few places that such a dichotomy is associated with an IETF protocol is FTP. The reason for this is historical, and still relevant in cross-platform contexts. When text files are transmitted between hosts, they are subject to >transformations designed to "preserve" the textual content (e.g., character codes, formatting, line breaks) across the transmission. By contrast, binary files are transmitted without change. 
>
>The transformations of "text" mode, while often useful, have been the source of innumerable accidents and problems since their advent. They were, and are necessary, but they should be viewed with caution.

IMHO, these are issues from the past .. luckily today, we have Unicode and UTF-8

>
>In the case of the WebSocket protocol, there is little to be gained by the text/binary dichotomy (as I noted previously, this is straightforward to implement as an "applications protocol"/understanding at the level of the WebSocket API).

The "gain" is supposed to be for one of the most important use cases:

client = browser / message = JSON / JSON = UTF-8

Honestly, I see WS as a _pragmatic_ effort to bring the web to the next level.

If it helps with that goal to have text/binary payload type distinction in the WS protocol, I am fine with that,
even if that is somehow unclean from an idealistic protocol design view (to which I agree).

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 16:43:57 +0200

> IMHO, these are issues from the past .. luckily today, we have Unicode and UTF-8

Upload/download a binary file (i.e. any custom app document) using FTP
in "binary" / "text" mode and you will get different results (for
example, a corrupted document). This issue still occurs today using
FTP clients which "magically" auto-choose the binary or text transfer.

> client = browser / message = JSON / JSON = UTF-8

Thanks for ignoring my previous comment so I must write them again:

or

  client = smartphone app / message = WHATEVER / WHATEVER = WHICHEVER_ENCODING

> Honestly, I see WS as a _pragmatic_ effort to bring the web to the next level.

That's your error, you just see "web" here. WebSocket is not a "web"
protocol, it's a network protocol.

> even if that is somehow unclean from an idealistic protocol design view (to which I agree).

The main question here is: WHY do you state that any other design but
the current ugly one would be worse or more difficult for WS
implementation?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Mon, 22 Aug 2011 11:02:09 -0400

> protocol, it's a network protocol.

What exactly do you think motivated the design of WebSockets?  For
applications not running in a browser where potentially hostile code can
execute, why isn't raw TCP or some existing protocol running on top of it
sufficient?  The reason WebSockets exists is to allow browsers to use it,
hence "Web" in the name.  Sure, it can be used for other things and they may
want to in the interest of interoperability, but that isn't the reason it
exists.

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 22 Aug 2011 08:02:23 -0700

> Upload/download a binary file (i.e. any custom app document) using FTP in
> "binary" / "text" mode and you will get different results (for example, a
> corrupted document). This issue still occurs today using FTP clients which
> "magically" auto-choose the binary or text transfer.

The point is: with Unicode/UTF-8 least denominator for text in transport,
there is no need for conversion, and thus no conversion problem.

I'm not talking about binary, which should not "converted" to UTF-8
obviously.

> >>In the case of the WebSocket protocol, there is little to be gained by the
> text/binary dichotomy (as I noted previously, this is straightforward to
> implement as an "applications protocol"/understanding at the level of the
> WebSocket API).
> >

That's fine. Use payload type binary and there you go.

> > If it helps with that goal to have text/binary payload type
> > distinction in the WS protocol, I am fine with that, even if that is somehow
> unclean from an idealistic protocol design view (to which I agree).
> 
> The main question here is: WHY do you state that any other design but the
> current ugly one would be worse or more difficult for WS implementation?

I did not say that.

If WS was an _academic_ endeavour I would be in the cleanness camp.

Let me put it like this: I don't think that moving the payload type out of opcodes
into say subprotocol negotiation is an improvement in terms of protocol design
cleanness big enough to outweigh the delay of WS widespread adoption and
set-back that such (yet another) fundamental change to WS would bring.

Honestly, we should all work together  to "just make it happen". There are
"other" guys who would be more than happy to see WS sink in endless
debates and never hit the road. They would say: "see, we told you - they
change stuff again" .. "not ready yet" .. "half-baked".

Is it really worth that?

---

From: Scott Ferguson <[email-masked]>
Date: Mon, 22 Aug 2011 09:56:14 -0700

On 08/22/2011 06:35 AM, John Tamplin wrote:
> On Mon, Aug 22, 2011 at 9:32 AM, Iñaki Baz Castillo <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     2011/8/22 Bob Gezelter <[email-masked]
>     <mailto:[email-masked]>>:
> I don't think this proves what you think it does -- in fact, to me it 
> argues for exactly what we have now, which is defining exactly how 
> text messages are transported and represented, rather than leaving it 
> up to each applcation.

+1

Text encoding complications are from underspecification or 
overgeneralization. WebSockets solves that problem upfront. Aesthetic 
arguments of "proper" layering aside (I like van Gogh, by the way), in 
actual practice, specifying One True Encoding is a great help to actual 
implementors.

Leaving it to the application? Contrast Java and C where Java made text 
a first-class primitive, while C left text up to the application. I'll 
just leave it at mb*.

   XML - parsers have to support every encoding despite UTF-8 being 
sufficient. (Plus the encoding is selected after parsing several text 
characters in a default encoding and added BOM complications.)

   CORBA - you really don't want to know their text encoding system. The 
horror.

   PHP - utter nightmare trying to backpatch unicode into a system 
designed around 8-bit strings.

   HTTP - URLs and forms underspecified in their encodings. text/html 
can be any encoding. Painful.

   Servlet/JSP - problems descended from HTTP underspecification, but 
eventually defaulted to latin-1 (!)

   SOAP - the inverse problem, not properly dealing with binary encoding 
upfront. Painfully dealt with many years and revisions later.

The counter example is FTP? That "text" mode isn't even modern unicode 
text. FTP "text" is an entirely different (and obsolete and broken) 
concept. It's irrelevant.

-- Scott

---

From: Brian <[email-masked]>
Date: Mon, 22 Aug 2011 22:19:59 -0700

> even if that is somehow unclean from an idealistic protocol design view (to

Yes.  This is *exactly* the purpose of WebSockets.  And having the type
distinction will be of *extremely* high value to JavaScript developers.
 Existing non-browser applications have TCP at their disposal directly and
can use it how they see fit.  The entire purpose for the WebSockets protocol
from the beginning was explicitly to allow *browsers* to use a real-time
bidirectional socket connection in a JavaScript environment.  That was and
is the *whole point* of this protocol.  Period.  End of story.  Other
non-browser applications can and will find good uses for this protocol, but
that's all gravy.  This WG has gone out of its way to make the protocol
flexible enough to allow for such uses, and for future enhancements in
browser capabilities, to the benefit of everyone.  Adding binary support in
the first place is a good example of this -- recall that draft-00 is a
text-only protocol!  But I tell you that removing text frames will not be to
the benefit of anyone.

It would be way beyond absurd to put utf-8 parsing into the application
layer of a JavaScript app.  It would only be barely tolerable if the
browsers started providing a native function that could internally convert
UTF-8 out of a slice of binary data - such a function does not yet exist -
but would cause JavaScript developers to have to think about how to work
with binary data.

The original goal of the WebSocket protocol was to be dead simple for
inexperienced implementers to trivially build WebSocket servers.  We've (I
think rightly) deviated some from that goal, but it is still crucial to make
it brain-dead simple for an inexperienced client-side developer to use the
JavaScript API.  Forcing these developers to think about parsing binary data
structures is setting the bar too high.  Working with binary formats is old
hat to many of us here, but remember that there are a lot of programmers out
there who have effectively no knowledge of binary data representations and
may never need to throughout their entire career.  Especially programmers
doing front-end web development.

Given its intended purpose, I think the WebSocket protocol rightly straddles
the line between a transport protocol and an application protocol.  In some
ways, it's an application protocol which is intended to have yet another
application protocol (the subprotocol) layered on top of it, but its primary
function is to transport data.  I don't think it can or should commit to
being on only one of those conceptual layers.

For someone already implementing the WebSocket protocol and its binary
frames, it also certainly isn't any more difficult to add support for the
text frame data type.  I really am actually rather flabbergasted that anyone
is even entertaining the idea of removing the text frames.

Brian McKelvey
https://github.com/Worlize/WebSocket-Node

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 23 Aug 2011 12:09:06 +0200

> distinction will be of *extremely* high value to JavaScript developers.

Is JavaScript the only scripting technology for a web page? Will it be
the only one?

> can use it how they see fit.

That's wrong IMHO. TCP is not a friendly protocol for high-level
developers since TCP is not message boundary. There are lot of buggy
programs implementing a protocol on top of TCP because the author
expected that a received TCP package must contain an entire
application level message (and no more).

HTTP is a good example. Even if HTTP/1.1 allows connection reuse and
the hyper-ugly-and-bad-designed pipeline feature, nobody uses it (but
just experimented and optimized web browsers). Nobody making an HTTP
library for language XXXX implements HTTP connection reuse or
pipeline. Lazy developers assume that each application request uses
single TCP connection which must be closed after getting the response.
Do you imagine a smartphone application using an HTTP library and
opening a new TCP connection for each "message"?

WebSocket could help here. If there appear (and there are) WebSocket
libraries for any language, a developer could use them to build an
application running out of a webbrowser (i.e. an smartphone). The
developer gets, by free, a message boundary protocol (simple API) and
a standarized protocol which, probably, will be implemented by many
services offering content not just via a web browser. Using raw TCP is
not the solution, as currently people prefers to build an ugly
HTTP-polling solution rather than dealing with TCP.

So, assuming that WebSocket must be designed just considering
JavaScript is a very very wrong asumption IMHO.

> The entire purpose for the WebSockets protocol
> is the *whole point* of this protocol.  Period.  End of story.

There are a lot of smartphone applications which "imitate" the web
browser interface (but they are not html browsers, but phone native
applications). Examples: Facebook chat, Twitter messenger, and so on.

Am I really the only one assuming that WebSocket is the perfect
protocol for those NON-web-browsers applications (think in smartphones
please) which currently use HTTP for communicating with the server?

> but would cause JavaScript developers to have to think about how to work

That's a good argument.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Tue, 23 Aug 2011 03:20:48 -0700

> >  Existing non-browser applications have TCP at their disposal directly
> > and can use it how they see fit.
> 
> That's wrong IMHO. TCP is not a friendly protocol for high-level developers
> since TCP is not message boundary. There are lot of buggy programs

use SCTP .. it has messages and multiplexing

> implementing a protocol on top of TCP because the author expected that a
> received TCP package must contain an entire application level message (and
> no more).

I assume you mean "TCP packet" .. but even then, there are only TCP segments
and IP packets.

If you want framing on TCP, use something that even was triggered by your
"security concerns"
 
http://cr.yp.to/proto/netstrings.txt

it's trivial, sufficient for framing (only) and available since 1997.

> So, assuming that WebSocket must be designed just considering JavaScript is
> a very very wrong asumption IMHO.

Who said "only"?

> > The entire purpose for the WebSockets protocol from the beginning was
> > explicitly to allow *browsers* to use a real-time bidirectional socket
> > connection in a JavaScript environment.  That was and is the *whole
> > point* of this protocol.  Period.  End of story.
> 
> There are a lot of smartphone applications which "imitate" the web browser
> interface (but they are not html browsers, but phone native applications).
> Examples: Facebook chat, Twitter messenger, and so on.
> 
> Am I really the only one assuming that WebSocket is the perfect protocol for
> those NON-web-browsers applications (think in smartphones

What serialization format will probably be the most widely used for JS _and_
"native" clients alike? If I had to make a bet, it would be JSON. You can
have your HTML5 frontend _and_ your native frontend talk to your
servers over the same transport/serialization.

Again: _no_ requirement to use JSON though. Using Thrift on ProtoBuffers
with WS is just as fine ..

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 23 Aug 2011 12:40:30 +0200

> use SCTP .. it has messages and multiplexing

Yes, I know SCTP, and I also know that it's a transport protocol just
present in big telephony providers networks for carrying ISUP or SIP
protocols and intercommunicate with other providers. So I don't think
we should suggest "use SCTP" in this thread.

> I assume you mean "TCP packet" .. but even then, there are only TCP segments

My fault.

> http://cr.yp.to/proto/netstrings.txt

I was asking for nothing, neither SCTP neither a framing solution for
TCP. I was describing existing issues when a lazy developer tries to
implement a *already* given protocol (so forget netstings stuff) on
top of TCP. For example, there are some XMPP clients assuming that a
TCP packet must contain an entire XMPP message (and just one).

Extracted from the last mail of Brian in this thread:

> The entire purpose for the WebSockets protocol from the beginning was explicitly to allow *browsers* to use a real-time bidirectional socket connection in a JavaScript environment.

> servers over the same transport/serialization.

I don't care which serialization format "native" clients would choose
for use within WebSocket protocol. Why is that important? I just meant
that, IMHO, WS is a perfect choice for a "generic" application (out of
a web page) communicating with a server which usually servers web
pages.

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Brian <[email-masked]>
Date: Tue, 23 Aug 2011 04:21:39 -0700

On Tue, Aug 23, 2011 at 3:40 AM, Iñaki Baz Castillo <[email-masked]> wrote:
> connection in a JavaScript environment.
>

My point is not that non-browsers shouldn't use WebSocket.  Nor is it that
we shouldn't consider such use cases in the design of the protocol.  My
point was that since the browser use case was the impetus for creating this
protocol in the first place, we shouldn't accept changes that would be in
any way detrimental to the browser or JavaScript developers for the sake of
a secondary use case, or even for a clean separation of application vs
transport protocol concepts.  This WG has frequently considered and accepted
changes and improvements motivated by forward-thinking people who envision a
broader use for WebSockets than the browser, and we all benefit from that,
but only when it causes no harm to the JS/HTML5 use case.

Brian

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Tue, 23 Aug 2011 13:37:20 +0200

>
>
> Nobody making an HTTP library for language XXXX implements HTTP connection
> reuse or pipeline.

No idea where you get that from. There's a difference between implementing a
feature and using it. Pipelining has proven to be complex too use but most
servers and clients do implement it.
Keep-alive is actually quite used.

> Lazy developers assume that each application request uses

Is it the role of specification to handle lazy developers?

> HTTP-polling solution rather than dealing with TCP.

People use HTTP-polling from a browser because you don't have any other
alternative due to the nature of HTTP. Raw sockets aren't available from
JavaScript hence a middle ground which is WS.

> JavaScript is a very very wrong asumption IMHO.

I don't believe this is assumed but that's the historical background of the
protocol.

> please) which currently use HTTP for communicating with the server?

WebSocket is probably an interesting protocol on mobile devices indeed and
I, for one, already play with it in that context.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 23 Aug 2011 13:49:51 +0200

> but only when it causes no harm to the JS/HTML5 use case.

I don't want to suggest *nothing* that would make more complex to use
WS within a common JS application. But that's not mean that current
spec cannot be improved in order to be more "ellegant" and correct.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 23 Aug 2011 13:56:04 +0200

> Keep-alive is actually quite used.

Sorry, let me refactor my statement:

"Nobody using an HTTP library implementing HTTP connection reuse or
pipeline makes usage if such features".

> Is it the role of specification to handle lazy developers?

In other mails I've clearly read that WS is intented for
"just-frontend-web-developers" (along with more experienced
developers, of course).
So yes, one of the WS specification targets is to make WS usage easy
for web developers (just 5-6 JavaScript functions, no more).

> JavaScript hence a middle ground which is WS.

I was speaking about non-web-browsers apps which use HTTP rather than
a bidirectional protocol on top of TCP.

> I, for one, already play with it in that context.

:)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 23 Aug 2011 14:01:01 +0200 (CEST)

On Tue, 23 Aug 2011, Iñaki Baz Castillo wrote:

> "Nobody using an HTTP library implementing HTTP connection reuse or pipeline 
> makes usage if such features".

Having implemented and worked with HTTP libraries for many years, I strongly 
disagree with this statement. I believe you're just guessing here without 
having any facts at hand.

[and now I'll go back and lurk again]

-- 

  / daniel.haxx.se

---

From: Joel Martin <[email-masked]>
Date: Tue, 23 Aug 2011 17:34:28 -0500

On Tue, Aug 23, 2011 at 6:21 AM, Brian <[email-masked]> wrote:

> My point was that since the browser use case was the impetus for creating
> this protocol in the first place, we shouldn't accept changes that would be
> in any way detrimental to the browser or JavaScript developers for the sake
> of a secondary use case, or even for a clean separation of application vs
> transport protocol concepts.
>

+1. Secondary use cases can just use binary set to true for all frames and
be done with it. Unless they are pretending to be a browser, in which case
... well, they should pretend to be a browser (and support text too if
needed).

Earlier someone asked if there would be many use cases of sending/receiving
binary data over WebSockets from Javascript. Absolutely! But most of them
haven't even been thought of yet. That's the real potential of WebSocket
(opening new applications domains rather than making the existing ones
incrementally better). My own noVNC project send/receives binary data over
the existing Hixie Websocket implementations and I'm very much looking
forward to being able to send binary data directly without base64
encoding/decoding first (which adds non-trivial latency).

I think the arguments in favor of retaining the binary/text distinction in
the spec are much more compelling than the "history teaches us to keep
layers distinct" line of arguments. Even more compelling to me is not
delaying the protocol over a purity vs practical argument.

Regards,

Joel Martin (kanaka)