Subject: [hybi] Binary/Text Distinction (Hybi Volume 30, Number 63, et al)

From: "Bob Gezelter" <[email-masked]>
Date: Mon, 22 Aug 2011 04:01:48 -0700

@Steve:

UTF-8 text != UTF-8 binary (actually, "UTF-8" binary is a misnomer).

This is likely redundant for many, but UTF-8 is identical to text, so
long as the text is composed of the 128 characters in the USASCII 7-bit
code. For 8-bit characters with a decimal value exceeding 127, the
result is a multiple byte encoding as defined by RFC 3629. Wikipedia has
a good quick reference to UTF-8 (with chart or illegal characters) at
http://en.wikipedia.org/wiki/Utf-8.

The strength of UTF-8 is that it permits backwards compatibility with
USASCII-7 while providing encoding of non-ASCII characters. This is why
it is becoming a IETF standard. However, it is not good for representing
binary data that has not been first encoded into a printable subset
(e.g., Base 64, MIME).

As one who has often been involved in transferring large volumes of
data, requiring MIME'ification represents a significant loss of
effective bandwidth.

(Everyone)

I have long ago stated, and my preference remains for the WebSocket
protocol to be defined as "UTF-8 for control messages (per IETF
convention) and "8-bit binary transparent" for the actual data channel.
The case of an application that wants to send some messages in UTF-8 and
some in binary is to me, not a compelling case for support within the
WebSocket protocol. It is an appropriate provision in an applications
level protocol that uses the WebSocket protocol as its underlying
transport.

The argument that "JavaScript does not support binary" is, IMHO, without
significant merit. Typed arrays are used in other HTML5 features (e.g.,
WebGL) and are supported by current versions of browsers (see
https://developer.mozilla.org/en/JavaScript_typed_arrays under
subheading "Specification"). 

It has been mentioned that "most applications will not need to exchange
anything but text messages". This is an unproven (and unprovable)
assertion. It is also true that what is true in the aggregate will most
likely not apply to individual application implementers. 

A small mezzanine protocol (layered on top of the WebSockets API, using
the WebSocket protocol for transport) can easily serve those
implementers that wish to have a text-only connection. It is thus
unnecessary to place that support within the WebSocket protocol.

That I am suggesting a change in the WebSockets API is unclear. There is
already a need to distinguish Text from Binary at the level of the API,
I merely suggest that the distinction be handled somewhat differently
than is currently envisioned.

- Bob Gezelter, http://www.rlgsc.com

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 13:20:42 +0200

2011/8/22 Bob Gezelter <[email-masked]>:
> significant merit.

It seems a joke for a protocol specification.
There is life out of HTTP and JavaScript.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 22 Aug 2011 04:38:58 -0700

Bob,

> It has been mentioned that "most applications will not need to exchange anything but text messages". This is an unproven (and unprovable) assertion. It is also true that what is true in the aggregate will most likely not apply to individual application implementers. 

It's indeed pointless to talk about proofs in this context, since the contrary (that most applications will want to only exchange binary messages) is also unprovable.

Here is my view:
 
Assumption 1: Most WS clients will be browsers.
Assumption 2: Most (JavaScript) WS apps running in the browser will exchange small, JSON encoded messages.

JSON is UTF-8-based.

IMHO we should avoid anything which makes that use case less workable/convenient.

You can of course question those assumptions ..

>A small mezzanine protocol (layered on top of the WebSockets API, using the WebSocket protocol for transport) can easily serve those implementers that wish to have a text-only connection. It is thus unnecessary to place that support within the WebSocket protocol.
>That I am suggesting a change in the WebSockets API is unclear. There is already a need to distinguish Text from Binary at the level of the API, I merely suggest that the distinction be handled somewhat differently than is currently envisioned.

It works today .. it is good enough .. why change?

There is a different issue (unrelated to WS protocol .. see my other post): 

There is upcoming support for typed array (ArrayBuffer and typed array views over that) in JS.

How is that transported in JSON payload and made transparent in the JSON.stringify/parse marshaling/demarshaling API?

Tobias

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 14:49:16 +0200

> IMHO we should avoid anything which makes that use case less workable/convenient.

This must be a joke. A protocol specification design based on
currently programming habits (mostly by developers with no more
knowledge other than a few JavaScript and HTML). Great.

Have you considered that many smartphone apps could/would use WS
protocol as a standard mechanism to communicate with the servers? will
you expect that smartphones apps are mainly coded in JavaScript and
they mostly use JSON for "everything"???

Let me say that assuming JavaScript and JSON "just because you expect
that" as a key for a protocol design seems really amateur.

> It works today .. it is good enough .. why change?

"What" is working today? WebSockets?

If a WS protocol has been negotiated during the WS handshake, both the
client and the server *alreay* know the kind of data they can send via
WS messages/frames (as such data must conform to the negotiated WS
protocol). Why should such frames indicate "binary" or "text"? it's
redundant information and provides NOTHING.

It also leaves the door open for stupic issues like:

- If I split a WS UTF-8 message in two frames, should each frame
contain pure UTF-8 data? or can I split a multibyte UTF-8 symbol in
both frames (so the data in each frame would be non valid UTF-8 text)?

- Should the receiver check the encoding of the received message to
ensure it's UTF-8 before sending the message to the application layer?
WHY?

> There is upcoming support for typed array (ArrayBuffer and typed array views over that) in JS.

I don't care JS. There is more life out of JavaScript and HTML.

> How is that transported in JSON payload and made transparent in the JSON.stringify/parse marshaling/demarshaling API?

Nobody making WS spec should care that.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 22 Aug 2011 05:59:14 -0700

>- If I split a WS UTF-8 message in two frames, should each frame contain pure UTF-8 data? or can I split a multibyte UTF-8 symbol in both frames (so the data in each frame would be non valid UTF-8 text)?

A frame from a valid, fragmented UTF-8 message can end (or start, consequently) within the UTF-8 encoding of a single Unicode code point.
In that case the frame by itself is only valid UTF-8 up to that last code point (or from the first complete encoded code point).

>
>- Should the receiver check the encoding of the received message to ensure it's UTF-8 before sending the message to the application layer?
>WHY?

Yes, it must do that. From the spec:

"""
8.1.  Handling Errors in UTF-8 Encoded Data

   When an endpoint is to interpret a byte stream as UTF-8 but finds
   that the byte stream is not in fact a valid UTF-8 stream, that
   endpoint MUST _Fail the WebSocket Connection_.
"""

The spec does not say whether this check must be done as soon as possible.

A strict implementation will fail as soon as possible.

When a frame end on an Unicode code point, that check can be done per frame.

When not, it can be done at least when receiving the first bytes of the follow up frame.

For UTF-8 any Unicode code point has an encoding of length <= 4.

So this is the "look ahead" you need to check UTF-8.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:16:55 +0200

> In that case the frame by itself is only valid UTF-8 up to that last code point (or from the first complete encoded code point).

Why should my WS *frame* parser be Unicode aware? it's insane. WS
frame is not a text protocol (but a binary protocol). When retrieving
the frame payload data I should just care about reading N bytes,
rather than inspecting such data with an Unicode interpreter to check
"invalid UTF-8 encoding".

Apart from that, your assertion is not present in the draft, is it?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Philipp Serafin <[email-masked]>
Date: Mon, 22 Aug 2011 15:19:54 +0200

> Apart from that, your assertion is not present in the draft, is it?
If WS frames *can* end inside a code pojnt, like he said, wouldn't that
mean your frame parser does *not* have to care? (Which sounds like the
reasoneable way to define it, IMO.)

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:28:49 +0200

> reasoneable way to define it, IMO.)

Of course. My frame parser is just byte-aware and I just read N bytes
(payload_length) from the frame payload data. This is how it MUST be
IMHO.

Unfortunatelly (and this is what I was trying to say in these mails)
the WS draft seems to state that frames (with opcode=1) MUST contain
valid UTF-8 data (not just the whole message, but each frame payload
data). So if an UTF-8 multibyte symbol gets splited within two
consecutive frames, each frame has invalid UTF-8 data. Is it an error
then? or not?

I know that Tobias says it's not an error, but honestly I don't find
such assertion in the WS draft.

PLEASE: Change the spec to say: "the *whole* message (after joining
all the frames payload) MUST be UTF-8 in case the first frame has
opcode=1."

But even better: just remove the opcode "1" and "2" values and let
just "1" (application data). The previously negotiated WS protocol
*already* defines the kind o data being carried within WS messages. WS
framing layer should not care about payload data encoding, not at all.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Mon, 22 Aug 2011 09:34:10 -0400

> (payload_length) from the frame payload data. This is how it MUST be

You can certainly do that, but then you may be doing extra buffering if you
need to convert text to UTF16/etc -- instead you could convert each frame as
it comes in and maintain state for the incomplete character at the end of a
frame.

> such assertion in the WS draft.

This has been discussed before -- framing can occur at arbitrary byte
boundaries in the payload, which necessarily means that each frame does not
contain only complete UTF8 characters.

> all the frames payload) MUST be UTF-8 in case the first frame has

Can you make a specific suggestion of what you think should be changed?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 22 Aug 2011 06:38:02 -0700

>If WS frames *can* end inside a code pojnt, like he said, wouldn't that mean your frame parser does *not* have to care? (Which sounds like the reasoneable way to define it, IMO.)

http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#page-45

it talks about "byte stream"

further:
- the spec nowhere forbids breaking up a text message on non-UTF-8 code point boundaries
- intermediaries can fragment as they like - again nowhere is written they'd need to honour boundaries of UTF-8 encoded Unicode code points

the spec does not make explicit everything that is legal, often only whats illegal. It would be daunting if the former was done. (You may send
a binary message where the first byte is x00 or x01 or x02 ... -- seems silly)

also: with a look-ahead of 4 bytes you can check for invalid UTF-8 byte sequences in a streaming fashion. You don't need the whole message.

==

anyway: I didn't write the spec .. just my interpretation .. probably the authors/editors or someone else can speak out

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:42:05 +0200

> Can you make a specific suggestion of what you think should be changed?

Yes:

---------------------------------------
4.6.  Data Frames

   Data frames (e.g. non-control frames) are identified by opcodes where
   the most significant bit of the opcode is 0.  Currently defined
   opcodes for data frames include 0x1 (Text), 0x2 (Binary).  Opcodes
   0x3-0x7 are reserved for further non-control frames yet to be
   defined.

   Data frames carry application-layer or extension-layer data.  The
   opcode determines the interpretation of the data:

   Text

      The payload data is text data encoded as UTF-8.
-------------------------------------

That section is clearly speaking about payload data in a frame (a not
about the whole message), and it states:

  "The payload data is text data encoded as UTF-8."

This seems to say that the payload data within each frame MUST be a
completely valid UTF-8 string (which is not true in the case I
explained: a multibyte UTF-8 symbol splited in the end of a frame).

So your previous assertion:

  "This has been discussed before -- framing can occur at arbitrary byte
   boundaries in the payload, which necessarily means that each frame
   does not contain only complete UTF8 characters."

does not fit well with the draft text.

It could be fixed by replacing:

   Text

      The payload data is text data encoded as UTF-8.

with:

   Text

      The whole message (after reassembling all the payload data in
the frames) is text data encoded as UTF-8.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:43:32 +0200

> the frames) is text data encoded as UTF-8.

Also:

7.4.1.  Defined Status Codes

----------------------------
   1007

      1007 indicates that an endpoint is terminating the connection
      because it has received data that was supposed to be UTF-8 (such
      as in a text frame) that was in fact not valid UTF-8 [RFC3629].
------------------------------

better:

----------------------------
   1007

      1007 indicates that an endpoint is terminating the connection
      because it has received data that was supposed to be UTF-8 (such
      as in a text message) that was in fact not valid UTF-8 [RFC3629].
------------------------------

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 15:48:38 +0200

> http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#page-45

It says:

> it talks about "byte stream"

It does not specify nothing about "inspecting a frame payload data" or
a "whole message". Nothing.

> - the spec nowhere forbids breaking up a text message on non-UTF-8 code point boundaries

Please read my two previous mails in which I point to sections
forbiding it. Specially:

  Text

> - intermediaries can fragment as they like - again nowhere is written they'd need to honour boundaries of UTF-8 encoded Unicode code points

Sorry, but what the f*** is an intermediary? All the people here talks
about "intermediaries" and nobody seems to explain "what it is".

> anyway: I didn't write the spec .. just my interpretation .. probably the authors/editors or someone else can speak out

Don't take me wrong but, what you are doing is looking for any
argument in order to justify the draft text. Isn't easier just to
improve it and make better? :)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Mon, 22 Aug 2011 16:15:35 +0200

>
>
>
> > - intermediaries can fragment as they like - again nowhere is written
> they'd need to honour boundaries of UTF-8 encoded Unicode code points
>

I assume that unless clearly stated for a given scenario, it would mean
"whatever may come in between two endpoints". In most cases, it doesn't seem
to matter much what they are or what they do other than the fact they can
interact with the frames. In fact I would consider best practices, among
intermediary vendors, will arise when WS becomes common. I wouldn't expect
the current spec to specify how they interact with WS.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 22 Aug 2011 16:50:48 +0200

> the current spec to specify how they interact with WS.

Ok, so nobody here wants to waste time explaining what a WS intermediary is.

I imagine a simple HTTP proxy which forwards the WS GET handshake and
101 response, and then just forward received bytes (from client or
server) being no WS aware at all, so it knows nothing about WS frames.

Or it could be a real WS server which *performs* the handshake with
the WS client, and later performs a new WS handshake with a remote WS
server, then it receives frames and messages and forwards them
(keeping the received frames or refactoring them) to the remote
server.

Or it could be a pseudo WS server which just forwards the initial GET
handshake to a remote server, but later it inspects the WS frames and
messages and, optionally, re-assembles them into new WS frames which
are forwarded to the remote server.

Ok, don't worry. I do know that nobody here will explain it.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Philipp Serafin <[email-masked]>
Date: Mon, 22 Aug 2011 19:23:51 +0200

> framing layer should not care about payload data encoding, not at all.

I wasnt aware that the text of the spec is like this, my apologies. I
agree, it should be changed.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 23 Aug 2011 02:47:57 +0900

> > all the frames payload) MUST be UTF-8 in case the first frame has

Also discussed here.
http://www.ietf.org/mail-archive/web/hybi/current/msg07811.html

I agree that this should be clarified in the spec. As I said in the thread,
I prefer allowing fragmentation in the middle of UTF-8 sequence.

---

From: Stephen Williams <[email-masked]>
Date: Mon, 22 Aug 2011 11:01:48 -0700

> UTF-8 text != UTF-8 binary (actually, "UTF-8" binary is a misnomer).

UTF-8 text, encoded as UTF-8 needs no further encoding.  Assuming you don't use BOM and leave no other options, like UTF-16 etc., 
then UTF-8 sent as "text" is indistinguishable from UTF-8 sent with "binary" "encoding".

So, UTF-8 text == UTF-8 binary.  Or did you mean something else?  Obviously UTF-8 text/binary != binary.

UTF-8 with "binary" "encoding" (i.e. simply sending UTF-8 as bytes) is not UTF-8 plus any random byte.  I wasn't suggesting that and 
I didn't see anyone else saying that either.

In the past, pre-UTF-8, a "text" mode (especially with IETF protocols) implied 7-bit, CRLF (with conversion to other line endings 
when necessary), line length limits often, etc.  "Clean" UTF-8, web style, has none of these limitations or conversions.  Therefore, 
when transmitted, it is treated as a simple sequence of bytes.

Of course, when created or consumed, it is often converted to 2 byte (Java especially, some C++ and most others) or 4 byte (some 
C++) characters internally.  But this is hidden from the protocol, and mostly hidden through language libraries.  API idioms are now 
common to allow this transformation without making application code handle details most of the time.

> USASCII-7 while providing encoding of non-ASCII characters. This is why

Except line endings are more liberal for UTF-8 than USASCII-7, at least in the sense that all UTF-8 receivers generally handle any 
combination of line endings, where it was common for ASCII processors to assume one of the 3 dominant styles.

While I understand, and mostly agree with IETF's affection with ASCII and CRLF, the inefficiency was always a little annoying.

> data, requiring MIME'ification represents a significant loss of

I agree.

> level protocol that uses the WebSocket protocol as its underlying

This was my point.

> - Bob Gezelter, http://www.rlgsc.com

sdw
-- 
Stephen D. Williams [email-masked] [email-masked]