Subject: [hybi] API for binary data (was: HTTP Upgrade - A Modest Proposal)

From: Joel Martin <[email-masked]>
Date: Thu, 7 Apr 2011 15:05:22 -0500

On Thu, Apr 7, 2011 at 1:45 PM, Brian McKelvey <[email-masked]>wrote:

> With UTF-8 it only takes two bytes for values over 127.  And it doesn't
> take two bytes on the wire at all if you use a binary message frame type,
> which I thought was what we were talking about.  The browser would be
> responsible for converting it to an appropriately encoded string for binary
> processing.
>

Is anybody here involved with the API side (standard or browser
implementation) that can comment on what the API side of things will look
like for sending binary data? Will it be automatic based on data-type (i.e.
ArrayBuffer vs string) or some sort of change to the API? If it is
automatic, how will the program know before sending if the API is capable of
sending binary data? A link to discussion on a different list works for me.

And just for reference for those considering binary data before the API
officially supports it, I recommend base64 encode/decode rather than UTF-8
for a few reasons:

- Using base64 is more space efficient than UTF-8 conversion of byte data if
it evenly spread across 0-255 (133% for base64 vs 150% for UTF-8).

- The meaning of 0 (zero) is somewhat uncertain in UTF-8 implementations.
Some implementations treat 0 in UTF-8 as string termination. Flash is one
example, which is problematic if you want to use the Flash WebSockets
fallback/polyfill web-socket-js. I consider this brokenness in Flash but it
does make transferring binary data via UTF-8 difficult. The combination of
length and binary data type in the new protocol will address this (because
then in Flash binary data can just be treated as binary data and not decoded
as UTF-8). However, it's not useful until the API supports binary data.

- If you are using a python WebSockets server, encoding to/from base64 is
much easier (and I faster too IIRC) then the encode/decode to/from UTF-8.

And if talking to raw TCP servers via the browser is your thing, then check
out https://github.com/kanaka/websockify which is a python WebSockets to raw
TCP socket proxy (or LD_PRELOAD wraps an existing server) plus a Javascript
lib that talks to it. It only speaks v76 currenly but I'm planning on
updating it to support 06 soon (or whatever is current when I get around to
it).

Regards,

Joel Martin (kanaka)

---

From: Patrick McManus <[email-masked]>
Date: Thu, 07 Apr 2011 13:15:14 -0700

On 4/7/11 1:05 PM, Joel Martin wrote:
> implementation) that can comment on what the API side of things will 
> look like for sending binary data? Will it be automatic based on 
> data-type (i.e. ArrayBuffer vs string) or some sort of change to the 
> API? If it is automatic, how will the program know before sending if 
> the API is capable of sending binary data? A link to discussion on a 
> different list works for me.
>
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12102

---

From: Stephen Williams <[email-masked]>
Date: Thu, 07 Apr 2011 14:24:38 -0700

On 4/7/11 1:05 PM, Joel Martin wrote:
> On Thu, Apr 7, 2011 at 1:45 PM, Brian McKelvey <[email-masked] <mailto:[email-masked]>> wrote:
>
>     With UTF-8 it only takes two bytes for values over 127.  And it doesn't take two bytes on the wire at all if you use a
>     binary message frame type, which I thought was what we were talking about.  The browser would be responsible for
>     converting it to an appropriately encoded string for binary processing.
>
>
> Is anybody here involved with the API side (standard or browser implementation) that can comment on what the API side of 
> things will look like for sending binary data? Will it be automatic based on data-type (i.e. ArrayBuffer vs string) or some 
> sort of change to the API? If it is automatic, how will the program know before sending if the API is capable of sending 
> binary data? A link to discussion on a different list works for me.

Websockets should have payloads that are binary only, just like TCP.  If the application wants to consider the data to be a 
particular encoding, mime-type, etc., that is fine but it should be layered on a binary transport.  Both Post and Get support 
binary bodies already.

>
> And just for reference for those considering binary data before the API officially supports it, I recommend base64 
> encode/decode rather than UTF-8 for a few reasons:
>
> - Using base64 is more space efficient than UTF-8 conversion of byte data if it evenly spread across 0-255 (133% for base64 vs 
> 150% for UTF-8).
>
> - The meaning of 0 (zero) is somewhat uncertain in UTF-8 implementations. Some implementations treat 0 in UTF-8 as string 
> termination. Flash is one example, which is problematic if you want to use the Flash WebSockets fallback/polyfill 
> web-socket-js. I consider this brokenness in Flash but it does make transferring binary data via UTF-8 difficult. The 
> combination of length and binary data type in the new protocol will address this (because then in Flash binary data can just 
> be treated as binary data and not decoded as UTF-8). However, it's not useful until the API supports binary data.
>
> - If you are using a python WebSockets server, encoding to/from base64 is much easier (and I faster too IIRC) then the 
> encode/decode to/from UTF-8.

Many environments, including Java, use layers of objects to work with UTF-8.  Encoding and decoding UTF-8 can be done in about 
10 lines of code.  Doing it inline, or at least in a focused, tight loop, should be about as fast as b64.

>
> And if talking to raw TCP servers via the browser is your thing, then check out https://github.com/kanaka/websockify which is 
> a python WebSockets to raw TCP socket proxy (or LD_PRELOAD wraps an existing server) plus a Javascript lib that talks to 
> it. It only speaks v76 currenly but I'm planning on updating it to support 06 soon (or whatever is current when I get around 
> to it).
>
> Regards,
>
> Joel Martin (kanaka)

sdw