Subject: WiSH: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols (HTTP)

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 20 Oct 2016 16:22:44 +0900

Hi HTTP WG,

We'd like to propose a general purpose message framing format named WiSH as
a solution for supporting bi-directional message-based communication on the
Web in the era of HTTP/2, QUIC, Fetch API, Streams API, etc.

https://tools.ietf.org/html/draft-yoshino-wish-00

The background/rationale for this proposal is explained in the section 2.

This effort started as a survey of protocol gaps (the summary was posted to
HyBi WG in Jan 2015 [1]). Wenbo has presented the proposal briefly in the
HTTP workshop 2016, and after that we've been incubating it at GitHub [2]
with great input from some parties. Input from this WG (in the thread about
the WS2/h2 proposal [3]) are also incorporated.

Comments are appreciated.

Thanks

[1] https://mailarchive.ietf.org/arch/msg/hybi/YnXi4j6cJhwuP5HEZ0Y3hKCONv8
[2] https://github.com/bidiweb/wish
[3] https://lists.w3.org/Archives/Public/ietf-http-wg/2016OctDec/0042.html

---

From: Loïc Hoguin <[email-masked]>
Date: Thu, 20 Oct 2016 12:11:28 +0200

> Hi HTTP WG,

Hello,

> Comments are appreciated.

Great work. I have two general comments and three typos:

You write:

    The Content-Type header value of the underlying HTTP request/response
    message MUST be "application/webstream".

With the current draft this media type is not enough to indicate what 
the frames will actually contain. Websocket has sec-websocket-protocol 
for that purpose. SSE allows defining event types for each events. WiSH 
currently only has payload frames with no meaning beyond text or binary.

Maybe have the media type define an optional parameter "protocol", for 
example? Then clients can use content negotiation to request protocols 
they accept. For example:

Client sends:

   Accept: application/webstream;protocol=v12.stomp;q=1,
           application/webstream;protocol=mqtt;q=0.5,
           text/event-stream;q=0.1, */*

Server picks the appropriate type and replies:

   Content-type: application/webstream;protocol=v12.stomp

Using content negotiation doesn't require extra logic which is a big 
plus. And we can still use the existing Websocket subprotocol registry, 
which will be required if Websocket compatibility is a concern.

You write:

    The message type distinction by the opcode field (text and binary) is
    kept to allow better Web support.  One of the possible use cases is
    to use the text type for exchaning meta data encoded in JSON, etc.,
    and the binary type for exchanging non-meta data messages.

In practice when using JSON and where performance is a concern, I have 
seen users use binary for everything. The problem being that Websocket 
text frames must be valid UTF-8; but so does JSON. The server not 
providing functionality to disable the Websocket UTF-8 check (and JSON 
decoders not able to), users just dropped text frames entirely.

Which brings me to my question: do you think it could be worth adding a 
note to implementers that perhaps they should consider optionally 
disabling the UTF-8 check when JSON is expected for text frames?

Typos:

* Page 3: there are two "furthur" that probably should be "further".
* Page 5: there is "exchaning" instead of "exchanging".

Cheers,

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Van Catha <[email-masked]>
Date: Thu, 20 Oct 2016 14:50:32 -0400

This is great and should be backward compatible quite easily!

On Thu, Oct 20, 2016 at 6:11 AM, Loïc Hoguin <[email-masked]> wrote:

> With the current draft this media type is not enough to indicate what the
> frames will actually contain. Websocket has sec-websocket-protocol for that
> purpose. SSE allows defining event types for each events. WiSH currently
> only has payload frames with no meaning beyond text or binary.
>
> example? Then clients can use content negotiation to request protocols they
> accept. For example:
>
> Client sends:
>
> Using content negotiation doesn't require extra logic which is a big plus.
> And we can still use the existing Websocket subprotocol registry, which
> will be required if Websocket compatibility is a concern.
>
>
> You write:
>
> seen users use binary for everything. The problem being that Websocket text
> frames must be valid UTF-8; but so does JSON. The server not providing
> functionality to disable the Websocket UTF-8 check (and JSON decoders not
> able to), users just dropped text frames entirely.
>
> note to implementers that perhaps they should consider optionally disabling
> the UTF-8 check when JSON is expected for text frames?
>
>
> Typos:
>

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 21 Oct 2016 22:04:01 +0900

Thanks for the comments, Loïc.

On Thu, Oct 20, 2016 at 7:11 PM, Loïc Hoguin <[email-masked]> wrote:

<snip>

> will be required if Websocket compatibility is a concern.

Good point. I was thinking about using the media type suffix convention
like +json, +xml, etc. (RFC 6839, RFC 7303) when we need to represent the
type of the messages. As the WebSocket subprotocol is required to conform
to the token ABNF, they can be embedded into the subtype part.

Regarding negotiation, my impression to the subprotocol mechanism of RFC
6455 has been that it's not really necessary thing. The server and JS (or
native application) may negotiate application level sub-protocol using e.g.
URL (some parameters in the query part or even the path part can encode
such info) and the initial message sent following the handshake response
immediately without any latency loss (this topic was discussed at the HyBi
WG sometimes e.g. https://www.ietf.org/mail-archive/web/hybi/current/
msg10347.html).

But this is important to ease migration of existing WebSocket apps, yes,
and I agree that the Accept header would be one of the good candidates for
carrying it.

> able to), users just dropped text frames entirely.

Oh, interesting. Does this mean that on the browser a JSON codec that takes
/ produces ArrayBuffers is used?

> the UTF-8 check when JSON is expected for text frames?

We could have removed the valid UTF-8 requirement of RFC 6455. This is
something need to be enforced at the binding between the Web API and a
WebSocket protocol stack, but not at the protocol level.

Until https://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00,
text frames had to be a valid UTF-8 (strictly speaking, data had to be
0xFF-free), but not from the version 01.

We can choose to omit the valid UTF-8 requirement from the WiSH spec and
instead have it in the spec for gluing WiSH with the WebSocket API in the
future. Then, server implementors won't be explicitly encouraged to check
validness when implementing WiSH.

> * Page 5: there is "exchaning" instead of "exchanging".

Thanks! I'll fix them on the next draft submission.

<snip>

---

From: Loïc Hoguin <[email-masked]>
Date: Fri, 21 Oct 2016 18:34:00 +0200

On 10/21/2016 03:04 PM, Takeshi Yoshino wrote:
> like +json, +xml, etc. (RFC 6839, RFC 7303) when we need to represent
> the type of the messages. As the WebSocket subprotocol is required to
> conform to the token ABNF, they can be embedded into the subtype part.

The problem with "+json" and "+xml" is that they do not represent the 
type of the messages, just their encoding.

This is why we have media types like "application/problem+json" instead 
of using "application/json" for everything, where "application/problem" 
indicates what the representation contains, and "+json" indicates its 
encoding.

Going back to WiSH, having application/webstream+json would only give 
information about the encoding of the response and frames bodies, not 
what they actually contain or how an endpoint should process the frames. 
We would still need more info to process it.

> 6455 has been that it's not really necessary thing. The server and JS
> (or native application) may negotiate application level sub-protocol
> using e.g. URL (some parameters in the query part or even the path part
> can encode such info) and the initial message sent following the
> handshake response immediately without any latency loss (this topic was
> discussed at the HyBi WG sometimes e.g.
> https://www.ietf.org/mail-archive/web/hybi/current/msg10347.html
> <https://www.ietf.org/mail-archive/web/hybi/current/msg10347.html>).

The good thing is that it's standard. There's only one way to select a 
sub-protocol, and you are guaranteed failure (or a normal HTTP response) 
if the endpoint doesn't speak the sub-protocol you want.

The use of a media type is precious to me as it makes it possible to 
write an autonomous client. Making the sub-protocol as part of the media 
type simplifies things greatly.

There are potential issues though and a number of things need to be defined:

* The method used to perform the request is important. A client will not 
be able to speak to the server if the method is GET. Similarly, no 
communication can occur if the method is HEAD. I would advise defining 
behavior for both GET and POST methods.

* The GET method would be used to enable a server->client stream, 
similar to text/event-stream, except with binary frames supported. The 
difference being the lack of event IDs and Last-Event-ID header. In some 
cases it doesn't matter.

* The POST method would be used to enable a bidirectional stream. But 
this implies that the client uses Content-Type: application/webstream in 
the request, along with the Accept header. Otherwise the server has no 
way to know what the request body will contain. Let content negotiation 
deal with both headers, it's already well defined.

* Technically this would allow client->server and server->client streams 
to select and use a different sub-protocol. I'm not sure it's worth 
preventing this in the spec; instead let the servers decide if they want 
to allow this. But we probably need to mention it.

* By the way, don't know if consistency is desirable, by maybe calling 
it application/web-stream is better. Maybe not.

* The HEAD method behaves as usual. The PUT method is probably not 
compatible with doing this. PATCH and DELETE are not compatible AFAIK.

> <snip>
> Oh, interesting. Does this mean that on the browser a JSON codec that
> takes / produces ArrayBuffers is used?

I have no idea about browsers, sorry. :-) Not dealing with them often. 
But as far as I understand it involves using a different type to get 
binaries, yes. I have no idea how this is done.

>     Which brings me to my question: do you think it could be worth
>     adding a note to implementers that perhaps they should consider
>     optionally disabling the UTF-8 check when JSON is expected for text
>     frames?
>
>
> Until
> https://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00
> <https://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00>,
> instead have it in the spec for gluing WiSH with the WebSocket API in
> the future. Then, server implementors won't be explicitly encouraged to
> check validness when implementing WiSH.

That sounds like a good idea. I think it should still be mentioned in 
the protocol spec (basically explaining what you said and referring to 
another document) so that implementors are aware of the UTF-8 check and 
decide whether to implement it. If it was only written in a browser RFC 
I would most likely never have seen it, personally.

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 25 Oct 2016 17:54:52 +0900

On Sat, Oct 22, 2016 at 1:34 AM, Loïc Hoguin <[email-masked]> wrote:

> The problem with "+json" and "+xml" is that they do not represent the type
> of the messages, just their encoding.
>
> This is why we have media types like "application/problem+json" instead of
> using "application/json" for everything, where "application/problem"
> information about the encoding of the response and frames bodies, not what
> they actually contain or how an endpoint should process the frames. We
> would still need more info to process it.
>

Oh, sorry for being unclear. I meant that we'll use
"application/subprotocol+webstream". I.e. introducing +webstream as a new
media type suffix.

> sub-protocol, and you are guaranteed failure (or a normal HTTP response) if
> the endpoint doesn't speak the sub-protocol you want.
>
> The use of a media type is precious to me as it makes it possible to write
> an autonomous client. Making the sub-protocol as part of the media type
> simplifies things greatly.
>

OK

> behavior for both GET and POST methods.

Ah, yeah. It's missing. Thanks for pointing it out.

>
> * The GET method would be used to enable a server->client stream, similar
> to text/event-stream, except with binary frames supported. The difference
> being the lack of event IDs and Last-Event-ID header. In some cases it
> doesn't matter.
>
>
Agreed

> * The POST method would be used to enable a bidirectional stream. But this
> implies that the client uses Content-Type: application/webstream in the
> request, along with the Accept header. Otherwise the server has no way to
> know what the request body will contain. Let content negotiation deal with
> both headers, it's already well defined.
>
>
Yeah. I'll add some text for discussing need for communicating subprotocols
and remark the need for both request and response.

> preventing this in the spec; instead let the servers decide if they want to
> allow this. But we probably need to mention it.
>

Allowing more options (combination of asymmetric subprotocols) wouldn't
block existing WebSocket users from migrating to it. If we want to enable
it for the Web, the WebSocket API needs to be updated to take the
client->server direction subprotocol in addition to the offered subprotocol
(converted into the Accept header and selected by the server using the
Content-Type header in the response to specify the server->client direction
subprotocol).

Anyway, let's mention that.

>
> * By the way, don't know if consistency is desirable, by maybe calling it
> application/web-stream is better. Maybe not.
>

Could you please elaborate the proposal?

> compatible with doing this. PATCH and DELETE are not compatible AFAIK.

I'm feeling that we should just limit the scope of the proposal to GET and
POST.

> I have no idea about browsers, sorry. :-) Not dealing with them often. But
> as far as I understand it involves using a different type to get binaries,
> yes. I have no idea how this is done.
>

OK. Thanks!

> That sounds like a good idea. I think it should still be mentioned in the
> protocol spec (basically explaining what you said and referring to another
> document) so that implementors are aware of the UTF-8 check and decide
> whether to implement it. If it was only written in a browser RFC I would
> most likely never have seen it, personally.

Yeah. I'll add some text for discussing this.

---

From: Loïc Hoguin <[email-masked]>
Date: Tue, 25 Oct 2016 12:08:56 +0300

On 10/25/2016 11:54 AM, Takeshi Yoshino wrote:
> "application/subprotocol+webstream". I.e. introducing +webstream as a
> new media type suffix.

Ah!

OK I have no problem with that.

> <snip>
>     * By the way, don't know if consistency is desirable, by maybe
>     calling it application/web-stream is better. Maybe not.
>
>

I mean there's already text/event-stream, so application/webstream is 
not consistent with it (missing the dash). But maybe it doesn't matter.

> I'm feeling that we should just limit the scope of the proposal to GET
> and POST.

Sounds good to me.

Thanks for the great work! I look forward to implementing this.

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 25 Oct 2016 18:10:15 +0900

On Tue, Oct 25, 2016 at 6:08 PM, Loïc Hoguin <[email-masked]> wrote:

> I mean there's already text/event-stream, so application/webstream is not
> consistent with it (missing the dash). But maybe it doesn't matter.

Oh, I see!

> Thanks for the great work! I look forward to implementing this.

Thank you so much for the review and suggestions!

---

From: Wenbo Zhu <[email-masked]>
Date: Thu, 27 Oct 2016 18:04:23 -0700

Thanks for the feedback.

> would still need more info to process it.
Agreed. It's also unclear how to apply a single suffix to text/binary
frames.

> allow this. But we probably need to mention it.
It's also possible to have a non-web-stream request with a web-stream
response (POST).

> application/web-stream is better. Maybe not.
Sounds good.

> compatible with doing this. PATCH and DELETE are not compatible AFAIK.
Not sure why a PUT/PATCH request can't have a streamed body.  I don't think
we want to over-spec how to use HTTP with this media type (which is not the
only stream-able media type either)

---

From: Wenbo Zhu <[email-masked]>
Date: Thu, 27 Oct 2016 18:13:02 -0700

> OK I have no problem with that.

is it true only registered suffixes are allowed, e.g. json, xml etc?

application/webstream; protocol=...  seems more legit and most subprotocols
are not registered media types either..

Re: the actual parameter "protocol", we may want to mention it similar to
utf-8 checking, as a future concern for providing websocket compatibility.

---

From: Loïc Hoguin <[email-masked]>
Date: Fri, 28 Oct 2016 11:29:34 +0300

On 10/28/2016 04:04 AM, Wenbo Zhu wrote:
> Not sure why a PUT/PATCH request can't have a streamed body.  I don't
> think we want to over-spec how to use HTTP with this media type (which
> is not the only stream-able media type either)

PUT can definitely have a streamed body, but protocols are a little more 
than that. PUT creates or replaces the resource with the enclosed 
representation, so whether PUT can be used depends on the protocol. If 
webstream is used like an event stream then there's definitely no 
problem; if it's used for MQTT the PUT semantics are lost.

PATCH expects a media type containing instructions on how to modify the 
resource, so again it depends on the protocol.

We should definitely not restrict it to specific methods, and that's not 
what I was trying to say. I was just trying to point out which methods 
should be mentioned in the document, even if only in an informative way 
or in examples.

A more general paragraph about request methods forbidding bodies should 
be more than enough to cover everything without going too much into the 
details of each method.

Cheers,

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 28 Oct 2016 17:59:30 +0900

> is it true only registered suffixes are allowed, e.g. json, xml etc?

Did you mean the prefix?

>
> application/webstream; protocol=...  seems more legit and most
> subprotocols are not registered media types either..
>
>
I skimmed RFC 6838. I think application/xxx+suffix would be subject to the
same rule for general rule for media types even if +suffix is registered.

We can choose to put the protocol parameter under control of IANA, but yes,
we'll have options to decide whether we make it so or not.

> utf-8 checking, as a future concern for providing websocket compatibility.
I'm adding a section about that. For WebSocket compatibility, its value
should follow the token ABNF.

---

From: Costin Manolache <[email-masked]>
Date: Fri, 28 Oct 2016 16:25:24 +0000

I think WiSH could be great as a fallback to the webpush protocol push
promises, in devices
that don't fully support http/2, and for webpush delivery receipts.

In the context of IoT: since continuation is available, any reason to
support 64-bit-length frames ?
Even 32 bit ( 4G ) frames may be unpractical.  I don't know how common this
int encoding scheme
is - but using varint or fixed may be easier to handle.

Costin

On Fri, Oct 28, 2016 at 2:06 AM Takeshi Yoshino <[email-masked]> wrote:

> We can choose to put the protocol parameter under control of IANA, but
> yes, we'll have options to decide whether we make it so or not.
>
>

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 29 Oct 2016 02:21:07 +0900

> that don't fully support http/2, and for webpush delivery receipts.

Great to hear that.

> Even 32 bit ( 4G ) frames may be unpractical.  I don't know how common
> this int encoding scheme

It's basically not to introduce any gap with WebSocket's representation
power for compatibility while also keeping it efficient for small frames.

Re: varint,
Actually, old WebSocket protocol I-Ds used base 128 varint for encoding
frame length for binary frames.
https://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00
We can choose this as this is efficient for small messages but considered
that WebSocket encoding is already common and honored input which led to
the WebSocket encoding. IIRC there was some feedback from hardware
developer against to 128 varint, but I cannot find it now.

Re: big frames,
We're also seeing that very long messages are uncommon from Chromium's UMA
stats. We might be able to share that.

Re: small over head for small frames,
For the use cases where WiSH is layered on top of HTTP2 overhead and TLS
record overhead, this difference might be negligible.

So, ... I'm open to changing the encoding, but want to hear more :)

---

From: Wenbo Zhu <[email-masked]>
Date: Fri, 28 Oct 2016 10:31:38 -0700

On Fri, Oct 28, 2016 at 1:29 AM, Loïc Hoguin <[email-masked]> wrote:

> webstream is used like an event stream then there's definitely no problem;
> if it's used for MQTT the PUT semantics are lost.
>
> should be mentioned in the document, even if only in an informative way or
> in examples.
>
> A more general paragraph about request methods forbidding bodies should be
> more than enough to cover everything without going too much into the
> details of each method.
>
Agreed.

Thanks.

---

From: Costin Manolache <[email-masked]>
Date: Fri, 28 Oct 2016 17:41:18 +0000

Current overhead is 2 bytes if frame is up to 125 bytes long - which I
think it's not very common,
4 bytes for up to 64k, and 10 bytes for anything larger.
IMHO adding one byte - i.e. making it fixed 5-byte, with first as is, and
next 4 fixed length would
be easiest to parse.

There are obviously too many options to encode and each has benefits - my
only concern was
that the choice of 1, 2, 8 bytes for length may not match common sizes.

( in webpush frames will be <4k ).

Costin

---

From: Costin Manolache <[email-masked]>
Date: Fri, 28 Oct 2016 17:43:00 +0000

Also 24 bit fixed length ( like in http2 frame ) may be a reasonable
choice, with a 4-byte fixed header.

Costin

---

From: Loïc Hoguin <[email-masked]>
Date: Fri, 28 Oct 2016 20:55:34 +0300

On 10/28/2016 08:41 PM, Costin Manolache wrote:
> IMHO adding one byte - i.e. making it fixed 5-byte, with first as is,
> and next 4 fixed length would
> be easiest to parse.

Is making it easy (or easier) to parse even a concern anymore?

Considering the number of agents and servers already supporting 
Websocket, the numerous libraries for nearly all languages and the great 
autobahntestsuite project validating it all, reusing the existing code 
is a very sensible solution.

> There are obviously too many options to encode and each has benefits -
> my only concern was

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 29 Oct 2016 03:02:27 +0900

Thanks. Yeah, they are good candidates.

Regarding why we need to have to be able to build so large frame though we
have the fragmentation feature, IIRC, the original motivation for this
during establishment of RFC 6455 was potential needs to avoid fragmentation
e.g. to be able to sendfile() a big file directly into a WebSocket message.
I now don't think we need to care this concern so much.

On Sat, Oct 29, 2016 at 2:43 AM, Costin Manolache <[email-masked]> wrote:

>> We're also seeing that very long messages are uncommon from Chromium's
>> UMA stats. We might be able to share that.
>>

---

From: Wenbo Zhu <[email-masked]>
Date: Fri, 28 Oct 2016 11:03:13 -0700

We are not designing the best framing here .. and Takeshi knows this the
best, being through the entire history of websocket discussion.

I think the current encoding scheme for small messages is fine, 1 byte <
125; 3 bytes < 64K; and then 9 bytes for everything else (by this point,
saving 4 bytes at the header of the message won't matter).

BTW, continuation is not widely used, e.g. there is not JS API for this.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 29 Oct 2016 03:12:52 +0900

On Sat, Oct 29, 2016 at 2:55 AM, Loïc Hoguin <[email-masked]> wrote:

> Considering the number of agents and servers already supporting Websocket,
> the numerous libraries for nearly all languages and the great
> autobahntestsuite project validating it all, reusing the existing code is a
> very sensible solution.
>
>
Yeah, I've been having similar feeling regarding cost for parser/encoder
implementation though I might be biased.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 29 Oct 2016 03:17:40 +0900

> BTW, continuation is not widely used, e.g. there is not JS API for this.

Yes, that's true regarding availability via the WebSocket Web API. In
Chromium, its WebSocket stack transparently fragments messages as it likes,
and because of that, most servers now support it.

>>> We can choose this as this is efficient for small messages but
>>> considered that WebSocket encoding is already common and honored input
>>> which led to the WebSocket encoding. IIRC there was some feedback from
>>> hardware developer against to 128 varint, but I cannot find it now.
>>>
>>> Re: big frames,

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 29 Oct 2016 04:05:58 +0900

Sorry for being ambivalent.

We can of course revisit each design decision we made for RFC 6455 framing
and search for the optimal again. But as:
- one of the main philosophies behind WiSH is compatibility with WebSocket
in terms of both spec and implementation
- the WebSocket is widely deployed and therefore we have a lot of
implementations in various languages/platform
- most browsers already have logic for the framing
- the framing is not considered to be so big pain
inheriting the WebSocket framing almost as-is is just good enough.
Basically, I'm leaning toward this plan.

Takeshi

---

From: Van Catha <[email-masked]>
Date: Sat, 29 Oct 2016 13:25:13 -0400

About changing the framing and other things, I think this is wrong. The way
WiSH currently is presented is perfect for quick adaptation.  Any changes
to framing or how it works will greatly delay browsers adding it as they
will have to change client code related to websockets.  Removing the mask
for example is fine, the browser will just always set the MASK bit to 0.
No need to change any client code.

---

From: Costin Manolache <[email-masked]>
Date: Sat, 29 Oct 2016 17:43:43 +0000

Good point - websocket is widely deployed, including IoT - and the header
is pretty easy to handle anyways.
+1.

One question: is this intended to be handled by browsers, and exposed using
the W3C websocket API ?
Will a regular app be able to make WiSH requests and parse the stream by
itself, without browser
interference ? And if yes, any advice on how it interact with CORS ?

Costin

---

From: Takeshi Yoshino <[email-masked]>
Date: Sun, 30 Oct 2016 21:27:28 +0900

Thanks, Van, Costin.

On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <[email-masked]> wrote:

> One question: is this intended to be handled by browsers, and exposed
> using the W3C websocket API ?

The first step would be using Streams based upload/download via the Fetch
API + protocol processing in JS.

The next step could be either introduction of an optimized native
implementation of WiSH parser/framer in the form of the TransformStream
which can be used as follows:

const responsePromise = fetch(url, init);
responsePromise.then(response => {
  const wishStream = response.body().pipeThrough(wishTransformStream);
  function readAndProcessMessage() {
    const readPromise = wishStream.read();
    readPromise.then(result => {
      if (result.done) {
        // End of stream.
        return;
      }

      const message = result.value;
      // Process the message
      // E.g. access message.opcode for opcode, message.body for the body
data
      readAndProcessMessage();
    });
  }
  readAndProcessMessage();
});

and provide a polyfill that presents this as the WebSocket API, and (or
skip the step and) go further i.e. native implementation for everything if
it turns out optimization is critical.

We need to discuss this also in W3C/WHATWG.

Regarding CORS, if the request includes non CORS-safelisted headers,
fetch() based JS polyfills will be basically subject to the CORS preflight
requirement. We could try to exempt some of well defined headers if any for
CORS like WebSocket handshake's headers and server-sent event's
Last-Event-Id are exempted. Regarding the proposed subprotocol negotiation
in the form of combination of the Accept header and the Content-Type
header, the Accept header is one of the CORS-safelisted headers, so it's
not a problem. The Content-Type header is considered to be
non-CORS-safelisted if it's value is none of the CORS-safelisted media
types. So, WiSH media type would trigger the preflight unless we exclude it.

Origin policy https://wicg.github.io/origin-policy/ might also help.

>> We can of course revisit each design decision we made for RFC 6455
>> framing and search for the optimal again. But as:
>> - one of the main philosophies behind WiSH is compatibility with
>> WebSocket in terms of both spec and implementation

---

From: Costin Manolache <[email-masked]>
Date: Sun, 30 Oct 2016 17:25:52 +0000

Thanks for the answer and pointers. From earlier responses, it seems
possible to use GET
or a non-web-stream request to would avoid the extra cost of the pre-flight.

One more question/issue: in some cases it would be good to send some
metadata (headers) along with binary frames. For example in webpush the
content is an encrypted
blob, and needs headers for the key/salt. I would assume a lot of other
'binary' messages would
benefit if simple metadata could be sent along. Would it be possible to use
one of the reserved
bits for 'has metadata' and add some encoded headers ? I know in websocket
they are intended
for 'extensions', but 'headers' seems a very common use case.

Having the binary frame use some MIME encoding to pass both text headers
and the binary blob
is possible - but has complexity and overhead.

Costin

---

From: Wenbo Zhu <[email-masked]>
Date: Sun, 30 Oct 2016 13:57:13 -0700

> benefit if simple metadata could be sent along. Would it be possible to
> use one of the reserved
Q about webpush: is the metadata different for each binary message?

We discussed about metadata and how to use one of RSV bits etc. For the
current version, let's make sure the WS compatibility is fully addressed
(with minimum wire encoding like WiSH)

> is possible - but has complexity and overhead.
OTOH, if the binary blob relies on text headers (metata) to be useful, then
you probably need define a dedicated MIME encoding.