Subject: Re: WiSH: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols (HTTP)

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 2 Nov 2016 14:34:26 +0900

On Mon, Oct 31, 2016 at 5:57 AM, Wenbo Zhu <[email-masked]> wrote:

>
>
> On Sun, Oct 30, 2016 at 10:25 AM, Costin Manolache <[email-masked]>
> wrote:
>
>> Thanks for the answer and pointers. From earlier responses, it seems
>> possible to use GET
>> or a non-web-stream request to would avoid the extra cost of the
>> pre-flight.
>>
>
>>
Yeah, at least the Content-Type in the HTTP request gets eliminated.

> One more question/issue: in some cases it would be good to send some
>> metadata (headers) along with binary frames. For example in webpush the
>> content is an encrypted
>> blob, and needs headers for the key/salt. I would assume a lot of other
>> 'binary' messages would
>> benefit if simple metadata could be sent along. Would it be possible to
>> use one of the reserved
>> bits for 'has metadata' and add some encoded headers ? I know in
>> websocket they are intended
>> for 'extensions', but 'headers' seems a very common use case.
>>
> Q about webpush: is the metadata different for each binary message?
>
> We discussed about metadata and how to use one of RSV bits etc. For the
> current version, let's make sure the WS compatibility is fully addressed
> (with minimum wire encoding like WiSH)
>

Agreed. Let's discuss the metadata needs separately. I agree it's important.

>
>
>>
>> Having the binary frame use some MIME encoding to pass both text headers
>> and the binary blob
>> is possible - but has complexity and overhead.
>>
> OTOH, if the binary blob relies on text headers (metata) to be useful,
> then you probably need define a dedicated MIME encoding.
>
>
>
>>
>> Costin
>>
>> On Sun, Oct 30, 2016 at 5:27 AM Takeshi Yoshino <[email-masked]>
>> wrote:
>>
>>> Thanks, Van, Costin.
>>>
>>> On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <[email-masked]>
>>> wrote:
>>>
>>> Good point - websocket is widely deployed, including IoT - and the
>>> header is pretty easy to handle anyways.
>>> +1.
>>>
>>> One question: is this intended to be handled by browsers, and exposed
>>> using the W3C websocket API ?
>>> Will a regular app be able to make WiSH requests and parse the stream by
>>> itself, without browser
>>> interference ? And if yes, any advice on how it interact with CORS ?
>>>
>>>
>>> The first step would be using Streams based upload/download via the
>>> Fetch API + protocol processing in JS.
>>>
>>> The next step could be either introduction of an optimized native
>>> implementation of WiSH parser/framer in the form of the TransformStream
>>> which can be used as follows:
>>>
>>> const responsePromise = fetch(url, init);
>>> responsePromise.then(response => {
>>>   const wishStream = response.body().pipeThrough(wishTransformStream);
>>>   function readAndProcessMessage() {
>>>     const readPromise = wishStream.read();
>>>     readPromise.then(result => {
>>>       if (result.done) {
>>>         // End of stream.
>>>         return;
>>>       }
>>>
>>>       const message = result.value;
>>>       // Process the message
>>>       // E.g. access message.opcode for opcode, message.body for the
>>> body data
>>>       readAndProcessMessage();
>>>     });
>>>   }
>>>   readAndProcessMessage();
>>> });
>>>
>>> and provide a polyfill that presents this as the WebSocket API, and (or
>>> skip the step and) go further i.e. native implementation for everything if
>>> it turns out optimization is critical.
>>>
>>> We need to discuss this also in W3C/WHATWG.
>>>
>>> Regarding CORS, if the request includes non CORS-safelisted headers,
>>> fetch() based JS polyfills will be basically subject to the CORS preflight
>>> requirement. We could try to exempt some of well defined headers if any for
>>> CORS like WebSocket handshake's headers and server-sent event's
>>> Last-Event-Id are exempted. Regarding the proposed subprotocol negotiation
>>> in the form of combination of the Accept header and the Content-Type
>>> header, the Accept header is one of the CORS-safelisted headers, so it's
>>> not a problem. The Content-Type header is considered to be
>>> non-CORS-safelisted if it's value is none of the CORS-safelisted media
>>> types. So, WiSH media type would trigger the preflight unless we exclude it.
>>>
>>> Origin policy https://wicg.github.io/origin-policy/ might also help.
>>>
>>>
>>>
>>> Costin
>>>
>>> On Fri, Oct 28, 2016 at 12:06 PM Takeshi Yoshino <[email-masked]>
>>> wrote:
>>>
>>> Sorry for being ambivalent.
>>>
>>> We can of course revisit each design decision we made for RFC 6455
>>> framing and search for the optimal again. But as:
>>> - one of the main philosophies behind WiSH is compatibility with
>>> WebSocket in terms of both spec and implementation
>>> - the WebSocket is widely deployed and therefore we have a lot of
>>> implementations in various languages/platform
>>> - most browsers already have logic for the framing
>>> - the framing is not considered to be so big pain
>>> inheriting the WebSocket framing almost as-is is just good enough.
>>> Basically, I'm leaning toward this plan.
>>>
>>> Takeshi
>>>
>>> On Sat, Oct 29, 2016 at 3:12 AM, Takeshi Yoshino <[email-masked]>
>>> wrote:
>>>
>>> On Sat, Oct 29, 2016 at 2:55 AM, Loïc Hoguin <[email-masked]> wrote:
>>>
>>> On 10/28/2016 08:41 PM, Costin Manolache wrote:
>>>
>>> Current overhead is 2 bytes if frame is up to 125 bytes long - which I
>>> think it's not very common,
>>> 4 bytes for up to 64k, and 10 bytes for anything larger.
>>> IMHO adding one byte - i.e. making it fixed 5-byte, with first as is,
>>> and next 4 fixed length would
>>> be easiest to parse.
>>>
>>>
>>> Is making it easy (or easier) to parse even a concern anymore?
>>>
>>> Considering the number of agents and servers already supporting
>>> Websocket, the numerous libraries for nearly all languages and the great
>>> autobahntestsuite project validating it all, reusing the existing code is a
>>> very sensible solution.
>>>
>>>
>>> Yeah, I've been having similar feeling regarding cost for parser/encoder
>>> implementation though I might be biased.
>>>
>>>
>>> There are obviously too many options to encode and each has benefits -
>>> my only concern was
>>> that the choice of 1, 2, 8 bytes for length may not match common sizes.
>>>
>>> ( in webpush frames will be <4k ).
>>>
>>>
>>> --
>>> Loïc Hoguin
>>> https://ninenines.eu
>>>
>>>
>>>
>>>
>

---

From: Costin Manolache <[email-masked]>
Date: Wed, 02 Nov 2016 18:20:10 +0000

Good timing -  http://httpwg.org/http-extensions/encryption-preview.html is
addressing my concerns for
webpush ( and general 'encrypted content' ), we're still discussing some
details, but for this use
case metadata won't be needed.

Costin

On Tue, Nov 1, 2016 at 10:34 PM Takeshi Yoshino <[email-masked]> wrote:

> bits for 'has metadata' and add some encoded headers ? I know in websocket
> they are intended
> Good point - websocket is widely deployed, including IoT - and the header
> is pretty easy to handle anyways.
> +1.
>
> The first step would be using Streams based upload/download via the Fetch
> API + protocol processing in JS.
>
>       // E.g. access message.opcode for opcode, message.body for the body
> data
>       readAndProcessMessage();
>     });
>   }
>   readAndProcessMessage();
> });
>
> We can of course revisit each design decision we made for RFC 6455 framing
> and search for the optimal again. But as:
> - one of the main philosophies behind WiSH is compatibility with WebSocket
> in terms of both spec and implementation
> Considering the number of agents and servers already supporting Websocket,
> the numerous libraries for nearly all languages and the great

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 18 Nov 2016 15:55:47 +0900

I'd like to share the feedback on WiSH from IETF 97.

----

Due to limited time, I got just one on-site comment from Martin about
comparison with Server-sent event (EventSource).

As mentioned in the I-D, yes, this is kinda full-duplex SSE with the WS
framing, and it might suffer from unexpected buffering by intermediaries if
any as Martin said.

WiSH should work well for deployment with TLS only (possibly with some
non-TLS part beyond server side front-end but are under control of the
service providers). Given the wide trend of encouraging TLS and browser
vendors' implementation status of H2, I think we should prioritize layering
simplicity than taking care of gain of WiSH/H2/TCP + transparent proxy
(with unexpected buffering) case. For H2-less TLS-less environment, we can
just use the WebSocket protocol.

There can still be some risk of MITM (trusted) proxy and unexpected
buffering with AntiVirus/Firewall for deployment with TLS, but other
WebSocket/H2 mapping proposals also have issues of possible blocking,
buffering, etc. WebSocket/TCP's handshake success rate for non-TLS port 80
was also not so good when it started getting deployed, and got improved
gradually. I think the problems will get resolved once WiSH is accepted
widely, and I believe the total pain and cost would be smaller.

----

Mark suggested that we should find some other right place than HTTP WG.
I'll discuss this with Mark. Maybe we'll consult the DISPATCH WG.

----

Thanks everyone for the feedback.

Takeshi

---

From: Van Catha <[email-masked]>
Date: Sat, 19 Nov 2016 22:26:51 -0500

I do not understand what this means.  Is the suggestion to ignore WiSH for
now in favor of SSE?

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 22 Nov 2016 13:14:32 +0900

Ah, no. Martin just warned us that we might face the same issue that SSE
faced.

Mark's suggestion is a separate thing. The co-chairs (Mark and Patrick)
said that this (WiSH) doesn't seems to be a topic that should be discussed
in the HTTP WG given the charter of the WG, I think.

On Sun, Nov 20, 2016 at 12:26 PM, Van Catha <[email-masked]> wrote:

>>>> Having the binary frame use some MIME encoding to pass both text
>>>> headers and the binary blob
>>>> Will a regular app be able to make WiSH requests and parse the stream
>>>> by itself, without browser
>>>> Yeah, I've been having similar feeling regarding cost for
>>>> parser/encoder implementation though I might be biased.
>>>>
>>>>

---

From: Van Catha <[email-masked]>
Date: Thu, 24 Nov 2016 13:39:04 -0500

Thanks for clarification. Unfortunate that so little attention was paid to
this.  Looks like some of us will be on HTTP1.1 for a long time.

On Mon, Nov 21, 2016 at 11:14 PM, Takeshi Yoshino <[email-masked]>
wrote:

>> I do not understand what this means.  Is the suggestion to ignore WiSH
>> for now in favor of SSE?
>>
>>>> Good timing -  http://httpwg.org/http-exte
>>>> nsions/encryption-preview.html is addressing my concerns for
>>>> webpush ( and general 'encrypted content' ), we're still discussing
>>>> some details, but for this use
>>>>> metadata (headers) along with binary frames. For example in webpush
>>>>> the content is an encrypted
>>>>> blob, and needs headers for the key/salt. I would assume a lot of
>>>>> other 'binary' messages would
>>>>> benefit if simple metadata could be sent along. Would it be possible
>>>>> to use one of the reserved
>>>>> We discussed about metadata and how to use one of RSV bits etc. For
>>>>> the current version, let's make sure the WS compatibility is fully
>>>>> addressed (with minimum wire encoding like WiSH)
>>>>>
>>>>>
>>>>> and provide a polyfill that presents this as the WebSocket API, and
>>>>> (or skip the step and) go further i.e. native implementation for everything
>>>>> if it turns out optimization is critical.
>>>>>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 25 Nov 2016 10:35:09 +1100

To clarify, we said that the future of WebSockets really isn't in scope for this WG; the proper venue for discussing that is:
  https://www.ietf.org/mailman/listinfo/hybi

What *is* in-scope here is how (if at all) that protocol interacts with HTTP, including HTTP/2; there are several ways you could implement WebSockets over HTTP/2, and a few pitfalls in doing so that the people on this list will be able to give you feedback on.

However, it's hard to do that before there's agreement in the WS community about what the requirements are. Ideally, that community would bring a single proposal that has broad support here for review.

Cheers,

> On 25 Nov. 2016, at 5:39 am, Van Catha <[email-masked]> wrote:
> 
> Thanks for clarification. Unfortunate that so little attention was paid to this.  Looks like some of us will be on HTTP1.1 for a long time.
> 
> On Mon, Nov 21, 2016 at 11:14 PM, Takeshi Yoshino <[email-masked]> wrote:
> Ah, no. Martin just warned us that we might face the same issue that SSE faced.
> 
> Mark's suggestion is a separate thing. The co-chairs (Mark and Patrick) said that this (WiSH) doesn't seems to be a topic that should be discussed in the HTTP WG given the charter of the WG, I think.
> 
> I do not understand what this means.  Is the suggestion to ignore WiSH for now in favor of SSE?
> 
> On Fri, Nov 18, 2016 at 1:55 AM, Takeshi Yoshino <[email-masked]> wrote:
> Due to limited time, I got just one on-site comment from Martin about comparison with Server-sent event (EventSource).
> 
> As mentioned in the I-D, yes, this is kinda full-duplex SSE with the WS framing, and it might suffer from unexpected buffering by intermediaries if any as Martin said.
> 
> WiSH should work well for deployment with TLS only (possibly with some non-TLS part beyond server side front-end but are under control of the service providers). Given the wide trend of encouraging TLS and browser vendors' implementation status of H2, I think we should prioritize layering simplicity than taking care of gain of WiSH/H2/TCP + transparent proxy (with unexpected buffering) case. For H2-less TLS-less environment, we can just use the WebSocket protocol.
> 
> There can still be some risk of MITM (trusted) proxy and unexpected buffering with AntiVirus/Firewall for deployment with TLS, but other WebSocket/H2 mapping proposals also have issues of possible blocking, buffering, etc. WebSocket/TCP's handshake success rate for non-TLS port 80 was also not so good when it started getting deployed, and got improved gradually. I think the problems will get resolved once WiSH is accepted widely, and I believe the total pain and cost would be smaller.
> 
> ----
> 
> Mark suggested that we should find some other right place than HTTP WG. I'll discuss this with Mark. Maybe we'll consult the DISPATCH WG.
> 
> ----
> 
> Good timing -  http://httpwg.org/http-extensions/encryption-preview.html is addressing my concerns for
> webpush ( and general 'encrypted content' ), we're still discussing some details, but for this use
> On Sun, Oct 30, 2016 at 10:25 AM, Costin Manolache <[email-masked]> wrote:
> Thanks for the answer and pointers. From earlier responses, it seems possible to use GET
> or a non-web-stream request to would avoid the extra cost of the pre-flight.
> 
> 
> metadata (headers) along with binary frames. For example in webpush the content is an encrypted
> blob, and needs headers for the key/salt. I would assume a lot of other 'binary' messages would
> benefit if simple metadata could be sent along. Would it be possible to use one of the reserved
> bits for 'has metadata' and add some encoded headers ? I know in websocket they are intended 
> We discussed about metadata and how to use one of RSV bits etc. For the current version, let's make sure the WS compatibility is fully addressed (with minimum wire encoding like WiSH)
> 
> Having the binary frame use some MIME encoding to pass both text headers and the binary blob
> OTOH, if the binary blob relies on text headers (metata) to be useful, then you probably need define a dedicated MIME encoding.
> 
>   
> 
> Costin
> 
> On Sun, Oct 30, 2016 at 5:27 AM Takeshi Yoshino <[email-masked]> wrote:
> Thanks, Van, Costin.
> 
> On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <[email-masked]> wrote:
> Good point - websocket is widely deployed, including IoT - and the header is pretty easy to handle anyways.
> +1.
> 
> One question: is this intended to be handled by browsers, and exposed using the W3C websocket API ?
> Will a regular app be able to make WiSH requests and parse the stream by itself, without browser
> The first step would be using Streams based upload/download via the Fetch API + protocol processing in JS.
> 
> The next step could be either introduction of an optimized native implementation of WiSH parser/framer in the form of the TransformStream which can be used as follows:
> 
>       // E.g. access message.opcode for opcode, message.body for the body data
>       readAndProcessMessage();
>     });
>   }
>   readAndProcessMessage();
> });
> 
> and provide a polyfill that presents this as the WebSocket API, and (or skip the step and) go further i.e. native implementation for everything if it turns out optimization is critical.
> 
> Regarding CORS, if the request includes non CORS-safelisted headers, fetch() based JS polyfills will be basically subject to the CORS preflight requirement. We could try to exempt some of well defined headers if any for CORS like WebSocket handshake's headers and server-sent event's Last-Event-Id are exempted. Regarding the proposed subprotocol negotiation in the form of combination of the Accept header and the Content-Type header, the Accept header is one of the CORS-safelisted headers, so it's not a problem. The Content-Type header is considered to be non-CORS-safelisted if it's value is none of the CORS-safelisted media types. So, WiSH media type would trigger the preflight unless we exclude it.
> 
> On Fri, Oct 28, 2016 at 12:06 PM Takeshi Yoshino <[email-masked]> wrote:
> We can of course revisit each design decision we made for RFC 6455 framing and search for the optimal again. But as:
> - one of the main philosophies behind WiSH is compatibility with WebSocket in terms of both spec and implementation
> - the WebSocket is widely deployed and therefore we have a lot of implementations in various languages/platform
> inheriting the WebSocket framing almost as-is is just good enough. Basically, I'm leaning toward this plan.
> 
> Takeshi
> 
> On Sat, Oct 29, 2016 at 3:12 AM, Takeshi Yoshino <[email-masked]> wrote:
> Considering the number of agents and servers already supporting Websocket, the numerous libraries for nearly all languages and the great autobahntestsuite project validating it all, reusing the existing code is a very sensible solution.
> 
> 
> Yeah, I've been having similar feeling regarding cost for parser/encoder implementation though I might be biased.
>  

--
Mark Nottingham   https://www.mnot.net/

---

From: Andy Green <[email-masked]>
Date: Fri, 25 Nov 2016 08:51:19 +0800

On Fri, 2016-11-25 at 10:35 +1100, Mark Nottingham wrote:
> To clarify, we said that the future of WebSockets really isn't in
> scope for this WG; the proper venue for discussing that is:

They kind of shut that down, the people managing it have gone away

https://mailarchive.ietf.org/arch/msg/hybi/YVBJTzJcvzytIY46KfIS8bchf5E

the ML is still running but is very quiet.

WS itself seems to be in rude health out there.

> What *is* in-scope here is how (if at all) that protocol interacts
> with HTTP, including HTTP/2; there are several ways you could
> implement WebSockets over HTTP/2, and a few pitfalls in doing so that
> the people on this list will be able to give you feedback on.

It's unfortunate that wasn't considered part of defining HTTP/2, so it
could be baked in.   The subject was certainly raised.  But I can
understand the desire to get the main business out of the door.

> However, it's hard to do that before there's agreement in the WS
> community about what the requirements are. Ideally, that community
> would bring a single proposal that has broad support here for review.

Any formal mechanism to manage that has gone away for hybi AFAICT.  So
this "let the community do it" feels like a bit of a cop-out / bullet
dodging.

At any rate I think the number of people interested in HTTP/2 WS is
still very low compared to the number of people interested in WS client
and server generally in my experience.  But at some point that will
change, possibly suddenly.

-Andy

> > Thanks for clarification. Unfortunate that so little attention was
> > paid to this.  Looks like some of us will be on HTTP1.1 for a long
> > time.
> > 
> > Mark's suggestion is a separate thing. The co-chairs (Mark and
> > Patrick) said that this (WiSH) doesn't seems to be a topic that
> > should be discussed in the HTTP WG given the charter of the WG, I
> > think.
> > 
> > Due to limited time, I got just one on-site comment from Martin
> > about comparison with Server-sent event (EventSource).
> > 
> > As mentioned in the I-D, yes, this is kinda full-duplex SSE with
> > the WS framing, and it might suffer from unexpected buffering by
> > intermediaries if any as Martin said.
> > 
> > WiSH should work well for deployment with TLS only (possibly with
> > some non-TLS part beyond server side front-end but are under
> > control of the service providers). Given the wide trend of
> > encouraging TLS and browser vendors' implementation status of H2, I
> > think we should prioritize layering simplicity than taking care of
> > gain of WiSH/H2/TCP + transparent proxy (with unexpected buffering)
> > case. For H2-less TLS-less environment, we can just use the
> > WebSocket protocol.
> > 
> > buffering with AntiVirus/Firewall for deployment with TLS, but
> > other WebSocket/H2 mapping proposals also have issues of possible
> > blocking, buffering, etc. WebSocket/TCP's handshake success rate
> > for non-TLS port 80 was also not so good when it started getting
> > deployed, and got improved gradually. I think the problems will get
> > resolved once WiSH is accepted widely, and I believe the total pain
> > and cost would be smaller.
> > 
> > ----
> > 
> > Mark suggested that we should find some other right place than HTTP
> > WG. I'll discuss this with Mark. Maybe we'll consult the DISPATCH
> > WG.
> > 
> > ----
> > 
> > OTOH, if the binary blob relies on text headers (metata) to be
> > useful, then you probably need define a dedicated MIME encoding.
> > 
> >   
> > 
> > Costin
> > 
> > On Sun, Oct 30, 2016 at 2:43 AM, Costin Manolache <[email-masked]
> > > wrote:
> > implementation of WiSH parser/framer in the form of the
> > TransformStream which can be used as follows:
> > 
> >   const wishStream =
> > response.body().pipeThrough(wishTransformStream);
> > (or skip the step and) go further i.e. native implementation for
> > everything if it turns out optimization is critical.
> > 
> > Regarding CORS, if the request includes non CORS-safelisted
> > headers, fetch() based JS polyfills will be basically subject to
> > the CORS preflight requirement. We could try to exempt some of well
> > defined headers if any for CORS like WebSocket handshake's headers
> > and server-sent event's Last-Event-Id are exempted. Regarding the
> > proposed subprotocol negotiation in the form of combination of the
> > Accept header and the Content-Type header, the Accept header is one
> > of the CORS-safelisted headers, so it's not a problem. The Content-
> > Type header is considered to be non-CORS-safelisted if it's value
> > is none of the CORS-safelisted media types. So, WiSH media type
> > would trigger the preflight unless we exclude it.
> > 
> > Websocket, the numerous libraries for nearly all languages and the
> > great autobahntestsuite project validating it all, reusing the
> > existing code is a very sensible solution.
> > 
> >

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 25 Nov 2016 13:12:36 +1100

Andy,

The "people managing it" are the IETF. "Let the community do it" is how the IETF works, with guidance from IETF leadership. I had a discussion with our Area Director about this topic in Seoul, and we agreed that the hybi list was the best place to talk about WebSockets.

Cheers,

> On 25 Nov. 2016, at 11:51 am, Andy Green <[email-masked]> wrote:
>

---

From: Andy Green <[email-masked]>
Date: Fri, 25 Nov 2016 10:32:03 +0800

On Fri, 2016-11-25 at 13:12 +1100, Mark Nottingham wrote:
> Andy,
> 
> The "people managing it" are the IETF. "Let the community do it" is
> how the IETF works, with guidance from IETF leadership. I had a 

No... when ws for http/1 was hashed out, there were some adults in the
room who could make judgments about what was or wasn't "a consensus" in
a kinda-definitive way.  Several times these calls were important in
stopping the definition going around in endless circles... even so you
will recall there were like 80+ drafts of it.

That has gone away now.

> discussion with our Area Director about this topic in Seoul, and we
> agreed that the hybi list was the best place to talk about
> WebSockets.

That's not necessarily a problem, but is either of you guys who decided
that going to restore the ML to have the kind of management it had when
it spat out RFC6455?  Or if it's chicken-and-egg, since the list is
moribund and no evident interest today to point to, some threshold for
that happening?  If not as I said it feels like it's just getting
punted over the wall.  It's going back to Hybi but the management of
Hybi is gone and it's nobody's problem...

-Andy

> > > > On Mon, Nov 21, 2016 at 11:14 PM, Takeshi Yoshino <tyoshino@goo
> > > > gle.
> > > > com> wrote:
> > > > I do not understand what this means.  Is the suggestion to
> > > > ignore
> > > > WiSH for now in favor of SSE?
> > > > 
> > > > On Fri, Nov 18, 2016 at 1:55 AM, Takeshi Yoshino <tyoshino@goog
> > > > le.c
> > > > om> wrote:
> > > > On Thu, Nov 3, 2016 at 3:20 AM, Costin Manolache <costin@gmail.
> > > > com>
> > > > wrote:
> > > > Good timing -  http://httpwg.org/http-extensions/encryption-pre
> > > > view
> > > > .html is addressing my concerns for
> > > > On Tue, Nov 1, 2016 at 10:34 PM Takeshi Yoshino <tyoshino@googl
> > > > e.co
> > > > m> wrote:
> > > > On Sun, Oct 30, 2016 at 10:25 AM, Costin Manolache <costin@gmai
> > > > l.co
> > > > m> wrote:
> > > > Agreed. Let's discuss the metadata needs separately. I agree
> > > > it's
> > > > important.
> > > >  
> > > >  
> > > > 
> > > > On Sun, Oct 30, 2016 at 5:27 AM Takeshi Yoshino <tyoshino@googl
> > > > e.co
> > > > m> wrote:
> > > > Thanks, Van, Costin.
> > > > 
> > > > interference ? And if yes, any advice on how it interact with
> > > > CORS
> > > > ? 
> > > > 
> > > >       // E.g. access message.opcode for opcode, message.body
> > > > for
> > > > the body data
> > > >       readAndProcessMessage();
> > > >     });
> > > >   }
> > > >   readAndProcessMessage();
> > > > });
> > > > 
> > > > On Fri, Oct 28, 2016 at 12:06 PM Takeshi Yoshino <tyoshino@goog
> > > > le.c
> > > > om> wrote:
> > > > On Sat, Oct 29, 2016 at 3:12 AM, Takeshi Yoshino <tyoshino@goog
> > > > le.c
> > > > om> wrote:
> > > > On Sat, Oct 29, 2016 at 2:55 AM, Loïc Hoguin <[email-masked]
> > > > u>
> > > > wrote:
> > > > IMHO adding one byte - i.e. making it fixed 5-byte, with first
> > > > as
> > > > is,
> > > > that the choice of 1, 2, 8 bytes for length may not match
> > > > common
> > > > sizes.
> > > >

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 25 Nov 2016 13:35:45 +1100

> On 25 Nov. 2016, at 1:32 pm, Andy Green <[email-masked]> wrote:
> 

Adding Alexey, the Area Directory who's most relevant to this.

You seem to be looking for a Working Group. WGs are formed when the IESG (i.e., all of the ADs) are convinced that it's useful to do so. The first step is usually to put together a charter proposal, and start gathering a community.

In the meantime, the adult supervision you're looking for isn't over here either; our charter restricts us to HTTP.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Nov 2016 07:52:08 +0100

On Fri, Nov 25, 2016 at 10:32:03AM +0800, Andy Green wrote:
> that happening?

No, the list isn't moribund. There are periods of the year where people
are more available for discussions and periods where they are less
available. Also before an IETF meeting people exchange less because
many will meet face to face and save time this way, after a meeting
they discuss more, but mostly on subjects presented at the meeting.
The HyBi WG was very active because it started bad and woke up passions
among many people who were scared to have to destroy their product just
to support it. But the period was devastating for everyone involved,
better never do that again.

These days there are several active drafts posted on the WG here and
people can assign 1 hour once in a while to review them and comment.
Many of us also rely on the others to do the first reading and see what
it speaks about without spending this first hour. You just need to be a
bit patient and to sometimes remind people when you need help or review
with your draft so that participants can reprioritize their planned
activities.

Regards,
willy

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 25 Nov 2016 17:25:20 +0900

Thanks all.

IESG and IETF don't have unlimited resource. Conclusion of HyBi did make
sense.

As noted by Barry in his mail about WG conclusion and as Mark said, we can
form a supervision again once there's enough interest.
https://mailarchive.ietf.org/arch/msg/hybi/vreF1jd3I-vsyWN1TiRnFSCEoVI

> What *is* in-scope here is how (if at all) that protocol interacts with
HTTP, including HTTP/2; there are several ways you could implement
WebSockets over HTTP/2, and a few pitfalls in doing so that the people on
this list will be able to give you feedback on.

One of the keys of the WiSH proposal is to focus on API level compatibility
with WebSocket. But except for that point, it's a general proposal of
application of HTTP semantics and HTTP/2's power for full-duplex messaging
in the Web. The proposal (one done by Yutaka in 2014 and Van's one also)
heavily depends on what the HTTP WG produces (specs, documents and possibly
any kind of official/unofficial communications). So, I think there
shouldn't be no doubt on need for close work with HTTP WG.

That said, I agree we need to have the right structure of the community to
have the "best" work mode based on various metrics (level of interest for
each proposal, their complexity, scope, etc.), and the IESG and the
co-chairs are trying to do the best in making the right decision, I think.

> However, it's hard to do that before there's agreement in the WS
community about what the requirements are. Ideally, that community would
bring a single proposal that has broad support here for review.

Mark, does this post of yours imply that you're seeing HyBi ML as one
effective representative of WS community at this point with HTTP WG chair
hat on?

Thanks

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 27 Nov 2016 17:53:26 +1100

> On 25 Nov. 2016, at 7:25 pm, Takeshi Yoshino <[email-masked]> wrote:
> 
> Thanks all.
> 
> IESG and IETF don't have unlimited resource. Conclusion of HyBi did make sense.
> 
> As noted by Barry in his mail about WG conclusion and as Mark said, we can form a supervision again once there's enough interest.
> One of the keys of the WiSH proposal is to focus on API level compatibility with WebSocket. But except for that point, it's a general proposal of application of HTTP semantics and HTTP/2's power for full-duplex messaging in the Web. The proposal (one done by Yutaka in 2014 and Van's one also) heavily depends on what the HTTP WG produces (specs, documents and possibly any kind of official/unofficial communications). So, I think there shouldn't be no doubt on need for close work with HTTP WG.
> 
> That said, I agree we need to have the right structure of the community to have the "best" work mode based on various metrics (level of interest for each proposal, their complexity, scope, etc.), and the IESG and the co-chairs are trying to do the best in making the right decision, I think.
> 
> Mark, does this post of yours imply that you're seeing HyBi ML as one effective representative of WS community at this point with HTTP WG chair hat on?

I'm seeing it as the obvious place to hold the discussion; it doesn't have any official status (beyond being the place where hybi happened before), but it's typical practice to keep IETF mailing lists open after a WG concludes, so that the interested parts of the community have a forum.

Cheers,

---

From: Van Catha <[email-masked]>
Date: Sun, 27 Nov 2016 09:36:24 -0500

So can we form a new WG then and focus on doing this right vs making
WebSocket2.  The focus earlier was to get the already coded clients and API
(websocket API) to be able to work with websockets layered on HTTP2/QUIC,
if we are in it for the long haul now we might as well form a new group and
create something more long term?

Long haul meaning maybe making websockets its own protocol, detaching from
HTTP2, having its own ALPN, etc.

On Sun, Nov 27, 2016 at 1:53 AM, Mark Nottingham <[email-masked]> wrote:

> > As noted by Barry in his mail about WG conclusion and as Mark said, we
> can form a supervision again once there's enough interest.
> with HTTP, including HTTP/2; there are several ways you could implement
> > One of the keys of the WiSH proposal is to focus on API level
> compatibility with WebSocket. But except for that point, it's a general
> proposal of application of HTTP semantics and HTTP/2's power for
> full-duplex messaging in the Web. The proposal (one done by Yutaka in 2014
> and Van's one also) heavily depends on what the HTTP WG produces (specs,
> documents and possibly any kind of official/unofficial communications). So,
> I think there shouldn't be no doubt on need for close work with HTTP WG.
> >
> > That said, I agree we need to have the right structure of the community
> to have the "best" work mode based on various metrics (level of interest
> for each proposal, their complexity, scope, etc.), and the IESG and the
> I'm seeing it as the obvious place to hold the discussion; it doesn't have
> any official status (beyond being the place where hybi happened before),
> but it's typical practice to keep IETF mailing lists open after a WG
> concludes, so that the interested parts of the community have a forum.
>
> Cheers,
>
>
> --

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 27 Nov 2016 21:18:50 +0000

--------
In message <[email-masked]>

>create something more long term?

Apologies for asking a stupid question, but isn't that exactly what QUIC
is all about in the first place ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 30 Nov 2016 19:29:53 +0000

QUIC's charter doesn't have anything directly to do with WebSockets.  But I agree that since WebSockets came from a different WG, it might be a reasonable question to the ADs whether that working group should be rechartered to do an HTTP/2 or HTTP/QUIC port.

HTTP/2 makes many of the pre-WebSocket solutions to this problem space much cheaper.  QUIC will probably make it even more so.  If there are people who feel strongly that WebSockets still meet a need over a modern HTTP, I'm happy to read and occasionally comment, but I don't feel called to be integral to that work.

-----Original Message-----
From: Poul-Henning Kamp [mailto:[email-masked]] 
Sent: Sunday, November 27, 2016 1:19 PM
To: Van Catha <[email-masked]>
Cc: Mark Nottingham <[email-masked]>; Takeshi Yoshino <[email-masked]>; Willy Tarreau <[email-masked]>; Andy Green <[email-masked]>; [email-masked] Group <[email-masked]>; Wenbo Zhu <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: WiSH: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols (HTTP)

--------
, Van Catha writes:

>WebSocket2.  The focus earlier was to get the already coded clients and 
>API (websocket API) to be able to work with websockets layered on 
>HTTP2/QUIC, if we are in it for the long haul now we might as well form 
>a new group and create something more long term?

Apologies for asking a stupid question, but isn't that exactly what QUIC is all about in the first place ?

---

From: Van Catha <[email-masked]>
Date: Wed, 30 Nov 2016 22:06:38 -0500

> HTTP/2 makes many of the pre-WebSocket solutions to this problem space
much cheaper.  QUIC will probably make it even more so.
> If there are people who feel strongly that WebSockets still meet a need
over a modern HTTP, I'm happy to read and occasionally comment,
> but I don't feel called to be integral to that work.

At this point I wonder if there is a need for a separate solution.  Making
websockets go over HTTP2 seems to bring up the bureaucratic problems of the
past.  If websocket can be its own thing, then have an implementation for
HTTP2 and another one for QUIC, I think it can see more success.  The
reason for this is both protocols (HTTP2 and QUIC) have their own way of
trammiting data frames.  Websocket in this case should become an
alternative frame type, basically one carrying a binary payload.  All the
previous baggage of websocket1 can be dropped like the distinction between
text/binary frames as well as ping/pong and close. So we end up with just 1
frame type.

Compression would be left up to the client js / server code.  The
websocket2 would be a super simple two way binary stream.

This is pretty self explanatory, the major problem is how you do handle
authentication now.

On Wed, Nov 30, 2016 at 2:29 PM, Mike Bishop <[email-masked]>
wrote:

> QUIC's charter doesn't have anything directly to do with WebSockets.  But
> I agree that since WebSockets came from a different WG, it might be a
> reasonable question to the ADs whether that working group should be
> rechartered to do an HTTP/2 or HTTP/QUIC port.
>
> HTTP/2 makes many of the pre-WebSocket solutions to this problem space
> much cheaper.  QUIC will probably make it even more so.  If there are
> people who feel strongly that WebSockets still meet a need over a modern
> HTTP, I'm happy to read and occasionally comment, but I don't feel called
> to be integral to that work.
>
> Cc: Mark Nottingham <[email-masked]>; Takeshi Yoshino <[email-masked]>;
> Willy Tarreau <[email-masked]>; Andy Green <[email-masked]>;
> [email-masked] Group <[email-masked]>; Wenbo Zhu <
> [email-masked]>; Martin Thomson <[email-masked]>

---

From: Andy Green <[email-masked]>
Date: Thu, 01 Dec 2016 11:36:16 +0800

On Wed, 2016-11-30 at 22:06 -0500, Van Catha wrote:
> > HTTP/2 makes many of the pre-WebSocket solutions to this problem
> space much cheaper.  QUIC will probably make it even more so. 
> > If there are people who feel strongly that WebSockets still meet a
> need over a modern HTTP, I'm happy to read and occasionally comment, 
> At this point I wonder if there is a need for a separate solution. 
> Making websockets go over HTTP2 seems to bring up the bureaucratic
> problems of the past.  If websocket can be its own thing, then have
> an implementation for HTTP2 and another one for QUIC, I think it can 

Yeah.

> see more success.  The reason for this is both protocols (HTTP2 and
> QUIC) have their own way of trammiting data frames.  Websocket in
> this case should become an alternative frame type, basically one
> carrying a binary payload.  All the previous baggage of websocket1
> can be dropped like the distinction between text/binary frames as 

The logical ws implemention in JS needs that distinction to be carried
over the link though, as it lets you choose and TEXT differs on the
other end by enforcing only valid UTF-8 in the message.  So that's not
a ws 1 wire-implementation-only type thing that can just be binned.

> well as ping/pong and close. So we end up with just 1 frame type.
> 
> This is pretty self explanatory, the major problem is how you do
> handle authentication now.  

Negotiating (or asserting) the ws protocol to interpret the message
with is also still needed, again that is a logical ws thing in JS ws
api (and is useful).  Atm in ws 1 initing the ws link in The Real World
looks like this

 - tcp connection establishment

 - GET some html with js

 - 200 comes back with payload

 - js gets to execute

 - tcp connection establishment

 - JS sends an UPGRADE with protocols

 - 101 comes back with agreed protocol

 - you can send ws now

It's two tcp connections (browsers don't pipeline the UPGRADE because
it changes the connection type) and two roundtrips.

HTTP/2 would remove the second tcp connection establishment, and you
could also replace the next two UPGRADE / negotiation steps with an
HTTP/2 new stream open.  The server can send the client a list of ws
protocols it supports when the http/2 link was created.  The client
then just asserts the ws protocol being used as the first thing you
send on the newly opened stream, all being well it continues or if
invalid the stream is killed by the server.

That saves a tcp connection establishment and a roundtrip compared to
http/1 ws...

-Andy

> On Wed, Nov 30, 2016 at 2:29 PM, Mike Bishop
> <[email-masked]> wrote:
> > QUIC's charter doesn't have anything directly to do with
> > WebSockets.  But I agree that since WebSockets came from a
> > different WG, it might be a reasonable question to the ADs whether
> > that working group should be rechartered to do an HTTP/2 or
> > HTTP/QUIC port.
> > 
> > HTTP/2 makes many of the pre-WebSocket solutions to this problem
> > space much cheaper.  QUIC will probably make it even more so.  If
> > there are people who feel strongly that WebSockets still meet a
> > need over a modern HTTP, I'm happy to read and occasionally
> > comment, but I don't feel called to be integral to that work.
> > 
> > Cc: Mark Nottingham <[email-masked]>; Takeshi Yoshino <tyoshino@goog
> > le.com>; Willy Tarreau <[email-masked]>; Andy Green <[email-masked]>; i
> > [email-masked] Group <[email-masked]>; Wenbo Zhu <wenboz@g
> > oogle.com>; Martin Thomson <[email-masked]>

---

From: Van Catha <[email-masked]>
Date: Wed, 30 Nov 2016 23:03:25 -0500

> a ws 1 wire-implementation-only type thing that can just be binned.

You are in control of the server and client.  Why can you not just cast
from ArrayBuffer to String (on js side) if you know both your client and
server are sending strings.  Its an unnecessary burden to have both text
and binary frames, its also unnecessarily confusing.

The problem is UPGRADE got dropped in HTTP2. The only reasonable option
lefts seems to be Alt-Svc.  ws-quic, ws-http2 Alt-Svc can become a thing
perhaps.  The requirement of sending browser headers (HTTP2) becomes
difficult to reasonably handle.  Maybe it can work by opening the new
stream then the first message Websocket will send is the headers as if it
was a normal request (over the ws channel).

We do not have anything in HTTP2 to handle an UPGRADE currently.  If the
websocket stream is opened first (client made aware through Alt-Svc), well
we've already upgraded!

---

From: Andy Green <[email-masked]>
Date: Thu, 01 Dec 2016 12:51:00 +0800

On Wed, 2016-11-30 at 23:03 -0500, Van Catha wrote:
> You are in control of the server and client.  Why can you not just 

That is not always the case.  For example we have a user using IBM's
Watson Test-to-Speech cloud service

https://www.ibm.com/watson/developercloud/doc/text-to-speech/websockets
.shtml

He is in control of the client, but he's just another consumer of IBM's
ws protocol.  I guess there are an increasing number of examples of
this that the server is somebody's service and there are independent
clients that may connect to it using the published protocol.

> cast from ArrayBuffer to String (on js side) if you know both your
> client and server are sending strings.  Its an unnecessary burden to
> have both text and binary frames, its also unnecessarily confusing.

I see your point, but it's in the JS ws api implemented in the
browsers.  It's desirable that should work as much the same as possible
regardless of the transport.  Some other guys decided the JS api.

> The problem is UPGRADE got dropped in HTTP2. The only reasonable
> option lefts seems to be Alt-Svc.  ws-quic, ws-http2 Alt-Svc can
> become a thing perhaps.  The requirement of sending browser headers
> (HTTP2) becomes difficult to reasonably handle.  Maybe it can work by
> opening the new stream then the first message Websocket will send is
> the headers as if it was a normal request (over the ws channel).  

But we have to keep the negotiation roundtrip then.  Originally a major
thought for ws 1 UPGRADE was to confirm nonsupporting servers would
choke on it and fail the connection, hence the required hash
computation thing on both sides.  So they accepted early there would be
http headers and a roundtrip opening a ws connection.

Aside from ws protocol negotiation it also lets the ws connection have
a URL space, and the client to send cookies.

I think you can maybe just assert those things that are necessary from
the client, like URL path, cookies and ws protocol, when you open the
ws stream.  If the server is happy, it accepts and processes traffic on
the stream, which may be pipelined.  If not it kills the stream before
processing anything.

> We do not have anything in HTTP2 to handle an UPGRADE currently.  If
> the websocket stream is opened first (client made aware through Alt-
> Svc), well we've already upgraded!

Yes but if we can signal it is a "ws stream" when the stream is opened,
actually the concept of UPGRADE and its roundtrip can maybe be
eliminated in favour of the client just asserting things that either
fly with the server or gets the stream killed.

About auth, usually the browser has authenticated to get to the html +
scripts to make the ws connection that needs auth.  Maybe the ws stream
can be a logical child of that and share any cookies the parent has
that are operable on the ws URL path.

-Andy

> > comment, 
> > If
> > > > le.com>; Willy Tarreau <[email-masked]>; Andy Green <[email-masked]
> > >; i
> > > > In message <CAG-EYCiVExcyHLoXB1ixQCKduxUPTVOnVX1XrmFJ3b72Y8AAFg
> > @mai
> > > > l.gmail.com>
> > > > , Van Catha writes:
> > > >
> > > > >HTTP2/QUIC, if we are in it for the long haul now we might as
> > well
> > > > form
> > > > Apologies for asking a stupid question, but isn't that exactly
> > what
> > > > QUIC is all about in the first place ?
> > > >
> > > > --

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 1 Dec 2016 16:36:36 +0900

> a ws 1 wire-implementation-only type thing that can just be binned.

Yes indeed. Even if it could be binned, it shouldn't. The distinction 
between TEXT (UTF-8 only!) and BINARY may be of no concern at all to the 
transport infrastructure, but it is essential for the next layer above. 
Without this distinction, the risk that JS code monkeys get the 
conversion from text to binary and back wrong is just way too high. Even 
just not having to do the conversion by hand is very helpful, and having 
the character encoding nailed down makes different servers and clients 
much more interoperable.

Regards,   Martin.