Subject: [hybi] permessage-priority extension draft

From: Tobias Oberstein <[email-masked]>
Date: Thu, 16 Jan 2014 08:39:29 -0800

Alright, I have written up a draft for a "permessage-priority" WebSocket extension:

https://github.com/oberstet/permessage-priority/blob/master/draft-oberstein-hybi-permessage-priority.txt

In writing this, I came to the conclusion that "message prioritization" on a single WebSocket connection
in fact is orthogonal to multiplexing multiple (logical) WebSocket connections over a single physical one
(be it via WS-MUX or a variant of WS/SPDY).

So this proposal should not be seen as a competitive proposal to WS-MUX or WS/SPDY, but as ... something different;)

The motivation and use case is outlined in the introduction.

I'd appreciate any feedback, suggestions, corrections or even PRs (the draft is on GitHub in a dedicated repo).

If the draft doesn't conform to RFC requirements or formatting rules, please forgive - this is my first RFC.

Cheers,
/Tobias

PS: For one thing I don't know, how do I submit this draft with the IETF so it can follow the proper way?

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 17 Jan 2014 15:10:29 -0800

There is a collection of open issues here:
https://github.com/oberstet/permessage-priority/issues

One of those is how to make permessage-priority compatible with permessage-deflate.

I guess it can be done by maintaining a separate compression context per priority class.
This allows intermediaries to prioritize and reorder frames without problems. However,
it multiplies memory requirements.

Was there any discussion/plan about how permessage-deflate would have worked with MUX
when compression was applied to the physical WebSocket connection?

I mean, when compression was applied to the logical MUXed WS connections, there
would have been a separate compression context for each (logical) WS connection also,
and intermediaries can reMUX without recompressing.

But when applied to the physical WS?

/Tobias

> -----Ursprüngliche Nachricht-----
> Von: hybi [mailto:[email-masked]] Im Auftrag von Tobias Oberstein
> Gesendet: Donnerstag, 16. Januar 2014 17:39
> An: [email-masked]
> Betreff: [hybi] permessage-priority extension draft
> 
> In writing this, I came to the conclusion that "message prioritization" on a
> single WebSocket connection in fact is orthogonal to multiplexing multiple
> (logical) WebSocket connections over a single physical one (be it via WS-MUX
> or a variant of WS/SPDY).
> 
> So this proposal should not be seen as a competitive proposal to WS-MUX or
> WS/SPDY, but as ... something different;)
> 
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Adam Rice <[email-masked]>
Date: Thu, 23 Jan 2014 17:32:32 +0900

This part contains a contradiction:

Intermediaries unaware of the permessage-priority extension

      1. MUST NOT change the message fragmentation (further fragment or
      coalesce non-control frames)

      2. MUST preserve strict frame order while processing and
      forwarding non-control frames

      3. MUST ignore WebSocket base protocol rules as they apply to
      continuation frames and the FIN bit

   or

      1. MUST remove the "permessage-priority" element from the client's
      "Sec-WebSocket-Extensions" header in the opening handshake

If the intermediary is unaware of the extension, then there is no way for
it to follow the rules in extension specification.

I think we can extrapolate a more general "all or nothing" rule from this.
A WebSocket intermediary must either treat all data as opaque and pass it
through unchanged, or completely implement the specification and all
extensions in use.

It follows that a WebSocket intermediary that understands the protocol must
strip any unrecognised extensions from the client handshake.

Thinking about compatibility with permessage-deflate, I came to the
conclusion that RFC6455 is too optimistic about extension compatibility.
The default expectation should be that extensions are incompatible, and the
WebSocket Extension Name Registery should have a list of "Known Compatible
Extensions" rather than "Known Incompatible Extensions".

As an implementer, my personal preference would be to not implement
permessage-priority. It requires changes in almost every layer of the
stack, adding complexity and overhead. I see my job as to get bytes from
the Javascript send() call into the kernel send buffer as efficiently as
reasonably possible; ideally I would like to avoid any additional queuing
altogether, never mind re-ordering of queued messages.

Thanks,
Adam Rice

On 18 January 2014 08:10, Tobias Oberstein <[email-masked]>wrote:

> > I'd appreciate any feedback, suggestions, corrections or even PRs (the
> draft is
> > on GitHub in a dedicated repo).
> >
> > If the draft doesn't conform to RFC requirements or formatting rules,
> please
> > forgive - this is my first RFC.
> >
> > Cheers,
> > /Tobias
> >
> > PS: For one thing I don't know, how do I submit this draft with the IETF
> so it can
> > follow the proper way?
> >

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 23 Jan 2014 02:37:00 -0800

Regarding intermediaries: you are right, the text is confusing. After rereading, I think RFC6455 already covers this:

"An intermediary MUST NOT change the fragmentation of any message in the context
of a connection where extensions have been negotiated and the intermediary is not
aware of the semantics of the negotiated extensions."

What is probably missing is:

"An intermediary that is not aware of the semantics of negotiated extensions MUST ignore the rules that
normally apply to continuation frames (opcode 0) and FIN bit"

Since without such rule, a (verifying) intermediary unaware of the extension semantics might comply
by not changing fragmentation, but still continue checking and (falsely) failing the WS connection if
it believes the continuation frame / FIN rules haven't been followed. And those rules might have
been changed by the extension.

===

Getting bytes as "efficiently into kernel as possible": not sure I understand your goals. What about efficiency
in the broader picture, e.g. minimize bytes on wire?

If an JS app in browser does

for (var i = 0; i < 1000000; ++i) socket.send("hello");

there will be buffering in userland (browser). The kernel will deny taking more bytes if the uplink cannot keep up.

Without coalescing WS messages in userland, it'll result in 1000000 syscalls and in 1000000 TCP segments being
generated (with the overhead that comes with that .. both in CPU cycles and bytes on wire).

With TLS, the situation becomes even worse: it'll burn lots of CPU cycles because 1000000 TLS records are
created (in userland) and lots of overhead on the wire .. I did measure this recently
http://tavendo.com/blog/post/dissecting-websocket-overhead/

If the sending app is decoupled from the TLS encryption / record framing and the syscall to TCP socket send()
using a frame-based queue (be it prioritized or not) and a background sender, those issues can be solved.

And if there is frame-based queue, the cost of having the queue prioritized seems quite small.

Sure, an app could do similar by layering another layer on top of WS, where WS messages are simply the
"new frames", and implement above _inside_ the app. This seems highly inefficient and duplication of efforts.

And in the end: why do we have frames at all in WS if we don't make good use of them?  ;)

/Tobias

Von: Adam Rice [mailto:[email-masked]]
Gesendet: Donnerstag, 23. Januar 2014 09:33
An: Tobias Oberstein
Cc: [email-masked]
Betreff: Re: [hybi] permessage-priority extension draft

   or

If the intermediary is unaware of the extension, then there is no way for it to follow the rules in extension specification.

I think we can extrapolate a more general "all or nothing" rule from this. A WebSocket intermediary must either treat all data as opaque and pass it through unchanged, or completely implement the specification and all extensions in use.

It follows that a WebSocket intermediary that understands the protocol must strip any unrecognised extensions from the client handshake.

Thinking about compatibility with permessage-deflate, I came to the conclusion that RFC6455 is too optimistic about extension compatibility. The default expectation should be that extensions are incompatible, and the WebSocket Extension Name Registery should have a list of "Known Compatible Extensions" rather than "Known Incompatible Extensions".

As an implementer, my personal preference would be to not implement permessage-priority. It requires changes in almost every layer of the stack, adding complexity and overhead. I see my job as to get bytes from the Javascript send() call into the kernel send buffer as efficiently as reasonably possible; ideally I would like to avoid any additional queuing altogether, never mind re-ordering of queued messages.

Thanks,
Adam Rice

On 18 January 2014 08:10, Tobias Oberstein <[email-masked]<mailto:[email-masked]>> wrote:
There is a collection of open issues here:
https://github.com/oberstet/permessage-priority/issues

/Tobias

> Von: hybi [mailto:[email-masked]<mailto:[email-masked]>] Im Auftrag von Tobias Oberstein
> An: [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
hybi mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/hybi

---

From: "John A. Tamplin" <[email-masked]>
Date: Thu, 23 Jan 2014 14:22:36 -0500

> been changed by the extension.

I disagree - I don't think an extension should be allowed to change those
rules.  If you don't even have the framing, then really there is no
standard WebSocket protocol.

-- 
John A. Tamplin

---

From: "John A. Tamplin" <[email-masked]>
Date: Thu, 23 Jan 2014 14:27:23 -0500

> And in the end: why do we have frames at all in WS if we don't make good

The primary reason is so we can write streams of data we don't know the
size of up front without having to buffer the whole thing.  Aside from
that, there are benefits for small-memory senders to limit the buffers
required.

-- 
John A. Tamplin

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 23 Jan 2014 12:29:20 -0800

>I disagree - I don't think an extension should be allowed to change those rules.  If you don't even have the framing, then really there is no standard WebSocket protocol.

With permessage-priority, those rules still apply .. only on a per-priority class basis.

The frame sequence _for each priority class_ follows the same rules as the frame sequence in the base protocol.

Since the frames for different priority classes are interleaved, an intermediary that is unaware of the extension cannot apply the rules of the base protocol to the sequence of frames as generated by the extension.

I think if we disallow such things (for which I cannot find text in RFC6455), this will only lead to extensions reinventing things on an embedded level:

E.g. with permessage-priority, we could simply ignore RSV etc altogether, and embed own Opcode/Fin bits inside the frame payload, and just sent every single frame as a binary frame with RSV=0 and FIN=1 at the base protocol level.

I'm not sure if this ignoring of the extension facilities of the base protocol would be in the spirit of the original WebSocket design ..

/Tobias

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 23 Jan 2014 12:33:34 -0800

>The primary reason is so we can write streams of data we don't know the size of up front without having to buffer the whole thing.  Aside from that, there are benefits for small-memory senders to limit the >buffers required.

Wasn't fragmentation also indented to limit head-of-line blocking / channel occupation with extensions like WebSocket MUX?

From http://tools.ietf.org/html/draft-ietf-hybi-websocket-multiplexing-11

"
  o  Fragment a large message into smaller frames to prevent a large
      message in a logical channel occupying the physical connection and
      thus delaying messages in other logical channels.
"

[Sender algorithm]

This is similar to permessage-priority's use of fragmentation.

/Tobias

---

From: Adam Rice <[email-masked]>
Date: Fri, 24 Jan 2014 15:41:08 +0900

> that, there are benefits for small-memory senders to limit the buffers

Another legitimate use is to allow the timely sending of control frames.

For example, a server sending a large message down a slow link might want
to interrupt the message with a "Ping" every few minutes to elicit a "Pong"
from the client and keep the NAT mapping alive.

Adam Rice

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 24 Jan 2014 16:32:47 +0900

> from the client and keep the NAT mapping alive.

Wouldn't the NAT mapping be kept alive anyway as long as data is flowing?

Regards,   Martin.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 Jan 2014 15:03:54 +0900

On Fri, Jan 24, 2014 at 5:33 AM, Tobias Oberstein <
[email-masked]> wrote:

> that, there are benefits for small-memory senders to limit the >buffers
> required.
>
> Wasn't fragmentation also indented to limit head-of-line blocking /
> channel occupation with extensions like WebSocket MUX?
>
>
Right. When looking for the way to interleave frames but keeping message
semantics, we said "yeah, there's frame!"

But in evolution of the mux spec, I
- stopped using frame-wise information (extension data)
- stopped use of inconsistent opcode/FIN combination

As a result, in this spec, everything are encapsulated into normal sequence
of fragments, and intermediaries which don't understand mux are allowed to
fragment/coalesce them.

I'd recommend encapsulation of frames into an encapsulating message
considering integration with other extensions and operability with old
intermediaries as John said.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 Jan 2014 15:13:01 +0900

> Wouldn't the NAT mapping be kept alive anyway as long as data is flowing?

Not so important, but a close frame may be sent in the middle of big
message to notify the reason of closure to the peer than just closing the
underlying connection.

I don't remember, but maybe close frames were made interjectable not based
on concrete requirement but considering future extensibility and belief on
this kind of feature (signaling should be able to be sent independent of
main flow. or something...).

---

From: Adam Rice <[email-masked]>
Date: Mon, 27 Jan 2014 19:08:21 +0900

> Wouldn't the NAT mapping be kept alive anyway as long as data is flowing?
I was thinking of NAT devices that only refresh the mapping when they see a
packet in the "outgoing" direction.

However, I had forgotten that the ACKs from the client of the data being
sent from the server would be sufficient to keep the NAT mapping alive.

So yes, there should be no need for Pings as long as data is in transit.

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 29 Jan 2014 06:16:32 -0800

>As a result, in this spec, everything are encapsulated into normal sequence of fragments, and intermediaries which don't understand mux are allowed to fragment/coalesce them.

RFC6455 forbids intermediaries to fragment/coalesce in the presence of a WS extension they don't understand:

"   o  An intermediary MUST NOT change the fragmentation of any message
      in the context of a connection where extensions have been
      negotiated and the intermediary is not aware of the semantics of
      the negotiated extensions.
"

>I'd recommend encapsulation of frames into an encapsulating message considering integration with other extensions and operability with old intermediaries as John said.

RFC6455 explicitly allows interleaving of messages for extensions:

"   o  The fragments of one message MUST NOT be interleaved between the
      fragments of another message unless an extension has been
      negotiated that can interpret the interleaving.
"
 
and

"
   o  A fragmented message consists of a single frame with the FIN bit
      clear and an opcode other than 0, followed by zero or more frames
      with the FIN bit clear and the opcode set to 0, and terminated by
      a single frame with the FIN bit set and an opcode of 0.
"

In my reading, taking both of above together means:

The opcode/FIN bit rule (2nd above) MUST always apply, but an extension that allows message interleaving (1st above) MUST apply it on a per-message basis.

Which is exactly what "permessage-priority" does.

So I'd argue, that applying the opcode/FIN rule on a per-message basis in the presence of an extension defining message interleaving is in full compliance with RFC6455.

===

An intermediary that does not conform to RFC6455 is not "old", but simply broken;)

If we start to restrict ourself beyond what's allowed by RFC6455 because of broken implementations, this is a road to insanity;) Why have a spec then anyway?

Rgd integration with other extensions: interoperability of permessage-priority with permessage-deflate seems to require a "compression context per message".
Encapsulating does not seem to change that nor help ..

/Tobias