Subject: [hybi] Per-frame compression spec draft 06

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 9 Mar 2012 16:15:57 -0800

Hi all,

Here's a revised version of the per-frame compression spec. I'm planning to
rename it to draft-ietf-hybi-websocket-perframe-compression-00 after the
meeting.
http://tools.ietf.org/html/draft-tyoshino-hybi-websocket-perframe-deflate-06

http://tools.ietf.org/rfcdiff?url2=draft-tyoshino-hybi-websocket-perframe-deflate-06.txt
Based on the feedback comments for the CfC, I've done
- reordering of sections
- abstracting the framing method (COMP bit and how to build compressed
frames) as general per-frame compression scheme
- renaming COMP bit to "per-frame compressed" and giving more general
description for it in both the main texts and IANA consideration section

Thanks
Takeshi

---

From: John Tamplin <[email-masked]>
Date: Fri, 9 Mar 2012 19:56:29 -0500

On Fri, Mar 9, 2012 at 7:15 PM, Takeshi Yoshino <[email-masked]> wrote:

> Here's a revised version of the per-frame compression spec. I'm planning
> to rename it to draft-ietf-hybi-websocket-perframe-compression-00 after the
> meeting.
>

>From the summary here, I expected it to be described as two separate pieces:
 1) a generic per-frame compression extension, including a way to negotiate
the algorithm to be used
 2) a particular compression algorithm available by default (and required
to be supported) - deflate

Instead, it still looks like an extension just for deflate, using slightly
more general language.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Gabriel Montenegro <[email-masked]>
Date: Sat, 10 Mar 2012 01:57:34 +0000

Thanks Takeshi,

Just a clarification that renaming the draft to draft-ietf-hybi-websocket-perframe-compression-00, of course, depends on the result of the current consensus call on adoption of this draft as a WG document. That call runs until next Mon March 12.

Gabriel

From: [email-masked] [mailto:[email-masked]] On Behalf Of John Tamplin
Sent: Friday, 09 March, 2012 16:56
To: Takeshi Yoshino
Cc: [email-masked]
Subject: Re: [hybi] Per-frame compression spec draft 06

On Fri, Mar 9, 2012 at 7:15 PM, Takeshi Yoshino <[email-masked]<mailto:[email-masked]>> wrote:
Here's a revised version of the per-frame compression spec. I'm planning to rename it to draft-ietf-hybi-websocket-perframe-compression-00 after the meeting.
http://tools.ietf.org/html/draft-tyoshino-hybi-websocket-perframe-deflate-06

http://tools.ietf.org/rfcdiff?url2=draft-tyoshino-hybi-websocket-perframe-deflate-06.txt
Based on the feedback comments for the CfC, I've done
- reordering of sections
- abstracting the framing method (COMP bit and how to build compressed frames) as general per-frame compression scheme
- renaming COMP bit to "per-frame compressed" and giving more general description for it in both the main texts and IANA consideration section

From the summary here, I expected it to be described as two separate pieces:
 1) a generic per-frame compression extension, including a way to negotiate the algorithm to be used
 2) a particular compression algorithm available by default (and required to be supported) - deflate

Instead, it still looks like an extension just for deflate, using slightly more general language.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 10 Mar 2012 12:50:04 -0800

On Fri, Mar 9, 2012 at 16:56, John Tamplin <[email-masked]> wrote:
...

> to be supported) - deflate

Sorry to be unclear.

> Instead, it still looks like an extension just for deflate, using slightly

Yes. I just factored out texts around comp bit handling so that other
extension specs can refer it.

----

Thanks for the clarification, Gabriel.

---

From: John Tamplin <[email-masked]>
Date: Tue, 13 Mar 2012 10:29:07 -0400

> extension specs can refer it.

Ok, could we get input from others on this?  Do we want something like this
document which defines only the per-frame deflate compression extension,
and any new per-frame compression algorithms would require their own
extension?  Or would we prefer to have a generic per-frame compression
extension, which allows negotiation of the compression algorithm and
parameters while mandating required support for the deflate algorithm as
proposed?

To me, the proposed deflate algorithm falls into the "better than nothing"
category, and really requires some way of detecting incompressible content
since otherwise it could increase the size.  Maintaining compression state
across frames for the most part mitigates that concern, as pretty much
anything except already compressed content will show improvement.  Also, I
expect state of the art for compression to improve over time (both new
algorithms and faster hardware making existing algorithms feasible) so I
expect we will want to support multiple compression algorithms.  For this
reason, I would prefer a more generic and extensible per-frame compression
extension.

---

From: "Arman Djusupov" <[email-masked]>
Date: Tue, 13 Mar 2012 18:08:28 +0200

The spec in the current state does not cover any methods other than the specific deflate-frame extension. Yes there are other compression methods available and there are alternative ways to apply the same DEFLATE compression method, for example flush sync can be used at the boundaries of the message rather than at frame boundary. But I’m not sure if the specification can define how to detect incompressible content, such functionality can/should be implementation specific. Sometimes even if the data appears to be incompressible it makes sense to include it into the sliding window and there is a chance it will produce matches with subsequent data. In general DEFLATE does not usually inflate frames by much, unless they are totally incompressible and extremely small.

I was thinking more about a specification that would define the way compression extensions are to be applied and reserves the RSV1 bit as compression flag for all extensions. In terms of negotiation, we already have a way to negotiate the compression method when the extension is being negotiated. As long as each compression method has its own extension name the WebSocket handshake will negotiate it anyway. Extensions can be named deflate-frame, bzip2-message, deflate-message-stateful and so on. What spec has to offer is some standard way of using those extensions. e.g. define that “compression is applied on payload prior masking”, “frame header is not compressed” and so on. Then all new compression extension specs would be able to reference the common specification defining the general principles on how it’s applied and negotiated, but each of them could have a different name and use a different compression method.

With best regards,

Arman

From: [email-masked] [mailto:[email-masked]] On Behalf Of John Tamplin
Sent: Tuesday, March 13, 2012 4:29 PM
To: Takeshi Yoshino
Cc: [email-masked]; Gabriel Montenegro
Subject: Re: [hybi] Per-frame compression spec draft 06

On Sat, Mar 10, 2012 at 3:50 PM, Takeshi Yoshino <[email-masked]> wrote:

Yes. I just factored out texts around comp bit handling so that other extension specs can refer it.

Ok, could we get input from others on this?  Do we want something like this document which defines only the per-frame deflate compression extension, and any new per-frame compression algorithms would require their own extension?  Or would we prefer to have a generic per-frame compression extension, which allows negotiation of the compression algorithm and parameters while mandating required support for the deflate algorithm as proposed?

To me, the proposed deflate algorithm falls into the "better than nothing" category, and really requires some way of detecting incompressible content since otherwise it could increase the size.  Maintaining compression state across frames for the most part mitigates that concern, as pretty much anything except already compressed content will show improvement.  Also, I expect state of the art for compression to improve over time (both new algorithms and faster hardware making existing algorithms feasible) so I expect we will want to support multiple compression algorithms.  For this reason, I would prefer a more generic and extensible per-frame compression extension.

---

From: John Tamplin <[email-masked]>
Date: Tue, 13 Mar 2012 12:19:31 -0400

On Tue, Mar 13, 2012 at 12:08 PM, Arman Djusupov <[email-masked]> wrote:

> I was thinking more about a specification that would define the way
> compression extensions are to be applied and reserves the RSV1 bit as
> compression flag for all extensions. In terms of negotiation, we already
> have a way to negotiate the compression method when the extension is being
> negotiated. As long as each compression method has its own extension name
> the WebSocket handshake will negotiate it anyway. Extensions can be named
> deflate-frame, bzip2-message, deflate-message-stateful and so on. What spec
> has to offer is some standard way of using those extensions. e.g. define
> that “compression is applied on payload prior masking”, “frame header is
> not compressed” and so on. Then all new compression extension specs would
> be able to reference the common specification defining the general
> principles on how it’s applied and negotiated, but each of them could have
> a different name and use a different compression method.
>
**
One complication of using treating each compression algorithm as a separate
WS extension is how do you negotiate exactly one of them?  Sure, you can
define the extensions as mutually incompatible, but that could complicate
server implementations to have to keep track of which ones use the same bit.

I also think it makes clearer specs to separate the mechanism of per-frame
compression from which algorithm to use.

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 14 Mar 2012 14:35:45 +0900

On Wed, Mar 14, 2012 at 01:19, John Tamplin <[email-masked]> wrote:

> One complication of using treating each compression algorithm as a
> separate WS extension is how do you negotiate exactly one of them?  Sure,
> you can define the extensions as mutually incompatible, but that could
> complicate server implementations to have to keep track of which ones use
> the same bit.
>

So, what do we do with multiple choices? Nesting?

Sec-WebSocket-Extensions: perframe-compress;
method="framewise-stateless-deflate; foo=bar; baz=123,
messagewise-stateful-deflate; e=mc2"

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 14 Mar 2012 10:37:23 +0200

IMO a server should be ready to handle situations where the use of the one extension prohibits the use of another. Just like with HTTP Accept-Encoding header “gzip, deflate” or with mux channel compression “deflate-frame, mux, deflate-frame” where deflate-frame can be applied either before or after mux. Grouping up supported extensions into mutuality exclusive groups would probably be simpler than parsing complex extension headers and creating specifications for negotiating the extensions of each type. 

The server can check the extension header for the presence of supported extensions from the same group. Once a supported extension found, other extensions of the same group are excluded. Such grouping of extensions can even be defined as comma separated list of extension names sorted by the order of preference.

With best regards,

Arman

From: John Tamplin [mailto:[email-masked]] 
Sent: Tuesday, March 13, 2012 6:20 PM
To: Arman Djusupov
Cc: Takeshi Yoshino; [email-masked]; Gabriel Montenegro
Subject: Re: [hybi] Per-frame compression spec draft 06

One complication of using treating each compression algorithm as a separate WS extension is how do you negotiate exactly one of them?  Sure, you can define the extensions as mutually incompatible, but that could complicate server implementations to have to keep track of which ones use the same bit.

I also think it makes clearer specs to separate the mechanism of per-frame compression from which algorithm to use.

---

From: John Tamplin <[email-masked]>
Date: Wed, 14 Mar 2012 11:09:21 -0400

> messagewise-stateful-deflate; e=mc2"

I was thinking a separate header for compression negotiation, but that
could work also.

---

From: John Tamplin <[email-masked]>
Date: Wed, 14 Mar 2012 11:11:15 -0400

On Wed, Mar 14, 2012 at 4:37 AM, Arman Djusupov <[email-masked]> wrote:

> IMO a server should be ready to handle situations where the use of the one
> extension prohibits the use of another. Just like with HTTP Accept-Encoding
> header “gzip, deflate” or with mux channel compression “deflate-frame, mux,
> deflate-frame” where deflate-frame can be applied either before or after
> mux. Grouping up supported extensions into mutuality exclusive groups would
> probably be simpler than parsing complex extension headers and creating
> specifications for negotiating the extensions of each type.
>
> **
>
> The server can check the extension header for the presence of supported
> extensions from the same group. Once a supported extension found, other
> extensions of the same group are excluded. Such grouping of extensions can
> even be defined as comma separated list of extension names sorted by the
> order of preference.
>
@Greg - IIRC, you were one of the ones who objected to requiring low-level
code to know such details due to server architectures you had in mind -- is
that accurate and if so still the case?

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 14 Mar 2012 18:05:00 +0200

The server will anyway have to go into such details. It can’t simply trust the Sec-WebSocket-Extensions header sent by the client and apply all extensions in the order specified by the client. If the server blindly trusts the order specified by the client, malicious or misbehaving clients will be able to trick the server in combining extensions in the worst possible order or in applying incompatible extensions together.

With best regards,

Arman

From: John Tamplin [mailto:[email-masked]] 
Sent: Wednesday, March 14, 2012 5:11 PM
To: Arman Djusupov; Greg Wilkins
Cc: Takeshi Yoshino; [email-masked]; Gabriel Montenegro
Subject: Re: [hybi] Per-frame compression spec draft 06

@Greg - IIRC, you were one of the ones who objected to requiring low-level code to know such details due to server architectures you had in mind -- is that accurate and if so still the case?