Subject: Re: [hybi] Payload only compression extension, again

From: Greg Wilkins <[email-masked]>
Date: Mon, 2 May 2011 19:27:31 +1000

In jetty trunk I have an implementation of an extension I've called
x-deflate-frame for now.

It uses 1 reserve bit to indicate if the payload is compressed.
If the payload is compressed, then it is sent as:

    uncompressed-length  deflated-data

The uncompressed-length is encoded in the same format as the websocket
frame length.  The uncompressed length is sent so that a buffer may be
allocated before starting to inflate the data.

The extension supports a minLength parameter (default 64) and will not
attempt to compress payloads that are smaller than that length.

When compressing the data, if the combined uncompressed-length and
deflated-data is longer than the original data, then the original data
frame is sent instead.

Currently I'm just using the standard java Inflater/Deflater  gzip
classes with a reset between frames.  Not exactly sure what that means
with regards to the algorithm details that Takeshi has specified, but
I think they are close.

cheers

---

From: John Tamplin <[email-masked]>
Date: Mon, 2 May 2011 10:51:16 -0400

> allocated before starting to inflate the data.

 Is sufficient value gained by this to justify the cost?

> attempt to compress payloads that are smaller than that length.

What is the value of making this configurable on a per-connection basis?
 Will this be exposed to the JS API?  If not, how will the browser come up
with a value here?

> deflated-data is longer than the original data, then the original data

Maintaining compression state across frames (which I think is required for
this to be useful) and doing this requires either being able to save/restore
the compression state or saying that even the case of the compressed data
growing in size and being sent uncompressed, it still affects the
compression state.  The former imposes some constraints on the
implementation, and the latter imposes a performance penalty on the receiver
(and the sender when it decides for other reasons, such as knowing the type
of data being sent, not to compress the data).

> with regards to the algorithm details that Takeshi has specified, but

See the compression experiments I posted last May - if you don't maintain
compression state between frames, then you lose most of the benefit of
compression.  Being able to send uncompressed frames cheaply avoids the cost
of this, but doesn't fix not getting the benefit of compression.

Unfortunately, Java's default deflate implementation doesn't allow this, as
you need to flush out the compressed bits at the end of a frame (though you
can play tricks with changing the compression level to 0 and back, but that
seems non-portable).  I believe you can do it with jzlib though.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 May 2011 07:59:36 +1000

>  Is sufficient value gained by this to justify the cost?

The cost is trivial.

The benefit is to be able to inflate into an allocated buffer of the
correct size.  Without knowing the size, then you will either over
allocate, or will need a growing buffer, which involves data copies.
  As the inflated size may be many many times larger than the deflated
size, I think that it is worth investing 1,3 or 9 bytes to avoid
another arbitrary limit or extra copying.

I started prototyping without this length and it was a good
simplification to the code when I decided to try it.  Sure there are
libraries that will hide the variable/unknown length better than the
java libraries, but they are just smoothing over significant costs in
allocation and data copying.

>  Will this be exposed to the JS API?  If not, how will the browser come up

Probably no great need to have this configurable.  I was mostly
testing extension parameters with this (but also modelling gzip
filters which do appear to make this configurable).   If the client
does not send the parameter, the server still responds with the
default value, so the client know it.

> of data being sent, not to compress the data).

The later also has cost on the server as it prevents simple buffer
management.    If can deflate into a buffer the same size as the raw
data, then you avoid reallocations and data copies - which are the
last things you want to do if you have discovered that the deflation
is increasing the size.

> seems non-portable).  I believe you can do it with jzlib though.

Yes, the java std lib does appear lacking in this regard.  As I said,
I'm just using it for prototyping and don't think that it's
deficiencies should greatly influence any effort to standardise such
an extension - specially if better libraries are available (argh! they
have license clashes and eclipse IP issues... still not a problem for
this group).

cheers

---

From: John Tamplin <[email-masked]>
Date: Mon, 2 May 2011 18:02:10 -0400

> have license clashes and eclipse IP issues... still not a problem for

My point was I see little value in having a compression extension which
starts the compression state afresh with every frame, as on real-world
applications this gives poor performance.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 May 2011 08:52:54 +1000

> applications this gives poor performance.

+1

My extension is more about prototyping plugable extension that work
well in a chain of extension, plus handling the negotiation and per
frame issues.    I'll follow the lead of the others/experience on the
exact algorithms to use (even though I think that is going to cause me
endless headaches with mismatched licenses and getting IP audits done
at eclipse :(

cheers

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 10 May 2011 16:30:51 +0900

I can change my proposal to accept last blocks (blocks with BFINAL=1) to
allow use of java.util.zip. In my proposal, receiver expects 3 header bits
0b000 (and padding to byte boundary) on the tail of the application data
portion. Not to complicate receiver logic much, I'd make change like this
- sender: send(<compressed data flushed with Z_FINISH> + "\x00")
- receiver: inflate(received_data + "\x00\x00\xff\xff"). every time inflater
encounters end of stream (the last octet of a last block), renew inflater
and keep decompressing the remainder.
or
- sender: send(<compressed data flushed with Z_FINISH>)
- receiver: inflate(received_data). every time inflater encounters end of
stream, renew inflater and keep decompressing the remainder. if the end of
received_data is not end of stream, inflate("\x00\x00\xff\xff").

Either algorithm works for Z_SYNC_FLUSH-ed and "\x00\x00\xff\xff" stripped
frame, too.

Takeshi

On Tue, May 3, 2011 at 07:52, Greg Wilkins <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>