Subject: [hybi] status of real world permessage-deflate implementations?

From: Joakim Erdfelt <[email-masked]>
Date: Wed, 14 May 2014 11:11:38 -0700

So, I've been testing the server and client side implementations of various
permessage-deflate implementations and come to realize that there is no
consistency (yet).

In Chrome 34, its client side decompress (received frames) is different
than pywebsocket client decompress (received frames) which is different
from autobahn testsuite's handling.

Been testing our Jetty side implementation, and for large compressed
messages (been using the Gutenberg project's text of "A Connecticut Yankee
in King Arthur's Court, Complete, by Mark Twain" as my data).
http://www.gutenberg.org/ebooks/86.txt.utf-8

All tests are done with client upgrading to Jetty side using ..

offer: permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
negotiated: permessage-deflate

I can make pywebsocket client happy, and it can send/receive single frame
messages or fragmented messages (using either 16k and 32k max frame sizes)
just fine.
However, Chrome 34 errors out consistently if it receives a large
fragmented message that is compressed.
The inter-frame handling of the tail bytes (0x00, 0x00, 0xFF, 0xFF) appears
to be different.
In the last spec document,
http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18 its
clear that we must include the tail bytes.
However, Chrome doesn't like that and fails the decompress w/abnormal
closure.
Inspection says:  failed to inflate a frame
If I strip the tail bytes for Chrome 34, then its happy and pywebsocket is
now broken.
autobahn apparently doesn't care, one way or another, it accepts (happily)
both with and without tail bytes between frames.

--
Joakim Erdfelt <[email-masked]>
webtide.com <http://www.webtide.com/> - intalio.com/jetty
Expert advice, services and support from from the Jetty & CometD experts
eclipse.org/jetty - cometd.org

---

From: Kevin <[email-masked]>
Date: Wed, 14 May 2014 14:41:33 -0400

On 5/14/2014 2:11 PM, Joakim Erdfelt wrote:
> So, I've been testing the server and client side implementations of 
> various permessage-deflate implementations and come to realize that 
> there is no consistency (yet).
>
> In Chrome 34, its client side decompress (received frames) is 
> different than pywebsocket client decompress (received frames) which 
> is different from autobahn testsuite's handling.
>
> messages (been using the Gutenberg project's text of "A Connecticut 
> Yankee in King Arthur's Court, Complete, by Mark Twain" as my data).
> I can make pywebsocket client happy, and it can send/receive single 
> frame messages or fragmented messages (using either 16k and 32k max 
> frame sizes) just fine.
> The inter-frame handling of the tail bytes (0x00, 0x00, 0xFF, 0xFF) 
> appears to be different.
> http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18 
> its clear that we must include the tail bytes.
> If I strip the tail bytes for Chrome 34, then its happy and 
> pywebsocket is now broken.
> autobahn apparently doesn't care, one way or another, it accepts 
> (happily) both with and without tail bytes between frames.
>
> --
> Joakim Erdfelt <[email-masked] <mailto:[email-masked]>>
> <http://intalio.com/jetty>
> eclipse.org/jetty <http://eclipse.org/jetty/> - cometd.org 
> <http://cometd.org/>
>
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
Let's then look at chrome.  Obviously that is the source of the 
problem.  Obviously something is wrong with the client side handshaking 
process.  I feel this should be explored.

-- 
Kevin

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 14 May 2014 12:22:09 -0700

Hi Joakim,

couple of thoughts ..

1)
The latest Autobahn testsuite did include testing of various deflate parameter sets (section 12)

http://autobahn.ws/testsuite/reports_20140314/clients/index.html

but does NOT yet include testing permessage-deflate combined with fragmentation and/or multiple deflate blocks.

That is missing. I guess we need it.

2)
FWIW, AutobahnPython allows to set auto-fragmentation of outgoing messages via

WebSocketClientFactory.setProtocolOptions(autoFragmentSize = 1024)

WebSocketServerFactory.setProtocolOptions(autoFragmentSize = 1024)

However, this works like this: first compress the complete message in one go and _then_ fragment.

This means, there will only be a _single_ deflate block that is then splitted between mulitple frames (see (B) below)

3)
In other words, there is another dimension: multiple deflate blocks in one or multiple frames  (see (C) below).

(A)
[FRAME 1 : DEFLATE BLOCK 1 - without tail bytes]

(B)
[FRAME 1 : DEFLATE BLOCK 1 (Part 1/2)] [FRAME 2 : DEFLATE BLOCK 1 (Part 2/2)- without tail bytes]

(C)
[FRAME 1 : DEFLATE BLOCK 1 with tail bytes] [FRAME 2 : DEFLATE BLOCK 2 without tail bytes]

Regarding the spec:

"""

Note that for non-final fragments, the removal of 0x00 0x00 0xff 0xff must not be done.
"""

I am now wondering which of the following would be valid:

(D)
[FRAME 1 : DEFLATE BLOCK 1 with tail bytes] [FRAME 2 : empty payload]

(E)
[FRAME 1 : DEFLATE BLOCK 1 (Part 1/2)] [FRAME 2 : DEFLATE BLOCK 1 (Part 2/2) with tail bytes | DEFLATE BLOCK 2 without tail bytes]

(F)
[FRAME 1 : DEFLATE BLOCK 1 with tail bytes | DEFLATE BLOCK 2 without tail bytes]

and whether the intention of the spec was more like

"""
The tail bytes of the last (and only the last) deflate block must be stripped.
"""

That is: no reference to frames at all, only to deflate blocks (last or non-last).

Anyway: which of (A) - (F) is intended to be valid/invalid?

My current understanding: all but (D) are valid.

/Tobias

Von: hybi [mailto:[email-masked]] Im Auftrag von Joakim Erdfelt
Gesendet: Mittwoch, 14. Mai 2014 20:12
An: [email-masked]
Betreff: [hybi] status of real world permessage-deflate implementations?

So, I've been testing the server and client side implementations of various permessage-deflate implementations and come to realize that there is no consistency (yet).

In Chrome 34, its client side decompress (received frames) is different than pywebsocket client decompress (received frames) which is different from autobahn testsuite's handling.

Been testing our Jetty side implementation, and for large compressed messages (been using the Gutenberg project's text of "A Connecticut Yankee in King Arthur's Court, Complete, by Mark Twain" as my data).
http://www.gutenberg.org/ebooks/86.txt.utf-8

I can make pywebsocket client happy, and it can send/receive single frame messages or fragmented messages (using either 16k and 32k max frame sizes) just fine.
However, Chrome 34 errors out consistently if it receives a large fragmented message that is compressed.
The inter-frame handling of the tail bytes (0x00, 0x00, 0xFF, 0xFF) appears to be different.
In the last spec document, http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18 its clear that we must include the tail bytes.
However, Chrome doesn't like that and fails the decompress w/abnormal closure.
Inspection says:  failed to inflate a frame
If I strip the tail bytes for Chrome 34, then its happy and pywebsocket is now broken.
autobahn apparently doesn't care, one way or another, it accepts (happily) both with and without tail bytes between frames.

--
Joakim Erdfelt <[email-masked]<mailto:[email-masked]>>
webtide.com<http://www.webtide.com/> - intalio.com/jetty<http://intalio.com/jetty>
Expert advice, services and support from from the Jetty & CometD experts
eclipse.org/jetty<http://eclipse.org/jetty/> - cometd.org<http://cometd.org/>

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 15 May 2014 15:15:03 +0200

Hi Joakim, Tobias,

On Wed, May 14, 2014 at 9:22 PM, Tobias Oberstein <
[email-masked]> wrote:

> but does NOT yet include testing permessage-deflate combined with
> fragmentation and/or multiple deflate blocks.
>
>
>
> This means, there will only be a _single_ deflate block that is then
> splitted between mulitple frames (see (B) below)
>
>
>
> 3)
>

This doesn't work since the concatenation of payloads ends with the tail
bytes. On the receiver, it adds the tail bytes and decodes and sees an
error. FRAME 2 must include 0x00. Please see
https://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-18#section-8.2.3.6

>
>
> (E)
>
> [FRAME 1 : DEFLATE BLOCK 1 (Part 1/2)] [FRAME 2 : DEFLATE BLOCK 1 (Part
> 2/2) with tail bytes | DEFLATE BLOCK 2 without tail bytes]
>
>
>

This is fine.

> [FRAME 1 : DEFLATE BLOCK 1 with tail bytes | DEFLATE BLOCK 2 without tail

This is also fine.

> The tail bytes of the last (and only the last) deflate block must be

Correct.

> That is: no reference to frames at all, only to deflate blocks (last or

Thanks for suggestion. I'll review my texts about this.

> My current understanding: all but (D) are valid.

Right.

>
>
> /Tobias
>
>
>
>
>
> *Von:* hybi [mailto:[email-masked]] *Im Auftrag von *Joakim Erdfelt
> *Gesendet:* Mittwoch, 14. Mai 2014 20:12
> *An:* [email-masked]
> *Betreff:* [hybi] status of real world permessage-deflate implementations?
>
>
>
> various permessage-deflate implementations and come to realize that there
> is no consistency (yet).
>
>
>

---

From: Joakim Erdfelt <[email-masked]>
Date: Thu, 15 May 2014 13:03:52 -0700

Some more updates...

This is looking like more and more like a *VERY SPECIFIC* chrome bug.
Would be happy to migrate this to a chrome specific bug tracker, but I just
wanted to update the rest of the group first on these results.

My test data:
http://joakim.erdfelt.com/jetty/websocket-intratail/

These are all from server, not masked, destined for client that is being
tested.
Each file is a single, complete, frame.
Each directory represents a single overall message.
Every message is the same...
 UTF8 e-book -> "A Connecticut Yankee in King Arthur's Court, Complete, by
Mark Twain"
 From Project Gutenberg: http://www.gutenberg.org/ebooks/86.txt.utf-8

Extension Negotiation:

  There were 2 overall client offer styles seen.
   a) permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
   b) permessage-deflate; client_max_window_bits

  Server always negotiated as: permessage-deflate

Once the connection is established, these raw frames are sent to the
client, as-is, with no extra interpretation by the server.
Connection is plain, http -> ws:// not https or wss:// or spdy.

  /by-spec-*/        - intra-frame tail is present (per spec)
  /by-spec-32k/      - fragmented, 20 frames,  32k max frame size
  /by-spec-128k/     - fragmented,  5 frames, 128k max frame size
  /by-spec-1-frame/  - not fragmented, 1 frame

  /strip-intratails-*/        - intra-frame tail is removed (in violation
of spec)
  /strip-intratails-32k/      - fragmented, 20 frames,  32k max frame size
  /strip-intratails-128k/     - fragmented,  5 frames, 128k max frame size
  /strip-intratails-1-frame/  - not fragmented, 1 frame

Tested Clients:

  A) PyWebSocket Trunk (@revision 798) - echo_client.py - Python 2.7.5
  B) Chrome 34.0.1847.132 - on Fedora 20 - Javascript/HTML
  C) Chromium 34.0.1847.132-1.fc20 - on Fedora 20 - Javascript/HTML
  D) Chrome 34.0.1847.131 - on OSX/10.9.2 (older MBP) - Javascript/HTML
  E) Chrome Canary 37.0.1991.0 - on OSX/10.9.2 (older MBP) - Javascript/HTML
  F) Chrome 34.0.1847.137 - on OSX/10.9.2 (first gen rMBP) - Javascript/HTML
  G) Chrome Canary 37.0.1991.0 - on OSX/10.9.2 (first gen rMBP) -
Javascript/HTML
  H) Chrome Canary 37.0.1994.2 - on OSX/10.9.2 (first gen rMBP) -
Javascript/HTML
  I) Chrome 34.0.1847.137 m - Windows 7 SP1 (x64) - Javascript/HTML
  J) Chrome 34.0.1847.131 m - Windows 8.1 (x64) - Javascript/HTML

Results:

  /by-spec-1-frame/          - all clients pass
  /by-spec-32k/              - B fails, all of the rest are successful.
  /by-spec-128k/             - all clients pass
  /strip-intratails-1-frame/ - all clients pass
  /strip-intratails-32k/     - only B passes, all other clients fail
  /strip-intratails-128k/    - all clients fail.

When a failure occurs, its always on the second frame sent.

As these results are just mind-boggling, I went ahead and installed a fresh
copy of fedora 20 in a vm (oracle virtualbox), updated all of the packages
and tried again with Chrome 34.0.1847.132.  Same results.
Important note! - When upgrading to stable version 34.0.1847.137-1 then
chrome behaves properly again.

As for the Fedora 20 main machine, it has a disabled firewall and selinux
(so those can be ruled out as the cause).

On Thu, May 15, 2014 at 6:15 AM, Takeshi Yoshino <[email-masked]>wrote:

>> 2/2)- without tail bytes]
>>
>>
>>
>> (C)
>>

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 21 May 2014 13:35:03 +0900

Thanks Joakim for the detailed report.

Are you using Chrome/Chromium with
enable-experimental-web-platform-features turned on or off?

On Thu, May 15, 2014 at 10:03 PM, Joakim Erdfelt <[email-masked]> wrote:

> Would be happy to migrate this to a chrome specific bug tracker, but I
> just wanted to update the rest of the group first on these results.
>
> My test data:

I tested by-spec-1-frame and by-spec-128k on my workstation with
enable-experimental-webplatform-features flag turned off.

- Chrome 34.0.1847.132 Ubuntu
- Chrome 37.0.2005.1 Windows 7
- Chrome 34.0.1847.137 Windows 7

All of them processed your inputs without any error.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 27 May 2014 11:09:08 +0900

> Thanks for suggestion. I'll review my texts about this.

I was afraid that this is not well clarified, but this text in -18 should
be enough.