Subject: [hybi] deflate-stream and masking

From: Greg Wilkins <[email-masked]>
Date: Mon, 20 Jun 2011 16:33:12 +1000

As part of my continuing campaign against including deflate-stream in
the specification as a standard extension, I did a quick test of how
well it works when applied to masked frames.

I took a days worth of traffic from an IRC channel and wrapped it up
as JSON messages sent as websocket frames.
There were 487 message that looked like:

     {channel:"#webtide", username:"tbecker", text:"joakime: jenkins
had issues pulling from github a couple of times  last week"}

As an unmasked WS stream, it was 50675 bytes, and as a masked stream
is was 52623 bytes.
I then compressed both these streams with gzip and got 13306 bytes for
unmasked and 51704 bytes for the masked!!!!

So for this very typical example, masking was sufficiently random to
completely negate the benefits of compression.

So the deflate-stream "extension" is:

 + next to useless for inbound traffic
 + breaks all the rules of what an extension can do
 + is potentially vulnerable to injection as attackers can send
repeated patterns that may subvert masking
 + can be replaced by the in-frame compression extension already proposed.
 + was inserted in the draft with little or no discussion and without
clear consensus.

Can I call for a straw poll of who wants to keep this extension in the spec?

regards

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Mon, 20 Jun 2011 09:11:45 +0100

On 06/20/2011 07:33 AM, Somebody in the thread at some point said:

Hi -

> completely negate the benefits of compression.

Isn't this just saying that it's dumb to mask-then-compress?

You could just compress-then-mask and get the 13Kbyte result directly 
and "safely".

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 20 Jun 2011 18:21:04 +1000

On 20 June 2011 18:11, "Andy Green (林安廸)" <[email-masked]> wrote:
> You could just compress-then-mask and get the 13Kbyte result directly and
> "safely".

That is exactly what I'm saying!

We should use a deflate-frame extension that does compress-then-mask,
rather than a deflate-stream "extension" that does mask-then-compress.

cheers

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Mon, 20 Jun 2011 09:33:47 +0100

On 06/20/2011 09:21 AM, Somebody in the thread at some point said:
> On 20 June 2011 18:11, "Andy Green (林安廸)"<[email-masked]>  wrote:

Actually I have to agree with you although I started out intending to 
just let it lie.

For good reasons we changed masking to be payload-only, and added an "I 
am masked" bit, and subordinated the mask to after the header.  All good 
stuff.  So now masking action is strongly defined inside a frame context 
and won't detach easily to be applied to deflate-stream.

Then to fix this the choices are either:

  1) add a feature to deflate-stream that it is always masked using a 
new mask scheme altogether and defeat frame masking if we know the thing 
has deflate-stream on it

  2) just operate deflate-stream on payloads before any per-frame 
masking gets applied

1) sounds like re-inventing the scary masking wheel.

The only little issue with 2) is that deflate-stream operates at bit 
level, you'll have to spill the bits up to a byte every frame payload.

So unless I missed something somewhere, to fix deflate-stream masking 
bloat it does seem to want delfate-stream action to move to only 
applying on payloads, and before any masking is done there.

-Andy

---

From: Dirkjan Ochtman <[email-masked]>
Date: Mon, 20 Jun 2011 10:37:43 +0200

> unmasked and 51704 bytes for the masked!!!!

Are you sure you got this right? From the RFC:

"The masking does not affect the length of the payload data."

Which means it's a little strange that your unmasked WS stream has a
different size from the masked stream.

(Still, it's fairly intuitive that masking would result in worse
compression, so I would tend to agree that compress-then-mask
makes more sense than mask-then-compress.)

Cheers,

Dirkjan

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 20 Jun 2011 18:41:11 +1000

> "The masking does not affect the length of the payload data."

But it does affect the length of the frame as it requires and extra 4
bytes per frame.

There were 487 frames, so that is 1948 extra bytes
52623-50675=1948

cheers

---

From: Bruce Atherton <[email-masked]>
Date: Tue, 21 Jun 2011 14:41:13 -0700

Although I agree with you that deflate-stream is a bad idea and 
recognize that the decision to include it was made before it had been 
determined that masking applies only to payload data (so deflate-stream 
has to be done on masked data), I think it might be difficult to get 
agreement to remove it as a known extension in the spec at this point.

But just because it is a known extension does not mean it needs to be 
implemented. My suggestion is that everyone that thinks it is a bad idea 
should remove deflate-stream from their implementations and add in 
deflate-frame. So long as deflate-frame is available I don't imagine 
there will be a groundswell of consumer desire for the far less useful 
deflate-stream.

On 19/06/2011 11:33 PM, Greg Wilkins wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 22 Jun 2011 09:31:37 +1000

On 22 June 2011 07:41, Bruce Atherton <[email-masked]> wrote:
> Although I agree with you that deflate-stream is a bad idea and recognize
> that the decision to include it was made before it had been determined that
> masking applies only to payload data (so deflate-stream has to be done on
> masked data), I think it might be difficult to get agreement to remove it as
> a known extension in the spec at this point.

Firstly - I really object to this repeated anti-pattern.... some
"feature" is slipped into the draft without much discussion and no
clear consensus, and then we have to argue to have it removed.  The
status quo is to have the non consensus "feature".
I think a lot of the angst in this WG has been due to this repeated
anti-pattern.

But for this particular issue, I see no reason why it cannot be
removed.  Servers have always been free to refuse an extension, so no
application can be written that depends on it's existence.   Clients
are free to keep asking for it, but that does not mean we have to
define it.

> deflate-frame. So long as deflate-frame is available I don't imagine there
> will be a groundswell of consumer desire for the far less useful
> deflate-stream.

the problem is more than just not using it (I have no plans of
implementing it for my server).

It is the only exemplar of an extension we have in the specification
and as an example to follow it is terrible!  It does not follow any of
the rules we set out for extensions and will just encourage developers
to think they can do a x-my-new-wire-protocol and send whatever they
like over the wire.    If people want to do arbitrary new wire
protocols, then they should open their own port. (I know that this
argument also applies to ws, but we have got to significant length to
make sure our protocol is compatible with a HTTP upgrade).

regards

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Wed, 22 Jun 2011 10:18:28 +0100

> and as an example to follow it is terrible!  It does not follow any of

I don't think we even need to go there... "performance is worthless on 
masked data" should be enough to see something needs to happen somewhere 
about it.

-Andy

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 22 Jun 2011 02:26:50 -0700

As an individual:

Yoshino-san sent out a proposal a while ago that did a much better job of
respecting the way extensions are supposed to work. Conceptually, I think
what we really should have is

WebSocket Frame [ Masked Data [ Compress [ Application Data ] ] ]

That is, the application data is first compressed, /then/ that compressed
data is masked and framed as per the rest of the spec.

http://www.ietf.org/id/draft-tyoshino-hybi-websocket-perframe-deflate-00.txt

-Ian

On Wed, Jun 22, 2011 at 2:18 AM, "Andy Green (林安廸)" <[email-masked]>wrote:

> ______________________________**_________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/hybi<https://www.ietf.org/mailman/listinfo/hybi>
>

---

From: "Andy Green (林安廸)" <[email-masked]>
Date: Wed, 22 Jun 2011 10:41:13 +0100

On 06/22/2011 10:26 AM, Somebody in the thread at some point said:
> As an individual:
>
> Yoshino-san sent out a proposal a while ago that did a much better job
> of respecting the way extensions are supposed to work. Conceptually, I
> think what we really should have is
>
> That is, the application data is first compressed, /then/ that
> compressed data is masked and framed as per the rest of the spec.
>

Although I don't really understand what those references devolve down to 
what zlib code and options, it looks like it solves exactly this problem.

-Andy

---

From: Bruce Atherton <[email-masked]>
Date: Wed, 22 Jun 2011 10:37:54 -0700

If the suggestion is to add deflate-frame as proposed in that document 
to the spec as a known extension, then that is a tune I can hum along 
to. There would be no cost to doing that since there would be no 
requirement to implement it if someone didn't want to. The only question 
is whether deflate-frame been tested enough in implementations that we 
can be confident it is going to be stable.

Removing deflate-stream from the spec as a known extension is a separate 
question. Sure, it is useless and a terrible example of how to create a 
websocket extension, but there are probably a few people that have 
implementations. Do these extensions suddenly become "unknown"? What 
would be the consequence of that?

Personally I would be fine with the removal of deflate-stream, but I 
recognize there are issues.