Subject: Re: [hybi] Compression/masking precedence

From: Bob Gezelter <[email-masked]>
Date: Thu, 31 Mar 2011 07:35:04 -0500

Masking should always be the last stage. Compression of masked data is
almost invariably far less efficient than compression of plaintext followed
by masking/encryption.

Example:   A(0x41) A(0x41) A(0x41) A(0x41) A(0x41) A(0x41) A(0x41) A(0x41)
     Mask:    0x01    0x02    0x03    0x04    0x01    0x02    0x03    0x04

Masked Text: 0x40    0x43    0x42    0x45    0x40    0x43    0x42    0x45

Compressed Text: 8*"A"
Masked: no successive character repeats (0x40, "CBE", 0x40, "CBE")

PGP is an example of this in practice. The text is compressed, then encoded.
Another illustration of this is daily backup sets. Compressing an
uncompressed backup set is often highly effective. Encrypting the resulting
data set does not take more steps. Reversing the precedence (encrypting
first, then compressing) produces far less effective compression.

- Bob
+--------------------------------------------------------------------------+
| Robert "Bob" Gezelter                       E-Mail:  [email-masked]  |
| Robert Gezelter Software Consultant                                      |
| 35-20 167th Street, Suite 215               Fax:       (on Request)      |
| Flushing, New York  11358-1731                                           |
| United States of America                    http://www.rlgsc.com         |
+--------------------------------------------------------------------------+

---

From: John Tamplin <[email-masked]>
Date: Thu, 31 Mar 2011 08:45:28 -0400

> almost invariably far less efficient than compression of plaintext followed

Nobody is arguing with that.  However, masking must be done at the frame
level, and the current deflate-stream extension operates at the connection
level, simply because that was something nobody objected to.  I don't think
anybody thought deflate-stream was the ultimate compression algorithm - just
something "better than nothing" that could be added without much debate.

See a post I made last May about the impact of different ways of doing
compression, even on small binary messages like those from GwtQuake.  I
certainly intend to propose a new compression extension once we get the base
spec out the door.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Patrick McManus <[email-masked]>
Date: Thu, 31 Mar 2011 14:56:33 +0200

On 3/31/11 2:45 PM, John Tamplin wrote:
> On Thu, Mar 31, 2011 at 8:35 AM, Bob Gezelter <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Nobody is arguing with that.  However, masking must be done at the 
> frame level, and the current deflate-stream extension operates at the 
> connection level, simply because that was something nobody objected 
> to.  I don't think anybody thought deflate-stream was the ultimate 
> compression algorithm - just something "better than nothing" that 
> could be added without much debate.
>
> compression, even on small binary messages like those from GwtQuake. 
>  I certainly intend to propose a new compression extension once we get 
> the base spec out the door.
>
this is all true, but lets not leave people with the impression that 
deflate-stream has no uses at all.

It certainly will provide help for largish messages (the xor won't 
change the distribution of anything within the message in the way real 
crypto would) as well as most flowing in the server->client direction. 
Note I said help - not "optimally help". A fair number of use cases 
would fall under these descriptions.