Subject: httpbis and deflate compression...

From: "Mahdavi, Jamshid" <[email-masked]>
Date: Mon, 22 Mar 2010 17:49:47 -0700

Hi,

I offered some assistance on this at the WG today.  Julian suggested that I send this information to the list.

I think RFC2616 is quite clear on what the correct behavior is for "Content-encoding: deflate".  However, because of the naming, I think there is an easy way for people to get confused.

Per 2616:

    deflate
        The "zlib" format defined in RFC 1950 [31] in combination with
        the "deflate" compression mechanism described in RFC 1951 [29].

However, it would be easy for someone to think that "Content-encoding: deflate" should simply map to the RFC for "deflate", which is just RFC1951 without the additional headers from 1950.

We could try to clarify this by adding an implementation note to the text.  Alternately, since some servers actually do implement *just* the 1951 version, we could add another content-coding which is specifically for this.  My suggestion would be to call that "deflate-bare" or something along those lines.  Then an implementer would have to actually think about what the difference between the two is, and anyone who currently implements the plain 1951 version could easily come into compliance without changing their actual compression code.

I'm not sure if there is any real demand for something like a "deflate-bare", although I could hypothesize that maybe someone chose this to avoid the extra checksum calculation required in RFC1950.

Just adding the note would be the simpler thing to do from a documentation perspective.  We could say something like:

NOTE:  Some implementations have incorrectly used only the RFC1951 encoding when returning data using the "deflate" content-coding.  Implementers should be aware the a compliant implementation of the "deflate" content-coding should follow the "ZLIB Compressed Data Format Specification" as defined in RFC1950.

--J

---

From: Dan Winship <[email-masked]>
Date: Mon, 22 Mar 2010 21:48:02 -0400

On 03/22/2010 08:49 PM, Mahdavi, Jamshid wrote:
> We could try to clarify this by adding an implementation note to the
> text. Alternately, since some servers actually do implement *just* the
> 1951 version, we could add another content-coding which is
> specifically for this.

My understanding is that the "bare" version is actually more common on
the web than the correct version, due to the fact that at least IE6 and
IE7 (not sure about 8 and 9) only support the bare version, and most
other browsers support both.

Given the confusion, and the fact that all current web browsers that
support deflate also support gzip, and that deflate has no real benefits
over gzip, another possibility would be to just deprecate "deflate" (and
"compress", which no one implements anyway) and encourage clients to
just send "Accept-Encoding: gzip".

-- Dan

---

From: Julian Reschke <[email-masked]>
Date: Tue, 23 Mar 2010 02:53:44 +0100

> other browsers support both.

Sounds like it would be cool to have test cases...

> ...

BR, Julian

---

From: David Morris <[email-masked]>
Date: Thu, 25 Mar 2010 18:08:58 -0700 (PDT)

I have a limited sample of wild web proxy data streams which I could
check for what is used.

In a data sample that lasted about 5 minutes, I captured 4500+ 
request/responses. Of these:
   648 has a non-empty Content-encoding
   638  gzip
     2  none
     1  UTF-8
     6  sdch,gzip
In the same data
  2436 have a non-empty accept encoding on the request
  2433 were gzip, deflate
     3 were gzip

There were 836 unique host names in the data. (x.google.com and 
y.google.com are unique). And roughly 350 unique sites.

In this sample, as processed, there were no deflate encoded responses.

If someone could provide sniff like rules I could apply to deflate
content, if I find any, to determine which variation it is, I can
examine the data again as well as some additional data I have to
see if deflate is used and how many sites use it. 

This user population is probably biased to variations of IE but
I don't have user agent analysis handy.

Ideally, you'd send a python regex I could just fold into my tool.

Dave Morris

---

From: "Mahdavi, Jamshid" <[email-masked]>
Date: Thu, 25 Mar 2010 18:12:51 -0700

That is consistent with what we've seen / heard; deflate never or almost never actually gets used because gzip is usually also an option and tends to be preferred.

--J

-----Original Message-----
From: David Morris [mailto:[email-masked]]
Sent: Thu 3/25/2010 6:08 PM
To: Julian Reschke
Cc: Dan Winship; Mahdavi, Jamshid; [email-masked]
Subject: Re: httpbis and deflate compression...

Dave Morris

---

From: Dan Winship <[email-masked]>
Date: Thu, 25 Mar 2010 23:36:46 -0400

On 03/25/2010 09:08 PM, David Morris wrote:
> content, if I find any, to determine which variation it is

zlib-wrapped-deflate has a small header, but there are no magic numbers
in it so there's not a lot to sniff. (The lower 3 bits of the first byte
are always 0x8 though...) Raw deflate data has no header at all and so
can look more or less like anything (including a valid zlib header). The
only way to actually tell which kind of deflate it is is to try decoding
it, and see if you get a "data is corrupt" error, and if so, that means
it was the other kind and you should try again. (At least, that's how
Firefox does it.)

-- Dan

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 27 Mar 2010 10:10:36 +1100

Thanks, Mahdavi. 

The proposal we've had in the ticket <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/73> for eight months is to add:

"Note that some incorrect implementations may send deflate encoding without a zlib wrapper when using this encoding."

The proposal below seems like an elaboration of this:

"NOTE:  Some implementations have incorrectly used only the RFC1951 encoding when returning data using the "deflate" content-coding.  Implementers should be aware the a compliant implementation of the "deflate" content-coding should follow the "ZLIB Compressed Data Format Specification" as defined in RFC1950."

Any objections to this?

--
Mark Nottingham     http://www.mnot.net/

---

From: Yves Lafon <[email-masked]>
Date: Sun, 28 Mar 2010 08:46:47 -0400 (EDT)

On Sat, 27 Mar 2010, Mark Nottingham wrote:

> Thanks, Mahdavi.
>
> The proposal we've had in the ticket 
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/73> for eight months 
> is to add:
>
> "Note that some incorrect implementations may send deflate encoding 
> without a zlib wrapper when using this encoding."
>
> "NOTE:  Some implementations have incorrectly used only the RFC1951 
> encoding when returning data using the "deflate" content-coding. 
> Implementers should be aware the a compliant implementation of the 
> "deflate" content-coding should follow the "ZLIB Compressed Data Format 
> Specification" as defined in RFC1950."
>
> Any objections to this?
>

Should we add that 'deflate' is deprecated because of interoperability 
issues? (ie: use 'gzip').

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 29 Mar 2010 10:49:54 +0900

On 2010/03/27 8:10, Mark Nottingham wrote:
> Thanks, Mahdavi.
>
> The proposal we've had in the ticket<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/73>  for eight months is to add:
>

Not if you fix "aware the a compliant implementation" to "aware that a 
compliant implementation".

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 28 Mar 2010 22:30:36 -0700

I think that's going too far. I have come across people using it.

On 28/03/2010, at 5:46 AM, Yves Lafon wrote:

> Should we add that 'deflate' is deprecated because of interoperability issues? (ie: use 'gzip').
>