Subject: Issue #73: definition of deflate encoding

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 29 Jul 2009 15:37:12 +0200

http://trac.tools.ietf.org/wg/httpbis/trac/ticket/73

As most of you know our definition of deflate is somewhat contradictory
with the referenced RFCs definitions, as is already stated in our
definition:

   deflate
        The "zlib" format defined in RFC 1950 [31] in combination with
        the "deflate" compression mechanism described in RFC 1951 [29].

As result of this inconsistent name selection there is several
implementations that implement "deflate" to literally mean the deflate
encoding without the zlib wrapper.

Proposed solution is to add a note on this fact

                Note that some incorrect implementations may send
                deflate encoding without a zlib wrapper when using this
                encoding.

On a related note I should perhaps also remind you that gzip is actually
also deflate but with a gzip wrapper instead of zlib. To be consistent
or "deflate" term should really had been "zlib". And both are well
defined file format which also happens to be the same name of the
application/library creating the format in question.

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Wed, 29 Jul 2009 22:17:03 -0700 (PDT)

While doing some recent packet capture / object extraction work as part of 
an effort to study the current degree of content-encoding by live web 
sites, I needed to remove gzip encoding ... naively thought I could just
save the data as a *.gz file (tried .zip also) and post process it. Found
about 800 gzip encoded responses of about 2400 and decoded them all 
assuming no gzip or zlib wrapper. I'm not 100% sure about my conclusions
(I probably could provide a couple example payloads still gzip-ed in file 
form if someone with more knowledge would care to look), but it appears to 
me that 'some incorrect' is likely 'most implementations are incorrect'.

Dave Morris

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 30 Jul 2009 10:29:10 +0200

Could you forward me some of the URLs that you think fail? REDbot <http://redbot.net 
 > has gzip checking built-in and I'd like to verify it...

Cheers,

On 30/07/2009, at 7:17 AM, David Morris wrote:

>
> While doing some recent packet capture / object extraction work as  
> part of an effort to study the current degree of content-encoding by  
> live web sites, I needed to remove gzip encoding ... naively thought  
> I could just
> (I probably could provide a couple example payloads still gzip-ed in  
> file form if someone with more knowledge would care to look), but it  
> appears to me that 'some incorrect' is likely 'most implementations  
> are incorrect'.
>
> Dave Morris
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 30 Jul 2009 10:53:58 +0200

> me that 'some incorrect' is likely 'most implementations are incorrect'.

Interesting. Are you saying that you saw this breakage for content
announced with an gzip content encoding as well, or did you mean to say
deflate encoding above?

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Thu, 30 Jul 2009 18:38:52 -0700 (PDT)

Ooops ... this is where I blush and appologize for not correctly 
remembering the sequence of issues I encountered. Going back to capture 
examples, I found that the content now ungzips correctly.  Digging 
further, it turns out that I had this problem before I discovered that
transfer encoding chunked was being incorrectly removed. 
Since I had already added built in gunzip in the extract tool, I had not 
gone back to try the gunzip.

Sorry about crying wolf without double checking my results.

Dave Morris