Subject: Clarification of the term "deflate"

From: "Paul Marquess" <[email-masked]>
Date: Tue, 7 Aug 2007 14:20:54 +0100

Below is the definition of "deflate" from RFC 2616, section 3.5 "Content
Codings"

  deflate
    The "zlib" format defined in RFC 1950 [31] in combination with
    the "deflate" compression mechanism described in RFC 1951 [29].

There is ambiguity in that definition because of the inconsistent use of the
term "deflate". This has resulted in a long standing confusion about how to
implement "deflate" encoding. 

There was a time a few years back when most of the high profile browser and
some http server implementations incorrectly implemented http "deflate"
encoding using RFC 1951 without the RFC 1950 wrapper. Admittedly most, if
not all, of the incorrect implementations have now been fixed, but the fix
applied recognises the reality that there are incorrect implementations of
"deflate" out in the wild. All browsers now seem to be able to cope with
"deflate" in both its RFC1950 or RFC1951 incarnations.

So I suggest there are two issues that need to be addressed

1. The definition of "deflate" needs to be rewritten to remove the
ambiguity.

2. Document the reality that there are incorrect implementations, and
recommend that anyone writing a "deflate" decoder should cope with both
forms.

cheers
Paul

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 07 Aug 2007 17:50:24 +0200

> 1. The definition of "deflate" needs to be rewritten to remove the

Any suggestion?

I don't see how the definition is ambiguous, apart from the poor choice
of name (should have been zlib, not deflate).

Regards
Henrik

---

From: "Paul Marquess" <[email-masked]>
Date: Tue, 7 Aug 2007 17:10:31 +0100

From: Henrik Nordstrom [mailto:[email-masked]]
 
> > 1. The definition of "deflate" needs to be rewritten to remove the

One possibility is to remove the reference to RFC 1951 completely.

  deflate
    The "zlib" format defined in RFC 1950 [31]. 

This variant keeps both RFC 1950 & 1951 but drops the troublesome labels.

  deflate
    The compressed data format defined in RFC 1950 [31] in combination with
    the compression mechanism described in RFC 1951 [29].

> of name (should have been zlib, not deflate).

Indeed - the choice of name is the source of the ambiguity - there are two
conflicting uses of "deflate" in the definition. If 2616 had used "zlib"
instead of "deflate" the problem wouldn't have happened. 

Paul

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 07 Aug 2007 23:36:23 +0200

>     The "zlib" format defined in RFC 1950 [31]. 

1950 doesn't reference 1951.

>     the compression mechanism described in RFC 1951 [29].

RFCs define formats and protocols, not implementation. So it should
refer to the format defined by 1951, not the mechanism used for
producing that format.

And "zlib" isn't a compressed data format even if it's title says so.
It's just a wrapper around compressed data (only deflate defined, others
may be added in future).

     The "zlib" format defined in RFC 1950 [31] in combination
     with the compressed data format described in RFC 1951 [29].

On a side note it's worth noting that the "gzip" compressed format is
also just a different wrapper around deflate. And that the gzip
compressor is a different implementation producing RFC1951 deflate
streams.

> instead of "deflate" the problem wouldn't have happened. 

Personally I don't really understand why the zlib wrapper is required
around the deflate stream, but that is a different question and too late
to do anything about.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 7 Aug 2007 15:56:37 -0700

> refer to the format defined by 1951, not the mechanism used for

Yes, but

<http://www.hpl.hp.com/personal/ange/archives/archives-96/http-wg- 
archive/1343.html>

is no help.  I had always thought deflate referred to the unwrapped
format, whereas gzip refers to the wrapped format.  The problem
is that there are advantages to storing the content in gzip format
and selectively delivering it according to which T-E's or C-E's
are listed as acceptable.

I think it is worthy of an issue number.

....Roy

---

From: "Paul Marquess" <[email-masked]>
Date: Wed, 8 Aug 2007 10:42:55 +0100

> 1950 doesn't reference 1951.

Indeed - it references this instead

   [3] Deutsch, L.P.,"DEFLATE Compressed Data Format Specification",
       available in ftp://ftp.uu.net/pub/archiving/zip/doc/  

The ftp link takes you to a list of files, one of which is 1951, but that
isn't a precise enough reference.

>      with the compressed data format described in RFC 1951 [29].

That's better.

So how about adding a note that tells the reader the reality of the
implementations that are out in the wild? Something like this perhaps?

Note: Experience has shown that some implementations of "deflate" encoding
have incorrectly used RFC 1951 without the RFC 1950 wrapper. In order to
maximise interoperability an implementation should consider accepting
"deflate" encoding that uses either RFC 1950 in combination with RFC 1951 or
"deflate" encoding that uses RFC 1951 on its own.

> around the deflate stream, but that is a different question and too late

Using the zlib wrapper means an implementation can quickly determine whether
the content is what it claims to be by checking the zlib signature. With
1951, you have to try to uncompress it before you can tell.

Paul

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 09 Aug 2007 14:47:35 +0200

On tis, 2007-08-07 at 15:56 -0700, Roy T. Fielding wrote:

> is no help.

And the mismatches in RFC references also doesn't help...

RFC1950 is zlib
RFC1951 is deflate
RFC1952 is gzip

discussion regarding the defalate & gzip entries was on RFC1951 &
RFC1952 it seems, which makes sense.

actual definition of deflate ended up to be RFC1950 using RFC1951.

> I had always thought deflate referred to the unwrapped
> format, whereas gzip refers to the wrapped format.

it's the intuitive choice.. but not how it got written.

> The problem

? how is that related?

> I think it is worthy of an issue number.

Probably.

But the question is what is the correct solution.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 9 Aug 2007 14:40:41 +0100

> 1951, you have to try to uncompress it before you can tell.

We have HTTP headers for that.  If they aren't trustworthy - what
makes the zlib signature trustworthy?

- Jamie

---

From: David Morris <[email-masked]>
Date: Thu, 9 Aug 2007 13:08:29 -0700 (PDT)

> makes the zlib signature trustworthy?

As part of a team that produces a value added proxy facility, I can tell
you that HTTP headers are frequently wrong when it comes to labeling
content, encoding, etc. Data streams are also corrupt, but much less often
and if the stream starts with a known label, we don't need to crash test
alternatives.

Dave Morris

---

From: "Paul Marquess" <[email-masked]>
Date: Fri, 17 Aug 2007 10:16:20 +0100

From: [email-masked] [mailto:[email-masked]] On

> are listed as acceptable.

Not sure I understand your point about gzip here Roy.  

> I think it is worthy of an issue number.

OK

Paul

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Aug 2007 13:40:27 +1000

http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i73

--
Mark Nottingham     http://www.mnot.net/