Subject: content-encoding and range headers

From: Stefan Eissing <[email-masked]>
Date: Tue, 10 Dec 2002 13:50:46 +0100

I have unsuccessfully tried to google an answer to this, maybe someone
on this list has an answer.

My understanding of 2616 is that range is applied to the
content-encoded entity. So if an entity is gzipped, the
byte selection is applied on the gzipped representation.
Given that:

What is a server to do when a client sends a request with
Accept-Encoding: gzip
Range: bytes=0-499

Is the server free to chose the content-encoding to its liking 
(identity or gzip)?

Assuming there is a second request later
Accept-Encoding: gzip
Range: bytes=500-1000

Is the server allowed to use another content-encoding in the response?

If yes, that would mean it is purely the client's responsibility
to detect varying content-encodings in partial responses and
act accordingly?

Or did I misunderstand the order of range and content-encoding
and the byte range is selected on the unencoded entity and
content-encoding is applied *after* the range selection process?

In that case, a server storing gzipped entities has some work to do
answering range request. It would then need to "uncompress"
(ungzip?) the entity before it can select the correct range.

  Thanks for any help.

//Stefan

---

From: Scott Lawrence <[email-masked]>
Date: 10 Dec 2002 09:33:51 -0500

> byte selection is applied on the gzipped representation.

I don't believe that this is correct.  The Range specifies bytes in
the 'entity'; the Content-Encoding refers to a coding method
_applied_to_ the 'entity', so the Content-Range header would refer to
the bytes in the original entity, and the Content-Encoding describes
how that has been encoded to create the message body.

> Is the server free to chose the content-encoding to its liking

Yes, but the result would be the first 500 bytes of the entity, sent
either with or without gzip encoding at the servers option.

> Is the server allowed to use another content-encoding in the response?

The 'identity' encoding is the only one that is implicitly allowed, so
it shouldn't send any others.
 
> (ungzip?) the entity before it can select the correct range.

This gets into an interesting question, and one that often causes
confusion.  Let's say that I have a file 'foo.txt.gz' on my web server
at URL '/foo.txt.gz'.  The (gzipped) file is 350 bytes long, but
expands to 1000 bytes when ungzipped.

 GET /foo.txt.gz HTTP/1.1
 Host: scott.example.com
 Accept-Encoding: gzip

I'll return 

 HTTP/1.1 200 Ok 
 Content-Length: 350
 Content-Type: application/x-gzip

 [350 bytes of the file]
 
Note that I did not send a Content-Encoding header, because the entity
itself is gzipped, which is expressed in the Content-Type.  

If the file were stored as 'foo.txt' and you sent:

 GET /foo.txt HTTP/1.1
 Host: scott.example.com
 Accept-Encoding: gzip

then I might compress the response and send:

 HTTP/1.1 200 Ok 
 Content-Length: 350
 Content-Type: text/plain
 Content-Encoding: gzip

 [350 bytes]

--
  Scott Lawrence

---

From: "Julian Reschke" <[email-masked]>
Date: Tue, 10 Dec 2002 16:20:50 +0100

> From: [email-masked] [mailto:[email-masked]]On
> Behalf Of Scott Lawrence
> Sent: Tuesday, December 10, 2002 3:34 PM
> To: Stefan Eissing
> Cc: [email-masked]
> Subject: Re: content-encoding and range headers
> 
> 
> 
> ...

I agree that it depends on the definition of the entity body, which is [1]:

	entity-body := Content-Encoding( Content-Type( data ) )

So I think Stefan's analysis is correct...

[1] <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.7.2.1>

--
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 10 Dec 2002 08:59:26 -0700 (MST)

Stefan,

	The following paper, written by one of the RFC 2616 authors,
may help you to navigate through the ranges, transfer, and content
encoding mess (see Figure 4-1b on page 8 if you do not want to read
the entire paper).

	"Clarifying the Fundamentals of HTTP"
	Jeffrey C. Mogul
	WWW Conference, 2002
	http://research.compaq.com/wrl/people/mogul/www2002/mogulwww2002preprint.pdf

HTH,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

---

From: Scott Lawrence <[email-masked]>
Date: 10 Dec 2002 11:28:08 -0500

> So I think Stefan's analysis is correct...

But Accept-Encoding does not _require_ the server to use the specified
encoding, it _offers_ to accept it.  So the request:

 GET /foo HTTP/1.1
 Host: example.com
 Accept-Encoding: gzip
 Range: 10000-19999

might return two completely different parts of /foo depending on
whether or not the server chose to apply the Content-Encoding.  Since
HTTP is stateless, you can't even count on any given server making the
same choice each time.

If you believe that interpretation, then I think you must conclude
that the combination is not very useful to the client.

The server I've worked on most didn't ever use Content-Encoding, so
it ignored Accept-Encoding.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 10 Dec 2002 09:59:20 -0700 (MST)

> that the combination is not very useful to the client.

I think you are right it general, though the above combination might
still be useful in some variations. For example,

  GET /foo HTTP/1.1
  Host: example.com
  Accept-Encoding: gzip
  Range: 0-5000

may be useful for a client that just needs to peek at the first 5000
bytes of an unencoded content (at most) and can uncompress the entity
if needed. The above will save bandwidth for content that compresses
to fewer than 5000 bytes. Again, this is clearly a rather exotic (but
HTTP-compliant?) usage.

Note that if the client really needs 10000-19999 unencoded content
bytes, the request below could be used instead:

  GET /foo HTTP/1.1
  Host: example.com
  TE: gzip
  Range: 10000-19999

Since transfer-coding is applied after the range is selected, the
client is guaranteed to receive the same content bytes, and the server
has an opportunity to compress the response to save bandwidth.
Transfer-coding will probably (but not necessarily) cost extra CPU
cycles.

Alex.

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 17 Dec 2002 14:54:38 +0100

Am Dienstag, 10.12.02, um 17:59 Uhr (Europe/Berlin) schrieb Alex 
Rousskov:

> [...]

While we're at it, maybe someone can confirm that I understood 
Transfer-Encoding
correctly:

If a server choses to use gzip transfer encoding in answer to the 
request above,
is it correct to assume that
a) the header
      Transfer-Encoding: gzip, chunked
    ot the combination
      Connection: close
      Transfer-Encoding: gzip
    would be sent. And that there is no other way to implement transfer 
encoding
    gzip than chunking the gzipped message body or closing the 
connection at the
   end of the gzipped body?
b) assuming chunked, gzip transfer encoding is used, the order in the
   header would be
      Transfer-Encoding: gzip, chunked
   and not
      Transfer-Encoding: chunked, gzip
c) if Content-Length is set in the response, it is the amout of octets 
in
    the ungzipped body (i.e. entity).

And last: is there a public server which implements gzip TE? Apache 
seems to
have  the module hooks for doing so at least...

Thanks for the help,

Stefan