Subject: if-range requests and compressed response

From: Ashok Kumar <[email-masked]>
Date: Thu, 28 Jul 2011 16:03:42 +0530

Hi All,

Can someone clarify on which representation the "Range" points to for an
"If-Range" request, when the original response was compressed.

 * Should the server return the specified range from the compressed
response (i.e. compress the original content again, if needed, and send the
relevant range, assuming the compression algorithm generates the same byte
stream again as the previous response!!) or
 * does the range point to bytes from the (decompressed/)uncompressed
response? if so, can the 206 partial response be non-compressed, though the
original response was compressed. How will client "fill" its cache with this
range? (i.e. will the client cache have a complete response after receiving
this for the given E-Tag?)

Also, If the Range points into compressed response, then I need a second
clarification, when if-range has an E-tag and when it carries a date.When
the If-Range carries an E-tag, we know if the original response was
compressed or not. But with last modified date, how to determine if the
original response was compressed or not?

Thanks in advance,
Ashok

---

From: J Ross Nicoll <[email-masked]>
Date: Thu, 28 Jul 2011 11:54:23 +0100

Hi Ashok, everyone,

I think the answer to your last question is to look the Content-Encoding
header (I'm unclear on how ETag would indicate compression?), which
indicates whether the server has compressed the content to be sent.

As to the range question, I'm not sure compressing the content and using
range headers is compatible in any readily sensible manner. If suppose if
the server stores a copy of the compressed content, so it's guaranteed to be
feeding the same data every time (I have just tried gzipping two copies of a
file and got different results  possibly a timestamp included?), that could
work, in which case the range headers would indicate a place in the
compressed stream.

Ross

From:  Ashok Kumar <[email-masked]>
Date:  Thu, 28 Jul 2011 16:03:42 +0530
To:  "[email-masked]" <[email-masked]>
Subject:  if-range requests and compressed response
Resent-From:  <[email-masked]>
Resent-Date:  Thu, 28 Jul 2011 10:34:25 +0000

Hi All,

 * Should the server return the specified range from the compressed response
(i.e. compress the original content again, if needed, and send the relevant
range, assuming the compression algorithm generates the same byte stream
again as the previous response!!) or
 * does the range point to bytes from the (decompressed/)uncompressed
response? if so, can the 206 partial response be non-compressed, though the
original response was compressed. How will client "fill" its cache with this
range? (i.e. will the client cache have a complete response after receiving
this for the given E-Tag?)

Thanks in advance,
Ashok

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 28 Jul 2011 23:09:56 +1200

my understanding is that a Content-Encoding representation of a URI
is independent of an unencoded version.

It should even have a different ETag.

So if you're talking about Content-Encoded compression, then the
range would refer to the representation which happens to be
compressed (e.g. after compression).

If you're referring to Transfer-Encoding compression, that may be
different

On 28/07/2011 10:54 p.m., J Ross Nicoll wrote:

Hi Ashok, everyone,

I think the answer to your last question is to look the
Content-Encoding header (I'm unclear on how ETag would indicate
compression?), which indicates whether the server has compressed
the content to be sent.

As to the range question, I'm not sure compressing the
content and using range headers is compatible in any readily
sensible manner. If suppose if the server stores a copy of the
compressed content, so it's guaranteed to be feeding the same
data every time (I have just tried gzipping two copies of a file
and got different results – possibly a timestamp included?),
that could work, in which case the range headers would indicate
a place in the compressed stream.

Ross

Date: Thu, 28 Jul 2011
16:03:42 +0530

To: "[email-masked]"
<[email-masked]>

Subject: if-range
requests and compressed response

Resent-Date: Thu, 28
Jul 2011 10:34:25 +0000

Hi All,

Can someone clarify on which representation the "Range"
points to for an "If-Range" request, when the original
response was compressed.

* Should the server return the specified range from the
compressed response (i.e. compress the original content again,
if needed, and send the relevant range, assuming the
compression algorithm generates the same byte stream again as
the previous response!!) or

* does the range point to bytes from the
(decompressed/)uncompressed response? if so, can the 206
partial response be non-compressed, though the original
response was compressed. How will client "fill" its cache with
this range? (i.e. will the client cache have a complete
response after receiving this for the given E-Tag?)

Also, If the Range points into compressed response, then I
need a second clarification, when if-range has an E-tag and
when it carries a date.When the If-Range carries an E-tag, we
know if the original response was compressed or not. But with
last modified date, how to determine if the original response
was compressed or not?

Thanks in advance,

Ashok

--
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 28 Jul 2011 23:14:52 +1200

On 28/07/11 22:33, Ashok Kumar wrote:
> response (i.e. compress the original content again, if needed, and send
> the relevant range, assuming the compression algorithm generates the
> same byte stream again as the previous response!!) or

Critical assumption. Have you noticed how two installs of same 
compression software can produce different output for the same input?

It seems to me _de_compressing might be safer. Anyone wiser able to say 
for sure?

> response? if so, can the 206 partial response be non-compressed, though
> the original response was compressed. How will client "fill" its cache
> with this range? (i.e. will the client cache have a complete response
> after receiving this for the given E-Tag?)

Does that client have identical existing ETag value? AND identical 
timestamps on both versions?

RFC 2616:
"
    A cache MUST NOT combine a 206 response with other previously cached
    content if the ETag or Last-Modified headers do not match exactly,
"

> clarification, when if-range has an E-tag and when it carries a
> date.When the If-Range carries an E-tag, we know if the original
> response was compressed or not. But with last modified date, how to
> determine if the original response was compressed or not?

see above.  BOTH ETag and Last-Modified must be identical for any 
merger. If either is missing or different no merge.

Someone will probably flame me about this, but ... IMO having a date in 
there means no Etag. So the cache can't use any version for which it has 
an ETag.

Following that implication, and combining it with:
   RFC 2616 section 10.2.7 206 Partial Content:
  "The response MUST include the following header fields: [...] ETag"

I conclude that no cache may legitimately respond to an If-Range 
containing a date.

Unless it were to synthesize an ETag for that reply. Which would then no 
longer match the requested object and thus erase the benefit of 
responding with 206 in the first place.

Or, would this be a valid 206 header value?
    ETag: ""

AYJ

---

From: Brian Pane <[email-masked]>
Date: Thu, 28 Jul 2011 09:23:26 -0700

> this for the given E-Tag?)

I looked into this same question a while ago and concluded that the
right interpretation is your first one: "return the specified range
from the compressed response."

The key sections of RFC 2616 are 14.35.1:

"Byte range specifications in HTTP apply to the sequence of bytes in
the entity-body (not necessarily the same as the message-body)."

and 14.11:

"The Content-Encoding entity-header field is used as a modifier to the
media-type. When present, its value indicates what additional content
codings have been applied to the entity-body."

Putting these two sections together, the server must interpret the
Range request relative to the entity-body, and the entity-body is the
document *after* compression and any other Content-Encodings have been
applied.

If I remember correctly, this interpretation is consistent with how
Apache and nginx do their response filtering: compression is applied
first, and then the requested byte-ranges are extracted.

As you noted, this implies that a server (or intermediary) that does
dynamic compression is implicitly required to produce the same output
every time it compresses the same response.  This means, for example,
that a server can't reuse its gzip dictionary state across multiple
responses on the same connection in hopes of achieving better
compression.  But the server shouldn't be doing that anyway, based on
the "considering messages in isolation" clarification in HTTPbis:
http://trac.tools.ietf.org/wg/httpbis/trac/ticket/288

-Brian

---

From: Ashok Kumar <[email-masked]>
Date: Fri, 29 Jul 2011 11:15:22 +0530

Thanks All,

J Ross Nicoll wrote
> I think the answer to your last question is to look the Content-Encoding
header
> (I'm unclear on how ETag would indicate compression?), which indicates
whether
> the server has compressed the content to be sent.
Ross, here I was talking about how a server(or intermediate cache doing
compression)
could identify if the if-range request was for the compressed entity or
uncompressed
one. i.e. whether the original response was served compressed or
uncompressed. If an
E-Tag is present, it will be different for compressed and uncompressed
entities and
the server can deduce if the original response was compressed or not. But if
all that
the server gets is a last modified date in the if-Range, then how to
determine which
content-encoding was served last time. (The assumption here is that the
server can
possibly skip compressing content even though A-E: gzip,deflate was received
from client,)

Amos Jeffries wrote
> see above.  BOTH ETag and Last-Modified must be identical for any merger.
If either is missing or different no merge.
> Someone will probably flame me about this, but ... IMO having a date in
there means no Etag. So the cache can't use
> any version for which it has an ETag.

If this is true, i.e. the client will merge the 206 if (and only if) the
E-Tag is also present
and matches, then my above worry about merging is unjustified. Thanks :)

Brian Pane wrote
> every time it compresses the same response.

So this also implies that for dynamically generated gzip compressed
response, when server
receives an If-Range, then server can only respond with 200 with
uncompressed response?

This begs another question. Though not specific to HTTP WG, but just to make
sure my
understanding is correct. IIRC, Apache mod_gzip fixed the E-Tag generation
problem
by appending "-gzip" or similar to end of E-tag of the original content. If
two runs of
gzip compression don't produce the same output, isn't the E-Tag generated
wrong? i.e.
when dynamically compressing the content, a completely new E-Tag must be
generated
rather than just appending a static string to E-tag of original uncompressed
content.

Thanks for clarifications again, Also please do let me know if HTTP WG is
not the right place for
discussion about these implementation issues and please point to right
discussion list.
-Ashok

On Thu, Jul 28, 2011 at 9:53 PM, Brian Pane <[email-masked]> wrote:

> the
> the

-- 
.- ... .... --- -.-

---

From: Brian Pane <[email-masked]>
Date: Fri, 29 Jul 2011 00:11:20 -0700

On Thu, Jul 28, 2011 at 10:45 PM, Ashok Kumar <[email-masked]> wrote:

> So this also implies that for dynamically generated gzip compressed response, when server
> receives an If-Range, then server can only respond with 200 with uncompressed response?

I think there are two situations where the server could respond with a 206:

1. The If-Range request specifies an entity-tag, and the server knows
(via implementation-specific means) that this entity-tag is the same
as the ETag the server would use for the response at the present time.

2. The If-Range request specifies an HTTP-date, and the server knows
(via implementation-specific means) that the Last-Modified date it
would use for the response at the present time is no later than the
specified HTTP-date.

> This begs another question. Though not specific to HTTP WG, but just to make sure my
> understanding is correct. IIRC, Apache mod_gzip fixed the E-Tag generation problem
> by appending "-gzip" or similar to end of E-tag of the original content. If two runs of
> gzip compression don't produce the same output, isn't the E-Tag generated wrong?

If it's a weak ETag, then it's okay to use the same ETag value even if
two applications of compression yield outputs that aren't bytewise
equal.  If it's a strong ETag, like you'd need in order to support
If-None-Match or If-Range, then yes, the generated ETag would be wrong
in this situation; however... what's the use case in which two runs of
gzip, both starting with freshly initialized state (as is the case
when gzip is invoked dynamically on a per-response basis in every
server and proxy implementation I've seen), will produce different
output from the same input?  I can only think of a couple of ways to
cause the output to differ while still producing a response the client
can uncompress:
- Change gzip's compression level between the two runs, or
- Swap in a different compression implementation (that produces
different but still valid gzip-formatted output) between the two runs.
Neither of those is impossible, but I don't imagine that they're
common cases either.  Am I missing any other use case?

Thanks,
-Brian

---

From: Ashok Kumar <[email-masked]>
Date: Fri, 29 Jul 2011 14:26:01 +0530

Thanks again Brian.

Brian Pane wrote
> what's the use case in which two runs of
> output from the same input?

for a proxy, won't gzip bytestream change because of the MTIME change (since
the proxy will fill in compression start time instead of original file
modification time because its working on a stream rather than a file?). For
ZLIB/Deflate, this should be fine.

On Fri, Jul 29, 2011 at 12:41 PM, Brian Pane <[email-masked]> wrote:

> wrong?

-- 
.- ... .... --- -.-

---

From: Brian Pane <[email-masked]>
Date: Fri, 29 Jul 2011 10:49:36 -0700

> ZLIB/Deflate, this should be fine.

Ah, thanks, now I see the use case where there's a potential problem.
The simplest solution is to set the MTIME in the gzip header to zero
when doing dynamic compression.  And, based on a quick check of the
code, that's exactly how it's done in nginx, Apache, and Varnish (the
first three open source proxies with gzip support that I could think
of).  These implementations call a function named deflateInit2 in zlib
to initialize their gzip state.  According to the comments in the zlib
source, this function sets up a gzip stream with a header suitable for
streaming compression of dynamic content:

"The gzip header will have no file name, no extra data, no comment, no
modification time (set to zero), no header crc, and the operating
system will be set to 255 (unknown)."

-Brian

---

From: Ashok Kumar <[email-masked]>
Date: Sat, 30 Jul 2011 10:19:23 +0530

Thanks Brian.

> system will be set to 255 (unknown)."

That should solve the problem.

-- 
.- ... .... --- -.-

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 30 Jul 2011 08:23:41 +0200

> common cases either.  Am I missing any other use case?

Well, even worse, a server (or an intermediary) might dynamically adjust
its compression level depending on CPU usage, which means that a single
compressed stream might be built from different compression levels.

Regards,
Willy