Subject: Re: i28 proposed replacement text

From: Mark Nottingham <[email-masked]>
Date: Fri, 9 May 2008 16:01:02 +1000

Last call. Current proposal:

ref: http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i28

In section 4.4 the current text is:
<<<
   2.If a Transfer-Encoding header field (section 14.41) is present and
     has any value other than "identity", then the transfer-length is
     defined by use of the "chunked" transfer-coding (section 3.6),
     unless the message is terminated by closing the connection.
Here is the porposed clarification text:
<<<
2: If a Transfer-Encoding header field (Section 14.41) is present, and
    has any value other than "identity", then the transfer-length is
    defined by use of the "chunked" transfer-coding (Section 3.6)

On 18/04/2008, at 10:14 PM, Yves Lafon wrote:

>
> On Thu, 3 Apr 2008, Mark Nottingham wrote:
>
>> Yves,
>>
>> Do you mean:
>>
>>> 2: If a Transfer-Encoding header field (Section 14.41) is present,  
>>> and
>>>  has any value other than "identity", then the transfer-length is
>>>  defined by use of the "chunked" transfer-coding (Section 3.6).
>>
>> ? (I think there was a cut-and-paste error, perhaps).
>
> Yes, as there is already a MUST (use chunked transfer-coding) in  
> part 02, section 3.4
>
>> If so, +1.
>>
>> One comment -- this is good, but my reading of the "unless the  
>> message is terminated..." text is that it was intended to cover the  
>> case where the connection is prematurely closed. It may be useful  
>> to have text added below the numbered list along these lines;
>>
>> """
>> If a message has a defined length (e.g., using chunked encoding,  
>> Content-Length, or multipart/byteranges), and the connection is  
>> prematurely closed, then the transfer-length will be less than  
>> indicated, and the message is incomplete.
>> """
>
> +1
> We can also add that it's an error (even if it seems obvious)
>
>> This leaves the question open of whether we want to place any  
>> additional requirements on incomplete messages (which should  
>> probably be considered separately).
>
> We can't say, from such an error, if it's used to signal an error,  
> or if it is an unplanned error, also there is the issue of partial  
> cache such responses to a GET, or what to do when it's a POST. It  
> all goes in the "error recovery" bucket...
> We have some error recovery (as in part 1, 7.2.4), something on the  
> same lines for this case should be ok.
> Cheers,
>
> -- 
> Baroula que barouleras, au tiéu toujou t'entourneras.
>
>        ~~Yves
>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 9 May 2008 08:10:04 +0200

>    defined by use of the "chunked" transfer-coding (Section 3.6)

+1.

<green/>bytes GmbH, Hafenweg 16, D-48155 Münster, Germany
Amtsgericht Münster: HRB5782

---

From: Yves Lafon <[email-masked]>
Date: Sat, 10 May 2008 18:32:06 -0400 (EDT)

+1

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 02:14:22 +0200

>     defined by use of the "chunked" transfer-coding (Section 3.6)

No, this is wrong. See the section on transfer encoding.

The following message header is perfectly legal and is what the "closing
the connection" part is about:

HTTP/1.1 200 OK
Transfer-Encoding: gzip
Connection: close

What it should read is something like the following:

2. If a Transfer-Encoding header field (Section 14.41) is present, and
   indicates that "chunked" was the last encoding applied to the
   message-body then the transfer-length is defined by use of the
   "chunked" transfer-coding (Section 3.6).

3. If a Transfer-Encoding header field (Section 14.41) is present, and
   has a value other than "identity", then the transfer-length
   is defined by closing the connection.

The 'Content-Length' and 'multipart/byteranges' rules do not apply to
messages using any form of transfer encoding. When transfer-encoding is
used the only methods available for delimiting the messages is chunked
transfer encoding or when chunked is not used by closing the connection.

Side Note: It's illegal to apply any transfer-encoding after chunked
encoding simply because it would be a complete waste. Technically the
message format do support inner levels of chunked encoding.

Regards
Henrik

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 12 May 2008 02:45:48 +0200

>   is defined by closing the connection.

I agree with your interpretation of the current text. I do note that,
if we change the text, we should use the opportunity to clarify how
to handle 

  Transfer-Encoding: identity;foo=bar

and perhaps

  Transfer-Encoding: gzip,identity

It is easy to misread "value other than".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 10:47:21 +0200

> It is easy to misread "value other than".

Hmm... lets see now. We have removed the "identity" encoding in #16

http://www3.tools.ietf.org/wg/httpbis/trac/ticket/16

so 3 becomes simply

If a Transfer-Encoding header field (Section 14.41) is present, then the
transfer-length is defined by closing the connection.

Or perhaps by just clarifying the existing rule 2 so it can not be
misread when closing the connection is a valid delimiting metod.

From:

        If a Transfer-Encoding header field is present, then the
        transfer-length is defined by use of the "chunked"
        transfer-coding unless the message is terminated by closing the
        connection.

To:

        If a Transfer-Encoding header field is present, then the
        transfer-length is defined by use of the "chunked"
        transfer-coding, unless "chunked" is not the last encoding
        applied to the message-body in which case the message is
        terminated by closing the connection.

Regards
Henrik

---

From: Yves Lafon <[email-masked]>
Date: Mon, 12 May 2008 05:42:37 -0400 (EDT)

>   is defined by closing the connection.

And in that case, you have no guarantee that the message in complete or 
not (unless you use a Transfer-Encoding that gives that property, if the 
recipient knows about this encoding, which is not granted.

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 12:06:59 +0200

> recipient knows about this encoding, which is not granted.

Ofcourse, except for the last part. Closing the connection as delimiting
method is only possible on responses and there any transfer encoding
other than chunked is negotiated using TE.

Both gzip & deflate (zlib) is self delimiting and checksummed by the
way, giving you this property. So the only transfer encoding available
today in HTTP/1.1 not having this property is the "identity"
non-encoding without content-length.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 12 May 2008 16:10:10 +0100

Henrik Nordstrom wrote:
> way, giving you this property.

Are you sure?  Concatenating gzip files (.gz) is allowed: when
decompressed it results in the concatenation of the decompressed
parts.  Therefore gzip _files_ aren't self delimiting.

I don't know if gzip-as-referenced-by-HTTP allows that, but given that
gzip files can, it would be inadvisable for the network protocol to be
different and depend on that difference.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 18:37:11 +0200

> parts.  Therefore gzip _files_ aren't self delimiting.

True. But each part/member is, and if the sender sends a single member
it can be sure that the recipient can tell if the message got truncated
even if there is no other forms of delimiting. It's the sender that
selects to send the message without any other form of delimiting.

And in this case the protocol do not really depend on the delimiting
being detected proper. The message delimiting is the same as identity
encoding without content-length, by closing the connection. But unlike
identity encoding the recipient can clearly tell if the message got
unexpectedly truncated, with the small exception of a sender sending
multiple gzip members in the same stream and the message getting
truncated exactly between two members.

Regards
Henrik

---

From: Brian Smith <[email-masked]>
Date: Mon, 12 May 2008 09:54:08 -0700

> truncated exactly between two members.

Does this mean that it isn't possible to use deflate or gzip 
Transfer-Encoding on a persistent connection unless chunked encoding is 
applied (last)? And, doesn't the following imply that any 
transfer-encoding (besides identity) must be chunked? That is, "deflate" 
would not be a valid Transfer-Encoding, but "deflate, chunked" would me.

Regards,
Brian

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 12 May 2008 18:06:46 +0100

> would not be a valid Transfer-Encoding, but "deflate, chunked" would me.

I believe that's correct: to use deflate on a persistent connection,
you must have "Transfer-Encoding: deflate, chunked".

However, it is by far the common practice to use "Content-Encoding:
deflate" (or gzip) instead.  User agents which decompress those send
the appropriate Accept-Encoding.  In principle this reduces the work
required by a proxy, but limits the ability of a proxy to compress
some connections when one endpoint doesn't have the capability.  But
in reality, Transfer-Encoding and Content-Encoding are virtually
interchangable in this regard.

So that makes compression independent of transfer encoding.

But then there's this small problem of bugs in old servers and user
agents either setting or parsing Content-Length as the length _after_
compression, which you might want to avoid.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 19:17:19 +0200

On mån, 2008-05-12 at 09:54 -0700, Brian Smith wrote:

> applied (last)?

Correct. If transfer encodings is applied to the message and chunked
encoding isn't used the connection MUST be closed after the message to
signal end-of-message.

> And, doesn't the following imply that any 

No. See the last section "unless the message is terminated by closing
the connection". It's a bit vague however and some people have misread
it and that is what this issue is about.

See also the section on Transfer Encoding. The rules for message
delimiting when using transfer encoding is explained quite well there:

p1 3.4 Transfer Encodings

Whenever a transfer-coding is applied to a message-body, the set of
transfer-codings MUST include "chunked", unless the message is
terminated by closing the connection. When the "chunked" transfer-coding
is used, it MUST be the last transfer-coding applied to the
message-body. The "chunked" transfer-coding MUST NOT be applied more
than once to a message-body. These rules allow the recipient to
determine the transfer-length of the message (Section 4.4).

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 19:20:47 +0200

> interchangable in this regard.

But they are not..  Content-Encoding creates a new entity, while
transfer-encoding is fully transparent.

> So that makes compression independent of transfer encoding.

?
> compression, which you might want to avoid.

Which is partly why specs clearly say that if Transfer-Encoding is used
then Content-Length MUST be ignored, with the small exception for the
now removed case of "Transfer-Encoding: identity".

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 12 May 2008 18:52:08 +0100

> transfer-encoding is fully transparent.

Sure, they are conceptually quite different.  But in practice it can
be a treated as a mere configuration issue.

The real work does not have to be different.  A proxy can forward a
gzip transfer-encoded entity without decoding and reencoding it (same
work/buffering as content-encoding), and just as it can use a
different encoding on the incoming and outgoing links, it can also
(non-transparently) be configured to change the content-encoding.  I
acknowledge issues with Etag, range requests, non-transparency etc.

Doing that is not a HTTP proxy per spec, but it done nonethless in
some configurations, and it is useful.

> > So that makes compression independent of transfer encoding.

In practice.

> now removed case of "Transfer-Encoding: identity".

I was meaning Content-Length in conjunction with Content-Encoding, not
Transfer-Encoding.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 12 May 2008 22:44:08 +0200

> some configurations, and it is useful.

And breaking the evolution of HTTP quite noticeably.. Try deploying a
for example a WebDAV client behind such transforming proxy, or a client
fetching ranges.

> > > So that makes compression independent of transfer encoding.

I disagree. There is a lot more to HTTP than plain browsing, and these
proxies bending the HTTP often do so without knowing HTTP or the bad
effects they cause, and the ones deploying it often considers HTTP
"browsing only, nothing critical if it gets a bit messed up as long as
browsing to the major sites works".

> I was meaning Content-Length in conjunction with Content-Encoding, not

And where is the confusion there?

Content-Length with Content-Encoding is the message length, nothing
else. Anyone getting this wrong is seriously flawed.

Content-Encoding is a property of the resource returned, not of how it's
transferred.  Content-Encoding does NOT change the message format, only
the resource transferred. To the protocol very similar to
Content-Language or Content-Type but on a different axis.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 12 May 2008 22:27:09 +0100

> for example a WebDAV client behind such transforming proxy, or a client

If a WebDAV client says "Accept-Encoding: gzip" it will probably get
similar issues even with no proxy.

Many generic HTTP servers act as transformative "pseudo-proxies" to
their backend content - consider Apache with mod_gzip for example.
Therefore, WebDAV clients for general purpose use should not say
"Accept-Encoding: gzip" unless they handle the consequences, which
typically means transparently decompressing what's received.  They
don't have to, but user expectations won't be met when connecting to
some servers, and editing may fail.

Range requests: if the proxy is written properly it can work.

HTTP evolution: Proxies for general HTTP use, such as at ISPs and
gateways, should not be configured that way.

Only proxies for specific applications would enable transformations
like that (we hope).  An example is Apache with mod_gzip+mod_proxy
acting as a reverse proxy in a server farm (I don't know if that
really works).  That is why I call it a configuration issue.

> browsing to the major sites works".

This is more like "as long as using major browsers (site irrelevant)
works, or as long as using a client intended to generally work with
sites found on the net (because mod_gzip is popular enough that even
non-browser clients must work with it, or not use Accept-Encoding)."
It is indeed dirty, but not as specifically dirty as you make out.

It's also not common to do this in proxies, so don't worry about it.
What is common is automatic compression a la mod_gzip, in what is
technically not a HTTP proxy, but is still a generic relay between
HTTP client and HTTP services, and similar non-transparency issues do
apply there.

Besides, I bet a HTTP proxy which opportunistically applies
"Transfer-Encoding: gzip" encoding when permitted, and adds "TE: gzip"
to requests removing the encoding from forwarded responses, will cause
problems too - maybe even bigger ones - even though it's fully
compliant and transparent according to spec.

> Content-Language or Content-Type but on a different axis.

I know.  But spec isn't everything.

The serious flaw is deployed.  I'm not surprised - it's a predictable
mistake given how HTTP systems are architected.  When writing code you
can't ignore the installed base of buggy agents if you want to
interoperate.  But as I've implied, that particular bug is found (as
far as I know) only in old agents which are dwindling in presence, so
you might choose to ignore it now, depending on how much you care
about reaching those remaining.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 13 May 2008 09:59:32 +1200

Transfer-encoding and Content-Encoding are fundamentally different.  It 
helps if you look at it from the point of view of who does the encoding.

Transfer-Encoding is performed on the fly by something in the stream 
(e.g. proxy or output conversion process).  In such cases it's often 
impossible (i.e. non-deterministic length of output of encoding) to know 
the length of the whole transformed entity.  You may have the 
content-length (an entity attribute) as input to the transformation 
process, but may not be able to calculate the length after encoding on 
the fly.  Therefore the only way to signal length is to use chunking or 
close the connection.

Content-Encoding is different because the sender should know the length, 
therefore can set Content-Length headers. It is deemed a separate entity 
- an attribute of which is an encoding, but as far as HTTP is concerned 
it may as well not be encoded.  The encoding is meant for the end 
consumer of the message.

Sure the underlying technology is similar, but the message semantics are 
completely different.

That's also why Content-length is banned with Transfer-Encoding, because 
Content-Length is an entity attribute, just like Content-Type, and 
Content-Encoding.

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 May 2008 01:03:43 +0200

> gateways, should not be configured that way.

And yet there is plenty doing this, especially in bandwidth scarse parts
of the world, by looping all their traffic via a gzip proxy in a well
connected co-location, and some major proxy vendors advertising it as a
feature..

> HTTP client and HTTP services, and similar non-transparency issues do

I know, as can be seen in my discussions with the Apache team..

> compliant and transparent according to spec.

Not sure on that. There isn't many implementing TE: gzip today.

> I know.  But spec isn't everything.

Not short-term no, but when working with a timespan of several years
it's important.

> about reaching those remaining.

Personally I have very little respect for old broken user agents. Those
generally have major gaping security flaws as well and really SHOULD get
upgraded.

I care more about broken servers.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 13 May 2008 15:50:05 +0100

> connected co-location, and some major proxy vendors advertising it as a

Interesting.  I didn't realise those exist.  It's quite
understandable: cheap fast web browsing is important, and
Transfer-Encoding is widely supported enough agents to be used.

Other HTTP uses on those networks are relatively unimportant and will
have to workaround those transformations.  Fortunately it is quite
straightforward and the least of one's worries - there's far more
annoying HTTP-in-reality issues!

I rather like the idea of proxies which use delta compression between
each other.  Infinitely more effective.  But alas not zero-install for
the end users.

> Not short-term no, but when working with a timespan of several years

I agree, but I think the unfolding reality suggest ways the spec
doesn't match the ways people want to actually use HTTP.  We push
sound principles but we work with the ecosystem too.  Same reason
we're writing applications in HTML + Javascript even though it's messy
and slow.

>From Google, I have the impression the problem is more prevalant on
servers, if you're counting different programs (as opposed to number
of running instances).  But that it's also rare, occurring around the
time people weren't really using compression and just starting to
dabble in it.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 13 May 2008 17:32:39 +0100

> Transfer-Encoding is widely supported enough agents to be used.
                      ^^                      "in"

I forgot "not" and "in".

Cheers,
-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 13 May 2008 18:16:21 +0100

> helps if you look at it from the point of view of who does the encoding.

Agreed, but I think everyone who spoke in this thread knows the
difference.  The discussion hasn't denied that.  I can see why it
might look that way.

> (e.g. proxy or output conversion process).

No, that's an implementation detail outside the scope of HTTP.  It
_can_ be implemented that way, but HTTP does not say anything about
that or require it.

> In such cases it's often impossible (i.e. non-deterministic length
> of output of encoding) to know the length of the whole transformed
> entity.

It's often impossible, but it's often possible.  Agents can internally
cache gzip transfer-encoded bodies and HTTP permits that (it says
nothing about it).

> therefore can set Content-Length headers.

No, that's another implementation detail outside the scope of HTTP.
HTTP does not require the sender to know the length; it places no such
requirement, not even in principle.

In practice, many senders using Content-Encoding: gzip don't know the
length when they start sending, and use chunked encoding or close the
connection, which is allowed.

> It is deemed a separate entity - an attribute of which is an
> encoding, but as far as HTTP is concerned it may as well not be
> encoded.  The encoding is meant for the end consumer of the message.

That's all correct, but it has no bearing on how servers, clients and
proxies implement and generate those entities.

In practice, many servers _don't_ know the compressed Content-Length
when they begin transmitting unless they have an internal cache and
hit it.  Compressed Content-Encoding is usually implemented as
dynamically generated content.

> Sure the underlying technology is similar, but the message semantics are 

Yes, the semantics are different.

But the underlying work is so similar that they often use the same
implementation.  That includes dynamically generating compressed
content in _either_ case, dynamically decompressing at the receiver in
_either_ case, caching compressed representations in _either_ case, or
storing pre-compressed representations in _either_ case.

All those are allowed with HTTP, and they are good strategies for a
high quality implementation too.

> Content-Length is an entity attribute, just like Content-Type, and 

That makes no sense.  Specifically, "it's an entity attribute
therefore it is banned with Transfer-Encoding" makes no sense, because
other entity attributes are not banned with Transfer-Encoding.

It's a basic principle that transfer encoding is supposed to be
transparent to entity headers.  But that transparency is broken with
Content-Length.

The end consumer of a message is affected by this.

For example, downloading tools can't show percent progress if any
proxy hop decides to use a transfer-encoding - proving that
transfer-encoding is not transparent.

Imho, RFC 2616 section 4.4 rule 3 (Content-Length is not allowed with
transfer-coding) is illogical and inconsistent with Content-Length
being an entity header.  The rule is there only for compatibility, and
might be safe to relax in some cases where the receiver is known to
implement HTTP/1.1.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 May 2008 08:40:35 +1200

> difference.  The discussion hasn't denied that.  I can see why it
Understood - some posts didn't appear clear on it, and there are many 
more reading this list than posting to it.

> _can_ be implemented that way, but HTTP does not say anything about
sure.  Insert "commonly" after "Transfer-Encoding is..."

> cache gzip transfer-encoded bodies and HTTP permits that (it says

OK.  That blurs the boundaries a fair bit.
> connection, which is allowed.
Another implementation detail I guess.  If a server is creating encoded 
versions of resources, and caching them (to save subsequent CPU cycles 
on re-encoding), they they may create another entity due to race 
conditions and synch issues (i.e. if the source is often changed, it 
becomes difficult to keep the encoded version in synch with the original).

Encoding it all on the fly as an output transformation process, sending 
chunked etc but using Content-Encoding is basically using 
Content-Encoding to do Transfer-Encoding's job.

There are major side-effects, not the least interop with HTTP/1.0 agents 
in the request chain, so I can see why people would take this route 
(although they then can't use chunking). 

I guess another fundamental difference, is it's legal for a proxy to 
convert a stream to Transfer-Encoded, but not to do so for 
Content-Encoded, since it breaks the entity tag for caching.

> transfer-encoding is not transparent.
OK.  I was trying to come up with the rationale why Content-Length was 
banned with T-E.  I guess the answer is due to backward compatibility 
issues.

> might be safe to relax in some cases where the receiver is known to
Actually I agree.  I've posted before about using Content-Length + 
chunking for instance to communicate at the start what the entity length 
(after removing any transfer-encoding) should be.  This is in many cases 
VERY important information, and precluding the possibility of 
transmitting that information when it may actually be available simply 
because of a T-E in use seems unnecessary and wasteful.

But that would break many HTTP/1.0 intermediaries, so I think we're 
stuck with using another header, or an extension to a T-E header.

Actually I'd prefer it entity headers could be associated with the 
entity.  E.g. with multipart responses, have a block of entity headers 
per entity, and keep them outside the general message header block.  
(e.g. another blank line, or mime multipart boundaries) This would mean 
intermediaries wouldn't even need to parse entity headers, just message 
headers.  But that's no longer HTTP.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 May 2008 23:20:03 +0200

> each other.  Infinitely more effective.  But alas not zero-install for

Why not zero-install for the end users? They talk to a proxy which does
delta to it's upstream proxy on the other side of the weak/expensive
link..

But delta encoding of HTTP is a different topic entirely, and also not
within the scope of this WG. But still interesting.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 13 May 2008 23:16:13 +0200

> other entity attributes are not banned with Transfer-Encoding.

It's banned because Content-Length is BOTH a entity header and a
transport header. But unfortunately the specs does not make a clear
distinction between transport and message..

Other transport headers are Transfer-Encoding, Connection, Trailer,
Upgrade, TE and Keep-Alive. Expect could also be counted as transport
but it's another odd beast..

> might be safe to relax in some cases where the receiver is known to

Maybe in some years. Something to revisit for HTTP/1.2.

The reals reason why it's there is to prevent badness when chunked
encoding is removed by an intermediary hop and the advertised
Content-Length does not match the acual length sent by chunked encoding.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 May 2008 00:48:04 +0100

> delta to it's upstream proxy on the other side of the weak/expensive

Because often the link from end user to the nearest proxy is the
slowest link.  (This is especially true of mobile phones.)  To get
gains, a delta coding proxy needs to be installed on the end user's
own equipment.

But if you're interested, look up rproxy which does exactly as you suggest.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 May 2008 00:54:57 +0100

> Content-Length does not match the acual length sent by chunked encoding.

Any intermediary which strips chunked (+ any other transfer-) encoding
and doesn't ensure what it sends matches Content-Length (and abort if
they don't) is buggy according to HTTP/1.1.

What makes you think HTTP/1.2 will fix that bug in implementations?

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 May 2008 01:02:07 +0100

> (although they then can't use chunking). 

Connection: close is used instead with HTTP/1.0 agents, if it's worth
compressing.

> Content-Encoded, since it breaks the entity tag for caching.

Yes.  Though some do it anyway.  Maybe they change the entity tag to
something unlikely to match a server generated tag (like map server's
"0405232430" to "proxy-generated-gzip-0405232430").  Or maybe they are
just broken.

> stuck with using another header, or an extension to a T-E header.

Why would it break HTTP/1.0 intermediaries?  Transfer-Encoding should
never be sent to HTTP/1.0 intermediaries.

It seems to me it's only to protect against buggy HTTP/1.1 receivers.
That sort of thing is normally left out of the RFC, and put into
documents like "Apache's list of necessary workarounds for deployed
buggy implementations".

-- Jamie

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 15 May 2008 21:04:53 +0200

Mark Nottingham wrote:

 [old text]
> unless the message is terminated by closing the connection.

Remotely related, I use a HTTP/1.0 GET script, and this
script, after sending its request, closed the sending
direction to indicate that it now only waits for an
answer.  Recently I stumbled over a HTTP/1.1 server
sending only what would be the first chunk in HTTP/1.1,
instead of the complete file.

Who got it wrong, my script or this server ?  I got the
complete file when I disabled the code terminating the
sending direction (from my POV).

 Frank

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 May 2008 21:58:23 +0200

> sending direction (from my POV).

Behaviour is somewhat unspecified if the client closes it's transmit
channel while still waiting for a response, so don't do it. Some servers
will give you the whole response, some will assume the client aborted
and abort the transaction.

There is arguments in favor for both server behaviours.

In general clients should only close it's transmit channel in response
to the server closing the connection.

"Connection: close" is used by the client if it want to signal that
there is no further requests to be expected on this connection. (or in
case of HTTP/1.0 no Connection: keep-alive header..)

Regards
Henrik

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 15 May 2008 23:46:58 +0200

Henrik Nordstrom wrote:
 
> Behaviour is somewhat unspecified if the client closes
> it's transmit channel while still waiting for a response,
> so don't do it.

Okay, and actually I forgot where this helped years ago...

> "Connection: close" is used by the client if it want to
> signal that there is no further requests to be expected
> on this connection. (or in case of HTTP/1.0 no 
> Connection: keep-alive header..)

...yes, I had this already.  Is the "unspecified HTTP/1.0
behaviour" something that should be mentioned in 2616bis ?

Thanks for info,

 Frank

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 17 May 2008 00:01:13 +0200

> behaviour" something that should be mentioned in 2616bis ?

"Connection: close" is fine in both versions.

The HTTP/1.0 keep-alive model is fairly well documented, and not
relevant to HTTP/1.1 beyond the little that is already there making
HTTP/1.1 servers behave properly in response to HTTP/1.0 requests.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 23 May 2008 14:30:25 +1000

Stepping back, the original issue was that the phrase "terminated by  
closing the connection" is vague; it doesn't say how the connection is  
closed, or who does it.

> unless the message is terminated by closing the connection.

In both of these cases, it may be better to say something like "unless  
the message indicates it is terminated by closing the connection  
(i.e., with Connection: close)."

As Henrik noted, we've removed "identity", so that should be factored  
out as well. This leaves (for 4.4):

If a Transfer-Encoding header field (section 14.41) is present, then  
the transfer-length is defined by use of the "chunked" transfer-coding  
(section 3.6), unless the message indicates it is terminated by  
closing the connection.

This addresses the case that H brought up;

HTTP/1.1 200 OK
Transfer-Encoding: gzip
Connection: close

but is unclear about this:

HTTP/1.1 200 OK
Transfer-Encoding, gzip, chunked
Connection: close

What determines the tranfser-length here?

Changing things to refer explicitly to "chunked" may help;

2. If the Transfer-Encoding header field (section 14.41) is present  
and contains the "chunked" value, the transfer-length is defined by  
the use of the "chunked" transfer encoding.

Note that 3.6 already requires that the chunked value be listed if  
chunking is in use.

3.6 would still benefit from:
Whenever a transfer-coding is applied to a message-body, the set of  
transfer-codings MUST include "chunked", unless the message indicates  
it is terminated by closing the connection.

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 23 May 2008 21:49:14 +0200

> it is terminated by closing the connection.

Looks fine to me.

Regards
Henrik

---

From: Joe Orton <[email-masked]>
Date: Fri, 23 May 2008 23:17:24 +0100

On Fri, May 23, 2008 at 02:30:25PM +1000, Mark Nottingham wrote:
> 2. If the Transfer-Encoding header field (section 14.41) is present and 
> contains the "chunked" value, the transfer-length is defined by the use of 
> the "chunked" transfer encoding.
>
> Note that 3.6 already requires that the chunked value be listed if chunking 
> is in use.
>
> transfer-codings MUST include "chunked", unless the message indicates it is 

I'm still just as confused.

What delimits the end of the message in the gzip transfer-coding?  Must 
it always used with chunked or not?  You gave the message:

HTTP/1.1 200 OK 
Transfer-Encoding: gzip
Connection: close

as an example, presumably to illustrate that the gzip coding is 
delimited by connection closure and not used with chunked?  So what 
about use of the gzip transfer-coding in request messages?  The client 
is supposed to do a half-shutdown of the TCP connection to indicate 
closure?!

I think someone mentioned the gzip transfer-coding is actually use by 
some implementations - what do they do?  Wrap it in chunked or not?

joe

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 24 May 2008 01:13:29 +0200

On fre, 2008-05-23 at 23:17 +0100, Joe Orton wrote:

> What delimits the end of the message in the gzip transfer-coding?

Chunked encoding, or if chunked is not used by closing the connection.

> about use of the gzip transfer-coding in request messages?

As closing the connection is not an option available for delimiting
requests chunking MUST be used on requests with transfer encoding.

half-closed connections is not supported as delimiting method.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Sat, 24 May 2008 09:52:42 +0200

Henrik Nordstrom wrote:
> ...

Which reminds me of a problem we've seen in the past...:

With chunked encoding in use, how does the sender of a message indicates 
an error? (such as when serving a file from the file system, and getting 
an IO error)

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 24 May 2008 11:51:56 +0200

> an error? (such as when serving a file from the file system, and getting 

By closing the connection in the middle of the response without sending
the last-chunk.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 24 May 2008 11:57:45 +0200

> By closing the connection in the middle of the response without sending

Or if you meant "Witch chunked encoding NOT in use" then it's still the
same, by closing the connection in the middle of the response. How easy
the error is to detect for the recipient then depends on the encoding or
format of the response.

T-E: gzip/deflate both have stream structure and clearly indicates
premature closure (except for the hypothetical case of a sender sending
multiple members in a gzip stream and crashing just between two
members).

If no T-E is used and Content-Length is used then it's trivially
detected by the response beeing too short.

If no T-E is used and no Content-Length then it's tricker as the receier
then has to fully parse the response and from that judge if it's
complete or not, and then guess if this is due to a server failure or if
the object at the server really is corrupted/truncated...

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Sat, 24 May 2008 12:01:50 +0200

> By closing the connection in the middle of the response without sending

Yes, if course (I shouldn't have phrased that as a question).

What I was trying to say is that, when streaming out a response using 
chunked encoding, this is the *only* way to signal an error, and

1) should make sure that httpbis still allows it, and

2) potentially point out that this is the only way to do it.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 24 May 2008 12:05:06 +0200

> the object at the server really is corrupted/truncated...

Well, in the generic case, the recipient hasn't got sufficient 
information for parsing (think browser doing file download).

BR, Julian

---

From: Yves Lafon <[email-masked]>
Date: Wed, 28 May 2008 10:41:51 -0400 (EDT)

> Chunked encoding, or if chunked is not used by closing the connection.

Then we must at least say that the encoding used (not chunked) must 
give the same characteristics as chunked wrt detection of the end of the 
message.

Is this particular case really used in deployed client/servers, and done 
the right way ?

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 28 May 2008 21:19:38 +0200

> give the same characteristics as chunked wrt detection of the end of the 

Why? The protocol will not fall down if a message is unexpectedly cut
short.

A note mentioning that this may downgrade the message integrity to the
level of a Connection: close without Content-Length message may be
acceptable, but not a must. In practice most file formats and encodings
easily detect truncation.

> Is this particular case really used in deployed client/servers, and done 

As already mentioned both gzip and deflate has this property to a
sufficient level.

Regards
Henrik