Subject: Content-Length and 304

From: Zhong Yu <[email-masked]>
Date: Wed, 19 Sep 2012 20:22:27 -0500

In the latest bis draft, a 304 response SHOULD set Content-Length
equal to the length of the would-be payload body.

====
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-20#section-3.3.2
3.3.2. Content-Length

... a Content-Length header field SHOULD be sent to indicate the
length of the payload body that ... would have been present had the
request been an unconditional GET.

...In the case of a 304 response to a GET request, Content-Length
indicates the size of the payload body that would have been sent in a
200 response.
====

However, RFC2616 was not specific on the matter. If a server
implementation always sets "Content-Length: 0" for 304 responses, it
was acceptable. But now it becomes non-compliant under the new spec.

Is this new requirement really necessary? Who would benefit from it?

According to the reasoning of

====
http://tools.ietf.org/html/draft-ietf-httpbis-p4-conditional-20#section-4.1
4.1. 304 Not Modified
Since the goal of a 304 response is to minimize information transfer
when the recipient already has one or more cached representations, the
response SHOULD NOT include representation metadata other than the
above listed fields
====

we SHOULD NOT include Content-Length in 304 responses either.

I think the meaning of Content-Length is confusing enough, in the case
of 204/304 response, it's better to require that
1. server SHOULD NOT set Content-Length
2. client MUST ignore Content-Length

Zhong Yu

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Sep 2012 22:05:17 -0400

I think that the Content-Length handling is illogical. I can say that as I
originally proposed the header.

In the wake of keep-alive, Content-Length has two purposes; 1) content
meta-data and 2) specifying how much data is to follow.

My original intention was that it would only be the second. Then people
decided that HEAD methods should return the content-length and then things
got ugly.

Problem is that now you have to understand the error code semantics to get
the framing right. That seems like a design error to me.

-- 
Website: http://hallambaker.com/

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Sep 2012 19:22:43 -0700

I had missed this. I just revisited 2616 and unless I misread, this is
a new thing.  This appears to me to add substantial workload to the
server and I don’t comprehend the benefit.  Either I missed something,
or YAGNI.  -T

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Sep 2012 22:35:53 -0400

I see no benefit either. I am certain that there is no benefit if
implementation is less than 100% and I am pretty sure there isn't a point
if there is 100% implementation.

It also boxes the use of the conditional get in. Say the conditional get is
for a generated page with lots of content that is user specific and an
article that is updated from time to time. The server would have to do a
lot of work to calculate the content length for the generated page but it
can check to see if it needs to refresh by checking the sources.

If we are going to make any change here I would suggest the opposite
approach and deprecate sending a Content-Length other than 0.

I would like to see those two purposes separate in HTTP/2.0 in any case.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 19 Sep 2012 22:01:30 -0500

On Wed, Sep 19, 2012 at 9:35 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> implementation is less than 100% and I am pretty sure there isn't a point if
> there is 100% implementation.
>
> article that is updated from time to time. The server would have to do a lot
> of work to calculate the content length for the generated page but it can
> check to see if it needs to refresh by checking the sources.

I guess the author intended the header to be optional - only set it if
convenient.

Nevertheless, previously Content-Length was meaningless in 204/304
responses; now it is assigned a meaning in 304. That can be
surprising.

Zhong Yu

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Sep 2012 23:13:44 -0400

Possibly, but I can't see much value to returning Content-Length in the
first place and only returning it sometimes seems like its completely
pointless.

On Wed, Sep 19, 2012 at 11:01 PM, Zhong Yu <[email-masked]> wrote:

> lot

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 20 Sep 2012 03:24:56 +0000

from a cache POV, it could potentially be yet another validator for 
correctness of the currently cached object.

But there's already Modified-Date, and ETag for that.  So it adds 
little there.

As for framing, yep it's a shame C-L isn't purely a framing field.  
Interesting to read parts of the spec I didn't recall, such as 
requirements to ignore C-L if Transfer-Encoding is not identity.

Makes following message boundaries a bit of a nightmare, surely we can 
do a lot better in HTTP/2.0

Adrien

------ Original Message ------
From: "Phillip Hallam-Baker" <[email-masked]>
To: "Zhong Yu" <[email-masked]>
Cc: "Tim Bray" <[email-masked]>;"HTTP Working Group" 
<[email-masked]>
Sent: 20/09/2012 3:13:44 p.m.
Subject: Re: Content-Length and 304
>Possibly, but I can't see much value to returning Content-Length in 
>the first place and only returning it sometimes seems like its 
>completely pointless.  
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 20 Sep 2012 07:11:47 +0200

Hi,

> Is this new requirement really necessary? Who would benefit from it?

I remind an old application I saw about 10 years ago which fell into
this trap of sending "Content-Length: 0" along with 304 responses. The
result was blank pages for clients fetching these responses from a cache
(apache 1.3), because the cache used the headers fetched from the response
to update the ones it had in the cache and subsequently delivered empty
objects only.

I think that *all* header fields must be sent with a 304 as they would
have been with the real response, it avoids such trouble.

> 2. client MUST ignore Content-Length

It's true that this part is confusing, we all have an exception for this
in client code to ignore the length (suggestion #2 above).

However, we also know that at least some clients break if we send a wrong
content length, so it should probably be the same if we don't send any.
In my opinion, better leave this untouched to remain compatible with what
is deployed. If you think that some points are still confusing in the spec,
maybe we should try to improve the wording to remove any confusion.

Regards,
Willy

---

From: Zhong Yu <[email-masked]>
Date: Thu, 20 Sep 2012 00:43:58 -0500

How about HTTP/1.0 clients, as far as you know - if the connection is
keep-alive, can they handle 204/304 framing correctly if
Content-Length is missing, or set to a non-zero value. What's the best
effort to keep a 1.0 connection alive when responding 204/304? Or is
it safer to just close the connection after responding.

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Sep 2012 23:05:27 -0700

> have been with the real response, it avoids such trouble.

This defeats a common optimization in Web frameworks where a request
comes in with an ETag, and you have a trick where you compute the ETag
based on last-update timestamps of a couple of tables and maybe a
global update sequence number, and if the ETag matches you fire off
the 304 and go to sleep, having saved lots of IO & templating.

So the right thing to do is to either send headers which match the
real response, or don’t send them. -T

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 20 Sep 2012 08:17:53 +0200

On Wed, Sep 19, 2012 at 11:05:27PM -0700, Tim Bray wrote:
> real response, or don?t send them. -T

Well, I don't know what the cache did when the header was not sent, but
I would not be surprized that it would consider the object as empty.
Don't get me wrong, I'm not saying this is fine nor a desired behaviour,
I'm talking about issues I've had to deal with in production.

What we're documenting in this spec is how to ensure the best
interoperability. If people have 100% positive experience with not
returning a content-length when the request contains an if-none-match,
maybe that's acceptable too, but I have not made the tests.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 20 Sep 2012 08:23:32 +0200

> it safer to just close the connection after responding.

I don't know but it would be a waste of resource to close on 304, since
it's designed to be fast. However, you should note that keep-alive was
not specified in 1.0 and was just implemented on top of the spec. So it
is very likely that some implementations will differ.

FWIW, on the implementation where I observed the truncated objects
behaviour with "Content-Length: 0", the connection was made in HTTP/1.0
and was not using keep-alive. I *think* it was Apache 1.3.12 with
mod_proxy.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 20 Sep 2012 06:39:21 +0000

In message <[email-masked]>, Willy Tarreau writes:

>behaviour with "Content-Length: 0"

In the testing I did with Varnish, I saw no credible differences
between a correct C-L, incorrect C-L or no C-L on 304 responses.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 11:46:51 +0200

On 2012-09-20 03:22, Zhong Yu wrote:
> ...

That was the case since -19 (just clarifying).

I also note that the requirements in P1 (Content-Length) and P4 (status 
code 304) do not seem to be totally in sync.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Sep 2012 10:27:31 -0700

My experience has always been that the C-L on a 304 represents the length of the response had it been whole, and I see that as just a clarification of current practice. 

However, I agree that the SHOULD is too strong here, as it makes several existing implementations non-conformant. 

Cheers,

On 20/09/2012, at 2:46 AM, Julian Reschke <[email-masked]> wrote:

> I also note that the requirements in P1 (Content-Length) and P4 (status code 304) do not seem to be totally in sync.
> 
> Best regards, Julian
> 

--
Mark Nottingham
http://www.mnot.net/

---

From: Tim Bray <[email-masked]>
Date: Thu, 20 Sep 2012 10:38:16 -0700

It seems like a CL of 0 (or anything other than the actual representation’s
length) is just wrong.  I suggest language to the effect of “The
Content-length value, if provided, MUST be accurate”.  I.e. if the server
chooses not to provide the correct answer, it should not send the header.
Or is there a case for a reserved value of -1 meaning “Not provided”?

I have to say that I don’t understand the motivation for encouraging
Content-length with 304 at all; that’s another argument, but I think we can
agree that it should either be accurate or absent.

FWIW, when I was making my living writing large-scale Web spiders, I
quickly learned to just treat Content-length as science fiction, suck it up
and deal with whatever data the server actually sent. But that was a decade
ago; maybe things are better now.

-Tim

On Thu, Sep 20, 2012 at 10:27 AM, Mark Nottingham <[email-masked]> wrote:

> My experience has always been that the C-L on a 304 represents the length
> of the response had it been whole, and I see that as just a clarification
> of current practice.
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 21 Sep 2012 22:11:47 -0700

On Sep 19, 2012, at 6:22 PM, Zhong Yu wrote:

> was acceptable.

No, that has never been acceptable --- a change in content-length
will cause a cache flush on many deployed implementations
(Netscape introduced that in the mid 90s), and will truncate
valid responses in others.

Depending on how you read 2616, the SHOULD send Content-Length
did not apply to messages that are not allowed to have a body.
Hence, it was considered valid to not send Content-Length on 304.
I messed that up when I merged the several paragraphs into
one requirement.

I have attempted a fix in

  http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1908

that should remove any ambiguity about when content-length is
sent.  Please review.

....Roy

---

From: Tim Bray <[email-masked]>
Date: Fri, 21 Sep 2012 22:59:06 -0700

Roy, that’s a hairy diff, I was having trouble with it, but I think it’s
saying that you MAY send a CL with a 304 containing the correct size, which
is what we want.  Given the record of bad practice in this space, might it
be worth while to say the server MUST NOT act stupid and send a 0 or
anything else inaccurate? -T

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 22 Sep 2012 00:07:12 -0700

On Sep 21, 2012, at 10:59 PM, Tim Bray wrote:

> Roy, that’s a hairy diff, I was having trouble with it,

It is easier if you skip the html part and just look at the
diff on the xml source.  Next time I will remember to include the
fragment ...

 http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1908#file1

> but I think it’s saying that you MAY send a CL with a 304 containing the correct size, which is what we want.  Given the record of bad practice in this space, might it be worth while to say the server MUST NOT act stupid and send a 0 or anything else inaccurate? -T

Given the unbounded set of stupid things a server might do,
I'd rather not.  However, I can restrict it further.

Instead of 

   A server &MAY; send a Content-Length header field in a 
   <x:ref>304 (Not Modified)</x:ref> response to a conditional GET request 
   (&status-304;) in order to indicate the size of the payload body, 
   excluding any potential transfer-coding, that would have been sent in a 
   <x:ref>200 (OK)</x:ref> response. 

how about

   A server &MAY; send a Content-Length header field in a 
   <x:ref>304 (Not Modified)</x:ref> response to a conditional GET request 
   (&status-304;); a server &MUST-NOT; send Content-Length in such a response
   unless its field-value equals the decimal number of octets that would
   have been sent in the payload body of a <x:ref>200 (OK)</x:ref> response
   to the same request.

Cheers,

....Roy

---

From: Tim Bray <[email-masked]>
Date: Sat, 22 Sep 2012 00:11:25 -0700

+1 -T

On Sat, Sep 22, 2012 at 12:07 AM, Roy T. Fielding <[email-masked]> wrote:

> > but I think it’s saying that you MAY send a CL with a 304 containing the
> correct size, which is what we want.  Given the record of bad practice in
> this space, might it be worth while to say the server MUST NOT act stupid
> and send a 0 or anything else inaccurate? -T
>

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 22 Sep 2012 09:47:55 +0200

+1 too

On Sat, Sep 22, 2012 at 12:11:25AM -0700, Tim Bray wrote:
> > > Roy, that?s a hairy diff, I was having trouble with it,
> >
> > > but I think it?s saying that you MAY send a CL with a 304 containing the

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 22 Sep 2012 12:09:48 -0400

That looks good to me:

The length of the body section MUST be specified whenever there is a body.
This MAY be indicated using Content-Length or Chunked ecoding [or MAY be
some other negotiated encoding]

Content-Length MAY be specified for a 304 response (and possibly a list of
others) and MUST be the length of the content referenced if specified.

I am a little worried though about this business of whether there is a body
or not. That seems confusing and possibly means that the client has to
understand the error code to know whether to look for the body or not.

I suggest that we grandfather the existing response codes and specify any
exceptions (304 being one, are there others) and for the rest have the rule
that if Content-Length (or a content-encoding) is specified then there MUST
be a body and it indicates the length of the body. If there is no body the
content length MUST be omitted.

Would it harm things to make this a universal rule?

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 22 Sep 2012 18:19:38 +0200

> content length MUST be omitted.

There are other conditions. Content-length is ignored for response messages
which do not include a body, which means intermediary codes 1xx, code 204,
code 304, and final responses to HEAD requests.

I don't think we should try to change the rules. We've had issues for years
with determining the body length but such issues have been fixed along the
time, so it would be better not to try to reformulate too much the rules,
which would risk to cause some confusion.

What Roy has proposed seems OK to me, it definitely closes the issue with
the 304 code, we should not try to go further in my opinion.

Regards,
Willy

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sat, 22 Sep 2012 09:27:01 -0700

On Sat, Sep 22, 2012 at 9:09 AM, Phillip Hallam-Baker <[email-masked]>wrote:

> I am a little worried though about this business of whether there is a
> body or not. That seems confusing and possibly means that the client has to

While I agree it can be confusing, I just wanted to note that clients
already have to understand the HTTP status code, as per section RFC 2616
section 4.3:
"""

For response messages, whether or not a message-body is included with
   a message is dependent on both the request method and the response
   status code (section 6.1.1). All responses to the HEAD request method
   MUST NOT include a message-body, even though the presence of entity-
   header fields might lead one to believe they do. All 1xx
   (informational), 204 (no content), and 304 (not modified) responses
   MUST NOT include a message-body. All other responses do include a
   message-body, although it MAY be of zero length.

"""

and
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-20#section-3.3.3
"""

   1.  Any response to a HEAD request and any response with a 1xx
       (Informational), 204 (No Content), or 304 (Not Modified) status
       code is always terminated by the first empty line after the
       header fields, regardless of the header fields present in the
       message, and thus cannot contain a message body.

"""

> On Sat, Sep 22, 2012 at 1:11 AM, Roy T. Fielding <[email-masked]>wrote:
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 22 Sep 2012 13:23:40 -0400

I would like to be able to add in new response codes if that looks like it
is useful.

That is not possible if sending an unknown response code causes an existing
client to lose framing. All that a client needs to know at present is how
to interpret the first digit of the code. Provided the unknown code is a
1xx, 2xx, 3xx, 4xx or 5xx as appropriate then the client should not break.

The rules seem to be designed to allow this behavior but some people seem
to have implemented them in ways that introduce ambiguity. What I am
suggesting is that we tighten the guidance so that the ambiguity is
eliminated. That does not cause any loss of backwards compatibility and it
improves forward compatibility. I think that is a win-win.

On Sat, Sep 22, 2012 at 12:27 PM, William Chan (陈智昌)
<[email-masked]>wrote:

>> The length of the body section MUST be specified whenever there is a
>> body. This MAY be indicated using Content-Length or Chunked ecoding [or MAY
>> be some other negotiated encoding]
>>
>> Content-Length MAY be specified for a 304 response (and possibly a list
>> of others) and MUST be the length of the content referenced if specified.
>>

---

From: Zhong Yu <[email-masked]>
Date: Sat, 22 Sep 2012 12:24:37 -0500

I did not know C-L in 304 had a meaning on many deployed
implementations, thanks for clarify and include that meaning in the
spec.

I'm still not quite clear about the benefit though. Any use case where
a C-L in 304 is better than no C-L? Is it alright for a server to
categorically exclude C-L in 304s?

Zhong Yu

---

From: Zhong Yu <[email-masked]>
Date: Sat, 22 Sep 2012 12:30:53 -0500

I guess any new response code would have to have a body. So the
body-less oddity is contained in present state forever.

On Sat, Sep 22, 2012 at 12:23 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> client to lose framing. All that a client needs to know at present is how to
> interpret the first digit of the code. Provided the unknown code is a 1xx,
> 2xx, 3xx, 4xx or 5xx as appropriate then the client should not break.
>
> The rules seem to be designed to allow this behavior but some people seem to
> have implemented them in ways that introduce ambiguity. What I am suggesting
> is that we tighten the guidance so that the ambiguity is eliminated. That
> does not cause any loss of backwards compatibility and it improves forward
> compatibility. I think that is a win-win.
>
>
>
>
> On Sat, Sep 22, 2012 at 12:27 PM, William Chan (陈智昌) <[email-masked]>
> wrote:
>>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 22 Sep 2012 11:15:03 -0700

On Sep 22, 2012, at 10:23 AM, Phillip Hallam-Baker wrote:

> I would like to be able to add in new response codes if that looks like it is useful. 
> 
> That is not possible if sending an unknown response code causes an existing client to lose framing. All that a client needs to know at present is how to interpret the first digit of the code. Provided the unknown code is a 1xx, 2xx, 3xx, 4xx or 5xx as appropriate then the client should not break.
> 
> The rules seem to be designed to allow this behavior but some people seem to have implemented them in ways that introduce ambiguity. What I am suggesting is that we tighten the guidance so that the ambiguity is eliminated. That does not cause any loss of backwards compatibility and it improves forward compatibility. I think that is a win-win.

See the quoted text. "All other responses do include a message body,
although it may be of zero length."

What you suggested (don't send Content-Length) means the body size
is unknown and the client continues reading until the connection is
closed.  That's the only way it could have been defined given that
HTTP/1.0 existed before C-L was defined for framing.  It seems
to work well enough.

....Roy

> On Sat, Sep 22, 2012 at 12:27 PM, William Chan (陈智昌) <[email-masked]> wrote:
> On Sat, Sep 22, 2012 at 9:09 AM, Phillip Hallam-Baker <[email-masked]> wrote:
> That looks good to me:
> 
> The length of the body section MUST be specified whenever there is a body. This MAY be indicated using Content-Length or Chunked ecoding [or MAY be some other negotiated encoding]
> 
> Content-Length MAY be specified for a 304 response (and possibly a list of others) and MUST be the length of the content referenced if specified.
> 
> I am a little worried though about this business of whether there is a body or not. That seems confusing and possibly means that the client has to understand the error code to know whether to look for the body or not.
> 
> While I agree it can be confusing, I just wanted to note that clients already have to understand the HTTP status code, as per section RFC 2616 section 4.3:
> """
> and http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-20#section-3.3.3
> """
> I suggest that we grandfather the existing response codes and specify any exceptions (304 being one, are there others) and for the rest have the rule that if Content-Length (or a content-encoding) is specified then there MUST be a body and it indicates the length of the body. If there is no body the content length MUST be omitted.
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 22 Sep 2012 15:02:22 -0400

Aw crud I had forgotten the connection close thing

304 post dated CL but predated keep alive.

Main thing is not making any more exceptions.

Sent from my iPhone

On Sep 22, 2012, at 2:15 PM, "Roy T. Fielding" <[email-masked]> wrote:

....Roy

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 23 Sep 2012 13:34:05 +1200

> Main thing is not making any more exceptions.

+1 to that. And noting this C-L oddity as one of the details to be 
corrected for a consistent syntax in 2.0+.

AYJ