Subject: Re: i28 proposed replacement text

From: Yves Lafon <[email-masked]>
Date: Mon, 2 Jun 2008 03:10:28 -0400 (EDT)

On Wed, 28 May 2008, Henrik Nordstrom wrote:

> On ons, 2008-05-28 at 10:41 -0400, Yves Lafon wrote:
>
>> Then we must at least say that the encoding used (not chunked) must
>> give the same characteristics as chunked wrt detection of the end of the
>> message.
>
> Why? The protocol will not fall down if a message is unexpectedly cut
> short.

Essentially for what it below, the need to verify the integrity of the 
message. It is needed when you don't cut the connection, but it is a 
desirable property even if the connection is cut, to detect an error case 
(like cutting the connection in the middle of a chunked stream) from a 
normal response.

> A note mentioning that this may downgrade the message integrity to the
> level of a Connection: close without Content-Length message may be
> acceptable, but not a must. In practice most file formats and encodings
> easily detect truncation.

partial or absolute detection ? If the format allows multiple concatenated 
entries, and integrity check is true if you reach exactly the end of one 
entry, you have a highly probable integrity check but not an absolute one.

In any case, a note explaining that it might be difficult to figure out 
the integrity of the message in this case would be sufficient.

>> Is this particular case really used in deployed client/servers, and done
>> the right way ?
>
> As already mentioned both gzip and deflate has this property to a
> sufficient level.

But it is used that way in deployed servers/clients ? Or is chunked 
encoding always the norm.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 02 Jun 2008 10:33:30 +0200

> Essentially for what it below, the need to verify the integrity of the 

Which in HTTP is provided by Content-MD5 (even if partly misunderstood
on 206 responses), or by Digest qop=auth-int (if implemented by
anyone..)

> It is needed when you don't cut the connection, but it is a 

Sure, there is many things which is desireable and nice, but it doesn't
make them MUSTs..

> entry, you have a highly probable integrity check but not an absolute one.

You don't hav an absolute one even if Content-Length matches or chunked
encoding is terminated proper. It all depends on where the failure was
and how communication was to that.. In any situation where there may be
some form or proxy inbetween (including servers running scripts) both
Content-Length and chunking is synthetic and hop-by-hop. As recipient
you can only be absolutely sure that if there is a mismatch then
something is wrong..

As sender you can be reasonably sure that if you explicitly indicate the
length by Content-Length then the recipient will get that, but chunking
does not provide this level of integrity as it's a hop-by-hop feature
which may be added/removed any number of times along the response path.

> encoding always the norm.

Not sure one yet can speak of a norm when talking about
transfer-encoding gzip/chunked...

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 2 Jun 2008 11:31:08 +0100

> encoding always the norm.

Chunked encoding clearly is not the norm when sending compressed
dynamic content to a HTTP/1.0 client.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 2 Jun 2008 11:35:53 +0100

Henrik Nordstrom wrote:
> and how communication was to that..

Agree, except we can at least identify when it's not the protocol
itself introducing undetected failure.

> In any situation where there may be

According to RFC2616, Content-Length is an entity header and an
end-to-end header, therefore should not be synthesised hop-by-hop.
Yeah, I know :-)

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 02 Jun 2008 18:14:23 +0200

> dynamic content to a HTTP/1.0 client.

We are talking about transfer-encoding and message delimiting, not
content-encoding.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 02 Jun 2008 18:19:25 +0200

> end-to-end header, therefore should not be synthesised hop-by-hop.

Content-Length is that odd entity header which is also a hop-by-hop
header.

See "Non-modifiable Headers" which among other things states

   The Content-Length field of a request or response is added or deleted
   according to the rules in section 4.4. A transparent proxy MUST
   preserve the entity-length (section 7.2.2) of the entity-body,
   although it MAY change the transfer-length (section 4.4).

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 2 Jun 2008 17:31:38 +0100

> We are talking about transfer-encoding and message delimiting, not

True, but the same message integrity issues w.r.t connection aborts
apply to Content-Encoding used for compressed dynamic content.  Since
that is how it's done most often right now, it seems relevant.

-- Jamie

---

From: Yves Lafon <[email-masked]>
Date: Tue, 3 Jun 2008 04:41:55 -0400 (EDT)

> that is how it's done most often right now, it seems relevant.

My issue is not about interaction between 1.0 and 1.1, but more about 
integrity/error detection in 1.1.

In p1 3.4:
<<
    Whenever a transfer-coding is applied to a message-body, the set of
    transfer-codings MUST include "chunked", unless the message is
    terminated by closing the connection.  When the "chunked" transfer-
    coding is used, it MUST be the last transfer-coding applied to the
    message-body.  The "chunked" transfer-coding MUST NOT be applied more
    than once to a message-body.  These rules allow the recipient to
    determine the transfer-length of the message (Section 4.4).

Should we add there or in Section 4.4 something like:
<<
   If the transfer-length is defined by closing the connection, the
   integrity of the message is not guaranteed, unless it is a
   characteristic of the transfer-coding used. 'Identity' does not have
   this characteristic.
?

         ~~Yves

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Jun 2008 10:01:01 +0100

>   characteristic of the transfer-coding used. 'Identity' does not have

<<
   Because closing is used to abort in some error scenarios, if the
   transfer-length is defined by closing the connection, the recipient
   cannot be sure the whole message is received, unless the
   transfer-coding unambiguously detects truncation.  The
   transfer-codings 'identity', 'deflate' and 'gzip' do not have this
   characteristic, although in most cases 'deflate' and 'gzip' will
   detect truncation.  Content-encoding and type-specific syntax of
   the received entity may also detect truncation, but truncation of
   entity syntax does not always indicate a transport error.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 03 Jun 2008 13:41:19 +0200

>    characteristic of the transfer-coding used. 'Identity' does not have

I don't see why as this isn't a guaranteed even if chunking is used in
the hop to the receiver.

Chain may be

server [Connection: close, no TE] -> proxy [Transfer-Encoding: chunked]
-> recipient

In which case receiving a properly terminated chunked encoding says
nothing about if the message was truncated in transit or not.

Denying upgrading from Connection: close to Transfer-Encoding: chunked
would work around this, but at a significant performance penalty and
also uncertain as such upgrading is actively done by deployed RFC2616
compliant implementations.

Integrity is not a property of the transport. The transport is
hop-by-hop and must only guarantee internal consistency, allowing
unambigious parsing of the data stream. Integrity checks should be done
at the message level.

But I am in favor for stating that if the transfer encoding could not be
properly decoded (decompression error in gzip/deflate, missing end chunk
in chunked etc..) or if network errors is detected while receiving the
message (i.e. read error or similar) the message SHOULD be considered
invalid and the connection closed. Implementations MUST NOT try to
continue using a connection after a transport error including
transfer-encoding related issues.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Jun 2008 13:09:00 +0100

> unambigious parsing of the data stream. Integrity checks should be done

True, but there is no practical mechanism in HTTP at the moment to
assure even the most basic message integrity: that you received all of it.

Content-MD5 is not useful for dynamically produced entities.
As a trailer it might be possible, but how compatible is that?

Content-Length, even, cannot be used.

Not even for static content.  When the server does provide
Content-Length, in your own example the proxy->client hop removes
Content-Length.

I'm thinking that the solution to these is allowing Content-Length in
a chunked trailer, and Content-MD5 too.

-- Jamie

---

From: Yves Lafon <[email-masked]>
Date: Tue, 3 Jun 2008 12:34:27 -0400 (EDT)

> nothing about if the message was truncated in transit or not.

Well, in that case, it would be better for the proxy to figure out there 
is a potential error from server to proxy and to something clever (cutting 
the connection without sending the last chunk ?) to signal this.

> unambigious parsing of the data stream. Integrity checks should be done

Integrity is also a hop-by-hop property. For end-to-end, Content-MD5 and 
others are here to help.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jun 2008 19:23:30 +0200

> As a trailer it might be possible, but how compatible is that?

Well, unless I'm missing something, it will be hard to send from a 
servlet (hey, Servlet EG, are you listening...?).

> a chunked trailer, and Content-MD5 too.

Well, what would they contain in case of a truncated response? Surely 
not the length/digest of the actual response, because that wouldn't help 
the client finding out about the truncation...

Maybe something like "final-status" as a new response header would make 
sense. That way, a server could send an initial 2xx, start sending 
content, and in case of internal errors could at least signal that 
something went fatally wrong...

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Jun 2008 18:34:05 +0100

> servlet (hey, Servlet EG, are you listening...?).

Implementation detail.  If you want this kind of message integrity,
fix the implementation :-)

> the client finding out about the truncation...

Truncated responses don't have trailers.  Trailers are at the _end_!

> something went fatally wrong...

Not a bad idea.  A final-status ought to be able to include an error
entity too.  Not sure how to fit that into the present syntax.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Jun 2008 18:35:34 +0100

> the connection without sending the last chunk ?) to signal this.

How can the proxy detect this potential error?

The proxy will not know anything about the entity being transferred.
Proxies are pure transport.

To the proxy, a truncated (Connection: close) message is
indistinguishable from one which isn't.  That's the problem.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jun 2008 19:40:16 +0200

Jamie Lokier wrote:
> ...

Depends on the definition of "truncated".

A server may not be able to complete sending the requested entity (file 
io error, for instance), but still be able to send trailers...

> ...

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 03 Jun 2008 20:07:10 +0200

> servlet (hey, Servlet EG, are you listening...?).

Content-MD5 either needs the server to buffer the response, or sent in
the trailer. Same for Content-Length except that it can't be sent in a
trailer.

Claiming that those isn't usable for a servlet is plain wrong. It's all
an implementation detail, and how feasible it is depends primarily on
the size of the response.

> the client finding out about the truncation...

Indeed.

> something went fatally wrong...

Problem there is that recipients are not required to care about trailers
and those who don't will misread the response as 100% successful.. so
you are better off simply closing the connection in the middle of the
response and log the error locally. The next-hop will notice the error,
but there is no guarantee the final recipient will..

The scope of this WG is to clarify HTTP/1.1 and correct errors, not
HTTP/1.2 (or 2.0) fixing the shortcomings of HTTP/1.1.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Jun 2008 20:00:20 +0100

> io error, for instance), but still be able to send trailers...

Ok, but then that's outside the bounds of HTTP message integrity.

If a server cannot complete a response for any internal reason, but
wants to send all the indications of a complete and finished responses
(whatever they are), there's any number of ways it can do that.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jun 2008 21:13:25 +0200

> the size of the response.

Of course the servlet container could supply those. What I was thinking 
of was a way for a servlet *instance* to return them.

BR, Julian

---

From: Robert Siemer <[email-masked]>
Date: Tue, 3 Jun 2008 23:29:14 +0200

> HTTP/1.2 (or 2.0) fixing the shortcomings of HTTP/1.1.

Exactly. - The parties who care about message truncation should not use 
a connection closure as message end indication. Likewise 
should intermediaries not change the message end indication method if 
they don't have the full message yet.

But one question stays: should a client/proxy retry if it detects a 
truncated message? - As I read RFC2616: yes (especially if the method is 
safe).

Firefox 2.x did not do that. - I didn't even warn the user or something 
if it had a clear indication that something is missing (response with 
Content-Length: and premature connection close).

What about unsafe methods?

Robert

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 04 Jun 2008 19:31:23 +0200

> Content-Length, in your own example the proxy->client hop removes

I am not aware of anyone besides the non-transparent content encoding
proxies which removes Content-Length.

My example meant to illustrate a response without Content-Length.

> a chunked trailer, and Content-MD5 too.

I don't see what having Content-Length in the trailer buys as it will be
the same as the dechunked length, and if you don't get all of it then
you don't get the trailer. Content-MD5 is fine, and allowed today
without any changes.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 04 Jun 2008 19:41:46 +0200

> of was a way for a servlet *instance* to return them.

To HTTP it doesn't matter who. They are both part of the server, and who
adds Content-Length to the response is an implementation detail.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 04 Jun 2008 19:55:21 +0200

> they don't have the full message yet.

Fully agreed.

The thing I oppose is texts indicating that chunked encoding provides a
guaranteed integrity. It does not. If you want something which provides
a reasonable guarantee then Content-Length must be used.

> truncated message? - As I read RFC2616: yes (especially if the method is 

Perfectly fine for user agents, but not so sure about proxies. But the
partial response MAY be cached and completed using range requests.

> Content-Length: and premature connection close).

Detected truncated messages really SHOULD be considered invalid. This
includes both chunked encoding closed without seeing the end chunk, or
connection closed without seeing all of the response as indicated by
Content-Length. Both cases indicates without any doubt that something
went very wrong with the message.

It's clearly wrong to process them as if they were completely valid
messages.

Regards
Henrik

---

From: Yves Lafon <[email-masked]>
Date: Thu, 5 Jun 2008 05:01:48 -0400 (EDT)

> partial response MAY be cached and completed using range requests.

Ok, so if the proxy gets a reply using connection closure to signal the 
end of the reply, should the proxy always treat this as an incomplete 
transaction ?

         ~~Yves

---

From: Robert Siemer <[email-masked]>
Date: Thu, 5 Jun 2008 12:17:34 +0200

> transaction?

Of course not (as the proxy can not detect truncated messages this way).

Robert

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 05 Jun 2008 22:34:56 +1200

connection closure only signals end of message if the message had no 
Content-Length header, and was not chunked.

Otherwise, if there is a Content-Length header, (and not chunked) then 
the message is complete when the number of bytes indicated have been 
received.  The connection can then be kept open (classic Connection: 
keep-alive without chunking).

Otherwise with chunking a message is only complete if you receive a 
terminating chunk (specified chunk length 0).

It's commonly not practical for a proxy to retry on truncation.  E.g if 
it has been spooling data through in the meantime, and gets a premature 
closure, with a retry it can't guarantee the same entity.  Plus the 
information that the connection was aborted is information the client 
may want.

Therefore it makes sense for the proxy to simply abortively close the 
connection to the client and let the client deal with it, that then 
conveys the (useful) information to the client that the transfer was 
aborted.

So IMO the proxy should not retry, as this can cause confusion for 
clients - and clients are generally equipped to retry anyway, but with 
some visual indication to a user

obviously this only goes for the small proportion of clients that are 
browsers with humans operating them.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Yves Lafon <[email-masked]>
Date: Thu, 5 Jun 2008 09:55:04 -0400 (EDT)

On Thu, 5 Jun 2008, Adrien de Croy wrote:

> Otherwise, if there is a Content-Length header, (and not chunked) then the 
> message is complete when the number of bytes indicated have been received. 
> The connection can then be kept open (classic Connection: keep-alive without 
> chunking).

[..]

> connection to the client and let the client deal with it, that then conveys 
> the (useful) information to the client that the transfer was aborted.

The issue seems to be message integrity (which can be somehow assessed by 
the receiving end) and entity integrity (only applications can try to do 
that).

In Henrik's example, the proxy wasn't able to assess that the received 
message was complete or not (at the HTTP level), and it won't try to 
inspect it. On the other side, the client talking to the proxy will 
receive a complete message, however the entity carried maynot be complete.

The proxy may add a Warning header to indicate this, it would be a better 
match than closing the connection in the middle of a chunk.

One question is... if a Transfer encoding of gzip is used + connection 
closure form the server to the proxy, and the client sent a TE: gzip to 
the proxy, should the proxy gunzip the content, then gzip it back (as 
TE/TEncoding is hop-by-hop) ? (and it that case an error may be noticed) ?

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 05 Jun 2008 15:55:34 +0200

> end of the reply, should the proxy always treat this as an incomplete 

No, only if the length disagrees with Content-Length or if chunked
encoding was not properly terminated.

A message without content-lenght or chunked encoding is validly
terminated by closing the connection.

Servers who require the client to ALWAYS be able to detect truncated
messages SHOULD use Content-Length. It's the only transmission method
where the message format guarantees possible end-to-end detection of
truncation (ignoring setups with non-transparent proxies perforing
content encoding in the mix..). Yes, this rules out using
Transfer-Encoding. Additionally it's wise to use Content-MD5 for real
integrity checks.

Servers only needing this at best effort level is free to use
transfer-encoding, any of the defined ones or combinations, but be aware
that the transfer encoding MAY be removed before the response reaches
the recipient. They SHOULD NOT delimit messages by closing the
connection.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 05 Jun 2008 16:02:21 +0200

> match than closing the connection in the middle of a chunk.

Most times it's too late to add a header as the header portion has
already been sent to the requesting client, and you can't send a trailer
header without  both the following

  - Indication of positive response without error
  - The headers you place in the trailer needs to be advertised in the
initial header portion.

> TE/TEncoding is hop-by-hop) ? (and it that case an error may be noticed) ?

Doesn't really matter for the initial request, but it should at least
verify the Transfer-Encoding if storing the object in cache and discard
the object if the Transfer-Encoding isn't decodeable.

Regards
Henrik