Subject: NEW ISSUE: Monitoring Connections text

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Nov 2007 16:50:55 +0100

Hi,

  Section "8.2.2 Monitoring Connections for Error Status Messages" has:

   An HTTP/1.1 (or later) client sending a message-body SHOULD monitor
   the network connection for an error status while it is transmitting
   the request. If the client sees an error status, it SHOULD
   immediately cease transmitting the body. If the body is being sent
   using a "chunked" encoding (section 3.6), a zero length chunk and
   empty trailer MAY be used to prematurely mark the end of the message.
   If the body was preceded by a Content-Length header, the client MUST
   close the connection.

This is somewhat confusing, it uses RFC 2119 keywords even though it
really just makes statements of fact (if you cease transmitting, you
cannot meaningfully maintain the connection) and it's unclear whether
clients may still send a non-empty trailer. Better would be e.g.

   A client sending a message-body SHOULD monitor the network connection
   for an error status while it is transmitting the request. If the
   client sees an error status, it SHOULD cease transmitting the body.
   Unless the body is being sent using the "chunked" encoding the client
   MUST close the connection.

regards,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 21 Nov 2007 17:28:00 +0000

>    MUST close the connection.

Both versions are confusing, and the second one is inconsistent with
itself.

First it says the client "SHOULD cease transmitting the body",
i.e. not MUST, but then it says the client MUST close the connection
unless it's chunked, unconditionally. This is not true: the client can
keep the connection open, if there's a good reason not to follow the
SHOULD, and send more messages.

It should say: _If_ the client ceases transmitting the body, then ...

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 21 Nov 2007 21:38:58 +0100

> clients may still send a non-empty trailer. Better would be e.g.

Personally I find the text quite clear. The way I read it it's a chain
of conditions.

1. client SHOULD monitor
2. If monitoring and seeing an error it SHOULD ..(if not monitoring it
won't see the error, so the rest do not apply then) 
3. If chunked then it MAY terminate the body using chunked encoding
4. Else (Content-Length) it MUST close the connection.

The only change I'd like to do there is in the last part, replacing

  of the message. If the body was.. [to the end] with simply

  else the client MUST close the connection.

to close the blank spot where the client is using chunked encoding but
do not want to use chunked encoding to prematurely mark the end of the
client message.

Extending the text to bind the first two SHOULD together in a chain
isn't needed imho. Should be rather obvious that you can't see things if
you don't monitor. 

>    MUST close the connection.

This is imho less clear than the original. Especially regarding how to
prematurely end a chunked message which in my experience isn't very
obvious to most people that you can.. (anyone remember the NTLM auth
discussion here some months ago?)

Regards
Henrik

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Nov 2007 21:51:10 +0100

>4. Else (Content-Length) it MUST close the connection.

My main concern is getting rid of the MAY as there is no new option
being offered by the protocol, and getting rid of the "empty trailer"
part. I take it Jamie also find the original text unclear in that it
has both "SHOULD" and "MAY" for the termination if chunked is used,
and "SHOULD" and "MUST" if Content-Length is used, though I would not
read the original this way.

>discussion here some months ago?)

If you want to say that, it would be better to say "by sending the
last-chunk at the next opportunity". There is no such thing as a
"zero length chunk". I'm fine with any text that does not have the
odd "MAY" and "empty trailer".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Nov 2007 21:53:04 +0100

>odd "MAY" and "empty trailer".

(Well there is, strictly speaking, but it's an odd notion of it.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 21 Nov 2007 22:04:53 +0100

On ons, 2007-11-21 at 21:51 +0100, Bjoern Hoehrmann wrote:

> part.

Terminating the message using chunked encoding is optional. The client
is always free to close the connection if it prefer to abandon the
connection and open a new one.

The point of terminating the message using chunked encoding is that it
allows the connection to be kept for the next message, and is a MAY as
it's entirely optional and up to the client to decide to do this instead
of closing the connection. But it MUST do one of the two if terminating
the message.

> odd "MAY" and "empty trailer".

Right.. should be last-chunk without a trailer.

Regards
Henrik

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Nov 2007 22:14:25 +0100

>of closing the connection. But it MUST do one of the two if terminating

I think we are all agreed on what should happen, we just don't have good
enough text for it yet. Could you make a proposal that would address my,
Jamie's, and your concerns?

>Right.. should be last-chunk without a trailer.

The point is that I don't want it to be illegal to send the trailer, or
if that needs to be illegal, that that is spellt out explicitly. It's
clear that clients should not generate unnecessary traffic, but there
may well be cases where sending the trailer is useful for diagnostics.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 22 Nov 2007 10:16:25 +1300

what sort of network error is being envisaged here?

One that still permits data to be sent?  I'm not aware of any sorts of 
network errors that you can actually monitor for, or receive 
notification of other than a reset or close.  Other types of errors 
typically don't require any action at all - i.e. don't pertain to that 
connection, nor affect the ability to send data on the connection.

In my experience, the only option upon discovery of a network error is 
to close the connection, no chance to send a final chunk or anything.

Otherwise it looks like the spec is suggesting we keep talking down the 
phone line when the other person has hung up... a bit pointless really.

Also, IMO it's a really BAD idea to send a final chunk if you didn't 
send the whole thing, because then if the other end receives it, it 
can't tell that the transfer was aborted.

So, IMO the only sensible option is to terminate the connection.

just my 2c

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 22 Nov 2007 10:40:12 +1300

ah, thanks.

OK, this is an issue close to my heart, since it affects POST / PUT 
through a proxy which uses any sort of challenge/response (i.e NTLM) 
authentication mechanism.

Our users have major problems with this.

It would be great if the concept of chunking could be stretched to 
include notification of an abort of transfer.

i.e. send a chunk with a length of "-1" or something.  Obviously that's 
a protocol level change, but it would allow the connection to be 
maintained after an abort being signaled.

I haven't seen a client that uses chunking for POST or PUT though - I 
guess in most if not all cases, the client knows the content length, and 
so specifies it.  Should we be putting a stronger emphasis on clients to 
use chunking in all cases?  That has down-sides as well, since then the 
client can't advertise apriori the length it will be sending (to allow 
for proxy / server policy evaluation based on upload sizes etc).

This actually could be a good use-case to use chunking AND specify a 
content-length header (I hear the groans) - something specifically 
deprecated in the spec.

People commonly want to limit download and upload sizes.  The only way 
to limit upload sizes efficiently is if clients sent Content-Length 
headers.  Otherwise you have to receive / process data until you hit the 
limit.

But actually I believe in the end, the issue can't be properly addressed 
without a major protocol version change, one that then incorporates the 
concept of negotiating (i.e. obtaining mutual consent for) transfer of 
message bodies from client to server (as opposed to only having 
negotiation in the other direction).

Regards

Adrien

Bjoern Hoehrmann wrote:
> * Adrien de Croy wrote:
> This is about the server telling you it won't handle the request while
> it is being sent (e.g., you are trying to PUT something to a resource
> that requires authentication, or after the 42nd chunk the server decides
> the entity is longer than it is willing to handle). It's not about any
> lower level network problems.
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Nov 2007 22:21:53 +0100

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 21 Nov 2007 23:27:11 +0100

> what sort of network error is being envisaged here?

none. an error status within http. I.e 404/401/407 etc.. we are talking
about "8.2.2 Monitoring Connections for Error Status Messages".

> to close the connection, no chance to send a final chunk or anything.

yes.

> can't tell that the transfer was aborted.

In this case the other end has already said the are not intresten in
what we are sending.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 22 Nov 2007 11:35:00 +1300

Henrik Nordstrom wrote:
> ...

actually depending on the structure of the proxy or server, this isn't 
always the case.

If you have a chain of filters installed in a proxy or a server, the 
data may have already passed through many filtering processes before it 
hits the one that bounces the request.

Signalling a final chunk, if that goes through the filter chain as well, 
will be telling all these filters that the resource they got was 
complete, and they may do all manner of processing on it.

I believe we need a way to clearly distinguish between a completion and 
an abort.

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 21 Nov 2007 23:59:11 +0100

> authentication mechanism.

Yes, and have already been discussed already here..

> Our users have major problems with this.

Yes, and they are not alone, and if NTLM-clients would utilize this they
would fare much better, but no one does even if specified..

> include notification of an abort of transfer.

Sending an last-chunk prematurely is an abort. But I guess we could
extend it with a chunk extension to clarify that it's an abort and not
just "I'm done".

> I haven't seen a client that uses chunking for POST or PUT though

Exactly. And not unsuprising. Chunked encoding of requests is a pretty
unchartered area, and quite heavily discouraged by the specs for legacy
reasons..

but it would benefit many things, not only NTLM. Especially when taking
100 Continue into account.

> content-length header (I hear the groans) - something specifically 

Needs to be a different informative header in such case. Content-Length
is to entangled into the transport layer of HTTP to be used here.

> negotiation in the other direction).

Sure it can. In fact it's already there in the form of 100 Continue and
chunked encoding. Just that nearly nobody is using it.

Client sends request header, waits for 100 Continue, gets a denial,
sends last-chunk to terminate the request and moves on to the next..

Problems:
1. Client only allowed to use chunked if it knows within reasonable
doubt that the next-hop is HTTP/1.1.

2. Client recommended to time out waiting for 100 Continue, as it's not
supported by HTTP/1.0 and there is no guarantee the path is fully
HTTP/1.1.

So currently it's not often both conditions to use this negotiation is
fulfilled.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 22 Nov 2007 13:53:54 +1300

> would fare much better, but no one does even if specified..
For anyone's interest, it looks like MS deprecated in IE7 the (IMO 
bogus) methodology they had implemented in IE6 of sending a POST with 
Content-Length: 0 in the expectation of an NTLM challenge.  IE7 now 
sends the whole message body in all cases.

Unfortunately IE6 still is the most common browser (hitting our sites 
anyway).

I think chunking uploads plus another optional header (maybe called 
"Chunked-Content-Length" or similar?) would be a good workaround in 
systems where the length is known,  and is needed by the receiver for 
policy reasons yet where chunking is desired for its capability to keep 
a connection alive.  I agree the existing Content-Length header is too 
entangled in the rest of HTTP to use in this case.

Being able to advertise an entitiy content length on chunked transfers 
would be useful in many cases (not just uploads).

> unchartered area, and quite heavily discouraged by the specs for legacy
I think there will be many broken implementations that don't work with 
chunked sends from clients.  Just a gut feel.

> doubt that the next-hop is HTTP/1.1.
This has major issues if the next hop is an HTTP/1.1 proxy.  The proxy 
then needs to either

a) maintain a database of versions of servers on the internet.
This is a heap of work, and can you guarantee that next hop will always 
have same capabilities?  Not with NLB you can't.

b) spool all chunked uploads, then send them to next hop not using 
chunked encoding. 
This has major issues with user responsiveness (i.e. upload to proxy is 
at LAN speed, then long wait with no progress while proxy uploads to 
next hop at internet connection speed).  This is another good use-case 
of my I-D for progress notifications.

various methodologies were proposed, such as getting the client to send 
a fake

> supported by HTTP/1.0 and there is no guarantee the path is fully
This is another big problem with proxies.  By the time the client has 
given up waiting for a 100 continue, the proxy may still be trying to 
connect to the end server. 

I don't think the current implementation of 100 continue with a timeout 
is very good at all, which is why I originally proposed an interim 
response to enable a proxy to explicitly prevent a UA from sending 
message body until a subsequent 100 continue was transmitted (ended up 
being 103 Wait-for-continue).  As was pointed out, this breaks message 
semantics (i.e. message length indicated but body not transferred). 
Being able to advertise lengths for chunked transfers would solve this 
issue if people used chunking.

With NTLM, the only way for recovery from a commenced send that must be 
aborted, is if the client sends chunked data.  So lack of a capability 
to advertise sizes with chunking precludes the possibility of a proxy 
using NTLM and enforcing upload size limits efficiently.  I know how 
some people feel about NTLM, but we're stuck with it.

Currently we are forced to just swallow and dump data in such cases.  It 
has a large negative impact on the user experiences of people uploading 
large files.  We also have a lot of customers that use a proxy across a 
WAN, and this is even worse.  Chained proxies plus servers requiring 
auth makes it pretty much impossible for users.  I've seen cases where 
the message body on POSTs had to be sent 6 times every time. 3 times is 
very common.

so, the "properly addressed" method I was referring to would be where 
the client establishes credentials and permission through to the end 
server before sending the message (PUT/POST) that transfers the body.  
It may have to be a new verb for that, like the concept of the HEAD 
command for uploads.  It would be of most benefit for large uploads, so 
could be an optional thing.  Clients could for instance choose to use it 
only when uploading files over a certain size.

Regards

Adrien

---

From: David Morris <[email-masked]>
Date: Wed, 21 Nov 2007 18:32:10 -0800 (PST)

> would be useful in many cases (not just uploads).

I'm really uncomfortable with the notion of two specifications of the
actual lenth. I'd prefer and advice notion or maximum notion where the
request or response could have a "Maximum-Length" or "Estimated-Length"
header to provide the recipient with resource management information
while not being an explicit conflict with the actual length sent via
chunked encoding.

Also, I thing the notion of a chunk length of -1 deserves further
discusion. Seems to me that if a recipient understands the new value as
abort, then the connection can continue to be used. If the recipient
doesn't understand the signal, it should be treated as a serious error
resulting in an abort of the request and connection. If I've not missed
something, that leaves older recipients protected while allowing a useful
tweak to the protocol.

Dave Morris

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 22 Nov 2007 07:29:34 +0100

> something, that leaves older recipients protected while allowing a useful

Should be a chunk extension, not new length.. i.e. something like

0; aborted

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Wed, 21 Nov 2007 22:33:29 -0800 (PST)

> Should be a chunk extension, not new length.. i.e. something like
Only if current recipients will not ignore the 'aborted' and consider the
entity complete.

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 22 Nov 2007 14:07:54 +0000

> something, that leaves older recipients protected while allowing a useful

I wouldn't be surprised if some implementation out there actually
parses it as a -1 length, leading to goodness knows what mayhem :-)

Any reason why "0;reason=aborted" cannot be used?  It's valid in RFC
2616 chunk-extension syntax, so anything which can accept chunked
requests really ought to accept this.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 23 Nov 2007 08:47:11 +0100

> Only if current recipients will not ignore the 'aborted' and consider the

Why?

What's important is that current recipients correctly understands that
as "end-of-message".

We are talking mainly about aborting the request entity after the server
have already said "I am not interested", not the response entity.

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Fri, 23 Nov 2007 15:06:11 -0800 (PST)

> have already said "I am not interested", not the response entity.

Because I believe it critical to safe operation that an abort be
recognized by the recipient of the 'message'. Passing an abort signal in
a way that it it ignored under extensiblity rules, etc. makes for the
possiblity of misunderstanding.