Subject: HTTP and half close (WAS: HTTP client abort detection)

From: Miles Sabin <[email-masked]>
Date: Wed, 14 Mar 2001 14:35:00 -0000

Scott Lawrence wrote,
> Miles Sabin wrote:
> > I've been trying (unsuccessfully) to decide whether or not
> > RFC 2616 permits a server to interpret the receipt of a FIN 
> > on the incoming-request side of it's connection to a client 
> > as a full client close ... 
>
> I think that it would have been poor form at best for the HTTP 
> WG to ignore the Tcp semantics in that way.  In fact, the 
> client is doing you a favor by sending the FIN as soon as it 
> knows it's done with that half-connection - the server can then 
> close its half right away even if it might otherwise have left 
> it open for persistence, perhaps suffering the time-wait 
> penalty when it times it out (if the client sends the first 
> FIN, then it has to hold the time-wait, and the server 
> doesn't).

Thinking about this some more, I'm coming to the conclusion that
there's a genuine and problematic gap in RFC 2616. Just to make 
sure there's no misunderstanding here, I want to emphasize that 
I'm quite well aware of the benefits for servers of clients 
sending the first FIN.

As far as I can make out there's nothing in RFC 2616 which 
unambiguously rules out any of the following possible client and 
server implementations,

1. Clients which send a FIN early ... ie. by doing a half close
   as soon as their last request is sent, but possibly before
   the corresponding response has been received.

2. Clients which don't (half or full) close until all pending
   responses have been received.

A. Servers which treat an early client FIN only as a half close 
   and continue processing and sending pending responses.

B. Servers which treat an early client FIN as an indicator of a 
   client abort, and hence abandon processing and sending pending
   responses.

The problem is that clients of type 1 aren't fully interoperable
with servers of type B ... if a type 1 client does an eager
half close a type B server would abandon it's response.

So at least one of these two should be ruled out by RFC 2616. 

Unfortunately I don't see that either unambiguously is. And if
neither are ruled out by the spec, then interoperability
considerations mean that it's not safe for client or server
implementors to adopt either. In real-world terms, I'm pretty
sure that most existing user-agents (nb. I'm not talking about 
clients generally) are of type 2, and most servers are of type
A.

Here are some pro and con considerations wrt type 1 clients and
type B servers.

* The pragmatic good citizenship argument that a client should 
  send the first FIN, and sending it immediately after its 
  request has been sent, before any or all of the response has 
  been received, makes that likely.

  On the other hand, this isn't the only way for a client to send
  the first FIN. In scenarios which allow a client to half close
  eagerly it should also be possible for it to send Connection: 
  close, in which case the server would know that the connection 
  is terminating. Hence the server, assuming it sends a Content-
  Length or chunks, could quite happily defer closing after 
  sending the response, on the assumption that the client will 
  close soon and that it can close it's own end when that
  happens.

  On balance, I don't think any conclusion can be drawn from the
  above. Early half close helps servers, but there are other
  ways of getting the same effect.

* A similar pragmatic argument that if servers are allowed to
  treat early client FINs as client aborts, then they might be
  able to avoid expensive response processing.

  In the particular case of a proxy server, such a FIN might be 
  detected during proxy-side DNS resolution, but before the 
  initiation of a connect to an origin server. If the proxy 
  isn't going to cache the now unwanted response (either because 
  it's not a caching proxy, or because it's confident the 
  response will be uncacheable) then it would make sense not to 
  attempt to forward the request to the origin server at all.

* From 8.1.4 Practical Considerations,

    Servers SHOULD always respond to at least one request per 
    connection, if at all possible. Servers SHOULD NOT close a 
    connection in the middle of transmitting a response, unless a 
    network or client failure is suspected.

  So long as we read 'middle' of a response liberally and allow 
  it to cover any point in time between the servers receipt of
  the request up to the completion of the sending of the 
  response, then this seems to support type 1 clients against 
  type B servers.

  Nevertheless, it's very hard to see the practical difference
  between a 'network or client failure' and, say, the behaviour
  of a user agent when a user hits the stop button because the
  server hasn't managed to deliver a response sufficiently
  quickly.

* If we aren't so liberal with the reading of 'middle' in the
  above quoted para, then we have,

    A client, server, or proxy MAY close the transport connection 
    at any time. For example, a client might have started to send 
    a new request at the same time that the server has decided to 
    close the "idle" connection. From the server's point of view, 
    the connection is being closed while it was idle, but from 
    the client's point of view, a request is in progress.

  This suggests that a server which detects an early client FIN 
  would be within it's rights in exploiting the persistent 
  connection close race condition to avoid processing or sending 
  any response at all: it's free to close the connection at any 
  time, and it's not yet started to send it's response. The only 
  difference between this scenario and the typical cases is the 
  exact location of the request message ... on the wire, in the 
  TCP receive buffers, or in a server buffer.

* If type 1 clients are legitimate, this parenthetical comment in 
  4.4 Message Length becomes quite baffling,

    the transfer-length of that body is determined by one of the 
    following (in order of precedence):

    5. By the server closing the connection. (Closing the 
       connection cannot be used to indicate the end of a request 
       body, since that would leave no possibility for the server 
       to send back a response.)

  because if an early half-close were legitimate, it would
  provide a perfectly respectable mechanism for delimiting a
  request body.

None of the above leaves me with any particularly clear idea of
what best practice might be. Unless I can be persuaded otherwise
I'm obliged to be cautious from an interoperability perspective
and assume,

* That a type 1 client implementation is inadvisable, because
  it wouldn't reliably interoperate with type B servers.

* That a type B server implementation is inadvisable, because
  it wouldn't reliably interoperate with type 1 clients.

Can anyone shed any more light on this?

Cheers,

Miles

-- 
Miles Sabin                               InterX
Internet Systems Architect                5/6 Glenthorne Mews
+44 (0)20 8817 4030                       London, W6 0LJ, England
[email-masked]                         http://www.interx.com/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 16 Mar 2001 17:33:55 -0800

"Server receives a FIN" and "HTTP connection is closed" are two
different things.  The first is one part of a TCP handshake for which
HTTP has no business defining the rules (HTTP is transport independent),
and the second is something that an application has to find out from
whatever network API it is using for sending/receiving HTTP bytes.

Using the typical socket API with TCP, an HTTP connection is considered
to be closed when it receives a fatal error or EOF either:

   1) when attempting to read an unfinished request and there are
      no pending responses being sent on the connection (this is easily
      determined because all HTTP requests are length-delimited), or

   2) when writing a response.

Note that the above applies to the socket API.  There may be a completely
different algorithm for determining when the connection is closed when
using other API, particularly event-based ones.

The reason this isn't defined in the HTTP spec is because it is not
an interoperability issue for the application protocol -- it is an
implementation detail that is entirely dependent on the nature of the
lower-layer API used by the application.  It is the stuff for a good
book on network programming.

BTW, the reason that clients don't close the connection is because
they are generally written to be selfish in regard to network resources.
Some clients (Windows-based) never send a FIN -- they always do an RST
because they don't want to incur the cost of TIME_WAIT, due to an
unusually low fixed limit on mbufs for some version of the OS.  This is
broken behavior, since the RST is easily lost and the server is left
hanging til a timeout expires, but that is fairly typical of clients.

Cheers,

Roy T. Fielding, Chief Scientist, eBuilt, Inc.
                 2652 McGaw Avenue
                 Irvine, CA 92614-5840  fax:+1.949.609.0001
                 ([email-masked])  <http://www.eBuilt.com>