Subject: Lingering Close

From: Zhong Yu <[email-masked]>
Date: Wed, 28 Nov 2012 11:08:42 -0600

http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-6.2.5
6.2.5. Tear-down

   A server that receives a close connection option *MUST initiate a
   lingering close* of the connection after it sends the final response
   to the request that contained close.

   A server that sends a close connection option *MUST initiate a
   lingering close* of the connection after it sends the response
   containing close.

The 2 MUSTs (of lingering close) are too strong here; a server may
reason that a simple close is enough since there shouldn't be any more
data from client, due to
  1. the request is HTTP/1.0 without "Connection:keep-alive".
  2. the request contains "Connection:close"
We can weaken the texts to "MUST close the connection", and discuss
later when a "lingering close" is needed.

The description of "lingering close" also seems a little off. We
shouldn't ask servers to "half-close" first, because

  1. Half-close is useless since an HTTP/1.1 client usually won't
detect it. The client reads to the end of the response and stops, it
won't read again to find out the EOF. (For HTTP/1.0, EOF is required,
but it's not part of the discussion of lingering close)

  2. Some APIs do not support half-close. Examples in Java: SSL
socket, Netty channel.

So we can remove the "half-close" step from "lingering close" process;
the process is simply: drain inbound data, then close the connection.

This "lingering close" process should work for abstract transport
protocols, as long as the server, while reading from client, can
detect that the client has closed the connection.

Zhong Yu

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 18:44:49 +0100

> but it's not part of the discussion of lingering close)

No, there are several issues with this, among which :
  - bogus clients which still send a late CRLF after a POST will trigger
    an immediate reset from the server's TCP stack before the end of the
    transfer if the connection is closed without lingering. This results
    in the common problem of truncated responses ;

  - "lingering" here clearly means the work performed by the TCP stack
    which is performed by default. If we remove this and talk about
    just a close, some people will believe that they can explicitly
    disable lingering, which is totally wrong as this also kills all
    pending data and truncates the response as well.

The main problem is that HTTP is not well suited for use with TCP (!).
But we have it now and we must do whatever we can to maximize chances of
successful transfers. The two problems of TCP are :
  - application is not aware that the remote end has received everything.
    An application-level ACK from the client would have solved this ;

  - client must never be encouraged to close first otherwise it quickly
    faces the 2 MSL problem with sockets in TIME-WAIT and cannot open new
    ports for several minutes whatever the setsockopt() it did. Still,
    sometimes we don't have much choice.

>   2. Some APIs do not support half-close. Examples in Java: SSL

Then these ones most likely perform the half-close transparently for you,
and certainly don't disable lingering!

I think the wording is appropriate here. If it's ambiguous, maybe it could
be improved, but I didn't find anything useless there at least.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 28 Nov 2012 17:59:14 +0000

--------

>The main problem is that HTTP is not well suited for use with TCP (!).

Indeed, this is a major problem in so many ways it's not even funny.

However, considering the penetration of HTTP, it's not inconceivable
that we could get a couple of much needed extensions to the socket
API to stick, possibly as a "TCP considerations for HTTP/2.0"
informal RFC.

If there is interest in this, I can make FreeBSD one of the reference
implementations.

One extension I have been pondering for a long time, is a true
per-socket idle timeout (Ie: no data-carrying packets in either
direction and no outstanding data to transmit for T time)

Poul-Henning

PS: And just in case some of you missed it, Queue had a couple of very
interesting spotlights on the dark buffer bloat issue some time
ago:

Article:
	http://queue.acm.org/detail.cfm?id=2071893

Interview:
	http://queue.acm.org/detail.cfm?id=2076798

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 19:13:42 +0100

> API to stick, possibly as a "TCP considerations for HTTP/2.0"

I agree. We've also discussed about this with Roberto Peon at IETF83 in
Paris and it seems we are several HTTP implementors to have ideas about
missing things (such as polling for all sent data to be acked, "true"
lingering close, etc...). Being able to pass a FIN with a full window
along the path would be nice too, as well as detecting it without polling
for reads.  Having recv() report the received shutdown without performing
a second attemtp would also help for the vast majority of situations where
the response comes with a close, etc... Well, that's a complete discussion
out of scope for this WG.

> If there is interest in this, I can make FreeBSD one of the reference

That could be an idea, yes.

> direction and no outstanding data to transmit for T time)

Indeed, that's one thing that could help.

> 	http://queue.acm.org/detail.cfm?id=2076798

I'll check this, thanks !

Willy

---

From: Roberto Peon <[email-masked]>
Date: Wed, 28 Nov 2012 10:14:16 -0800

I understand the specification of lingering close, however, specifying that
the HTTP implementation should make an effort to ensure that the data is
received before closing (and that lingering close is one such mechanism) is
potentially better.
Lingering close is nasty for a number of reasons.The best option here (at
least for servers) would be to have some knowledge about when the last ack
is ack'd so that it can close the connection from userspace, safe in the
knowledge that everything has arrived at the client safely. That is
obviously not an issue for this WG< however.
-=R

---

From: Zhong Yu <[email-masked]>
Date: Wed, 28 Nov 2012 12:20:06 -0600

Willy, I thought the term "lingering closed" in the section is self-defined

    a server can perform a lingering close on a connection by [steps...]

the word linger here has nothing to do with its usage elsewhere, like
SO_LINGER option.

I might be wrong if the word "linger" carries a meaning not defined
here but well known to the readers. Please clarify. At least the word
is not in the TCP spec.

Regardless, I certainly do not suggest to abortively close a
connection without seeing outbound data ACK-ed. My point is, the
half-close step is not necessary.

More inline comments below...

>     in the common problem of truncated responses ;

It's solved by the draining step, not by half-close

>     pending data and truncates the response as well.

I do not suggest to disable "lingering" in that sense.

> and certainly don't disable lingering!

It's impossible in these APIs to half-close first, then read;
therefore the current description of "lingering close" cannot be
implemented in these APIs.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 19:22:39 +0100

> received before closing (and that lingering close is one such mechanism) is

I agree but there is no portable way of doing so at the moment unfortunately,
so basically what the spec requires implies suboptimal processing on all
standard-compliant HTTP stacks (which basically means draining data from the
client until it closes).

> obviously not an issue for this WG< however.

Exactly, I would love to have this ! We could even imagine an option so
that the FIN could be sent by the stack while the app layer is notified.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 20:08:05 +0100

> the word linger here has nothing to do with its usage elsewhere, like

OK.

> here but well known to the readers. Please clarify. At least the word

No, you're right.

> half-close step is not necessary.

I disagree, precisely because if you only close, you risk facing
the RST.

> It's solved by the draining step, not by half-close

No, because if you don't send the half-close, you can't expect the client
to close first as it might very well be only relying on the close to get
the end of the message (old client). Also, even if it's a new client, you
absolutely don't want it to close first as you might cause it to run out
of source ports very quickly (think about a proxy).

> I do not suggest to disable "lingering" in that sense.

OK.

> implemented in these APIs.

But I do think that if you simply close, the underlying layers will do
the lingering for you. However this still means that these APIs are not
compatible with HTTP/1.0 clients, which is somewhat problematic :-/

Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 28 Nov 2012 13:24:15 -0600

Thanks Willy, I think I get what you mean by now. FIN should be
initiated by server to avoid the TIME_WAIT problem. Therefore the
half-close step is important.

The current text makes perfect sense to me now.

Unfortunately, this lingering close process cannot be implemented on
top of some APIs; these APIs don't do transparent lingering close upon
close() either. But it is their fault, not the spec's.

Zhong Yu

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 28 Nov 2012 19:36:11 +0000

> The current text makes perfect sense to me now.

That's great.  I knew about the RST problem (and waiting for data-ACK
isn't enough to solve that), but I didn't think about the client-side
TIME_WAIT problem before now.  Thanks Willy!

> close() either. But it is their fault, not the spec's.

Unfortunately because the old HTTP specs didn't mention this, it seems
to be widely believed that ordinary close is fine for HTTP, and APIs
were created with HTTP in mind to run over them with this limitation.
It's unfortunate that lingering close wasn't made clear in the earlier
specs.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 28 Nov 2012 19:39:47 +0000

> that the FIN could be sent by the stack while the app layer is notified.

I'm not sure it would work.  When the last ACK is sent from client to
server, the client kernel has the data but the client application
generally doesn't.  If there's post-request data still in flight from
the client, or the client app will send a few bytes more, RST can
still happen, and the client kernel may abort the socket before the
client application has read from its kernel.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 21:22:40 +0100

> half-close step is important.

exactly.

> The current text makes perfect sense to me now.

With this in mind, do you think that something in the text should be
updated for future readers ?

> close() either. But it is their fault, not the spec's.

Indeed. However the fallback to full close with the kernel's lingering is
still working in practice. Just not perfectly reliable.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Nov 2012 21:24:30 +0100

> client application has read from its kernel.

Indeed, that's why I proposed a linux patch years ago to avoid sending
this RST until client data were acked, but it was rejected, not being
standards compliant. Anyway, I do think that we'll need to update some
TCP stacks to use HTTP better in the future.

Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 28 Nov 2012 15:13:03 -0600

> updated for future readers ?

The text gives motivation for draining (to avoid RST), it probably
should give motivation for half-close as well. The text may become too
long and out of scope, but I agree with Jamie Lokier that it's better
to warn implementers about the issues.

Also, I would probably use a different word than "linger"; at first
read I though it means kernel's lingering.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 29 Nov 2012 01:29:43 +0100

> to warn implementers about the issues.

I too was caught in the past by this and have had to perform several
incremental changes on haproxy to address this, including for responses
caught from some servers.

> read I though it means kernel's lingering.

It was my case too when reading this sentence out of context.

Maybe a small change like this would be appropriate ?

-   To avoid the TCP reset problem, a server can perform a lingering
-   close on a connection by closing only the write side of the read/
-   write connection (a half-close) and continuing to read from the
-   connection until the connection is closed by the client or the server

+   To avoid the TCP reset problem, a server can shut down the write side
+   of the connection (also called a half-close) and continuing to read from
+   the connection until the connection is closed by the client or the server

Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 28 Nov 2012 20:11:12 -0600

> +   the connection until the connection is closed by the client or the server

"lingering close" is also mentioned in previous texts. give it a new
distinct name?