Subject: Comments on Section 6.1 (Persistent Connections) of HTTPbis Part 1, version 17

From: Jonathan Billington <[email-masked]>
Date: Tue, 20 Dec 2011 12:52:01 +1030

We are currently working on developing a model of HTTP's operation over TCP based on Part 1 of HTTPbis. This has led us to make the following 5 comments on section 6.1 of draft 17. We have also included possible replacement text to help clarify section 6.1 of Part 1, where we thought it may be useful. The final comment may touch on design, and we are not confident about our assumptions, but we set out some issues that we feel need to be resolved, and are willing to prepare replacement text if the issues can be resolved.

Regards

Jonathan Billington
Professor of Computer Systems Engineering
University of South Australia

Sonya Arnold
Research Student
University of South Australia

------------------
Comments on Section 6.1 (Persistent Connections) of HTTPbis Part 1, version 17, 31 October 2011
Input to: HTTPbis Working Group.
20 December 2011
J. Billington and S. Arnold

1.  Section 6.1.1 Purpose
Change Type: Editorial clarification
Rationale: This section includes a list of advantages of using persistent HTTP connections. The third and fourth points state:
"Network congestion is reduced by reducing the number of packets caused by TCP opens, and by allowing TCP sufficient time to determine the congestion state of the network."
"Latency on subsequent requests is reduced since there is no time spent in TCP's connection opening handshake."

These points concentrate on the opening phase of TCP causing delay, because of the 3 way handshake, but equally important is the graceful closing phase, which requires handshakes in both directions (FIN followed by ACK for each side), and the holding of the TIMEWAIT state, causing further delay. Because it is equally important to the closing phase, we feel it is useful to include this point in the list of advantages.

Proposed Replacement text:
Third point:
"Network congestion is reduced by eliminating the packets associated with TCP's three way handshake and graceful close procedures, and by allowing TCP sufficient time to determine the congestion state of the network."
Fourth Point:
"Latency on subsequent requests is reduced since there is no time spent in the double handshakes required for gracefully closing TCP connections and holding the TIMEWAIT state, and in TCP's connection opening handshake, when reopening the connection."

2. Section 6.1.2 Overall Operation, second paragraph

Change Type: Editorial clarification
Rationale: The second paragraph of this section states:
  " Persistent connections provide a mechanism by which a client and a
   server can signal the close of a TCP connection.  This signaling
   takes place using the Connection header field (Section 8.1).  Once a
   close has been signaled, the client MUST NOT send any more requests
  on that connection."

The last sentence contains a mandatory requirement that may not be able to be met by a client using pipelining. If the client signals the close, then it is OK. However, if the server signals the close in a response, further pipelined requests may be sent by the client until it receives the response with the close token. Hence we need to rephrase the requirement so that the client can meet it.

Proposed Replacement text:
Replace the last sentence by:
"Once a close has been signaled by the client in a request, or the close token has been received by the client in a response, the client MUST NOT send any more requests on that connection."

3. Section 6.1.2.1 Negotiation, second paragraph

Change Type: Editorial clarification
Rationale: The second paragraph of this section states:
  " An HTTP/1.1 client MAY expect a connection to remain open, but would
   decide to keep it open based on whether the response from a server
   contains a Connection header field with the connection-token close."

It is not clear what the intent of this sentence is. Does it imply that on the receipt of a response with a close token, the client will close the connection? Does it also imply that if the response did not contain the close token, then the client would keep the connection open? If this is the intent, then a clearer statement of it would help to exclude other interpretations, as the current text seems to allow the client latitude to close or not on either of these events.

Proposed Additional text if the above interpretation is the intent:
Add a sentence to clarify the intent at the end of the above quoted sentence:
"If the response does not contain the close token the client will keep the connection open, otherwise, on receipt of the close token in the response, the client will close the connection."

4. Section 6.1.2.1 Negotiation, third paragraph (second paragraph on page 42)

Change Type: Editorial clarification
Rationale: The paragraph states:
  "If either the client or the server sends the close token in the
   Connection header field, that request becomes the last one for the
   connection."

This is similar to item 2 above. The problem is knowing what "that request" is in the case when the server includes the close token and pipelining occurs, and what is intended by the phrase "becomes the last one for the connection". We expect that the following is the intent:
A. If the client receives a response with the close token, it will not send any more requests (as in item 2 above); and
B. Once the server has sent a response with the close token, it will discard any further requests it receives.

Proposed Replacement text, given that previous items 2 and 3 are accepted
Replace this sentence by:
"Once the server has sent a response with the close token, it will discard any further requests it receives."

5. Section 6.1.5 (page 46)

Change Type: Design/Editorial clarification
Rationale: The section states:
   "Senders can close the transport connection at any time.  Therefore,
   clients, servers, and proxies MUST be able to recover from
   asynchronous close events.  Client software MAY reopen the transport
   connection and retransmit the aborted sequence of requests without
   user interaction so long as the request sequence is idempotent (see
   Section 6.1.2 of [Part2]).  Non-idempotent request methods or
   sequences MUST NOT be automatically retried, although user agents MAY
   offer a human operator the choice of retrying the request(s).
   Confirmation by user-agent software with semantic understanding of
   the application MAY substitute for user confirmation.  The automatic
   retry SHOULD NOT be repeated if the second sequence of requests
   fails."

We believe it is useful to define terms associated with "sequence of requests". We think it is clear that "the second sequence of requests" refers to "the aborted sequence of requests" mentioned earlier in the paragraph. Thus we need to understand and define what an "aborted sequence of requests" is. To make a start on this we consider the following example:

We begin with the premise that all sequences are idempotent. Consider that the client has pipelined 5 requests and has received 2 responses when the transport connection is closed by the server. It seems clear at this stage that the aborted sequence comprises the remaining 3 requests (that have not received responses). Now we think the following occurs, where we have numbered the requests from 1 to 5 in the order they were sent.

The client automatically reopens the transport connection and resends request 3. (It cannot pipeline until it receives the response to request 3 - see paragraph 2 of section 6.1.2.2 Pipelining.) The following scenarios are possible:
a) If the server closes the transport connection before the response is received, then "the second sequence of requests has failed". The HTTP client closes its half of the transport connection, the remaining 3 requests are removed from the HTTP client, and the HTTP client user is informed of the failure. The connection could be reopened by the client user and the requests resent. In this case the HTTP client would consider the sending of these requests to be a new sequence, and hence retry them if the server prematurely closed.
b) If the response is received without the close token, then the client can resume pipelining, and could send requests 4 and 5 (and further requests). Let's say it sends requests 4, 5 and 6. This is a new sequence of requests (no longer [3,4,5]). Now if the server closes the connection before sending a response, then the automatic retry is allowed because this is now a new sequence, rather than an aborted sequence (or second sequence), even though it includes requests 4 and 5 that have been sent twice. Further, if only requests 4 and 5 were sent and the server closed, it would still be a new sequence [4,5],  rather than [3,4,5], and hence the retry would be invoked. Is this the intended meaning of section 6.1.5?
c) If the response is received with the close token, then the connection is no longer persistent and must be closed without any further requests being sent. In this case, request 4 will be sent over a new connection, initially without pipelining to align with the spirit of the approach expressed in paragraph 2 of section 6.1.2.2. If the HTTP client receives a response without the close token, then pipelining can recommence, starting with request 5. If the response does include the close token, then request 5 is treated the same way as request 4. As in a), on reopening a connection HTTP considers any sequence of requests to be new, and hence will retry if the server closes prematurely.

Proposed Additional text: We are not sure if the assumptions made above are correct or not. Once the intended meaning has been ascertained we are willing to prepare additional text to clarify the retry operation.

-------------------------------end of comments on HTTP Part 1 section 6.1--------------------------

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 20 Dec 2011 08:25:41 +0100

Hello,

> These points concentrate on the opening phase of TCP causing delay, because of the 3 way handshake, but equally important is the graceful closing phase, which requires handshakes in both directions…

In fact it depends who closes first and how the close is performed. If the
server closes first, you have the usual 3-way handshake and there is no
delay imposed by the TIME_WAIT state because while the socket remains in
this state on the server, it can immediately be reopened when the client
provides a new SYN above the end of the last window.

If the client closes first using the 3-way handshake, then what you describe
happens and in practice the client is stuck. For this reason, when clients
absolutely needs to close, they usually close with an RST which saves them
from the TIME_WAIT delay and at the same time saves bandwidth since only one
packet is sent.

> "Latency on subsequent requests is reduced since there is no time spent in the double handshakes required for gracefully closing TCP connections and holding the TIMEWAIT state, and in TCP's connect…

I think the 4th point can be left intact without the TIME_WAIT add-on.

> It is not clear what the intent of this sentence is. Does it imply that on the receipt of a response with a close token, the client will close the connection?

No, it indicates that the client should expect the connection to be
closed when it receives this response. This means it should still poll
it to detect the close, and should not send anything else over it when
it receives the full response. We should never expect a client to close
first for the TIME_WAIT issues discussed above.

> Does it also imply that if the response did not contain the close token, then the client would keep the connection open?

No, as it can never be assumed that any agent in the chain will kill a
connection open, this is just best effort from everyone. It means that
the client MAY keep the connection open unless there is a close token
in the Connection header.

> If this is the intent, then a clearer statement of it would help to exclude other interpretations, as the current text seems to allow the client latitude to close or not on either of these events.
> 

It is not the intent. Also, be careful with the wording "the client will",
because it does not look like something mandatory for the client while it
implies that the server can safely rely on this. Prefer the MUST, SHOULD,
MAY etc... wording to indicate who is responsible for what.

> B. Once the server has sent a response with the close token, it will discard any further requests it receives.

I think both are true, indeed.

> "Once the server has sent a response with the close token, it will discard any further requests it receives."

You removed the point related to the client's close token. I think it
should be better to add a sentence after the original one :
"Clients MUST not send any new request over a connection once the last
 one is identified, and servers MUST discard any new request received
 after the last one."

That's all for me now.

Thanks,
Willy

---

From: Dan Winship <[email-masked]>
Date: Tue, 20 Dec 2011 10:22:52 -0500

> 1.  Section 6.1.1 Purpose

So I've also been reviewing the -17 specs, and was going to suggest that
6.1.1 is no longer relevant (except for the "SHOULD implement" part at
the end) and should just be removed. Everyone agrees that persistent
connections are a good thing now; we don't need to spend a whole page
citing 15-year-old research to convince the reader.

-- Dan

---

From: Jonathan Billington <[email-masked]>
Date: Wed, 21 Dec 2011 09:17:28 +1030

Hi Dan,

You wrote:
>So I've also been reviewing the -17 specs, and was going to
>suggest that 6.1.1 is no longer relevant (except for the "SHOULD
>implement" part at the end) and should just be removed.

I tend to agree with the following provisos:
1. The history of why persistent connections were introduced is valuable to newcomers to HTTP, and it would be nice to retain this information somewhere. Perhaps Appendix A is the right place for it.
2. It would also be helpful to provide a succinct definition of persistent connections. Apart from the table of contents, the term appears twice (firstly in section 3.4 Handling Incomplete Messages and then in Section 5.1.1 (Chunking)) before section 6.1, but it is not defined, and there is no forward reference to section 6.1 in section 3.4 (or 5.1.1) for a definition. The current 6.1 provides a feel for what they are, but a more succinct definition would be useful. Perhaps this definition should appear in section 2 (Architecture), due to it being an important part of the rationale behind version 1.1. This would then mean that the term was defined before it is first used in section 3.4.

Jonathan

-----Original Message-----
From: Dan Winship [mailto:[email-masked]]
Sent: Wednesday, 21 December 2011 1:53 AM
To: Jonathan Billington
Cc: [email-masked]; Sonya Arnold
Subject: Re: Comments on Section 6.1 (Persistent Connections) of HTTPbis Part 1, version 17

-- Dan

---

From: Jonathan Billington <[email-masked]>
Date: Wed, 21 Dec 2011 11:50:24 +1030

Hello Willy,

In response to our remark:
>> These points concentrate on the opening phase of TCP causing
>>delay, because of the 3 way handshake, but equally important is
>>the graceful closing phase, which requires handshakes in both
>>directions (FIN followed by ACK for each side), and the holding
>>of the TIMEWAIT state, causing further delay. Because it is
>>equally important to the closing phase, we feel it is useful to
>>include this point in the list of advantages.

>provides a new SYN above the end of the last window.

This last sentence does not correspond to TCP in RFC793.
1. When closing you have two relatively independent two way handshakes (FIN then ACK) as described above, rather than a 3-way handshake.
2. On page 69 of RFC 793 it describes the conditions under which a segment is acceptable in the TIME-WAIT state. It is not acceptable if it is outside the window, which is the case if the sequence number of the new SYN is "above the end of the last window". In this case the RFC states:

        If an incoming segment is not acceptable, an acknowledgment
        should be sent in reply (unless the RST bit is set, if so drop
        the segment and return):

          <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>

        After sending the acknowledgment, drop the unacceptable segment
        and return.

This is further backed up in the case of SYNs on the bottom of page 71. Thus on receipt of a SYN in this case, the server would return an ACK and remain in the TIME-WAIT state, until the 2MSL timer expires. There is also a clarification of this situation in RFC 1122 on page 88.

            When a connection is closed actively, it MUST linger in
            TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime).
            However, it MAY accept a new SYN from the remote TCP to
            reopen the connection directly from TIME-WAIT state, if it:

            (1)  assigns its initial sequence number for the new
                 connection to be larger than the largest sequence
                 number it used on the previous connection incarnation,
                 and

            (2)  returns to TIME-WAIT state if the SYN turns out to be
                 an old duplicate.

Note the mandatory requirement (MUST) to stay in the TIME-WAIT state for 2MSL. The additional text allows (MAY) a SYN to be received (and not dropped) in TIME-WAIT, with presumably a transition to SYN_RCVD (and sending of the SYNACK) after the 2MSL timer expires (to obey the mandatory requirement). Thus TCP does not have to return to LISTEN, it can transition directly from TIME-WAIT to SYN_RCVD. Ignoring the 2MSL requirement is done at your peril, as this could lead to old duplicate segments being accepted instead of the new ones (not good for financial transactions or safety critical applications).

>from the TIME_WAIT delay and at the same time saves bandwidth since only one

The use of RST to close is not mentioned in Part 1 as far as I am aware. On the contrary, one gets the impression in section 6.1.4 Practical Considerations, that the graceful close should be used, at least for timeouts:

   When a client or server wishes to time-out it SHOULD issue a graceful
   close on the transport connection.

This seems to relate to active ticket #176 where the use of graceful close (half-close) is also suggested, to avoid loss of responses.

>I think the 4th point can be left intact without the TIME_WAIT add-on.

See discussion above regarding the mandatory requirement of holding TIME-WAIT in RFCs 793 and 1122. With respect to IETF standards, I think the case is made. That implementers are ignoring the TIME-WAIT state to improve performance at the expense of safety makes me uncomfortable.

This is long enough for this message. I shall address the other points in my next message.

Jonathan

-----Original Message-----
From: Willy Tarreau [mailto:[email-masked]]
Sent: Tuesday, 20 December 2011 5:56 PM
To: Jonathan Billington
Cc: [email-masked]; Sonya Arnold
Subject: Re: Comments on Section 6.1 (Persistent Connections) of HTTPbis Part 1, version 17

Hello,

That's all for me now.

Thanks,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 21 Dec 2011 08:04:32 +0100

Hello Jonathan,

> 1. When closing you have two relatively independent two way handshakes (FIN then ACK) as described above, rather than a 3-way handshake.

There are multiple ways of closing a connection, and the most common in
HTTP when the server closes is a 3-way handshake :
   - server sends FIN
   - client acks and sends FIN
   - server acks client's FIN and puts itself in TIME_WAIT state.

> 2. On page 69 of RFC 793 it describes the conditions under which a segment is acceptable in the TIME-WAIT state. It is not acceptable if it is outside the window, which is the case if the sequence number of the new SYN is "above the end of the last window".

This rule applies to a segment which belongs to an active session. This
is not what I was talking about, I was talking about re-opening a session
which was in TIME_WAIT. Those are two completely different things.

(...)
> This is further backed up in the case of SYNs on the bottom of page 71. Thus on receipt of a SYN in this case, the server would return an ACK and remain in the TIME-WAIT state, until the 2MSL timer expires.

No, not at all. A server would return an ACK to a SYN in TIME_WAIT only if
the SYN was within the last window because it would look like a retransmit.
That's precisely why you need to have it above.

> There is also a clarification of this situation in RFC 1122 on page 88.
> 

It's exactly this rule I'm talking about.

>             (2)  returns to TIME-WAIT state if the SYN turns out to be

It's this one you're confusing it with.

> Note the mandatory requirement (MUST) to stay in the TIME-WAIT state for 2MSL.

Look at the "However" term above :-)

> The additional text allows (MAY) a SYN to be received (and not dropped) in TIME-WAIT, with presumably a transition to SYN_RCVD (and sending of the SYNACK) after the 2MSL timer expires (to obey the mandatory requirement). Thus TCP does not have to return to LISTEN, it can transition directly from TIME-WAIT to SYN_RCVD. Ignoring the 2MSL requirement is done at your peril, as this could lead to old duplicate segments being accepted instead of the new ones (not good for financial transactions or safety critical applications).

I can assure you that you will not find a single TCP stack which does not
apply the rule above, otherwise it would basically not be usable. Also,
when you read RFC793 you have to understand that it was written with
single connections in mind (hence the "return to LISTEN"). Implementations
do not "return to LISTEN", they have one socket always in LISTEN state
from which they instantiate as many sessions as they need upon each new
SYN. The rule above makes perfect sense if you're using TCP on top of a
serial line with a single connection at a time, but this is not what we're
doing here.

Please make a test if you don't believe me. Currently the web would not work
with the rule as you state it. 2 MSL is 240 seconds by default, you would
exhaust the 64k source ports at only 260 connections per second ! I'm commonly
playing with servers which I push up to 200000 connections per second, this is
1000 times above the limit which would be imposed by the rule if implementations
did not apply the "MAY" above.

>    close on the transport connection.

But if the client does this, it is the one which gets the TIME_WAIT state
and this time it is forced to respect the 2*MSL timer. Doing this on a
browser is not critical, but doing this on a proxy stops its operations
after 64k connections. Closing with an RST is not clean at all but sometimes
you have no other choice.

> This seems to relate to active ticket #176 where the use of graceful close (half-close) is also suggested, to avoid loss of responses.

No, this is different, the half-close to avoid loss of response is for the
server, because if the server closes too fast, the last pending data may not
linger long enough in the TCP stack for the client to retrieve them, which
can end up with the client getting a truncated response, especially if the
client uses pipelining and tries to send another request before it gets the
full response.

Regards,
Willy