Subject: Closing on CONNECTION_CLOSE

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jul 2017 06:47:17 -0700

There have been a number of issues raised about the exact semantics of
CONNECTION_CLOSE [0], but we have never really closed on them. As Ryan
says, one might have think that CONNECTION_CLOSE is like TCP FIN or
like TCP RST.

In the FIN-like model:
- The closing side sends a CONNECTION_CLOSE
- The non-closing side ACKs it
- The closing side keeps retransmitting until it gets an ACK or it
  times out.

In the RST-like model:
- The closing side sends a CONNECTION_CLOSE
- The non-closing side just terminates the connection
- The closing side responds to new packets from the non-closing side
  with a retransmitted CONNECTION_CLOSE (perhaps just a stored packet)
  for some time.

The specification seems kind of vague on this point (there is a big
TODO for TIME_WAIT) and in places suggests retransmission [1] I had
previously assumed the FIN-like model but after talking to Ian this
morning I think the RST-like model is better. It's easier to implement
because you don't need to manage a post-close state machine on either
side. The only downside I'm really aware of is that the closing side
has to keep state for something like 2MSL (so that it can properly
respond to late packets), rather than being able to clean up on
receiving an ACK. However, this isn't that big a deal, because as
noted above, you can throw away the connection and just send a stored
packet, or alternately, just send public reset (or just go silent).

Given that ID1 does include CONNECTION_CLOSE, albeit with limited
semantics, it would be good to resolve this soon. Are there people who
want to argue in favor of the FIN-like model?

-Ekr

[0] https://github.com/quicwg/base-drafts/issues/330
https://github.com/quicwg/base-drafts/issues/328

[1] See also the spec for ID1 which says:
"The packet containing a connection close does not need to be
retransmitted."

---

From: Ian Swett <[email-masked]>
Date: Thu, 27 Jul 2017 09:58:14 -0400

Nice summary.  As you implied, I favor the RST-like model.

And as you stated, it'd be very helpful to specify something soon, since
it's about to become an implementation blocker.

---

From: Christian Huitema <[email-masked]>
Date: Thu, 27 Jul 2017 07:02:33 -0700

> want to argue in favor of the FIN-like model?

Yes the text is vague. My ID1 code implements the RST behavior --
immediate close on receipt, no ACK. It seemed natural.

I have yet to implement a proper zombie state, but that would be needed
whether we want the RST or FIN semantic, since FIN packets can be lost.

RST behavior is cleaner there: repeat the RST frame if packets are still
received after some timeout.

-- 
Christian Huitema

---

From: Subodh Iyengar <[email-masked]>
Date: Thu, 27 Jul 2017 14:08:33 +0000

> The only downside I'm really aware of is that the closing side
receiving an ACK

I favor the RST model as well.
You'll end up having to keep a loss timeout even in the FIN model because
of ACKs being lost / the remote side shutting for unrelated reasons / silently closing
their connection.

However what is the right response to late packets? The only response I can think of
is to resend the connection close if you still have the connection state. Is there anything
else that could be sent that I'm missing?

As an aside on the topic of TIME_WAIT itself:

The reason I'm aware that TIME_WAIT exists in TCP is to protect us from the local port
being re-bound to another connection and the data from the old connection interfering with
the new connection. With QUIC this seems less of an issue though because a
combination of (connection id + packet type + packet encryption) can be used to determine whether or
not the packet was valid to be received in the context of a new connection. Going with the
philosophy of dropping packets on the floor that are malformed that martin proposed during
the last WG meeting, do we even need a TIME_WAIT state in QUIC?

The advantage of TIME_WAIT that I see is this ability to send the CONNECTION_CLOSE in a timely
manner to prevent the client from needing to retry till the RTO limit and realize the connection
needs to be closed. However if we go with the close model of graceful close always driven by the app, then
this won't be a common case and we might deliver a PUBLIC RESET instead of a CLOSE.

Subodh

________________________________
From: QUIC <[email-masked]> on behalf of Christian Huitema <[email-masked]>
Sent: Thursday, July 27, 2017 7:02:33 AM
To: [email-masked]
Subject: Re: Closing on CONNECTION_CLOSE

--
Christian Huitema

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jul 2017 07:12:18 -0700

> else that could be sent that I'm missing?

There seem to be three options, which we can leave to the implementation:

- Re-send CONNECTION_CLOSE
- Send Stateless Reset
- Drop the packet.

> this won't be a common case and we might deliver a PUBLIC RESET instead of

Yes, that would probably also work. As above, do we need to decide?

-Ekr

>
> Subodh
>
> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Christian Huitema <
> [email-masked]>
> *Sent:* Thursday, July 27, 2017 7:02:33 AM
> *To:* [email-masked]
> *Subject:* Re: Closing on CONNECTION_CLOSE
>
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 27 Jul 2017 16:20:02 +0200

> of ACKs being lost / the remote side shutting for unrelated reasons / silently closing

I prefer the RST model as well in general, but it's important to keep in
mind that it doesn't apply perfectly everywhere. In TCP, when the receiver
is waiting for data and has nothing to say and the sender suddenly sends
an RST which is lost, the sender definitely closes and the receiver stays
open forever. This is even more visible through firewalls because if the
RST is lost between the firewall and the receiver, the firewall will quickly
close its session and any receiver's packets will then be blocked. The usual
case is when a client/proxy has to close an idle connection going to a server
and cannot use the regular FIN model for TIME_WAIT accumulation reasons. In
this case using massive RSTs occasionally exhibits zombie connections on the
server.

I don't think it would be an issue with QUIC since the use cases are very
dynamic connections where timeouts could be expected to be reasonably low
(ie no need to have multi-days timeouts there as is sometimes done with
SSH connections).

> the last WG meeting, do we even need a TIME_WAIT state in QUIC?

Agreed, TIME_WAIT is a protection against fast reuse of a scarce resource
which could cause confusion (source ports, sequence numbers). If we have
no risk of accidently mixing up connections, better get rid of it.

Willy

---

From: "Swindells, Thomas (Nokia - GB/Cambridge, UK)" <[email-masked]>
Date: Thu, 27 Jul 2017 14:47:28 +0000

> > However, this isn't that big a deal, because as
Should/would you have to generate a newly encrypted packet with a new packet number each time (as per other retransmissions)?
Or is this a special case because it is the final packet and so doesn't matter if it is duplicated?

---

From: Ian Swett <[email-masked]>
Date: Thu, 27 Jul 2017 10:51:02 -0400

I would recommend sending an identical packet every time, since there's no
value to encrypting a new one with a new packet number, but it increases
the amount of state you have to keep and the CPU to respond to spurious
packets.

On Thu, Jul 27, 2017 at 10:47 AM, Swindells, Thomas (Nokia - GB/Cambridge,
UK) <[email-masked]> wrote:

> Should/would you have to generate a newly encrypted packet with a new
> packet number each time (as per other retransmissions)?

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jul 2017 07:56:35 -0700

I just filed a PR for this at:
https://github.com/quicwg/base-drafts/pull/705

It explicitly permits duplication.

-Ekr

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 27 Jul 2017 15:35:49 +0000

I'm also in favor of the RST model. Subodh -- to your point about
time_wait, this dovetails with a timeout we need to have in place for
regular connection close, where we must allow for retransmissions to be
sent and for responses to peer's retransmissions. (It's what I call
DRAIN_TIME).

So basically, once a connection close is sent, the sender waits for
DRAIN_TIME and then discards state. The receivers sends an ack and
immediately discards state on receipt.

As ekr notes, receipt of this ack could circumvent the DRAIN_TIME timeout.
TCP time wait accounts for network lifetime of packets, which we don't have
to handle, since once both sides are torn down, there's no danger of an old
connection interfering with a new one. That said, I'm still thinking about
if it's possible for an old packet circling the network to create
connection state when received after all state is torn down... If we want
to ensure that doesn't happen, we'd want to account for network lifetime (2
msl).

On Jul 27, 2017 7:59 AM, "Eric Rescorla" <[email-masked]> wrote:

>> I would recommend sending an identical packet every time, since there's
>> no value to encrypting a new one with a new packet number, but it increases
>> On Thu, Jul 27, 2017 at 10:47 AM, Swindells, Thomas (Nokia -
>> GB/Cambridge, UK) <[email-masked]> wrote:
>>

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 27 Jul 2017 08:47:57 -0700

> immediately discards state on receipt.

Jana,

I'm finding myself a bit confused. Can you perhaps draw me a diagram of
what you mean when
you say "allow for retransmissions to be sent"...?

-Ekr

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 28 Jul 2017 08:49:51 -0400

On Thu, Jul 27, 2017 at 03:35:49PM +0000, Jana Iyengar wrote:
> That said, I'm still thinking about if it's possible for an old packet
> circling the network to create connection state when received after
> all state is torn down... If we want to ensure that doesn't happen,
> we'd want to account for network lifetime (2
> msl).

Unlikely, because version negotiation occurs when new connection is
established and thus first few incoming packets would have to have the
VERSION bit set:

  " A QUIC connection begins with a client sending a handshake packet. The
  " details of the handshake mechanisms are described in {{handshake}},
  " but all of the initial packets sent from the client to the server
  " MUST use the long header format and MUST specify the version of the
  " protocol being used.

  - Dmitri.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 28 Jul 2017 13:15:11 -0400

I agree that the rst-like semantic in the pull request is sensible here.
Some more language around expecting applications to do graceful shutdown
before this happens would be helpful - especially with the definition of
GOAWAY migrating out of the transport document. This works well with the
silent close concept so that makes me happy.

I would like the text to say that the receiver should not discard data that
it has acknowledged before receipt of the CLOSE (discarding it is the TCP
receiver RST behavior). Are there objections to adding that?

I disagree a little bit with (I think) Subodh and Wily regarding what to
send when you recv data after you have sent a CLOSE. I want to strongly
encourage implementations to have a TIME_WAIT like period of being able to
a fully protected CLOSE (it can be a memcpy retransmit). public reset ought
to be left as a last resort.

---

From: Christian Huitema <[email-masked]>
Date: Fri, 28 Jul 2017 11:49:09 -0700

On 7/28/2017 10:15 AM, Patrick McManus wrote:
> I disagree a little bit with (I think) Subodh and Wily regarding what
> to send when you recv data after you have sent a CLOSE. I want to
> strongly encourage implementations to have a TIME_WAIT like period of
> being able to a fully protected CLOSE (it can be a memcpy retransmit).
> public reset ought to be left as a last resort.

As Subodh pointed out, "TIME_WAIT exists in TCP is to protect us from
the local port being re-bound to another connection and the data from
the old connection interfering with the new connection." We don't have
that constraint of old packets resurfacing 30 seconds later and
interfering with a new data flow. Encryption protects against that. The
"waiting" or "draining" period would mostly be a courtesy to the peer,
to ensure that the CLOSE frame is well received, and that the peer
actually stops sending.

The normal duration of the draining period would be "sufficiently more
than 1 RTT". With the current spec, the closer knows that the CLOSE has
not been received yet if it keeps receiving packets from the peer. This
should normally last for 1 RTT after sending the CLOSE frame. If it
receives any packets more than 1 RTT after sending the CLOSE, it can
assume that the CLOSE frame was not received. It would make sense to
resend it then.

Of course, there is a question of how long to wait after 1 RTT. Probably
"some time depending on the application".

We could be tempted to specify acknowledgements, etc. But then we have
to remember the two generals...

-- 
Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 28 Jul 2017 16:31:56 -0400

As Subodh pointed out, "TIME_WAIT exists in TCP is to protect us from the
local port being re-bound to another connection and the data from the old
connection interfering with the new connection."

That would be another argument for making the connection independent of
5-tuple. It would also simplify error handling during early handshake and
decouple protocol for from lower layer.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 28 July 2017 at 20.49.25, Christian Huitema ([email-masked]) wrote:

As Subodh pointed out, "TIME_WAIT exists in TCP is to protect us from the
local port being re-bound to another connection and the data from the old
connection interfering with the new connection." We don't have that
constraint of old packets resurfacing 30 seconds later and interfering with
a new data flow. Encryption protects against that. The "waiting" or
"draining" period would mostly be a courtesy to the peer, to ensure that
the CLOSE frame is well received, and that the peer actually stops sending.

The normal duration of the draining period would be "sufficiently more than
1 RTT". With the current spec, the closer knows that the CLOSE has not been
received yet if it keeps receiving packets from the peer. This should
normally last for 1 RTT after sending the CLOSE frame. If it receives any
packets more than 1 RTT after sending the CLOSE, it can assume that the
CLOSE frame was not received. It would make sense to resend it then.

We could be tempted to specify acknowledgements, etc. But then we have to
remember the two generals...

--
Christian Huitema