Subject: [multipathtcp] Multipath TCP FAST_CLOSE - abrupt closing

From: François Finfe <[email-masked]>
Date: Wed, 13 Sep 2017 15:39:20 +0200

Hi,

RFC6824 and RFC6824bis use the FAST_CLOSE option to abruptly close a
Multipath TCP connection. The motivation for the procedure defined in
section 3.5 is to enable a host to quickly terminate a Multipath TCP
connection without being forced to send DATA_FINs.

The current procedure defined in section 3.5 is that the host that
wishes to abruptly terminate a connection must send a RST on all
subflows except one where it sends an ACK with the FAST_CLOSE option and
waits for a RST before closing the Multipath TCP connection. The
FAST_CLOSE option includes the receiver’s key to authenticate the abrupt
release of the subflow. This prevents attacks where a remote attacker
could send spoofed packets that would terminate the Multipath TCP
connection.

Unfortunately, the procedure defined in section 3.5 has a very important
drawback. Although the host wants to abruptly close the Multipath TCP
connection, it needs to maintain state for this connection until the
other host has confirmed (with a RST) the termination of the connection.
This goes against the idea of abruptly closing connections due to lack
of ressources for example. We have faced some issues with applications
using SO_LINGER set to zero that were expecting the socket to have
disappeared after an abrupt release while it did not disappear.

It would make sense to reconsider the abrupt connection termination
procedure defined in section 3.5 to enable a host to immediately
terminate a Multipath TCP connection and release the associated data
structures. One possibility would be to work as follows.

Consider a Multipath TCP connection established between a client and a
server. There are two subflows, A and B. The server wants to abruptly
close the connection. For this, we propose to continue to use the
FAST_CLOSE option, but place it inside the RST packet sent over all
subflows. By using a RST instead of an ACK as in section 3.5, the server
does not need to maintain any state and can remove the data structures
associated with the closed connection immediately. If one of the RST
packets reaches the client, it will know that the entire connection has
been closed by the server.

The only issue with this new approach is when none of the RST packets
sent by the server reach the client. We would be in a situation where
the client considers that the Multipath TCP is active while it does not
exist anymore on the server. After some time, the client will send a
packet to the server over one of the subflows. Since the Multipath TCP
connection does not exist anymore on the server, it will reply with a
RST packet that will close this subflow. The client will receive the
same RST over all the established subflows. When all subflows have been
closed, the client should try to establish a new subflow to the server
by sending a SYN with MP_JOIN. Upon reception of this SYN, the server
will try to match the connection token with the established Multipath
TCP connection and will find that there is no matching connection.
According to RFC6824, the server should reply with a RST packet to close
the subflow. We could add to this RST packet a new MP_TCPRST option
indicating that the server could not match the received token with an
existing connection. Upon reception of a RST carrying this option, the
client would know that the corresponding connection has been closed by
the server and would close it itself.

We believe that this approach is simpler on the server side while still
allowing clients to detect closed Multipath TCP connections. The abrupt
closing (e.g. socket option SO_LINGER with zero timeout) is also
properly implemented. In contrast with the current approach, it places
the burden on the client instead of on the server, but we believe this
is beneficial for servers that often need to abruptly close connections.

François

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 13 Sep 2017 21:16:14 -0700

Hello,

> is beneficial for servers that often need to abruptly close connections.

I strongly support this proposal.

Whenever the stack decides to force-fully close a connection due to
resource-limitations, it is important for the scalability of the system to
get rid of the state as quickly as possible. Otherwise, servers accumulate
huge numbers of these lingering connections.

Christoph

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 14 Sep 2017 15:33:46 -0700

Hi Francois,

This is an interesting proposal for the bis draft. But, I guess the
receiver of FAST_CLOSE (will be clients in most cases) may need to
wait longer time in some cases.

I am thinking that the main motivation of SO_LINGER with zero timeout
approach is to avoid going into time_wait.
I think MPTCP's FAST_CLOSE already takes care of this point to some
extent and senders don't go into M_TIMEWAIT, although it requires at
least 1 RTT.
So, I am still not very sure if we should save this 1 RTT or should
avoid unfortunate cases for long waiting.

BTW, I happened to find that the state transition diagram in the draft
doesn't contain M_FASTCLOSE state, although it is mentioned in Section
3.5.
Is this OK?

Thanks,
--
Yoshi

On Wed, Sep 13, 2017 at 6:39 AM, François Finfe
<[email-masked]> wrote:
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 15 Sep 2017 10:25:49 +0200

Hi Yoshifumi,
> approach is to avoid going into time_wait.

I totally agree

> extent and senders don't go into M_TIMEWAIT, although it requires at

This requires one RTT when there are no packet losses. Unfortunately, 
with mobile nodes, there is no guarantee that the ACK packet sent with 
the FAST_CLOSE option will reach the client. If this packet is lost, the 
server needs to wait for the expiration of its timer to retransmit the 
ACK with the FAST_CLOSE option. This ACK could again be lost as well...

> avoid unfortunate cases for long waiting.

The issue if when there is no reply to the ACK sent by the server. This 
forces the server to maintain state for a potentially long duration to 
retransmit the ACK with FAST_CLOSE. Francois' proposal avoids this and 
forces the client to retransmit a SYN later to verify that the 
connection does not exist anymore. When there are no packet losses, the 
client is informed immediately with the RST that includes FAST_CLOSE.

Olivier

--

---

From: François Finfe <[email-masked]>
Date: Mon, 18 Sep 2017 13:58:54 +0200

Hi Yoshi,

> avoid unfortunate cases for long waiting.
Yes, in most likely cases, it will last 1 RTT. In worst case, with 3
retransmissions and exp. backoff enabled, it will linger 8 RTO's.

1 RTT is relatively very high compared to the time to send an
MP_FASTCLOSE. On the current linux MPTCP implementation and on my
computer, the time required to fast close a MPTCP connection (send
MP_FASTCLOSE and RST's, without taking account the retransmissions or
waiting for RST from the other peer) is 1 ms.

This time depends of a lot of factor but it is quite small compared to
typical values of RTT for DSL/LTE.

In the case of the SO_LINGER with zero timeout, the user space will
expect that "all queued messages for the socket have been successfully
sent or the linger timeout has been reached" and nothing is done on
background. By waiting for a RST in response to a MP_FASTCLOSE, it is
not possible to match the timing requirement, or it must be done it in
the background.

Abrupt connection closing can be explicitly set by the user space
software explicitly.

But, it can be tacitly set as well:
- on GNU/Linux, if a process is killed, all the established connection
  will be abruptly closed.
- Other scenarii are when the system is under pressure.

In these cases, quickly close a connection  is important.
> doesn't contain M_FASTCLOSE state, although it is mentioned in Section
I don't know why it's not mentioned. Maybe, it is omitted for the sake
of clarity, because you can switch from a lot of state (not only
established) to closed via fast close. On most TCP state transition
diagram, this transition (via a RST instead of a fast close) is not
mentioned either.

François

--

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 20 Sep 2017 00:37:20 -0700

Hi Oliver, Francois,

Thanks for the comments.
One concern I have is it seems that people are expecting to expose this
feature to apps although this won't be a recommended way to terminate an
MPTCP session.
I think It won't be good if we end up always using it even when there's no
resource limitations.
--
Yoshi

On Mon, Sep 18, 2017 at 4:58 AM, François Finfe <[email-masked]
> wrote:

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 20 Sep 2017 11:14:54 +0200

Yoshifumi,
> I think It won't be good if we end up always using it even when there's 
> no resource limitations.

An implementation could send RST with an option on all paths when 
SO_LINGER is set to zero or under pressure and send RST without option 
on all paths except one when SO_LINGER is larger

Olivier

--

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 21 Sep 2017 11:00:25 -0700

Hi Olivier,

On Wed, Sep 20, 2017 at 2:14 AM, Olivier Bonaventure <
[email-masked]> wrote:

> SO_LINGER is set to zero or under pressure and send RST without option on
> all paths except one when SO_LINGER is larger

Right. I just thought it won't be good if applications always do SO_LINGER
with zero timeout, although it'll be tricky for us to prevent it.
In unfortunate cases, I think client will need to wait well enough amount
of time or need to check possible paths to see if the session is really
gone.

I have a bit concern about the MP_TCPRST approach suggested by
Francois, because
MP_TCPRST probably doesn't have key info for authentication.
So, I think we had better not close the session unless we receive MP_TCPRST
from all possible paths (or at least more than 1)

OTOH, if the client try to reduce the waiting time because of optimization,
it might hurt break-before-make scenarios.
--
Yoshi