Subject: [multipathtcp] MPTCP FAST_CLOSE

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 13 Nov 2017 15:30:23 +0100

Hello,

Sorry for being late with this, but the beginning of the fall semester 
is always too busy for me.

During the summer, there was a discussion on the FAST_CLOSE option that 
was triggered by Francois.

The current text of RFC6824bis uses the FAST_CLOSE option inside ACK 
packets, but deployment experience has shown that this results in 
additional complexity since the ACK must be retransmitted to ensure its 
reliable delivery.

Here is a proposed update for section 3.5 that allows a sender to either 
send FAST_CLOSE inside ACK or RST while a receiver must be ready to 
process both. Changes are marked with * *

------------------
3.5.  Fast Close

    Regular TCP has the means of sending a reset (RST) signal to abruptly
    close a connection. With MPTCP, a *reguler* RST only has the scope 
of the
    subflow and will only close the concerned subflow but not affect the
    remaining subflows.  MPTCP's connection will stay alive at the data
    level, in order to permit break-before-make handover between
    subflows.  It is therefore necessary to provide an MPTCP-level
    "reset" to allow the abrupt closure of the whole MPTCP connection,
    and this is the MP_FASTCLOSE option.

    MP_FASTCLOSE is used to indicate to the peer that the connection will
    be abruptly closed and no data will be accepted anymore.  The reasons
    for triggering an MP_FASTCLOSE are implementation specific.  Regular
    TCP does not allow sending a RST while the connection is in a
    synchronized state [RFC0793].  Nevertheless, implementations allow
    the sending of a RST in this state, if, for example, the operating
    system is running out of resources.  In these cases, MPTCP should
    send the MP_FASTCLOSE.  This option is illustrated in Figure 14.

                             1                   2                   3
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +---------------+---------------+-------+-----------------------+
        |     Kind      |    Length     |Subtype|      (reserved)       |
        +---------------+---------------+-------+-----------------------+
        |                      Option Receiver's Key                    |
        |                            (64 bits)                          |
        |                                                               |
        +---------------------------------------------------------------+

                 Figure 14: Fast Close (MP_FASTCLOSE) Option

    If Host A wants to force the closure of an MPTCP connection, *it has 
two different options* :

    - *Option A (ACK) : *Host A sends an ACK containing the MP_FASTCLOSE 
option on one
       subflow, containing the key of Host B as declared in the initial
       connection handshake.  On all the other subflows, Host A sends a
       regular TCP RST to close these subflows, and tears them down.
       Host A now enters FASTCLOSE_WAIT state.

    - *Option R (RST) : Host A sends a RST containing the MP_FASTCLOSE 
option on all
       subflows, containing the key of Host B as declared in the initial
       connection handshake.  Host A can tear the subflows and the 
connection down immediately.*

    *If a host receives a packet with a valid MP_FASTCLOSE option, it 
shall process it as follows :*

    o  Upon receipt of an *ACK* with MP_FASTCLOSE, containing the valid 
key, Host B answers on the same subflow with a TCP RST and tears down all
       subflows.  Host B can now close the whole MPTCP connection (it
       transitions directly to CLOSED state).

    o  As soon as Host A has received the TCP RST on the remaining
       subflow, it can close this subflow and tear down the whole
       connection (transition from FASTCLOSE_WAIT to CLOSED states).  If
       Host A receives an MP_FASTCLOSE instead of a TCP RST, both hosts
       attempted fast closure simultaneously.  Host A should reply with a
       TCP RST and tear down the connection.

    o  If Host A does not receive a TCP RST in reply to its MP_FASTCLOSE
       after one retransmission timeout (RTO) (the RTO of the subflow
       where the MPTCP_RST has been sent), it SHOULD retransmit the
       MP_FASTCLOSE.  The number of retransmissions SHOULD be limited to
       avoid this connection from being retained for a long time, but
       this limit is implementation specific.  A RECOMMENDED number is 3.
       If no TCP RST is received in response, Host A SHOULD send a TCP
       RST *with the MP_FASTCLOSE option* itself when it releases state 
in order to clear any remaining state at middleboxes.

  *o  Upon receipt of a RST with MP_FASTCLOSE, containing the valid key, 
Host B tears down all subflows. Host B can now close the whole MPTCP 
connection (it transitions directly to CLOSED state).*

----------

The burden of the MP_FASTCLOSE is on the host that decides to close the 
connection (typically the server, but not necessarily). The text above 
allows it to opt for either ACK or RST without adding complexity to the 
other side.

Another point that might be discussed in RFC6824bis is how a host should 
react when there are all the subflows associates to a Multipath TCP 
connection are in the CLOSED state, but the connection is still alive. 
At this point, it is impossible to send or receive data over the 
connection. It should attempt to reestablish one subflow to keep the 
connection alive.

Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 13 Nov 2017 14:47:40 -0800

Hi Olivier,

Thanks for the proposed texts. I have two comments on this.

1: We now have two options for host A, is there any guideline which options
should be chosen in a certain cases?'
2: In case of option R, I think host A cannot receive packets from host B
and cannot wait for timeouts. So, in some unfortunate cases, host B might
need to wait for a long time.
    Is this fine? Or, we have some mitigations?

--
Yoshi

On Mon, Nov 13, 2017 at 6:30 AM, Olivier Bonaventure <
[email-masked]> wrote:

> Hello,
>
> Sorry for being late with this, but the beginning of the fall semester is
> always too busy for me.
>
> send FAST_CLOSE inside ACK or RST while a receiver must be ready to process
> both. Changes are marked with * *
>
> ------------------
> 3.5.  Fast Close
>
>    close a connection. With MPTCP, a *reguler* RST only has the scope of
> the
>    *If a host receives a packet with a valid MP_FASTCLOSE option, it shall
> process it as follows :*
>
>       RST *with the MP_FASTCLOSE option* itself when it releases state in
> order to clear any remaining state at middleboxes.
>
> connection are in the CLOSED state, but the connection is still alive. At
> this point, it is impossible to send or receive data over the connection.
> It should attempt to reestablish one subflow to keep the connection alive.
>
>
> Olivier
>

---

From: Alan Ford <[email-masked]>
Date: Tue, 14 Nov 2017 11:12:45 +0000

Hi Olivier,

Can you clarify what issue we are trying to solve here? Is this a purely a matter of shutting down a connection rapidly without maintaining state?

The minimum to be compliant as written requires RSTing every subflow bar one, then sending MP_FASTCLOSE on one subflow. The retransmit is only a SHOULD, so you could theoretically RST that subflow after a RTO too. Or is that RTO still too much to keep?

I don’t have an issue with adding this methodology - it does not require any significant changes, just specifies how to behave when receiving RST + MP_FASTCLOSE.

The acknowledged model was designed so that it was less likely that any state would be left at middleboxes and at the client. This does not ensure it was received at the client so in theory the client could sit there not knowing the connection had gone away. However, this is no worse than regular TCP so I don’t immediately see any problems here.

Regards,
Alan

> On 13 Nov 2017, at 14:30, Olivier Bonaventure <[email-masked]> wrote:
> 
> Hello,
> 
> Sorry for being late with this, but the beginning of the fall semester is always too busy for me.
> 
> During the summer, there was a discussion on the FAST_CLOSE option that was triggered by Francois.
> 
> The current text of RFC6824bis uses the FAST_CLOSE option inside ACK packets, but deployment experience has shown that this results in additional complexity since the ACK must be retransmitted to ensure its reliable delivery.
> 
> Here is a proposed update for section 3.5 that allows a sender to either send FAST_CLOSE inside ACK or RST while a receiver must be ready to process both. Changes are marked with * *
> 
> ------------------
> 3.5.  Fast Close
> 
>   close a connection. With MPTCP, a *reguler* RST only has the scope of the
>   If Host A wants to force the closure of an MPTCP connection, *it has two different options* :
> 
>   - *Option A (ACK) : *Host A sends an ACK containing the MP_FASTCLOSE option on one
>   - *Option R (RST) : Host A sends a RST containing the MP_FASTCLOSE option on all
>      connection handshake.  Host A can tear the subflows and the connection down immediately.*
> 
> 
> 
>   *If a host receives a packet with a valid MP_FASTCLOSE option, it shall process it as follows :*
> 
>   o  Upon receipt of an *ACK* with MP_FASTCLOSE, containing the valid key, Host B answers on the same subflow with a TCP RST and tears down all
>      RST *with the MP_FASTCLOSE option* itself when it releases state in order to clear any remaining state at middleboxes.
> 
> *o  Upon receipt of a RST with MP_FASTCLOSE, containing the valid key, Host B tears down all subflows. Host B can now close the whole MPTCP connection (it transitions directly to CLOSED state).*
> 
> ----------
> 
> 
> The burden of the MP_FASTCLOSE is on the host that decides to close the connection (typically the server, but not necessarily). The text above allows it to opt for either ACK or RST without adding complexity to the other side.
> 
> 
> Another point that might be discussed in RFC6824bis is how a host should react when there are all the subflows associates to a Multipath TCP connection are in the CLOSED state, but the connection is still alive. At this point, it is impossible to send or receive data over the connection. It should attempt to reestablish one subflow to keep the connection alive.
> 
> 
> Olivier

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 14 Nov 2017 04:48:18 -0800

On Tue, Nov 14, 2017 at 3:12 AM, Alan Ford <[email-masked]> wrote:

> Hi Olivier,
>
> Can you clarify what issue we are trying to solve here? Is this a purely a
> matter of shutting down a connection rapidly without maintaining state?
>
> The minimum to be compliant as written requires RSTing every subflow bar
> one, then sending MP_FASTCLOSE on one subflow. The retransmit is only a
> SHOULD, so you could theoretically RST that subflow after a RTO too. Or is
> that RTO still too much to keep?
>
> I don’t have an issue with adding this methodology - it does not require
> any significant changes, just specifies how to behave when receiving RST +
> MP_FASTCLOSE.
>
> The acknowledged model was designed so that it was less likely that any
> state would be left at middleboxes and at the client. This does not ensure
> it was received at the client so in theory the client could sit there not
> knowing the connection had gone away. However, this is no worse than
> regular TCP so I don’t immediately see any problems here.
>
> I think there're situations where it get worse than regular TCP.
When the client send some data to the server due to new data or
retransmission, the server will respond with reset,which make the client
realize the connection has already gone.
However, in case of mptcp, even the client receives reset, it cannot be
sure if the mptcp session has gone or not.

--
Yoshi

---

From: Christoph Paasch <[email-masked]>
Date: Sun, 26 Nov 2017 21:33:45 -0800

Hello Alan,

> Can you clarify what issue we are trying to solve here? Is this a purely a matter of shutting down a connection rapidly without maintaining state?

the main motivation is that when a host decides to force-close
an MPTCP-connection, in many cases it is due to resource-shortage (running
out of memory). In that case, we really want to be able to get rid of the
state as quickly as possible. Sending ACK+MP_FASTCLOSE, immediately followed
by a RST would do the job as well, but I think the RST+MP_FASTCLOSE option
is cleaner.

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Sun, 26 Nov 2017 21:35:01 -0800

Hello Yoshifumi,

> should be chosen in a certain cases?'

this is probably implementation specific, depending on the reason why the
stack decides to force-close the connection. For the cases where the forced
closure is due to resource-limitations option R should be used.

Christoph