Subject: Re: [multipathtcp] RFC6824bis: knowing if a Data ACK should have been received?

From: Bob Briscoe <[email-masked]>
Date: Mon, 1 Dec 2014 18:33:27 +0000

Olivier, Yoshifumi [I've pasted your reply into Olivier's below],

At 21:08 30/11/2014, Olivier Bonaventure wrote:
>Bob,
>>
>>I am trying to determine how an MPTCP endpoint will know if TCP options
>>start to be stripped part-way through a subflow (e.g. due to a mobile IP
>>handover that MPTCP is not aware of, which brings a new middlebox into
>>the path).
>
>This could indeed be a problem. For simplicity, let us assume that 
>the new middlebox removes all MPTCP options from all segments (it 
>seems safe to assume that this middlebox is not MPTCP aware and thus 
>should treat all MPTCP options in the same way). Then, we have to 
>considered two different cases :
>
>1- The MPTCP connection uses two subflows, one affected by the 
>middlebox and the other not
>
>2- The MPTCP connection uses a single subflow, affected by the middlebox
>
>In case 1, the Data_ack will not arrive anymore on the affected 
>subflow and after some time the data (at the MPTCP level) will be 
>retransmitted on the other subflow. Data flow will continue on one 
>subflow after a slowdown. This is what we already see with some FTP 
>ALG as described in 
>http://inl.info.ucl.ac.be/publications/are-tcp-extensions-middlebox-proof
>
>In case 2, the problem is different. The data cannot be 
>retransmitted over another subflow. Maybe an implementation could 
>detect this problem after N attemtps to retransmit the data.

I may be missing sth (I admit I only have partial knowledge of 
MPTCP), but why does suddenly starting to strip options prevent data 
being sent? I imagine that stripping options would only prevent data 
being acknowledged at the Data ACK level. Each subflow will still ACK 
data, so why will there be any retransmission needed?

>Since MPTCP options are stripped, mp_fail and infinite mappings 
>cannot be exchanged to solve the problem. Maybe one option could be 
>to close the subflow and restablish another one.

Before we get to solutions, the first problem to get over is how to 
detect there is a problem - per subflow. That's why I thought some 
degree of deterministic Data ACK behaviour would be useful. More..

>>I guess the best way to tell would be absence of DATA ACKs, however the
>>RFC6842bis draft does not encourage an implementation to send DATA ACKs
>>unless the cumulative Data ACK has advanced (see the para quoted below).
>>
>>* Question: If all the cumulative ACKs of all the subflows have
>>advanced, can an implementation be sure that the Data ACK should have
>>advanced and therefore be certain that it should have received a Data ACK?
>>
>>* Edit request: While reading RFC6842bis I noticed a MUST seems to be
>>missing...
>>
>><https://tools.ietf.org/html/draft-ietf-mptcp-rfc6824bis-02#section-3.3.2>
>>3.3.2. Data Acknowledgments
>>CURRENT:
>>     The Data ACK MAY be included in all segments; however, optimizations
>>     SHOULD be considered in more advanced implementations, where the Data
>>     ACK is present in segments only when the Data ACK value advances, and
>>     this behavior MUST be treated as valid.  This behavior ensures the
>>     sender buffer is freed, while reducing overhead when the data
>>     transfer is unidirectional.
>>SUGGESTED:
>>     The Data ACK MUST be included when the Data ACK value advances. It MAY
>>     be included in all segments; however, optimizations SHOULD be
>>considered
>>     in more advanced implementations, where the Data ACK is present only
>>when
>>     it advances, and this behavior MUST be treated as valid.  This
>>behavior ensures the
>>     sender buffer is freed, while reducing overhead when the data
>>     transfer is unidirectional.

At 20:50 30/11/2014, Yoshifumi Nishida wrote:
>What if there is no enough option space to put MPTCP option?

OK, a MUST would be too strong.

[Alternatively we could use one of the option space schemes being 
proposed in tcpm (hint: if you need to pick one, Inner Space is really cool ;)
]

>I'm not in favor of forcing a data_ack each time the data ack 
>advances because this would prevent the utilisation of delayed acks 
>that are useful to reduce the cpu consumption. If we go in this 
>direction, we should at least agree to not send more data_ack than 
>subflow level tcp acks

That's understood. The wording was only a suggestion. The underlying needs are:
- determinism in at least one MPTCP option, so that the endpoints can 
tell if they stop;
- clarity that Data ACKs do have to be sent eventually, most probably 
by specifying how much they can be deferred by.

Bob

>Olivier
>
>--
>INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 02 Dec 2014 08:11:28 +0100

Bob,
>> This could indeed be a problem. For simplicity, let us assume that the
>> new middlebox removes all MPTCP options from all segments (it seems
>> safe to assume that this middlebox is not MPTCP aware and thus should
>> treat all MPTCP options in the same way). Then, we have to considered
>> two different cases :
>>
>> In case 2, the problem is different. The data cannot be retransmitted
>> over another subflow. Maybe an implementation could detect this
>> problem after N attemtps to retransmit the data.
>
> I may be missing sth (I admit I only have partial knowledge of MPTCP),
> but why does suddenly starting to strip options prevent data being sent?

Data is sent and acked at the subflow level but not acked at the MPTCP 
level since the new middlebox strips all MPTCP options.

> I imagine that stripping options would only prevent data being
> acknowledged at the Data ACK level. Each subflow will still ACK data, so
> why will there be any retransmission needed?

The MPTCP-level retransmissions are included in MPTCP to cope notably 
with problems due to middlebox interference.

>> Since MPTCP options are stripped, mp_fail and infinite mappings cannot
>> be exchanged to solve the problem. Maybe one option could be to close
>> the subflow and restablish another one.
>

Basically if a subflow suddenly strips all MPTCP options, then there is 
nothing useful that MPTCP can do with this subflow. If the DSS option 
does not pass through anymore, then MPTCP cannot send data over this 
subflow.

>
>> I'm not in favor of forcing a data_ack each time the data ack advances
>> because this would prevent the utilisation of delayed acks that are
>> useful to reduce the cpu consumption. If we go in this direction, we
>> should at least agree to not send more data_ack than subflow level tcp
>> acks
>

My concern is that if all MPTCP options that passed through are suddenly 
stripped, then I don't see what can be done

> - clarity that Data ACKs do have to be sent eventually, most probably by
> specifying how much they can be deferred by.

The Data ACKs should be sent when both :

  - a subflow level ACK is sent according to the TCP rules
  - and the Data_ack advances

The wording that we use should be compatible with large DSS mappings 
that could be used by hosts that support TSO/GRO.

Olivier

Olivier

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 17 Dec 2014 17:33:35 +0000

Olivier,

At 07:11 02/12/2014, Olivier Bonaventure wrote:
>Data is sent and acked at the subflow level but not acked at the 
>MPTCP level since the new middlebox strips all MPTCP options.

That was my point. The sender can still /send/ data until the 
sender's flow-control window reduces to zero due to the Data ACKs 
being stripped. So we can specify how the receiver detects lack of 
MPTCP options on this 'dying gasp' of data.

>The MPTCP-level retransmissions are included in MPTCP to cope 
>notably with problems due to middlebox interference.

Understood. Thx

>Basically if a subflow suddenly strips all MPTCP options, then there 
>is nothing useful that MPTCP can do with this subflow. If the DSS 
>option does not pass through anymore, then MPTCP cannot send data 
>over this subflow.

If case 1, other combinations of interfaces could be tried.
(I don't think we should add the ability to close one subflow over a 
different subflow - adds an unnecessary new vulnerability.)
If case 2, an error can at least be thrown up to the app and the 
subflow closed at the TCP level.

Whatever, you keep jumping to solution mode. But I'm trying to keep 
the focus on problem detection first. The reason I sent the email 
originally was because the MPTCP spec is not deterministic enough to 
even determine when there is a problem.

On one hand, an MPTCP session doesn't need a solution if it is not 
yet certain there is a problem.
On the other, once both ends of a session can deterministically know 
the subflow's dead, they can at least stop trying to be alive. Both 
ends can shut themselves down with a timeout if data has been / is 
being sent, but no MPTCP options are getting through and no Data ACKs 
are getting back.

>My concern is that if all MPTCP options that passed through are 
>suddenly stripped, then I don't see what can be done
>

Someone needs to propose the precise wording (I'm not expert enough 
on MPTCP), and failure detection will need to be implemented against 
the wording.

Cheers

Bob

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 17 Dec 2014 22:52:37 +0100

Bob,

>>> degree of deterministic Data ACK behaviour would be useful. More..

Let's go back to detection first as you suggest. For simplicity, I 
assume that data flows only in the A->B direction and there is a single 
subflow, which is the worst case. If there are two subflows, one 
affected by a middlebox that suddently strips MPTCP options and not the 
other, then MPTCP will eventually stop using the path where options are 
stripped.

The connection has been running for some time and A sends data.

A sends data with one DSS per segment for simplicity.

The middlebox strips the MPTCP option from all TCP segments.

B receives data in sequence at subflow level without DSS options. This 
data is acked at the subflow level. B includes a Data_ack that does not 
advance in each TCP ack, but this option is stripped by the middlebox.

Given that the DSS option can only map up to 64 KBytes of data, B can be 
sure that there is a problem once it has received 64 KBytes of 
in-sequence data without any DSS. This cannot happen on a valid MPTCP 
connection.

Host A could detect the problem earlier when it receives regular acks 
but no data_acks for the same data. To prevent false negatives, we'd 
probably need to wait for some number of mappings or some amount of data.

If there are two or more subflows, the problem is more complex since 
data is not in sequence on each subflow and it is normal to receive 
regular acks that are decoupled from the data_acks

Olivier