Subject: [multipathtcp] RFC6824bis edits based on implementation feedback

From: Alan Ford <[email-masked]>
Date: Wed, 1 Jan 2020 22:51:32 +0000

Hi all,

We’d love to get this to a state of completion as soon as possible, and to this end I am starting a new thread on this topic. In discussion with the chairs, it is possible to make the desired changes in AUTH48 as long as there is WG consensus. The discussion so far has been fairly limited in terms of participation.

I would ask the chairs please if it was possible to specify a time bound for this discussion and a default conclusion.

Regarding the changes, in summary, there are two areas where changes have been requested by the implementation community. As we are the IETF we obviously have strong focus on “running code” and so ease of implementing standards-compliant code is strongly desirable. However, we do not wish to reduce functionality agreed by the IETF community if it is considered a required feature by the community.

Change 1

Change the sentence reading:

   If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3). 

To:

   If B has data to send first, then the reliable delivery of the ACK + MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the MP_CAPABLE (which is the first octet of the data sequence space).

What this means:

The current text is concerned only with ensuring a path is MPTCP capable, and so only cares that DSS option occurs on a data packet. However, the MP_CAPABLE option is defined to occupy the first octet of data sequence space and thus, if analogous to TCP, must be acknowledged. From an implementation point of view it would make sense not to have this hanging around forever and instead define it is acknowledged at the connection level as soon as received. This change ensures the first data packet also DATA_ACKs this MP_CAPABLE octet.

Change 2

Change the sentence reading:

   A Data Sequence Mapping does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver.

To:

   The mapping provided by a Data Sequence Mapping MUST apply to some or all of the subflow sequence space in the TCP segment which carries the option. It does not need to be included in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping known at the receiver.

What this means:

The current text does not place any restrictions on where a mapping could appear. In theory a sender could define a thousand different mappings up front, send them all, and expect a receiver to store this and reassemble data according to these mappings as it arrives. Indeed, this was never explicitly disallowed since it “might have been useful”. The implementation community, however, has expressed concerns over the difficulty of implementing this open-endedly. How many mappings is it reasonable to store? Is there a DoS risk here? Instead, it has been requested that thee specification restricts the placement of the DSS option to being within the subflow sequence space to which it applies.

Please can members of the WG express whether they are happy with these changes, or concerned.

Best regards,
Alan

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 02 Jan 2020 10:26:55 -0800

+ MPTCP upstreaming community

Hello,

> Please can members of the WG express whether they are happy with these changes, or concerned.

As an implementor and having been the one kicking off this discussion, I fully
support this change.

Thanks,
Christoph

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 7 Jan 2020 00:41:38 -0800

Hi,
The texts look good to me. (as an individual contributor, just in case.)
Thanks for the efforts!
--
Yoshi

On Thu, Jan 2, 2020 at 10:27 AM Christoph Paasch <cpaasch=
[email-masked]> wrote:

> > We’d love to get this to a state of completion as soon as possible, and
> to this end I am starting a new thread on this topic. In discussion with
> the chairs, it is possible to make the desired changes in AUTH48 as long as
> there is WG consensus. The discussion so far has been fairly limited in
> terms of participation.
> >
> > I would ask the chairs please if it was possible to specify a time bound
> for this discussion and a default conclusion.
> >
> > Regarding the changes, in summary, there are two areas where changes
> have been requested by the implementation community. As we are the IETF we
> obviously have strong focus on “running code” and so ease of implementing
> standards-compliant code is strongly desirable. However, we do not wish to
> reduce functionality agreed by the IETF community if it is considered a
> required feature by the community.
> >
> >
> > Change 1
> >
> >    If B has data to send first, then the reliable delivery of the ACK +
> MP_CAPABLE can be inferred by the receipt of this data with an MPTCP Data
> Sequence Signal (DSS) option (Section 3.3).
> >
> > To:
> >
> >    If B has data to send first, then the reliable delivery of the ACK +
> MP_CAPABLE is ensured by the receipt of this data with an MPTCP Data
> Sequence Signal (DSS) option (Section 3.3) containing a DATA_ACK for the
> MP_CAPABLE (which is the first octet of the data sequence space).
> >
> > What this means:
> >
> > The current text is concerned only with ensuring a path is MPTCP
> capable, and so only cares that DSS option occurs on a data packet.
> However, the MP_CAPABLE option is defined to occupy the first octet of data
> sequence space and thus, if analogous to TCP, must be acknowledged. From an
> implementation point of view it would make sense not to have this hanging
> around forever and instead define it is acknowledged at the connection
> level as soon as received. This change ensures the first data packet also
> DATA_ACKs this MP_CAPABLE octet.
> >
> >
> > Change 2
> >
> >    A Data Sequence Mapping does not need to be included in every MPTCP
> packet, as long as the subflow sequence space in that packet is covered by
> a mapping known at the receiver.
> >
> > To:
> >
> >    The mapping provided by a Data Sequence Mapping MUST apply to some or
> all of the subflow sequence space in the TCP segment which carries the
> option. It does not need to be included in every MPTCP packet, as long as
> the subflow sequence space in that packet is covered by a mapping known at
> the receiver.
> >
> > What this means:
> >
> > The current text does not place any restrictions on where a mapping
> could appear. In theory a sender could define a thousand different mappings
> up front, send them all, and expect a receiver to store this and reassemble
> data according to these mappings as it arrives. Indeed, this was never
> explicitly disallowed since it “might have been useful”. The implementation
> community, however, has expressed concerns over the difficulty of
> implementing this open-endedly. How many mappings is it reasonable to
> store? Is there a DoS risk here? Instead, it has been requested that thee
> specification restricts the placement of the DSS option to being within the
> subflow sequence space to which it applies.
> >
> >
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: <[email-masked]>
Date: Wed, 8 Jan 2020 09:42:05 +0000

All - if you have any concerns, please send your comments as soon as possible. The deadline is 14th January. Given there's already been some discussion on the list, silence is considered as agreement.

Alan - thanks for summarising these changes and the reasons behind them.

Thanks,
Phil & Yoshi

-----Original Message-----
From: multipathtcp <[email-masked]> On Behalf Of Christoph Paasch
Sent: 02 January 2020 18:27
To: Alan Ford <[email-masked]>
Cc: MultiPath TCP - IETF WG <[email-masked]>; mptcp Upstreaming <[email-masked]>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hello,

> Please can members of the WG express whether they are happy with these changes, or concerned.

As an implementor and having been the one kicking off this discussion, I fully support this change.

Thanks,
Christoph

---

From: V Anil Kumar <[email-masked]>
Date: Mon, 13 Jan 2020 23:55:21 +0530 (IST)

Hi, 

I have some points related to the modifications (Change 2) being proposed on data sequence map. Please see them inline. Though I am putting forward the below points, if the consensus is in favour of the proposed change for reducing implementation complexity, I am also OK with that as well. 

From: "alan ford" <[email-masked]> 
To: [email-masked] 
Sent: Thursday, January 2, 2020 4:21:32 AM 
Subject: [multipathtcp] RFC6824bis edits based on implementation feedback 

Hi all, 

Change 1 

To: 

What this means: 

Change 2 

To: 

What this means: 

Below are my comments on this. I had shared some of these points in a previous thread that you had initiated in the same context. 

Transmitting large number of non-contiguous data sequence maps could be a misbehaviour (map-flooding), though it is not clear whether this can go to the extent of causing a potential DoS to the data receiver. So some sort of restriction on this could be useful. One approach could be to insist that the data sender should ensure that the map being transmitted is for in-window data, as per the receiver advertised window. A receiver should anyhow be willing to store the maps for in-window data to deal with packet loss. For example, when a window of data segments (say 1 to 64) is transmitted, each carrying its corresponding map, and segment-1 is lost, the maps for the remaining 63 need to be stored till the lost segment is retransmitted. Of course, in this case the maps will be stored at the receiver side along with their corresponding data. But the need to store multiple maps for in-window data would still be there. 

The problem with the proposed change (restriction) is that a data sender may find it difficult, in case a need arise to slightly delay the map delivery by few segments, i.e., sending some data first without map, and then send the corresponding map in a later segment, as shown below: 

subflow-1: segment-1 segment-3 segment-4 segment-7 

bytes:1-100 bytes:201-300 bytes:301-400 bytes:601-700 

no map map for 1-100 map for 201-400 map for 601-700 

subflow-2: segment-2 segment-5 segment-6 segment-8 

bytes: 101-200 bytes:401-500 bytes: 501-600 bytes:701-800 

map for 101-200 map for 401-600 no map map for 701-800 

In the above case, segment-1 goes without map and its map is included later in segment-3, the next data segment in the same subflow. Further, in the above scheduling pattern, the map in segment-3 cannot cover the data in segment-1 and segment-3, as some data in between (segment-2) is transmitted through another subflow. With the proposed change, the map in segment-3 will become invalid and this will eventually break subflow-1, though this could be a corner case. 

The question at this stage is why would segment-1 be transmitted without its map. In the case of bidirectional data transfer, there could be a need to pack both timestamp and SACK options in a data segment, i.e., piggybacking of SACK with data. If we consider that timestamp takes 12 bytes and SACK, even with single block, takes another 10 bytes, the remaining 18 bytes option space is not adequate to carry data sequence signal with map, especially when DSN is 64 bit long. So the delivery of either of the two (SACK or map) would be delayed. 

As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options) implies that SACK should not be delayed. It states "If sent at all, SACK options SHOULD be included in all ACKs which do not ACK the highest sequence number in the data receiver's queue". It also says "If data receiver generates SACK options under any circumstance, it SHOULD generate them under all permitted circumstances". So, as part of meeting the RFC 2018 requirements, if the combination of SACK and timestamp is given preference over DSS, data segments could be transmitted without their map. 

Another case of delaying map could arise if the data sender prefers to send ADD_ADDR option, instead of map, in a data segment. It is nice that ADD_ADDR option can be delivered reliably in a pure ACK, but I think this is not the case with DSS at present. 

If we adopt the proposed change, I think it might also be helpful to spell out how the receiver is supposed to behave, if it gets maps not meeting the MUST condition in the proposed change. For example termination of the subflow with MP_TCPRST option (section 3.6 in RFC 6824-bis) with appropriate reason code and T flag value to intimate the data sender the cause for subflow termination. 
With regards, 

Anil 

Best regards, 
Alan

---

From: <[email-masked]>
Date: Wed, 15 Jan 2020 09:12:06 +0000

+ including the MPTCP upstreaming community

Phil

Ps Anil, Thanks for your comments

From: multipathtcp <[email-masked]> On Behalf Of V Anil Kumar
Sent: 13 January 2020 18:25
To: alan ford <[email-masked]>
Cc: [email-masked]
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi,

________________________________
From: "alan ford" <[email-masked]<mailto:[email-masked]>>
To: [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 2, 2020 4:21:32 AM
Subject: [multipathtcp] RFC6824bis edits based on implementation feedback

Hi all,

Change 1

To:

The current text is concerned only with ensuring a path is MPTCP capable, and so only cares that DSS option occurs on a data packet. However, the MP_CAPABLE option is defined to occupy the first octet of data sequence space and thus, if analogous to TCP, must be acknowledged. >From an implementation point of view it would make sense not to have this hanging around forever and instead define it is acknowledged at the connection level as soon as received. This change ensures the first data packet also DATA_ACKs this MP_CAPABLE octet.

Change 2

To:

What this means:

With regards,

Anil

Best regards,
Alan

_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: <[email-masked]>
Date: Thu, 23 Jan 2020 10:51:11 +0000

I haven’t seen any responses about this. Please follow-up if you care about the issue.

Alan – please consider Change 1 as agreed.

Hi,

Hi all,

Change 1

To:

Change 2

To:

What this means:

With regards,

Anil

Best regards,
Alan

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 31 Jan 2020 14:09:51 -0800

Hi Anil,

I have a question about your proposed text.
I am actually wondering if we really want to terminate connection here.

The packets without proper mappings will be treated as invalid and will be
discarded.
If an implementation failed to attach proper mapping for some reasons (e.g.
option space), it might be able to attach the proper one when it
retransmits the packets. This also looks ok to me.

I don't have strong preference for this. But, do we have a reason to
terminate connection?

Thanks,
--
Yoshi

On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar <[email-masked]> wrote:

> Hi,
>
> I have some points related to the  modifications (Change 2) being proposed
> on data sequence map. Please see them inline. Though I am putting forward
> the below points, if the consensus is in favour of the proposed change for
> reducing implementation complexity, I am also OK with that as well.
>
> ------------------------------
> *From: *"alan ford" <[email-masked]>
> *To: [email-masked]
> *Sent: *Thursday, January 2, 2020 4:21:32 AM
> *Subject: *[multipathtcp] RFC6824bis edits based on implementation
> feedback
>
> Hi all,
>
> We’d love to get this to a state of completion as soon as possible, and to
> this end I am starting a new thread on this topic. In discussion with the
> chairs, it *is *possible to make the desired changes in AUTH48 as long as
> Regarding the changes, in summary, there are two areas where changes have
> been requested by the implementation community. As we are the IETF we
> *Change 1*
> *   If B has data to send first, then the reliable delivery of the ACK +
> Sequence Signal (DSS) option (Section 3.3). *
> To:
>
>
> *   If B has data to send first, then the reliable delivery of the ACK +
> MP_CAPABLE (which is the first octet of the data sequence space).*
> What this means:
>
> The current text is concerned only with ensuring a path is MPTCP capable,
> and so only cares that DSS option occurs on a data packet. However, the
> MP_CAPABLE option is defined to occupy the first octet of data sequence
> space and thus, if analogous to TCP, must be acknowledged. >From an
> *Change 2*
> *   A Data Sequence Mapping does not need to be included in every MPTCP
> a mapping known at the receiver.*
>
> To:
>
>
> *   The mapping provided by a Data Sequence Mapping MUST apply to some or
> the receiver.*
> The current text does not place any restrictions on where a mapping could
> appear. In theory a sender could define a thousand different mappings up
> front, send them all, and expect a receiver to store this and reassemble
> Below are my comments on this. I had shared some of these points in a
> previous thread that you had initiated in the same context.
>
> Transmitting large number of non-contiguous data sequence maps could be a
> misbehaviour (map-flooding), though it is not clear whether this can go to
> the extent of causing a potential DoS to the data receiver. So some sort of
> restriction on this could be useful.  One approach could be to insist that
> the data sender should ensure that the map being transmitted is for
> in-window data, as per the receiver advertised window. A receiver should
> anyhow be willing to store the maps for in-window data to deal with packet
> loss. For example, when a window of data segments (say 1 to 64) is
> transmitted, each carrying its corresponding map, and segment-1 is lost,
> the maps for the remaining 63 need to be stored till the lost segment is
> retransmitted. Of course, in this case the maps will be stored at the
> receiver side along with their corresponding data. But the need to store
> multiple maps for in-window data would still be there.
>
> The problem with the proposed change (restriction) is that a data sender
> may find it difficult, in case a need arise to slightly delay the map
> delivery by few segments, i.e., sending some data first without map, and
> then send the corresponding map in a later segment, as shown below:
>
>
> In the above case, segment-1 goes without map and its map is included
> later in segment-3, the next data segment in the same subflow. Further,  in
> the above scheduling pattern, the map in segment-3 cannot cover the  data
> in segment-1 and segment-3, as some  data in between (segment-2) is
> transmitted through another subflow.  With the proposed change, the map in
> segment-3 will become invalid and this will eventually break subflow-1,
> though this could be a corner case.
>
> The question at this stage is why would segment-1 be transmitted without
> its map. In the case of bidirectional data transfer, there could be a need
> to pack both timestamp and SACK  options in a data segment, i.e.,
> piggybacking of  SACK with data. If we consider that timestamp takes 12
> bytes and SACK, even with single block,  takes another 10 bytes, the
> remaining 18 bytes option space is not adequate to carry data sequence
> signal with map, especially when DSN is 64 bit long. So the delivery of
> either of the two (SACK or map) would be delayed.
>
> As far as I understand, RFC 2018 (TCP Selective Acknowledgement Options)
> implies that SACK should not be delayed. It states "If sent at all, SACK
> options SHOULD be included in all ACKs which do not ACK the highest
> sequence number in the data receiver's queue". It also says "If data
> receiver generates SACK options under any circumstance, it SHOULD generate
> them under all permitted circumstances".   So, as part of meeting the RFC
> 2018 requirements, if the combination of SACK and timestamp is given
> preference over DSS, data segments could be transmitted without their map.
>
> Another case of delaying map could arise if the data sender prefers to
> send ADD_ADDR option, instead of map, in a data segment. It is nice that
> ADD_ADDR option can be delivered reliably in a pure ACK, but I think this
> is not the case with DSS at present.
>
> If we adopt the proposed change, I think it might also be helpful to spell
> out how the receiver is supposed to behave, if it gets maps not meeting the
> MUST condition in the proposed change.  For example termination of the
> subflow with MP_TCPRST option (section 3.6 in RFC 6824-bis) with
> appropriate reason code and T flag value to intimate the data sender the
> cause for subflow termination.
>
> With regards,
>
> Anil
>