Subject: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 16 Jun 2011 10:03:07 -0500

All,

We have uploaded a new I-D.

Title: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Authors: Georg Hampel, Thierry Klein

Filename: draft-hampel-mptcp-applicability-wireless-networks-00.txt

URL: http://datatracker.ietf.org/doc/draft-hampel-mptcp-applicability-wireless-networks/

Abstract:

This document analyses the applicability of Multipath TCP to wireless access networks with overlapping coverage area, and it discusses potential protocol extensions that aim to improve operation in such environments.  The analysis attempts to identify use cases, benefits as well as technical and functional obstacles encountered in the current version of the protocol.  Based on this analysis, recommendations are made on feature-, signaling- and policy extensions that promise to enhance Multipath-TCP's value, versatility and market acceptance in wireless access networks.

IMPORTANT COMMENTS: Apart from a variety of recommendations for research, the I-D proposes the following signaling and policy enhancements:

MP_SELECT option:

The host sends this option to enforce path-selective operation on its peer.  The option is generally sent on the subflow to be used for data transmission. It may enclose an address id in case it is sent on another subflow, e.g. in break-before-make scenarios before the designated path becomes available. The option permits low-complexity MPTCP receiver solutions (Section 4.4) as well as dynamic overhead shedding for heavily loaded servers (Section 4.5).

AVAIL_ADDR/UNAVAIL_ADDR option(s):

The host sends this option to inform its peer about the availability/unavailability status of an interface referred to via an address id.  The enclosed address-id permits sending the option from an available interface to refer to an unavailable interface. The option permits the peer to swiftly adjust data transmission when the host's interface availability changes. There are multiple use cases for this option (Section 3.6, Section 4.4 and Section 4.5).

DEFER_ADDR option:

The host sends this option to instruct its peer to use a specific address referred to via an address id as the destination for all future subflows. This option is required for transparent-proxy operation (section 5).

ADD_ADDR option:

The present version of this option should be modified: The option should only provide a mapping between address id and address value without further advice or request for action. When the ADD_ADDRESS option refers to the source address of the packet it is enclosed in, it can omit the address value. The re-interpretation of this option is necessary to support multiple instructions as provided by the options above (and the one below).

JOIN_ADDR option:

The host sends this option to request subflow creation (via MP_JOIN) to an address referred to via the enclosed address id. Currently, the functionality of this option is melted into the ADD_ADDR option.

DSS insertion policy for bulk transfer:

To reduce receiver complexity, DSS options should be inserted into all packets of a bulk until the first data ACK is received for a packet contained in the bulk (Section 3.7).

Please take the details from the corresponding sections of the document. Comments are welcome. Thank you.

Regards,
Georg Hampel

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 21 Jun 2011 10:49:35 +0100

Hello Georg, Thierry,

Thanks for writing this - it covers an important and interesting topic.
I've had a chance to read it, now, and have some comments.

3.3: It is definitely intentional that you can use MPTCP with no changes
to send data resiliently on many paths. I'll expand the text in the
protocol draft slightly to discuss processing the same mappings on
different subflows.

3.4: Agreed, this (fluctuations in path throughput/delay) is a very
valid concern and one that we discuss considerably in the API draft.  It
is an issue where users are best placed to drive the development of
solutions.

3.7: This idea is sensible, but I would be reluctant to mandate it.  I
think some text like this may be sufficient: "An implementation SHOULD
send DSS options that cover multiple segments multiple times, to reduce
the need for unnecessary buffering at the receiver.  It is RECOMMENDED
that a sender repeats the DSS option until a DATA ACK for some data
covered by this mapping is received."

"Path Availability" - this is where I get confused as to what the goal
of this is. Can you give an example scenario where this would be useful?

Throughout the rest of this document, I struggle to see what difference
there is between path (or interface) availability, and address
availability.  We have the ADD_ADDR and REMOVE_ADDR options to indicate
the availability or loss of availability of an address (interface) on a
host.  What is (UN)AVAIL_ADDR giving over and above this? Why separate
the creation of the mapping and the signalling of the availability?

Some points on 4.4:

Yes, MP_PRIO is not binding, although the expectation is that a host
should have a good reason for ignoring it (such as if the path appears
dead to the host, or it costs too much money) - I can't see that we can
change that.  If an end host was desperate to stop receiving traffic on
an interface, it could drop the subflow.

If I understand your comment correctly, you'd like to be able to switch
a single path to B=0 and all the others to B=1 in one go, and this is
the purpose of MP_SELECT?  I guess we could adapt MP_PRIO to achieve
this, by having another value that means "use me and me alone".
Although I think I need more convincing that it adds value!

In 4.4.2, you talk about BBM in the scenario of a radio knowing that it
is going to take down its interface to attach to a new network.  You can
do this already cleanly - just FIN the old path (without a DATA FIN) and
bring up the new one.  What is your proposal adding?

5.1 is an interesting problem and one that Mark led some discussion of
at the last IETF.  As with the discussion about AVAIL_ADDR, I don't see
the point of JOIN_ADDR and ADD_ADDR separations, but DEFER_ADDR (or
"USE_THIS_ADDR!") is an interesting idea that I need to think about some
more!

Regards,

Alan

From: [email-masked]
[mailto:[email-masked]] On Behalf Of Hampel, K Georg (K
Georg)
Sent: 16 June 2011 16:03
To: [email-masked]
Cc: Klein, Thierry E (Thierry)
Subject: [multipathtcp] New I-D: Enhancements to Improve the
Applicability of Multipath TCP to Wireless Access Networks

All,

Title: Enhancements to Improve the Applicability of Multipath TCP to
Wireless Access Networks

URL:
http://datatracker.ietf.org/doc/draft-hampel-mptcp-applicability-wireles
s-networks/

Abstract:

This document analyses the applicability of Multipath TCP to wireless
access networks with overlapping coverage area, and it discusses
potential protocol extensions that aim to improve operation in such
environments.  The analysis attempts to identify use cases, benefits as
well as technical and functional obstacles encountered in the current
version of the protocol.  Based on this analysis, recommendations are
made on feature-, signaling- and policy extensions that promise to
enhance Multipath-TCP's value, versatility and market acceptance in
wireless access networks.

IMPORTANT COMMENTS: Apart from a variety of recommendations for
research, the I-D proposes the following signaling and policy
enhancements: 

MP_SELECT option:

The host sends this option to enforce path-selective operation on its
peer.  The option is generally sent on the subflow to be used for data
transmission. It may enclose an address id in case it is sent on another
subflow, e.g. in break-before-make scenarios before the designated path
becomes available. The option permits low-complexity MPTCP receiver
solutions (Section 4.4) as well as dynamic overhead shedding for heavily
loaded servers (Section 4.5).

The host sends this option to inform its peer about the
availability/unavailability status of an interface referred to via an
address id.  The enclosed address-id permits sending the option from an
available interface to refer to an unavailable interface. The option
permits the peer to swiftly adjust data transmission when the host's
interface availability changes. There are multiple use cases for this
option (Section 3.6, Section 4.4 and Section 4.5).

DEFER_ADDR option: 

The host sends this option to instruct its peer to use a specific
address referred to via an address id as the destination for all future
subflows. This option is required for transparent-proxy operation
(section 5).

ADD_ADDR option:

The present version of this option should be modified: The option should
only provide a mapping between address id and address value without
further advice or request for action. When the ADD_ADDRESS option refers
to the source address of the packet it is enclosed in, it can omit the
address value. The re-interpretation of this option is necessary to
support multiple instructions as provided by the options above (and the
one below).

JOIN_ADDR option: 

The host sends this option to request subflow creation (via MP_JOIN) to
an address referred to via the enclosed address id. Currently, the
functionality of this option is melted into the ADD_ADDR option.

To reduce receiver complexity, DSS options should be inserted into all
packets of a bulk until the first data ACK is received for a packet
contained in the bulk (Section 3.7).

Please take the details from the corresponding sections of the document.
Comments are welcome. Thank you.

Regards,

Georg Hampel

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 21 Jun 2011 13:14:04 +0200

Hi all,

I took also some time to read the draft and have some comments on your draft 
and on Alan's comments.

3.6:
"The ADD_ADDR option can be simplified when it refers to the packet's source 
address."
I don't see a case where the ADD_ADDR-option is sent, refering to the packet's 
source address, because the address-id of this source-address has already been 
provided by the MP_JOIN when establishing the subflow.

> covered by this mapping is received."

In fact, if the first packet (carrying the DSS) is lost, packets will be 
stored in the subflow-specific out-of-order queue, because we have a loss at 
the subflow-level.

In any case, as long as the whole mapping of the payload has not been 
received, we need to buffer the packets for the checksum-verification.

4.3:
"In case the old subflow becomes unavailable, retransmissions can..."

Maybe you should note, that the retransmissions have to be done on the new 
subflow before the regular data-stream. Otherwise you might have problems, in 
the next paragraph.

Concerning your removal of the DSS-options. Is it intended that after removing 
the DSS-options the host is still able to switch to another subflow?
If this is the case, then we have problems with middleboxes modifying the 
content of packets (e.g., IP-address rewriting). If they add/remove a byte, 
the data-sequence mapping between the two hosts isn't the same anymore.

4.3:
"Using only one flow/congestion engine significantly..."

So, the intention is to have the same congestion-window and slow-start-
threshhold on the new subflow as on the old-one, and thus you do not slow-
start on the new subflow?
I'm not sure if this is a good idea. You will probably overload the network in 
the beginning. It may also be, that statefull firewalls are dropping packets, 
because they do not fit in the expected window.

> the creation of the mapping and the signalling of the availability?

Agree with Alan on this point. I don't really see the difference between 
ADD/REMOVE and AVAIL/UNAVAIL.

4.4.2:
"Obviously, the host most have announced the mapping between address id and 
address value prior to the handover using the ADD_ADDRESS option."

Here, we are in the BBM-scenario. Thus, we do not yet have the new address. So 
we cannot announce it with ADD_ADDR, because we don't know which address we 
will get.

Cheers,
Christoph

--
Christoph Paasch
PhD Student

IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp
Université Catholique de Louvain

www.rollerbulls.be
--

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 21 Jun 2011 10:43:51 -0500

Hello Alan,

Thanks for reading the draft.  We are currently implementing a low-complexity MPTCP version for path-selective mode (see section 4 of draft). Many of the issues in section 4 came up when we thought through the specific steps for such a design. There are multiple ways to tackle these issues. I have added comments below.

Regards,
Georg

________________________________
From: Ford, Alan [mailto:[email-masked]]
Sent: Tuesday, June 21, 2011 5:50 AM
To: Hampel, K Georg (K Georg); [email-masked]
Cc: Klein, Thierry E (Thierry)
Subject: RE: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hello Georg, Thierry,

Thanks for writing this - it covers an important and interesting topic. I've had a chance to read it, now, and have some comments.

3.3: It is definitely intentional that you can use MPTCP with no changes to send data resiliently on many paths. I'll expand the text in the protocol draft slightly to discuss processing the same mappings on different subflows.

3.4: Agreed, this (fluctuations in path throughput/delay) is a very valid concern and one that we discuss considerably in the API draft.  It is an issue where users are best placed to drive the development of solutions.

3.7: This idea is sensible, but I would be reluctant to mandate it.  I think some text like this may be sufficient: "An implementation SHOULD send DSS options that cover multiple segments multiple times, to reduce the need for unnecessary buffering at the receiver.  It is RECOMMENDED that a sender repeats the DSS option until a DATA ACK for some data covered by this mapping is received."

"Path Availability" - this is where I get confused as to what the goal of this is. Can you give an example scenario where this would be useful?
Throughout the rest of this document, I struggle to see what difference there is between path (or interface) availability, and address availability.  We have the ADD_ADDR and REMOVE_ADDR options to indicate the availability or loss of availability of an address (interface) on a host.  What is (UN)AVAIL_ADDR giving over and above this? Why separate the creation of the mapping and the signalling of the availability?

[G.H.] The "Path availability" option shows up in sections 3.6 and 4.4.2. Here is an example that applies to section 3.6:

A multi-homed mobile downloads a data stream from a server using 2 interfaces, say,  Wifi & 4G. Suddenly, a lower-layer signal on the mobile indicates that the 4G interface has dropped off (coverage hole or fade). The server does not know about this signal and keeps pumping data along both paths. The data sent to the Wifi interface arrive but they cannot get delivered since all 4G-data get dropped. When RTO is exceeded for the 4G path, the server starts to realize that something is wrong and engages into retransmission, most likely on the same subflow first and after even more time, it engages into cross-subflow retransmission (dependent on local policy).
The whole mess could get resolved much faster if the mobile quickly informed the server about the sudden unavailability of its 4G interface. In this case, the server would be certain within 1/2 RTT that the "4G" interface is currently NOT good and it can immediately redirect all data including 4G-retransmissions to the "WiFi" path. The mobile should NOT tear down the "4G" path since it can be back up any moment. Further, MP_PRIO cannot be used since it cannot be sent on an unavailable path. Moreover, the MP_PRIO refers to individual paths, while the present problem applies to a specific interface, i.e. which may affect multiple paths.

Some points on 4.4:

Yes, MP_PRIO is not binding, although the expectation is that a host should have a good reason for ignoring it (such as if the path appears dead to the host, or it costs too much money) - I can't see that we can change that.  If an end host was desperate to stop receiving traffic on an interface, it could drop the subflow.

If I understand your comment correctly, you'd like to be able to switch a single path to B=0 and all the others to B=1 in one go, and this is the purpose of MP_SELECT?  I guess we could adapt MP_PRIO to achieve this, by having another value that means "use me and me alone".  Although I think I need more convincing that it adds value!

[G.H] I think you summarized the essence of MP_SELECT rather well. There are multiple ways of doing this. The goal of all this is that the host can ENFORCE path-selective operation on the peer's sender since this simplifies the host's receiver, AND that path-reselection can be done in the FASTEST and MOST RELIABLE possible way. Constraints are: (1) MP_PRIO with B=1 cannot be sent on a path that is already down, (2) uncertainty when multiple MP_PRIO message with different B-setting arrive out of sync due to the different RTTs on the various paths, (3) conflict resolution, i.e. host A wants receive data on path 1 but host B wants to send data on path 2, (4) long delay in case "switching messages" get lost.

In 4.4.2, you talk about BBM in the scenario of a radio knowing that it is going to take down its interface to attach to a new network.  You can do this already cleanly - just FIN the old path (without a DATA FIN) and bring up the new one.  What is your proposal adding?

[G.H.] We had an MPTCP mailing list discussion on BBM. From this discussion I understand that BBM on lower layer should NOT be followed on L4. The reasons given were: When a host with one radio interface rapidly ping-pongs between two networks (lower-layer BBM at every switch), MPTCP should keep both subflows up and solely change the activity of each path. In other words, BBM on lower layers is translated to MBB on MPTCP. Your proposal, i.e. sending FIN on the old path, would make a switchback to the old path very slow (since you have to MP_JOIN again). Further, the remote peer may not respond to FIN if it still has data to send (!). The BBM ping-pong scenario is another reason for AVAIL/UNAVAIL_ADDR.

5.1 is an interesting problem and one that Mark led some discussion of at the last IETF.  As with the discussion about AVAIL_ADDR, I don't see the point of JOIN_ADDR and ADD_ADDR separations, but DEFER_ADDR (or "USE_THIS_ADDR!") is an interesting idea that I need to think about some more!

[G.H.] I think Mark had a more generic proxy solution in mind. The "transparent" proxy we are proposing has much more stringent deployment requirements. However, there is a very important use case (put it on PDN-GW) and it can be realized through a very simple signalling enhancement ("USE_THIS_ADDR" may indeed be clearer).

Regards,
Alan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Hampel, K Georg (K Georg)
Sent: 16 June 2011 16:03
To: [email-masked]
Cc: Klein, Thierry E (Thierry)
Subject: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

All,

Abstract:

MP_SELECT option:

DEFER_ADDR option:

ADD_ADDR option:

JOIN_ADDR option:

Regards,
Georg Hampel

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 21 Jun 2011 10:57:33 -0500

Hello Christoph,

You are right! I have overseen this. Thanks.

Georg

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Tuesday, June 21, 2011 7:14 AM
To: [email-masked]
Cc: Ford, Alan; Hampel, K Georg (K Georg); Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi all,

3.6:

4.3:

4.3:

4.4.2:

Cheers,
Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 21 Jun 2011 17:34:21 +0100

Hi Georg, all,

To extract two key points from your mail below:

[G.H.] The "Path availability" option shows up in sections 3.6 and
4.4.2. Here is an example that applies to section 3.6:

A multi-homed mobile downloads a data stream from a server using 2
interfaces, say,  Wifi & 4G. Suddenly, a lower-layer signal on the
mobile indicates that the 4G interface has dropped off (coverage hole or
fade). The server does not know about this signal and keeps pumping data
along both paths. The data sent to the Wifi interface arrive but they
cannot get delivered since all 4G-data get dropped. When RTO is exceeded
for the 4G path, the server starts to realize that something is wrong
and engages into retransmission, most likely on the same subflow first
and after even more time, it engages into cross-subflow retransmission
(dependent on local policy). 

The whole mess could get resolved much faster if the mobile quickly
informed the server about the sudden unavailability of its 4G interface.
In this case, the server would be certain within 1/2 RTT that the "4G"
interface is currently NOT good and it can immediately redirect all data
including 4G-retransmissions to the "WiFi" path. The mobile should NOT
tear down the "4G" path since it can be back up any moment. Further,
MP_PRIO cannot be used since it cannot be sent on an unavailable path.
Moreover, the MP_PRIO refers to individual paths, while the present
problem applies to a specific interface, i.e. which may affect multiple
paths.

Whilst I understand the case you are describing here, I think you may be
underestimating the flexibility that could be implemented in the MPTCP
retransmission algorithm.  At the moment , we do not have any
significant retransmission recommendations since we have relatively
little real-world experience to base it on.  But an aggressive
retransmission strategy would cope with this scenario quite well.  For
example, if the Wifi continues well, there will soon be a point where
the buffer is rapidly filling at the receiver and the discrepancy
between the DATA_ACK and the subflow ACK is large enough to trigger a
retransmission on the functioning subflow. (This should probably not
exceed max(RTT), given the receive buffer sizing).

[G.H.] We had an MPTCP mailing list discussion on BBM. From this
discussion I understand that BBM on lower layer should NOT be followed
on L4. The reasons given were: When a host with one radio interface
rapidly ping-pongs between two networks (lower-layer BBM at every
switch), MPTCP should keep both subflows up and solely change the
activity of each path. In other words, BBM on lower layers is translated
to MBB on MPTCP. Your proposal, i.e. sending FIN on the old path, would
make a switchback to the old path very slow (since you have to MP_JOIN
again). Further, the remote peer may not respond to FIN if it still has
data to send (!). The BBM ping-pong scenario is another reason for
AVAIL/UNAVAIL_ADDR.

In this case, you are relying on the wireless network infrastructure
retaining state (i.e.  NATs) for the connection while the host uses
another network.  This is by no means guaranteed, and will require an
adjustment to the subflow-level state machines at each host, to
determine that a lost packet on an interface is not cause for
terminating a subflow when that same connection-level data has since
been successfully transmitted on another subflow (if this was applied,
however, the desired result could be achieved with a MP_PRIO signal
either downgrading the background subflow or promoting the foreground
one to be the "use me and me alone" subflow).

If we were to modify the MP_PRIO option to take an Address ID (which is
something we've thought about doing before but the use cases were not
clear), then you could downgrade one path from another. That may be
sufficient (at least, with tolerant flow control).  Shortcuts of a
second bit to say "Do not use" or "use exclusively" could also help
here.  Might this cover all cases?

Regards,

Alan

From: Hampel, K Georg (K Georg) [mailto:[email-masked]]

Sent: 21 June 2011 16:44
To: Ford, Alan; [email-masked]
Cc: Klein, Thierry E (Thierry)
Subject: RE: [multipathtcp] New I-D: Enhancements to Improve the
Applicability of Multipath TCP to Wireless Access Networks

Hello Alan,

Thanks for reading the draft.  We are currently implementing a
low-complexity MPTCP version for path-selective mode (see section 4 of
draft). Many of the issues in section 4 came up when we thought through
the specific steps for such a design. There are multiple ways to tackle
these issues. I have added comments below.

Regards,

Georg

Subject: RE: [multipathtcp] New I-D: Enhancements to Improve the

Hello Georg, Thierry,

[G.H.] The "Path availability" option shows up in sections 3.6 and
4.4.2. Here is an example that applies to section 3.6:

A multi-homed mobile downloads a data stream from a server using 2
interfaces, say,  Wifi & 4G. Suddenly, a lower-layer signal on the
mobile indicates that the 4G interface has dropped off (coverage hole or
fade). The server does not know about this signal and keeps pumping data
along both paths. The data sent to the Wifi interface arrive but they
cannot get delivered since all 4G-data get dropped. When RTO is exceeded
for the 4G path, the server starts to realize that something is wrong
and engages into retransmission, most likely on the same subflow first
and after even more time, it engages into cross-subflow retransmission
(dependent on local policy). 

The whole mess could get resolved much faster if the mobile quickly
informed the server about the sudden unavailability of its 4G interface.
In this case, the server would be certain within 1/2 RTT that the "4G"
interface is currently NOT good and it can immediately redirect all data
including 4G-retransmissions to the "WiFi" path. The mobile should NOT
tear down the "4G" path since it can be back up any moment. Further,
MP_PRIO cannot be used since it cannot be sent on an unavailable path.
Moreover, the MP_PRIO refers to individual paths, while the present
problem applies to a specific interface, i.e. which may affect multiple
paths.

Some points on 4.4:

[G.H] I think you summarized the essence of MP_SELECT rather well. There
are multiple ways of doing this. The goal of all this is that the host
can ENFORCE path-selective operation on the peer's sender since this
simplifies the host's receiver, AND that path-reselection can be done in
the FASTEST and MOST RELIABLE possible way. Constraints are: (1) MP_PRIO
with B=1 cannot be sent on a path that is already down, (2) uncertainty
when multiple MP_PRIO message with different B-setting arrive out of
sync due to the different RTTs on the various paths, (3) conflict
resolution, i.e. host A wants receive data on path 1 but host B wants to
send data on path 2, (4) long delay in case "switching messages" get
lost.

[G.H.] We had an MPTCP mailing list discussion on BBM. From this
discussion I understand that BBM on lower layer should NOT be followed
on L4. The reasons given were: When a host with one radio interface
rapidly ping-pongs between two networks (lower-layer BBM at every
switch), MPTCP should keep both subflows up and solely change the
activity of each path. In other words, BBM on lower layers is translated
to MBB on MPTCP. Your proposal, i.e. sending FIN on the old path, would
make a switchback to the old path very slow (since you have to MP_JOIN
again). Further, the remote peer may not respond to FIN if it still has
data to send (!). The BBM ping-pong scenario is another reason for
AVAIL/UNAVAIL_ADDR.

[G.H.] I think Mark had a more generic proxy solution in mind. The
"transparent" proxy we are proposing has much more stringent deployment
requirements. However, there is a very important use case (put it on
PDN-GW) and it can be realized through a very simple signalling
enhancement ("USE_THIS_ADDR" may indeed be clearer).

Regards,

Alan

All,

URL:

Abstract:

 

MP_SELECT option:

DEFER_ADDR option: 

ADD_ADDR option:

JOIN_ADDR option: 

 

Regards,

Georg Hampel

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jun 2011 09:38:58 -0700

Interesting draft, but this - like the original work - continues to 
confuse IP addresses with paths.

I raised this point many times during the original multipath discussion, 
but it continues to be lost at each successive addition to the work - 
partly because the work continues to me misnamed "multipathTCP", vs. 
multiaddressTCP

*When* (not if) wireless links are one hop upstream of a device (e.g., 
in 3G/4G WiFi hotspot boxes), information about an IP address tells you 
nothing about the availability of the **path**.

It'd be useful for this group, esp. those discussing this doc, to 
consider previous IETF BoF attempts in this area, e.g., trigtran from 
summer 2003 (unfortunately RFC5184 apparently didn't, since it doesn't 
address the above limitations unfortunately).

Joe

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 21 Jun 2011 14:11:24 -0500

Hi Alan, all,

On address availability:
I think we agree that this applies to situations where the host has knowledge from lower-layer signaling that one of its links is down or has insufficient "link quality". (This does not necessarily mean that the IP address is down.) Presently, the host has no means to inform its peer about this knowledge. If the MP_PRIO message could be equipped with an address-id, as you suggested, the host could forward this knowledge via another path pertaining to an interface that is up and has sufficient link quality. This would take 1/2RTT. It may be possible that an aggressive cross-subflow retransmission schedule would lead to similar performance. This, however, is currently unknown.

Enforcement of path-selective operation for host with simplified receiver: When the host wishes to hand over from one of its interfaces (old) to another (new), the host sends two MP_PRIO options, where one has B=1 and refers to the old interface (implicitly or via address id) and the other has B=0 and refers to the new path (implicitly or via address id). The host MAY send both options on the same path to ensure that they arrive at the same time. The host MAY enclose these options onto every subsequent packet until it receives the first packet on the new path. This allows the host to ensure fast and reliable delivery.

In case of conflicts, i.e. the peer does not like the new path, it can simply respond with its own MP_PRIO message-pair to choose another path. To avoid ping-ponging, the unhappy peer should pick a path that holds the host's new interface.

In summary: The address-id-equipped MP_PRIO option should do the job.

On BBM:
You are bringing up a good point. Given the uncertainty about middle-box state on a link that has been broken, the host is best advised to discontinue all subflows pertaining to this broken link. I sent a question on the ML some weeks ago: In case of an imminent BBM, should a host terminate all affected subflows via RST as a courtesy to clear middleboxes?

Thanks,

Regards,
Georg

Sent: Tuesday, June 21, 2011 12:34 PM

Hi Georg, all,

Whilst I understand the case you are describing here, I think you may be underestimating the flexibility that could be implemented in the MPTCP retransmission algorithm.  At the moment , we do not have any significant retransmission recommendations since we have relatively little real-world experience to base it on.  But an aggressive retransmission strategy would cope with this scenario quite well.  For example, if the Wifi continues well, there will soon be a point where the buffer is rapidly filling at the receiver and the discrepancy between the DATA_ACK and the subflow ACK is large enough to trigger a retransmission on the functioning subflow. (This should probably not exceed max(RTT), given the receive buffer sizing).

In this case, you are relying on the wireless network infrastructure retaining state (i.e.  NATs) for the connection while the host uses another network.  This is by no means guaranteed, and will require an adjustment to the subflow-level state machines at each host, to determine that a lost packet on an interface is not cause for terminating a subflow when that same connection-level data has since been successfully transmitted on another subflow (if this was applied, however, the desired result could be achieved with a MP_PRIO signal either downgrading the background subflow or promoting the foreground one to be the "use me and me alone" subflow).

If we were to modify the MP_PRIO option to take an Address ID (which is something we've thought about doing before but the use cases were not clear), then you could downgrade one path from another. That may be sufficient (at least, with tolerant flow control).  Shortcuts of a second bit to say "Do not use" or "use exclusively" could also help here.  Might this cover all cases?

Regards,
Alan

Hello Alan,

Regards,
Georg

Hello Georg, Thierry,

Some points on 4.4:

Regards,
Alan

All,

Abstract:

MP_SELECT option:

DEFER_ADDR option:

ADD_ADDR option:

JOIN_ADDR option:

Regards,
Georg Hampel

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 21 Jun 2011 14:21:42 -0500

Hi Joe, all,

Indeed, the AVAIL/UNAVAIL_ADDR options pursue similar intentions as proposed by trigtran (draft-dawkins-trigtran-framework-00, dawkins-trigtran-linkup-01).
Obviously, such messages can only refer to local-link information and not to next-hop links. This may not be perfect but pretty good in many use cases (as is also stated in the trigtran drafts). 

I further think that referring to addresses rather than paths is correct in the context of AVAIL/UNAVAIL_ADDR.

If you see any inconsistencies and flaws in the present draft, please let us know. Thanks.

Georg

-----Original Message-----
From: Joe Touch [mailto:[email-masked]] 
Sent: Tuesday, June 21, 2011 12:39 PM
To: Ford, Alan
Cc: Hampel, K Georg (K Georg); [email-masked]; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jun 2011 13:11:54 -0700

> If you see any inconsistencies and flaws in the present draft, please let us know. Thanks.

FWUW, I think the key would be to focus on the fact that these are 
addresses, and include a brief discussion on the limitations of the 
approach - i.e., it works for only directly attached interfaces.

Joe

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 22 Jun 2011 09:31:12 -0500

Hi Christoph, all,

I missed Christoph's comments on Alan's comments in my last reply on Christoph's comments. There is a lot of discussion material. 

I have inserted my comments to Christoph's comments below (marked with [GH]). Thanks.

Regards,

Georg

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Tuesday, June 21, 2011 7:14 AM
To: [email-masked]
Cc: Ford, Alan; Hampel, K Georg (K Georg); Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi all,

3.6:

[GH] Our intention is to avoid this out-of-order queue if possible.

[GH] I was under the impression that the checksum applies to the packet only. In my opinion this would make things much simpler. Is there a specific reason to apply the checksum to the entire bulk rather than solely the packet? Things would get much simpler if the checksum were packet-specific.

4.3:

[GH] One can support 2 distinct scenarios: In one, no cross-subflow retransmissions occur since the old subflow is still in good shape. In the other, all unacknowledged data on the old subflow are assumed lost and retransmitted on the new subflow in one bulk. This would make things substantially simpler.

[GH] Yes, we want to keep switching. We were thinking of using an infinite mapping between re-selection events. In other words: The hosts operate in a "fall-back mode" in between re-selection events. I have copied/pasted the corresponding text from draft-ietf-mptcp-multiaddressed-03:

[draft-ietf-mptcp-multiaddressed-03.txt: section 3.3.1]
...An "infinite" mapping can be used to fallback to
   regular TCP by mapping the subflow-level data to the connection-level
   data for the remainder of the connection (see Section 3.5).  This is
   achieved by setting the data-level length field to the reserved value
   of 0. ...

[draft-ietf-mptcp-multiaddressed-03.txt: section 3.5]
... This connection will then continue to appear as a regular TCP
   session, and a middlebox may change the payload without causing
   unintentional harm....
...When a connection is in fallback mode, only one subflow can send data
   at a time.  Otherwise, the receiver would not know how to reorder the
   data.  However, subflows can be opened and closed as necessary, as
   long as a single one is active at any point.

[GH] Quite honestly, we haven't really considered payload-rewriting middleboxes. I don't know how frequent they are and I don't know how well applications work with them. I further don't understand how the "TCP-fallback-with-consecutive-subflow-switch" proposed in the above draft-ietf-mptcp-multiaddressed-03 reference addresses this issue. I could imagine a solution, where the host "exits" TCP-fallback mode (i.e. re-enters MPTCP mode) by inserting a re-normalizing DSS, i.e. a new mapping based on the existing SN (rather than vice versa). This would work as long as payload-rewriting does not happen during the re-selection event.

4.3:

[GH] We have done tests on this. The new path may receive a brief burst which the network may not be able to handle. This burst takes at most RTO. Then the host goes into slow start. Note that 3GPP's WLAN-interoperability solutions provide the same functionality with one flow engine. The same applies to the Mobile IP family.
In turn, I would like to see real measurements on how much better the full-fledge MPTCP host performs in this case.
Don't worry about firewalls: The simplified host has to ensure each subflow's integrity, i.e. that all SNs fit in expected window etc.

[GH] ADD/REMOVE_ADDR create/delete subflows. AVAIL/UNAVAIL_ADDR indicate throughput conditions of addresses pertaining to existing subflows. No subflow-teardown/setup necessary. However, the remote host may quickly adapt the subflow subset it uses for data transmission.

4.4.2:

[GH] The BBM situation needs more discussion: Does lower-layer BBM imply subflow BBM? If not, it is possible that moving back to an old network provides you with the same IP address.

Cheers,
Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 22 Jun 2011 17:15:47 +0200

Hi Georg,

thanks for your reply.

In another mail you said, that you are implementing the path-selective version 
of MPTCP. Do you implement it on top of our Linux Kernel implementation of 
MPTCP?
Is the source-code available in a public git?

Some comments inline.

On Wednesday 22 June 2011 wrote Hampel, K Georg (K Georg):
> stored in the subflow-specific out-of-order queue, because we have a loss
> at the subflow-level.
> 

From an implementation point-of-view, I don't really see how to avoid this 
out-of-order queue. We still need to be sure that the TCP subflow behaves as  
regular TCP. Thus, if some packets of a bulk of packets get lost, we need to 
store the out-of-order packets (I mean out-of-order at a subflow-level) in a 
subflow out-of-order queue to be sure that the ack's we will send back to the 
sender are correct (e.g., holes in the out-of-order queue need to be filled up 
correctly).
Of course, if these out-of-order packets cover already a full DSS-mapping, we 
may optimize the implementation so that it already pushes the payload to the 
data-level. But still, the packet needs to stay in the subflow out-of-order-
queue.

> [GH] I was under the impression that the checksum applies to the packet
> only. In my opinion this would make things much simpler. Is there a
> specific reason to apply the checksum to the entire bulk rather than
> solely the packet? Things would get much simpler if the checksum were
> packet-specific.

If the DSS-mapping covers multiple packets, the checksum in the DSS-option is 
on the whole payload of these packets plus the pseudo-header.

> Concerning your removal of the DSS-options. Is it intended that after
> removing the DSS-options the host is still able to switch to another
> subflow? If this is the case, then we have problems with middleboxes
> modifying the content of packets (e.g., IP-address rewriting). If they
> add/remove a byte, the data-sequence mapping between the two hosts isn't
> the same anymore.
> 
> [GH] Yes, we want to keep switching. We were thinking of using an infinite
> mapping between re-selection events. In other words: The hosts operate in
> a "fall-back mode" in between re-selection events. I have copied/pasted
> the corresponding text from draft-ietf-mptcp-multiaddressed-03:

Ok, assuming there are no content-changing middleboxes, you can indead skip 
from sending DSS's by doing like in the draft-ietf-mptcp-multiaddressed-03. 
But you then do not need to wait for the DATA_ACK's as described in your 
draft, as the infinite mapping is "announced" with data_len == 0.

> [GH] Quite honestly, we haven't really considered payload-rewriting
> middleboxes. I don't know how frequent they are and I don't know how well
> applications work with them. I further don't understand how the
> "TCP-fallback-with-consecutive-subflow-switch" proposed in the above
> draft-ietf-mptcp-multiaddressed-03 reference addresses this issue. I could
> imagine a solution, where the host "exits" TCP-fallback mode (i.e.
> re-enters MPTCP mode) by inserting a re-normalizing DSS, i.e. a new
> mapping based on the existing SN (rather than vice versa). This would work
> as long as payload-rewriting does not happen during the re-selection
> event.

True, I'm also asking myself how "TCP-fallback-with-consecutive-subflow-
switch" of draft-ietf-mptcp-multiaddressed-03 can work.
The re-normalizing DSS should be ok without these middleboxes.

Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 22 Jun 2011 12:25:27 -0500

Hi Christoph,

It is a fresh implementation. We are trying to put the first version on netfilter_queue which allows doing packet processing in userspace. (Operation w/o out-of-order queue is very important in this case.)

The source code will be available when done. By then, we would also like to test interoperability with other MPTCP implementations.

Comments inline. Thanks.

Regards,
Georg 

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Wednesday, June 22, 2011 11:16 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ford, Alan; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi Georg,

thanks for your reply.

Some comments inline.

[GH2] Here's the idea: bulk-size is either infinity or packet size. If infinity, the mapping is always the same until explicitly changed through a new DSS. The new mapping (i.e. DSS) must be inserted on all consecutive packets until the sender receives the first data ACK confirming the latest mapping change (this motivates section 3.7. in our draft). If bulk = packet size, things are straightforward since they only apply to the present packet. The consecutive packets are expected to carry a DSS until end-to-end synch is achieved again.

[GH2] And why not simply put one DSS in each packet of the bulk? Then the checksum only has to cover the packet and all out-of-order queuing is eliminated! (Am I missing something here?)

[GH2] That's the plan. The advantage is that path-selective MPTCP becomes conventional TCP in between re-selection events.

[GH2] The re-normalizing DSS should actually work WITH these payload-changing middleboxes. Example:
Mapping on active subflow 1, established via a DSS: 
	Sender: DSN = 1000, SN1 = 100, Receiver: SN1' = 100 => DSN = 1000 
200 bytes later, middlebox adds 8 bytes. The mapping would be:
	Sender: DSN = 1200, SN1 = 300, Receiver: SN1' = 308 => DSN = 1208.
The hosts do not notice the mismatch since DSSs are not sent.
At DSN = 1400, the sender re-normalizes the mapping by sending a DSS:
	Sender: DSN 1400, SN1 = 400, Receiver: SN1'= 408 => DSN = 1400
The receiver may notice that it somehow received a few bytes more but it does not care. In fact, it does not even need to keep the notion of a DSN between DSS transmissions.

Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 24 Jun 2011 12:02:07 +0200

Hi,

On Wednesday 22 June 2011 wrote Hampel, K Georg (K Georg):
> The source code will be available when done. By then, we would also like to
> test interoperability with other MPTCP implementations.

You can get our kernel-implementation from http://inl.info.ucl.ac.be/mptcp/ if 
you want to test interoperability. (however we do not yet support the hmac-
based security-mechanism)

I'm looking forward to have a look at your code.

Comments inline.

> [GH2] Here's the idea: bulk-size is either infinity or packet size. If
> infinity, the mapping is always the same until explicitly changed through
> a new DSS. The new mapping (i.e. DSS) must be inserted on all consecutive
> packets until the sender receives the first data ACK confirming the latest
> mapping change (this motivates section 3.7. in our draft). If bulk =
> packet size, things are straightforward since they only apply to the
> present packet. The consecutive packets are expected to carry a DSS until
> end-to-end synch is achieved again.

> If the DSS-mapping covers multiple packets, the checksum in the DSS-option
> is on the whole payload of these packets plus the pseudo-header.
> 
> [GH2] And why not simply put one DSS in each packet of the bulk? Then the
> checksum only has to cover the packet and all out-of-order queuing is
> eliminated! (Am I missing something here?)

Yes, if the DSS-mapping is per-packet you do not need to buffer the packets 
for DSS-checksum verification.

However, concerning the subflow-out-of-order queue. Consider the following 
packet-sequence arriving with subflow-sequence-numbers:

9 - 8 - 6 - 5 - 3 - 2

Packets 1, 4 and 7 are lost.
We need to store all the packets in the subflow ofo-queue. Duplicate acks 
trigger a fast-retransmit and packet 1 will be retransmitted.
At reception of packet 1, we can push 1, 2 and 3 to the data-receive-queue and 
thus these packets are available to the application. The subflow-acks will 
have sequence-number 4. Again, the dup-acks will trigger a fast-retransmit of 
packet 4 (assuming SACK is disabled). And so on...

How can you handle the gaps in the subflow-sequence space and send correct 
subflow-acks without a subflow ofo-queue? (even if the DSS-mapping covers only 
single packets)

> [GH2] The re-normalizing DSS should actually work WITH these
> payload-changing middleboxes. Example: Mapping on active subflow 1,
> established via a DSS:
> The receiver may notice that it somehow received a few bytes more but it
> does not care. In fact, it does not even need to keep the notion of a DSN
> between DSS transmissions.

Yes, that should be ok. The DSS should maybe be marked as "re-normalizer" to 
make the renormalisation explicit.
The old subflow however needs to be correctly closed with a FIN, because 
otherwise both hosts may be unsure about where they are in the sequence-space.
Imagine, if the old subflow breaks, and the sender received the subflow-acks 
until 300 but the receiver got subflow-data until 408.
Then, sending the re-normalizing DSS on the new-subflow will result in a 
corrupted data-stream at the receiver, because he will append the 
retransmitted data which goes from 1300 to 1400 (of the old subflow - because 
only subflow-ack 300 has been received by the sender) to the received 1408.

Maybe, re-normalisation can be achieved by changing the subflow-sequence 
number inside the re-normalizing DSS-option to be the one of the old subflow?

But this change of the DSS-option and may be not a so good idea. We will have 
problems, if this renormalizing DSS-packet get's split by a middlebox or TSO. 
As the subflow-sequence number in the DSS-option is the one of the old 
subflow, we will overwrite the data-stream.

Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Fri, 24 Jun 2011 09:15:07 -0500

Hi Cristoph, all,

Thanks for the link to the MPTCP code. We're skipping HMACs as well.

Further comments inserted below.

Regards,
Georg

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Friday, June 24, 2011 6:02 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ford, Alan; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi,

[GH3] We also look forward to having a look at our (finished) code... 

Comments inline.

9 - 8 - 6 - 5 - 3 - 2

[GH3] We use one TCP control block which has an ofo queue and handles all reordering and retransmissions, etc. Since it operates in the DSN space, we only have to swap out DSN<=>SN and vv underneath. This can be done sequentially (i.e. packet by packet) as long as we have the corresponding mapping. The only thing we have to cache is this mapping information. Without cross-flow retransmissions that's only one integer. With cross-flow retransmission it can become a little longer. 

[GH3] The whole payload-adding-middlebox is a mess. Any solution you propose only works if the middlebox behaves "well" (i.e. does not add packets) during subflow switching or mapping re-normalization.
Buy the way: I thought that the mapping re-normalizer is sent on the old subflow just before switching. This creates a trustworthy global DSN which can be used to synchronize both subflows during switching. After switching, the mapping can be "forgotten" until the next switching event approaches. You don't have to close the old subflow. You're right in that this approach only works in clean MBB scenarios. It fails if the old subflow breaks before you have sent the re-normalizer.

Christoph

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: <[email-masked]>
Date: Tue, 28 Jun 2011 08:46:58 +0100

Hi,
I summarise where I think the discussion (triggered by the email below) has got to. This is just to test whether we have consensus on how to change the protocol draft, so please kick.
Firstly, the MP-PRIO message is altered to include Address ID - and (I think) a second flag meaning 'use only this subflow'  (not sure discussion finalised on this point)
Secondly, the other new msgs (see below) suggested by this draft are not needed
Thirdly, add some text something like: "An implementation SHOULD send DSS options that cover multiple segments multiple times, to reduce the need for unnecessary buffering at the receiver.  It is RECOMMENDED that a sender repeats the DSS option until a DATA ACK for some data covered by this mapping is received."
Also, there may be some extra /amended discussion about content-altering middleboxes (I couldn't tell whether the discussion concluded on this point)
Sorry where i misunderstood
Thanks
Phil/

All,

Abstract:

MP_SELECT option:

DEFER_ADDR option:

ADD_ADDR option:

JOIN_ADDR option:

Regards,
Georg Hampel

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 28 Jun 2011 15:56:15 -0500

Hi Phil, all,

Below is a summary on the present level of agreement. This summary is not complete but I hope I captured the essential comments. There seems to be convergence on some points, others are still open and there are contradicting opinions on BBM. It may make sense to get as much agreement as possible via ML prior to the meeting. Thanks.

Regards,
Georg

MP_SELECT:
-          Alan Ford: 'I guess we could adapt MP_PRIO to achieve this, by having another value that means "use me and me alone".  Although I think I need more convincing that it adds value!"
-          Georg Hampel: This would do the job. The "use me and me alone" MUST be binding though.

AVAIL/UNAVAIL_ADDRESS:
-          Alan Ford: 'If we were to modify the MP_PRIO option to take an Address ID (which is something we've thought about doing before but the use cases were not clear), then you could downgrade one path from another. That may be sufficient (at least, with tolerant flow control).  Shortcuts of a second bit to say "Do not use" or "use exclusively" could also help here.'
-          Georg Hampel:  MP_PRIO with Address ID changes the semantic meaning since it refers to an address rather than a path. I'm fine with this as long as everybody is aware of it. We may have to file out the details.
-          Joe Touch: "I think the key would be to focus on the fact that these are addresses, and include a brief discussion on the limitations of the approach - i.e., it works for only directly attached interfaces". Joe further recommended to cite trigtran work in this context.

DEFER_ADDR:
-          Alan Ford: I don't see the point of JOIN_ADDR and ADD_ADDR separations, but DEFER_ADDR (or "USE_THIS_ADDR!") is an interesting idea that I need to think about some more!
-          Georg Hampel: USE_THIS_ADDR instead of DEFER_ADDR is fine with me and may even be clearer.

ADD_ADDRESS:
-          Christoph Paasch: "I don't see a case where the ADD_ADDR-option is sent, refering to the packet's source address, because the address-id of this source-address has already been provided by the MP_JOIN when establishing the subflow.
-          Georg Hampel: That's right I've overseen this.

JOIN_ADDRESS to be separated from ADD_ADDR:
-          Georg Hampel: We still need to separate these two functions to support DEFER_ADDR (or USE_THIS_ADDR) unless the latter options include the address value explicitly.

Break before make: NOT CLARIFIED!!!
-          Georg Hampel: If host undergoes lower-layer BBM, it should send RST on the old interface to close state on middleboxes.
-          Yoshifumi Nishida: "I think L4 should be conservative to react on L2/L3 incidents, so I'm a bit skeptical about sending RST here."
-          Olivier Bonaventure: "In the case that you mentioned, the host should not send a RST when it looses connectivity. Instead, it should leave the MPTCP session up, switch to the new address and send an MP_JOIN. After that, it can use REMOVE_ADDR to indicate that the old address does not work anymore (section 3.4).
-          According to Christoph Paasch lower-layer BBM should lead to subflow discontinuation since it is not certain that middleboxes will keep state in case the interface is re-connected.
-          Georg Hampel: Christoph's statement means that the host must send RST since it cannot count on its peer to react to FIN.
-          to be continued...

DSS insertion policy for bulk transfer: NEEDS MORE DISCUSSION
-          Alan Ford: "An implementation SHOULD send DSS options that cover multiple segments multiple times, to reduce the need for unnecessary buffering at the receiver.  It is RECOMMENDED that a sender repeats the DSS option until a DATA ACK for some data covered by this mapping is received."
-          Georg Hampel: We need this to avoid an additional out-of-order queue for path-selective operation.
-          Christoph Paasch had longer comments on this...

________________________________
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, June 28, 2011 3:47 AM
To: Hampel, K Georg (K Georg); [email-masked]
Cc: Klein, Thierry E (Thierry)
Subject: RE: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi,

All,

Abstract:

MP_SELECT option:

DEFER_ADDR option:

ADD_ADDR option:

JOIN_ADDR option:

Regards,
Georg Hampel

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 29 Jun 2011 09:34:10 +0200

Alan,

> think some text like this may be sufficient: “An implementation SHOULD
> covered by this mapping is received.”

I would suggest to wait until we get more experience with the MPTCP
implementations and their behaviour in the real Internet before
discussing optimisations like this one. A reasonable solution based on
our understanding of the operations of MPTCP today is to put a DSS
option inside each segment. I would suggest to put text that recommends
experiments with lower frequency transmission of the DSS option but
would wait for the results of these experiments before mandating
anything in the spec. There are subtle interactions between DSS and the
TCP segment offload capable itnerfaces that need to be analysed before
mandating anything on this

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 29 Jun 2011 03:09:35 -0700

Hi Georg,

Thanks for the summary. Could you elaborate the binding in MP_SELECT?
I'm wondering how a peer has a strong control to the other peer.
If the other peer disagree with it, what will happen?

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/6/28 Hampel, K Georg (K Georg) <[email-masked]>:
> Hi Phil, all,
>
>
>
> Below is a summary on the present level of agreement. This summary is not
> complete but I hope I captured the essential comments. There seems to be
> convergence on some points, others are still open and there are
> contradicting opinions on BBM. It may make sense to get as much agreement as
> possible via ML prior to the meeting. Thanks.
>
>
>
> Regards,
>
> Georg
>
>
>
>
>
> MP_SELECT:
>
> -          Alan Ford: ‘I guess we could adapt MP_PRIO to achieve this, by
> having another value that means “use me and me alone”.  Although I think I
> need more convincing that it adds value!”
>
> -          Georg Hampel: This would do the job. The “use me and me alone”
> MUST be binding though.
>
>
>
> AVAIL/UNAVAIL_ADDRESS:
>
> -          Alan Ford: ‘If we were to modify the MP_PRIO option to take an
> Address ID (which is something we’ve thought about doing before but the use
> cases were not clear), then you could downgrade one path from another. That
> may be sufficient (at least, with tolerant flow control).  Shortcuts of a
> second bit to say “Do not use” or “use exclusively” could also help here.’
>
> -          Georg Hampel:  MP_PRIO with Address ID changes the semantic
> meaning since it refers to an address rather than a path. I’m fine with this
> as long as everybody is aware of it. We may have to file out the details.
>
> -          Joe Touch: “I think the key would be to focus on the fact that
> these are addresses, and include a brief discussion on the limitations of
> the approach - i.e., it works for only directly attached interfaces”. Joe
> further recommended to cite trigtran work in this context.
>
>
>
> DEFER_ADDR:
>
> -          Alan Ford: I don’t see the point of JOIN_ADDR and ADD_ADDR
> separations, but DEFER_ADDR (or “USE_THIS_ADDR!”) is an interesting idea
> that I need to think about some more!
>
> -          Christoph Paasch: “I don't see a case where the ADD_ADDR-option
> is sent, refering to the packet's source address, because the address-id of
> this source-address has already been provided by the MP_JOIN when
> establishing the subflow.
>
> -          Georg Hampel: That’s right I’ve overseen this.
>
>
>
> -          Georg Hampel: We still need to separate these two functions to
> support DEFER_ADDR (or USE_THIS_ADDR) unless the latter options include the
> address value explicitly.
>
>
>
> -          Georg Hampel: If host undergoes lower-layer BBM, it should send
> RST on the old interface to close state on middleboxes.
>
> -          Yoshifumi Nishida: “I think L4 should be conservative to react on
> L2/L3 incidents, so I'm a bit skeptical about sending RST here.”
>
> -          Olivier Bonaventure: “In the case that you mentioned, the host
> should not send a RST when it looses connectivity. Instead, it should leave
> the MPTCP session up, switch to the new address and send an MP_JOIN. After
> that, it can use REMOVE_ADDR to indicate that the old address does not work
> anymore (section 3.4).
>
> -          According to Christoph Paasch lower-layer BBM should lead to
> subflow discontinuation since it is not certain that middleboxes will keep
> state in case the interface is re-connected.
>
> -          Georg Hampel: Christoph’s statement means that the host must send
> RST since it cannot count on its peer to react to FIN.
>
> -          to be continued…
>
>
>
> -          Alan Ford: “An implementation SHOULD send DSS options that cover
> multiple segments multiple times, to reduce the need for unnecessary
> buffering at the receiver.  It is RECOMMENDED that a sender repeats the DSS
> option until a DATA ACK for some data covered by this mapping is received.”
>
> -          Christoph Paasch had longer comments on this…
>
>
>
>
>
> I summarise where I think the discussion (triggered by the email below) has
> got to. This is just to test whether we have consensus on how to change the
> protocol draft, so please kick.
>
> Firstly, the MP-PRIO message is altered to include Address ID – and (I
> think) a second flag meaning ‘use only this subflow’  (not sure discussion
> finalised on this point)
>
> Thirdly, add some text something like: “An implementation SHOULD send DSS
> options that cover multiple segments multiple times, to reduce the need for
> unnecessary buffering at the receiver.  It is RECOMMENDED that a sender
> repeats the DSS option until a DATA ACK for some data covered by this
> mapping is received.”
>
> Also, there may be some extra /amended discussion about content-altering
> middleboxes (I couldn’t tell whether the discussion concluded on this point)
>
> Subject: [multipathtcp] New I-D: Enhancements to Improve the Applicability
> of Multipath TCP to Wireless Access Networks
>
>
>
> All,
>
>
>
> http://datatracker.ietf.org/doc/draft-hampel-mptcp-applicability-wireless-networks/
>
>
>
> Abstract:
>
> This document analyses the applicability of Multipath TCP to wireless access
> networks with overlapping coverage area, and it discusses potential protocol
> extensions that aim to improve operation in such environments.  The analysis
> attempts to identify use cases, benefits as well as technical and functional
> obstacles encountered in the current version of the protocol.  Based on this
> analysis, recommendations are made on feature-, signaling- and policy
> extensions that promise to enhance Multipath-TCP's value, versatility and
> market acceptance in wireless access networks.
>
>
>
>
>
> IMPORTANT COMMENTS: Apart from a variety of recommendations for research,
> the I-D proposes the following signaling and policy enhancements:
>
>
>
> MP_SELECT option:
>
> The host sends this option to enforce path-selective operation on its peer.
> The option is generally sent on the subflow to be used for data
> available interface to refer to an unavailable interface. The option permits
> the peer to swiftly adjust data transmission when the host’s interface
> availability changes. There are multiple use cases for this option (Section
> 3.6, Section 4.4 and Section 4.5).
>
>
>
> DEFER_ADDR option:
>
> The host sends this option to instruct its peer to use a specific address
> referred to via an address id as the destination for all future subflows.
> This option is required for transparent-proxy operation (section 5).
>
>
>
> ADD_ADDR option:
>
> only provide a mapping between address id and address value without further
> advice or request for action. When the ADD_ADDRESS option refers to the
> source address of the packet it is enclosed in, it can omit the address
> value. The re-interpretation of this option is necessary to support multiple
> instructions as provided by the options above (and the one below).
>
>
>
> JOIN_ADDR option:
>
> The host sends this option to request subflow creation (via MP_JOIN) to an
> address referred to via the enclosed address id. Currently, the
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>
>

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 29 Jun 2011 10:05:43 -0400

Going further, I would argue that there is no point to make this path preference binding for the other endhost.  I think the other host SHOULD use this information if possible. If this is not possible, and the receving  host feels so strong about path preference, it can always drop the subflow, or send congestion marks. 

Re: BBM. I don't see it as contentious - as Olivier and Yoshifumi mention, the host should not reset subflows when the interface dies ; it should let the transport stack time out the corresponding subflow.
Re: Christoph's note about middelboxes timing out state: that is also true for silent flow (e.g. an ssh session not use for a few minutes). It is orthogonal to BBM, and if it happens the flow will crash miserably and does today's SSH, and that;'s that.

Cheers,
Cosin

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 29 Jun 2011 09:36:12 -0500

Hi Yoshifumi, all

MP_SELECT combines two functions: 1) enforce path-selective operation on the remote host and 2) recommend a specific path the remote host should use for data transmissions. 

The first function MUST be binding. It allows the host to reduce its receive buffer size since it can count on data to arrive on only one path (outside path-reselection windows at least).

The second function represents a path recommendation. The remote host may disagree with this recommendation. The draft discusses a conflict-resolution scheme in this case:
 
- If the remote host is not happy with the path proposed by MP_SELECT, it can respond with its own MP_SELECT message proposing a new path.

- To avoid ping-pong in conflict cases, every host can choose the local interface and impose this selection to the peer. (The local interface is generally the source address of the MP_SELECT message unless an explicit ADDRESS-ID is included as suggested by Alan). This policy resolves conflict within one MP_SELECT exchange.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Wednesday, June 29, 2011 6:10 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; [email-masked]; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi Georg,

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/6/28 Hampel, K Georg (K Georg) <[email-masked]>:
> -          Alan Ford: 'I guess we could adapt MP_PRIO to achieve this, by
> having another value that means "use me and me alone".  Although I think I
> need more convincing that it adds value!"
>
> -          Alan Ford: 'If we were to modify the MP_PRIO option to take an
> Address ID (which is something we've thought about doing before but the use
> second bit to say "Do not use" or "use exclusively" could also help here.'
>
> meaning since it refers to an address rather than a path. I'm fine with this
> -          Joe Touch: "I think the key would be to focus on the fact that
> the approach - i.e., it works for only directly attached interfaces". Joe
> -          Alan Ford: I don't see the point of JOIN_ADDR and ADD_ADDR
> separations, but DEFER_ADDR (or "USE_THIS_ADDR!") is an interesting idea
> -          Christoph Paasch: "I don't see a case where the ADD_ADDR-option
> -          Yoshifumi Nishida: "I think L4 should be conservative to react on
> L2/L3 incidents, so I'm a bit skeptical about sending RST here."
>
> -          Olivier Bonaventure: "In the case that you mentioned, the host
> -          Georg Hampel: Christoph's statement means that the host must send
> -          to be continued.
> -          Alan Ford: "An implementation SHOULD send DSS options that cover
> option until a DATA ACK for some data covered by this mapping is received."
>
> -          Christoph Paasch had longer comments on this.
>
>
>
>
>
> Firstly, the MP-PRIO message is altered to include Address ID - and (I
> think) a second flag meaning 'use only this subflow'  (not sure discussion
> finalised on this point)
>
> Thirdly, add some text something like: "An implementation SHOULD send DSS
> mapping is received."
> middleboxes (I couldn't tell whether the discussion concluded on this point)
>
> the peer to swiftly adjust data transmission when the host's interface

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 29 Jun 2011 10:09:28 -0500

Hi Costin, all,

Here is the questionable BBM scenario:

HostA having one radio interface homes on network_1 with IP_1 and starts connection via subflow_1 to HostB.

Then HostA moves to network_2, discontinues IP_1 and obtains IP_2. After that it starts subflow_2. We assume that subflow_1 is still up but without conductivity.

Then HostA moves back to network_1, discontinues IP_2 and obtains IP_1 again. Alan's point is (if I understand it right) that HostA has no knowledge if middleboxes on network_1 still hold state pertaining to subflow_1. Therefore, HostA should start a new subflow from IP_1 right away instead of wasting time on pushing data against a closed firewall.

Do we agree on this behavior? (Y/N)

In case we agree: Since HostA has to start a new subflow after every lower-layer BBM, it should immediately terminate the old subflow when lower-layer BBM occurs. In other words: There is no reason to keep the old subflow up any longer. The old subflow can be terminated via RST (prior to BBM) or via REMOVE_ADDR (after BBM).

Am I missing something?

Regards,

Georg 

-----Original Message-----
From: Costin Raiciu [mailto:[email-masked]] 
Sent: Wednesday, June 29, 2011 10:06 AM
To: Yoshifumi Nishida
Cc: Hampel, K Georg (K Georg); [email-masked]; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Cheers,
Cosin

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 29 Jun 2011 11:18:03 -0400

Hi,

> Do we agree on this behavior? (Y/N)

No. If host A comes back really quickly, or there is just a temporary failure of the interface, it is good to keep the subflow up. Also, if middlebox state has vanished, the host will receive a RST from that middlebox and the subflow will be killed. So there is no hazard here.

> In case we agree: Since HostA has to start a new subflow after every lower-layer BBM, it should immediately terminate the old subflow when lower-layer BBM occurs. In other words: There is no reason…

I see no real reason for this. Instead of saving bandwidth, it could have the opposite effect. As Yoshifumi pointed out, the transport stack should be conservative about reactive to low level status changes.

Costin

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 29 Jun 2011 12:51:14 -0500

Hi Costin, all,

Okay. Unless anybody disagrees BBM can be considered resolved. Thanks.

Georg

-----Original Message-----
From: Costin Raiciu [mailto:[email-masked]]
Sent: Wednesday, June 29, 2011 11:18 AM
To: Hampel, K Georg (K Georg)
Cc: Yoshifumi Nishida; [email-masked]; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi,

Costin

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Wed, 29 Jun 2011 19:52:18 -0500

Hi all,

I'm proposing to revise the MP_PRIO option as outlined below. The revised version tries to accommodate the issues discussed recently.

Please comment.

Regards,

Georg

========================================

MP_PRIO sent by Host_A to Host_B contains the following fields:

Path = (ADDR-ID-A, ADDR-ID-B), where

ADDR-ID-A: one of Host_A's addresses; value 255 = all;

ADDR-ID-B: one of Host_B's addresses; value 255 = all;

B-flag: (use/don't use) path contained in this option

S-flag: (use only one path/use any number of paths)

Operation:

- Host_A can refer to unavailable paths via available paths due to the explicit use of address-ids. This also permits sending an MP_PRIO option redundantly on multiple paths.

- Host_A can refer to a specific interface, e.g. IP_A, by selecting all paths that terminate at this interface, i.e. path = (IP_A, all).

- Through consecutive MP_PRIO messages, Host_A can specify (and update) the subset of paths it wants Host_B to consider for data transmission.

- Host_B is not bound to use this subset of paths for data transmission. However, if a certain interface of Host_A is not contained in any of the paths of this subset, Host_B should consider this interface unavailable and not use any of the corresponding paths for data transmission.

- The S-flag setting "use only one path" is binding. This path can be selected by Host_B subject to the same constraints as mentioned in the prior bullet. When an MP_PRIO option arrives, path-re-selection should always occur AFTER the B-flag- and path-fields of the MP_PRIO option have been evaluated, since they may change the subset of recommended paths.

- If multiple MP_PRIO options are contained in one TCP options header, they should be evaluated in the order they appear in the header.

- There is no explicit confirmation of MP_PRIO. Host_A can implicitly derive MP_PRIO's delivery from acks and/or packets arriving from Host B.

Use cases/examples:

1. Set [IP_A, IP_B] as backup path:

MP_PRIO: path = [IP_A, IP_B], B =(don't use), S =(use any number)

2. IP_A has become unavailable:

MP_PRIO: path = [IP_A, all], B =(don't use), S =(use any number)

3. IP_A is available again:

MP_PRIO: path=[IP_A, all], B =(use), S =(use any number)

4. Use only one path, doesn't matter which one:

MP_PRIO: path=[all, all], B =(use), S =(use only one path)

5. Use only one path among [IP_A, IP_B] and [IP_A', IP_B']:

Add 2 MP_PRIOs into header:

MP_PRIO: path = [IP_A, IP_B], B =(use), S =(either setting)

MP_PRIO: path = [IP_A', IP_B'], B =(use), S =(use only one path)

6. Switch one selected path from [IP_A, IP_B] to [IP_A', IP_B']:

Add 2 MP_PRIOs into header:

MP_PRIO: [IP_A, IP_B], B = (don't use), S = (either setting)

MP_PRIO: [IP_A', IP_B'], B = (use), S = (use only one path)

7. Switch one path from interface IP_A to IP_A':

Add 2 MP_PRIOs into header:

MP_PRIO: [IP_A, all], B = (don't use), S = (either setting)

MP_PRIO: [IP_A', all], B = (use), S = (use only one path)

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 30 Jun 2011 03:03:59 -0700

Hi Georg,

In this proposal, if we specify a non-existing sub-connection and set
"use only this path", what will happen?
Also, I'm still wondering how to solve binding loop where both ends
have conflicted policies. (sorry i couldn't parse your previous
e-mail.)

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/6/29 Hampel, K Georg (K Georg) <[email-masked]>:
> Hi all,
>
>
>
> I’m proposing to revise the MP_PRIO option as outlined below. The revised
> version tries to accommodate the issues discussed recently.
>
>
>
> Please comment.
>
> Regards,
>
> Georg
>
>
>
> ADDR-ID-A: one of Host_A’s addresses; value 255 = all;
>
> ADDR-ID-B: one of Host_B’s addresses; value 255 = all;
>
> B-flag: (use/don’t use) path contained in this option
>
> - Host_A can refer to unavailable paths via available paths due to the
> explicit use of address-ids. This also permits sending an MP_PRIO option
> redundantly on multiple paths.
>
>
>
> - Host_A can refer to a specific interface, e.g. IP_A, by selecting all
> paths that terminate at this interface, i.e. path = (IP_A, all).
>
>
>
> - Through consecutive MP_PRIO messages, Host_A can specify (and update) the
> subset of paths it wants Host_B to consider for data transmission.
>
>
>
> - Host_B is not bound to use this subset of paths for data transmission.
> However, if a certain interface of Host_A is not contained in any of the
> paths of this subset, Host_B should consider this interface unavailable and
> not use any of the corresponding paths for data transmission.
>
>
>
> - The S-flag setting “use only one path” is binding. This path can be
> selected by Host_B subject to the same constraints as mentioned in the prior
> bullet. When an MP_PRIO option arrives, path-re-selection should always
> occur AFTER the B-flag- and path-fields of the MP_PRIO option have been
> evaluated, since they may change the subset of recommended paths.
>
>
>
> - If multiple MP_PRIO options are contained in one TCP options header, they
> should be evaluated in the order they appear in the header.
>
>
>
> - There is no explicit confirmation of MP_PRIO. Host_A can implicitly derive
> MP_PRIO’s delivery from acks and/or packets arriving from Host B.
>
>
>
>
>
> Use cases/examples:
>
>
>
> MP_PRIO: path = [IP_A, IP_B], B =(don’t use), S =(use any number)
>
>
>
> MP_PRIO: path = [IP_A, all], B =(don’t use), S =(use any number)
>
>
>
> 4. Use only one path, doesn’t matter which one:
>
> 5. Use only one path among [IP_A, IP_B] and [IP_A’, IP_B’]:
>
> MP_PRIO: path = [IP_A’, IP_B’], B =(use), S =(use only one path)
>
>
>
>
>
> 6. Switch one selected path from [IP_A, IP_B] to [IP_A’, IP_B’]:
>
> MP_PRIO: [IP_A, IP_B], B = (don’t use), S = (either setting)
>
> MP_PRIO: [IP_A’, IP_B’], B = (use), S = (use only one path)
>
>
>
>
>
> 7. Switch one path from interface IP_A to IP_A’:
>
> MP_PRIO: [IP_A, all], B = (don’t use), S = (either setting)
>
> MP_PRIO: [IP_A’, all], B = (use), S = (use only one path)
>
>
>
>
>

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 30 Jun 2011 08:42:25 -0500

Hi Yoshifumi,

There is no "use only THIS path". Instead, there is "use only ONE path". While this command is binding, it does not specify the path itself. Host_B can select this path from all available paths while honoring the associated use/don't use recommendations sent by Host_A. 

(For a simplified receiver on Host_A it is important to restrict data delivery to one path. Path selection is of lower priority and subject to the same constraints as in full-fledged MPTCP.)

"Use/Don't use" recommendations for non-existent paths should have no effect.

Conflicting interests: 
"Use/Don't use" flags are primarily recommendations which SHOULD be honored (this is the same as for the present MP_PRIO). Host_B may overwrite this SHOULD due to economical reasons, for instance.

If a certain interface on Host_A is not part of any recommended path, Host_B can assume that it is not up or has very poor local link quality. In this case, Host_B SHOULD not use the associated paths for technical reasons. 

We COULD replace SHOULD with MUST in this latter case. However, if a certain interface is completely inaccessible for extended time, Host_A can simply tear down the associated paths.

Georg 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Thursday, June 30, 2011 6:04 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; [email-masked]; Klein, Thierry E (Thierry)
Subject: Re: [multipathtcp] New I-D: Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks

Hi Georg,

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/6/29 Hampel, K Georg (K Georg) <[email-masked]>:
> Hi all,
>
>
>
> I'm proposing to revise the MP_PRIO option as outlined below. The revised
> - The S-flag setting "use only one path" is binding. This path can be
> MP_PRIO's delivery from acks and/or packets arriving from Host B.
>
>
>
>
>
> Use cases/examples:
>
>
>