Subject: [multipathtcp] MPTCP carrying UDP

From: <[email-masked]>
Date: Tue, 15 Nov 2016 11:51:17 +0000

(I changed the Subject line of the email)

Do people have any experimental results /experiences they could share of running UDP applications over MPTCP sub-flows?  Would be interested to hear about the issues. I guess VoIP and Quic would be the most interesting ones. Whether the experiments are with only one path being used at once, or multiple. etc

From: Robert Skog [mailto:[email-masked]]
Sent: 15 November 2016 10:09
To: [email-masked]; [email-masked]; Eardley,PL,Philip,TUB8 R <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: RE: [multipathtcp] Towards a Multipath TCP Proxy work item

+1
We see a need to solve UDP in several places, so having a separate specification can help this moving forward.

Cheers,
/Robert

From: multipathtcp [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: den 15 november 2016 09:57
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [multipathtcp] Towards a Multipath TCP Proxy work item

Hi Markus, all,

I’m on the same page that non-tcp traffic should be considered in scope. At least, the UDP case should be investigated. I remember there was interest in BA meeting about the UDP part (https://www.ietf.org/proceedings/95/minutes/minutes-95-mptcp).

An option I see is to consider advancing the use of MPTCP to convey UDP flows in a separate EXPERIMENTAL specification that will be cross-reviewed with other WGs (intarea, tsvwg). Having that base specification will help having interoperable implementations that will be used to carry experimentations.

Those experiments will help to determine whether the proposed solution is complex/simple, viable/non-viable, failed/successful, etc.

Let’s give it a try.

Cheers,
Med

De : multipathtcp [mailto:[email-masked]] De la part de [email-masked]<mailto:[email-masked]>
Envoyé : mardi 15 novembre 2016 08:58
À : [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc : [email-masked]<mailto:[email-masked]>
Objet : Re: [multipathtcp] Towards a Multipath TCP Proxy work item

Phil,

If you feel it would take too long and to be too complex to achieve in a reasonable timeframe, we are ok with excluding non-tcp traffic, however, for our total solution we do requires also non-tcp traffic to be sent over multiple paths. We could argue that we can use a different mechanisms specifically for UDP and other protocols, but it would be nice to have as single MPTCP-based solution. (my gut feeling is that we would need to replicate a lot of MPTCP scheduling features into a new mechanism).

Marcus

Von: multipathtcp <[email-masked]<mailto:[email-masked]>> im Auftrag von "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Datum: Montag, 14. November 2016 um 07:32
An: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Betreff: Re: [multipathtcp] Towards a Multipath TCP Proxy work item

i think we should move beyond "exploring whether it would be useful".

i'd like us to assess proposed solutions. i think we should say this is what we're doing - at the moment we've had quite a lot of discussion about one proposal. we should give the chance for other proposals, and make the discussion more structured (what are the assessment criteria).

i also think we should explicitly exclude non-tcp traffic  (i think non-tcp traffic is too big a topic for our WG)

phil

________________________________
From: Alan Ford <[email-masked]<mailto:[email-masked]>>
Sent: 14 November 2016 06:11
To: Eardley,PL,Philip,TUB8 R
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [multipathtcp] Towards a Multipath TCP Proxy work item

I think this work item is achievable by simply removing references to “at least one end” from the existing charter item on the proxy. So the item would now read:

Finally, the working group will explore whether an MPTCP-aware
middlebox would be useful. For example, potentially helping MPTCP’s
incremental deployment by allowing only one end host to be MPTCP-enabled
and the middlebox acts as an MPTCP proxy for the other end host, which runs
TCP; and potentially helping some mobility scenarios, where the middle box
acts as an anchor between two MPTCP-enabled hosts. Alternatively, neither
end host could be MPTCP-enabled but a pair of proxies could work together to
bring MPTCP benefits to such connections. The working group will detail what real
problems an MPTCP-enabled middlebox might solve, how it would impact the
Multipath TCP architecture (RFC6182), what proxy approach might be
justified as compared against alternative solutions to the problems, and
the likely feasibility of solving the technical and security issues.

In some ways, the two ended proxy work could even be seen as an extension of the previous operational experience work within this WG.

Regards,
Alan

On 10 Nov 2016, at 19:17, [email-masked]<mailto:[email-masked]> wrote:

Hi,
Perhaps this is speaking too soon, but it looks like the very active discussion is reaching some common understanding?

We’re trying to work out what a work item might look like, so would like to understand what assumptions we would make, eg about the scenario, & what common agreements we’d assume & restrictions on how the solution works. This seems important to frame work by WG. If possible we’d like discussion on these points to avoid getting into the fine details of one particular existing proposal.

What we’d appreciate is a summary of what the assumptions /understandings are about:
•         The scenario (for instance: the MPTCP-enabled host knows the address of the proxy (eg through configuration); and it knows the address of the ‘legacy’ host it wants to communicate with)
•         If any impact is already envisaged on the current MPTCP protocol’s fallback behaviour and coping with middleboxes
•         If we can agree that the solution is based on a new MPTCP option
•         If any impact is already envisaged on the current MPTCP protocol’s semantics (other than the new option) eg in terms of https://tools.ietf.org/html/rfc6824#section-4
•         If any impact is already envisaged on TCP’s semantics, or any mods are needed, or assumptions about its behaviour, etc
•         If any impact is already envisaged on other existing transport protocol’s semantics (presuming people still would like non-TCP in scope?)
•         Anything else that you think is needed in order to frame the work item

It may be clearer to do this for the two use cases (single-ended proxy, ie where only one host is MPTCP-enabled; and double-ended proxy, ie where neither host is MPTCP-enabled).

This may seem like a long list, but most of the answers can be “none” – we’ll end up with just a short paragraph or a few bullets in the charter.

We’d also have to work out interactions with non-MPTCP WGs, but Mirja and IESG will probably want the main input on this.

Thanks
Phil & Yoshi
_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: <[email-masked]>
Date: Tue, 15 Nov 2016 12:40:01 +0000

Re-,

I have one comment about the target applications: QUIC is definitely one.

For the VoIP case (RTP to be more accurate), there is already a candidate solution: Multipath RTP (https://tools.ietf.org/html/draft-ietf-avtcore-mprtp-03).

I don’t have results to share because we are in a chicken/egg situation: implementers are asking for a stable specification to play with.

Cheers,
Med

De : [email-masked] [mailto:[email-masked]]
Envoyé : mardi 15 novembre 2016 12:51
À : [email-masked]; BOUCADAIR Mohamed IMT/OLN; [email-masked]; [email-masked]
Cc : [email-masked]
Objet : MPTCP carrying UDP

To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Eardley,PL,Philip,TUB8 R <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: Alan Ford <[email-masked]>
Date: Tue, 15 Nov 2016 15:55:13 +0000

As I said at IETF95 and will say again now, MPTCP provides a TCP service: stream-oriented, reliable, whereas UDP is datagram-oriented, and unreliable.

"Mapping" one to the other cannot be directly done since the transports have fundamentally different semantics.

You can tunnel UDP over TCP, as long as you maintain packet boundaries. This gains reliability (which may or may not be a gain, depending on what you are trying to achieve with UDP - e.g. with RTP, you really don’t care about reliability, you are about timeliness). However, you must maintain the packet boundaries - UDP is not a stream.

Mapping a datagram-oriented transport to a stream-oriented transport is a generic problem and well out of scope of the MPTCP working group. There is (deliberately) nothing about MPTCP’s service model which is different to TCP, and therefore it is unlikely there would be anything different regarding mapping UDP to MPTCP as mapping UDP to TCP.

A lot of UDP uses - such as RTP - would be distinctly unhappy with the TCP model of retransmissions. Transparently providing this mapping would break assumption from the application layer.

Regards,
Alan

> On 15 Nov 2016, at 12:40, [email-masked] wrote:
> 
> Re-,
>  
> For the VoIP case (RTP to be more accurate), there is already a candidate solution: Multipath RTP (https://tools.ietf.org/html/draft-ietf-avtcore-mprtp-03 <https://tools.ietf.org/html/draft-ietf-avtcore-mprtp-03>).
>  
> From: Robert Skog [mailto:[email-masked] <mailto:[email-masked]>] 
> To: [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>; Eardley,PL,Philip,TUB8 R <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> Cc: [email-masked] <mailto:[email-masked]>
> From: multipathtcp [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of [email-masked] <mailto:[email-masked]>
> To: [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> Cc: [email-masked] <mailto:[email-masked]>
> I’m on the same page that non-tcp traffic should be considered in scope. At least, the UDP case should be investigated. I remember there was interest in BA meeting about the UDP part (https://www.ietf.org/proceedings/95/minutes/minutes-95-mptcp <https://www.ietf.org/proceedings/95/minutes/minutes-95-mptcp>).
>  
> De : multipathtcp [mailto:[email-masked] <mailto:[email-masked]>] De la part de [email-masked] <mailto:[email-masked]>
> À : [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> Cc : [email-masked] <mailto:[email-masked]>
> Von: multipathtcp <[email-masked] <mailto:[email-masked]>> im Auftrag von "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> An: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> From: Alan Ford <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>
> On 10 Nov 2016, at 19:17, [email-masked] <mailto:[email-masked]> wrote:
>  
> Hi,
> ·         The scenario (for instance: the MPTCP-enabled host knows the address of the proxy (eg through configuration); and it knows the address of the ‘legacy’ host it wants to communicate with)
> ·         If any impact is already envisaged on the current MPTCP protocol’s fallback behaviour and coping with middleboxes
> ·         If we can agree that the solution is based on a new MPTCP option
> ·         If any impact is already envisaged on the current MPTCP protocol’s semantics (other than the new option) eg in terms of https://tools.ietf.org/html/rfc6824#section-4 <https://tools.ietf.org/html/rfc6824#section-4>  
> ·         If any impact is already envisaged on TCP’s semantics, or any mods are needed, or assumptions about its behaviour, etc
> ·         If any impact is already envisaged on other existing transport protocol’s semantics (presuming people still would like non-TCP in scope?)
> ·         Anything else that you think is needed in order to frame the work item
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/multipathtcp <https://www.ietf.org/mailman/listinfo/multipathtcp>
>

---

From: <[email-masked]>
Date: Tue, 15 Nov 2016 16:12:48 +0000

Hi Alan,

Please see inline.

Cheers,
Med

De : Alan Ford [mailto:[email-masked]]
Envoyé : mardi 15 novembre 2016 16:55
À : BOUCADAIR Mohamed IMT/OLN
Cc : [email-masked]; [email-masked]; [email-masked]; [email-masked]
Objet : Re: MPTCP carrying UDP

You can tunnel UDP over TCP, as long as you maintain packet boundaries.
[Med] The proposed design will preserve the packet boundaries since the length of the UDP payload is also placed in the payload of MPTCP messages.

This gains reliability (which may or may not be a gain, depending on what you are trying to achieve with UDP
[Med] Yes. Another use is to pass firewalls when only TCP is allowed, but that’s another story. Let’s not open that one.

- e.g. with RTP
[Med] As I mentioned below, RTP is not a target use case since there is already a multipath RTP solution.

, you really don’t care about reliability, you are about timeliness). However, you must maintain the packet boundaries - UDP is not a stream.
[Med] That is what we are proposing. Here is an extract from an old version of the plain mode draft:

==
   1.  Extract the payload and its length from the UDP datagram.

   2.  Send the length (as a 16 bits field in network byte order)
       followed by the payload of the UDP datagram over the bound MPTCP
       connection.
==

[Med] That story may need to be revisited given the increase of UDP traffic and the definition of TCP-like primitives over UDP.

A lot of UDP uses - such as RTP - would be distinctly unhappy with the TCP model of retransmissions. Transparently providing this mapping would break assumption from the application layer.
[Med] This is a fair comment. Polices will be used to elect flows that will be placed over an MPTCP connection. This is exactly the same problem as for TCP. All is policy-based.

Regards,
Alan

On 15 Nov 2016, at 12:40, [email-masked]<mailto:[email-masked]> wrote:

Re-,

Cheers,
Med

De : [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Envoyé : mardi 15 novembre 2016 12:51
À : [email-masked]<mailto:[email-masked]>; BOUCADAIR Mohamed IMT/OLN; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc : [email-masked]<mailto:[email-masked]>
Objet : MPTCP carrying UDP

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: <[email-masked]>
Date: Tue, 15 Nov 2016 16:21:43 +0000

Hi Phil, all,

Our Swisscom Labs have done some playing around on the topic running UDP over SOCKS5 + MPTCP (Thanks to Olaf, Silka, Dominique) for feasibility not for performance measurements.

The learnings are that it works, however it is naturally more complex to do than TCP.

-          Since UDP is stateless, every single packet has to be examined for its destination (IP and port). This also requires a timeout associated with each MPTCP session in order to manage the lifetime (the client or destination will not send a FIN).

-          Obtaining the actual destination address of a packet after it was redirected to a proxy is not trivial (that sound a bit like an implementation issue of the kernels and the interplay with the proxy used).

Kind regards/Viele Grüsse

Marcus

——-
Marcus Brunner
Swisscom
+41798764409

Von: "[email-masked]" <[email-masked]>
Datum: Dienstag, 15. November 2016 um 12:51
An: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, SCS-SIS-INI Brunner Markus <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Betreff: MPTCP carrying UDP

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 15 Nov 2016 09:21:59 -0800

> I don’t have results to share because we are in a chicken/egg situation: implementers are asking for a stable specification to play with.

I don't buy this argument. Implementors can build a prototype to feed
information/feedback into the standardization process.

Christoph

> [email-masked]

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 05:25:18 +0000

Hi,

Von: multipathtcp [mailto:[email-masked]] Im Auftrag von Alan Ford
Gesendet: Dienstag, 15. November 2016 16:55
An: [email-masked]
Cc: [email-masked]; [email-masked]
Betreff: Re: [multipathtcp] MPTCP carrying UDP

> You can tunnel UDP over TCP, as long as you maintain packet boundaries. This gains reliability (which may or may not be a gain,
>  depending on what you are trying to achieve with UDP - e.g. with RTP, you really don’t care about reliability, you are about
>  timeliness). However, you must maintain the packet boundaries - UDP is not a stream.
I don’t think that tunneling UDP over TCP is really an option as it breaks the expected behavior of UDP. There are also certain applications which do implement
there own retransmission and flow control mechanisms for UDP (e.g. QUIC or for certain IPTV applications) and I guess you will see a different behavior on application level if you tunnel UDP over a TCP connection in parts of the network.  In addition the tunnel might be not end-to-end but used between two proxies (in that case you might lose a UDP packet which is than being retransmitted by an mechanism within the application using UDP.

Regards

Nic

Re-,

Cheers,
Med

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 06:46:43 +0000

Hi Markus,

Thank you for sharing this information.

Please see inline.

Cheers,
Med

De : [email-masked] [mailto:[email-masked]]
Envoyé : mardi 15 novembre 2016 17:22
À : [email-masked]; [email-masked]; BOUCADAIR Mohamed IMT/OLN; [email-masked]
Cc : [email-masked]
Objet : Re: MPTCP carrying UDP

Hi Phil, all,

[Med] This is why we are propossing to have a lifetime entry in the binding table maintained by an MPTCP proxy:

   Lifetime:  This information indicates the remaining validity lifetime
      for the session entry.  When the lifetime expires, this session
      entry is deleted from the table.  If all sessions bound to a given
      binding entry expired, that binding entry must be deleted.
      Recommendations for setting this parameter are defined in
      [RFC7857<https://tools.ietf.org/html/rfc7857>][RFC5382][RFC4787<https://tools.ietf.org/html/rfc4787>].

We are also proposing a solution to sync the binding entries between two proxies:

==4.3.3<https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-08#section-4.3.3>.  Expiration of a Binding Entry

   A configurable parameter MAY be supported by the CPE and the
   concentrator to terminate MPTCP connections with the FASTCLOSE
   procedure (Section 3.5 of [RFC6824]<https://tools.ietf.org/html/rfc6824#section-3.5>) when a binding entry expires.

   If there is no binding state that matches a received non-SYN segment,
   the CPE/concentrator SHOULD reply with a RST segment.  This behavior
   aims to synchronize the binding tables between the CPE and the
   concentrator by clearing bindings that are present either in the CPE
   or in the concentrator.

==================

And also this behavior:

==
5.1.3<https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-08#section-5.1.3>.  Terminating UDP-Triggered Subflows
   UDP-triggered subflows SHOULD be terminated by an MPTCP endpoint (CPE
   or concentrator) if no UDP packet matching the corresponding binding
   entry is received for at least 5 minutes (see Section 4.3 of<https://tools.ietf.org/html/rfc4787#section-4.3>
   [RFC4787]<https://tools.ietf.org/html/rfc4787#section-4.3>).  Consequently, the procedure in Section 4.4.4<https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-08#section-4.4.4> MUST be

   followed to terminate the MPTCP connection and the associated
   subflows.  The transport coordinates of the FASTCLOSE messages are
   set according to the information maintained in the binding table.

=

[Med] I see this is as implementtaion-specific.

Marcus

——-
Marcus Brunner
Swisscom
+41798764409

Von: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Datum: Dienstag, 15. November 2016 um 12:51
An: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, SCS-SIS-INI Brunner Markus <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Betreff: MPTCP carrying UDP

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 07:07:58 +0000

Hi Nic,

Of course, there are applications that are not suitable to be injected inside an MPTCP connection.
It is also clear that some issues are to be handled correctly if we want to have a decent solution.

All that is fully agreed.

My point is that instead of speculating about the foreseen results, let’s document an EXPERIMENTAL solution so that people can play with and assess. This is fundamental because all is about technical details.

If it appears the results are bad, that particular solution will be then stamped as historic.

Cheers,
Med

De : [email-masked] [mailto:[email-masked]]
Envoyé : mercredi 16 novembre 2016 06:25
À : [email-masked]; BOUCADAIR Mohamed IMT/OLN
Cc : [email-masked]; [email-masked]
Objet : AW: [multipathtcp] MPTCP carrying UDP

Hi,

Von: multipathtcp [mailto:[email-masked]] Im Auftrag von Alan Ford
Gesendet: Dienstag, 15. November 2016 16:55
An: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Betreff: Re: [multipathtcp] MPTCP carrying UDP

Regards

Nic

Re-,

Cheers,
Med

+1

Cheers,
/Robert

Hi Markus, all,

Let’s give it a try.

Cheers,
Med

Phil,

Marcus

phil

Regards,
Alan

Hi,

Thanks
Phil & Yoshi

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 07:11:55 +0000

Christoph, 

I would like to make two comments: 
* The implementers I have in mind are CPE vendors and Carrier-Grade platform manufacturers.
* Implementation does not come for free. 

Cheers,
Med

> -----Message d'origine-----
> De : [email-masked] [mailto:[email-masked]]
> Envoyé : mardi 15 novembre 2016 18:22
> Cc : [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> Objet : Re: [multipathtcp] MPTCP carrying UDP
> 
> > For the VoIP case (RTP to be more accurate), there is already a
> candidate solution: Multipath RTP (https://tools.ietf.org/html/draft-ietf-
> avtcore-mprtp-03).
> >
> > I don’t have results to share because we are in a chicken/egg situation:
> implementers are asking for a stable specification to play with.
> 
> > À : [email-masked]; BOUCADAIR Mohamed IMT/OLN;
> [email-masked]; [email-masked]
> > Do people have any experimental results /experiences they could share of
> running UDP applications over MPTCP sub-flows?  Would be interested to
> hear about the issues. I guess VoIP and Quic would be the most interesting
> ones. Whether the experiments are with only one path being used at once,
> or multiple. etc
> >
> >
> >
> > To: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>;
> Eardley,PL,Philip,TUB8 R
> <[email-masked]<mailto:[email-masked]>>;
> [email-masked]<mailto:[email-masked]>
> > We see a need to solve UDP in several places, so having a separate
> specification can help this moving forward.
> >
> > Cheers,
> > /Robert
> >
> > From: multipathtcp [mailto:[email-masked]] On Behalf Of
> [email-masked]<mailto:[email-masked]>
> > To: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>
> > I’m on the same page that non-tcp traffic should be considered in scope.
> At least, the UDP case should be investigated. I remember there was
> interest in BA meeting about the UDP part
> (https://www.ietf.org/proceedings/95/minutes/minutes-95-mptcp).
> >
> > An option I see is to consider advancing the use of MPTCP to convey UDP
> flows in a separate EXPERIMENTAL specification that will be cross-reviewed
> with other WGs (intarea, tsvwg). Having that base specification will help
> having interoperable implementations that will be used to carry
> experimentations.
> >
> > Those experiments will help to determine whether the proposed solution
> is complex/simple, viable/non-viable, failed/successful, etc.
> >
> > Let’s give it a try.
> >
> > Cheers,
> > Med
> >
> > De : multipathtcp [mailto:[email-masked]] De la part de
> [email-masked]<mailto:[email-masked]>
> > À : [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>
> > If you feel it would take too long and to be too complex to achieve in a
> reasonable timeframe, we are ok with excluding non-tcp traffic, however,
> for our total solution we do requires also non-tcp traffic to be sent over
> multiple paths. We could argue that we can use a different mechanisms
> specifically for UDP and other protocols, but it would be nice to have as
> single MPTCP-based solution. (my gut feeling is that we would need to
> replicate a lot of MPTCP scheduling features into a new mechanism).
> >
> > Marcus
> >
> > Von: multipathtcp <[email-masked]<mailto:multipathtcp-
> [email-masked]>> im Auftrag von
> "[email-masked]<mailto:[email-masked]>"
> <[email-masked]<mailto:[email-masked]>>
> > An: "[email-masked]<mailto:[email-masked]>"
> <[email-masked]<mailto:[email-masked]>>
> > Cc: "[email-masked]<mailto:[email-masked]>"
> <[email-masked]<mailto:[email-masked]>>
> > i'd like us to assess proposed solutions. i think we should say this is
> what we're doing - at the moment we've had quite a lot of discussion about
> one proposal. we should give the chance for other proposals, and make the
> discussion more structured (what are the assessment criteria).
> >
> >
> >
> > i also think we should explicitly exclude non-tcp traffic  (i think non-
> tcp traffic is too big a topic for our WG)
> >
> >
> >
> > phil
> >
> >
> >
> >
> >
> > I think this work item is achievable by simply removing references to
> “at least one end” from the existing charter item on the proxy. So the
> item would now read:
> >
> > In some ways, the two ended proxy work could even be seen as an
> extension of the previous operational experience work within this WG.
> >
> > Regards,
> > Alan
> >
> > On 10 Nov 2016, at 19:17,
> [email-masked]<mailto:[email-masked]> wrote:
> >
> > Hi,
> > Perhaps this is speaking too soon, but it looks like the very active
> discussion is reaching some common understanding?
> >
> > We’re trying to work out what a work item might look like, so would like
> to understand what assumptions we would make, eg about the scenario, &
> what common agreements we’d assume & restrictions on how the solution
> works. This seems important to frame work by WG. If possible we’d like
> discussion on these points to avoid getting into the fine details of one
> particular existing proposal.
> >
> > •         The scenario (for instance: the MPTCP-enabled host knows the
> address of the proxy (eg through configuration); and it knows the address
> of the ‘legacy’ host it wants to communicate with)
> > •         If any impact is already envisaged on the current MPTCP
> protocol’s fallback behaviour and coping with middleboxes
> > •         If any impact is already envisaged on the current MPTCP
> protocol’s semantics (other than the new option) eg in terms of
> https://tools.ietf.org/html/rfc6824#section-4
> > •         If any impact is already envisaged on TCP’s semantics, or any
> mods are needed, or assumptions about its behaviour, etc
> > •         If any impact is already envisaged on other existing transport
> protocol’s semantics (presuming people still would like non-TCP in scope?)
> > It may be clearer to do this for the two use cases (single-ended proxy,
> ie where only one host is MPTCP-enabled; and double-ended proxy, ie where
> neither host is MPTCP-enabled).
> >
> > This may seem like a long list, but most of the answers can be “none” –
> we’ll end up with just a short paragraph or a few bullets in the charter.
> >
> > We’d also have to work out interactions with non-MPTCP WGs, but Mirja
> and IESG will probably want the main input on this.
> >
> > Thanks
> > Phil & Yoshi

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 16 Nov 2016 10:06:44 +0100

Med,
> * Implementation does not come for free.

Independently of the topic of this email, I'd like to point out that in 
the MPTCP working group most of the important decisions have been 
validated by early prototypes that have been evaluated and discussed in 
the working group. I think that this has served as an important "reality 
check" in the working group and has probably contributed to the good 
results that we have achieved with the various deployments that have 
been documented. With the open-source implementation of Multipath TCP in 
the Linux kernel, the cost of developing a prototype is not as high as 
developing a hardware implementation in some working groups.

Coming back to the proxy topic, TCP/MPTCP proxies have already been 
demonstrated, e.g. in
http://inl.info.ucl.ac.be/publications/multipath-middlebox
and extensions of this idea are deployed in the field.
I'm not aware of similar work showing that UDP can be safely 
encapsulated inside MPTCP for reactive applications.

Best regards,

Olivier

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 09:53:28 +0000

Hi Olivier, 

I'm not questioning the importance of implementations. Their importance is obvious to me. 

If we want to proceed to a "reality check" as you mentioned, there should be an agreement of what "reality" is to be checked at the first place!

"tunnel UDP over MPTCP" is too generic. We need to point to a given specification so that interpreting results is not biased. For example, comparing the experimental results of these proposals does not make sense: 

* nc command to relay TCP to UDP and vice versa: http://www.tutorialspoint.com/unix_commands/nc.htm 
* http://www.komodia.com/komodia-relay 
* http://www.cs.columbia.edu/~lennox/udptunnel/ 
* SOCKS (as mentioned by Markus). 

Cheers,
Med

> De : Olivier Bonaventure [mailto:[email-masked]]
> Envoyé : mercredi 16 novembre 2016 10:07
> À : BOUCADAIR Mohamed IMT/OLN; [email-masked]

---

From: William Ivancic <[email-masked]>
Date: Wed, 16 Nov 2016 09:23:08 -0500

I think Med’s proposal is very reasonable.  I think whatever would be learned from and EXPERIMENTAL solution would be valuable for many working groups / research groups.  One might consider taking this to some research group, but I think we are so close to an EXPERIMENTAL solution already in MPTCP, that  such a move wouldn’t be a good use of time and energy.

Will

From: multipathtcp <[email-masked]> on behalf of <[email-masked]>
Date: Wednesday, November 16, 2016 at 2:07 AM
To: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [multipathtcp] MPTCP carrying UDP

Hi Nic, 

Cheers,

Med

Hi,

> "Mapping" one to the other cannot be directly done since the transports have fundamentally different semantics.

…..

---

From: Sébastien Noel <[email-masked]>
Date: Wed, 23 Nov 2016 12:09:26 +0100

Phil,

> running UDP applications over MPTCP sub-flows?  Would be interested to hear
> about the issues.
> I guess VoIP and Quic would be the most interesting ones.

To understand the interactions between QUIC and an underlying MPTCP 
transport, we performed some experiments by running QUIC over OpenVPN 
that runs itself over an MPTCP connection. This is the closest scenario 
to what you are discussing based on existing open-source software.

OpenVPN includes some framing to carry UDP and 
encryption/authentication. These mechanisms add CPU overhead and byte 
overhead compared to transporting QUIC over a plain MPTCP connection, 
but this does not change the results of the experiments.

Our measurement setup was the following.

                       /-----\
[client] --- [router1]         [router2] --- [server]
                       \-----/

An openvpn tunnel over MPTCP mode was setup between router 1 & 2.

The client & server were not running an MPTCP kernel.

We simply recorded the time needed to transfer a random 10M file
from server to client while tweaking speed/latency/reliability of each 
link on router 1 & 2 with netem.

We used quic_server & quic_client
from https://github.com/google/proto-quic

In the following tables, Download/Upload are in Mbps, time in seconds &
performance is a percentage of a perfect theoretical result/max speed
(without any IP or TCP overhead).

1) "no" latency (< 1ms) & no packet loss; different link bandwidths
		
Link 1		| Link2		|
--------------------------------|----------------
Down	Upload	| Down	Upload	| time	| perf (%)
-------------------------------------------------
1	1	| 1	1	| 58,1	| 68,85 %
1	1	| 2	1	| 48,4	| 55,10 %
1	1	| 4	1	| 24,3	| 65,84 %
1	1	| 8	1	| 12,3	| 72,27 %
1	1	| 10	1	| 10,3	| 70,61 %
1	1	| 15	5	| 7,2	| 69,44 %
1	1	| 20	5	| 6,5	| 58,61 %
1	1	| 25	5	| 18,2	| 16,91 %
1	1	| 30	10	| 20,1	| 12,84 %
1	1	| 50	10	| 15,5	| 10,12 %
2	1	| 2	1	| 31,7	| 63,09 %
4	1	| 4	1	| 17,1	| 58,48 %
8	1	| 8	1	| 10	| 50,00 %
10	1	| 10	1	| 7,3	| 54,79 %
15	5	| 15	5	| 6,4	| 41,67 %
20	5	| 20	5	| 3,6	| 55,56 %
25	5	| 25	5	| 3,3	| 48,48 %
30	10	| 30	10	| 2,9	| 45,98 %

2) Fixed bandwidth ; no latency ; different packet loss on Link2

Link 1		| Link2			|
----------------------------------------|----------------
Down	Upload	| Down	Upload	P-L(%)	| time	| perf (%)
---------------------------------------------------------
2	1	| 10	2	0	| 9,9	| 67,34 %
2	1	| 10	2	1	| 11,2	| 59,52 %
2	1	| 10	2	2	| 11,3	| 59,00 %
2	1	| 10	2	3	| 11,7	| 56,98 %
2	1	| 10	2	4	| 12,3	| 54,20 %
2	1	| 10	2	5	| 15,9	| 41,93 %
2	1	| 10	2	6	| 16,6	| 40,16 %
2	1	| 10	2	7	| 18,7	| 35,65 %
2	1	| 10	2	8	| 20,1	| 33,17 %
2	1	| 10	2	9	| 23,5	| 28,37 %
2	1	| 10	2	10	| 27,4	| 24,33 %

3) Fixed bandwidth ; no packet loss ; variable latency (in ms) on Link2

Link 1		| Link2			|
----------------------------------------|----------------
Down	Upload	| Down	Upload	Latency	| time	| perf (%)
---------------------------------------------------------
2	1	| 10	2	0	| 9,9	| 67,34 %
2	1	| 10	2	5	| 10	| 66,67 %
2	1	| 10	2	10	| 10,1	| 66,01 %
2	1	| 10	2	15	| 10,2	| 65,36 %
2	1	| 10	2	20	| 10,6	| 62,89 %
2	1	| 10	2	30	| 12,2	| 54,64 %
2	1	| 10	2	40	| 14,3	| 46,62 %
2	1	| 10	2	50	| 16,2	| 41,15 %
2	1	| 10	2	60	| 18,2	| 36,63 %

IMHO those results confirm the intuition that running a protocol like 
QUIC that includes its own congestion control and retransmission 
mechanisms are a reliable bytestream protocol like MPTCP is not a good idea.

As you can see, with QUIC over MPTCP sub-flows, performance seems to 
drop as soon as you have an unreliable medium or as soon as you have latency

The sames tests were performed again, but this time with HTTP over 
end-to-end MPTCTP, to have a point of comparison

1b) "no" latency (< 1ms) & no packet loss; variable bandwidth
		
Link 1		| Link2		|
--------------------------------|----------------
Down	Upload	| Down	Upload	| time	| perf (%)
-------------------------------------------------
2	1	| 2	1	| 23,5	| 85,11 %
4	1	| 4	1	| 11,2	| 89,29 %
8	1	| 8	1	| 5,7	| 87,72 %
10	1	| 10	1	| 4,9	| 81,63 %
15	5	| 15	5	| 3	| 88,89 %
20	5	| 20	5	| 2,5	| 80,00 %
25	5	| 25	5	| 1,8	| 88,89 %
30	10	| 30	10	| 1,5	| 88,89 %

This should not surprise the readers of this list, but confirms that 
MPTCP works well in this environment.

2b) Fixed bandwidth ; no latency ; variable packet loss (P-L) on Link2

Link 1		| Link2			|
----------------------------------------|----------------
Down	Upload	| Down	Upload	P-L(%)	| time	| perf (%)
---------------------------------------------------------
2	1	| 10	2	0	| 8,2	| 81,30 %
2	1	| 10	2	1	| 7,8	| 85,47 %
2	1	| 10	2	2	| 7,7	| 86,58 %
2	1	| 10	2	3	| 7,8	| 85,47 %
2	1	| 10	2	4	| 7,7	| 86,58 %
2	1	| 10	2	5	| 8,8	| 75,76 %
2	1	| 10	2	6	| 8	| 83,33 %
2	1	| 10	2	7	| 7,8	| 85,47 %
2	1	| 10	2	8	| 7,9	| 84,39 %
2	1	| 10	2	9	| 8	| 83,33 %

Again, MPTCP adapts correctly to packet losses in the environment.

Given the bad results of running QUIC over MPTCP, we don't plan to 
analyse this in more details.

Best regards,

Sébastien

---

From: <[email-masked]>
Date: Wed, 23 Nov 2016 13:07:13 +0000

Dear Sébastion, 

Thank you for sharing these details. 

Please see inline.

Cheers,
Med

> Sébastien Noel
> Envoyé : mercredi 23 novembre 2016 12:09
> À : [email-masked]
> hear

[Med] The schema we are investigating does not include this cascaded layers. It is only UDP payload transported in plain MPTCP connections.  

> but this does not change the results of the experiments.

[Med] I wouldn't drop that conclusion as the overall performance depends also on the overhead prepended to packets to be injected over the tunnel. 

Putting that aside, can you please indicate how the traffic is distributed among available subflows? FWIW, the target traffic distribution policy for the hybrid access is to use the fixed line first, and then grab some resources from the cellular link if needed. So the target objective is not 1+1! 

> 1) "no" latency (< 1ms) & no packet loss; different link bandwidths

[Med] When do you say "end-to-end MPTCP" do you mean MPTCP is enabled by the client and the server? 

> 1b) "no" latency (< 1ms) & no packet loss; variable bandwidth

[Med] What if you had an option in the MPTCP implementation to relax TCP reliability checks on QUIC-triggered MPTCP connections? 

==
   The CPE and the Concentrator MUST establish a set of subflows that
   are maintained alive.  These subflows are used to transport UDP
   datagrams that are distributed among existent subflows.  TCP session
   tracking is not enabled for the set of subflows that are dedicated to
   transport UDP traffic.
===

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 23 Nov 2016 15:07:32 +0100

Med,
> [Med] The schema we are investigating does not include this cascaded layers. It is only UDP payload transported in plain MPTCP connections.

The cascading layers increase the CPU and byte overhead, but the 
interactions between the congestion control schemes and the reliablity 
mechanisms remains. In the experiment, the CPU is not a concern given 
that PCs were used as routers. The byte overhead influences the maximum 
efficiency of the solution but not how it degrades when there is latency 
or losses. The degration comes form the coupling between the congestion 
control and the reliability mechanisms of both QUIC and MPTCP.

> Putting that aside, can you please indicate how the traffic is distributed among available subflows? FWIW, the target traffic distribution policy for the hybrid access is to use the fixed line firs…

That's a policy issue. The experiment was whether both links can be used 
efficiently when running QUIC over MPTCP to transport a long file. If 
your policy delays the utilisation of the second link when the first is 
full, then you will get even lower performance since MPTCP will delay 
the utilisation of the subflows.

> [Med] When do you say "end-to-end MPTCP" do you mean MPTCP is enabled by the client and the server?

Yes. Given our experience with TCP/MPTCP proxies, the result would have 
been the same with TCP/MPTCP proxies running on the two routers, but 
these proxies could not be integrated in this setup for practical reasons.

> [Med] What if you had an option in the MPTCP implementation to relax TCP reliability checks on QUIC-triggered MPTCP connections?

If you disable reliability from MPTCP, you end up using a protocol that 
is very different from MPTCP. In TCP, congestion control and flow 
control are closely coupled with reliable delivery and removing this 
coupling would result in a protocol that would not be TCP anymore.

Olivier

---

From: <[email-masked]>
Date: Wed, 23 Nov 2016 17:52:52 +0000

Sebastien, Olivier,
Thanks for doing these experiments- and so quickly - and for sharing the results - very interesting.

phil

-----Original Message-----
From: Sébastien Noel [mailto:[email-masked]] 
Sent: 23 November 2016 11:09
To: Eardley,PL,Philip,TUB8 R <[email-masked]>
Cc: [email-masked]
Subject: Re: [multipathtcp] MPTCP carrying UDP

Phil,

> Do people have any experimental results /experiences they could share 
> of running UDP applications over MPTCP sub-flows?  Would be interested 
> to hear about the issues.

To understand the interactions between QUIC and an underlying MPTCP transport, we performed some experiments by running QUIC over OpenVPN that runs itself over an MPTCP connection. This is the closest scenario to what you are discussing based on existing open-source software.

OpenVPN includes some framing to carry UDP and encryption/authentication. These mechanisms add CPU overhead and byte overhead compared to transporting QUIC over a plain MPTCP connection, but this does not change the results of the experiments.

                       /-----\

We simply recorded the time needed to transfer a random 10M file from server to client while tweaking speed/latency/reliability of each link on router 1 & 2 with netem.

In the following tables, Download/Upload are in Mbps, time in seconds & performance is a percentage of a perfect theoretical result/max speed (without any IP or TCP overhead).

Link 1		| Link2		|

Link 1		| Link2			|

Link 1		| Link2			|

IMHO those results confirm the intuition that running a protocol like QUIC that includes its own congestion control and retransmission mechanisms are a reliable bytestream protocol like MPTCP is not a good idea.

As you can see, with QUIC over MPTCP sub-flows, performance seems to drop as soon as you have an unreliable medium or as soon as you have latency

The sames tests were performed again, but this time with HTTP over end-to-end MPTCTP, to have a point of comparison

Link 1		| Link2		|

This should not surprise the readers of this list, but confirms that MPTCP works well in this environment.

Link 1		| Link2			|

Given the bad results of running QUIC over MPTCP, we don't plan to analyse this in more details.

Best regards,

Sébastien

---

From: <[email-masked]>
Date: Thu, 24 Nov 2016 10:32:09 +0000

Hi Olivier,

Please see inline.

Cheers,
Med

> Envoyé : mercredi 23 novembre 2016 15:08
> À : BOUCADAIR Mohamed IMT/OLN; Sébastien Noel; [email-masked]
> of
> > [Med] The schema we are investigating does not include this cascaded
> layers. It is only UDP payload transported in plain MPTCP connections.
> 
> 
> mechanisms remains.

[Med] My point is that we need to isolate the impact of those cascaded layers on the CPU/Performance efficiency from pure interactions between the CCs.  

 In the experiment, the CPU is not a concern given
> > [Med] I wouldn't drop that conclusion as the overall performance depends
> also on the overhead prepended to packets to be injected over the tunnel.
> >
> > Putting that aside, can you please indicate how the traffic is
> distributed among available subflows? FWIW, the target traffic
> distribution policy for the hybrid access is to use the fixed line first,
> and then grab some resources from the cellular link if needed. So the
> target objective is not 1+1!
> >
> 

[Med] It is a policy, that's true... but it is a key point to decide what are acceptable performance or not. Assuming 1+1 is putting the bar too high for the deployments plans I'm aware of. 

[SNIP]

> Yes.

[Med] I wouldn't use that as a reference. In order to have comparable results the reference IMHO is to have the same cascade layers for TCP flows that will be proxied into MPTCP connections. 

 Given our experience with TCP/MPTCP proxies, the result would have
> these proxies could not be integrated in this setup for practical reasons.

[Med] It would be more interesting if similar setup conditions are used to have fair comparisons.

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 24 Nov 2016 12:24:53 +0100

Med,

> [Med] My point is that we need to isolate the impact of those cascaded layers on the CPU/Performance efficiency from pure interactions between the CCs.

Given the performance of the PCs used as router, the CPU has no impact here.

The lessons that we've learned from this experiment, and others where 
users have reported experiments with running TCP over OpenVPN running 
over MPTCP is that it is a bad idea to run a protocol that uses a 
congestion control scheme above a protocol that provides a reliable 
delivery before the reliable delivery comes from retransmissions and 
those retransmissions hide packet losses that serve as indication for 
the congestion controller. This lessons applies to QUIC and similar 
protocols.

If you have other experimental data, please share it on the list so that 
we can discuss based on real data.

Olivier

---

From: <[email-masked]>
Date: Fri, 25 Nov 2016 08:46:47 +0000

Hi Sebastien,

Thanks for sharing the results! I guess that shows quite well that "layering" protocols where both use a congestion control has an significant impact on the overall throughput.
	
regards

Nic

-----Ursprüngliche Nachricht-----
Von: multipathtcp [mailto:[email-masked]] Im Auftrag von Sébastien Noel
Gesendet: Mittwoch, 23. November 2016 12:09
An: [email-masked]
Cc: [email-masked]
Betreff: Re: [multipathtcp] MPTCP carrying UDP

Phil,

                       /-----\

		
Link 1		| Link2		|

Link 1		| Link2			|

Link 1		| Link2			|

		
Link 1		| Link2		|

Link 1		| Link2			|

Best regards,

Sébastien