Subject: Re: [Anima] Mirja Kuehlewind's Discuss on draft-ietf-anima-stable-connectivity-07: (with DISCUSS)

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Thu, 1 Feb 2018 14:42:29 +0100

HI Toerless,

thanks so much for these edits. Unfortunately I have one more question. And sorry again for the delay.

Why does your case 2 need an MPTCP connection instead of just opening a second separate TCP data plan connection (that of course fails when it fails..)?

Mirja

> Am 27.01.2018 um 00:06 schrieb Toerless Eckert <[email-masked]>:
> 
> On Fri, Jan 26, 2018 at 11:43:08AM +0100, Mirja Kuehlewind (IETF) wrote:
>> Hi Toerless,
>> 
>> sorry for my late reply but I spend only limited time on AD stuff on the telechat agenda was crowded this week.
> 
> Thanks for all the hard work!
> 
>> Okay that is fine. I mainly wanted to double-check as that was still not fully clear from the provided text. Can you maybe propose a slight rewording that explicitly say that there is this backup while using the word ???only??? above in the cited text seems to indicated differently.
> 
> I wanted to take the full extend of your feedback into account, so
> rewrote the paragraph in question maybe a bit more than slightly
> (couldn't figure out a sane "slightly" ;-):
> 
> - Moved up the last sentence about per-connection policy up front,
>  so the examples are more logical scoped.
> 
> - Made the examples into three separate bullet points
>  and expanded them to hopefully eliminate the misunderstanding
>  wrt to backup.
> 
> - Not only mentioning the fall back (backup), but also added
>  desirable behavior under recovery of preferred subflow path. 
>  (important for both 1) and 2).
> 
> - explaining in bullet point 3) how this one is actually a
>  simple TCP connection (As you pointed out).
>  (whereas 1) and 2) rely on the MPTCP benefit of two subflows and address signaling).
> 
> -09 uploaded with this fix as well as fix for Yohi's comment.
> 
> Diff:
> 
> http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-08.txt&url2=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-09.txt
> 
> Here is the changed paragraph:
> 
> The above described behavior/policy for MPTCP must be controllable by	
> applications or libraries acting on behalf of applications.  APIs
> and/or data models for this control need to be defined. It should be
> sufficient to make these policies work on a per connection	
> basis, and not change during the lifetime of a connection for	
> different data items:
> 
> 1) The policy for likely most connections would be to use the data-plane
>   subflow and fall back using the ACP subflow when the data-plane fails.
>   This could be the default. It reduces load on the ACP but would continue
>   to run connection traffic at likely reduced throughput when the
>   data-plane fails.  Ideally such connections would also revert back to
>   using a data-plane subflow once its connectivity recovers.
> 
> 2) Connections for non-urgent bulk transers (for example most routine 
>   firmware updates or cached log collection) may use a policy where
>   the connection is made to fail when the data-plane fails or have
>   transfers suspend until another data-plane subflow can successfully
>   be built. This avoids over-taxing the ACP when the data plane fails. 
> 
> 3) Connections for critical network configuration change operations 
>   known to impact the data-plane might want to only use the ACP and
>   could therefore map to a (non-MP)TCP connection.
> 
> Cheers
>    Toerless
> 
> 
> m 22.01.2018 um 19:42 schrieb Toerless Eckert <[email-masked]>:
>>> 
>>> 
>>> Thanks, Mirja. Inline.
>>> 
>>> On Mon, Jan 22, 2018 at 01:37:58PM +0100, Mirja Kuehlewind (IETF) wrote:
>>>> Hi Toerless,
>>>> 
>>>> thanks a lot for the updated text. I still have one little point I would like to discuss a bit further on this text:
>>>> 
>>>> "The above described behavior/policy for MPTCP must be controllable by	
>>>> applications or libraries acting on behalf of applications.  APIs	
>>>> and/or data models for this control need to be defined.  As outlined	
>>>> above, applications for example may choose to only perform transfers	
>>>> if the data-plane is actually available because of performance	
>>>> limitations of the GACP, so the application needs to be made aware if	
>>>> the setup of the data-plane subflow fails.  Or transfers may want to	
>>>> only use the GACP because the connection performs configuration	
>>>> changes that are likely known to bring down the data-plane.  It	
>>>> should be sufficient to make these policies work on a per connection	
>>>> basis, and not change during the lifetime of a connection for	
>>>> different data items.???
>>>> 
>>>> 
>>>> When you say that there is a need for an application to send data only on one path, today that???s not possible with MPTCP as you may fallback to the other path silently during the transmission. Yes, of course this could be changed and an extended interface could indicate this. 
>>> 
>>>> 
>>>> So my question is still what the ???only??? really means in this text. If you???d like to just indicate a preference that might be okay. If you really want to rule out the possibility to fallback to the other path, then I don???t think you need MPTCP and two separate TCP connections would be the better option.
>>>> 
>>>> Does that make sense to you?
>>> 
>>> Not really, the goal is always to leverage existing MPTCP signaling of
>>> addresses, indicating backup for flows, etc. I think two TCP flows would
>>> never be better:
>> 
>> Okay that is fine. I mainly wanted to double-check as that was still not fully clear from the provided text. Can you maybe propose a slight rewording that explicitly say that there is this backup while using the word ???only??? above in the cited text seems to indicated differently.
>> 
>> Thanks!
>> 
>>> 
>>> The GACP address is the primary identifier of a device known in DNS.
>>> Data-Plane addresses can change over time subject to operator configuration
>>> and could also pose more of a security issue to be in DNS (GACP is isolated
>>> network).
>>> 
>>> GACP performance may be quite low because some network device may route/forward
>>> it in software. data-plane routing/forwarding is expected to be fast/HW-accelerated.
>>> 
>>> With MPTCP as suggested this gives for example:
>>> Initial subflow must use GACP because thats the only "known/stable address" in DNS
>>> Then MPTCP signaling is used to indicate mutually the data-plane addresses
>>> and make GACP subflow backup.
>>> Then you transfer lets say a few Gigabyte of data (e.g.: firmware update) which
>>> goes over data-plane.
>>> Then some uncorrelated error happens and data-plane fails. This particular app
>>> should then fail because it would just slow down to insufferable slow across
>>> GACP and would overload software forwarding in GACP.
>>> 
>>> In another application connection, fallback to GACP subflow for data if dat-plane
>>> subflow fails is fine, but you would only prefer data-plane for performance.
>>> 
>>> You could consider the GACP potential software only forwarding like a very high
>>> cost of using it while data-plane is free.
>>> 
>>> If i would just use 2 * TCP instead of MPTCP i would have to reinvent a lot of
>>> MPTCP to get the address signaling etc. Right ?
>>> 
>>> Cheers
>>>   Toerless
>>>> 
>>>> Mirja
>>>> 
>>>> 
>>>> 
>>>>> Am 17.01.2018 um 03:36 schrieb Toerless Eckert <[email-masked]>:
>>>>> 
>>>>> Mirja, Yoshifumi
>>>>> 
>>>>> I just posted -08: https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-08.txt
>>>>> 
>>>>> I have reworked the MPTCP text based on your threads feedback with
>>>>> the intention to fix errors and have it answer the questions/concerns raised,
>>>>> but without otherwise changing the scope of it:
>>>>> 
>>>>> - What are they key features making MPTCP interesting 
>>>>> - How could it be used to solve the stable-connectivity issue
>>>>> - Disclaimer that THIS REQUIRES ADDITIONAL SPECIFICATION WORK
>>>>> beyond the scope of this document
>>>>> - Describe the areas of specification work required
>>>>> - API/policy, control by apps
>>>>> - dealing with dual VRF addresses
>>>>> 
>>>>> Please check diff in this URL (not complete diff of -08, just fix for your discuss/comment):
>>>>> 
>>>>> http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/14d5f9b66b8318bc160cee74ad152c0b926b4042/draft-ietf-anima-stable-connectivity/draft-ietf-anima-stable-connectivity-08.txt&url2=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/c02252710fbd7aea15aff550fb393eb36584658b/draft-ietf-anima-stable-connectivity/draft-ietf-anima-stable-connectivity-08.txt
>>>>> 
>>>>> I hope this resolves your DISCUSS/comments. 
>>>>> 
>>>>> Note that the term ACP was changed in the doc to GACP based on resolving Alvaros discus. See:
>>>>> 
>>>>> https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/master/draft-ietf-anima-stable-connectivity/08-01-alvaro-retana.txt
>>>>> 
>>>>> Thank you
>>>>>  Toerless
>>>>> 
>>>>> On Thu, Jan 11, 2018 at 03:42:53PM +0100, Mirja K?hlewind wrote:
>>>>>> Hi Toerless,
>>>>>> 
>>>>>> the point I'm wondering about is your point (b) below. Yes, you can
>>>>>> set the ACP subflow to backup but that would still mean that if the
>>>>>> other link fails, it would automatically switch over to the ACP
>>>>>> subflow (without exposing this to the upper layer). Is that what you
>>>>>> want? Because my understanding was rather that there are cases where
>>>>>> you'd probably would like to know over which link the OAM packets
>>>>>> where actually sent...?
>>>>>> 
>>>>>> Mirja
>>>>>> 
>>>>>> On 08.01.2018 22:06, Toerless Eckert wrote:
>>>>>>> Thanks, Mirja
>>>>>>> 
>>>>>>> (a) If the systems socket API does not transparently make TCP sockets
>>>>>>> to use MPTCP, then you would want a shim library. According to
>>>>>>> draft-hesmans-mptcp-socket, this is be the case on Apple (iOS).
>>>>>>> 
>>>>>>> (b) Making the ACP subflow not carry traffic after establishing the
>>>>>>> data plane subflow should easily be possible possible by setting its MP_PRIO
>>>>>>> to backup.
>>>>>>> 
>>>>>>> (c) How exactly to specify or implement the desired policy of only
>>>>>>> establishing a subflow between the ACP addresses and the data plane addresses
>>>>>>> (but not full-mesh) seem to be a subject for further spec work. It could
>>>>>>> be defined as a specific in-MPTCP policy, or it could be done via a shim
>>>>>>> library (orthogonal to (a)). draft-hesmans-mptcp-socket might be sufficient.
>>>>>>> 
>>>>>>> But in general: i would like for this (informational!) draft to just motivate
>>>>>>> the concept and not specify the solution: MPTCP would be a simple way to
>>>>>>> make "TCP" applications automatically "upgrade" from ACP to a data-plane path
>>>>>>> and switch back when data-plane fails... because MP-TCP can signal the
>>>>>>> additional data-plane addresses, establish transparently another subflow and switch
>>>>>>> traffic between the subflows - all by using the right shim-library+API or
>>>>>>> in-MP-TCP address/subflow policies - and the ability to establish subflows
>>>>>>> across two VRFs.
>>>>>>> 
>>>>>>> Cheers
>>>>>>>  Toerless
>>>>>>> 
>>>>>>> On Mon, Jan 08, 2018 at 05:18:54PM +0100, Mirja Kuehlewind (IETF) wrote:
>>>>>>>>>>> Suggested replacement text last two paragraphs of 2.1.5:
>>>>>>>>> A (shim) library for aplications maps TCP connections to MPTCP without the applications having to be aware of it.
>>>>>>>> It???s not the (shim) library/policy framework that opens/maps the TCP connection. MPTCP itself opens eventually multiple connections but exposes only one connection to the layer above. That means everything above MPTCP does not have any real control about which data is send over which connection.
>>>>>>>> 
>>>>>>>> A policy shim layer could only implement rules about which new subflows should be established when and what the priority is over which subflow data should be sent but it generally does not control which data is send of which flow. You can???t say I want this data to be sent over subflow one and this data to be sent over sub flow two.
>>>>>>>> 
>>>>>>>> I think what you want is actually a view on the different TCP connections and you try to use MPTCP only for announcing the other IP address but that is not what MPTCP meant to be.
>>>>>>>> 
>>>>>>>> Mirja
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Names in DNS use only the ACP IPv6 addresses of network devices. Thererefore, the initial MPTCP subflow will use the ACP. After it is operating, the shim libraries on both ends add their data-plane address (MPTCP ADD_ADDR) and attempt to build a new subflow between those addresses. If that (data plane) subflow is successfully built, the shim libraries could shift all traffic over this subflow which should be forwarded hardware accelerated by the network - and use the ACP subflow across the ACP solely for signaling - beause it is most resilient against failure.
>>>>>>>>> 
>>>>>>>>> This MPTCP approach is only an outline and would need to be fully speficied for interoperable implementations. It may also require extensions to MP-TCP. This mechanism must not be used without providing for encryption of subflows not running across the ACP.
>>>>>>>>> 
>>>>>>>>>>> Brian: ... can not use capital MUST NOT in an information draft (i think)
>>>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> Toerless
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 08, 2018 at 11:30:30AM +0100, Mirja Kuehlewind (IETF) wrote:
>>>>>>>>>> Hi Micheal,
>>>>>>>>>> 
>>>>>>>>>> to clarify one part below:
>>>>>>>>>> 
>>>>>>>>>>> Am 05.01.2018 um 23:30 schrieb Michael Richardson <[email-masked]>:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Mirja KÃ¼hlewind <[email-masked]> wrote:
>>>>>>>>>>>> "DNS naming is set up to provide the ACP IPv6 address of network
>>>>>>>>>>>> devices.  Unbeknownst to the application, MPTCP is used.  MPTCP
>>>>>>>>>>>> mutually discovers between the NOC and network device the data-plane
>>>>>>>>>>>> address and caries all traffic across it when that MPTCP subflow
>>>>>>>>>>>> across the data-plane can be built."
>>>>>>>>>>> Section 2.1.5 is discussion, it discusses ways in which the
>>>>>>>>>>> anticipated low performance (compared to what the box might do with its
>>>>>>>>>>> hardware accelerated forwarding).
>>>>>>>>>>> 
>>>>>>>>>>> If we have an application that needs the bandwidth of the native hardware,
>>>>>>>>>>> the connection can be initated over the ACP (that's what would be in DNS).
>>>>>>>>>>> One presumes that an MPTCP layer could then enumerate the available IPs at
>>>>>>>>>>> each end and then start off additional flows on the other destinations.
>>>>>>>>>> MPTCP adda an additional TCP flow but for the application that still looks like one flow. As I said I???m not sure if that is what you want.
>>>>>>>>>> 
>>>>>>>>>> Mirja
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> The application would have to include application security, since it would
>>>>>>>>>>> not be protected by the ACP.
>>>>>>>>>>> 
>>>>>>>>>>> Perhaps MPTCP doesn't work this way.
>>>>>>>>>>> 
>>>>>>>>>>>> However, I'm actually uncertain how this is supposed to work and what
>>>>>>>>>>>> "Unbeknownst to the application" should mean. If another address should be
>>>>>>>>>>>> signaled to the other host, this needs to be indicated by the application or at
>>>>>>>>>>>> least some kind of policy framework above MPTCP. Also MPTCP will by default use
>>>>>>>>>>>> both paths simultaneously while still looking like one connection to the
>>>>>>>>>>>> application, meaning the application has no control which path is used for
>>>>>>>>>>>> which traffic. I guess you can open a second subflow and then configure the
>>>>>>>>>>>> first subflow as backup path but I'm not sure if that's what you want (given
>>>>>>>>>>>> the application/policy framework will still not know which path is used)..?
>>>>>>>>>>>> Please provide more information about what the expected usage scenario is here.
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Michael Richardson <[email-masked]>, Sandelman Software Works
>>>>>>>>>>> -= IPv6 IoT consulting =-
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Anima mailing list
>>>>>>>>>> [email-masked]
>>>>>>>>>> https://www.ietf.org/mailman/listinfo/anima
>>>>>>>>> -- 
>>>>>>>>> ---
>>>>>>>>> [email-masked]
>>>>> 
>>>>> -- 
>>>>> ---
>>>>> [email-masked]
> 
> -- 
> ---
> [email-masked]
>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 01 Feb 2018 10:31:56 -0500

Mirja Kuehlewind (IETF) <[email-masked]> wrote:
    > Why does your case 2 need an MPTCP connection instead of just opening a
    > second separate TCP data plan connection (that of course fails when it
    > fails..)?

I think the idea is that there is some long running connection; for instance
a log flow or a mirror port flow.  Or maybe even an SDN control connection.

I believe (but I could be wrong) that MPTCP would allow a router that has
multiple data-plane circuits to (pre-)establish those flows and then quickly
switch to them.   If all data-plane flows fail, then it might suspend
traffic.

I think that there is some advantage to starting the connection on the ACP,
because:
  a) no TCP SYN listener on the data-plane interface(s) reduces attack
     surface (assuming NOC->router initialized connections, but the same
     argument applies to keeping the NOC devices isolated)

  b) the IP addresses of the router on the data-plane might be hard to know
     in advance, being subject to routing algorithms and address assignment
     ASAs. Also some interfaces might exist only with LL addresses until the
     need to grow a GUA occurs.

     The ACP addresses are essentially locked down by being embedded in
     certificate DNs. (see the ACP draft).  That also means that a TLS
     connection to the router will be validatable by the router's domain
     certificate, while the (stable? temporary?) address assigned on the dataplane
     interfaces will not.

So that's what MPTCP gets us: the ability to start on the ACP, do security,
and then move the bulk traffic to another place.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Thu, 1 Feb 2018 17:14:58 +0100

Hi Michael,

see below.

> Am 01.02.2018 um 16:31 schrieb Michael Richardson <[email-masked]>:
> 
> 

The case 2 I’ve been referring to is that:

"2) Connections for non-urgent bulk transers (for example most routine 
  firmware updates or cached log collection) may use a policy where
  the connection is made to fail when the data-plane fails or have
  transfers suspend until another data-plane subflow can successfully
  be built. This avoids over-taxing the ACP when the data plane fails.“

I think for non-urgent bulk transfers it would be no problem to open a new TCP connection, send the data, and close it again.

> switch to them.  

You can also pre-establish normal non-MPTCP connections (and send keep-alives); there is no difference or advantage when using MPTCP. However, I don’t think you need that for non-urgent bulk transfers.

> If all data-plane flows fail, then it might suspend
> traffic.

That will not happen with MPTCP; it will fail back to use the ACP connection (if there is a subflow for it).

>     argument applies to keeping the NOC devices isolated)

Also with MPTCP you need a TCP SYN lister on the remote data-plane interface because creating a MPTCP subflow means sending a TCP SYN with an MPTCP option.

>     certificate, while the (stable? temporary?) address assigned on the dataplane

What I’m basically proposing is that you use MPTCP for case 1 (data that needs a fallback to the ACP connection) but for all other transmission (that should be done only over one of the paths, either data plane or ACP), you have additional plain TCP connections to use. You can still add an interface to MPTCP to extract the IP address information you need to start these additional connections.

Mirja

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 1 Feb 2018 12:14:54 -0600

This is Mirja's Discuss thread, and MPTCP is her working group, and she
knows MPTCP better than I do, but I'm confused about one thing, as I'm
reading this discussion ...

On Thu, Feb 1, 2018 at 10:14 AM, Mirja Kuehlewind (IETF) <
[email-masked]> wrote:

> I think for non-urgent bulk transfers it would be no problem to open a new
> TCP connection, send the data, and close it again.
>
>
> You can also pre-establish normal non-MPTCP connections (and send
> keep-alives); there is no difference or advantage when using MPTCP.
> However, I don’t think you need that for non-urgent bulk transfers.
>

If I'm understanding the discussion, the desire is to use MPTCP to handle
failover to another pre-established flow quickly, and Mirja's point is that
normal MPTCP would just use both flows - so, not actually failing over, but
if you provision for each flow to carry all your traffic, which you would
have to do in order to handle failover anyway, I'm not sure why it's bad to
let MPTCP do what it would do normally, and load-share across
overprovisioned paths.

Is that the discussion people think they're having?

Thanks,

Spencer, the irresponsible AD, in this case ...

> That will not happen with MPTCP; it will fail back to use the ACP
> connection (if there is a subflow for it).
>
> >
> Also with MPTCP you need a TCP SYN lister on the remote data-plane
> interface because creating a MPTCP subflow means sending a TCP SYN with an
> MPTCP option.
>
>
> >
> What I’m basically proposing is that you use MPTCP for case 1 (data that
> needs a fallback to the ACP connection) but for all other transmission
> (that should be done only over one of the paths, either data plane or ACP),
> you have additional plain TCP connections to use. You can still add an
> interface to MPTCP to extract the IP address information you need to start
> these additional connections.
>
> Mirja
>
> >

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 1 Feb 2018 19:22:14 +0100

Inline

> Why does your case 2 need an MPTCP connection instead of just opening a second separate TCP data plan connection (that of course fails when it fails..)?

How do the peers know each others data plane address ?

=> draft text:
  (Only) the GACP address of GACP devices could to be put into DNS.
  When MPTCP builds the first subflow across the GACP, it exchanges the
  data-plane address.

Without MPTCP i need to reinvent a good part of it and i can not imagine how
i would get anyone interested to do that signaling part in a standardized
fashion given how MPTCP does exist.

-> Build TCP connection to responder GACP address.
-> Signal with responder the data-plane address - in a fashion that would
   be backward compatible with a legacy responder/initiator that doesn't
   speak this extension. I wouldn't even know how i could do this via
   TCP payload. Must be at the transport level.
-> Build TCP connection via data-plane addresses.
-> Use policy and/or new signaling to decide to prefer using that connection.
...
-> Data-plane connection fails. 
-> need some scheme to figure out how to rebuild. if - if and when
   available. Including some exponential backoff trying to reconnect. Or
   more likely in our case the data-plane address was unconfigured and gets
   reconfigured maybe differently.  (some SDN controller doing some change
   of addressing to the data-plane), So when one side signals to the other
   side that the data-plane address is gone, then no connection probing is
   necessary.  And when it is available again, maybe different one, gets
   signalled, and we're building the connection again and using it.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 1 Feb 2018 19:34:47 +0100

> reading this discussion ...
[...] 

> let MPTCP do what it would do normally, and load-share across

I have not seen Mirja rejecting the statement of the draft that we
can make one path a backup (not written out in the draft: via MPTCP MP_PRIO).

> Is that the discussion people think they're having?

I think Mirja is trying to do due diligence that the problem can not be
solved with just TCP, and i may not have been doing textually a good enough
job highlighting how the need of an MPTCP like signaling for additional
addresses (data-plane addresses) is necessary when you only have the
GACP address in DNS but actually prefer to use the data-plane (see -09
draft version, MPTCP options 1), 2)).

Cheers
    Toerless

> > What I???m basically proposing is that you use MPTCP for case 1 (data that

-- 
---
[email-masked]

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 1 Feb 2018 12:46:14 -0600

Hi, Toerless,

> can make one path a backup (not written out in the draft: via MPTCP

I'll let her follow up here. What I was asking about, was (see next)

> draft version, MPTCP options 1), 2)).

Thanks, that's helpful.

I do see the due diligence (she's a diligent AD), but I thought Mirja was
(also?) concerned that the draft was assuming that MPTCP could be made to
handle two paths in failover mode, when it actually doesn't work that way.
Perhaps I was misreading (at least some of the conversation happened while
I was in and out of China, and that never helps me do e-mail).

But I'll let you two keep typing :-)

Spencer

> > > > I think that there is some advantage to starting the connection on
> the
> > > ACP,
> > > > because:
> > > >     ASAs. Also some interfaces might exist only with LL addresses
> until
> > > the
> > > >     certificate, while the (stable? temporary?) address assigned on
> the
> > > dataplane
> > > >     interfaces will not.
> > >

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 1 Feb 2018 20:05:26 +0100

> handle two paths in failover mode, when it actually doesn't work that way.

I am not sure if/what would be missing in MP_TCP wrt. to the failover itself,
thats what signaling of addresses and and MP_PRIO are about, right ?

But the draft itself is an informational use-case validation draft, and
the notions about MP-TCP should be understood as requirements. If somethiing
is missing in MPTCP or enabllin shim-libraries on top of it, then this should
come through standards specs. draft alread says:

"Making this idea work requires additional specification work outside
 the scope of this document. "

So i hope the due diligence is mostly around whethere there is something fundamentally
wrong as opposed to try to go further down into the potentially necessary additional
specification elements (which i think are mostly about policies for MPTCP
and/or abstract APIs).

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 1 Feb 2018 20:30:27 +0100

On Thu, Feb 01, 2018 at 05:14:58PM +0100, Mirja Kuehlewind (IETF) wrote:
> > If all data-plane flows fail, then it might suspend traffic.
> 

Right. This is one of the pieces that i think would require additional
spec/extension to MPTCP - or could be handled by a shim library
that gets enough notificatations from MPTCP.

I was hoping we could move discussions about the "howto to an appropviate
draft in the MPTCP WG, eg: the to-be-renewed socket draft i mentione din
before.

> Also with MPTCP you need a TCP SYN lister on the remote data-plane interface because creating a MPTCP subflow means sending a TCP SYN with an MPTCP option.

The data-plane subflow MPTCP SYN/ACK can ony happen after the GACP
subflows ADD_ADDR, so one should be able to enable selective filtering
of data-plane SYN packets based on the address learned from the ADD_ADDR.

Can we somehow move all this discussion as mentioned above over to
some good (hopefully standards track) target draft in MPTCP ? 
Stable connectivity is really just the informative use-case discussion
draft and should be read as a requirements draft containing a
high level description of the reasons why & how to use & expand MPTCP
(and/or shim library).

> What I???m basically proposing is that you use MPTCP for case 1 (data that needs a fallback to the ACP connection) but for all other transmission (that should be done only over one of the paths, either data plane or ACP), you have additional plain TCP connections to use. You can still add an interface to MPTCP to extract the IP address information you need to start these additional connections.

See my previous reply why i think there is no difference between 1) and
2) re the need to leverage MPTCP. As said above, the main difference IMHO
is that "suspending or aborting" the connection upon failure of a subflow
is seemingly new and would require additional work for MPTCP.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Thu, 01 Feb 2018 15:19:06 -0500

> I think for non-urgent bulk transfers it would be no problem to open a
    > new TCP connection, send the data, and close it again.

Yes, but to what address does one open the new TCP connection?
The dataplane addresses are potentially ephemeral, and don't have any
TLS certificates bound to them.

    > connection (if there is a subflow for it).

okay.

    > interface because creating a MPTCP subflow means sending a TCP SYN with
    > an MPTCP option.

okay, I'm honestly ignorant of such things.
But the address of the data-plane interface doesn't have to be known.

    > What I’m basically proposing is that you use MPTCP for case 1 (data
    > that needs a fallback to the ACP connection) but for all other
    > transmission (that should be done only over one of the paths, either
    > data plane or ACP), you have additional plain TCP connections to
    > use. You can still add an interface to MPTCP to extract the IP address
    > information you need to start these additional connections.

so, you start TCP on the ACP, do security.
Then you add the data-plane interface to get the addresses.
Then you close it all and continue using the plain data-plane TCP?
Did I understand this correctly?

---

From: Michael Richardson <[email-masked]>
Date: Thu, 01 Feb 2018 15:21:18 -0500

Spencer Dawkins at IETF <[email-masked]> wrote:
    > This is Mirja's Discuss thread, and MPTCP is her working group, and
    > she knows MPTCP better than I do, but I'm confused about one thing, as
    > I'm reading this discussion ...

...

    > If I'm understanding the discussion, the desire is to use MPTCP to
    > handle failover to another pre-established flow quickly, and Mirja's
    > point is that normal MPTCP would just use both flows - so, not
    > actually failing over, but if you provision for each flow to carry all
    > your traffic, which you would have to do in order to handle failover
    > anyway, I'm not sure why it's bad to let MPTCP do what it would do
    > normally, and load-share across overprovisioned paths.

I also don't know why it would be bad.

My other question is, what happens in MPTCP is one path is significantly
faster (or less lossy) than the other path?  Won't the window open up
significantly on that path and simply attrack more traffic?

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 1 Feb 2018 22:59:50 +0100

> significantly on that path and simply attrack more traffic?

I am changing subject because i am hoping/claiming, that we
do not need this discussion to proceed stable-connectivity
draft because it explicitly excludes that option. But of course
i am very intersted in this discussion. Maybe better had on
MPTCP mailing lists...

That said:

MPTCP implementations are supposed to use couple
rfc6824 does not seem to mandate anything but just suggests to use
coupled congstion control according to rfc6356 if the policy goal
is to maximize utilization of both paths, see
https://www.ietf.org/proceedings/77/slides/mptcp-9.pdf

If i correctly understand it though, MPTCP is not really
fair, so if you have lets say 5 flows and they all happen to
share a single bottleneck link, then i think MPTCP stil gives
you as much as 5 separate flows. If for example ACP and data plane
where both hardware accelerated then this would be a possible
discussion point if we wanted to have the policy of load splitting
traffic across both ACP and data plane. SCTP should be "fairer"
in such a case if i remember correctly (i may be wrong).

In any case, i didn't think we needed to investigate these options and
their possible difficulties for the stable-connectivity use case.
That drat is solely suited on resilience and avoiding overload of
devices due to possible short term limited ACP implementations. Therefore
siple policy to always only transfer data across one subflow.

Your point of looking into best methods to  speed up switchover
between sublflows of course is valid for stable connectivity,
and maybe one of the enhanced policies to achieve this is to not
wait for connection reset of the data-plane flow but to start
sharing load on the ACP much faster, eg: jut measuring the
congestion window.

Cheers
    toerless

-- 
---
[email-masked]

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 09:53:21 +0100

Hi Toreless,

see below.

> Am 01.02.2018 um 19:22 schrieb Toerless Eckert <[email-masked]>:
> 
> Inline
> 

No. For what ever you want to do, you at least have to change something in the MPTCP interface. And you can just expose the address you’ve got from MPTCP and open a new TCP connection for the build transfer.
>   TCP payload. Must be at the transport level.

This is not a transport function. So you would need to build a minimal app protocol on top of TCP. However it you be as minimal as, if you open a TCP connection to a certain port, the other end with send you an address in the first payload packet and closes the connection again.

Changing the MPTCP interface is also not compatible with legacy systems that does support that yet.

> …

This is the part you also need with MPTCP as you need some kind of framework policy on top of MPTCP that decides when to build with subflow.

> -> need some scheme to figure out how to rebuild.

However, my reading is that is what you want in case of bulk transfers. Fail instead of fail-over. In MPTCP if one or your subflows fails it will also not be rebuild automatically; this is again part of the policy framework what to do in such a case and when and how to try to build up another subflow. 

> if - if and when

Again basically what I’m proposing it that you use MPTCP for connection where the fallback makes sense, and at the same time expose the address you’ve learned over MPTCP and build up additional TCP connection for transfers where no fallback is wanted. Using only one subflow of MPTCP is not MPTCP anymore and therefore a functionality we probably don’t want to provide with MPTCP. Also see my next mail in this point please.

Mirja

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 10:08:56 +0100

Hi Spencer,

see below

> Am 01.02.2018 um 21:21 schrieb Michael Richardson <[email-masked]>:
> 
> 
>> point is that normal MPTCP would just use both flows

No. That was not my point at all. You can use both flow simultaneously with MPTCP pr configure it to automatically failover but making one or some of the subflow backup subflows. These are other valid use cases. However for transmission where you want neither of these two options (but fails when one specific connection fails), you should not use MPTCP but a single TCP connection. 

>> - so, not

If you use coupled connection control to would sent more data over the faster path. But I don’t this was the use case here rather than automatic failover (with can be done with MPTCP as explained above).

Mirja

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 10:04:39 +0100

Hi Michael,

see below.

> Am 01.02.2018 um 21:19 schrieb Michael Richardson <[email-masked]>:
> 
> 

see below.

> But the address of the data-plane interface doesn't have to be known.

Of course you need an address.

> Did I understand this correctly?

No. you open the MPTCP connection, get the address and open the subflow. This connection you can use for any transfer where you want to fallback to the ACP connection. However, if you also have transfers where you don’t want to fallback, you expose the address you learnt over the MPTCP connection and use that to open a second (non-MPTCP) TCP connection to the same address (which then has automatically the desired feature to fail when needed). Having multiple TCP connection opens at once is the right solution here.

Mirja

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 10:00:34 +0100

Hi Toerless, 

see below again.

> Am 01.02.2018 um 20:30 schrieb Toerless Eckert <[email-masked]>:
> 

This is something that should not be done with MPTCP because that exactly not what MPTCP is meant for. In this case just use a standard TCP connection. And that’s my whole point. I see that there is a use case for MPTCP in you setup when fallback is needed but don’t use MPTCP if not needed and then don’t require changes to MPTCP that would make it behave like single TCP because that really not the design goal of MPTCP. 

> draft in the MPTCP WG, eg: the to-be-renewed socket draft i mentione din

Yes, we really do not need to design the whole solution here. But I don’t want to discuss the use of MPTCP when it is not necessary to use MPTCP and more over I don’t want to phrase requirement for MPTCP that doesn’t make sense because it would be easier to just use TCP instead.

> high level description of the reasons why & how to use & expand MPTCP

Yes, those details really don’t need to go in the draft. I was just replying to something Michael said.

> 2) re the need to leverage MPTCP.

And this is exactly my point and where we disagree.

> As said above, the main difference IMHO

I don’t this this should be a requirement for MPTCP and I don’t think this should be spell out in this draft as such because you can just use a TCP connection in this case. That’s the whole point.

Mirja

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 6 Feb 2018 03:52:57 +0100

Thanks Mirja, also for our talk earlier today.

-10 just posted.

https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-10.txt

http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-09.txt&url2=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-10.txt

As discussed today, and to summarize for the thread here:

As changelog explains, i changed the wording of the multipath text to
be a set of requirements and not to suggest specific ways how to use
MPTCP. Without changing the scope & goal of the multipath text, this
was IMHO the correct approach because through the review from Mirja and
Yoshi it became clear that there is more work to be done before there
is a mutually agreeable solution. Which i had expected from the beginning,
but the text nevertheless became too suggestive of specific ways how to 
leverage MPTCP that where non-agreeable. Fixed now!

 [ Working for a vendor, i usually have to tell customers
  "don't describe the solution, describe the requirements". Takes 
  sometimes longer to recognize when you are the customer. ]

Also added the fix about need for end-to-end encryption (as brought up
by BrianC earlier). 

Cheers
    toerless

On Fri, Feb 02, 2018 at 10:00:34AM +0100, Mirja Kuehlewind (IETF) wrote:
> This is something that should not be done with MPTCP because that exactly not what MPTCP is meant for. In this case just use a standard TCP connection. And that???s my whole point. I see that there is a use case for MPTCP in you setup when fallback is needed but don???t use MPTCP if not needed and then don???t require changes to MPTCP that would make it behave like single TCP because that really not the design goal of MPTCP. 
> 
> > 
> Yes, we really do not need to design the whole solution here. But I don???t want to discuss the use of MPTCP when it is not necessary to use MPTCP and more over I don???t want to phrase requirement for MPTCP that doesn???t make sense because it would be easier to just use TCP instead.
> 
> > 
> Yes, those details really don???t need to go in the draft. I was just replying to something Michael said.
> 
> > 
> I don???t this this should be a requirement for MPTCP and I don???t think this should be spell out in this draft as such because you can just use a TCP connection in this case. That???s the whole point.
> 
> Mirja
> 
> 
> 
> > 
> > Cheers
> >    Toerless
> > 
> >> Mirja
> >> 
> >>>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 9 Feb 2018 13:42:32 +0100

Thanks! I think know we really captured everything. Thanks for writing this up clearly! Will release my discuss now!

Mirja

> Am 06.02.2018 um 03:52 schrieb Toerless Eckert <[email-masked]>:
>

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 9 Feb 2018 21:24:33 +0100

Thanks, Mirja

-- 
---
[email-masked]