Subject: [Anima] Mirja Kühlewind's Discuss on draft-ietf-anima-stable-connectivity-07: (with DISCUSS)

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 05 Jan 2018 10:23:36 -0800

Mirja Kühlewind has entered the following ballot position for
draft-ietf-anima-stable-connectivity-07: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-anima-stable-connectivity/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Section 2.1.5 talks about use of MPTCP:
"DNS naming is set up to provide the ACP IPv6 address of network
   devices.  Unbeknownst to the application, MPTCP is used.  MPTCP
   mutually discovers between the NOC and network device the data-plane
   address and caries all traffic across it when that MPTCP subflow
   across the data-plane can be built."
However, I'm actually uncertain how this is supposed to work and what
"Unbeknownst to the application" should mean. If another address should be
signaled to the other host, this needs to be indicated by the application or at
least some kind of policy framework above MPTCP. Also MPTCP will by default use
both paths simultaneously while still looking like one connection to the
application, meaning the application has no control which path is used for
which traffic. I guess you can open a second subflow and then configure the
first subflow as backup path but I'm not sure if that's what you want (given
the application/policy framework will still not know which path is used)..?
Please provide more information about what the expected usage scenario is here.

---

From: Michael Richardson <[email-masked]>
Date: Fri, 05 Jan 2018 17:30:35 -0500

> across the data-plane can be built."

Section 2.1.5 is discussion, it discusses ways in which the
anticipated low performance (compared to what the box might do with its
hardware accelerated forwarding).

If we have an application that needs the bandwidth of the native hardware,
the connection can be initated over the ACP (that's what would be in DNS).
One presumes that an MPTCP layer could then enumerate the available IPs at
each end and then start off additional flows on the other destinations.

The application would have to include application security, since it would
not be protected by the ACP.

Perhaps MPTCP doesn't work this way.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 6 Jan 2018 13:21:47 +1300

I think Mirja is right. The current text doesn't really make sense,
but I think it's mainly an issue of English. However, the notion
that MPTCP might spontaneously and silently decide to bypass ACP
security by adding a data plane address pair is a major issue that
requires a very large red flag. Possibly even a MUST NOT?

(Mirja suggested a policy framework above MPTCP. Is there any such
framework defined? And in any case, do we want *autonomic* network
management to be dependent on some external traditional policy
mechanism? I don't think so.)

Regards
   Brian

On 06/01/2018 11:30, Michael Richardson wrote:
> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima
>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Mon, 8 Jan 2018 11:33:49 +0100

Hi Brian,

no there is no policy framework defined. There has be discussion about path selection in the working group but often it really depends on the application.

Mirja

> Am 06.01.2018 um 01:21 schrieb Brian E Carpenter <[email-masked]>:
>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Mon, 8 Jan 2018 11:30:30 +0100

Hi Micheal,

to clarify one part below: 

> Am 05.01.2018 um 23:30 schrieb Michael Richardson <[email-masked]>:
> 
> 

MPTCP adda an additional TCP flow but for the application that still looks like one flow. As I said I’m not sure if that is what you want.

Mirja

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 8 Jan 2018 16:47:25 +0100

Let me know what you think:

>> 2.5.1 third to last paragraph:...
>> For example, a set-up for non-MPTCP aware applications could be as follows:
>>
>> Suggested replacement text last two paragraphs of 2.1.5:

A (shim) library for aplications maps TCP connections to MPTCP without the applications having to be aware of it. Names in DNS use only the ACP IPv6 addresses of network devices. Thererefore, the initial MPTCP subflow will use the ACP. After it is operating, the shim libraries on both ends add their data-plane address (MPTCP ADD_ADDR) and attempt to build a new subflow between those addresses. If that (data plane) subflow is successfully built, the shim libraries could shift all traffic over this subflow which should be forwarded hardware accelerated by the network - and use the ACP subflow across the ACP solely for signaling - beause it is most resilient against failure.

This MPTCP approach is only an outline and would need to be fully speficied for interoperable implementations. It may also require extensions to MP-TCP. This mechanism must not be used without providing for encryption of subflows not running across the ACP.

>> Brian: ... can not use capital MUST NOT in an information draft (i think)
>>

Cheers
    Toerless

On Mon, Jan 08, 2018 at 11:30:30AM +0100, Mirja Kuehlewind (IETF) wrote:
> MPTCP adda an additional TCP flow but for the application that still looks like one flow. As I said I???m not sure if that is what you want.
> 
> Mirja
> 
> 
> > 

-- 
---
[email-masked]

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Mon, 8 Jan 2018 17:18:54 +0100

Hi Toerless,

I don’t think that’s how MPTCP works. See blow.

> Am 08.01.2018 um 16:47 schrieb Toerless Eckert <[email-masked]>:
> 
> A (shim) library for aplications maps TCP connections to MPTCP without the applications having to be aware of it.

It’s not the (shim) library/policy framework that opens/maps the TCP connection. MPTCP itself opens eventually multiple connections but exposes only one connection to the layer above. That means everything above MPTCP does not have any real control about which data is send over which connection.

A policy shim layer could only implement rules about which new subflows should be established when and what the priority is over which subflow data should be sent but it generally does not control which data is send of which flow. You can’t say I want this data to be sent over subflow one and this data to be sent over sub flow two.

I think what you want is actually a view on the different TCP connections and you try to use MPTCP only for announcing the other IP address but that is not what MPTCP meant to be.

Mirja

> Names in DNS use only the ACP IPv6 addresses of network devices. Thererefore, the initial MPTCP subflow will use the ACP. After it is operating, the shim libraries on both ends add their data-plane address (MPTCP ADD_ADDR) and attempt to build a new subflow between those addresses. If that (data plane) subflow is successfully built, the shim libraries could shift all traffic over this subflow which should be forwarded hardware accelerated by the network - and use the ACP subflow across the ACP solely for signaling - beause it is most resilient against failure.
>

---

From: Michael Richardson <[email-masked]>
Date: Mon, 08 Jan 2018 12:09:30 -0500

> MPTCP adda an additional TCP flow but for the application that still
    > looks like one flow. As I said I’m not sure if that is what you want.

I think that this might be exactly what the application might want.
As stable-connectivity doesn't detail the requirements of the application,
the point of this section is to point out that the bandwidth limitations of
the ACP need not be a constraint when the network is healthy.

If you feel that 2.1.5 doesn't explain this well, then please say so.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 9 Jan 2018 07:30:05 +1300

> the ACP need not be a constraint when the network is healthy.

But that ducks the security issue. I don't think that's OK. Since the ACP
secures connections *below* TCP and MPTCP allows paths that bypass the ACP,
this is a security hole you can drive a large truck through.

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Mon, 08 Jan 2018 15:19:00 -0500

> this is a security hole you can drive a large truck through.

Yes, AFAIK, MPTCP doesn't just appear without awareness of the application.

Use of MPTCP implies application layer security if there are non-ACP paths.

I think that 95% of ASAs will live in a container/network-namespace/VRF that
sees ACP interfaces only.  Those that do not need to take care anyway.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 8 Jan 2018 22:06:17 +0100

Thanks, Mirja

(a) If the systems socket API does not transparently make TCP sockets
to use MPTCP, then you would want a shim library. According to
draft-hesmans-mptcp-socket, this is be the case on Apple (iOS).

(b) Making the ACP subflow not carry traffic after establishing the 
data plane subflow should easily be possible possible by setting its MP_PRIO
to backup.

(c) How exactly to specify or implement the desired policy of only
establishing a subflow between the ACP addresses and the data plane addresses
(but not full-mesh) seem to be a subject for further spec work. It could
be defined as a specific in-MPTCP policy, or it could be done via a shim
library (orthogonal to (a)). draft-hesmans-mptcp-socket might be sufficient.

But in general: i would like for this (informational!) draft to just motivate 
the concept and not specify the solution: MPTCP would be a simple way to
make "TCP" applications automatically "upgrade" from ACP to a data-plane path 
and switch back when data-plane fails... because MP-TCP can signal the
additional data-plane addresses, establish transparently another subflow and switch
traffic between the subflows - all by using the right shim-library+API or
in-MP-TCP address/subflow policies - and the ability to establish subflows
across two VRFs.

Cheers
    Toerless

On Mon, Jan 08, 2018 at 05:18:54PM +0100, Mirja Kuehlewind (IETF) wrote:
> It???s not the (shim) library/policy framework that opens/maps the TCP connection. MPTCP itself opens eventually multiple connections but exposes only one connection to the layer above. That means everything above MPTCP does not have any real control about which data is send over which connection.
> 
> A policy shim layer could only implement rules about which new subflows should be established when and what the priority is over which subflow data should be sent but it generally does not control which data is send of which flow. You can???t say I want this data to be sent over subflow one and this data to be sent over sub flow two.
> 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 9 Jan 2018 09:31:01 +0100

On Mon, Jan 08, 2018 at 03:19:00PM -0500, Michael Richardson wrote:
> Yes, AFAIK, MPTCP doesn't just appear without awareness of hearsay from folks i
talked to and e.g.: the intro of draft-hesmans-mptcp-socket, and that states
for example that MP-TCP in linux kernel does seem to work without knowledge of
the (TCP) app.

Time to enable MP-TCP myself and play around with it ;-))

> Use of MPTCP implies application layer security if there are non-ACP paths.

Right. I did include that based on Brians comment into the proposed text in my
last email to Mirja.

> sees ACP interfaces only.  Those that do not need to take care anyway.

Good question... Need to design more example ASA to know which of them
would need access to data-plane. Certainly the ones that want to monitor the
data-plane (IP-SLA like ASA).

Cheers
    Toerless

---

From: Michael Richardson <[email-masked]>
Date: Tue, 09 Jan 2018 08:57:14 -0500

> data-plane (IP-SLA like ASA).

Depends upon what kind of monitoring; it could all be available via counters
and controls which are "remote" a la SNMP.  I think that a lot depends upon
the router OS.

I agree that we need more examples.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 09 Jan 2018 16:01:54 -0500

> in-MP-TCP address/subflow policies - and the ability to establish subflows

Perhaps we need some examples to motivate why the ACP wouldn't be enough.
Some obvious ones to me are:
    a) firmware updates
    b) transfering VM images onto the
       target equipment --- the ASA is a priviledged domain in a hypervisor
       system.
    c) things like mirror port/captures that need to be offloaded quickly
       as they can not be stored.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 10 Jan 2018 11:29:23 +1300

>        as they can not be stored.

Isn't that discussion broader in scope than the stable-connectivity draft?

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 09 Jan 2018 18:31:46 -0500

> Isn't that discussion broader in scope than the stable-connectivity draft?

Maybe.... I am suggesting that maybe the motivation would help explain the
utility of the connectivity.

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 11 Jan 2018 15:42:53 +0100

Hi Toerless,

the point I'm wondering about is your point (b) below. Yes, you can set 
the ACP subflow to backup but that would still mean that if the other 
link fails, it would automatically switch over to the ACP subflow 
(without exposing this to the upper layer). Is that what you want? 
Because my understanding was rather that there are cases where you'd 
probably would like to know over which link the OAM packets where 
actually sent...?

Mirja

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 17 Jan 2018 03:36:55 +0100

Mirja, Yoshifumi

I just posted -08: https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-08.txt

I have reworked the MPTCP text based on your threads feedback with
the intention to fix errors and have it answer the questions/concerns raised,
but without otherwise changing the scope of it:

- What are they key features making MPTCP interesting 
- How could it be used to solve the stable-connectivity issue
- Disclaimer that THIS REQUIRES ADDITIONAL SPECIFICATION WORK
  beyond the scope of this document
- Describe the areas of specification work required
  - API/policy, control by apps
  - dealing with dual VRF addresses

Please check diff in this URL (not complete diff of -08, just fix for your discuss/comment):

http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/14d5f9b66b8318bc160cee74ad152c0b926b4042/draft-ietf-anima-stable-connectivity/draft-ietf-anima-stable-connectivity-08.txt&url2=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/c02252710fbd7aea15aff550fb393eb36584658b/draft-ietf-anima-stable-connectivity/draft-ietf-anima-stable-connectivity-08.txt

I hope this resolves your DISCUSS/comments. 

Note that the term ACP was changed in the doc to GACP based on resolving Alvaros discus. See:

https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/master/draft-ietf-anima-stable-connectivity/08-01-alvaro-retana.txt

Thank you
    Toerless

On Thu, Jan 11, 2018 at 03:42:53PM +0100, Mirja K?hlewind wrote:
> Hi Toerless,
> 
> the point I'm wondering about is your point (b) below. Yes, you can
> set the ACP subflow to backup but that would still mean that if the
> other link fails, it would automatically switch over to the ACP
> subflow (without exposing this to the upper layer). Is that what you
> want? Because my understanding was rather that there are cases where
> you'd probably would like to know over which link the OAM packets
> where actually sent...?
> 
> Mirja
> 
> >>>>>Mirja KÃ¼hlewind <[email-masked]> wrote:

-- 
---
[email-masked]

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Mon, 22 Jan 2018 13:37:58 +0100

Hi Toerless,

thanks a lot for the updated text. I still have one little point I would like to discuss a bit further on this text:

"The above described behavior/policy for MPTCP must be controllable by	
 applications or libraries acting on behalf of applications.  APIs	
 and/or data models for this control need to be defined.  As outlined	
 above, applications for example may choose to only perform transfers	
 if the data-plane is actually available because of performance	
 limitations of the GACP, so the application needs to be made aware if	
 the setup of the data-plane subflow fails.  Or transfers may want to	
 only use the GACP because the connection performs configuration	
 changes that are likely known to bring down the data-plane.  It	
 should be sufficient to make these policies work on a per connection	
 basis, and not change during the lifetime of a connection for	
 different data items.“

The idea of MPTCP is really to use multiple network path without the need for the high-layer to care about this. Yes, of course there are cases where one or another path is preferred (e.g. because of cost) but usually this is rather a preference than a hard requirement given MPTCP's goal really is to keep connection up with the best performance available. 

When you say that there is a need for an application to send data only on one path, today that’s not possible with MPTCP as you may fallback to the other path silently during the transmission. Yes, of course this could be changed and an extended interface could indicate this. However, why don’t you just use two different TCP connection in this case. You could use a (new) higher level protocol to announce the second address over the first connection and then start a second new TCP connection. Or you could potentially even use the MPTCP option and expose that address to the application that then start a new TCP connection but I’m not sure if that’s a clean solution...

So my question is still what the „only“ really means in this text. If you’d like to just indicate a preference that might be okay. If you really want to rule out the possibility to fallback to the other path, then I don’t think you need MPTCP and two separate TCP connections would be the better option.

Does that make sense to you?

Mirja

> Am 17.01.2018 um 03:36 schrieb Toerless Eckert <[email-masked]>:
> 
> Mirja, Yoshifumi
>

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 22 Jan 2018 19:42:50 +0100

Thanks, Mirja. Inline.

On Mon, Jan 22, 2018 at 01:37:58PM +0100, Mirja Kuehlewind (IETF) wrote:
>  different data items.???
> When you say that there is a need for an application to send data only on one path, today that???s not possible with MPTCP as you may fallback to the other path silently during the transmission. Yes, of course this could be changed and an extended interface could indicate this. 

> 
> So my question is still what the ???only??? really means in this text. If you???d like to just indicate a preference that might be okay. If you really want to rule out the possibility to fallback to the other path, then I don???t think you need MPTCP and two separate TCP connections would be the better option.
> 

Not really, the goal is always to leverage existing MPTCP signaling of
addresses, indicating backup for flows, etc. I think two TCP flows would
never be better:

  The GACP address is the primary identifier of a device known in DNS.
  Data-Plane addresses can change over time subject to operator configuration
  and could also pose more of a security issue to be in DNS (GACP is isolated
  network).

  GACP performance may be quite low because some network device may route/forward
  it in software. data-plane routing/forwarding is expected to be fast/HW-accelerated.

With MPTCP as suggested this gives for example:
  Initial subflow must use GACP because thats the only "known/stable address" in DNS
  Then MPTCP signaling is used to indicate mutually the data-plane addresses
  and make GACP subflow backup.
  Then you transfer lets say a few Gigabyte of data (e.g.: firmware update) which
  goes over data-plane.
  Then some uncorrelated error happens and data-plane fails. This particular app
  should then fail because it would just slow down to insufferable slow across
  GACP and would overload software forwarding in GACP.

In another application connection, fallback to GACP subflow for data if dat-plane
subflow fails is fine, but you would only prefer data-plane for performance.

You could consider the GACP potential software only forwarding like a very high
cost of using it while data-plane is free.

If i would just use 2 * TCP instead of MPTCP i would have to reinvent a lot of
MPTCP to get the address signaling etc. Right ?

Cheers
    Toerless

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 26 Jan 2018 11:43:08 +0100

Hi Toerless,

sorry for my late reply but I spend only limited time on AD stuff on the telechat agenda was crowded this week.

See below.

> Am 22.01.2018 um 19:42 schrieb Toerless Eckert <[email-masked]>:
> 
> 
> Thanks, Mirja. Inline.
> 

Okay that is fine. I mainly wanted to double-check as that was still not fully clear from the provided text. Can you maybe propose a slight rewording that explicitly say that there is this backup while using the word ‚only‘ above in the cited text seems to indicated differently.

Thanks!

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 27 Jan 2018 00:06:51 +0100

> sorry for my late reply but I spend only limited time on AD stuff on the telechat agenda was crowded this week.

Thanks for all the hard work!

> Okay that is fine. I mainly wanted to double-check as that was still not fully clear from the provided text. Can you maybe propose a slight rewording that explicitly say that there is this backup while using the word ???only??? above in the cited text seems to indicated differently.

I wanted to take the full extend of your feedback into account, so
rewrote the paragraph in question maybe a bit more than slightly
(couldn't figure out a sane "slightly" ;-):

- Moved up the last sentence about per-connection policy up front,
  so the examples are more logical scoped.

- Made the examples into three separate bullet points
  and expanded them to hopefully eliminate the misunderstanding
  wrt to backup.

- Not only mentioning the fall back (backup), but also added
  desirable behavior under recovery of preferred subflow path. 
  (important for both 1) and 2).

- explaining in bullet point 3) how this one is actually a
  simple TCP connection (As you pointed out).
  (whereas 1) and 2) rely on the MPTCP benefit of two subflows and address signaling).

-09 uploaded with this fix as well as fix for Yohi's comment.

Diff:

http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-08.txt&url2=https://tools.ietf.org/id/draft-ietf-anima-stable-connectivity-09.txt

Here is the changed paragraph:

The above described behavior/policy for MPTCP must be controllable by	
applications or libraries acting on behalf of applications.  APIs
and/or data models for this control need to be defined. It should be
sufficient to make these policies work on a per connection	
basis, and not change during the lifetime of a connection for	
different data items:

1) The policy for likely most connections would be to use the data-plane
   subflow and fall back using the ACP subflow when the data-plane fails.
   This could be the default. It reduces load on the ACP but would continue
   to run connection traffic at likely reduced throughput when the
   data-plane fails.  Ideally such connections would also revert back to
   using a data-plane subflow once its connectivity recovers.

2) Connections for non-urgent bulk transers (for example most routine 
   firmware updates or cached log collection) may use a policy where
   the connection is made to fail when the data-plane fails or have
   transfers suspend until another data-plane subflow can successfully
   be built. This avoids over-taxing the ACP when the data plane fails. 

3) Connections for critical network configuration change operations 
   known to impact the data-plane might want to only use the ACP and
   could therefore map to a (non-MP)TCP connection.

Cheers
    Toerless

m 22.01.2018 um 19:42 schrieb Toerless Eckert <[email-masked]>:
> Okay that is fine. I mainly wanted to double-check as that was still not fully clear from the provided text. Can you maybe propose a slight rewording that explicitly say that there is this backup while using the word ???only??? above in the cited text seems to indicated differently.
> 
> Thanks!
> 
> > 

-- 
---
[email-masked]