Subject: [Anima] The Data Plane ACP vs the tunnelled ACP.

From: Michael Richardson <[email-masked]>
Date: Sun, 26 Jul 2015 21:06:07 -0400

{I often save emails that I feel deserve a longer reply aside,
and here I am at the cottage trying to empty out that folder.
This topic is still alive, so I'm replying}

"Data Plane ACP" has crept into draft-behringer-anima-autonomic-control-plane,
but the text is not yet mature enough to understand what it really is,
and how it operates.  I found this email thread from March, between Brian and
Michael that sheds some more light on it:

mbehring> The ACP is an overlay network, essentially. Each node maintains a
mbehring> VRF (or similar), and the tunnels mentioned above get plugged into
mbehring> this VRF. Inside the VRF you define a loopback address, and route
mbehring> through those tunnels. The net effect is that you have an overlay
mbehring> network, which only depends on link local addresses in the data
mbehring> plane, yet provides network wide reachability.

mbehring> So, please tell me:  - does this explanation still not answer your
mbehring> question? In which case I don't understand your concern. Or:  

brian> My concern is that I don't see why you would do this rather than just
brian> running a native ULA prefix and a regular security protocol. What does
brian> the overhead of a VRF, with routing support in every autonomic node,
brian> buy you? What failure modes and threats does it exclude, in exchange
brian> for the overhead and scaling issues?

mbehring> - does it, in which case I need to make the text clearer.

brian> I think it does need some extra words, because at the moment it only
brian> becomes clear on the second or third reading of the document as a
brian> whole. Mentioning a VRF helps a lot.

...

mbehring> One key feature of the ACP is that the traffic inside doesn't need
mbehring> to be further protected. You could of course use non-encrypted
mbehring> tunnels, and secure all protocols through it separately. In the end
mbehring> that's a tradeoff. 

brian> Yes. And with my GDNP hat on, I believe that we must assume that there
brian> are bad actors on the layer below. We'd end up with double
brian> encryption.

brian> As I said, I understand your model, I just don't understand why it's
brian> required. 

brian> (But this comes from someone who believes that VLANs are a really bad
brian> idea, too.)

later email:
mbehring> So, to your question "what does it buy me?" - you can mess up all
mbehring> addressing, routing, AAA, etc, on a device and still have access to
mbehring> it through this ACP.

I would go a bit further than Michael Behringer in describing this as a "VRF
(or similar)", and suggest that we want the ACP to be on seperate routing
engine.  I am aware of multiple ASIC forwarding engines that can not forward
traffic while being repartitioned between L2, L3, IPv4 and IPv6
allocations. (And some that didn't have VRFs for IPv6, because, well, aren't
"VRFs just to deal with multiple RFC1918 contexts?", the developers said)

A straight VRF is *not* going to work in a lot of situations.
For a VRF to work, the interface has to be up, it has to be receiving
packets, it has to have no layer-2 or layer-3 ACLs, it needs to have all
the right VLANs configured....

To get a link-layer IPsec SA going on that interface is also challenging!
It also requires many layer-2 things to work too... I think that getting
a untagged-layer-2 port-500 hole created would work.  We might even want
to decide it's always ESP-over-UDP-port-4500, which leaves us with a single
UDP port to open.  It would be ideal if that filter could be implemented
in the line cards as some kind of unremoveable bypass... but that itself has
downsides.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 26 Jul 2015 23:28:58 -0700

1. +1 on where you'd like to go. IMHO it would make sense to
document possible implementation options of the ACP with different
"VRF" approaches in some separate "implementation" draft
later on when the actual VRF draft is stable. We do have
for example the OpenDayLight Open Source work that is looking into
two models.

2.  My suggestion during the WG meeting was to move the data-plane ACP
option in the next revision of the ACP draft into an appendix,
add text explaining why we would like to rathrer remove it, and
then hopefully remove it in a revision later. That way, we have
some -xx draft version we can point to if someone wonders later (as
opposed to just some mail thread which is a lot harder to find).

As i tried to outline during the stable connectivity preso at
thursdays WG meeting:

The main issue with data-plane ACP is that with data-plane ACP,
the ASA have ONLY the addressing as the context to know when
traffic uses the ACP vs. Data-Plane. In the VRF model, an ASA
has an explicit context: "VRF". So when ASA wants to make sure
that traffic it sends or received is secured by the ACP, in the
VRF model, the ASA just sends/receives with the ACP context.

In the data-plane model, we need the following:

- Packets send and received are only protected by the ACP
  if both source AND destination address are in the ACP prefix.
  - this also means that it will be lot tougher to connect
    ACPs of multiple ANs together, eg: under mergers & aquisitions
    or the like.
- In the single Data-Plane+ACP routing protocol, case must be
  taken that no non-ACP interface has any ACP prefix or that
  any routes with ACP prefixes are injected.
- Traffic from an ACP address to a non-ACP address or vice versa
  will partially travel the ACP and therefore leverage the
  encryption resources of the ACP, but it can not be considered
  to be ACP traffic, because we can not control where the traffic
  entered or left the ACP - As opposed to the VRF model, where
  we can clearly model where ACP traffic enter/leaves: the dedicated
  NOC interfaces, where we do want it to happen.

This text, maybe improved would be what i'd suggest to prepend
the data-plane model with and it also serve as justification
why we should let go of it.

Cheers
   Toerless

> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

-- 
---
Toerless Eckert, [email-masked]

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Mon, 27 Jul 2015 07:21:07 +0000

Hi MichaelR, 

Thanks for your very analytic mail. You make a few good points, and Toerless already responded to them.

Fundamentally, as Toerless also describes, there are many more shades of grey between the two models, and we may need to describe them too. 

> engine.  

Clarification: You mean a separate routing process for the ACP address space, but running in the same context (what I call the "data plane"). Right? 

> I am aware of multiple ASIC forwarding engines that can not
> forward traffic while being repartitioned between L2, L3, IPv4 and IPv6
> allocations. 

I don't understand that statement. Can you expand? 

> (And some that didn't have VRFs for IPv6, because, well, aren't

OK, there is also our washing machine that couldn't implement it either. Do we want to support ALL devices? :-)   

> For a VRF to work, the interface has to be up, it has to be receiving packets,
> it has to have no layer-2 or layer-3 ACLs, it needs to have all the right VLANs
> configured....

OK, also that, I need to understand better. Can you educate me please? Which platforms have such restrictions? 

What does it mean " For a VRF to work, the interface has to be up"? 

> It also requires many layer-2 things to work too... I think that getting a
> untagged-layer-2 port-500 hole created would work.  We might even want
> UDP port to open.  It would be ideal if that filter could be implemented in
> the line cards as some kind of unremoveable bypass... but that itself has
> downsides.

So, I don't really understand your concerns yet in detail, but of course if this VRF model proves too complicated for many platforms, we may have to allow for other options. 

Thanks for calling that out!
Michael

---

From: Michael Richardson <[email-masked]>
Date: Mon, 27 Jul 2015 12:10:43 -0400

Michael Behringer (mbehring) <[email-masked]> wrote:
    >> I would go a bit further than Michael Behringer in describing this as
    >> a "VRF (or similar)", and suggest that we want the ACP to be on
    >> seperate routing
    >> engine.

    > Clarification: You mean a separate routing process for the ACP address
    > space, but running in the same context (what I call the "data
    > plane"). Right?

I am saying that I think that the tunnelled ACP should be routed by a process
in the control plane CPU, completely seperate from the data plane.
(As has been said, MPTCP could result in most data going via the data plane
when it has been set up)

    > I don't understand that statement. Can you expand?

For instance, some systems need to have their tCAMs need be partitioned
between L2 and L3 tables, and between IPv4 and IPv6 entries.
This is one of those activities which requires a system restart, although I'm
sure that vendor has worked hard to make it less of a big deal.
(While other patches and reconfigurations can now be applied live, so-called
"hitless" to use one vendor's terminology)

It would be nice if the ACP did not go down while the tCAMs are reset
as part of the figuring out how to implement the provided Intent... I'm sure
that there are other situations like this.

    > OK, there is also our washing machine that couldn't implement it
    > either. Do we want to support ALL devices? :-)

Some of those devices that do not support VRFs for IPv6 can do things that
some of the devices that do support VRFs for IPv6 can't do.  For instance,
receive a packet, remove two layers of 1q tags, and L3-route the packet...

    >> packets,

    > Which platforms have such restrictions?

Well, all the platforms have the restriction that the the port has to be
in some kind of L3-forwarding mode.  If the previous configuration of the
router/port is that it was used for an L2 function, how will the system know
that it should poke itself in with IPv6?
I think it no surprise that much of this ANIMA push comes from Cisco,
where the LLDP/CDP was "invented" --- I think of ANIMA as L3-CDP.

    > What does it mean " For a VRF to work, the interface has to be up"?

The interface has to be enabled.

We can discuss ways in which we might cheat --- let the tunnelled or
data-plane ACP flow in the same way that CDP/LLDP flows on a port even when
it is administratively down, but if we do that, then really what I think we
are doing is pretending that we have a third (and fourth at the other end)
machine on what otherwise looks like a point to point link.

Let me draw a diagram:
    http://www.sandelman.ca/tmp/anima/anima-acp-1.png

Shows router 1 and router 2, connected together by a cable, that goes through
a PHY (the connector on the box, along with the electronics+optics).  It
plugs into a normal data plane routing engine at each side.

If one uses an "ordinary" VRF to build a data-plane ACP, then this is all you
need, as the VRFs are implemented inside the routing engine(s).  On the other
hand the port has to be enabled for L3 routing, and there can't be any other
knobs or configurations involved that might "steal" your ACP traffic.
(For instance, per-port L2 VLANs, L2 RBridges, etc.)

Here is diagram anima-acp-2:
    http://www.sandelman.ca/tmp/anima/anima-acp-2.png

It shows that the ACP router, located in the control plane CPU, is connected
in parallel to the routing engine, and gets traffic in a different way.
One can also conceptualize this that the ACP router would have a different
L2 identity (MAC address) than the routing engine.

Now, would I implement it literally this way (with a switch behind every
port)  No, it would get implemented as a L2 bridge inside the routing engine
installed by the firmware in the line card.  (Of course, we'd need "no acp"
option to turn even that off)

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-