Subject: Re: [Anima] ANIMA when there is a system-wide issue

From: Toerless Eckert <[email-masked]>
Date: Tue, 26 Jan 2021 20:17:46 +0100

On Mon, Nov 30, 2020 at 06:36:16PM -0500, Michael Richardson wrote:
> 
> Brian E Carpenter <[email-masked]> wrote:
>     > Perhaps there is something we should specify in ANIMA to prevent the
>     > ANIMA infrastructure falling into this sort of trap: when there is a
>     > system-wide issue (such as hitting an O/S resource limit everywhere at
>     > the same time) it also prevents the autonomic mechanisms from working.
> 
> I think you mean, that it should not also prevent the autonomic system?
> I think that key is:
>   1) allocate resources (including threads) up-front
>   2) do not dynamically allocate threads per interface, but rather use async
>     routines.

Reminds me of university times in the 80th. In my university, they had built
a network layer packet accounting system (midle box like the firewall, i let you guess the
network layer protocol). Using:

https://en.wikipedia.org/wiki/PEARL_(programming_language)

I took a semester programming PEARL. No dynamic resource management. Pain
in undisclosed body parts. But gets you into the mood to think about how
to handle fixed constraints.

Didn't help to avoid buggy software though. One month it claimed i had
used 60,000 German Marks in network resources. Turned out it was "only"
6,000 though. Programming bug related to insufficient testing with corner
situations (high numbers).

I thought Facebook praised itself by creating regularily
"A Series of Unfortunate Events" into its software to ensure the system
is continuously hardened.

> But, I don't know what we could write into the specification to make this
> happen.  It seems that we really just need smart implementers.

Unless there is documentation for what constitutes "smart", you can only count
on "experienced".  Oh wait: Networking is a commodity. "Experienced" is too
expensive for our industry.

Cheers
    Toerless

> --
> Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
>            Sandelman Software Works Inc, Ottawa and Worldwide
> 
> 
> 
> 

> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 28 Jan 2021 17:03:56 +0100

Laurent, *:

In german: "Am Ast saegen auf dem man sitzt", aka:

https://www.fotocommunity.de/photo/am-ast-saegen-auf-dem-man-sitzt-smithi/32932119

Not sure what the best english equivalent saying is. In german
there are actually multiple of these idioms.

In networking this problem is inband management of IP networks.
You use use IP network that you configure to provide you connectibity
for that configuration. Epic Fail. Thats why in Data Centers and
SPs you stil often have separate out-of-band management networks.
ACP is the best current attempt to solve this issue without the
cost overhead of a separate out-of-band-network.

The google incident is exactly solved by ACP: The ACP is absent of any
configuration. Therefore also absent of any misconfiguration.

The magic of resilience lies in the correct layering. In the best
of implementation designs, the whole ACP implementation on a node
is running on resources that nobody can misconfigure and any user
traffic and contrl plane configuration does not touch it.

This is perfectly possible by combining ACP IETF spec with appropriate
implementation techniques on the nodes. At IEEE IRIC 2016 conference,
i had some slides, showing for example, how one could consider to put
ACP into the firmware of a BMC on every node. And you could segment
the NPU as well. Not rocket science. Just better engineering than what's
done today.

Cheers
    Toerless

On Fri, Dec 18, 2020 at 08:42:02AM +0000, Ciavaglia, Laurent (Nokia - FR/Paris-Saclay) wrote:
> Hi Brian,
> 
> Thanks for sharing interesting incidents and reflecting on the role of ANIMA technologies.
> 
> Reading the report, it seems the outage results from a series of indirectly linked events 
> leading to isolation of a portion of the GCP network.
> 
> For the incident you refer here, how/where would you see ANIMA components to have (helped) avoided the outage?
> Could we expect ANIMA networks to provide better/longer data plane operation in case of failed control plane/control plane functions? Beyond the pure ability to do so, there is a gain/risk trade-off to let a DP run out of sync of its CP.
> Could we expect ANIMA components to have reacted differently and circumvent the issue, preventing the full disconnection of the GCP network portion? Or mitigated at intermediate points? 
> The incident seems to have been triggered by a legitimate/valid configuration change but with resulting with a functionality loosing its access to files it needed to perform. The config. change validation basically didn't notice there was a possible issue.
> Would such a miss been caught by ANIMA components? (e.g. via a different validation-dependencies approach?)
> 
> Why I'm a bit doubtful here is because even if we deploy robust autonomic functions/agents, the above problem seems to originate from an issue in how the system has been configured.
> And even ASAs would still need to get some form/level of initial configuration or guidance.
> 
> 
> Best regards, 
> Laurent
> 
> > -----Original Message-----
> > From: Anima <[email-masked]> On Behalf Of Brian E Carpenter
> > Sent: Thursday, December 17, 2020 02:47
> > To: Anima WG <[email-masked]>
> > Subject: Re: [Anima] ANIMA when there is a system-wide issue
> > 
> > And here's what happens when the control plane itself falls over:
> > 
> > https://status.cloud.google.com/incident/zall/20011#20011006
> > 
> > It seems pretty clear that Cloud needs ANIMA.
> > 
> > Regards
> >    Brian
> > 
> > On 01-Dec-20 11:02, Brian E Carpenter wrote:
> > > "AWS reveals it broke itself by exceeding OS thread limits"
> > >
> > > https://www.theregister.com/2020/11/30/aws_outage_explanation/
> > >
> > > Especially:
> > > "The TIFU-like post also outlines why Amazon's dashboards offered only
> > scanty info about the incident ??? because they, too, depend on a service
> > that depends on Kinesis."
> > >
> > system-wide issue (such as hitting an O/S resource limit everywhere at the
> > same time) it also prevents the autonomic mechanisms from working.
> > >
> > > Regards
> > >    Brian Carpenter
> > >
> > 

-- 
---
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Thu, 28 Jan 2021 15:35:07 -0500

> The magic of resilience lies in the correct layering. In the best of
    > implementation designs, the whole ACP implementation on a node is
    > running on resources that nobody can misconfigure and any user traffic
    > and contrl plane configuration does not touch it.

Right, so in keeping this thought, is the WG interested in designing a
way to interact with L2 switches?
In my experience, 90% of mis-configuration issues are due to L2 switch
issues.
(I think that it's because there is no "traceroute" or "ping" which
works at L2, and so it's really hard to diagnose/determine if one did the
right thing.)

My current idea is to attempt to prototype this in DPDK, but not really any
point if nobody cares about this.

Name:		draft-richardson-anima-l2-friendly-acp

was about to expire, so I have reposted it.

Html:           https://www.ietf.org/archive/id/draft-richardson-anima-l2-friendly-acp-01.html
Diff:           https://www.ietf.org/rfcdiff?url2=draft-richardson-anima-l2-friendly-acp-01

Abstract:
   This document details the challenges with building an Autonomic
   Control Plane on Campus/Enterprise networks which are built out of
   layer-two (Ethernet) switched technologies.

   This document does not propose a specific solution as yet, but
   details a number of possibilities, and what it would take to
   standardize each possibility.