Subject: [Anima] recursive system dependencies (Was: Re: New Version Notification for draft-trossen-rtgwg-impact-of-dlts-00.txt)

From: Toerless Eckert <[email-masked]>
Date: Thu, 10 Mar 2022 09:52:59 +0100

[adding anima]

One should not be surprised to see a lot of outages to be related to
loss of connectivity and/or control due to non-understood circular dependencies.
This is certainly true for several of the big outages reported, such as
some complex mutual automation control failure in Google a few years back,
and then the typical "routing broke, and we couldn't remotely access the
node we needed to fix... because routing broke". That issue of course
is resolveable by ACP/RFC8994 and lists a documented example of that issue
as investigated by FCC because the network was running 911 services and
was thus subject to regulation. When big OTT networks have outages, you
will of course almost never see a good root cause analysis publically because
there is no regulation that forces them to. Maybe a bit like how the banking
sector i think also has managed to keep a lot of its security issues under wraps.

We do also have in ANIMA participants interested in having those dependency
chains modelled, which would allow to discover loops that could result in blockage,
and to manage service instantiation in the order necessary to resolve dependencies
when they are resolvable. However:

I always look with marvel and dismay at those dependency resolution systems
found in linux distributions to (compile and) install packages. And that sounds
so simple that one should expect that it should not result in dependency resolution
failures when there are no actual circular ones, but in practice those issues occur
repeatedly, and one has to take really unnecessary brute-force resolution
measures even though in detailled analysis one could see a limited dependency
resolution chain, which the platform package management system just couldn't figure
out. Admittedly, this is more of a problem with more flexible systems such as
gentoo where you can have multiple versions of packages and also differently
parameterized packages(through parameterized compilation). But those are exactly
the minimum complexity requirements i would expect in an actual network services
(ever seen a netork service without per-deployment parameters ?).
So i fear that the more we're automating networks with SDN, the more we will run
in the buildup of system dependencies that can only be resolved automatically with
 "please reboot the continent wide network... now!".

Or else we could go back building simpler networks. Maybe bottom up like we
used to do instead of orchestrating a random complexity zoo like we do like to do
now in the times of automation...

Cheers
    Toerless

On Thu, Mar 10, 2022 at 06:49:07AM +0000, Dirk Trossen wrote:
> Have you seen anything like this and what do you think are the consequences?
> [DOT] What the consequences of this may be is difficult to judge IMO. Maybe an SDN-like situation, i.e., still having an IP-based control overlay which then manages the remaining IP service plane? Another possible direction to look into is using ANIMA concepts to bootstrap (and maintain) a DLT-like control overlay. But this is also why we are asking the community for possible network innovations that may help improve on the efficacy of DLTs. One approach we looked into, as an example but this still needs more work, is 'L3-based service routing' as a form of semantic routing (hence the hint to the semantic routing discussions in the draft). Here, we interpret the DLT as a service provided across the network. I could see this being provided as a baseline/bootstrap service through which to manage the (service routing) data that is being enriched over time by other services. But again, we were hoping to find more folks in the community who have thought deeper along those lines; we are ju
>  st at the start of doing so.
> 
> Thanks,
> Adrian
> 
> -----Original Message-----
> From: rtgwg <[email-masked]> On Behalf Of Dirk Trossen
> Sent: 02 March 2022 13:16
> To: Dirk Trossen <dirk.trossen=[email-masked]>; [email-masked]; [email-masked]
> Subject: RE: New Version Notification for draft-trossen-rtgwg-impact-of-dlts-00.txt
> 
> All,
> 
> We have posted an updated to the draft below at https://datatracker.ietf.org/doc/draft-trossen-rtgwg-impact-of-dlts/. 
> 
> We now welcome Mike McBride and Xinxin Fan as additional co-authors and have also added language into the introduction to clarify our initial insights being limited to PoW -based DLT systems, which lead to some characteristic communication patterns and associated inefficiencies compared to, say, PoS-based systems. 
> 
> Please provide any comments you may have on the draft and/or its insights, including any interests in contributing in its expansion (e.g., to other DLT systems).
> 
> Best,
> 
> Dirk
> 
> -----Original Message-----
> From: routing-discussion [mailto:[email-masked]] On Behalf Of Dirk Trossen
> Sent: 14 February 2022 16:16
> To: [email-masked]; [email-masked]
> Subject: FW: New Version Notification for draft-trossen-rtgwg-impact-of-dlts-00.txt
> 
> All,
> 
> We posted the draft below to continue a piece of work initiated in the Industry IoT Consortium (IIC) on understanding the "Impact of DLTs on provider networks", leading to the whitepaper at https://www.iiconsortium.org/pdf/2022-01-10-Impact-of-Distributed-Ledgers-on
> -Provider-Networks.pdf
> 
> With this draft, we solicit feedback from the wider IETF community on our insights regarding DLTs with the desire to broaden our findings with the expertise we can find here but also to capture possible network and routing innovations that may improve on the impacts we have identified.
> 
> If you have any comments or would like to contribute to this work, please do let us know, either on the list or directly to the authors.
> 
> Best,
> 
> Dirk (on behalf of the authors)
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: 14 February 2022 16:10
> To: David Guzman <[email-masked]>; Dirk Trossen <[email-masked]>
> Subject: New Version Notification for
> draft-trossen-rtgwg-impact-of-dlts-00.txt
> 
> 
> A new version of I-D, draft-trossen-rtgwg-impact-of-dlts-00.txt
> has been successfully submitted by Dirk Trossen and posted to the IETF
> repository.
> 
> Name:		draft-trossen-rtgwg-impact-of-dlts
> Revision:	00
> Title:		Impact of DLTs on Provider Networks
> Document date:	2022-02-14
> Group:		Individual Submission
> Pages:		16
> URL:
> https://www.ietf.org/archive/id/draft-trossen-rtgwg-impact-of-dlts-00.txt
> Status:
> https://datatracker.ietf.org/doc/draft-trossen-rtgwg-impact-of-dlts/
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-trossen-rtgwg-impact-of-dlts
> 
> 
> Abstract:
>    This document discusses the impact of distributed ledger technologies
>    being realized over IP-based provider networks.  The focus here lies
>    on the impact that the DLT communication patterns have on efficiency
>    of resource usage in the underlying networks.  We provide initial
>    insights into experimental results to quantify this impact in terms
>    of inefficient and wasted communication, aligned along challenges
>    that the DLT realization over IP networks faces.
> 
>    This document is intended to outline this impact but also
>    opportunities for network innovations to improve on the identified
>    impact as well as the overall service quality.  While this document
>    does not promote specific solutions that capture those opportunities,
>    it invites the wider community working on DLT and network solutions
>    alike to contribute to the insights in this document to aid future
>    research and development into possible solution concepts and
>    technologies.
> 
>    The findings presented here have first been reported within the
>    similarly titled whitepaper released by the Industry IoT Consortium
>    [IIC_whitepaper].
> 
>  
> 
> 
> 
> The IETF Secretariat
> 
> 
> _______________________________________________
> routing-discussion mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/routing-discussion
> 
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
> 
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg

-- 
---
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 11 Mar 2022 08:40:39 +1300

> loss of connectivity and/or control due to non-understood circular dependencies.

This problem is as old as distributed computing. I remember the difficulty
of restarting the CERN computer centre after a power outage back in the
1980s, when nobody knew the correct restart sequence and the dependencies
turned out to be different each time. While having great hope for
autonomic networking, I don't think we have a magic solution for this.
See draft-ciavaglia-anima-coordination from 2016, and add in the case of a
general outage. Probably ANIMA needs to restart work on that topic.

    Brian

---

From: Jon Crowcroft <[email-masked]>
Date: Fri, 11 Mar 2022 07:35:41 +0000

the cambridge distributed system was an integrated programmable architecture where routers
(called ring-ring bridges), nameservers, bootservers, fileservers, were all treated in a
coherent sand consistent fashion - no-one thought there was anything odd about this - we even
had an internet protocol (the universe data gram protocol) which allowed interworking with non
cambridge networks (indeed, we connected several universities in the UK with satellite and
later land lines)...

ref
Needham, Roger Michael; Herbert, Andrew J. (1983). The Cambridge Distributed Computing System. Addison Wesley.
also coverd in 
https://dl.acm.org/doi/10.1145/6041.6074

i have to admit that one upgrade in the mid 80s, one of my colleagues forgot about
dependencies, and managed to require the nameserver to allow the bootserver to boot - this was
tricky after power outages given the nameserver booted from the bootserver(s) - luckily, we
could rollback a version, reboot everything, fix the bug, and roll forward again

(this wasn't as idiotic as it might seem, since the nameserver was also an integral part of
how some (stateful) routing was done...so booting from a bootserver on a remote ring required
a path.....hmmmmi think i wrote a paper about this back then...)

i can't get to exciting by people thinking routers (and switches and NICs, and AI
accelerators) aren't just yet more computers....

however, there are specialisations (and aforementioned dependencies) which may require extreme
optimisations, which render some devices rather inappropriate for some kinds of
computations...

like running a renderfarm on a bunch of P4 programmable boxes might be a bit daft...

>         [DOT] What the consequences of this may be is difficult to judge 
> IMO. Maybe an SDN-like situation, i.e., still having an IP-based control 
> overlay which then manages the remaining IP service plane? Another 
> possible direction to look into is using ANIMA concepts to bootstrap (and 
> maintain) a DLT-like control overlay. But this is also why we are asking 
> the community for possible network innovations that may help improve on 
> the efficacy of DLTs. One approach we looked into, as an example but this 
> still needs more work, is 'L3-based service routing' as a form of 
> semantic routing (hence the hint to the semantic routing discussions in 
> the draft). Here, we interpret the DLT as a service provided across the 
> network. I could see this being provided as a baseline/bootstrap service 
> through which to manage the (service routing) data that is being enriched 
> over time by other services. But again, we were hoping to find more folks 
> in the community who have thought deeper along those lines; we are
> 
>       ju
> 
>         To: Dirk Trossen <dirk.trossen=[email-masked]>; 
> [email-masked]; [email-masked]
>         Subject: RE: New Version Notification for 
>         We have posted an updated to the draft below at 
> https://datatracker.ietf.org/doc/draft-trossen-rtgwg-impact-of-dlts/.
> 
>         We now welcome Mike McBride and Xinxin Fan as additional 
> co-authors and have also added language into the introduction to clarify 
> our initial insights being limited to PoW -based DLT systems, which lead 
> to some characteristic communication patterns and associated 
> inefficiencies compared to, say, PoS-based systems.
> 
>         Please provide any comments you may have on the draft and/or its 
> insights, including any interests in contributing in its expansion (e.g., 
> to other DLT systems).
> 
>         Best,
> 
>         Dirk
> 
>         From: routing-discussion 
> [mailto:[email-masked]] On Behalf Of Dirk Trossen
>         Subject: FW: New Version Notification for 
>         We posted the draft below to continue a piece of work initiated 
> in the Industry IoT Consortium (IIC) on understanding the "Impact of DLTs 
> on provider networks", leading to the whitepaper at 
> https://www.iiconsortium.org/pdf/2022-01-10-Impact-of-Distributed-Ledgers-on
>         -Provider-Networks.pdf
> 
>         With this draft, we solicit feedback from the wider IETF 
> community on our insights regarding DLTs with the desire to broaden our 
> findings with the expertise we can find here but also to capture possible 
> network and routing innovations that may improve on the impacts we have 
> identified.
> 
>         If you have any comments or would like to contribute to this 
> work, please do let us know, either on the list or directly to the 
> authors.
> 
>         Best,
>