Subject: [netmod] [CCAMP][TEAS][NETMOD] Efficiency issues with YANG model data in integration

From: yuchaode <[email-masked]>
Date: Tue, 15 Mar 2022 01:30:07 +0000

Hello Friends,

I have a question about YANG data model efficiency issues discovered during integration with OSS.
This issue is also mentioned in section2.2 of draft-yg3bp-ccamp-network-inventory-yang that we are planning to present at the CCAMP WG during IETF 113.

YANG data models define schema trees such as:

module: example
   +--rw root
      +--rw leaf-a?   string
      +--rw list-b* [leaf-c]
      |  +--rw leaf-c    string
      |  +--rw list-d* [leaf-e]
      |     +--rw leaf-e    string
      |     +--rw list-f* [leaf-g]
      |        +--rw leaf-g    string
      |        +--rw leaf-h?   string
      +--rw list-i* [leaf-j]
         +--rw leaf-j    string
         +--rw leaf-k?   string

From the point of view of data model definition, it is convenient to define objects and their relationship using a tree structure.

However, from a practical perspective, relational databases, in which these objects are saved in different tables, are widely used by network controllers and OSS/BSS. In this type of implementations, the model data need to be cut into several pieces of small object data for management and this is causing some efficiency issues with the integration between systems.

For example, in full data synchronization scenario, the YANG server needs to assemble the data of each small object into a tree structure based on the YANG model defined for the communication with its YANG client(s). The YANG client then disassembles the data into its own storage structure. This assembly-disassembly process is actually time consuming, especially when the storage structures between the upper and lower system are consistent, they don’t need this assembly-disassembly process at all. This efficiency issue is more prominent in large scale networks.

RESTCONF can also support querying small objects. For example, in the above YANG model, the YANG server can provide the list of list-f instances ({{restconf}}/data/example:root/list-b={leaf-c}/list-d={leaf-e}/list-f) to a YANG client, with no need to perform any assembly-disassembly process.

However, since list-f is a child list under list-d and list-b, if you want to get access the list-f list, the list-b’s identifier (leaf-c) and the list-d’s identifier (leaf-e) shall be specified in the URI. Therefore, in the full data synchronization scenario, a loop is required to get all the elements of list-f. If there are 100 list-b instances and there’re 100 list-d instances under each list-b instance, then 100*100 requests are needed to get all of the list-f instances.
Assume that there are only 10 instances by average in each list-d instance, there are totally 100,000 list-f instances. If, instead, it would possible to retrieve all the list-f instances without specifying the list-f’s parent and grandparent’s identifier, but using RESTFUL’s pagination, each RESTFUL request, can provide far more than 10 list-f instances, regardless of whether they belong to a same list-d or list-b instance or not. If RESTFUL interface can return 100 list-f instances per request, only 1000 requests are required, one tenth of RESTCONF interface. This approach will reduce the time taken by the integration.

So my question is whether, to address this efficiency issue, the only viable option is to define the YANG model with a more flat structure, as shown below:

module: example
   +--rw root
      +--rw root-summay-information
      |  +--rw leaf-a?             string
      |  +--rw contained-list-b*   string
      |  +--rw contianed-list-i*   string
      +--rw list-b* [leaf-c]
      |  +--rw leaf-c              string
      |  +--rw contained-list-d*   string
      +--rw list-d* [leaf-c leaf-e]
      |  +--rw leaf-c              leafref  (path "/exmp:root/exmp:list-b/exmp:leaf-c")
      |  +--rw leaf-e              string
      |  +--rw contained-list-f*   string
      +--rw list-f* [leaf-c leaf-e leaf-g]
      |  +--rw leaf-c    leafref  (path "/exmp:root/exmp:list-b/exmp:leaf-c")
      |  +--rw leaf-e    leafref  (path "/exmp:root/exmp:list-d/exmp:leaf-e")
      |  +--rw leaf-g    string
      |  +--rw leaf-h?   string
      +--rw list-i* [leaf-j]
         +--rw leaf-j    string
         +--rw leaf-k?   string

The leaf-list object contained-list-* is a list of identifier reference to list-* instance. With this kind of modeling, all the list-f instances in the whole network can be retrieved without the need to specify its parent’s and grandparent’s identifiers in URI.

We would appreciate receiving your comments and suggestions on whether there are other ways to resolve these integration efficiency issues.

Thank you!

B.R.
Chaode

玉朝德  Tel: +86-15002728675
华为技术有限公司
[Huawei]

---

From: tom petch <[email-masked]>
Date: Thu, 17 Mar 2022 16:32:17 +0000

Trimming the distribution to NETMOD (if I send a query to everyone then everyone may think that everyone  else will respond so no-one does)

From: netmod <[email-masked]> on behalf of yuchaode <yuchaode=[email-masked]>
Sent: 15 March 2022 01:30
To: [email-masked]'; [email-masked]'; [email-masked]'
Cc: Fatai Zhang

module: example
   +--rw root
      +--rw leaf-a?   string
      +--rw list-b* [leaf-c]
      |  +--rw leaf-c    string
      |  +--rw list-d* [leaf-e]
      |     +--rw leaf-e    string
      |     +--rw list-f* [leaf-g]
      |        +--rw leaf-g    string
      |        +--rw leaf-h?   string
      +--rw list-i* [leaf-j]
         +--rw leaf-j    string
         +--rw leaf-k?   string

<tp>
When the data definition language that we know as YANG was being specified, the question did arise of how object-oriented it should be and the consensus was that it should not be.  Seeking to retrofit such a concept might be a bit like finding late in the day that the class definitions that have been chosen do not go high up enough the tree:-(

An interesting idea but I am not sure how feasible it will prove to be.

Tom Petch

module: example
   +--rw root

Thank you!

B.R.
Chaode

玉朝德  Tel: +86-15002728675
华为技术有限公司
[Huawei]

---

From: Randy Presuhn <[email-masked]>
Date: Thu, 17 Mar 2022 10:29:46 -0700

Hi -

On 2022-03-17 9:32 AM, tom petch wrote:
...
> I have a question about YANG data model efficiency issues discovered during integration with OSS.
...
> An interesting idea but I am not sure how feasible it will prove to be.
...

The efficiency issue (mapping an object-oriented model to a relational
implementation) occurs even with slightly object-oriented modeling
techniques like SNMP's SMI, and doesn't get substantially worse with
full-on object-oriented ones like GDMO.  (I think the problem is
actually easier if one starts from the GDMO perspective.)  Our
experience over 30 years ago was that it is tractable, but it
requires careful thought about the representation of fully distinguished
names and of values of types with complex syntax.  If you think about
it in the right way, the depth of nesting doesn't really matter.

Randy

---

From: Robert Varga <[email-masked]>
Date: Fri, 25 Mar 2022 23:16:43 +0100

> An interesting idea but I am not sure how feasible it will prove to be.

RFC6095 might be a starting point for that sort of work. I am not sure 
anybody ever implement that, though.

Regards,
Robert

---

From: Robert Varga <[email-masked]>
Date: Fri, 25 Mar 2022 23:22:10 +0100

On 15/03/2022 02:30, yuchaode wrote:
> This issue is also mentioned in section2.2 of 
> draft-yg3bp-ccamp-network-inventory-yang that we are planning to present 
> at the CCAMP WG during IETF 113.
> 
>  From the point of view of data model definition, it is convenient to 
> define objects and their relationship using a tree structure.
> 
> However, from a practical perspective, relational databases, in which 
> these objects are saved in different tables, are widely used by network 
> controllers and OSS/BSS. In this type of implementations, the model data 
> need to be cut into several pieces of small object data for management 
> and this is causing some efficiency issues with the integration between 
> systems.

[snip]

> So my question is whether, to address this efficiency issue, the only 
> viable option is to define the YANG model with a more flat structure, as 
> shown below:
> 
> module: example
>     +--rw root
> The leaf-list object contained-list-* is a list of identifier reference 
> to list-* instance. With this kind of modeling, all the list-f instances 
> in the whole network can be retrieved without the need to specify its 
> parent’s and grandparent’s identifiers in URI.

Well, this is flat, but it brings 'leafref' and some amount of XPath 
evaluation to the runtime -- btw. I assume those are 'require-instance 
true'.

Sounds like a trade-off implementers should weigh in on...

Regards,
Robert

---

From: Andy Bierman <[email-masked]>
Date: Fri, 25 Mar 2022 15:32:42 -0700

On Fri, Mar 25, 2022 at 3:17 PM Robert Varga <[email-masked]> wrote:

> > When the data definition language that we know as YANG was being
> specified, the question did arise of how object-oriented it should be and
> the consensus was that it should not be.  Seeking to retrofit such a
> concept might be a bit like finding late in the day that the class
> definitions that have been chosen do not go high up enough the tree:-(
> >
This issues way back (at least)  to the SMI-NG days circa 2002
https://tools.ietf.org/search/rfc3780

SMIv2 was never replaced for the very reasons cited in this thread.

But NETCONF was intended for configuration, and CLI is hierarchical.
So YANG is hierarchical.

It is not that hard to unfold nested lists into an SMIv2-style table.
You may have to rename key leafs and table names so they are unique.

Regards,
> Robert

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Balázs Lengyel <[email-masked]>
Date: Fri, 25 Mar 2022 23:03:21 +0000

Hello,
There have been a number of SDOs and other groups that were requiring object-oriented constructs in YANG. However we were out-voted.

Myself I am trying to map 3GPP objects to YANG (3gpp TS 32.160 section 6.2 https://www.3gpp.org/ftp/Specs/archive/32_series/32.160/32160-h30.zip).  I am trying to find a balance between mimicking object-orientation and keeping YANG readable. It is not always easy. 

I know that there are similar efforts in ITU and other organizations. Scott may have a full list of such efforts.
Regards Balazs

-----Original Message-----
From: netmod <[email-masked]> On Behalf Of Robert Varga
Sent: Friday, 25 March, 2022 23:17
To: tom petch <[email-masked]>; yuchaode <yuchaode=[email-masked]>; [email-masked]' <[email-masked]>
Cc: Fatai Zhang <[email-masked]>
Subject: Re: [netmod] [CCAMP][TEAS][NETMOD] Efficiency issues with YANG model data in integration

On 17/03/2022 17:32, tom petch wrote:
> specified, the question did arise of how object-oriented it should be 
> and the consensus was that it should not be.  Seeking to retrofit such 
> a concept might be a bit like finding late in the day that the class 

RFC6095 might be a starting point for that sort of work. I am not sure anybody ever implement that, though.

Regards,
Robert