YANG Catalog Latest Development (IETF 98 Hackathon)

Before delving into the hackathon results, let’s briefly review the YANG « state of the union ». As you know, YANG became the standard data modeling language of choice. Not only is it used by the IETF for specifying models, but also in many Standard Development Organizations (SDOs), consortia, and open-source projects: the IEEE, the Broadband Forum (BFF), DMTF, MEF, ITU, OpenDaylight, Open ROADM, Openconfig, sysrepo, and more. Here is a nice summary presentation “SDN and Metrics from SDOs” by Dave Ward during the Open Networking Summit 2017.

This data model-driven management applies throughout the automation « stack »: from data plane programmability with the fd.io honeycomb open-source project that exposes YANG models for VPP functionality via NETCONF and RESTCONF, to operating systems with the sysrepo open-source project (Sysrepo is a YANG-based configuration and operational datastore for Unix/Linux applications), to network control via the networking specifications in IETF/IEEE/BBF/openconfig/etc., to service models specification (YANG Data Model for L3VPN Service Delivery as an example), to controller and orchestrator and open-source projects such as openconfig), to cloud and virtualization management, without forgetting the orchestration and policy aspects (for  example, the MEF Livecycle Service Orchestration).

With the rise of data modeling-driven management and the success of YANG as a key piece comes a challenge: the entire industry develops YANG models but those models must work together in order for operators to automate coherent services. And they must work together NOW. We don’t have the luxury to work in well planned sequences and all the modules. On one side, the YANG modules are constantly improved, and on the other side, they depend on each others.

In order to resolve this challenge, we’ve been working during IETF hackathons to provide the right open-source tools for the industry. Previous results after the IETF 97 have been highlighted.   At this IETF, a team of around 10 people went one step further by integrating tools around a YANG catalog.

Note: I went the easy way to create this blog with « we » as opposed to stress individual achievements, but many thanks to Joe Clarke, William Lupton, Einar Nilsen-Nygaard, Gary Wu, Mahesh Jethanandani, Radek Kreji,  Sudhir Rustogi, Abhi Ramesh Keshav, Carl Moberg, Rob Wilton, Miroslav Kovac, Vladimir Vassilev, and more.

What is the idea behind a YANG catalog?

From a high-level point of this YANG catalog goal is become a reference for all YANG modules available in the industry, for both YANG developers (to search on what exists already) and for operators (to discover the more mature YANG models to automate services). This YANG catalog should not only contain pointers to the YANG modules themselves, but also contains metadata related to those YANG modules: What is the module type (service model or not?) What is the maturity level? (for the IETF: is this a RFC, a working group document or an individual draft?), Is this module implemented? Who is the contact? Is there open-source code available? And we expect much more in the future. The industry starts to understand that the metadata related to those YANG modules become equally important as the YANG modules themselves. We based on work on openconfig catalog, as a starting point but we realized that we have slightly different goals.

The YANG catalog added value, compared to a normal Github repository resides in the toolchain and the additional metadata:

• the ability to validate YANG modules (including IETF drafts) with multiple validators.
• the related metadata regarding implementation
• the ability to visualize the dependencies between YANG modules, including the bottleneck in case of standardization
• the search capabilities on any YANG type and metadata, based on http://yangcatalog.org/yang-search/yang-search.php. Avoiding this way, models or model parts redefinition, which is costly to integrate.
• the REST APIs to query and post any content
• the demonstration of data model-driven management with open source tools:
  YANG Explorer (a GUI-based tool to explore modules, generate some code, and connect the devices)
  YANG Development Kit (a more advanced tool for code generation)

Using this one-stop set of tools, the typical flow for a YANG module designer is to validate the YANG module and to populate the YANG catalog (via an IETF drafts, via Github, or directly via the YANG catalog).

And the typical flow for a YANG module user is to search for an existing YANG module, to look up the metadata (such as maturity level, implementation, etc.), and to look up the import and include dependencies if any. Once the YANG module of choice is found, the YANG module user would browse the YANG module content, then load the YANG module in the YANG Explorer and test it by connecting to a NETCONF or RESTCONF server, and finally generate python scripts to start the automation.

This is obviously work in progress and contributions are welcome, as everything is open-source.

Practically, what have we done during this IETF 98 hackathon?

The YANG validator improved, with yanglint as an additional validator and with specific plugins that will check the correct format for the IEEE, MEF, BBF, and Cisco-specific plugins (practically, they will check the urn and prefix).

The YANG DB search laid a framework for the multi-SDO impact analysis, including a color scheme for the standard maturity levels.  Below is an example.

Regarding YANG module validation, William Lupton from the Broadband Forum added all the BBF modules to the YANG catalog, including the work-in-progress ones (this means 192 modules in total). While validating those modules, we discovered some issues with some validators. Those issues are now solved and most BBF YANG modules validate correctly. Finally, William updated the YANG catalog With the BBF modules maturity level that will distinguish draft and ratified YANG modules. The example here BBF work-in-progress YANG modules depending on the IETF interface YANG module [RFC 7223].

We created a script to push, as a cronjob, all extracted YANG modules from IETF drafts into Github (we still have to integrate it in the tool chain btw). Background: it should be a priority for the IETF to separate the draft text from its « code » content, such as a YANG module, so that we don’t have to extract YANG module via tooling, and so that the YANG module could progress in Github independently of the draft version.

Another major achievement for this hackathon is the integration of the YANG Development Kit into the YANG Explorer. In this example below, you see the YANG Explorer with the GUI on the left-hand side and the generated python script.

In term of YANG module transition, we also created a script to help with the latest IETF YANG module guideline: a script to convert the -config/-state into a combined tree, to convert from a combined tree to generating an additional -state tree, and to convert from a combined tree to generating an Openconfig style tree. This will be pretty handy for the transition phase, and a good addition to the toolchain.

I have personally great hope for this YANG catalog, as it will solve a real issue in this industry. Now, we should not wait until the next hackathon to continue improving it. « Running code and consensus » I heard someone say!

And finally here is Packet Pushers podcast “YANG Models & Telemetry At IETF 98”, recorded at the very end of this IETF week. Always a pleasure to speak with Ethan Banks.