Subject: [yang-doctors] lighten the process?

From: Andy Bierman <[email-masked]>
Date: Tue, 25 Jul 2017 16:35:11 -0700

Hi,

I missed the plenary, but I heard the topic of YANG doctors came up.
Not sure what transpired but I suspect it might be that YD is
a bottleneck and holding everyone back.

IMO we should radically change the YD review procedures.
We should focus on more online tools and more validation checks in tools.

The problem is that people expect YD to examine their module
in fine detail and help perfect it for the domain-specific problem at hand.
This is always going to be slow and time-intensive and it will keep
getting worse as more doctors quit because it is too much work.

We should ignore the module for domain-specific correctness.
This is not our problem and we do not have enough cycles or expertise.

We should only review the YANG usage to sanity check it for
reuse/duplication and 6087bis adherence.
Eventually these checks should all be done by software.
(Well, maybe not description-stmt proofing, which is done by RFC Editor
anyway).

Then YD only has to focus on  exception handling and proactive consultation.
MIB Police had its day, but the YANG language and YANG tools are much
better now than SMIv2.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Wed, 26 Jul 2017 00:36:35 +0000

Agreed, YDs cannot be expected to be domain experts in anything but YANG.

Some of the tooling you describe is mentioned in Benoit's YD minutes under the "Instance document" heading.  The idea here is that tooling could:

·         extract instance examples in drafts.   How?  Should we document something like EXAMPLE-BEGIN/END in Section 3.12 in rfc6087bis...of course, after raising it on the list?

·         validate extracted instance examples.  (tools can do this already)

·         generate something like node-coverage statistics, percentage of the nodes in a module (inc. rpcs/actions and notifications) that appear in all the extract instance examples, regardless if the example passed validation or not  (we need to write a tool to do this)

·         ensure import statements include a 'reference' substatement.  Should we document this as best practice in rfc6087bis?

·         ensure RFC's referenced inside a module file also appear as references in the draft itself.  This is something that would have to be done in as an update to datatracker.

These are things that Juergen found in a YD review.  Were tools in place, the draft-author likely would've caught the issues before it ever got to the YD.  We should constantly improve our tooling to automate as much as possible.  Not only does it scale better, but its also more consistent and less error-prone.

Kent

On 7/25/17, 7:35 PM, "yang-doctors on behalf of Andy Bierman" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Hi,

We should only review the YANG usage to sanity check it for
reuse/duplication and 6087bis adherence.
Eventually these checks should all be done by software.
(Well, maybe not description-stmt proofing, which is done by RFC Editor anyway).

Andy

---

From: Jan Lindblad <[email-masked]>
Date: Wed, 26 Jul 2017 14:35:10 +0200

Every aspect of YANG modeling that can be validated automatically certainly should be. For all the reasons mentioned, and more.

That said, I believe that completely automatic "YD review" will remain impractical/insufficient for a long time to come. Many important classes of issues that I and other YDs have been pointing out are beyond automatic tools of the kind we have been using so far. We'd need to bring in AI or big-data pattern analysers to get coverage.

Some random examples:

- ietf-l3vpn-svc.yang (now obsolete RFC 8049) was using a legal but very unfortunate choice of key for ACL rules. The choice also differed from how this was done in ietf-acl.yang
- draft-ietf-trill-yang-oam-05.yang had a choice statement with the closing brace in the wrong location, making three separate cases of something that should have been a single case with three elements.
- it was also using a number of legal must expressions, but with a quite different meaning than the authors intended.
- ietf-conn-oam.yang had some unqualified identityref equality comparisons in when statements. Not likely to work/be interoperable.
- Lada's review of draft-ietf-i2rs-yang-l3-topology-10.yang a few minutes ago showed that network type would be better modeled using identities
- and that router-id should be dotted-quad, not ip-address to get the desired semantics
- almost every model slated for review has a lot of optional leafs with no description of what happens if the leaf isn't specified. Many should probably be mandatory for implementations to work

I don't think we can expect the respective subject matter experts themselves to find and resolve all of these issues (they didn't), and I find it unlikely we could do this with improved tooling in the near future. IMO letting through any of these issues would be a very bad thing for the YANG model in question.

When it comes to example instance data, I agree it would be great to have that tested against the models. There are frequently errors in the examples. In my experience it's not trivial to automate this process, however. I have verified quite a few of these using some semi-automatic methods, and it turns out that the examples are generally incomplete and routinely depend on configurations not included in the example having certain values. To not include all details in every example may be ok/a good thing, but it complicates automated testing.

/jan

> On 26 Jul 2017, at 02:36, Kent Watsen <[email-masked]> wrote:
> 
> On 7/25/17, 7:35 PM, "yang-doctors on behalf of Andy Bierman" <[email-masked] <mailto:[email-masked]> on behalf [email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi, 
>  
> _______________________________________________
> yang-doctors mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/yang-doctors

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 26 Jul 2017 14:44:28 +0200

> That said, I believe that completely automatic "YD review" will remain impractical/insufficient for a long time to come. Many important classes of issues that I and other YDs have been pointing out…

+1

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 26 Jul 2017 15:15:48 +0200

> On 26 Jul 2017, at 14:35, Jan Lindblad <[email-masked]> wrote:
> 

Agreed.

> I don't think we can expect the respective subject matter experts themselves to find and resolve all of these issues (they didn't), and I find it unlikely we could do this with improved tooling in…

Yes, and it can even be a security issue. On the other hand, Andy is correct that it is probably too ambitious to expect that YANG doctors can properly review all YANG modules passing through the IETF in a timely manner. I was late with my review of draft-ietf-i2rs-yang-l3-topology but Mehmet told me that some reviews are pending since March.

> When it comes to example instance data, I agree it would be great to have that tested against the models. There are frequently errors in the examples. In my experience it's not trivial to automate…

Yup, it would be necessary to add YANG library or equivalent info as metadata to each example, but such metadata is mostly not desirable in the RFC text.

Lada

>> On 7/25/17, 7:35 PM, "yang-doctors on behalf of Andy Bierman" <[email-masked] on behalf [email-masked]> wrote:
>>  
>> Hi, 
>>  

--
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67