Subject: [yang-doctors] YD review and yang-push and friends

From: Kent Watsen <[email-masked]>
Date: Thu, 8 Mar 2018 02:45:25 +0000

Doctors,

I just submitted a request for YD reviews on the following drafts that are currently in WG Last Call:

  https://tools.ietf.org/html/draft-ietf-netconf-yang-push-15
  https://tools.ietf.org/html/draft-ietf-netconf-subscribed-notifications-10
  https://tools.ietf.org/html/draft-ietf-netconf-netconf-event-notifications-08

Undoubtedly Dan and Mehmet will look to the YD queue to pick one of us, but perhaps one of you were already planning to review the YANG in these documents carefully?  

If you would additionally like to be shepherd for one or more drafts, please let me know.

Thanks!
Kent

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 Mar 2018 08:56:54 +0100 (CET)

>   https://tools.ietf.org/html/draft-ietf-netconf-subscribed-notifications-10

Ok.

>   https://tools.ietf.org/html/draft-ietf-netconf-netconf-event-notifications-08

This one doesn't have a YANG module, so it shouldn't need YD review.

(side note - I am surprised that this draft made it to WGLC with the
interleave issue unresolved)

> Undoubtedly Dan and Mehmet will look to the YD queue to pick one of
> us, but perhaps one of you were already planning to review the YANG in
> these documents carefully?

Yes, for sure, but then I'm also high up in the queue so I might get
one of them anyway...

/martin

---

From: "Mehmet Ersue" <[email-masked]>
Date: Thu, 8 Mar 2018 16:12:19 +0100

I did not start review for netconf-event-notifications-08.

Netconf co-chairs: Please clarify whether a review is required.

Cheers,
Mehmet

> -----Original Message-----
> From: yang-doctors <[email-masked]> On Behalf Of Martin
> Bjorklund
> Sent: Thursday, March 8, 2018 8:57 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [yang-doctors] YD review and yang-push and friends
> 
interleave
> issue unresolved)
> Yes, for sure, but then I'm also high up in the queue so I might get one
of
> them anyway...
> 
> 
> /martin
> 
> _______________________________________________
> yang-doctors mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/yang-doctors

---

From: "Mehmet Ersue" <[email-masked]>
Date: Thu, 8 Mar 2018 16:26:31 +0100

Dear Sue, Kent,

I extended the deadlines to 3/21.

Our regular review time is 3 weeks.
For LC reviews we should give the reviewers as a minimum (approx.) two
weeks.

Thanks,
Mehmet

> From: yang-doctors <[email-masked]> On Behalf Of Kent
> Watsen
> Sent: Thursday, March 8, 2018 3:45 AM
> To: yang-doctors <[email-masked]>
> Subject: [yang-doctors] YD review and yang-push and friends
> 
> 
> Doctors,
> 
https://tools.ietf.org/html/draft-ietf-netconf-netconf-event-notifications-
> 08
> 
> Undoubtedly Dan and Mehmet will look to the YD queue to pick one of us,
> but perhaps one of you were already planning to review the YANG in these
> documents carefully?
> 
> If you would additionally like to be shepherd for one or more drafts,
please
> let me know.

---

From: "Reshad Rahman (rrahman)" <[email-masked]>
Date: Thu, 8 Mar 2018 15:54:47 +0000

I am planning on going over these YANG models, but I have holidays before and after IETF so I can't be YD for these docs.

Regards,
Reshad.

On 2018-03-08, 10:26 AM, "yang-doctors on behalf of Mehmet Ersue" <[email-masked] on behalf of [email-masked]> wrote:

    Dear Sue, Kent,

    
    Thanks,
    Mehmet
    
    please

---

From: Kent Watsen <[email-masked]>
Date: Thu, 8 Mar 2018 17:09:05 +0000

> Netconf co-chairs: Please clarify whether a review is required.

What's in a YANG Doctor review?  Is it just syntax, or semantics too?
If it includes semantics, then does that then entail needing to read the
draft text as well, to determine if the YANG module expresses the correct
semantics or find that the draft text is wrong?  Would it also extend to
reviewing the examples in the draft, to further ensure that the semantics
are understood correctly or, possibly, that there is an error in the
example?

Yes, I am aware that netconf-event-notifications does not define a YANG
module, but it does have examples that for the YANG modules in the 
yang-push and subscriber-notifications drafts.  In that sense, I'm 
wondering if they need to be reviewed, or do we expect the YD reviewers
of those other two drafts to look at this draft already?  

FWIW, I not talking about what might be found via validation.  I've
already asked the authors to post a script that validates the 14 
examples in this draft...

K.

---

From: Kent Watsen <[email-masked]>
Date: Fri, 9 Mar 2018 02:05:43 +0000

> interleave issue unresolved)

The authors stated that they responded to the concerns raised and, 
after a week of not hearing any objection, assumed that it was be
okay.  As my LC kickoff message said, the authors indicated to the
chairs that they believed the document ready.  Perhaps not, but I
can't be second-guessing everyone either.

FWIW, I also raised the :interleave capability issue again in the 
review I posted yesterday, let's ensure it gets fixed this time.

Kent

---

From: "Mehmet Ersue" <[email-masked]>
Date: Fri, 9 Mar 2018 19:49:41 +0100

One question coming up in my mind is against which criteria should such drafts be reviewed.
A YANG module has its review criteria defined in YANG RFCs.
However examples may be manifold and imperfect.

Cheers,
Mehmet

> From: Kent Watsen <[email-masked]>
> Sent: Thursday, March 8, 2018 6:09 PM
> To: Mehmet Ersue <[email-masked]>; 'Martin Bjorklund' <mbj@tail-
> f.com>
> If it includes semantics, then does that then entail needing to read the draft
> text as well, to determine if the YANG module expresses the correct
> reviewing the examples in the draft, to further ensure that the semantics are
> understood correctly or, possibly, that there is an error in the example?
> 
> module, but it does have examples that for the YANG modules in the yang-
> push and subscriber-notifications drafts.  In that sense, I'm wondering if they
> need to be reviewed, or do we expect the YD reviewers of those other two
> drafts to look at this draft already?
> 
> FWIW, I not talking about what might be found via validation.  I've already
> asked the authors to post a script that validates the 14 examples in this
> draft...
> 
> 
> K.
>

---

From: Kent Watsen <[email-masked]>
Date: Tue, 13 Mar 2018 01:58:02 +0000

Now that the YD page has been restored, here's what it says:

"""

What to look for during a review

The most important item is to give the AD a sense of how important it is that they pay attention to the document. 
For YANG reviews the YANG Doctors will apply the RFC6087bis document on the Guidelines for Authors and Reviewers of YANG Data Model Documents ​https://datatracker.ietf.org/doc/draft-ietf-netmod-rfc6087bis/. The YANG language syntax and semantics should be analyzed. The compliance with ​Network Management Datastore Architecture should to be ensured (see also ​NMDA guidelines).

Review Information

Under some circumstances, the YANG doctors might discover open issues or provide feedback worth documenting for the larger community. While the NETMOD WG still work on RFC6087bis, updating this document is preferred. If the topic is not appropriate for the RFC6087bis or if RFC6087bis has already been published, then this must be documented on the YANG questions/answers WIKI https://trac.ietf.org/trac/ops/wiki/YANGDoctorsFAQ.

"""

The scope of the YD's review is unclear. 

K.

===== original message =====

Cheers,
Mehmet

---

From: Benoit Claise <[email-masked]>
Date: Tue, 13 Mar 2018 09:05:16 +0100

Why not review the document, even if there is no YANG module, and see if 
there is something to pay attention to? The examples, for example, are 
important to review and validate.

Regards, B.

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 13 Mar 2018 09:22:18 +0100 (CET)

Benoit Claise <[email-masked]> wrote:
> Why not review the document, even if there is no YANG module, and see
> if there is something to pay attention to? The examples, for example,
> are important to review and validate.

Yes, but is this something for the YANG doctors in general?

In this particular case, it doesn't really matter, since most likely
several YDs will review the document anyway.

/martin

> > The most important item is to give the AD a sense of how important it
> > is that they pay attention to the document.
> > For YANG reviews the YANG Doctors will apply the RFC6087bis document
> > on the Guidelines for Authors and Reviewers of YANG Data Model
> > Documents
> > ​https://datatracker.ietf.org/doc/draft-ietf-netmod-rfc6087bis/. The
> > YANG language syntax and semantics should be analyzed. The compliance
> > with ​Network Management Datastore Architecture should to be ensured
> > (see also ​NMDA guidelines).
> >
> > Review Information
> >
> > Under some circumstances, the YANG doctors might discover open issues
> > or provide feedback worth documenting for the larger community. While
> > the NETMOD WG still work on RFC6087bis, updating this document is
> > preferred. If the topic is not appropriate for the RFC6087bis or if
> > RFC6087bis has already been published, then this must be documented on
> > the YANG questions/answers WIKI
> > https://trac.ietf.org/trac/ops/wiki/YANGDoctorsFAQ.
> >
> > """
> >
> >> yang-
> >> already

---

From: "Mehmet Ersue" <[email-masked]>
Date: Tue, 13 Mar 2018 16:10:52 +0100

As YANG secretary I have an issue with reviewing draft which do not include YANG modules.

> However examples may be manifold and imperfect.

If the group decides to review such documents the review criteria needs to be defined first.

Cheers,
Mehmet

> From: Martin Bjorklund <[email-masked]>
> Sent: Tuesday, March 13, 2018 9:22 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> In this particular case, it doesn't really matter, since most likely several YDs
> will review the document anyway.
> 
> 
> /martin
> 
> >
> > Regards, B.
> >
> >
> > > The most important item is to give the AD a sense of how important
> > > it is that they pay attention to the document.
> > > YANG language syntax and semantics should be analyzed. The
> > > compliance with ​Network Management Datastore Architecture should to
> > > be ensured (see also ​NMDA guidelines).
> > >
> > > Review Information
> > >
> > > Under some circumstances, the YANG doctors might discover open
> > > issues or provide feedback worth documenting for the larger
> > > community. While the NETMOD WG still work on RFC6087bis, updating
> > > this document is preferred. If the topic is not appropriate for the
> > > RFC6087bis or if RFC6087bis has already been published, then this
> > > must be documented on the YANG questions/answers WIKI
> > >> If it includes semantics, then does that then entail needing to
> > >> read the draft text as well, to determine if the YANG module
> > >> expresses the correct semantics or find that the draft text is
> > >> wrong?  Would it also extend to reviewing the examples in the
> > >> draft, to further ensure that the semantics are understood
> > >> correctly or, possibly, that there is an error in the example?
> > >>
> > >> Yes, I am aware that netconf-event-notifications does not define a
> > >> YANG module, but it does have examples that for the YANG modules in
> > >> the
> > >> yang-
> > >> push and subscriber-notifications drafts.  In that sense, I'm
> > >> wondering if they need to be reviewed, or do we expect the YD
> > >> reviewers of those other two drafts to look at this draft already?
> > >>

---

From: Kent Watsen <[email-masked]>
Date: Tue, 13 Mar 2018 22:29:49 +0000

Fine, but what about the examples that are contained within a draft that defines the YANG module.  Do we expect YANG Doctors to review the examples or not?   What I'm looking for is a definition of what all a YANG Doctor looks at, if anything less than the entire draft.  Can the YANG Doctor function be automated, or is function more than validators could ever hope to do?

K.

=====

Cheers,
Mehmet

> > > ​https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Dnetmod-2Drfc6087bis_&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-eRBVyomnwn2DvqhWonaBc&s=a93P1wWzQy2YYDl9KQTai1EbHdIRYwH_EwYv-TcUYjU&e=. The
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__trac.ietf.org_trac_ops_wiki_YANGDoctorsFAQ&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-eRBVyomnwn2DvqhWonaBc&s=s7Yh35PaMXizxNvkc0_aLaDV1FDcJgoj2_IuPiEoRlc&e=.
> > >
> > > """
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_yang-2Ddoctors&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-eRBVyomnwn2DvqhWonaBc&s=3pky8v7zSfdi3HNvorvvT3Y60l7ZxfBUm6K8ulTV3r8&e=
> >

---

From: Andy Bierman <[email-masked]>
Date: Wed, 14 Mar 2018 12:52:50 -0700

On Tue, Mar 13, 2018 at 3:29 PM, Kent Watsen <[email-masked]> wrote:

>
> Fine, but what about the examples that are contained within a draft that
> defines the YANG module.  Do we expect YANG Doctors to review the examples
> or not?   What I'm looking for is a definition of what all a YANG Doctor
> looks at, if anything less than the entire draft.  Can the YANG Doctor
> function be automated, or is function more than validators could ever hope
> to do?
>
> K.
>
> =====
>

IMO the scope needs to be fairly tight, and needs to focus on the aspects
that
no tool could ever hope to automate:

  Review YANG module(s) from 3 POVs
      A) standard POV -- consistent with all related standards; reusing
existing YANG correctly?
      B) server POV -- it is clear to server implementors what to do
      C) client POV -- it is clear to client developers what to do, and
what a server is expected to do

Since normative text is spread all over the place, determining if (2) is
correct
can be a massive undertaking.

So I will focus only only these aspects during my reviews.
If you want to check if the indentation is exactly right or
every single reference is fully named, then an automated tool should do
that.

Cheers,
> Mehmet

Andy

> > > > ​https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dnetmod-
> 2Drfc6087bis_&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=
> 9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-
> eRBVyomnwn2DvqhWonaBc&s=a93P1wWzQy2YYDl9KQTai1EbHdIRYwH_EwYv-TcUYjU&e=.
> The
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__trac.
> ietf.org_trac_ops_wiki_YANGDoctorsFAQ&d=DwIFaQ&c=
> HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=
> 9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-
> eRBVyomnwn2DvqhWonaBc&s=s7Yh35PaMXizxNvkc0_aLaDV1FDcJgoj2_IuPiEoRlc&e=.
> > > >
> > > > """
> > > >
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
> ietf.org_mailman_listinfo_yang-2Ddoctors&d=DwIFaQ&c=
> HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=
> 9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=9NbPiPD-CVHGtQCZXZJQf-
> eRBVyomnwn2DvqhWonaBc&s=3pky8v7zSfdi3HNvorvvT3Y60l7ZxfBUm6K8ulTV3r8&e=
> > >
>
>
>

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 15 Mar 2018 08:33:39 +0100 (CET)

> every single reference is fully named, then an automated tool should do

I agree.  But there are two issues right now: (i) who checks that
these tools have been run or who runs the tools; and (ii) such tools
currently do not do a 100% good job.

Until we have these tools 100% automated, I think it is part of the YD
review to check these things.

For example, to check indentation of YANG itself you can do:

   $ pyang -f yang --keep-comments <IN> > <OUT>
   $ diff <IN> <OUT>

... and manually check the result (it won't be empty).

The reference checker "should" be built into idnits, or similar.

/martin

---

From: Benoit Claise <[email-masked]>
Date: Thu, 15 Mar 2018 12:20:46 +0100

Dear all,
> The reference checker "should" be built into idnits, or similar.
We all agree that everything that could be automated should be automated.
In the end, it boils down to IETF hackathon and community members 
investing their time. Looking at the IETF hackathon participants this 
coming week-end ... I guess that Joe Clarke, Vladimir Vassilev, and I 
will be lonely at our YANG tools development table.

I was able to get some Cisco funding for one developer/9 months for the 
yangcatalog.org and related integration in the datatracker. That money 
stream dried out.

Next step: I hope for some ISOC money for the next phase of development 
on the yangcatalog.org and related tools. I've been trying for 3 months 
but the process is very slow.
In the mean time, see you guys at the hackathon?

Regards, Benoit

---

From: Christian Hopps <[email-masked]>
Date: Thu, 15 Mar 2018 15:20:58 +0000

Hi,

I was recently discussing yang versioning with some colleagues and I was saying that one should try and construct modules to allow for later revision (i.e., "minor version updates"), trying to avoid doing things that would require backward incompatible changes (i.e., "major version updates") later to add functionality. Poorly placed "mandatory true" is the first example that jumps to mind here. In any case this seems like something that it would be hard for an automated tool to recognize that a yang doctor review might catch.

Thanks,
Chris.