Subject: Why NETCONF needs a data modeling language

From: "David Harrington" <[email-masked]>
Date: Fri, 30 Nov 2007 10:35:13 -0500

> Hi,
> 
> There are a few questions that the IESG and others have asked,
> which I will try to address:
> 
>    Q1) Why does NETCONF need a DML at all?
>    Q2) Why is NETCONF special?
>    Q3) Why won't lots of other WGs want to define their own
>        protocol-specific DMLs?
>    Q4) Why isn't XSD or RelaxNG good enough?

I'll take a whack at answering the same questions.

A1) Why does NETCONF need a DML at all?

to promote vendor-neutral interoperable management. 

It is much easier to develop standards when everybody uses the same
basic language to communicate; this "common language for shared
communication" is also reflected in the IETF decision to use English
text and ASCII documents.

This is not just about being able to standardize **device
management**; it is a basic step to permit the standardization of
**network management** and possibly **services management". The DML is
a basic building block.

A2) Why is NETCONF special?

It is and it isn't. Netconf is only one protocol used for network
management. 

It has some unique requirements, such as using a document-based
approach and differentiating the data for config versus state and for
dealing with different time-defined contexts such as running and
startup configs. This differs from other network management protocols,
such as SNMP and syslog and ipfix, which use their own data formats,
and usually deal only with the currently-running config. Netconf is a
tool designed to meet the special requirements of configuration, and
the DML needs to support special features not found in other NM-DMLs. 

Netconf is not special, in that any language used for network
management is likely to have certain common requirements. NM data
models are commonly used directly by humans, in their raw form, such
as when they troubleshoot problems using network sniffers. NM data
models are also commonly used by NMS applications that can handle the
translations into a more human-readable format. Designers of NM tools
(e.g., protocols and data models) thus need to pay close attention to
who will use the information, and to assume that the data will be used
both directly by humans and by applications. Operators have complained
strongly that OIDs are very hard to work with in raw form, yet
operators frequently need to deal with the raw form of the data. 

A3) Why won't lots of other WGs want to define their own
        protocol-specific DMLs?

To paraphrase a wise man from the SNMP community, when you fill a room
with protocol designers and ask for a solution to a problem, is it a
surprise when they recommend designing a new protocol?

The decision about whether to use an existing protocol/DML or develop
a new protocol/DML should be made after a careful analysis of the
requirements of the solution. SNMP was designed when CMIP was found to
not quite address the needs. The SMI was designed when ASN.1 was found
to not quite address the needs. XML was designed when other DMLs were
found to not quite address the needs.

The decision to explore an XML-based DML and to explore a C-like DML
follows years and years and years of debate over the requiremnts of
network management, and configuration in particular. 

If every WG spends as much time analyzing the requirements that the
OPS area has spent considering this decision, it might be good for
ensuring no new protoocls are designed that are not really needed, but
the IETF would also grind to a halt, much as the OPS community has
done over our many years of debate. And the delay caused by our
debates has made IETF network management largely irrelevant to the
operator community (our customers).

As always, we need to be vigilant and determine whether enough thought
has been given to reuse of existing protocols. We also need to
consider the tradeoffs between new protocols and reusing existing
protocols in ways they were not designed to be used.

A4) Why isn't XSD or RelaxNG good enough?

As mentioned in A2, NM data models need to be both machine- and
human-readable. XSD is machine-readable, but it is a tough language
for humans. RelaxNG seems better. 

As discussed further in a different email, Netconf will almost
certainly need a DML suited to its requirements, and if RelaxNG is
found to be human-friendly-enough, we would almost certainly still
need to select a subset and adapt it to meet configuration
requirements. So the benefit of using RelaxNG over a domain-specific
DML may be lost by using an adapted-subset of RelaxNG.

Most operators already understand languages like Perl and C and
Javascript, because they already need to write lots of scripts to
manage their networks. Most implementers of NM support in
internetworking devices work in C or a variant of C. It makes a lot of
sense to use a language with a C-like syntax for these people, rather
than forcing them to learn yet another language that was designed for
some other purpose.

[soap]
somebody commented that the designers of XSD and RelaxNG really
understand how to design DMLs, implying that the OPS community does
not. Many of the people involved in the ongoing DML discussions over
the years, and now, are MIB Doctors and operators and protocol
designers, and have had years of experience designing and working with
SMI and network management. They understand the requirements of a DML
for NM far more than the designers of XSD or RelaxNG, who were not
designing their DMLs for NM purposes.
[end soap]

David Harrington
[email-masked]
[email-masked]

---

From: "David Harrington" <[email-masked]>
Date: Fri, 30 Nov 2007 09:31:37 -0500

Hi comments inline. 

> -----Original Message-----
> From: Juergen Schoenwaelder 
> [mailto:[email-masked]] 
> Sent: Thursday, November 29, 2007 6:35 AM
> To: Rohan Mahy
> Cc: [email-masked]; [email-masked]; NETCONF Goes On
> Subject: [YANG] Re: Why NETCONF needs a data modeling language
> 
> On Wed, Nov 28, 2007 at 08:17:04PM -0800, Rohan Mahy wrote:
> 
> > > NETCONF needs
> > > its own MIB language, just like SNMP needs SMIv2.
> > 
> > I don't think you've made a strong case here. You might be 
> right, but your
> > statements above do not strongly motivate a new language.
> 
> NETCONF needs a new language; whether it is a domain specific
language
> like YANG or an "adapted subset" of some existing XML schema
language
> does not changFrom [email-masked] Sat Dec 01 20:38:14 2007
Date: Fri, 30 Nov 2007 11:27:42 -0800
From: Jim Fenton <[email-masked]>
To: [email-masked]
Subject: Re: My XML session Sunday 3PM
Cc: Apps Discuss <[email-masked]>

Sounds interesting.  Where will you be?

-Jim

Tim Bray wrote:
> Since my agenda isn't posted on the IETF70 site, it might be helpful
> for people planning their Sundays to know it, and be sure to stay
> away:
>
> - How to think about whether XML is a good choice or not for use in an
> Internet protocol.  There are alternatives which are often superior
> choices.
> - If you are going to use XML, how to think about inventing your own
> language as opposed to using an existing one, with a review of some
> popular languages you might wane that fact that we will need a NETCONF data modeling
> language and that we will need tools to understand that language.

+1. 
Since this discussion goes outside the Network Management community in
the IETF, I think some background is important.

ASN.1 was a general-purpose modeling langugae that did not meet the
requirements of network management. For network management standards,
it is important to limit the required resources (code footprint,
memory, CPU usage) to make implementtions feasible for
resource-constrained devices like set-top boxes. That's why SMI is a
subset of ASN.1, rather than all of ASN.1. 

There are aspects of the data models needed for network management
that are not necessaily needed for all ASN.1 modeling. Some examples
are given below. That's why SMI is an "adapted" subset of ASN.1

Now we are looking to move from the binary ASN.1 to the textual XML as
the base language, to meet operators' requests (see RFC3535). Network
management still needs an "adapted subset" of a general purpose
modeling language or a domain-specific language.

>  
> > I don't think we can take it on blind faith that we need to 
> do things in
> > Netconf the way SNMP did them. 

I think the way SNMP did things is very inappropriate for
configuration purposes (and so most operators apparently). While I
strongly support reuse, especially reuse of existing standards, we
need to choose the right tools for the job to be done. SNMP is not a
viable candidate for confguration in most networks, for reasos
described below.

>I think it is very important 
> to look at the
> > reasons we thought that SNMP needed to be a certain way, 
> but we also need to
> > acknowledge that SNMP was hardly a success as a 
> configuration protocol, and
> > try to find out why. It is possible that the data model was 
> too rigid for
> > practical configuration use.
> 
> There has been a long discussion about why SNMP failed before the
> NETCONF effort was started, all going back to IAB workshop [RFC
3535]
> and lots of followup email discussions. I am not sure it is terrible
> useful to redo this exercise. From what I recall, the rigidity of
the
> SMI was never mentioned as a major issue why SNMP failed to be a
good
> configuration protocol.

+1. Let me do a quick review for those who haven't followed those
discussions, so we don't have to redo all these discussions from
scratch. 

SNMP was not the only approach for configuration that failed, but we
can start there. The 2002 IAB Workshop was toward the end of long
discussions about configuration, not the beginning. COPS-PR, SNMPCONF,
the Dilbert mailing list, the ops-nm mailing list, and many other
discussions focused a lot on configuration requirements. The IAB
Workshop achieved consesnus to explore an XML-based approach. 

The "SNMP standard" has three parts - the protocol, the MIB, and the
SMI.

The SNMP protocol was found inadequate for configuration for a number
of reasons. SNMPv1 lacked security, so many vendors did not support
SETs, so operators could not manipulate the configuration of many
vendors' devices with SNMP. Even when SET was supported, SNMP was
designed to run over unfragmented UDP, which limited message sizes so
much it would be impossible to configure a large device in a
reasonable amount of time.

The major issue with MIBs for configuration was lack of vendor support
for MIB modules for all features in their products, which left SNMP
without MIB modules to configure everything in a device. And if the
CLI could configure everything, and the CLI was needed to configure
those things without MIB modules, then it made sense to simply use CLI
for everything. A related issue was the long time it takes to get a
standard MIB module for a new protocol versus developing a proprietary
CLI command set for a new protocol; the delay made the MIB module
irrelevant since the problem of configuring the feature would have
been already resolved using the CLI.

There were some discussions of the rigidity of the SMI, mostly around
complex data structures such as nested tables, but there are
workarount to consider adopting.
> - If you are going to design your own language, how you might go about
> doing so.  This would include discussion of:
> -- verbosity vs. terseness
> -- the use of schemas, and the choice of available schema languages
> -- extensibility
> -- i18n and encoding issues
>
> Plus, the crowd will probably be small, so if you have your own
> XML/IETF issue that needs discussion, bring it along.  -T
>
>

---

From: "David Harrington" <[email-masked]>
Date: Fri, 30 Nov 2007 09:31:37 -0500

Hi comments inline. 

language
language
> does not changds for this. AFAIK, relational databases typically do not
support nested tables either, so I never considered this a major
fault; it just makes it harder to represent structures that can be
expressed easily in C, which was the preferred language for agent
developers.

I think the biggest problem with the SMI is that everybody hates OIDs
and wants something more human-readable. But that's not just about
configuration.

See RFC3535 for more detailed summaries of the IAB Workshop
discussions.

> 
> > Look at the following Relax NG in compact form, 
> side-by-side with YANG.
> > 
> > container aaa {
> >    container bbb {
> >       leaf foo { type string; }
> >       leaf bar { type uint32; }
> >    }
> > }
> > 
> > element aaa {
> >    element bbb {
> >       element foo { xsd:string },
> >       element bar { xsd:unsignedInt }
> >    }
> > }
> > 
> > This hardly seems like a big difference to me.
> 
> I am not sure which point you are trying to make using examples that
> ignore almost all NETCONF specifics. But even then, if you spell out
> the additional NETCONF specific semantics which are hidden because
> there are suitable defaults for the most common cases, then the YANG
> example above actually expands to the following:
> 
>  container aaa {
>     config true;
>     status current;
>     container bbb {
>        status current;
>        config true;
>        leaf foo { 
>           type string; 
>           config true;
> 	  status current;
> 	  mandatory false;
>        }
>        leaf bar { 
>           type uint32; 
>           config true;
> 	  status current;
> 	  mandatory false;
>        }
>     }
>  }
> 
> Statements like 'config true;' are essential for the processing of
> NETCONF messages and also for validating content exchanged in
NETCONF
> messages (and this is why for example generic XML validation tools
are
> only of limited value for NETCONF implementors).
> 
> I am sure one can define an adapted subset of RNG which takes care
of
> all this and then at the end this adapted subset likely works like
> YANG. However, given the SMI and ASN.1 experience, I do question
> 
> a) that doing so will lead to a large reuse of available RNG tools
for
>    processing NETCONF data models (since they don't know the RNG
>    subset nor the semantic additions)

+1. Tools for XSD or RNG will only be able to do partial validation
and processing. Additional validation will be needed that is
domain-specific. If XSD forces us to put much of the "adapted" stuff
into app-notes, we will need special tools that can validate the
content of the app-notes.

> 
> b) that the effort to specify such an extended subset will be less
>    than the effort for defining a domain specific language like YANG

+1; I think it might take longer to develop a RNG-standard-compliant
extended subset than working with a domain specific language, because
we will constantly be faced with finding workarounds for the
constraints imposed by the RNG standard.

> 
> c) that the effort to implement tools understanding the adapted
subset
>    of RNG will be less than writing proper tools for a domain
specific
>    language from scratch

+1.

> 
> d) that there will be long term (say 20 year) reliable change
control
>    exercised by the organization that has change control over
NETCONF

+1. ASN.1 changed a lot after the SMI was developed as an adapted
subset of ASN.1-1988. I think ASN.1 became more complex, adding new
features for additional uses, and changed in ways that made the SMI no
longer compatible with the ASN.1 standard. The separation of the SMI
from the underlying langauge has allowed the SMI to remain stable for
twenty years. That stability is critically important to motivating
vendors to utilize our standards, since they are not faced with a
constantly-changing base language.

The SMING WG tried to update the SMI, and found that, even with strong
pressures to remain backwards compatible with SMIv1 and SMIv2, there
is a need to move forward to a new base language that is more
human-readable than SMIv1 or SMIv2 (and ASN.1). But whether XML or
somee that fact that we will need a NETCONF data modeling

+1. 

3535]
the
good

There were some discussions of the rigidity of the SMI, mostly around
complex data structures such as nested tables, but there are
workarounthing else, the solution needs to be able to address the
requirements of network management, especially human-readability.

> 
> /js
> 
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> 
> 
> _______________________________________________
> YANG mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/yang
> 

ds for this. AFAIK, relational databases typically do not
support nested tables either, so I never considered this a major
fault; it just makes it harder to represent structures that can be
expressed easily in C, which was the preferred language for agent
developers.

I think the biggest problem with the SMI is that everybody hates OIDs
and wants something more human-readable. But that's not just about
configuration.

See RFC3535 for more detailed summaries of the IAB Workshop
discussions.

> 
> > Look at the following Relax NG in compact form, 
> side-by-side with YANG.
> > 
> > container aaa {
> >    container bbb {
> >       leaf foo { type string; }
> >       leaf bar { type uint32; }
> >    }
> > }
> > 
> > element aaa {
> >    element bbb {
> >       element foo { xsd:string },
> >       element bar { xsd:unsignedInt }
> >    }
> > }
> > 
> > This hardly seems like a big difference to me.
> 
> I am not sure which point you are trying to make using examples that
> ignore almost all NETCONF specifics. But even then, if you spell out
> the additional NETCONF specific semantics which are hidden because
> there are suitable defaults for the most common cases, then the YANG
> example above actually expands to the following:
> 
>  container aaa {
>     config true;
>     status current;
>     container bbb {
>        status current;
>        config true;
>        leaf foo { 
>           type string; 
>           config true;
> 	  status current;
> 	  mandatory false;
>        }
>        leaf bar { 
>           type uint32; 
>           config true;
> 	  status current;
> 	  mandatory false;
>        }
>     }
>  }
> 
> Statements like 'config true;' are essential for the processing of
> NETCONF messages and also for validating content exchanged in
NETCONF
> messages (and this is why for example generic XML validation tools
are
> only of limited value for NETCONF implementors).
> 
> I am sure one can define an adapted subset of RNG which takes care
of
> all this and then at the end this adapted subset likely works like
> YANG. However, given the SMI and ASN.1 experience, I do question
> 
> a) that doing so will lead to a large reuse of available RNG tools
for
>    processing NETCONF data models (since they don't know the RNG
>    subset nor the semantic additions)

+1. Tools for XSD or RNG will only be able to do partial validation
and processing. Additional validation will be needed that is
domain-specific. If XSD forces us to put much of the "adapted" stuff
into app-notes, we will need special tools that can validate the
content of the app-notes.

> 
> b) that the effort to specify such an extended subset will be less
>    than the effort for defining a domain specific language like YANG

+1; I think it might take longer to develop a RNG-standard-compliant
extended subset than working with a domain specific language, because
we will constantly be faced with finding workarounds for the
constraints imposed by the RNG standard.

> 
> c) that the effort to implement tools understanding the adapted
subset
>    of RNG will be less than writing proper tools for a domain
specific
>    language from scratch

+1.

> 
> d) that there will be long term (say 20 year) reliable change
control
>    exercised by the organization that has change control over
NETCONF

+1. ASN.1 changed a lot after the SMI was developed as an adapted
subset of ASN.1-1988. I think ASN.1 became more complex, adding new
features for additional uses, and changed in ways that made the SMI no
longer compatible with the ASN.1 standard. The separation of the SMI
from the underlying langauge has allowed the SMI to remain stable for
twenty years. That stability is critically important to motivating
vendors to utilize our standards, since they are not faced with a
constantly-changing base language.

The SMING WG tried to update the SMI, and found that, even with strong
pressures to remain backwards compatible with SMIv1 and SMIv2, there
is a need to move forward to a new base language that is more
human-readable than SMIv1 or SMIv2 (and ASN.1). But whether XML or
something else, the solution needs to be able to address the
requirements of network management, especially human-readability.

> 
> /js
> 
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> 
> 
> _______________________________________________
> YANG mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/yang
>

---

From: Balazs Lengyel <[email-masked]>
Date: Sun, 02 Dec 2007 10:20:07 -0800

Hello,
 >>    Q1) Why does NETCONF need a DML at all?

If we have an IETF standard DML for NETCONF, device vendors (like my company) will be much 
more comfortable, much more willing to adapt NETCONF itself. With a standard DML we would 
see a better chance at having available 3rd party or even freeware toolkits for NETCONF.

The above would be important even for a company which would not care about interoperability.

Balazs

---

From: Jon Saperia <[email-masked]>
Date: Sun, 2 Dec 2007 14:00:43 -0500

Thanks
/jon
----------------------
Jon Saperia
TSP NM
(mobil) 617-201-2655
(office) 978-461-0249
[email-masked]

On Dec 2, 2007, at 1:20 PM, Balazs Lengyel wrote:

> If we have an IETF standard DML for NETCONF, device vendors (like  
> my company) will be much more comfortable, much more willing to  
> adapt NETCONF itself. With a standard DML we would see a better  
> chance at having available 3rd party or even freeware toolkits for  
> NETCONF.

Yes, but does that mean that we will begin to develop a standard  
configuration objects for say, BGP or DiffServ that would work from  
one vendor to the next?  It would be nice to think so, that would  
really help interoperability and should be the target of 'our'  
collective efforts.
/jon
>> to promote vendor-neutral interoperable management. It is much  
>> easier to develop standards when everybody uses the same
>> management. It has some unique requirements, such as using a  
>> document-based
>> the DML needs to support special features not found in other NM- 
>> DMLs. Netconf is not special, in that any language used for network
>> network management, and configuration in particular. If every WG  
>> spends as much time analyzing the requirements that the
>> for humans. RelaxNG seems better. As discussed further in a  
>> different email, Netconf will almost

---

From: Balazs Lengyel <[email-masked]>
Date: Sun, 02 Dec 2007 12:19:01 -0800

Hello,
Naturally our aim is to develop standard NETCONF configuration models, and the more people 
use NETCONF the easier that will be.
Balazs

Jon Saperia wrote:
> 
> Thanks
> /jon
> ----------------------
> Jon Saperia
> TSP NM
> (mobil) 617-201-2655
> (office) 978-461-0249
> [email-masked] <mailto:[email-masked]>
> 
> 
> 
>> If we have an IETF standard DML for NETCONF, device vendors (like my 
>> company) will be much more comfortable, much more willing to adapt 
>> NETCONF itself. With a standard DML we would see a better chance at 
>> having available 3rd party or even freeware toolkits for NETCONF.
> 
> configuration objects for say, BGP or DiffServ that would work from one 
> vendor to the next?  It would be nice to think so, that would really 
> help interoperability and should be the target of 'our' collective efforts.
> /jon
>>
>>> to promote vendor-neutral interoperable management. It is much easier 
>>> to develop standards when everybody uses the same
>>> operators frequently need to deal with the raw form of the data. A3) 
>>> Why won't lots of other WGs want to define their own
>>>         protocol-specific DMLs?
>>> for humans. RelaxNG seems better. As discussed further in a different 
>>> email, Netconf will almost
>>> [email-masked] <mailto:[email-masked]>
>>> [email-masked] <mailto:[email-masked]>
>>
>>
>

---

From: Jon Saperia <[email-masked]>
Date: Sun, 2 Dec 2007 17:43:53 -0500

Wrong order - people will push (not just the vendors for convenience)  
when there is something that makes a real difference.
On Dec 2, 2007, at 3:19 PM, Balazs Lengyel wrote:

> Hello,
> Naturally our aim is to develop standard NETCONF configuration  
> models, and the more people use NETCONF the easier that will be.
> Balazs
>
>
> Jon Saperia wrote:
>> Thanks
>> /jon
>> ----------------------
>> Jon Saperia
>> TSP NM
>> (mobil) 617-201-2655
>> (office) 978-461-0249
>>> chance at having available 3rd party or even freeware toolkits  
>>> for NETCONF.
>> configuration objects for say, BGP or DiffServ that would work  
>> from one vendor to the next?  It would be nice to think so, that  
>> would really help interoperability and should be the target of  
>> 'our' collective efforts.
>> /jon
>>>

Thanks,
/jon
----
Jon Saperia
[email-masked]
(mobil)	617-201-2655
(office)	978-461-0249