Subject: [YANG] key clause issues

From: Andy Bierman <[email-masked]>
Date: Mon, 07 Jan 2008 16:38:30 -0800

Hi,

I have a few concerns with the key clause...

 From -00:
7.8.2.  The list's key Statement

    The "key" statement, which MUST be present if the list represents
    configuration, and MAY be present otherwise, takes as an argument a
    string which specifies a space separated list of leaf identifiers of
    this list.  A leaf identifier MUST NOT appear more than once in the
    key.

Issue 1)

   From an interoperability perspective, the key needs to
   be defined any time there are multiple named instances
   of an element (i.e., every list).  Config vs. read-only is irrelevant.
   Operators and manager applications need to know how list entries
   are indexed for interoperability.  SNMP is mostly for monitoring.
   I can't imagine how operators and DM readers would be able to talk about
   tables and rows if all the INDEX clauses were removed from read-only tables.

Issue 2)

   Why is the CLR about a leaf appearing more than once in the key needed?
   It should be up to the DM writer to decide if this makes sense or not:

    list addr-pairs {
      key "addr-type src-addr addr-type dest-addr";

      leaf addr-type { type ietf:InetAddressType; }
      leaf src-addr { type ietf:InetAddress; }
      leaf dest-addr { type ietf:InetAddress; }
      // ... rest of list entry
    }

    Why is this illegal?
    It may be bad practice (according to RFC 4001), but should it be illegal?

Issue 3)

 From the ABNF:

      key-stmt               = key-keyword sep key-arg-str stmtend

      key-arg-str            = < a string which matches the rule
                                 key-arg >

      key-arg                = 1*(identifier sep)

   Why is the key statement limited to top-level sibling nodes
   within the list?  The unique-stmt (and keyref) allows
   descendant-schema-nodeid instead of just identifier:

       unique "foo/item1 foo/bar/item2";

   With this constraint on the key-stmt, YANG cannot support
   indexing using reusable containers.  (There needs to be
   exactly one instance of the key component per list entry,
   so 'list' and 'choice' nodes cannot be used within a key.)

   There are many use-cases, such as reusable (addr-type, addr) or
   (addr, port) data types.  The whole point of the container
   is that it provides the DM writer with an abstraction that
   represents a containerized set of objects, bound together for some reason.

   YANG has no support direct support for reusable complex types,
   but there is a workaround with grouping/uses:

    grouping addr-pair {
      leaf addr-type { type ietf:InetAddressType; }
      leaf addr { type ietf:InetAddress; }
    }

    list addr-pairs {
      key "src/addr-type src/addr dest/addr-type dest/addr";

      container src {
        uses addr-pair;
      }

      container dest {
        uses addr-pair;
      }
      // ... rest of list entry
    }

   This sort of indexing is very useful and needs to be supported by YANG.

Andy

_______________________________________________
YANG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/yang

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 08 Jan 2008 11:29:13 +0100

See below

PS.:
The reason for the existence of YANG is that it is more simple then XSD." quote from Balazs

> I have a few concerns with the key clause...
...

>    It may be bad practice (according to RFC 4001), but should it be 
[BALAZS]: When the user/programmer uses this in an edit-config he might think that the two 
add-types are two distinct pieces of data, while they are actually the same.

Would the user actually have to provide the addr-type twice in a netconf message?
How do you handle if the two provided values in a netconf request are not the same?

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 08 Jan 2008 12:11:47 +0100 (CET)

Hi Andy,

Andy Bierman <[email-masked]> wrote:
>    of an element (i.e., every list).

Exactly what will not be interoperable by not having keys for
non-config data?

(One problem might be how to define access control.  W/o keys you
might have to fall back to generic XPath, which we want to avoid...)

>    Config vs. read-only is irrelevant.

In SNMP, the INDEX is needed by the protocol.  Not so in NETCONF for
non-config data.

>       key "addr-type src-addr addr-type dest-addr";

What does this mean?  What is the difference between this and:

        key "addr-type src-addr dest-addr";

What does an instance identifier for your example look like?

>    within the list?

Several reasons.  Have you seen any CLIs with arbitrarly deep keys?
(is it really needed?)

What does this mean:

   list foo {
     key "a/b/c";
     container a {
        container b {
           precense "...";
	   leaf c;
        }
     }
               
Or this:

   list foo {
     key "a/b/c";
     container a {
        list b {
           ...
	   leaf c;
        }
     }

Or this:

   list foo {
     key "a/b";
     container a {
        choice x {
	   leaf b;
	   leaf c;
        }
     }

Also, by allowing deep keys, you force an agent implementation to do
buffering of - in the worst case - the entire configuration.  You
influence the protocol characteristics to the worse.  W/o deep keys
implementations _can_ do minimal buffering, and thus make the NETCONF
protocol usable in smaller devices.  I know at least three
implementations that work this way today (and I know one that does
buffering).

>    so 'list' and 'choice' nodes cannot be used within a key.)

I can also argue that this is an CLR.  Why don't you allow:

   list foo {
      key address;
      container address {
         choice type {
            leaf ipv4address;
            leaf ipv6address;
            leaf name;
         };
         leaf port;
      }
   }

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 08 Jan 2008 05:24:41 -0800

Balazs Lengyel wrote:
> [BALAZS]: When the user/programmer uses this in an edit-config he might 
> think that the two add-types are two distinct pieces of data, while they 
> are actually the same.
> 

Why would they think that?
The path to the node (/addr-pairs/addr-type) is the same.
That is enough info for the manager to know they are the same object.

> Would the user actually have to provide the addr-type twice in a netconf 

no -- the key is conceptual -- not actually identifiable in
any NETCONF PDU as a key.

> How do you handle if the two provided values in a netconf request are 

The point of this DM design would be that the addr-pairs are
always the same addr-type so listed it twice is always redundant.
If the addr-pairs could ever be different addr types for each addr,
then a different table design would be used.

We need to give the DM writer the tools to make good decisions.
We do not need a ton of CLRs that assume the DM writer does
not know what to do.  CLRs should be used when the consensus
is that more often than not, bad things will happen without the CLR.
What bad things happen by not forbidding a key component from
appearing more than once?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 08 Jan 2008 05:57:17 -0800

> Exactly what will not be interoperable by not having keys for

The feature intent needs to be explained in the draft.
If a list has no key defined, then that means that
there is no distinction in the agent between any of
the entries, other than their relative position (foo[1], foo[2], etc.)

If this is true, and the manager knows that, then there is no
problem leaving out the key.  (Other DMLs call this multiple
instances of a container, not a list though).

But if the agent has some way of identifying list entries,
other than their relative position, then a key MUST be specified.
(Is this what the draft intends?)

> In SNMP, the INDEX is needed by the protocol.  Not so in NETCONF for

Not true at all.  If the agent is using named entries
instead of position-identified entries, the manager needs to know.

Consider any XML table like the ifTable counters.
What if the agent has entries for interfaces 1, 2, and 3,
but after awhile, interface 2 is deleted.  If the agent
knows about 'ifIndex', but not the manager, how well do
you think the manager application will handle this common scenario?

Config vs. non-config is irrelevant here.
Identification via named keys or identification via node position
is the issue.

> What does an instance identifier for your example look like?

This table obviously cannot use the nested address.
By design, it uses separated fields.

> Several reasons.  Have you seen any CLIs with arbitrarly deep keys?

The CLR is simple -- just like the use-case.
Only static container nodes can be listed as interior nodes
in a key.  All your fabricated cases below are invalid.
The key indicates the mandatory naming components for the list entry.
All key components MUST be present in every list entry.

> implementations that work this way today (and I know one that does

This is really irrelevant in the context of standardization.
In other implementations (like mine) the position of the key
within the list element sub-tree is not a factor.

BTW, there is no text in the draft that says that the key components
must be defined first in the list.  In order to optimize the DML
design in favor of this implementation choice, the key components
can never be in reusable groupings (like address/port) and they
must always be the first leafs, defined in the same order as the key.

SMIv2 never had any massive CLRs like this for tables.
You can specify 'key' clauses in XSD for arbitrary nodes.

There is so much complexity in YANG wrt/ Xpath, local typedefs/groupings,
augments, that this hardly seems like an implementation burden
worthy of such drastic restrictions on data model design.

> I can also argue that this is an CLR.  Why don't you allow:

We need to understand what a key is, and when it is used.
A key identifies the static naming components so multiple
instances of a list node can be distinguished by an application.

I am not proposing that the key components be complex and/or non-static,
like you are, with this straw-man example.   It is not a CLR to
specify that key components must be static leafs.  This is
how SMIv2 works, and it should be how YANG works as well.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 08 Jan 2008 15:38:53 +0100 (CET)

> (Is this what the draft intends?)

I dont't follow.  How does it matter to interoperability if the agent
has some way of identifying list entries?  Which protocol operation
will suffer?

> you think the manager application will handle this common scenario?

ifIndex is a leaf that the manager knows about, and so does the
agent.  What will not work well?

[I am playing the devil's advocate here.  I would be perfectly happy
with keys for non-config data as well.]

Ah, now I remember the other reason.  If I use a list in an rpc or
notification, do I still need to have a key?
  
> By design, it uses separated fields.

You lost me again.  What does nested address have to do with the
single-occurance of leaf in the key?

I still don't understand what the key above actually is supposed to
mean.

> >>    Why is the key statement limited to top-level sibling nodes

[...]

> This is really irrelevant in the context of standardization.

Is it?  Even XSD puts restrictions on the XPath used in keys "In order
to reduce the burden on implementers, in particular implementers of
streaming processors"
[http://www.w3.org/TR/xmlschema-1/#cIdentity-constraint_Definitions].

Wasn't one of the reasons for not doing XPath filtering only in
NETCONF that it would require more resources on the device?

If we get a YANG WG, I think it is very important the WG really
understands the implementation/resource implications of constructs
like this.

> must always be the first leafs, defined in the same order as the key.

The keys don't have to be defined first.  They will be encoded first
in the XML.

> SMIv2 never had any massive CLRs like this for tables.

Well, in SMI there were no deep structures.

> You can specify 'key' clauses in XSD for arbitrary nodes.

Right.  Are you arguing that we should support the same flexibility of
keys as XSD?

> worthy of such drastic restrictions on data model design.

First of all, YANG does not require an XPath engine on the device, for
the same reason - to reduce the implementation burden on the device.

Local typedefs / groupings might add complexity to the compiler, but
not to the engine.

And again, it is the resource consumption on the device that I'm
worried about.  By allowing this you essentially say that streaming
processing is out of the question.

> how SMIv2 works, and it should be how YANG works as well.

Again, SMI does not have deep structures in tables so I don't think
the comparison helps.

Out of curiosity - how do you handle deep keys in your CLI?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 08 Jan 2008 08:07:11 -0800

See below

> agent.  What will not work well?

Yes, the manager knows about all the leafs (ifIndex, ifInOctets, ifMtu, etc.)
Without the magic text 'key ifIndex;', the manager has no idea what
makes one blob called <ifEntry> from another one.

Without a key, the manager has to identify each list node
by its position:

   <ifEntry>
     <ifIndex>1</ifIndex>
     <ifMtu>1500</ifMtu>
     ...
   </ifEntry>
   <ifEntry>
     <ifIndex>2</ifIndex>
     <ifMtu>9600</ifMtu>
     ...
   </ifEntry>
   <ifEntry>
     <ifIndex>3</ifIndex>
     <ifMtu>9600</ifMtu>
     ...
   </ifEntry>

Now, if the manager application has no idea that 'ifIndex' is the
key, and the entire <ifEntry> for entry '2' is deleted,
then the manager has no idea it is collecting data
for entries 1 and 3, not 1 and 2.

> notification, do I still need to have a key?

yes -- if there are named entries in the list
no -- if the list has unnamed entries, just position tells them apart

> I still don't understand what the key above actually is supposed to

Instance ID:

   /addr-pairs[addr-type='ipv4'][src-addr='192.168.0.1'][dest-addr='192.168.0.4']

> understands the implementation/resource implications of constructs

I do not think buffering input is an implementation burden
that should be a big factor.  The size of the data models
on the device is going to be relative to the device.

WGs and vendors should certainly attempt to place the index leafs
first instead of last, but should not be forced to do so.

Most importantly though,  WGs and vendors should not be forced
to undo (cut-and-paste) data placed in containers, just so
a list can use that data in the key.

It must be up to the DM writer where to use containers
for data, instead of spreading everything out in a flat table
like SMIv2.  Simple containers like (type, addr, port)
are going to be reused, and going to be needed in list keys.

SMIv2 never had any nesting.
That is a bug, not a feature.
XML DM writers need to be able to use nesting anywhere
it makes sense for a particular data model.

> Right.  Are you arguing that we should support the same flexibility of

no.  XSD allows the key to be outside the list.
In YANG, the key components MUST be defined within the list,
either directly, or via expansion of 'uses' statements.

I am proposing:
   1) The key leafs do no have to be first, or in any particular
      order within the list
   2) leafs nested only within the 'container' construct may
      be used as key components

> Out of curiosity - how do you handle deep keys in your CLI?

Yet another implementation detail, but the CLI processing
is different than NETCONF.  For CLI, all the index components
are listed first, like:

  ifEntry 1 {
    # ifIndex would not be listed since it is the index
    ifMtu 1500
    ifName Ethernet0/0
  }

Since the 'ifIndex' node has a back-ptr to its index header,
it can be skipped correctly no matter where it is located
in the subtree.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 08 Jan 2008 17:22:55 +0100 (CET)

> no -- if the list has unnamed entries, just position tells them apart

Why don't the same rules apply to non-config data?  In the ifIndex
example above, by this rule you would specify a key.

>    /addr-pairs[addr-type='ipv4'][src-addr='192.168.0.1'][dest-addr='192.168.0.4']

So the key statement above has addr-type twice, but in the IID it
occurs just once?

/martin

---

From: Phil Shafer <[email-masked]>
Date: Tue, 08 Jan 2008 17:20:39 -0500

>the entries, other than their relative position (foo[1], foo[2], etc.)

This is exactly what it's there for.  Some lists are position, and
no key is available and we want to avoid having to create one for
data that can't be manipulated individually.

A simple example would be syslog entries (or any list of events),
which consist of many non-unique leafs, having no unique key.

>problem leaving out the key.

Yup, exactly.

>(Other DMLs call this multiple

Isn't it cool that YANG got this right? ;^)

>(Is this what the draft intends?)

MUST?  It's the designer's data model.  Are we going to MUST them?
This is a CBAR.

>instead of position-identified entries, the manager needs to know.

Yup, and the lack of a key tells the manager that there's no key
for this list.

>you think the manager application will handle this common scenario?

So the modeler will say "Hey! I have a key" and use the in their
data model.

>Config vs. non-config is irrelevant here.

Config vs. non-config is _very_ relevant because we explicitly do
not allow lists in config data to be non-keyed.  The need to
manipulate the data overrides the modeler's need to avoid keys.

>Identification via named keys or identification via node position

The more important use case is where position (in the absolute)
simply doesn't matter.  Events in a syslog file may matter in
relative terms, but never in absolute terms even those numbers
are subject to change as files roll over and low priority events
are discarded.

>By design, it uses separated fields.

I'm lost here.  Why would you ever need to list the leaf more than once?

>worthy of such drastic restrictions on data model design.

It's funny which features trigger our individual "running with
scissors" sensors, but this one (deep keys) fires off mine.  I can
see a couple of fairly weak use cases, but the number of potential
abuses is overwhelming.  Someone's guaranteed to put their eye out
with deep keys.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 08 Jan 2008 15:18:48 -0800

>> problem leaving out the key.

Then the concept is correct and the text is wrong.
The intent is:
All config lists must have named entries and therefore must a key.
A non-config list must have a key if it has named entries,
otherwise the non-config list has only position-identified entries.

> MUST?  It's the designer's data model.  Are we going to MUST them?

I didn't mean the agent -- I meant the DM writer.

> So the modeler will say "Hey! I have a key" and use the in their

Good -- we just need to make the text more clear

> I'm lost here.  Why would you ever need to list the leaf more than once?

Fine -- leave this CLR in the draft.  Just explain why it is needed.

> abuses is overwhelming.  Someone's guaranteed to put their eye out

IMO, the 'my implementation uses more memory' argument
is really weak to justify huge CLRs that are
more restrictive than SMIv2 tables.

The draft does not force all the key components to be first.
That means that an agent implementation has to buffer NETCONF PDUs
that contain non-key nodes (deep or not) until all the key nodes
are received anyway.  What difference does it make if
the static key is 'foo' or 'foo/bar' or '/foo/bar/baz'.

I do not see the ability to index even 1 layer deep into a reusable
container as a weak use case.  Not every DM writer is going
to want to place every leaf at the same sibling level.
That is what SMIv2 has to offer, and it isn't good enough.

>  Phil

Abdy