Subject: Re: [Netconf] restconf confusing text

From: [email-masked] (Dale R. Worley)
Date: Mon, 05 Sep 2016 16:35:58 -0400

The whole situation is annoying and non-systematic.

draft-ietf-netconf-restconf-15 section 3.5.3 says, in regard to
constructing request URLs:

   If a data node in the path expression is a YANG list node, then the
   key values for the list (if any) MUST be encoded according to the
   following rules:

   o  The key leaf values for a data resource representing a YANG list
      MUST be encoded using one path segment [RFC3986].

   o  If there is only one key leaf value, the path segment is
      constructed by having the list name, followed by an "=" character,
      followed by the single key leaf value.

   o  If there are multiple key leaf values, the path segment is
      constructed by having the list name, followed by the value of each
      leaf identified in the "key" statement, encoded in the order
      specified in the YANG "key" statement.  Each key leaf value except
      the last one is followed by a comma character.

   o  The key value is specified as a string, using the canonical
      representation for the YANG data type.  Any reserved characters
      MUST be percent-encoded, according to [RFC3986], section 2.1.

   o  All the components in the "key" statement MUST be encoded.
      Partial instance identifiers are not supported.

Though the wording is a bit awkward, it's disambiguated by the example:

    /restconf/data/example-top:top/list1=key1,key2,key3/list2=key4,key5/X

The problem is that non-config lists need not have keys, and the
semantics of a URL derived from an unkeyed list and the URL from a keyed
list are different:

    /restconf/data/example-top:top/keyed-list=key1,key2,key3

    /restconf/data/example-top:top/unkeyed-list

The first identifies one element of the list, the second identifies the
entire list.  Which is not what one would expect at first look.  But
given the text

   If a retrieval request for a data resource representing a YANG leaf-
   list or list object identifies more than one instance, and XML
   encoding is used in the response, then an error response containing a
   "400 Bad Request" status-line MUST be returned by the server.

it's clear that's what the authors intended.

If the response will be provided in XML, the problem is that the list as
a whole is not represented as an XML element, but rather as a series of
XML elements contained within an element for the parent data node of the
list.  (draft-ietf-netmod-rfc6020bis-14, section 7.8.3.1) And an XML
document must have a single topmost element.  So, as you say, if there
is no <collection> element that can be wrapped around the set of list
items, there is no way to represent a list as a whole (unless it
contains exactly one element).  (I'm not sure *why* there is no element
for the list as a whole.  That would seem natural to me, but apparently
it's not the practice in device configuration data.)

OTOH, in the JSON representation of a Yang data tree (which I recently
learned is specified in RFC 7951), the list as a whole *does* have a
first-class syntax representation:

    "keyed-list" : [
      { "key1" : "value1a",
	"key2" : "value2a",
	"key3" : "value3a",
	"field" : "value" } ,
      { "key1" : "value1b",
	"key2" : "value2b",
	"key3" : "value3b",
	"field" : "value" } ]

and so it can be returned in a JSON response.

Dale

---

From: Kent Watsen <[email-masked]>
Date: Thu, 8 Sep 2016 17:08:51 +0000

My recommendation is for us to not ignore this issue.  I think that the draft should define what is the expected response for when a list or leaf-list is referenced.

Assuming we define the often-discussed fake-root called “collection”, the XML response for Dale’s example would be:

  <collection xmlns=”urn:ietf:params:xml:ns:yang:ietf-restconf">
    <keyed-list xmlns=”http://example.com/test”>
      <key1>value1a</key1>
      <key2>value2a</key2>
      <key3>value3a</key3>
      <field>value</field>
    </keyed-list>
    <keyed-list xmlns=”http://example.com/test”>
      <key1>value1b</key1>
      <key2>value2b</key2>
      <key3>value3b</key3>
      <field>value</field>
    </keyed-list>
  </collection>

Even though it’s not necessary, we could do something similar to the JSON response, rationalizing that it’s more a RESTCONF-thing than an encoding-thing, but I can also see it being an encoding thing.  Either way is fine with me.

Thoughts?

Kent

On 9/5/16, 4:35 PM, "Netconf on behalf of Dale R. Worley" <[email-masked] on behalf of [email-masked]> wrote:

    

    

    

    

    

    
    
        "keyed-list" : [
          { "key1" : "value1a",
    	"key2" : "value2a",
    	"key3" : "value3a",
    	"field" : "value" } ,
          { "key1" : "value1b",
    	"key2" : "value2b",
    	"key3" : "value3b",
    	"field" : "value" } ]

    Dale
    
    _______________________________________________
    Netconf mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/netconf

---

From: "Steve Truong (struong)" <[email-masked]>
Date: Thu, 8 Sep 2016 17:17:02 +0000

IMO the following constructs should be supported (using Dale's example) to retrieve an entire list:

        /restconf/data/example-top:top/keyed-list

Thanks,
steve

-----Original Message-----
From: Netconf [mailto:[email-masked]] On Behalf Of Kent Watsen
Sent: Thursday, September 08, 2016 10:09 AM
To: Dale R. Worley; [email-masked]
Subject: Re: [Netconf] restconf confusing text

Thoughts?

Kent

    

    

    

    

    

    

        "keyed-list" : [
          { "key1" : "value1a",
    	"key2" : "value2a",
    	"key3" : "value3a",
    	"field" : "value" } ,
          { "key1" : "value1b",
    	"key2" : "value2b",
    	"key3" : "value3b",
    	"field" : "value" } ]

    Dale

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 Sep 2016 10:23:51 -0700

Hi,

What is the issue that needs to be addressed?

There is already text that says responses that would produce invalid XML
are not allowed.
IMO this is sufficient.  We could add a sentence that says the encoding of
list keys only
applies to lists that define keys.  That seems obvious but maybe not to
everyone.

I do not want to add any more features to RESTCONF.
The RESTCONF Collections draft expired.
That work can either get dropped or that draft finished, but it
cannot get moved back into the RESTCONF draft.

Andy

On Thu, Sep 8, 2016 at 10:08 AM, Kent Watsen <[email-masked]> wrote:

> My recommendation is for us to not ignore this issue.  I think that the
> draft should define what is the expected response for when a list or
> leaf-list is referenced.
>
> Assuming we define the often-discussed fake-root called “collection”, the
> XML response for Dale’s example would be:
>
> Even though it’s not necessary, we could do something similar to the JSON
> response, rationalizing that it’s more a RESTCONF-thing than an
> encoding-thing, but I can also see it being an encoding thing.  Either way
> is fine with me.
>
> Thoughts?
>
> Kent
>
>
> On 9/5/16, 4:35 PM, "Netconf on behalf of Dale R. Worley" <
> [email-masked] on behalf of [email-masked]> wrote:
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 Sep 2016 10:25:09 -0700

Hi,

I strongly object to adding new features at this time.
This issue was moved to the RESTCONF Collections draft.
It should be solved there.

Andy

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 8 Sep 2016 19:19:48 +0000

> draft should define what is the expected response for when a list or

+1

Thanks

Michael

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 Sep 2016 12:26:01 -0700

Hi,

There is this text in sec 4.3

   If any content is returned to the client, then the server MUST send a
   valid response message-body.  More than one element MUST NOT be
   returned for XML encoding.  If multiple elements are sent in a JSON
   message-body, then they MUST be sent as a JSON array.  In this case
   any timestamp or entity-tag returned in the response MUST be
   associated with the first element returned.

   If a retrieval request for a data resource representing a YANG leaf-
   list or list object identifies more than one instance, and XML
   encoding is used in the response, then an error response containing a
   "400 Bad Request" status-line MUST be returned by the server.  The
   error-tag value "invalid-value" is used in this case.  Note that a
   non-configuration list is not required to defined any keys.  In this
   case, retrieval of a single list instance is not possible.

Seems clear to me.
Do you have OLD and NEW suggested text?

Andy

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 8 Sep 2016 19:42:47 +0000

From my point of view, Section 4.3 lacks examples.

Michael

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, September 08, 2016 9:26 PM
To: Scharf, Michael (Nokia - DE)
Cc: Kent Watsen; Dale R. Worley; [email-masked]
Subject: Re: [Netconf] restconf confusing text

Hi,

Seems clear to me.

Andy

+1

Thanks

Michael
_______________________________________________
Netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 Sep 2016 12:52:04 -0700

> From my point of view, Section 4.3 lacks examples.

examples of what?

There are lots of GET examples in section D.1

Andy

>
>
> *From:* Andy Bierman [mailto:[email-masked]]
> *Sent:* Thursday, September 08, 2016 9:26 PM
> *To:* Scharf, Michael (Nokia - DE)
> *Cc:* Kent Watsen; Dale R. Worley; [email-masked]
> *Subject:* Re: [Netconf] restconf confusing text
>
>
>
> Hi,
>
>
>

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 8 Sep 2016 20:14:16 +0000

The examples for lists (in JSON) are pretty well hidden in appendix D.1. And unlike in other Sections, in Section 4.3 there is not even a reference to the relevant examples in D.1.

Michael

Sent: Thursday, September 08, 2016 9:52 PM

On Thu, Sep 8, 2016 at 12:42 PM, Scharf, Michael (Nokia - DE) <[email-masked]<mailto:[email-masked]>> wrote:
From my point of view, Section 4.3 lacks examples.

examples of what?

Michael

Andy

From: Andy Bierman [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, September 08, 2016 9:26 PM
To: Scharf, Michael (Nokia - DE)
Cc: Kent Watsen; Dale R. Worley; [email-masked]<mailto:[email-masked]>
Subject: Re: [Netconf] restconf confusing text

Hi,

Seems clear to me.

Andy

+1

Thanks

Michael

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 Sep 2016 13:18:42 -0700

On Thu, Sep 8, 2016 at 1:14 PM, Scharf, Michael (Nokia - DE) <
[email-masked]> wrote:

> The examples for lists (in JSON) are pretty well hidden in appendix D.1.
> And unlike in other Sections, in Section 4.3 there is not even a reference
> to the relevant examples in D.1.
>
>
>

This is an oversight.
I will add a sentence in 4.3 about the examples in D.1.

There is a 21 page section called "RESTCONF Message Examples".
That doesn't seem to be well-hidden.

Andy

> *Sent:* Thursday, September 08, 2016 9:52 PM

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 8 Sep 2016 20:29:03 +0000

Well, it took me a lot of time to reverse-engineer the handling of GETs for lists from the whole I-D. From an implementer’s perspective, the document is quite hard to read for that specific case of list handling. And the lack of examples for lists is quite surprising to me. Of course, your mileage may vary.

Michael

Sent: Thursday, September 08, 2016 10:19 PM

On Thu, Sep 8, 2016 at 1:14 PM, Scharf, Michael (Nokia - DE) <[email-masked]<mailto:[email-masked]>> wrote:
The examples for lists (in JSON) are pretty well hidden in appendix D.1. And unlike in other Sections, in Section 4.3 there is not even a reference to the relevant examples in D.1.

This is an oversight.

Michael

Andy

examples of what?

Michael

Andy

Hi,

Seems clear to me.

Andy

+1

Thanks

Michael

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 09 Sep 2016 10:45:48 +0200 (CEST)

Kent Watsen <[email-masked]> wrote:
> My recommendation is for us to not ignore this issue.  I think that
> the draft should define what is the expected response for when a list
> or leaf-list is referenced.

The WG agreed to not do 'collections' in the base RESTCONF spec, but
do it later.  This means that the base spec must allow for this future
extensibility (or we need to rev the RESTCONF protocol).

I'm worried that the current text is too restrictive.  Also, I don't
think it is a good idea to allow a certain resource to be returned
only if the format is json, but not if it is xml (see more below).

> Assuming we define the often-discussed fake-root called “collection”,
> the XML response for Dale’s example would be:
> 
> Even though it’s not necessary, we could do something similar to the
> JSON response, rationalizing that it’s more a RESTCONF-thing than an
> encoding-thing, but I can also see it being an encoding thing.  Either
> way is fine with me.

I prefer to do something similat in the JSON response.  I.e., don't
introduce clever optimizations for some formats which lead to
inconsistencies in what the protocol can do ("oh, if want to do X you
need to use JSON").

/martin

> 
> Thoughts?
> 
> Kent
> 
> 
> On 9/5/16, 4:35 PM, "Netconf on behalf of Dale R. Worley"
> <[email-masked] on behalf of [email-masked]> wrote:
>

---

From: Kent Watsen <[email-masked]>
Date: Fri, 9 Sep 2016 13:38:31 +0000

On 9/9/16, 4:45 AM, "Martin Bjorklund" <[email-masked]> wrote:

[KENT] understood.  here I’m suggesting we explore the grey area.  Let’s define the fake “collections” root now, but not define anything else, such as query parameters for sorting, paging, or complex filtering.  I’m thinking that we might define just this bit, in order to resolve Dale’s issue, and let the rest be defined later.

[KENT] +1

Kent

---

From: Andy Bierman <[email-masked]>
Date: Fri, 9 Sep 2016 07:02:10 -0700

On Fri, Sep 9, 2016 at 6:38 AM, Kent Watsen <[email-masked]> wrote:

> [KENT] understood.  here I’m suggesting we explore the grey area.  Let’s
> define the fake “collections” root now, but not define anything else, such
> as query parameters for sorting, paging, or complex filtering.  I’m
> thinking that we might define just this bit, in order to resolve Dale’s
> issue, and let the rest be defined later.
>

>     only if the format is json, but not if it is xml (see more below).

This problem has existed from Day One.
News Flash: JSON has arrays and XML does not.
YANG-API had a <data> container,
People did not want an extra container added so it got removed,
even though I pointed out (and probably others) that taking out the
container means that XML cannot return more than one data node.

This was understood and accepted.
Now after 4 years it is suddenly unacceptable and we need to go back
to what YANG-API defined in the first place? Except now the 4 byte name
"data"
is not good enough, we need to waste 11 bytes on the name instead?

Andy

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 09 Sep 2016 14:10:44 -0400

Andy Bierman <[email-masked]> writes:
> There is already text that says responses that would produce invalid
> XML are not allowed.  IMO this is sufficient.  We could add a sentence
> that says the encoding of list keys only applies to lists that define
> keys.  That seems obvious but maybe not to everyone.

IMO it would be far clearer if the discussion of the encoding of list
nodes noted:  For a keyed list, the key values are mandatory (which is
already said) and that the resource that is specified is the specific
list element (which is intuitive).  But for an unkeyed list, the key
values are omitted (obviously) but that the resource that is specified
is the collection of items in the list (which is much less intuitive).
The complexity arises when one then applies further path components
after an unkeyed list component.

All of this is "intuitive", as that is about the only sensible meaning
that the text has, and it is reinforced by bits of text scattered
throughout the draft.  But the complexity of the situation isn't
immediately apparent on the first reading, particularly the fact that
you can write URLs that mean "the component 'foo' from each element of
the list 'bar'".  So I think it would be helpful if some sentences of
explanation and warning were added.

Dale

---

From: Andy Bierman <[email-masked]>
Date: Fri, 9 Sep 2016 11:30:24 -0700

> explanation and warning were added.

I don't see any obvious edits here.
"the key values are omitted" -- this is confusing because these
lists do not have any keys defined so there is nothing to omit.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 9 Sep 2016 13:51:58 -0700

> issue, and let the rest be defined later.
I prefer to define standards carefully, and only release "real code", not
hooks for
future projects.

I cannot find the emails from 2012 that discussed why the <data> container
was taken out,
but from memory:

   - when should the <data> container be returned?
      A - always
      B - all XML and never JSON
      C - just list and leaf-list for XML
      D - just leaf and leaf-list for XML that have more than one instance
      E - never

I don't think any consensus was reached and it was just removed (E)

IMO (C) would be an operationally acceptable solution.
So what if XML and JSON representations are not the same.
They are already not the same. (That's the problem  ;-).

>     only if the format is json, but not if it is xml (see more below).

This has been the situation for 4 years. Why is this issue just coming up
(again)
after WGLC?

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 12 Sep 2016 17:54:16 -0400

> News Flash: JSON has arrays and XML does not.

Actually, XML *can*, but the use of XML that Yang has chosen does not.

For instance, a container of three items is one XML element wrapped
around the elements for the three items:

    <container>
        item 1
        item 2
        item 3
    </container>

but a list of three items is three XML elements with no enwrapping
element:

    <list>
        item 1
    </list>
    <list>
        item 2
    </list>
    <list>
        item 3
    </list>

Dale

---

From: Phil Shafer <[email-masked]>
Date: Mon, 12 Sep 2016 20:19:27 -0400

>Actually, XML *can*, but the use of XML that Yang has chosen does not.

Viewed another way, XML allows you to encode anything, but YANG
gives context to allow the data to be interpreted as arrays.  It's
the rules and data model that allow us to "know" what the data
means.

JSON gives some indication of JSON type (true/false, lack of quotes
for numbers, [] for arrays) (which may not be the real datatype,
e.g. uint64) but doesn't tell us vital information like keys, range,
real datatype, etc.  I'll see an array called interfaces, but I
won't know that the "name" field is the key, or that it's an IP
address.

All the real meta-data is in the data model, and the real magic is
having a YANG-aware decoder that turns the interfaces list into a
dictionary with a the interface name as the index.  Once you have
that, the presence of an containing element around the list is
pointless.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Sat, 24 Sep 2016 09:35:56 -0700

Hi,

There has been no further discussion on this issue so there
will not be any changes to the related text in RESTCONF-17.

Andy