Subject: Re: [Netconf] 4741bis: issue 13, confirmed-commit (new capability)

From: Martin Bjorklund <[email-masked]>
Date: Tue, 16 Mar 2010 22:24:42 +0100 (CET)

Hi,

Andy Bierman <[email-masked]> wrote:
> Hi,
> 
> First there was the Robust Configuration draft, which proposed
> a verified-commit procedure.  Then there was some consensus
> towards decoupling the verify and commit procedures.  Then there
> was a suggestion to improve the commit procedure with a new
> capability.  Old clients would still work because there would
> be 1 or more new parameters that must be present to select the
> new behavior.  The new confirmed-commit-1.1
> capability would add a <persist> parameter to the
> <commit> operation.
> 
> There are no proposed edits at this time.
> The WG needs to decide on a basic solution approach first:
> 
>   A) add a new capability which changes the commit procedure
>      to allow the original server to terminate without
>      canceling the confirmed commit;  detailed proposal
>      from Martin is still needed, and WG consensus on
>      all the details is needed

I think there are 4 problems with the :confirmed-commit:1.0
capability.

  1.  It is not clear if the confirming commit MUST be sent on the
      same session which sent the confirmed commit.

      Since the confirmed commit is aborted if the original session is
      terminated, it seems to make sense that the confirming commit
      MUST be sent on the original session.  Also, allowing the
      confirming commit to be sent on other sessions opens up
      interesting access control questions.

      FWIW, I know two different implementations that has interpreted
      the text this way.

  2.  The text says:

         Thus to cancel a confirmed commit and revert changes without
         waiting for the confirm timeout to expire, the client can
         explicitly restore the configuration to its state before the
         confirmed commit was issued.

      But there is no way to "explicitly restore" the configuration.

  3.  If another client takes a lock when the confirming commit is
      pending, it will be surprised if the config changes due to a
      rollback.

      Proposal: do not allow lock when a confirming-commit is pending. 

  4.  Since the confirmed commit is aborted if the session terminates,
      there is no way to use the powerful confirmed commit operation
      if the interface on which the NETCONF session is transported is
      modified so that the NETCONF session dies (e.g. changing the IP
      address).  But in these situations the confirmed commit feature
      is needed the most - if the configuration is broken in some way
      so that the interface actually is down, we want the device to
      roll back when the timer expires.

I propose a new capability, :confirmed-commit:1.1 to solve 2 and 4
above:

** Overview

This capabaility introduces two new parameters to the <commit>
operation; <persist> and <persist-id>.

The <persist> parameter is used to make the confirmed commit survive a
session termination, and set a cookie on the ongoing confirmed
commit.

The <persist-id> parameter is used to issue a confirming commit from
another session, using the cookie from the previous commit operation.

It also adds a new RPC operation, <cancel-commit>, which can be used
to abort a confirmed commit.

** YANG data model

This data model defines the new commit capability.

  augment /nc:commit/nc:input {
    leaf persist {
      type persist-id;
      description
        "This parameter is used to make a confirmed commit
        persistent.  This means that the confirmed commit is not
        aborted if the NETCONF session terminates.  The only way to
        abort a persistent confirmed commit it to let the timer
        expire, or to use the cancel-commit operation.

        The value of this parameter is a cookie that must be given
        in the 'persist-id' parameter of commit or cancel-commit in
        order to confirm or cancel the persistent confirmed
        commit.

        The cookie should be a random string.";
    }
    leaf persist-id {
      type persist-id;
      description
       "This parameter must be given in order to commit a persistent
        confirmed commit.  The value must match the value given in
        the 'persist' parameter to the commit operation.  If it does
        not match, a 'bad-persist-id' error-app-tag is returned.";
    }
  }

  rpc cancel-commit {
    description
     "This operation is used to cancel an ongoing confirmed commit.
      If the confirmed commit is persistent, the parameter
      'persist-id' must be given, and it must match the value of the
      'persist' parameter.";
    input {
      leaf persist-id {
        type persist-id;
        description
         "This parameter must be given in order to cancel a persistent
          confirmed commit.  The value must match the value given in
          the 'persist' parameter to the commit operation.  If it does
          not match, a 'bad-persist-id' error-app-tag is returned.";
      }
    }
  }

** Example

The following RPC starts a persistent confirmed-commit:

  <rpc message-id="108"
       xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <commit>	
      <confirmed/>
      <persist>foo</persist>
    </commit>
  </rpc>

The following RPC confirms a persistent confirmed-commit:

  <rpc message-id="109"
       xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <commit>	
      <persist-id>foo</persist-id>
    </commit>
  </rpc>

The following RPC aborts a persistent confirmed-commit:

  <rpc message-id="110"
       xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <cancel-commit>
      <persist-id>foo</persist-id>
    </cancel-commit>
  </rpc>

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 16 Mar 2010 14:44:51 -0700

>       FWIW, I know two different implementations that has interpreted

It is clear -- the confirmed commit is rolled back
if the session that started it is terminated.

>       But there is no way to "explicitly restore" the configuration.

close-session

>       Proposal: do not allow lock when a confirming-commit is pending. 

This seems fine, along with many other small changes,
except the WG already decided not to change the base capability,
so there is no way for a client to tell an RFC4741 server
from a 4741bis server.  An old client is not going
to be looking for a 1.1 confirmed-commit capability,
so anything that worked before (even gray areas like this)
need to be considered compliant.

What is a server supposed to do?
Return this error only if 1.1 is advertised?
That does not help, since the client only advertises
the protocol base capability, and is only looking
for the confirmed-commit:1.0 capability, which has no rule
to send an error in this corner-case.

There is a simple solution -- create a 1.1 protocol
and obsolete the 1.0 version.  Happens all the time.
Let 4741 servers keep doing whatever they are doing
if the client advertises only the 1.0 base uri.
Use the highest version in common to settle on a
protocol version in use.

If a 1.1 server is talking to a 1.1 client, it must follow
all the new rules that 4751bis adds, otherwise it is
using 1.0 rules and can do whatever is compliant for 1.0.
The capability-changed problem can be solved, warnings
can be solved, etc.

> I propose a new capability, :confirmed-commit:1.1 to solve 2 and 4

I think this work should be chartered separately, and be published
in a different RFC.  The requirements for Robust Configuration
are broader in scope and more functionally complete.
I would like the charter to be crafted carefully,
not just another ad-hoc add-on to NETCONF.

I would like the NETCONF protocol to stabilize soon so
we can advance to Draft Standard on the base protocol
and add bells and whistles in a way that does not
require the base to keep cycling at Proposed Standard.

Andy

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Johan Rydberg <[email-masked]>
Date: Wed, 17 Mar 2010 12:46:21 +0100

> I propose a new capability, :confirmed-commit:1.1 to solve 2 and 4

Why not make confirmation of a commit a first class operation?

The semantics that a commit confirms another commit can be confusing
to some people (including me).

---

From: Phil Shafer <[email-masked]>
Date: Wed, 17 Mar 2010 07:44:00 -0400

>if the session that started it is terminated.

I would like to see this declared as a bug in the spec:

   If the session issuing the confirmed commit is terminated for any
   reason before the confirm timeout expires, the server MUST restore
   the configuration to its state before the confirmed commit was
   issued.

I dislike this because it means that changes that affect the session's
path to the device are not possible.  I can't commit confirmed,
discard my now useless session, open a new one, see that all is
good, and issue a confirming commit.   I see this as useful
functionality is it's how the commit confirmed originally worked
in JUNOS.

>>       But there is no way to "explicitly restore" the configuration.

That's not explicit.  Explicit means you reload or rollback the
config by hand (from a saved copy).

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 13:22:07 +0100 (CET)

> if the session that started it is terminated.

*That* is clear, I agree.  But that's not what I wrote.  The question
 is if the confirming commit can be sent over a different session.

> >       But there is no way to "explicitly restore" the configuration.

If this is what the text really refers to, the text should mention it,
and not call it explicit:

          Thus to cancel a confirmed commit and revert changes without
          waiting for the confirm timeout to expire, the client can
          restore the configuration to its state before the
          confirmed commit was issued by closing the session.

But I don't think close-session is what the original text refered
to...

> need to be considered compliant.

What does this have to do with confirmed-commit:1.1???

This problem (3 in my list), points out a problem with interaction
between lock and confirmed commit.  Do you agree that if a session has
a lock, the db MUST NOT change?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 05:25:54 -0700

> I would like to see this declared as a bug in the spec:

It was not considered a bug when it was written.
The WG was quite aware how confirmed-commit.
The text clearly says that if the session
that starts the commit is terminated for any reason,
the confirmed commit fails and is rolled back.
This is not a bug.  This is how the standard is defined.

> functionality is it's how the commit confirmed originally worked

This should have been addressed when the RFC was being written.

I object to any changes to 4741 that cause an old server
or old client (compliant to RFC 4741) to be considered a bug
in the new version.

Adding a new capability as Marting suggests will break
existing clients and servers, so it is a non-starter.
We can obsolete 4741 and start over with 1.1 -- that is an acceptable
migration path.

> config by hand (from a saved copy).

Explicit just means by client command;
this was the choice of the WG at the time.
The addition of more commands was discussed and rejected.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 05:29:40 -0700

> What does this have to do with confirmed-commit:1.1???

You cannot add 1.1 because it changes the way 1.0 works.
An old client doesn't know about 1.1, only 1.0.
Advertising 1.1 breaks the 1,0 client.
IMO, this is a non-starter.

Advertising 1.1 and not 1.0 breaks old clients as well.

You cannot declare previously acceptable protocol behavior
to now be an error.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 13:45:24 +0100 (CET)

> You cannot add 1.1 because it changes the way 1.0 works.

I'm sorry, but how does 1.1, as I proposed, chnage the way 1.0 works?

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 13:50:53 +0100 (CET)

> or old client (compliant to RFC 4741) to be considered a bug

The problem with most of the things we're discussing lies in the term
"compliant".  If the spec is unclear so that different implementations
behave differently, it should be clarified.  We are chartered to fix
bugs, and we can even do extensions in order to fix bugs.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 08:41:31 -0700

> I'm sorry, but how does 1.1, as I proposed, chnage the way 1.0 works?

I already explained it.
For a client that doesn't know about your 1.1 capability
or the verified-commit capability from my draft, that it
is similar, this changes the 1.0 contract.

Behavior which is unspecified on a gray area in 1.0 needs
to stay that way in 1.0.

If a 1.0 client starts getting errors that didn't happen
before (e.g., return an error for a lock request when
a c-commit is pending), then the server is broken.

If the server advertises 1.0 and 1.1, how does it maintain
its 1.0 behavior for old clients?
I understand how the new client can opt-in by using a new parameter,
but new errors are different.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 17 Mar 2010 17:05:11 +0100

> a c-commit is pending), then the server is broken.

If the behaviour of some protocol feature is unspecified, should a
well written client not be able to deal with any behaviour?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 17:15:17 +0100 (CET)

> is similar, this changes the 1.0 contract.

No!  Please read my proposal again.  The confirmed-commit:1.1 does not
in any way change any error code -- it just adds new optional
operations & parameters.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 09:18:59 -0700

> well written client not be able to deal with any behaviour?

No -- the client will end up dealing with specific server
behavior that it needs to support.

You and Martin seem to be completely missing my point, so
I will repeat it.  There are well understood mechanisms
for distinguishing 2 separate versions of a protocol.

This WG does not want to use any of them, and instead
insists that a 1.0 client will just deal will all the
changes in a 1.1 server, and if it doesn't,
it must be the 1.0 client's fault for using
a poorly specified standard.

Adding to an existing set of capabilities does nothing
to tell an old client it is dealing with a new server
and a new version of the protocol.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 17 Mar 2010 19:21:48 +0100

> behavior that it needs to support.

But since there is no standard behaviour, such a client will misbehave
anyway if it hits a certain type of servers.

> a poorly specified standard.

I suggest you leave it to chairs to determine concensus. That said, if
a certain protocol behaviour is not defined and a client then silently
assumes a certain behaviour, the client is at its own risk.

> and a new version of the protocol.

Yes, but for the old client, nothing has changed since the expected
server behaviour was undefined. Now we can go on in circles, but
perhaps we should also look at how many deployed clients are affected
by this. The larger this number and the more mission critical they
are, the better for all of us.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 20:09:55 +0100 (CET)

> for distinguishing 2 separate versions of a protocol.

Andy, I think *you* are missing *my* point.  The proposed
:confirmed-commit:1.1 capability does not change existing behavior at
all.  It adds some optional parameters and one optional rpc.

Further, the nice thing with all the capabilities is that we *can*
update single capabilities without updating the base protocol.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 14:03:19 -0700

> all.  It adds some optional parameters and one optional rpc.

I am talking about the error for locking the candidate,
and other changes in the spec that are not covered
by your capability.

I understand that lots of little 'non-changes', and protocol versions,
and protocol permutations, do not make interoperability easier for the client,
but rather makes it much worse.

> update single capabilities without updating the base protocol.

I completely disagree that this is a feature.
The nice thing about a version update is that it
is clear which protocol version you are using.

The confirmed-commit:1.1 capability is optional,
and only relevant if candidate and cc:1.0 are also
supported, it cannot be used as the simple way for all clients
to tell which RFC version each server supports.

Since a base:1.1 server is required to act like a
base:1.0 server if the client only requests base:1.0,
how does this even affect clients that do not wish
to update?  Why do you see a version update as harmful,
when it has no affect whatsoever on old clients
and old servers?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 17 Mar 2010 22:16:44 +0100 (CET)

> and other changes in the spec that are not covered

Ok, good, so you do agree that the proposed :confirmed-commit:1.1 does
not break old clients.

Re. the lock, I will start by (re-)asking a simple question:

  Do you agree that when the lock is taken, the db MUST NOT be
  modified by anyone but the session that owns the lock?

> but rather makes it much worse.

For better or worse, NETCONF has many capabilities, each with its own
version number.

> when it has no affect whatsoever on old clients

I haven't said that I think a base:1.1 is harmful.  But I do think we
should avoid it if we can.  It all boils down to the changes we want
to make.  For example, if we decide we want to add/redesign warnings,
we must do a base:1.1.  I do not think that any of the changes we
currently have require a new base version.

[If we do make a base:1.1, we can get rid of all capabilities and use
features instead.  Then the protocol will work the way you describe
above - the client can immediately know which rfc the server
implements.]

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 15:40:49 -0700

> Ok, good, so you do agree that the proposed :confirmed-commit:1.1 does

no,  I am not against adding the new functionality,
especially since it looks so similar to the proposal
I had in an earlier draft.

IMO, we should use base:1.1 for the entirety of the 4741bis
updates.  There is no possibility of hiding a new version.
All the WEB pages and RFC Index will say

   RFC 4741 (Obsoleted by RFC 5xxx).

>   modified by anyone but the session that owns the lock?

I agree with you that the lock should not be granted to any session
while the candidate rollback is possible.

I just want to make it clear to users and clients which
protocol version has the mandatory server error, and which
is TBD by the client.

> For better or worse, NETCONF has many capabilities, each with its own

But these are not all bad.
We started out (guided by Phil) making sure that
we could model real-world native platforms.
(IMO, we may have done that too well to get any standard content ;-)

We should consider all the c-commit:1.1 changes
to be bugfixes, because we have consensus on the need
for a more robust solution.  That means they are automatic
if the server supports c-commit:1.0 and base:1.1.

> above - the client can immediately know which rfc the server

The client will know either way.

IMO, a base:1.1 is cleaner because we keep wanting
to fix things that were not quite right in 1.0.
We cannot possibly add capabilities for every behavior
change, and in the long run, nobody will want that.

I would like to get rid of all the little capabilities
and use YANG features instead.  Right now, I have not bothered
to implement ietf-netconf.yang directly, since nobody supports
it and I have to know all the capability values anyway for 1.0.

Unless we put a stake in the ground and say we want people
to use the 1.1 version, we have to support the 1.0 flaws forever.

Users need to know how to tell the different RFCs apart,
based on protocol version, or something every server supports.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 17 Mar 2010 15:51:56 -0700

> Ok, good, so you do agree that the proposed :confirmed-commit:1.1 does

Is there some reason why this cannot be added as a YANG feature
and not a capability at all, since ietf-netconf.yang is
normative?  It's just different strings in the <hello> content
to the protocol.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 18 Mar 2010 08:39:13 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> normative?

The capability string for base 1.0 must not contain any query
components.  An old client will probably not understand

 <capability>
   urn:ietf:params:netconf:base:1.0?features=confimred-commit-1.1
 </capability>

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 18 Mar 2010 09:18:22 -0700

>    urn:ietf:params:netconf:base:1.0?features=confimred-commit-1.1

This would never happen because a 4741-compliant server does
not get to pick and choose pieces of 4741bis to implement.

But actually, doesn't the ietf-netconf.yang module capability
break this supposed rule?  According to the YANG spec,
a server using ietf-netconf.yang would need to advertise:

 <capability>
   urn:ietf:params:netconf:base:1.0?module=ietf-netconf&revision=2010-xx-xx&feature=candidate ...
 </capability>

I know my code would not handle this right now because the
spec says this specific URI must exactly match the string.

IMO, this is another reason base:1.1 would help.
The rule needs to be refined for 4741bis, so the XML namespace
is an exact match to the capability URI base part (w/o any parameters).

The client needs to do this already for YANG module capability URIs.

Andy