Subject: Re: [Netconf] 4741bis, issue 14, capabilities changed error

From: Andy Bierman <[email-masked]>
Date: Tue, 12 Jan 2010 09:32:34 -0800

Martin Bjorklund wrote:
> Andy Bierman <[email-masked]> wrote:
...
> This is why I proposed the following text in 
> http://www.ietf.org/mail-archive/web/netconf/current/msg04512.html:
> 
>   The capabilities of a NETCONF server may change over time.  However,
>   once a NETCONF server has announced its capabilities in the <hello>
>   message, the capabilities for that session MUST NOT change.  A
>   server MUST reply with a 'capabilities-changed' error if the client
>   sends a request which is affected by a modified capability.
> 
> And maybe add:
> 
>   A server MAY choose to send 'capabilities-changed' as the response
>   to any request other than <close-session> if its capabilities has
>   changed.
> 
> 
> Provided that 'capabilities-changed' is an error-app-tag, this is
> backwards compatible.
> 
> If we want to create a new fancy mechanism to resync modified
> capabilities, I agree we can do that in netconf 1.1.
> 
> 

We need to close all these 4741 open issues.
This is the last one (because a new confirmed-commit
capability is punted to another release).

My concern is the 2nd sentence (MUST NOT change).

Here are some use-cases where this is not a feature:

  1) load module command
      A server may support run-time loading of a YANG module
      and its corresponding instrumentation.  The 'MUST not change'
      CLR makes it impossible for a NETCONF session to change the
      system capabilities.

  2) preserved session utility
      It is impossible to preserve every open session's particular
      view of the system.  It is also misleading, and may cause the
      operator to do incorrect things, or be limited to some
      un-discoverable vendor-specific set of operations.

  3) plug in a new HW module
     The addition of more ports, or an redundant power supply
     is not enough reason to disrupt all the NETCONF sessions.
     This is normal operation.

The standard SNMP approach to this problem seems to
be the best choice here as well -- take steps to inform
the client that things have changed, but do not disrupt
any management operations.

It would be trivial to add another leaf to the
ietf-netconf-monitoring module:

  leaf capability-last-changed-time {
      description
         "Contains the date and time that the capabilities
          of the NETCONF server were last changed.";
      type yang:date-and-time;
      mandatory true;
      config false;
  }

A notification should also be added.
This might take longer to complete than a time-stamp,
so it may have to wait.  This is the one I am using:

http://www.netconfcentral.org/modules/yuma-system/2009-12-27#sysCapabilityChange.244

IMO, it is the client's problem to deal with a dynamic system,
not the server's problem to pretend the system is static.

I still don't know what text to add to 4741bis, so I think
I will punt this issue to the March meeting, unless any good
suggestions show up on the mailing list.

> /martin

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 12 Jan 2010 22:46:09 +0100 (CET)

Hi,

> capability is punted to another release).

There are a few other open issues as well.  And why is a new
confirmed-commit capability punted to another release?  IMO the
current one is broken and needs to be fixed.  But that's a separate
thread!

>       CLR makes it impossible for a NETCONF session to change the

Yes, a NETCONF session can change the system capabilities, as long as
the current session is not affected.

>       un-discoverable vendor-specific set of operations.

That's why a server can send the 'capabilities-changed' error for any
request on open sessions.

>      This is normal operation.

Do you agree that if capabilities change, the client needs to somehow
be informed, and it needs to get a chance to resync?  There are a
couple of alternatives:

  1.  As proposed; send an error, and let the client resync by opening
      a new session (can be handled by opening a new channel and
      closing the old one; this is almost as cheap and easy as having
      an explicit 'resync-capabilites' rpc.)

  2.  Send an error as above, plus add a new 'resync-capabilites'
      rpc.  Until the session has done resync, it will get the error
      as in option 1.
 
  3.  Send an error as above, and resync with a notification?  The
      problem with this approach has been discussed before - the
      client needs two sessions, one for the notification and one for
      rpcs.  And it needs to sync between the two.

> any management operations.

With NETCONF and YANG, we have a number of mechanisms to let the
client learn what the server can do, so it does not have to do trial
and error.  The capability list is the contract between the server and
client.  If the contract can change in random ways at any time, it is
worthless.

I think option 1 above is simple, and good enough.  Option 2 should
work, but adds a new rpc, which means a new capability.  If it is
covered by our charter and the WG think it is necessary, fine.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 12 Jan 2010 14:26:46 -0800

> current one is broken and needs to be fixed.  But that's a separate

There was not much interest in changing the commit
operation in this document.  I would not say it is broken.
The limitations were understood at the time it was published.

> the current session is not affected.

That is impossible for the server to do.
Besides, if I load the 'foo' module,
it's because I want to start using it.

We have a major disconnect between the protocol and reality.
The capabilities represent the system capabilities at the
time the session was started.  It makes no sense at all
to talk about freezing them for some server.

If the operator removes a redundant power supply,
there is no way the server is going to 'fake it'
for all the NETCONF sessions that are open.
The capabilities can change over time.

> request on open sessions.

The server could send a capabilities-changed notification
and set the time-stamp.  This gives the client to figure
out what changed and if it matters to that session.

> be informed, and it needs to get a chance to resync?  There are a

The client needs to poll for changes or listen
for notifications.

>       an explicit 'resync-capabilites' rpc.)

but what is the error exactly?
"hey client, you may be doing something wrong!
 not such if and what it is, but time to end the session."

IMO, the server should not assume the client
is clueless and will break.  The server should
just execute the <rpc> requests, in order.

  <rpc>
    <check-the-backup-power-supply />
  </rpc>

If the capabilities change, and the redundant power supply
is gone, then data-model specific content may be affected
like the RPC above.  If the client tries to use this RPC,
then the server can send a definitive error because the
hardware is not available.  The client can poll for changes,
and realize not to try this operation, or just try it and get an error.
I don't see any reason to force one way or the other.

>       rpcs.  And it needs to sync between the two.

4.  set a last-changed time-stamp and send a notification.

> client.  If the contract can change in random ways at any time, it is

This notion that the system capabilities at the time the session
starts is a static contract is fatally flawed because not
all capabilities are the same.  They are just generic markers.
The contract that says "I implement the RFC 4741 version
of NETCONF" is the only one that cannot really change.

Adding a module or changing the HW configuration
are not capability contracts.  They are merely the set of
dynamic capabilities at the current time.

> covered by our charter and the WG think it is necessary, fine.

NMS applications are used to 'directed polling' for system
changes (notification + polling 'change-marker MIB objects').
We have notifications and a <get> operation, so there is no
reason to invent new mechanisms.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 12 Jan 2010 23:53:14 +0100 (CET)

> The client needs to poll for changes or listen

So you do not agree with what I wrote.  Does this means that you view
the capabilities list not as a contract, but more as a hint for the
client?

I strongly object to such an interpretation.  Why bother at all with
keeping track of capabilities if they may change at any time, at any
rate, in any way?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 12 Jan 2010 15:10:28 -0800

> keeping track of capabilities if they may change at any time, at any

I think the capabilities need refinement in order to assume
a particular capability change breaks the current session.
If the client is receiving notifications, then just
sending a capability-change event is good enough.
The <resynch> RPC and capabilities-changes error stuff is overkill.

I think it is up to the client to decide if the session is
usable after a particular capability change.

We can say that the base-NETCONF capability MUST NOT change
during a session.  I don't see any others that represent
contracts -- they represent current configuration
settings or current system hardware state.

If we had a capability-stmt in YANG (like I wanted ;-)
then we could have a static vs. dynamic property,
and an allowed-to-change vs. must-not-change property.
Without that, I do not agree the server MUST do anything
wrt/ a capability change.  It is data model dependent.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 13 Jan 2010 09:36:54 +0100

[...]

I am in favour of adding an rpc called hello, a client initiated
version of the initial hello exchange before the RPC protocol is
started.

rpc hello {
  input {
    container capabilities {
      leaf-list capability {
        type inet:uri;
      }
    }
  }
  output {
    container capabilities {
      leaf-list capability {
        type inet:uri;
      }
    }
    leaf session-id {
      type session-id-type;
    }
  }
}

This hello RPC allows a client to resync capabilities, also covering
the case where client capabilities might have changed dynamically or
where the client lost (for whatever reason) the capabilities announced
at the beginning of the session.

With this hello RPC in place, all we need is a mechanism to let the
server notify clients that capabilities have changed. (And I agree
with the remark I think by Andy that it is the client which should
take a decision whether it is affected by capability change).

- If there is a NETCONF notification channel, all that is needed is a
  notification, like Andy suggested.

- So the remaining case is where the client does not listen to NETCONF
  notifications. One proposed conservative approach is to fail RPCs
  until the hello RPC has been executed to resynchronize. Another
  option is to allow further <hello> messages between <rpc-reply>
  messages - I have not checked whether there is text in 4741
  disallowing this. The third option is to wait until we have a
  working mechanism to send warnings, means NETCONF 1.1.

At this point in time, I would be in favour of (a) adding the hello
RPC outlined above and (b) adding text discussing the issue and
stating that server implementations MAY send capability-changed errors
if capabilities have changed and there is no other mechanism to notify
the client about the capability change.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 13 Jan 2010 01:01:50 -0800

> the client about the capability change.

We already have the capabilities in the monitoring module.
All that is missing is a last-changed timestamp to make
<get> retrieval of the capabilities more efficient.

How many different optional mechanisms can one add,
each time claiming an N+1th optional mechanism is
needed because all N existing solutions are optional?

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 13 Jan 2010 10:15:17 +0100

> <get> retrieval of the capabilities more efficient.

The proposed <hello> RPC is not equivalent to a <get> on the
monitoring module since it also passes the client capabilities to the
server, which I am sure at some point we will find valuable to have.
I believe wrapping the initial <hello> exchange into an RPC is a very
clean and straight forward solution for client initiated synchronization
of capabilities.

/js

PS: I am not a big fan of a polling solution for detecting capability
    changes.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 13 Jan 2010 01:44:42 -0800

> PS: I am not a big fan of a polling solution for detecting capability

There is no requirement now for the server to save
the client capabilities.  The client can remember what
it sent in its <hello>.  The current <hello> obviously
has just the server capabilities, so this will add code
to clients, to deal with a new type of <hello> message.

So how does a client know it needs to call the <hello> RPC?
It seems to me you are replacing a <get> poll with a <hello> poll.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 13 Jan 2010 10:51:10 +0100

> to clients, to deal with a new type of <hello> message.

Take a look at the YANG definition and tell me where you think it does
not follow the NETCONF <hello>. I am talking about the situation where
the client capabilities change and the server should be notified.
 
> It seems to me you are replacing a <get> poll with a <hello> poll.

I am separating the synchronization mechanism from the question how it
is triggered. See my original message.

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 13 Jan 2010 02:47:19 -0800

> the client capabilities change and the server should be notified.

There are no description clauses, just some syntax.

The only server requirement in the protocol is that
the client <hello> contain the base 4741 capability.
Why would the client capabilities change, and why does
that matter to the server?

> is triggered. See my original message.

Sending the <hello> more than once is not backward compatible and
will break existing clients.  I do not see any value in
your RPC.  We already have this data in the monitoring module.

I do not agree that a capability change should alter the current
session at all.  The server can update a timestamp, change the
retrievable capabilities and send out a notification.
This has always been good enough for SNMP and I see no reason
why it isn't good enough for NETCONF.

I do not agree that NETCONF sessions are static,
or that they should be static.  If a client does not
want to poll for system state changes, then it should use notifications.

Nobody has actually demonstrated how an arbitrary capability
change breaks the client.  I agree that the base capability
cannot change during a session, but that is the only one.
Everything else is just part of normal management of dynamic systems.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 13 Jan 2010 12:35:48 +0100 (CET)

> a particular capability change breaks the current session.

I agree.  But currently there is no way to tell.  Some capability
changes might be completely harmless, while others break the
contract.

Your way of dealing with this (ignore capability changes) is trivial
to implement if a capability-changed error is returned.   Your client
code can automatically resync if it gets a capability-changed error.

> The <resynch> RPC and capabilities-changes error stuff is overkill.

I think it is simple to implement, and we get a more robust model.

> usable after a particular capability change.

Sure.  The server informs the client that a capability has changed by
sending the capability-changed error, and the client decides what to
do.

> settings or current system hardware state.

Suppose the server stops supporting the :notification capability.  You
won't even know since you won't get the capability-changed
notification.  No capability in 4741 represents config or system hw
state.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 13 Jan 2010 12:43:56 +0100 (CET)

> >> So how does a client know it needs to call the <hello> RPC?

It knows that it needs to resync when it gets the capabilities-changed
error.

> will break existing clients.

Read Juergen's proposal again.  His proposal is backwards compatible.
He did not propose that <hello> as defined in 4741 should be sent more
than once.

> I do not see any value in

I agree with Jurgen that if we want to support a way to resync, we
should have an explicit operation, not polling some data.  The problem
with polling some data is that it is one-way.  The server does not
know that the client has synced.  This, of course, is only a problem
if we believe that the capability list is more than a hint.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 13 Jan 2010 06:36:14 -0800

> It knows that it needs to resync when it gets the capabilities-changed

I do not agree this is a good idea.
If I have a script that does a small number of steps,
it isn't going to handle a capabilities-changed and resync.

I do not agree that the condition you are describing
represents an error and the session needs to be disrupted.
There are dozens of potential monitoring objects that
one might need to be aware of within the managed server.
The server capabilities are just 1 piece of that data.

If the client attempts an operation that is no longer
supported, then the server has a reason to issue an error.

> if we believe that the capability list is more than a hint.

I don't agree that a capability resync is useful and needs
a new operation added to the protocol.

I don't agree that the server should assume all open sessions
need to have every operation request rejected with an
operation-failed error, as a way of preventing problems
due to a capability change.  This in itself will cause
problems for the client.  The client never needed this
feature in SNMP.  Notifications and polling are sufficient.

Client applications that work now will stop working
as soon as they get an operation-failed error
that never happened before, completely unrelated
to anything the client did or didn't send.

Without a clear definition of a generic capability,
the NETCONF text needs to specify exactly which
capability changes cause particular behavior changes
in the protocol.

Andy