Subject: [NGO] multi-session locks

From: Andy Bierman <[email-masked]>
Date: Mon, 09 Apr 2007 07:56:10 -0700

Hi Balazs,

I am very interested in the partial database locking problem
that we discussed at the last interim.  I think that you are
right wrt/ your concerns about the "1 user global lock" design
for large devices.

There are many issues with implicit locking, especially given
our current architecture and lock operations.  This approach
won't work on the <running> target unless operations are added
to make it work like the <candidate> model instead.  This
would defeat the purpose of having both <running> and <candidate>
targets in the protocol.

IMO it will take a lot of work to re-define either the <candidate>
or the <running> write models to allow for collision-proof multi-user
write access, using implicit locking.

Yet, explicit locking doesn't look so great either.  Is the agent
going to reject an edit-config operation if part of it touches
a locked sub-tree and part of it touches an unlocked sub-tree?
The edit-config operation is designed to work with or without locks,
so rejecting an edit-config operation this way seems wrong.
What happens if another session comes along and locks the altered
but unlocked sub-tree?  At least explicit locking works the same
whether the target is <running> or <candidate>.

One workaround that occurred to me:

Why not let the managers worry about database collisions with a 
multi-session
lock mechanism that works like an access control list, except the
resource is the lock, not the data itself.  In other words, the
manager configures which sessions can share a lock, based on data
outside the scope of the standard.  For example, what if the group-id
was used by the agent instead of the session-id, when checking lock access?
(Access control, users and groups, etc. out of scope for now.)

The grouping could also be an internal implementation detail,
such as all 'sibling' virtual routers associated with one real router.

IMO, this approach is better than not using locks at all,
because unwanted edit-config access will be rejected if the 'group'
is configured correctly.  Also, no other application will be able
to grab the global lock, ruining the cooperative locking in progress.

How do you think the protocol should support multi-session locks?

thanks,
Andy

_______________________________________________
NGO mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ngo

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 10 Apr 2007 10:49:34 +0200

Hello Andy,
For us the question is even more interesting, as parts of our big node configuration might be 
administered by different people/organizations using different management tools; so allowing 
operator A to deny configuration operations for operator B sounds really bad.

IMO we have here two separate questions:
- What can be locked: full data-store, a sub-tree, any group of managed objects?
- Who can used a locked entity: one session, any session connected to one userId or groupId?

IMO the first step in any lock related work is to allow the node to reject the global lock, 
else some old manager application will come and lock the whole node.

The next step (as you point out below) is that partial locking has a difficulty with candidate 
configurations. At least commit, discard-changes, delete-config and copy-config needs a full 
lock (or rather an absence of any lock on any part of the datastore). Theoretically we could 
have a partitioned candidate data-store with partial commit etc. but I think that's too 
complicated to be usable.

All in all I see the following possibilities:
- A complete SQL-database-like implicit locking mechanism works well, but might be a heavy task 
for small nodes
- I also see an important use-case in dividing up the configuration into disjunct parts and 
locking those (e.g. a sub-tree). This way each operator could configure his own part. It is not 
feasible to define the sub-trees at design time.
- Multi-session locks can also be useful, if it is the same person/group using the same lock 
simultaneously. We could utilize something like the groupId or userId for this purpose.

In our implementation we will have just a writable running config. The size of this will be so 
big, that that alone precludes keeping a candidate. The running datastore will work somewhat 
like a normal SQL database. It locks what is needed without explicit lock operations.

See also some comments below:

> write access, using implicit locking.
[Balazs]: One solution is to say locking is a completely internal, implementation specific 
issue, so we can avoid redefining most parts of the protocol. If part or the whole of an 
operation (edit-config, copy-config, etc.) fails due to internal locking, that is handled the 
same way as any other internal error. Even today we can not guarantee that all set/create 
sub-operations will succeed. Still it would be good to tell the client that the operation 
failed (temporarily?) due to locking, so he will not try to find bugs in the operation's data.
> a locked sub-tree and part of it touches an unlocked sub-tree?
[Balazs}: IMHO failure due to locking can be handled as just any other internal failure. E.g. 
one due to running out of memory. In my understanding the edit-config error-option will be able 
to handle this. Do you agree?
> How do you think the protocol should support multi-session locks?
[Balazs]: Multi-session locks are useful, but they solve only part of the problem. We already 
have big nodes where different people configure different parts of the system, so we would like 
to be able to lock just part of the node. If the involved people are geographically separated 
multi-session locks don't solve this problem.

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Tue, 10 Apr 2007 07:36:11 -0700

Balazs Lengyel wrote:
> Hello Andy,
> For us the question is even more interesting, as parts of our big node 
> configuration might be administered by different people/organizations 
> using different management tools; so allowing operator A to deny 
> configuration operations for operator B sounds really bad.
> 

Related config data can be spread around the entire XML tree.
Limiting access to a single subtree will be too restrictive.
Locking access by namespace or 'namespace owner' may also
be too restrictive, given the various ways namespaces are used.

You end up with an arbitrary Xpath expression for the lock,
which can include all kinds of bizarre corner-cases where
parts of the config transition in and out of locked state,
as different clauses in the Xpath expression change value over time.
(Not a fan of this 12 pound hammer for every job!)

> - Who can used a locked entity: one session, any session connected to 

Clearly the unit of measure is the session.
The group of writers is an integral number of NETCONF sessions.

> 
> IMO the first step in any lock related work is to allow the node to 
> reject the global lock, else some old manager application will come and 
> lock the whole node.

Yes, but this falls out for free if the reason the agent is denying
the lock is because some form of lock is already held.  The logic
for explicit locking is simple, and fits in with the current scheme:

   - If no-locks-held then grant lock request
   - If global lock held then deny lock request
   - If partial lock held then deny global lock request, but
     analyze partial lock request to determine if it can be granted
   - Dropped session only releases the lock held by that session,
     so if it is partial then other partial locks by other sessions
     are not affected.  (This makes lock discovery an issue.)

> 
> The next step (as you point out below) is that partial locking has a 
> difficulty with candidate configurations. At least commit, 
> discard-changes, delete-config and copy-config needs a full lock (or 
> rather an absence of any lock on any part of the datastore). 
> Theoretically we could have a partitioned candidate data-store with 
> partial commit etc. but I think that's too complicated to be usable.

I think partial locking on a global candidate is very problematic.
I don't want to rewrite any text in RFC 4741 related to commit,
discard-changes, and confirmed-commit.  We would need to introduce
a candidate-per-session architecture, which is very TBD.

> - A complete SQL-database-like implicit locking mechanism works well, 
> but might be a heavy task for small nodes
> - I also see an important use-case in dividing up the configuration into 
> disjunct parts and locking those (e.g. a sub-tree). This way each 
> operator could configure his own part. It is not feasible to define the 
> sub-trees at design time.
> - Multi-session locks can also be useful, if it is the same person/group 
> using the same lock simultaneously. We could utilize something like the 
> groupId or userId for this purpose.
> 
> In our implementation we will have just a writable running config. The 
> size of this will be so big, that that alone precludes keeping a 
> candidate. The running datastore will work somewhat like a normal SQL 
> database. It locks what is needed without explicit lock operations.

Even with implicit locking, it is important that the manager
give explicit indications to the agent that it is:
  - starting a transaction
  - canceling a transaction
  - committing a transaction

We already have the candidate model work this way for a global config,
and changing the running config model to also work that way seems
complicated and redundant.

> [Balazs]: One solution is to say locking is a completely internal, 
> implementation specific issue, so we can avoid redefining most parts of 
> the protocol. If part or the whole of an operation (edit-config, 
> copy-config, etc.) fails due to internal locking, that is handled the 
> same way as any other internal error. Even today we can not guarantee 
> that all set/create sub-operations will succeed. Still it would be good 
> to tell the client that the operation failed (temporarily?) due to 
> locking, so he will not try to find bugs in the operation's data.
>>
> [Balazs}: IMHO failure due to locking can be handled as just any other 
> internal failure. E.g. one due to running out of memory. In my 
> understanding the edit-config error-option will be able to handle this. 
> Do you agree?
> [Balazs]: Multi-session locks are useful, but they solve only part of 
> the problem. We already have big nodes where different people configure 
> different parts of the system, so we would like to be able to lock just 
> part of the node. If the involved people are geographically separated 

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 10 Apr 2007 17:06:16 +0200

Hello Andy,

> be too restrictive, given the various ways namespaces are used.
[Balazs]: It can be spread out, but often it is not. I believe a subtree based locking could be 
useful for many users. E.g. our node includes both a gateway and a router. In this case there 
is a natural division of config data into sub-trees. Also while working with British Telecom I 
saw that they had separate (human) operators for performance management. In many nodes 
administering PM data was in a separate sub-tree from the rest of configuration.
> (Not a fan of this 12 pound hammer for every job!)
[Balazs]: I agree that the full XPATH is a heavy hammer, but we could leave it to the 
implementors to decide if they want just sub-trees, or the full heavy hammer or nothing.
> The group of writers is an integral number of NETCONF sessions.
[Balazs]: Agree.
>> reject the global lock, else some old manager application will come 
>> and lock the whole node.
> 
[Balazs]: I might want to disallow global locking, because I always want the partial lock to be 
always available, but maybe this can be left to the specific access control model.
> a candidate-per-session architecture, which is very TBD.
[Balazs]: Agree, it would be too complicated.
>> - I also see an important use-case in dividing up the configuration 
>> into disjunct parts and locking those (e.g. a sub-tree). This way each 
>> operator could configure his own part. It is not feasible to define 
>> the sub-trees at design time.
>> - Multi-session locks can also be useful, if it is the same 
>> person/group using the same lock simultaneously. We could utilize 
>> something like the groupId or userId for this purpose.
>>
[Balazs]: You don't need explicit transactions for this. You just lock the config and handle 
each edit-config as a separate transaction.
IMO we already have 2 transaction models in Netconf. One with the candidate configuration the 
other without. Without all you have is: one operation is one transaction with roll-back-on-error.
Also there are nodes out there that already work with explicit start-cancel-commit transaction 
operations (not yet on Netconf). I know of a company (not mine) who is already implementing 
explicit start-cancel-commit transaction Netconf operations.

---

From: Andy Bierman <[email-masked]>
Date: Tue, 10 Apr 2007 10:37:47 -0700

Balazs Lengyel wrote:
> Hello Andy,
> 
> Andy Bierman wrote:
>> Balazs Lengyel wrote:
>>> Hello Andy,
>>> For us the question is even more interesting, as parts of our big 
>>> node configuration might be administered by different 
>>> people/organizations using different management tools; so allowing 
> [Balazs]: It can be spread out, but often it is not. I believe a subtree 
> based locking could be useful for many users. E.g. our node includes 
> both a gateway and a router. In this case there is a natural division of 
> config data into sub-trees. Also while working with British Telecom I 
> saw that they had separate (human) operators for performance management. 
> In many nodes administering PM data was in a separate sub-tree from the 
> rest of configuration.
>>
> [Balazs]: I agree that the full XPATH is a heavy hammer, but we could 
> leave it to the implementors to decide if they want just sub-trees, or 
> the full heavy hammer or nothing.

What if the Xpath expression include sub-expressions that have nothing
to do with identifying the conceptual objects and instances being locked?
If it was limited to an absolute path expression, that simply identified
a single XML element within the tree, or an Instance Identifier
that just included the XML elements that made up the key,
then I would like it much better.

Using a 12 lb. hammer drives the little finishing nail real quick,
but it leaves a big dent in the furniture if slightly misused.
IMO, arbitray Xpath locks will create many unintended consequences
than we should avoid rather than cope with forever.

> [Balazs]: I might want to disallow global locking, because I always want 
> the partial lock to be always available, but maybe this can be left to 
> the specific access control model.
>>
>>

What session-id do you put in the <rpc-error> when you reject the lock
request for no reason?  IMO, the partial lock has to work in a way
that is compatible with the global lock, in every way, even the
required fields in the <rpc-error> info.

>>> into disjunct parts and locking those (e.g. a sub-tree). This way 
>>> each operator could configure his own part. It is not feasible to 
>>> define the sub-trees at design time.
>>> In our implementation we will have just a writable running config. 
>>> The size of this will be so big, that that alone precludes keeping a 
> [Balazs]: You don't need explicit transactions for this. You just lock 
> the config and handle each edit-config as a separate transaction.

The whole point of locking is to have an edit transaction
that includes multiple edit-configs, maybe some <get> operations, etc.

> IMO we already have 2 transaction models in Netconf. One with the 
> candidate configuration the other without. Without all you have is: one 
> operation is one transaction with roll-back-on-error.
> Also there are nodes out there that already work with explicit 
> start-cancel-commit transaction operations (not yet on Netconf). I know 
> of a company (not mine) who is already implementing explicit 
> start-cancel-commit transaction Netconf operations.
>>
>>
>>>

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 11 Apr 2007 09:23:43 +0200

Andy Bierman wrote:
>> [Balazs]: It can be spread out, but often it is not. I believe a 
>> subtree based locking could be useful for many users. E.g. our node 
>> includes both a gateway and a router. In this case there is a natural 
>> division of config data into sub-trees. Also while working with 
>> British Telecom I saw that they had separate (human) operators for 
>> performance management. In many nodes administering PM data was in a 
>> separate sub-tree from the rest of configuration.
>>>
[Balazs]: I agree that the filter can become dangerously complicated. So selecting just one 
element and the underlying sub-tree is a good choice. I would also allow selecting multiple 
disjoint sub-trees as I believe thats not so complicated, but nothing more complicated.
In Hungary we also have 40 lb hammers at least better locksmiths :-)
>> [Balazs]: I might want to disallow global locking, because I always 
>> want the partial lock to be always available, but maybe this can be 
>> left to the specific access control model.
>>>
>>>
> 
> 
[Balazs]: I will say access-denied and provide no session info. This also indicates that the 
denial is not temporary.
>>>> - A complete SQL-database-like implicit locking mechanism works 
>>>> well, but might be a heavy task for small nodes
>>>> candidate. The running datastore will work somewhat like a normal 
>>>> SQL database. It locks what is needed without explicit lock operations.
>>>
>>>
[Balazs]: Agree, but having a one-operation transaction is better then nothing, and for some 
nodes using candidate-configuration is simply not possible; but we still love Netconf :-). The 
explicit transaction operations will also help.
>> candidate configuration the other without. Without all you have is: 
>> one operation is one transaction with roll-back-on-error.
>> start-cancel-commit transaction operations (not yet on Netconf). I 
>> know of a company (not mine) who is already implementing explicit

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 20 Apr 2007 11:24:15 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> that we discussed at the last interim.

So am I.

> a locked sub-tree and part of it touches an unlocked sub-tree?

No.  

> so rejecting an edit-config operation this way seems wrong.

I agree.

> but unlocked sub-tree?

Nothing special.

> At least explicit locking works the same

Yes, IMO explicit partial locking with no special semantics (like
rolling back uncommitted changes) is sufficient.

> discard-changes, and confirmed-commit.  We would need to introduce a
> candidate-per-session architecture, which is very TBD.

Could you elaborate on this a bit?  Why is partial locking on the
global candidate problematic?

IMO, the semantics of edit-configs should be the same with a partial
lock as without a lock when noone else is modifying the datastore.

I also like the idea of using the xpath-based instance identifier
syntax to lock subtrees.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Apr 2007 04:48:43 -0700

> global candidate problematic?

The transaction sequence (lock+edit+commit+unlock) does not
work independently like it would if the candidate config was
per-session, instead of global.   Once the first writer
does a commit, the entire config is committed, not just the
part that this writer locked (or wrote).

Also, the way candidate is written, once the candidate is written
with an edit, nobody else can take out the lock, even if that
first writer didn't take out a lock at all.

There is also a access control deadlock concern with candidate,
that gets a little worse with partial locks.  Let's say session A can
modify one sub-tree and session B can modify a different subtree,
but not the other way around.  If both sessions write to the
candidate config, then neither one of them will be able
to commit it, because it contains edits in an area each
one is denied write access.

If this data is also locked, then a third session with write access
to everything cannot undo part of the edits and commit the rest.
The operations are applied rather unevenly, since you can configure
access control to allow a 'root' user to commit another user's edits,
but there is no way for root to undo another user's lock.
There is only 'kill-session', which causes the entire candidate
config to to a discard-changes when the session is killed.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 20 Apr 2007 14:00:24 +0200 (CEST)

> but there is no way for root to undo another user's lock.

The same situation occurs if no partial locks are involved - user A
modifies parts of the shared candidate, and B some other parts.
Neither A or B can commit.  So 'root' comes in and either commits or
reverts.

> config to to a discard-changes when the session is killed.

...provided that the session which is killed has the *global* lock.
My point is that IMO we don't have to use this semantics for the
partial lock mechanism.

> > lock as without a lock when noone else is modifying the datastore.

and

/martin