Subject: [NGO] Re: multi-session locks

From: Andy Bierman <[email-masked]>
Date: Thu, 03 May 2007 14:05:45 -0700

Hi Balazs,

I like your proposal, except for a few details.

---------------------------
To summarize, a partial lock is a lock on one particular subtree
in the conceptual data model. (Or is is an arbitrary Xpath expression?)

For example:

   lock (/interfaces) locks the interfaces node and all interface entries

   lock (/interfaces/interface) locks all interface nodes

   lock (/interfaces/interface[ifIndex=3]) locks the entire ifEntry.3

   lock (/interfaces/interface[ifIndex=3]/ifMtu) locks just the ifMtu.3 leaf

Each session can have zero or more partial locks on the same target config.

A partial unlock only affects the specified sub-tree and no other locks.

A session termination only affects the partial locks held by that session.

If any partial locks are active, then a request for a full lock will fail.

If a full lock is active then any request for a partial lock on the same
target will fail.

If no lock is active on the target, then a request for a full lock or
a partial lock will succeed.

If any partial locks are active on the target, then a request for a partial
lock will only succeed if the Xpath expression does not identify any nodes
with are already partially locked.
-----------------------------------------------

I believe the only difference between this and your proposal is
the requirement to support full locks at all.  The protocol is
quite clear about this conformance requirement, and compliant managers
that do not know about the partial locks will break if the <lock>
command always fails no matter what.

Andy

> Hello Andy,
> 
> Andy Bierman wrote:
>> Balazs Lengyel wrote:
>>> Hello Andy,
>>> For us the question is even more interesting, as parts of our big 
>>> node configuration might be administered by different 
>>> people/organizations using different management tools; so allowing 
>>> operator A to deny configuration operations for operator B sounds 
>>> really bad.
>>>
>>> IMO we have here two separate questions:
>>> - What can be locked: full data-store, a sub-tree, any group of 
>>> managed objects?
>>
>> Related config data can be spread around the entire XML tree.
>> Limiting access to a single subtree will be too restrictive.
>> Locking access by namespace or 'namespace owner' may also
>> be too restrictive, given the various ways namespaces are used.
> [Balazs]: It can be spread out, but often it is not. I believe a subtree 
> based locking could be useful for many users. E.g. our node includes 
> both a gateway and a router. In this case there is a natural division of 
> config data into sub-trees. Also while working with British Telecom I 
> saw that they had separate (human) operators for performance management. 
> In many nodes administering PM data was in a separate sub-tree from the 
> rest of configuration.
>>
>> You end up with an arbitrary Xpath expression for the lock,
>> which can include all kinds of bizarre corner-cases where
>> parts of the config transition in and out of locked state,
>> as different clauses in the Xpath expression change value over time.
>> (Not a fan of this 12 pound hammer for every job!)
> [Balazs]: I agree that the full XPATH is a heavy hammer, but we could 
> leave it to the implementors to decide if they want just sub-trees, or 
> the full heavy hammer or nothing.
>>
>>> - Who can used a locked entity: one session, any session connected to 
>>> one userId or groupId?
>>
>> Clearly the unit of measure is the session.
>> The group of writers is an integral number of NETCONF sessions.
> [Balazs]: Agree.
>>
>>>
>>> IMO the first step in any lock related work is to allow the node to 
>>> reject the global lock, else some old manager application will come 
>>> and lock the whole node.
>>
>> Yes, but this falls out for free if the reason the agent is denying
>> the lock is because some form of lock is already held.  The logic
>> for explicit locking is simple, and fits in with the current scheme:
>>
>>   - If no-locks-held then grant lock request
>>   - If global lock held then deny lock request
>>   - If partial lock held then deny global lock request, but
>>     analyze partial lock request to determine if it can be granted
>>   - Dropped session only releases the lock held by that session,
>>     so if it is partial then other partial locks by other sessions
>>     are not affected.  (This makes lock discovery an issue.)
> [Balazs]: I might want to disallow global locking, because I always want 
> the partial lock to be always available, but maybe this can be left to 
> the specific access control model.
>>
>>
>>>
>>> The next step (as you point out below) is that partial locking has a 
>>> difficulty with candidate configurations. At least commit, 
>>> discard-changes, delete-config and copy-config needs a full lock (or 
>>> rather an absence of any lock on any part of the datastore). 
>>> Theoretically we could have a partitioned candidate data-store with 
>>> partial commit etc. but I think that's too complicated to be usable.
>>
>>
>> I think partial locking on a global candidate is very problematic.
>> I don't want to rewrite any text in RFC 4741 related to commit,
>> discard-changes, and confirmed-commit.  We would need to introduce
>> a candidate-per-session architecture, which is very TBD.
> [Balazs]: Agree, it would be too complicated.
>>
>>>
>>> All in all I see the following possibilities:
>>> - A complete SQL-database-like implicit locking mechanism works well, 
>>> but might be a heavy task for small nodes
>>> - I also see an important use-case in dividing up the configuration 
>>> into disjunct parts and locking those (e.g. a sub-tree). This way 
>>> each operator could configure his own part. It is not feasible to 
>>> define the sub-trees at design time.
>>> - Multi-session locks can also be useful, if it is the same 
>>> person/group using the same lock simultaneously. We could utilize 
>>> something like the groupId or userId for this purpose.
>>>
>>> In our implementation we will have just a writable running config. 
>>> The size of this will be so big, that that alone precludes keeping a 
>>> candidate. The running datastore will work somewhat like a normal SQL 
>>> database. It locks what is needed without explicit lock operations.
>>
>>
>> Even with implicit locking, it is important that the manager
>> give explicit indications to the agent that it is:
>>  - starting a transaction
>>  - canceling a transaction
>>  - committing a transaction
>>
>> We already have the candidate model work this way for a global config,
>> and changing the running config model to also work that way seems
>> complicated and redundant.
> [Balazs]: You don't need explicit transactions for this. You just lock 
> the config and handle each edit-config as a separate transaction.
> IMO we already have 2 transaction models in Netconf. One with the 
> candidate configuration the other without. Without all you have is: one 
> operation is one transaction with roll-back-on-error.
> Also there are nodes out there that already work with explicit 
> start-cancel-commit transaction operations (not yet on Netconf). I know 
> of a company (not mine) who is already implementing explicit 
> start-cancel-commit transaction Netconf operations.
>>
>>
>>>
>>> See also some comments below:
>>>
>>> Andy Bierman wrote:
>>>> Hi Balazs,
>>>>
>>>> I am very interested in the partial database locking problem
>>>> that we discussed at the last interim.  I think that you are
>>>> right wrt/ your concerns about the "1 user global lock" design
>>>> for large devices.
>>>>
> 
> 

_______________________________________________
NGO mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ngo

---

From: Andy Bierman <[email-masked]>
Date: Thu, 03 May 2007 14:14:19 -0700

> I like your proposal, except for a few details.

Forgot one important 'rule' (to make Wes happy ;-)

If a session does not have write access to any node identified in
the Xpath expression for a partial lock request, the lock will
not be granted.

This doesn't fix the DoS attack for a full lock, just a partial lock.

Andy

>> [Balazs]: It can be spread out, but often it is not. I believe a 
>> subtree based locking could be useful for many users. E.g. our node 
>> includes both a gateway and a router. In this case there is a natural 
>> division of config data into sub-trees. Also while working with 
>> British Telecom I saw that they had separate (human) operators for 
>> performance management. In many nodes administering PM data was in a 
>> separate sub-tree from the rest of configuration.
>>>
>>>> - Who can used a locked entity: one session, any session connected 
>>>> to one userId or groupId?
>>>
>> [Balazs]: I might want to disallow global locking, because I always 
>> want the partial lock to be always available, but maybe this can be 
>> left to the specific access control model.
>>>
>>>
>>>>
>>>> - A complete SQL-database-like implicit locking mechanism works 
>>>> well, but might be a heavy task for small nodes
>>>> candidate. The running datastore will work somewhat like a normal 
>>>> SQL database. It locks what is needed without explicit lock operations.
>>>
>>>
>> candidate configuration the other without. Without all you have is: 
>> one operation is one transaction with roll-back-on-error.
>> start-cancel-commit transaction operations (not yet on Netconf). I 
>> know of a company (not mine) who is already implementing explicit

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 04 May 2007 14:54:44 +0200

Hello Andy,
See answers below!
Would you support the draft in the IETF?
regards Balazs

> the Xpath expression for a partial lock request, the lock will
{BALAZS]: What if we want to a lock a subtree, then read some data and release the lock. The 
aim is to guarantee we read a consistent set of data. Do we still need write access rights? 
Maybe this issue is just an unneeded complication, but I would like your opinion about it.
> This doesn't fix the DoS attack for a full lock, just a partial lock.
{BALAZS]:Locking, candidate configuration and access control together can become complicated 
enough; so I  propose locking one or more subtrees. If a node is locked, the complete subtree 
below it should also be locked. I feel that's simple enough and powerful enough.
>>   lock (/interfaces/interface[ifIndex=3]/ifMtu) locks just the ifMtu.3 
{BALAZS]:Yes that's how I see it as well.
>> A partial unlock only affects the specified sub-tree and no other locks.
{BALAZS]:Speaking with Martin he convinced me, that it would be better to use a lockId instead 
of a filter based unlocking. The lock operation would return the lockId (unique at least for 
the session, but rather for the node). In the unlock operation the the manager would specify 
this lockId, but no filter. This simpler, less error prone and would avoid some difficult cases:
- what if the lock and unlock filters don't match
- what happens if I lock /interfaces and later I unlock /interfaces/interface[ifIndex=3]
- what if I lock /interfaces/interface[ifIndex=3] and later I unlock /interfaces
>> with are already partially locked.
{BALAZS]: Strictly speaking my draft does not speak about the possibility to reject global 
locking, although you are right about my hidden ideas :-) But as for now it is not in the draft 
I believe.

{BALAZS]:Some other rules:
- If a locked subtree is deleted, the lock is still valid, meaning that another session can not 
re-create the same subtree while the lock is still held. (You could argue the other way as 
well, but at least it must be stated.)
- I would also like to follow Martin with avoiding any special interactions between the partial 
lock and the candidate configuration. Anything like we have in rfc4741:
"When a client fails with outstanding changes to the candidate
    configuration, recovery can be difficult.  To facilitate easy
    recovery, any outstanding changes are discarded when the lock is
    released, whether explicitly with the <unlock> operation or
    implicitly from session failure."
- others ?
>>> leave it to the implementors to decide if they want just sub-trees, 
>>> or the full heavy hammer or nothing.
>>>>
>>>>> The next step (as you point out below) is that partial locking has 
>>>>> a difficulty with candidate configurations. At least commit, 
>>>>> discard-changes, delete-config and copy-config needs a full lock 
>>>>> (or rather an absence of any lock on any part of the datastore). 
>>>>> The size of this will be so big, that that alone precludes keeping 
>>>>> a candidate. The running datastore will work somewhat like a normal 
>>> [Balazs]: You don't need explicit transactions for this. You just 
>>> lock the config and handle each edit-config as a separate transaction.

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 May 2007 07:23:33 -0700

> Would you support the draft in the IETF?

The one you haven't published yet? ;-)
I'm interested in implementing it.

> {BALAZS]: What if we want to a lock a subtree, then read some data and 
> release the lock. The aim is to guarantee we read a consistent set of 
> data. Do we still need write access rights? Maybe this issue is just an 
> unneeded complication, but I would like your opinion about it.
>>

The lock is for all operations.
If you want to take a lock out and just read, nothing to stop that.

> {BALAZS]:Locking, candidate configuration and access control together 
> can become complicated enough; so I  propose locking one or more 
> subtrees. If a node is locked, the complete subtree below it should also 
> be locked. I feel that's simple enough and powerful enough.

Okay -- so it is just an Xpath expression, not constrained in any way.
It identifies the nodes to be locked.
When issuing an unlock, the Xpath expression must exactly match
the one used in the lock operation.

This feature requires that the optional Xpath capability be supported
by the agent BTW, unless your draft supports a subtree filter partial lock.

> {BALAZS]:Speaking with Martin he convinced me, that it would be better 
> to use a lockId instead of a filter based unlocking. The lock operation 
> would return the lockId (unique at least for the session, but rather for 
> the node). In the unlock operation the the manager would specify this 
> lockId, but no filter. This simpler, less error prone and would avoid 
> some difficult cases:

they have to exactly match to be considered the same lock

> - what if I lock /interfaces/interface[ifIndex=3] and later I unlock 

Don't do that. :-)
The agent will keep track of the Xpath expressions and match them.
The lock ID adds complexity and make the agent authoritative wrt/ naming,
when in fact the manager should be in charge of this.

The added feature of incrementally increasing or decreasing the
partial lock scope is too complicated to support.  You have to
pick the right scope and then release that lock, not modify it.

> {BALAZS]: Strictly speaking my draft does not speak about the 
> possibility to reject global locking, although you are right about my 
> hidden ideas :-) But as for now it is not in the draft I believe.

I think this is a critical issue.
IMO you cannot silently break compliant managers and still
be a compliant agent.

A NETCONF 1.0 manager is extremely likely to use the global locking.
I don't see a good way around this problem.

> - If a locked subtree is deleted, the lock is still valid, meaning that 
> another session can not re-create the same subtree while the lock is 
> still held. (You could argue the other way as well, but at least it must 
> be stated.)

Good point -- but makes the implementation a challenge.
The agent will have to grant locks for nodes that are
not even supported by the agent, since it may not be able
to distinguish between these and nodes it does support,
and real nodes for which there is no sub-agent installed at the moment.

> - I would also like to follow Martin with avoiding any special 
> interactions between the partial lock and the candidate configuration. 
> Anything like we have in rfc4741:

This is also a problem with NETCONF 1.0 compatibility.
What about the <discard-changes> and <confirmed-commit> operations?
These are designed for a global candidate with a global lock.

I think adding partial locking would require a significant re-write
of the protocol, and probably require obsoleting RFC 4741 instead
of extending it.  That would be unacceptable IMO.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 04 May 2007 19:52:13 +0200 (CEST)

> If you want to take a lock out and just read, nothing to stop that.

... except for the rule you suggested above - if a node does not have
*write* access the lock request is rejceted.  We must change that to
just 'access'.

> the one used in the lock operation.

So essentially the *exact* string is used as an identifier, including
any prefixes, which are local in each instance document?

I think it's cleaner and less error prone to return an identifier
which is used in unlock.

> when in fact the manager should be in charge of this.

Same as session-id.  I disagree that this adds complexity; I think
it's the other way around.  I don't understand your comment about
"authoritative wrt/ naming".   Could you elaborate?  What's the
problem?

> pick the right scope and then release that lock, not modify it.

I agree.

> I don't see a good way around this problem.

What exactly is the problem?

> These are designed for a global candidate with a global lock.

Yes.  So the partial locking would not affect this at all.  Ok, I
re'-read Balaxs draft, and he actually modifies the existing
lock/unlock operations.  I think it's better to define new
operations.  Then the old operations will continue to work as
specified in rfc4741.

> of extending it.  That would be unacceptable IMO.

I agree.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 May 2007 12:12:15 -0700

> I think it's cleaner and less error prone to return an identifier

ok

> "authoritative wrt/ naming".   Could you elaborate?  What's the

the manager knows the string it passed in for the lock.
Now it has the string and an ID to keep track of.

> What exactly is the problem?

If there are no locks held at all by anybody,
and a session requests a global lock, it should be granted.
Refusing to grant the global lock ever, under any circumstances,
does not seem to be compliant to RFC 4741 to me.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 04 May 2007 21:21:47 +0200 (CEST)

> does not seem to be compliant to RFC 4741 to me.

This is what the rfc says, in section 7.5

      An attempt to lock the configuration MUST fail if an existing
      session or other entity holds a lock on any portion of the lock
      target.

So I think it covers the partial lock.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 May 2007 15:32:22 -0700

> So I think it covers the partial lock.

I don't -- if an agent never grants the full lock.
I am concerned about the plan of simply never granting the
full lock and claiming that RFC 4741 is intended to support this,
just because the opposite form of this sentence is not also present:

(An attempt to lock the configuration MUST succeed if no existing
  session or other entity holds a lock on any portion of the lock
  target.)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sat, 05 May 2007 00:43:46 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> full lock

Ok, I missed that plan, sorry about that.  I never read that in

> hidden ideas :-) But as for now it is not in the draft I believe.

...in which Balazs seems to suggest that global locking should always
be rejected.

I agree with you on this.  I don't think the presence of partial
locking support should turn off global locking support.

/martin

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 07 May 2007 10:13:14 +0200

Hello Andy,

>  session or other entity holds a lock on any portion of the lock
[BALAZS]: You can not have the "MUST sentence" as access control must be able to deny full 
locking of the configuration even if no one is holding any kind of lock at the moment.

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 07 May 2007 10:15:07 +0200

Hello,
Actually there is nothing in my draft about denying the full lock, or please point out the 
relevant text and I will remove it.
Balazs

---

From: Andy Bierman <[email-masked]>
Date: Mon, 07 May 2007 08:24:26 -0700

Balazs Lengyel wrote:
> Hello,
> Actually there is nothing in my draft about denying the full lock, or 
> please point out the relevant text and I will remove it.

Can you send a URL to your draft?
I can only find draft-lengyel-netconf-granular-locking-00.txt from 2005.

Andy