Subject: Byte Range Locking, What's OK?

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Sep 2000 13:54:24 -0700

I'm doing (server-side) byte range locking support and I'm not
sure about what types of lock and unlock requests are valid.

I originally adopted one interpretation and after thinking about
it, now have a different interpretation.  At most one of my
interpretations is right, so I'm wondering if there is a
difference of opinion out there.

Some background.  Until fairly late in the development in the 
spec, it was assumed (at least by me) that you might unlock, 
upgrade, or downgrade only a particular range of bytes that you 
previously locked and that each range was indivisible (and 
uncombinable with other ranges even if adjacent).  Then it was 
realized that the unlocking of a sub-range could not be simulated 
by the client as a combination of unlock and lock requests because 
of atomicity considerations.

The result was that the spec was changed to allow clients who
needed to make such requests to make them with the server allowed
to reject them with an error (because NT, for example, could not
support them).

The question is how far should V4 servers should go to accommodate
the peculiarities of flock; is "sub-range semantics" to be interpreted
strictly as only covering actual sub-ranges or is more encompassed
here.

At first I just assumed that servers could just implement flock's
locking semantics (although they didn't have to).  This includes 
lots of strange things besides modifying the locking of a sub-range.

Now I've changed my mind.  I'm inclinded to interpret what the
spec says more strictly and I'm wondering how others are dealing
with this issue.

The reason's I've changed my view of this are:

     The spec does say "sub-range semantics".

     The other wierdnesses of flock can (I think) be simulated by
     the client without the same atomicity issues that occur
     with subranges (ar at least the atomicity issues don't
     have such significant consequences).

     The protocol's treatment of unlock, in that it requires
     the lock type, doesn't fit very well with the flock
     handling of unlock allowing to unlock a range that 
     includes a number a separate locks (since those locks
     might have different types).

What's other people's view of this?  In particular, are those
doing unix-ish clients trying to simulate the peculiarities 
of flock locally (except for sub-ranges strictly defined)
or are they sending the flock requests over essentialy
unmodified as I believe they would have done with nlm. 

The spec says:

   The protocol allows a lock owner to request a lock with one byte
   range and then either upgrade or unlock a sub-range of the initial
   lock.  It is expected that this will be an uncommon type of request.
   In any case, servers or server file systems may not be able to
   support sub-range lock semantics.  In the event that a server
   receives a locking request that represents a sub-range of current
   locking state for the lock owner, the server is allowed to return the
   error NFS4ERR_LOCK_RANGE to signify that it does not support sub-
   range lock operations.  Therefore, the client should be prepared to
   receive this error and, if appropriate, report the error to the
   requesting application.

This gives rise to a number of issues.

First unlock and upgrade are mentioned but downgrade is not.  Since
downgrade of a subrange gives rise to the same atomicity issues
as unlock of a sub-range (if the client were to try to simulate it)
I'm assuming this is an oversight and that downgrade of a sub-range
(i.e. by getting a read lock of an area previously locked for write)
is OK, but the server may return a LOCK_RANGE error.   Comments?

In connection with the above, I had been assuming that ordinary 
upgrade and downgrade of existing locks (by doing a new lonk
request for the same range with a different type) but I can't 
find anywhere were this is said explicitly.  Are these operations
part of what server must support.  Can NT servers support them?
If not, what do they do?  LOCK_RANGE doesn't seem to be the right
error, but if not that, what?

Some examples of operations that I (now) don't think the spec
allows, but would be required by simple implementations of flock 
that simply send the corresponding requests to the server (rather
than keeping track of the clients locks locally and transforming
flock actions into corresponding protocol requests).  These
are not really within the scope of "sub-range semantics".  Should
these be errors and if so what error?

1) Locking an area that subsumes totally a number of previously
   locked areas, locking any areas not previously locked and
   possibly upgrading or downgrading the locked area (if they 
   are of a different locking type locking type).

2) Locking an area we already have locked with the same locking 
   type and not getting an error (a no-op).

3) Unlocking an area currently locked when the lock type specified
   in the unlock doesn't match.

3a) What about when it matches except for the "wait" portion of 
    type (i.e. WRITE_LT vs. WRITEW_LT).

4) Unlocking an area that includes a locked area plus additional
   non-locked bytes.

5) Unlocking an area that subsumes totally a number of previously
   locked areas.

6) Unlocking an area that you don't currently have locked at all.

What I'm currently planning to do for the bake-off is to return
INVAL for 1) through 6), but not 3a).  Will there be (and also
should there be) clients that expect some different behavior?

---

From: Jim Rees <[email-masked]>
Date: Thu, 21 Sep 2000 02:27:38 -0400

My own opinion is that byte-range locks add unnecessary cruft to the
protocol and should be removed.  Any application that needs them is likely
to need other database semantics too, and would be better off just using a
real database instead of trying to overload a file system.

Yes, I realize this is an unpopular view and that it's too late.

---

From: Mike Eisler <[email-masked]>
Date: Thu, 21 Sep 2000 07:14:10 -0700 (PDT)

> Yes, I realize this is an unpopular view and that it's too late.

It's not unpopular at all, has been discussed before, and the conclusion was
that telling application developers that what they are doing is stupid
is a losing battle.

	-mre

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Fri, 22 Sep 2000 10:51:37 -0400

I'm implementing byte-range locking, and have decided to punt on
the subrange issue (at least for now) and use this as my server's policy:

   range lock operations. 

it's clear that in order to implement subrange locking, the possible subrange 
cases that dave has enumerated need to be addressed in the spec. is INVAL what 
applications expect as an error from subrange locking, or are there more 
informative error returns? LOCK_RANGE?

>is OK, but the server may return a LOCK_RANGE error.   Comments?

that's how i read the spec - it's legal for the server to return LOCK_RANGE error on any subrange lock request.

-->Andy

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 22 Sep 2000 08:44:12 -0700

> the subrange issue (at least for now)

I see you're reserving the right to suddenly change your mind about 
punting and run on fourth and long.  I wouldn't be surprised to see
your server supporting subrange locking at the bake-off.

> and use this as my server's policy:
>

>From this paragraph, it appears that you believe the the
cases I enumerated are "subrange cases" while I would describe
them as "wierd cases which aren't subrange cases".  If they are
indeed "subrange cases", then the according to the spec,

     non-subrange-supporting servers should return LOCK_RANGE

     sub-range supporting servers should just implement the request

I don't have any problem with this, but if this is the consensus,
the spec should make this clear.

The other interpretation, which is that my wierd cases are not
"subrange cases", which has at least as much support within the
spec, implies,

     all servers should return INVAL for these cases

     clients need to keep track of the locks that they hold and
     use that information to determine how to simulate the wierd
     (but non-subrange) cases.

>that's how i read the spec - it's legal for the server to return 
>LOCK_RANGE error on any subrange lock request.

Right but my main concern is with the other half of it: that servers
that do support sub-range semantics should supprot downgrade of a 
subrange as well as upgrade and unlock.

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Fri, 22 Sep 2000 12:39:10 -0400

>your server supporting subrange locking at the bake-off.

perhaps..... :0

>    (but non-subrange) cases.

OK. here's what i would do.

i'd return LOCK_RANGE in these cases, in each case the range is 'wrong',

>   are of a different locking type locking type)

i'd return INVAL for these cases where there is no more informative error.

>6) Unlocking an area that you don't currently have locked at all.

in a twisted sort of way, this is a subrange (of 100%!), but i guess if an 
error is to be returned INVAL is ok.

>>is OK

yes, i agree.

-->Andy

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 22 Sep 2000 10:16:20 -0700

> There needs to be an agreement on the interaction between
> the lock sequence number, lease, and stateid error checking.
> 
> One issue is when does the client upon receiving an error
> on a lock sequence number mutating operation decrement
> it's lockowner sequence number?
> 
> I've divided the operations into three catagories and shown
> the server's processing logic with questions for each, and 
> would like to hear comments....
> 
> In what order is everyone else doing this processing on the 
> server?
> 
> 
> Operations that have a stateid and a sequence number 
> ---------------------------------------------------- 
> CLOSE, LOCK, LOCKU, OPEN_DOWNGRADE, 
> OPEN_CONFIRM (stateid in verifier otherwise server 
> can't locate lockowner)
> 
> The stateid has to be checked first - it is the only way
> to find the lockowner.  If the stateid is bad, the sequence
> number cannot be bumped because the lockowner can't be 
> identified. So, the client when getting a NFS4ERR_XXX_STATEID
> error needs to decrement the lockowners sequence number otherwise
> it will be inconsistant for other lockowner operations.

Right.  My server does not bump the sequence number or check
for a replay, since it can't find the lockowner structure.

In the case of NFS4ERR_STALE_STATEID, the client should decide that 
the server has rebooted so it should reset the sequence numbers,
shouldn't it.

The cases of NFS4ERR_{BAD,OLD}_STATEID should not occur if the
client and server are functioning properly but I agree the client
should undo any sequnce number increment.
 
> Should the lease be renewed on a good stateid and bad sequence number?

I don't.  I do sequence number checking as early as I can
(but I may have to check the stateid first) and try to make
sure that requests that have bad sequence numbers don't 
cause anything to happen.

> If so, what error should be returned in the case that the
> lease has also expired (NFS4ERR_BAD_SEQ or NFS4ERR_EXPIRED)?
> 
> Should the lease be renewed on a replay? Yes, the stateid is valid.

I don't do this now.  I'm not sure if this is the right answer or not,
however.

> What if the lease has expired on a replay?

Interesting question.  What I currently do is to return the old
response without checking for the current validity of the lease.
If the lease had expired and the some of all of the associated
locks were released, I would know.  Maybe that's wrong.

However, if it is, what about the case in which a replay is
recognized by the ordinary RPC replay logic, rather than the
special sequence number version.  I guess I would argue that
if it is OK to give back the orginal response with the ordinary
replay logic (and no lease checking and renewal), then the same
thing is OK if you recogize a replay through the sequence checking.
 
> Should the sequence number be bumped when the lease is expired?

I say yes it should.

<Diagram Deleted>

> 
> Operations that have a sequence number, and no stateid
> ------------------------------------------------------
> 
> OPEN 
> 
> Should the lease be renewed on a valid clientid and a bad 
> sequence number? If so, what error should be returned in the 
> case that the lease has also expired 
> (NFS4ERR_BAD_SEQ or NFS4ERR_EXPIRED)?

I return BAD_SEQ and do nothing with lease in this case.

> 
> Should the lease be renewed on a valid clientid and replay? Yes, 
> the clientid is valid.  What if the lease has expired on a replay?

I just return the old response and do nothing with leases.

> Should the sequence number be bumped when the lease is expired?

I say it should.

<Diagram Deleted>

> 
> Operations that have a stateid and no sequence number
> ------------------------------------------------------
> READ, WRITE, RENEW

SETATTR (for truncate) is in this class.

> 
> This case is clear, the lease is renewed only if the 
> stateid is valid.

The stateid being valid is a neceassry codition to do
lease renewal, but is it a sufficient condition?

What about a GRACE error?

If you have a valid lease (which you got with an OPEN
specifying reclaim) and a WRITE gets GRACE, should the
lease be renewed.  If the stateid is valid, I guess I
don't see any reason why not.

BTW, I've been meaning to send a message out about
GRACE errors in V4.  While the spec explains why you
have to return GRACE for READ, WRITE, LOCK, the
list of ops that can return GRACE is larger.

It isn't clear to me why you need to return GRACE errors
on LOCKU, CLOSE or RENEW.  I don't think the spec even
says that you do, but it does list GRACE as errors for 
these ops.

<Diagram Deleted>

---

From: Mike Eisler <[email-masked]>
Date: Fri, 22 Sep 2000 12:09:30 -0700 (PDT)

> > Should the sequence number be bumped when the lease is expired?

Perhaps I didn't understand the context of Andy's context.

I understood it in the context of the server side { r, L } processing
that that protocol specification discusses.

If the server has a nfs_lockowner for an expired lease, then, I don't
think it should increment L. Otherwise, if it did, then when it gets a
request from the client for r == L, what is it going to return?

	-mre

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 22 Sep 2000 13:50:44 -0700

> request from the client for r == L, what is it going to return?

I was saying that if a server gets a request with a correct sequence
number (that is, one more then the last one received), and that
request get a NFS4ERR_EXPIRED error, then the last sequence number received
for that lockowner should be incremented to reflect the operation
just performed and that therefore the client should not decrement
his sequence number when he receives this error.

I suspect that we understood the question differently but at this 
point I am very uncertain which of us misunderstood Andy (and it
could be both!).