Subject: [nfsv4] LOCK and locker

From: Thomas Haynes <[email-masked]>
Date: Sun, 26 Jun 2011 11:50:37 -0500

Rick and I have been going over an issue with the following sequence:

> - I'll use labels like O1, L1 for the stateids, to save typing
> Packet#
> 119 open lockfile1279
> 120 ok fh f7ca5653 stateid O1
> 135 open lockfile1279
> 136 ok (should be same fh) stateid O2
> 137 Lock stateid O1 new_lock_owner lock_seqid 0
> 138 ok stateid L1
> 143 LockU fh f7ca5653 stateid L1 lock_seqid 1
> 144 ok
> 147 Close fh f7ca5653 stateid O2
> 148 ok
> 149 Close fh f7ca5653 stateid O1
> 150 ok
> 
> - now on to the next test which creates the file again, etc
>  but does pretty much the same thing
> 159 open lockfile1279
> 160 ok fh f6ca5753 stateid O3
> 175 open lockfile1279
> 176 ok (should be same fh) stateid O4
> 177 Lock f6ca5356 stateid O3 new_lock_owner lock_seqid 0
> 178 ok stateid L2
> 183 LockU fh f6ca5356 stateid L2 lock_seqid 1
> 184 NFS4ERR_BAD_SEQID
> 
> It's the NFS4ERR_BAD_SEQID that gets turned into EACCES.
> 
> For some reason, the Netapp server thinks a lock_seqid == 1
> is incorrect the second time this is done. (I surmise it is
> because the lock_owner string is the same for #137 and #177
> and the Netapp server didn't reset lock_seqid back to 0 when
> it did #177, even though the client specified a new initial
> value of 0 in the #177 request.)
> 
> As far as I know, reuse of lock_owner strings is allowed. If
> not, a client would have to remember them forever to avoid reuse.

In summary, a lock-owner is created for file #1, that file is closed (and
removed), and then the same lock-owner is presented for the next file.

3530 has this to say about it:

>    The locker argument specifies the lock_owner that is associated with
>    the LOCK request.  The locker4 structure is a switched union that
>    indicates whether the lock_owner is known to the server or if the
>    lock_owner is new to the server.  In the case that the lock_owner is
>    known to the server and has an established lock_seqid, the argument
>    is just the lock_owner and lock_seqid.  In the case that the
>    lock_owner is not known to the server, the argument contains not only
>    the lock_owner and lock_seqid but also the open_stateid and
>    open_seqid.  The new lock_owner case covers the very first lock done
>    by the lock_owner and offers a method to use the established state of
>    the open_stateid to transition to the use of the lock_owner.

The crux of this text is whether the lock_owner is known to the server
or is new to the server. The NetApp server already knows about the
lock-owner via the first file open. It is easy enough for the server to
let the client be smart and reuse the lock_owner string. If the client
did it incorrectly, i.e., the lock-owner was based on the PID and in use
elsewhere, then there will be problems found later on.

How many client implementors reuse the lock_owner string?

Or considering this is the first time we've hit this issue in our server,
how many client implementors think it is reasonable to reuse the
lock_owner string?

Second, I think an error has been introduced into 3530bis, for its description
of this has this:

> In the
>    case in which it has, the argument is just a stateid representing the
>    set of locks associated with that open file and lock-owner, together
>    with a lock_seqid value that MAY be any value and MUST be ignored by
>    the server.

I.e., this has to have been copied over from 5661, where it makes sense to
ignore the explicit sequencing.

Anyone have a different opinion on this difference in meaning?

---

From: Rick Macklem <[email-masked]>
Date: Sun, 26 Jun 2011 18:12:07 -0400 (EDT)

> elsewhere, then there will be problems found later on.
I think that the key issue here is that the Open stateid used for the
open_to_lock_owner4 in the first case (at packet #120) has been closed
(at packet #149), so it's my understanding that a server cannot be
expected to "know" the lock_owner/lock_stateid after that.

This paragraph from pg 224 of RFC3530 hints at the same thing:
(It's describing ReleaseLockOwner)

   The client may choose to use this operation to ease the amount of
   server state that is held.  Depending on behavior of applications at
   the client, it may be important for the client to use this operation
   since the server has certain obligations with respect to holding a
   reference to a lock_owner as long as the associated file is open.
   Therefore, if the client knows for certain that the lock_owner will
   no longer be used under the context of the associated open_owner4, it
   should use RELEASE_LOCKOWNER.

It seems to me that a client will reuse a lock_owner string sooner or
later, it just happens that mine does it sooner. (It does use the
open_to_lock_owner4 case for packet #177.)

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Tue, 28 Jun 2011 13:35:56 -0400

> not, a client would have to remember them forever to avoid reuse.

That would be true if release-_lockowner did not exist.

Basically, that is why release_lockowner was added; so that the client would not have to remember this stuff forever, simply bcause the the server might retain knowledge of it.

If release by the client (unilaterally) were allowed, the integrity of the sequencing mechanism wood be compromised.

I agree with you about the rfc3530bis text.from 5661..  it's flat wrong in a v4.0 context.

i think the whole RFC3530 paragraph is Wong/confusing and I would suggest he following for RFC3530bis.

>  indicates whether the lock_owner is known to the server as the holder
>  of a lock for the current open file or or if it is new to the server
>  in that context. In the case that the lock_owner is known to the server
>  in that context, it must be known to the server in general and have
>  an established lock_seqid.  In that case, the argument is a lock
>  stateid previously associated with the current lock_owner and
>  open file and the current seqid associated with the lock_owner.
>  In the case that the lock_owner is not known to the server in that
>  context, the argument contains not only the lock_owner and lock_seqid
>  but also the open_stateid and open_seqid.  This case, the new lock_owner
>  case covers the very first lock done by the lock_owner and also
>  offers a method to use the established state of the open_stateid to
>  transition to the use of a lock_owner already known to the server in
>  another context.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
Sent: Sunday, June 26, 2011 12:51 PM
To: [email-masked]
Cc: Rick Macklem
Subject: [nfsv4] LOCK and locker

In the

---

From: Rick Macklem <[email-masked]>
Date: Tue, 28 Jun 2011 16:01:25 -0400 (EDT)

> Basically, that is why release_lockowner was added; so that the client
> would not have to remember this stuff forever, simply bcause the the
> server might retain knowledge of it.
>  
> If release by the client (unilaterally) were allowed, the integrity of
> the sequencing mechanism wood be compromised.  
>  
Wouldn't close of the Open Stateid that the LockStateid is associated with
(the association established by open_to_lock_owner4 and used to check for
"If locks are held...") imply release of the lock owner?

To me, the description of ReleaseLockOwner in RFC3530 (I haven't looked
at RFC3530bis, I'll admit:-) reads that the client can use this to notify
the server that it isn't going to use the lock_owner4 before it gets around
to closing the OpenStateid. (It talks about "certain obligations with repsect
to holding a reference to a lock_owner as long as the associated file is open"
in the IMPLEMENTATION para. for ReleaseLockOwner.)

Also, since ReleaseLockOwner takes the lock_owner4 as an argument, what about
the case where the same lock_owner4 string happens to still hold locks on a
different Lock_stateid related to an OpenStateid for a different file?
I don't see how I can use ReleaseLockOwner for this case?

Also, note that the second cycle (where it fails) is actually a different file
(that just happens to have the same name as the one deleted at the end of the
first sequence, that works), so it must be a different Open with a different
Open stateid and that, as I understand it (from the locks related to an Open
stateid that is being Closed) forces it to do a fresh open_to_lock_owner4 which
specifies the new initial lock_seqid#.

For example, take the case where a posix process on the client is doing locks
on 2 different files concurrently:
Doesn't the client have to do separate Lock/open_to_lock_owner4 Ops for each of
the two open files. It then gets 2 Lock stateids each with a separate initial
lock_seqid set (must be 0, it says in RFC3530) when the first Lock Op for that
file is done.
For this case, it seems that there are 2 lock_seqids, one for each Lock stateid,
although both have the same lock_owner4 string (which represents that POSIX process
in the client).

> I a gree with you about the rfc3530bis text.from 5661..  it's flat
> wrong in a v4.0 context.
>  
> I a gree with you about the rfc3530bis text.from 5661..  it's flat
> wrong in a v4.0 context.
>

---

From: Rick Macklem <[email-masked]>
Date: Tue, 28 Jun 2011 16:39:15 -0400 (EDT)

> server might retain knowledge of it.
Btw, if it turns out that ReleaseLockOwner is the "magic bullet", my
plan is to change my generation of lock_owner4 strings from something
that represents a POSIX process to something like "POSIX-process,FH"
so that they are different for different files locked by the same
process. That would allow a ReleaseLockOwner for the string when the
file is Closed and also avoid reuse of the same string in this case.
(since it is actually a different file with the same name)

Does anyone think using different lock_owner4 strings for different
files being locked by the same POSIX process will cause problems for
their server? (I won't have a chance to test against many servers for
quite a while...)

Thanks for help with this, rick
[good stuff snipped for brevity]

---

From: <[email-masked]>
Date: Wed, 29 Jun 2011 13:52:15 -0400

There are two possible  models here.

1)  a lockowner is specific to a fh/open-file/pid combination.
2)  a lockowner represents a pid and it can be used by many fh/open-file combinations.

I think it is clear that RFC3530 uses model #2.  the client can still choose to use model #1, but it does then you may have to deal specially with the case in which the same fh is opened by two different open owners.  This will result in two different lockowners which conflict although you may not want them to since they are the same pid.

Release_lockowner notifies the server that the lockowner will not be used any more.  If there are open files, you have to be sure that the lockowner will not be used subsequently with that open file or with additional open files.  Although the case of existing open files is stressed, because it is trickier, the basic purpose remains.  You get rid of the lockowner and that means you don't have to save the information indefinitely.

> "If locks are held...") imply release of the lock owner?

No, it would not.

If a lockowner is a process, then would close any open file for which you had 
gotten a lock, imply he process has gone?  You may have other such open files
And get looks on other ones or ones not even open yet.

RFC3530 is pretty clear about what a lockowner is.  It is a process or similar thing
And that means that RFC3530 is meant to embody model #2.

> ner4 as an argument, what about

What about it?

> I don't see how I can use ReleaseLockOwner for this case?

You can't do if that lock stateid might be used subsequently, as the text indicates.

One thing to do is only to do a Release_lockowner when the process exits, or when all
Files with locks for that owner are closed and you have some additional assurance that no more 
will be created.  Using  process description pid+pid-generation with a release
at exit. Also, note that the second cycle (where it fails) is actually a different file
(that just happens to have the same name as the one deleted at the end of the
first sequence, that works), so it must be a different Open with a different
Open stateid and that, as I understand it (from the locks related to an Open
stateid that is being Closed) forces it to do a fresh open_to_lock_owner4 which
specifies the new initial lock_seqid#.

I understand you specify a lock-seqid in this case,. But that it is not "initial".
It should be the correct next seqid for that lockowner.

> the two open files. 

Yes it does, but the second one is not "initial".  It is the next seqid for that lockowner and there is only one valid such next seqid, which is very good thing, sequencing-wise.

> It then gets 2 Lock stateids each with a separate initial

It gets two lock stateids and each uses some subset of the seqids for that lockowner.  Since the two sets
are disjoint, the "initial" one for each will be different.

> although both have the same lock_owner4 string (which represents that POSIX process

You don't have two lockowners with the same lock_owner4-string.  That is a recipe for chaos/confusion.

My reading of the spec is that there is one lockowner per clientid/lock_ownwer4-string pair.  Anything
else give me nightmares (I wake up screaming, thinking I've been made editor of RFC3010-henkiliakis. :-(

---

From: <[email-masked]>
Date: Wed, 29 Jun 2011 14:05:27 -0400

I think that the only negative consequence of making this fh-specific would have to do with the amount of information that client and server would have to maintain.  That doesn't seem awful.

If the same fh were opened by multiple open owners, you would have to only do the release when there were no relevant opens left.  The first close of two is not a good place to do the release.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 29 Jun 2011 15:02:49 -0400

> Release_lockowner notifies the server that the lockowner will not be used any more.  If there are open files, you have to be sure that the lockowner will not be used subsequently with that open fil…

Yes, but the issue is that the open_to_lockowner mechanism means that
lock stateids are always established on a fh/open-stateid/lockowner
basis. Unlike NFSv4.1, in NFSv4.0 those lock stateids get to hang around
until one of 2 events:

      * The client calls CLOSE on the associated open stateid
OR 
      * The client calls RELEASE_LOCKOWNER on the associated lockowner

The result is that if the client waits until the process dies before it
calls RELEASE_LOCKOWNER (as per model #2) then that can cause a
proliferation of old and unused lock stateids that the server has to
keep around until the associated open stateid finally gets CLOSEd.
Wasn't that precisely the scenario that RELEASE_LOCKOWNER was supposed
to cure?

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Jun 2011 20:25:01 -0400 (EDT)

David Noveck wrote:
> I think it is clear that RFC3530 uses model #2. the client can still
> choose to use model #1, but it does then you may have to deal
> specially with the case in which the same fh is opened by two
> different open owners. This will result in two different lockowners
> which conflict although you may not want them to since they are the
> same pid.
> 
> Release_lockowner notifies the server that the lockowner will not be
> used any more. If there are open files, you have to be sure that the
> lockowner will not be used subsequently with that open file or with
> additional open files. Although the case of existing open files is
> stressed, because it is trickier, the basic purpose remains. You get
> rid of the lockowner and that means you don't have to save the
> information indefinitely.
> 
Ok, so you are saying that Close of the associated Open stateid doesn't
release the lock stateid(s) created from the Open stateid via open_to_lock_owner4?
(The RFC does say that it must check that there are no locks when the Close is done.
 If the lock stateid outlives the Close, then there could be locks done on it after
 the Close and those would never get checked for. I think it makes more sense if
 the Close checks for "no locks" and then the lock stateid is discarded so further
 locks that wouldn't be associated with any Open can't be done?)

My understanding of what Trond said is that he understands that the lock stateid
hangs around until Close or ReleaseLockOwner, but not after it?

Also, if the Close doesn't release the lock stateid, then what about clients that
never do ReleaseLockOwner? From what I've seen, Solaris10 is such a client. Is it
guaranteed that the Solaris10 client will eventually run any server out of lock_owner space?

> It should be the correct next seqid for that lockowner.
Ok, but at the Bakeathon all the server's except one were happy with my
client specifying a new initial value of 0 and allowed the next Op to be
done with a lock_seqid == 1.

It seems to me that some servers must be maintaining a separate lock_seqid
for each lock stateid and not just one shared by all lock stateids for a
given lock_owner4 string? (Or they are allowing the open_to_lock_owner4 case
of Lock to re-initialize it to 0, which would produce the same outcome for
the test that fails for one server.)

Bottom line. I need to know what works for the servers that are "on the street
to-day". A few questions some of the server implementors can maybe answer?
- Can I assume that a lock stateid outlives the open stateid it was created from
  and continue to use it to do locking after that open stateid is closed?
- Can I use the same lock stateid to do locks on different files or do I need
  a separate one for each file being locked?
- If a client has more than one lock stateid for a given lock_owner4 string,
  do these lock stateids share the same lock_seqid or are there separate ones
  for each of the lock stateids?

> Yes it does, but the second one is not "initial". It is the next seqid
> for that lockowner and there is only one valid such next seqid, which
> is very good thing, sequencing-wise.
> 
When you say "lockowner" here, are you referring to the lock_owner4-string
or one of the possibly many lock stateids generated from it via the
open_to_lock_owner case of the Lock Op?

You seem to imply a one<->one relationship between lock_owner4-string and
lock stateid. However, when I test against servers, I normally see different
lock stateids each time I do an open_to_lock_owner4 case of Lock, even for the
same lock_owner4-string.

I also believe that the requirement that there be no locks related to an open
stateid when a Close Op is done pretty well forces a one<->many relationship
between lock_owner4-string and lock stateid.
> My reading of the spec is that there is one lockowner per
> clientid/lock_ownwer4-string pair. Anything
So, I have to ask "what are you referring to when you use the term lockowner"?
(The string in open_to_lock_owner4 or a lock stateid or a set of lock stateids or???)

Sorry, but every time I see "lockowner" I just don't know what you are referring to, rick

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Jun 2011 22:35:07 -0400 (EDT)

> is very good thing, sequencing-wise.
Ok, I just ran a small test program on my client (the one that uses <pid>
as a lock_owner4-string and assumes there can be multiple instances of that
string used for different open_to_lock_owner4 Lock Ops.) against a Solaris10
server. Looking at the packet trace, it is clear that the Solaris10 server
issues different Lock Stateids for multiple open_to_lock_owner4 Lock Ops
using the same lock_owner4 string and uses different lock_seqid #s for each
of the Lock Stateids.

  http://people.freebsd.org/~rmacklem/try2morelocks.c - the little program
  http://people.freebsd.org/~rmacklem/try2morelocks.snoop-v - the "snoop -v" text packet trace
  http://people.freebsd.org/~rmacklem/try2morelocks.pcap - binary capture for wireshark

If you look at the above, you'll see the program opens 2 files, does interleaved
lock ops on them both, with each using lock_seqid 0, 1, 2 (no NFS4ERR_BAD_SEQID replies).
Then it closes the files, reopens them and does fresh open_to_lock_owner4 Lock Ops
(still using the same lock_owner4 string) and restarts the lock_seqid at 0 for both.

So, either the Solaris10 server doesn't bother to check lock_seqid (I vaguely recall
getting NFS4ERR_BAD_SEQID from it in the past when I had client bugs) or it assumes
that there are separate lock_seqid numbers for each lock stateid and not one for
the lock_owner4 string.

I'll run the same test against a Fedora15 server as soon as I get it set up.
If anyone else would like to run the same test, it can be done with a recent
snapshot of FreeBSD9 from ftp.freebsd.org. (I can email more details if you are
interested.)

rick
> are disjoint, the "initial" one for each will be different.
As tested above, that's not the way the Solaris10 server does it.

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 30 Jun 2011 05:04:46 -0400

> I'll run the same test against a Fedora15 server as soon as I get it set up.

I'd expect the same results.  Each open_to_lockowner4 lock gets you a
new stateid with its own seqid stream.  (In fact I think that's true
even of multiple open_to_lockowner4's on the same file.)

Hm, I notice pynfs has some tests for this kind of thing that aren't
run, for some reason.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 08:46:34 -0400

> even of multiple open_to_lockowner4's on the same file.)

That makes sense if you do multiple open_to_lockowner4 on different
open_owners on the same file, but what does it mean to have different
stateids for the same lockowner/open_owner/filehandle combination?
Shouldn't the server return an error if a client tries to do that?

Cheers
  Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 08:56:39 -0400

> Shouldn't the server return an error if a client tries to do that?

In case it wasn't clear: I'm talking about the case of NFSv4.0 where
there is an existing lock stateid for that combination, and the client
tries to start a new sequence id stream.

For the case of NFSv4.1, the server can and should just return the
existing lock stateid after bumping the latter's sequence id.

Cheers
  Trond

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Thu, 30 Jun 2011 09:40:45 -0400 (EDT)

> > Shouldn't the server return an error if a client tries to do that?
One thing to be careful about here is that when you say open_owner are
you referring to an extant open stateid? The open stateid goes away when
it's Closed and it's the open stateid that is used for open_to_lock_owner4
in a Lock Op. (My understanding is that the open_owner4 string that was used
to get the open stateid "might" persist after all open stateids acquired by it
are closed. If it no longer persists, the next Open op using the same
string will be required to OpenConfirm.)

> tries to start a new sequence id stream.
I think (related to the above) that a large part of the confusion for me
is how long these things must/shall/might persist. And you have to differentiate
between the strings and the stateids, since at least for opens, the "strings"
can outlive the stateids. (I put "strings" in quotes, since I suspect it is
actually some data structure referenced by the open/lock_owner4 string.)

In other words, when you say "existing", this comes back to the question
"Does the lock stateid exist after the associated open stateid is Closed?"
and, if it does exist after the Close, when does it go away?

The problem my client has with this server appears to be that it assumes that the lock
stateid has gone away when the open stateid is closed and that the
lock_owner4-string can be reused for a open_to_lock_owner4 Lock Op after that,
for possibly the same file (and maybe same open_owner4 string) on a new open stateid
acquired by an Open Op later.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 30 Jun 2011 10:07:27 -0400

> tries to start a new sequence id stream.

Understood.  Yes, the current code lets the client call
open_to_lockowner multiple times for the same (file, lockowner string)
pair, and the result will be a new stateid and sequence id stream.

Well, that's just from a quick look back at the code.  Might be
interesting to write a few tests to confirm what happens in this case.

> existing lock stateid after bumping the latter's sequence id.

And we don't do anything differently in the 4.1 case.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 10:10:09 -0400

> and, if it does exist after the Close, when does it go away?

My understanding from the spec is that the lock stateid cannot outlive
the open stateid. Otherwise, the implication is that you would be able
to LOCK the file using your lock stateid even after the open stateid has
been CLOSEed, which would appear to be forbidden by sections 8.10 and
14.2.2.

> acquired by an Open Op later.

That appears to be correct behaviour to me. The Linux client also
behaves like this.

I was rather discussing the situation where the client tries a new
open_to_lockowner using the same open stateid and the same lockowner
that it used in a previous LOCK request. Unless both the open sequence
id and the lock seqence id match the server expectations, I don't see
how the request can be allowed to succeed.

Cheers
  Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 10:14:19 -0400

> pair, and the result will be a new stateid and sequence id stream.

Urgh... Shouldn't it result in a NFS4ERR_BAD_SEQID?

> And we don't do anything differently in the 4.1 case.

That would imply that a client can't send LOCK requests in parallel.

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 30 Jun 2011 10:30:13 -0400

> Urgh... Shouldn't it result in a NFS4ERR_BAD_SEQID?

Why should it?  Doesn't the seqid stream associated with the open
provide any necessary sequencing?

> That would imply that a client can't send LOCK requests in parallel.

Why not?  (Not arguing, I just don't understand.)

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 10:52:50 -0400

> provide any necessary sequencing?

Yes, but you now have 2 lock stateids and hence 2 lock seqid streams for
the same lockowner on the same file. How do you order LOCK and LOCKU
requests for that file?

IOW: how do the client and server then manage to agree on the lock state
if a LOCK is sent on one seqid stream and a LOCKU are sent on the
other? 

> Why not?  (Not arguing, I just don't understand.)

Same problem: ordering of LOCK and LOCKU.

Trond

NetApp

---

From: <[email-masked]>
Date: Thu, 30 Jun 2011 10:57:47 -0400

That's one of them.  The other is that the lokowner information has to stay around even past CLOSE.

To solve the stateid issue you would have to put FREE_STATEID in v4.0.  It is in v4.1, but it isn't needed for his purpose in v.1.

Sigh.  

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Wednesday, June 29, 2011 3:03 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] LOCK and locker

Cheers
  Trond

NetApp

---

From: <[email-masked]>
Date: Thu, 30 Jun 2011 11:02:46 -0400

No .  I'm saying he lock stateid goes away at CLOSE.

The issue is when the lockowner an its seqid goes away.

RFC3530 sort of implies it can go away when you lose your lease and stuff like that but h only ay to b sure it goes away is RELEASE_LOCKOWNER.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 30 Jun 2011 11:05:59 -0400

> if a LOCK is sent on one seqid stream and a LOCKU are sent on the

Then if I understand correctly, on its own it's harmless for a server to
permit this--there are no replay cases that allow this to happen by
accident--but it would be nice for servers to catch this case if only to
help identify buggy clients.

> Same problem: ordering of LOCK and LOCKU.

... and ditto for this case; the session ordering means a server won't
see two such open_to_lockowner's without a client actually sending them,
but catching this case on the server side could be useful as a debugging
aid.

(Correct me if I'm missing something.)

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 11:06:43 -0400

> That's one of them.  The other is that the lokowner information has to stay around even past CLOSE.

I don't understand this statement.

      * There is already language in RFC3530 that describes how the
        server should deal with a LOCKU replay after CLOSE. 
      * We definitely don't want to allow a new LOCK request using an
        old lock stateid after CLOSE.

So what is the point of keeping the lockowner after CLOSE, and where is
the text that says we need to do so?

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 11:33:44 -0400

> RFC3530 sort of implies it can go away when you lose your lease and stuff like that but h only ay to b sure it goes away is RELEASE_LOCKOWNER. 

Section 8.1.7 states explicitly that the server can free it as soon as
there is no more associated locking state.

I'd assume that most servers will want to do this anyway, since there is
no real penalty involved for the client in sending an open_to_lockowner
compared to reusing the old stateid.
Clients that _do_ try to reuse the old stateid are rather the ones that
risk getting penalised, since they may get a NFS4ERR_BAD_STATEID reply,
and so have to resend the LOCK request with an open_to_lockower. Why
would a client want to risk that?

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 11:37:58 -0400

> but catching this case on the server side could be useful as a debugging

Err... Where is the text that says the client has to send all LOCK/LOCKU
requests on the same slot?

There is no session ordering, and the client isn't buggy if it sends
LOCK and LOCKU on different slots.

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 30 Jun 2011 11:53:10 -0400

> requests on the same slot?

Agreed.  What the session does for us here is tells the server that if
it gets two open_to_lockowner's for the same (file, lockowner string)
pair, then that must be because the client actually sent both of them.

> LOCK and LOCKU on different slots.

So this ambiguous lock ordering will occur if a client actually chooses
it by sending two such open_to_lockowners.

Are we in violent agreement here, or am I missing something else?  Does
a server catching this case do anything more than catch buggy clients,
or are there legitimate reasons why a client would do two
open_to_lockowners for the same (file, lockowner string) ?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 12:04:33 -0400

> pair, then that must be because the client actually sent both of them.

Agreed.

> it by sending two such open_to_lockowners.

Why? Where does the spec allow you to do this?

> open_to_lockowners for the same (file, lockowner string) ?

If I have two threads that are trying to grab byte range locks on the
same file (but for different ranges) then why should I be forced to
order them?
I don't have to do that for the OPEN calls, or LAYOUTGET calls so why
are LOCKs such a special case?

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 30 Jun 2011 12:17:53 -0400

> help identify buggy clients.

BTW: You do send a NFS4ERR_BAD_SEQID if the client tries to OPEN a file
using an established open_owner (i.e. one for which the client already
holds share locks on another file), if it sends the wrong sequence id,
right?
So again, why are you treating lock_owner any differently, and how are
you handling the case of a lock_owner with multiple sequence id streams?
Does the client get to arbitrarily choose one stream or the other?

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Thu, 30 Jun 2011 20:10:28 -0400 (EDT)

> the text that says we need to do so?
I just read Sec. 8.1.7 (of rfc3530) and it seems to say "may" be released.
(And David, my appologies, I see it does define lock_owner, although the definition is
 a bit confusing to me because it uses the term for both open and byte range
 locks.)

However, I agree with Trond in that I don't see a reason to keep it around
after the Close.

Note that, for an open_owner4 string, when the server sees one that it still
has, but the client uses a different open_seqid#, it uses this open_seqid# as the new
initial value and requires an OpenConfirm to verify that it should be the new
one and this isn't an out of order replay.
(When all the open stateids for the open_owner4 string have been Closed.)

It seems to me that the server should do the same for a lock_owner4 string.
ie. When it gets a open_to_lock_owner4 Lock Op and it already has a record
for the lock_owner4 string but there are no lock stateids associated with it
(they've been zapped by a Close), then it should accept the new lock_seqid# as
its new initial value. It does not need a confirmation, since the
open_to_lock_owner4 Lock Op is ordered by the open_seqid# in the open_to_lock_owner4.
--> If you don't do this, you get inconsistent behaviour between the servers that
    choose to throw the lock_owner4 away when the Close happens and those that
    choose to keep it. (Which is what I believe I got bit by.;-)
    This also allows a client to not try and remember a lock_owner4 string until
    it knows it will "never" use it again. (It seems the Solaris10 and Linux client
    use different lock_owner4 strings each time, but I have no idea what algorithm
    they use and, as such, whether they ever reuse the same string? Linux appears
    to do an explicit ReleaseLockOwner whereas Solaris10 does not.)

I don't believe there are any words in rfc3530 that state what the use of the
lock_seqid inside open_to_lock_owner4 is, so it seems reasonable to clarify that
for rfc3530bis.

> > To solve the stateid issue you would have to put FREE_STATEID in
> > v4.0. It is in v4.1, but it isn't needed for his purpose in v.1.
> >
> > Sigh.
> >
> > > I think it is clear that RFC3530 uses model #2. the client can
> > > still choose to use model #1, but it does then you may have to
> > > deal specially with the case in which the same fh is opened by two
> > > different open owners. This will result in two different
> > > lockowners which conflict although you may not want them to since
> > > they are the same pid.
> > >
> > > Release_lockowner notifies the server that the lockowner will not
> > > be used any more. If there are open files, you have to be sure
> > > that the lockowner will not be used subsequently with that open
> > > file or with additional open files. Although the case of existing
> > > open files is stressed, because it is trickier, the basic purpose
> > > remains. You get rid of the lockowner and that means you don't
> > > have to save the information indefinitely.
> >