Subject: Re: [nfsv4] NFS client behavior on lock owner

From: Saul Tamari <[email-masked]>
Date: Thu, 1 May 2014 14:24:52 -0400

Hi Rick,

Do you know if that also applies to current NFSv4.1 clients/servers?
Lock-owners are not used with different open files at the same time?

Thanks,
Saul
On Apr 28, 2014 6:56 PM, "Rick Macklem" <[email-masked]> wrote:

> Yanbei Wang wrote:
> >
> >
> >
> >
> > Dell - Internal Use - Confidential
> >
> > Hi,
> >
> >
> >
> > I observed same behavior regarding lock owner from several NFS
> > 4.0/4.1 clients from different OS, including CentOS, Fedora and
> > Solaris. That is, when a range lock is outstanding, the lock owner
> > will not be shared. Here is an example.
> >
> > 1. Open file1
> >
> > 2. Lock range 0-100
> >
> > 3. Unlock range 0-10
> >
> > 4. Open file2
> >
> > 5. Lock range 0-100
> >
> >
> >
> > The lock owners from 2 and 5 will be different even if a single
> > process is opening the two files. If the unlock in 3 unlocks the
> > full range 1-100, the lock owner in 2 will be same as in 5.
> >
> >
> >
> > Is this true for all the NFS 4.0/4.1 clients? NFS spec doesn’t seem
> > to address it clearly. Can server implementation reply on this
> > behavior?
> >
> I believe the answer is "yes" for all of these questions.
> The spec. didn't specifically allow multiple instances of any
> particular lock_owner. By multiple instances, I am referring
> to using the same lock_owner string again for the open_to_lock_owner
> case of Lock, if the server might still have state for that
> lock_owner string.
>
> This means that a client cannot re-use the same lock_owner
> string until there is no state (locks) related to that lock_owner
> string and the client has done a ReleaseLockOwner operation
> to ensure that the server no longer has the lock_owner string
> in its state structure.
>
> I ran into this late in the development cycle for the FreeBSD
> client because most servers allowed multiple instances (related
> to different open_stateids) of the same lock_owner string
> and my client was doing that.
> When I found out that not all servers allowed this (and it
> was not specifically required to be supported), I fixed the
> FreeBSD client by adding a FH to the lock_owner string (so
> it refers to a process/file and not just a process) and by
> doing explicit ReleaseLockOwner ops when the client is no
> longer going to be using the lock_owner.
>
> I suspect that, if you look at the case where you unlock the
> entire range at 3, you will find a ReleaseLockOwner op being
> done for the lock_owner between 3 and 5, so it can safely
> use the same lock_owner string again.
>
> All of this comes from one sentence in RFC3530's description
> of Close, where it notes that the server must check for
> "associated locks". After some time, it became apparent that
> this association can only be established by the open_to_lock_owner
> case of Lock and that means that this case must be used for
> 4, since it is using a different open_stateid for a different
> file (file2).
>
> This took quite a while to resolve during interoperability
> testing and I'll admit I've never looked at rfc3530bis, to
> see how it is described there.
>
> Now I'll see if anyone disagrees with my view of this, rick
> ps: I can at least tell you that the answer is "yes" for the
>     FreeBSD client.
>
> >
> >
> > Thanks,
> >
> >
> >
> > Yanbei
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nfsv4
> >
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 1 May 2014 15:36:42 -0400

Hi Saul,

There is no permanent state associated with lock owners in NFSv4.1. It
all disappears when the last associated stateid disappears.

In any case, why are you guys asking about implementations in the
first place? Your question _should_ be "what does the spec
allow/disallow". In 5 years, new client implementors won't remember a
thing about what was said in this email exchange.

---

From: <[email-masked]>
Date: Thu, 1 May 2014 20:53:00 +0000

Hi Trond,

I consider state-ids associated with lock-owners as state. In NFSv4.1 when does their lifetime end? When the last locked byte has been unlocked or when FREESTATEID has been called?

I agree that we should have an RFC compliant implementation but the uniqueness of lock-owners among multiple files is not clear to me, especially when we inspect the current implementations of different clients that seem to have unique lock-owners for each file.

Thanks,

Saul

Sent from Windows Mail

From: Trond Myklebust
Sent: ‎Thursday‎, ‎May‎ ‎1‎, ‎2014 ‎3‎:‎36‎ ‎PM
To: Saul Tamari
Cc: Rick Macklem, nfsv4

Hi Saul,

---

From: Rick Macklem <[email-masked]>
Date: Thu, 1 May 2014 20:55:27 -0400 (EDT)

> Lock-owners are not used with different open files at the same time?
Well, the question is whether or not the server must allow multiple
Lock ops with the open_to_lock_owner case for the same lock_owner and
different open_stateids. I believe the answer to this is "no" and my
way of avoiding this in the client was putting the FH in the lock_owner
string (there are other ways a client could avoid this).

I don't believe NFSv4.1 servers are required to allow multiple Lock ops
with the open_to_lock_owner case for the same lock_owner and different
open_stateids either. (But I'll admit I haven't read through RFC-5661
to see if this is clearly stated in it.)

rick

> Thanks,
> Saul
> On Apr 28, 2014 6:56 PM, "Rick Macklem" < [email-masked] >
> wrote:
> 
> 
> Yanbei Wang wrote:
> > 
> > 
> > 
> >

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 1 May 2014 23:56:17 -0400

> longer going to be using the lock_owner.

The lock _stateid_ is definitely restricted to a <file, lockowner,
open stateid> triplet, however why is the lockowner similarly
restricted?

> 4, since it is using a different open_stateid for a different

That sentence definitely links specific lock stateids to the open
stateids, but how does it do the same for lock owners?

According to http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-33#section-9.1.9
there is nothing preventing the association of multiple lock stateids
to a single lock owner. The only expectation there is that the lock
owner MAY disappear once the lock stateids are all trivial (i.e. point
to no locking state). As far as I can tell, the idea is that lock
owners have very much the same properties as open owners, except that
they live in a different name space.

The only restriction in
http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-33#section-15.12.5
is that "In all of the cases, there is a lock_seqid specified, whether
the lock-owner is specified explicitly or  implicitly.  This seqid
value is used for checking lock-owner sequencing/replay issues.  When
the given lock-owner is not known to the server, this establishes an
initial sequence value for the new lock-owner."

Cheers
  Trond

---

From: Saul Tamari <[email-masked]>
Date: Fri, 2 May 2014 12:48:50 -0400

Hi,

Can we get the requirement of the uniqueness of lock-owners changed in the
bis to match current client implementations?
We've checked Linux, FreeBSD and Solaris. They all use unique lock-owners
per file.

Thanks,
Saul
On May 1, 2014 11:56 PM, "Trond Myklebust" <[email-masked]> wrote:

> According to
> http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-33#section-9.1.9

---

From: Tom Haynes <[email-masked]>
Date: Fri, 2 May 2014 13:12:24 -0400

Hi,

What about AIX?

And what motivates the change?

Tom

On May 2, 2014, at 12:48 PM, Saul Tamari <[email-masked]> wrote:

> Hi,
> 
> Can we get the requirement of the uniqueness of lock-owners changed in the bis to match current client implementations?
> We've checked Linux, FreeBSD and Solaris. They all use unique lock-owners per file.
> 
> Thanks,
> Saul
>

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 2 May 2014 14:21:34 -0400

Either way, it would be wrong to change the bis based on a
verification of the client implementations alone. What do you do about
servers that suddenly become spec-incompliant?

Trond

---

From: Saul Tamari <[email-masked]>
Date: Fri, 2 May 2014 14:48:09 -0400

The change I'm suggesting is for a client-side spec change. Current
clients already function that way so they can interoperate with
current servers.
This change would actually tell future client implementors what is
currently needed to interoperate with current server implementations.

I don't see how servers can be break with this change as unique
lock-owners support are a subset of non-unique lock-owners support.

---

From: Saul Tamari <[email-masked]>
Date: Fri, 2 May 2014 15:12:15 -0400

...
[snip]
> longer going to be using the lock_owner.

Do you remember which server required this change on your client implementation?

Thanks,
Saul

---

From: Saul Tamari <[email-masked]>
Date: Fri, 2 May 2014 15:21:45 -0400

Hi,

We didn't test with an AIX. We'll try and get access to one and test this.

The motivation for this is for faster and more scaleable distributed
server implementations. The current client implementations already
work that way so I assume it would be best to document this practice
instead of burdening servers with an unused flexibility.

Thanks,
Saul

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 2 May 2014 16:45:03 -0400

> lock-owners support are a subset of non-unique lock-owners support.

In NFSv4.1, I agree that the differences are trivial.

In NFSv4, the replay semantics differ: if the lockowner is globally
unique, then so is the sequence id, and so clients would need to
serialise even if the stateids differ because the lockowner (and hence
the sequence id) is not attached to the file.
Servers have to enforce these semantics.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 2 May 2014 20:44:13 -0400 (EDT)

> open stateid> triplet, however why is the lockowner similarly
Well, I probably described it poorly. As I recall, the problem is
the NFSv4.0 seqid that is for "each" lock_owner. If/when a client
uses the open_to_lock_owner case of the Lock Op. for a lock_owner
string that the server still holds the seqid for...
- What does the server do with the lock_seqid in the open_to_lock_owner4,
  which is the initial value for the seqid?
  - Some servers seem to keep multiple instances of the
    lock_owner string/lock_stateid/lock_seqid (one for each open_stateid)
    and uses this lock_seqid as the initial value of this new seqid.
   (ie. A separate seqid for each of the lock_stateids for a lock_owner string.)
  - One I tested against did not. I can't remember if it ignores this new initial value
    for the seqid or resets the lock_owner's seqid to this new initial
    value, but what I would get was an NFS4ERR_BAD_SEQID from this server.
   (ie. One seqid for the lock_owner string shared by the lock_stateids.)
I avoided the NFS4ERR_BAD_SEQID problem by making the lock_owner string
unique for <fh, client process> instead of just <client process> and by
making sure the client did a ReleaseLockOwner when done with a lock_owner
string, so that the server wouldn't retain a record of the string's
seqid value. (That way, I avoided ever having more than one lock_stateid
for a given lock_owner string.)

Put another way, it is the seqid for a lock_owner string/lock_stateid that
was problematic.

I haven't looked to see if rfc-3530bis clarifies this or not?

I don't think there is a problem with NFSv4.1, since the lock_seqid isn't
used.

rick

---

From: Saul Tamari <[email-masked]>
Date: Sun, 4 May 2014 07:30:52 -0400

I agree with your point regarding NFSv4.0.
The fact that NFSv4.0 clients use a per-file unique lock-owner
(instead of a single lock-owner/seqid for several files) makes
concurrent non-serialized lock operations on multiple files possible
and provides better performance.