Subject: [nfsv4] Some locking issues in draft-minorversion1-10 ?

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 10 Apr 2007 16:00:43 -0400

I noticed some inconsistencies in the draft while investigating the
descriptions of mandatory versus advisory locking. I am tossing
them to the list because I'm not sure of the state of the issue
tracker, I'll leave that to the editors. Of course, I could be all
wet, too.

- Neither v4.0 nor 4.1 seem to offer the server any way to
indicate to the client whether it supports mandatory or advisory
semantics on its locks. In 8.1.4, there is mention of a "mandatory
lock attribute" (lines 7545-6) being "set" on a file, but this appears
to refer to internal server state rather than protocol. Is it the
intention that the client needs to probe each lock, perhaps by
reading the file from a new owner? Section 9.3.3 seems to say
this.

- Speaking of section 9.3.3, it also says the client is supposed
to treat its cache differently when mandatory file locking is in
effect on a given file and the file has locks. However, the text
is very unclear whether the salient locks are held by the client
doing the reading, or another client. As written, one possible
interpretation is that clients should attempt a read from the
server in all cases before using cache, which would appear to
defeat the purpose.

- There is no definition in the draft of the nfs_lock_type4 data
type, nor of its possible values READ_LT, etc, you have to look
in the .x for them. These are also referenced at several points
in the text, sometimes as (quote) "READ*_LT"!

Tom.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Tue, 10 Apr 2007 13:43:49 -0700 (PDT)

> indicate to the client whether it supports mandatory or advisory

At one point in an NFSv4.0 draft, there was a bit in the OPEN reply.
The problem with this approach is that mandatory locking is
subject to, on some operating environments, to being toggled
at will, depending on, for example, the mandatory lock attribute.

> to refer to internal server state rather than protocol. Is it the

It refers to the mandatory lock attribute settable on many UNIX or
UNIX-like systems via
   chmod +l <file name>

This is a bit that overloads the setgid bit. See a man page for
how this is done.

Clients can get GETATTR for this bit, and if set modify behavior
accordingly.

Could we do something better, such as a callback? Yes. Is there
rough consensus we should? I haven't seen it.

> reading the file from a new owner? Section 9.3.3 seems to say

9.3.3 is just saying if READ or WRITE returns NFS4ERR_LOCKED
on a file the client successfully OPENed, then this indicates
mandatory locking is going on (some other client has a byte range
lock on the file). The client then needs to obtain a byte range
lock; or it can just keep trying the I/Os.

> server in all cases before using cache, which would appear to

It means whether the client has a lock on the affected range.
I've no problem with making the text more explicit ... please file an
issue and suggest some text.

> in the text, sometimes as (quote) "READ*_LT"!

Right now we cut and paste all the data types (in most
cases from the .x into the xml, arguments and results
are one exception), and that is clearly error prone. One of the things
I want to do is to make sure every
data type appears in the i-d. I'm thinking that a shell script that
spits out .x and .xml fragments would be the most maintianble without
having to add 100 or so files to the CVS tree.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 10 Apr 2007 16:48:19 -0400

On Tue, Apr 10, 2007 at 04:00:43PM -0400, Talpey, Thomas wrote:
> descriptions of mandatory versus advisory locking.

A basic question: does anyone implement mandatory locking over NFS?  If
so, has it been used?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 10 Apr 2007 17:12:53 -0400

> A basic question: does anyone implement mandatory locking over NFS?  If

We've certainly been testing mandatory locking at the Bakeathons. IIRC
it was mainly against the EMC server.

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Apr 2007 14:49:57 -0400 (EDT)

> at will, depending on, for example, the mandatory lock attribute.
Here's a simple example of the problem as I see it.
Client A Opens foo for writing
- writes on foo are cached in the client for write-back
Client B sets "mandatory locking" and write locks all bytes of foo
Client A's write backs fail NFS4ERR_LOCKED
     --> all it can do is keep trying, hoping that Client B will unlock
         the byte range
         --> this could result in a very constipated client side data
             cache

Now, how can Client A avoid this:
1 - Write through caching (not very attractive, from a performance
     point of view).
OR
2 - Grab write locks on all byte ranges being written, when they are
     written into the client cache. Note that this is BEFORE Client B
     sets the "mandatory locking" attribute.
     In other words, it must do so for all writes to all files if there
     is any possibility of "mandatory locking" being subsequently set
     for the file. (A lot of overhead grabbing all those locks just in
     case the "mandatory locking" attribute might get set.)

There is a similar problem for read caching,
where a process on Client A will successfully read data from the cache
when Client B holds a mandatory write lock on that data.

I think the Open Reply flag was the correct solution, but it required
servers to not change the "mandatory locking" attribute on a file while
that file was Open, so others didn't like it. (It was in RFC3010, which I
suppose you could call a draft;-)

The only other thing I can think of that would help a client would be
an Attribute that says "this file system will never do mandatory locking",
so the client knows it can cache files on the file system.

> rough consensus we should? I haven't seen it.
Agreed. No-one seems concerned about the above problem. My "fix" is to
have a "Disable Caching" flag on the client mount and document the use
of that flag for servers that do mandatory locking.

A long time favourite nit of mine, rick

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 11 Apr 2007 15:55:11 -0400

> so the client knows it can cache files on the file system.

Reality check: when would the above scenario _ever_ make sense? ...and
if it does, do we really care that the user on client A that was
"incorrectly" caching without holding locks obtains stale data?

Cheers,
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Apr 2007 16:48:27 -0400 (EDT)

>>          --> this could result in a very constipated client side data
[stuff snipped]
> "incorrectly" caching without holding locks obtains stale data?
When Client A opens foo and starts writing to it, there isn't mandatory
locking, so why should it get write locks? So, why do you say that Client 
A was "incorrectly" caching? (Does the Linux client get
exclusive byte range locks on all files it is writing?)

Although I can't see why anyone would "chmod +l foo", if users on clients
do do that, why not on Client B at that point in time, followed by Opening
and exclusive locking the entire file?

I don't see why you think this scenario is bogus? rick

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 11 Apr 2007 17:06:04 -0400

> I don't see why you think this scenario is bogus? rick

My 2 points were

1) What would be the point of switching on mandatory locking on an
existing file? Particularly one which has been written with no locking
at all? I just don't see when in a shared environment you would ever
want to transition a file from a state where clients initially don't use
locking to one where they MUST.

and

2) Even if you did, why would you then care about serving up cached data
to a user that then fails to use locking? Particularly so if the file
was unlocked when the user opened it (a 'chmod +l' would usually cause
the file change attribute to be modified, which would trigger a cache
flush if the file was marked as mandatory locked when client A opened it
again). NFS caching rules state that we only guarantee cache correctness
for changes that happen _before_ the call to OPEN.

Cheers
  Trond

PS: "The Windows server always does mandatory locking" scenario is more
worrying. How would a client know that it is talking to a Windows
server? Are they required to always set the 'mandatory lock' mode bit?

---

From: Mike Eisler <[email-masked]>
Date: Wed, 11 Apr 2007 14:17:33 -0700 (PDT)

> >>          --> this could result in a very constipated client side

And adding the "mandatory lock" result to OPEN doesn't
change anything. Only a callback, acknowledged by each
client that has the file OPENed, will solve this.
I doubt it was ever intended that mandatory locking would
be arbitrarily turned on and off. I can't think of an
example yet where such arbitrary toggling would break apps
accessing a local filesystem, but it would not surprise me
if there is such a case.

> >
> > Reality check: when would the above scenario _ever_ make sense?

The point being that doing chmod +l a file currently being opened
by a mission critical application is a pretty risky, inadvisable thing
to do. Indeed, unless one is absolutely certain no other process
or client has the file open (e.g. you OPEN the file with a deny
mode of all, and once successfully OPENed, then set the l bit) 
the only safe time to set the bit is when the file is created. 

> ...and

Advisory and mandatory locking are of limited use unless the apps
are intelligent enough to use it. If I intend for byte range
locking to be used, and by app modifies or reads a record via
a sequence of I/Os, I had better get a byte range lock on the range
the I/Os will encompass before issuing the sequence of I/Os,
regardless whether mandatory locking is on or not. So if two
apps on two clients are contending for the same file, then
I agree with Trond that the app on client A was incorrect under
your scenario.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Apr 2007 17:27:44 -0400 (EDT)

> for changes that happen _before_ the call to OPEN.
Cache correctness (I assume you mean coherency) is one thing, but a write
back failing unexpectedly with NFS4ERR_LOCKED is beyond the coherency
guarantees (or lack thereof, if you prefer). It's akin to ENOSPACE.

Ok, based on your 1), would you have no problem with the "chmod +l foo"
being delayed until the file is not open by any client? If so, then the
RFC3010 solution of replying OPEN4_RESULT_MLOCK to any Open where the
server will be applying mandatory locking to the file is just fine. (see 
below)

> server? Are they required to always set the 'mandatory lock' mode bit?
No problem, with the OPEN4_RESULT_MLOCK reply flag to Opens, per RFC3010.

Alternately, a read-only attribute that says if the server is applying
mandatory locking to a file at that point in time, would be better than
nothing? (The setgid without group execute mode bit scenario is just
mentioned as one way a server might choose to do mandatory locking, so
they aren't defined as a MUST for servers doing mandatory locking.)

Have fun with it, rick

---

From: "Yoder, Alan" <[email-masked]>
Date: Wed, 11 Apr 2007 14:43:16 -0700

All locks in Windows are mandatory.  A routine
PITA on filers is clearing locks left behind by
clients that have gone off in the weeds while
holding locks.  

Alan 

> -----Original Message-----
> From: Trond Myklebust [mailto:[email-masked]] 
> Sent: Wednesday, April 11, 2007 2:06 PM
> To: Rick Macklem
> Cc: Mike Eisler; Talpey, Thomas; [email-masked]
> Subject: Re: [nfsv4] Some locking issues in draft-minorversion1-10 ?
>

---

From: Mike Eisler <[email-masked]>
Date: Wed, 11 Apr 2007 14:50:47 -0700 (PDT)

> No problem, with the OPEN4_RESULT_MLOCK reply flag to Opens, per

You'll never give up on that will you? :-)

Gee, what if I upgrade my Windows-based NFS server to
FreeBSD, in-place? :-)

> 
> Alternately, a read-only attribute that says if the server is

You mean like the mandatory lock bit in the mode attribute? :-)

> applying

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 11 Apr 2007 17:51:17 -0400

> guarantees (or lack thereof, if you prefer). It's akin to ENOSPACE.

...or to someone doing a 'chmod a-w' on the server while your client was
caching the write. It is just one of those things that can happen. I
don't see how the client can recover in that situation: grabbing a write
lock would ensure that it could write out that data, but does nothing to
ensure that it _should_ (i.e. that it is the correct thing to do in that
context).
I'd argue that punting, and returning an error to the application is
more likely to be the correct method for recovering since the user has
violated the locking conventions on that file.

> server will be applying mandatory locking to the file is just fine. (see 

I'd say that is still a dodgy scenario.

> they aren't defined as a MUST for servers doing mandatory locking.)

I'd prefer the former over the latter, since that would clean up the
problem on the UNIX server of relying on a mode bit protocol that is not
defined in any standard.

Cheers,
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Apr 2007 18:01:45 -0400 (EDT)

On Wed, 11 Apr 2007, Mike Eisler wrote:

[stuff snipped]
> change anything.
If the server is not permitted to do "chmod +l foo" while the file is 
Open, it does. (This was why OPEN4_RESULT_MLOCK was removed from RFC3530,
I was told.)

> Only a callback, acknowledged by each
Yep, but as you mentioned before, more complex.

> be arbitrarily turned on and off.
Then why must it be doable while a file is Open?

Can I summarize the Reality Check as "clients that do read/write
sharing of a file without using byte range locking are asking for 
trouble"? If so, another tack might be to state something like:

 	If clients are read/write sharing a file, the should hold
 	byte range lock(s) on all byte ranges being accessed/modified.
 	If the clients do not do this, there can be indeterminite
 	results from read/write Ops, particularily if the server
 	chooses to enforce mandatory locking.

Clients still need to follow what is descibed in Sec 9.3.3 of RFC3530
either always OR when the server is enforcing mandatory locking, so
it would be really nice if the client knows. (Otherwise the client
side cache has to be maintained byte granular with regions that start/end
on arbitrary byte boundaries, because other clients may be locking/using
arbitrary non-overlapping byte regions. Try building that instead of
the traditional unix-style block/page sized client buffer cache.:-)

Keep having fun with it, rick

---

From: Rick Macklem <[email-masked]>
Date: Thu, 12 Apr 2007 12:26:11 -0400 (EDT)

> You'll never give up on that will you? :-)
Actually, I gave up on it years ago. However, since Tom brought it
up again, I couldn't resist kicking the can one more time:-)

> You mean like the mandatory lock bit in the mode attribute? :-)
I can live with the mode bit stuff, so long as it is a MUST for any
server doing mandatory locking and a MUST NOT send that combination
of "setgid and not set group execute" when not doing mandatory locking.

Honestly, I don't care about mandatory locking, because modifying the
client side buffer caches to support it is just too much work. If the
"mode bit stuff" becomes required, I'll just change my mount point flag
to one automagically set for that combination of mode bits (set + not 
set), which disables client side caching.

Has any client implementor actually constructed a byte granular data
cache that supports Sec 9.3.3 of RFC3530? (If not, then mandatory locking
won't work correctly for those clients, unless caching is disabled.)
By byte granular, I'm referring to the fact that a block could consist
of several separately locked regions on arbitrary byte boundaries, that
have to be handled separately. (Might require separate Read/Write Ops,
since the locks are related to different LockStateID's, for example.)
And then, how do you handle the case where the App unlocks part of the
block? And so on, and so forth...

Lots'a'fun. More than I'm interested in trying to code, rick