Subject: [nfsv4] Last call for RFC3530bis-13 (ends on Sept 2nd)

From: Spencer Shepler <[email-masked]>
Date: Wed, 17 Aug 2011 05:37:30 +0000

Tom has dropped version -13 of RFC3530bis.

Given the removal of LIPKEY/SPKM3 and for the fact that
the I-D is huge, I am going to run another last call.

This last call will run until the end of the day Sept 2, 2011.

Editorial comments may be sent to Tom directly.  All others
should be directed to the WG alias.

This IETF tool is very nifty in "diffing" I-D versions.

http://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc3530bis/history/

and will let you run diffs back to the original.

Have fun.
Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 17 Aug 2011 09:05:23 -0400

> and will let you run diffs back to the original.

Does it contain any advice on the OPEN return value when applied to a
file that is not a regular file, symlink or directory? I'll be back in
active duty and able to check for myself tomorrow, but I thought I'd
draw it to your attention now.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 17 Aug 2011 13:30:28 -0500

> draw it to your attention now.

I assume you are speaking with respect to the recent thread:

Re: [nfsv4] open() of device special files

And the answer would be no, we made no changes from 3530
there.

You want to add text to the effect to return NFS4ERR_SYMLINK
on an OPEN of an incorrect type?

---

From: Rick Macklem <[email-masked]>
Date: Wed, 17 Aug 2011 17:32:56 -0400 (EDT)

> on an OPEN of an incorrect type?
Well, the FreeBSD server returns NFS4ERR_INVAL for an Open attempt
on a device special file. If every other extant NFSv4.0 server out
there returns NFS4ERR_SYMLINK for this case, I can easily change it.

However, I think "odd" is an understatement for returning NFS4ERR_SYMLINK
for this case and if other servers are not doing this, I don't see how
that can be required now? (Maybe suggested or recommended, but I think
that NFS4ERR_INVAL sounds closer as a suggested error return for this case.)

It seems to me that a client could put:
Lookup "name"
Verify (type==NFREG)
before the Open in a compound for the case of an Open without Create,
to avoid doing an extra RPC of (Lookup, Getattr) before the compound with
the Open in it, if the VFS allows that. I suspect Opens of device special
files won't specify O_CREATE. (The FreeBSD VFS pretty well requires
a (Lookup, Getattr) RPC before the Open anyhow, so I'm stuck with the
extra RPC.)

Alternately, the client will have to accept pretty well any error return that
isn't obviously something else as an indication of this and do a
(Lookup, Getattr) RPC to find out, I think?

rick
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 17 Aug 2011 18:57:46 -0700

> -----Original Message-----
> From: Rick Macklem [mailto:[email-masked]]
> Sent: Wednesday, August 17, 2011 5:33 PM
> To: Haynes, Tom
> Cc: [email-masked]; Myklebust, Trond
> Subject: Re: [nfsv4] Last call for RFC3530bis-13 (ends on Sept 2nd)
> 
> Thomas Haynes wrote:

No, that is no where near race free. I can easily substitute a new file between your lookup and open.

You need open by filehandle in order to make your suggestion work, and that is only available in minor version 1.

> (Lookup, Getattr) RPC to find out, I think?

Yech. Also not race free.

Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 09:40:10 -0400 (EDT)

Trond Myklebust wrote:
> > > file that is not a regular file, symlink or directory? I'll be
> > > back
> > in
> > that NFS4ERR_INVAL sounds closer as a suggested error return for
> > this
> > case.)
> >
Sure, and you can also replace the file object after a successful Open
(for most cases, see below) while it is Open/open, with one of the
same name and same or different type of object.
(lower case open refers to the client's open syscall, not an NFS Op)

> You need open by filehandle in order to make your suggestion work, and
> that is only available in minor version 1.
> 
Well, that's all true, but has been that way for NFS all along.
(ie. If another client replaces a file with one of the same name,
 things break.)
As I understand it, the only exceptions are:
 - if the client Open'd a regular file successfully with DENY_WRITE
   or DENY_BOTH (and that is only a SHOULD and not a MUST, at least
   for RFC3530), the remove fails.
 - If the client holds a delegation issued when Open'd with DENY_NONE
   or DENY_READ, the delegation will be recalled. However, since this
   happens for a lot of reasons and not just Remove, it doesn't really
   help the client recognize the file has been replaced. (Which puts the
   client in another weird quirky corner of the protocol, in that it can
   have a valid Open_StateID for a file that may or may not still exist
   at the server. At least for RFC3530, it uses MAY for this case w.r.t.
   the file's data still being extant after the Remove. ie. A server MAY
   implement POSIX-like semantics for this case.)

My understanding of your problem was that the client attempted an Open
Op for what was a non-NFREG file at that point in time and then the
open syscall failed because the server replied NFS4ERR_INVAL instead of
NFS4ERR_SYMLINK for this case.
I think it's true that the client can't just try Opens over and over and over
again when they fail (another client could conceivably just keep replacing
the object over and over and...), but it can do a (Lookup, Getattr) RPC
when the Open fails, to see if there is a non-NFREG object of the correct
name extant at the point of time the Lookup/Getattr is done.
If that's the case, all I think a client can do is assume the object will
not be replaced while the client is using it. (Same goes for most Opens
of NFREG files after the Open Op, until it sees NFS4ERR_STALE.)

rick

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Aug 2011 09:56:48 -0400

> (lower case open refers to the client's open syscall, not an NFS Op)

So what? By then you have a filehandle, and so can still reference the
original object. This situation is completely provided for in the
protocol.

> NFS4ERR_SYMLINK for this case.

No. The client attempted an OPEN for an _unknown_ object. Our client
doesn't ever do a LOOKUP if it can optimise it away.

If the server replies "that's an invalid argument", then that is a
completely unhelpful reply.

> of NFREG files after the Open Op, until it sees NFS4ERR_STALE.)

This is fixed in RFC5661 with the 'open by filehandle' cases. In
RFC3530, we should at least allow the client to obtain sane error
returns that give it a hint as to what is wrong.
The OPEN operation in RFC3530 is basically "LOOKUP" with frills. For
LOOKUP, NFS4ERR_INVAL has a very specific meaning; it means you supplied
a screwed up filename. Why is it at all useful to make that mean "you
supplied a screwed up filename, or I looked up a wrong file type on the
filesystem" in the case of OPEN when there are better alternatives?

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2011 10:11:29 -0400

> (lower case open refers to the client's open syscall, not an NFS Op)

To implement the open() syscall, you need either

	- A filehandle and major/minor number, or
	- A filehandle and open stateid

You can't request both in the same operation, so you have to try one or
the other, and your only choice is to bounce back and forth (presumably
some limited number of times) when failure of one indicates you should
have tried the other.

If there's an agreed upon "this is a device" error return from OPEN,
then you know on return from OPEN that the path pointed to a device.
But you still have to go back to the server with a PUTFH, GETATTR,
GETFH, and that could race.

On the other hand, having an agreed-upon error would at least save you
in those cases when the OPEN failed with some *other* error--you'd know
you could fail right away instead of having to go make a race-prone test
to decide whether it was a fatal error or a sign you should have treated
it as a device.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2011 11:07:11 -0400

> on an OPEN of an incorrect type?

Also not a spec issue unless this goes unfixed in our implementations,
but note rfc 5661 requires NFS4ERR_WRONG_TYPE in the case of OPEN of a
device (p. 455 in the OPEN implementation section, "If the component is
neither of those but not an ordinary file, the error NFS4ERR_WRONG_TYPE
is returned.")

One other 3530bis question:

I'm trying to decide where else the server could return NFS4ERR_SYMLINK.
3530bis says COMMIT, LOOKUP, LOOKUPP, OPEN, READ, WRITE.

I wonder about:

	CLOSE, OPEN_CONFIRM, LOCK, LOCKU, LOCKT: as with READ, WRITE,
	and COMMIT, these cases where the client should have known
	better, and we may as well allow (but not require)
	NFS4ERR_SYMLINK, as we do NFS4ERR_ISDIR.  (Also NFS4ERR_SYMLINK
	is on the error lists for some of these in rfc 5661).

	CREATE, READDIR, REMOVE, RENAME, SECINFO, possibly some others:
	currently require NFS4ERR_NOTDIR.  I wonder if NFS4ERR_SYMLINK
	should also be allowed.  I guess these are also buggy-client
	cases, so I think it's safely left up to the implementation
	whether to return INVAL, NOTDIR, or SYMLINK.

(And, a really nitty nit: why are the page numbers in the table of
contents at

	http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-13

off by one?)

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Aug 2011 11:13:02 -0400

> 	whether to return INVAL, NOTDIR, or SYMLINK.

I can live with those returning NFS4ERR_INVAL. Those are all cases where
the error applies to an invalid object which was passed in the form of a
filehandle. IOW: the client can and should know better.

> 	http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-13

Are they? They seem correct to me. Remember that RFCs list the page
number at the bottom of the page and not the top.

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2011 11:18:56 -0400

> filehandle. IOW: the client can and should know better.

Yeah, agreed.

(In fact, INVAL may be the preferred error--but as long as the operation
fails in these cases I think any reasonable error is fine.  E.g. the
linux server is currently going out of its way to map lower-level
symlink errors to INVAL in the above cases, and I don't see the point,
so I'm inclined to just delete that extra code.)

> number at the bottom of the page and not the top.

Doh!

--b.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 18 Aug 2011 18:37:26 +0000

Returning to the original question.  Do we need any updates
to the rfc3530bis (and symmetrical an errata filed for 5661)?

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> J. Bruce Fields
> Sent: Thursday, August 18, 2011 8:19 AM
> To: Trond Myklebust
> Cc: [email-masked]
> > > > > Does it contain any advice on the OPEN return value when applied
> > > > > to a file that is not a regular file, symlink or directory? I'll
> > > > > be back in active duty and able to check for myself tomorrow,
> > > > > but I thought I'd draw it to your attention now.
> > > > >
> > > >
> > > > And the answer would be no, we made no changes from 3530 there.
> > > >
> > > > You want to add text to the effect to return NFS4ERR_SYMLINK on an
> > > > OPEN of an incorrect type?
> > >
> > > Also not a spec issue unless this goes unfixed in our
> > > implementations, but note rfc 5661 requires NFS4ERR_WRONG_TYPE in
> > > the case of OPEN of a device (p. 455 in the OPEN implementation
> > > section, "If the component is neither of those but not an ordinary
> > > file, the error NFS4ERR_WRONG_TYPE is returned.")
> > >
> > I can live with those returning NFS4ERR_INVAL. Those are all cases
> > where the error applies to an invalid object which was passed in the
> > form of a filehandle. IOW: the client can and should know better.
> 
> Yeah, agreed.
> 
> fails in these cases I think any reasonable error is fine.  E.g. the linux
> server is currently going out of its way to map lower-level symlink errors
> to INVAL in the above cases, and I don't see the point, so I'm inclined to
> just delete that extra code.)
>

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 17:24:06 -0400 (EDT)

J. Bruce Fields wrote:
> > > > > to a
> > > > > active duty and able to check for myself tomorrow, but I
> > > > > thought
> > > > > I'd
> > > > for this case and if other servers are not doing this, I don't
> > > > see
> > > > how
> > > > to avoid doing an extra RPC of (Lookup, Getattr) before the
> > > > compound
> > > > with
> > > > the Open in it, if the VFS allows that. I suspect Opens of
> > > > device
> > > > special
> > > > a (Lookup, Getattr) RPC before the Open anyhow, so I'm stuck
> > > > with
> > > > the
> > > > extra RPC.)
> > >
What about the following compound:
   PutFH <dir>
   Lookup "name"
   Getattr (type, rdev, etc)
   GetFH
   Verify (type==NFREG)
   PutFH <dir>
   Open "name"

for the non-create case? (For the create case, I agree that you can't do
it all in one.)

rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Aug 2011 14:35:30 -0700

> Sent: Thursday, August 18, 2011 5:24 PM
> To: J. Bruce Fields
> Cc: Haynes, Tom; Myklebust, Trond; [email-masked]
> > > > > there returns NFS4ERR_SYMLINK for this case, I can easily
> change
> > > > > it.
> > > > >
> > > > > that can be required now? (Maybe suggested or recommended, but
> I
> > > > > think
> > > > No, that is no where near race free. I can easily substitute a
> new
> > > > file between your lookup and open.
> > > >

Still racy (and ugly as hell!)... Operations are not executed atomically within a COMPOUND, so all you have done is reduced the possibility of the race at an exhorbitant cost to the case where the file to be opened actually is a regular file.

IOW: this would be optimizing for entirely the wrong case.

Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2011 17:36:49 -0400

>    Getattr (type, rdev, etc)

You could end the compound here and do the type==NFREG check on the
client and then send the following as a separate compound, and it
wouldn't change anything.

You can never fix races with verify/nverify--they're just an
optimization.

>    Open "name"

So here you may be opening something different from what you looked up
above.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 17:41:52 -0400 (EDT)

Trond Myklebust wrote:
> > > > Alternately, the client will have to accept pretty well any
> > > > error
> > > > return that

The problem is simply that the RFC has been published for 8 years and
there are lots of servers out there now. Even if you specify a sane error
return now, 4.1 will be in use by the time the world's servers get
converted. (4.1 does have a sane error return for this.)

I noted that if, by coincidence, all the other servers out there are
currently returning NFS4ERR_SYMLINK for the case of an Open attempt on
a device special file, I could easily change the FreeBSD server to do that.
(And since its first non-experimental release is just approaching, that
 change would propagate quickly.)

However, I suspect that other server vendors (who have been silent on
this discussion sofar) have shipped servers that do not return NFS4ERR_SYMLINK for
this case and would not be able to get a patch into the field quickly?

Therefore, it seems to me that an NFSv4.0 client will have to deal with
the "not sane" error returns.

If your original post simply meant that the Open works for whatever error,
but works more efficiently when NFS4ERR_SYMLINK is replied and, therefore,
would NFSv4.0 server implementors please use that error if possible, then
I have no problem with that. (I haven't looked, but I don't think the
FreeBSD client cares which error gets returned for this case.)

> filesystem" in the case of OPEN when there are better alternatives?
Beyond the fact that NFS4ERR_SYMLINK seems weird, the only problem I
see w.r.t. this is stated above.

rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Aug 2011 14:44:26 -0700

> Sent: Thursday, August 18, 2011 5:42 PM
> To: Myklebust, Trond
> Cc: Haynes, Tom; [email-masked]
> > > > You need open by filehandle in order to make your suggestion
> work,
> > > > and
> > >    help the client recognize the file has been replaced. (Which
> puts
> > >    the
> > >    client in another weird quirky corner of the protocol, in that
> it
> > >    can
> > >    the file's data still being extant after the Remove. ie. A
> server
> > >    MAY
> > > open syscall failed because the server replied NFS4ERR_INVAL
> instead
> > > of
> > > I think it's true that the client can't just try Opens over and
> over
> > > and over
> > > If that's the case, all I think a client can do is assume the
> object
> > > will

Errmmm.... Isn't that turning the problem entirely on its head?????

How many clients are there per server, and how much more work is it going to be to change all them?

Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 17:49:06 -0400 (EDT)

Trond Myklebust wrote:
> > > > > > Subject: Re: [nfsv4] Last call for RFC3530bis-13 (ends on
> > > > > > Sept
> > > > > > 2nd)
> > > > > >
> > > > > > Thomas Haynes wrote:
> > > > > > > file that is not a regular file, symlink or directory?
> > > > > > > I'll
> > be
> > > > > > > back
> > > > > > in
> > > > > > on a device special file. If every other extant NFSv4.0
> > > > > > server
> > > > > > out
> > > > > > before the Open in a compound for the case of an Open
> > > > > > without
> > > > > > Create,
> > > > Sure, and you can also replace the file object after a
> > > > successful
> > > > Open
> > > You can't request both in the same operation, so you have to try
> > > one
> > > or
> > for the non-create case? (For the create case, I agree that you
> > can't
> > do
> > it all in one.)
> 
> Still racy (and ugly as hell!)... Operations are not executed
> atomically within a COMPOUND, so all you have done is reduced the
> possibility of the race at an exhorbitant cost to the case where the
> file to be opened actually is a regular file.
> 
I still don't see why you care if it's "racy", except for exclusive open,
which couldn't be done reliably for NFSv4.0 anyhow. (The point I was
trying to make in my original post was "yea, it's racy, and it always
will be". As you've noted, these things got fixed for 4.1, but that's
not what's on "the street" right now.)

The object of the above compound was not to be "non-racy", but to avoid
the extra (also racy, by necessity) RPC to do Lookup/Getattr.

Your point w.r.t. a specific error for this case making it easier for
the client seems correct to me. However, again, it just seems too late
for 4.0.

rick

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 17:55:49 -0400 (EDT)

> You can never fix races with verify/nverify--they're just an
Yes, agreed. The point wasn't to fix the race (I didn't think the
original discussion was related to fixing a race and never understood
why Trond kept bringing up the business about atomicity). It was just
meant to avoid the extra RPC, which was what I thought the original
discussion was about.

I thought the original discussion w.r.t. what error code was returned
was so that the client could avoid the extra Lookup/Getattr RPC for the
case where the Open was failing for some other reason than "not a regular
file".

rick
> > can't do
> > > you could fail right away instead of having to go make a
> > > race-prone
> > > test

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2011 17:57:58 -0400

> the client seems correct to me. However, again, it just seems too late

Well, all I know is that an open of a device special file that doesn't
already happen to be in the client's cache fails unless I return
NFS4ERR_SYMLINK, probably for every Linux client out there.  We can
patch the client as well, but that's a lot of clients already out
there--so I'm inclined to think it's worth documenting.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Aug 2011 18:05:58 -0400 (EDT)

> there--so I'm inclined to think it's worth documenting.
Ok, documenting the above I can buy into. (I'm not sure that
it's something that should go in the updated RFC, but I'll let the
editors decide that.)

It also implies that I should try and get a FreeBSD server patch for
this in the pipe, once I test it against FreeBSD and Solaris10 clients.

Is there any client implementor out there that thinks having a server
return NFS4ERR_SYMLINK for an Open attempt on a device special file will
break their client?

rick

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 19 Aug 2011 05:23:40 +0000

Bruce,

If the clients are "can not patch" and the some of the
servers are going to be changed to accommodate, then this 
seems like a "MUST return NFS4ERR_SYMLINK"
statement in the RFCs so that everyone else can know what to do.

Ah, the power of the installed base.

If anyone disagrees with a statement like the above being added
to the RFCs, they should speak up so we can decide what an 
alternate path might be.

Spencer

> Sent: Thursday, August 18, 2011 2:58 PM
> To: Rick Macklem
> Cc: Tom Haynes; Trond Myklebust; [email-masked]
> > I still don't see why you care if it's "racy", except for exclusive
> > open, which couldn't be done reliably for NFSv4.0 anyhow. (The point I
> > was trying to make in my original post was "yea, it's racy, and it
> > always will be". As you've noted, these things got fixed for 4.1, but
> > that's not what's on "the street" right now.)
> >
> > The object of the above compound was not to be "non-racy", but to
> > avoid the extra (also racy, by necessity) RPC to do Lookup/Getattr.
> >
> NFS4ERR_SYMLINK, probably for every Linux client out there.  We can patch
> the client as well, but that's a lot of clients already out there--so I'm
> inclined to think it's worth documenting.
> 
> --b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 19 Aug 2011 11:10:08 -0400

> meant to avoid the extra RPC, which was what I thought the original

OK, fair enough.

> case where the Open was failing for some other reason than "not a regular

Yes, there are a number of ways to fix this on the client side as well.

As a practical matter the servers are probably easier to update, unless
there's another client out there that the new error return will break.

It would be useful to hear something about other client and server
implementations.

--b.