Subject: RE: [nfsv4] RE: Finding hardlinks

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 5 Jan 2007 12:28:15 -0500

For now, I'm not going to address the controversial issues here,
mainly because I haven't decided how I feel about them yet.

     Whether allowing multiple filehandles per object is a good
     or even reasonably acceptable idea.

     What the fact that RFC3530 talks about implies about what
     clients should do about the issue.

One thing that I hope is not controversial is that the v4.1 spec
should either get rid of this or make it clear and implementable.
I expect plenty of controversy about which of those to choose, but
hope that there isn't any about the proposition that we have to 
choose one of those two.

> SECINFO information is, for instance, given
> out on a per-filehandle basis, does that mean that the server will
have
> different security policies? 

Well yes, RFC3530 does say "The new SECINFO operation will allow the 
client to determine, on a per filehandle basis", but I think that
just has to be considered as an error rather than indicating that if
you have two different filehandles for the same object, they can have 
different security policies.  SECINFO in RFC3530 takes a directory fh
and a name, so if there are multiple filehandles for the object with
that name, there is no way for SECINFO to associate different policies
with different filehandles.  All it has is the name to go by.  I think
this should be corrected to "on a per-object basis" in the new spec no 
matter what we do on other issues.

I think the principle here has to be that if we do allow multiple 
fh's to map to the same object, we require that they designate the 
same object, and thus it is not allowed for the server to act as if 
you have multiple different object with different characteristics.

Similarly as to:

> In some places, people haven't even started
> to think about the consequences: 
>
>     If GETATTR directed to the two filehandles does not return the
>     fileid attribute for both of the handles, then it cannot be
>     determined whether the two objects are the same.  Therefore,
>     operations which depend on that knowledge (e.g., client side data
>     caching) cannot be done reliably.

I think they (and maybe "they" includes me, I haven't checked the
history
here) started to think about them, but went in a bad direction.

The implication here that you can have a different set of attributes
supported for the same object based on which filehandle is used to 
access the attributes is totally bogus.

The definition of supp_attr says "The bit vector which would retrieve
all mandatory and recommended attributes that are supported for this 
object.  The scope of this attribute applies to all objects with a
matching fsid."  So having the same object have different attributes
supported based on the filehandle used or even two objects in the same
fs having different attributes supported, in particular having fileid
supported for one and not the other just isn't valid.

> The fact is that RFC3530 contains masses of rope with which
> to allow server and client vendors to hang themselves. 

If that means simply making poor choices, then OK.  But if there are 
other cases where you feel that the specification of a feature is simply

incoherent and the consequences not really thought out, then I think 
we need to discuss them and not propagate that state of affairs to v4.1.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, January 05, 2007 5:29 AM
To: Benny Halevy
Cc: Jan Harkes; Miklos Szeredi; [email-masked];
[email-masked]; Mikulas Patocka;
[email-masked]; Jeff Layton; Arjan van de Ven
Subject: Re: [nfsv4] RE: Finding hardlinks

On Fri, 2007-01-05 at 10:28 +0200, Benny Halevy wrote:
> Trond Myklebust wrote:
> > Exactly where do you see us violating the close-to-open cache
> > consistency guarantees?
> > 
> 
> I haven't seen that. What I did see is cache inconsistency when
opening
> the same file with different file descriptors when the filehandle
changes.
> My testing shows that at least fsync and close fail with EIO when the
filehandle
> changed while there was dirty data in the cache and that's good.
Still,
> not sharing the cache while the file is opened (even on a different
file
> descriptors by the same process) seems impractical.

Tough. I'm not going to commit to adding support for multiple
filehandles. The fact is that RFC3530 contains masses of rope with which
to allow server and client vendors to hang themselves. The fact that the
protocol claims support for servers that use multiple filehandles per
inode does not mean it is necessarily a good idea. It adds unnecessary
code complexity, it screws with server scalability (extra GETATTR calls
just in order to probe existing filehandles), and it is insufficiently
well documented in the RFC: SECINFO information is, for instance, given
out on a per-filehandle basis, does that mean that the server will have
different security policies? In some places, people haven't even started
to think about the consequences:

      If GETATTR directed to the two filehandles does not return the
      fileid attribute for both of the handles, then it cannot be
      determined whether the two objects are the same.  Therefore,
      operations which depend on that knowledge (e.g., client side data
      caching) cannot be done reliably.

This implies the combination is legal, but offers no indication as to
how you would match OPEN/CLOSE requests via different paths. AFAICS you
would have to do non-cached I/O with no share modes (i.e. NFSv3-style
"special" stateids). There is no way in hell we will ever support
non-cached I/O in NFS other than the special case of O_DIRECT.

...and no, I'm certainly not interested in "fixing" the RFC on this
point in any way other than getting this crap dropped from the spec. I
see no use for it at all.

Trond

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Fri, 5 Jan 2007 09:58:15 -0800 (PST)

> should either get rid of this or make it clear and implementable.

Getting rid of multiple filehandles per object is going to be
difficult for any NFS server that allows multiple export policies
for the same filesystem.

> hope that there isn't any about the proposition that we have to 

RFC3530 gives clients a way to optimize caching if the client cares.
Trond is saying he doesn't care. If none of the other client's care,
then deprecation is worth considering.

> you have two different filehandles for the same object, they can have

Again, making SECINFO return a single security policy for file on
a filesystem that has multiple export policies is difficult.

> you have multiple different object with different characteristics.

Easy to say, hard to do.

Now if we want to say that multiple export policies per
fsid is now banned in NFSv4.1, that's something worth
debating. If we agree to do that, I will start of betting
pool as to how long it will take for the first server implementor
to violate the spec.

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 5 Jan 2007 12:01:17 -0600

> hope that there isn't any about the proposition that we have to 

If client implementors believe that requiring a hash table by {fsid,
fileid} in order to deal with many-FHs-per-file (as opposed to different
FHs across server reboots), then that should be considered.

It's possible that we could add a new FH-like item or items that serve
the purpose of letting the client cache information that the server
needs when the client uses that FH.  This would very much require a new
minor version, but it has its appeal.  E.g., the server could compute
and encrypt the access rights for the client principal that obtained the
FH so that it need not re-compute that later unless the file's
ACL/owner/... changes -- i.e., an access token/cookie.  I imagine this
as new GET/PUT operations (GETTOKEN/PUTTOKEN).

Nico
--

---

From: Trond Myklebust <[email-masked]>
Date: Sat, 06 Jan 2007 02:15:14 +0100

> a filesystem that has multiple export policies is difficult.

Asking the client to track that security information via multiple
filehandles is worse. How am I for instance supposed to figure out which
security policy to use when several users write to the same file, having
opened the file using different paths via shared mmap()?

> pool as to how long it will take for the first server implementor

Your servers are already violating said spec given that they want to
return the same fsid+fileid for files that are not hardlinks (i.e.
snapshots). That violation is one of the gripes I listed with the
algorithm outlined in section 9.3.4 of RFC3530.

Trond