Subject: [nfsv4] Request for review of NFSv4 DNS SRV draft

From: James Lentini <[email-masked]>
Date: Wed, 5 Aug 2009 10:49:48 -0400 (EDT)

Hello,

At the NFSv4 WG meeting last week, Lars directed us to obtain a review 
of

 http://www.ietf.org/id/draft-ietf-nfsv4-federated-fs-dns-srv-namespace-01.txt

by the DNS Directorate.

This draft along with the NFSv4 WG drafts "Requirements for Federated 
File Systems", "NSDB Protocol for Federated Filesystems", and 
"Administration Protocol for Federated Filesystems" define FedFS, a 
system for creating a federated namespaces across NFSv4 servers.

A WG last call on these drafts is scheduled for October.

We look forward to your feedback on our use of DNS in 
draft-ietf-nfsv4-federated-fs-dns-srv-namespace-01.

-james

--
James Lentini | NetApp | 781-768-5359 | [email-masked]

---

From: Thomas Narten <[email-masked]>
Date: Tue, 18 Aug 2009 15:54:27 -0400

Here are my review comments on this document. I am also not subscribed
to the nfsv4 list, so please cc me in responses.

Note: I'm reviewing this mostly from a DNS perspective and what little
I know about NFS goes pretty far back. So some of my
questions/concerns may stem from a lack of understanding of NFSv4.

>    [NB0510]   Noveck, D. and R. Burnett, "Next Steps for NFSv4
>               Migration/Replication", October 2005, <ftp://www.ietf.org/
>               internet-drafts/draft-noveck-nfsv4-migrep-00.txt>.

This reference seems pretty important to this document. Yet the
document is 4 years old and expired. What is the status of it? Does it
make sense to advance this document without the reference? Or has it
been replaced by a newer document?

>    3.1.  Deployment of the Resource Record
> 
>    As with any DNS resource, any server installation needs to concern
>    itself with the likely loads and effects of the presence of the
>    resource record.  The answers to requests for RRs might differ
>    depending on what the server can tell about the client.  For example,
>    some RRs might be returned only to those clients inside some network
>    perimeter (to provide an intranet service) and requests from other
>    clients might be denied.  As the RR directs the clients to ask for
>    service from a given set of servers, the administrator should ensure
>    that the identified servers can handle the expected load.
>    Fortunately, the definition of the DNS SRV resource record offers a
>    mechanism to distribute the load to multiple servers within a
>    priority ordering.

The above makes me a little nervous, but maybe I'm missreading the
intention. From a DNS point of view, it is generally frowned upon to
return differing answers to a DNS query depending on criteria (like
the above). (I know there are DNS load balancers that do that, but
they only work if one disables DNS caching, etc.).

Besides, the above doesn't really buy you much. Isn't the better place
to perform access control on the server itself when a client attempts
to contact it?

Is that what you really mean to happen above?

> 4.  Integration with Use of NFS Version 4
> 
>    There are at least two remaining questions: whether this DNS SRV
>    record evaluation is done in the NFS server or client, and also how
>    the domain names of the organizations are passed to client or server.
>    A third question is how this might produce a uniform global file name
>    space, and what prefix should be used for such file names.

I don't get the first question. SRV processing is always done by the
client (the requestor). The NFS server does nothing... The DNS server
does respond to the DNS	queries, but the NFS server is not involved in
that....

Also, what does the last sentence (referring to "what prefix should be
used") mean?

>    This specification anticipates that these SRV records will most
>    commonly be used to define the second directory level in an inter-
>    organizational file name space.  This directory will be populated
>    with domain names pointing to the file systems published for use
>    under those domain names.  Thus, the root directory for the file
>    system published by example.net will effectively be mounted
>    underneath the example.net name in a second-level directory.

I don't really understand this part at all. The SRV resolution will
convert an SRV into a machine name (and eventually an address) and
port number. That is what the client contacts. But you are talking
about "mounting". What exactly does the client pass to the server in
the mount request? Doesn't that need to be specified here? 

I also don't understand what "second-level directory" means or why
anything needs to be said. (This probably just reflects my ignorance
about NFS mounts.) What does it mean to populate a directory with DNS
names? 

>    Thus, the root directory for the file system published by
>    example.net will effectively be mounted underneath the
>    example.net name in a second-level directory.

Going back to your example, the SRVs point to nfs2ex.example.net. So
it doesn't seem right to say (above) "will effectively be mounted
underneath the example.net name..." I don't understand what you are
trying to say here.

> 4.1.  Globally-useful names: conventional mount point
> 
>    For the inter-organizational name space to be a global name space, it
>    is useful for its mount point in local systems to be uniform as well.
>    The name /nfs4/ SHOULD be used so that names on one machine will be
>    directly usable on any machine.  Thus, the example.net published file
>    system would be accessible as
> 
>            /nfs4/example.net/
> 
>    on any client.  Using this convention, "/nfs4/" is a mount for a
>    special file system that is populated with the results of SRV record
>    lookups.

I don't understand the above at all. Are you getting into NFS
implementation details that are uniform and well known? Does NFS have
a well-known name space where /nfs4/foo.com/ would have the second
component recognized to be a a DNS name? Where is this specified?

>    One extension of this rule that we might choose to inherit from AFS,
>    though, is to give a special meaning to the domain name of an
>    organization preceded by a period (".").  It might be reasonable to
>    have names mounting the filesystem for a period-prefixed domain name
>    (e.g., ".example.net") attempt to mount only a read-write instance of
>    that organization's root filesystem, rather than permitting the use
>    of read-only instances of that filesystem.  Thus,

Seems like a bad idea.. You are using non compliant DNS names...

> 4.3.  File system integration issues
> 
>    The result of the DNS search SHOULD appear as a (pseudo-)directory in
>    the client name space, cached for a time no longer than the RR's TTL.
>    A further refinement is advisable, and SHOULD be deployed: that only
>    fully-qualified domain names appear as directories.  That is, in many

The above seems at odds with the documents earlier support for using
short TTLs in order to return client-specific results. And, when the
TTL expires, what is supposed to happen?

>    environments, DNS names may be abbreviated from their fully-qualified
>    form.  In such circumstances, multiple names might be given to file
>    system code that all resolve to the same DNS SRV RRs.  The
>    abbreviated form SHOULD be represented in the client's name space
>    cache as a symbolic link, pointing to the fully-qualified name, case-
>    canonicalized when appropriate.

This seem like it would be hard for clients to implement, since the
DNS algorithm (which may append various names as part of trying to
resolve the query) are not done by the client directly, but as part of
more generic DNS routines. Is this really feasible to implement?

And how is the above problem worse with SRV records than it is today
when the DNS name of the server is given?

Is the  "name space cache" something very specific to NFS?

> This will allow pathnames obtained
>    with, say, getcwd() to include the DNS name that is most likely to be
>    usable outside the scope of any particular DNS abbreviation
>    convention.

> 5.  Where is this integration carried out?
> 
>    Another consideration is what agent should be responsible for
>    interpreting the SRV records.  It could be done just as well by the
>    client or by the server, though we expect that most clients will
>    include this function themselves.  Using something like Automounter

I am very worried that this documen seems to have some pretty basic
misconceptions about DNS/SRVs.  NFS servers DO NOT interpret SRV
records... only clients do that...

Thomas

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Aug 2009 16:03:25 -0500

> been replaced by a newer document?

It's informational.  The referral mechanism in NFSv4 was originally
conceived as a way to indicate to clients that a filesystem migrated to
a different location, and later it was recognized that this mechanism
could be used to form a "global namespace", quite independently of
migration and replication.

> >    clients might be denied.  [...]
> 

Well, people do tend to keep fake DNS roots inside their perimeters,
with different zone file contents inside and outside.  I agree with you
that this I-D should really not mention that.  For load balancing the
SRV RR weight and priority RDATA fields should suffice.

> to perform access control on the server itself when a client attempts

I didn't read that text as being about access control at all, but about
separating "inside" and "outside" namespaces, infrastructures.  But I
agree that it seems completely out of place.

> does respond to the DNS	queries, but the NFS server is not involved in

I think the idea is that servers could dynamically generate NFSv4
referrals on the basis of these SRV RRs.  Consider this example:

 - client A mounts B:/foo on /bar
 - B:/foo/ is a "magic" directory in that all names looked up in it are
   treated as domainnames, their corresponding SRV RRs are looked up,
   and, if found, referrals are instantiated with those domainnames as
   their name.

   So when the client looks for /bar/baz.example/a/b/c the
   _nfs4._tcp.baz.example SRV RR is looked up by the _server_.

Clients could also implement a similar magic directory, in which case
it's the _clients_ that do the SRV RR lookup.

(If you've ever used a /net/ automounter path on a Unix-like system then
you know what I mean by "magic".  In the /net/ case the automounter
interprets all names looked up in it as hostnames, and tries to mount
<hostname>:/ on /net/<hostname>.  The same thing would happen here, only
with names interpreted as domain names and resolved to server names via
the _nfs4 SRV RR.  In the Windows world the closest equivalent to /net/
would be UNCs.)

> Also, what does the last sentence (referring to "what prefix should be

I believe it refers to the /net/ automounter path.  On Unix-like systems
it is quite common that /net/ is as described above.  Now, where should
the new magic directory get mounted?  On /domain/?  /dom/??  It's a
local matter.  Therefore this I-D has little business specifying it, but
since we're going to want to converge on a common convention, it does
make sense to make a recommendation (in this case /nfs4/).

I'm not sure that I really like /nfs4/ for this -- as a user I really
could care less what protocol is being used to get to some file...  But
I don't really have a better recommendation either.

> the mount request? Doesn't that need to be specified here? 

There's no "mount request".

> about NFS mounts.) What does it mean to populate a directory with DNS

I *think* they mean that /nfs4/example.net/foo/ will really map to "the
filesystem at foo.example.net".  But frankly, I find that text confusing
too.

> underneath the example.net name..." I don't understand what you are

I think they mean to say that a server for example.net named, say, foo,
will be accessible as /nfs4/example.net/foo -- /nfs4/example.net/ being
a magic /net/-like filesystem containing only referrals.

The key issue here though is this: say you lookup the _nfs4._tcp SRV RR
for example.net and find a list of servers, now, what path from any one
of those fileservers should be mounted on /nfs4/example.net??

In the /net/ case the answer to that question is always "/".  I.e.,
/net/foo.example.net/ should be a mount of foo.example.net:/

Section 4.2 is explicit about this: use "the pseudo-root" (meaning "the
root FH").  I don't think that's a great answer for this new magic
mount, but the SRV RR has no way to tell us what that path should be, so
we must pick one.

I see the quoted text as getting to that issue in a round-about way, but
without actually giving us an answer.  I really don't see how the
servers for example.net can have two roots: one for clients arriving via
an SRV RR lookup, and one for clients arriving as usual.

> component recognized to be a a DNS name? Where is this specified?

See above.

> Seems like a bad idea.. You are using non compliant DNS names...

I agree.  I don't understand the paragraph immediately preceding the one
you quote.

I think the issue here is that a referral can point to more than one
server, so what to do if one is writeable and the others are not?  But
it gets worse than that: the referral data in the NFSv4 protocol doesn't
tell you which is writeable, so to find out the client would have to go
test each one.

But it gets worse: in NFSv4.0 (RFC3530) there's no way to find out if a
filesystem is read-only without actually attempting a write!

So, IMO the paragraph immediately preceding the one you quote is wrong.
The one you quote is inadvisable, IMO.

What should happen is that the client should assume the mount is
read-write *if* there's a single location, else the client should assume
read-write *if* the client is willing to retry writes against each
location until one succeeds, else the client should assume read-only.

(The I-D could provide a lot more background, including an *informative*
copy of the fs_locations XDR, so that readers who are not familiar with
NFSv4 can recognize some of these issues more easily.)

> TTL expires, what is supposed to happen?

The path should be unmounted.  If the path is busy and can't easily be
unmounted then the client should be free to leave it as is and say "oh
well".  (This happens today in the case of /net/.)

> more generic DNS routines. Is this really feasible to implement?

IMO this is really referring to the resolver "search" features.  I see
no reason why it should be difficult to use a resolver search in this
context.

> when the DNS name of the server is given?

IMO it's not *unless* the client can't cope with mounting the same
resource in multiple places on the client-side namespace.  But I don't
think that's an issue nowadays.

> Is the  "name space cache" something very specific to NFS?

No, it's a local issue (though I don't know where "cache" comes from here).

Gathering recommendations for _local_ behavior should be in a separate
section would be a good idea.  And they should be as OS-agnostic as
possible, though even then, POSIX-specific recommendations are a good
idea.  (And what about Windows?  I dunno.  Something like network
neighborhood needs to be added that knows about this scheme.  But we can
hardly tell MSFT what to do, whereas in the POSIX case it's much more
likely that vendors will follow these recommendations.)

> records... only clients do that...

See above.  I can certainly see both, clients and servers doing it.  I
think it's best for clients to do it exclusively, but consider older
clients: they won't know how, therefore it'd be useful to have servers
that export a "magic" (automounter) directory where the servers do what
the client should be doing.

IMO the I-D needs wordsmithing.  The DNS-based _mechanism_ described
is fine, though it has shortcomings:

 - no way to specify a path
 - no way to specify read-only vs. read-write (but this is really a
   local issue)
 - no way to specify other local options that would nonetheless be very
   convenient to be able to specify centrally

These shortcomings can be dealt with easily enough.  Path? -> /.
Read-write? -> guess read-write and if you fail, return a suitable error
to the app.  Other local options? -> use suitable defaults.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 18 Aug 2009 17:46:08 -0400

Thanks for the review, Thomas.  My responses are interspersed.

> -----Original Message-----
> From: Thomas Narten [mailto:[email-masked]] 
> Sent: Tuesday, August 18, 2009 3:54 PM
> To: Lentini, James
> Cc: [email-masked]; Everhart, Craig; [email-masked]; Lars Eggert
> Subject: Re: [dns-dir] Request for review of NFSv4 DNS SRV draft
> 
> Here are my review comments on this document. I am also not 
> subscribed to the nfsv4 list, so please cc me in responses.
> 
> Note: I'm reviewing this mostly from a DNS perspective and 
> what little I know about NFS goes pretty far back. So some of 
> my questions/concerns may stem from a lack of understanding of NFSv4.
> 
> This reference seems pretty important to this document. Yet 
> the document is 4 years old and expired. What is the status 
> of it? Does it make sense to advance this document without 
> the reference? Or has it been replaced by a newer document?

The NFS4.1 draft, draft-ietf-nfsv4-minorversion1-29, describes
replacement functionality.  It is in the RFC Ed queue.
 
> The above makes me a little nervous, but maybe I'm 
> missreading the intention. From a DNS point of view, it is 
> generally frowned upon to return differing answers to a DNS 
> query depending on criteria (like the above). (I know there 
> are DNS load balancers that do that, but they only work if 
> one disables DNS caching, etc.).
> 
> Besides, the above doesn't really buy you much. Isn't the 
> better place to perform access control on the server itself 
> when a client attempts to contact it?
> 

I'm happy to take the operational advice about DNS.  Probably the better
answer is to defer to the definition of the SRV record as a mechanism
for load balancing.

I totally expect that the file servers will perform their own access
control; that isn't really what I was concerned about, though.

> I don't get the first question. SRV processing is always done 
> by the client (the requestor). The NFS server does nothing... 
> The DNS server

Actually, I'm arguing that the NFS server could, architecturally, be the
DNS client here.  The functionality being described could be implemented
by an NFS server.  The semantics of the SRV record could be captured in
an NFS4/NFS4.1 referral.

Even if most of the NFSv4 list currently thinks that the right
implementation to use is the client-based one, I feel that it's sensible
to describe the NFS-server-based one.  Given that it makes semantic
sense, is this not appropriate?
 
> Also, what does the last sentence (referring to "what prefix should be

The question is what's described below: basically, whether one wants to
have a convention for what amounts to the mount point for the directory
whose contents are given by all the SRV records.
 
> I don't really understand this part at all. The SRV 
> resolution will convert an SRV into a machine name (and 
> eventually an address) and port number. That is what the 
> client contacts. But you are talking about "mounting". What 
> exactly does the client pass to the server in the mount 
> request? Doesn't that need to be specified here? 

Yes, and every NFSv4 server exports a root file system, which is what I
expect the clients would mount.
 
> I also don't understand what "second-level directory" means 
> or why anything needs to be said. (This probably just 
> reflects my ignorance about NFS mounts.) What does it mean to 
> populate a directory with DNS names? 

I guess I'm not being clear.  Let's talk about the client-based
implementation of this, as opposed to the server-based one.

I'm proposing a scheme where, on the NFSv4 client, there's a special
directory under the client's root directory.  Suppose that directory is
called "/nfs4".  Then, under this special directory, whenever some
application does a lookup for a domain name, like "/nfs4/mit.edu", the
NFSv4 client asks DNS for the corresponding SRV record for that
domain--in this case, _nfsv4.mit.edu.  This DNS lookup produces a list
of SRV RRs, and I'm trying to describe the process by which those SRV
RRs can be interpreted as mounts of file systems on NFSv4 servers.  So
our NFS client carries out the indicated server mount, using
"/nfs4/mit.edu" as the place in the client's name space where that
server (those servers) are mounted.  If subsequent applications do
lookups for other domain names under /nfs4/, that local directory (the
/nfs4/ one) will appear to be populated with the domain names that have
been looked up.

> Going back to your example, the SRVs point to 
> nfs2ex.example.net. So it doesn't seem right to say (above) 
> "will effectively be mounted underneath the example.net 
> name..." I don't understand what you are trying to say here.

Right.  The NFS client does an auto-mount of the given file server(s) in
the machine's local name space, as /nfs4/example.net.

> implementation details that are uniform and well known? Does 
> NFS have a well-known name space where /nfs4/foo.com/ would 
> have the second component recognized to be a a DNS name? 
> Where is this specified?

This draft is proposing that /nfs4/ be used conventionally, and it
proposes that there be a well-known name space such as you describe.
All that the above paragraph is trying to describe is the name of that
conventional mount point, proposing the "nfs4" name for it.

> Seems like a bad idea.. You are using non compliant DNS names...

It's true that ".example.net" is not a legal DNS name.  My expectation,
perhaps not well-spelled-out, is that the agent interpreting
".example.net" (the NFS client, commonly) would strip off the leading
"." and proceed as before to obtain the SRV records from DNS for the
"example.net" name.  Subsequent processing of the servers so indicated
might show that some servers were read-only, and if so, they should not
be mounted by such a request.
 
> The above seems at odds with the documents earlier support 
> for using short TTLs in order to return client-specific 
> results. And, when the TTL expires, what is supposed to happen?

This is an inconsistency.  I should not have suggested any short TTL
values earlier.
 
> This seem like it would be hard for clients to implement, 
> since the DNS algorithm (which may append various names as 
> part of trying to resolve the query) are not done by the 
> client directly, but as part of more generic DNS routines. Is 
> this really feasible to implement?

I remember doing this some years ago.  In the layer I had to deal with,
it was totally possible, if not obvious.  If that's no longer true,
perhaps the SHOULD will put its featherweight on the scale of making it
possible again.
 
> And how is the above problem worse with SRV records than it 
> is today when the DNS name of the server is given?
> 

I was thinking of a "name space cache" specifically in the NFS client,
so that the content fetched from DNS would be cached.  The alternative
seems to be that the interpretation of every pathname would require a
DNS lookup/reply.  Even if this is a local operation, in most
implementations it would require a trip from kernel to userspace and
back, which is a substantial penalty.  This speaks to how the DNS result
might be represented in-kernel.

> I am very worried that this documen seems to have some pretty 
> basic misconceptions about DNS/SRVs.  NFS servers DO NOT 
> interpret SRV records... only clients do that...

As I say, one obvious use for this convention is that an NFS client
could consult the DNS client.  Architecturally, though, it would be
perfectly possible for an NFS server to use this convention.

The convention I'm advocating is that admins wishing to publish file
servers for a domain name should be able to do so by publishing suitable
SRV records in that domain.  Those SRV records could then be interpreted
by either the NFS-client-based scheme or by the NFS-server-based one, in
principle.
 
> Thomas

		Craig

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Aug 2009 16:52:53 -0500

>  http://www.ietf.org/id/draft-ietf-nfsv4-federated-fs-dns-srv-namespace-01.txt

My advice:

 - remove all references to internal and external DNS; these are not
   necessary

 - collect all recommendations for purely local behavior into a separate
   section/subsection:

   X.  Local Behavior Recommendations

   X.1.  Local Behavior Recommendations for POSIX

   This includes:

    - the name of the mount point on clients for a /net/-like automount

    - local options like "ro" (read-only), "intr" (interruptible), ...

 - obtain consensus on where on the client to mount a domain-based
   "automount map"

   /nfs4 seems odd.  Why not /nfs4dom, or some such.  Why should one
   think that /nfs4 is not an NFSv4-specific variant of /net?

   Why do users need an NFSv4-specific mountpoint here?  Shouldn't there
   also be a protocol-agnostic mountpoint?

 - obtain consensus on whether clients should mount the root FH for each
   root server, or some other path.

   I strongly recommend something like /.nfs4-globalnamespace-root
   (POSIX allows for special . files/dirs in the root of any
   filesystem), or some such.

 - work out a suitable read-only/read-write option setting

   Possible schemes:

   a) Assume read-only for all namespace roots.  (They're very likely to
      be read-onlyanyways.)

   b) Assume read-write and pass on any NFS4ERR_ROFS as EROFS.

   c) Heuristic:

    - if there's only one root location -> assume read-write; if the
      server returns NFS4ERR_ROFS then mark the mount as read-only

    - if there's more than one root location and if the client is
      willing to re-do LOOKUPs/OPENs (as during recovery) on
      NFS4ERR_ROFS -> assume read-write; if all servers return
      NFS4ERR_ROFS then mark the mount as read-only

    - if there's more than one root location and the client is not
      willing to do as above -> mark the mount as read-only

 - recommend that clients do the SRV lookups, but allow (MAY) servers to
   export filesystems where names are interpreted as domainnames and
   which result in referrals to those domainnames' namespace root
   locations.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Wed, 19 Aug 2009 10:01:39 -0400

> From: Nicolas Williams [mailto:[email-masked]] 
> Sent: Tuesday, August 18, 2009 5:03 PM
> To: Thomas Narten
> 
...
> Well, people do tend to keep fake DNS roots inside their 
> perimeters, with different zone file contents inside and 
> outside.  I agree with you that this I-D should really not 
> mention that.  For load balancing the SRV RR weight and 
> priority RDATA fields should suffice.

Agreed.
 
...
> I *think* they mean that /nfs4/example.net/foo/ will really 
> map to "the filesystem at foo.example.net".  But frankly, I 
> find that text confusing too.

OK, it must really be confusing.  "[S]econd-level directory" is just
meant as the suggested /nfs4/ directory name on the NFS client, where
the "magic" automount-like mounts would appear.

It doesn't mean what Nico interprets it as.  The only thing that appears
*under* the /nfs4/example.net/ directory, or mount, or name, is a
reflection of what's on a file server.  So the only way
/nfs4/example.net/foo/ appears is if there's a "/foo" exported from the
file server(s) pointed to by the SRV records for "example.net".

> nfs2ex.example.net. So 
> I think they mean to say that a server for example.net named, 
> say, foo, will be accessible as /nfs4/example.net/foo -- 
> /nfs4/example.net/ being a magic /net/-like filesystem 
> containing only referrals.
> 
> The key issue here though is this: say you lookup the 
> _nfs4._tcp SRV RR for example.net and find a list of servers, 
> now, what path from any one of those fileservers should be 
> mounted on /nfs4/example.net??
> 
> In the /net/ case the answer to that question is always "/".  
> I.e., /net/foo.example.net/ should be a mount of foo.example.net:/
> 
> Section 4.2 is explicit about this: use "the pseudo-root" 
> (meaning "the root FH").  I don't think that's a great answer 
> for this new magic mount, but the SRV RR has no way to tell 
> us what that path should be, so we must pick one.
> 
> I see the quoted text as getting to that issue in a 
> round-about way, but without actually giving us an answer.  I 
> really don't see how the servers for example.net can have two 
> roots: one for clients arriving via an SRV RR lookup, and one 
> for clients arriving as usual.

I agree.  If you go to any of the servers (named in SRV) for example.net
and mount their "/", you should get the same thing as if you had asked
your NFS client to interpret the pathname /nfs4/example.net/.
 
> I agree.  I don't understand the paragraph immediately 
> preceding the one you quote.
> 
> I think the issue here is that a referral can point to more 
> than one server, so what to do if one is writeable and the 
> others are not?  But it gets worse than that: the referral 
> data in the NFSv4 protocol doesn't tell you which is 
> writeable, so to find out the client would have to go test each one.
> 
> But it gets worse: in NFSv4.0 (RFC3530) there's no way to 
> find out if a filesystem is read-only without actually 
> attempting a write!
> 

This isn't a central feature of the draft, but a bell/whistle.  It could
go without leaving a mark.  But it's a handy kind of thing for actually
*using* a system where some filesystem instances are readonly and some
are read-write.
 
> What should happen is that the client should assume the mount 
> is read-write *if* there's a single location, else the client 
> should assume read-write *if* the client is willing to retry 
> writes against each location until one succeeds, else the 
> client should assume read-only.

I buy the first half.  I don't buy the second half.  What if some of the
R/O instances are out-of-date with respect to the R/W, even if they will
eventually converge?  Then you have updates arriving at a file server
where the update is based on a potentially stale copy of the data.
Think "echo 'foo' >> /etc/passwd".
 
> (The I-D could provide a lot more background, including an 
> *informative* copy of the fs_locations XDR, so that readers 
> who are not familiar with

(Sure, though I don't want to break with IETF tradition here.)
 
> > >    fully-qualified domain names appear as directories.  
> That is, in 
> > > many
> > 
> for using 
> The path should be unmounted.  If the path is busy and can't 
> easily be unmounted then the client should be free to leave 
> it as is and say "oh well".  (This happens today in the case 
> of /net/.)

OK, this sort of thing needs more clarity: noted.  Certainly the path
could be un-pseudo-mounted if it's idle.  If it's busy, probably the
(NFS) client is best off leaving it there for a while.  But this section
was insufficiently qualified; it's supposed to apply to idle
pseudo-mounts.
 
> > >    canonicalized when appropriate.
...
> No, it's a local issue (though I don't know where "cache" 

I'm just talking about the collection of pseudo-mounts under /nfs4/,
which works like a cache of the SRV records.
 
> Gathering recommendations for _local_ behavior should be in a 
> separate section would be a good idea.  And they should be as 
> OS-agnostic as possible, though even then, POSIX-specific 
> recommendations are a good idea.  (And what about Windows?  I 
> dunno.  Something like network neighborhood needs to be added 
> that knows about this scheme.  But we can hardly tell MSFT 
> what to do, whereas in the POSIX case it's much more likely 
> that vendors will follow these recommendations.)

Not a bad idea.  Certainly the location for the nest of pseudo-mounts is
local.  I'm not so sure about the suggested behavior of an NFS client
with respect to, say, the TTL of the SRV RRs; that doesn't feel entirely
local.
 
...

		Craig

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 20 Aug 2009 12:06:52 -0500

> file server(s) pointed to by the SRV records for "example.net".

OK, but see below.

> > [...]

I agree that that's what the I-D says :) but am not sure that's what it
should say.

The namespace that someone deploying this should want to appear in
/nfs4/foo.example/ is most likely very different than what one should
want to appear in /net/randomserver.foo.example/.  That means that one
could not turn randomserver.foo.example into a root server for
foo.example.

I.e., one would need to deploy new servers (possibly using
virtualization) in order to add domain root servers to a domain.

That's very annoying.  But suppose you were to either define a new
special FH, like the ROOT FH, or a special directory to be found in the
server's pseudo-root directory, and have the clients mount _that_ on
/nfs4/foo.example/.  That would avoid the above problem.

But I realize that either of those two solutions (new special FH or
reserved special directory name in the ROOT FH directory) are
problematic by themselves as well.  So I won't push them.

> > [...]

But you can't even know which ones are read-write and which are
read-only until you try to write.

I strongly recommend that you remove this feature.

> Think "echo 'foo' >> /etc/passwd".

I see the following options:

a) more than one location -> assume read-only.

b) more than one location -> assume all are writable or none are; try to
			     write but don't do fallback on EROFS.

c) more than one location -> on EROFS fallback on other locations,
			     searching for a writable one, and if one is
			     found, then use that location exclusively
			     (fallback requires re-doing LOOKUPs, OPENs,
			     as replicas are not required to understand
			     each other's FHs, IIRC).

(c) copes with the problem you mention.  If a writable copy fails then
fallback on any other replica may mean that the filesystem appears to go
backwards in time if the replica is not up to date with the writable
copy at that point.  That problem can be avoided by never falling back
on another replica once a writable replica is found (a very unsatisfying
solution, IMO).

> with respect to, say, the TTL of the SRV RRs; that doesn't feel entirely

Behavior w.r.t. TTL is definitely generic.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Fri, 21 Aug 2009 11:27:07 -0400

> Sent: Thursday, August 20, 2009 1:07 PM
> 
...
> _nfs4._tcp SRV 
> > > RR for example.net and find a list of servers, now, what 
> path from 
> > > any one of those fileservers should be mounted on 
> > > /nfs4/example.net??
> > > 
> > > [...]
> > > (meaning "the root FH").  I don't think that's a great answer for 
> > > this new magic mount, but the SRV RR has no way to tell 
> us what that 
> > I agree.  If you go to any of the servers (named in SRV) for 
> > example.net and mount their "/", you should get the same 
> thing as if 
> > you had asked your NFS client to interpret the pathname 
> /nfs4/example.net/.
> 
> I agree that that's what the I-D says :) but am not sure 
> that's what it should say.
> 
> The namespace that someone deploying this should want to 
> appear in /nfs4/foo.example/ is most likely very different 
> than what one should want to appear in 
> /net/randomserver.foo.example/.  That means that one could 
> not turn randomserver.foo.example into a root server for foo.example.
> 
> That's very annoying.  But suppose you were to either define 
> a new special FH, like the ROOT FH, or a special directory to 
> be found in the server's pseudo-root directory, and have the 
> clients mount _that_ on /nfs4/foo.example/.  That would avoid 
> the above problem.
> 
> But I realize that either of those two solutions (new special 
> FH or reserved special directory name in the ROOT FH 
> directory) are problematic by themselves as well.  So I won't 
> push them.

You've convinced me that a reserved name for the name to be exported as
an organizational root seems perfectly reasonable.  That is, to
interpret /nfs4/foo.example/., you'd get the "randomserver.foo.example"
name from the SRV RR and mount a reserved server-side name like
"/.org-root", which is "/net/randomserver.foo.example/.org-root" in your
/net/ parlance.  I agree that a reserved FH would be far harder to
administer.  But what's wrong with a reserved name?  I don't see the
downside.

> > > [...]
> > This isn't a central feature of the draft, but a bell/whistle.  It 
> > could go without leaving a mark.  But it's a handy kind of 
> thing for 
> > actually
> But you can't even know which ones are read-write and which 
> are read-only until you try to write.
> 

Yeah, there are probably other ways of achieving comparable
functionality (finding an RW instance of a replicated root) that don't
require NFS-client involvement.  They'd be more straightforward to
execute.

> > > read-write *if* there's a single location, else the client should 
> > > assume read-write *if* the client is willing to retry 
> writes against 
> > > each location until one succeeds, else the client should assume 
> > > read-only.
> > 
> > I buy the first half.  I don't buy the second half.  What 
> if some of 
> > the R/O instances are out-of-date with respect to the R/W, even if 
> > they will eventually converge?  Then you have updates arriving at a 
> > file server where the update is based on a potentially 
> stale copy of the data.
> (c) copes with the problem you mention.  If a writable copy 
> fails then fallback on any other replica may mean that the 
> filesystem appears to go backwards in time if the replica is 
> not up to date with the writable copy at that point.  That 
> problem can be avoided by never falling back on another 
> replica once a writable replica is found (a very unsatisfying 
> solution, IMO).

Admittedly, this is something of a dead horse, if the draft simply
abandons the /nfs4/.domain.name semantics, and I'll let it go I promise,
but this still fails my example.  That example would read, say,
"/etc/passwd" (really "/nfs4/.domain.name/etc/passwd", but no matter)
from an aged R/O site, possibly out-of-date; it's only when the
application attempts to write an updated copy that the NFS client tries
to find a writable copy of the file system.  Now, it could be that the
NFS client can notice that, moving to a writable copy, it is in fact
failing forward in time--which is a fine thing to do.  What's not so
fine is that the updated "passwd" file is an update to what may have
been an obsolete version of the file.  Re-doing the LOOKUP, OPEN, READ
(to the kernel cache, say) isn't really sufficient, as it's the
application which will have done the read and will have composed new
content to be written to the file.  There may be differences whether one
updates with "echo 'foo' >> /etc/passwd" or "cat /etc/passwd
/tmp/new.entry > /etc/passwd.new; rename /etc/passwd.new /etc/passwd",
but both mechanisms should work, and at least one breaks.

If, as you say, the NFS client avoids failing over to an older version
of the file system by refusing to leave the R/W copy, that at least
operates with correct semantics, though as you say, it's unsatisfying.

I believe that the longer-term cure is more complicated.  There's a
place for the NFS client in navigating a set of R/O file system copies
of different ages for reading purposes, but it's fundamentally different
from mixing in the attempts to write/update the file system.  It's
probably too much semantics for this little feature in any case.

		Craig

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 21 Aug 2009 11:45:54 -0500

> administer.  But what's wrong with a reserved name?  I don't see the

Great.  IMO there's nothing wrong with having such a reserved name,
provided it starts with a '.'.

(Although arguably there's nothing forcing us to make the reserved name
start with a period, I think sticking to the spirit of the POSIX rule
that filesystems may only have special reserved names if they're in
filesystem root directories and start with periods, is a good idea.)

> but this still fails my example.  That example would read, say,

:)

Multi-replica synchronoization is way outside the scope here.  To be
fair, I think one can design protocols that make this possible.

However, the way in which a client picks a replica to write to need not
be out of scope.  Consider a replication protocol that involves having a
single master (and, say, elections, or whatever), then a single replica
will be writable, and clients that want to write need to look for it.
Or consider a replication protocol that will synchronize all copies
provided that clients have open file delegations, or do byte-range
locking: then all replicas might be writable and clients will
[obviously] quickly find a suitable replica to write to.  And so on.

The bigger problem is: how to use an FH obtained from one replica on
another.  Currently there's no way to do that -- the client has to
essentially do lookupd-based recovery, and that is generally
unsatisfying.  But that too is out of scope.

I'd be happy to leave handling of read-only vs. read-write out of scope
entirely for this I-D, just as the base NFSv4 spec did in the first
place.  But I wouldn't want to force clients to do read-only when
there's multiple replicas.

Nico
--

---

From: Thomas Narten <[email-masked]>
Date: Fri, 21 Aug 2009 14:01:41 -0400

I'm not going to respond in detail to the thread, because most of the
followup is NFS-specific.

That said, I think:

> My advice:

seems generally on the right track.

I see now that a lot of my confusion has to do with the document not
being clear about what is purely NFS business vs. what is the DNS SRV
behavior. It would be good to more clearly separate these two aspects.

For example, you really only need to describe the DNS client behavior
w.r.t. to SRV. Whether this is done on an NFS client or NFS server
depends on the context, but from the perspective of DNS, when doing
SRV processing, the entity is a DNS "client", even if it is an NFS
*server* doing the processing. *This* document presumably only needs
to be specifying aspects of SRV lookup behavior.

Also, the talk about read/write and trying various alternative servers
to find one that is writable seems fundamentally like a Bad Idea to
mix up with SRV processing. For one thing, a basic definition of SRV
is that all entities that are returned via an SRV lookup provide
exactly the same service. One should not be trying various ones
looking for a "different" service.

If you want to distinguish read only from read-write directories, the
proper way would be to give them different service names (I would
think).

Thomas

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 21 Aug 2009 13:08:01 -0500

> to be specifying aspects of SRV lookup behavior.

+1

> proper way would be to give them different service names (I would

Ah, very good point.  Yes, so let's have _two_ SRV RR names:

 - _nfs4._tcp.@ORIGIN for read-only domain namespace roots
 - _write._nfs4._tcp.@ORIGIN for read-write domain namespace roots

Also, the "nfs4" in the service name is a bit misleading.  Perhaps we
should have it be:

 - _nfs4._domainroot._tcp.@ORIGIN for read-only domain namespace roots
 - _write. _nfs4._domainroot._tcp.@ORIGIN for read-write ... roots

I.e., we should clearly separate "domain namespace distributed
filesystem root" from "distributed filesystem protocol".

That way we could regularize CIFS, AFS, NFS, Lustre, ... domain root SRV
RR naming.  (Yes, I am aware of the existing AFSDB RR type.)

Nico
--