Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-flex-files-10.txt

From: [email-masked]
Date: Mon, 17 Jul 2017 14:17:47 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Network File System Version 4 of the IETF.

        Title           : Parallel NFS (pNFS) Flexible File Layout
        Authors         : Benny Halevy
                          Thomas Haynes
	Filename        : draft-ietf-nfsv4-flex-files-10.txt
	Pages           : 38
	Date            : 2017-07-17

Abstract:
   The Parallel Network File System (pNFS) allows a separation between
   the metadata (onto a metadata server) and data (onto a storage
   device) for a file.  The flexible file layout type is defined in this
   document as an extension to pNFS which allows the use of storage
   devices in a fashion such that they require only a quite limited
   degree of interaction with the metadata server, using already
   existing protocols.  Client side mirroring is also added to provide
   replication of files.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-flex-files/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-10
https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-flex-files-10

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-flex-files-10

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 17 Jul 2017 21:25:37 +0000

My sole intent this week is to catch up on all of the review comments that I have been avoiding.

I want to thank Dave for all of the comments in general, but very specifically for the complete rewrite of Section 2.3.

> On Jul 17, 2017, at 11:17 PM, [email-masked] wrote:
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Rick Macklem <[email-masked]>
Date: Mon, 17 Jul 2017 22:39:41 +0000

First off. I'd like to thank both Thomas and David for their work on this. I am happy to see
active work on it.

1 - I think I mentioned this, but I no longer care about the stuff I posted last year.
      (Basically the case where a server might only require one mirror to be written and
       how quickly changes to a mirror were to be propagated. Others may still feel that
       this is useful, but it no longer matters to me;-)

I am thinking of posting one issue at a time, since they probably require discussion.
(If you'd prefer one post with everything in it, just let me know and I'll hold off on
 further individual posts and make one "collective" post.)
Having said the above, here's the first one:

>2.1.  LAYOUTCOMMIT
>
>   When tightly coupled storage devices are used, the metadata server
>   has the responsibility, upon receiving a LAYOUTCOMMIT (see
>   Section 18.42 of [RFC5661]), of ensuring that the semantics of pNFS
>   are respected (see Section 12.5.4 of [RFC5661]).  These do not
>   include a requirement that data written to data storage device be
>   stable upon completion of the LAYOUTCOMMIT.

- The above sentence seems to imply that the client is responsible to do COMMITs
   and seems to contradict the following para., which suggests that the COMMITs
   are required for "loose coupling" only.

>  In the case of loosely coupled storage devices, it is the
>  responsibility of the client to make sure the data file is stable
>  before the metadata server begins to query the storage devices about
>  the changes to the file.  If any WRITE to a storage device did not
>  result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the
>  metadata server MUST be preceded by a COMMIT to the storage devices
>  written to.  Note that if the client has not done a COMMIT to the
>   storage device, then the LAYOUTCOMMIT might not be synchronized to
>   the last WRITE operation to the storage device.
- I think it would be simpler for a client implementor (like me;-) to just apply
  this para. to both loose and tight coupling (ie. just drop the part of the first
  sentence up to the "," and say "It is the responsibility of the client.."
OR
- Make it conditional on a flag returned by the MDS.

In general, most of the "tight" vs "loose" coupling seems to be confusing to me.
I can see that there may be the two cases w.r.t. security:
1 - Use the synthetic uid/gid and AUTH_SYS only.
OR
2 - Use the same model as the MDS (same uid/gid/authenticator) and let the
     MDS<->DS figure out how to implement it.
Again, I think a flag returned by the MDS to indicate which security model is being
used would be simpler than the "tight" vs "loose" concept. (The document can mention
that some control protocol may be needed for the server implementation, but the
client only cares what it needs to do and not how the server chooses to implement it.)

I just find the "loose" vs "tight" coupling a somewhat confusing "catch all" and I think
a few flags returned by the MDS to the client will clarify what the client needs to do,
but this is just a suggestion and I'm happy with anything that works. (I think this also
applies to Data Server vs Storage Server. To be honest, I'd just get rid of all mention
of these and just call them all data servers?)

Again, thanks for working on this, rick

---

From: Rick Macklem <[email-masked]>
Date: Mon, 17 Jul 2017 23:15:47 +0000

Here's one more:
Last para of 2.3.1
   As the client never receives a stateid generated by a storage device,
   there is no client lease on the storage device and no prospect of
   lease expiration, even when access is via NFSv4 protocols.  Clients
   will have leases on the metadata server.  In dealing with lease
   expiration, the metadata server may need to use fencing to prevent
   revoked stateids from being relied upon by a client unaware of the
   fact that they have been revoked.

What about a loosely coupled NFSv4.1 storage server?
If won't have any stateids, but it will have a clientid and session.
I doubt that the server wants to hold onto the clientid forever?
- I'd suggest that this case might require a simple recovery of
  ExchangeID, CreateSession when the client receives a
  NFS4ERR_BAD_SESSION error from the storage server.

I don't know if you call this "lease expiration", but I would have thought
of it that way, rick

________________________________________
From: nfsv4 <[email-masked]> on behalf of Rick Macklem <[email-masked]>
Sent: Monday, July 17, 2017 6:39:41 PM
To: Thomas Haynes; [email-masked]
Subject: Re: [nfsv4] I-D Action: draft-ietf-nfsv4-flex-files-10.txt

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 18 Jul 2017 08:27:21 +0000

> On Jul 18, 2017, at 12:39 AM, Rick Macklem <[email-masked]> wrote:
> 

The client is always responsible for doing COMMITs.

In a loosely coupled storage device, the client is responsible for doing the COMMIT because it has the knowledge of which write to commit.

With a tightly coupled storage device, if the client sends the COMMIT to the MDS, the MDS can ensure through the control protocol that the commit happens.

Actually, that is wrong, RFC 5661 states in Section 12.5.4 (on page 289): 

   The data should be written and
   committed to the appropriate storage devices before the LAYOUTCOMMIT
   occurs. 

And that completely invalidates the last sentence of the first paragraph of Section 2.1.

(see below)

>  sentence up to the "," and say "It is the responsibility of the client..”

It is safe to say/implement that the client does the COMMIT in either scenario.

Taking both of these points into consideration, I’ve rewritten the section as:

2.1.  LAYOUTCOMMIT

   The metadata server has the responsibility, upon receiving a
   LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), of ensuring that the
   semantics of pNFS are respected (see Section 12.5.4 of [RFC5661]).
   These do include a requirement that data written to data storage
   device be stable before the occurance of the LAYOUTCOMMIT.

   It is the responsibility of the client to make sure the data file is
   stable before the metadata server begins to query the storage devices
   about the changes to the file.  If any WRITE to a storage device did
   not result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the
   metadata server MUST be preceded by a COMMIT to the storage devices
   written to.  Note that if the client has not done a COMMIT to the
   storage device, then the LAYOUTCOMMIT might not be synchronized to
   the last WRITE operation to the storage device.

I.e., no mention of the model as they both have to obey the same requirements.

> client only cares what it needs to do and not how the server chooses to implement it.)

The concepts are presented to highlight the difficulties that a server developer will face.

I don’t think the client has to be aware of whether it is loosely or rightly coupled.

Consider a NFSv3 storage device made by X. A MDS made by Y can use NFSv3 as
 the control protocol. It could CREATE to instantiate the data files, it could SETATTR
the mode bits/uid/gid to fence, it can GETATTR to determine the change attribute,
the mtime, etc.

But if Y can recognize X and X provides some management APIs, then Y might be
able to leverage those APIs as an additional control protocol.

And as a client implementor, you are completely unaware of all of that.

> of these and just call them all data servers?)

In practice, they are the same. In the original 3 standard documents, they are
treated a bit differently. A data server has an OS, a storage device may or may not
have an OS.

> Again, thanks for working on this, rick

And thanks for making me think!

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 18 Jul 2017 09:38:26 +0000

> On Jul 18, 2017, at 1:15 AM, Rick Macklem <[email-masked]> wrote:
> 
> Here's one more:
> Last para of 2.3.1

While true, I’m not sure it needs to be called out.

---

From: Rick Macklem <[email-masked]>
Date: Tue, 18 Jul 2017 12:45:53 +0000

Thomas Haynes wrote:
[lots of stuff snipped for brevity]
>   the last WRITE operation to the storage device.

Yes. This looks good to me. (except for the spelling of occurance;-)

> The concepts are presented to highlight the difficulties that a server developer will face.
Ok, I can buy into this. Maybe a sentence such as this near the beginning of the
document would clarify it for dumb-ass reader like me;-)

> And as a client implementor, you are completely unaware of all of that.
I did a partial implementation of the client side last year and, at least for normal
operation, I think this statement is correct.

One case where the client "might" need to know is when the server is doing
fencing.
- Unless I have misunderstood it, for the loosely coupled model, the client will
  see a NFS4ERR_ACCESS error when trying to do I/O on the storage server?
- But what error does the client see for the tightly coupled model?

Put another way, I think the client will need to be able to recognize the difference
between "fencing" and a typical NFS4ERR_ACCESS due to permissions on a tightly
coupled server.
(My partial implementation didn't include code to deal with mirrors or the case
 where one mirror has failed or been fenced off, so I didn't deal with this case.)

I'll dig into -11 later to-day.

Thanks again, rick

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 19 Jul 2017 06:50:49 +0000

> On Jul 18, 2017, at 2:45 PM, Rick Macklem <[email-masked]> wrote:
> 
> Thomas Haynes wrote:
>> 
>> 

That would actually depend on how the fencing was done (it is totally up to
the control protocol).

The implementation might cause the storage device to return NFS4ERR_ACCESS
or NFS4ERR_BAD_STATEID. The key point is that when the client gets an error
from the storage device, it is responsible for returning the layout to the mds.

The exception being NFS4ERR_DELAY - the client may want to retry the IO.

> between "fencing" and a typical NFS4ERR_ACCESS due to permissions on a tightly

I don’t think so. Consider that the client already passed permission checking to
get the file open and the layout. Now it could get the error either because it
does not have write permission or the permissions on the file was closed.

In any event, on any error, the client should return the layout (along with the error)
to the MDS. It can then ask for the LAYOUT again. At which point the mds either
assigns the new layout or does not provide a new layout (forcing IO to the mds)
or it could return NFS4ERR_ACCESS if the permissions had changed.

---

From: David Noveck <[email-masked]>
Date: Wed, 19 Jul 2017 03:01:11 -0400

> The exception being NFS4ERR_DELAY - the client may want to retry the IO.

Another possible exception is the NFSv4.0 error NFS4ERR_LEASE_MOVED.