Subject: [nfsv4] NFSv4.2 client does Seek to pNFS DS or MDS for File Layout?

From: Rick Macklem <[email-masked]>
Date: Thu, 22 Aug 2019 01:04:44 +0000

Hi,

In reading RFC-7862, I can clearly see that, for a pNFS server, Seek may be done
on the DS. (Actually, it's MAY in upper case.)

What I can't find is any indication of whether the client should do Seek on the DS
or MDS for this case. Is there some understanding along the lines of...
--> NFSv4.2 client should attempt Seek on the DS. If NFS4ERR_NOTSUPP is replied,
      then it should attempt Seek on the MDS.
or ???
(It doesn't seem to say that they are implemented on both MDS and DS or neither
 MDS or DS.)

Also, am I to assume similar rules apply to a NFSv4.2 data server for Flexible File
Layout or do all these operations go to the MDS for Flexible File Layout?

Sorry if I just missed something in the RFC, rick

---

From: David Noveck <[email-masked]>
Date: Thu, 22 Aug 2019 12:06:36 -0400

> on the DS. (Actually, it's MAY in upper case.)

I think the "MAY" you are referring to is basically saying that, unlike
operation such
as RENAME/REMOVE/OPEN, which are MDS-only, SEEK/READ/READ_PLUS
are allowed in COMPOUNDs sent to the DS.

In the case of the OPTIONAL operations among these (e.g. SEEK, READ_PLUS),
the
DS or the MDS may choose to support or not support them.

> or MDS for this case.

It's not there.

> Is there some understanding along the lines of...

You could do that but it is highly unlikely to work.   As a general matter,
operations typically
sent to the DS (e.g. READ) can be sent to the MDS who acts as a proxy.  My
understanding
is that SEEK would be handled similarly, but I don't think that's currently
made explicit.

It is possible that an MDS could simulate SEEK on a DS what desn't support
it but I don't
expect MDS's to devote the necessaryeffort to that.   The MDS is not under
an obligation to
support the OPTIONAL operation SEEK and trying to support it when the DS
does not seems
to me like goinhg the extra 26.2 miles.  Sigh!.

> (It doesn't seem to say that they are implemented on both MDS and DS or
neither

It doesn't.   It may be worth filing an erratta on this issue.   I think
you could say something
useful/helpful without re-issung 7862.

> Layout

i would think that what I said above would apply to flexible files as
well.   In the tight
coupling case, I don't see any difference beween flexible files and the
files layout
described in Section 13 of RFc5661.

> or do all these operations go to the MDS for Flexible File Layout?

In the case of a NFSv3 DS with weak coupling, you can try the MDS, but I
expect it is
unlikely to provide support, for the reasons stated above.

On Wed, Aug 21, 2019 at 9:05 PM Rick Macklem <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Rick Macklem <[email-masked]>
Date: Tue, 27 Aug 2019 02:13:39 +0000

David Noveck wrote:
>> rmacklem wrote:
>I think the "MAY" you are referring to is basically saying that, unlike operation such
>In the case of the OPTIONAL operations among these (e.g. SEEK, READ_PLUS), the
Since others haven't replied to this, here's a couple of data points:
- The FreeBSD server will support Seek on the MDS and DSs when it is configured
  as a pNFS server and the servers support NFSv4.2.
- The FreeBSD NFSv4.2 client does Seek to the MDS at this time. (I may add
  doing Seek to the DS someday, but not unless I see enough need to justify
  doing it to avoid the overhead of Seek being done on the MDS as a proxy to the DS.)
- The Linux NFSv4.2 client appears to do Seek to the MDS only. (From a quick look
  at Linux-4.17-rc2.)
  Trond, is this going to continue to be the case?

>You could do that but it is highly unlikely to work.   As a general matter, operations >typically
>sent to the DS (e.g. READ) can be sent to the MDS who acts as a proxy.  My >understanding
>is that SEEK would be handled similarly, but I don't think that's currently made >explicit.
>
>It is possible that an MDS could simulate SEEK on a DS what desn't support it but I >don't
>expect MDS's to devote the necessaryeffort to that.   The MDS is not under an >obligation to
>support the OPTIONAL operation SEEK and trying to support it when the DS does not >seems
>It doesn't.   It may be worth filing an erratta on this issue.   I think you could say >something
I suppose a useful question might be..
Is there an extant or planned NFSv4.2 pNFS File or Flexible File layout server
that will support Seek in one of the MDS or DSs, but not both of them.

If not, I think it might be nice to clarify that either both or neither support Seek.

>i would think that what I said above would apply to flexible files as well.   In the tight
>coupling case, I don't see any difference beween flexible files and the files layout
Yes, I feel that the Flexible File layout should optionally support Seek for NFSv4.2
storage servers as well. (With "optionally" having the same meaning as File layout.
Hopefully supported by both MDS and DS or neither of them.)

>In the case of a NFSv3 DS with weak coupling, you can try the MDS, but I expect it is
I could see an argument for a weakly coupled NFSv4.2 DS supporting it without
any MDS support, but I think it would be better to have a rule of
"both support it or neither MDS nor DS do".

Can anyone who knows the answer for other NFSv4.2 pNFS File or Flexible File
layout pNFS servers please reply w.r.t. which of MDS and/or DS supports Seek?

Thanks, rick

On Wed, Aug 21, 2019 at 9:05 PM Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 26 Aug 2019 23:25:16 -0400

> On Aug 26, 2019, at 22:13, Rick Macklem <[email-masked]> wrote:
> 
> David Noveck wrote:
>>> rmacklem wrote:

If you are asking if I have any plans to change this, then the answer for now is “no”. However if others are interested in working on it, then I’m happy to take patches.

That said, I have a number of questions here. How are you dealing with striped files on the client? What about dense vs. sparsely packed layouts?
In particular a DS serving a sparsely packed layout is supposed to deny you access to the holes that lie between stripes (see NFS4ERR_PNFS_IO_HOLE) so what would a server do when you ask it to seek across those boundaries?

I’m not really seeing any of those questions being answered directly in RFC7862. There is a sentence or two describing how READ_PLUS should operate (but details are missing), but there is nothing that describes SEEK behaviour w.r.t. pNFS, affaics.

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Tue, 27 Aug 2019 04:23:51 +0000

Trond Myklebust wrote:
>On Aug 26, 2019, at 22:13, Rick Macklem ><[email-masked]<mailto:[email-masked]>> wrote:
>
>David Noveck wrote:
>rmacklem wrote:
>If you are asking if I have any plans to change this, then the answer for now is “no”. >However if others are interested in working on it, then I’m happy to take patches.
>
>That said, I have a number of questions here. How are you dealing with striped files >on the client?
Good point. I hadn't thought about striping...

I'll admit I don't know how Seek to a DS would work for striped layouts?
Would it only return data/hole offsets for the stripes stored on that DS?
And for the densely packed case, would the offset be a logical offset for the
file or one mapped to the offset for that stripe?

--> This suggests that an MDS only implementation of Seek makes sense for striped
      File layouts.

> What about dense vs. sparsely packed layouts?
>In particular a DS serving a sparsely packed layout is supposed to deny you access to >the holes that lie between stripes (see NFS4ERR_PNFS_IO_HOLE) so what would a >server do when you ask it to seek across those boundaries?
--> I would have guessed it would ignore these I/O holes, since they aren't really
      a part of the file. Fortunately, I didn't need to worry about how this might
      be implemented because...
The FreeBSD server never does striping and simply provides the entire file on a
single DS for File layout (sparse, but no I/O holes caused by striping).

For Flexible file layout, the file can be mirrored, but that results in a copy
of the entire file on each of the mirrors. Again, no striping.

>I’m not really seeing any of those questions being answered directly in RFC7862. >There is a sentence or two describing how READ_PLUS should operate (but details >are missing), but there is nothing that describes SEEK behaviour w.r.t. pNFS, affaics.
>
Yep. Seems like a "hole" in the RFC (sorry, I couldn't resist;-).

rick