Subject: RE: [nfsv4] New pNFS ID available

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 12:47:21 -0400

Garth Goodson wrote:
>Mark Eshel wrote:

...

> > I know a single metadata server but I did not see anything that will

> > preclude it from also acting as a data server.
> 
> You are correct.  It is a requirement that data can also be sent to
the 
> metadata server through normal V4 commands.  Additionally, there isn't

> anything precluding the metadata server from naming itself in a layout

> to be used as a data server.

The potential problem I see with naming yourself as a data server is
that
it makes it very hard to implement or even make much sense of cases
where
the spec tells you that the data server can do one set of things and the
metadata server a different set of things.

I think to distinguish two cases.  One is that the same CPU's, memory,
disks, network adapters are used to implement both a metadata server
and a data server.  I don't see any problem with that.  If the metadata
server's IP is meta.pnfs.netapp.com and it puts a data1.pnfs.netapp.com
in a layout and those two addresses are really the same pile of
hardware,
then you have no problem.  For any request you receive, you know whether
it is for the metadata server or the data server and the spec tells you
what you can and must do.

If you put meta.pnfs.netapp.com in the layout, things get much worse.
For each op, you  might be able to make the distinction based on the 
current filehandle (assuming the layout fh's are different), but the
prospect of a request in which some of the ops are directed at the 
metadata server and others at the data server strikes me as quite
unappetizing.

As there some reason distinct IP's will not work?  If you do need the
same IP, I would think we need some way of establishing separate 
sessions with the metadata server and the data server so that each
request is clearly directed at one or the other.

-----Original Message-----
From: Goodson, Garth 
Sent: Thursday, July 21, 2005 9:12 PM
To: Marc Eshel
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

Marc Eshel wrote:
> Garth Goodson <[email-masked]> wrote on 07/21/2005 05:01:20
PM:
> 
> 
>>>The high level requirement that I have to be able to support cluster 
> 
> file 
> 
>>>system is that a metadata server can also act as a data server. From 
>>>reading this document I don't see a problem doing it but if someone 
> 
> think 
> 
>>>otherwise please let me know why. 
>>
>>The issues outlined in Dave Noveck's previous comments still remain.
As 
> 
> 
>>far as the spec is concerned there is a single metadata server and
only 
>>I/O ops are to be issued to data servers (the wording still exists).
> 
> 
> I know a single metadata server but I did not see anything that will 
> preclude it from also acting as a data server.
> 
You are correct.  It is a requirement that data can also be sent to the 
metadata server through normal V4 commands.  Additionally, there isn't 
anything precluding the metadata server from naming itself in a layout 
to be used as a data server.

> 
>>>Is there a draft for file storage protocol? where can I find it?
>>>
>>
>>It is included in the doc.  Section 5, page 26.
> 
> 
> That is what I thought but I wanted to make sure. So I don't see how
the 
> NFS client interface with the different layout drivers and for the
file 
> layout driver how it establishes communications with the NFS data
server. 
> Few things need to happen before you can send a read/write request to
the 
> NFS data server. In other words I should be able to implement the NFS
file 
> base protocol using only this document but I am sure I can.
> Marc.

We don't define how the NFS client interfaces with layout drivers, that 
is client side implementation specific and can be done in many ways 
(although arguably useful for interoperability), but I don't believe it 
is required to build a working file's-based client.

-Garth

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 1 Aug 2005 13:23:27 -0400 (EDT)

> request is clearly directed at one or the other.

The metadata server must accept NFSv4 I/O, BUT, must it accept NFSv4 I/O
while a client has a layout?  I think it should be stated in the spec that
a client should not issue NFSv4 I/O while it has a layout for a file.  It
is then an easy check for the server to see if the client has a layout or
not, and where to direct the I/O request.  Is there an use case that would
warrant it?  I can't think of one.
Dean

---

From: Marc Eshel <[email-masked]>
Date: Mon, 1 Aug 2005 11:44:09 -0700

I thought we already agreed that the data server does not have to check if 
the client is using the right fh for data server only operations. The 
draft states that it "SHOULD" and I was asking to down grade it to 
"should" but it is not "MUST". In any case we can add a flag in the fh 
that will indicate that it should be used only for data server operations. 
like you suggested, if we wanted to do more checking on the data server. 
Using multiple ip address for the same server complicates cluster 
management for especially for failure detection and fail over. Is there a 
real problem that you see other than unappetizing ?
Marc.

---

From: Jim Zelenka <[email-masked]>
Date: Mon, 01 Aug 2005 15:43:10 -0400

>I think it should be stated in the spec that
>a client should not issue NFSv4 I/O while it has a layout for a file.

I'm not sure that this is a good idea. We've discussed
handling most of the "hard" error cases of clients having
trouble writing directly to storage by having the client
issue a regular NFSv4 write directly to the server. I don't
like saying that the client does a LAYOUTRETURN before writing,
even in a compound request, because there's no atomicity there,
which raises the possibility of another client getting bogus
read results between the LAYOUTRETURN and the WRITE.

-jim

-- 
Jim Zelenka
Software Engineer, Panasas, Inc.
Accelerating Time to Results(TM) with Clustered Storage
www.panasas.com
412.323-3500

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 1 Aug 2005 16:20:03 -0400 (EDT)

I think this restriction would only be for the NFSv4 file layout case.
(this is the confusing problem of lumping the NFSv4 file layout in the
general spec, but yes there are advantages too...)  With two I/O
protocols, there isn't any problems with "what is this I/O command really
for?"  Anyways, I think I like Marc's idea of putting stuff in the
filehandle.

> read results between the LAYOUTRETURN and the WRITE.

Just to nitpick, I don't see the atomicity requirement here.
LAYOUTRETURN doesn't sync the data or do anything interesting that would
cause results to change (at least, not that I can remember).  As well, if
you don't lock a region, another client can always sneak in some data.
Dean

---

From: Craig Everhart <[email-masked]>
Date: Mon, 1 Aug 2005 14:24:00 -0600

> request is clearly directed at one or the other.

Seems like the meta server could make the choice itself.  I kinda like the
idea of distinct FHs, myself, but if the meta server cared to make the
distinction, it could do so in the manner of its choice.

Is anybody still interested in the one-time-use FHs that were originally
proposed for data filers?

                Craig

Craig Everhart
+1 919 543 2169 (tie 441 2169)
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Craig Everhart <[email-masked]>
Date: Mon, 1 Aug 2005 16:40:42 -0400

> read results between the LAYOUTRETURN and the WRITE.

So is the client responsible for cache consistency between its
own data buffers and the server data buffers?  How can it do
this?  When the client sends data to the metadata NFS server
to store it, but it has a layout that it could use to read
or write the data, should it expect that the data it could
read via the layout will be what it wrote?

Or should the server recall the layout in the presence of
fetches and stores from the client that has the layout?

                Craig

Craig Everhart

---

From: Jim Zelenka <[email-masked]>
Date: Mon, 01 Aug 2005 17:45:04 -0400

If client A holds a write layout on a region in a file, and client B
does a READ to the metadata server that contains that region of that
file, the server must ("MUST") recall the layout from client A.

If client A does the read, I would expect the server to return
bits that are not more out-of-date than the most recent LAYOUTCOMMIT
from client A for the layout. If client A wants that read to return
up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
A layout is identified by {ClientID,FileHandle,offset,length,layout_type,iomode
}*,
so the server should have enough information to figure out if the
requestor holds an outstanding layout.

In the case where client A holds a read-only layout, and client B
does a read, there is no conflict.

In the case where the reading client hasn't set a client ID, the
server should (IMO) assume that any/all outstanding write layouts
are held by clients other than the reading client, and recall them
all before servicing the read.

-jim

* Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
assuming that it is included for the purposes of this e-mail.

> <br><font size=2><tt>[email-masked] wrote on 08/01/2005 03:43:10
> PM:<br>
> <br>
> &gt; &gt;I think it should be stated in the spec that<br>
> &gt; &gt;a client should not issue NFSv4 I/O while it has a layout for
> a file.<br>
> &gt; <br>
> &gt; I'm not sure that this is a good idea. We've discussed<br>
> &gt; handling most of the &quot;hard&quot; error cases of clients having<br>
> &gt; trouble writing directly to storage by having the client<br>
> &gt; issue a regular NFSv4 write directly to the server. I don't<br>
> &gt; like saying that the client does a LAYOUTRETURN before writing,<br>
> &gt; even in a compound request, because there's no atomicity there,<br>
> &gt; which raises the possibility of another client getting bogus<br>
> &gt; read results between the LAYOUTRETURN and the WRITE.</tt></font>
> <br>
> <br><font size=2><tt>So is the client responsible for cache consistency
> between its</tt></font>
> <br><font size=2><tt>own data buffers and the server data buffers? &nbsp;How
> can it do</tt></font>
> <br><font size=2><tt>this? &nbsp;When the client sends data to the metadata
> NFS server</tt></font>
> <br><font size=2><tt>to store it, but it has a layout that it could use
> to read</tt></font>
> <br><font size=2><tt>or write the data, should it expect that the data
> it could</tt></font>
> <br><font size=2><tt>read via the layout will be what it wrote?</tt></font>
> <br>
> <br><font size=2><tt>Or should the server recall the layout in the presence
> of</tt></font>
> <br><font size=2><tt>fetches and stores from the client that has the layout?</tt></font>
> <br>
> <br><font size=2><tt>&gt; -jim<br>
> &gt; <br>
> &gt; -- <br>
> &gt; Jim Zelenka<br>
> &gt; Software Engineer, Panasas, Inc.<br>
> &gt; Accelerating Time to Results(TM) with Clustered Storage<br>
> &gt; www.panasas.com<br>
> &gt; 412.323-3500</tt></font>
> <br>
> <br><font size=2><tt>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; Craig</tt></font>
> <br>
> <br><font size=2 face="sans-serif">Craig Everhart<br>
> +1 919 543 2169 (tie 441 2169)<br>
> </font>
>

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 1 Aug 2005 18:09:09 -0400 (EDT)

> file, the server must ("MUST") recall the layout from client A.
"MUST" if you want to kill your write peformance.  In the NFS world,
reading out of date info happens all the time, are we being more strict
here?

> requestor holds an outstanding layout.
LAYOUTCOMMIT and LAYOUTRETURN have different semantics.  LAYOUTCOMMIT
commits the data, updates the mtime, etc.  LAYOUTRETURN should simply just
return the layout.  If it wants the read to return up to date data, it
just needs to do a LAYOUTCOMMIT, the LAYOUTRETURN should not affect what
types of bits are returned.
Dean

---

From: Craig Everhart <[email-masked]>
Date: Mon, 1 Aug 2005 18:36:51 -0400

> requestor holds an outstanding layout.

Right.  Good.  Now how about the case I cited, where the client does a
write to the meta server.  Should it assume that its layout is useless,
since it can't tell when it could read its data back from the
storage?  Clearly the NFS meta server can implement persistent
storage any way it intends, and the only consistency on the data
is that it shouldn't hand out further layouts to a device or a
set of devices until its pending data could be read by future
fetches from that device or those devices.

                Craig

Craig Everhart

---

From: Benny Halevy <[email-masked]>
Date: Tue, 02 Aug 2005 15:48:01 +0300

> fetches from that device or those devices.

One (maybe obvious) thing that needs to be said is that the 
client can expect to read back the same data it wrote 
through the metadata server but cannot assume anything about 
the written data if reading from the data servers using the 
layout.

I think it makes sense to require from nfsv4 servers which 
support pnfs that COMMIT (and synchronous writes) will 
commit dirty data to stable storage on the data servers so 
that it could be read via layouts.

Benny

> ------------------------------------------------------------------------
>