Subject: RE: [nfsv4] New pNFS ID available

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 19:21:39 -0400

> > If client A holds a write layout on a region in a file, and client B
> > does a READ to the metadata server that contains that region of that
> > file, the server must ("MUST") recall the layout from client A.
> > 
> > If client A does the read, I would expect the server to return
> > bits that are not more out-of-date than the most recent LAYOUTCOMMIT
> > from client A for the layout. If client A wants that read to return
> > up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
> > A layout is identified by {ClientID,FileHandle,offset,length,
> > layout_type,iomode
> > }*,
> > so the server should have enough information to figure out if the
> > requestor holds an outstanding layout. 

> Right.  Good.  

Not as far as I'm concerned.

> Now how about the case I cited, where the client does a 
> write to the meta server.  Should it assume that its layout is useless, 
> since it can't tell when it could read its data back from the 
> storage?  

Most certainly not.  The layout is a promise by the server that data 
can be read using it, and if that promise ceases to ever be valid, then
it will be recalled.  Once the WRITE returns, if the layout has not 
been recalled, you can use it.  If anything, including the WRITE,
renders it useless, it is the responsibility of the server to recall
it. 

> Clearly the NFS meta server can implement persistent 
> storage any way it intends, 

OK so far.

> and the only consistency on the data 
> is that it shouldn't hand out further layouts to a device or a 
> set of devices until its pending data could be read by future 
> fetches from that device or those devices. 

Absolutely false as far as I'm concerned.  It also has a consistency
constraint with regard to layouts already outstanding (in addition
to "further layouts").  While the WRITE is pending reads to the
data server can give you the old or the new data (or a mixture I
guess -- no atomicity requirement).  After the WRITE completes, they
have to give you the new data.  If the server implements its storage
in any way such that it cannot obey those constraints then it must
recall the layouts to prevent reads being done that cannot be handled
correctly.

-----Original Message-----
From: Craig Everhart [mailto:[email-masked]]
Sent: Monday, August 01, 2005 6:37 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

[email-masked] wrote on 08/01/2005 05:45:04 PM:

> If client A holds a write layout on a region in a file, and client B
> does a READ to the metadata server that contains that region of that
> file, the server must ("MUST") recall the layout from client A.
> 
> If client A does the read, I would expect the server to return
> bits that are not more out-of-date than the most recent LAYOUTCOMMIT
> from client A for the layout. If client A wants that read to return
> up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
> A layout is identified by {ClientID,FileHandle,offset,length,
> layout_type,iomode
> }*,
> so the server should have enough information to figure out if the
> requestor holds an outstanding layout. 

Right.  Good.  Now how about the case I cited, where the client does a 
write to the meta server.  Should it assume that its layout is useless, 
since it can't tell when it could read its data back from the 
storage?  Clearly the NFS meta server can implement persistent 
storage any way it intends, and the only consistency on the data 
is that it shouldn't hand out further layouts to a device or a 
set of devices until its pending data could be read by future 
fetches from that device or those devices. 

> In the case where client A holds a read-only layout, and client B
> does a read, there is no conflict.
> 
> In the case where the reading client hasn't set a client ID, the
> server should (IMO) assume that any/all outstanding write layouts
> are held by clients other than the reading client, and recall them
> all before servicing the read.
> 
> -jim
> 
> * Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
> assuming that it is included for the purposes of this e-mail.
> 
> -- 
> Jim Zelenka
> Software Engineer, Panasas, Inc.
> Accelerating Time to Results(TM) with Clustered Storage
> www.panasas.com
> 412.323-3500 

                Craig

Craig Everhart
+1 919 543 2169 (tie 441 2169)

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Tue, 02 Aug 2005 16:00:31 +0300

> renders it useless, it is the responsibility of the server to recall

I agree, just the guarantee is on the mapping not on the 
contents...  I'm not sure that your definition of "useless" 
and Craig's are the same.  Reading possibly stale data using 
the layout may not be considered useless for many 
applications that use NFS today.  At the same time many such 
applications may use fsync() and close() to flush the client 
and server buffers onto stable storage and expect basic 
level of distributed data consistency so that after client A 
writes data and syncs it using fsync()/close() the data will 
be visible (persistently) to any other client.  Hence, my 
proposal to require pnfs enabled servers to sync dirty data 
to stable storage on the data servers on COMMIT and 
synchronous writes.

---

From: Craig Everhart <[email-masked]>
Date: Tue, 2 Aug 2005 17:01:10 -0400

> > Not as far as I'm concerned.

Perhaps I should have said "Well, seems reasonable, but how about this?"

> to stable storage on the data servers on COMMIT and 

This is just the kind of semantic issue I was trying to flush out.
I think "sync dirty data to stable storage on the data servers"
captures the requirement, so that pnfs reads get the updates, whether
they're done to the data servers or to the meta server.

What I was getting at was whether the data written by WRITE
would be the same as that fetched via a subsequent read to the
data server.

> > recall the layouts to prevent reads being done that cannot be handled

Sorry, I overspoke with "the *only* consistency".  I was trying to
flush out issues with existing layouts, and you've helped out.
This is exactly the semantic issue that needs to be clear.  It sounds like
folks are happy to live within the requirement.  What's not quite
clear is whether data is on the data servers after a WRITE or
after a COMMIT, but let me guess that you meant a synchronous WRITE
or a COMMIT.

                Craig

Craig Everhart
+1 919 543 2169