Subject: RE: [nfsv4] New pNFS ID available

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 3 Aug 2005 17:24:14 -0400

> > As to "cannot assume anything" with regard to asynchronous writes,
> > I'm wondering if that might be too liberal.  For example, if I
> > write 4K, an implementation might propagate all the odd bytes
> > immediately and the even bytes at COMMIT time.  Even though 
> > nobody would be that wierd, I'm wondering whether some restrictions
> > might make some applications easier.  For example, the blocks
> > protocol might reasonably require that all of the write that is
> > within one block happen at one time, so that if the write is on
> > a block boundary, you can't read a partially updated block.  
> 
> That's a good property to have yet I'm not sure whether the 
> client can tell what the server's atomic block size is.

For pnfs, you could add it to the layout or the device info.  In the
case of blocks, the client is going to have to now the physical block
size to access the disk, so a statement in the definition of the 
storage to the effect would at least make it as big as that, even
if our would still need additional info to define it as a multiple
of that.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Tuesday, August 02, 2005 2:52 PM
To: Noveck, Dave
Cc: Craig Everhart; [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

Noveck, Dave wrote:
> Benny Halevy:
> 
>>One (maybe obvious) thing that needs to be said is that the 
>>client can expect to read back the same data it wrote 
>>through the metadata server but cannot assume anything about 
>>the written data if reading from the data servers using the 
>>layout.
> 
> 
>>I think it makes sense to require from nfsv4 servers which 
>>support pnfs that COMMIT (and synchronous writes) will 
>>commit dirty data to stable storage on the data servers so 
>>that it could be read via layouts.
> 
> 
> I agree.
> 
> As to "cannot assume anything" with regard to asynchronous writes,
> I'm wondering if that might be too liberal.  For example, if I
> write 4K, an implementation might propagate all the odd bytes
> immediately and the even bytes at COMMIT time.  Even though 
> nobody would be that wierd, I'm wondering whether some restrictions
> might make some applications easier.  For example, the blocks
> protocol might reasonably require that all of the write that is
> within one block happen at one time, so that if the write is on
> a block boundary, you can't read a partially updated block.  

That's a good property to have yet I'm not sure whether the 
client can tell what the server's atomic block size is.

> 
> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]]
> Sent: Tuesday, August 02, 2005 8:48 AM
> To: Craig Everhart
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] New pNFS ID available
> 
> 
> Craig Everhart wrote:
> 
>>[email-masked] wrote on 08/01/2005 05:45:04 PM:
>>
>> > If client A holds a write layout on a region in a file, and client
B
>> > does a READ to the metadata server that contains that region of
that
>> > file, the server must ("MUST") recall the layout from client A.
>> >
>> > If client A does the read, I would expect the server to return
>> > bits that are not more out-of-date than the most recent
LAYOUTCOMMIT
>> > from client A for the layout. If client A wants that read to return
>> > up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>> > A layout is identified by {ClientID,FileHandle,offset,length,
>> > layout_type,iomode
>> > }*,
>> > so the server should have enough information to figure out if the
>> > requestor holds an outstanding layout.
>>
>>Right.  Good.  Now how about the case I cited, where the client does a
>>write to the meta server.  Should it assume that its layout is
useless,
>>since it can't tell when it could read its data back from the
>>storage?  Clearly the NFS meta server can implement persistent
>>storage any way it intends, and the only consistency on the data
>>is that it shouldn't hand out further layouts to a device or a
>>set of devices until its pending data could be read by future
>>fetches from that device or those devices.
> 
> 
> One (maybe obvious) thing that needs to be said is that the 
> client can expect to read back the same data it wrote 
> through the metadata server but cannot assume anything about 
> the written data if reading from the data servers using the 
> layout.
> 
> I think it makes sense to require from nfsv4 servers which 
> support pnfs that COMMIT (and synchronous writes) will 
> commit dirty data to stable storage on the data servers so 
> that it could be read via layouts.
> 
> Benny
> 
> 
>>
>> > In the case where client A holds a read-only layout, and client B
>> > does a read, there is no conflict.
>> >
>> > In the case where the reading client hasn't set a client ID, the
>> > server should (IMO) assume that any/all outstanding write layouts
>> > are held by clients other than the reading client, and recall them
>> > all before servicing the read.
>> >
>> > -jim
>> >
>> > * Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but
I'm
>> > assuming that it is included for the purposes of this e-mail.
>> >
>> > --
>> > Jim Zelenka
>> > Software Engineer, Panasas, Inc.
>> > Accelerating Time to Results(TM) with Clustered Storage
>> > www.panasas.com
>> > 412.323-3500
>>
>>                Craig
>>
>>Craig Everhart
>>+1 919 543 2169 (tie 441 2169)
>>
>>
>>----------------------------------------------------------------------
--
>>
>>_______________________________________________
>>nfsv4 mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Thu, 04 Aug 2005 12:35:33 +0300

> if our would still need additional info to define it as a multiple

Sometimes the metadata server's virtual memory system may 
impose an atomic block size which different than the 
physical block size on disk (one defined by allocation 
granularity) so I think this attribute of the file system 
should be separated out.  Why not add it as a recommended 
filesystem attribute?

Benny

---

From: David Robinson <[email-masked]>
Date: Thu, 04 Aug 2005 10:58:35 -0500

Benny Halevy wrote:
>>>> immediately and the even bytes at COMMIT time.  Even though nobody 
>>>> would be that wierd, I'm wondering whether some restrictions

>>> That's a good property to have yet I'm not sure whether the client 
>>> can tell what the server's atomic block size is.

> Sometimes the metadata server's virtual memory system may impose an 
> atomic block size which different than the physical block size on disk 
> (one defined by allocation granularity) so I think this attribute of the 
> file system should be separated out.  Why not add it as a recommended 
> filesystem attribute?

Please don't do that.  From a classic NFS perspective the blocksize
has always been just an advisory piece of data.  If the client used
the preferred block size it may get better performance, but any size
of data would work just fine.  It has always been that the server
made it right for its needs.

In this particular example, only the direct block method needs this
sort of information. As was suggested above, the correct place to
put this sort of information is in the layout.

	-David