Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: [email-masked]
Date: Tue, 5 Jul 2005 17:46:31 -0400

Going back to Garth's original set of issues, with some picking/choosing
from a block layout perspective ... 

> 3.1.1 Device IDs
> 
> Proposal: device ids are valid only while a server is up, remappings 
> while server is up must use a different device id.  (I prefer this)
> 
> Alternative: device ids are attached to leases and may be timed-out 
> (probably also need to be recallable or invalidated)

I think we've beaten this one to a pulp.  IMHO, the basic principles
should be:
- A server cannot reuse device IDs without at a minimum recalling all
	the layouts that contain the invalid IDs, and it may have to do
	something special to get clients to discard the old device ID
	info.
- We'll need to define the circumstances under which a client MUST
	discard old device ID info (e.g., because server has rebooted
	and forgotten).  Existing techniques for dealing with server
	reboot are the place to start.
- Device IDs should not be leaseable, recallable, invalidatable, etc.
	separately from layouts and existing mechanisms to deal with
	server amnesia.
The server can always use new device IDs, which will cause clients to
fetch the device info for them anew.  This should be a rare event.

> 3.1.2 Aggregation Schemes
> 
> Proposal: aggregation scheme (e.g., striping is part of opaque 
> layout-type defined structure)
> 
> Alternative: make a general striping aggregation that is outside of the 
> opaque structure; push striping up a level... (it references layout-type 
> opaque specific structures which contain the devices)

I prefer like the alternative, so that there's a layout-independent way
of striping layouts, because all three initial layouts (file, block, object)
would like this.  I can live with layout-independent striping having
support for the sparse striping that the file layout wants, even though
it's of essentially no use for block layouts.

> 3.2.2 Operation sequencing
> 
> Issue: Race condition exists between LAYOUTGET/LAYOUTRECALL passing each 
> other on wire.  Sessions does not solve this since they are on different 
> channels.  May require a seq-id to be returned in layout ops.

I think we need the equivalent of the layout sequence id that was in the
earlier versions of the ops draft.  I hesitate to reuse an existing seqid
for this purpose, but need to check.

> 3.3.1 Identifying layouts
> 
> Proposal: layouts are identified by <ClientID, FH, offset, length>
> 
> Alternative: may need to distinguish between read vs. read/write layout 
> types, e.g., block/obj layouts may have separate read vs. read/write 
> layouts and will want to handle them separately.  Propose adding iomode 
> to layout identification (and allowing recall to specify the specific 
> mode or any mode).

At least for the block layouts, read vs. write would be opaque to pNFS
- we'd hand back extra data structure in the layout to get this done.
An important case in the block world is that the read block is valid
and read-only, but the write block is invalid until written.  OTOH,
I'm not sure I can get the details of how to do this into the block
draft in time for Paris.  This may be block-only, as this particular
issue (reads have to go to a different place than writes for correctness)
probably does not occur in files or objects.

> 3.4 Recalling a LAYOUT
> 
> Addition: Add a recall to recall all layouts pertaining to a 
> specific fsid.
> 
> Issue: Long callback recalls (if client has dirty data that needs to be 
> flushed).  Mostly wording on how long client should have to write data 
> while holding the layout.  It can always revert to sending the data 
> through the metadata server if I/Os to data servers fail (due to server 
> revoking layout).

This is needed to support a server taking a filesystem offline quickly
(but not immediately) with clients having some chance to clean up.  On
the client side, it tells the client that the filesystem is going away;
this enables the client to concentrate on writes, something it may not
know to do when it gets hit with a bunch of individual layout recalls.
We'll need wording on "how long client should have to write data" which
in practice translates into how much dirty data the client can cache
per fsid.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jim Zelenka <[email-masked]>
Date: Tue, 05 Jul 2005 19:01:15 -0400

Read versus write distinctions for layout identification
are important for objects as well. I think it's the case
that it's only don't-care for files.

-Jim Zelenka

-- 
Jim Zelenka
Software Engineer, Panasas, Inc.
Accelerating Time to Results(TM) with Clustered Storage
www.panasas.com
412.323-3500