Subject: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: Garth Goodson <[email-masked]>
Date: Mon, 20 Jun 2005 12:39:59 -0700

There are a number of open issues and changes that were suggested for 
the pnfs draft ID (draft-welch-pnfs-ops-02.txt).

Please post any feedback or opinions (or feel free to add more).

Thanks,
-Garth

3.1.1 Device IDs

Proposal: device ids are valid only while a server is up, remappings 
while server is up must use a different device id.  (I prefer this)

Alternative: device ids are attached to leases and may be timed-out 
(probably also need to be recallable or invalidated)

3.1.2 Aggregation Schemes

Proposal: aggregation scheme (e.g., striping is part of opaque 
layout-type defined structure)

Alternative: make a general striping aggregation that is outside of the 
opaque structure; push striping up a level... (it references layout-type 
opaque specific structures which contain the devices)

3.2.2 Operation sequencing

Issue: Race condition exists between LAYOUTGET/LAYOUTRECALL passing each 
other on wire.  Sessions does not solve this since they are on different 
channels.  May require a seq-id to be returned in layout ops.

3.3.1 Identifying layouts

Proposal: layouts are identified by <ClientID, FH, offset, length>

Alternative: may need to distinguish between read vs. read/write layout 
types, e.g., block/obj layouts may have separate read vs. read/write 
layouts and will want to handle them separately.  Propose adding iomode 
to layout identification (and allowing recall to specify the specific 
mode or any mode).

3.3.3 Copy-on-write

Same as discussion for 3.3.1

3.4 Recalling a LAYOUT

Addition: Add a recall to recall all layouts pertaining to a specific fsid.

Issue: Long callback recalls (if client has dirty data that needs to be 
flushed).  Mostly wording on how long client should have to write data 
while holding the layout.  It can always revert to sending the data 
through the metadata server if I/Os to data servers fail (due to server 
revoking layout).

3.5 Committing a layout

Addition: add mtime/atime hints to LAYOUTCOMMIT (server can use them or 
not based on current state -- e.g., server should not allow time to go 
backwards and should not set the mtime if the mtime is already higher 
than that being set).  SETATTR used to specify an exact time and is 
constrained by regular V4 semantics.  The main difference between the 
time in LAYOUTCOMMIT and SETATTR is that the times set by SETATTR are 
mandatory vs. hints.

3.5.1 LAYOUTCOMMIT and EOF

Change: instead of specifying newEOF with a flag (depending on whether 
client thinks it is setting a new EOF), client will specify the last 
byte to which it wrote.

5.1 File Striping and Data Access

Change: simplify striping layout -- have enum for SPARSE vs. DENSE 
layout instead of skip and start offset

Issue: think about what error gets returned if a client performs a 
non(READ/WRITE/PUTFH/COMMIT) at a data server; issue: this may be a 
problem if a regular nfsv4 data server is used as it has no way to 
differentiate accesses.

5.2 Global Stateids

Issue: does not provide for unmodified NFSv4 data servers.  More 
thinking must be done if unmodified V4 servers are to be used as data 
servers.  See section: 5.7 for discussion.

5.3 I/O mode

Change: Don't restrict I/O mode to be RW.  It may be useful to have 
read-only replicas to which a client can be directed if the iomode is READ.

5.4.1 Lock State Propagation

Issue: can seq ids in stateids be ignored on the data servers (what 
about if sessions are used?)

5.4.3 Access State Propagation

Issue: NFSv4 spec says that READs/WRITEs do not require same principal 
as OPEN.  This opens a security hole, but some implementations depend on 
it.  pNFS should probably go with the spec on this and not change the 
semantics.

6.3 pnfs_devaddr4

Change: switch on layouttype4 instead of devaddrtypes4

Change?: add disk signature to list of types

7 pnfs File Attributes

Add: PREFERRED_ALIGNMENT, PREFERRED_BLOCKSIZE as FSID level attributes

9.2 LAYOUTCOMMIT

Add: mtime/atime time attribute hints to args

Change: neweof in result as per Object Internet Draft.  Basically, have 
a specific structure for each layout type that is the layoutcommit 
layout (vs. the layout received by layout get).  This allows extra 
opaque data to be sent on layoutcommit.

General:

IANA - think about whether additional layouttypes go through IANA or 
specification process.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 20 Jun 2005 16:01:19 -0400

> it.  pNFS should probably go with the spec on this and not change the 

I still disagree here.  In my opinion this was a mistake in the
protocol--users shouldn't be able to close other users' opens.  At the
very least, we shouldn't allow it to propogate further.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 20 Jun 2005 16:04:14 -0400

> (probably also need to be recallable or invalidated)

Note the language in 3.1.1 also needs to be clarified to make sure which
of the above is proposed.  (E.g. it says the mapping may change on
reboot, but doesn't explicitly forbid changes between reboots; in fact
the last sentence implies that such changes are allowable.)

--b.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 20 Jun 2005 13:50:58 -0700

> the last sentence implies that such changes are allowable.)

The last sentence is saying something different.  It is saying that if 
the mapping between data and the deviceID changes, then the layout 
should be recalled -- not the mapping between the deviceID and the device.

There are a number of wording changes that will be made (throughout the 
document).  I have not commented on these types of changes.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 Jun 2005 13:52:38 -0700

> problem if a regular nfsv4 data server is used as it has no way to 

Why is should it be an error in the first place. I would like all the 
nodes in my cluster filesystem to take the roles of metadata server or 
data server for each file independently.
Marc. 
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 20 Jun 2005 16:55:42 -0400

> should be recalled -- not the mapping between the deviceID and the device.

OK, I guess I see.  Though maybe that "SHOULD" should be a "MUST".

> document).  I have not commented on these types of changes.

Fair enough.--b.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 20 Jun 2005 13:59:34 -0700

I agree.  I don't think it need be an error, but if the system isn't 
designed to handle it, it would be nice if the data servers did/could 
reply with an error.

-Garth

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 20 Jun 2005 17:03:47 -0400 (EDT)

If they are stock, how can they return a new error unless they know
something, in which case they are no longer stock.
Dean

---

From: Garth Goodson <[email-masked]>
Date: Mon, 20 Jun 2005 15:06:10 -0700

Correct, that is one of the issues I mentioned.  But I think there are 
cases where the data servers are neither stock, nor where it is 
desirable that they service more than I/O requests (i.e., metadata 
requests) in which case they may return an error (and that error should 
be decided upon).

-Garth