Subject: [nfsv4] draft notes from pNFS meeting held in Vancouver

From: Mike Eisler <[email-masked]>
Date: Wed, 5 Dec 2007 07:55:34 -0800 (PST)

(these notes are draft. please send me any corrections)

Date of Meeting: December 4, 2007
Where: Vancouver
Purpose: to make headway on pile of pNFS issues

Attendees

- Andy Adamson - CITI
- Mike Eisler - NetApp
- Sam Falkner - Sun
- Jason Glasgow - EMC
- Garth Goodson - NetApp
- Robert Gordon - Sun
- Suchit Kaura
- Trond Myklebust - NetApp
- Anatoly Pinchuk
- Spencer Shepler - Sun
- Renu Tewari -- IBM
- Brent Welch - Panasas

Phone

- Marc Eschel - IBM
- Benny Halevy - Panasas
- Dean Hildebrand - IBM
- Rahul Iyer - NetApp
- Lisa Week - Sun

Major Issues

- Recall processing and referential items to fore operations
-- refers to language in pNFS and sessions about how
   callbacks refer to fore operations

- Sessions major/minor and impact on pNFS
- persistent sessions usage (required or not)
- layout hint usage (OPEN and/or SETATTR)
- Overall model of device map stateids (one per major
  device) needs to be explained (dependence on GETDEVICELIST)

-- filehandle to be used for recall
-- Recall by FSID, ALL, etc.

- Layout stateids (one per clientID/layout-type/filehandle) needs
  description

- Relationship of open mode and mode of layout needs
  description in LAYOUTGET

- Layout ranges and accounting: strict or forgetful
-- Layout return not being complete until all the range
   is returned

- Layout return not being complete until all of the range
  is returned

- LAYOUTCOMMIT and access-time
- GETDEVICELIST and device map stateids
- Obtaining the device list for a major id
- sparse versus dense layouts
- granularity of device id for the file layouts
- layoutcommit vs. commit

------------------------------------------------------------

Decided to deal with device ID issues first.

Spencer: issues with device ID mapping are around recovery
in the event of server restart. How do clients and
servers discover inconsistencies. General idea was to uses
stateids, leases, etc. to deal with issues. For layouts,
this is solved, but for deviceIDs, the issues was how to
recall individual deviceID mappings without recalling
layouts.  We also changed devices to 64 bits (from 32
bits). Also wanted to deal with recalling deviceIDs
associated with a particular fsid without perturbing the
layouts. This led to major and minor devices. The server
could align major deviceIDs to fsids to get the desired
effect. We are now looking at 128 bit deviceIDs.

Tom: If we have something as wide as a uuid why not call
it a uuid?

Andy: Don't we have to fence off I/O when recalling device
IDs. If an I/O is in progress to one data server address,
and I change the device ID to another data server, how is
this different from layout recall.

Andy: Who wants this feature, blocks and objects don't
need it, files don't need it?

Brent: if you have a model where device ID is 1-1 to the
layout, then the device ID recall is superfluous.

Jason: If the IP address changes, you should be able to
recall the device ID without the layout.

(Editor: this was the ah hah moment).

Jason: The clearest thing is to support the recall of a
device ID.

Brent: We would like to be able recall device ID to handle
the IP address case.

Brent: Do we need major/minor device IDs?

Garth: I see the need to recall major device IDs to deal
with case where all dev mappings for an fsid are affected.

Mike: We have two methods for breaking device ID to device
address mappings.

Brent: Notify vs. recall: Notify is for optimizing GETATTRs
in the directory case.  Recalling deviceIDs should rare

Mike: My rationale for notifications was to avoid having
zillions of device IDs.

Suchit: the most common use case is to update a device
address for device ID. An update notification is the
least intrusive.

Trond: stateids only make sense if you are going to recall
by stateid.

Sam: Would it make sense to state that recall of layout
by fsid does not invalidate deviceIDs.

Garth/Sam: What does a notify delete callback response
mean: The client got the notify, or it deleted the
mapping.

Jason: We need to very clear: does deleting device ID
mapping mean the client's I/O is drained?

Trond: You should be able to fence the client.

Andy: Here is the client sending I/O to a device. Gets
device ID delete notify, the client's data falls on
the floor.

Brent: It's actually a benefit, because if the mapping is
deleted, the client gets told that writing to the device
address is futile.

Garth: If you get a recall of a device ID, the server
shout allow the I/O to be flushed complete until the recall
completes.

Trond: But it should be allowed to have a server fence
the client.

Spencer: Recall processing. There is fundamental issue
how long a client is allowed to process the callback? In
NFSv4.0, the client could acknowledge the recall, and then do
delegreturn. In NFsv4.1 we seem to have added the notion
that a client gets a device ID recall, and the client can
wait for I/O flush before acknowledging the recall.

Mike: If deviceIDs are delegated, then we cannot have
a different model for device IDs than for file object
recalls.

Spencer: But layouts don't behave delegations

Mike: Right, because they aren't delegations. I can get
behind making them behave the same, but for most non-file
(objects excepted) layout types, making them equivalent
doesn't work (because of the fencing issue; most native
data access protocols don't use NFSv4.1 stateids in the
I/O operation).

Spencer: Note: we don't use layout stateid on READ/WRITE

Andy: What is a layout stateid used for?

Spencer: It is used to keep track of the set of layouts
for a file (stateids have seqids)

Jason: If client sent a LAYOUTGET/LAYOUTRETURN at the same
time, won't the sequence id be wrong?

Spencer: In that case, the client set the seqid to zero,
to indicate it doesn't care about sequencing.

Trond: What does it mean when the current stateid is
used? What does that do the seqid?

Spencer: Need to clarify. If is undefined what the
seqid is.

Spencer: If we use recall to recall a device ID delegation,
then we need an fh for every device ID

Mike: This is why I wanted notifications for the entire
device map versus a delegation on a device ID.

Jason: How about: server should do CB_NOTIFY remove device
ID. Client immediately responds. When client has drained
I/O, it does a delegreturn.

Sam: If server knows no client is using device ID 1,
what does the server do with that knowledge?

Mike/Trond: so add a generation number to device ID.

Marc: At 128 bit device IDs, there is room for generation
number.

Spencer: We need to establish requirements now and then
move forward.

Requirements:

- large opaque device IDs to accommodate generation IDs

- want to allow the server to stop using a data server
  and switch to another

- recalling layouts to change device ID is too burdensome

-- because shared deviceIDs result in zillions of
   layoutgets

- The spec should not mandate that when
  changing the deviceID mapping the client MUST drain I/O.

-- Server gives client an indication whether old mapping
   exists for a while or is gone. Regardless, the indication
   of mapping change says that the client needs to stop
   sending new I/Os.

-- spec should allow server, if it wants, to preserve the
   writes to the old data server

- Should allow device ID to device address mapping to
  change, and device IDs to be re-used without recalling
  layouts because file layouts use a device ID

- recall layouts by fsid/clientID. Previously the notion
  was to use major deviceIDs to abstract this concept but
  as we've seen that introduced much complexity. Recall
  layout by fsid is seen as the lesser evil.

- recall layouts by clientID (ALL) required.

- if you delete a device ID, you have to recall the layouts
  that use the device ID (also must be addressed in IANA
  requirements section for new layout types. Not all layout
  types encode deviceIDs the same way)

- we will delete one deviceID at a time
-- no need for major/minor device IDs
-- no stateids for device IDs

(document GETDEVICEINFO / CB_NOTIFY race and how the client
deals with it).

- new notification op for deviceID changes (not CB_NOTIFY). This
  eliminates need for filehandle and stateid.

- deviceID pertains to a clientID not an fsid.
- no stateids needed for entire deviceID list.

------------------------------------------------------------

Strict and/or forgetful layout ranges

Spencer: Client can forget it has a layout on the range.

Spencer: recommendation: client can give up accounting for ranges.

Spencer: client can either return  to LAYOUTREECALL: "I
don't have that range", or issue LAYOUTRETURN on the range.

Brent: client still has to track seqid field in layout
stateid

Mike: that means an error "I don't have that range" might
cause the layout stateid seqid to be out of sync.

Consensus: client must do an explicit LAYOUTRECALL

------------------------------------------------------------

Sparse and Dense

Garth - Needs to clear that that if two layout ranges
on a file have two different device IDs, the ranges are
not mergeable.

Trond: If you can't merge layout ranges, you can't append
(mre: huh?)

Brent: 3 choices
  - when you add devices to your device set, you have to
    recall the layout
  - fix the math with dense
  - throw out dense

Mike: Having different striping patterns for different
ranges of a file is an absolute requirement. Cannot require
the entire file to be re-striped when new devices come
online in order to benefit from those devices.

Brent: I don't really have a stake in this, since it is a file
layout issue.

Mike: What happens if Panasas adds an NFSv4.1 wrapper to its data
server>

Brent: OK, in that case, since objects uses dense, it
would be much better if I give a file and object layout
that they both use the same packing. So I would prefer
DENSE be supported.

Rough consensus is that DENSE is needed.

Mike: A simple proposal is desired. The proposal Anatoly
emailed to the [email-masked] on November 28, 2007 looked
complex.

Anatoly/Suchit: We will produce a proposal.

------------------------------------------------------------

Persistent Sessions and Layout Hints

Spencer: current text requires persistent sessions.

Spencer: wording need to change, and justify why persistent
sessions.

Jason: This sentence:

  Once the client has a client ID that supports pNFS,
  it creates a persistent session over the client ID,
  requesting persistent.

seemed to come out of no where.

Rough consensus: add an arm to the OPEN arguments and
results "EXCL2" that encodes a bitmap of attributes that
can be specified in exclusive create. Would include ACL
and layout hint.

(mre: I realized over night that we could add a
recommended, read-only attribute that is the bitmap of
attributes permitted to be specified in the new EXCL2 arm).

------------------------------------------------------------

- Recall processing and referential items to fore operations

Mike: Asserts the seqid in the layout stateid and the
removal of device ID stateids makes this moot.

Mike: need text to understand how the client deals with
the race.

Tom: What if the server loses its mind, the client has to
cache some bogus, temporal state that the server.

Mike: The slot table on the fore channel has N slots. The
client has at most N layoutgets outstanding. When the
server sends N+1 recalls that don't match any stateids,
the client knows the server is insane, and destroys the
session and client ID.

Spencer: This issues applies to delegations.

Mike: Need common text in spec.

------------------------------------------------------------

- LAYOUTCOMMIT and access-time

Spencer/Brent: this is in place

Spencer: will verify text.

Mike: found this text in the i-d:

   "The loca_time_modify and loca_time_access"

------------------------------------------------------------

   - layoutcommit vs. commit

Mike: asserts that until layoutcommit succeeds, the data
is not committed.

Tom: what about a client that writes beyond its stripe?

Mike: need text to describe that NFSv4.1 layout type must
prevent this.

Issue tabled for now.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Wed, 5 Dec 2007 16:41:10 -0800 (PST)

> Anatoly/Suchit: We will produce a proposal.

I met with Anatoly and Suchit after the WG meeting today.

All three of us came to the conclusion (though Anatoly wants to
do some test cases) that the existing results in LAYOUTGET
are sufficient for calculating the relative offset. 

Cutting and pasting Anatoly's example which kicked off the DENSE
packing issue:

struct nfsv4_1_file_layout_ds_addr4
dev0 = { // ID is 0
    {0},
    {{A}}
},
dev1 = { // ID is 1
    {0},
    {{B}}
};
 
File Layouts for a client file
struct nfsv4_1_file_layout4
fl0 = {
    0,
    x, // stripe unit size of 0x400 is coded with x
    0,
    fh0
},
fl1 = {
    1,
    y, // stripe unit size of 0x1000 and dense packing are coded with y
    0,
    fh1
};
 
fl0 and fl1 correspond to file ranges [0, 0x400-1] and [0x400,
0x1400-1];
 
Now a client requests layout for an offset of 0x800 and length 0x400
and the reply is:
 
struct layout4
lo = {
    0x800,
    0x400,
    lo_iomode, // some IO mode
    lo_content // see below
};
 
struct layout_content4
lo_content = {
    LAYOUT4_NFSV4_1_FILES,
    loc_body // see below
};
loc_body contains fl1 defined above.
 
So, the client knows the server (B) and the file handle (fh1) to do IO
on. Offset for the fh1 is needed as well. The formula from paragraph
14.5 does not provide correct offset since
 
data_file_offset = floor(file_offset / stripe_width)
* stripe_unit_size + file_offset % stripe_unit_size =
floor(0x800/0x1000) * 0x1000 + 0x800 % 0x1000 = 0x800;
 
and the correct offset is

rel_offset = (file_offset - layout_offset) = (0x800 - 0x400) = 0x400;
data_file_offset = floor(rel_offset / stripe_width)
* stripe_unit_size + rel_offset % stripe_unit_size =
floor(0x400/0x1000) * 0x1000 + 0x400 % 0x1000 = 0x400;
 
Offset of first byte the layout is applicable to (layout_offset =
0x400) is used in the calculation of the data file offset. Therefore,
it should be stored in the nfsv4_1_file_layout4 structure returned to
the client.

The layout_offset is available in results from LAYOUTGET.
So the spec needs to be updated to reflect proper calculation.

Have we missed anything?

--------------

---

From: Garth Goodson <[email-masked]>
Date: Thu, 06 Dec 2007 11:21:15 -0800

I think you are assuming that the offset in each layout4 structure  is 
always the beginning offset for that layout -- this means that if I have 
a layout covering say 0-0x80000 then I can't give a layout4 structure 
covering just 0x40000-0x80000 since there is no way for the client to 
know where the layout really started.  I think we need this flexibility; 
that is if I'm right about this.

-Garth

---

From: Mike Eisler <[email-masked]>
Date: Thu, 6 Dec 2007 14:54:53 -0800 (PST)

> that is if I'm right about this.

I think the problem is that the term "layout" is not mutually
understood.

To me, a layout is an object of type layout4.

So I parse your words and read:

``I think you are assuming that the offset in each layout is 
always the beginning offset for that layout -- this means that if I
have a layout covering say 0-0x80000 then I can't give a layout
covering just 0x40000-0x80000 since there is no way for the client to
know where the layout really started.''

It makes no sense to me.

---

From: Garth Goodson <[email-masked]>
Date: Thu, 06 Dec 2007 15:10:46 -0800

Mike Eisler wrote:
>> have 

I agree with your definition.  Let me reword.

I think you are assuming that the offset in each layout4 structure is 
the beginning offset of that striping pattern.  This means that if I 
have a striping pattern that can be described in a single layout4 
structure (the minimum layout4 structures required for this pattern) 
that covers 0-0x80000 then the server can not give a client a layout4 
structure that only covers 0x40000-0x80000 even though the client 
doesn't care about 0-0x40000-1 (e.g., this is necessary for block-based 
layouts). There is no way for the client to know that the real start 
offset for this striping pattern is 0 and not 0x40000.

-Garth

---

From: Mike Eisler <[email-masked]>
Date: Sun, 9 Dec 2007 10:01:43 -0800 (PST)

Final notes from pNFS meeting held in Vancouver

> (these notes are draft. please send me any corrections)

No corrections were received. These notes are FINAL. Corrections
should be directed to this thread, I will not make further updates.

These are NOT the notes from the IETF working group meeting.
I expect the chairs will send those soon.

-------------------------------------------------------------------

Copyright (C) The IETF Trust (2007)

Attendees

- Andy Adamson - CITI
- Mike Eisler - NetApp
- Sam Falkner - Sun
- Jason Glasgow - EMC
- Garth Goodson - NetApp
- Robert Gordon - Sun
- Suchit Kaura - BlueArc
- Trond Myklebust - NetApp
- Anatoly Pinchuk - BlueArc
- Spencer Shepler - Sun
- Renu Tewari -- IBM
- Brent Welch - Panasas

Phone

- Marc Eschel - IBM
- Benny Halevy - Panasas
- Dean Hildebrand - IBM
- Rahul Iyer - NetApp
- Lisa Week - Sun

Major Issues

Requirements:

Sparse and Dense

Brent: 3 choices

Jason: This sentence:

Issue tabled for now.

---

From: Mike Eisler <[email-masked]>
Date: Sun, 9 Dec 2007 10:12:53 -0800 (PST)

> attributes permitted to be specified in the new EXCL2 arm).

Attached are the diffs I will be checking in today to resolve this
issue.

Index: nfsv41_middle_fileattributes.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_fileattributes.xml,v
retrieving revision 1.31
diff -c -w -b -r1.31 nfsv41_middle_fileattributes.xml
*** nfsv41_middle_fileattributes.xml	25 Nov 2007 00:35:58 -0000	1.31
--- nfsv41_middle_fileattributes.xml	9 Dec 2007 17:41:40 -0000
***************
*** 156,162 ****
  	  The per file system attributes are:
  	  <list style='empty'>
  	    <t>
! 	      supp_attr, fh_expire_type, link_support,
  	      symlink_support, unique_handles, aclsupport,
  	      cansettime, case_insensitive, case_preserving,
  	      chown_restricted, files_avail, files_free,
--- 156,162 ----
  	  The per file system attributes are:
  	  <list style='empty'>
  	    <t>
! 	      supported_attrs, suppattr_exclcreat, fh_expire_type, link_support,
  	      symlink_support, unique_handles, aclsupport,
  	      cansettime, case_insensitive, case_preserving,
  	      chown_restricted, files_avail, files_free,
***************
*** 202,208 ****
        <ttcol align='left' >Acc.</ttcol>
        <ttcol align='left' >Defined in:</ttcol>
  
!       <c>supp_attr</c><c>0</c><c>bitmap</c><c>RD</c>
        <c>
  	<xref target="attrdef_supp_attr" />
        </c>
--- 202,208 ----
        <ttcol align='left' >Acc.</ttcol>
        <ttcol align='left' >Defined in:</ttcol>
  
!       <c>supported_attrs</c><c>0</c><c>bitmap4</c><c>RD</c>
        <c>
  	<xref target="attrdef_supp_attr" />
        </c>
***************
*** 266,271 ****
--- 266,277 ----
        <c>
  	<xref target="attrdef_filehandle" />
        </c>
+ 
+       <c>suppattr_exclcreat</c><c>75</c><c>bitmap4</c><c>RD</c>
+       <c>
+ 	<xref target="attrdef_suppattr_exclcreat" />
+       </c>
+ 
      </texttable>
    </section>
    <section anchor="recommended_attributes" 
***************
*** 302,315 ****
  	<xref target="attrdef_case_insensitive" />
        </c>
  
!       <c>change_policy</c><c>60</c><c>chg_policy4</c><c>RD</c>
        <c>
! 	<xref target="attrdef_change_policy" />
        </c>
        
!       <c>case_preserving</c><c>17</c><c>bool</c><c>RD</c>
        <c>
! 	<xref target="attrdef_case_preserving" />
        </c>
  
        <c>chown_restricted</c><c>18</c><c>bool</c><c>RD</c>
--- 308,321 ----
  	<xref target="attrdef_case_insensitive" />
        </c>
  
!       <c>case_preserving</c><c>17</c><c>bool</c><c>RD</c>
        <c>
! 	<xref target="attrdef_case_preserving" />
        </c>
  
!       <c>change_policy</c><c>60</c><c>chg_policy4</c><c>RD</c>
        <c>
! 	<xref target="attrdef_change_policy" />
        </c>
  
        <c>chown_restricted</c><c>18</c><c>bool</c><c>RD</c>
***************
*** 600,606 ****
  						 Definitions">
  
      <section toc="exclude" anchor="attrdef_supp_attr" 
! 	     title="Attribute 0: supp_attr">
  	<t>
  	The bit vector which would retrieve all mandatory and
  	recommended attributes that are supported for this object.
--- 606,612 ----
  						 Definitions">
  
      <section toc="exclude" anchor="attrdef_supp_attr" 
! 	     title="Attribute 0: supported_attrs">
  	<t>
  	The bit vector which would retrieve all mandatory and
  	recommended attributes that are supported for this object.
***************
*** 609,614 ****
--- 615,631 ----
  	</t>
      </section>
  
+     <section toc="exclude" anchor="attrdef_suppattr_exclcreat" 
+ 	     title="Attribute 75: suppattr_exclcreat">
+ 	<t>
+ 	The bit vector which would set all mandatory and
+ 	recommended attributes that are supported by the EXCLUSIVE4_1
+         method of file creation via the OPEN operation.
+ 	The scope of this attribute applies to all objects with a
+ 	matching fsid.
+ 	</t>
+     </section>
+ 
      <section toc="exclude" anchor="attrdef_type" 
  	     title="Attribute 1: type">
  	<t>
Index: nfsv41_middle_fileattributes_acls.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_fileattributes_acls.xml,v
retrieving revision 1.23
diff -c -w -b -r1.23 nfsv41_middle_fileattributes_acls.xml
*** nfsv41_middle_fileattributes_acls.xml	21 Nov 2007 01:12:01 -0000	1.23
--- nfsv41_middle_fileattributes_acls.xml	9 Dec 2007 17:41:40 -0000
***************
*** 1402,1407 ****
--- 1402,1408 ----
                SHOULD take place, but the mode MUST be applied to the
                inherited ACL as described in <xref target="setmode"
  						  />, thereby modifying the ACL.
+ 		      <vspace blankLines='1' />
              </t>
              <t>If just the ACL is given in the call:
                <vspace blankLines="1" />
***************
*** 1409,1419 ****
--- 1410,1422 ----
                the ACL as defined in the CREATE or OPEN will be set
                without modification, and the mode modified as in
                <xref target="settingacl" />
+ 		      <vspace blankLines='1' />
              </t>
              <t>If both mode and ACL are given in the call:
                <vspace blankLines="1" /> In this case, inheritance
                SHOULD NOT take place, and both attributes will be set
                as described in <xref target="setboth" />.
+ 		      <vspace blankLines='1' />
              </t>
              <t>
                If neither mode nor ACL are given in the call:
***************
*** 1421,1427 ****
                In the case where an object is being created without
                any initial attributes at all, e.g. an OPEN operation
                with an opentype4 of OPEN4_CREATE and a createmode4 of
!               EXCLUSIVE4, inheritance SHOULD NOT take place.
                Instead, the server SHOULD set permissions to deny all
                access to the newly created object. It is expected
                that the appropriate client will set the desired
--- 1424,1432 ----
                In the case where an object is being created without
                any initial attributes at all, e.g. an OPEN operation
                with an opentype4 of OPEN4_CREATE and a createmode4 of
!               EXCLUSIVE4, inheritance SHOULD NOT take place (note that
!               EXCLUSIVE4_1 is a better choice of createmode4, since it
!               does permit initial attributes).
                Instead, the server SHOULD set permissions to deny all
                access to the newly created object. It is expected
                that the appropriate client will set the desired
***************
*** 1444,1449 ****
--- 1449,1455 ----
                If either the dacl or sacl attribute is supported,
                then the ACL4_DEFAULTED flag SHOULD be set on the
                newly created attributes.
+ 		      <vspace blankLines='1' />
              </t>
            </list>
          </t>
Index: nfsv41_middle_op_open.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_open.xml,v
retrieving revision 1.23
diff -c -w -b -r1.23 nfsv41_middle_op_open.xml
*** nfsv41_middle_op_open.xml	21 Nov 2007 01:12:01 -0000	1.23
--- nfsv41_middle_op_open.xml	9 Dec 2007 17:41:41 -0000
***************
*** 15,65 ****
  
    <section toc="exclude" anchor="OP_OPEN_DESCRIPTION" title="DESCRIPTION">
      <t>
!       The OPEN operation creates and/or opens a regular file in a directory
!       with the provided name.  If the file does not exist at the server and
!       creation is desired, specification of the method of creation is
!       provided by the openhow parameter.  The client has the choice of three
!       creation methods: UNCHECKED, GUARDED, or EXCLUSIVE.
      </t>
      <t>
!       If the current filehandle is a named attribute directory, OPEN will
!       then create or open a named attribute file.  Note that exclusive
!       create of a named attribute is not supported.  If the createmode is
!       EXCLUSIVE4 and the current filehandle is a named attribute directory,
        the server will return EINVAL.
      </t>
      <t>
!       UNCHECKED means that the file should be created if a file of that name
!       does not exist and encountering an existing regular file of that name
!       is not an error.  For this type of create, createattrs specifies the
!       initial set of attributes for the file.  The set of attributes may
!       include any writable attribute valid for regular files.  When an
!       UNCHECKED create encounters an existing file, the attributes specified
!       by createattrs are not used, except that when an size of zero is
!       specified, the existing file is truncated.  If GUARDED is specified,
!       the server checks for the presence of a duplicate object by name
!       before performing the create.  If a duplicate exists, an error of
!       NFS4ERR_EXIST is returned as the status.  If the object does not
!       exist, the request is performed as described for UNCHECKED.  For each
!       of these cases (UNCHECKED and GUARDED) where the operation is
!       successful, the server will return to the client an attribute mask
!       signifying which attributes were successfully set for the object.
!     </t>
!     <t>
!       EXCLUSIVE specifies that the server is to follow exclusive creation
!       semantics, using the verifier to ensure exclusive creation of the
!       target.  The server should check for the presence of a duplicate
!       object by name.  If the object does not exist, the server creates the
!       object and stores the verifier with the object.  If the object does
!       exist and the stored verifier matches the client provided verifier,
!       the server uses the existing object as the newly created object.  If
!       the stored verifier does not match, then an error of NFS4ERR_EXIST is
!       returned.  No attributes may be provided in this case, since the
!       server may use an attribute of the target object to store the
!       verifier.  If the server uses an attribute to store the exclusive
!       create verifier, it will signify which attribute by setting the
!       appropriate bit in the attribute mask that is returned in the results.
      </t>
      <t>
        For the target directory, the server returns change_info4 information
        in cinfo.  With the atomic field of the change_info4 data type, the
--- 15,190 ----
  
    <section toc="exclude" anchor="OP_OPEN_DESCRIPTION" title="DESCRIPTION">
      <t>
!       The OPEN operation opens a regular file in a
!       directory with the provided name or filehandle.
!       OPEN can also create a file if a name is provided,
!       and the client specifies it wants to create a file.
!       Specification whether a file is be created or not,
!       and the method of creation is via the openhow
!       parameter. The openhow parameter consists of
!       a switched union (data type opengflag4), which
!       switches on the value of opentype (OPEN4_NOCREATE
!       or OPEN4_CREATE). If OPEN4_CREATE is specified,
!       this leads to another switched union (data type
!       createhow4) that supports four cases of creation
!       methods: UNCHECKED4, GUARDED4, EXCLUSIVE4,
!       or EXCLUSIVE4_1. If opentype is OPEN4_CREATE,
!       then the claim field of the claim field (sic)
!       MUST be one of CLAIM_NULL, CLAIM_DELEGATE_CUR, or
!       CLAIM_DELEGATE_PREV, because these claim methods
!       include a component of a file name.
! 
      </t>
+    
      <t>
!       If the current filehandle is a named attribute
!       directory, OPEN will then create or open a named
!       attribute file.  Note that exclusive create
!       of a named attribute is not supported.  If the
!       createmode is EXCLUSIVE4 or EXCLUSIVE4_1 and the
!       current filehandle is a named attribute directory,
        the server will return EINVAL.
+ 
+ <cref>mre: ugh: OPEN can be used to create a named attribute directory?</cref>
+ 
+     </t>
+ 
+     <t>
+       UNCHECKED4 means that the file should be created if a
+       file of that name does not exist and encountering an
+       existing regular file of that name is not an error.
+       For this type of create, createattrs specifies the
+       initial set of attributes for the file.  The set
+       of attributes may include any writable attribute
+       valid for regular files.  When an UNCHECKED4
+       create encounters an existing file, the attributes
+       specified by createattrs are not used, except that
+       when createattrs specifies the size attribute
+       with a size of zero, the existing file is truncated.
+ 
+     </t>
+     <t>
+       If GUARDED4 is specified, the server checks for
+       the presence of a duplicate object by name before
+       performing the create.  If a duplicate exists, an
+       error of NFS4ERR_EXIST is returned as the status.
+       If the object does not exist, the request is
+       performed as described for UNCHECKED4.
+ 
+     </t>
+     <t>
+       For the UNCHECKED4 and GUARDED4 cases, where the
+       operation is successful, the server will return
+       to the client an attribute mask signifying which
+       attributes were successfully set for the object.
+ 
+     </t>
+ 
+      <t>
+       EXCLUSIVE4_1 and EXCLUSIVE4
+       specify that the server is to follow exclusive
+       creation semantics, using the verifier to ensure
+       exclusive creation of the target.  The server should
+       check for the presence of a duplicate object by name.
+       If the object does not exist, the server creates
+       the object and stores the verifier with the object.
+       If the object does exist and the stored verifier
+       matches the client provided verifier, the server
+       uses the existing object as the newly created object.
+       If the stored verifier does not match, then an error
+       of NFS4ERR_EXIST is returned.
+      </t>
+      <t>
+       If using EXCLUSIVE4, and if the server uses attributes to
+       store the exclusive create verifier, the server will signify
+       which attributes it used by setting the appropriate bits in
+       the attribute mask that is returned in the results.
+       Unlike UNCHECKED4, GUARDED4, and EXCLUSIVE4_1, EXCLUSIVE4 does
+       not support the setting of attributes at file creation, and
+       after a successful OPEN via EXCLUSIVE4, the client MUST
+       issue a SETATTR to set attributes to a known state.
+      </t>
+      <t>
+       In NFSv4.1, EXCLUSIVE4 has been deprecated in favor
+       of EXCLUSIVE4_1.
+       Unlike EXCLUSIVE4, attributes may be provided
+       in the EXCLUSIVE4_1 case, but because the server
+       may use attributes of the target object to store
+       the verifier, the set of allowable attributes
+       may be fewer than the set of attributes SETATTR
+       allows. The allowable attributes for EXCLUSIVE4_1
+       are indicated in the suppattr_exclcreat (<xref
+       target="attrdef_suppattr_exclcreat"/>) attribute. If the client
+       attempts to set in cva_attrs an attribute that is not in
+       suppattr_exclcreate, the server MUST return NFS4ERR_INVAL.
+       The response field, attrset indicates both which attributes
+       the server set from cva_attrs, and which attributes the
+       server used to store the verifier. The client can logically
+       AND cva_attrs.attrmask with attrset to determine which attributes
+       were used to store the verifier.
+ 
+      </t>
+      <t>
+       With the addition of persistent sessions and
+       pNFS, under some conditions EXCLUSIVE4 MUST NOT
+       be used by the client or supported the server.
+       The following table summarizes the appropriate and
+       mandated exclusive create methods for implementations
+       of NFSv4.1:
+ 
+      </t>
+ 
+       <texttable anchor='exclusive_create'>
+ 
+       <preamble>
+          Required methods for exclusive create
+       </preamble>
+ 
+       <ttcol>Persistent Reply Cache</ttcol>
+       <ttcol>pNFS server</ttcol>
+       <ttcol>Server REQUIRED</ttcol> <ttcol>Client Allowed</ttcol>
+ 
+       <c>no</c> <c>no</c>
+ 	<c>EXCLUSIVE4_1 and EXCLUSIVE4</c>
+ 	<c>EXCLUSIVE4_1 (SHOULD) or EXCLUSIVE4 (SHOULD NOT)</c>
+ 
+       <c>no</c> <c>yes</c>
+ 	<c>EXCLUSIVE4_1</c>
+ 	<c>EXCLUSIVE4_1</c>
+ 
+       <c>yes</c> <c>no</c>
+ 	<c>GUARDED4</c>
+ 	<c>GUARDED4</c>
+ 
+       <c>yes</c> <c>yes</c>
+ 	<c>GUARDED4</c>
+ 	<c>GUARDED4</c>
+ 
+       </texttable>
+ 
+     <t>
+      If CREATE_SESSION4_FLAG_PERSIST is set in the results
+      of CREATE_SESSION the reply cache is persistent.
+      If the EXCHGID4_FLAG_USE_PNFS_MDS flag is set in the
+      results from EXCHANGE_ID, the server is a pNFS server.
+      If the client attempts to use EXCLUSIVE4 on a persistent session,
+      or a session derived from a 
+      EXCHGID4_FLAG_USE_PNFS_MDS client ID, the server MUST return
+      NFS4ERR_INVAL.
      </t>
+ 
      <t>
! 
!      With persistent sessions, exclusive create semantics
!      are fully achievable via GUARDED4, and so EXCLUSIVE4
!      or EXCLUSIVE4_1 MUST NOT be used.  When pNFS is
!      being used, the layout_hint attribute might
!      not be supported after the file is created. Only the
!      EXCLUSIVE4_1 and GUARDED methods of exclusive file
!      creation allow the atomic setting of attributes.
! 
      </t>
+ 
      <t>
        For the target directory, the server returns change_info4 information
        in cinfo.  With the atomic field of the change_info4 data type, the
***************
*** 218,232 ****
        read-only file system.
      </t>
      <t>
!       As with the CREATE operation, the server MUST derive the owner, owner
!       ACE, group, or group ACE if any of the four attributes are required
!       and supported by the server's file system.  For an OPEN with the
!       EXCLUSIVE4 createmode, the server has no choice, since such OPEN calls
!       do not include the createattrs field.  Conversely, if createattrs is
!       specified, and includes owner or group (or corresponding ACEs) that
!       the principal in the RPC call's credentials does not have
!       authorization to create files for, then the server may return
!       NFS4ERR_PERM.
      </t>
      <t>
        In the case of a OPEN which specifies a size of zero (e.g. truncation)
--- 343,361 ----
        read-only file system.
      </t>
      <t>
!       As with the CREATE operation, the server MUST derive
!       the owner, owner ACE, group, or group ACE if any
!       of the four attributes are required and supported
!       by the server's file system.  For an OPEN with the
!       EXCLUSIVE4 createmode, the server has no choice,
!       since such OPEN calls do not include the createattrs
!       field.  Conversely, if createattrs (UNCHECKED4 or
!       GUARDED4) or cva_attrs (EXCLUSIVE4_1) is specified,
!       and includes an owner, or owner_group, or ACE that
!       the principal in the RPC call's credentials does
!       not have authorization to create files for, then
!       the server may return NFS4ERR_PERM.
! 
      </t>
      <t>
        In the case of a OPEN which specifies a size of zero (e.g. truncation)
***************
*** 390,413 ****
    </section>
    <section toc="exclude" anchor="OP_OPEN_IMPLEMENTATION" title="IMPLEMENTATION">
      <t>
!       The OPEN operation contains support for EXCLUSIVE create.  The
!       mechanism is similar to the support in NFSv3 <xref
!       target="RFC1813"/>.  However, this mechanism is not needed
!       if a server stores its reply cache in stable storage. If the
!       server indicates (via the csr_persist field in the response to
!       CREATE_SESSION) the client SHOULD NOT use OPEN's approach to
!       exclusive create.
!      </t>
!      <t>
!       In absence of csr_persist being TRUE, the client invokes exclusive
!       create by setting the how parameter is EXCLUSIVE.  In this
!       case, the client provides a verifier that can reasonably be
!       expected to be unique.  A combination of a client identifier,
!       perhaps the client network address, and a unique number
!       generated by the client, perhaps the RPC transaction identifier,
!       may be appropriate.  This mechanism allows reliable exclusive
!       create semantics even when the server does not support the storing
!       session reply information in stable storage.
      </t>
      <t>
        If the object does not exist, the server creates the object and stores the
--- 519,532 ----
    </section>
    <section toc="exclude" anchor="OP_OPEN_IMPLEMENTATION" title="IMPLEMENTATION">
       <t>
!       In absence of a persistent session, the client
!       invokes exclusive create by setting the how parameter
!       to EXCLUSIVE4 or EXCLUSIVE4_1.  In these cases, the
!       client provides a verifier that can reasonably be
!       expected to be unique.  A combination of a client
!       identifier, perhaps the client network address,
!       and a unique number generated by the client, perhaps
!       the RPC transaction identifier, may be appropriate.
      </t>
      <t>
        If the object does not exist, the server creates the object and stores the
***************
*** 418,469 ****
        erroneous failure on retransmission of the request. It is assumed that
        an exclusive create is being performed because exclusive semantics are
        critical to the application. Because of the expected usage, exclusive
!       CREATE does not rely solely on the normally volatile duplicate request
!       cache for storage of the verifier. The duplicate request cache in
!       volatile storage does not survive a crash and may actually flush on a
!       long network partition, opening failure windows.  In the UNIX local
!       file system environment, the expected storage location for the
!       verifier on creation is the meta-data (time stamps) of the object. For
!       this reason, an exclusive object create may not include initial
!       attributes because the server would have nowhere to store the
!       verifier.
!     </t>
!     <t>
!       If the server can not support these exclusive create semantics,
!       possibly because of the requirement to commit the verifier to stable
!       storage, it should fail the OPEN request with the error,
!       NFS4ERR_NOTSUPP.
!     </t>
!     <t>
!       During an exclusive CREATE request, if the object already exists, the
!       server reconstructs the object's verifier and compares it with the
!       verifier in the request. If they match, the server treats the request
!       as a success. The request is presumed to be a duplicate of an earlier,
!       successful request for which the reply was lost and that the server
!       duplicate request cache mechanism did not detect. If the verifiers do
!       not match, the request is rejected with the status, NFS4ERR_EXIST.
!     </t>
!     <t>
!       Once the client has performed a successful exclusive create, it must
!       issue a SETATTR to set the correct object attributes.  Until it does so,
!       it should not rely upon any of the object attributes, since the server
!       implementation may need to overload object meta-data to store the
!       verifier.  The subsequent SETATTR must not occur in the same COMPOUND
!       request as the OPEN.  This separation will guarantee that the
!       exclusive create mechanism will continue to function properly in the
!       face of retransmission of the request.
!     </t>
!     <t>
!       Use of the GUARDED attribute does not provide exactly-once semantics.
!       In particular, if a reply is lost and the server does not detect the
!       retransmission of the request, the operation can fail with
!       NFS4ERR_EXIST, even though the create was performed successfully.  The
!       client would use this behavior in the case that the application has
!       not requested an exclusive create but has asked to have the file
!       truncated when the file is opened.  In the case of the client timing
!       out and retransmitting the create request, the client can use GUARDED
!       to prevent against a sequence like: create, write, create
!       (retransmitted) from occurring.
      </t>
      <t>
        For SHARE reservations, the client must specify a value for
--- 537,665 ----
        erroneous failure on retransmission of the request. It is assumed that
        an exclusive create is being performed because exclusive semantics are
        critical to the application. Because of the expected usage, exclusive
!       CREATE does not rely solely on the server's reply cache
!       for storage of the verifier. A nonpersistent reply cache
!       does not survive a crash and the session and reply cache
!       may be deleted after a network partition that exceeds the
!       lease time, thus opening failure windows. 
!     </t>
!     <t>
!       An NFSv4.1 server SHOULD NOT store the verifier in
!       any of the file's RECOMMENDED or REQUIRED attributes.
!       If it does, the server SHOULD use time_modify_set or
!       time_access_set to store the verifier.
!       The server SHOULD NOT store the verifier in the
!       following attributes:
! 
! 	acl (it is desirable for access control to
! 	be established at creation),
! 
! 	dacl (ditto),
! 
! 	mode (ditto),
! 
! 	owner (ditto),
! 
! 	owner_group (ditto),
! 
! 	retentevt_set (it may be desired to
! 	establish retention at creation)
! 
! 	retention_hold (ditto),
! 
! 	retention_set (ditto),
! 
! 	sacl (it is desirable for auditing control
! 	to be established at creation),
! 
! 	size (on some servers, size may have a
! 	limited range of values),
! 
! 	mode_set_masked (as with mode),
! 
! 		and
! 
! 	time_creation (a meaningful file creation
! 	should be set when the file is created).
! 
!      Another alternative for the server is to use named attribute
!      to store the verifier.
! 
!     </t>
! 
!     <t>
!      Because the EXCLUSIVE4 create method does not specify
!      initial attributes, when processing an EXCLUSIVE4 create,
!      the server
! 
!      <list style="symbols">
!      <t>
!        SHOULD set the
!        owner of the file to that corresponding to the credential of
!        request's RPC header.
!      </t>
! 
!      <t>
!       SHOULD not leave the file's access control to anyone
!       but the owner of the file.
!      </t>
!      </list>
! 
!     </t>
! 
!     <t>
!       If the server cannot support exclusive create
!       semantics, possibly because of the requirement to
!       commit the verifier to stable storage, it should fail
!       the OPEN request with the error, NFS4ERR_NOTSUPP.
! 
!     </t>
!     <t>
!       During an exclusive CREATE request, if the object
!       already exists, the server reconstructs the object's
!       verifier and compares it with the verifier in
!       the request. If they match, the server treats the
!       request as a success. The request is presumed to
!       be a duplicate of an earlier, successful request
!       for which the reply was lost and that the server
!       duplicate request cache mechanism did not detect. If
!       the verifiers do not match, the request is rejected
!       with the status, NFS4ERR_EXIST.
! 
!     </t>
!     <t>
!       After the client has performed a successful
!       exclusive create, the attrset response indicates
!       which attributes were used to store the verifier.
!       If EXCLUSIVE4 was used, any attribute set in
!       attrset was used for the verifier. If EXCLUSIVE4_1
!       was used, the client determines the attributes
!       used for the verifier by comparing attrset with
!       cvs_attrs.attrmask; any bits set in former but
!       not the latter identify the attributes used store
!       the verifier.  The client MUST immediately issue a
!       SETATTR on attributes used to store the verifier.
!       Until it does so, the attributes used to store the
!       verifier cannot be relied upon.  The subsequent
!       SETATTR must not occur in the same COMPOUND request
!       as the OPEN.
! 
!     </t>
!     <t>
!       Unless a persisent session is used, use of the
!       GUARDED4 attribute does not provide exactly-once
!       semantics.  In particular, if a reply is lost and
!       the server does not detect the retransmission of the
!       request, the operation can fail with NFS4ERR_EXIST,
!       even though the create was performed successfully.
!       The client would use this behavior in the case that
!       the application has not requested an exclusive create
!       but has asked to have the file truncated when the
!       file is opened.  In the case of the client timing
!       out and retransmitting the create request, the client
!       can use GUARDED4 to prevent against a sequence like:
!       create, write, create (retransmitted) from occurring.
! 
      </t>
      <t>
        For SHARE reservations, the client must specify a value for
***************
*** 499,505 ****
        After the server returns OPEN4_RESULT_PRESERVE_UNLINKED, if a client
        issues a REMOVE operation that would reduce the file's link count to
        zero, the server SHOULD report a value
!       of zero for the FATTR4_NUMLINKS attribute on the file.
      </t>
      <section toc="exclude" anchor="open_getfh_issue"
  	title="WARNING TO CLIENT IMPLEMENTORS">
--- 695,701 ----
        After the server returns OPEN4_RESULT_PRESERVE_UNLINKED, if a client
        issues a REMOVE operation that would reduce the file's link count to
        zero, the server SHOULD report a value
!       of zero for the numlinks attribute on the file.
      </t>
      <section toc="exclude" anchor="open_getfh_issue"
  	title="WARNING TO CLIENT IMPLEMENTORS">
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.52
diff -c -w -b -r1.52 nfsv41_middle_pnfs.xml
*** nfsv41_middle_pnfs.xml	20 Nov 2007 21:30:04 -0000	1.52
--- nfsv41_middle_pnfs.xml	9 Dec 2007 17:41:41 -0000
***************
*** 495,518 ****
    files. Setting this attribute separately,
    after the file has been created might make
    it difficult, or impossible, for the server
!   implementation to comply. This further
!   complicates the exclusive file creation via
!   OPEN, which when done via the EXCLUSIVE4
!   createmode does not allow the setting of
!   attributes at file creation time. However as
!   noted in <xref target="OP_OPEN_DESCRIPTION"
!   />, if the server supports a persistent reply
!   cache, the EXCLUSIVE4 createmode is not
!   needed. Therefore, a metadata server that
!   supports the layout_hint attribute SHOULD
!   support a persistent session reply cache.
!   If a session is created on the metadata
!   server with a persistent
!   reply cache,
!   a pNFS client that wants to set layout_hint
!   at file creation (OPEN) time MUST NOT use the
!   EXCLUSIVE4 createmode, and instead MUST use
!   GUARDED for an exclusive regular file creation.
  
  </t>
  </section>
--- 495,513 ----
    files. Setting this attribute separately,
    after the file has been created might make
    it difficult, or impossible, for the server
!   implementation to comply.
! </t>
! <t>
!   Because the EXCLUSIVE4 createmode4 does not allow the
!   setting of attributes at file creation time, NFSv4.1
!   introduces the EXCLUSIVE4_1 createmode4, which does
!   allow attributes to be set at file creation time. In
!   addition, if the session is created with persistent
!   reply caches, EXCLUSIVE4_1 is neither necessary
!   nor allowed. Instead, GUARDED4 both works better and is
!   prescribed. <xref target="exclusive_create" /> in <xref
!   target="OP_OPEN_DESCRIPTION" />, summarizes how a client
!   is allowed to issue an exclusive create.
  
  </t>
  </section>
***************
*** 1048,1078 ****
    the client, the client knows that pNFS is a possibility for the file
    system.  If, from the server that returned the new FSID, the client
    does not have a client ID that came from an EXCHANGE_ID result that
!   returned EXCHGID4_FLAG_USE_PNFS_MDS, it must send an EXCHANGE_ID to
    the server with the EXCHGID4_FLAG_USE_PNFS_MDS bit set. If the
    server's response does not have EXCHGID4_FLAG_USE_PNFS_MDS, then
    contrary to what the fs_layout_type attribute said, the server does
    not support pNFS, and the client will not be able use pNFS to that
!   server; in this case, the server should return NFS4ERR_NOTSUPP in
    response to any pNFS operation.
  </t>
  <t>
!  Once the client has a client ID that supports pNFS, it creates
!  a persistent session over the client ID, requesting persistent.
! </t>
! <t>
!   If a file is to be created on a pNFS enabled file system, the client
!   uses the OPEN operation as it would normally -- specifying for
!   create type either GUARDED4 or UNCHECKED4.  With the normal set of
!   attributes that may be provided upon OPEN used for creation, there
!   is an optional layout_hint attribute.  The client's use of
!   layout_hint allows the client to express its preference for layout
!   type its associated layout details.  The client is advised to
!   combine a GETATTR operation after the OPEN within the same
!   COMPOUND.  The GETATTR may then retrieve the layout_type attribute
!   for the newly created file.  The client will then know what layout
!   type the server has chosen for the file and therefore what storage
!   protocol the client must use.
  </t>
  <t>
    If the client wants to open an existing file, then it also includes
--- 1043,1081 ----
    the client, the client knows that pNFS is a possibility for the file
    system.  If, from the server that returned the new FSID, the client
    does not have a client ID that came from an EXCHANGE_ID result that
!   returned EXCHGID4_FLAG_USE_PNFS_MDS, it MUST send an EXCHANGE_ID to
    the server with the EXCHGID4_FLAG_USE_PNFS_MDS bit set. If the
    server's response does not have EXCHGID4_FLAG_USE_PNFS_MDS, then
    contrary to what the fs_layout_type attribute said, the server does
    not support pNFS, and the client will not be able use pNFS to that
!   server; in this case, the server MUST return NFS4ERR_NOTSUPP in
    response to any pNFS operation.
  </t>
  <t>
!   The client then creates a session, requesting a persistent session, so
!   that exclusive creates can be done with single round trip via the
!   createmod4 of GUARDED. If the session ends up not being persistent,
!   the client will used EXCLUSIVE4_1 for exlusive creates.
! </t>
! <t>
!   If a file is to be created on a pNFS enabled file
!   system, the client uses the OPEN operation.  With the
!   normal set of attributes that may be provided upon OPEN
!   used for creation, there is an optional layout_hint
!   attribute.  The client's use of layout_hint allows the
!   client to express its preference for layout type its
!   associated layout details. The use a createmode4 of
!   UNCHECKED4, GUARDED4, or EXCLUSIVE4_1 will allow the
!   client to provide the layout_hint attribute at create
!   time. The client MUST NOT use EXCLJUSIVE4 (see <xref
!   target="exclusive_create"/>).  The client is RECOMMENDED
!   to combine a GETATTR operation after the OPEN within
!   the same COMPOUND.  The GETATTR may then retrieve
!   the layout_type attribute for the newly created file.
!   The client will then know what layout type the server has
!   chosen for the file and therefore what storage protocol
!   the client must use.
! 
  </t>
  <t>
    If the client wants to open an existing file, then it also includes
Index: dotx.d/head.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/head.x,v
retrieving revision 1.48
diff -c -w -b -r1.48 head.x
*** dotx.d/head.x	21 Nov 2007 01:12:05 -0000	1.48
--- dotx.d/head.x	9 Dec 2007 17:41:41 -0000
***************
*** 382,387 ****
--- 382,388 ----
   * NFSv4.1 attributes
   */
  typedef bitmap4		fattr4_supported_attrs;
+ typedef bitmap4		fattr4_suppattr_exclcreat;
  typedef nfs_ftype4	fattr4_type;
  typedef uint32_t	fattr4_fh_expire_type;
  typedef changeid4	fattr4_change;
***************
*** 476,481 ****
--- 477,483 ----
  const FATTR4_LEASE_TIME		= 10;
  const FATTR4_RDATTR_ERROR	= 11;
  const FATTR4_FILEHANDLE		= 19;
+ const FATTR4_SUPPATTR_EXCLCREAT = 75;
  
  /*
   * Recommended Attributes
***************
*** 527,537 ****
  const FATTR4_DIRENT_NOTIF_DELAY = 57;
  const FATTR4_DACL		= 58;
  const FATTR4_SACL		= 59;
! const FATTR4_CHANG_POLICY	= 60;
  const FATTR4_FS_STATUS		= 61;
! const FATTR4_FS_LAYOUT_TYPES	= 62;
  const FATTR4_LAYOUT_HINT	= 63;
! const FATTR4_LAYOUT_TYPES	= 64;
  const FATTR4_LAYOUT_BLKSIZE	= 65;
  const FATTR4_LAYOUT_ALIGNMENT	= 66;
  const FATTR4_FS_LOCATIONS_INFO	= 67;
--- 529,539 ----
  const FATTR4_DIRENT_NOTIF_DELAY = 57;
  const FATTR4_DACL		= 58;
  const FATTR4_SACL		= 59;
! const FATTR4_CHANGE_POLICY	= 60;
  const FATTR4_FS_STATUS		= 61;
! const FATTR4_FS_LAYOUT_TYPE	= 62;
  const FATTR4_LAYOUT_HINT	= 63;
! const FATTR4_LAYOUT_TYPE	= 64;
  const FATTR4_LAYOUT_BLKSIZE	= 65;
  const FATTR4_LAYOUT_ALIGNMENT	= 66;
  const FATTR4_FS_LOCATIONS_INFO	= 67;
***************
*** 543,549 ****
  const FATTR4_RETENTION_HOLD	= 73;
  const FATTR4_MODE_SET_MASKED	= 74;
  
- 
  /*
   * File attribute container
   */
--- 545,550 ----
Index: dotx.d/open_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/open_args.x,v
retrieving revision 1.6
diff -c -w -b -r1.6 open_args.x
*** dotx.d/open_args.x	21 Nov 2007 01:12:05 -0000	1.6
--- dotx.d/open_args.x	9 Dec 2007 17:41:41 -0000
***************
*** 4,10 ****
  enum createmode4 {
  	UNCHECKED4	= 0,
  	GUARDED4	= 1,
! 	EXCLUSIVE4	= 2
  };
  
  union createhow4 switch (createmode4 mode) {
--- 4,22 ----
  enum createmode4 {
  	UNCHECKED4	= 0,
  	GUARDED4	= 1,
!         /* Deprecated in NFSv4.1. */
! 	EXCLUSIVE4	= 2,
! 	/*
!          * New to NFSv4.1. If session is persistent,
!          * GUARDED4 MUST be used. Otherwise, use
!          * EXCLUSIVE4_1 instead EXCLUSIVE4.
!          */
! 	EXCLUSIVE4_1	= 3
! };
! 
! struct creatverfattr {
! 	 verifier4	cva_verf;
! 	 fattr4		cva_attrs;
  };
  
  union createhow4 switch (createmode4 mode) {
***************
*** 13,18 ****
--- 25,32 ----
  	 fattr4		createattrs;
   case EXCLUSIVE4:
  	 verifier4	createverf;
+  case EXCLUSIVE4_1:
+ 	 creatverfattr	ch_createboth;
  };
  
  enum opentype4 {

---

From: Benny Halevy <[email-masked]>
Date: Sun, 09 Dec 2007 21:25:34 +0200

I reviewed the changes (quickly) and they look OK to me.

Benny

Mike Eisler wrote:
> ------------------------------------------------------------------------
>

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Dec 2007 00:41:09 -0800 (PST)

> - no stateids needed for entire deviceID list.

Attached are the diffs and new files (for the new
CB_NOTIFY_DEVICEID operation) that I will check in later today.

In going through the changes, I think more needs to be said about 
layout stateids, and when I have changes to post, I will post them. 

<!-- 	$Id: nfsv41_middle_op_cb_notify.xml,v 1.12 2007/11/10 04:53:25 mre Exp $	 -->
<!-- Copyright (C) The IETF Trust (2007) -->
<!-- Copyright (C) The Internet Society (2006) -->
<section anchor="OP_CB_NOTIFY_DEVICEID" title="Operation 6: CB_NOTIFY_DEVICEID - Notify directory changes" >
  <t>
    Tell the client of directory changes.
  </t>

  <section toc="exclude" anchor="OP_CB_NOTIFY_DEVICEID_ARGUMENT" title="ARGUMENT">
   <?rfc include='autogen/cb_notify_deviceid_args.xml'?>
  </section>
  <section toc="exclude" anchor="OP_CB_NOTIFY_DEVICEID_RESULT" title="RESULT">
   <?rfc include='autogen/cb_notify_deviceid_res.xml'?>
  </section>
  <section toc="exclude" anchor="OP_CB_NOTIFY_DEVICEID_DESCRIPTION" title="DESCRIPTION">
    <t>
      The CB_NOTIFY_DEVICEID operation is used by the
      server to send notifications to clients about
      changes to pNFS device IDs.  The registration of
      device ID notifications occurs when the device
      mapping stateid is established using GETDEVICEINFO
      or GETDEVICELIST.  These notifications are sent
      over the backchannel. The notification is sent
      once the original request has been processed
      on the server. The server will send an array of
      notifications, cnda_changes, as a bitmap.  See <xref
      target="fattr4" /> for a description of how NFSv4.1
      bitmaps work.

   </t>

   <t>
    All device ID notifications contain a device ID and a
    layout type.  The layout type is necessary because two
    different layout types can share the same device ID,
    and the common device ID can have completely different
    mappings for each layout type.

   </t>

    <t>
     The server will send the following notifications:

      <list style="hanging">
	<t hangText="NOTIFY_DEVICEID4_ADD"><vspace />
	  Adds a device ID to the mappings.  This is
	  used for a device that has not been previously
	  returned to the client.  The client uses
	  GETDEVICEINFO or GETDEVICELIST to obtain the new
	  mapping. The notification is encoded in value of
	  data type notify_deviceid_add_or_delete4.

	</t>

	<t hangText="NOTIFY_DEVICEID4_CHANGE"><vspace />
	  A previously provided device ID to device
	  address mapping has changed and the client uses
	  GETDEVICEINFO or GETDEVICELIST to obtain the
	  updated mapping.

          The notification is encoded in a value of data
          type notify_deviceid_chg_or_del4. This data type
          also contains a boolean field, ndd_immediate,
          which if TRUE indicates that the change will be
          enforced immediately, and so the client might not
          be able to complete any pending I/O to the device
          ID. If ndcod_immediate is FALSE, then for an
          indefinite time, the client can complete pending
          I/O. After pending I/O is complete, the client
          SHOULD get the new device ID to device addrress
          mappings before issuing new I/O to the device ID.

	</t>

	<t hangText="NOTIFY4_DEVICEID_DELETE"><vspace />
	  Deletes a device ID from the mappings.  This
	  notification MUST NOT be sent if the client has
	  a layout that refers to the device ID. In other
	  words one of the following is true for layouts
	  associated with the layout type:

          <list style="symbols">

          <t>
           The client never had a layout for the device ID.
          </t>
          
          <t>
           The client has returned all layouts with that device ID.
          </t>

          <t>
           The server has revoked all layouts with that device ID.
          </t>

          </list>

	  The notification is encoded in value of data
	  type notify_deviceid_add_or_delete4.

	</t>
      </list>
    </t>
  </section>
</section>
Index: Makefile
===================================================================
RCS file: /cvs/stdk/NFSv41/Makefile,v
retrieving revision 1.99
diff -c -w -b -r1.99 Makefile
*** Makefile	21 Nov 2007 07:39:48 -0000	1.99
--- Makefile	10 Dec 2007 08:19:47 -0000
***************
*** 250,255 ****
--- 250,257 ----
  		autogen/cb_notify_res.xml \
  		autogen/cb_notify_lock_args.xml \
  		autogen/cb_notify_lock_res.xml \
+ 		autogen/cb_notify_deviceid_args.xml \
+ 		autogen/cb_notify_deviceid_res.xml \
  		autogen/cb_push_deleg_args.xml \
  		autogen/cb_push_deleg_res.xml \
  		autogen/cb_recall_any_args.xml \
***************
*** 462,467 ****
--- 464,470 ----
  	nfsv41_middle_op_cb_sequence.xml \
  	nfsv41_middle_op_cb_wants_cancelled.xml \
  	nfsv41_middle_op_cb_notify_lock.xml \
+ 	nfsv41_middle_op_cb_notify_deviceid.xml \
  	nfsv41_middle_op_cb_illegal.xml \
  	nfsv41_middle_op_cb_zzz.xml \
  	nfsv41_middle_security_considerations.xml \
Index: nfsv41_middle_op_cb_layoutrecall.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_layoutrecall.xml,v
retrieving revision 1.16
diff -c -w -b -r1.16 nfsv41_middle_op_cb_layoutrecall.xml
*** nfsv41_middle_op_cb_layoutrecall.xml	10 Nov 2007 04:53:25 -0000	1.16
--- nfsv41_middle_op_cb_layoutrecall.xml	10 Dec 2007 08:19:47 -0000
***************
*** 129,133 ****
--- 129,141 ----
        specify the full range being recalled (see <xref
        target="recall_robustness" /> for details).
      </t>
+     <t>
+       If a server needs to delete a device ID, and there are layouts
+       referring to the device ID, LAYOUTRECALL MUST be invoked to
+       cause the client to return all layouts referring to device ID
+       before the server can delete the device ID. If the client
+       does not return the affected layouts, the server MAY revoke
+       the layouts.
+     </t>
    </section>
  </section>
Index: nfsv41_middle_op_cb_notify.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_notify.xml,v
retrieving revision 1.12
diff -c -w -b -r1.12 nfsv41_middle_op_cb_notify.xml
*** nfsv41_middle_op_cb_notify.xml	10 Nov 2007 04:53:25 -0000	1.12
--- nfsv41_middle_op_cb_notify.xml	10 Dec 2007 08:19:47 -0000
***************
*** 16,44 ****
      <t>
        The CB_NOTIFY operation is used by the server to
        send notifications to clients about changes to
-       certain objects.  NFSv4.1 currently supports two such objects:
- 
-       <list style="symbols">
- 
-       <t>
         delegated directories
!       </t>
! 
!       <t>
!        delegated pNFS device ID to device address mappings.
!       </t>
!       </list>
!       The registration of notifications for the former
        occurs when the delegation is established using
        GET_DIR_DELEGATION.
!       The registration of notifications for the latter
!       occurs when the device mapping stateid is established using
!       GETDEVICEINFO or GETDEVICELIST. In either case,
!       these notifications are sent over the backchannel. The
        notification is sent once the original request has been
        processed on the server. The server will send an array of
        notifications for all changes that might have occurred in the
!       directory. The notify_type4 can only have one bit set
        for each notification in the array (see <xref target="fattr4" />
        for a description of how NFSv4.1 bitmaps works). 
  
--- 16,30 ----
      <t>
        The CB_NOTIFY operation is used by the server to
        send notifications to clients about changes to
        delegated directories
!       The registration of notifications for the direcories
        occurs when the delegation is established using
        GET_DIR_DELEGATION.
!       These notifications are sent over the backchannel. The
        notification is sent once the original request has been
        processed on the server. The server will send an array of
        notifications for all changes that might have occurred in the
!       directory. The notify4 can only have one bit set
        for each notification in the array (see <xref target="fattr4" />
        for a description of how NFSv4.1 bitmaps works). 
  
***************
*** 46,64 ****
  
     <t>
  
!       For directory notifications, if the client holding the
!       delegation makes any changes in the directory that cause files
!       or sub directories to be added or removed, the server will
!       notify that client of the resulting change(s). If the client
!       holding the delegation is making attribute or cookie verifier
!       changes only, the server does not need to send notifications to
!       that client.  The server will send the following information for
        each operation:
  
        <list style="hanging">
! 	<t hangText="ADDING A FILE"><vspace /> 
  	  The server will send
! 	  information about the new entry being created along with the
  	  cookie for that entry.  The entry information (data type
  	  notify_add4) includes the component name of the entry and
  	  attributes.  The server will send this type of entry when a
--- 32,51 ----
  
     <t>
  
!       If the client holding the delegation makes any
!       changes in the directory that cause files or sub
!       directories to be added or removed, the server will
!       notify that client of the resulting change(s). If the
!       client holding the delegation is making attribute
!       or cookie verifier changes only, the server does
!       not need to send notifications to that client.
!       The server will send the following information for
        each operation:
  
        <list style="hanging">
! 	<t hangText="NOTIFY4_ADD_ENTRY"><vspace /> 
  	  The server will send
! 	  information about the new directory entry being created along with the
  	  cookie for that entry.  The entry information (data type
  	  notify_add4) includes the component name of the entry and
  	  attributes.  The server will send this type of entry when a
***************
*** 78,90 ****
  	  nad_new_entry_cookie (another variable length array of up to
  	  one element).
  	</t>
! 	<t hangText="REMOVING A FILE"><vspace />
  	  The server will send information about the directory entry
  	  being deleted. The server will also send the cookie value
  	  for the deleted entry so that clients can get to the cached
  	  information for this entry.
  	</t>
! 	<t hangText="RENAMING A FILE"><vspace />
  	  The server will send information about both
  	  the old entry and the new entry. This includes name and
  	  attributes for each entry. This notification is only sent if
--- 65,77 ----
  	  nad_new_entry_cookie (another variable length array of up to
  	  one element).
  	</t>
! 	<t hangText="NOTIFY4_REMOVE_ENTRY"><vspace />
  	  The server will send information about the directory entry
  	  being deleted. The server will also send the cookie value
  	  for the deleted entry so that clients can get to the cached
  	  information for this entry.
  	</t>
! 	<t hangText="NOTIFY4_RENAME_ENTRY"><vspace />
  	  The server will send information about both
  	  the old entry and the new entry. This includes name and
  	  attributes for each entry. This notification is only sent if
***************
*** 93,99 ****
  	  notification to one directory and an add notification to the
  	  other directory, assuming both have a directory delegation.
  	</t>
! 	<t hangText="FILE/DIR ATTRIBUTE CHANGE"><vspace />
  	  The client will use the attribute
  	  mask to inform the server of attributes for which it wants to
  	  receive notifications. This change notification can be
--- 80,89 ----
  	  notification to one directory and an add notification to the
  	  other directory, assuming both have a directory delegation.
  	</t>
! 	<t
! 	hangText="NOTIFY4_CHANGE_CHILD_ATTRS/NOTIFY4_CHANGE_DIR_ATTRS"><vspace
! 	/>
! 
  	  The client will use the attribute
  	  mask to inform the server of attributes for which it wants to
  	  receive notifications. This change notification can be
***************
*** 113,150 ****
  	  notification is permitted for directory or child attributes
  	  by setting the dir_notif_delay and
  	  dir_entry_notif_delay attributes respectively.
  	</t>
! 	<t hangText="COOKIE VERIFIER CHANGE"><vspace />
  	  If the cookie verifier changes while
  	  a client is holding a delegation, the server will notify the
  	  client so that it can invalidate its cookies and reissue a
  	  READDIR to get the new set of cookies.
  	</t>
        </list>
      </t>
-     <t>
-      For delegations of device ID to device address mappings, the server will sent the
-      following notifications:
-       <list style="hanging">
- 	<t hangText="DEVICE ID ADD"><vspace />
- 	  Adds a device ID to the mappings.  This is used for a device
- 	  that has not been previously returned to the client.  The
- 	  client uses GETDEVICEINFO or GETDEVICELIST to obtain the new
- 	  mapping.
- 	</t>
- 	<t hangText="DEVICE ID CHANGE"><vspace />
- 	  A previously provided device ID mapping has changed and the
- 	  client uses GETDEVICEINFO or GETDEVICELIST to obtain the
- 	  updated mapping.
- 	</t>
- 	<t hangText="DEVICE ID DELETE"><vspace />
- 	  A device ID mapping has been removed and all mappings MUST
- 	  NOT be used.  New mappings are obtained with GETDEVICEINFO
- 	  or GETDEVICELIST.
- 	</t>
-       </list>
-       All device ID to device address mappings present a value of
-       data type notify_deviceid4.
-     </t>
    </section>
  </section>
--- 103,119 ----
  	  notification is permitted for directory or child attributes
  	  by setting the dir_notif_delay and
  	  dir_entry_notif_delay attributes respectively.
+ 
  	</t>
! 	<t hangText="NOTIFY4_CHANGE_COOKIE_VERIFIER"><vspace />
! 
  	  If the cookie verifier changes while
  	  a client is holding a delegation, the server will notify the
  	  client so that it can invalidate its cookies and reissue a
  	  READDIR to get the new set of cookies.
+ 
  	</t>
        </list>
      </t>
    </section>
  </section>
Index: nfsv41_middle_op_getdeviceinfo.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_getdeviceinfo.xml,v
retrieving revision 1.12
diff -c -w -b -r1.12 nfsv41_middle_op_getdeviceinfo.xml
*** nfsv41_middle_op_getdeviceinfo.xml	13 Nov 2007 18:42:21 -0000	1.12
--- nfsv41_middle_op_getdeviceinfo.xml	10 Dec 2007 08:19:47 -0000
***************
*** 32,85 ****
        The notification mask is composed in the same
        manner as the bitmap for file attributes (<xref
        target="fattr4" />).  The numbers of bit positions
!       are listed in the notify_type4 enumeration type
!       (<xref target="OP_GET_DIR_DELEGATION" />). Only
        three enumerated values of notify_type4 currently
        apply to GETDEVICEINFO (and GETDEVICELIST,
        see <xref target="OP_GETDEVICELIST" />):
!       NOTIFY4_DEVICE_ID_ADD NOTIFY4_DEVICE_ID_CHANGE,
!       and NOTIFY4_DEVICE_ID_DELETE (see <xref
!       target="OP_CB_NOTIFY" />).
! 
!       The server responds with
!       gdir_notification, a bitmap of which notifications
!       it will actually send to the server (via CB_NOTIFY
!       (<xref target="OP_CB_NOTIFY" />).
  
      </t>
      <t>
!       If an invalid device ID is given in gdia_device_id, the
!       server returns NFS4ERR_INVAL.  Otherwise, the device address
!       information is returned in gdir_device_addr.  The server also
!       returns the device stateid to be used in subsequent recall or
!       return operations.  Finally, if the server supports
        notifications for device ID mappings, the gdir_notification
!       result will contain the notification types the server will
!       provide for this device.
      </t>
      <t>
        If NFS4ERR_TOOSMALL is returned, the results also contain
        gdir_mincount.  The value of gdir_mincount represents the
        minimum size necessary to obtain the device information.
      </t>
      <t>
!       The current filehandle is updated so that the
!       device stateid may be recalled. Whether this device
!       filehandle maps to a real file or an abstraction in
!       the server or some other mapping is implementation
!       dependent. The server implementation can also return
!       a persistent or volatile filehandle. Clients SHOULD
!       issue COMPOUND requests with the GETFH operation
!       immediately following the GETDEVICEINFO operation so
!       that they can respond to recalls of the delegation
!       and return the delegation. Otherwise, the server is
!       free revoke the delegation.  The server MAY return
!       NFS4ERR_UNSAFECOMPOUND if the client fails to issue
!       GETFH after GETDEVICEINFO. Note that everything
!       discussed in this paragraph applies to GETDEVICELIST
!       too.

      </t>
    </section>
  </section>
--- 32,136 ----
        The notification mask is composed in the same
        manner as the bitmap for file attributes (<xref
        target="fattr4" />).  The numbers of bit positions
!       are listed in the notify_device_type4 enumeration type
!       (<xref target="OP_CB_NOTIFY_DEVICEID" />). Only
        three enumerated values of notify_type4 currently
        apply to GETDEVICEINFO (and GETDEVICELIST,
        see <xref target="OP_GETDEVICELIST" />):
!       NOTIFY_DEVICEID4_ADD NOTIFY_DEVICEID4_CHANGE,
!       and NOTIFY_DEVICEID4_DELETE (see <xref
!       target="OP_CB_NOTIFY_DEVICEID" />).
! 
  
      </t>
      <t>
!       If an invalid device ID is given in gdia_device_id,
!       the server returns NFS4ERR_INVAL. <cref>mre: _INVAL
!       isn't the right error for an unknown device ID</cref>
! 
!       Otherwise, the device address
!       information is returned in gdir_device_addr. 
!       Finally, if the server supports
        notifications for device ID mappings, the gdir_notification
!       result will contain a bitmap of which notifications
!       it will actually send to the server (via CB_NOTIFY_DEVICEID,
!       see <xref target="OP_CB_NOTIFY_DEVICEID" />).
      </t>
      <t>
        If NFS4ERR_TOOSMALL is returned, the results also contain
        gdir_mincount.  The value of gdir_mincount represents the
        minimum size necessary to obtain the device information.
      </t>
+   </section>
+   <section toc="exclude" anchor="OP_GETDEVICEINFO_IMPLEMENTATION" title="IMPLEMENTATION">
+   <t>
+     The client SHOULD request a notification for changes or
+     deletion of a device ID to device address mapping so
+     that the server can allow the client gracefully use a
+     new mapping, without having pending I/O fail abruptly,
+     or force layouts using the device ID to be recalled
+     or revoked.
+ 
+   </t>
+ 
+   <t>
+     It is possible that GETDEVICEINFO (and
+     GETDEVICELIST) will race with CB_NOTIFY_DEVICEID,
+     i.e.  CB_NOTIFY_DEVICEID arrives before the client
+     gets and processes the response to GETDEVICEINFO or
+     GETDEVICELIST.  The analysis of the race leverages the
+     fact that the server MUST NOT delete a device ID that
+     is referred to by a layout the client has.
+ 
+   </t>
+   <t>
+     If CB_NOTIFY_DEVICEID refers to a device ID that is
+     not currently referenced in any layout granted to the
+     client, then the server MAY be adding or deleting a
+     device ID, or changing a device ID's device address
+     mapping.
+ 
+     Regardless, the client will need to issue a possibly
+     redundant GETDEVICEINFO or GETDEVICELIST request after
+     the CB_NOTIFY_DEVICE request is processed.
+ 
+   </t>
    <t>
!     Unless the client has had a layout revoked, if
!     CB_NOTIFY_DEVICEID refers to a device ID is currently
!     referenced in a layout granted to the client then by
!     process of elimination, the notification MUST be for a
!     change of the device ID's device address mappings. The
!     notification cannot be to add a device ID, because the
!     client already knows of the device ID. The notification
!     cannot be to delete a device ID because the server
!     MUST NOT delete a device ID that is referred to by
!     an outstanding layout.  At worst, during the race,
!     the client will use stale device addresses, resulting
!     in failed I/Os.
  
+     Regardless, the client will need to issue a possibly
+     redundant GETDEVICEINFO or GETDEVICELIST request after
+     the CB_NOTIFY_DEVICE request is processed.
  
    </t>
+   <t>
+     If the client had a layout revoked, it is possible
+     the CB_NOTIFY_DEVICE refers to a device ID the
+     client thinks it has a layout for, in which case the
+     notification could be for adding or deleting a device
+     ID. In the case of adding a device ID, the server
+     SHOULD avoid reusing device IDs unless it revokes
+     the client ID. Instead, because device IDs are 16
+     bytes long, the server should some of of the device
+     ID's width for a generation number. Regardless,
+     if the client does get a notification to add or
+     delete a device ID it is already aware of, the
+     client MUST assume the layouts that referred to that
+     device ID have been revoked, and recover (see <xref
+     target="revoke_layout" />).
+ 
+   </t> 
+ 
    </section>
  </section>
Index: nfsv41_middle_op_getdevicelist.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_getdevicelist.xml,v
retrieving revision 1.13
diff -c -w -b -r1.13 nfsv41_middle_op_getdevicelist.xml
*** nfsv41_middle_op_getdevicelist.xml	10 Nov 2007 04:53:25 -0000	1.13
--- nfsv41_middle_op_getdevicelist.xml	10 Dec 2007 08:19:47 -0000
***************
*** 20,28 ****
      </t>
      <t>
        The client provides the layout type in gdia_layout_type.  Since
!       this operation is similar to the <xref target="OP_READDIR">
!       READDIR</xref> operation, the client also provides gdia_cookie
!       and gdia_cookieverf to specify where to obtain device mappings.
        The client provides gdla_maxcount to limit the number of bytes
        for the result.  This maximum size represents all of the data
        being returned within the GETDEVICELIST4resok structure and
--- 20,31 ----
      </t>
      <t>
        The client provides the layout type in gdia_layout_type.  Since
!       this operation may require multiple calls to enumerate all the
!       device ID to device address mappings (and is thus
!       similar to the <xref target="OP_READDIR">
!       READDIR</xref> operation), the client also provides gdia_cookie
!       and gdia_cookieverf to specify the current cursor position in the
!       mappings.
        The client provides gdla_maxcount to limit the number of bytes
        for the result.  This maximum size represents all of the data
        being returned within the GETDEVICELIST4resok structure and
***************
*** 43,51 ****
        cookie, gdlr_cookie, and cookie verifier, gdlr_cookieverf, to be
        used on the subsequent GETDEVICELIST.  A gdlr_eof value of TRUE
        signifies that there are no remaining entries in the server's
!       device list.  Of course, the gdlr_devinfo_list contains the list
!       of device IDs, the corresponding address information, and
!       associated device stateid. The notifications mask is also
        returned by the server in gdlr_notification.
      </t>
      <t>
--- 46,54 ----
        cookie, gdlr_cookie, and cookie verifier, gdlr_cookieverf, to be
        used on the subsequent GETDEVICELIST.  A gdlr_eof value of TRUE
        signifies that there are no remaining entries in the server's
!       device list.  Each element of gdlr_devinfo_list contains
!       a device ID, and the corresponding device address information.
!       The notifications mask is also
        returned by the server in gdlr_notification.
      </t>
      <t>
***************
*** 54,66 ****
        minimum size necessary to obtain a single entry device list
        item.
      </t>
      <t>
!       The current filehandle is updated so that
!       the device stateid may be recalled. See <xref
!       target="OP_GETDEVICEINFO" /> for a discussion of
!       the considerations for constructing and using the
!       device filehandle.
  
      </t>
    </section>
  </section>
--- 57,93 ----
        minimum size necessary to obtain a single entry device list
        item.
      </t>
+   </section>
+   <section toc="exclude" anchor="OP_GETDEVICELIST_IMPLEMENTATION" title="IMPLEMENTATION">
+   <t>
+     The client SHOULD request a notification for additions,
+     changes, deletions of to device ID to device address
+     mappings so that:
+ 
+     <list style="symbols">
      <t>
! 
!       The server can allow the client gracefully use new
!       mappings, without having pending I/O fail abruptly,
!       or force layouts using the device IDs to be recalled
!       or revoked.
  
      </t>
+     <t>
+ 
+       The client can maintain an up to date listing of
+       all device IDs and their device addresses by invoke
+       GETDEVICEINFO on single device IDs versus making
+       periodic calls to GETDEVICELIST.
+ 
+     </t>
+     </list>
+ 
+   </t>
+   <t>
+    See <xref target="OP_GETDEVICEINFO_IMPLEMENTATION"/> for a discussion
+    of races between GETDEVICELIST and CB_NOTIFY_DEVICEID.
+   </t>
+ 
    </section>
  </section>
Index: nfsv41_middle_op_layoutget.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutget.xml,v
retrieving revision 1.16
diff -c -w -b -r1.16 nfsv41_middle_op_layoutget.xml
*** nfsv41_middle_op_layoutget.xml	15 Nov 2007 13:21:22 -0000	1.16
--- nfsv41_middle_op_layoutget.xml	10 Dec 2007 08:19:47 -0000
***************
*** 62,67 ****
--- 62,72 ----
       logr_layout will contain just one entry. Otherwise, if the
       requested range overlaps more than one striping pattern,
       logr_layout will contain the required number of entries.
+      Each element of logr_layout MUST have the same iomode. The elements
+      of logr_layout MUST be sorted in ascending order of the value of
+      lo_offset field of each element. There MUST be no gaps in the
+      range between two successive elements of logr_layout. The lo_iomode
+      field in each element of logr_layout MUST be the same.
      </t>
      <t>
        The metadata server may adjust
***************
*** 97,103 ****
        For the server that knows a layout must be returned before a
        close of the file, this return value can be used to communicate
        the desired behavior to the client and thus removing one extra
!       step from the client and server's interaction.
      </t>
      <t>
        The logr_stateid, as with all stateid processing, is returned to
--- 102,108 ----
        For the server that knows a layout must be returned before a
        close of the file, this return value can be used to communicate
        the desired behavior to the client and thus removing one extra
!       step from the client's and server's interaction.
      </t>
      <t>
        The logr_stateid, as with all stateid processing, is returned to
***************
*** 160,164 ****
--- 165,174 ----
        kind of layout the server will return.  This prevents servers from
        issuing layouts that are unusable by the client.
      </t>
+     <t>
+       Once the client has a obtained a layout referring to a particular
+       device ID, the server MUST NOT delete the device ID until the
+       layout is returned or revoked.
+     </t>
    </section>
  </section>
Index: nfsv41_middle_op_layoutreturn.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutreturn.xml,v
retrieving revision 1.19
diff -c -w -b -r1.19 nfsv41_middle_op_layoutreturn.xml
*** nfsv41_middle_op_layoutreturn.xml	19 Nov 2007 20:33:12 -0000	1.19
--- nfsv41_middle_op_layoutreturn.xml	10 Dec 2007 08:19:47 -0000
***************
*** 170,174 ****
--- 170,178 ----
      LAYOUTRETURN sequence.  See <xref target="recall_robustness" />
      for more details.
    </t>
+   <t>
+     Once the client has returned all layouts referring to a particular
+     device ID, the server MAY delete the device ID.
+   </t>
    </section>
  </section>
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.53
diff -c -w -b -r1.53 nfsv41_middle_pnfs.xml
*** nfsv41_middle_pnfs.xml	9 Dec 2007 18:13:37 -0000	1.53
--- nfsv41_middle_pnfs.xml	10 Dec 2007 08:19:48 -0000
***************
*** 273,279 ****
  <section title="Device IDs" anchor="device_ids">
    <t>
      The device ID (data type deviceid4, see
!     <xref target="deviceid4"/>) names a storage device. In practice, a
      significant amount of information may be required to fully address
      a storage device.  Rather than embedding all such information in a
      layout, layouts embed device IDs.  The NFSv4.1 operation
--- 273,280 ----
  <section title="Device IDs" anchor="device_ids">
    <t>
      The device ID (data type deviceid4, see
!     <xref target="deviceid4"/>) names a storage device. The scope
!     of a device ID is per pair of client ID and layout type. In practice, a
      significant amount of information may be required to fully address
      a storage device.  Rather than embedding all such information in a
      layout, layouts embed device IDs.  The NFSv4.1 operation
***************
*** 291,312 ****
      recovery works in that situation.
    </t>
    <t>
!     To clearly define the lifetime of a device ID to storage address
!     mapping and to assist in modifications to those mappings over the
!     duration of server operation, the stateid mechanism is applied to
!     device IDs.  Device ID mappings represent another form of stateid
!     <xref target="stateid_types" />.  The GETDEVICEINFO and
!     GETDEVICELIST operations each return a device stateid.  Like file
!     delegations, the device stateid is recallable.  A recall of the
!     device stateid will remove or invalidate the device ID mappings as
!     well as lease expiration.  The GETDEVICEINFO and GETDEVICELIST
!     operations update the current filehandle to facilitate the recall
!     of the device stateid.  To reduce the need to recall the device ID
!     stateid during mapping modifications, the notifications mechanism
!     may be used by the server to update the client on changes that
!     occur.  The server must support notifications and the client must
      request them before they can be used.  For further information
!     about the notification types <xref target="OP_CB_NOTIFY" />.
    </t>
  </section>
  
--- 292,312 ----
      recovery works in that situation.
    </t>
    <t>
!     A device ID to device address mapping lives as long as
!     there is a layout referring to the device ID.
!     Via a notification mechanism
!     (see <xref target="OP_CB_NOTIFY_DEVICEID" />),
!     device ID to device address mappings can change over the duration
!     of server operation without recalling or revoking the layouts that
!     refer to device ID. The notification mechanism can also delete
!     a device ID, but only if the client has no layouts referring
!     to the device ID.
!     A notification of a change to a device ID to device address
!     mapping will immediately or eventually invalidate some or all of
!     the device ID's mappings.
!     The server MUST support notifications and the client must
      request them before they can be used.  For further information
!     about the notification types <xref target="OP_CB_NOTIFY_DEVICEID" />.
    </t>
  </section>
  
***************
*** 758,763 ****
--- 758,764 ----
  <list style="symbols">
  <t>
    It may be useful for clients to be able to discard layout
+ <cref>mre: wrong</cref>
    information without calling LAYOUTRETURN.  If conflicts that require
    callbacks are very rare, and a server can use a multi-file callback
    to recover per-client resources (e.g., via a FSID recall, or a
***************
*** 1007,1013 ****
--- 1008,1019 ----
  </section>
  </section>
  
+ <section title="Revoking Layouts" anchor="revoke_layout">
+ <t>
+ <cref>mre: need text. Since layouts have stateids, revocation is not layout type specfic.</cref>
+ </t>
  
+ </section>
  <section title="Metadata Server Write Propagation" anchor="async_writes">
  <t>
    Asynchronous writes written through the metadata server may be
***************
*** 1127,1132 ****
--- 1133,1139 ----
    server and the fact that unlike delegations, layouts are not bound
    to stateids introduces subtleties that must be handled correctly if
    the possibility of file system corruption is to be avoided.
+   <cref>mre: layouts are bound to stateids</cref>
  </t>
   <section title="Client Recovery" anchor="pnfs_client_recovery">
   <t>
Index: dotx.d/Makefile
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/Makefile,v
retrieving revision 1.39
diff -c -w -b -r1.39 Makefile
*** dotx.d/Makefile	21 Nov 2007 01:12:05 -0000	1.39
--- dotx.d/Makefile	10 Dec 2007 08:19:48 -0000
***************
*** 129,135 ****
  		cb_wants_cancelled_args.x \
  		cb_wants_cancelled_res.x \
  		cb_notify_lock_args.x \
! 		cb_notify_lock_res.x
  
  SPITGEN =	basic_types.x \
  		type_nfstime4.x \
--- 129,137 ----
  		cb_wants_cancelled_args.x \
  		cb_wants_cancelled_res.x \
  		cb_notify_lock_args.x \
! 		cb_notify_lock_res.x \
! 		cb_notify_deviceid_args.x \
! 		cb_notify_deviceid_res.x
  
  SPITGEN =	basic_types.x \
  		type_nfstime4.x \
Index: dotx.d/cb_notify_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/cb_notify_args.x,v
retrieving revision 1.7
diff -c -w -b -r1.7 cb_notify_args.x
*** dotx.d/cb_notify_args.x	10 Nov 2007 04:53:32 -0000	1.7
--- dotx.d/cb_notify_args.x	10 Dec 2007 08:19:48 -0000
***************
*** 1,3 ****
--- 1,15 ----
+ /*
+  * Directory notification types.
+  */
+ enum notify_type4 {
+ 	NOTIFY4_CHANGE_CHILD_ATTRS = 0,
+ 	NOTIFY4_CHANGE_DIR_ATTRS = 1,
+ 	NOTIFY4_REMOVE_ENTRY = 2,
+ 	NOTIFY4_ADD_ENTRY = 3,
+ 	NOTIFY4_RENAME_ENTRY = 4,
+ 	NOTIFY4_CHANGE_COOKIE_VERIFIER = 5
+ };
+ 
  /* Changed entry information.  */
  struct notify_entry4 {
  	component4	ne_file;
***************
*** 38,46 ****
  	verifier4	nv_new_cookieverf;
  };
  
! typedef deviceid4 notify_deviceid4;
! 
! /* Objects of type struct notify_<>4 are encoded in this */
  typedef opaque notifylist4<>;
  
  struct notify4 {
--- 50,56 ----
  	verifier4	nv_new_cookieverf;
  };
  
! /* Objects of type notify_<>4 and notify_device_<>4 are encoded in this */
  typedef opaque notifylist4<>;
  
  struct notify4 {
Index: dotx.d/get_dir_delegation_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/get_dir_delegation_args.x,v
retrieving revision 1.10
diff -c -w -b -r1.10 get_dir_delegation_args.x
*** dotx.d/get_dir_delegation_args.x	19 Nov 2007 09:56:44 -0000	1.10
--- dotx.d/get_dir_delegation_args.x	10 Dec 2007 08:19:48 -0000
***************
*** 1,19 ****
  /*
-  * Notification types.
-  */
- enum notify_type4 {
- 	NOTIFY4_CHANGE_CHILD_ATTRS = 0,
- 	NOTIFY4_CHANGE_DIR_ATTRS = 1,
- 	NOTIFY4_REMOVE_ENTRY = 2,
- 	NOTIFY4_ADD_ENTRY = 3,
- 	NOTIFY4_RENAME_ENTRY = 4,
- 	NOTIFY4_CHANGE_COOKIE_VERIFIER = 5,
- 	NOTIFY4_DEVICE_ID_ADD = 6,
- 	NOTIFY4_DEVICE_ID_CHANGE = 7,
- 	NOTIFY4_DEVICE_ID_DELETE = 8
- };
- 
- /*
   * Mask of notification types.
   */
  typedef bitmap4 notification_mask4;
--- 1,4 ----
Index: dotx.d/getdeviceinfo_res.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/getdeviceinfo_res.x,v
retrieving revision 1.6
diff -c -w -b -r1.6 getdeviceinfo_res.x
*** dotx.d/getdeviceinfo_res.x	19 Nov 2007 09:56:44 -0000	1.6
--- dotx.d/getdeviceinfo_res.x	10 Dec 2007 08:19:48 -0000
***************
*** 1,5 ****
  struct GETDEVICEINFO4resok {
-         stateid4		gdir_stateid;
          device_addr4		gdir_device_addr;
  	notification_mask4	gdir_notification;
  };
--- 1,4 ----
Index: dotx.d/head.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/head.x,v
retrieving revision 1.49
diff -c -w -b -r1.49 head.x
*** dotx.d/head.x	9 Dec 2007 18:13:41 -0000	1.49
--- dotx.d/head.x	10 Dec 2007 08:19:48 -0000
***************
*** 33,38 ****
--- 33,39 ----
  const NFS4_FHSIZE		= 128;
  const NFS4_VERIFIER_SIZE	= 8;
  const NFS4_OPAQUE_LIMIT		= 1024;
+ const NFS4_SESSIONID_SIZE	= 16;
  
  /*
   * File types
Index: dotx.d/spit_types.sh
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/spit_types.sh,v
retrieving revision 1.37
diff -c -w -b -r1.37 spit_types.sh
*** dotx.d/spit_types.sh	8 Dec 2007 05:47:58 -0000	1.37
--- dotx.d/spit_types.sh	10 Dec 2007 08:19:48 -0000
***************
*** 32,38 ****
  typedef :opaque:sec_oid4<>:Security Object Identifier The sec_oid4 data type is not really opaque. Instead it contains an ASN.1 OBJECT IDENTIFIER as used by GSS-API in the mech_type argument to GSS_Init_sec_context. See <xref target="RFC2743" /> for details.
  typedef :uint32_t:sequenceid4:sequence number used for various session operations (EXCHANGE_ID, CREATE_SESSION, SEQUENCE, CB_SEQUENCE).
  typedef :uint32_t:seqid4:Sequence identifier used for file locking
! typedef :opaque:sessionid4[16]:Session identifier
  typedef :uint32_t:slotid4:sequencing artifact for various session operations (SEQUENCE, CB_SEQUENCE).
  typedef :opaque:utf8string<>:UTF-8 encoding for strings
  typedef :utf8string:utf8str_cis:Case-insensitive UTF-8 string
--- 32,38 ----
  typedef :opaque:sec_oid4<>:Security Object Identifier The sec_oid4 data type is not really opaque. Instead it contains an ASN.1 OBJECT IDENTIFIER as used by GSS-API in the mech_type argument to GSS_Init_sec_context. See <xref target="RFC2743" /> for details.
  typedef :uint32_t:sequenceid4:sequence number used for various session operations (EXCHANGE_ID, CREATE_SESSION, SEQUENCE, CB_SEQUENCE).
  typedef :uint32_t:seqid4:Sequence identifier used for file locking
! typedef :opaque:sessionid4[NFS4_SESSIONID_SIZE]:Session identifier
  typedef :uint32_t:slotid4:sequencing artifact for various session operations (SEQUENCE, CB_SEQUENCE).
  typedef :opaque:utf8string<>:UTF-8 encoding for strings
  typedef :utf8string:utf8str_cis:Case-insensitive UTF-8 string
***************
*** 260,269 ****
  	type_deviceid4.x )
  
  cat << EOF > $i
! struct deviceid4 {
! 	uint64_t did_major;
! 	uint64_t did_minor;
! };
  EOF
  	;;
  
--- 260,268 ----
  	type_deviceid4.x )
  
  cat << EOF > $i
! const NFS4_DEVICEID4_SIZE = 16;
! 
! typedef opaque  deviceid4[NFS4_DEVICEID4_SIZE];
  EOF
  	;;
  
***************
*** 440,446 ****
  cat << EOF > $i
  struct devlist_item4 {
          deviceid4                 dli_id;
- 	stateid4		  dli_stateid;
          device_addr4              dli_device_addr;
  };
  EOF
--- 439,444 ----
***************
*** 753,758 ****
--- 751,757 ----
  	OP_CB_SEQUENCE		= 11,
  	OP_CB_WANTS_CANCELLED	= 12,
          OP_CB_NOTIFY_LOCK       = 13,
+         OP_CB_NOTIFY_DEVICEID   = 14,
  
  	OP_CB_ILLEGAL		= 10044
  };
***************
*** 785,790 ****
--- 784,791 ----
        CB_WANTS_CANCELLED4args   opcbwants_cancelled;
   case OP_CB_NOTIFY_LOCK:
        CB_NOTIFY_LOCK4args       opcbnotify_lock;
+  case OP_CB_NOTIFY_DEVICEID:
+       CB_NOTIFY_DEVICEID4args   opcbnotify_deviceid;
   case OP_CB_ILLEGAL:            void;
  };
  EOF
***************
*** 840,845 ****
--- 841,850 ----
  			CB_NOTIFY_LOCK4res
  					opcbnotify_lock;
  
+  case OP_CB_NOTIFY_DEVICEID:
+ 			CB_NOTIFY_DEVICEID4res
+ 					opcbnotify_deviceid;
+ 
   /* Not new operation */
   case OP_CB_ILLEGAL:	CB_ILLEGAL4res	opcbillegal;
  };
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Dec 2007 00:59:38 -0800 (PST)

> CB_NOTIFY_DEVICEID operation) that I will check in later today.

The .x files did not come over well. I've fixed them (I think)
and re-attached them.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 10 Dec 2007 17:47:54 +0200

!       directory. The notify4 can only have one bit set
        for each notification in the array (see <xref target="fattr4" />
        for a description of how NFSv4.1 bitmaps works). 
If only one bit is allowed to be set for each notification in the array
then why use a bitmap? A simple type field will be much simpler, no?

                             ^^^^^^^^^^^^^^^^^
typo?

1. slight phrasing issue: the client may be aware of a deleted
device ID while having no layout refer to it so "aware" is
probably a poor choice.

How about "if the client does get a notification to delete
a device ID and the client is aware of layouts still referring to 
it, the client MUST assume ..."

2. CB_NOTIFY_DEVICE may race also with LAYOUTGET which may
return a new device ID.  What's the issue with getting an
update for a new device ID the client is already aware of?
Can't the client handle this exactly like the update case
or just ignore the update?

Benny

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Dec 2007 11:51:55 -0800 (PST)

> then why use a bitmap? A simple type field will be much simpler, no?

I changed it to use multiple bits in the bitmaps.

> +     bytes long, the server should some of of the device

yes.

> or just ignore the update?

Good point. I looked at that issue, and found a race condition
involving
LAYOUTGET, CB_NOTIFY_DEVICE, and GETDEVICEINFO.

I concluded that the race conditions are much simpler if the spec
mandates that once a device ID is deleted, the server MUST NOT 
re-use the device ID (for that client ID, layout type pair).
This made describing the race conditions in GETDEVICEINFO and
LAYOUTGET easier.

If people are unhappy with no re-use of device IDs, then we
can make device IDs as big as people want. 

A new set of diffs (incremental to what was last emailed)
is attached.

Index: nfsv41_middle_core_infrastructure.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_core_infrastructure.xml,v
retrieving revision 1.72
diff -u -w -b -r1.72 nfsv41_middle_core_infrastructure.xml
--- nfsv41_middle_core_infrastructure.xml	28 Nov 2007 20:04:09 -0000	1.72
+++ nfsv41_middle_core_infrastructure.xml	10 Dec 2007 19:36:19 -0000
@@ -1976,7 +1976,7 @@
      bit unsigned value, and is therefore in the range 0..0xFFFFFFFF (2^32 - 1).
      The first time a slot is used, the requester must specify
      a sequence id of one (1) (<xref target="OP_CREATE_SESSION" />).
-     Each time a slot is re-used, the request MUST specify a sequence id
+     Each time a slot is reused, the request MUST specify a sequence id
      that is one greater than that of the previous request on the
      slot. If the previous sequence id was 0xFFFFFFFF, then the next
      request for the slot MUST have the sequence id set to zero (i.e.
Index: nfsv41_middle_op_cb_notify.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_notify.xml,v
retrieving revision 1.13
diff -u -w -b -r1.13 nfsv41_middle_op_cb_notify.xml
--- nfsv41_middle_op_cb_notify.xml	10 Dec 2007 09:27:06 -0000	1.13
+++ nfsv41_middle_op_cb_notify.xml	10 Dec 2007 19:36:19 -0000
@@ -24,9 +24,10 @@
       notification is sent once the original request has been
       processed on the server. The server will send an array of
       notifications for all changes that might have occurred in the
-      directory. The notify4 can only have one bit set
-      for each notification in the array (see <xref target="fattr4" />
-      for a description of how NFSv4.1 bitmaps works). 
+      directory. The notifications are sent as list of pairs of
+      bitmaps and values.
+      See <xref target="fattr4" />
+      for a description of how NFSv4.1 bitmaps work.
 
    </t>
 
Index: nfsv41_middle_op_cb_notify_deviceid.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_notify_deviceid.xml,v
retrieving revision 1.1
diff -u -w -b -r1.1 nfsv41_middle_op_cb_notify_deviceid.xml
--- nfsv41_middle_op_cb_notify_deviceid.xml	10 Dec 2007 09:27:06 -0000	1.1
+++ nfsv41_middle_op_cb_notify_deviceid.xml	10 Dec 2007 19:36:19 -0000
@@ -23,9 +23,9 @@
       over the backchannel. The notification is sent
       once the original request has been processed
       on the server. The server will send an array of
-      notifications, cnda_changes, as a bitmap.  See <xref
-      target="fattr4" /> for a description of how NFSv4.1
-      bitmaps work.
+      notifications, cnda_changes, as a list of pairs of
+      bitmaps and values.  See <xref target="fattr4" />
+      for a description of how NFSv4.1 bitmaps work.
 
    </t>
 
@@ -47,7 +47,7 @@
 	  used for a device that has not been previously
 	  returned to the client.  The client uses
 	  GETDEVICEINFO or GETDEVICELIST to obtain the new
-	  mapping. The notification is encoded in value of
+	  mapping. The notification is encoded in a value of
 	  data type notify_deviceid_add_or_delete4.
 
 	</t>
@@ -60,44 +60,52 @@
 
           The notification is encoded in a value of data
           type notify_deviceid_chg_or_del4. This data type
-          also contains a boolean field, ndd_immediate,
+          also contains a boolean field, ndc_immediate,
           which if TRUE indicates that the change will be
           enforced immediately, and so the client might not
           be able to complete any pending I/O to the device
-          ID. If ndcod_immediate is FALSE, then for an
+          ID. If ndc_immediate is FALSE, then for an
           indefinite time, the client can complete pending
           I/O. After pending I/O is complete, the client
           SHOULD get the new device ID to device addrress
           mappings before issuing new I/O to the device ID.
 
+	  The notification is encoded in a value of
+	  data type notify_deviceid_change4.
+
 	</t>
 
 	<t hangText="NOTIFY4_DEVICEID_DELETE"><vspace />
 	  Deletes a device ID from the mappings.  This
 	  notification MUST NOT be sent if the client has
 	  a layout that refers to the device ID. In other
-	  words one of the following is true for layouts
+	  words if the server is sending a delete device ID
+          notification, one of the following is true for layouts
 	  associated with the layout type:
 
           <list style="symbols">
 
           <t>
-           The client never had a layout for the device ID.
+           The client never had a layout referring to that device ID.
           </t>
           
           <t>
-           The client has returned all layouts with that device ID.
+           The client has returned all layouts referring to that device ID.
           </t>
 
           <t>
-           The server has revoked all layouts with that device ID.
+           The server has revoked all layouts referring to that device ID.
           </t>
 
           </list>
 
-	  The notification is encoded in value of data
+	  The notification is encoded in a value of data
 	  type notify_deviceid_add_or_delete4.
 
+          After a server deletes a device ID, it MUST NOT
+          reuse that device ID for the same layout type until the
+          client ID is deleted.
+
 	</t>
       </list>
     </t>
Index: nfsv41_middle_op_getdeviceinfo.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_getdeviceinfo.xml,v
retrieving revision 1.13
diff -u -w -b -r1.13 nfsv41_middle_op_getdeviceinfo.xml
--- nfsv41_middle_op_getdeviceinfo.xml	10 Dec 2007 09:27:06 -0000	1.13
+++ nfsv41_middle_op_getdeviceinfo.xml	10 Dec 2007 19:36:19 -0000
@@ -81,56 +81,68 @@
     GETDEVICELIST.  The analysis of the race leverages the
     fact that the server MUST NOT delete a device ID that
     is referred to by a layout the client has.
-
-  </t>
+    <list style="symbols">
   <t>
-    If CB_NOTIFY_DEVICEID refers to a device ID that is
-    not currently referenced in any layout granted to the
-    client, then the server MAY be adding or deleting a
-    device ID, or changing a device ID's device address
-    mapping.
-
-    Regardless, the client will need to issue a possibly
-    redundant GETDEVICEINFO or GETDEVICELIST request after
-    the CB_NOTIFY_DEVICE request is processed.
+       CB_NOTIFY_DEVICEID deletes a device ID.
+       If the client believes it has layouts that refer to the
+       device ID, then it is possible the layouts have been revoked.
+       The client should send a TEST_STATEID request using the
+       stateid for each layout that might have been revoked. If
+       TEST_STATEID indicates any layouts have been revoked, the
+       client must recover from layout revocation as described in
+       (see <xref
+       target="revoke_layout" />). If TEST_STATEID indicates at least
+       one layout has not been revoked, the client should issue
+       a GETDEVICEINFO on the device ID to verify that the device ID
+       has been deleted. If GETDEVICEINFO indicates the device ID
+       does not exist, the client then assumes the server is broken,
+       and recovers issuing EXCHANGE_ID.
+   
+       If the client does not have layouts that refer to the
+       device ID, no harm is done.
+       The client should mark the device ID as deleted, and when
+       the GETDEVICEINFO or GETDEVICELIST results are finally
+       received for the device ID, delete the device ID from
+       client's cache.
 
   </t>
-  <t>
-    Unless the client has had a layout revoked, if
-    CB_NOTIFY_DEVICEID refers to a device ID is currently
-    referenced in a layout granted to the client then by
-    process of elimination, the notification MUST be for a
-    change of the device ID's device address mappings. The
-    notification cannot be to add a device ID, because the
-    client already knows of the device ID. The notification
-    cannot be to delete a device ID because the server
-    MUST NOT delete a device ID that is referred to by
-    an outstanding layout.  At worst, during the race,
-    the client will use stale device addresses, resulting
-    in failed I/Os.
-
-    Regardless, the client will need to issue a possibly
-    redundant GETDEVICEINFO or GETDEVICELIST request after
-    the CB_NOTIFY_DEVICE request is processed.
 
+    <t>
+       CB_NOTIFY_DEVICEID adds a device ID.  If the client
+       believes it has layouts that refer to the device
+       then the client likely received the
+       device ID in a recent response to a LAYOUTGET,
+       and CB_NOTIFY_DEVICEID is racing with LAYOUTGET
+       as well as GETDEVICEINFO. The client can ignore
+       the CB_NOTIFY_ADD.
+
+       If the client does not have layouts that refer to
+       the added device ID, and GETDEVICEINFO (and not
+       GETDEVICELIST) is racing with CB_NOTIFY_DEVICEID,
+       this begs the question why client is issuing a
+       GETDEVICEINFO on the device ID. A legitimate
+       scenario is that the layout was recalled and
+       returned before the layout could be used. An
+       associated scenario is that the server is also
+       sending a CB_NOTIFY_DEVICEID to delete the device
+       ID, yet another race.  The client should deal with
+       the race by issuing GETDEVICEINFO on the device ID
+       and update (or delete, if GETDEVICEINFO indicates
+       the device ID is not in use) the device ID's device
+       address mappings.
   </t>
+
   <t>
-    If the client had a layout revoked, it is possible
-    the CB_NOTIFY_DEVICE refers to a device ID the
-    client thinks it has a layout for, in which case the
-    notification could be for adding or deleting a device
-    ID. In the case of adding a device ID, the server
-    SHOULD avoid reusing device IDs unless it revokes
-    the client ID. Instead, because device IDs are 16
-    bytes long, the server should some of of the device
-    ID's width for a generation number. Regardless,
-    if the client does get a notification to add or
-    delete a device ID it is already aware of, the
-    client MUST assume the layouts that referred to that
-    device ID have been revoked, and recover (see <xref
-    target="revoke_layout" />).
+       CB_NOTIFY_DEVICEID indicates a device ID's device
+       addressing mappings have changed. The client should assume
+       that the results from the in progress GETDEVICEINFO or
+       GETDEVICELIST will be stale for the device ID
+       once received, and so it should issue a GETDEVICEINFO
+       on the device ID.
 
   </t> 
+   </list>
+  </t> 
 
   </section>
 </section>
Index: nfsv41_middle_op_layoutget.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutget.xml,v
retrieving revision 1.17
diff -u -w -b -r1.17 nfsv41_middle_op_layoutget.xml
--- nfsv41_middle_op_layoutget.xml	10 Dec 2007 09:27:06 -0000	1.17
+++ nfsv41_middle_op_layoutget.xml	10 Dec 2007 19:36:19 -0000
@@ -16,7 +16,7 @@
       byte range specified by offset and length.  Layouts are
       identified by the client ID (derived from the sessionid in the
       preceding SEQUENCE operation), current filehandle, layout type
-      (loga_layout_type), and the layout stateid (logo_stateid).  The
+      (loga_layout_type), and the layout stateid (loga_stateid).  The
       use of the loga_iomode depends upon the layout type, but should
       reflect the client's data access intent.
     </t>
@@ -166,9 +166,27 @@
       issuing layouts that are unusable by the client.
     </t>
     <t>
-      Once the client has a obtained a layout referring to a particular
+      Once the client has obtained a layout referring to a particular
       device ID, the server MUST NOT delete the device ID until the
       layout is returned or revoked.
     </t>
+    <t>
+      CB_NOTIFY_DEVICEID can race with LAYOUTGET. One race scenario is that
+      LAYOUTGET returns a device ID the client does not have device
+      address mappings for, and the server sends a CB_NOTIFY_DEVICEID
+      to add the device ID to the client's awareness and meanwhile
+      the client issues GETDEVICEINFO on the device ID.
+      This scenario is discussed in
+      <xref target="OP_GETDEVICEINFO_IMPLEMENTATION" />.
+      Another scenario is that the CB_NOTIFY_DEVICEID is processed
+      by the client before it processes the results from LAYOUTGET.
+      The client will issue a GETDEVICEINFO on the device ID.
+      If the results from GETDEVICEINFO are received before the
+      client gets results from LAYTOUTGET, then there is no longer a
+      race. If the results from LAYOUTGET are received before the results
+      from GETDEVICEINFO, the client can either wait for results
+      of GETDEVICEINFO, or issue another one to get possibly
+      more up to date device address mappings for the device ID.
+    </t>
   </section>
 </section>
Index: nfsv41_middle_op_setclientid_confirm.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_setclientid_confirm.xml,v
retrieving revision 1.5
diff -u -w -b -r1.5 nfsv41_middle_op_setclientid_confirm.xml
--- nfsv41_middle_op_setclientid_confirm.xml	1 Mar 2007 23:52:22 -0000	1.5
+++ nfsv41_middle_op_setclientid_confirm.xml	10 Dec 2007 19:36:19 -0000
@@ -47,7 +47,7 @@
 	  just include record locks and share reservations.
 	</t>
 	<t>
-	  The client's re-use of an old, previously confirmed, shorthand client
+	  The client's reuse of an old, previously confirmed, shorthand client
 	  identifier, a new callback value, and a new callback_ident value.  The
 	  client's use of SETCLIENTID_CONFIRM in this case MUST NOT result in
 	  the removal of any previous leased state (locks, share reservations,
Index: nfsv41_middle_op_test_stateid.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_test_stateid.xml,v
retrieving revision 1.8
diff -u -w -b -r1.8 nfsv41_middle_op_test_stateid.xml
--- nfsv41_middle_op_test_stateid.xml	25 Nov 2007 00:35:58 -0000	1.8
+++ nfsv41_middle_op_test_stateid.xml	10 Dec 2007 19:36:20 -0000
@@ -71,6 +71,7 @@
       The errors which are validly returned within the status_code array 
       are: NFS4ERR_OK, NFS4ERR_BAD_STATEID, NFS4ERR_OLD_STATEID, 
       NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, and NFS4ERR_DELEG_REVOKED.
+     <cref>_LAYOUT_REVOKED</cref>.
     </t>
   </section>
   <section toc="exclude" anchor="OP_TEST_STATEID_IMPLEMENTATION" title="IMPLEMENTATION">
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.54
diff -u -w -b -r1.54 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	10 Dec 2007 09:27:06 -0000	1.54
+++ nfsv41_middle_pnfs.xml	10 Dec 2007 19:36:22 -0000
@@ -308,6 +308,17 @@
     request them before they can be used.  For further information
     about the notification types <xref target="OP_CB_NOTIFY_DEVICEID" />.
   </t>
+  <t>
+    Once a device ID is deleted by the server, the server MUST NOT
+    reuse the device ID for the same layout type and client ID again.
+    This requirement is feasible because the device ID is 16 bytes
+    long, leaving sufficient room to store a generation number if
+    server's implementation requires most of the rest of the device ID's
+    content to be reused. This requirement is necessary because
+    otherwise the race conditions between asynchronous notification
+    of device ID addition and deletion would be too difficult to
+    sort out.
+  </t>
 </section>
 
 </section>
@@ -348,7 +359,7 @@
  </list>
 </t>
 <t>
-  The pNFS client MUST implement these callback operations.  A pNFS
+  The pNFS client and server MUST implement these callback operations.  A pNFS
   server MAY use them as appropriate.  Also note that recalls can be
   expected for the various layout types such as object and
   block/volume layouts.
@@ -372,6 +383,17 @@
  </t>
  </list>
 </t>
+<t>
+  The pNFS client MAY implement this callback operation, and the
+  pNFS server MUST implement it.
+
+  <list style='hanging'>
+  <t hangText="CB_NOTIFY_DEVICEID">
+   (<xref target="OP_CB_NOTIFY_DEVICEID" />) Notifies the client of
+   changes to device IDs.
+ </t>
+ </list>
+</t>
 
 </section>
 <section title="pNFS Attributes" anchor="pnfs_attr">
Index: dotx.d/cb_notify_deviceid_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/cb_notify_deviceid_args.x,v
retrieving revision 1.1
diff -u -w -b -r1.1 cb_notify_deviceid_args.x
--- dotx.d/cb_notify_deviceid_args.x	10 Dec 2007 09:27:14 -0000	1.1
+++ dotx.d/cb_notify_deviceid_args.x	10 Dec 2007 19:36:22 -0000
@@ -14,10 +14,10 @@
 };

 /* For NOTIFY4_DEVICEID4_CHANGE */

-struct notify_deviceid_delete4 {

-	layouttype4	ndd_layouttype;

-	deviceid4	ndd_deviceid;

-	bool		ndd_immediate;

+struct notify_deviceid_change4 {

+	layouttype4	ndc_layouttype;

+	deviceid4	ndc_deviceid;

+	bool		ndc_immediate;

 };

 struct CB_NOTIFY_DEVICEID4args {

---

From: Benny Halevy <[email-masked]>
Date: Thu, 13 Dec 2007 08:54:05 +0200

> can make device IDs as big as people want. 

I have no problem with not reusing deleted device IDs (for the record :)

> A new set of diffs (incremental to what was last emailed)

The diffs look good.
Thanks.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Fri, 14 Dec 2007 19:42:10 -0800 (PST)

> Mike: Need common text in spec.

Attached are my diffs for this issue.

Index: nfsv41_middle_errors.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_errors.xml,v
retrieving revision 1.48
diff -w -b -u -r1.48 nfsv41_middle_errors.xml
--- nfsv41_middle_errors.xml	5 Dec 2007 15:05:23 -0000	1.48
+++ nfsv41_middle_errors.xml	15 Dec 2007 03:33:26 -0000
@@ -354,6 +354,10 @@
       <c>10068</c>
       <c><xref target="err_RETRY_UNCACHED_REP" /></c>
 
+      <c>NFS4ERR_RETURNCONFLICT</c>
+      <c>10086</c>
+      <c><xref target="err_RETURNCONFLICT" /></c>
+
       <c>NFS4ERR_ROFS</c>
       <c>30</c>
       <c><xref target="err_ROFS" /></c>
@@ -1191,6 +1195,16 @@
         layout.
       </t>
     </section>
+    <section title="NFS4ERR_RETURNCONFLICT (Error Code 10086)" 
+             anchor="err_RETURNCONFLICT">
+      <t>
+        A layout
+        is unavailable due to an attempt to perform the LAYOUTGET
+        before a pending LAYOUTRETURN on the file has been received.
+        See <xref target="layout_server_consider" />.
+
+      </t>
+    </section>
     <section title="NFS4ERR_UNKNOWN_LAYOUTTYPE (Error Code 10062)" 
              anchor="err_UNKNOWN_LAYOUTTYPE">
       <t>
Index: nfsv41_middle_file_layout.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_file_layout.xml,v
retrieving revision 1.28
diff -w -b -u -r1.28 nfsv41_middle_file_layout.xml
--- nfsv41_middle_file_layout.xml	8 Dec 2007 05:47:53 -0000	1.28
+++ nfsv41_middle_file_layout.xml	15 Dec 2007 03:33:26 -0000
@@ -1166,17 +1166,16 @@
 
 <section title="Global Stateid Requirements" anchor="global_stateid">
 <t>
-  Note, there are no stateids embedded within the layout returned
-  by the metadata server to the pNFS client. When the client issues
-  I/O to a data server, the stateid used must be one
-  previously returned by the
-  metadata server. Permitted stateids are based on one of the following:
+  When the client issues
+  I/O to a data server, the stateid used MUST NOT be a layout stateid
+  as returned by LAYOUTGET or sent by CB_LAYOUTRECALL. 
+  Permitted stateids are based on one of the following:
   an open stateid
   (the stateid field of data type OPEN4resok as returned by OPEN),
   a delegation stateid (the stateid field of data types open_read_delegation4
   and open_write_delegation4 as returned by OPEN or WANT_DELEGATION,
   or as sent by CB_PUSH_DELEG), or a stateid returned by the LOCK or LOCKU
-  operations.  The stateid sent to the data server must be sent with
+  operations.  The stateid sent to the data server MUST be sent with
   the seqid set to zero, indicating the most current version of that
   stateid, rather one indicating a specific non-zero seqid value.  In
   no case is the use of special stateid values allowed.
@@ -1385,33 +1384,53 @@
 </t>
 </section>
 
-<section title="Recovery from Loss of Layout" anchor="file_crash_recovery">
+<section title="Layout Revocation and Fencing" anchor="file_layout_revoke" >
 <t>
   As described in <xref target="crash_recovery" />, the
   layout type-specific storage protocol is responsible
   for handling the effects of I/Os started before
   lease expiration, extending through lease expiration.
-  The NFSv4.1 file layout type prevents all I/Os from
+  The LAYOUT4_NFSV4_1_FILES layout type 
+  can prevents all I/Os to data servers from
   being executed after lease expiration, without relying
   on a precise client lease timer and without requiring
-  data servers to maintain lease timers.
-
+  data servers to maintain lease timers. However, while
+  LAYOUT4_NFSV4_1_FILES pNFS server is free to deny the client all
+  access to the data servers, because it supports
+  revocation of layouts, it is also free to perform a denial
+  on a per file basis only when revoking a layout.
+  layout.
+</t>
+<t> 
+  In addition to lease expiration,
+  the reasons a layout can be revoked include: client fails to respond to
+  a CB_LAYOUTRECALL,
+  the
+  metadata server restarts, or administrative intervention. Regardless
+  of the reason, once a client's layout has been revoked, the pNFS
+  server MUST prevent the client from issuing I/O for the affected file
+  from and to all data servers, in other words, it MUST fence the
+  client from the affected file on the data servers.
 </t>
 <t>
-  It works as follows. As described in <xref
+  Fencing works as follows. As described in <xref
   target="pnfs_session_stuff" />, in COMPOUND procedure
   requests to the data server, the data filehandle provided
   by the PUTFH operation and the stateid in the READ or
   WRITE operation are used to validate that the client has
   a valid layout for the I/O being performed, if it does
   not, the I/O is rejected with NFS4ERR_PNFS_NO_LAYOUT.
+  The server can simply check the stateid, and additionally,
+  make the data filehandle stale if the layout specified
+  a data filehandle that is different from the metadata server's
+  filehandle for the file (see the nfl_fh_list description in
+  <xref target="file_data_types" />).
+</t>
+<t>
   Before the metadata server takes any action to invalidate
   layout state given out by a previous instance, it must make
   sure that all layout state from that previous instance are
   invalidated at the data servers.
-
-</t>
-<t>
   This means that a metadata server may not restripe a
   file until it has contacted all of the data servers
   to invalidate the layouts from the previous instance
@@ -1419,10 +1438,12 @@
   layouts from the previous instance without either doing
   a specific invalidation (as it would have to do anyway)
   or doing a global data server invalidation.
-
 </t>
+
+
 </section>
 
+
 <section title="Security Considerations for the File Layout Type" anchor="file_security_considerations">
 <t>
   The NFSv4.1 file layout type MUST adhere to the security
Index: nfsv41_middle_iana_considerations.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_iana_considerations.xml,v
retrieving revision 1.13
diff -w -b -u -r1.13 nfsv41_middle_iana_considerations.xml
--- nfsv41_middle_iana_considerations.xml	20 Nov 2007 23:48:48 -0000	1.13
+++ nfsv41_middle_iana_considerations.xml	15 Dec 2007 03:33:26 -0000
@@ -111,8 +111,8 @@
  </section>
  <section anchor="notifyiana" title="Defining New Notifications">
     <t>
-      New notification types may be added to the CB_NOTIFY operation
-      <xref target="OP_CB_NOTIFY" />. This can be done via changes to
+      New notification types may be added to the CB_NOTIFY_DEVICEID operation
+      <xref target="OP_CB_NOTIFY_DEVICEID" />. This can be done via changes to
       the operations that register notifications, or by adding new operations
       to NFSv4. This requires a new minor version of NFSv4, and requires
       a standards track document from IETF. Another way to
@@ -168,6 +168,9 @@
 	      the data servers 
 	    </t>
 	    <t>
+	      Describe whether revocation of layouts is supported.
+	    </t>
+	    <t>
 	      At a minimum, describe the methods of recovery from:
 	      <list style="numbers">
 		<t> Failure and restart for client, server, storage device.
@@ -182,7 +185,7 @@
 	      </list>
 	    </t>
             <t>
-              A list of any new notification values for CB_NOTIFY. 
+              A list of any new notification values for CB_NOTIFY_DEVICEID. 
             </t>
 	    <t>
 	      Include an IANA considerations section.
Index: nfsv41_middle_op_cb_layoutrecall.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_layoutrecall.xml,v
retrieving revision 1.17
diff -w -b -u -r1.17 nfsv41_middle_op_cb_layoutrecall.xml
--- nfsv41_middle_op_cb_layoutrecall.xml	10 Dec 2007 09:27:06 -0000	1.17
+++ nfsv41_middle_op_cb_layoutrecall.xml	15 Dec 2007 03:33:26 -0000
@@ -57,6 +57,8 @@
         are also recalled. The respective LAYOUTRETURN with either
         LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL acknowledges to the
         server that the client invalidated the said device mappings.
+        See <xref target="bulk_layouts" /> for considerations with
+        "bulk" recall of layouts.
       </t>
       <t>
         The NFS4ERR_NOMATCHING_LAYOUT error is only returned when the
@@ -84,6 +86,12 @@
       case, the client writes and commits data through the metadata
       server.
     </t>
+    <t>
+      See <xref target="layout_stateid"/> for a description
+      of how the lor_stateid field in the arguments is to
+      be constructed.
+
+    </t>
   </section>
   <section toc="exclude" anchor="OP_CB_LAYOUTRECALL_IMPLEMENTATION" title="IMPLEMENTATION">
     <t>
@@ -93,7 +101,7 @@
       the request before actually returning layouts with the
       LAYOUTRETURN operation.  While the client responds to the
       CB_LAYOUTRECALL immediately, the operation is not considered
-      complete until all affected layouts are returned to the server
+      complete (i.e. considered pending) until all affected layouts are returned to the server
       with the LAYOUTRETURN operation.
     </t>
     <t>
Index: nfsv41_middle_op_getdeviceinfo.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_getdeviceinfo.xml,v
retrieving revision 1.14
diff -w -b -u -r1.14 nfsv41_middle_op_getdeviceinfo.xml
--- nfsv41_middle_op_getdeviceinfo.xml	10 Dec 2007 19:58:58 -0000	1.14
+++ nfsv41_middle_op_getdeviceinfo.xml	15 Dec 2007 03:33:26 -0000
@@ -90,8 +90,8 @@
        stateid for each layout that might have been revoked. If
        TEST_STATEID indicates any layouts have been revoked, the
        client must recover from layout revocation as described in
-       (see <xref
-       target="revoke_layout" />). If TEST_STATEID indicates at least
+       <xref
+       target="revoke_layout" />. If TEST_STATEID indicates at least
        one layout has not been revoked, the client should issue
        a GETDEVICEINFO on the device ID to verify that the device ID
        has been deleted. If GETDEVICEINFO indicates the device ID
Index: nfsv41_middle_op_layoutcommit.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutcommit.xml,v
retrieving revision 1.16
diff -w -b -u -r1.16 nfsv41_middle_op_layoutcommit.xml
--- nfsv41_middle_op_layoutcommit.xml	8 Dec 2007 16:23:56 -0000	1.16
+++ nfsv41_middle_op_layoutcommit.xml	15 Dec 2007 03:33:26 -0000
@@ -65,7 +65,9 @@
     </t>
     <t>
       The loca_stateid is a layout stateid value as returned by
-      previously successful layout operations.
+      previously successful layout operations (
+      see <xref target="layout_stateid"/>).
+
     </t>
     <t>
       The loca_last_write_offset field specifies the offset of the
Index: nfsv41_middle_op_layoutget.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutget.xml,v
retrieving revision 1.18
diff -w -b -u -r1.18 nfsv41_middle_op_layoutget.xml
--- nfsv41_middle_op_layoutget.xml	10 Dec 2007 19:58:58 -0000	1.18
+++ nfsv41_middle_op_layoutget.xml	15 Dec 2007 03:33:26 -0000
@@ -43,12 +43,17 @@
       NFS4ERR_BADLAYOUT will be returned.
     </t>
     <t>
-      The loga_stateid field specifies a valid stateid.  If a layout
-      is not currently held by the client, the loga_stateid field
-      represents a stateid reflecting the correspondingly valid open,
-      record lock, or delegation stateid.  Once a layout is held by
-      the client for the file, the loga_stateid field is a stateid as
-      returned from a previous LAYOUTGET or LAYOUTRETURN operation.
+      The loga_stateid field specifies a valid stateid.
+      If a layout is not currently held by the client,
+      the loga_stateid field represents a stateid
+      reflecting the correspondingly valid open,
+      record lock, or delegation stateid.  Once a
+      layout is held by the client for the file, the
+      loga_stateid field is a stateid as returned from
+      a previous LAYOUTGET or LAYOUTRETURN operation or
+      provided by a CB_LAYOUTRECALL operation (see <xref
+      target="layout_stateid" />).
+
     </t>
     <t>
       The loga_maxcount field specifies the maximum layout size (in bytes)
Index: nfsv41_middle_op_layoutreturn.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutreturn.xml,v
retrieving revision 1.20
diff -w -b -u -r1.20 nfsv41_middle_op_layoutreturn.xml
--- nfsv41_middle_op_layoutreturn.xml	10 Dec 2007 09:27:06 -0000	1.20
+++ nfsv41_middle_op_layoutreturn.xml	15 Dec 2007 03:33:26 -0000
@@ -43,6 +43,8 @@
       byte range does not designate any bytes for which a layout
       is held for the specified file, client ID, layout type and
       mode, no error results.
+        See <xref target="bulk_layouts" /> for considerations with
+        "bulk" return of layouts.
     </t>
     <t>
       The layout being returned may be a subset
@@ -96,16 +98,20 @@
     <t>
       If the client is returning the layout in response to a
       CB_LAYOUTRECALL where the lor_recalltype was LAYOUTRECALL4_FILE, the
-      client should include the lor_stateid value as the
-      lora_recallstateid value.  This is done to indicate the point
+      client should include use lor_stateid value from CB_LAYOUTRECALL
+      as the value for lrf_stateid. Otherwise, it should use
+      logr_stateid (from a previous LAYOUTGET result) or
+      lorr_stateid (from a previous LAYRETURN result)
+      This is done to indicate the point
       in time (in terms of layout stateid transitions) when the recall
       was issued.  The client must use the precise lora_recallstateid
       value and not set the seqid to zero.  Otherwise NFS4ERR_BAD_STATEID
-      will be returned.  This stateid is not subject to checking
-      for the NFS4ERR_OLD_STATEID error.  When this LAYOUTRETURN is
-      not in response to CB_LAYOUTRECALL where the lor_recalltype 
-      was LAYOUTRECALL4_FILE, the lor_stateid should be the 
-      special stateid consisting of all zeros.
+      will be returned. NFS4ERR_OLD_STATEID can be returned if the
+      client is using an old seqid, and the server knows the
+      client should not be using the old seqid. E.g. the client uses
+      the seqid on slot 1 of the session, received the response with
+      the new seqid, and uses the slot to send another request with the
+      old seqid.
     </t>
     <t>
       If a client fails to return a layout
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.55
diff -w -b -u -r1.55 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	10 Dec 2007 19:58:58 -0000	1.55
+++ nfsv41_middle_pnfs.xml	15 Dec 2007 03:33:27 -0000
@@ -49,7 +49,7 @@
   delegations are managed.  For example, the layout is leased,
   recallable and revocable.  However, layouts are distinct
   abstractions and are manipulated with new operations.  When a client
-  holds a layout, is delegated the ability to access the data location
+  holds a layout, it is delegated the ability to access the data location
   directly using the location information specified in the layout.
 </t>
 <t>
@@ -292,8 +292,35 @@
     recovery works in that situation.
   </t>
   <t>
-    A device ID to device address mapping lives as long as
-    there is a layout referring to the device ID.
+    A device ID lives as long as there is a layout
+    referring to the device ID.  If there are no layouts
+    referring to the device ID, the server is free to
+    delete the device ID whenever it decides to.
+
+  </t>
+  <t>
+    Device ID to device address mappings are not leased,
+    and can be changed at any time. (Note that while
+    device ID to device address mappings are likely
+    to change after the metadata server restarts the
+    server is not required to change the mappings.)
+    A server has two
+    choices for changing mappings.  It can recall all
+    layouts referring to the device ID or it can use a
+    notification mechanism.
+
+  </t>
+  <t>
+    The NFSv4.1 protocol has no optimal way to recall
+    all layouts that referred to a particular device ID
+    (unless the server associates a single device ID with
+    a single fsid or a single client ID; in which case,
+    CB_LAYOUTRECALL has options for recalling all layouts
+    associated with the fsid, client ID pair or just the
+    client ID).
+
+  </t>
+  <t>
     Via a notification mechanism
     (see <xref target="OP_CB_NOTIFY_DEVICEID" />),
     device ID to device address mappings can change over the duration
@@ -307,6 +334,7 @@
     The server MUST support notifications and the client must
     request them before they can be used.  For further information
     about the notification types <xref target="OP_CB_NOTIFY_DEVICEID" />.
+
   </t>
   <t>
     Once a device ID is deleted by the server, the server MUST NOT
@@ -487,15 +515,24 @@
 
 </t>
 <t>
-  There is no required ordering between getting a layout and performing
-  a file OPEN.  For example, a layout may first be retrieved by
-  placing a LAYOUTGET operation in the same COMPOUND as the initial
-  file OPEN.  Once the layout has been retrieved, it can be held
-  across multiple OPEN and CLOSE sequences.  Therefore, a client may
-  hold a layout for a file that is not currently open by any user on
-  the client.  This allows for the caching of layouts beyond CLOSE
-  such that CLOSE to OPEN sequences do not incur additional layout
-  overhead.
+  In order to get a layout, the client must first have
+  opened the file via the OPEN operation. When a client has
+  no layout on a file, it presents a stateid as returned by
+  OPEN, a delegation stateid, or a byte-range lock stateid
+  in the loga_stateid argument. A successful LAYOUTGET
+  result includes a layout stateid. The first successful LAYOUTGET processed
+  by the server using a non-layout stateid as an argumenht MUST
+  have the "seqid" field of the layout stateid in the response set
+  to one. Thereafter, the client
+  uses a layout stateid (see <xref target="layout_stateid"
+  />) on future invocations of LAYOUTGET on the file, and the "seqid"
+  MUST NOT ever be set to zero.
+  Once the layout has been retrieved, it can be held across
+  multiple OPEN and CLOSE sequences.  Therefore, a client
+  may hold a layout for a file that is not currently open
+  by any user on the client.  This allows for the caching
+  of layouts beyond CLOSE.
+
 </t>
 <t>
   The storage protocol used by the client to access the data on the
@@ -534,13 +571,59 @@
 </t>
 </section>
 
+<section title="Layout Stateid" anchor="layout_stateid">
+<t>
+ As with all other stateids, the stateid consists of a
+ "seqid" and "other" field. Once a layout stateid is
+ changed, the "other" field will stay constant unless
+ the stateid is revoked, or the client returns all layouts
+ on the file and the server disposes of the stateid.
+ The "seqid" field is initially set to one, and is
+ never zero on any NFSv4.1 operation that uses layout
+ stateids. After the layout stateid is
+ established, the "seqid" is incremented by the server
+ in each subsequent LAYOUTGET and LAYOUTRETURN response,
+ and in each CB_LAYOUTRECALL request. When the
+ client fully processes the response to a LAYOUTGET or LAYOUTRETURN,
+ or fully processes the arguments of a CB_LAYOUTRECALL, it
+ MUST use the seqid of the stateid of the reply from LAYOUTGET and
+ LAYOUTRETURN, or the stateid in the arguments of CB_LAYOUTRECALL,
+ the client MUST use the seqid on subsequent calls to LAYOUTGET
+ or LAYOUTRETURN.
+ The client and server use
+ the "seqid" of the layout stateid for the following.
+ <list style="symbols">
+ <t>
+  Permit the client to issue parallel LAYOUTGET operations
+  on the same file.  As with parallel opens (see <xref
+  target="parallel_opens" />) the use of the sequence ID
+  allows a client to avoid serializing operations. This
+  way the client does not have to serialize I/O by in turn
+  serializing LAYOUTGET calls.  LAYOUTGET was executed
+  last. In the event parallel LAYOUTGET operations are
+  send with a non-layout stateid (because the client does
+  not yet have a layout stateid), the successful responses
+  MUST have the same "other" field in the LAYOUTSTATEID,
+  and each response has a unique "seqid", where the lowest
+  "seqid" is one, and the highest "seqid" is equal to the
+  count of parallel LAYOUTGET operations invoked on the
+  non-layout stateid.
+
+ </t>
+ <t>
+  Allow the client and server to detect race
+  conditions. See <xref target="pnfs_operation_sequencing"/>.
+ </t>
+ </list>
+</t>
+</section>
 
 <section title="Committing a Layout" anchor="committing_layout">
 <t>
   Allowing for varying storage protocols capabilities, the pNFS
   protocol does not require the metadata server and storage devices to
   have a consistent view of file attributes and data location
-  mappings.  Data location mapping refers to things like which offsets
+  mappings.  Data location mapping refers to aspects such as which offsets
   store data as opposed to storing holes (see <xref
   target="sparse_dense" /> for a discussion).  Related issues arise
   for storage protocols where a layout may hold provisionally
@@ -569,7 +652,7 @@
   the same time frame.
 </t>
 <t>
-  For block/volume-based layouts, the LAYOUTCOMMIT may require
+  For block/volume-based layouts, LAYOUTCOMMIT may require
   updating the block list that comprises the file and committing this
   layout to stable storage.  For file-layouts synchronization of
   attributes between the metadata and storage devices primarily the
@@ -585,33 +668,33 @@
   between a WRITE to a storage device and the LAYOUTCOMMIT may observe
   a size that does not reflect the actual data written.
 </t>
-<section title="LAYOUTCOMMIT and change/time_modify/time_change">
+<section title="LAYOUTCOMMIT and change/time_modify">
 <t>
-  The change, time_modify, and time_access attributes may be updated
+  The change and time_modify attributes may be updated
   by the server when the LAYOUTCOMMIT operation is processed.  The
   reason for this is that some layout types do not support the update
   of these attributes when the storage devices process I/O operations.
   The client is capable providing suggested values to the server for
-  time_access and time_modify with the arguments to LAYOUTCOMMIT.
+  time_modify with the arguments to LAYOUTCOMMIT.
   Based on layout type, the provided values may or may not be used.
   The server should sanity check the client provided values
   before they are used.  For example, the server should ensure that
   time does not flow backwards.  The client always has the option to
-  set these attributes through an explicit SETATTR operation.
+  set time_modify through an explicit SETATTR operation.
 </t>
 <t>
   For some layout protocols, the storage device is able to notify the
   metadata server of the occurrence of an I/O and as a result the
-  change, time_modify, and time_access attributes may be updated at
+  change and time_modify attributes may be updated at
   the metadata server.  For a metadata server that is capable of
-  monitoring updates to the change, time_modify, and time_access
+  monitoring updates to the change and time_modify
   attributes, LAYOUTCOMMIT processing is not required to update the
   change attribute; in this case the metadata server must ensure that
   no further update to the data has occurred since the last update of
   the attributes; file-based protocols may have enough information to
   make this determination or may update the change attribute upon each
-  file modification.  This also applies for the time_modify and
-  time_access attributes.  If the server implementation is able to
+  file modification.  This also applies for the time_modify
+  attributes.  If the server implementation is able to
   determine that the file has not been modified since the last
   time_modify update, the server need not update time_modify at
   LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, the updated attributes
@@ -765,7 +848,7 @@
   the layouts and thus the amount of outstanding modified data.
 </t>
 
-<section title="Recall Callback Robustness" anchor="recall_robustness">
+<section title="Layout Recall Callback Robustness" anchor="recall_robustness">
 <t>
   It has been assumed thus far that pNFS client
   state for a file exactly matches the pNFS server state for that file
@@ -779,16 +862,14 @@
 <t>
 <list style="symbols">
 <t>
-  It may be useful for clients to be able to discard layout
-<cref>mre: wrong</cref>
-  information without calling LAYOUTRETURN.  If conflicts that require
+  If conflicts that require
   callbacks are very rare, and a server can use a multi-file callback
   to recover per-client resources (e.g., via a FSID recall, or a
   multi-file recall within a single compound), the result may be
   significantly less client-server pNFS traffic.
 </t>
 <t>
-  It may be similarly useful for servers to maintain information about
+  It may be useful for servers to maintain information about
   what ranges are held by a client on a coarse-grained basis, leading
   to the server's layout ranges being beyond those actually held by
   the client.
@@ -797,6 +878,17 @@
   clients may request and be granted sub-file ranges.
 </t>
 <t>
+  It may be useful for clients to "forget" details about what
+  layouts and ranges the client actually has,
+  leading to the server's layout ranges being beyond those what the
+  client "thinks" it has. As long as the client does not
+  assume it has layouts that are beyond what the server has
+  granted, this is a safe practice. Regardless, when a client
+  forgets what ranges and layouts it has, when the server
+  issues a LAYOUTRECALL, the client MUST follow up with
+  a LAYOUTRETURN for the what the server asked for.
+</t>
+<t>
   In order to avoid errors, it is vital that a client not assign
   itself layout permissions beyond what the server has granted and
   that the server not forget layout permissions that have been granted.
@@ -822,20 +914,23 @@
 </t>
 <t>
   When a layout is returned, the client MUST NOT have any outstanding
-  write requests to the storage devices involved in the layout.
+  I/O requests to the storage devices involved in the layout.
   Rephrasing, the client MUST NOT return the layout while it has
-  outstanding write requests to the storage device.
+  outstanding I/O requests to the storage device.
 </t>
 <t>
-  Even with this requirement for the client, it is possible that write
+  Even with this requirement for the client, it is possible that I/O
   requests may be presented to a storage device no longer allowed to
   perform them.  Since the server has no strict control as to when the
   client will return the layout, the server may later decide to
-  unilaterally revoke the client's access provided by the layout.  In
+  unilaterally revoke the client's access to the storage devices
+  as provided by the layout.  In
   choosing to revoke access, the server must deal with the possibility
-  of lingering writes; those outstanding writes still in flight to
-  storage servers identified by the revoked layout.  Each layout
-  specification MUST define whether unilateral layout revocation by
+  of lingering I/O request; those outstanding I/O requests still in flight to
+  storage servers identified by the revoked layout.
+
+  All layout
+  specifications MUST define whether unilateral layout revocation by
   the metadata server is supported; if it is, the specification must
   also describe how lingering writes are processed.  For example,
   storage devices identified by the revoked layout could be fenced off
@@ -855,74 +950,92 @@
 </t>
 </section>
 
-<section title="Serialization of Layout Operations" anchor="pnfs_operation_sequencing">
+<section title="Sequencing of Layout Operations" anchor="pnfs_operation_sequencing">
 <t>
   As with other stateful operations, pNFS requires the correct
-  sequencing of layout operations.  pNFS uses the sessions
-  feature of NFSv4.1 to provide the correct sequencing between regular
+  sequencing of layout operations.  pNFS uses the "seqid" in the
+  layout stateid to provide the correct sequencing between regular
   operations and callbacks.  It is the server's responsibility to
   avoid inconsistencies regarding the layouts provided and the
   client's responsibility to properly serialize its layout requests
   and layout returns.
 </t>
+<section title="Recall/Return Sequencing">
+<t>
+  <xref target="sessions_callback_races"/>
+  describes the sessions mechanism for allowing the
+  client to detect such situations in order to delay processing such a
+  CB_LAYOUTRECALL. The server MUST reference all conflicting LAYOUTGET
+  operations in the CB_SEQUENCE that precedes the CB_LAYOUTRECALL.  A
+  zero length array of referenced operations is used by the server to
+  tell the client that the server does not know of any LAYOUTGET
+  operations that conflict with the recall.
 
-<section title="Get/Return Serialization">
+</t>
 <t>
-  The protocol allows the client to send concurrent LAYOUTGET and
-  LAYOUTRETURN operations to the server. However, the protocol does
-  not provide any means for the server to process the requests
-  in the same order in which they were created, nor does
-  it provide a way for the client to determine the
-  order in which parallel outstanding
-  operations were processed by the server. Thus, when a layout
-  retrieved by an outstanding LAYOUTGET operation intersects with
-  a layout returned by an outstanding LAYOUTRETURN the order
-  in which the two conflicting operations are processed determines
-  the final state of the overlapping layout. To disambiguate between
-  the two cases the client MUST serialize LAYOUTGET operations and
-  voluntary LAYOUTRETURN operations for the same file.
+  While referencing conflicting operations in CB_SEQUENCE
+  conveys to the client that the server is aware of
+  races, one critical issue with regard to operation sequencing
+  concerns callbacks.  The protocol must defend against
+  races between the reply to a LAYOUTGET or LAYOUTRETURN
+  operation and a subsequent CB_LAYOUTRECALL.  A client
+  MUST NOT process a CB_LAYOUTRECALL that implies one or
+  more outstanding LAYOUTGET or LAYOUTRETURN operations to
+  which the client has not yet received a reply. The client
+  detects such a CB_LAYOUTRECALL by examining the "seqid"
+  field of the recall's layout stateid. If the "seqid"
+  is not what the client currently has recorded, and the
+  client has at least one LAYOUTGET and/or LAYOUTRETURN
+  operation outstanding, the client knows the recall is
+  for a response to an outstanding LAYOUTGET or LAYOUTRETURN.
+
+</t>
+
+<section title="Get/Return Sequencing">
+<t>
+  The protocol allows the client to send concurrent
+  LAYOUTGET and LAYOUTRETURN operations to the server. The
+  protocol does not provide any means for the server to
+  process the requests in the same order in which they
+  were created. However, through the use of the "seqid"
+  field in the layout stateid, the client can determine
+  the order in which parallel outstanding operations were
+  processed by the server. Thus, when a layout retrieved
+  by an outstanding LAYOUTGET operation intersects with
+  a layout returned by an outstanding LAYOUTRETURN on
+  the same file, the order in which the two conflicting
+  operations are processed determines the final state of
+  the overlapping layout. The client determines the order,
+  by looking at the "seqid" returned in the layout stateid
+  by each operation. The result with the highest "seqid"
+  is the one that was executed last.
+
 </t>
 <t>
   It is permissible for the client to send in parallel multiple
   LAYOUTGET operations for the same file or multiple LAYOUTRETURN
-  operations for the same file; but never a mix of both. It is also
-  permissible for the client to combine LAYOUTRETURN and LAYOUTGET
+  operations for the same file, and a mix of both.
+</t>
+<t>
+  It is permissible for the client to combine LAYOUTRETURN and LAYOUTGET
   operations for the same file in the same COMPOUND request since the
-  server MUST process these in order.  If a client does issue such
-  requests, it MUST NOT have more than one outstanding for the same
+  server MUST process these in order. The client uses the
+  current stateid (see <xref target="current_stateid"/>). However,
+  if a client does issue such
+  COMPOUND requests, it MUST NOT have more than one outstanding for the same
   file at the same time and MUST NOT have other LAYOUTGET or
   LAYOUTRETURN operations outstanding at the same time for that same
   file.
 </t>
 </section>
 
-<section title="Recall/Return Sequencing">
-<t>
-  One critical issue with regard to operation sequencing concerns
-  callbacks.  The protocol must defend against races between the reply
-  to a LAYOUTGET operation and a subsequent CB_LAYOUTRECALL.  A client
-  MUST NOT process a CB_LAYOUTRECALL that identifies an outstanding
-  LAYOUTGET operation to which the client has not yet received a
-  reply.  Intersecting LAYOUTGET operations are identified in the
-  CB_SEQUENCE preceding the CB_LAYOUTRECALL.
-</t>
-<t>
-  <xref target="sessions_callback_races"/>
-  describes the sessions mechanism for allowing the
-  client to detect such situations in order to delay processing such a
-  CB_LAYOUTRECALL. The server MUST reference all conflicting LAYOUTGET
-  operations in the CB_SEQUENCE that precedes the CB_LAYOUTRECALL.  A
-  zero length array of referenced operations is used by the server to
-  tell the client that the server does not know of any LAYOUTGET
-  operations that conflict with the recall.
-</t>
 <section title="Client Considerations">
 <t>
   Consider a pNFS client that has issued a LAYOUTGET and then receives
-  an overlapping CB_LAYOUTRECALL for the same file.  There are two
-  possibilities, which the client would be unable to distinguish
-  without additional information provided by the sessions
-  implementation.
+  a CB_LAYOUTRECALL for the same file with an overlapping range.  There are two
+  possibilities, which the client can distinguish
+  via the layout stateid in the recall.
+
 <list style="numbers">
 <t>
   The server processed the LAYOUTGET before issuing the recall, so the
@@ -931,110 +1044,130 @@
   with the CB_LAYOUTRECALL.
 </t>
 <t>
-  The server issued the callback before receiving the LAYOUTGET. The
-  server will not respond to the LAYOUTGET until the CB_LAYOUTRECALL
-  is processed.
+    The
+    server issued the callback before receiving the
+    LAYOUTGET. The server will not respond to the LAYOUTGET
+    until the CB_LAYOUTRECALL is processed.
+
 </t>
 </list>
-  These possibilities could cause deadlock, as the client must wait
-  for the LAYOUTGET response before processing the recall in the first
-  case, but that response will not arrive until after the recall is
-  processed in the second case.  Via the CB_SEQUENCE operation, the
-  server provides the client with the { sessionid, slotid , sequence
-  id } of any earlier LAYOUTGET operations which remain unconfirmed at
-  the server by the session slot usage rules.  This allows the client
-  to disambiguate between the two cases. In case 1 the server will
-  provide the operation reference(s); whereas in case 2 it will not
-  because there are no dependent client operations.  Therefore, the
-  action at the client will only require waiting in the case that the
-  client has not yet seen the server's earlier responses to the
-  LAYOUTGET operation(s).
-
-  <vspace blankLines='1' />
 
-  The following requirements apply to avoid this deadlock: by adhering
-  to the following requirements:
+  If these possibilities cannot be distinguished, a
+  deadlock could result, as the client must wait for the
+  LAYOUTGET response before processing the recall in the
+  first case, but that response will not arrive until after
+  the recall is processed in the second case. Note that
+  in the first case, the "seqid" in the layout stateid
+  of the recall is one greater than what the client has
+  recorded and in the second case, the "seqid" is equal to
+  what the client has recorded.  This allows the client
+  to disambiguate between the two cases. The client thus
+  knows precisely which possibility applies.
 
-  <list style="symbols">
-  <t>
-    A LAYOUTGET MUST be rejected with the error NFS4ERR_RECALLCONFLICT
-    if there is an overlapping outstanding CB_LAYOUTRECALL to the same
-    client.
   </t>
   <t>
-    When processing a recall, the client MUST wait for a response to
-    all conflicting outstanding LAYOUTGETs that are referenced in the
-    CB_SEQUENCE for the recall before performing any LAYOUTRETURN that
-    could be affected by any such response.
-  </t>
-  <t>
-    The client SHOULD wait for responses to all operations required to
-    complete a recall before sending any LAYOUTGETs that would
-    conflict with the recall because the server is likely to
-    return errors for them (see the first item above).
+
+  In case 1 the client knows it needs to wait for
+  the LAYOUTGET response before processing the recall
+  (or the client can return NFS4ERR_DELAY). 
   </t>
   <t>
-    Before sending a new LAYOUTGET for a range covered by a layout
-    recall, the client SHOULD wait for responses to any outstanding
-    LAYOUTGET that overlaps any portion of the new LAYOUTGET's range .
-    This is because it is possible (although unlikely) that the prior
-    operation may have arrived at the server after the recall
-    completed and hence will succeed.
+  In case 2 the client will not wait for the LAYOUTGET
+  response before processing the recall, because waiting
+  would cause deadlock.  Therefore, the action at the
+  client will only require waiting in the case that the
+  client has not yet seen the server's earlier responses
+  to the LAYOUTGET operation(s).
+
   </t>
   <t>
-    The recall process can be considered completed, by the client, when
+  The recall process can be considered completed when
     the final LAYOUTRETURN operation for the recalled range is completed.
+  The LAYOUTRETURN uses the layout stateid (with seqid) specified in
+  CB_LAYOUTRECALL.
   </t>
-  </list>
-</t>
+
 </section>
 
-<section title="Server Considerations">
+<section title="Server Considerations" anchor="layout_server_consider">
 <t>
-  Consider a related situation from the metadata server's point of
+  Consider the race the metadata server's point of
   view.  The metadata server has issued a CB_LAYOUTRECALL and receives
   an overlapping LAYOUTGET for the same file before the
-  LAYOUTRETURN(s) that respond to the CB_LAYOUTRECALL.  Again, there
-  are two cases:
+  LAYOUTRETURN(s) that respond to the CB_LAYOUTRECALL. There are
+  are three cases:
 
 <list style="numbers">
 <t>
   The client issued the LAYOUTGET before processing the
   CB_LAYOUTRECALL.
+  The "seqid" in the layout stateid
+  of LAYOUTGET is less than that of the "seqid" in
+  CB_LAYOUTRECALL. The server
+  returns NFS4ERR_RECALLCONFLICT to the client, which
+  indicates to the client that there is a pending recall.
 </t>
 <t>
   The client issued the LAYOUTGET after processing the
-  CB_LAYOUTRECALL, but it arrived before the LAYOUTRETURN that
+  CB_LAYOUTRECALL, but the LAYOUTGET arrived before the LAYOUTRETURN and
+  response to CB_LAYOUTRECALL that
   completed that processing.
-</t>
-</list>
+  The "seqid" in the layout stateid
+  of LAYOUTGET is equal to that of the "seqid" in
+  CB_LAYOUTRECALL.
+  The server has not received a response to the CB_LAYOUTRECALL,
+  so it returns NFS4ERR_RECALLCONFLICT.
 </t>
 <t>
-   There is nothing the session sequence logic can do to disambiguate
-   between these two cases because both operations are independent of
-   one another. They are simply asynchronous events which crossed. The
-   situation can even occur if the session is configured to use a
-   single connection for both operations and callbacks.
+  The client issued the LAYOUTGET after processing the
+  CB_LAYOUTRECALL, the server received the CB_LAYOUTRECALL
+  response, but the LAYOUTGET arrived before the LAYOUTRETURN that
+  completed that processing.
+  The "seqid" in the layout stateid
+  of LAYOUTGET is equal to that of the "seqid" in
+  CB_LAYOUTRECALL.
+  The server has received a response to the CB_LAYOUTRECALL,
+  so it returns NFS4ERR_RETURNCONFLICT.
 </t>
+</list>
+</t>
+</section>
+
+<section title="Bulk Recall and Return" anchor="bulk_layouts">
 <t>
-  Given no method to disambiguate these cases the metadata server MUST
-  reject the overlapping LAYOUTGET with the error
-  NFS4ERR_RECALLCONFLICT.  The client has two ways to avoid this
-  result.  It can issue the LAYOUTGET as a subsequent element of a
-  COMPOUND containing the LAYOUTRETURN that completes the
-  CB_LAYOUTRECALL or it can wait for the response to that
-  LAYOUTRETURN.
+ PNFS supports recalling and returning all layouts that
+ are for files belonging to a particular fsid
+ (LAYOUTRECALL4_FSID, LAYOUTRETURN4_FSID) or client ID
+ (LAYOUTRECALL4_ALL, LAYOUTRETURN4_ALL).
+ There are no "bulk" stateids, so detection of races
+ via the seqid is not possible. 
+ The server MUST not initiate bulk recall while another
+ recall is in progress, or the corresponding LAYOUTRETURN
+ is in progress or pending.
+ In the event the server sends a bulk recall
+ while the client has pending or in progress LAYOUTRETURN,
+ CB_LAYOUTRECALL, or LAYOUTGET, the client returns
+ NFS4ERR_DELAY. In the event the client sends a LAYOUTGET
+ or LAYOUTRETURN while a bulk recall is in progress, the
+ server return NFS4ERR_RECALLCONFLICT.
 </t>
 </section>
+
 </section>
 </section>
 </section>
 
 <section title="Revoking Layouts" anchor="revoke_layout">
 <t>
-<cref>mre: need text. Since layouts have stateids, revocation is not layout type specfic.</cref>
-</t>
 
+Parallel NFS permits servers to revoke layouts from clients
+that fail to response to recalls and/or fail to renew their
+lease in time. Whether the server revokes the layout or
+not depends on the layout type, and what actions are taken
+with respect to the client's I/O to data servers is also
+layout type specific.
+
+</t>
 </section>
 <section title="Metadata Server Write Propagation" anchor="async_writes">
 <t>
@@ -1192,32 +1325,6 @@
 
  <section title="Dealing with Lease Expiration on the Client"
    anchor="lease_expiration_client">
- <t>
-   The mappings between device IDs and device addresses are what
-   enables a pNFS client to safely write data to and read data from a
-   storage device. These mappings are leased (in a manner similar to
-   locking state) from the metadata server, and as long as the lease
-   is valid, the client has a ability to issue I/O to the storage
-   devices.  The lease on device ID to device address mappings is
-   renewed when the metadata server receives a SEQUENCE operation from
-   the pNFS client. The same is not specified to be true for the data
-   server receiving a SEQUENCE operation, and the client MUST NOT
-   assume that a SEQUENCE sent to a data server will renew its lease;
-   this applies to files-based layouts only.
- </t>
- <t>
-   The expiration of the lease leads to the loss of the device ID to
-   device address mappings.  In the event of lease expiration, the
-   server may choose to fence a client's access to the storage device
-   thus preventing the use of expired device ID to device address
-   mappings.  To avoid lease expiration, the client should start its
-   lease timer based on the time that it issued the operation to the
-   metadata server rather than based on the time the response was
-   received.  It is also necessary to take propagation delay into
-   account as described in <xref target="lease_propagation_delay" />.
-   Thus, the client MUST be aware of the one-way propagation delay and
-   should issue renewals well in advance of lease expiration.
- </t>
  <t anchor="pnfs_clnt_case1">
    If a client believes its lease has expired, it MUST NOT issue I/O
    to the storage device until it has validated its lease. The client
@@ -1225,12 +1332,12 @@
    SEQUENCE operation is successful, but sr_status_flag has
    SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
    SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or
-   SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST not use
-   currently held layout or device id mappings.  The client has two
+   SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST NOT use
+   currently held layouts. The client has two
    choices to recover from the lease expiration. First, for all
    modified but uncommitted data, write it to the metadata server
    using the FILE_SYNC4 flag for the WRITEs or WRITE and
-   COMMIT. Second, the client reestablish client ID and session with
+   COMMIT. Second, the client reestablishes a client ID and session with
    the server and obtain new layouts and device ID to device address
    mappings for the modified data ranges and then write the data to the
    storage devices with the newly obtained layouts.
@@ -1255,7 +1362,7 @@
  </t>
  <t anchor="pnfs_clnt_case4">
    If sr_status_flags reports no loss of state, then the lease for the
-   mappings the client has with the metadata server are valid and
+   layouts the client has are valid and
    renewed, and the client can once again issue I/O requests to the
    storage devices.
  </t> 
@@ -1281,7 +1388,7 @@
      the metadata server has not restarted
    </t>
    <t>
-     a pNFS client's device ID to device address mappings and/or
+     a pNFS client's device ID to
      layouts have been discarded (usually because the client's lease
      expired) and are invalid
    </t>
@@ -1294,7 +1401,7 @@
    operations from the client to the storage devices after layout
    state loss.  The details of how fencing is done are specific to the
    layout type.  The solution for NFSv4.1 file-based layouts is
-   described in (<xref target="file_crash_recovery" />), and for other
+   described in (<xref target="file_layout_revoke" />), and for other
    layout types in their respective external specification documents.
  </t>
  </section>
@@ -1306,7 +1413,7 @@
     target="server_failure" /> and discussed in a pNFS-specific
     context in <xref target="pnfs_clnt_case2" />, of <xref
     target="lease_expiration_client" />.  The client MUST stop using
-    layouts and delete device ID to device address mappings it
+    layouts and delete the device ID to device address mappings it
     previously received from the metadata server.  Having done that,
     if the client wrote data to the storage device without committing
     the layouts via LAYOUTCOMMIT, then the client has
@@ -1349,7 +1456,7 @@
     the device addresses could have been assigned to a different
     storage device.  The option of retrieving the data from the
     storage device and writing it to the metadata server per the
-    recovery scenario described above in the previous two bullets is
+    recovery scenario described above is
     not available because, again, the mappings of range to device ID,
     device ID to device address, device address to physical device are
     stale and new mappings via new LAYOUTGET do not solve the problem.
Index: nfsv41_middle_state_mgmt.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_state_mgmt.xml,v
retrieving revision 1.13
diff -w -b -u -r1.13 nfsv41_middle_state_mgmt.xml
--- nfsv41_middle_state_mgmt.xml	29 Nov 2007 15:50:21 -0000	1.13
+++ nfsv41_middle_state_mgmt.xml	15 Dec 2007 03:33:27 -0000
@@ -261,23 +261,13 @@
           When a stateid is sent by the server to client as part of
           a callback operation, it is not subject to checking for
           a current seqid and returning NFS4ERR_OLD_STATEID.  This
-          is because he client is not in a position to know the
+          is because the client is not in a position to know the
           most up-to-date seqid and thus cannot verify it.  Unless
           specially noted, the seqid value for a sateid sent by the
           server to the client as part of a callback is required
           to be zero with NFS4ERR_BAD_STATEID returned if it is
           not.
         </t>
-        <t>
-          The
-          exceptions are stateid values, such as the lor_stateid value,
-          passed in CB_LAYOUTRECALL, whose purpose is to be passed on,
-          unmodified, in later requests to the server (in the case of
-          lor_stateid, as the lora_recallstateid value passed to
-          LAYOUTRETURN).  In such cases, no check of the seqid is
-          done, and it will not be zero, but in this case as well
-          a return of NFS4ERR_OLD_STATEID is not possible.
-        </t>
       </section> <!-- "Stateid Structure" -->
       <section anchor="special_stateid" title="Special Stateids">
         <t>
Index: dotx.d/head.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/head.x,v
retrieving revision 1.50
diff -w -b -u -r1.50 head.x
--- dotx.d/head.x	10 Dec 2007 09:27:14 -0000	1.50
+++ dotx.d/head.x	15 Dec 2007 03:33:27 -0000
@@ -165,7 +165,8 @@
  NFS4ERR_WRONG_CRED     = 10082,/* op done by wrong cred   */
  NFS4ERR_WRONG_TYPE     = 10083,/* op on wrong type object */
  NFS4ERR_DIRDELEG_UNAVAIL= 10084,/* delegation not avail.  */
- NFS4ERR_REJECT_DELEG   = 10085 /* cb rejected delegation  */
+ NFS4ERR_REJECT_DELEG   = 10085,/* cb rejected delegation  */
+ NFS4ERR_RETURNCONFLICT = 10086 /* layout get before return*/
 };
 
 /*
Index: dotx.d/layoutreturn_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/layoutreturn_args.x,v
retrieving revision 1.3
diff -w -b -u -r1.3 layoutreturn_args.x
--- dotx.d/layoutreturn_args.x	30 Oct 2007 09:00:39 -0000	1.3
+++ dotx.d/layoutreturn_args.x	15 Dec 2007 03:33:27 -0000
@@ -1,14 +1,7 @@
-union layoutreturn_stateid switch (bool lrs_present) {
-case TRUE:
-        stateid4                lrs_stateid;
-case FALSE:
-        void;
-};
 
 struct LAYOUTRETURN4args {
         /* CURRENT_FH: file */
         bool                    lora_reclaim;
-        layoutreturn_stateid    lora_recallstateid;
         layouttype4             lora_layout_type;
         layoutiomode4           lora_iomode;
         layoutreturn4           lora_layoutreturn;
Index: dotx.d/layoutreturn_res.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/layoutreturn_res.x,v
retrieving revision 1.4
diff -w -b -u -r1.4 layoutreturn_res.x
--- dotx.d/layoutreturn_res.x	10 Nov 2007 04:53:32 -0000	1.4
+++ dotx.d/layoutreturn_res.x	15 Dec 2007 03:33:27 -0000
@@ -1,3 +1,10 @@
+union layoutreturn_stateid switch (bool lrs_present) {
+case TRUE:
+        stateid4                lrs_stateid;
+case FALSE:
+        void;
+};
+
 union LAYOUTRETURN4res switch (nfsstat4 lorr_status) {
 case NFS4_OK:
         layoutreturn_stateid    lorr_stateid;
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Sat, 15 Dec 2007 13:17:28 +0200

Mike, thanks for your updates!

My general impression is that they make the device mapping and
layout state models much clearer.

I particularly like the changes to use the layout stateid for sequencing
of  layout get, return, and recall.

Some minor comments and nitpicking follow.
One issue that still concern me is processing of voluntary LAYOUTRETURNs
(with seqid==0) during the processing of a conflicting CB_LAYOUTRECALL.
See comment at the end for more details...

Thanks,

Benny

nfsv41_middle_file_layout.xml:

      ^^^^^

nfsv41_middle_pnfs.xml:
"LAYOUTGET was executed last."
I'm not sure I got the context of this sentence...
Did you mean "The layout stateid allows the client
to determine which of parallel LAYOUTGET operations
was executed last."

"operations are send" should be "are sent", no?

-  ... This also applies for the time_modify and
-  time_access attributes.
+  This also applies for the time_modify
+  attributes.

attributes => attribute

+  ... Regardless, when a client
+  forgets what ranges and layouts it has, when the server
+  issues a LAYOUTRECALL, the client MUST follow up with

LAYOUTRECALL => CB_LAYOUTRECALL (helps finding all occurrences
if the reader is searching for the latter)

Or return NFS4ERR_NOMATCHING_LAYOUT if it has no layout
matching the CB_LAYOUTRECALL arguments.

still => "are still"

+  The protocol allows the client to send concurrent
+  LAYOUTGET and LAYOUTRETURN operations to the server. The
+  protocol does not provide any means for the server to
+  process the requests in the same order in which they
+  were created
in the case the client set the "seqid" field of all concurrent
LAYOUTGET and LAYOUTRETURN operations to zero.

[they client may choose to set the seqid to the last one he received
and if it dispatched a new layout get/return to the same file before
receiving the reply for any outstanding such op it can bump the seqid
hoping the parallel ops never cross on the wire. However, If they do
the server detects that are fails the first op he sees out of order.]

Couldn't the client have bumped the seqid on its own
when sending multiple LAYOUTGET/RETURNs in parallel?
If so the seqid may be equal to or greater than the
seqid in CB_LAYOUTRECALL.  I think it will be simpler
to require that the client MUST use seqid of zero
when for LAYOUTGET and any voluntary LAYOUTRETURN (that is
not in response to CB_LAYOUTRECALL).  seqid != 0 MUST be
used for all LAYOUTRETURNs responding to a CB_LAYOUTRECALL.

This way the CB_LAYOUTRECALL processing can be easily
serialized by the server since by rejecting any LAYOUTGET
or LAYOUTRETURN it receives with a seqid of zero
with NFS4ERR_RECALLCONFLICT until the CB_LAYOUTRECALL processing
is done.  The layout stateid will mutate by the CB_LAYOUTRECALL
and following LAYOUTRETURNs (that will carry a non-zero seqid)
and any LAYOUTGET or LAYOUTRETURN that might have been in flight
during the CB_LAYOUTRECALL processing (and carried seqid of zero)
and that were processed after the layout recall was fully processed
will return a seqid that follows the last seqid returned on a
LAYOUTRETURN that responded to the CB_LAYOUTRECALL, or the
seqid in the CB_LAYOUTRECALL in case the client returned 
NFS4ERR_NOMATCHING_LAYOUT to it.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Sat, 15 Dec 2007 07:56:38 -0800 (PST)

> Mike, thanks for your updates!

Benny,

Very useful comments. I think each of us has
somewhat different models of layout stateids.
For now at least, I've stuck with my more
restrictive model and enhanced the description.
New diffs are attached (relative to the previous
diffs).

> See comment at the end for more details...

I assert that seqid==0 is an unsupported value. 

``  to one. Thereafter, the client
 uses a layout stateid (see <xref target="layout_stateid"
 />) on future invocations of LAYOUTGET on the file, and the "seqid"
 MUST NOT ever be set to zero. ''

'' The "seqid" field is initially set to one, and is
 never zero on any NFSv4.1 operation that uses layout
 stateids. After the layout stateid is''

The attached diffs make this even clearer.

Why do we need to support seqid==0?

> to determine which of parallel LAYOUTGET operations

Yes. Hopefully it is clearer now.

> matching the CB_LAYOUTRECALL arguments.

The client cannot both forget and return NFS4ERR_NOMATCHING_LAYOUT.
Granted, if the client remembers enough to know for sure
it does not have that range, it could return an error. The
diffs clarify this.

> the server detects that are fails the first op he sees out of order.]

I disagree. The client isn't allowed to construct seqids. It
is only allowed to use seqids it has seen and processed. The new diffs
make this clearer.

> when sending multiple LAYOUTGET/RETURNs in parallel?

No for same reason given above. My model is that
the seqid is used by the server to tell determine the
client's view of layout stateid. The seqid is the "clock"
that tells the server how up to date the client's view
is.

> to require that the client MUST use a seqid of zero

I think allowing "seqid==0" makes the model
more complex. I can sort of see why you want want to link
the seqid in the recall with the return, but nothing
prevents the client from issuing multiple returns to
satisfy the recall. In the end, the server has to decide
when the client has complied with the recall.

I've also added text that states that bulk recall
prevents the client from using the layout stateids except
for LAYOUTCOMMIT. Bulk return frees all the affected
layout stateids.
Index: nfsv41_middle_file_layout.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_file_layout.xml,v
retrieving revision 1.30
diff -u -b -w -r1.30 nfsv41_middle_file_layout.xml
--- nfsv41_middle_file_layout.xml	15 Dec 2007 03:47:51 -0000	1.30
+++ nfsv41_middle_file_layout.xml	15 Dec 2007 15:51:54 -0000
@@ -1003,7 +1003,7 @@
       current fh (or the one to be made the current fh in the
       case of PUTFH) is one that is only valid on the metadata
       server or where a stateid is used that is incompatible 
-      with the data server, i.e. is a spcial stateid or has
+      with the data server, i.e. is a special stateid or has
       a non-zero seqid value.
     </t>
   </list>
@@ -1404,7 +1404,6 @@
   access to the data servers, because it supports
   revocation of layouts, it is also free to perform a denial
   on a per file basis only when revoking a layout.
-  layout.
 </t>
 <t> 
   In addition to lease expiration,
Index: nfsv41_middle_op_cb_layoutrecall.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_cb_layoutrecall.xml,v
retrieving revision 1.18
diff -u -b -w -r1.18 nfsv41_middle_op_cb_layoutrecall.xml
--- nfsv41_middle_op_cb_layoutrecall.xml	15 Dec 2007 03:47:51 -0000	1.18
+++ nfsv41_middle_op_cb_layoutrecall.xml	15 Dec 2007 15:51:54 -0000
@@ -89,7 +89,8 @@
     <t>
       See <xref target="layout_stateid"/> for a description
       of how the lor_stateid field in the arguments is to
-      be constructed.
+      be constructed. Note that the "seqid" field of lor_stateid
+      MUST NOT be zero.
 
     </t>
   </section>
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.56
diff -u -b -w -r1.56 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	15 Dec 2007 03:47:51 -0000	1.56
+++ nfsv41_middle_pnfs.xml	15 Dec 2007 15:51:56 -0000
@@ -243,7 +243,7 @@
   (<xref target="OP_LAYOUTGET" />) time.  For example, block/volume
   based protocols, block allocation could occur when a READ/WRITE
   iomode is specified.  A special LAYOUTIOMODE4_ANY iomode is defined
-  and can only be used for LAYOUTRETURN and LAYOUTRECALL, not for
+  and can only be used for LAYOUTRETURN and CB_LAYOUTRECALL, not for
   LAYOUTGET.  It specifies that layouts pertaining to both READ and
   READ/WRITE iomodes are being returned or recalled, respectively.
 </t>
@@ -521,7 +521,7 @@
   OPEN, a delegation stateid, or a byte-range lock stateid
   in the loga_stateid argument. A successful LAYOUTGET
   result includes a layout stateid. The first successful LAYOUTGET processed
-  by the server using a non-layout stateid as an argumenht MUST
+  by the server using a non-layout stateid as an argument MUST
   have the "seqid" field of the layout stateid in the response set
   to one. Thereafter, the client
   uses a layout stateid (see <xref target="layout_stateid"
@@ -580,7 +580,8 @@
  on the file and the server disposes of the stateid.
  The "seqid" field is initially set to one, and is
  never zero on any NFSv4.1 operation that uses layout
- stateids. After the layout stateid is
+ stateids, whether it is fore channel or backchannel
+ operation. After the layout stateid is
  established, the "seqid" is incremented by the server
  in each subsequent LAYOUTGET and LAYOUTRETURN response,
  and in each CB_LAYOUTRECALL request. When the
@@ -597,14 +598,15 @@
   Permit the client to issue parallel LAYOUTGET operations
   on the same file.  As with parallel opens (see <xref
   target="parallel_opens" />) the use of the sequence ID
-  allows a client to avoid serializing operations. This
-  way the client does not have to serialize I/O by in turn
-  serializing LAYOUTGET calls.  LAYOUTGET was executed
-  last. In the event parallel LAYOUTGET operations are
-  send with a non-layout stateid (because the client does
+  allows a client to avoid serializing LAYOUTGET operations. If
+  LAYOUTGETs were serialized, especially non-overlapping LAYOUTGETs,
+  then non-overlapping I/Os to storage devices
+  would in turn be effectively serialized with each other.
+  In the event parallel LAYOUTGET operations are
+  sent with a non-layout stateid (because the client does
   not yet have a layout stateid), the successful responses
   MUST have the same "other" field in the LAYOUTSTATEID,
-  and each response has a unique "seqid", where the lowest
+  and each response with a unique "seqid", where the lowest
   "seqid" is one, and the highest "seqid" is equal to the
   count of parallel LAYOUTGET operations invoked on the
   non-layout stateid.
@@ -616,6 +618,23 @@
  </t>
  </list>
 </t>
+<t>
+ The client MUST always use a seqid that was returned by the
+ server in a LAYOUTGET or LAYOUTRETURN operation, or sent by
+ the server in a CB_LAYOUTRECALL operation. It MUST only use such
+ a seqid after processing the LAYOUTGET and LAYOUTRETURN results,
+ or the CB_LAYOUTRECALL request. Simply seeing the result or
+ CB_LAYOUTRECALL request is not sufficient cause to use the
+ seqid. For LAYOUTGET results, if the client is not
+ using the forgetful model (<xref target="recall_robustness"/>),
+ it MUST first update its
+ record of what ranges of the file's layout it has before using
+ the seqid. For LAYOUTRETURN results, the client MUST cease any
+ I/O on the affected range and delete the
+ range from its record of what ranges of the file's layout it has
+ before using the seqid. For CB_LAYOUTRECALL arguments, the
+ client MUST send a response to the recall before using the seqid.
+</t>
 </section>
 
 <section title="Committing a Layout" anchor="committing_layout">
@@ -694,7 +713,7 @@
   the attributes; file-based protocols may have enough information to
   make this determination or may update the change attribute upon each
   file modification.  This also applies for the time_modify
-  attributes.  If the server implementation is able to
+  attribute.  If the server implementation is able to
   determine that the file has not been modified since the last
   time_modify update, the server need not update time_modify at
   LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, the updated attributes
@@ -884,9 +903,14 @@
   client "thinks" it has. As long as the client does not
   assume it has layouts that are beyond what the server has
   granted, this is a safe practice. Regardless, when a client
-  forgets what ranges and layouts it has, when the server
-  issues a LAYOUTRECALL, the client MUST follow up with
-  a LAYOUTRETURN for the what the server asked for.
+  forgets what ranges and layouts it has, and it gets
+  a CB_LAYOUTRECALL recall and is not certain whether it has a layout
+  for the range specified when the server
+  issues a CB_LAYOUTRECALL, the client MUST follow up with
+  a LAYOUTRETURN for what the server asked for. If the client
+  is partially forgetting and partially remembering, and it is
+  certain it does not have the range being recalled, it MUST
+  return NFS4ERR_NOMATCHING_LAYOUT.
 </t>
 <t>
   In order to avoid errors, it is vital that a client not assign
@@ -926,8 +950,9 @@
   unilaterally revoke the client's access to the storage devices
   as provided by the layout.  In
   choosing to revoke access, the server must deal with the possibility
-  of lingering I/O request; those outstanding I/O requests still in flight to
-  storage servers identified by the revoked layout.
+  of lingering I/O request; those outstanding I/O requests are
+  still in flight to
+  storage devices identified by the revoked layout.
 
   All layout
   specifications MUST define whether unilateral layout revocation by
@@ -953,7 +978,7 @@
 <section title="Sequencing of Layout Operations" anchor="pnfs_operation_sequencing">
 <t>
   As with other stateful operations, pNFS requires the correct
-  sequencing of layout operations.  pNFS uses the "seqid" in the
+  sequencing of layout operations.  PNFS uses the "seqid" in the
   layout stateid to provide the correct sequencing between regular
   operations and callbacks.  It is the server's responsibility to
   avoid inconsistencies regarding the layouts provided and the
@@ -1005,10 +1030,9 @@
   a layout returned by an outstanding LAYOUTRETURN on
   the same file, the order in which the two conflicting
   operations are processed determines the final state of
-  the overlapping layout. The client determines the order,
-  by looking at the "seqid" returned in the layout stateid
-  by each operation. The result with the highest "seqid"
-  is the one that was executed last.
+  the overlapping layout. The order is determined by
+  the "seqid" returned in each operation: the operation with the
+  higher seqid was executed later.
 
 </t>
 <t>
@@ -1151,6 +1175,30 @@
  or LAYOUTRETURN while a bulk recall is in progress, the
  server return NFS4ERR_RECALLCONFLICT.
 </t>
+<t>
+ When a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is done,
+ the client MUST NOT use any layout stateid except for
+ LAYOUTCOMMIT.
+ When a LAYOUTRETURN of LAYOUTRETURN4_ALL is done,
+ all layout stateids granted to the client ID are freed,
+ and the client MUST NOT use the layout stateids again. It
+ MUST use LAYOUTGET to obtain new layout stateids.
+
+</t>
+<t>
+ When a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is done,
+ the client MUST NOT use any layout stateid on a file
+ with the referenced fsid except for
+ LAYOUTCOMMIT.
+ 
+ When a LAYOUTRETURN of LAYOUTRETURN4_FSID is done,
+ all layout stateids granted to the referenced fsid for
+ the client ID are freed, and the client MUST NOT use the
+ layout stateids associated with files with the referenced
+ fsid again. It MUST use LAYOUTGET to obtain new layout
+ stateids on files with the referenced fsid.
+
+</t>
 </section>
 
 </section>

---

From: Benny Halevy <[email-masked]>
Date: Sun, 16 Dec 2007 10:14:39 +0200

> Why do we need to support seqid==0?

In the case the client already established the layout stateid
it use the last seqid it got back from the server on all parallel
layout operations until it receives a new seqid, right?

So this clause allows the server to handle this case:

+      ... NFS4ERR_OLD_STATEID can be returned if the

But there's a bit of a contrast to what's being said in section 8.2.2.
"Stateid Structure":

   When a client sends a stateid to the server, it has two choices with
   regard to the seqid sent.  It may set the seqid to zero to indicate
   to the server that it wishes the most up-to-date seqid for that
   stateid's "other" field to be used.  ... It
   also may set a non-zero value in which case the server checks if that
   seqid is the correct one.  In that case the server is required to
   return NFS4ERR_OLD_STATEID if the seqid is lower than the most
   current value and NFS4ERR_BAD_STATEID if the seqid is greater than
   the most current value.

First, the new, more permissive rule appears in a paragraph talking about
"If the client is returning the layout in response to a CB_LAYOUTRECALL" and it
seems to apply also to normal mode LAYOUTGET/RETURN operations.

Also, it is worth mentioning in the "Stateid Structure" section, either
as a special case, or even better as a general rule applicable to
other/all kinds of stateids.

One more thing about 8.2.2, it mentions setting the seqid to zero as a valid
choice (always), so adding a caveat about layout stateid's seems appropriate.

Other comments about the most recent changes:

(and similarly for FSID)

"done" is a loaded term.  It can be interpreted as the time
when the server received the reply for the CB_LAYOUTRECALL
callback, yet I guess you meant the time when the client
either returned NFS4ERR_NOMATCHING_LAYOUT or sent the final LAYOUTRETURN
matching the callback (with same recall/return type and iomode).
The term "completed" is used in other places, like in section 10.2:
   The recall is considered complete when the client returns the delegation ..."
or 12.5.4.2.2.1.  Client Considerations:
   o  The recall process can be considered completed, by the client,
      when the final LAYOUTRETURN operation for the recalled range is
      completed.

with this interpretation, I don't think the client is allowed to send
LAYOUTCOMMITs after it returned the layout (only in the server recovery case
in which the client uses the "reclaim" option).  If it needs to send
LAYOUTCOMMIT in response to CB_LAYOUTRECALL it must do so before 
returning the layout and use the seqid it got from CB_LAYOUTRECALL
for any resulting LAYOUTCOMMIT or LAYOUTRETURN.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Sun, 16 Dec 2007 13:21:06 -0800 (PST)

The 3rd round of layout diffs are attached.

Index: nfsv41_middle_op_layoutreturn.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_layoutreturn.xml,v
retrieving revision 1.24
diff -u -w -b -r1.24 nfsv41_middle_op_layoutreturn.xml
--- nfsv41_middle_op_layoutreturn.xml	16 Dec 2007 21:07:51 -0000	1.24
+++ nfsv41_middle_op_layoutreturn.xml	16 Dec 2007 21:15:48 -0000
@@ -101,7 +101,7 @@
       client should include use lor_stateid value from CB_LAYOUTRECALL
       as the value for lrf_stateid. Otherwise, it should use
       logr_stateid (from a previous LAYOUTGET result) or
-      lorr_stateid (from a previous LAYRETURN result)
+      lorr_stateid (from a previous LAYRETURN result).
       This is done to indicate the point
       in time (in terms of layout stateid transitions) when the recall
       was issued.  The client must use the precise lora_recallstateid
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.60
diff -u -w -b -r1.60 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	16 Dec 2007 21:07:51 -0000	1.60
+++ nfsv41_middle_pnfs.xml	16 Dec 2007 21:15:49 -0000
@@ -1166,22 +1166,22 @@
  server return NFS4ERR_RECALLCONFLICT.
 </t>
 <t>
- When a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is done,
+ Once a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is issued,
  the client MUST NOT use any layout stateid except for
  LAYOUTCOMMIT.
- When a LAYOUTRETURN of LAYOUTRETURN4_ALL is done,
+ Once a LAYOUTRETURN of LAYOUTRETURN4_ALL is issued,
  all layout stateids granted to the client ID are freed,
  and the client MUST NOT use the layout stateids again. It
  MUST use LAYOUTGET to obtain new layout stateids.
 
 </t>
 <t>
- When a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is done,
+ Once a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is issued,
  the client MUST NOT use any layout stateid on a file
  with the referenced fsid except for
  LAYOUTCOMMIT.
  
- When a LAYOUTRETURN of LAYOUTRETURN4_FSID is done,
+ Once a LAYOUTRETURN of LAYOUTRETURN4_FSID is issued,
  all layout stateids granted to the referenced fsid for
  the client ID are freed, and the client MUST NOT use the
  layout stateids associated with files with the referenced
Index: nfsv41_middle_state_mgmt.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_state_mgmt.xml,v
retrieving revision 1.18
diff -u -w -b -r1.18 nfsv41_middle_state_mgmt.xml
--- nfsv41_middle_state_mgmt.xml	16 Dec 2007 21:07:51 -0000	1.18
+++ nfsv41_middle_state_mgmt.xml	16 Dec 2007 21:15:49 -0000
@@ -219,7 +219,8 @@
           was returned.
         </t> 
         <t>
-	  When a client sends a stateid to the server, it has two
+	  Except for layout stateids (<xref target="layout_stateid"/>)
+          when a client sends a stateid to the server, it has two
           choices with regard to the seqid sent.  It may set the seqid
           to zero to indicate to the server that it wishes the most
           up-to-date seqid for that stateid's "other" field to be
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Mon, 17 Dec 2007 20:55:50 +0200

@@ -1166,22 +1166,22 @@

By "issued" you mean the time when the client receives the callback?
If so, can it send LAYOUTRETURNs for individual files in the recalled scope?
The current spec/implementation uses that to show progress while the bulk
recall is being processed.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 13:26:33 -0800 (PST)

No that isn't what I mean.

The verb issue means here:

   the act of sending or giving out.
   library.thinkquest.org/J003358F/terms.html

I'll change "issue" to "send".

> If so, can it send LAYOUTRETURNs for individual files in the recalled

No that isn't what I mean.

> recall is being processed.

I can see value in a progress indicator, but using the
file-based LAYOUTRETURN seems like the wrong way to do it.
It is a very chatty progress indicator.

Perhaps we could add to CB_LAYOUTRERECALL's result a count of
how may layouts the client still has to drain I/O for and/or 
LAYOUTCOMMIT. The count would apply to recall by FSID and ALL cases:

When the count reached zero, or the bulk
LAYOUTRETURN is received, the client is done. If a server gets 
impatient, it sends another CB_LAYOUTRECALL. If the server doesn't like
the progress, (e.g. it finds the client lowering the slope of
line, it revokes.)

---

From: [email-masked]
Date: Mon, 17 Dec 2007 16:45:45 -0500

Mike,

Unless I've lost the larger context, I cannot see how the act of the
server sending a CB_LAYOUTRECALL can imply the behavior written below,
i.e. "the client MUST NOT use any layout stateid except for
LAYOUTCOMMIT."

What if the client had simultaneously used the statid for another
operation before receiving the CB_LAYOUTRECALL?

IMHO, it is once the client replies to the CB_LAYOUTRECALL that the
restriction begins to apply.

-Jason

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, December 17, 2007 4:27 PM
To: [email-masked]
Subject: Re: Layout races (was Re: [nfsv4] Final notes from pNFS meeting
heldin Vancouver)

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 14:18:53 -0800 (PST)

--- [email-masked] wrote:

> server sending a CB_LAYOUTRECALL can imply the behavior written

Keep in mind that is a CB_LAYOUTRECALL for the _FSID or _ALL case.

> below,

``In the event the client sends a LAYOUTGET or LAYOUTRETURN while a
bulk recall is in progress, the server return NFS4ERR_RECALLCONFLICT.''

(Will fix the typo: return --> returns)

> restriction begins to apply.

Clearly if the client doesn't see it, it can't honor the restriction.
It gets an recoverable error instead.
 
> > If so, can it send LAYOUTRETURNs for individual files in the
> recalled
> > scope?
> 
> > The current spec/implementation uses that to show progress while
> the
> > bulk

---

From: Benny Halevy <[email-masked]>
Date: Tue, 18 Dec 2007 09:14:12 +0200

>> the progress, (e.g. it finds the client lowering the slope of

Yeah, that's good idea.

Benny

>> operation before receiving the CB_LAYOUTRECALL?

Yeah, that's the problem I saw with restricting the client behavior
on an event it isn't aware of (yet).

Semantically, I think it'd better to say something like:
"Once the client receives the bulk recall it MUST NOT use any layout
stateid except for LAYOUTCOMMIT.  The server MUST NOT allow that once
the bulk recall has been issued and in case it receives LAYOUTGET or
LAYOUTRETURN from the recalled client ID, and for a file in the recalled
fsid in the RECALL_FSID case, the server returns NFS4ERR_RECALLCONFLICT."

Benny

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 08:30:15 -0800 (PST)

> >> the progress, (e.g. it finds the client lowering the slope of

To quote Spencer: ``The words and intent I have tried to communicate is
that unless something
is broken that there be no more functional changes.''

If others really want it and think what is there now is broken, 
I'll make the changes.

> fsid in the RECALL_FSID case, the server returns

More diffs are attached.
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.63
diff -u -w -b -r1.63 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	17 Dec 2007 22:17:39 -0000	1.63
+++ nfsv41_middle_pnfs.xml	18 Dec 2007 14:58:29 -0000
@@ -1167,28 +1167,47 @@
 </t>
 <t>
  Once a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is sent,
- the client MUST NOT use any layout stateid except for
- LAYOUTCOMMIT.
- Once a LAYOUTRETURN of LAYOUTRETURN4_ALL is sent,
- all layout stateids granted to the client ID are freed,
- and the client MUST NOT use the layout stateids again. It
+ the server MUST NOT allow the client to use any layout
+ stateid except for LAYOUTCOMMIT. Once the client receives
+ a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL, it MUST NOT use
+ any layout stateid except for LAYOUTCOMMIT.
+
+ Once a LAYOUTRETURN of LAYOUTRETURN4_ALL is sent, all
+ layout stateids granted to the client ID are freed.
+ The client MUST NOT use the layout stateids again. It
  MUST use LAYOUTGET to obtain new layout stateids.
 
 </t>
 <t>
- Once a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is sent,
- the client MUST NOT use any layout stateid on a file
- with the referenced fsid except for
- LAYOUTCOMMIT.
- 
- Once a LAYOUTRETURN of LAYOUTRETURN4_FSID is sent,
- all layout stateids granted to the referenced fsid for
- the client ID are freed, and the client MUST NOT use the
- layout stateids associated with files with the referenced
- fsid again. It MUST use LAYOUTGET to obtain new layout
- stateids on files with the referenced fsid.
+ Once a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is sent, the
+ server MUST NOT allow the client to use any layout stateid
+ that refers to a file with the specified fsid except for
+ LAYOUTCOMMIT. Once the client receives a CB_LAYOUTRECALL
+ of LAYOUTRECALL4_ALL, it MUST NOT use any layout stateid
+ that refers to a file with the specified fsid except
+ for LAYOUTCOMMIT.
+
+ Once a LAYOUTRETURN of LAYOUTRETURN4_FSID is sent, all
+ layout stateids granted to the referenced fsid are freed.
+ The client MUST NOT use the layout stateids for files
+ with the referenced fsid again. It MUST use LAYOUTGET to
+ obtain new layout stateids files with the referenced fsid.
 
 </t>
+<t>
+ LAYOUTGET and LAYOUTRETURN on a specific stateid are both
+ prone to racing with a bulk recall. If the server has
+ sent a bulk CB_LAYOUTRECALL, and receives a LAYOUTGET,
+ or a LAYOUTRETURN with a stateid, the server MUST return
+ NFS4ERR_RECALLCONFLICT. If the server has sent a bulk
+ CB_LAYOUTRECALL, and receives a LAYOUTRETURN with an
+ lr_returntype that is not equal to the lor_recalltype
+ of the CB_LAYOUTRECALL, the server MUST return
+ NFS4ERR_RECALLCONFLICT.
+
+</t>
+ 
+<section title="Sequencing of Layout Operations" anchor="pnfs_operation_sequencing">
 </section>
 
 </section>
Index: dotx.d/cb_layoutrecall_args.x
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/cb_layoutrecall_args.x,v
retrieving revision 1.4
diff -u -w -b -r1.4 cb_layoutrecall_args.x
--- dotx.d/cb_layoutrecall_args.x	30 Oct 2007 05:20:37 -0000	1.4
+++ dotx.d/cb_layoutrecall_args.x	18 Dec 2007 14:58:29 -0000
@@ -3,9 +3,9 @@
  */
 
 enum layoutrecall_type4 {
-        LAYOUTRECALL4_FILE = 1,
-        LAYOUTRECALL4_FSID = 2,
-        LAYOUTRECALL4_ALL  = 3
+        LAYOUTRECALL4_FILE = LAYOUT4_RET_REC_FILE,
+        LAYOUTRECALL4_FSID = LAYOUT4_RET_REC_FSID,
+        LAYOUTRECALL4_ALL  = LAYOUT4_RET_REC_ALL
 };
 
 struct layoutrecall_file4 {
Index: dotx.d/spit_types.sh
===================================================================
RCS file: /cvs/stdk/NFSv41/dotx.d/spit_types.sh,v
retrieving revision 1.40
diff -u -w -b -r1.40 spit_types.sh
--- dotx.d/spit_types.sh	16 Dec 2007 21:07:55 -0000	1.40
+++ dotx.d/spit_types.sh	18 Dec 2007 14:58:29 -0000
@@ -428,10 +428,15 @@
 
 cat << EOF > $i
 %
+/* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
+const LAYOUT4_RET_REC_FILE	= 1;
+const LAYOUT4_RET_REC_FSID	= 2;
+const LAYOUT4_RET_REC_ALL	= 3;
+%
 enum layoutreturn_type4 {
-	LAYOUTRETURN4_FILE = 1,
-	LAYOUTRETURN4_FSID = 2,
-	LAYOUTRETURN4_ALL  = 3
+	LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
+	LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
+	LAYOUTRETURN4_ALL  = LAYOUT4_RET_REC_ALL
 };
 
 struct layoutreturn_file4 {
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Tue, 18 Dec 2007 19:17:23 +0200

> If others really want it and think what is there now is broken, 

Monitoring the client's progress is really important if the client is
flushing data since the time that could take it is indefinite.
Rather than polling it by resending CB_LAYOUTRECALL we can add
the counter to LAYOUTRETURN for LAYOUTRETURN4_{FSID,ALL} and
the client will send these periodically (this will also explicitly
renew its lease) until done (when the counter reaches zero).

    union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
            case LAYOUTRETURN4_FILE:
                    layoutreturn_file4      lr_layout;
+           case LAYOUTRETURN4_FSID:
+           case LAYOUTRETURN4_ALL:
+		   uint32_t                lr_count;
+		   uint32_t                lr_total;
            default:
                    void;
    };

>> fsid in the RECALL_FSID case, the server returns

New diffs look good to me :)