Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

From: <[email-masked]>
Date: Tue, 5 Jul 2011 17:02:29 -0400

> > Has anyone volunteered to present something on the LAYOUT_COMMIT issues
> > we are turning into a 5661 errata and a NFSv4.2 update?

Uhm - I guess he means me ;-).

It looks like the change is confined to one paragraph, in section 18.42.3 of RFC 5661:

OLD

   The LAYOUTCOMMIT operation commits changes in the layout represented
   by the current filehandle, client ID (derived from the session ID in
   the preceding SEQUENCE operation), byte-range, and stateid.  Since
   layouts are sub-dividable, a smaller portion of a layout, retrieved
   via LAYOUTGET, can be committed.  The byte-range being committed is
   specified through the byte-range (loca_offset and loca_length).  This
   byte-range MUST overlap with one or more existing layouts previously
   granted via LAYOUTGET (Section 18.43), each with an iomode of
   LAYOUTIOMODE4_RW.  In the case where the iomode of any held layout
   segment is not LAYOUTIOMODE4_RW, the server should return the error
   NFS4ERR_BAD_IOMODE.  For the case where the client does not hold
   matching layout segment(s) for the defined byte-range, the server
   should return the error NFS4ERR_BAD_LAYOUT.

NEW

   The LAYOUTCOMMIT operation commits changes in the layout represented
   by the current filehandle, client ID (derived from the session ID in
   the preceding SEQUENCE operation), and stateid.  As a layout-independent
   operation, LAYOUTCOMMIT commits the entire layout; layout-specific data
   (loca_layoutupdate) may specify a layout subset that requires layout-
   specific actions (e.g., for the block layout, see RFC 5663 [41]).
   The loca_offset and loca_length arguments have been deprecated.  The
   client SHOULD set loca_offset to 0 and SHOULD set loca_length to the
   current end-of-file offset.  The server MUST ignore loca_offset and
   loca_length.  The client MUST hold one or more existing layouts
   previously    granted via LAYOUTGET (Section 18.43), with an iomode of
   LAYOUTIOMODE4_RW.  In the case where no previously granted layout
   has an iomode of LAYOUTIOMODE4_RW, the server should return the error
   NFS4ERR_BAD_IOMODE.  For the case where the client does not hold
   any previously granted layout, the server should return the error
   NFS4ERR_BAD_LAYOUT.

** IMPORTANT - pNFS implementers **
Please check that your implementation does or will comply with:

   The client SHOULD set
   loca_offset to 0 and SHOULD set loca_length to the current end-of-
   file offset.  The server MUST ignore loca_offset and loca_length.

The problematic scenario is new client talking to an unmodified server, as an
unmodified server will detect the new client trying to commit portions of the file
for which the client has no layout (particularly for block pNFS).  If we have to
accommodate unmodified servers, we probably cannot change RFC 5661.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Benny Halevy
> Sent: Friday, June 10, 2011 8:10 AM
> To: Thomas Haynes
> Cc: [email-masked]
> Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT
> 
> On 2011-06-09 12:09, Thomas Haynes wrote:
> > Has anyone volunteered to present something on the LAYOUT_COMMIT issues
> > we are turning into a 5661 errata and a NFSv4.2 update?
> >
> 
> I'm not sure yet if I can make it to this meeting or not.
> I'm all for 5661 errata and a NFSv4.2 update as there's no
> wire format change required and there were no reservations
> about it from any server vendor.
> 
> Benny
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Spencer Shepler" <[email-masked]>
Date: Tue, 5 Jul 2011 14:04:40 -0700

How does this correlate with what Ricardo captured and passed
by the WG earlier this year?

The text of which can be found here:

http://www.rfc-editor.org/errata_search.php?eid=2751

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Tuesday, July 05, 2011 2:02 PM
> To: [email-masked]
> Cc: [email-masked]
> > > Has anyone volunteered to present something on the LAYOUT_COMMIT
> > > issues we are turning into a 5661 errata and a NFSv4.2 update?
> 
> The problematic scenario is new client talking to an unmodified server, as
> an unmodified server will detect the new client trying to commit portions
> of the file for which the client has no layout (particularly for block
> pNFS).  If we have to accommodate unmodified servers, we probably cannot
> change RFC 5661.
> 
> Thanks,
> --David
> 
> 
> > > Has anyone volunteered to present something on the LAYOUT_COMMIT
> > > issues we are turning into a 5661 errata and a NFSv4.2 update?
> > >
> >
> > I'm all for 5661 errata and a NFSv4.2 update as there's no wire format
> > change required and there were no reservations about it from any
> > server vendor.
> >
> > Benny

---

From: <[email-masked]>
Date: Tue, 5 Jul 2011 17:13:42 -0400

That's a completely separate and distinct issue.

Thanks,
--David

> From: Spencer Shepler [mailto:[email-masked]]
> Sent: Tuesday, July 05, 2011 5:05 PM
> To: Black, David; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] Quebec and LAYOUT_COMMIT
> 
>

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 5 Jul 2011 21:54:40 +0000

Oh, good that we refer to them as LAYOUTCOMMIT issue. :-)

Having straightened that out.

Why add language specific to 5663 or other non-file layout types
in 5661?  If special handling is needed, presumably it is in the
context of the layout type and can be captured in the RFC for
that layout type.  Therefore, shouldn't the 5661 update be more
generic and then make it clear that additional MUST/SHOULD handling
of LAYOUTCOMMIT will be captured in 5663, etc.?

Spencer

> Sent: Tuesday, July 05, 2011 2:14 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> > How does this correlate with what Ricardo captured and passed by the
> > WG earlier this year?
> >
> > > From: [email-masked] [mailto:[email-masked]] On
> > > Behalf Of [email-masked]
> > > The problematic scenario is new client talking to an unmodified
> > > server, as an unmodified server will detect the new client trying to
> > > commit portions of the file for which the client has no layout
> > > (particularly for block pNFS).  If we have to accommodate unmodified
> > > servers, we probably cannot change RFC 5661.
> > >
> > > Thanks,
> > > --David
> > >
> > >
> > > > I'm all for 5661 errata and a NFSv4.2 update as there's no wire
> > > > format change required and there were no reservations about it
> > > > from any server vendor.
> > > >
> > > > Benny

---

From: <[email-masked]>
Date: Tue, 5 Jul 2011 18:16:07 -0400

Spencer,

> Oh, good that we refer to them as LAYOUTCOMMIT issue. :-)

Two's company, three's a crowd :-).  Let's not go there [three] ;-).

> of LAYOUTCOMMIT will be captured in 5663, etc.?

That's not what's going on.  Most of the changed text, including
deprecation of the two arguments, is layout-type-independent, and hence
belongs in 5661 because the arguments are deprecated for all layout types
 - file, block, object and any that may be defined in the future.  The
text changes take the functionality out of the common protocol rather than
requiring each layout type to independently work around it.

The only language specific to 5663 (block layout) is the following:

    layout-specific data
    (loca_layoutupdate) may specify a layout subset that requires layout-
    specific actions (e.g., for the block layout, see RFC 5663 [41])

The words up to "actions" need to be in 5661 to avoid having 5663 conflict
with 5661, and I included the parenthetical example to help out the reader
who wonders why it might be a good idea for the layout-specific data to be
used for something like that.

Thanks,
--David

> From: Spencer Shepler [mailto:[email-masked]]
> Sent: Tuesday, July 05, 2011 5:55 PM
> To: Black, David; [email-masked]; [email-masked]
> Cc: [email-masked]
> > > > > Behalf Of Benny Halevy

---

From: [email-masked]
Date: Wed, 6 Jul 2011 07:28:48 +0000

Why is the change specific to the block layout? The analysis we did turned out that the LAYOUTCOMMIT byte range is redundant for all layout types, therefore I see no good reason to fork the standard semantics per layout type in this case.

Benny
Sent from my BlackBerry® smartphone from orange

-----Original Message-----
From: <[email-masked]>
Sender: [email-masked]
Date: Tue, 5 Jul 2011 18:16:07 
To: <[email-masked]>; <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

Spencer,

    layout-specific data

Thanks,
--David

---

From: <[email-masked]>
Date: Wed, 6 Jul 2011 08:50:13 -0400

Let's separate the discussion of what the proposed change does do as far as layout-type specificity, which has been confused, from the separate issue of what it should say.  Until Benny's message, I had thought there was agreement on the following.

     The general rule is that the range is not (SHOULD NOT BE) used.

     Specific layout types may make exceptions, for non-file layout types presumably in 
     documents other than RFC5661 and bis-like descendants.

There may also have been disagreement about whether David's text by simply referring to the possibility of exceptions in specific other documents has inappropriately crossed some sort of line, but regardless of that I hadn't seen any disagreement on the two points above.

I'm not sure if Benny is disagreeing since he is only saying that he knows of no reason to make this exception for block.

So I think the point here is that allowing the exceptions is an OK thing in general, but that if Benny wants to go on from seeing no reason to use this range in pNFS block to saying that there is none, he needs to press his point in the context of RFC5663 making it easy for me to remain uninvolved in that discussion.

Or is Benny really objecting to the fact that layout-type-specific exceptions are allowed, in which case I'm going to have to be involved in the discussion in order to explain to him why he is quite wrong :-)
 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, July 06, 2011 3:29 AM
To: Black, David; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

Benny
Sent from my BlackBerry(r) smartphone from orange

-----Original Message-----
From: <[email-masked]>
Sender: [email-masked]
Date: Tue, 5 Jul 2011 18:16:07 
To: <[email-masked]>; <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

Spencer,

    layout-specific data

Thanks,
--David

---

From: [email-masked]
Date: Wed, 6 Jul 2011 14:40:46 +0000

I have no objection to the general rule of allowing layout-type specific exceptions and to documenting those outside RFC5661 for non-file layout types as long as RFC5661 allows such freedom and documents that.

In this particular case I think there's no good reason to allow such freedom and the errata should state that the range SHOULD NOT be used by any layout type, and that byte-range information for the particular ranges of the layout that are being LAOUTCOMMITTED SHOLD be encoded in the layout-type specific payload.

Benny

 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, July 06, 2011 3:29 AM
To: Black, David; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

Benny

-----Original Message-----
From: <[email-masked]>
Sender: [email-masked]
Date: Tue, 5 Jul 2011 18:16:07 
To: <[email-masked]>; <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [nfsv4] Quebec and LAYOUT_COMMIT

Spencer,

    layout-specific data

Thanks,
--David

---

From: <[email-masked]>
Date: Wed, 6 Jul 2011 14:35:21 -0400

The text that I wrote does what Benny describes:

1) Deprecates use of the byte range in the NFSv4.1 LAYOUTCOMMIT operation
	- client SHOULD set the byte range arguments to cover entire file
	- server MUST ignore the byte range arguments (yes, "MUST" is necessary).

2) Allows layout-specific actions on specific byte to be specified in layout
	specific data, and cites the block layout as an example (the only one at
	this time).

The actual change is not specific to the block layout.

I believe that Benny, Dave Noveck and I are in agreement.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, July 06, 2011 10:41 AM
> To: Noveck, David; Black, David; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Cc: [email-masked]
> I have no objection to the general rule of allowing layout-type specific exceptions and to documenting
> those outside RFC5661 for non-file layout types as long as RFC5661 allows such freedom and documents
> that.
> 
> In this particular case I think there's no good reason to allow such freedom and the errata should
> state that the range SHOULD NOT be used by any layout type, and that byte-range information for the
> particular ranges of the layout that are being LAOUTCOMMITTED SHOLD be encoded in the layout-type
> specific payload.
> 
> Benny
> To: <[email-masked]>; <[email-masked]>; <[email-masked]>; <[email-masked]>;
> <[email-masked]>; <[email-masked]>
> Cc: <[email-masked]>
> Let's separate the discussion of what the proposed change does do as far as layout-type specificity,
> which has been confused, from the separate issue of what it should say.  Until Benny's message, I had
> thought there was agreement on the following.
> 
> There may also have been disagreement about whether David's text by simply referring to the
> possibility of exceptions in specific other documents has inappropriately crossed some sort of line,
> but regardless of that I hadn't seen any disagreement on the two points above.
> 
> So I think the point here is that allowing the exceptions is an OK thing in general, but that if Benny
> wants to go on from seeing no reason to use this range in pNFS block to saying that there is none, he
> needs to press his point in the context of RFC5663 making it easy for me to remain uninvolved in that
> discussion.
> 
> Or is Benny really objecting to the fact that layout-type-specific exceptions are allowed, in which
> case I'm going to have to be involved in the discussion in order to explain to him why he is quite
> wrong :-)
> 
> Why is the change specific to the block layout? The analysis we did turned out that the LAYOUTCOMMIT
> byte range is redundant for all layout types, therefore I see no good reason to fork the standard
> semantics per layout type in this case.
> 
> Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 06:55:14 +0300

Good, I'm convinced we're all in agreement.

One comment regarding your proposed text:
It is kind of tricky to say that the client SHOULD set loca_length to
the current end-of-file offset, and later in section 18.42.3. it is said
"loca_last_write_offset MUST overlap the range described by loca_offset and loca_length"

One, I bet you wanted "the current end-of-file offset" to be the one
the client wishes, i.e., loca_last_write_offset+1, when present.

Two, if we deprecate loca_offsett and loca_length we should also clean up
the other constraint for loca_last_write_offset.
The constraint should be that it MUST overlap with a LAYOUTIOMODE4_RW layout
the client holds and in particular, if loca_layoutupdate4 restricts the
parts of the layout that are being LAYOUTCOMMiTTed, loca_last_write_offset
must overlap one of the said ranges.

Also, other required changes in section 
12.7.4. Recovery from Metadata Server Restart

OLD

      If the metadata server's
      consistency checks on loca_layoutupdate succeed, then the metadata
      server MUST commit the data (as described by the loca_offset,
      loca_length, and loca_layoutupdate fields of the arguments) that
      was written to the storage device.

OLD

      If the metadata server's
      consistency checks on loca_layoutupdate succeed, then the metadata
      server MUST commit the data (as described by the loca_layoutupdate
      field of the arguments) that
      was written to the storage device.

and further down in the same section:

OLD

      However, any time LAYOUTCOMMIT with loca_reclaim
      TRUE fails, the pNFS client has lost all the data in the range
      defined by <loca_offset, loca_length>.

NEW

      However, any time LAYOUTCOMMIT with loca_reclaim
      TRUE fails, the pNFS client may have lost data which layout was not previously
      committed using LAYOUTCOMMIT.
      Note that for the files layout, as described in Section 13,
      it is sufficient to commit the data either using COMMIT to the MDS
      when NFL4_UFLG_COMMIT_THRU_MDS is set or to rely on the Data Server returning
      FILE_SYNC4 for the respective WRITE operations.

And finally:

OLD

      As with the scenario in
      the above bullet point, the failure of LAYOUTCOMMIT means the data
      in the range <loca_offset, loca_length> lost.

NEW

      As with the scenario in
      the above bullet point, the failure of LAYOUTCOMMIT means data
      loss.

Thanks,

Benny

---

From: <[email-masked]>
Date: Thu, 7 Jul 2011 12:32:51 -0400

All are good catches, although I think the sentence about the last write offset overlapping
the byte range belongs in the bit-bucket.  I'll send a consolidated set of changes out later today.

Thanks,
--David

> From: Benny Halevy [mailto:[email-masked]]
> Sent: Wednesday, July 06, 2011 11:55 PM
> To: Black, David
> Cc: [email-masked]
> >> I'm not sure if Benny is disagreeing since he is only saying that he knows of no reason to make
> this
> >> exception for block.
> >>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 19:42:28 +0300

> the byte range belongs in the bit-bucket.  I'll send a consolidated set of changes out later today.

I'm afraid that otherwise a client could expose an EOF which falls
in a byte range it had not yet committed, e.g. in the case that due to message size
limitation it cannot LAYOUTCOMMIT all the stable ranges in a single call.

Benny