Subject: [nfsv4] use of current stateid for layout ops

From: Benny Halevy <[email-masked]>
Date: Wed, 05 Mar 2008 15:51:54 +0200

Here's a proposal for specifying the correct usage of current stateid
for layout operations.

Benny

diff a/nfsv41_middle_pnfs.xml b/nfsv41_middle_pnfs.xml
--- a/nfsv41_middle_pnfs.xml
+++ b/nfsv41_middle_pnfs.xml
@@ -579,7 +579,7 @@
 
 <section title="Layout Stateid" anchor="layout_stateid">
 <t>
- As with all other stateids, the stateid consists of a
+ As with all other stateids, the layout stateid consists of a
  "seqid" and "other" field. Once a layout stateid is
  changed, the "other" field will stay constant unless
  the stateid is revoked, or the client returns all layouts
@@ -590,11 +590,39 @@
  operation. After the layout stateid is
  established, the "seqid" is incremented by the server
  in each subsequent LAYOUTGET and LAYOUTRETURN response,
- and in each CB_LAYOUTRECALL request. When the
+ and in each CB_LAYOUTRECALL request.
+</t>
+<t>
+ The client MUST use either one of the following stateids for layout stateids:
+ <list style="symbols">
+ <t>
+ When a client has no layout on a file,
+ it MUST present a stateid as returned by OPEN,
+ a delegation stateid, or a byte-range lock stateid.
+ The current stateid MAY be used for the first
+ LAYOUTGET operation following an OPEN or LOCK operation
+ in the same COMPOUND procedure.
+ </t>
+ <t>
+ Once the client establishes a layout stateid,
+ it MUST always use a "seqid" that was returned by the
+ server in a reply to a LAYOUTGET or LAYOUTRETURN operation, or sent by
+ the server in a CB_LAYOUTRECALL operation.
+ </t>
+ <t>
+ When multiple LAYOUTGET, LAYOUTCOMMIT, and/or LAYOUTRETURN operations
+ are sent in the same COMPOUND procedure,
+ the client MUST use the current stateid on all but the first layout operations
+ so to distinguish them from a set of layout operations sent in parallel.
+ </t>
+ </list>
+</t>
+<t>
+ When the
  client fully processes the response to a LAYOUTGET or LAYOUTRETURN,
  or fully processes the arguments of a CB_LAYOUTRECALL, it
  MUST use the seqid of the stateid of the reply from LAYOUTGET and
- LAYOUTRETURN, or the seqid of the stateid in the arguments of CB_LAYOUTRECALL,
+ LAYOUTRETURN, or the seqid of the stateid in the arguments of CB_LAYOUTRECALL
  on subsequent calls to LAYOUTGET
  or LAYOUTRETURN.
  The client and server use
@@ -625,10 +653,10 @@
  </list>
 </t>
 <t>
- The client MUST always use a "seqid" that was returned by the
+ When using a "seqid" that was returned by the
  server in a reply to a LAYOUTGET or LAYOUTRETURN operation, or sent by
- the server in a CB_LAYOUTRECALL operation. It MUST only use such
- a seqid after processing the LAYOUTGET and LAYOUTRETURN results,
+ the server in a CB_LAYOUTRECALL operation, the seqid MUST only be used
+ after processing the LAYOUTGET and LAYOUTRETURN results,
  or the CB_LAYOUTRECALL request. Simply seeing the result or
  the CB_LAYOUTRECALL request is not sufficient cause to use the
  seqid. For LAYOUTGET results, if the client is not
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 6 Mar 2008 21:09:42 -0500

> + When a client has no layout on a file,  it MUST present a stateid as

> +returned by OPEN,  a delegation stateid, or a byte-range lock
stateid.

I assume it is also valid to modify the stateid to put zero in the seqid
field just as may be done for READs and WRITEs for example.

> + The current stateid MAY be used for the first  LAYOUTGET operation 
> +following an OPEN or LOCK operation  in the same COMPOUND procedure.

If you use the current stateid, zero will be substituted for the seqid
so if that is an issue for the previous sentence, it is an issue here.

I think trying to specify the exact permutations is going to get you in
trouble (e.g. what about LAYOUTGET following LOCKU, OPEN_DOWNGRADE?).  I
think the right answer here is to specify that in general current
stateid may be used and any constraints apply to the stateid after
replacement of the special stateid designating the current stateid by
the current stateid itself. 

> + </t>
> + <t>
> + Once the client establishes a layout stateid,  it MUST always use a 
> +"seqid" that was returned by the  server in a reply to a LAYOUTGET or

> +LAYOUTRETURN operation, or sent by  the server in a CB_LAYOUTRECALL 
> +operation.

Why?  I have heard no explanation for this restriction.  Spencer has
promised a writeup but I don't know when that will be forthcoming and
even without a detailed writeup of this, Spencer  should be able to
provide us a brief explanation of the motivation for this restriction.

The big problem with it, besides it being unmotivated, is that it
prevents parallel opens from working right.  If you issue two parallel
requests that both have LAYOUTGETs there is no way of making sure they
are processed in the same order that you used for assinging seqids and
they may be rejected unnecessarily.  Also, even if the order is right,
an error in the first one will mean the seqid for the second stateid is
wrong and thus rejected.

> +are sent in the same COMPOUND procedure,  the client MUST use the 
> +current stateid on all but the first layout operations  so to 
> +distinguish them from a set of layout operations sent in parallel.

I think this is a good idea but why the "MUST"?  If I issue a LAYOUTGET
with "other" field O1 and seqid S why can't I issue a second with
"other" field O1 and seqid S+1, or one with "other" field O1 and seqid
0.  After all the the current stateid substituted if I use that special
stateid will have O1 in the "other" field and seqid 0.

I don't understand "so to distinguish them from a set of layout
operations sent in parallel".  Why does the server need to distinguish
these cases, he may very well have multiple parallel requests each of
which has multiple LAYOUTGETs.  The obvious thing to do is to send these
all with seqids of zero either explicitly or implicitly, because the
current stateid is used.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Wednesday, March 05, 2008 8:52 AM
To: NFSv4
Subject: [nfsv4] use of current stateid for layout ops

Benny

 <section title="Layout Stateid" anchor="layout_stateid">  <t>
- As with all other stateids, the stateid consists of a
+ As with all other stateids, the layout stateid consists of a
  "seqid" and "other" field. Once a layout stateid is
  changed, the "other" field will stay constant unless
  the stateid is revoked, or the client returns all layouts @@ -590,11
+590,39 @@
  operation. After the layout stateid is
  established, the "seqid" is incremented by the server
  in each subsequent LAYOUTGET and LAYOUTRETURN response,
- and in each CB_LAYOUTRECALL request. When the
+ and in each CB_LAYOUTRECALL request.
+</t>
+<t>
+ The client MUST use either one of the following stateids for layout
stateids:
+ <list style="symbols">
+ <t>
+ When a client has no layout on a file,  it MUST present a stateid as 
+returned by OPEN,  a delegation stateid, or a byte-range lock stateid.
+ The current stateid MAY be used for the first  LAYOUTGET operation 
+following an OPEN or LOCK operation  in the same COMPOUND procedure.
+ </t>
+ <t>
+ Once the client establishes a layout stateid,  it MUST always use a 
+"seqid" that was returned by the  server in a reply to a LAYOUTGET or 
+LAYOUTRETURN operation, or sent by  the server in a CB_LAYOUTRECALL 
+operation.
+ </t>
+ <t>
+ When multiple LAYOUTGET, LAYOUTCOMMIT, and/or LAYOUTRETURN operations

+are sent in the same COMPOUND procedure,  the client MUST use the 
+current stateid on all but the first layout operations  so to 
+distinguish them from a set of layout operations sent in parallel.
+ </t>
+ </list>
+</t>
+<t>
+ When the
  client fully processes the response to a LAYOUTGET or LAYOUTRETURN,
  or fully processes the arguments of a CB_LAYOUTRECALL, it
  MUST use the seqid of the stateid of the reply from LAYOUTGET and
- LAYOUTRETURN, or the seqid of the stateid in the arguments of
CB_LAYOUTRECALL,
+ LAYOUTRETURN, or the seqid of the stateid in the arguments of 
+ CB_LAYOUTRECALL
  on subsequent calls to LAYOUTGET
  or LAYOUTRETURN.
  The client and server use
@@ -625,10 +653,10 @@
  </list>
 </t>
 <t>
- The client MUST always use a "seqid" that was returned by the
+ When using a "seqid" that was returned by the
  server in a reply to a LAYOUTGET or LAYOUTRETURN operation, or sent by
- the server in a CB_LAYOUTRECALL operation. It MUST only use such
- a seqid after processing the LAYOUTGET and LAYOUTRETURN results,
+ the server in a CB_LAYOUTRECALL operation, the seqid MUST only be used

---

From: Benny Halevy <[email-masked]>
Date: Fri, 07 Mar 2008 11:17:11 +0200

> field just as may be done for READs and WRITEs for example.

Agreed.

> so if that is an issue for the previous sentence, it is an issue here.

Agreed.

> the current stateid itself.

OK. That makes sense.
 
> even without a detailed writeup of this, Spencer  should be able to

My take on this is that the server needs the seqids to serialize
the ops across a CB_LAYOUTRECALL so that it can reject and op
that carries a seqid which is earlier than the one recalled.

That said, I think that is we only mandate presenting the layout stateid
given by CB_LAYOUTRECALL on all LAYOUTRETURNs (and LAYOUTCOMMITs actually)
that are in response to it, the server can still implement the layout recall
"barrier" by rejecting all GETs and RETURNs with any other seqid (zero or not).
Voluntary LAYOUTGET, COMMIT, and RETURNs are explicitly ordered by the seqid
on their respective replies anyway so it doesn't really matter what seqid they
carry in the args, as long as it is never equal to a seqid given on a 
CB_LAYOUTRECALL.

> an error in the first one will mean the seqid for the second stateid is

I see the problem and I'm not sure how to solve it gracefully with layout
stateids. We may want to advise the client to serialize the first LAYOUTGET
it sends for a file with any other one (FWIW, that's what we decided to do
for the linux pnfs client)

> stateid will have O1 in the "other" field and seqid 0.

Yeah, that's the same issue of allowing zero seqid.

> all with seqids of zero either explicitly or implicitly, because the

A bit more on this in 12.5.5.2.1.1:

   It is permissible for the client to combine LAYOUTRETURN and
   LAYOUTGET operations for the same file in the same COMPOUND request
   since the server MUST process these in order.  The client uses the
   current stateid (see Section 16.2.3.1.2).  However, if a client does
   send such COMPOUND requests, it MUST NOT have more than one
   outstanding for the same file at the same time and MUST NOT have
   other LAYOUTGET or LAYOUTRETURN operations outstanding at the same
   time for that same file.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 7 Mar 2008 08:25:56 -0500

> My take on this is that the server needs the seqids 
> to serialize the ops across a CB_LAYOUTRECALL so that 
> it can reject and op that carries a seqid which is 
> earlier than the one recalled.

But that doesn't provide the requisite serialization either, at least in
the presence of parallel LAYOUTGETs.  Suppose I issue LAYOUTGETs (all
the same "other" field) with seqids
S, S+1, S+2, all in parallel.  Now a CB_LAYOUTRECALL with stateid seqid
S+1 is issued and the server gets the LAYOUTGET with S+2, which is not
earlier than the one recalled.  

> That said, I think that is we only mandate presenting the 
> layout stateid given by CB_LAYOUTRECALL on all LAYOUTRETURNs 
> (and LAYOUTCOMMITs actually) that are in response to it, the 
> server can still implement the layout recall "barrier" by 
> rejecting all GETs and RETURNs with any other seqid (zero or 
> not).

I think this makes sense.  But if you have multiple layout operations in
response to the CB_LAYOUTRECALL, you either have to make special
exception for seqids in layout operations and allow the server to accept
one which is not current but is the seqid sent with an earlier recall,
or you stick with the current checking for OLD_STATEID and the server
will allow a seqid that is either sent in the CB_LAYOUTRECALL or is one
sent in the response for operation sent to resolve a layout recall.

One interesting case to think about is the case of multiple parallel
CB_LAYOUTRECALLs.  When the first recall occurs, it is fine that that
causing pending voluntary operations to be rejected.  OTOH, if you have
a recall in effect and operations are being done pursuant to it, it
wouldn't be good if another layout recall interfered with return
operations.
 
> Voluntary LAYOUTGET, COMMIT, and RETURNs are explicitly 
> ordered by the seqid on their respective replies anyway 
> so it doesn't really matter what seqid they carry in the 
> args, as long as it is never equal to a seqid given on a 
> CB_LAYOUTRECALL. 

Right.  To prevent that, the client MAY use zero or a stateid-seqid it
got from the server but it MUST NOT take a seqid it got from the server
and modify it (e.g. by incrementing it) since that may cause the client
to inadvertently "forge" the seqid for a subsequent recall.

> I see the problem and I'm not sure how to solve it 
> gracefully with layout stateids. We may want to advise 
> the client to serialize the first LAYOUTGET it sends for 
> a file with any other one (FWIW, that's what we decided 
> to do for the linux pnfs client)

When I speak of parallel LAYOUTGETs, I'm assuming that these are after
the first one which must be serialized.  Subsequent ones require that
you know the "other" field that the server will generate and unless your
client is psychic, how can it guess that 96-bit value before it gets it
from the server?

If this needs to be more explicit in the spec, let's make it so.

One other issue is that if I issue two initial LAYOUTGETs (same fh,
different open/lock stateids), then presumably the second one is in
error but does the spec make clear what error?

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, March 07, 2008 4:17 AM
To: Noveck, Dave
Cc: NFSv4
Subject: Re: [nfsv4] use of current stateid for layout ops

On Mar. 07, 2008, 4:09 +0200, "Noveck, Dave" <[email-masked]>
wrote:
>> + When a client has no layout on a file,  it MUST present a stateid 
>> + as
> 
> I assume it is also valid to modify the stateid to put zero in the 
> seqid field just as may be done for READs and WRITEs for example.

Agreed.

Agreed.

> 
> I think trying to specify the exact permutations is going to get you 
> in trouble (e.g. what about LAYOUTGET following LOCKU, 
> OPEN_DOWNGRADE?).  I think the right answer here is to specify that in

> general current stateid may be used and any constraints apply to the 
> stateid after replacement of the special stateid designating the 
> current stateid by the current stateid itself.

OK. That makes sense.

>> +"seqid" that was returned by the  server in a reply to a LAYOUTGET 
>> +or
> 

My take on this is that the server needs the seqids to serialize the ops
across a CB_LAYOUTRECALL so that it can reject and op that carries a
seqid which is earlier than the one recalled.

That said, I think that is we only mandate presenting the layout stateid
given by CB_LAYOUTRECALL on all LAYOUTRETURNs (and LAYOUTCOMMITs
actually) that are in response to it, the server can still implement the
layout recall "barrier" by rejecting all GETs and RETURNs with any other
seqid (zero or not).
Voluntary LAYOUTGET, COMMIT, and RETURNs are explicitly ordered by the
seqid on their respective replies anyway so it doesn't really matter
what seqid they carry in the args, as long as it is never equal to a
seqid given on a CB_LAYOUTRECALL.

> an error in the first one will mean the seqid for the second stateid 
> is wrong and thus rejected.

I see the problem and I'm not sure how to solve it gracefully with
layout stateids. We may want to advise the client to serialize the first
LAYOUTGET it sends for a file with any other one (FWIW, that's what we
decided to do for the linux pnfs client)

> I think this is a good idea but why the "MUST"?  If I issue a 
> LAYOUTGET with "other" field O1 and seqid S why can't I issue a second

> with "other" field O1 and seqid S+1, or one with "other" field O1 and 
> seqid 0.  After all the the current stateid substituted if I use that 
> special stateid will have O1 in the "other" field and seqid 0.

> which has multiple LAYOUTGETs.  The obvious thing to do is to send 
> these all with seqids of zero either explicitly or implicitly, because

> the current stateid is used.

Benny

stateid.

> + When multiple LAYOUTGET, LAYOUTCOMMIT, and/or LAYOUTRETURN 
> +operations
> 
> + the server in a CB_LAYOUTRECALL operation, the seqid MUST only be 
> + used
>

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 7 Mar 2008 09:38:19 -0500

> earlier than the one recalled.  

My understanding is that in the current model the client MUST not increment
the seqid as you described but rather send the last seqid it received
on all concurrent layout requests.

> sent in the response for operation sent to resolve a layout recall.

Like above, I think all responses the a specific layout recall should carry
its seqid.

> wouldn't be good if another layout recall interfered with return

I think it'd simpler and OK to implicitly invalidate a previous layout
recall when a new one is sent.  If the server still needs to recall
whatever has not been returned it should cover that range in the
new cb_layoutrecall it produces.

> to inadvertently "forge" the seqid for a subsequent recall.

Right.

> client is psychic, how can it guess that 96-bit value before it gets it

We're in agreement then.  I referred to sending multiple OPENs in parallel,
each carrying a LAYOUGET that uses the current stateid.

> error but does the spec make clear what error?

Good question.  I'm not sure.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, March 07, 2008 4:17 AM
To: Noveck, Dave
Cc: NFSv4
Subject: Re: [nfsv4] use of current stateid for layout ops

Agreed.

Agreed.

OK. That makes sense.

> operations

Benny

stateid.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 7 Mar 2008 10:14:00 -0500

> on all concurrent layout requests.

agreed. note that in this model the client can send decreasing
stateid.seqids due to out of order replys from the server. this is not bad,
just different an perhaps should be mentioned.

> Good question.  I'm not sure.

this is not an error. the server simply checks to see if the new open
stateid is valid, and then returns the established layout stateid with an
incremented seqid.

-->Andy

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 7 Mar 2008 10:45:21 -0500

> My understanding is that in the current model 
> the client MUST not increment the seqid as you 
> described but rather send the last seqid it 
> received on all concurrent layout requests.

But if it does that, the general rules for stateid seqids would cause
some of these to be rejected with an OLD_STATEID error.  We have a
chapter that explains stateids and the seqid fields and if layout
stateids are going to use a different model, then the state management
chapter needs at least a mention that the general stateid model for this
doesn't apply and an xref to section that clearly explains the layout
stateid model for this.  Does this section exist in the spec currently
or is this what we are waiting for Spencer to write?  In that latter
event, I think we need a clear deadline for completion of that.  Mike
Eisler has suggested COB today, but there has been no confirmation from
Spencer on that or a proposal for an alternate deadline.  

-----Original Message-----
From: Halevy, Benny [mailto:[email-masked]] 
Sent: Friday, March 07, 2008 9:38 AM
To: Noveck, Dave
Cc: NFSv4
Subject: RE: [nfsv4] use of current stateid for layout ops

Noveck, Dave <[email-masked]> Wrote:
 
> > > Why?  I have heard no explanation for this restriction.  Spencer 
> > > has promised a writeup but I don't know when that will be 
> > > forthcoming and even without a detailed writeup of this, Spencer  
> > > should be able to
> 
> > My take on this is that the server needs the seqids to serialize the

> > ops across a CB_LAYOUTRECALL so that it can reject and op that 
> > carries a seqid which is earlier than the one recalled.
> 
> But that doesn't provide the requisite serialization either, at least 
> in the presence of parallel LAYOUTGETs.  Suppose I issue LAYOUTGETs 
> (all the same "other" field) with seqids S, S+1, S+2, all in parallel.

> Now a CB_LAYOUTRECALL with stateid seqid

My understanding is that in the current model the client MUST not
increment the seqid as you described but rather send the last seqid it
received on all concurrent layout requests.

> > That said, I think that is we only mandate presenting the layout 
> > stateid given by CB_LAYOUTRECALL on all LAYOUTRETURNs (and 
> > LAYOUTCOMMITs actually) that are in response to it, the server can 
> > still implement the layout recall "barrier" by rejecting all GETs 
> > and RETURNs with any other seqid (zero or not).
> 
> I think this makes sense.  But if you have multiple layout operations 
> in response to the CB_LAYOUTRECALL, you either have to make special 
> exception for seqids in layout operations and allow the server to 
> accept one which is not current but is the seqid sent with an earlier 
> recall, or you stick with the current checking for OLD_STATEID and the

> server will allow a seqid that is either sent in the CB_LAYOUTRECALL 
> or is one sent in the response for operation sent to resolve a layout
recall.

Like above, I think all responses the a specific layout recall should
carry its seqid.

> causing pending voluntary operations to be rejected.  OTOH, if you 
> have a recall in effect and operations are being done pursuant to it, 
> it wouldn't be good if another layout recall interfered with return 
> operations.

I think it'd simpler and OK to implicitly invalidate a previous layout
recall when a new one is sent.  If the server still needs to recall
whatever has not been returned it should cover that range in the new
cb_layoutrecall it produces.

>  
> > Voluntary LAYOUTGET, COMMIT, and RETURNs are explicitly ordered by 
> > the seqid on their respective replies anyway so it doesn't really 
> > matter what seqid they carry in the args, as long as it is never 
> > equal to a seqid given on a CB_LAYOUTRECALL.
> 

> got from the server but it MUST NOT take a seqid it got from the 
> server and modify it (e.g. by incrementing it) since that may cause 
> the client to inadvertently "forge" the seqid for a subsequent recall.

Right.

> > layout stateids. We may want to advise the client to serialize the 
> > first LAYOUTGET it sends for a file with any other one (FWIW, that's

> > what we decided to do for the linux pnfs client)
> 

> you know the "other" field that the server will generate and unless 
> your client is psychic, how can it guess that 96-bit value before it 
> gets it from the server?

We're in agreement then.  I referred to sending multiple OPENs in
parallel, each carrying a LAYOUGET that uses the current stateid.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]]
Sent: Friday, March 07, 2008 4:17 AM
To: Noveck, Dave
Cc: NFSv4
Subject: Re: [nfsv4] use of current stateid for layout ops

Agreed.

Agreed.

OK. That makes sense.

Benny

stateid.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 7 Mar 2008 10:53:31 -0500

> Spencer on that or a proposal for an alternate deadline.

Yes - the use of non-zero stateid.seqid means that OLD_STATEID error does
not apply, the rules are different for layout stateids, and indeed we need a
new section describing the parallel use of layout stateids.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 07 Mar 2008 10:32:16 -0600

On Mar 6, 2008, at 8:09 PM, Noveck, Dave wrote:

> seqid

The scenario that I was able to devise that requires the client to
send non-zero stateid.seqid values for layout stateids is related
to the layout recall and potential issues with concurrency with
layoutget and layoutreturn.

One of the things that we need to define is if layout stateids
are always assumed to be sent in parallel and the client is
always required to deal with sorting the results.  In the case
of open/lock stateids, the client has a choice by specifying
the stateid.seqid == 0 such that this value is an indication
of the client's intent and ability to deal with the results.
This differentiation helps allow for the requirements that
stateid.seqid values for layout stateids be non-zero.  This isn't
the sole motivation but it helps with the following motivation.

As is apparent, the stateid.seqid is acting as a store barrier,
if you will, for the sequence of layoutget, layoutreturn, and
cb_layoutrecall operations exchanged between the client and server.
I believe the barrier requirements we had revolved around
concurrent layoutget/layoutreturn processing as well as
cb_layoutrecall processing being sequenced with layoutgets/ 
layoutreturns.
My memory isn't clear if there was a desire or a requirement
from correctness for these barriers but if that is the requirement,
then the use of the stateid.seqid provides for it.

The processing rules would be something like this:
If the client is sending layoutgets, the stateid.seqid for
the requests must be greater than or equal to the last value
returned for a layoutreturn.  If the client is sending
layoutreturns, the stateid.seqid must be greater than or equal
to the last received layoutget or the value provided in the
cb_layoutrecall.

As long as the above are met, the server processes the
layout requests in parallel and the client sorts the results.

If the client were to send stateid.seqid == 0, then the
server can not provide for the barrier functionality given
that it doesn't know when the client sent the result.

With this approach, the burden of ordering/recovering
from parallel layoutget/layoutreturn is shared between
the client and server (server has additional checks and
the client has to sort and deal with oldstateid errors).

If we decide that the barriers are unnecessary, then
the client is always responsible for sorting out the
returns but loses a method of parallelism in that it
has to serialize layoutget/layoutreturns itself if
attempting to avoid additional work.

While I can see an advantage to the current wording
(non-zero stateid.seqid values required),  I don't have
a strong opinion about having to keep it nor can I find
a scenario that it must be this way.  Having said that,
it seems prudent to require the client, in the processing
of a cb_layoutrecall, to use the stateid.seqid provided
in cb_layoutrecall in the layoutreturn to ensure that the
loop is closed or signified as processed by the client.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 07 Mar 2008 10:37:54 -0600

On Mar 7, 2008, at 3:17 AM, Benny Halevy wrote:
> layout

I read the stateid.seqid non-zero requirement as only applying to
the layout stateids and not the open stateids.  And given that
the use of current stateid would zero the seqid portion of the
stateid, then concurrent open/layoutget requests should process
as expected.

In the non-zero seqid requirement case, it is only when the client
does the layoutreturn the server would/should (?) reject the use
of open/lock/delegation stateids.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 7 Mar 2008 11:50:49 -0500

I think this will work but we need to get it clearly written up in a
section on stateid-seqid rules for layout stateids in the pnfs chapter
and then the state management chapter can say something like:

    Layout-related have their own set of rules for the handling
    of seqid fields in stateids.  For these types of stateids,
    zero is not an acceptable value for the seqid in stateid 
    arguments and the rules on when NFS4ERR_OLD_STATEID is 
    returned reflect the specifics of the get/return/recall
    concurrency model for layouts.  See ... for details.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 07, 2008 11:32 AM
To: NFSv4
Subject: Re: [nfsv4] use of current stateid for layout ops

Spencer