Subject: [nfsv4] layout stateid handling/processing updates

From: Spencer Shepler <[email-masked]>
Date: Wed, 19 Mar 2008 14:06:21 -0500

Included is the updated/proposed text to clarify layout stateid  
handling.
The diffs were a little difficult to read so I have included the  
sections
that have been updated.

Spencer

----

12.5.3.  Layout Stateid

    As with all other stateids, the layout stateid consists of a "seqid"
    and "other" field.  Once a layout stateid is changed, the "other"
    field will stay constant unless the stateid is revoked, or the  
client
    returns all layouts on the file and the server disposes of the
    stateid.  The "seqid" field is initially set to one, and is never
    zero on any NFSv4.1 operation that uses layout stateids, whether it
    is fore channel or backchannel operation.  After the layout stateid
    is established, the server increments by one the value of the  
"seqid"
    in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each
    CB_LAYOUTRECALL request.

    Given the design goal of pNFS to provide parallelism, the layout
    stateid differs from other stateid types in that the client is
    expected to send LAYOUTGET and LAYTOUTRETURN operations in parallel.
    Since the server is incrementing the "seqid" value on each layout
    operation, the client may determine the order of operation  
processing
    by inspecting the "seqid" value.  In the case of overlapping ranges
    in the client's requests, the ordering information will provide the
    client the knowledge of which layout ranges are held.  Additional
    layout stateid sequencing requirements are provided in
    Section 12.5.5.2.

    Once the client receives a layout stateid, it MUST use the correct
    "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations.  The
    correct "seqid" is defined as the highest "seqid" value from fully
    processed responses to LAYOUTGET or LAYOUTRETURN operations or fully
    processed arguments of a CB_LAYOUTRECALL operation.

    The client's receipt of a "seqid" is not sufficient for subsequent
    use.  The client must fully process the operations before the  
"seqid"
    can be used.  For LAYOUTGET results, if the client is not using the
    forgetful model (Section 12.5.5.1), it MUST first update its record
    of what ranges of the file's layout it has before using the seqid.
    For LAYOUTRETURN results, the client MUST delete the range from its
    record of what ranges of the file's layout it had before using the
    seqid.  For CB_LAYOUTRECALL arguments, the client MUST send a
    response to the recall before using the seqid.  The fundemental
    requirement in client processing is that the "seqid" is used to
    provide the ordering of processing.  LAYOUTGET results may be
    processed in parallel.  LAYOUTRETURN results may be processed in
    parallel.  LAYOUTGET and LAYOUTRETURN responses may be processed in
    parallel as long as the ranges do not overlap.  CB_LAYOUTRECALL
    request processing MUST be processed in "seqid" order at all times.

    Once a client has no more layouts on a file, the layout stateid  
is no
    longer valid, and MUST NOT be used.  Any attempt to use such a  
layout
    stateid will result in NFS4ERR_BAD_STATEID.

------

12.5.5.2.1.  Layout Recall and Return Sequencing

    One critical issue with regard to layout operations sequencing
    concerns callbacks.  The protocol must defend against races between
    the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
    CB_LAYOUTRECALL.  A client MUST NOT process a CB_LAYOUTRECALL that
    implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations
    to which the client has not yet received a reply.  The client  
detects
    such a CB_LAYOUTRECALL by examining the "seqid" field of the  
recall's
    layout stateid.  If the "seqid" is not one higher than what the
    client currently has recorded, and the client has at least one
    LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client  
knows
    the server sent the CB_LAYOUTRECALL after sending a response to an
    outstanding LAYOUTGET or LAYOUTRETURN.  The client MUST wait before
    processing such a CB_LAYOUTRECALL until it processes all replies for
    outstanding LAYOUTGET and LAYOUTRETURN operations for the
    corresponding file with seqid less than the seqid given by
    CB_LAYOUTRECALL (lor_stateid, see Section 20.3.)

    In addition to the seqid-based mechanism, Section 2.10.5.3 describes
    the sessions mechanism for allowing the client to detect callback
    race conditions and delay processing such a CB_LAYOUTRECALL.  The
    server MAY reference conflicting operations in the CB_SEQUENCE that
    precedes the CB_LAYOUTRECALL.  Because the server has already sent
    replies for these operations before issuing the callback, the  
replies
    may race with the CB_LAYOUTRECALL.  The client MUST wait for all the
    referenced calls to complete and update its view of the layout state
    before processing the CB_LAYOUTRECALL.

12.5.5.2.1.1.  Get/Return Sequencing

    The protocol allows the client to send concurrent LAYOUTGET and
    LAYOUTRETURN operations to the server.  The protocol does not  
provide
    any means for the server to process the requests in the same  
order in
    which they were created.  However, through the use of the "seqid"
    field in the layout stateid, the client can determine the order in
    which parallel outstanding operations were processed by the server.
    Thus, when a layout retrieved by an outstanding LAYOUTGET operation
    intersects with a layout returned by an outstanding LAYOUTRETURN on
    the same file, the order in which the two conflicting operations are
    processed determines the final state of the overlapping layout.  The
    order is determined by the "seqid" returned in each operation: the
    operation with the higher seqid was executed later.

    It is permissible for the client to send in parallel multiple
    LAYOUTGET operations for the same file or multiple LAYOUTRETURN
    operations for the same file, and a mix of both.

    It is permissilble for the client to use the current stateid (see
    Section 16.2.3.1.2) for LAYOUTGET operations for example when
    compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs.  It is
    also permissible to use the current stateid when compounding
    LAYOUTRETURNs.

    It is permissible for the client to current stateid when combining
    LAYOUTRETURN and LAYOUTGET operations for the same file in the same
    COMPOUND request since the server MUST process these in order.
    However, if a client does send such COMPOUND requests, it MUST NOT
    have more than one outstanding for the same file at the same time  
and
    MUST NOT have other LAYOUTGET or LAYOUTRETURN operations outstanding
    at the same time for that same file.

----

12.5.5.2.1.2.  Client Considerations

    Consider a pNFS client that has sent a LAYOUTGET and before it
    receives the reply to LAYOUTGET, it receives a CB_LAYOUTRECALL for
    the same file with an overlapping range.  There are two
    possibilities, which the client can distinguish via the layout
    stateid in the recall.

    1.  The server processed the LAYOUTGET before issuing the recall, so
        the LAYOUTGET must be waited for because it may be carrying
        layout information that will need to be returned to deal with  
the
        CB_LAYOUTRECALL.

    2.  The server sent the callback before receiving the LAYOUTGET.   
The
        server will not respond to the LAYOUTGET until the
        CB_LAYOUTRECALL is processed.

    If these possibilities cannot be distinguished, a deadlock could
    result, as the client must wait for the LAYOUTGET response before
    processing the recall in the first case, but that response will not
    arrive until after the recall is processed in the second case.  Note
    that in the first case, the "seqid" in the layout stateid of the
    recall is two greater than what the client has recorded and in the
    second case, the "seqid" is one greater than what the client has
    recorded.  This allows the client to disambiguate between the two
    cases.  The client thus knows precisely which possibility applies.

    In case 1 the client knows it needs to wait for the LAYOUTGET
    response before processing the recall (or the client can return
    NFS4ERR_DELAY).

    In case 2 the client will not wait for the LAYOUTGET response before
    processing the recall, because waiting would cause deadlock.
    Therefore, the action at the client will only require waiting in the
    case that the client has not yet seen the server's earlier responses
    to the LAYOUTGET operation(s).

    The recall process can be considered completed when the final
    LAYOUTRETURN operation for the recalled range is completed.  The
    LAYOUTRETURN uses the layout stateid (with seqid) specified in
    CB_LAYOUTRECALL.  If the client uses multiple LAYOUTRETURNs in
    processing the recall, the first LAYOUTRETURN will use the layout
    stateid as specified in CB_LAYOUTRECALL.  Subsequent LAYOUTRETURNs
    will use the highest seqid as is the usual case.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Thu, 20 Mar 2008 16:44:44 -0400

Nits...

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Spencer Shepler
> Sent: Wednesday, March 19, 2008 3:06 PM
> To: NFSv4
> Subject: [nfsv4] layout stateid handling/processing updates
> 

/is fore/is a fore/

Also, the spec uses both "back channel" and "backchannel".  It can be
particularly odd in a sentence that uses both "fore channel" and
"backchannel".

>     expected to send LAYOUTGET and LAYTOUTRETURN operations in
parallel.

s/LAYTOUT/LAYOUT

>     processed arguments of a CB_LAYOUTRECALL operation.

s/fully processed arguments of a/arguments of a fully processed/

>     provide the ordering of processing.  LAYOUTGET results may be

s/ordering/order/

>     It is permissible for the client to current stateid when combining

s/client to current/client to use the current/

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 20 Mar 2008 16:52:42 -0400

> particularly odd in a sentence that uses both "fore channel" and

Am I the only one that finds "fore channel" really odd-looking?

I would have expected fore-channel or forechannel.  (And ideally
something parallel for back-channel/backchannel.)

--b.

---

From: Jim Rees <[email-masked]>
Date: Thu, 20 Mar 2008 18:31:05 -0400

J. Bruce Fields wrote:

I would have expected channel and back-channel.  Fore-channel seems very
awkward.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 20 Mar 2008 15:48:37 -0700 (PDT)

> particularly odd in a sentence that uses both "fore channel" and

My understanding is that backchannel is a word and forechannel is not.

Currently the only uses of "back channel" I'm aware are:

$ grep "back channel" *.xml
nfsv41_middle_core_infrastructure.xml:   fore and back channels.
nfsv41_middle_op_sequence.xml:          complete on the fore and back
channel, and then

I think these are appopriate.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 21 Mar 2008 10:04:36 -0400

What about NFS4ERR_OLD_STATEID? If the client sends parallel
LAYOUTGET/RETURN operations, they could all have the same stateid. The
server will process the first one, and all remaining will have old stateids.
Should the server ignore this error for layout stateids?

--->Andy

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Mar 2008 10:58:36 -0400

Caveat: I haven't got a chance to read Spencer's email on this subject
in detail;

My expectation though was that OLD_STATEID would follow the rules
Spencer set out in his mail of 3/7:

> The processing rules would be something like this:
> If the client is sending layoutgets, the stateid.seqid for
> the requests must be greater than or equal to the last value
> returned for a layoutreturn.  If the client is sending
> layoutreturns, the stateid.seqid must be greater than or equal
> to the last received layoutget or the value provided in the
> cb_layoutrecall. 

My expectation was that if the seqid you sent did not follow the
appropriate "equal or greater" clause, you would get OLD_STATEID while
if it greater than the last one currently given out, you would get
BAD_STATEID.  Chapter 8 would say that the rules for seqid on layouts
are slightly different from those for IO, opens, byte-range locks, etc.
and have an xref to chapter 12 where this was discussed.

Appropriate wordsmithing of the above to deal with the fact that seqids
are subject to wraparound is left as an exercize for anybody who doesn't
pass the buck quickly enough :-)

-----Original Message-----
From: William A. (Andy) Adamson [mailto:[email-masked]] 
Sent: Friday, March 21, 2008 10:05 AM
To: Spencer Shepler
Cc: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

What about NFS4ERR_OLD_STATEID? If the client sends parallel
LAYOUTGET/RETURN operations, they could all have the same stateid. The
server will process the first one, and all remaining will have old
stateids. Should the server ignore this error for layout stateids?

--->Andy

On Wed, Mar 19, 2008 at 3:06 PM, Spencer Shepler
<[email-masked]> wrote:

Spencer

----

12.5.3.  Layout Stateid

------

----

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 10:30:34 -0500

> pass the buck quickly enough :-)

Having reviewed all of the layout stateid material in the I-D I came
to the conclusion that seqid checking/validation the server needed
to do was limited to what is stated in:
	12.5.5.2.1.  Layout Recall and Return Sequencing
that I sent out (mainly unchanged).  Otherwise, the server just
processes the requests as they arrive and ensures that the seqid
is updated for each resposne.  It is then the client's responsibility
to sort out the responses to deal with overlapping layoutget/ 
layoutreturn
processing.

My apologies for not making this clear.

I would suggest this wording be added to the second paragraph of
"12.5.3.  Layout Stateid"

   Note the seqid processing for layout stateid differs from the stateid
   types.  To support the parallelism desired, seqid processing rules
   are limited to those specified in the pNFS chapters (mainly this
   section and Section 12.5.5.2).

Spencer

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 21 Mar 2008 11:33:59 -0400

On Fri, Mar 21, 2008 at 10:30:34AM -0500, Spencer Shepler wrote:
>    types.

Did you mean to write "the other stateid types"?

--b.

> To support the parallelism desired, seqid processing rules

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 10:47:53 -0500

> Did you mean to write "the other stateid types"?

Heh, yes.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Mar 2008 11:59:12 -0400

I don't get it.

Are you saying that the processing rules that you sent out on 3/7 are
not enforced by the server and I can send any random seqid I choose
(except zero perhaps).  But if any other value is OK, why isn't zero OK.

If that is so, then you would need to say so explicitly and that
OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not
returned.

The section you mention just lists cases in which using the current
stateid is permissible and doesn't seem to say anything about things
which are impermissable.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 21, 2008 11:31 AM
To: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 11:39:16 -0500

> (except zero perhaps).  But if any other value is OK, why isn't  

The rules I suggested on 3/7 are different and more restrictive than
what is in the text I proposed a couple of days ago.

Briefly, the barrier that the layout stateid.seqid attempting to
provide is for the CB_LAYOUTRECALL case.  The stateid.seqid != 0
requirement derives from this in that the client must be using
a stateid.seqid that was valid at some point in the past.
The reason for relaxing the rules I proposed on 3/7 is that
the proposed text places the full burden on the client to
sort out the execution order of LAYOUTGET/LAYOUTRETURN.
This way, the client can send parallel LAYOUTGET/LAYOUTRETURNs
and I expect the most common case for this usage to be
for non-overlapping ranges.  Note, the client is required to
use the highest stateid.seqid that was fully processed.  Therefore,
it can't pick values at random, it must use something reasonably
recent.

> OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not

Okay, I will review the text again for further updates.

> stateid is permissible and doesn't seem to say anything about things

The larger section does include the specifics.  12.5.5.2.1.3 does
list what the server will do with respect to the CB_LAYOUTRECALL barrier

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Mar 2008 13:23:01 -0400

> Note, the client is required to
> use the highest stateid.seqid that was fully processed.  

But the server is not allowed to enforce that in your approach.  It has
to accept anything that is not zero.

> Therefore,

In terms of obeying the spec, yes.  But if he does pick a value at
random, it will be accepted (except for one in four billion times :-).
At that point you wonder what the point of the spec requirement is.  The
server doesn't check the seqid or use it and the only reason to not pick
a random value is to comply with the spec, but what is the spec's
motivation for the requirement?

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 21, 2008 12:39 PM
To: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 12:33:09 -0500

> motivation for the requirement?

In the even the server does a CB_LAYOUTRECALL, the server will be
checking the stateid.seqid against the stateid.seqid provided
in the CB_LAYOUTRECALL.  If the client is choosing seqids at
random then that check is meaningless.   I agree that in the
general case, the stateid.seqid is not validated as strongly
as it is for the other stateid types it is still critical that
the client use appropriate seqid values.

What you observing is that for layout requests (stateids) it is
a free-for-all until the server does a CB_LAYOUTRECALL (if ever) and
it is entirely the client's responsibility for sorting out the
execution order.

Even if we were to return to the additional barrier of LAYOUTRETURN  
vs. LAYOUTGET,
there would still be a lack of strict ordering in the case
that the client is sending a stream of LAYOUTGET requests.

While the model is very different from "normal" stateid.seqid
processing, is something broken with what I have proposed?

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Mar 2008 13:42:31 -0400

OK.  I see the issue with seqid and CB_LAYOUTRECALL.  I now don't see a
problem with the approach, but I think we need more clarity about it in
terms of what constraints on seqid exist from the server's point of view
(very few) and why.  In other words, I think we want the strategy set
out:

    The main use of seqid is in the server-to-client path,
    for determining the order that requests were processed.

    The seqid is sent by the client to the server but there
    the purpose is determing whether this is or isn't a 
    result of a CB_LAYOUTRECALL.  

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 21, 2008 1:33 PM
To: Noveck, Dave
Cc: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 15:17:54 -0500

>     result of a CB_LAYOUTRECALL.

I think the intent is in the text but I have reworked it a litte
in an attempt to make it obvious.

I have included all relevant text but the updates are contained
within paragraphs 2, 3, 4 below:

Spencer

-------

12.5.3.  Layout Stateid

    is a fore channel or backchannel operation.  After the layout  
stateid

    Given the design goal of pNFS to provide parallelism, the layout
    stateid differs from other stateid types in that the client is
    expected to send LAYOUTGET and LAYOUTRETURN operations in parallel.
    The "seqid" value is used by the client to properly sort  
responses to
    LAYOUTGET and LAYOUTRETURN.  The "seqid" is also used to prevent  
race
    conditions between LAYOUTGET and CB_LAYOUTRECALL.  Given the
    processing rules differ from layout stateids and other stateid  
types,
    only the pNFS sections of this document should be considered to
    determine proper layout stateid handling.

    Once the client receives a layout stateid, it MUST use the correct
    "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations.  The
    correct "seqid" is defined as the highest "seqid" value from
    responses of fully processed LAYOUTGET or LAYOUTRETURN operations or
    arguments of a fully processed CB_LAYOUTRECALL operation.  Since the
    server is incrementing the "seqid" value on each layout operation,
    the client may determine the order of operation processing by
    inspecting the "seqid" value.  In the case of overlapping ranges in
    the client's requests, the ordering information will provide the
    client the knowledge of which layout ranges are held.  Additional
    layout stateid sequencing requirements are provided in
    Section 12.5.5.2.

    The client's receipt of a "seqid" is not sufficient for subsequent
    use.  The client must fully process the operations before the  
"seqid"
    can be used.  For LAYOUTGET results, if the client is not using the
    forgetful model (Section 12.5.5.1), it MUST first update its record
    of what ranges of the file's layout it has before using the seqid.
    For LAYOUTRETURN results, the client MUST delete the range from its
    record of what ranges of the file's layout it had before using the
    seqid.  For CB_LAYOUTRECALL arguments, the client MUST send a
    response to the recall before using the seqid.  The fundemental
    requirement in client processing is that the "seqid" is used to
    provide the order of processing.  LAYOUTGET results may be processed
    in parallel.  LAYOUTRETURN results may be processed in parallel.
    LAYOUTGET and LAYOUTRETURN responses may be processed in parallel as
    long as the ranges do not overlap.  CB_LAYOUTRECALL request
    processing MUST be processed in "seqid" order at all times.

----

12.5.5.2.  Sequencing of Layout Operations

    As with other stateful operations, pNFS requires the correct
    sequencing of layout operations.  PNFS uses the "seqid" in the  
layout
    stateid to provide the correct sequencing between regular operations
    and callbacks.  It is the server's responsibility to avoid
    inconsistencies regarding the layouts provided and the client's
    responsibility to properly serialize its layout requests and layout
    returns.

    It is permissible for the client to use the current stateid when
    combining LAYOUTRETURN and LAYOUTGET operations for the same file in
    the same COMPOUND request since the server MUST process these in
    order.  However, if a client does send such COMPOUND requests, it
    MUST NOT have more than one outstanding for the same file at the  
same
    time and MUST NOT have other LAYOUTGET or LAYOUTRETURN operations
    outstanding at the same time for that same file.

12.5.5.2.1.3.  Server Considerations

    Consider a race from the metadata server's point of view.  The
    metadata server has sent a CB_LAYOUTRECALL and receives an
    overlapping LAYOUTGET for the same file before the LAYOUTRETURN(s)
    that respond to the CB_LAYOUTRECALL.  There are three cases:

    1.  The client sent the LAYOUTGET before processing the
        CB_LAYOUTRECALL.  The "seqid" in the layout stateid of LAYOUTGET
        is two less than that of the "seqid" in CB_LAYOUTRECALL.  The
        server returns NFS4ERR_RECALLCONFLICT to the client, which
        indicates to the client that there is a pending recall.

    2.  The client sent the LAYOUTGET after processing the
        CB_LAYOUTRECALL, but the LAYOUTGET arrived before the
        LAYOUTRETURN and the response to CB_LAYOUTRECALL that completed
        that processing.  The "seqid" in the layout stateid of LAYOUTGET
        is equal to that of the "seqid" in CB_LAYOUTRECALL.  The server
        has not received a response to the CB_LAYOUTRECALL, so it  
returns
        NFS4ERR_RECALLCONFLICT.

    3.  The client sent the LAYOUTGET after processing the
        CB_LAYOUTRECALL, the server received the CB_LAYOUTRECALL
        response, but the LAYOUTGET arrived before the LAYOUTRETURN that
        completed that processing.  The "seqid" in the layout stateid of
        LAYOUTGET is equal to that of the "seqid" in CB_LAYOUTRECALL.
        The server has received a response to the CB_LAYOUTRECALL, so it
        returns NFS4ERR_RETURNCONFLICT.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Mar 2008 18:06:30 -0400

Looks good.

> determine proper layout stateid handling. 

Is that too broad?  Is it only with regard to seqid handling that layout
stateids should be considered  only from the chapter 12-13 viewpoint.
I'd hate for us to have to make sure that everything we said about
stateid's in chaper 8 (which would not apply to layouts) would have to
be resaid in chapters 12-13 or would otherwise be effectively not said.

Could we say:

    Processing rules for layout stateids differ from those for other 
    stateid types, so that where there is a conflict between general
    descriptions of stateids and those in the pNFS sections regarding
    those for layouts, those pNFS sections should be considered 
    definitive in determining proper layout stateid handling.  With 
    regard to treatment of the "seqid" field of stateid layouts, the 
    discussion in the pNFS sections is complete and anything said in 
    other sections does not apply. 

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 21, 2008 4:18 PM
To: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

On Mar 21, 2008, at 12:42 PM, Noveck, Dave wrote:

> OK.  I see the issue with seqid and CB_LAYOUTRECALL.  I now don't see 
> a problem with the approach, but I think we need more clarity about it

> in terms of what constraints on seqid exist from the server's point of

> view (very few) and why.  In other words, I think we want the strategy

> set

I think the intent is in the text but I have reworked it a litte in an
attempt to make it obvious.

I have included all relevant text but the updates are contained within
paragraphs 2, 3, 4 below:

Spencer

-------

12.5.3.  Layout Stateid

    The "seqid" value is used by the client to properly sort responses
to

    Once a client has no more layouts on a file, the layout stateid is
no
    longer valid, and MUST NOT be used.  Any attempt to use such a
layout
    stateid will result in NFS4ERR_BAD_STATEID.

----

    any means for the server to process the requests in the same order
in

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Mar 2008 17:39:31 -0500

> be resaid in chapters 12-13 or would otherwise be effectively not  

Good point.

>     other sections does not apply.

Further wordsmithing:

     Processing rules for layout stateids differ from those for other
     stateid types.  The pNFS sections are definitive in determining
     layout stateid handling if a conflict exists between the  
descriptions
     for general layouts and descriptions of layout stateids.  With
     regard to treatment of the "seqid" field of stateid layouts, the
     discussion in the pNFS sections is complete and anything discussed
     in other sections does not apply.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Mon, 24 Mar 2008 14:16:42 +0200

>     client the knowledge of which layout ranges are held.  Additional

Overlapping ranges are actually with the server replies since the server may
grant the client wider range and/or stronger iomode than the client requested.

How about the following?
     In case outstanding LAYOUTRETURN operations conflict with responses
     to outstanding LAYOUTGETs, i.e. their respective ranges overlap, and
     iomodes match (see <xref 12.5.5.2.1.1. />), the ordering information will provide the
     client the knowledge of which layout ranges are held.

>     response to the recall before using the seqid.  The fundemental

nit: fundamental.

>     It is permissilble for the client to use the current stateid (see

nit: permissible

>         returns NFS4ERR_RETURNCONFLICT.

Proposed amendment to the text above:

Since the size of the seqid is limited, it is expected that the seqid may
wrap around.  Therefore, it is important that the server correctly determines
the respective ordering of seqids, i.e. what comes "before" or "after" a
particular seqid.

To prevent a situation where LAYOUTGETs or LAYOUTRETURNs sent before a
CB_LAYOUTRECALL are mistaken to be in response to that CB_LAYOUTRECALL, or
just sent after it.  The server SHOULD NOT just accept any seqid.
The server MUST allow seqids which are in the range of the current seqid
(as maintained by the server) less a reasonable threshold, to the current seqid,
all modulo |seqid| (2^32).  If the seqid is outside this range, the server
SHOULD return either NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID respectively.

Benny

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 25 Mar 2008 12:54:45 -0500

>> On Mar 21, 2008, at 12:42 PM, Noveck, Dave wrote:
...

>      client the knowledge of which layout ranges are held.

Good point.  I suggest the following as modification to your  
suggested text.

   In the case of overlapping layout ranges, the ordering information
   will provide the client the knowledge of which layout ranges are  
held.
   Note that layout ranges may occur because of the client's specific
   requests or because the server is allowed to expand the range of
   a requested layout and notify the client in the LAYOUTRETURN results.

...

> SHOULD return either NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID  

I agree with the sentiment and suggest that the following paragraph
be added to the "Sequencing of Layout Operations":

    Given the possiblity of wraparound of the "seqid" value, it is
    helpful to the server that the client is required to use the highest
    "seqid" value of processed server responses or callbacks (see
    Section 12.5.3).  The server can then determine if incoming  
LAYOUTGET
    and LAYOUTRETURN operations are within a reasonable range of "seqid"
    values.  The NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID errors would
    be returned to the client if not.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Wed, 26 Mar 2008 11:32:59 +0200

>    Note that layout ranges may occur because of the client's specific

Your proposal sounds better indeed.
One phrase that doesn't seem to parse quite right get is "ranges may occur"
Did you mean "Note that layout ranges may overlap because ..."?

>     be returned to the client if not.

Sounds good!

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 26 Mar 2008 06:18:40 -0400

Spencer wrote:
>    Section 12.5.3).  

The introductory clause seems to add confusion rather than clarification
about what is meant, sort of like, "A well-regulated militia ..." in the
Second Amendment.  Apologies to non-US readers and others who are not
Supreme Court junkies.

"it is helpful to the server that the client is required" is not clear.
First, if it is required, say so.  Are you saying that the client MUST
do this?  If so, say it directly.  As to the motivation, is it the
server that benfits or is this just tightening things up to the general
benefit of the protocol.

>   The server can then determine if incoming LAYOUTGET
>   values.  

He could if the spec defined what a reasonable range is.  If it doesn't,
expect different definitions, and a later need to clarify.

> The NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID errors would

I sort of understand "MUST" and "SHOULD" but what exactly does "would"
mean here.  Also, since we have ruled out what chapter 8 says about
seqids, we need some explanation about when each of
NFS4ERR_{OLD,BAD}_STATEID are returned and how to deal with seqid
wraparound. 

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Tuesday, March 25, 2008 1:55 PM
To: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

...

>>     responses of fully processed LAYOUTGET or LAYOUTRETURN operations

>> or
> Overlapping ranges are actually with the server replies since the 
> server may grant the client wider range and/or stronger iomode than 
> the client requested.
>
> How about the following?

Good point.  I suggest the following as modification to your suggested
text.

...

>>         completed that processing.  The "seqid" in the layout stateid

>> of
>>         The server has received a response to the CB_LAYOUTRECALL, so

>> it
> Since the size of the seqid is limited, it is expected that the seqid 
> may wrap around.  Therefore, it is important that the server correctly

> determines the respective ordering of seqids, i.e. what comes "before"

> or "after" a particular seqid.
>

> CB_LAYOUTRECALL are mistaken to be in response to that 
> CB_LAYOUTRECALL, or just sent after it.  The server SHOULD NOT just 
> accept any seqid.
> The server MUST allow seqids which are in the range of the current 
> seqid (as maintained by the server) less a reasonable threshold, to 
> the current seqid, all modulo |seqid| (2^32).  If the seqid is outside

> this range, the server SHOULD return either NFS4ERR_OLD_STATEID or 
> NFS4ERR_BAD_STATEID respectively.

I agree with the sentiment and suggest that the following paragraph be
added to the "Sequencing of Layout Operations":

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 27 Mar 2008 14:25:40 -0500

As Dave pointed out, this paragraph was a little lame:

     Given the possiblity of wraparound of the "seqid" value, it is
     helpful to the server that the client is required to use the  
highest
     "seqid" value of processed server responses or callbacks (see
     Section 12.5.3).  The server can then determine if incoming  
LAYOUTGET
     and LAYOUTRETURN operations are within a reasonable range of  
"seqid"
     values.  The NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID errors  
would
     be returned to the client if not.

I have replaced it with the paragraph below  (12.5.5.2.1.4)
and included the previous section for context.

At this point, the server will return BAD_STATEID but I can
not think of a reason OLD_STATEID would be returned for
LAYOUTGET/LAYOUTRETURN.  Let me know anyone believes otherwise.

Spencer

>        is two less than the "seqid" in CB_LAYOUTRECALL.
>        The server returns NFS4ERR_RECALLCONFLICT to the client, which
>        is equal to or greater than that of the "seqid" in
>        CB_LAYOUTRECALL.  The server has not received a response to the
>        CB_LAYOUTRECALL, so it returns NFS4ERR_RECALLCONFLICT.
>
> 12.5.5.2.1.4.  Wraparound of sequence id
>
>    The rules for layout stateid processing differ from other  
> stateids in
>    the protocol because the "seqid" value can not be zero and the
>    stateid's "seqid" value changes in a CB_LAYOUTRECALL operation.   
> The
>    non-zero requirement combined with the inherent parallelism of  
> layout
>    operations means that a set of LAYOUTGET and LAYOUTRETURN  
> operations
>    may contain the same value for "seqid" and the value will represent
>    the span of parellism achieved by the client.  To account for this
>    parellism, the server validates that the "seqid" is non-zero.   
> If the
>    server uses a CB_LAYOUTRECALL, then the "seqid" is validated  
> further
>    by applying the rules listed above in Section 12.5.5.2.1.3.
>
>    The server uses a slightly modified version of the modulo  
> arithmetic
>    as described in Section 2.10.5.1 when incrementing the layout
>    stateid's "seqid".  The modification to that modulo arithmetic
>    description is to not use zero.  The modulo arithmetic is also used
>    for the comparisons of "seqid" values in the processing of
>    CB_LAYOUTRECALL events as described above in Section 12.5.5.2.1.3.

---

From: Benny Halevy <[email-masked]>
Date: Fri, 28 Mar 2008 12:58:11 +0300

> LAYOUTGET/LAYOUTRETURN.  Let me know anyone believes otherwise.

For non-layout stateid's we say the following:
   the server is required to return
   NFS4ERR_OLD_STATEID if the seqid is lower than the most current value
   and NFS4ERR_BAD_STATEID if the seqid is greater than the most current
   value.

I think that could be applied in the same spirit to a valid range, modulo
wraparound.  Although returning NFS4ERR_BAD_STATEID for anything outside
the valid range would work too, if the client sent a seqid it thought OK
and due to a storm of layout operations it fell outside the range the server
allows (and that's really not the client's fault), NFS4ERR_OLD_STATEID seems
like a more polite error to return.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 28 Mar 2008 09:20:06 -0400

I don't think OLD_STATEID makes sense for layouts.  To explain why, let
me go back to the reason we have OLD_STATEID in v4.1.  At one point, we
were thinking of removing it.  The real case that showed we had to have
it was the case of OPEN/CLOSE.  If I issue a CLOSE, I have the problem
of a possibly racing OPEN.  I don't want to close a file when the racing
open might have been an open-upgrade.  The result would be that we would
either wind up with a close file or an open based on which (open or
close) was processed first.

The important point is that in the case of open, the spec can't simply
say "Do't do that".  I cannot force the client to defer operations by
saying you can only have an open or close outstanding and not both.  The
reason is that because of hard links, I may have no idea that a file I
am opening is the same one as one for which a close is in progress. 

Because of that, we have to distingush between BAD_STATEID which means
"The seqid you sent in the stateid is WRONG.  RTFRFC.  Get with the
program." and OLD_STATEID which means "Unforseen circumstances have
caused the seqid you sent in the stateid to be inappropriate, making it
not possible to execute the operation as sent.  It is therefore
necessary for you to reissue the operation with a new seqid in the
stateid.  In pointing this out, the server intends not to cast any
aspersions whatsoever on your most excellent client implementation."
Please don't ask me to update the spec's error definitions. :-) 

In the case of layouts, I don't see that there is an analogous
situation.  If something is outside the valid range, then that means the
client is broken, doesn't it?  If there can exist cases in which you can
issue a request obeying the protocol, and the seqid is not in the valid
range, I think that means that the valid range is not correctly
specified (i.e. it is too small).  In other words, if we partition the
seqids in any given situation into three sets: those that are valid and
result in OK, those that are wrong and result in BAD_STATEID, and those
that sort of valid but not quite and result in OLD_STATEID, we have to
ask what is the distinction between the first and third classes.  If we
can define the boundary between OLD_STATEID and BAD_STATEID so that
anything getting OLD_STATEID indicates the client is not violating the
protocol, why are not the seqids in that class not in the OK class?

> if the client sent a seqid it thought OK and due to a 
> storm of layout operations it fell outside the range 
> the server allows (and that's really not the client's 
> fault), NFS4ERR_OLD_STATEID seems like a more polite 
> error to return.

I think the valid range has to be defined so that can't happen.  As to
the layout storm, we should be able to bound the outstanding requests by
the set of (forward channel) slots for all sessions associated with the
current client.  In this regard we have to be careful about requests
replied to but where the response with its layout is not "fully
processed".  The client must not be allowed to issue a new layout
operation beyond the set imposed by the slot limits until it has
processed at least one of the set of the earliest performed for a given
layout "other" field, as evidenced by the seqid returned by the server. 

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, March 28, 2008 5:58 AM
To: Spencer Shepler
Cc: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

On Mar. 27, 2008, 21:25 +0200, Spencer Shepler <[email-masked]>
wrote:
> I have replaced it with the paragraph below  (12.5.5.2.1.4) and 
> included the previous section for context.
> 
> At this point, the server will return BAD_STATEID but I can not think 
> of a reason OLD_STATEID would be returned for LAYOUTGET/LAYOUTRETURN.

> Let me know anyone believes otherwise.

I think that could be applied in the same spirit to a valid range,
modulo wraparound.  Although returning NFS4ERR_BAD_STATEID for anything
outside the valid range would work too, if the client sent a seqid it
thought OK and due to a storm of layout operations it fell outside the
range the server allows (and that's really not the client's fault),
NFS4ERR_OLD_STATEID seems like a more polite error to return.

Benny

>>    The rules for layout stateid processing differ from other stateids

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 28 Mar 2008 09:54:47 -0400

Thanks Dave. I'm convinced.  I agree that the number of slots
can be used to deterministically restrict the valid layout stateid 
range and that anything outside this range can (MUST) be considered
invalid, resulting in BAD_STATEID error.

Benny

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, March 28, 2008 5:58 AM
To: Spencer Shepler
Cc: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates

Benny

---

From: [email-masked]
Date: Fri, 28 Mar 2008 10:09:55 -0400

Each slot is used for a compound, but the compound may have many layout
operations (LAYOUTGET or LAYOUTRETURN), each of which when processed
will modify the seq id.

Thus the bound is the maximum number of layout operations times the
number of slots.

-Jason

Of
> Halevy, Benny
> Sent: Friday, March 28, 2008 9:55 AM
> To: Noveck, Dave; Spencer Shepler
> Cc: NFSv4
stateids

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 28 Mar 2008 09:23:09 -0500

> Thus the bound is the maximum number of layout operations times the

And to refer back to a subtle point that Dave mentioned, the
response has to be processed.  Therefore, the client may have
made a request on a slot that is "stuck" at the client and that
slot has not been reused by another request.

For the reasons that Dave mentioned, I did not include use
of the OLD_STATEID.  I also did not specify a valid range for
BAD_STATEID usage because of the various caveats that we have
identified.  The range may large and based on client behaviors
(as Jason has an example of multiple LAYOUTGETs in a single
COMPOUND/slot) calculating the range of "valid" seqid's is
hard.

In the cases of CB_LAYOUTRECALL use, the specification is clear
about what errors to return and why.  If the client is ahead
of the server with a seqid, then that is a case of BAD_STATEID
usage.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 28 Mar 2008 10:43:42 -0400

> I also did not specify a valid range for
> identified. 

But that leads to a problem.  If we can't determine the valid range (and
I agree that given the issues that have been mentioned it is difficult
to do), then how is the client to do so?  And if it can't reliably do
so, then BAD_STATEID, which generally tells the client he has made a
serious mistake does not seem appropriate.

If the protocol were to say that you can issue 50 layoutgets in parallel
(for a given "other" value) and that you had to stop process and process
responses to move the window along, then that would define a valid range
and the server could envorce that and the client would be obliged to
respect that window and would have to accept that if he went outside it
BAD_STATEID would result.

Clearly any fixed number like 50 is not going to work and if I suggested
adding a new attribute to specify that value at this point, I'd worry
about Mike Eisler being rather upset at me.  So this is where the number
of slots on all sessions for the client comes in.  It is not an
automatically enforced limit, since, as Jason has pointed out, you can
issue multiple layou operations in a single COMPOUND.  But it would be a
value that would naturally scale with the amount of parallelism that
people wanted and so if you used this value (or twice this value or half
of this value) and made it obligatory on the client to stay within it,
then we could establish a valid range that the server could enforce,
while still allowing as much LAYOUTGET parallelism as any reasonable
client could want.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, March 28, 2008 10:23 AM
To: [email-masked]
Cc: [email-masked]; Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] layout stateid handling/processing updates

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 28 Mar 2008 09:58:47 -0500

> while still allowing as much LAYOUTGET parallelism as any reasonable

The client knows what to do in that it is required to use the
highest seqid from the set of fully processed responses to
LAYOUTGET/LAYOUTRETURN and the call of CB_LAYOUTRECALL.

As you say, we want to allow for the maximum amount of parallelism.
As I said, BAD_STATEID will be returned when the the seqid == 0
and when seqid is "in the future".  The problem with the "in the
future" is with wraparound, as we have noted.  I would suggest
that the server use something very large, like 10 million, as the
limit for the number of seqid's "in the past" and everything else
is "in the future".  Of course, I don't want to put this in the spec.

The major point is that the server should be be very broad in
its interpretation of what is in the past to allow for the
parallelism.

Spencer

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 31 Mar 2008 17:24:02 -0700 (PDT)

>> about Mike Eisler being rather upset at me.  So this is where the

Indeed Dave should worry if he goes that route. :-)

>> number

Let's pursue this notion a bit more though. As Jason points out,
technically it is the summation of i=1..N of  ca_maxrequests(i) *
(ca_maxoperations(i) - 1) where N is the number of forechannels
(the - 1 is because SEQUENCE uses a operation in a COMPOUND).
That however might be too large a number. Or it might not. It depends on
the server (keeping in mind that this is a global limit per client
ID, not per client ID/fh pair). So the range of parallelism is
thus between summation of i=1..N ca_maxoperations(i) - 1,
and summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)

It is fair game for the client to attempt to play in this range, with
the understanding that the server MUST support summation of i=1..N
ca_maxoperations(i) - 1, and MAY support up to summation of i=1..N
(ca_maxoperations(i) - 1) * ca_requests(i).

Of course,
 ca_maxrequestsize;
 ca_maxresponsesize;
 ca_maxresponsesize_cached;

might limit the MAY further, because layout operations are not
exactly compact.

But regardless server can limit the range by simply
returning NFS4ERR_DELAY on a layout operation that
returns a layout stateid.

> is "in the future".  Of course, I don't want to put this in the spec.

I am having trouble imagining servers that support 10 million
outstanding, parallel layout operations. But I think we are
talking about four separate concepts:

1. the minimum number of parallel layout operations a client
   MUST be allowed

2. the maximum number a parallel layout operations a server MAY
   allow

3. what values of seqid get _BAD_STATEID returned?

4. what values of seqid get _OLD_STATEID returned?

I described 1 and 2.

I think 3 and 4 determined by the number of seqids in the
past. This number, call it, PAST_SEQ, is determined by the maximum value of
summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
that has ever existed on the client ID. This allows for additional
fore channels to be added to the client ID and then deleted without
introducing problems, as well as the highest slot number on a fore channel
to be increased and reduced on a server.

Of course, a server implementer might decide: "to heck with that,
I'm just going to set PAST_SEQ to 10 million." Fine, as long as it is never
less than maximum value of
  summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)

I willing to believe that the "to heck with that" server dude
can make that guarantee by doing stuff like limiting the maximum number
of fore channels to 100, and the maximum number of slots per
fore channel to 100. 10K is much less than 1M, and 10K slots is
probably enough for most MDS clients. Though I think THWT ought to
be more elegant with his #define for PAST_SEQ.

> its interpretation of what is in the past to allow for the

But the point that trumps it is that one server's definition of
broad will inevitably be at odds with that of one client's. Since
we don't want to codify 1M in the spec, and instead
can deterministically define PAST_SEQ, and without .x changes,
we should do it.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 01 Apr 2008 10:47:52 +0300

> (the - 1 is because SEQUENCE uses a operation in a COMPOUND).

This is true unless we mandate that the client uses the current stateid
when issuing multiple such ops in the same compound.

> can deterministically define PAST_SEQ, and without .x changes,

Agreed.

---

From: [email-masked]
Date: Tue, 1 Apr 2008 10:55:36 -0400

I too agree that we should define PAST_SEQ in the spec.
-Jason

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Benny Halevy
Sent: Tuesday, April 01, 2008 3:48 AM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] layout stateid handling/processing updates

Agreed.