Subject: [nfsv4] pNFS Issues document

From: <[email-masked]>
Date: Tue, 1 Feb 2011 19:25:20 -0500

As promised, this (attached) is the initial version of the document to guide discussion of pNFS issues at the interim meeting.

I appreciate corrections and additions.  I think it will increase the productivity if people read it early and get their updates to me by 2/11.  I'm got saying that updates that after won't make but I'm not going to guarantee that they will.  OTOH, anything received before then will be included in a version that goes out a few days a before the meeting.  How's that for a deal?

---

From: Fred Isaman <[email-masked]>
Date: Tue, 1 Feb 2011 22:25:48 -0500

On Tue, Feb 1, 2011 at 7:25 PM,  <[email-masked]> wrote:
> I appreciate corrections and additions.

Some additions:

What does it mean if the server sends a bulk CB_LAYOUTRECALL with
iomode != ANY?  There appears to be an intent to support this, but as
written, it seems impossible given the strict "dump all layoutstate"
requirements of the bulk recalls.

How should the server respond to a LAYOUTGET with an open stateid when
it has already handed out a layoutstateid?  It would be helpful from
the forgetful point of view if it were considered a signal that the
client has forgotten all previous layouts and the layout stateid for
that file.  At the very least, spec needs to clarify what happens if
the client sends multiple parallel LAYOUTGET(openstateid) compounds.
(I believe the consensus here at least is that the client MUST NOT do
that.)

If the server sends a CB_LAYOUTRECALL, what client responses
necessitate updating the clients layoutstateid seqid?  Clearly OK, but
isn't NOMATCHING also a "successful" response that should update the
clients view?  What about other errors?  For that matter, why can't
the client just always send a 0 seqid in all layout stateids?

Fred

---

From: <[email-masked]>
Date: Wed, 2 Feb 2011 12:51:43 -0500

Thanks.

The first two will be in Version 2 which should be out at the end of the week.

As to the third:

> clients view? 

I do discuss NOMATCHING in connection with CB_LAYOUTRECALL in discussing section 12.5.5.2.1.3.

It seems to me that the update of the client's layoutstateid is unconditional and it doesn't depend on what the client is going to return.  After all, it also updates its stateid when it gets seqid values in the responses to LAYOUTGET and LAYOUTRETURN.  Those are unconditional because the client doesn't respond to those responses.  Is there somewhere which suggests that the updating of the seqid in the RECALL case is different?  After all, if the client is not to update his view, the server better not have updated his, but I thought he did it when sending the callback.  ??? 

> What about other errors? 

I have mentioned DELAY in the context of 12.5.5.2.1.3 but not otherwise.

> For that matter, why can't

He can but the server MUST return BAD_STATEID in that case (section 12.5.5.2.1.4).

If you mean why could the protocol not have been designed otherwise, then that question is out-of-scope for this inquiry.  The spec does talk about various races and if something is said in that regard that isn't true, then it should be fixed.  However, I don't want get into the position of judging the overall adequacy of any arguments that this or some other design was the right choice.

By the way, with regard to the question of errors and incrementing the seqid, the following passages (from 12.5.3) do address the issue:

>  Once the client receives a layout stateid, it MUST use the correct
>  "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations.  The
>  correct "seqid" is defined as the highest "seqid" value from
>  responses of fully processed LAYOUTGET or LAYOUTRETURN operations or
>  arguments of a fully processed CB_LAYOUTRECALL operation. 

>  For CB_LAYOUTRECALL arguments, the client MUST send a
>  response to the recall before using the seqid.

It says "send a response" and there is an opportunity to make it conditional on errors or whatever, but it just says a response.  The only conclusion is that the error sent doesn't matter as long as the response is sent.

I think I've seen this phenomenon before.  The spec is clear on a particular point, once you find the passage, but there is confusion about it due to spec organization or size or something.  Maybe a general discussion of why this is would be good to have.   I have some theories but they are just that.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Fred Isaman
Sent: Tuesday, February 01, 2011 10:26 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS Issues document

Some additions:

Fred

---

From: Fred Isaman <[email-masked]>
Date: Wed, 2 Feb 2011 13:01:53 -0500

> If you mean why could the protocol not have been designed otherwise, then that question is out-of-scope for this inquiry.  The spec does talk about various races and if something is said in that re…

That is what I meant, but I can accept it being out-of-scope.

> It says "send a response" and there is an opportunity to make it conditional on errors or whatever, but it just says a response.  The only conclusion is that the error sent doesn't matter as long a…

I agree.

Thanks,
Fred

---

From: Fred Isaman <[email-masked]>
Date: Wed, 2 Feb 2011 16:17:21 -0500

> I appreciate corrections and additions.  I think it will increase the productivity if people read it early and get their updates to me by 2/11.  I'm got saying that updates that after won't make bu…

Here are a few more comments.

Item 1 - My understanding of the "forgetful model" seems to map to
what you call the "normal memory" model.  In fact, I really don't
understand what you believe the forgetful model to be.  The text you
quote from 12.5.3 is poorly written, and if interpreted as you do,
that the clause allows the client to use seqid prematurely, it is just
plain wrong.

Item 6: What about a choice D, that overlapping segments with
differing roc bits conflict, and thus indicate server error?

Item 8: I suspect that client ID pair refers to the pair previously
mentioned in 12.2.10 of <client ID, layout type>.  So I think really
the question here is "do bulk returns pay attention to clora_type"?
This is similar to my other question "do bulk returns pay attention to
clora_iomode"?

Item 9:  Note that section 12.5.5.2.1.2 has problems similar to
12.5.5.2.1.3.  In particular, the algorithm for deciding based on
seqid+1 is implicitly assuming only a single LAYOUTGET outstanding.

Fred

---

From: <[email-masked]>
Date: Wed, 2 Feb 2011 17:57:58 -0500

Thanks.

> understand what you believe the forgetful model to be. 

I don't have a clue what it is.  But the phrase does appear in the spec and it is often used and I think we need to get clear about it.

As you note, there is some quoted text which is badly written.  But I decided not to take on the task of changing everything badly written or just plain wrong.  I'm limiting myself to citing the cases  where something is ambiguous or doesn't make sense and we'll be looking for agreement from the group and then we'll be looking for volunteers to write some new text. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Fred Isaman
Sent: Wednesday, February 02, 2011 4:17 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS Issues document

Fred
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 02 Feb 2011 18:01:16 -0500

> that the clause allows the client to use seqid prematurely, it is just

The text in 12.5.3 appears to assume that the client will always end up
doing a LAYOUTRETURN after a CB_LAYOUTRECALL, and because of this
assumption, it appears to say that the _client_ needs to bump the seqid
before it can send a new LAYOUTGET.

The text in 12.5.5.2.1.3, however clarifies this in case (3): the client
may have sent a LAYOUTGET _after_ it processed the CB_LAYOUTRECALL, but
_before_ it sends the LAYOUTRETURN, in which case it still has a seqid
that is the same as the one from the CB_LAYOUTRECALL
.
Conversely, 12.5.5.2.1.3 implies that if the client isn't expected to
send the LAYOUTRETURN, then the layout stateid's seqid isn't going to
get bumped by either the client or the server. This is irrespective of
whether or not the client uses the forgetful model.

IOW: there is no case (as has previously been stated on this mailing
list) where the client is supposed to bump the seqid without having
received a new stateid containing that new seqid from the server.

Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 02 Feb 2011 18:24:43 -0500

> received a new stateid containing that new seqid from the server.

In summary, the forgetful client model basically boils down to the
following:

If the client receives a CB_LAYOUTRECALL for a layout segment that it
has forgotten (or wants to forget), then it returns
NFS4ERR_NO_MATCHING_LAYOUT. The new layout stateid is set to whatever
value was received in the CB_LAYOUTRECALL (unless the client holds the
same stateid with a higher seqid already).

If there is a parallel LAYOUTGET, then cases (1) and (2) from section
12.5.5.2.1.3 apply. Case 3 doesn't, because the client isn't going to
send a LAYOUTRETURN (the same is true no matter what non-NFS_OK error
value the client replied to the CB_LAYOUTRECALL).

If the server is able to determine from the NFS4ERR_NO_MATCHING_LAYOUT
that the client no longer holds any layouts for this file, then the
layout stateid is dropped by the server, as per section 12.5.3. If the
client tries to use it again in a future LAYOUTGET, then the server
should return NFS4ERR_BAD_STATEID (again as per 12.5.3).

--------------------------------------

The question then remains, what does the server do if the client sends a
LAYOUTGET with an open stateid when the server believes the client holds
a valid layout?

I propose that there are 2 possible solutions:

     1. The server does a CB_LAYOUTRECALL on the layout stateid that it
        believes the client already holds (and replies to the LAYOUTGET
        with a NFS4ERR_RECALLCONFLICT???). If the client responds with
        NFS4ERR_NO_MATCHING layout, then the matter is moot. 
     2. The server assumes that the client has forgotten the layout
        stateid, and forgets it too. The problem here is that if the
        server's assumption is wrong, then it has no way of knowing that
        until the client tries to use that stateid for a future
        LAYOUTGET.

Unless the server knows that there is no risk associated with applying
solution (2) (for instance if the client is asking for an RW layout that
covers the entire file), then it should probably apply solution (1).

Trond

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Thu, 03 Feb 2011 10:31:03 +0200

> > I appreciate corrections and additions.  I think it will increase the productivity if people read it early and get their updates to me by 2/11.  I'm got saying that updates that after won't make…
Dave, thanks for the awesome summary!

A few comments:

> > What about the case of a CB_LAYOUTRECALL? Suppose that overlapped some existing
> > layout plus the one being gotten.  If the overall LAYOUTRETURN were done with
> > the new seqid signaling completion of the recall, and this happened
> > before the update of what ranges of the layout it has, we would be left with a
> > situation in which the client held a layout range that the server did not think
> > the client had. Am I missing something here?
According to 12.5.5.2.1:
   A client MUST NOT process a CB_LAYOUTRECALL that
   implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations
   to which the client has not yet received a reply.  The client detects
                                   ^^^^^^^^^^^^^^^^ (weak, inconsistent with below)
   such a CB_LAYOUTRECALL by examining the "seqid" field of the recall's
   layout stateid.  If the "seqid" is not exactly one higher than what
   the client currently has recorded, and the client has at least one
   LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client knows
   the server sent the CB_LAYOUTRECALL after sending a response to an
   outstanding LAYOUTGET or LAYOUTRETURN.  The client MUST wait before
   processing such a CB_LAYOUTRECALL until it processes all replies for
                                              ^^^^^^^^^^^^^^^^^^^^^
   outstanding LAYOUTGET and LAYOUTRETURN operations for the
   corresponding file with seqid less than the seqid given by
   CB_LAYOUTRECALL (lor_stateid; see Section 20.3.)

The point is that the replies MUST be fully processed including updating
the client's view of the layout before using the associated stateid,
including for the purpose of synchronizing with CB_LAYOUTRECALL.

> > What to do about "Forgetful Model"? (Consistency issue)
Another option not mentioned is dropping the forgetful model entirely and
mandate the use of LAYOUTRETURN to synchronize the client and server layout state.
This would make CB_NOMATCHING_LAYOUT problematic for pretty much anything
except the recall of a whole file layout (with iomode==IOMODE_ANY) and bulk recalls
with iomode_ANY.

> > Recall by client ID pair
eh? :)

> > Other Layout Seqid Issues
The spec must be clarified and inconsistencies be fixed regarding the cases where
the layout stateid is invalidated.  The only place where it is currently explicitly
invalidate is the lrs_present==false in the reply of LAYOUTRETURN.

1. Parallelism and use of initial (open/lock/deleg) stateid.
   - Sending multiple LAYOUTGETs with the initial stateid in parallel should be disallowed
     as this, in combination with intervening LAYOUTRETURNs, may lead to cases where the
     client and server get out of sync and the client ends up holding layout the server thinks it does not.
   - The simplest solution is to allow only a single LAYOUTGET with the initial stateid,
     synchronized with any layout operation (LAYOUTGET, LAYOUTCOMMIT, LAYOUTRETURN, CB_LAYOUTRECALL
     processing at the client side)

2. Currently the spec disallows (12.5.3, 18.43.3) sending a LAYOUTGET with the initial stateid
   after the layout stateid has been established, but at the same time it requires the client
   to stop using the layout stateid once it holds no layout.
   - We should explicitly allow the client to send (a singular) LAYOUTGET with a non-layout
     stateid mid-stream, i.e. without sending a LAYOUTRETURN and receiving lrs_present=false.
     This should imply a return of the layout for the whole file.

3. Clarification of the seqid value after CB_LAYOUTRECALL, as Fred mentioned on another thread,
   in particular:
   a. when an error is returned. In this case the stateid should be updated as the state already
      changed on the server as indicated by lor_stateid.
   b. Special case when NFS4ERR_NOMATCHING_LAYOUT is returned and the client has no layout
      the stateid on the client side can be ditched and implicitly reset to the initial (non-layout)
      stateid.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 03 Feb 2011 10:37:48 +0200

>> It seems to me that the update of the client's layoutstateid is unconditional and it doesn't depend on what the client is going to return.  After all, it also updates its stateid when it gets seqi…

Agreed.

Benny

> That is what I meant, but I can accept it being out-of-scope.

Not to derail the discussion, the seqid is used to distinguish between operations
sent before and after a CB_LAYOUTRECALL.  That said, it might be possible theoretically
to do it differently.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 03 Feb 2011 10:45:38 +0200

> should return NFS4ERR_BAD_STATEID (again as per 12.5.3).

The server may do that.

> covers the entire file), then it should probably apply solution (1).

The definition of risk here is fuzzy.  This protocol should be robust
and straight forward.
I suggest that it is up to the client to serialize any LAYOUTGET with
an open(/deleg/lock) stateid and the that the server to interpret that
as an indication that the client forgot any layout it had, i.e. it is
equivalent to an implicit LAYOUTRETURN with iomode=IOMODE_ANY for the whole file.

Benny

---

From: <[email-masked]>
Date: Fri, 4 Feb 2011 13:43:41 -0500

> In summary, the forgetful client model basically boils down to the

I think we need get to the principal point.

It is not how you would refer to your or other users of the "forgetful" model.  It is whether there is or should be am explicit definition within the spec of a client which is "using the forgetful model", whether that consideration has any consequences (for example exceptions of various sorts), and whether there is a restriction on changing from or to the forgetful model.  The alternative is that we simply define the options for the protocol and let them do whatever they want that fits, whether forgetful at some points or non-forgetful at others.

If you have this complete concept of how this should and will be used, then you have a server working with two distinct clients protocols (essentially), and it sound like that is a recipe for interoperability problems. 

> has forgotten (or wants to forget), 

?? What if it doesn't want to forget it.

> then it returns NFS4ERR_NO_MATCHING_LAYOUT. 

Understood that that is allowed in the protocol.

> The new layout stateid is set to whatever

Is this going to process new seqids not in order?  I don't see how it works.

A client gets a layout recall with seqid M+1.  Instead of processing a LAYOUTGET whose response has seqid M first, you are saying we just go ahead and propagate that.  Then get layout gotten can be added to the set of layouts after the CB_LAYOUTRECALL was processed and you wind up with these layout segments when you might not be allowed to have them.

> value the client replied to the CB_LAYOUTRECALL).

Right but are you saying that we want 12.5.5.2.1.3.1 and 12.5.5.2.1.3.2 to deal with forgetful and no-forgetful models.  Isn't better to fix the sets of cases so it deals with all the case that either model might use, and it doesn't need to explicit reference the various "models".  For example, do we want anything saying that case 3 doesn't apply to the "forgetful model".

> should return NFS4ERR_BAD_STATEID (again as per 12.5.3).

But there's about the case in which the server can't determine that the client no longer holds a layout but it doesn't hold one.  This is just a another consequence of the weak client-server layout agreement model.

> LAYOUTGET with an open stateid when the server believes the client holds

I'll add this information to the document since it doesn't appear dealt with in RFC5661.

Not sure it affects RFC5661 all that much but someone has got to explain to me why we chose to do this.

+ It avoids do LAYOUTRETURNs but the overhead of these are very low in
  that the server can be notified asynchronously and COMPOUNDs can be
  used to forget a lot of layouts without a big response count.

- By not doing its own forgets the client is deprived of the ability to
  easily return the least-recently-used layouts.  Instead of asking the 
  metadata sever to pick layouts at random (unless he expect to  co-ordinate
  LRU lists from data servers.

- If there is some reason to conflict the another layout, you can never have the
  conflict reserved in advance.  Latency to resolve the issue is at maximum.

- Additional complexity in RFC5661 and further complexity we find and
  it isn't clear when it will stop.

I guess saying it is "useful" for people to not forget their layouts seems not very convincing to me.  Am I missing something that make this desirable?

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Wednesday, February 02, 2011 6:25 PM
To: Fred Isaman; Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS Issues document

Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 04 Feb 2011 15:03:33 -0500

> ?? What if it doesn't want to forget it.

The server is recalling the layout segment, so the only options
available to the client under normal circumstances are:

     1. Assume the layout is forgotten and return
        NFS4ERR_NO_MATCHING_LAYOUT 
     2. Return NFS4_OK, and proceed to schedule a LAYOUTRETURN 
     3. Return NFS4ERR_DELAY

> Is this going to process new seqids not in order?  I don't see how it works.

The client may have received a reply to a non-conflicting LAYOUTGET in
that was processed in parallel with the recall on the server. If that is
the case, the seqid on the resulting stateid will be higher than the
seqid on the recall.

> A client gets a layout recall with seqid M+1.  Instead of processing a LAYOUTGET whose response has seqid M first, you are saying we just go ahead and propagate that.  Then get layout gotten can be…

The client can distinguish between what was processed before and after
the recall. According to section 12.5.5.2.1. the server is supposed to
bump the seqid when it sends a CB_LAYOUTRECALL, so if the LAYOUTGET was
processed before the CB_LAYOUTRECALL (and overlaps the segment being
recalled) then the client can detect that and know that it needs to be
treated as part of the recall. There is no difference here between the
forgetful model or the LAYOUTRETURN model.

The difference lies in the fact that the forgetful client has the option
of simply forgetting _all_ LAYOUTGET results that it processes after the
CB_LAYOUTRECALL, but that were processed before the CB_LAYOUTRECALL on
the server (i.e. they have seqids that precede the CB_LAYOUTRECALL).
IOW: instead of caching all CB_LAYOUTRECALL arguments in order to check
whether or not outstanding LAYOUTGET results are valid, the client just
needs to cache the seqid of the last LAYOUTRECALL, and resend all
LAYOUTGET results that have seqids that are older.

> Right but are you saying that we want 12.5.5.2.1.3.1 and 12.5.5.2.1.3.2 to deal with forgetful and no-forgetful models.  Isn't better to fix the sets of cases so it deals with all the case that eit…

I think it might be helpful to add sections in 12.5.3. to explain how
the client and server might deal with an error being returned in the
reply to CB_LAYOUTRECALL. The most important is
NFS4ERR_NO_MATCHING_LAYOUT, but it might be useful to consider something
like NFS4ERR_DELAY too (does the client update its seqid in that case?).

I'm not sure I see a need to update 12.5.5.2., unless someone thinks
that there is value in adding the paragraph to explain that the client
can sequence by forgetting and resending the LAYOUTGETs that were
processed before the CB_LAYOUTRECALL on the server (as explained above).

> I guess saying it is "useful" for people to not forget their layouts seems not very convincing to me.  Am I missing something that make this desirable?

Returning NFS4ERR_NO_MATCHING_LAYOUT to a CB_LAYOUTRECALL is much faster
than going through the whole LAYOUTRETURN process. It allows the client
to avoid a lot of the sequencing complexity that LAYOUTRETURN
introduces.

If the client needs more time to finish the outstanding READ or WRITE
requests on the segment that is being recalled, it has the option of
returning NFS4ERR_DELAY to the recall. Since the MDS will likely have to
return that anyway to the client that sent the LAYOUTGET that caused the
conflict with those outstanding READs/WRITEs, there really isn't much of
an advantage to using LAYOUTRETURN in that situation...

I agree that managing layouts is an argument for using LAYOUTRETURN, and
we will probably introduce it to the Linux client at some point in order
to do so. However the fact that the "forgetful" model is simpler to code
up and maintain on the client is a strong argument for implementing that
first, then adding the optimisations later when we start to see what
conflicts actually occur in the field.

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 04 Feb 2011 15:35:49 -0500

> conflicts actually occur in the field.

By the way. Is it possible to use FREE_STATEID to achieve the same goal
as the LAYOUTRETURN in the above case? The RFC appears to suggest that
it is...

NetApp

---

From: <[email-masked]>
Date: Mon, 7 Feb 2011 13:18:09 -0500

If we are clear that that this is a question of practicalities for implementations, then that's fine.

I just don't want the spec embracing two somewhat inconsistent universes, and any servers that only support one.

I'll check out the FREE-STATEID issues.

Thanks.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, February 04, 2011 3:36 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] pNFS Issues document

NetApp

---

From: Casey Bodley <[email-masked]>
Date: Fri, 11 Feb 2011 11:33:51 -0500

Regarding item 8, my interpretation of "CB_LAYOUTRECALL has options
for recalling all layouts associated with the fsid, client ID pair, or
just the client ID)" is slightly different.  I believe it's just
talking about the difference between LAYOUTRECALL4_ALL (for "just the
client ID") and LAYOUTRECALL4_FSID (for "the fsid, client ID pair").