Subject: [nfsv4] forgetfulness and multiple layoutget's wth open stateid

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 23 Feb 2012 08:50:46 +0000

There was some discussion about this at connectathon today. 

As you know, to get to the layout, a client needs to issue a LAYOUTGET with the open stateid. According to the spec, a client MUST use the layout it got on subsequent LAYOUTGET operations.

12.5.2 Getting a Layout
....

   In order to get a layout, the client must first have opened the file
   via the OPEN operation.  When a client has no layout on a file, it
   MUST present an open stateid, a delegation stateid, or a byte-range
   lock stateid in the loga_stateid argument.  A successful LAYOUTGET
   result includes a layout stateid.  The first successful LAYOUTGET
   processed by the server using a non-layout stateid as an argument
   MUST have the "seqid" field of the layout stateid in the response set
   to one.  Thereafter, the client MUST use a layout stateid (see
   Section 12.5.3) on future invocations of LAYOUTGET on the file, and
   the "seqid" MUST NOT be set to zero.  Once the layout has been
   retrieved, it can be held across multiple OPEN and CLOSE sequences.
   Therefore, a client may hold a layout for a file that is not
   currently open by any user on the client.  This allows for the
   caching of layouts beyond CLOSE.
...

Apparently, there are use cases (or maybe conditions) where a client would forget the layout it has and that could result in issuing a LAYOUTGET with the same open stateid that it used before to get the layout stateid. The catch here is that the server did not forget the layout and client did not do a LAYOUTRETURN. So, oblivious to the fact that the client lost track of it, the server returns the exisiting layout stateid. That begs for few questions:

1). Is it a bad thing for server to return existing layout stateid and potentially return a seqid that may not start with 1?

2).Should/Must the server assume that the second LAYOUTGET with open stateid from the client is a hint in itself for the server to remove the existing layout (and generate new layout) and/or set the seqid back to one?

I'm of the opinion that, server cannot assume that the second LAYOUTGET(open stateid) is an indication for server to delete earlier instance of layout. Any thoughts?

- Pranoop

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 04:25:52 -0800

> Apparently, there are use cases (or maybe conditions) where a client would forget the layout it has and that could result in issuing a LAYOUTGET with the same open stateid that it used before to ge…

We've discussed this issue in the past, for example here:
http://www.spinics.net/lists/linux-nfs/msg19053.html

According to the proposed errata, the client must serialize layoutgets
with the open stateid. This allows the server to release the layout
state the client might had as if it implicitly sent a layoutreturn for
the file.

> 1). Is it a bad thing for server to return existing layout stateid and potentially return a seqid that may not start with 1?

given the serialization requirement on the client the assumption (and
the simulatio I ran) was that this would work.

> 2).Should/Must the server assume that the second LAYOUTGET with open stateid from the client is a hint in itself for the server to remove the existing layout (and generate new layout) and/or set th…

Yes

> I'm of the opinion that, server cannot assume that the second LAYOUTGET(open stateid) is an indication for server to delete earlier instance of layout. Any thoughts?

The errata should say the opposite.
For the forgetful client model to work this protocol issue must be fixed.

Benny

>
> - Pranoop
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 23 Feb 2012 17:07:00 +0000

Benny,

You mention a proposed errata; do you have a reference for that?
If so, great, if not, then something needs to hit this alias
so we can have a comment and then submit it to the ietf errata site.

Spencer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benny Halevy
Sent: Thursday, February 23, 2012 4:26 AM
To: Erasani, Pranoop
Cc: [email-masked]
Subject: Re: [nfsv4] forgetfulness and multiple layoutget's wth open stateid

Yes

Benny

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 23 Feb 2012 17:27:08 +0000

I searched through errata and It doesn't look like the errata ever made it. NOTE: there are some objections to this in the referred thread.

I agree with Spencer that this needs to hit alias again for a meaningful discussion.

- Pranoop

> From: Spencer Shepler [mailto:[email-masked]]
> Sent: Thursday, February 23, 2012 9:07 AM
> To: Benny Halevy; Erasani, Pranoop
> Cc: [email-masked]
> Subject: RE: [nfsv4] forgetfulness and multiple layoutget's wth open stateid
> 
> 
> Benny,
> 
> If so, great, if not, then something needs to hit this alias so we can have a
> comment and then submit it to the ietf errata site.
> 
> Spencer
> 
> 
> > As you know, to get to the layout, a client needs to issue a LAYOUTGET with
> the open stateid. According to the spec, a client MUST use the layout it got
> on subsequent LAYOUTGET operations.
> >
> > 12.5.2 Getting a Layout
> > ....
> >
> > Apparently, there are use cases (or maybe conditions) where a client
> would forget the layout it has and that could result in issuing a LAYOUTGET
> with the same open stateid that it used before to get the layout stateid. The
> catch here is that the server did not forget the layout and client did not do a
> LAYOUTRETURN. So, oblivious to the fact that the client lost track of it, the
> server returns the exisiting layout stateid. That begs for few questions:
> 
> According to the proposed errata, the client must serialize layoutgets with
> the open stateid. This allows the server to release the layout state the client
> might had as if it implicitly sent a layoutreturn for the file.
> 
> 
> >
> > 1). Is it a bad thing for server to return existing layout stateid and
> potentially return a seqid that may not start with 1?
> 
> given the serialization requirement on the client the assumption (and the
> simulatio I ran) was that this would work.
> 
> >
> > 2).Should/Must the server assume that the second LAYOUTGET with open
> stateid from the client is a hint in itself for the server to remove the existing
> layout (and generate new layout) and/or set the seqid back to one?
> 
> Yes
> 
> >
> > I'm of the opinion that, server cannot assume that the second
> LAYOUTGET(open stateid) is an indication for server to delete earlier instance
> of layout. Any thoughts?
>

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 23 Feb 2012 18:23:17 +0000

> 1). Is it a bad thing for server to return existing layout stateid and potentially return a seqid that may not start with 1?

No. It would seem like a reasonable thing to me for the server to be
allowed to continue to track a layout if it believes that layout is
still valid. A forgetful client cannot assume that the server will
always forget too.

That said, I believe that it would be wrong for the server to refuse the
LAYOUTGET request on the basis that it thinks the client knows about a
layout that has been forgotten.

> I'm of the opinion that, server cannot assume that the second LAYOUTGET(open stateid) is an indication for server to delete earlier instance of layout. Any thoughts?

I don't see that there is much of a consequence for the client either
way. A forgetful client must always be prepared for a recall that it
doesn't recognise.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 10:32:09 -0800

> You mention a proposed errata; do you have a reference for that?

Unfortunately, no.  If I recall correctly we discussed it verbally in
Maastricht but I can't find a written reference od the discussion.
(Probably summarized in my Panasas mailbox which I have no access to)

I can draft a proposal and send to the list.

Benny

---

From: Fred Isaman <[email-masked]>
Date: Thu, 23 Feb 2012 13:48:17 -0500

> way. A forgetful client must always be prepared for a recall that it

We went through this last year.  Consensus was reached then that the
client sending a LAYOUTGET(openstateid) implied that the client had
forgotten all previous layout state.  The current linux client is
written using this assumption.  As discussed before, without this
assumption, a client could send parallel LAYOUTGET(openstateids), and
there are then cases where the server and client get out of synch.  So
the client has to serialize LAYOUTGET(openstateids).  But once it has
sent the first, it has a layoutstateid and may not send another
openstateid until it has forgotten its layout state.

Fred

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 23 Feb 2012 18:59:19 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Fred Isaman
> Sent: Thursday, February 23, 2012 10:48 AM
> To: Myklebust, Trond
> Cc: Erasani, Pranoop; [email-masked]
> >> As you know, to get to the layout, a client needs to issue a LAYOUTGET
> with the open stateid. According to the spec, a client MUST use the layout it
> got on subsequent LAYOUTGET operations.
> >>
> >> 12.5.2 Getting a Layout
> >> ....
> >>
> > That said, I believe that it would be wrong for the server to refuse
> > the LAYOUTGET request on the basis that it thinks the client knows
> > about a layout that has been forgotten.
> >
> LAYOUTGET(open stateid) is an indication for server to delete earlier
> instance of layout. Any thoughts?
> >
> We went through this last year.  Consensus was reached then that the client
> sending a LAYOUTGET(openstateid) implied that the client had forgotten all
> previous layout state.  The current linux client is written using this
> assumption.  As discussed before, without this assumption, a client could
> send parallel LAYOUTGET(openstateids), and there are then cases where the
> server and client get out of synch.  So the client has to serialize
> LAYOUTGET(openstateids).  But once it has sent the first, it has a
> layoutstateid and may not send another openstateid until it has forgotten
> its layout state.

Consensus between whom though? Linux client and Linux server?

Errata has not been filed. It needs to go through discussion again. I'd wait for Benny's proposed errata.

- Pranoop

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 23 Feb 2012 18:59:41 +0000

Reference the thread of discussion please and then someone will need to volunteer to provide the updated text to submit as an errata.

Anyone picking up on a theme?

Spencer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Fred Isaman
Sent: Thursday, February 23, 2012 10:48 AM
To: Myklebust, Trond
Cc: Erasani, Pranoop; [email-masked]
Subject: Re: [nfsv4] forgetfulness and multiple layoutget's wth open stateid

Fred

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 11:46:43 -0800

> Anyone picking up on a theme?

I volunteer.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 14:41:00 -0800

My proposed errata follows below.
The gist of it is:
1. Initial LAYOUTGETs using non-layout stateid reset the layout state the client
   holds on the server.
2. Therefore these LAYOUTGETs MUST be serialized with any other layout operation
   on the file.

Benny

Section 12.5.2. says:
   The first successful LAYOUTGET
   processed by the server using a non-layout stateid as an argument
   MUST have the "seqid" field of the layout stateid in the response set
   to one.  Thereafter, the client MUST use a layout stateid (see
   Section 12.5.3) on future invocations of LAYOUTGET on the file, and
   the "seqid" MUST NOT be set to zero.

It should say:
   The first successful LAYOUTGET
   processed by the server using a non-layout stateid as an argument
   MUST have the "seqid" field of the layout stateid in the response set
   to one.  Thereafter, the client MUST use a layout stateid (see
   Section 12.5.3) on future invocations of LAYOUTGET on the file, and
   the "seqid" MUST NOT be set to zero.
|  The client MUST serialize LAYOUTGET operations using a non-layout
|  stateid with any other operation affecting the layout state on the file,
|  including CB_LAYOUTRECALL, to allow consistent initialization of the
|  layout state.

Add the following paragraph to section 12.5.3.:
   When the client is using the forgetful model (Section 12.5.5.1) it MAY
   forget its layout state and associated layout stateid at any time.
   In such a case, the client MUST use a non-layout stateid for the next
   LAYOUTGET operation.  This will signal the server that the client has
   no more layouts on the file and its respective layout state can be
   released before issuing a new layout in response to LAYOUTGET.

Section 12.5.5.2.1. says:
   One critical issue with regard to layout operations sequencing
   concerns callbacks.  The protocol must defend against races between
   the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
   CB_LAYOUTRECALL.  A client MUST NOT process a CB_LAYOUTRECALL that
   implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations
   to which the client has not yet received a reply.  The client detects
   such a CB_LAYOUTRECALL by examining the "seqid" field of the recall's
   layout stateid.  If the "seqid" is not exactly one higher than what
   the client currently has recorded, and the client has at least one
   LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client knows
   the server sent the CB_LAYOUTRECALL after sending a response to an
   outstanding LAYOUTGET or LAYOUTRETURN.

It should say:
   One critical issue with regard to layout operations sequencing
   concerns callbacks.  The protocol must defend against races between
   the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
   CB_LAYOUTRECALL.  A client MUST NOT process a CB_LAYOUTRECALL that
   implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations
   to which the client has not yet received a reply.  The client detects
   such a CB_LAYOUTRECALL by examining the "seqid" field of the recall's
   layout stateid.  If the "seqid" is not exactly one higher than what
   the client currently has recorded, and the client has at least one
   LAYOUTGET and/or LAYOUTRETURN operation outstanding,
|  or if the client has a outstanding LAYOUTGET with a non-layout stateid,
   the client knows
   the server sent the CB_LAYOUTRECALL after sending a response to an
   outstanding LAYOUTGET or LAYOUTRETURN.

Section 12.5.5.2.1.1. says:
   It is permissible for the client to send multiple parallel LAYOUTGET
   operations for the same file or multiple parallel LAYOUTRETURN
   operations for the same file or a mix of both.

It should say:
    It is permissible for the client to send multiple parallel LAYOUTGET
    operations for the same file
|   using the layout stateid
    or multiple parallel LAYOUTRETURN
    operations for the same file or a mix of both.

Section 12.5.5.2.1.2. says:
   Note
   that in the first case, the "seqid" in the layout stateid of the
   recall is two greater than what the client has recorded;

It should say:
   Note
   that in the first case, the "seqid" in the layout stateid of the
   recall is two greater than what the client has recorded,
|  or the client has an outstanding LAYOUTGET using a non-layout stateid;

After the following paragraph in Section 18.43.3:
   The loga_stateid field specifies a valid stateid.  If a layout is not
   currently held by the client, the loga_stateid field represents a
   stateid reflecting the correspondingly valid open, byte-range lock,
   or delegation stateid.  Once a layout is held on the file by the
   client, the loga_stateid field MUST be a stateid as returned from a
   previous LAYOUTGET or LAYOUTRETURN operation or provided by a
   CB_LAYOUTRECALL operation (see Section 12.5.3).

Add the following paragraph:
|  When loga_stateid represents a non-layout stateid the server MUST
|  release any layout state previously held by the client as if the client
|  explicitly returned the layout using a corresponding LAYOUTRETURN
|  with lora_iomode of LAYOUTIOMODE4_ANY, lr_returntype of LAYOUTRETURN4_FILE,
|  lrf_offset value 0, and lrf_length value NFS4_UINT64_MAX. This is
|  particularly useful to clients implementing the forgetful model (Section
|  12.5.5.1) when acquiring a layout after unilaterally forgetting the layout
|  state.  This requirement mandates that the client serializes sending
|  a LAYOUTGET with non-layout statid with any other layout operation on
|  the file (See section 12.5.2).

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 18:12:19 -0500

>  forget its layout state and associated layout stateid at any time.

And when it is not using the forgetful model, it MAY NOT?  Aren't you
really saying "A client MAY always forget its layout state and associated 
layout stateid at any time"?  If it was a mistake for RFC5661 not to say 
that, then it was an awfully big one, and we should think very carefully
before making this correction/change.

Note that section 15.5.5.1 is entitled "Layout Recall Callback Robustness".

It does say it "may be useful for clients to 'forget' details about what
layouts and ranges the client actually has, leading to the
server's layout ranges being beyond those that the client "thinks"
it has".

I'm not clear that the fact that you have a layout for a particular file
or not is the kind of detail that is being referred to. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benny Halevy
Sent: Thursday, February 23, 2012 5:41 PM
To: Erasani, Pranoop
Cc: Myklebust, Trond; [email-masked]
Subject: Re: [nfsv4] forgetfulness and multiple layoutget's wth open stateid

Benny

Section 12.5.2. says:

It should say:

It should say:

It should say:

It should say:
   Note

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 15:18:19 -0800

> before making this correction/change.

Agreed.

> or not is the kind of detail that is being referred to. 

Unfortunately, I think it is indeed the right reference.  It might be
useful to devote a complete section for the forgetful client model and
its various considerations for the client and the server.

Benny

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 24 Feb 2012 00:05:39 +0000

> > before making this correction/change.

There is no choice. Section 12.5.5.1 says that

"As long as the client does not assume it has layouts that are beyond
what the server has granted, this is a safe practice."

...and so we have coded the Linux client based on that principle.

> its various considerations for the client and the server.

We need to reconcile the RFC5661 errata with the (often spirited)
discussions that have been held on this mailing list, as well as
official IETF meetings. We also need to reconcile it with the facts on
the ground.

NetApp

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 24 Feb 2012 00:46:34 +0000

As to the question of a new section.  Please consider a short errata to 
correct any immediate inconsistencies and then publication of a separate
RFC that would address the larger issues.  Essentially, that RFC would
become part of an "bis" document but the content/agreement/behavior can
be captured now and published in a way that is useful to the larger 
community.

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Myklebust, Trond
> Sent: Thursday, February 23, 2012 4:06 PM
> To: Benny Halevy
> > >>  When the client is using the forgetful model (Section 12.5.5.1) it
> > >> MAY  forget its layout state and associated layout stateid at any
> time.
> > >
> > > And when it is not using the forgetful model, it MAY NOT?  Aren't
> > > you really saying "A client MAY always forget its layout state and
> > > associated layout stateid at any time"?  If it was a mistake for
> > > RFC5661 not to say that, then it was an awfully big one, and we
> > > should think very carefully before making this correction/change.
> >
> > Agreed.
> 
> "As long as the client does not assume it has layouts that are beyond what
> the server has granted, this is a safe practice."
> 
> > > It does say it "may be useful for clients to 'forget' details about
> > > what layouts and ranges the client actually has, leading to the
> > > I'm not clear that the fact that you have a layout for a particular
> > > file or not is the kind of detail that is being referred to.
> >
> discussions that have been held on this mailing list, as well as official
> IETF meetings. We also need to reconcile it with the facts on the ground.
> 
> --
> Trond Myklebust

---

From: Benny Halevy <[email-masked]>
Date: Tue, 28 Feb 2012 12:32:49 +0200

Changes from previous version:

* Incorporating David Noveck's comment
http://www.ietf.org/mail-archive/web/nfsv4/current/msg10699.html

Version 1:
http://www.ietf.org/mail-archive/web/nfsv4/current/msg10697.html

---

Section 12.5.2. says:

It should say:

Add the following paragraph to section 12.5.3.:
   A client MAY always forget its layout state and associated
   layout stateid at any time (See also section 12.5.5.1).
   In such case, the client MUST use a non-layout stateid for the next
   LAYOUTGET operation.  This will signal the server that the client has
   no more layouts on the file and its respective layout state can be
   released before issuing a new layout in response to LAYOUTGET.

It should say:

It should say:

It should say:
   Note

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 28 Feb 2012 14:20:25 +0000

>    implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations

This is not a "MUST NOT".

A forgetful client just needs to remember to discard the results of
those outstanding LAYOUTGETs with seqid < CB_LAYOUTRECALL seqid. We
already do this in the existing Linux pNFS client.

>    outstanding LAYOUTGET or LAYOUTRETURN.

The client knows that all outstanding LAYOUTGETs which return a seqid <
that of the CB_LAYOUTRECALL are affected by the recall. All other
LAYOUTGETs will either return NFS4ERR_RECALLCONFLICT, or are unaffected
by the recall.

If the server processes an outstanding LAYOUTRETURN after it has sent
the recall, then presumably it will return NFS4ERR_OLD_STATEID, since
the seqid send by the client will be < that recorded in the recall.
Again, there should be no ambiguity.
If it was a bulk recall, then the server will return
NFS4ERR_RECALLCONFLICT in response a non-bulk LAYOUTRETURN. (BTW: we
need to add NFS4ERR_RECALLCONFLICT to the allowed return values for
LAYOUTRETURN - it is mentioned in section 12.5.5.2.1.5, but not in the
allowed return values sections).

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Wed, 29 Feb 2012 16:38:04 +0200

> already do this in the existing Linux pNFS client.

I see your point but I'm afraid this is out of scope of this
particular errata topic.

We better encapsulate this in a different errata.
Are you aware of any other changes required in the area of
forget client behavior?

Benny