Subject: [nfsv4] layout return - final update ?

From: Marc Eshel <[email-masked]>
Date: Tue, 28 Feb 2012 13:39:24 -0800

Hopefully the final correction to the logr_return_on_close result field in 

18.43.3 that should go into v4.1 errata.
Marc.

section 18.43.3 say:

The logr_return_on_close result field is a directive to return the layout 
before closing the file.  When the metadata server sets this  return value
to TRUE, it MUST be prepared to recall the layout in the case in which the
client fails to return the layout before close.  For the metadata server 
that knows a layout must be returned before a close of the file, this 
return value can be used to communicate the desired behavior to the client
and thus remove one extra step from the client's and metadata server's 
interaction.

it should say:

The logr_return_on_close result field is a directive to return or forget 
the layout before closing the file . When the metadata server sets this 
return value to TRUE, the client MUST NOT use the layout after sending 
the last CLOSE of that file. Any outstanding COMMIT to the data servers
or LAYOUTCOMMIT to the metadata server MUST be sent prior to closing the
file. This return value is used to communicate to the client that any 
layout or segment of a layout that were given with logr_return_on_close 
set to TRUE will result in the server invalidating all layouts of 
layouttype4 of loga_layout_type for the file after it was closed. Once 
the server returns logr_return_on_close set to TRUE to a client for a
given file it MUST set logr_return_on_close to TRUE for all subsequent 
layouts or segments of layoutype4 of the respective loga_layout_type for
that file and client until that file is closed.

Notes:

This directive will remove one or more extra steps from the client's and 
metadata server's interaction.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Tue, 28 Feb 2012 18:47:18 -0500

The last few sentences seem to be written assuming that that three will be only one
open file per fh/client pair.  It seem that it says that if owner A opens a file and
Gets some layouts, then of owner B opens the same file then all the layout he gets should be 
marked logr_return_on_close.  Is that what is intended?  And if so, when should 
these be invalidated.  On the first close, which the text seems to suggest?  I think the
right direction, not having thought about it terribly much, is that logr_return_on_close
layouts are open-file-associated and if that is something that you can agree with, you 
would b better off if the text said that explicitly.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Marc Eshel
Sent: Tuesday, February 28, 2012 4:39 PM
To: NFSv4
Subject: [nfsv4] layout return - final update ?

section 18.43.3 say:

it should say:

Notes:

---

From: Benny Halevy <[email-masked]>
Date: Wed, 29 Feb 2012 21:07:04 +0200

On Feb 29, 2012 1:47 AM, <[email-masked]> wrote:
> marked logr_return_on_close.  Is that what is intended?

I believe so.

And if so, when should
> these be invalidated.  On the first close, which the text seems to
suggest?

Previously what we discussed was the last CLOSE for the file/client pair.
That said, I'm not sure about multiple filehandles representing the same
file.

I think the
> would b better off if the text said that explicitly.

This approach seems valid but it complicates the case of layouts not marked
as return_on_close that survived a previous last CLOSE as these will not be
affected now.
Since we are talking about losing the layout on the last CLOSE from the
client of that file a catch all approach would be simpler.

Benny

---

From: <[email-masked]>
Date: Wed, 29 Feb 2012 15:30:48 -0500

> Previously what we discussed was the last CLOSE for the file/client pair.

OK, but the proposed text you supplied is not clear on that point.

> That said, I'm not sure about multiple filehandles representing the same file.

Another issue that needs to be clarified in the text.

> This approach seems valid but it complicates the case of layouts not marked as return_on_close that survived a previous last CLOSE as these will not be affected now.

If we are talking about layout not marked return-on-close that were sent before the first return-on-close layout, then I think that the text you proposed is exactly the same in this respect.  Or am I missing something here?

> Since we are talking about losing the layout on the last CLOSE from the client of that file a catch all approach would be simpler.

I'm not sure that that is so.  First of all, it is very natural/required for clients and servers to have data structures representing open files, and linking return-on-close layout  to these seems relatively straightforward.  on the other hand, I would not expect clients and servers to always have an infrastructure to keep track of client/fh or client/file pairs and thus always be able to see when a last close is being done.  To top it all off, you may have racing closes so last-close-at-issue may not be the same as last-close-at-receipt.  Also closes and opens can race.  Those sorts of considerations led to me thinking associating a layout with a specific open file was simpler or at least it made my head hurt less.

________________________________
From: Benny Halevy [mailto:[email-masked]]
Sent: Wednesday, February 29, 2012 2:07 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] layout return - final update ?

I believe so.

And if so, when should
> these be invalidated.  On the first close, which the text seems to suggest?

Previously what we discussed was the last CLOSE for the file/client pair.
That said, I'm not sure about multiple filehandles representing the same file.

I think the
> would b better off if the text said that explicitly.

This approach seems valid but it complicates the case of layouts not marked as return_on_close that survived a previous last CLOSE as these will not be affected now.
Since we are talking about losing the layout on the last CLOSE from the client of that file a catch all approach would be simpler.

Benny

> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Marc Eshel
> [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 29 Feb 2012 21:25:12 +0000

> Another issue that needs to be clarified in the text.

We should do the same as we do for other state: LAYOUTs for different
filehandles should be represented by different stateids and should not
be combined.

IOW: since you can't choose to CLOSE the file using either filehandle,
but MUST use the filehandle that you received when you did the OPEN, I
suggest that we extend that principle to layouts too.

> > This approach seems valid but it complicates the case of layouts not
> marked as return_on_close that survived a previous last CLOSE as these
> will not be affected now.
> 
> 
> If we are talking about layout not marked return-on-close that were
> sent before the first return-on-close layout, then I think that the
> text you proposed is exactly the same in this respect.  Or am I
> missing something here?
> 
> 
> > Since we are talking about losing the layout on the last CLOSE from
> the client of that file a catch all approach would be simpler.
> 
> I'm not sure that that is so.  First of all, it is very
> natural/required for clients and servers to have data structures
> representing open files, and linking return-on-close layout  to these
> seems relatively straightforward.  on the other hand, I would not
> expect clients and servers to always have an infrastructure to keep
> track of client/fh or client/file pairs and thus always be able to see
> when a last close is being done.  To top it all off, you may
> have racing closes so last-close-at-issue may not be the same as
> last-close-at-receipt.  Also closes and opens can race.  Those sorts
> of considerations led to me thinking associating a layout with a
> specific open file was simpler or at least it made my head hurt less. 

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Feb 2012 18:47:52 -0500 (EST)

Trond Myklebust wrote:
> > > Previously what we discussed was the last CLOSE for the
> > > file/client
> > pair.
> >
> >
> > > That said, I'm not sure about multiple filehandles representing
> > > the
> > same file.
> >
Isn't it also possible to have multiple OPENs for the same FH, but
different open_owner4's?

(Just fyi, this doesn't affect the FreeBSD
client much, since it holds onto all OPENs for a given FH until all
of the file descriptors are closed and the CLOSEes them all at the
same time. As such, the layout can be returned at that time, if any
case of return_on_closed == true has been received for the layout.)

rick

---

From: Marc Eshel <[email-masked]>
Date: Wed, 29 Feb 2012 16:25:35 -0800

Lets try to keep it simple. Can we agree that once the server returns a 
layout with return_on_closed == true for a file A (what ever way it is 
represented) to client B the serve will be allowed to invalidate the 
layouts on the last close  of file A by client B. If you disagree please 
propose a clear and complete alternative.
Marc.

From:
Rick Macklem <[email-masked]>
To:
Trond Myklebust <[email-masked]>
Cc:
[email-masked], [email-masked], david noveck <[email-masked]>
Date:
02/29/2012 03:48 PM
Subject:
Re: [nfsv4] layout return - final update ?

rick

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Feb 2012 19:53:47 -0500 (EST)

Marc Eshel wrote:
> layout with return_on_closed == true for a file A

> (what ever way it is represented)
If, by "what ever way it is represented" you mean a file
handle, then that sounds fine to me.

If you mean "any file with the same fileid and fsid from
the same server", I'm not so sure, since it would be difficult
to find all cases of this, imho. Yes, fileid and fsid are attributes,
but a client tends to cache them (with no guarantee that
they are up-to-date) and even an exhaustive search of the attribute
cache might not find them all. (I believe this is only an issue for
cases where a file has multiple FH's, possibly after a RENAME
against certains servers?)

rick

> to client B the serve will be allowed to invalidate the

---

From: Marc Eshel <[email-masked]>
Date: Wed, 29 Feb 2012 17:37:15 -0800

[email-masked] wrote on 02/29/2012 04:53:47 PM:
> Trond Myklebust, nfsv4

Yes, any fh that maps to the same file (inode). btw, files might also have 
different fh becuse of hard links not just rename.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 1 Mar 2012 03:15:40 +0000

> case of return_on_closed == true has been received for the layout.)

Yes, the fact that you can have several open_owners for the same filehandle does affect the 'last closer' issue. I'm not denying that is a complication, however I believe that the multiple filehandles issue is easily solvable thanks to the precedent that we already have for all the other existing state (open/lock state and delegation state).

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Feb 2012 22:16:01 -0500 (EST)

Marc Eshel wrote:
> > > Lets try to keep it simple. Can we agree that once the server
> > > returns
> > > a
Well, I think this is too hard for a client to handle. (ie.
Recognizing that multiple FH's are for the same file.)

When I said FH, I meant that the rule would apply to one FH,
not all FHs that map to the same file.

rick

> > > > filehandles should be represented by different stateids and
> > > > should
> > > > not
> > > > be combined.
> > > >
> > > > but MUST use the filehandle that you received when you did the
> > > > OPEN,
> > > > I
> > > > > > This approach seems valid but it complicates the case of
> > > > > > layouts
> > > > > > not
> > > > > marked as return_on_close that survived a previous last CLOSE
> > > > > as
> > > > > these
> > > > > sent before the first return-on-close layout, then I think
> > > > > that
> > > > > the
> > > > > > Since we are talking about losing the layout on the last
> > > > > > CLOSE
> > > > > > from
> > > > > track of client/fh or client/file pairs and thus always be
> > > > > able to
> > > > > see
> > > > > specific open file was simpler or at least it made my head
> > > > > hurt
> > > > > less.
> > > >
> > > >
> > > > --
> > > > Trond Myklebust

---

From: Rick Macklem <[email-masked]>
Date: Wed, 29 Feb 2012 22:19:29 -0500 (EST)

Trond Myklebust wrote:
> >> OPEN, I
> Yes, the fact that you can have several open_owners for the same
> filehandle does affect the 'last closer' issue. I'm not denying that
> is a complication, however I believe that the multiple filehandles
> issue is easily solvable thanks to the precedent that we already have
> for all the other existing state (open/lock state and delegation
> state).
> 
If I understand you, I think we agree. Above you state that different
FHs have different stateids and shouldn't be combined. To me, that sounds
like you are saying that the different FHs for the same file shouldn't
be handled as the same one.

Did I get that correct? rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 1 Mar 2012 03:51:14 +0000

ACK. :-)

Rick Macklem <[email-masked]> wrote:

---

From: Benny Halevy <[email-masked]>
Date: Thu, 01 Mar 2012 07:01:15 +0200

On 2012-02-29 22:30, [email-masked] wrote:
> *> Previously what we discussed was the last CLOSE for the file/client pair.*
> 
> *> That said, I'm not sure about multiple filehandles representing the same file.*
> 

Yeah.  Sorry for bringing that up ;-)
Anyhow, given that all layout operations, as well as CLOSE, are done on a filehandle
we can safely refer to the filehandle as representing the file.  Different filehandle
may represent the same inode (fsid:fileid) on the server but they'll be associated
with separate layout states.

> *> This approach seems valid but it complicates the case of layouts not marked as return_on_close that survived a previous last CLOSE as these will not be affected now.*
> 
> *> Since we are talking about losing the layout on the last CLOSE from the client of that file a catch all approach would be simpler.*
> 

The root problem is that the layout stateid is not tied to any specific open stateid.
We require the client to first OPEN the filehandle to get the first layout stateid but from then on
the layout stateid is independent and layout operations affect it regardless of any OPENs and CLOSEs
on that filehandle.

Benny

> 
> 
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--
> *From:* Benny Halevy [mailto:[email-masked]]
> *Sent:* Wednesday, February 29, 2012 2:07 PM
> *To:* Noveck, David
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [nfsv4] layout return - final update ?
> 
> 
> On Feb 29, 2012 1:47 AM, <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> From: [email-masked] <mailto:[email-masked]> [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Marc Eshel
>> [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>

---

From: Marc Eshel <[email-masked]>
Date: Wed, 29 Feb 2012 21:59:22 -0800

>From my server point of view either options are fine I am just trying to 
make as many of you happy which is very tricky. So let me try again. What 
I think most people agree on is the last statement by Benny. Given that 
all layout operations, as well as CLOSE, are done on a fh we can safely 
refer to the fh as representing the file. So all the rules regarding 
return-on-close refer to (fh, client).
If I don't hear more objection I will try to write it into the text and 
probably trigger some other issue :)
Marc.

From:
Benny Halevy <[email-masked]>
To:
[email-masked]
Cc:
[email-masked], [email-masked]
Date:
02/29/2012 09:01 PM
Subject:
Re: [nfsv4] layout return - final update ?
Sent by:
[email-masked]

Yeah.  Sorry for bringing that up ;-)
Anyhow, given that all layout operations, as well as CLOSE, are done on a 
filehandle
we can safely refer to the filehandle as representing the file.  Different 
filehandle
may represent the same inode (fsid:fileid) on the server but they'll be 
associated
with separate layout states.

> *> This approach seems valid but it complicates the case of layouts not 
marked as return_on_close that survived a previous last CLOSE as these 
will not be affected now.*
> 
> If we are talking about layout not marked return-on-close that were sent 
before the first return-on-close layout, then I think that the text you 
proposed is exactly the same in this respect.  Or am I missing something 
here?
> 
> 
> *> Since we are talking about losing the layout on the last CLOSE from 
the client of that file a catch all approach would be simpler.*
> 
> I'm not sure that that is so.  First of all, it is very natural/required 
for clients and servers to have data structures representing open files, 
and linking return-on-close layout  to these seems relatively 
straightforward.  on the other hand, I would not expect clients and 
servers to always have an infrastructure to keep track of client/fh or 
client/file pairs and thus always be able to see when a last close is 
being done.  To top it all off, you may have racing closes so 
last-close-at-issue may not be the same as last-close-at-receipt.  Also 
closes and opens can race.  Those sorts of considerations led to me 
thinking associating a layout with a specific open file was simpler or at 
least it made my head hurt less. 

The root problem is that the layout stateid is not tied to any specific 
open stateid.
We require the client to first OPEN the filehandle to get the first layout 
stateid but from then on
the layout stateid is independent and layout operations affect it 
regardless of any OPENs and CLOSEs
on that filehandle.

Benny

suggest?
file.
>> From: [email-masked] <mailto:[email-masked]> [
mailto:[email-masked] <mailto:[email-masked]>] On Behalf 
Of Marc Eshel