Subject: [nfsv4] layout return

From: Marc Eshel <[email-masked]>
Date: Wed, 22 Feb 2012 21:15:05 -0800

It looks like some developers understood the return_on_close to mean that 
if the server sets it on the client doesn't have to return the layout and 
the close is an indication that it is not holding the layout anymore. I 
think this is a good idea but from the following section of the spec it 
sounds that if the client doesn't return the layout the server still needs 
to recall the layout. Is there a reason why the server can't just assume 
that close implies that the client gave up the layout?
Marc.

18.43.3

The logr_return_on_close result field is a directive to return the
  layout before closing the file.  When the metadata server sets this
  return value to TRUE, it MUST be prepared to recall the layout in the
  case in which the client fails to return the layout before close.
  For the metadata server that knows a layout must be returned before a
  close of the file, this return value can be used to communicate the
  desired behavior to the client and thus remove one extra step from
  the client's and metadata server's interaction.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 03:53:13 -0800

> that close implies that the client gave up the layout?

We've discussed this in the past but it seems the ball got dropped.

http://www.ietf.org/mail-archive/web/nfsv4/current/msg08817.html

Benny

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 23 Feb 2012 17:08:25 +0000

Marc,

Given the previous discussion, you should draft the needed errata, send to the WG and once
we have concensus, it can be posted to the ietf errata site.

Spencer

From: [email-masked] [mailto:[email-masked]] On Behalf Of Benny Halevy
Sent: Thursday, February 23, 2012 3:53 AM
To: Marc Eshel
Cc: NFSv4
Subject: Re: [nfsv4] layout return

Benny

> [email-masked]<mailto:[email-masked]>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Feb 2012 11:17:14 -0800

So here is my correction to the logr_return_on_close result field in 
18.43.3 that should go into v4.1 errata.

section 18.43.3 say:
The logr_return_on_close result field is a directive to return the layout 
before closing the file.  When the metadata server sets this  return value 
to TRUE, it MUST be prepared to recall the layout in the case in which the 
client fails to return the layout before close.  For the metadata server 
that knows a layout must be returned before a close of the file, this 
return value can be used to communicate the desired behavior to the client 
and thus remove one extra step from the client's and metadata server's 
interaction.
it should say:
The logr_return_on_close result field is a directive to return or forget 
the layout before closing the file . When the metadata server sets this 
return value to TRUE, it can assume that the client MUST not use the 
layout after the file was closed. This return value is used to communicate 
to the client that any layout or segment of a layout that were given with 
logr_return_on_close set to TRUE will result in the server invalidating 
all layouts for the file after it was closed. 
Notes:
This directive will remove one or more extra steps from the client's and 
metadata server's interaction.

Marc.

From:
Spencer Shepler <[email-masked]>
To:
Benny Halevy <[email-masked]>, Marc Eshel <[email-masked]>
Cc:
NFSv4 <[email-masked]>
Date:
02/23/2012 09:14 AM
Subject:
Re: [nfsv4] layout return
Sent by:
[email-masked]

Marc,
 
Given the previous discussion, you should draft the needed errata, send to 
the WG and once
we have concensus, it can be posted to the ietf errata site.
 
Spencer
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
Benny Halevy
Sent: Thursday, February 23, 2012 3:53 AM
To: Marc Eshel
Cc: NFSv4
Subject: Re: [nfsv4] layout return

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 23 Feb 2012 19:34:26 +0000

> return value to TRUE, it can assume that the client MUST not use the 

I think we need a stronger statement:

"When the metadata server sets the logr_return_on_close value to TRUE,
the client MUST NOT use the layout after sending the last CLOSE of that
file. Any outstanding COMMIT to the DSes or LAYOUTCOMMIT to the MDS MUST
be sent prior to closing the file."

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Robert Gordon <[email-masked]>
Date: Thu, 23 Feb 2012 13:43:19 -0600

> be sent prior to closing the file."

This is good. 

-- Robert.

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Feb 2012 14:26:34 -0800

Update to my correction to the logr_return_on_close result field in 
18.43.3 that should go into v4.1 errata.
Marc.

section 18.43.3 say:

it should say:

The logr_return_on_close result field is a directive to return or forget 
the layout before closing the file . When the metadata server sets this 
return value to TRUE, the client MUST NOT use the layout after sending 
the last CLOSE of that file. Any outstanding COMMIT to the data servers
or LAYOUTCOMMIT to the metadata server MUST be sent prior to closing the
file. This return value is used to communicate to the client that any 
layout or segment of a layout that were given with logr_return_on_close 
set to TRUE will result in the server invalidating all layouts for the 
file after it was closed. 

Notes:

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 17:54:07 -0500

> This return value is used to communicate to the client that any

The use of "any" and "all" above reads as logr_return_on_close being a global
flag - if server sets the flag once, all layouts for the file are invalidated
on close, even those for which the flag was set to FALSE.  That's not what was
intended, so the following text would be better:

This return value is used to inform the client that any layout or segment
of a layout that were given with logr_return_on_close set to TRUE will be
invalidated by the server when the client closes the file.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Marc Eshel
> Sent: Thursday, February 23, 2012 5:27 PM
> To: Myklebust, Trond
> Cc: NFSv4
> Subject: [nfsv4] layout return - update
>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2012 15:13:18 -0800

> invalidated by the server when the client closes the file.

I agree that this is a valid approach, but it complicates both the client
and server and I don't think this complexity is worthwhile.

In addition, this will not be as useful for the forgetful client model
as the server will not know whether the client released all its layout state
or just the segments that were marked with return_on_close.

Another corner case might be with overlapping segments that are incompatible
with regards to return_on_close.

Based on the oral discussions we had on previous meetings I'm inclined
to prefer a simpler, catch-all approach over a more complex, optimized approach.

In most cases, I assume that the server will either mark all segments with
return_on_close or none of them. In some cases, the server might mark
read-write layouts as return_on_close and not read-only layouts.
This may be the most common case that will benefit from the fine grain solution
and again, I'm not sure it's worth the trouble.

Benny

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Feb 2012 15:15:43 -0800

The intended was the way it was written. After talking to few developers 
at the connectathon the agreement was the once the file is closed all 
layouts will be invalidated, and as I hear it was the consensus in the 
last IETF meeting. Typically this will be a server policy for all files 
but it is flexible enough for a given file, and definitely not needed for 
some layouts of a file and not others. It is an unnecessary complexity on 
both server and client that don't seem to be required. 
Marc.

> > The logr_return_on_close result field is a directive to return the 
layout

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 18:19:10 -0500

Hi Benny,

> and again, I'm not sure it's worth the trouble.

I'd prefer requiring all-or-none marking by the server as it reduces the
opportunity for the client to get confused.  Assuming that the client is
implemented with a single state bit, the error case interoperability (server
marks only some of the segments as return_on_close) is a bit odd:
- Server has to retain all the segments (just in case client didn't notice).
	A server that doesn't like this pays for its failure to mark consistently
	with a recall callback.
- Client can forget what it feels like forgetting.

Thanks,
--David

> From: Benny Halevy [mailto:[email-masked]]
> Sent: Thursday, February 23, 2012 6:13 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] layout return - update
>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Feb 2012 15:36:27 -0800

If you insist I will change it to all-or-none but let me try one more time 
to convince you. The client needs only one bit to keep track of this flag 
since once it is TRUE it applies to all layouts of the file and it will 
stay true until you close the file. It will also be nice for a server that 
starts by allowing clients to keep layouts by setting this flag to FALSE 
will switch police one it is low on memory to return layout at close to 
reduce the number of layouts that it needs to remember.
Marc.

layout
> > >> return value can be used to communicate the desired behavior tothe 
client

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 24 Feb 2012 01:08:50 +0100

> - Client can forget what it feels like forgetting.

I agree that this would greatly simplify the model. That said, the Linux
client shouldn't currently care: we can deal with servers that do the
'partial return-on-close' thing.

Cheers
  Trond

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 19:23:50 -0500

Marc,

The disconnect I'm having with this is that a client that forgets the layout
has to remember this flag from the layout.  OTOH, I understand your concern
about the policy switch case - having to issue a heavy duty recall at that
point just to switch policy would not be good.

I observe that once the server has set the flag to TRUE, that behavior applies
going forward no matter what the server does (i.e., if the flag is FALSE on the
next layout for the same client and file, the flag is ignored because the
required behavior is as if the flag were TRUE).

Would it be reasonable to require that if the server ever sets the flag to TRUE
for a <client, layout> pair, then it MUST set the flag to true for all subsequent
layouts or segments of layouts for that <client, layout> pair?

Thanks,
--David

> From: Marc Eshel [mailto:[email-masked]]
> Sent: Thursday, February 23, 2012 6:36 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]
> Subject: RE: [nfsv4] layout return - update
>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Feb 2012 16:33:14 -0800

Sure, I think this is a reasonable compromise, If there are no more 
comments in a day or two I will make another try at the wording of the 
errata.
Thanks, Marc. 

From:
<[email-masked]>
To:
<[email-masked]>
Cc:
[email-masked]
Date:
02/23/2012 04:24 PM
Subject:
Re: [nfsv4] layout return - update
Sent by:
[email-masked]

Marc,

The disconnect I'm having with this is that a client that forgets the 
layout
has to remember this flag from the layout.  OTOH, I understand your 
concern
about the policy switch case - having to issue a heavy duty recall at that
point just to switch policy would not be good.

I observe that once the server has set the flag to TRUE, that behavior 
applies
going forward no matter what the server does (i.e., if the flag is FALSE 
on the
next layout for the same client and file, the flag is ignored because the
required behavior is as if the flag were TRUE).

Would it be reasonable to require that if the server ever sets the flag to 
TRUE
for a <client, layout> pair, then it MUST set the flag to true for all 
subsequent
layouts or segments of layouts for that <client, layout> pair?

Thanks,
--David

> > > In most cases, I assume that the server will either mark all 
segments
> with
> > > >> set to TRUE will result in the server invalidating all layouts 
for
> the
> > > as the server will not know whether the client released all its 
layout
> state
> > > In most cases, I assume that the server will either mark all 
segments
> with
> > > >> before closing the file.  When the metadata server sets this 
return
> value
> > > >> the layout before closing the file . When the metadata server 
sets
> this
> > > >> set to TRUE will result in the server invalidating all layouts 
for
> the

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 24 Feb 2012 01:53:33 +0100

> point just to switch policy would not be good.

Ermm... Why?

The forgetful client shouldn't need to care about what the server state
_as_long_as_it_takes_a_conservative_approach. I don't see how
return-on-close changes that.

> layouts or segments of layouts for that <client, layout> pair?

The client won't ever care. This whole discussion is about allowing the
server to drop the layout recall in a situation where the client _knows_
that it is unnecessary...

Cheers
  Trond

---

From: <[email-masked]>
Date: Fri, 24 Feb 2012 12:18:54 -0500

Hi Trond,

> > point just to switch policy would not be good.

I was thinking that an all-or-nothing marking discipline would require a recall
for a server to return_on_close=TRUE.  As the server's under load at that point,
it's more effective to mark the next layout with that and drop the data structures
to track all the layouts for that <client, file> pair.

> return-on-close changes that.

I agree - I was thinking about the server, where we both agree that we're trying
to avoid recalls.

> that it is unnecessary...

I observe that return_on_close=TRUE is one-way sticky - after the server sets that
to TRUE once, it's TRUE forever for that <client, file> pair.  I'd prefer that the
protocol require the server to tell the truth on the wire (be conservative in what
is sent), but I also agree that the client should ignore a FALSE value that comes after
a TRUE value (be liberal in what is accepted).

Thanks,
--David

> From: Trond Myklebust [mailto:[email-masked]]
> Sent: Thursday, February 23, 2012 7:54 PM
> To: Black, David