Subject: [nfsv4] Layout recall by deviceid

From: "Erasani, Pranoop" <[email-masked]>
Date: Wed, 19 Aug 2009 23:11:18 -0700

Hi Folks.
 
I have been told that this was already discussed and was not well
received. I'm not sure of the exact reasons. I have a server use case in
mind that is worth exploring. I believe layout recalls by deviceid will
be beneficial in such cases.
 
- Let's say I have a striping configuration of 1 MDS and 4 DS's (DS1,
DS2, DS3 and DS4) and let's say there we have created some files on the
filesystem that follows that configuration. 
 
- Let's say, I restripe this device configuration by adding one more DS.
So, the configuration now becomes 1 MDS and 5 DS's (DS1, DS2, DS3, DS4
and DS5). All the new files that I create from now on, will be created
with new configuration.
 
- So, essentially we have a filesystem when served over pNFS could have
multiple device mappings. In that respect, the layout of files created
via first and the subsequent configuration will be different.
 
Now, let's say if a device address on DS5 fails over to some other place
(another node in the cluster), I'd like to recall the layouts for that
deviceid only. Recalling layouts for that FSID as a whole does not make
sense in this case. I believe that we should provide layout recall by
deviceid as an option as well for such use cases.
 
An obvious way of looking at this situation is to have server initiate a
CB_NOTIFY_DEVICEID (DELETE) and let the clients issue a GETDEVICEINFO,
but it still doesn't ensure cover the clients which don't support the
device notification. NOTE: servers MUST implement device notification,
but clients don't have to.
 
It would be nice to recall layouts by deviceid, so that cases like above
can be optimized. If there is enough interest, we could add it to
NFSv4.2 work item list. Let me know, what you guys think.
 
- Pranoop

---

From: Tom Haynes <[email-masked]>
Date: Thu, 20 Aug 2009 01:36:47 -0500

Erasani, Pranoop wrote:
> received. I'm not sure of the exact reasons. I have a server use case 
> in mind that is worth exploring. I believe layout recalls by deviceid 
> will be beneficial in such cases.
>  
> DS2, DS3 and DS4) and let's say there we have created some files on 
> the filesystem that follows that configuration. 
>  
> - Let's say, I restripe this device configuration by adding one more 
> DS. So, the configuration now becomes 1 MDS and 5 DS's (DS1, DS2, DS3, 
> DS4 and DS5). All the new files that I create from now on, will be 
> created with new configuration.
>  
> - So, essentially we have a filesystem when served over pNFS could 
> have multiple device mappings. In that respect, the layout of files 
> created via first and the subsequent configuration will be different.
>  
> Now, let's say if a device address on DS5 fails over to some other 
> place (another node in the cluster), I'd like to recall the layouts 
> for that deviceid only.

Why? If it fails over, doesn't that imply that the change in DSes should 
be transparent
to the client? I.e., the node that was failed over to should be 
accepting network
requests to the IP of the down node.

Or are you stating that for some clustered file systems, the box acting 
as the DS
doesn't matter because they all get to the same storage?

> Recalling layouts for that FSID as a whole does not make sense in this 
> case. I believe that we should provide layout recall by deviceid as an 
> option as well for such use cases.
>

Conceivably DS5 could belong to multiple device ids. I.e., before you 
added it to
be packaged with DS1, DS2, DS3, and DS4, it could have been in a package 
with
DS6 and DS7.

So how do you find all of the files that reside on DS5 to get all of the 
possible
device ids that you want to recall?

Also, nothing mandates that a DS only has one dataset being used by the MDS.
Our implementation certainly doesn't have that limitation.

Is there much difference in recalling layouts for N device ids than for 
1? Nope.

> An obvious way of looking at this situation is to have server initiate 
> a CB_NOTIFY_DEVICEID (DELETE) and let the clients issue a 
> GETDEVICEINFO, but it still doesn't ensure cover the clients which 
> don't support the device notification. NOTE: servers MUST implement 
> device notification, but clients don't have to.
>  
> It would be nice to recall layouts by deviceid, so that cases like 
> above can be optimized. If there is enough interest, we could add it 
> to NFSv4.2 work item list. Let me know, what you guys think.
>  
> - Pranoop
> ------------------------------------------------------------------------
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 20 Aug 2009 07:44:48 +0100

Hi Pranoop,

It seems like you're describing a rather narrow use case.
I think we should encourage clients to implement
CB_NOTIFYDEVICEID rather than provide an alternative to it.

One question, after adding the new DS do you keep the old
device so that new layoutget's to files that were created
over the old device info will still refer to it.  I.e.
do you persistently keep old device info's or will you
use the current device when giving out new layouts
to files striped over the old device?

Benny

---

From: Tom Haynes <[email-masked]>
Date: Thu, 20 Aug 2009 01:53:41 -0500

> to files striped over the old device?

You have to give the old "device", right?

Or you get data corruption.

Whether you persistently store the device infos or the
layouts is your choice.

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Wed, 19 Aug 2009 23:58:14 -0700

> -----Original Message-----
> From: Tom Haynes [mailto:[email-masked]] 
> Sent: Wednesday, August 19, 2009 11:37 PM
> To: Erasani, Pranoop
> Cc: [email-masked]
> Subject: Re: [nfsv4] Layout recall by deviceid
> 
> Erasani, Pranoop wrote:
> > Hi Folks.
> >  
> Why? If it fails over, doesn't that imply that the change in 
> DSes should be transparent to the client? I.e., the node that 
> was failed over to should be accepting network requests to 
> the IP of the down node.
> 
> Or are you stating that for some clustered file systems, the 
> box acting as the DS doesn't matter because they all get to 
> the same storage?

It could be transperant. But, I don't want to serve stripe data over 
cluster path. I want to give a new layout with an optimal path (which
 is network and storage residing on the same physical hardware).

> Conceivably DS5 could belong to multiple device ids. I.e., 
> before you added it to be packaged with DS1, DS2, DS3, and 
> DS4, it could have been in a package with
> DS6 and DS7.
> 
> So how do you find all of the files that reside on DS5 to get 
> all of the possible device ids that you want to recall?

I have a cluster that can give me all the information. How do I
find it isn't something we should care.

> Our implementation certainly doesn't have that limitation.

Our server doesn't have that limitation either.

> Is there much difference in recalling layouts for N device 
> ids than for 1? Nope.

I'm not sure I follow that comment.

- Pranoop

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 00:06:17 -0700

Hi Benny, 

Thanks for the reply. Replies inline...

> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Wednesday, August 19, 2009 11:45 PM
> To: Erasani, Pranoop
> Cc: [email-masked]

Do you expect restriping to be a narrow use case? What is
Your view or definition of restriping.

> CB_NOTIFYDEVICEID rather than provide an alternative to it.

How do we do this? I suspect we can start by mandating the
device notifications itself. ;) Since we don't want to do that,
you are suggesting that servers don't need that option to
recall layouts by deviceid.

Fundamentally, I think spec is incorrect to not provide that
option. I see references to the fact that a filesystem (fsid)
could have multiple device mappings. Saying that, and not
Letting servers recall on per-deviceid seems to be quite
broken. I don't believe, we should expect servers to workaround
such things, especially when we have a chance to fix it.

> do you persistently keep old device info's or will you use 
> the current device when giving out new layouts to files 
> striped over the old device?

I persistently keep the old device information. As I mention
below, I also have the existing files follow the old device
information and new files (creating after restriping event)
follow the new device information.

- Pranoop

> > for that deviceid only. Recalling layouts for that FSID as a whole 
> > does not make sense in this case. I believe that we should provide 
> > layout recall by deviceid as an option as well for such use cases.
> >  
> > to

---

From: Tom Haynes <[email-masked]>
Date: Thu, 20 Aug 2009 02:10:51 -0500

Erasani, Pranoop wrote:
>>> received. I'm not sure of the exact reasons. I have a 
>>>       
>> server use case 
>>     
>>> in mind that is worth exploring. I believe layout recalls 
>>>       
>> by deviceid 
>>     
>>> - Let's say I have a striping configuration of 1 MDS and 4 
>>>       
>> DS's (DS1, 
>>     
>>> - Let's say, I restripe this device configuration by adding 
>>>       
>> one more 
>>     
>>> DS. So, the configuration now becomes 1 MDS and 5 DS's 
>>>       
>> (DS1, DS2, DS3,
>>     
>>> created via first and the subsequent configuration will be 
>>>       
>> different.
>>     
>>>  

And that sounds exactly like something CB_NOTIFYDEVICEID is
designed for.

The clients which support it will get an optimal path to the new node
and those which do not support CB_NOTIFYDEVICEID will take
a hit.

> find it isn't something we should care.

My point is that you present a simplified use case and I doubt it
is that easy.

The real question is how complex does the scenario get before we
either throw up our hands to help clients do as Benny suggested or
we say recall by fsid is much faster?

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 00:15:49 -0700

> The clients which support it will get an optimal path to the 
> new node and those which do not support CB_NOTIFYDEVICEID 
> will take a hit.

Ok. Fair enough. But, I don't want to serve because I'm wasting my
precious
cluster resources also. No?

It benefits both servers and clients. ;)

> before you 
> >> added it to be packaged with DS1, DS2, DS3, and DS4, it could have 
> >> been in a package with
> >> DS6 and DS7.
> >>
> >> So how do you find all of the files that reside on DS5 to 
> get all of 
> >> the possible device ids that you want to recall?
> >>     
> >
> > I have a cluster that can give me all the information. How 
> do I find 
> > it isn't something we should care.
> >
> >   
> 
> My point is that you present a simplified use case and I 
> doubt it is that easy.
> 
> The real question is how complex does the scenario get before 
> we either throw up our hands to help clients do as Benny 
> suggested or we say recall by fsid is much faster?

Wouldn't client be slower given it is recalling more layouts than it
should?
Or maybe I didn't get the faster part.

- Pranoop

---

From: Benny Halevy <[email-masked]>
Date: Thu, 20 Aug 2009 08:27:43 +0100

> Your view or definition of restriping.

No, it's not restriping by itself that's narrow, but
rather one data server failure effecting only one
deviceid and where the client does not support
the deviceid callback.

> device notifications itself. ;) Since we don't want to do that,

We may want to reconsider that in v4.2 if we see the need
and change CB_NOTIFY_DEVICEID to be REQUIRED.

> follow the new device information.

So a failed data server could belong to several deviceid's.
Over time, the older the data server is, it will belong to more
devices. I.e. on average you'll have to recall half the deviceid's!

This is potentially less than recalling the individual layouts
but this is certainly more complex than recalling all layouts
by fsid.

Benny

>> one more DS.

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 00:48:25 -0700

> No, it's not restriping by itself that's narrow, but rather 
> one data server failure effecting only one deviceid and where 
> the client does not support the deviceid callback.

In a cluster, you could put your striped filesystem only on
few nodes. Our cluster accomadates creating multiple striped
file systems, whose constituents could be placed on the 
different nodes in the cluster and will/could be moved around
as the customer load matters/changes. Thus, it will give you 
an effect that a data server failure effecting only one incarnation
of the filesystem that was striped over a period of time i.e
To put it differently, I expect customers to add more nodes to the
cluster and in the process may want to add more constituents to
the existing striped file system(s). As that happens, we have
just one filesystem serving multiple sets of files with different
data servers. So, it's quite normal that one data server effect
only one deviceid generated for that filesystem. It may effect
Other file system, but that's not what I'm interested in this for
The purpose of this discussion.

> >> I think we should encourage clients to implement CB_NOTIFYDEVICEID 
> >> rather than provide an alternative to it.
> > 
> > How do we do this? I suspect we can start by mandating the device 
> > notifications itself. ;) Since we don't want to do that,
> 
> We may want to reconsider that in v4.2 if we see the need and 
> change CB_NOTIFY_DEVICEID to be REQUIRED.

Ok. Good. But, it still doesn't give me flexibility to invalidate
selective deviceid's belonging to a particular filesystem. No?

> device so 
> >> that new layoutget's to files that were created over the 
> old device 
> >> info will still refer to it.  I.e.
> >> do you persistently keep old device info's or will you use the 
> >> current device when giving out new layouts to files 
> striped over the 
> >> old device?
> > 
> > I persistently keep the old device information. As I 
> mention below, I 
> > also have the existing files follow the old device 
> information and new 
> > files (creating after restriping event) follow the new device 
> > information.
> 
> Over time, the older the data server is, it will belong to 
> more devices. I.e. on average you'll have to recall half the 
> deviceid's!
> This is potentially less than recalling the individual 
> layouts but this is certainly more complex than recalling all 
> layouts by fsid.

Ok. I see your point. But, why would you want client to return more
layouts and go through the layout get path again, when it is not needed?
Servers should be perfectly fine recalling based on deviceid, as it
looks
at different deviceid's as different configurations and it could easily
detect failure of DS's belonging to a particular deviceid.

I guess I'm still not convinced, that you don't need this. Your argument
sounds fine to me, but law of averages doesn't seem like a compelling
argument to not provide this *option*.

So, what are client vendors hesitate about? Is it the mappings you have
to maintain between deviceid's and layouts? Or something else?

- Pranoop

> ---------------------------------------------------------------------
> >> -
> >>> --
> >>>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 20 Aug 2009 09:40:09 +0100

> The purpose of this discussion.

You got me lost in the details... :-/
It'd help if you provide a concrete example.

> selective deviceid's belonging to a particular filesystem. No?

The deviceid namespace is per server, per layout-type.
Not per fsid. So a particular deviceid always belong
to a particular filesystem.

> argument to not provide this *option*.

I'm not against having an efficient way to deal with the scenario you
describe.  What I'm saying is that there is already a way to deal with it,
using CB_NOTIFY_DEVICEID and we should not add more complexity to the
protocol by providing yet another way to do the same thing.

Please read the spec. for how CB_NOTIFY_DEVICEID works and in
particular the race condition that are to be considered.
Note that adding a CB_LAYOUTRECALL for a deviceid may have to
deal with similar races.  We already defined a pretty elaborate
stateid-based mechanism to help resolve recall/get/return races
for individual layouts and we chose to simplify the recall by
fsid and recall-all case by allowing a batched way to recall
all layouts in this scope and that has no stateid's.

> to maintain between deviceid's and layouts? Or something else?

The mapping between a deviceid and layouts that use it is
actually useful also for service CB_NOTIFY_DEVICEID.
If the client implementation wants to be lazy about it
it can invalidate all layouts, and that is true also
about a possible CB_LAYOUT_RECALL for a deviceid.
Making the operation mandatory does not force the client
implementation to do the most efficient thing...

To summarize, I think it is important that you first try
to implement your solution using the available mechanisms
and see how they work for you.  If they indeed are valuable
both to the server and to the client than there will be
incentive for client implementors to actually implement them
even though they are optional to implement.  To take it
to the extreme, you are basically saying that if an optional
feature is so important I think it should have been defined
as mandatory, I'll just go and add a new, mandatory, feature
that does the same thing so that clients will have to
implement it.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 20 Aug 2009 10:26:59 -0400

> It seems like you're describing a rather narrow 

It's a matter of perspective.

> CB_NOTIFYDEVICEID rather than provide an alternative 

OK, but saying that Pranoop's case, which could be addressed by using
CB_NOTIFYDEVICEID is narrow (and thus not important) does not sound like
a way to encourage people to implement it.

What exactly is the reason for making this optional?  What am I missing
here?

1) CB_NOTIFY_DEVICEID would solve this problem if mandatory-to-implement
(T/F).

2) But if it optional.  Why?

3) Assume for the moment we need something to address this.

4) If we added recall-by-deviceid, to solve this issue, it would have to
be mandatory, or else we have the same problem.

5) Ergo, we have to add something mandatory.

6) Why not just make CB_NOTIFY_DEVICEID mandatory in v4.2 and avoid
adding anything new. 

I guess I'm asking whether, if "we should encourage clients to implement
CB_NOTIFYDEVICEID", the best way to do that might be to say, "Clients
MUST implement the CB_NOTIFY_DEVICEID callback in v4.2 implementations."

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, August 20, 2009 2:45 AM
To: Erasani, Pranoop
Cc: [email-masked]
Subject: Re: [nfsv4] Layout recall by deviceid

Hi Pranoop,

One question, after adding the new DS do you keep the old device so that
new layoutget's to files that were created over the old device info will
still refer to it.  I.e.
do you persistently keep old device info's or will you use the current
device when giving out new layouts to files striped over the old device?

Benny

> - Let's say, I restripe this device configuration by adding one more
DS.

---

From: Tom Haynes <[email-masked]>
Date: Thu, 20 Aug 2009 10:45:03 -0500

> MUST implement the CB_NOTIFY_DEVICEID callback in v4.2 implementations."

Yes - the use case Pranoop presented reads exactly like CB_NOTIFY_DEVICEID.

Make it mandatory.

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 09:08:44 -0700

> Sent: Thursday, August 20, 2009 8:45 AM
> To: Noveck, Dave
> Cc: Benny Halevy; Erasani, Pranoop; [email-masked]
> > I guess I'm asking whether, if "we should encourage clients to 
> > implement CB_NOTIFYDEVICEID", the best way to do that might 
> be to say, 
> > "Clients MUST implement the CB_NOTIFY_DEVICEID callback in 
> v4.2 implementations."
> >   
> 
> 

Similar, but I wouldn't jump to conclusion that says "exactly". I
suspect we'd need some more clarifications along with that.

> Make it mandatory.

That's a good start.

But, what should the server do to the layouts that is already granted to
the client on the deviceid that is being notified for a change? 

- Should it mark them revoked?

- Should it wait for the client to recall those layouts before it serves
the new GETDEVICEINFO?

And what should clients do, when it gets a CB_NOTIFY_DEVICEID?

- MUST clients return the layouts on receiving the notification?

- Should it return them before responding to CB_NOTIFY_DEVICEID.

- Pranoop

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 11:24:51 -0700

> It'd help if you provide a concrete example.

Let me type one up and send it in a day or two.

> CB_NOTIFYDEVICEID 
> Not per fsid. So a particular deviceid always belong to a 
> particular filesystem.

We voilently agree. But, in a cluster an fsid is shared/accessible
 across nodes, which in turn could be served by different IP's 
configured on different nodes and those IP's being considered as 
different servers from client perspective. I think this will be
clear when I type the example that I mention above.

> >> This is potentially less than recalling the individual layouts but 
> >> this is certainly more complex than recalling all layouts by fsid.
> > 
> > looks at different deviceid's as different configurations 
> and it could 
> > easily detect failure of DS's belonging to a particular deviceid.
> > 
> > I guess I'm still not convinced, that you don't need this. Your 
> > argument sounds fine to me, but law of averages doesn't seem like a 
> > compelling argument to not provide this *option*.
> > 
> 
> I'm not against having an efficient way to deal with the 
> scenario you describe.  What I'm saying is that there is 
> already a way to deal with it, using CB_NOTIFY_DEVICEID and 
> we should not add more complexity to the protocol by 
> providing yet another way to do the same thing.

In it's current state, the CB_NOTIFY_DEVICEID, cares very less
to describe how clients and servers should behave w.r.t layouts
that exist already, when a server sends a CB_NOTIFY_DEVICEID.
How about elaborating those cases? Also, look at the latest e-mail
I sent to Tom.

> deal with similar races.  We already defined a pretty 
> elaborate stateid-based mechanism to help resolve 
> recall/get/return races for individual layouts and we chose 
> to simplify the recall by fsid and recall-all case by 
> allowing a batched way to recall all layouts in this scope 
> and that has no stateid's.

You are telling me what exists. I'm telling you that what exists
in NFSv4.1 is not optimzed to address the cases I described.

Whether we'l end up saying that CB_NOTIFY_DEVICEID is mandatory
and some clarifications around how clients and servers should behave,
OR we'l end up inventing a whole new way is a secondary question.

> > So, what are client vendors hesitate about? Is it the mappings you 
> > have to maintain between deviceid's and layouts? Or something else?
> 
> If the client implementation wants to be lazy about it it can 
> invalidate all layouts, and that is true also about a 
> possible CB_LAYOUT_RECALL for a deviceid.
> To summarize, I think it is important that you first try to 
> implement your solution using the available mechanisms and 
> see how they work for you.  

No one is arguing against adjusting to NFSv4.1 reality. You see,
we shoudlnt' wait for implementors to implement everything
before we see the problem. If there is a chance to fix this, it
is now (thinking v4.2). I don't think we should wait for another
decade before we think about a problem here. Personally, I'm surprised
that WG doesn't consider this as an issue.

> If they indeed are valuable both 
> to the server and to the client than there will be incentive 
> for client implementors to actually implement them even 
> though they are optional to implement.  To take it to the 
> extreme, you are basically saying that if an optional feature 
> is so important I think it should have been defined as 
> mandatory, I'll just go and add a new, mandatory, feature 
> that does the same thing so that clients will have to implement it.

So, don't make CB_NOTIFY_DEVICEID mandatory. But, provide an option
to CB_LAYOUTRECALL to recall by deviceid's, so that both clients
and servers' life becomes easier.

Seems like I'm flexible on both fronts. And I'm still not convinced
why you are reluctant in seeing this as a problem/optimization/need.

- Pranoop

---

From: Benny Halevy <[email-masked]>
Date: Fri, 21 Aug 2009 12:49:21 +0300

> suspect we'd need some more clarifications along with that.

So I actually went back to the spec. to refresh my memory.
This is how CB_NOTIFY_DEVICEID with NOTIFY4_DEVICEID_DELETE is specified
as follows:

20.12. Operation 14: CB_NOTIFY_DEVICEID - Notify Client of Device ID Changes
...
   NOTIFY4_DEVICEID_DELETE
      Deletes a device ID from the mappings.  This notification MUST NOT
      be sent if the client has a layout that refers to the device ID.
      In other words if the server is sending a delete device ID
      notification, one of the following is true for layouts associated
      with the layout type:

      *  The client never had a layout referring to that device ID.

      *  The client has returned all layouts referring to that device
         ID.

      *  The server has revoked all layouts referring to that device ID.

Houston (or Austin, actually ;-), do we have a problem? :-)
In the revocation case, is the server allowed to send NOTIFY4_DEVICEID_DELETE
while the client holds the revoked layouts that refer to the deleted deviceid?

In any case, since revoking the layout also requires fencing the client
off so that it cannot use the revoked layout anymore the spirit of this
is to first attempt to recall the layouts gracefully and revocation is
the last resort.

>> Make it mandatory.

Agreed.

> - Should it mark them revoked?

yes.

> - Should it wait for the client to recall those layouts before it serves

The new deviceid is valid regardless. Why should the server wait?

> - Should it return them before responding to CB_NOTIFY_DEVICEID.

CB_NOTIFY_DEVICEID is not recalling the layout.
Since the layout is already revoked I believe that its layout stateid
is not valid anymore so there's no need to return it.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Aug 2009 15:40:29 -0400

> Houston (or Austin, actually ;-), do we have a 
> problem? :-) In the revocation case, is the server 
> allowed to send NOTIFY4_DEVICEID_DELETE while the 
> client holds the revoked layouts that refer to the 
> deleted deviceid? 

I don't think the client can hold revoked layouts.  Once they are
revoked, they cannot be held.  He may still have the stateids marked as
revoked and need to free them, but that isn't the same thing as holding
a layout.  Once layouts are revoked, they are not held.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, August 21, 2009 5:49 AM
To: Erasani, Pranoop; Tom Haynes; Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Layout recall by deviceid

20.12. Operation 14: CB_NOTIFY_DEVICEID - Notify Client of Device ID
Changes ...
   NOTIFY4_DEVICEID_DELETE

Houston (or Austin, actually ;-), do we have a problem? :-) In the
revocation case, is the server allowed to send NOTIFY4_DEVICEID_DELETE
while the client holds the revoked layouts that refer to the deleted
deviceid?

Agreed.

> That's a good start.
> 
> But, what should the server do to the layouts that is already granted 
> to the client on the deviceid that is being notified for a change?
> 

yes.

> 
> - Should it wait for the client to recall those layouts before it 
> serves the new GETDEVICEINFO?

CB_NOTIFY_DEVICEID is not recalling the layout.
Since the layout is already revoked I believe that its layout stateid is
not valid anymore so there's no need to return it.

Benny

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 24 Aug 2009 18:01:32 -0700

> while the client holds the revoked layouts that refer to the deleted deviceid?
I'm picking up on this late, but in the file layout case, adding or 
removing a data server may simply result in a NOTIFY_DEVICEID4_CHANGE, 
since you simply want to change the set of DSs that are associated with 
a particular device id.  This applies to the restripe case I think 
doesn' it?  So you can change the DSs a client is talking to while they 
hold layouts referring to those DSs.  This change can be immediate or not.

Another point is that the server should know the device<->fsid mappings, 
and can therefore (and slightly less efficiently) recall everything it 
needs to by way of multiple layoutrecalls using stateids/fsids.
Dean

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 27 Aug 2009 14:07:42 -0700

On Fri, August 21, 2009 2:49 am, Benny Halevy wrote:

> Changes

Vancouver actually. Not that I agree "we" have a problem. :-)

http://www.ietf.org/mail-archive/web/nfsv4/current/msg05177.html

> while the client holds the revoked layouts that refer to the deleted

Yes: the part the spec you quoted says:

[...] if the server is sending a delete device ID
notification, one of the following is true for layouts associated
       with the layout type:
[..]

What isn't clear?

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Benny Halevy <[email-masked]>
Date: Fri, 28 Aug 2009 10:17:17 +0300

>        *  The server has revoked all layouts referring to that device ID.

OK. That particular part is clear.

The problem I meant is that the server is not allowed to use
CB_NOTIFY_DEVICEID with NOTIFY4_DEVICEID_DELETE while the client
holds valid layouts referring to the deleted deviceid.
The server must recall or revoke the layouts first.
I consider revocation the last resort and I expect the server
to recall the layouts first.  This brings us back to square
one with regards to Pranoop's original issue.

Benny

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sat, 29 Aug 2009 12:37:52 -0700

> >> In the revocation case, is the server allowed to send 
> >> NOTIFY4_DEVICEID_DELETE while the client holds the revoked layouts 
> >> that refer to the deleted deviceid?
> > 
> notification, one of 
> > the following is true for layouts associated
> >        with the layout type:
> > [..]
> > 
> CB_NOTIFY_DEVICEID with NOTIFY4_DEVICEID_DELETE while the 
> client holds valid layouts referring to the deleted deviceid.

Yes, Precisely. If revocation is so preferable, then why do we even have
recalls?

- Pranoop

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 29 Aug 2009 18:13:14 -0400

Benny wrote:
> I consider revocation the last resort and I expect the 
> serever to recall the layouts first.  This brings us 
> back to square one with regards to Pranoop's original 
> issue.

Does "This" in "This brings us back to square one" refer to your
expectation that the server will recall layouts first? :-)  Is there a
good reason to expect that in this particular case?

Pranoop wrote:
> Yes, Precisely. If revocation is so preferable, then 
> why do we even have recalls? 

Normally we have recalls because that is the only way the client is
going to find out about the state chhange.  If the client is going to
find out about the revocation through the notification, is there a
reason to do the recall first?

What if we said?

> [other cases]
>
>   * The server has revoked all layouts referring to 
>     that device ID.  The server may revoke these layouts
>     without an initial attempt to recall them, since the
>     client, upon receiving the notification, is informed
>     that any layouts referring to that device ID must have 
>     been revoked by the server.

-----Original Message-----
From: Erasani, Pranoop 
Sent: Saturday, August 29, 2009 3:38 PM
To: Benny Halevy; Mike Eisler
Cc: Tom Haynes; Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Layout recall by deviceid

> CB_NOTIFY_DEVICEID with NOTIFY4_DEVICEID_DELETE while the client holds

> valid layouts referring to the deleted deviceid.
> I consider revocation the last resort and I expect the server to 
> recall the layouts first.  This brings us back to square one with 
> regards to Pranoop's original issue.

- Pranoop

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sat, 29 Aug 2009 20:28:36 -0700

> From: Noveck, Dave 
> Sent: Saturday, August 29, 2009 3:13 PM
> To: Erasani, Pranoop; 'Benny Halevy'; 'Mike Eisler'
> Cc: 'Tom Haynes'; [email-masked]'
> > I consider revocation the last resort and I expect the serever to 
> Does "This" in "This brings us back to square one" refer to 
> your expectation that the server will recall layouts first? 
> :-)  Is there a good reason to expect that in this particular case?
> 
> Pranoop wrote:
> > Yes, Precisely. If revocation is so preferable, then why do we even 
> > have recalls?
> 
> Normally we have recalls because that is the only way the 
> client is going to find out about the state chhange.  If the 
> client is going to find out about the revocation through the 
> notification, is there a reason to do the recall first?

Unerstood. 

You know, that basically tells me that we have two mechanisms:

    recalls vs notifications

Working similarly, and the second one (i.e notifications) is a 
Forcible version of "recall" (or let's call it revocation).

We could have just implemented CB_NOTIFY_DEVICEID by enhancing
CB_LAYOUTRECALL with deviceid as argument.

> >     been revoked by the server.

This text sounds promising to me.

- Pranoop

> > that device ID.
> client holds 
> Yes, Precisely. If revocation is so preferable, then why do 
> we even have recalls?
> 
> - Pranoop
> 
> > 
> > Benny
> > 
>

---

From: Benny Halevy <[email-masked]>
Date: Sun, 30 Aug 2009 09:29:27 +0300

> good reason to expect that in this particular case?

I presume that revocation may require extra work from the server
to communicate with the data servers to invalidate the revoked
layouts and that recalling them will save this work and therefore
be cheaper.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Sun, 30 Aug 2009 10:05:53 +0300

> CB_LAYOUTRECALL with deviceid as argument.

Note that LAYOUTRECALL4_FSID and LAYOUTRECALL4_ALL also recall
the respective deviceids and in practice they achieve this effect
just with a bigger hammer.

In theory, we can define a LAYOUTRECALL4_DEVICEID case that will
recall all layouts that refer to the specified deviceid and
have a flag to indicate whether the deviceid is deleted
(re-use clora_changed or add a flag for the new case).
In the revocation case, the server should just use CB_NOTIFY_DEVICEID
w/ NOTIFY_DEVICEID4_DELETE.

Benny

> This text sounds promising to me.

Yeah, it does clarify this case.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 31 Aug 2009 11:54:29 -0400

> We could have just implemented CB_NOTIFY_DEVICEID 
> by enhancing CB_LAYOUTRECALL with deviceid as argument.

I think we could have for the delete case.  I think the modify case
would have remained. 

-----Original Message-----
From: Erasani, Pranoop 
Sent: Saturday, August 29, 2009 11:29 PM
To: Noveck, Dave; 'Benny Halevy'; 'Mike Eisler'
Cc: 'Tom Haynes'; [email-masked]'
Subject: RE: [nfsv4] Layout recall by deviceid

> expectation that the server will recall layouts first?

Unerstood. 

    recalls vs notifications

Working similarly, and the second one (i.e notifications) is a Forcible
version of "recall" (or let's call it revocation).

- Pranoop