Subject: Re: [nfsv4] Device stateids

From: Benny Halevy <[email-masked]>
Date: Mon, 29 Oct 2007 17:45:07 +0200

During the most recent bakeathon concern has been raised
about the global device ID namespace and Marc Eshel from
IBM requested that we retain the ability to recall
devices per-fsid.  To satisfy this requirement as well
as the requirement for a simple global device ID namespace
the updated proposal below introduces a device-set identifier
(devsetid) that qualifies the device ID.

This provides for a more flexible deviceid namespace topology that
can cover either a global deviceid namespace, per-fsid namespace, or
anything in between (e.g. when several filesystems reside on (and are
confined to) a set of devices, like in EMC or Panasas' cases).

The proposal also introduces new operations to manage device mapping
state rather than unnecessarily overloading existing operations.

Plus, I simplified the proposal by not requiring any new sequence flags
as the server should use the proposed callback operation to recall
device mappings and use the SEQ4_STATUS_RECALLABLE_STATE_REVOKED and 
SEQ4_STATUS_CB_PATH_DOWN sequence flags as appropriate the same it
may use them for other recallable state objects (locks and layouts).

One last change is providing for a "wildcard" value (0) for device ID
that specifies all the device IDs in a devset.  This means that
the server MUST NOT return a zero device ID in GETDEVICELIST
or in any layout and it must reject it as invalid in GETDEVICEINFO.
I guess that this won't be an issue for anyone; otherwise, please
holler.

Benny

--
1. Introduction

The NFS client maintains a mapping of device IDs contained
in a layout to the corresponding storage device address.
Device IDs and filesystem IDs are associated with device set
IDs such that each fsid is associated with a single devsetid
as indicated by a new, recommended filesystem attribute, and
each deviceid is associated with a single devsetid.

The state of the mapping is represented by a recallable device
stateid (devstateid). There is at most one devstateid per
{ client ID, layout type, devsetid } nexus.

A recommended filesystem attribute - devsetid is introduced.
All deviceids associated with any file in the filesystem are
associated with this devsetid. 
If the server does not support device sets, it can omit this attribute,
making the deviceid namespace global and implicitly associating all
deviceids with the default devset (having the implicit devsetid
ID of 0)

2. Getting device information

GETDEVICEINFO
SYNOPSIS: devstateid, devsetid, deviceid, layout_type, maxcount, notify_ask ->
	devstateid, notify_ack, device_addr

GETDEVICELIST
SYNOPSIS: devstateid, devsetid, layout_type, maxcount, cookie, cookieverf, notify_ask ->
	devstateid, notify_ack, cookie, cookieverf, device info list<>

GETDEVICEINFO and GETDEVICELIST each take the a devstateid,
a devsetid, and the layout type, and return a non-zero devstateid.
In case the client holds a valid devstateid it MUST use it for
subsequent GETDEVICEINFO and GETDEVICELIST that update the device ID
mappings.  The devstateid argument must be set to zero whenever the
client has no valid devstateid (e.g. after reboot or after
the devstateid was previously returned via DEVICERETURN).

GETDEVICEINFO and GETDEVICELIST also take a boolean flag
optionally asking for device notifications and return a
corresponding flag confirming the notification.

3. Returning device IDs

DEVICERETURN
SYNOPSIS: devstateid, devsetid, deviceid -> devstateid

DEVICERETURN returns a device ID mapping.  When called, this
client MUST guarantee not to make further use of that device ID mapping.
If the deviceid argument is set to zero, all device ID mappings
in the referred devset are returned.  The server updates the devstateid
and returns the updated value.

4. Sequencing rules

Device stateids follow the sequencing and use rules of all other
stateid types.  The devstateid is invalidated when all devices
associated with it are returned via DEVICERETURN. The client can
then "forget" about that devstateid and use a value of zero for
the devstateid for any subsequent operation that takes a devstateid
argument and allows the uninitialized value.

5. Recalling device IDs

CB_DEVICERECALL
SYNOPSIS: devstateid, devsetid, deviceid, deleted ->

CB_DEVICERECALL takes a devstateid, devsetid, and a deviceid to notify
the client it needs to return the specified deviceid (when non-zero) or all device
IDs in the devsetid when the deviceid is set to zero, by using a respective
DEVICERETURN. CB_DEVICERECALL does not change the device stateid.

All layouts using the affected device IDs remain in force, but the
server SHOULD fence the client from accessing the affected
devices until the client has returned the recalled device IDs.

When the "deleted" boolean flag is set to false the client MAY use
GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if it
has valid layouts the refer to the changed deviceid.  If the "deleted" flag
is set to true the specified deviceid will not be reused by the server and
all outstanding layouts referring to it are expected to be recalled via
CB_LAYOUTRECALL; hence the client SHOULD NOT attempt to call GETDEVICEINFO or
GETDEVICELIST for the specified deviceid. However, if it does do so the server
MUST return a NFS4ERR_INVAL error.

6. Device notification

CB_DEVICENOTIFY
SYNOPSIS: devstateid, devsetid, deviceid ->

If the server supports CB_DEVICENOTIFY for devices and the client has
requested device id notifications via GETDEVICEINFO or GETDEVICELIST
the server can inform the client of additional deviceids that were
not previously returned by GETDEVICELIST.

The purpose of this operation is to reduce the latency for getting
new device mappings.  Rather than waiting until the client accesses
a file that uses the new device ID, issue a LAYOUTGET, and then a
GETDEVICEINFO on the new device, the server can notify the client
of any new device ID in advance and have the client get the device
address mapping by calling GETDEVICEINFO at its convenience.

CB_DEVICENOTIFY takes a device stateid, a device set ID
and a device ID in the device set for a device ID that was not
returned in a previous series GETDEVICELIST calls (where the
series ended in gdlr_eof == TRUE).  The client calls GETDEVICEINFO
or GETDEVICELIST to determine the mappings for the new device ID.
CB_DEVICENOTIFY does not change the device stateid.

If the client has already issued a GETDEVICEINFO for the specified deviceid
then it SHOULD just ignore the device id notification.
Updating an existing device ID mapping should be done using a sequence of
CB_DEVICERECALL, DEVICERETURN, and GETDEVICEINFO (or GETDEVICELIST).

7. SEQUENCE flags

In addition to informing the client of mapping changes via
CB_DEVICERECALL and CB_DEVICENOTIFY, the server MUST use
the SEQ4_STATUS_RECALLABLE_STATE_REVOKED SEQUENCE flag
to indicate that one or more device IDs have been
revoked without expiration of the lease period, due to the
client's failure to return them when recalled. This status bit
remains set on all SEQUENCE replies until the loss of all such
device ID mappings has been acknowledged by use of DEVICERETURN.

On Aug. 09, 2007, 10:22 +0300, Benny Halevy <[email-masked]> wrote:
> Robert Gordon wrote:
>> In general i think this is good.
>>
>> I've started to think that the SEQUENCE status flags could just be
>> two bits.
>>
>> SEQ4_STATUS_DEVID_DELETED_ALL
>> 	This flag stays on until the client issues a DELEGRETURN
>> 	using the devstateid.
>>
>> SEQ_STATUS_DEVID_CHANGED
>> 	This flag stay on until a GETDEVICELIST or GETDEVICEINFO
>> 	(if using CB_NOTIFY) gets the affected mappings.
> 
> The DEVID_DELETED cases should be covered by CHANGED as the client will
> have to revalidate all the device mappings anyway to see what actually changed
> and should either not see the deleted deviceid returned by GETDEVICELIST
> or get an error from GETDEVICEINFO for that deviceid.
> 
> The DEVID_ADDED case is covered if the client is doing GETDEVICELIST
> and the server supports it.  If the server does not support GETDEVICELIST
> is must not set SEQ_STATUS_DEVID_CHANGED for added devices as the client
> has no way to get the new device ID.  These will be discovered only
> when the client will get new layouts referring to the new devices
> (which is perfectly fine)
> 
>> Robert.
>>
>> On Aug 8, 2007, at 10:58 AM, Benny Halevy wrote:
>>
>>> The NFS client maintains a mapping of device ids contained
>>> in a layout, to the corresponding storage device address.
>>> The state of the mapping is represented by a single
>>> recallable device stateid (devstateid). There is at most
>>> one devstateid per { client ID, layout type } pair.
>>>
>>> GETDEVICEINFO and GETDEVICELIST each take the layout type and
>>> an optional devstateid and return the devstateid.  In case
>>> the client holds a valid devstateid it MUST use it for subsequent
>>> GETDEVICEINFO and GETDEVICELIST that update the device ID
>>> mappings.  The devstateid argument is invalid whenever the
>>> client has no valid devstateid (e.g. after reboot or after
>>> the devstateid was previously returned via DELEGRETURN).
>>> GETDEVICEINFO or GETDEVICELIST that operate on a device stateid
>>> that was already recalled via CB_RECALL MUST be rejected by the
>>> server with NFS4ERR_BADSTATEID.
>>>
>>> GETDEVICEINFO and GETDEVICELIST also take a boolean flag
>>> optionally asking for device notifications and return a
>>> corresponding flag confirming the notification.
>>>
>>> Device stateids follow the sequencing and use rules of all other
>>> stateid types.  The mappings are invalidated when the devstateid is
>>> returned via CB_DELEGRETURN or explicitly deleted via CB_NOTIFY.
>>>
>>> All layouts using the device IDs remain in force, but the
>>> server MUST fence the client from accessing the affected
>>> devices until the client has obtained the re-mapped device
>>> IDs via GETDEVICEINFO or GETDEVICELIST.
>>>
>>> CB_RECALL takes a devstateid to notify the client it needs
>>> to return all device IDs identified by the devstateid.  The
>>> filehandle argument has no use in this case.  It does not change
>>> the device stateid.
>>>
>>> DELEGRETURN takes a devstateid to return all device IDs associated
>>> with the devstateid. The current filehandle has no use in this case.
>>>
>>> If the server supports CB_NOTIFY for devices and the Client has
>>> requested device id notifications via GETDEVICEINFO
>>> or GETDEVICELIST; The server can inform the client
>>> of a delete, add, or change mapping.
>>>
>>> CB_NOTIFY takes a device stateid qualifying a device ID
>>> for the following device notifications:
>>>
>>> NOTIFY4_DELETE_DEVICE_ID
>>> 	Deletes all mappings using the specified device ID.
>>> 	The server MUST stop using the device ID. Any
>>> 	layouts remain in force, but aren't useful until
>>> 	new mappings are available. The server fences the
>>> 	client from the affected data server until all
>>> 	layouts with the device ID are returned, or the
>>> 	client obtains new mappings. The server provides
>>> 	a hint as to whether new mappings will arrive and
>>> 	if so how long.
>>>
>>> NOTIFY4_ADD_DEVICE_ID
>>> 	Adds a device ID to address mapping. This is
>>> 	for a device ID the that was not returned in a
>>> 	previous series GETDEVICELIST calls (where the
>>> 	series ended in gdlr_eof == TRUE).  The client
>>> 	calls GETDEVICEINFO or GETDEVICELIST to determine
>>> 	the mappings for the new device ID.
>>>
>>> NOTIFY4_CHANGE_DEVICE_ID
>>> 	The client call GETDEVICEINFO or GETDEVICELIST
>>> 	to get updated device addresses for the device
>>> 	ID. Layouts remain in force. The client is fenced
>>> 	from any devices that were in the old address list,
>>> 	but not the new address list.
>>>
>>> CB_NOTIFY does not change the device stateid.
>>>
>>> In addition to informing the client of mapping changes via
>>> CB_NOTIFY and CB_RECALL, SEQUENCE carries status flags to
>>> indicate that device mappings have changed.
>>>
>>> The flags are :-
>>>
>>> SEQ4_STATUS_DEVID_DELETED
>>> 	Mapping has been partially deleted. If the client is
>>> 	using CB_NOTIFY it may issue a GETDEVICEINFO for each
>>> 	notified deletion or GETDEVICELIST to refresh the entire
>>> 	map. This flag stays on until the client responds to
>>> 	any and all _DELETE notifications, or it has issued a
>>> 	GETDEVICELIST.
>>>
>>> SEQ4_STATUS_DEVID_DELETED_ALL
>>> 	This flag stays on until the client issues a DELEGRETURN
>>> 	using the devstateid.
>>>
>>> SEQ4_STATUS_DEVID_CHANGED
>>> SEQ4_STATUS_DEVID_ADDED
>>> 	These flags stay on until a GETDEVICELIST or GETDEVICEINFO
>>> 	(if using CB_NOTIFY) gets the affected mappings.
>>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 29 Oct 2007 15:57:44 -0400

I've been hearing a lot talk about getting together a spec by 11/19
(final final deadline for next ietf) that would serve as the basis for a
group last-call, although I've been saying "how about 'group
penultimate-call'".  As part of that, I'm trying to get to closure on
the op to error mapping and I didn't expect to see a proposal at this
point that would add 3 new ops, and a whole bunch of errors to two
existing ops.  If this is really needed, then we'll do what we have to
do, but I'm not clear on the justification for something so different
from what had been previously agreed to. 

Is this stuff really a requirement, or is this a case of preferring to
have bells and whistles, that we can do without?

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Monday, October 29, 2007 11:45 AM
To: Robert Gordon; Marc Eshel
Cc: NFSv4
Subject: Re: [nfsv4] Device stateids

Benny

--
1. Introduction

GETDEVICEINFO
SYNOPSIS: devstateid, devsetid, deviceid, layout_type, maxcount,
notify_ask ->
	devstateid, notify_ack, device_addr

GETDEVICELIST
SYNOPSIS: devstateid, devsetid, layout_type, maxcount, cookie,
cookieverf, notify_ask ->
	devstateid, notify_ack, cookie, cookieverf, device info list<>

3. Returning device IDs

DEVICERETURN

4. Sequencing rules

5. Recalling device IDs

CB_DEVICERECALL

CB_DEVICERECALL takes a devstateid, devsetid, and a deviceid to notify
the client it needs to return the specified deviceid (when non-zero) or
all device
IDs in the devsetid when the deviceid is set to zero, by using a
respective
DEVICERETURN. CB_DEVICERECALL does not change the device stateid.

When the "deleted" boolean flag is set to false the client MAY use
GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if it
has valid layouts the refer to the changed deviceid.  If the "deleted"
flag
is set to true the specified deviceid will not be reused by the server
and
all outstanding layouts referring to it are expected to be recalled via
CB_LAYOUTRECALL; hence the client SHOULD NOT attempt to call
GETDEVICEINFO or
GETDEVICELIST for the specified deviceid. However, if it does do so the
server
MUST return a NFS4ERR_INVAL error.

6. Device notification

CB_DEVICENOTIFY

If the client has already issued a GETDEVICEINFO for the specified
deviceid
then it SHOULD just ignore the device id notification.
Updating an existing device ID mapping should be done using a sequence
of
CB_DEVICERECALL, DEVICERETURN, and GETDEVICEINFO (or GETDEVICELIST).

7. SEQUENCE flags

---

From: "dean hildebrand" <[email-masked]>
Date: Mon, 29 Oct 2007 13:47:42 -0700

> from what had been previously agreed to.

I'm probably behind the times, but what is the previous agreement?
Leave draft 14 devices as is?  The changes suggested in the other post
included below Benny's?

Dean

---

From: Marc Eshel <[email-masked]>
Date: Mon, 29 Oct 2007 14:28:36 -0700

Yes, we all want to get to a final draft but it can not be an excuse to 
selectively choose changes. The last set of changes to getdevicelist and 
layoutget were a major changes that some how got in very late and in the 
last bakeathon I did not find one person that was happy with those 
changes, so I am not sure how the decision are made. I explicitly 
requested that the ability to recall devices per-fsid will be maintained, 
but some how it was dropped without general agreement. So I think we 
should consider this proposal or undo some changes that took away some 
functionality that we had in previous drafts. In general the approach on 
the Linux server side is to allow the different implementation as much 
flexibility as possible and this proposal helps us to do that.
Marc.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 30 Oct 2007 20:24:39 -0400

> Plus, I simplified the proposal by not requiring any new sequence
> flags as the server should use the proposed callback operation to
> recall device mappings and use the
SEQ4_STATUS_RECALLABLE_STATE_REVOKED 
> and SEQ4_STATUS_CB_PATH_DOWN sequence flags as appropriate the 
> same it may use them for other recallable state objects (locks and 
> layouts).

As far as I can see you don't you them the same and that is an issue. 

> stateid (devstateid). There is at most one devstateid per { client ID,

> layout type, devsetid } nexus.

> A recommended filesystem attribute - devsetid is introduced.

Notice that this arrangement means that fsid A and fsid B share a 
devsetid or layout type X iff they share it for layout type Y.
It seems that layout type is the one that should have the flexibility
to figure out how it arranges its devices.

I think a better model is to divide device id into major and minor
fields and then all devices sharing a major device value become a device
set.  I think this would give you a lot more flexibility.

> DEVICERETURN returns a device ID mapping.  When called, this client 
> MUST guarantee not to make further use of that device ID mapping.

Do you really mean that?  Isn't this like a CLOSE if which if you used 
the stateid for the open file, you would get an error.  Do you really
mean that he server can depend on you not using this?

> If the deviceid argument is set to zero, all device ID mappings in the

> referred devset are returned.  The server updates the devstateid and 
> returns the updated value.

If the deviceid were zero, the returned value of devstateid would be
zero.
Correct?

notify_ask ->

> SYNOPSIS: devstateid, devsetid, layout_type, maxcount, cookie,
cookieverf, notify_ask ->>

> GETDEVICEINFO and GETDEVICELIST each take the a devstateid, a
devsetid, 
> and the layout type, and return a non-zero devstateid.

This looks inconsistent to me.

If I do a GETDEVICEINFO(a), I get a non-zero stateid.

If I then do a DEVICERETURN(a), I get a zero stateid.

OK, now I start with a zero stateid again and do a GETDEVICELIST and
get a non-zero stateid.

If I do a GETDEVICELIST, I get another non-zero stateid.

If I then do a DEVICERETURN(a), I get ... a non-zero stateid because
the GETDEVICELIST has some effect.  Is that true.

It seems the DEVICERETURN is the inverse GETDEVICEINFO but that 
GETDEVICELIST has no inverse.  At the very least we need more clarity
about this stuff.

> GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally 
> asking for device notifications and return a corresponding flag 
> confirming the notification.

Isn't this new function?  Is it really needed?

>Device stateids follow the sequencing and use rules of all other
stateid 
>types. 

I assume that that means I use the normal "other" value with a seqid of
zero as a wildcard.
 
> The devstateid is invalidated when all devices associated with 
>it are returned via DEVICERETURN. The client can then "forget" about
that 
> devstateid and use a value of zero for the devstateid for any
subsequent 
> operation that takes a devstateid argument and allows the
uninitialized 
> value.

I suppose but the server can return that devstateid as zero.  Then the 
client can either remember that zero and use it in which case he is
using 
the correct thing or he can specially note it is zero and this allows
him
to forget it (but then he has to specially note that he has forgotten it
and should use zero in its stead).  Seems like the code would be simpler
if it just remembered the last thing it got back (if the client did
parallel once it would have to compare state seqids and treat zero
specially
as later than all normal ones.

> all device IDs in the devsetid when the deviceid is set to zero, by 
> using a respective DEVICERETURN. CB_DEVICERECALL does not change the 
> device stateid.

A couple things are unclear.  It says it needs to return the specified
device id, but is that true when 'deleted' is false?  Also, what happens
when the device id is zero and 'deleted' is false.  What is supposed to
happen?

> server SHOULD fence the client from accessing the affected devices 
> until the client has returned the recalled device IDs.

And after it has returned it?  Should the server not fence the client
form a device it has returned.

> GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if 
> it has valid layouts the refer to the changed deviceid.  

After or instead of doing the return?

> If the "deleted" flag is set to true the specified deviceid will not
be
> reused by the server and all outstanding layouts referring to it are 
> expected to be recalled via CB_LAYOUTRECALL; hence the client SHOULD 
> NOT attempt to call GETDEVICEINFO or GETDEVICELIST for the specified 
> deviceid. However, if it does do so the server MUST return a
NFS4ERR_INVAL 
> error.

> not previously returned by GETDEVICELIST.

Again, what is the need here?  The client will see the device id's
when he gets a layout.

> CB_DEVICERECALL and CB_DEVICENOTIFY, the server MUST use the 
> SEQ4_STATUS_RECALLABLE_STATE_REVOKED SEQUENCE flag to indicate 
> that one or more device IDs have been revoked without expiration 
> of the lease period, due to the client's failure to return them 
> when recalled. 

First of all, in the case of CB_DEVICENOTIFY, nothing has been 
recalled or is supposed to be returned, as far as I can see.

Then it speaks of devices being "revoked".  Devices do not have
stateids and are not, in that sense, locking objects.  If the 
current protocol, when a stateid has multiple locks (.e.g. byte-range
locks), it is carefully stateid that when one is revoked, all are as
a unit and that is tied to the associated stateid being made invalid.

When SEQ4_STATUS_RECALLABLE_STATE_REVOKED is set in the current 
protocol, the client is supposed to use TEST_STATEID to determine
what state has been lost and acknowledge that loss using FREE_STATEID.

I think it would be much better if revocation continued to follow
that model.  If a devstateid is revoked, then, at least for that,
you start from scratch. 

> This status bit remains set on all SEQUENCE replies until the 
> loss of all such device ID mappings has been acknowledged by use 
> of DEVICERETURN.

I think it should stay until all revoked stateids are freed via
FREE_STATEID, just as it is today.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Monday, October 29, 2007 11:45 AM
To: Robert Gordon; Marc Eshel
Cc: NFSv4
Subject: Re: [nfsv4] Device stateids

During the most recent bakeathon concern has been raised about the
global device ID namespace and Marc Eshel from IBM requested that we
retain the ability to recall devices per-fsid.  To satisfy this
requirement as well as the requirement for a simple global device ID
namespace the updated proposal below introduces a device-set identifier
(devsetid) that qualifies the device ID.

This provides for a more flexible deviceid namespace topology that can
cover either a global deviceid namespace, per-fsid namespace, or
anything in between (e.g. when several filesystems reside on (and are
confined to) a set of devices, like in EMC or Panasas' cases).

Plus, I simplified the proposal by not requiring any new sequence flags
as the server should use the proposed callback operation to recall
device mappings and use the SEQ4_STATUS_RECALLABLE_STATE_REVOKED and
SEQ4_STATUS_CB_PATH_DOWN sequence flags as appropriate the same it may
use them for other recallable state objects (locks and layouts).

One last change is providing for a "wildcard" value (0) for device ID
that specifies all the device IDs in a devset.  This means that the
server MUST NOT return a zero device ID in GETDEVICELIST or in any
layout and it must reject it as invalid in GETDEVICEINFO.GETDEVICEINFO
and GETDEVICELIST also take a boolean flag optionally asking for device
notifications and return a corresponding flag confirming the
notification.
I guess that this won't be an issue for anyone; otherwise, please
holler.

Benny

--
1. Introduction

The NFS client maintains a mapping of device IDs contained in a layout
to the corresponding storage device address.
Device IDs and filesystem IDs are associated with device set IDs such
that each fsid is associated with a single devsetid as indicated by a
new, recommended filesystem attribute, and each deviceid is associated
with a single devsetid.

The state of the mapping is represented by a recallable device stateid
(devstateid). There is at most one devstateid per { client ID, layout
type, devsetid } nexus.

A recommended filesystem attribute - devsetid is introduced.
All deviceids associated with any file in the filesystem are associated
with this devsetid. 
If the server does not support device sets, it can omit this attribute,
making the deviceid namespace global and implicitly associating all
deviceids with the default devset (having the implicit devsetid ID of 0)

GETDEVICEINFO

GETDEVICELIST

GETDEVICEINFO and GETDEVICELIST each take the a devstateid, a devsetid,
and the layout type, and return a non-zero devstateid.
In case the client holds a valid devstateid it MUST use it for
subsequent GETDEVICEINFO and GETDEVICELIST that update the device ID
mappings.  The devstateid argument must be set to zero whenever the
client has no valid devstateid (e.g. after reboot or after the
devstateid was previously returned via DEVICERETURN).

GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally
asking for device notifications and return a corresponding flag
confirming the notification.

3. Returning device IDs

DEVICERETURN

DEVICERETURN returns a device ID mapping.  When called, this client MUST
guarantee not to make further use of that device ID mapping.
If the deviceid argument is set to zero, all device ID mappings in the
referred devset are returned.  The server updates the devstateid and
returns the updated value.

4. Sequencing rules

Device stateids follow the sequencing and use rules of all other stateid
types.  The devstateid is invalidated when all devices associated with
it are returned via DEVICERETURN. The client can then "forget" about
that devstateid and use a value of zero for the devstateid for any
subsequent operation that takes a devstateid argument and allows the
uninitialized value.

5. Recalling device IDs

CB_DEVICERECALL

CB_DEVICERECALL takes a devstateid, devsetid, and a deviceid to notify
the client it needs to return the specified deviceid (when non-zero) or
all device IDs in the devsetid when the deviceid is set to zero, by
using a respective DEVICERETURN. CB_DEVICERECALL does not change the
device stateid.

All layouts using the affected device IDs remain in force, but the
server SHOULD fence the client from accessing the affected devices until
the client has returned the recalled device IDs.

When the "deleted" boolean flag is set to false the client MAY use
GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if it
has valid layouts the refer to the changed deviceid.  If the "deleted"
flag is set to true the specified deviceid will not be reused by the
server and all outstanding layouts referring to it are expected to be
recalled via CB_LAYOUTRECALL; hence the client SHOULD NOT attempt to
call GETDEVICEINFO or GETDEVICELIST for the specified deviceid. However,
if it does do so the server MUST return a NFS4ERR_INVAL error.

6. Device notification

CB_DEVICENOTIFY

If the server supports CB_DEVICENOTIFY for devices and the client has
requested device id notifications via GETDEVICEINFO or GETDEVICELIST the
server can inform the client of additional deviceids that were not
previously returned by GETDEVICELIST.

The purpose of this operation is to reduce the latency for getting new
device mappings.  Rather than waiting until the client accesses a file
that uses the new device ID, issue a LAYOUTGET, and then a GETDEVICEINFO
on the new device, the server can notify the client of any new device ID
in advance and have the client get the device address mapping by calling
GETDEVICEINFO at its convenience.

CB_DEVICENOTIFY takes a device stateid, a device set ID and a device ID
in the device set for a device ID that was not returned in a previous
series GETDEVICELIST calls (where the series ended in gdlr_eof == TRUE).
The client calls GETDEVICEINFO or GETDEVICELIST to determine the
mappings for the new device ID.
CB_DEVICENOTIFY does not change the device stateid.

If the client has already issued a GETDEVICEINFO for the specified
deviceid then it SHOULD just ignore the device id notification.
Updating an existing device ID mapping should be done using a sequence
of CB_DEVICERECALL, DEVICERETURN, and GETDEVICEINFO (or GETDEVICELIST).

7. SEQUENCE flags

In addition to informing the client of mapping changes via
CB_DEVICERECALL and CB_DEVICENOTIFY, the server MUST use the
SEQ4_STATUS_RECALLABLE_STATE_REVOKED SEQUENCE flag to indicate that one
or more device IDs have been revoked without expiration of the lease
period, due to the client's failure to return them when recalled. This
status bit remains set on all SEQUENCE replies until the loss of all
such device ID mappings has been acknowledged by use of DEVICERETURN.

On Aug. 09, 2007, 10:22 +0300, Benny Halevy <[email-masked]> wrote:
> The DEVID_DELETED cases should be covered by CHANGED as the client 
> will have to revalidate all the device mappings anyway to see what 
> actually changed and should either not see the deleted deviceid 
> returned by GETDEVICELIST or get an error from GETDEVICEINFO for that
deviceid.
> and the server supports it.  If the server does not support 
> GETDEVICELIST is must not set SEQ_STATUS_DEVID_CHANGED for added 
> devices as the client has no way to get the new device ID.  These will

> be discovered only when the client will get new layouts referring to 
> the new devices (which is perfectly fine)
> 
>> Robert.
>>
>>> The NFS client maintains a mapping of device ids contained in a 
>>> layout, to the corresponding storage device address.
>>> The state of the mapping is represented by a single recallable 
>>> device stateid (devstateid). There is at most one devstateid per { 
>>> client ID, layout type } pair.
>>>
>>> GETDEVICEINFO and GETDEVICELIST each take the layout type and an 
>>> optional devstateid and return the devstateid.  In case the client 
>>> holds a valid devstateid it MUST use it for subsequent GETDEVICEINFO

>>> and GETDEVICELIST that update the device ID mappings.  The 
>>> devstateid argument is invalid whenever the client has no valid 
>>> devstateid (e.g. after reboot or after the devstateid was previously

>>> returned via DELEGRETURN).
>>> GETDEVICEINFO or GETDEVICELIST that operate on a device stateid that

>>> was already recalled via CB_RECALL MUST be rejected by the server 
>>> with NFS4ERR_BADSTATEID.
>>>
>>> GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally 
>>> asking for device notifications and return a corresponding flag 
>>> confirming the notification.
>>>
>>> All layouts using the device IDs remain in force, but the server 
>>> MUST fence the client from accessing the affected devices until the 
>>> client has obtained the re-mapped device IDs via GETDEVICEINFO or 
>>> GETDEVICELIST.
>>>
>>> CB_RECALL takes a devstateid to notify the client it needs to return

>>> all device IDs identified by the devstateid.  The filehandle 
>>> argument has no use in this case.  It does not change the device 
>>> stateid.
>>>
>>> requested device id notifications via GETDEVICEINFO or 
>>> GETDEVICELIST; The server can inform the client of a delete, add, or

>>> change mapping.
>>>
>>> CB_NOTIFY takes a device stateid qualifying a device ID for the 
>>> following device notifications:
>>>
>>> NOTIFY4_DELETE_DEVICE_ID
>>> In addition to informing the client of mapping changes via CB_NOTIFY

>>> and CB_RECALL, SEQUENCE carries status flags to indicate that device

>>> mappings have changed.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 31 Oct 2007 19:59:44 +0200

Dave, thanks for the comments!
My answers inline below.

Benny

> As far as I can see you don't you them the same and that is an issue. 

OK. we can define SEQ4_STATUS_DEVID_DELETED* in the spirit of the
last proposal [that I see is now part of draft-15 which seems to
introduce inconsistencies with the side effects of LAYOUTRETURN
for RETURN_{FSID,ALL} that return the respective device mappings
but do not change the device stateid]

> set.  I think this would give you a lot more flexibility.

Yeah, I see your point. Good idea.

> mean that he server can depend on you not using this?

same as LAYOUTRETURN. the server is not required to fence off the client after
LAYOUTRETURN.

> If the deviceid were zero, the returned value of devstateid would be

I'm not sure.  What does the server return on CLOSE?

> the GETDEVICELIST has some effect.  Is that true.

if you didn't return all device IDs using deviceid==0 then you
shouldn't get a zero devstateid.

> GETDEVICELIST has no inverse.  At the very least we need more clarity

The way I see it DEVICERETURN w/ deviceid==0 is sort of the inverse
of GETDEVICELIST as it "resets" (or reinitializes the device state.

> Isn't this new function?  Is it really needed?

It was in the previous proposal.
It is not strictly needed as the new device ID can always be introduced
at LAYOUTGET time and be resolved using GETDEVICEINFO.
The motivation is to optimize this case and reduce the latency by
allowing the client to get the device info in advance (it could do
so opportunistically, only if it's idle).

> as later than all normal ones.

Agreed. That makes a lot of sense.

> when the device id is zero and 'deleted' is false.  What is supposed to

I meant that the client should send a DEVICERETURN in both cases,
although it doesn't seem harmful if it issues a GETDEVICEINFO to
refresh the data when "deleted" is set to false.  But the server will
need to detect that case.

> form a device it has returned.

I don't think so.  We have access control mechanism to limit what
the client can do on the device but fencing the client off may hurt
performance needlessly.

> After or instead of doing the return?

See answer above.  This needs to be clarified and I think I'll go
with the permissive approach of allowing to get the info without
and explicit DEVICERETURN.

> Again, what is the need here?  The client will see the device id's

Agreed.  This is an optimization we can live without of.
see above...

> recalled or is supposed to be returned, as far as I can see.

right.

> that model.  If a devstateid is revoked, then, at least for that,

OK

> FREE_STATEID, just as it is today.

OK

---

From: Robert Gordon <[email-masked]>
Date: Wed, 31 Oct 2007 16:44:36 -0500

On Oct 31, 2007, at 12:59 PM, Benny Halevy wrote:

>>> { client ID,

An alternative way of looking at this is to say a device stateid  
represents
a collection of device id's. The Server can then have the flexibility to
arrange the grouping how it likes. One server implementation may wish to
group things by MDS FSID while another implementation may wish to group
things by some other criteria or just have one stateid that represents
all device ids.

Robert.