Subject: [nfsv4] Device stateids

From: Spencer Shepler <[email-masked]>
Date: Fri, 27 Jul 2007 00:12:11 -0500

We seem to have consensus on Mike's device stateid proposal
and I will work to integrate the changes into the next draft.
If there are any tweaks needed, please speak up.

I have included the original text for reference.

Spencer

Recallable Device ID to Device Address Mappings
-----------------------------------------------

Proposal:

Make each MDS server's device ID to device adddress
mappings a single recallable object.

Rationale:

The specification is unclear as to when device ID to device
address mappings can change. There are attempts single
tie these to layouts and/or leases. They should be tied to
leases, but layouts are separate, albeit, related concept,
and besides which, the layout-type independent part of a
layout doesn't have a device ID.

More Details of Proposal:

I have previously observed that device mappings and
their semantics closely follow directories. The desire to
have recall of all of some mappings resembles directory
delegations and notifications represent a model that device
mappings could use. Thus it seems possible use directory
operations for manipulating device mappings. However doing
this would not save any operations because a "PUTDEVFH"
operation (with a layouttype argument) would have to
be added to allow READDIR to replace GETDEVICELIST
(and then we have to figure out how to have directory
entries represent device ID to device address mappingfs --
presumably with attributes). And an operation to map a
device ID to an address would still be needed, unless we
make device IDs file handles (a 4 byte length followed by
a 4 byte handle).

Regardless, the complexity of simulating a directory,
when there aren't POSIX APIs that want directory semantics
(e.g. per entry cookies) for device mappings might not be
worth it, nor is the timing great for doing this scale
of change.  It also dubious if any of the READDIR code
would be shared for real directories and device maps.

An alternative is proposed below.

GETDEVICEINFO or GETDEVICELIST each return a device
stateid that represents a recallable, read-only hold on
the mappings. Recall of the device stateid will remove the
mappings. Lease expiration can remove the mappings. The
mappings stay in force until recall of the stateid or
lease expiration. GETDEVICEINFO and GETDEVICELIST change
the current file handle so the CB_RECALL can be used to
recall the device stateid. All layouts using the device IDs
remain in force, but the server MUST fence the client
from accessing the affected data servers until
the device IDs are re-mapped (via GETDEVICEINFO or
GETDEVICELIST).

There is at most one device stateid per { client ID,
layout type } pair.

CB_NOTIFY can modify, delete, or add mappings,
if GETDEVICEINFO and GETDEVICELIST request notifications and
if the server supports device notifications.
The notifications available are:

NOTIFY4_DELETE_DEVICE_ID
	Deletes all mappings using the specified device ID.
	The server MUST stop using the device ID. Any
	layouts remain in force, but aren't useful until
	new mappings are available. The server fences the
	client from the affected data server until all
	layouts with the device ID are returned, or the
	client obtains new mappings. The server provides
	a hint as to whether new mappings will arrive and
	if so how long.

NOTIFY4_ADD_DEVICE_ID
	Adds a device ID to address mapping. This is
	for a device ID the that was not returned in a
	previous series GETDEVICELIST calls (where the
	series ended in gdlr_eof == TRUE).  The client
	calls GETDEVICEINFO or GETDEVICELIST to determine
	the mappings for the new device ID.

NOTIFY4_CHANGE_DEVICE_ID
	The client call GETDEVICEINFO or GETDEVICELIST
	to get updated device addresses for the device
	ID. Layouts remain in force. The client is fenced
	from any devices that were in the old address list,
	but not the new address list.

SEQUENCE will have a status flag added to indicate that
the device mappings have been partially deleted, fully
deleted (e.g. recall or lease expiry), changed, or added
to. The partially deleted flag stays on until the client
responds to any and all _DELETE notifications, or it does
a GETDEVICELIST. The fully deleted flag stays on until
DELEGRETURN returns the device stateid. The changed and
added flags stay on until a GETDEVICELIST or GETDEVICEINFO
gets the affected mappings.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 27 Jul 2007 09:26:25 -0400

I will make a change to the state management chapter to add this to the
list of things that stateids can designate. 

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, July 27, 2007 1:12 AM
To: [email-masked]
Subject: [nfsv4] Device stateids

We seem to have consensus on Mike's device stateid proposal and I will
work to integrate the changes into the next draft.
If there are any tweaks needed, please speak up.

Spencer

Proposal:

Make each MDS server's device ID to device adddress mappings a single
recallable object.

Rationale:

The specification is unclear as to when device ID to device address
mappings can change. There are attempts single tie these to layouts
and/or leases. They should be tied to leases, but layouts are separate,
albeit, related concept, and besides which, the layout-type independent
part of a layout doesn't have a device ID.

I have previously observed that device mappings and their semantics
closely follow directories. The desire to have recall of all of some
mappings resembles directory delegations and notifications represent a
model that device mappings could use. Thus it seems possible use
directory operations for manipulating device mappings. However doing
this would not save any operations because a "PUTDEVFH"
operation (with a layouttype argument) would have to be added to allow
READDIR to replace GETDEVICELIST (and then we have to figure out how to
have directory entries represent device ID to device address mappingfs
-- presumably with attributes). And an operation to map a device ID to
an address would still be needed, unless we make device IDs file handles
(a 4 byte length followed by a 4 byte handle).

Regardless, the complexity of simulating a directory, when there aren't
POSIX APIs that want directory semantics (e.g. per entry cookies) for
device mappings might not be worth it, nor is the timing great for doing
this scale of change.  It also dubious if any of the READDIR code would
be shared for real directories and device maps.

GETDEVICEINFO or GETDEVICELIST each return a device stateid that
represents a recallable, read-only hold on the mappings. Recall of the
device stateid will remove the mappings. Lease expiration can remove the
mappings. The mappings stay in force until recall of the stateid or
lease expiration. GETDEVICEINFO and GETDEVICELIST change the current
file handle so the CB_RECALL can be used to recall the device stateid.
All layouts using the device IDs remain in force, but the server MUST
fence the client from accessing the affected data servers until the
device IDs are re-mapped (via GETDEVICEINFO or GETDEVICELIST).

There is at most one device stateid per { client ID, layout type } pair.

CB_NOTIFY can modify, delete, or add mappings, if GETDEVICEINFO and
GETDEVICELIST request notifications and if the server supports device
notifications.
The notifications available are:

NOTIFY4_DELETE_DEVICE_ID

NOTIFY4_ADD_DEVICE_ID

NOTIFY4_CHANGE_DEVICE_ID

SEQUENCE will have a status flag added to indicate that the device
mappings have been partially deleted, fully deleted (e.g. recall or
lease expiry), changed, or added to. The partially deleted flag stays on
until the client responds to any and all _DELETE notifications, or it
does a GETDEVICELIST. The fully deleted flag stays on until DELEGRETURN
returns the device stateid. The changed and added flags stay on until a
GETDEVICELIST or GETDEVICEINFO gets the affected mappings.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 30 Jul 2007 10:16:53 -0700

> I have included the original text for reference.

I think one of the areas still under debate was about the ability to 
recall and return multiple devices at once and I want to make sure 
everyone is aware of the implication of this proposal.

There was discussion about breaking the deviceID in two sections 
(major,minor) such that a recall could recall all IDs associated with a 
major num.  I don't see this in the proposal below.

So my question is, is it possible to recall all device IDs for a given 
FSID in a single call, and return in a single call (like it is in the 
current text)?  If not, is everyone ok with this.

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 11:54:46 +0300

> current text)?  If not, is everyone ok with this.

For us it's sub-optimal but we can live with this restriction.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 31 Jul 2007 07:40:58 -0700

> For us it's sub-optimal but we can live with this restriction.
I missed the answer to Garth question. Did you loose the ability to recall 
all device IDs for a given FSID?
Marc.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 17:55:12 +0300

Apparently, one more thing the new proposal loses is the layout_type
granularity.  CB_NOTIFY will delete all device mappings for
all layout types, even if the MDS is serving heterogeneous file
systems exported over different sets of devices and layout types.

The idea behind my proposal for composite device ID is to be
able to group devices into "device sets" that you can recall
and return.  The server can then map the device sets according
to its architecture and topology so one server can have a single
device set for all devices, the other can have one device set
per filesystem, and a third could map several filesystems onto
one device set as they they may share the same devices.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 17:50:05 +0300

> all device IDs for a given FSID?

It appears so in Spencer's most recent "device stateids" proposal.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 17:55:12 +0300

Benny

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 17:50:05 +0300

---

From: Marc Eshel <[email-masked]>
Date: Tue, 31 Jul 2007 08:09:53 -0700

> >>> (major,minor) such that a recall could recall all IDs associated 
with 
> > a 

Given that few of us would prefer to have this ability Is there a good 
reason why it is being dropped? 
I assume that recall all layouts for a given FSID is still there.
Marc.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 18:58:50 +0300

> reason why it is being dropped? 

If I recall correctly, Mike Eisler's original reason was to simplify
the client so it wouldn't have to track the device mappings per fsid.
FWIW, this doesn't look too complicated to implement IMO...

> I assume that recall all layouts for a given FSID is still there.

Yes it's still there.

---

From: Robert Gordon <[email-masked]>
Date: Tue, 31 Jul 2007 11:12:27 -0500

On Jul 30, 2007, at 12:16 PM, Garth Goodson wrote:

> So my question is, is it possible to recall all device IDs for a  
> given FSID in a single call, and return in a single call (like it  
> is in the current text)?  If not, is everyone ok with this.

it is, and i think that the current proposal is good.

Robert..

---

From: Garth Goodson <[email-masked]>
Date: Tue, 31 Jul 2007 09:50:47 -0700

> it is, and i think that the current proposal is good.

I guess there is still some confusion if you are claiming that it is 
still possible to recall all device IDs and return them all in a single 
call (how are all their stateids specified?).  It seems a separate op is 
needed for each stateid.

A few messages back Benny stated it was not possible and I tend to agree.

-Garth

---

From: Garth Goodson <[email-masked]>
Date: Tue, 31 Jul 2007 09:53:19 -0700

Benny Halevy wrote:
>> given 
But since LAYOUTRETURN now takes a stateid, it is not possible to return 
them all at once as was possible before.  Correct?

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Tue, 31 Jul 2007 20:22:42 +0300

> them all at once as was possible before.  Correct?

I assumed that you can return for FSID/ALL as before with a null lora_stateid
and a non-null lora_recallstateid but I'm not sure what was the intent.
Obviously that changes the layout state of each affected file, but on the
other hand, the client doesn't really need the layout stateid per-file
after returning the whole layout.  On a subsequent LAYOUTGET it can
send the same stateid it sent after opening the file and if the file
is closed by now it must open it again anyway.

---

From: Robert Gordon <[email-masked]>
Date: Tue, 31 Jul 2007 12:34:56 -0500

On Jul 31, 2007, at 11:50 AM, Garth Goodson wrote:
> I guess there is still some confusion if you are claiming that it  
> is still possible to recall all device IDs and return them all in a  
> single call (how are all their stateids specified?).  It seems a  
> separate op is needed for each stateid.
>

> gets the affected mappings.

I took it to mean:

if in SEQUENCE the server has indicated that the deviceid
mappings have been fully deleted ; then the client should
dump its current device list and issue GETDEVICELIST.

Robert.

---

From: Garth Goodson <[email-masked]>
Date: Tue, 31 Jul 2007 10:38:54 -0700

Robert Gordon wrote:
>>>> given FSID in a single call, and return in a single call (like it is 
>>>> in the current text)?  If not, is everyone ok with this.
>> still possible to recall all device IDs and return them all in a 
The thing that has me hung up is the sentence:

'The fully deleted flag stays on until DELEGRETURN returns the device 
stateid.'

Which seems to imply that each stateid needs to be returned.

Spencer/Mike clarification?

-Garth

---

From: Garth Goodson <[email-masked]>
Date: Tue, 31 Jul 2007 10:40:28 -0700

> is closed by now it must open it again anyway.

I have no problem with this (if this is what was intended).  But again, 
it looks like there are details that are missing from the proposals that 
are causing us to make assumptions.

-Garth

---

From: Robert Gordon <[email-masked]>
Date: Tue, 31 Jul 2007 12:58:58 -0500

Ah..

On Jul 27, 2007, at 12:12 AM, Spencer Shepler wrote:

> the mappings.

I was thinking that each device would have a stateid, and the
entire list would also be represented by a state id, such that
recalling the 'list' stateid would recall ALL the mapping..

.. Robert.