Subject: [nfsv4] pNFS: Open issues (from chapter 12 & 13 reviews)

From: Garth Goodson <[email-masked]>
Date: Wed, 20 Jun 2007 16:06:54 -0700

Pulled from meeting minutes thanks to Lisa and Sam.

Chapter 12: Common pNFS
-----------------------

* Open Issue 1: Layout hint attribute on OPEN (or a later SETATTR)

   (Line 12890 of draft 10)
   - Dictates the use of a persistent reply cache if layout hints are to
     be used. General agreement that this is not good.
   - SETATTR after an exclusive create may be a viable option (servers
    may be fine with this)

Noveck's proposed solution: Encourage, but not require a persistent 
reply cache for layout_hint.  A client can do a layout_hint on EXCLUSIVE 
CREATE, but the server may or may not be able to handle it effectively.

* Open Issue 2: Device ID Mapping and Leasing
   - Mike Eisler started a thread on the WG (04/17/07) however no
     resolution has arisen -- need to revisit and conclude

* Open Issue 3: Layout segment type
   - Benny made a proposal to the WG list that a new type be added called
     a layoutsegment.  This introduces an iomode into LAYOUTCOMMIT.
   - Currently there has not yet been much discussion (is there a
     consensus?)

* Open Issue 4: adding stateids back to layouts
   - Came up in discussion of Chapter 13
   - Mike Eisler to write up proposal for layoutget to return stateids to
     enforce opens with the use of layouts
     - requirement that layouts can still be held across OPEN/CLOSE
     - requirement that the same stateid be returned for any layoutget
       request on the same file (e.g., if multiple ranges are requested
       separately).

Chapter 13: (file layout)
-------------------------

* Open Issue 1: new device type mappings in layout (replace
   SIMPLE/COMPLEX)
   - Mike Eisler made new proposal.  Waiting until Fri 22nd to get more
     comments.  Currently close to reaching consensus.

* Open Issue 2: data server trunking via equivalent multipathing nodes
   - From the previous discussion a question was asked as to whether
     trunking across equivalent data servers was allowed.
   - This was not the original intent, but the spec should clearly state
     as to how equivalent DSs may be used.

If I have left any out, feel free to add them.  None are currently in 
the issue tracker.

-Garth

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Thu, 21 Jun 2007 21:33:47 -0700 (PDT)

>      resolution has arisen -- need to revisit and conclude
[...]
>        request on the same file (e.g., if multiple ranges are

Here are my ideas for each of the above two issues (presented in
opposite order).

Layout StateIDs
---------------

Proposal:

Add a layout stateid to NFSv4.1.

Rationale:

Because layouts do not take stateids, it is possible for
clients using non-NFSv4.1 layout types to perform I/O
to files:

	LOOKUP "file", ACCESS, LAYOUTGET

The layout is then used to find the storage device
and I/O commences.

This is bad because OPEN is there to coordinate with other
openers that establish a deny mode, whether they be NFSv4
or CIFS.

More Details of Proposal:

LAYOUTGET will take a normal share reservation stateid
(as returned by OPEN) or a delegation stateid as
an additional argument. If successful, it will return
a layout stateid. For a given file/layout type pair,
the same layout stateid is always returned by LAYOUTGET,
except when there is a server restart.

LAYOUTRETURN is doing too much ... it supports return by
_FILE, _FSID, and _ALL. LAYOUTRETURN will now return just a
single file, and will take a layout stateid as an argument.

LAYOUTCOMMIT takes a layout stateid.

Layout stateids persist across CLOSE and DELEGRETURN.

The model is then like delegation stateids.

For the NFSv4.1 layout type, READ and WRITE to a data
server will continue to require a delegation stateid,
a share reservation stateid, or a lock stateid.  The data
server SHOULD be able to map these stateids to the granted
layout to make sure the client has a layout, but in any
case the data server MUST reject I/Os to files that do
not have layouts.

As with other state, layout stateids might become invalid
when the lease expires.

For all layout types, the rule is: no layout stateid, no
I/O to data servers.

CB_LAYOUTRECALL will be augmented to recall all layouts
of tied to one or more device IDs.

Recallable Device ID to Device Address Mappings
-----------------------------------------------

Proposal:

Make each MDS server's device ID to device adddress
mappings a single recallable object.

Rationale:

The specification is unclear as to when device ID to device
address mappings can change. There are attempts single
tie these to layouts and/or leases. They should be tied to
leases, but layouts are separate, albeit, related concept,
and besides which, the layout-type independent part of a
layout doesn't have a device ID.

More Details of Proposal:

I have previously observed that device mappings and
their semantics closely follow directories. The desire to
have recall of all of some mappings resembles directory
delegations and notifications represent a model that device
mappings could use. Thus it seems possible use directory
operations for manipulating device mappings. However doing
this would not save any operations because a "PUTDEVFH"
operation (with a layouttype argument) would have to
be added to allow READDIR to replace GETDEVICELIST
(and then we have to figure out how to have directory
entries represent device ID to device address mappingfs --
presumably with attributes). And an operation to map a
device ID to an address would still be needed, unless we
make device IDs file handles (a 4 byte length followed by
a 4 byte handle).

Regardless, the complexity of simulating a directory,
when there aren't POSIX APIs that want directory semantics
(e.g. per entry cookies) for device mappings might not be
worth it, nor is the timing great for doing this scale
of change.  It also dubious if any of the READDIR code
would be shared for real directories and device maps.

An alternative is proposed below.

GETDEVICEINFO or GETDEVICELIST each return a device
stateid that represents a recallable, read-only hold on
the mappings. Recall of the device stateid will remove the
mappings. Lease expiration can remove the mappings. The
mappings stay in force until recall of the stateid or
lease expiration. GETDEVICEINFO and GETDEVICELIST change
the current file handle so the CB_RECALL can be used to
recall the device stateid. All layouts using the device IDs
remain in force, but the server MUST fence the client
from accessing the affected data servers until
the device IDs are re-mapped (via GETDEVICEINFO or
GETDEVICELIST).

There is at most one device stateid per { client ID,
layout type } pair.

CB_NOTIFY can modify, delete, or add mappings,
if GETDEVICEINFO and GETDEVICELIST request notifications and
if the server supports device notifications.
The notifications available are:

NOTIFY4_DELETE_DEVICE_ID
	Deletes all mappings using the specified device ID.
	The server MUST stop using the device ID. Any
	layouts remain in force, but aren't useful until
	new mappings are available. The server fences the
	client from the affected data server until all
	layouts with the device ID are returned, or the
	client obtains new mappings. The server provides
	a hint as to whether new mappings will arrive and
	if so how long.

NOTIFY4_ADD_DEVICE_ID
	Adds a device ID to address mapping. This is
	for a device ID the that was not returned in a
	previous series GETDEVICELIST calls (where the
	series ended in gdlr_eof == TRUE).  The client
	calls GETDEVICEINFO or GETDEVICELIST to determine
	the mappings for the new device ID.

NOTIFY4_CHANGE_DEVICE_ID
	The client call GETDEVICEINFO or GETDEVICELIST
	to get updated device addresses for the device
	ID. Layouts remain in force. The client is fenced
	from any devices that were in the old address list,
	but not the new address list.

SEQUENCE will have a status flag added to indicate that
the device mappings have been partially deleted, fully
deleted (e.g. recall or lease expiry), changed, or added
to. The partially deleted flag stays on until the client
responds to any and all _DELETE notifications, or it does
a GETDEVICELIST. The fully deleted flag stays on until
DELEGRETURN returns the device stateid. The changed and
added flags stay on until a GETDEVICELIST or GETDEVICEINFO
gets the affected mappings.

---

From: Robert Gordon <[email-masked]>
Date: Fri, 22 Jun 2007 13:00:53 -0500

I'm 'fore' both proposed changes.

Robert.

---

From: Garth Goodson <[email-masked]>
Date: Fri, 22 Jun 2007 11:41:45 -0700

Couple of questions inline:

> single file, and will take a layout stateid as an argument.

1) If I return a portion of a layout (or a portion is recalled), I must 
still hold on to the stateid until the whole layout is returned, but 
each return will pass back the same stateid, correct?

> Layout stateids persist across CLOSE and DELEGRETURN.

2) If layout stateids persist across CLOSE, how does holding one across 
an OPEN/CLOSE sequence ensure that the file has been opened 
appropriately the second time around?

> of tied to one or more device IDs.

3) Can you be more explicit on this change?  Will it specify multiple 
file handles and byte ranges each of which will have to be returned via 
a separate LAYOUTRETURN (given only one layout range can be returned at 
a time)?

...

> the mappings.

4) Is a single stateid returned by GETDEVICELIST that maps to all the 
devices returned?

Thanks,
-Garth

---

From: Mike Eisler <[email-masked]>
Date: Fri, 22 Jun 2007 11:51:12 -0700 (PDT)

> each return will pass back the same stateid, correct?

Yes. This does mean that the state of layouts are therefore
"all or nothing". Any administrative intervention to revoke
a layout segment affects the entire range.

I'd be OK with multiple stateids per file,layouttype if people
think we need it.

> appropriately the second time around?

It doesn't. I may be naive, but if a NFSv4.1 client is deliberately
going through two code paths to destroy the protocol, that
requires a degree of maliciousness no protocol spec can guard against.

> a separate LAYOUTRETURN (given only one layout range can be returned

It will specify an array of device IDs.
All layouts that use those device IDs need to be returned via
LAYOUTRETURN.

> 4) Is a single stateid returned by GETDEVICELIST that maps to all the

Yes, a single stateid per { client ID, layoutttype } pair.

---

From: "Iyer, Rahul" <[email-masked]>
Date: Fri, 22 Jun 2007 11:53:28 -0700

Hi Mike,
I had a few questions regarding the proposals. My questions are inline.

> clients using 
> > non-NFSv4.1 layout types to perform I/O to files:
> >

Wouldn't it be simpler to state in the spec thhat the client MUST OPEN a
file before using it, regardless of pNFS. Because, I don't see how this
is different from a client doing a LOOKUP and then doing a special
stateid READ/WRITE in non-pNFS v4.1 (Unless we plan to ban special
stateids). 
Stating it in the spec seems to be a simpler way to deal with it.

> >
> > The layout is then used to find the storage device and I/O 
> commences.
> >
> > This is bad because OPEN is there to coordinate with other openers 
> > that establish a deny mode, whether they be NFSv4 or CIFS.
> >
> > LAYOUTGET will take a normal share reservation stateid (as 
> returned by 
> > OPEN) or a delegation stateid as an additional argument. If 
> > successful, it will return a layout stateid. For a given 
> file/layout 
> > type pair, the same layout stateid is always returned by LAYOUTGET, 
> _FILE, _FSID, 
> > and _ALL. LAYOUTRETURN will now return just a single file, and will 
> > take a layout stateid as an argument.
> >
> > For the NFSv4.1 layout type, READ and WRITE to a data server will 
> > continue to require a delegation stateid, a share 
> reservation stateid, 
> > or a lock stateid.  The data server SHOULD be able to map these 
> > stateids to the granted layout to make sure the client has 
> a layout, 
> > but in any case the data server MUST reject I/Os to files 
> that do not 
> > have layouts.
> >
> > As with other state, layout stateids might become invalid when the 
> > lease expires.
> >
> I/O to data 
> > servers.
> > CB_LAYOUTRECALL will be augmented to recall all layouts of 
> tied to one 
> > or more device IDs.
> >
> > The specification is unclear as to when device ID to device address 
> > mappings can change. There are attempts single tie these to layouts 
> > and/or leases. They should be tied to leases, but layouts are 
> > separate, albeit, related concept, and besides which, the 
> layout-type 
> > independent part of a layout doesn't have a device ID.
> >
> > I have previously observed that device mappings and their semantics 
> > closely follow directories. The desire to have recall of 
> all of some 
> > mappings resembles directory delegations and notifications 
> represent a 
> > model that device mappings could use. Thus it seems possible use 
> > directory operations for manipulating device mappings. 
> However doing 
> > operation (with a layouttype argument) would have to be 
> added to allow 
> > READDIR to replace GETDEVICELIST (and then we have to 
> figure out how 
> > to have directory entries represent device ID to device address 
> > mappingfs -- presumably with attributes). And an operation to map a 
> make device 
> > IDs file handles (a 4 byte length followed by a 4 byte handle).
> >
> > Regardless, the complexity of simulating a directory, when there 
> > aren't POSIX APIs that want directory semantics (e.g. per entry 
> > cookies) for device mappings might not be worth it, nor is 
> the timing 
> > great for doing this scale of change.  It also dubious if 
> any of the 
> > READDIR code would be shared for real directories and device maps.
> >
> > GETDEVICEINFO or GETDEVICELIST each return a device stateid that 
> > represents a recallable, read-only hold on the mappings. 
> Recall of the 
> > device stateid will remove the mappings. Lease expiration 
> can remove 
> > the mappings. The mappings stay in force until recall of 
> the stateid 
> > or lease expiration. GETDEVICEINFO and GETDEVICELIST change the 
> > current file handle so the CB_RECALL can be used to recall 
> the device 
> > stateid. All layouts using the device IDs remain in force, but the 
> > server MUST fence the client from accessing the affected 
> data servers 
> > until the device IDs are re-mapped (via GETDEVICEINFO or 
> > GETDEVICELIST).
> >
> > There is at most one device stateid per { client ID, layout type } 
> > pair.

This means that all GETDEVICELISTs and GETDEVICEINFOs return the same
stateid, right? I'm also confused as to why we need the stateid.

Thanks
Regards
Rahul

---

From: Mike Eisler <[email-masked]>
Date: Fri, 22 Jun 2007 12:13:12 -0700 (PDT)

--- "Iyer, Rahul" <[email-masked]> wrote:

> file before using it, regardless of pNFS. Because, I don't see how

Yes we will say that regardless.

> this

We do indeed plan to ban special stateids for pNFS; my understanding
was we agreed to that during the NFSv4.1 layout inspection last week.

> Stating it in the spec seems to be a simpler way to deal with it.

Every other recallable object has some kind of volatile handle
attached to it. It seems prudent that layouts should not be
the exception. I worry about situations where the client and
server disagree about whether there is a layout in force or not.
The stateid is useful to detect issues. For instance,
let's say a client for some reason dropped a CB_LAYOUTRECALL
request. The server eventually revokes the layout. The next
time the client calls LAYOUTGET on the file, with no stateid,
the client doesn't recognize the proble,. With a stateid,
the server can return a different stateid than it had returned
before the revoke. The client now knows something is wrong.
He knows that the future LAYOUTCOMMIT might not commit what he
thought would get committed.

> stateid, right? I'm also confused as to why we need the stateid.

Yes; I thought that was clear from "at most one device stateid per {
client ID, layout type }. :-) You guys are really bumming me out 
about my apparent inability to articulate myself. :-(.

CB_RECALL passes a stateid and a file handle (GETDEVICE* will change
the
current file handle), so we need a stateid. We could add another
op, CB_RECALL_DEV that just passes a layouttype. There doesn't seem
to be much need for that. If it looks like a delegation, and
acts like a delegation ...

---

From: "Iyer, Rahul" <[email-masked]>
Date: Fri, 22 Jun 2007 14:01:48 -0700

> -----Original Message-----
> From: Mike Eisler [mailto:[email-masked]] 
> Sent: Friday, June 22, 2007 12:13 PM
> To: [email-masked]
> Subject: RE: [nfsv4] Re: pNFS: Open issues (from chapter 12 & 
> 13 reviews)
> 
> 
> > Wouldn't it be simpler to state in the spec thhat the 
> client MUST OPEN 
> > a file before using it, regardless of pNFS. Because, I don't see how
> 
> We do indeed plan to ban special stateids for pNFS; my 
> understanding was we agreed to that during the NFSv4.1 layout 
> inspection last week.
> 

Yes, but there was some confusion about banning them for pNFS only
versus banning them altogether. I personally prefer the latter.

> Every other recallable object has some kind of volatile 
> handle attached to it. It seems prudent that layouts should 
> not be the exception. I worry about situations where the 
> client and server disagree about whether there is a layout in 
> force or not.
> The stateid is useful to detect issues. For instance, let's 
> say a client for some reason dropped a CB_LAYOUTRECALL 
> the client doesn't recognize the proble,. With a stateid, the 
> server can return a different stateid than it had returned 
> He knows that the future LAYOUTCOMMIT might not commit what 
> he thought would get committed.
>

Ok. I understand now. Is it valid for the client to use the layout
stateid to access the file on the DS? If I understand correctly, the
layout stateid is independent of the stateid used for file access and so
the answer to my question should be "no".

> Yes; I thought that was clear from "at most one device 
> stateid per { client ID, layout type }. :-) You guys are 
> really bumming me out about my apparent inability to 
> articulate myself. :-(.
> 

I did understand this. I was just trying to be a 100% sure. Here's my
worry... The entire mapping seems to have 1 stateid. Which means that
either the entire mapping is valid or the entire mapping is invalid.
Doesn't that seem a bit coarse grained to you?

Thanks
Regards
Rahul

---

From: Mike Eisler <[email-masked]>
Date: Fri, 22 Jun 2007 14:43:08 -0700 (PDT)

> versus banning them altogether. I personally prefer the latter.

If there is WG consensus we can move the all 1s / all 0s
stateids to SHOULD NOT in this iteration, and then 
eliminate them in NFSv4.2. For new features
like pNFS it is OK to make them MUST NOT.

> the answer to my question should be "no".

As proposed, only share reservation stateids, delegation stateids,
and byte range lock stateid can be used on READ and WRITE to 
data servers.

> worry... The entire mapping seems to have 1 stateid. Which means that

Yes.

> Doesn't that seem a bit coarse grained to you?

Having no stateid seemed too coarse grained. :-)

Having a stateid per device ID seemed too fine grained
(and begs the question if a device ID should simply be a stateid),
and too much last minute burden on the protocol.
Intead, I've proposed the the client and server can optionally use
notifications to remove device IDs,
change device ID to address mappings, or add a new device ID.
The server is required to fence clients from using device addresses
of removed device IDs, or from using
addresses that have been removed from device ID's address list.
The fencing is enforced until the client gets a new mapping
for the device ID, or if the device ID is simply removed,
until the client returns the layouts with the affected device ID.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 22 Jun 2007 17:55:52 -0400

Is anybody using the all-ones stateid, i.e. treating
it differently from all-zeros?  If we get rid of 
those special stateids, there might need to be some
feature to what all-ones is supposed to do (allow 
read despite an existing DENY).  Presumably this
could be done with a flag at OPEN, if people need it.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, June 22, 2007 5:43 PM
To: [email-masked]
Subject: RE: [nfsv4] Re: pNFS: Open issues (from chapter 12 & 13
reviews)

Yes.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Jun 2007 16:23:41 +0300

If we move to a global device identifiers space I'd like to request 
to change deviceid4 from uint32_t to uint64_t to make it easier
for us to map our device identifiers to this space (not that we
typically need to map more than 2^32 devices in our systems, not yet :)

Benny

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Jun 2007 17:24:41 +0300

Mike, I'm a bit concerned that your proposal for adding a layout stateid
is too radical as it changes the current model pretty significantly rather
than incrementally improving it.  Please see my inline comments below.

Benny

> Here are my ideas for each of the above two issues (presented in

Hmm, I thought we're going in a direction where the
seqid part of the stateid is incremented by the server
to enforce strict ordering of LAYOUTGETs and LAYOUTRETURNs.

> single file, and will take a layout stateid as an argument.

We currently use LAYOUTRETURN {FSID,ALL} as the final response to
a corresponding CB_LAYOUTRECALL callback.  In your proposal, what
is the use of the layout stateid in the LAYOUTRETURN and CB_LAYOUTRECALL
contexts?

> Layout stateids persist across CLOSE and DELEGRETURN.

This seems to be inconsistent with disallowing a LAYOUTGET before OPEN
which is the rationale for this proposal.

The server can always recall the layout when it gets an OPEN with any share deny
mode from another client (or just any OPEN).  Then the issue boils down
to rejecting a LAYOUTGET before OPEN when there's a share deny (or any OPEN)
on that file. I think that this requires only an open stateid argument for LAYOUTGET
and a new error code telling the client it needs to OPEN the file first.

> of tied to one or more device IDs.
Requiring a layout stateid for CB_LAYOUTRECALL is problematic
for the FSID and ALL recall types.  Although it is always possible
to recall layouts for each file separately when handling a
filesystem or system wide event it is sub-optimal for a large
number of files.

<snip>

---

From: Mike Eisler <[email-masked]>
Date: Wed, 27 Jun 2007 08:46:43 -0700 (PDT)

> to enforce strict ordering of LAYOUTGETs and LAYOUTRETURNs.

I don't recall such a discussion, but that seems like a reasonable
thing to do; consider my proposal as so amended.

> is the use of the layout stateid in the LAYOUTRETURN and

In my proposal it would be to deal with the situation where the
client for some reason missed a CB_LAYOUTRECALL. The stateid
mismatch would let the client know.  

> which is the rationale for this proposal.

I think if a client is implemented to comply with the spec on
the first open(), and then fall back to using ACCESS on
subsequent open()s then that is a degree of malicious intent
that the spec can never protect customers from.

Recalling layouts after CLOSE seems very heavy handed, and
will impact performance.

> mode from another client (or just any OPEN).  Then the issue boils

Yes but most clients won't issue OPEN with DENY modes, nor will
most servers recall after CLOSE.

> down

Which is part of my proposal.

> for the FSID and ALL recall types.  Although it is always possible

True, and in my proposal, FSID and ALL recall types are gone.

> filesystem or system wide event it is sub-optimal for a large

A similar argument can be made for delegations, but we've
managed without CB_RECALL by FSID or by client ID. My
recollection during the pNFS inspections is that client implementors
found the recall by FSID burdensome because it required them
to maintain a data structure that a server might never use.

I can see why a server might need to recall layouts by device ID,
and I can also see that the additional burden for that is low. If
the layout type of the layout includes device IDs, then both
the client and server are maintaining a device ID to layout table.

Is there something about non-NFSv4.1 layout types that makes
recall by FSID necessary? 
Is the real issue that for some layout types, it makes more sense
for sets of files to share layouts? If so, associating the
files with a common device ID would seem to be sufficient. 

Regarding the ALL recall type, we do have CB_RECALL_ANY ... issue it
with a count of zero.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Jun 2007 19:18:43 +0300

> recall by FSID necessary? 

Our use case for that is taking a snapshot of a filesystem.
I'm not sure about other layout types.  I imagine that you might want
to do it also for block based filesystems if you want to force the
snapshot of the volume into some self-consistent state.

> files with a common device ID would seem to be sufficient. 

We could definitely use layoutrecall by device ID when this device
is broken but this doesn't answer out need for efficiently recalling
layouts by fsid.

> Regarding the ALL recall type, we do have CB_RECALL_ANY ... issue it

Agreed.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 27 Jun 2007 09:41:49 -0700

> --- Benny Halevy <[email-masked]> wrote:
....
> True, and in my proposal, FSID and ALL recall types are gone.

I missed it in your proposal and I object to removing recall by FSID

> to maintain a data structure that a server might never use.

I don't think that delegation and layouts are in the same category, and 
even if they were doing it wrong because we did it wrong last time is no 
justification.  The server can change the layout as nodes leave or join 
the cluster which is a frequent enough event in a large cluster and 
requires an efficient way to recall layouts per FSID. The layout can 
change without impacting delegations or any other state because there was 
a change in the list of Data Servers.

---

From: Garth Goodson <[email-masked]>
Date: Wed, 27 Jun 2007 10:58:14 -0700

...
>> recall by FSID necessary? 

I thought one of the other original reasons for returning by FSID was to 
return all layouts on an unmount in a single call, instead of returning 
thousands of individual layouts.  But this must also be a problem with 
delegations and other open state, so maybe it is a non-issue?

-Garth

---

From: Mike Eisler <[email-masked]>
Date: Wed, 27 Jun 2007 11:08:48 -0700 (PDT)

--- Marc Eshel <[email-masked]> wrote:

> > [...] The server can change the layout as nodes leave or
> join 

If node leaves the cluster that's going affect multiple
file systems (FSIDs). Indeed, in some deployments, it
will affect all FSIDs. Moreoever, deleting a node shouldn't
necessarily affect all files of a particular FSID.

So I see the motivation, but I don't see recall by FSID as the 
best solution. Recall by device ID seems closer to the solution.

Benny suggests that the device ID be made into 64 bits. What if
it were optionally partitioned somehow, such that the upper bits were
for a "major" device ID, and the lower bits for the "minor" device
ID. When a client gets a layout, the device ID is encoded
to indicate whether the device ID is partitioned or not.
If the device ID is partitioned, then this is a clue to the
client that when a RECALL_DEVICEID CB_LAYOUTRECALL occurs,
the client must be prepared to search by the major deviceID
if the recall specifies so.

Encoding the device ID:

Alternative 1: split the the 64 bit device ID into two
32 bit fields, di_major and di_minor. If the highest order of
di_major is set to one, then this means the remaining 63 bits are 
considered flat, otherwise they are partitioned. If partitioned,
a di_minor == 0 means di_major refers to all device IDs with the
same 

Alternative 2: Allow for major and minor device IDs of different
sizes, and have single uint64_t represent the
device ID. As with alternative 1, if the higher order bit is 1,
the remaining 63 bits are a flat device ID. Otherwise, the
next 7 bit represent how many bits of the remain 56 are
for the major device ID, with the rest then being for  the minor 
device ID. As with alternative 1, a minor devide ID of zero means
the major device ID refers to all devices with the same major device
ID.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 27 Jun 2007 13:18:27 -0700

> necessarily affect all files of a particular FSID.

A better example would be when you need to umount or disable a particular 
file system for some internal problems.
 
> the major device ID refers to all devices with the same major device

So are you saying that we will have a call that or a given device id 
recall all layouts that included this device id ? That should work.

---

From: Garth Goodson <[email-masked]>
Date: Wed, 27 Jun 2007 13:41:34 -0700

> recall all layouts that included this device id ? That should work.

Isn't the problem that you still have to return them (the layouts) one 
by one since they are now represented by stateids?  Or would you propose 
that the server keep a mapping that would allow a return of this type 
with no stateid.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Wed, 27 Jun 2007 15:02:16 -0700

> >>> change without impacting delegations or any other state because 
there
> >>> was 

> that the server keep a mapping that would allow a return of this type 

Yes that would be nice, I assume that the server will have to keep layout 
list for a given client and the layout includes the device id so it will 
be possible to support in one call.
Marc.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 28 Jun 2007 10:19:42 +0300

> be possible to support in one call.

Since we don't have any identifier for a layoutrecall "process", we currently
rely on the final layoutreturn for a particular recall to echo exactly the
same parameters that were recalled to release the server's internal recall
state so if there's a layoutrecall by deviceid there should be corresponding
layoutreturn arguments.