Subject: [nfsv4] pNFS: Device ID Mappings Issue

From: Mike Eisler <[email-masked]>
Date: Tue, 17 Apr 2007 12:49:23 -0700 (PDT)

In the inspection of section 12 of draft-10, an issue arose
regarding when a client can rely on the mappings returned by 
GETDEVICELIST and GETDEVICEINFO.

As you may recall from a previous email from me, I've determined that
mappings between device IDs and device addresses are leased, and
this is explicitly noted in draft-10. 

The question then is when can a client depend on what GETDEVICELIST
and GETDEVICEINFO return.

My intepretation is that a device ID to device address mapping
from GETDEVICELIST or GETDEVICEINFO is valid only if there is already
in force a layout granted to the same client that refers to the
same deviceID. Hencefore, as long as the layout (and any other 
furture or present layout) is in force, the mapping exists.
Lease expiration, metadata server restart, and returning all layouts
that refer to that device ID cause the mapping to go away.

If you do agree, what does this mean for
GETDEVICELIST? The justification for GETDEVICELIST is:

"In some applications, especially SAN environments, it is convenient to
find out about all the devices associated with a file system. This lets
a client determine if it has access to these devices, e.g., at mount
time."

If a device ID to device address mapping can go away at anytime
(unless a layout on the device ID is in force), then is
GETDEVICELIST of any use?

----------------

Now let's assume you disagree about my interpretation that a
device ID to device address mapping can be valid even if there is
no layout in force on that device ID. Then what do people
suggest we do about it? It seems to me we would need a CLOSEDEVICELIST
operation to free up a list up mappings. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Sam Falkner <[email-masked]>
Date: Tue, 17 Apr 2007 14:09:29 -0600

> that refer to that device ID cause the mapping to go away.

In other words, assume the simple case, where client only does:

<mount>
LAYOUTGET
GETDEVICEINFO on a device ID in the above layout
LAYOUTRETURN the full layout segment that was granted above
LAYOUTGET on the same file, getting the same layout segment

Upon the second LAYOUTGET, the client cannot depend on the device  
info obtained via GETDEVICEINFO to still be valid, right?  I (and I  
suspect others) have been assuming that it would be valid, in the  
absence of an intervening CB_LAYOUTRECALL of type FSID or ANY.

I'm not saying that this is necessarily a good assumption, it's just  
what I've been assuming.

> GETDEVICELIST of any use?

Yes, if the only way to hold a device ID mapping valid is to hold a  
layout containing the given device ID, then GETDEVICELIST is useless.

Do you see CLOSEDEVICELIST being a callback, or is it something a  
client would do to alleviate burden on the metadata server of keeping  
all device ID mappings valid?

- Sam

---

From: Benny Halevy <[email-masked]>
Date: Wed, 18 Apr 2007 00:11:08 +0300

> operation to free up a list up mappings. 

I'm not objecting to having an explicit CLOSEDEVICELIST operation,
but what's the justification to add it when calling CB_LAYOUTRECALL
for RECALL_FSID or RECALL_ALL already invalidates the corresponding
device mappings?  Do you anticipate a need to invalidate the mappings
while keeping the layouts (that refer to them) around?

Note that the callback can be sent by the server even when there are
no layouts at the client and we continue to say in section 19.3.4:

   Device mappings are
   invalidated also when no layouts are found for LAYOUTRECALL4_FSID or
   LAYOUTRECALL4_ALL and NFS4ERR_NOMATCHING_LAYOUT is returned.

This addresses the state implicitly created by GETDEVICEINFO and 
GETDEVICELIST that is disassociated from the layout that references
it.  E.g. if GETDEVICELIST is called before any LAYOUTGET, or the
state left behind after the last layout had been returned (with
per-file CB_LAYOUTRETURNs

Benny

---

From: Garth Goodson <[email-masked]>
Date: Tue, 17 Apr 2007 15:21:28 -0700

This seems to make caching device IDs very hard.  Since device IDs are 
not going to be changing constantly it seems that we are optimizing for 
the uncommon case and are making the client do repeated GETDEVICEINFO 
calls for IDs whose mappings aren't changing.

Initially Device IDs weren't tied to leases and we avoided all this 
problems and had the benefits.  Could we introduce a 
CB_DEVICEID_INVALIDATE to invalidate all device IDs (and remove the 
invalidation on FSID)?  This would be a heavy weight operation, but 
would allow the client to cache mappings efficiently and allow the 
server to not track individual IDs.  However, this only solves part of 
the problem.

The other issue is that of recovery (hence the introduction of the lease 
association). Leases came into the picture to prevent device ID mappings 
changing when there was no recall path (e.g., no session to the client). 
  However, isn't tying device ID invalidation to the sequence state 
expiration good enough?  If the client is ever encounters an error 
sequence that indicates state expiration it MUST invalidate ALL device 
ID mappings.  As well, the client MUST first contact the MDS after any 
lease expiration before contacting a DS using a layout (as must be 
currently done).

In summary, I like GETDEVICELIST and I like the ability to cache device 
ID mappings.  Some of these mappings are going to be large (several KB 
in size) and I would hate for the client to have to refetch them just 
because it has returned a layout in the interim.

-Garth

---

From: Mike Eisler <[email-masked]>
Date: Tue, 17 Apr 2007 17:18:20 -0700 (PDT)

The feedback from Sam, Benny, and Garth has been valuable.

Here is my proposal that I think addresses all their concerns
plus the requirement/goal for zero to little .x changes:

- When a GETDEVICEINFO or GETDEVICELIST is done, since it is always
  done inside a session, the device ID to device address mapping(s)
  stays valid until invalidated. A
  LAYOUTGET does not have to precede or proceed GETDEVICE{INFO,LIST}.

- Device ID to device address mappings are potentially invalidated
  when the lease expires or when the metadata server resets (e.g.
  reboots). Section 12.7.2 covers this. An issue with this
  is that even if a metadata server revokes just one device ID to
  device address mapping, the client has to re-establish all the
  mappings it needs. I don't see any alternative, but I'm might
  have missed something.

- Add new cases to CB_LAYOUTRECALL's arguments:

  case LAYOUTRECALL4_FSID_DEVMAPPINGS:
  case LAYOUTRECALL4_FSID: /*existed before in draft -10*/
        fsid4       lor_fsid;
  case LAYOUTRECALL4_ALL_DEVMAPPINGS:
        void;

- Replace this text,

``   In addition, LAYOUTRECALL4_FSID and LAYOUTRECALL4_ALL specify that
all the storage device ID to storage device address mappings in the
affected file system(s) are also recalled. The respective LAYOUTRETURN
with either LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL acknowledges to the
server that the client invalidated the said device mappings. Device
mappings are invalidated also when no layouts are found for
LAYOUTRECALL4_FSID or LAYOUTRECALL4_ALL and NFS4ERR_NOMATCHING_LAYOUT
is returned. ''
  
  with something like:

   LAYOUTRECALL4_FSID_DEVMAPPINGS and LAYOUTRECALL4_ALL_DEVMAPPINGS
   specify that all the storage device ID to storage device address
   mappings in the affected file system(s) are also recalled. The 
   client MUST issue LAYOUTRETURN with either LAYOUTRETURN4_FSID or
   LAYOUTRETURN4_ALL to return all layouts that are using the said
   device mappings. The client then issues a reply to
   to CB_LAYOUTRECALL4.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 17 Apr 2007 20:54:19 -0700

This change is find with me. If I understand it right you added an option 
to recall layouts without invalidating the device mapping. I am jus not 
sure which problem did you solve.
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 17 Apr 2007 20:59:52 -0700 (PDT)

> sure which problem did you solve.

I've tried to solve the following problems:

- make GETDEVICELIST useful
- reduce the frequency of unnecessarily deleting 
  device ID to device address mappings.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 09:52:09 -0400

>    device mappings. The client then issues a reply to

If the client doesn't hold any layouts on a given fsid then how does it
know that the devices are attached to that particular fsid? ...or are we
keeping this rule that clients need to keep a device list per fsid? IIRC
we had problems justifying this on the confcall yesterday.

Also, why would the server be wanting to recall all layouts on a
filesystem unless it was going to invalidate the device list too?

Cheers
  Trond

---

From: Robert Gordon <[email-masked]>
Date: Wed, 18 Apr 2007 09:56:03 -0500

> filesystem unless it was going to invalidate the device list too?

Backing up a bit;

Are we saying that the device list itself should be imbedded within the
layout and we should not have the device id abstraction ?

(pre-coffee) Robert..

---

From: Mike Eisler <[email-masked]>
Date: Wed, 18 Apr 2007 08:20:44 -0700 (PDT)

> we had problems justifying this on the confcall yesterday.

We also had trouble reaching concensus to remove the per-fsid
stuff. So my proposal fits with is currently in draft 10.

A separate thread of discussion is needed to decide if there
is any tying of fsid to device ID 9and if so, to what
degree). If the answer is no tying, then
LAYOUTRECALL4_FSID_DEVMAPPINGS is not needed.

> filesystem unless it was going to invalidate the device list too?

Perhaps the filesystem's topology is being altered. That
doesn't mean the device ID to device address mappings will change.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 18 Apr 2007 19:07:26 +0300

> layout and we should not have the device id abstraction ?

No.  Fortunately, this hasn't come up (yet ;-)).

One idea was to get rid of GETDEVICELIST and use GETDEVICEINFO
per device ID (present in the layout).  I think there's
consensus not to do that and leave GETDEVICELIST in the protocol.

Another idea is to disassociate the device IDs from the FSID
and to have in a single device ID space (per MDS of course).
There is no consensus about that and we're going to
discuss this idea on a separate thread.

Benny

---

From: Marc Eshel <[email-masked]>
Date: Wed, 18 Apr 2007 09:10:48 -0700

Trond Myklebust <[email-masked]> wrote on 04/18/2007 06:52:09 

> we had problems justifying this on the confcall yesterday.

The reason I was asking to keep association between deviceid and fsid is 
because I wanted to be able to recall layouts and device list for a 
specific fsid. Some failures on the server can impact just one file system 
and the subset of DSs that serve that file system. From day one I asked 
for a way to invalidate just the device list, some people objected and the 
compromise was that it will be done as a side effect of recalling layouts. 
Give the new options to recall just device list without recalling the 
layouts I think that I can accept just the option of recall all device 
lists if the fsid option is causing the client implementation to much 
management problems, but I would prefer to keep both (all and per fsid) 
since it is more efficient.
Marc.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 18 Apr 2007 19:12:59 +0300

Trond Myklebust wrote:
> know that the devices are attached to that particular fsid? 

For both GETDEVICE(INFO|LIST) the current filehandle identifies the
file system.

Not that in principal, the device IDs in the current model are not
associated with any layout, but rather with a file system (that's
identified by a fsid).

To be on the safe side, the client can make sure it knows the
fsid of the current filehandle by doing a GETATTR for the fsid
in the same COMPOUND with GETDEVICE(INFO|LIST).

Benny

---

From: Benny Halevy <[email-masked]>
Date: Wed, 18 Apr 2007 19:28:50 +0300

> since it is more efficient.

Note that Mike E. proposed above that LAYOUTRECALL4_*_DEVMAPPINGS
recall both layouts and device IDs (~"specify that all ... mappings
... are _also_ recalled"~)

We can separate recalling layouts from device IDs if there's a need
and people agree it's simpler to understand and/or implement.

We can organize the device IDs differently, not by fsid.
If we do that, then I'm going to propose a model closer to what Panasas
currently implements which I think is flexible enough to satisfy everybody.
This model introduces a "device-set ID", which can be represented by a
read-only file system attribute (several file systems can be stored
on the same set of devices).
The device IDs are associated with the device-set ID.  The association
is determined at GETDEVICE* time (we can either use the current file handle
as we do today or add a devicesetid field)  and CB_LAYOUTRECALL could
recall all device ID mappings of a specified device-set ID, or all device IDs
(e.g. if the given devicesetid is zero)

---

From: Sam Falkner <[email-masked]>
Date: Wed, 18 Apr 2007 11:28:27 -0600

> filesystem unless it was going to invalidate the device list too?

Perhaps an administrator-driven event to migrate or restripe all data  
for a given fsid?  The device IDs themselves won't have to change,  
but the layouts will.  I'm sure there are other possibilities.

- Sam

---

From: Sam Falkner <[email-masked]>
Date: Wed, 18 Apr 2007 11:39:37 -0600

On Apr 17, 2007, at 6:18 PM, Mike Eisler wrote:

>   reboots). Section 12.7.2 covers this.

Up to here describes what I was assuming happened before, so if  
that's what we want, then specifying this exactly would be very helpful!

> An issue with this

>   case LAYOUTRECALL4_ALL_DEVMAPPINGS:

Here in CB_LAYOUTRECALL, we could add a case to recall one specific  
device ID mapping.

>    device mappings. The client then issues a reply to

Would the client reply to the CB_LAYOUTRECALL4 first, then get around  
to LAYOUTRETURN asap?

* * *

This seems a good proposal to me.

- Sam

---

From: Mike Eisler <[email-masked]>
Date: Wed, 18 Apr 2007 10:49:11 -0700 (PDT)

> Would the client reply to the CB_LAYOUTRECALL4 first, then get around

I agree the client could respond to CB_LAYOUTRECALL { *DEVMAPPINGS }
before returning layouts. But it can't use the layouts once it
gets CB_LAYOUTRECALL, or send requests to the affected storage devices.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 18 Apr 2007 21:05:22 +0300

> gets CB_LAYOUTRECALL, or send requests to the affected storage devices.

This is a delicate issue.  The spec currently says the client can use the
layouts until LAYOUTRETURN, and that it SHOULD not attempt to do that if
the clora_changed field is TRUE (but it can try if it wants to)

 24774	   If the clora_changed field is TRUE, then the client SHOULD not write
 24775	   and commit its modified data to the storage devices specified by the
 24776	   layout being recalled.  Instead, it is preferable for the client to
 24777	   write and commit the modified data through the metadata server.

If the client does not return the layout(s) in a timely manner, the server
is allowed to forcefully revoke the layout(s).  For some layout-types the
client will be fenced off at this point.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 18 Apr 2007 11:31:13 -0700

> (e.g. if the given devicesetid is zero)

I missed the "also" in the following:
"LAYOUTRECALL4_FSID_DEVMAPPINGS and LAYOUTRECALL4_ALL_DEVMAPPINGS
 specify that all the storage device ID to storage device address
 mappings in the affected file system(s) are also recalled"

So if we still have the requirement that invaliding deviceid can only be 
accomplished as a side effect of recalling layouts than we must keep the 
fsid deviceid association and be able to recall layout and deviceid for a 
specific file system.

A much better solution would be to separate recalling of layouts from 
deviceid like Benny suggested (and I misunderstood the proposal to be)so 
we can recall all layouts, recall all layouts for an fsid, recall all 
deviceid mapping, and recall all deviceid for an fsid. 
The last one would require the deviceid fsid association and we can argue 
if we need it or just the recall all deviceid when any deviceid mapping 
change on the MDS.
Marc.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 14:41:41 -0400

> since it is more efficient.

Specifics please, Marc!

      * For what topology of server is this a _likely_ failure scenario?
      * Why do we need to optimise the protocol for that failure
        scenario instead of allowing clients to issue just one
        GETDEVICE_LIST call per MDS under normal operation?

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 14:51:18 -0400

> but the layouts will.  I'm sure there are other possibilities.

Migration is a good point except if Marc's per-fsid device IDs are
allowed. If that is the case, then a migration event would necessarily
imply invalidating the device IDs.
As for restriping: the cost of the extra GETDEVICEINFO is negligeable
compared to the cost of recovering all the layouts after the restriping
is done (again, assuming the per-fsid device IDs).

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 15:20:01 -0400

> As for restriping: the cost of the extra GETDEVICEINFO is negligeable
					   ^^^^^^^^^^^^^ I mean "GETDEVICELIST"

---

From: Robert Gordon <[email-masked]>
Date: Wed, 18 Apr 2007 14:38:35 -0500

> discuss this idea on a separate thread.

(post coffee)

What was the intent of having the device information abstracted from  
the layout ?

   If it was intended to alter the device information but keep the  
layout
   data "the same", then presumably MDS would need to recall all  
effected
   layouts ?? -- so that doesn't seem to work..

   If it was intended to reduce the memory foot print at the client,  
then
   perhaps we should revisit that thought, since it has a effect on the
   protocol complexity..

Okay, so if we must have the level of indirection .. Then, why don't we
treat the device list as a 'special' directory, use a change attribute,
use directory delegations etc..

Robert..

---

From: Marc Eshel <[email-masked]>
Date: Wed, 18 Apr 2007 13:13:59 -0700

Trond Myklebust <[email-masked]> wrote on 04/18/2007 11:41:41 

>       * For what topology of server is this a _likely_ failure scenario?

The server can use different set of DSs from the total set of nodes in the 
cluster to service different exported file systems. Now if one DS fails 
and another node takes over for that failing DS we need ti invalidate it 
deviceid mapping. 

>         GETDEVICE_LIST call per MDS under normal operation?

Although I prefer the invalidate of deviceid per fsid this would work too, 
but only if we can just invalidate all the deviceid and not have to recall 
all layouts on all file systems for that MDS.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 18:38:14 -0400

> and another node takes over for that failing DS we need ti invalidate it 

Why? In most cases, won't the other node simply take over the identity
of the failed DS and simply host its IP address?

More to the point, though, what prevents you from simply recalling all
the existing layouts on that fsid, and have the MDS re-issue them using
the existing deviceID of the replacement DS? You have 32-bits worth of
device IDs, that is enough to address the entire IPv4 network if you
really need to.

> all layouts on all file systems for that MDS.

Why do you need to invalidate any deviceIDs at all? As long as the new
DS already exists in your deviceID space (either because you supplied it
on the first GETDEVICELIST call, or because you took over the identity
of the failing DS) then you are fine.

Alternatively, you could simply state that if the client receives a
deviceID that it doesn't previously know about when it reclaims the
layout, then it should issue a GETDEVICEINFO call in order to resolve
it. That allows you to dynamically add new deviceIDs as you seem fit.

  Trond

---

From: Marc Eshel <[email-masked]>
Date: Wed, 18 Apr 2007 17:03:04 -0700

Trond Myklebust <[email-masked]> wrote on 04/18/2007 03:38:14 

> device IDs, that is enough to address the entire IPv4 network if you

It is possible that a replacement node in the cluster will takeover the 
identity (deviceid) of a node that failed or removed but will have a 
different IP address. So yes I can do many things to solve this problem 
with different implementations, I could even use the IP address in IPv4, 
but I still would like to keep the option for the server to be able to 
change it deviceid mapping without recalling all layouts. If I understand 
your motivation here is that you would like to get rid of the fsid to 
deviceid mapping, so I already said that we can do that if we separate 
layout recall from deviceid mapping invalidation and I will be happy with 
an invalidate all mapping for an MDS that doesn't require recall of all 
layouts.
Marc.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Apr 2007 21:28:31 -0400

> an invalidate all mapping for an MDS that doesn't require recall of all 

If you are just talking about replacing a single deviceID that is dead,
then things can be made much easier: just fence off the old DS so that
in-flight read/write requests to that DS are rejected and issue a
callback to all clietns that says something along the lines of 'replace
this device info with that device info'.

If you are talking about shuffling around the entire list of deviceIDs,
then it would be up to you to set up rules for client and server
behaviour during the transition from one list of devices to another.

Trond

---

From: Benny Halevy <[email-masked]>
Date: Thu, 19 Apr 2007 10:47:41 +0300

> this device info with that device info'.

You don't have to do that.
The spirit of the current spec. as I understand it is the device IDs
are just volatile indices into the device table.  If the the device address
changes there are possibilities:  either the device keeps its ID and the
address changes or the device is given a new ID (and address).

If the device retains its ID then the mapping need not be recalled
(and the old address fenced off until the new mapping goes into effect)
if the client can discover that its talking to the wrong device at the
old address.  E.g. with OSD/iSCSI the client will lose the
iSCSI session, will reestablish it and verify the device rootid
If it discovers that, the issues a GETDEVICEINFO to the MDS to get the
new address.  I think that this approach is safer in general and
protects you also against lingering requests.

If that cannot be done, the server should recall all layouts using the
old mapping, optionally, it can recall the device address mappings as
well.  If it doesn't want to recall all mappings for a single device,
it can just generate a new ID with the new mapping, and start using
the new ID instead of the old one (and never use the old mapping
again, until all device IDs are recalled)

---

From: Robert Gordon <[email-masked]>
Date: Thu, 19 Apr 2007 08:58:38 -0500

What problem or issue did the protocol solve in providing the Device ID
abstraction from the layout ?

Robert.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Thu, 19 Apr 2007 10:45:35 -0400

> abstraction from the layout ?

some device addresses are long and complicated - for instance volume
topologies for a block based storage protocol. the deviceID gives a short
hand reference, and the complicated device addresses are parsed once per
GETDEVICEINFO/LIST as opposed to parsing for each LAYOUTGET.

-->Andy

Robert.