Subject: [nfsv4] proposed RFC5661 errata: CB_NOTIFY_DEVICEID semantics

From: Christoph Hellwig <[email-masked]>
Date: Wed, 3 Sep 2014 18:08:48 +0200

Type: Technical

Section 20.12.3 says:

   NOTIFY_DEVICEID4_CHANGE
      A previously provided device-ID-to-device-address mapping has
      changed and the client uses GETDEVICEINFO to obtain the updated
      mapping.  The notification is encoded in a value of data type
      notify_deviceid_change4.  This data type also contains a boolean
      field, ndc_immediate, which if TRUE indicates that the change will
      be enforced immediately, and so the client might not be able to
      complete any pending I/O to the device ID.  If ndc_immediate is
      FALSE, then for an indefinite time, the client can complete
      pending I/O.  After pending I/O is complete, the client SHOULD get
      the new device-ID-to-device-address mappings before sending new
      I/O requests to the storage devices addressed by the device ID.

It should say:

   NOTIFY_DEVICEID4_CHANGE
      A previously provided device-ID-to-device-address mapping has
      changed and the client uses GETDEVICEINFO to obtain the updated
      mapping.  The notification is encoded in a value of data type
      notify_deviceid_change4.  This data type also contains a boolean
      field, ndc_immediate, which SHOULD be ignored by the client.
      The client may finish any outstanding I/Os that reference the
      previously provided device-ID-to-device-address mapping and SHOULD
      use GETDEVICEINFO to obtain the updated mapping for the previously
      provided device-ID-to-device-address mapping before requesting new
      layouts.

Notes:

   Clarify what DEVICEID4_CHANGE means vs layouts instead of I/Os.
   Drop the under specified ndc_immediate flag, which can't be enforced.

   Maybe add a note that NOTIFY_DEVICEID4_DELETE must be used
   for any incompatible changes that force invalidation of outstanding
   layouts?

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 3 Sep 2014 13:07:39 -0400

Hi Christoph,

>       provided device-ID-to-device-address mapping before requesting new

I assume that any reasonably sane server will recall the layouts
before changing the device id, so is the comment about finishing
outstanding I/O warranted?

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 3 Sep 2014 19:14:17 +0200

> outstanding I/O warranted?

Why would it?  By keeping the same device ID a _CHANGE implies that
we can keep existing layouts, otherwise you'd do a _DELETE which explicitly
requires recalling the layouts.

The use case for block is that we want to be able to grow a volume,
which just adds additional blocks to the device, and keeps all existing
ones valid.  Benny also mentioned a use case for objects, in which
targets change IP addresses.

I think all layouts should document for what uses cases they allow a _CHANGE
notification, and I plan to do so for the RFC5663 errata / RFC5663bis.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 3 Sep 2014 13:41:28 -0400

> notification, and I plan to do so for the RFC5663 errata / RFC5663bis.

For the "change of IP" use case, I can't see how you can complete I/O,
unless the DSes are listening on both IP addresses (in which case you
can't drop the "obsolete" address until the layout is returned
anyway).

For your case, you are saying that you are essentially just appending
entries to the bda_volumes<> array, is that correct? OK, I can see how
it would be nice to avoid a recall in that case.

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 3 Sep 2014 19:50:02 +0200

> it would be nice to avoid a recall in that case.

A little more than that.  Quoting my own mail to the nfsv4 list from
monday:

> Talking in block layout terms this is what I allow for now (and plan
> to write down as an informative errata for RFC5663 in nicer words):
>
>  - extending a SLICE volume to a larger size
>  - implicitly extending a SIMPLE volume (there is no size in it, so the
>    client has to rely on the storage device defined size)
>  - extending the last member in a CONCAT volume
>  - adding new members to the end of a CONCAT volume
>  - extending all members of a STRIPE volume by the same amount

So it's basically adding new bda_volumes<> entries in places where they don't
cause existing ones to move, or increasing the size of entries where that
doesn't cause others to move.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 3 Sep 2014 14:03:16 -0400

> doesn't cause others to move.

OK. It might be nice to capture that general requirement that all
outstanding layouts must remain valid across a NOTIFY_DEVICEID4_CHANGE
notification. I believe that requirement is already implicit both in
the existing text and in your proposed errata, however it would be
good to state it explicitly.

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 4 Sep 2014 01:33:31 +0200

> good to state it explicitly.

I'll come up with wording that makes it more explicit.

---

From: Christoph Hellwig <[email-masked]>
Date: Mon, 8 Sep 2014 01:37:26 +0200

Type: Technical

Section 20.12.3 says:

   NOTIFY_DEVICEID4_CHANGE

It should say:

   NOTIFY_DEVICEID4_CHANGE
      A previously provided device-ID-to-device-address mapping has
      changed and the client uses GETDEVICEINFO to obtain the updated
      mapping.  The notification is encoded in a value of data type
      notify_deviceid_change4.  This data type also contains a boolean
      field, ndc_immediate, which SHOULD be ignored by the client.
      The client may finish any outstanding I/Os that reference the
      previously provided device-ID-to-device-address mapping and SHOULD
      use GETDEVICEINFO to obtain the updated mapping for the previously
      provided device-ID-to-device-address mapping before requesting new
      layouts.  All outstanding layouts remain valid after a notification
      of type NOTIFY_DEVICEID4_CHANGE.  If the device-ID-to-device-address
      mapping changed in an incompatible way that would invalidate
      outstanding layouts, the server MUST recall all outstanding layouts
      and send a NOTIFY_DEVICEID4_DELETE notification instead.
      
Notes: