Subject: Re: [nfsv4] End-to-end corruption checks

From: <[email-masked]>
Date: Sat, 3 Sep 2011 14:57:08 -0400

So to give some substance to this discussion, basically to enable Tom and me
to figure out whether we are in violent agreement or vigorous disagreement, 
Let me propose some XDR (with minimal verbiage) and we can see if this is 
What Tom meant.  My goal here is to enable the stuff I was talking about, 
as an option without in any way interfering with the existing ADB stuff at all.

So is this what Tom was sort of thinking of, and if not, how?  Anybody
can answer but Tom's answers to questions regarding his own thinking have 
to be considered dispositive in the event of conflict :-)

    /// /*
    ///  * Use an enum such that we can extend new types.
+   ///  *
+   ///  * assuming for now, that additions require a minor version.
+   ///  *
+   ///  * allowing new formats via IANA and  stds-track RFC would be
+   ///  * nice but we would not want to thus allow an arm to be added
+   ///  * to unions that are switched on a data_content4.
+   ///  *
+   ///  * one approach to allow a new type if it is treated exactly the
+   ///  * as an exiting type with unions defined by a minor version.
    ///  */
    /// enum data_content4 {
    ///         NFS4_CONTENT_DATA = 0,
    ///         NFS4_CONTENT_APP_BLOCK = 1,
-   ///         NFS4_CONTENT_HOLE = 2
+   ///         NFS4_CONTENT_HOLE = 2,
+   ///         NFS4_CONENT_DATA_WITH_PI = 3
    /// };  

    ///  * We use data_content4 in case we wish to
    ///  * extend new types later. Note that we
-   ///  * are explicitly disallowing data.
+   ///  * are explicitly disallowing data, defined as a
+   ///  * data_content4 value not explicitly listed below, so
+   ///  * the default are will never be used
    ///  */
    /// union initialize_arg4 switch (data_content4 content) {
    /// case NFS4_CONTENT_APP_BLOCK:
    ///         app_data_block4 ia_adb;
    /// case NFS4_CONTENT_HOLE:
    ///         hole_info4      ia_hole;
    /// default:
    ///         void;
    /// };

    /// union read_plus_content switch (data_content4 content) {
    /// case NFS4_CONTENT_DATA:
    ///         opaque          rpc_data<>;
    /// case NFS4_CONTENT_APP_BLOCK:
    ///         app_data_block4 rpc_block;
    /// case NFS4_CONTENT_HOLE:
    ///         hole_info4      rpc_hole;
+   /// case NFS4_CONTENT_DATA_WITH_PI:
+   ///         opaque          rpc_wpidata<>;
+   ///         opaque          rpc_pinfo<>;
    /// default:
    ///         void;
    /// };

+   /// struct WRITE_PLUS4args {
+   ///         /* CURRENT_FH: file */
+   ///         stateid4        wpa_stateid;
+   ///         offset4         wpa_offset;
+   ///         stable_how4     wpa_stable;
+   ///         count4          wpa_count;
+   ///         write_plus_content wpa_content;
+   /// };
+   ///
+   /// /*
+   ///  * Could have mad this the mirror image of INITIALIZE and not
+   ///  * allow non-data but decided not to.
+   ///  *
+   ///  * Punching a hole is allowed but adbs would require too much
+   ///  * data structure surgery, at least for now.
+   ///  * 
+   /// union write_plus_plus_content switch (data_content4 content) {
+   /// case NFS4_CONTENT_DATA:
+   ///          opaque          wpc_data<>;
+   /// case NFS4_CONTENT_HOLE:
+                void;          /* range from wpa_offset and wpa_count */
+   /// case NFS4_CONTENT_DATA_WITH_PI:
+   ///          opaque         wpc_wpidata<>;
+   ///          opaque         wpc_pinfo<>;
+   /// default:
+   ///    
+   /// union WRITEPLUS4res switch (nfsstat4 status) {
+   ///  case NFS4_OK:
+   ///          WRITE4resok    wpr_resok4;  /* sharing is good */
+   ///  default:
+   ///          void;
+   /// };

    ///  OP_READ_PLUS           = 65,
+     ///  OP_WRITE_PLUS          = 66,
    ///
    ///  case OP_READ_PLUS:     READ_PLUS4args opread_plus;
+   ///  case OP_WRITE_PLUS:    WRITE_PLUS4args owrite_plus;

-   ///  /* Operations not new to NFSv4.1 */
+   ///  /* Operations not new to NFSv4.2 */

    ///  case OP_READ_PLUS:      READ_PLUS4res opread_plus;
+   ///  case OP_WRITE_PLUS:     WRITE_PLUS4res opwrite_plus;
    ///
-   ///  /* Operations not new to NFSv4.1 */
+   ///  /* Operations not new to NFSv4.2 */

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
Sent: Monday, August 29, 2011 2:10 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] End-to-end corruption checks

On Aug 29, 2011, at 11:02 AM, <[email-masked]> wrote:

> Your questions couldn't have been all that rhetorical, since you came
> up with some answers.
> 
> Now I want to make sure that I understand the answers.
> 
>> Yes, but define how you want to detect corruption and where?
> 
> It doesn't matter where (and how intensively) I would want to check for
> corruption.  People differ in the value that they place on their
> data's correctness and their degree of "paranoia" (in quotes because 
> I'm not trying to judge levels of pathology here; anything the user is
> willing to pay for works for me).
> 
> I think our architecture should accommodate user choice to the maximum 
> degree possible.  Some people have always taken a dim view of "wasting" 
> effort on correctness.  Recall "Parity is for farmers"?
> 
> The things I don't want is to have some valid/useful sort of checking
> that is available in block environments but not in NFS environments.    
> 
>> Do you want end-to-end checking? Then no, not provided.
> 
> I understand it is not provided.  The issue I'm concerned about is whether
> it is disallowed/prevented.  I really don't want that latter to be
> the case.
> 
> Are you saying it is?

I'm saying as it currently stands, we have no way for the application/client
to tell the server to do checking.

As you point out, we can do that. And the new arm of the discriminated
union that I mentioned would do the same thing.

> 
> As far as I can see, if the ADB allows you to define this virtual
> disk format (e.g each 256-byte block of data consists of 239 bytes of
> data and 17 bytes of checksum -- two prime numbers :-), with the 
> checksum format accessible, there is no protocol reason that bits
> in the adb can't specifically allow you to:
> 1) deputize the client to check data correctness at write time, if
>   the user is willing to pay the cost.
> 2) allow the server to be deputize to cheek data correctness on write
>   and to have the disk hardware check that.
> 3) allow a NFSv2/v4.0,v4.1 client to cheek the data validity when it
>   is read by someone who doesn't support the adb format.
> 
> Note that all of the above would be optional.
> 
>> Do you want the application (the VM manager in this case) to be
>> able to detect corruption on every read? Then yes, it is provided.
> 
> Personally, I would find that fine.  Being the cynic that I am, I'd
> rather have the client/server/fs/disks know nothing about how to make
> the data correct.  If they know what it takes to check it, they will
> know how to make it (look) correct, which may mean that they can somehow 
> inadvertently make the data look correct, when it shouldn't be.  
> 
> 
> -----Original Message-----
> From: Thomas Haynes [mailto:[email-masked]] 
> Sent: Friday, August 26, 2011 5:49 PM
> To: Noveck, David
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] End-to-end corruption checks
> 
> I've asked somewhat rhetorical questions until I get to the payload at
> the end, i.e., I know where you are going and I basically have tried
> to file in some history.
> 
> Right now the ADB work provides a generic framework to allow
> enterprise applications to detect corruption at their level. Just as
> we eventually decided split holes out of the ADB format and modify
> READ_PLUS to handle sparse files explicitly, we could also provide
> a form of READ_PLUS (and WRITE_PLUS) for other use cases
> that are willing to pay the price of corruption checks at every layer
> of the network stack (application, client OS, server OS, and server
> storage layer, etc.).
> 
> But just as we finally reached consensus to not force normal sparse
> files to pay the price for ADB, we should maintain the current ADB
> format for enterprise applications which are interested in performance
> over corruption checks at every layer.
> 
> 
> On Aug 26, 2011, at 12:19 PM, <[email-masked]> <[email-masked]> wrote:
> 
>> A lot of the description of adb currently assumes (or sometimes explicitly
>> States) that this is for hole-equivalent data.  Given that, it is natural
>> to assume that it only works for the special case of an empty virtual disk.
>> 
>> Is that a correct assumption.
> 
> No, it also works for a database file.
> 
>> 
>> The important issue is that if the framework provided allows
>> 1) "any application to describe that virtual disk"
>> 2) quick initialization of it, and to 
>> 3) detection of corruption based on the state of the block.
> 
> Yes, but define how you want to detect corruption and where?
> 
> Do you want end-to-end checking? Then no, not provided.
> 
> Do you want the application (the VM manager in this case) to be
> able to detect corruption on every read? Then yes, it is provided.
> 
> 
> 
>> 
>> What is missing that most virtual disks provide?
>> 4) ability to write data in the virtual disk (and provide pi data)
> 
> Somewhat agreed, for application data checking (the 'a' in 'ADB'),
> it is assumed that initialization writes (via INITIALIZE) can cover
> a range of the file, if not the entire file. Likewise, a hole punch can
> be accomplished by INITIALIZE.
> 
> When actual data is placed in the block, the application reads the
> ADB format and places the pi data in the appropriate place. As this
> is *not* end-to-end semantics, that suffices and the normal WRITE
> op is trivial to use.
> 
> 
>> 5) ability to read from the virtual disk  (and get and check pi data)
>> 
>> So what is the state of adb and do we need a separate adb+9 to use
>> disk pi facilities.
> 
> It meets the requirements of describing a block format that the application
> can use to detect corruption.
> 
> It does not meet a requirement that allows the server and client NFS
> implementations (or indeed even the underlying storage layer) to
> detect corruption in flight.
> 
> The new READ_PLUS framework would make adding this easier,
> i.e., we don't need READ_PLUS_PLUS. We would need a WRITE_PLUS,
> but I'm think we might want to bite that bullet in 4.2 anyway.
> 
> 
> 
> 
> 
> 
>> 
>> As an example, if we want the file equivalent of the oracle block-oriented
>> stuff that somebody posted a pointer to (there's a lot of powerpontification 
>> but there is some real data there too), how could we do it?  
>> 
>> I'd like to understand people's position about the role of adb in doing this.
>> Some possibilities:
>> A) the adb approach can do this given a new adb format and some editorial work
>>  like changing all the text that suggests adb's and holes are basically alike.
> 
> 
> I want to change the text anyway. :->
> 
> 
> 
>> B) The adb approach can be built on but needs to be rethought for this 
>>  different sort of challenge, leading to an adb+9 that is designed
>>  differently and incorporates adb as a subset.
>> C) You should drop adb for this and start fresh.
>> 
>> I'm pretty sure A) is not right but I'm basically guessing.
>> 
>> I'm not sure if Tom is arguing for A) or B) or whether he is just saying that C) is wrong.
>> 
> 
> I'm arguing more towards A. We can think of the current ADB as:
> 
> header layout information (what are the offsets for info, how big are they, how big is the block, etc.)
> logical block number
> state to detect corruption
> data
> 
> We could easily add a new arm in the discriminated union to describe another ADB as:
> 
> header layout information
> logical block number
> enum describing the algorithm to calculate corruption
> state for corruption
> data
> 
> The enum would be an IANA registry of algorithmic approaches to detecting corruption. (Okay,
> one could be "application" and indicate that the client, server, and storage not do any checks.)
> 
> For example, one could be CRC and the layers could all do an agreed upon crc check to
> detect corruption since the last layer. We would probably want to define a new error code
> for corruption.
> 
> And as you suggest above, both WRITE_PLUS and READ_PLUS would have to understand
> this encapsulation.
> 
> 
> 
>> 
>> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dean Hildebrand
>> Sent: Wednesday, May 18, 2011 3:58 PM
>> To: Thomas Haynes
>> Cc: [email-masked]
>> Subject: Re: [nfsv4] End-to-end corruption checks
>> 
>> 
>>> 
>>> But this framework we are presenting does allow for end-to-end checksums
>>> to be built on top of it. We could define a new ADB format in a future 
>>> version
>>> and it could provide the location, length, and algorithm needed to compute
>>> the end-to-end checksum.
>>> 
>>> The ADB approach can be thought of as a virtual disk in a file. We provide
>>> a framework which allows any application to describe that virtual disk,
>>> to allow quick initialization of it, and to detect corruption based on the
>>> state of the block.
>>> 
>> ADB can only detect corruption in 'empty' blocks correct?  How does it 
>> detect corruption is blocks that have data in them?
>> 
>> Dean
>>> 
>>> 
>>> 
>>> 
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>> 
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Thomas Haynes <[email-masked]>
Date: Sat, 3 Sep 2011 19:53:52 -0500

The main difference is that I envisioned an enum which would tell
the different layers how to compute the rpc_pinfo. From what I read
when doing some research, it seems everyone has a favorite
algorithm.

I like the XDR down below, it is forcing me to think. The only other
tweak I can think of is to always allow for the end-to-end checks,
regardless of the data type:

enum pi_check4 {
  NFS4_PI_CHECK_NONE = 0,
  NFS4_PI_CHECK_CRC = 1,
...
};

struct WRITE_PLUS4args {
         /* CURRENT_FH: file */
        stateid4        wpa_stateid;
        offset4         wpa_offset;
        stable_how4     wpa_stable;
         count4          wpa_count;
         write_plus_content wpa_content;
         pi_check4     wpa_check;
         opaque          wpa_pinfo<>;
 };

That would allow us to add pi to ADB or data.

---

From: Thomas Haynes <[email-masked]>
Date: Sat, 3 Sep 2011 23:51:36 -0500

How were you envisioning the PI data would be saved? In the file, some
hidden file, or not at all?

The ADB allows for this by describing the block and both the offset and length
of where metadata for the block resides.

---

From: <[email-masked]>
Date: Tue, 6 Sep 2011 17:03:22 -0400

As the intent is to pass this through to Protection-Information-enabled SCSI storage, the answer is some version of "in the file".

Note that there are multiple SCSI formats for protection information, and multiple ways in which it can be used, some of which allow regeneration of the protection information on read (long term storage is not required).  For an example related to the latter, see the ATO bit in the SCSI Control Mode page (SPC-3 or a current draft of SPC-4).

I'm somewhat concerned about how much of the SCSI control structure for protection information will have to be imported into NFSv4 to make this functionality useful.

Thanks,
--David

> Sent: Sunday, September 04, 2011 12:52 AM
> To: Thomas Haynes
> Cc: [email-masked]

---

From: <[email-masked]>
Date: Wed, 7 Sep 2011 11:37:26 -0400

I think we are in violent agreement as far as such an enum
goes;  it is great to envision somebody else doing it!
It does eventually have to be done, though.  With the adb
Describing the data format, there is a similar need but it is not as
critical in that there is no interoperability issue in that model.
Still, if the goal is for the adb to describe the data format, some
way of specifying where a checksum resides and how it is computed makes
sense.

I know everybody has a favorite algorithm but billions of algorithms are
too many.  Most likely it will be like songs.  Everybody has his own
favorite but there are relatively few at the top of the charts.

Q: what is the checkum-equivaelnt of rap music? :-)

BTW, the PI should include not only the data but desired location info as well.
The goal is to be able to detect cases where the data block is stored in the
wrong place.

If there is such an enum, it would not be in the WRITE_PLUS.  The idea of each
write choosing its own protection scheme is not a good one, from the implementation
sanity point of view.  This should be uniform for a file or maybe a filesystem.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
Sent: Saturday, September 03, 2011 8:54 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] End-to-end corruption checks

enum pi_check4 {
  NFS4_PI_CHECK_NONE = 0,
  NFS4_PI_CHECK_CRC = 1,
...
};

struct WRITE_PLUS4args {
         /* CURRENT_FH: file */
        stateid4        wpa_stateid;
        offset4         wpa_offset;
        stable_how4     wpa_stable;
         count4          wpa_count;

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 7 Sep 2011 13:10:10 -0500

> sanity point of view.  This should be uniform for a file or maybe a filesystem.

Okay, that makes sense. With the ADB, it is possible to have multiple block formats
within a file. But I agree that we would probably only want one enum per file.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 7 Sep 2011 21:47:41 -0500

David,

Putting words into your mouth, this appears like the target for general purpose end-to-end
checking is NFSv4.3.

As we have a valid use case for ADB and an easy upgrade path for future expansions
for READ_PLUS, I still want to deliver ADB in NFSv4.2.

In other words, I view the requirements to be orthogonal.

And by no means am I trying to keep the general purpose end-to-end checks out
of NFSv4.2. But I am claiming if we have to throw them off the train, that does not
mean we need to do so with ADB.

Thanks,
Tom