Subject: [nfsv4] Review of draft-ietf-nfsv4-rfc5666bis-05

From: David Noveck <[email-masked]>
Date: Sat, 23 Apr 2016 13:54:15 -0400

*General Comments*

*Overall Evaluation*

I think this document is in good shape and can go forward after a small
number of issues are addressed.

Although I have suggested a lot of changes, the vast majority of these are
basically editorial matters.  I don't know how this fits into the
"consensus" model but basically I feel that editorial matters are best left
to the editor.  When I use the word "suggest", it indicates that, while I
think the change would improve the document, my suggestion is offered on a
take-it-or-leave-it-basis and a specific response is not required.

*Important Issues*

I think the following issues do need to be addressed before proceeding

   - The issue with unions discussed in *4.4.6.2.  Unused Write Chunks*.
   - The SHOULD-vs.-MUST issue mentioned in *4.4.6.  Write Chunks*.
   - The position ambiguity discussed in *4.4.5.  Read Chunks*.

*Stylistic Issues*

There are a number of stylistic issues that I mention here either because
they may be controversial or because there are so many instances of the
issue that it doesn't make sense to address them individually.

   - The use of ASCII-art diagrams mentioned in *4.4.3.  RDMA
Segments*, *4.4.5.
   Read Chunks*, and *5.2.  Fixed Header Fields*.   These did not appear om
   RFC5666 and were added in rfc5666bis, but I'm not clear why they were added.
   - There are a large number of MUSTs in the document where it isn't clear
   that MUST is appropriate.  In some cases, it is a case of describing how
   the protocol works and certain elements being chosen, for no clear reason,
   to be described by a MUST.  In other cases, it isn't clear exactly who is
   being told he/she/it MUST do a particular thing or in which a statement of
   fact that X may occur is phrased as ".. MUST NOT assume" X will not occur.

This may be an issue in during IESG review.  In the past, the focus was on
asking people why RFC2119 terms were *not *used.  This may have happened
when RFC5666 was reviewed.  Recently, things have switched around and one
is often asked for a clear explanation of why particular uses of these
terms are appropriate.

*Comments by Section*

*1.  Introduction*

Suggest replacing the last sentence of the second paragraph by the
following:

It clarifies the role of Upper Layer Bindings within RPC-over-RDMA and
describes what they are to contain.

*1.2.  Remote Procedure Calls on RDMA Transports*

In the second sentence of the third paragraph, suggest deleting the "a" in
"a reliable".

In the second sentence of the third fourth, suggest:

   - replacing "discussed" by "presented".
   - replacing "NFSv4-related issues are discussed" by "some RDMA-related
   matters with regard to NFSv4 are dealt with"

In the last sentence of the last paragraph, suggest deleting "further"

*2.2.  Changes to the Protocol*

In the the first bullet, propose splitting up the second (run-on) sentence
as follows:

   - Replacing the comma by a period.
   - Replacing "thus" by "As a result,"

In the second bullet propose:

   - Replacing the first sentence by "There is not an adequate definition
   of RDMA_MSGP in [RFC5666], although some incomplete implementation work has
   been done".
   - Replacing the the first clause of the second sentence by "even if an
   adequate definition were provided and implementation produced"

*3.1.2.  Requesters and Responders*

Suggest adding a new paragraph after the current fourth paragraph, as
follows:

CALL messages are sent by requesters to responders to initiate an RPC
request.  REPLY messages are sent by responders to requesters to complete
processing on an RPC request.

In the existing fifth paragraph suggest replacing "or 'requester'" by "in
the role of requester".

Similarly, in the existing sixth paragraph suggest replacing "or
'responder'" by "in the role of responder".

*3.1.3.  RPC Transports*

In the last sentence suggest replacing "requesters initiate transport
connections, while responders" by "clients initiate transport connections,
while servers".

*4.1.  Transfer Models*

In the last sentence of the last paragraph suggest replacing "Other
Transfer Models" by "Transfer Models other than Read-Write"

*4.4.  XDR Encoding With Chunks *

In the first sentence of the first paragraph, suggest making the following
changes:

   - Replace "during XDR encoding it can be determined" by "it can be
   determined during XDR encoding"
   - Replace "content" by "contents".
   - Replace "directly in the" by "directly into the".

*4.4.1.  Reducing an XDR Stream*

The last sentence describes something that doesn't really happen.  I
suggest the following as a replacement

The receiver considers the data items conveyed via RDMA operations as part
of the Payload stream.  Generally, because these data items are opaque and
have been directly placed into appropriate locations, they are skipped in
the process of decoding the Payload stream.

The first sentence of the second paragraph appears misplaced  The previous
paragraph has just told you what is done.  It doesn't make sense to now
tell someone that what is done can be done.

The second sentence of the second paragraph is confusing.  Propose the
following as a possible replacement:

The piece of memory containing the portion of the data stream that is split
out and directly placed is referred  to as a "chunk".  In some contexts,
data in the RPC-over-RDMA header that describes such pieces of memory is
also referred to as a "chunk".

With regard to the last sentence, I don't understand why the data item
itself, as opposed to the payload stream, is referred to as "reduced".  It
is exactly the same size as it would be if it were transferred inline.

*4.4.3.  RDMA Segments*

In the second sentence of the first paragraph, suggest replacing "It" by
"When it chooses to do so, it".

I don't see the point of the diagram in this section.

*4.4.4.  Chunks*

Suggest the following replacement for the second sentence of the second
paragraph:

The requester MAY divide a chunk into segments using any boundaries that
are convenient.

Suggest the following replacement for the last sentence of the last
paragraph:

Only a subset of RPC-over-RDMA messages have one or more chunks associated
with it.

*4.4.5.  Read Chunks*

As is the case with the diagram on section 4.4.3, I don't see the point of
this diagram.  Unlike that case, this diagram adds confusion in that, while
it accurately shows what is on the wire, it obscures the XDR structure
(i.e, that a plain segment is a substructure of a read segment).

Citing the XDR here would be better.

Suggest replacing the second paragraph by the following;

A Read chunk is a list of one or more RDMA read segments. Each RDMA read
segment in a Read chunk consists of a position field followed by a plain
segment.

It appears to me that there is an ambiguity regarding the term position.
Basically, with all the reduction/re-insertion going on, it isn't clear
which payload stream is being referred to and thus what position within it
is appropriate.

As a way of illustrating the issue consider a COMPOUND containing two WRITE
operations and thus two read(!) chunks.  Now suppose that:

   - the payload stream contains X bytes (including the length word),
   before the bulk data.
   - The bulk data to be written in the first WRITE is 8K in length.
   - There are Y bytes following the bulk data for the first WRITE and
   preceeding the bulk data for the second WRITE.
   - The bulk data to be written in second first WRITE is 8K in length.

Clearly the position for the first read chunk is X.  The question is what
is the right position for the second read chunk.  Some possibilities:

   - X + Y
   - X + Y + 8K

*4.4.5.2.  Decoding Read Chunks*

In the first sentence, suggest replacing "transport" with "responder".

*4.4.6.  Write Chunks*

In the second paragraph, suggest adding the following sentence after the
current second sentence:

In many cases, the location of a particular XDR item in the response may
not be predictable at the time a request is issue.

The last sentence of the fifth paragraph has a number of issues:

   - The word "filled' is unclear in that data shorter that the segment
   length might be considered as not "filling" it.  Suggest replacing "not
   filled by" "receives no data from".
   - The "SHOULD" here seems wrong.  Propose replacing it by "MUST".

*4.4.6.2.  Unused Write Chunks*

A number of things should be noted about the second sentence:

   - Although the it says "the requester is required", it is not clear
   where else in the document this requirement is actually stated.
   - It is particularly important to clarify this, since, if the precise
   formulation here is definitive, every COMPOUND would need write chunk(s)
   since the response to COMPOUND is a counted array where each element is a
   switched union, that will inevitably contain arms containing DDP-eligible
   items.

My current proposal, subject to change, is to start this sentence "The
requester may be REQUIRED (by the ULB)" and then make a corresponding
addition to  Section 7.

BTW, RFC5666 contains the following sentence: "Additionally, when making an
RPC call that can result in bulk data transferred in the reply, write
chunks MAY be provided to accept the data directly via RDMA Write".

There are a number of issues with the last sentence of the lat paragraph,
which appears to have come directly from RFC5666

   - If the responder tells you a write chunk is unused, and he is telling
   the truth, I can't see why it is not valid to decide/assume that the
   corresponding buffers have not been modified.
   - I don't see now the requester assuming this, even if this is not true,
   would affect interoperability, so "MUST" is probably not valid.
   - I'm not sure exactly where the boundaries of the "RPC layer" are,
   making the whole thing even more confusing

*4.5.  Message Size.*

Given that the Protocol has a minimum size of 1K and has no means of
negotiating any other size, the second sentence which argues that the
inline threshold be left small has no point.  Suggest deleting it.

The sentence in question appears to derive from one in Section 5 in
RFC5666, which reads "Since RPC messages are frequently small, memory
savings can be achieved by posting small buffers".

*5.1. XDR Protocol Definition*

Because there is no longer a possibility of OPTIONAL extensions to Version
One :-(, suggest that "without any OPTIONAL extensions" be deleted from the
last sentence of this section.

*5.1.2.  RPC-over-RDMA Version One XDR*

There needs to be some sort of comments indicating that struct
rpc_rdma_header_padded is not to be used.  I also propose deleting the
existing comments in this structure.

In enum rdma_proc, it isn't clear what "Reserved" means. Suggest "Not to be
used on encode".

In rdma_body, if we aren't prepared to delete cases RDMA_MSGP and RDMA_DONE
comments should indicate that these are not for general use.  Propose
"Vestigial; not to be used on encode "

In rdma_msg,

   - rdma_{xid,vers,credit} should have comments indicating hat they are
   fixed for all versions.
   - rdma_body should have a comment indicating that the discriminator
   field is fixed for all versions.

*5.2.  Fixed Header Fields*

While the diagram here is accurate, it doesn't help resolve the fundamental
problem here which is that these four field are not a single XDR item put
rather a part of one.  Propose deleting it and putting in its place the
following.

These four words are part of the rdma_msg structure:

   - The first three words are the individual fields within rdma_msg.
      - The last of the four word is the first word of the rdma_body union
      and appears there as the discriminator for the switched union.
The current
      contents of this field are described in Section 5.2.4.  The position of
      this field are fixed but the contents can changed, but only via
extension,
      in subsequent RPC-over-RDMA versions.

*5.4.2. Communicating DDP-Eligibility*

In the last sentence of the last paragraph, suggest replacing "requirement"
by "requirement, beyond the constraints imposed by the Upper Layer Binding,"

---

From: Chuck Lever <[email-masked]>
Date: Mon, 25 Apr 2016 12:40:01 -0400

Hi Dave-

Thanks for your careful analysis. I'll limit my responses
below to just the major issues.

> On Apr 23, 2016, at 1:54 PM, David Noveck <[email-masked]> wrote:
> 
> General Comments
> 
> Overall Evaluation
> 
> I think this document is in good shape and can go forward after a small number of issues are addressed.
> 
> Although I have suggested a lot of changes, the vast majority of these are basically editorial matters.  I don't know how this fits into the "consensus" model but basically I feel that editorial matters are best left to the editor.  When I use the word "suggest", it indicates that, while I think the change would improve the document, my suggestion is offered on a take-it-or-leave-it-basis and a specific response is not required.
> 
> Important Issues
> 
> 	• The issue with unions discussed in 4.4.6.2.  Unused Write Chunks.
> 	• The SHOULD-vs.-MUST issue mentioned in 4.4.6.  Write Chunks.
> 	• The position ambiguity discussed in 4.4.5.  Read Chunks.

I'll respond to these three inline below.

> Stylistic Issues
> 
> There are a number of stylistic issues that I mention here either because they may be controversial or because there are so many instances of the issue that it doesn't make sense to address them individually.  
> 	• The use of ASCII-art diagrams mentioned in 4.4.3.  RDMA Segments, 4.4.5.  Read Chunks, and 5.2.  Fixed Header Fields.   These did not appear om RFC5666 and were added in rfc5666bis, but I'm not clear why they were added.

Bill suggested adding these diagrams to clarify how
the fields fit in a header, the same way as some other
protocol documents do. I was interested in improving
clarity in general, so I added them.

However, since then, I've realized that:

- NFS and RPC related specs use XDR for this purpose.
There are no recent NFS or RPC specs that contain
box diagrams that I can find.

- The diagrams are redundant, because they state
exactly what the XDR definition states but in a
more clumsy fashion.

- Anyone who reads this document has to have a
strong working knowledge of XDR anyway.

- Section 5.7 provides similar clarification,
making the box diagrams even more redundant.
This section was added for similar reasons of
clarity of explanation, and is perhaps a deeper
description of the field contents than is the
box diagrams.

- The ladder diagrams that were removed to make way
for the box diagrams convey information that is not
represented by the XDR. These have been restored
where appropriate.

Therefore I agree that the box diagrams do not add
value, and I can remove them.

> 	• There are a large number of MUSTs in the document where it isn't clear that MUST is appropriate.  In some cases, it is a case of describing how the protocol works and certain elements being chosen, for no clear reason, to be described by a MUST.  In other cases, it isn't clear exactly who is being told he/she/it MUST do a particular thing or in which a statement of fact that X may occur is phrased as ".. MUST NOT assume" X will not occur.

If I may impose on Tom Talpey to review the use of
MUST / MUST NOT in this document. As one of the
original authors of RFC 5666, he can say with some
authority whether the additional requirements in
rfc5666bis close real holes, or indeed introduce
over-specification.

> This may be an issue in during IESG review.  In the past, the focus was on asking people why RFC2119 terms were not used.  This may have happened when RFC5666 was reviewed.  Recently, things have switched around and one is often asked for a clear explanation of why particular uses of these terms are appropriate.
> 
> Comments by Section
> 
> 1.  Introduction
> 
> Suggest replacing the last sentence of the second paragraph by the following:
> 
> It clarifies the role of Upper Layer Bindings within RPC-over-RDMA and describes what they are to contain.
> 
> 1.2.  Remote Procedure Calls on RDMA Transports
> 
> In the second sentence of the third paragraph, suggest deleting the "a" in "a reliable".
> 
> 	• replacing "discussed" by "presented".
> 	• replacing "NFSv4-related issues are discussed" by "some RDMA-related matters with regard to NFSv4 are dealt with"
> 2.2.  Changes to the Protocol
> 
> In the the first bullet, propose splitting up the second (run-on) sentence as follows:
> 	• Replacing the comma by a period.
> 	• Replacing "thus" by "As a result,"
> 	• Replacing the first sentence by "There is not an adequate definition of RDMA_MSGP in [RFC5666], although some incomplete implementation work has been done".
> 	• Replacing the the first clause of the second sentence by "even if an adequate definition were provided and implementation produced"
> 3.1.2.  Requesters and Responders
> 
> Suggest adding a new paragraph after the current fourth paragraph, as follows:
> 
> CALL messages are sent by requesters to responders to initiate an RPC request.  REPLY messages are sent by responders to requesters to complete processing on an RPC request.
> 
> In the existing fifth paragraph suggest replacing "or 'requester'" by "in the role of requester".
> 
> Similarly, in the existing sixth paragraph suggest replacing "or 'responder'" by "in the role of responder".
> 
> 3.1.3.  RPC Transports
> 
> In the last sentence suggest replacing "requesters initiate transport connections, while responders" by "clients initiate transport connections, while servers".
> 
> 4.1.  Transfer Models
> 
> In the last sentence of the last paragraph suggest replacing "Other Transfer Models" by "Transfer Models other than Read-Write"
> 
> 4.4.  XDR Encoding With Chunks 
> 
> In the first sentence of the first paragraph, suggest making the following changes:
> 	• Replace "during XDR encoding it can be determined" by "it can be determined during XDR encoding"
> 	• Replace "content" by "contents".
> 	• Replace "directly in the" by "directly into the".
> 4.4.1.  Reducing an XDR Stream
> 
> The last sentence describes something that doesn't really happen.  I suggest the following as a replacement
> 
> The receiver considers the data items conveyed via RDMA operations as part of the Payload stream.  Generally, because these data items are opaque and have been directly placed into appropriate locations, they are skipped in the process of decoding the Payload stream.
> 
> The first sentence of the second paragraph appears misplaced  The previous paragraph has just told you what is done.  It doesn't make sense to now tell someone that what is done can be done.
> 
> The second sentence of the second paragraph is confusing.  Propose the following as a possible replacement:
> 
> The piece of memory containing the portion of the data stream that is split out and directly placed is referred  to as a "chunk".  In some contexts, data in the RPC-over-RDMA header that describes such pieces of memory is also referred to as a "chunk".
> 
> With regard to the last sentence, I don't understand why the data item itself, as opposed to the payload stream, is referred to as "reduced".  It is exactly the same size as it would be if it were transferred inline.

I want to use the same term to apply to both the Payload
Stream and the extracted data items, since the reduced
Payload Stream and the extracted data items are intimately
tied together; they are parts of the same message.

"Reduced" is a mathematical way to describe it, similar
to the term Map-Reduce. Reduced is used here to mean
"simplified".

I'm interested in discussing a replacement if this is
an ongoing concern. Is "decomposition" a better way to
put it?

> 4.4.3.  RDMA Segments
> 
> In the second sentence of the first paragraph, suggest replacing "It" by "When it chooses to do so, it".
> 
> 4.4.4.  Chunks
> Suggest the following replacement for the second sentence of the second paragraph:
> 
> The requester MAY divide a chunk into segments using any boundaries that are convenient.
> 
> Suggest the following replacement for the last sentence of the last paragraph:
> 
> Only a subset of RPC-over-RDMA messages have one or more chunks associated with it.
> 
> 4.4.5.  Read Chunks
> 
> As is the case with the diagram on section 4.4.3, I don't see the point of this diagram.  Unlike that case, this diagram adds confusion in that, while it accurately shows what is on the wire, it obscures the XDR structure (i.e, that a plain segment is a substructure of a read segment).
> 
> A Read chunk is a list of one or more RDMA read segments. Each RDMA read segment in a Read chunk consists of a position field followed by a plain segment.
> 
> It appears to me that there is an ambiguity regarding the term position.  Basically, with all the reduction/re-insertion going on, it isn't clear which payload stream is being referred to and thus what position within it is appropriate.
> 
> As a way of illustrating the issue consider a COMPOUND containing two WRITE operations and thus two read(!) chunks.  Now suppose that:
> 	• the payload stream contains X bytes (including the length word), before the bulk data.
> 	• The bulk data to be written in the first WRITE is 8K in length.
> 	• There are Y bytes following the bulk data for the first WRITE and preceeding the bulk data for the second WRITE.
> 	• The bulk data to be written in second first WRITE is 8K in length.
> Clearly the position for the first read chunk is X.  The question is what is the right position for the second read chunk.  Some possibilities:
> 	• X + Y
> 	• X + Y + 8K

The position field is referenced to the unreduced payload
stream. I can add text that clarifies this.

> 4.4.5.2.  Decoding Read Chunks
> 
> 4.4.6.  Write Chunks
> In the second paragraph, suggest adding the following sentence after the current second sentence:
> 
> In many cases, the location of a particular XDR item in the response may not be predictable at the time a request is issue.
> 
> 	• The word "filled' is unclear in that data shorter that the segment length might be considered as not "filling" it.  Suggest replacing "not filled by" "receives no data from".
> 	• The "SHOULD" here seems wrong.  Propose replacing it by "MUST".

My memory is fuzzy, but I think I was documenting
current practice here. That is, the existing server
implementations do not behave the same as each
other.

I agree, though, that MUST is better protocol.
I can replace the SHOULD.

> 4.4.6.2.  Unused Write Chunks
> 
> 	• Although the it says "the requester is required", it is not clear where else in the document this requirement is actually stated.

The exact text is:

For example, an Upper Layer Protocol may define a
union result where some arms of the union contain
a DDP-eligible data item while other arms do not.
The requester is required to provide a Write chunk
in this case, but if the responder returns a result
that uses an arm of the union that has no DDP-eligible
data item, the Write chunk remains unused.

The second sentence should start with:

The responder is required to use provided Write
chunks in this case, 

I can clarify that sentence.

> 	• It is particularly important to clarify this, since, if the precise formulation here is definitive, every COMPOUND would need write chunk(s) since the response to COMPOUND is a counted array where each element is a switched union, that will inevitably contain arms containing DDP-eligible items.  
> My current proposal, subject to change, is to start this sentence "The requester may be REQUIRED (by the ULB)" and then make a corresponding addition to  Section 7.

Is this change still needed?

> BTW, RFC5666 contains the following sentence: "Additionally, when making an RPC call that can result in bulk data transferred in the reply, write chunks MAY be provided to accept the data directly via RDMA Write".

If I am reading it correctly, this text is referring to
a Long Reply, not to using a Write chunk to return a
single DDP-eligible data item.

> There are a number of issues with the last sentence of the last paragraph, which appears to have come directly from RFC5666
> 	• If the responder tells you a write chunk is unused, and he is telling the truth, I can't see why it is not valid to decide/assume that the corresponding buffers have not been modified.
> 	• I don't see how the requester assuming this, even if this is not true, would affect interoperability, so "MUST" is probably not valid.
> 	• I'm not sure exactly where the boundaries of the "RPC layer" are, making the whole thing even more confusing

This sentence means that even if a Write chunk is
marked as unused, the server may have written some
data into it before deciding not to return that
data item. For example, a problem reading the local
storage can occur while a server is filling Write
chunks. This would interrupt the stream of RDMA
Write operations, but a ULP error needs to be
returned that reflects the underlying error.

If the memory region provided for that Write chunk
contained live data, the application is responsible
for ensuring the contained data is consistent before
using it, even if the Write chunk is returned unused.

A requester can't treat unused Write chunks as if
they do not contain data changes. HW caches have to
be flushed and invalidation is necessary before the
underlying memory is re-used, with the same
restrictions as would be the case with a Write chunk
that returned actual data.

That is my understanding, and I can clarify this
paragraph if others agree this is the appropriate
intention.

> 4.5.  Message Size.
> 
> Given that the Protocol has a minimum size of 1K and has no means of negotiating any other size, the second sentence which argues that the inline threshold be left small has no point.  Suggest deleting it.
> 
> The sentence in question appears to derive from one in Section 5 in RFC5666, which reads "Since RPC messages are frequently small, memory savings can be achieved by posting small buffers". 
>   
> 5.1. XDR Protocol Definition
> 
> Because there is no longer a possibility of OPTIONAL extensions to Version One :-(, suggest that "without any OPTIONAL extensions" be deleted from the last sentence of this section.
> 
> 5.1.2.  RPC-over-RDMA Version One XDR
> 
> There needs to be some sort of comments indicating that struct rpc_rdma_header_padded is not to be used.  I also propose deleting the existing comments in this structure.
> 
> In enum rdma_proc, it isn't clear what "Reserved" means. Suggest "Not to be used on encode".
> 
> In rdma_body, if we aren't prepared to delete cases RDMA_MSGP and RDMA_DONE comments should indicate that these are not for general use.  Propose "Vestigial; not to be used on encode "
> 
> In rdma_msg,
> 	• rdma_{xid,vers,credit} should have comments indicating hat they are fixed for all versions.
> 	• rdma_body should have a comment indicating that the discriminator field is fixed for all versions.
> 5.2.  Fixed Header Fields
> 
> While the diagram here is accurate, it doesn't help resolve the fundamental problem here which is that these four field are not a single XDR item put rather a part of one.  Propose deleting it and putting in its place the following.
> 
> 		• The first three words are the individual fields within rdma_msg.
> 		• The last of the four word is the first word of the rdma_body union and appears there as the discriminator for the switched union.  The current contents of this field are described in Section 5.2.4.  The position of this field are fixed but the contents can changed, but only via extension, in subsequent RPC-over-RDMA versions.
> 5.4.2. Communicating DDP-Eligibility
> 
> In the last sentence of the last paragraph, suggest replacing "requirement" by "requirement, beyond the constraints imposed by the Upper Layer Binding,"
> 
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 25 Apr 2016 15:23:30 -0400

>       • The issue with unions discussed in 4.4.6.2.  Unused Write Chunks.

I think your wording change makes that go away.

>       • The SHOULD-vs.-MUST issue mentioned in 4.4.6.  Write Chunks.

I think we've agreed on "MUST".  Unless, there is further dispute about
this, we can consider the issue closed.

>       • The position ambiguity discussed in 4.4.5.  Read Chunks.

I think your proposed clarification addresses this.

> tied together; they are parts of the same message.

The result of that is that one of the uses will wind up being
inappropriate, because, despite their connection, they have
different roles.

For example, "extracted" applies to the data items, but would not
applied to the payload stream.  It is extracted from.  "Reduced"  has
the complementary problem.

> to the term Map-Reduce. Reduced is used here to mean

But when something is reduced in that sense, there is a loss of information.
Here information is not lost but rather rearranged.

> an ongoing concern.

It isn't.

> Is "decomposition" a better way to put it?

That describes the process but it wouldn't work to described either the
payload stream or the data items as "decomposed".

> The responder is required to use provided Write

That addresses my issue, but if you make that change, the sentence will
still
need some work.  The problems is with "the Write chunk remains unused."
It may remain unused, but further DDP-eligible data item could be matched
up
with it.

> Is this change still needed?

Given your correction, no.

> > BTW, RFC5666 contains the following sentence: "Additionally, when
making an RPC call that can result in bulk data transferred in the reply,
write chunks MAY be provided to accept the data directly via RDMA Write".

> single DDP-eligible data item.

I read it differently, but at this point, it doesn't matter.

> returned that reflects the underlying error.

That's very clear and helpful.  The problem is drafting
it an way that retains the clarity while still leaving it
acceptable in a standards-track document :-(

> contained live data,

Now I'm confused.  I'm not sure what exactly what is meant
by "live data" in this context.  No sane implementation is
going to be depending on the content of data in a write chunk
between the time you issue RPC and it returns.  There is
no place for "live data" here.

> the application

or "implementation"??

> is responsible
> using it,

I'm not sure what "consistent" means in this case, or how
an implementation might ensure it,

> even if the Write chunk is returned unused.

I think the principle is that the same things that have to be done
for used chunks have to be done for unused ones.  That's OK.

> they do not contain data changes.

At the very least, he is very unwise to assume so.

HW caches have to
be flushed and

I don't see what HW Caches would need to be flushed.  In any
case, such cache flushing is not mentioned anywhere else.

> invalidation is necessary before the

Agree with that.

> paragraph if others agree this is the appropriate

I think we at least agree on the basic issue, but I'm concerned
that we may have a lingering issue about caches/consistency.

On Mon, Apr 25, 2016 at 12:40 PM, Chuck Lever <[email-masked]>
wrote:

> > Although I have suggested a lot of changes, the vast majority of these
> are basically editorial matters.  I don't know how this fits into the
> > There are a number of stylistic issues that I mention here either
> because they may be controversial or because there are so many instances of
> the issue that it doesn't make sense to address them individually.
> >       • The use of ASCII-art diagrams mentioned in 4.4.3.  RDMA
> Segments, 4.4.5.  Read Chunks, and 5.2.  Fixed Header Fields.   These did
> not appear om RFC5666 and were added in rfc5666bis, but I'm not clear why
> they were added.
>
> >       • There are a large number of MUSTs in the document where it isn't
> clear that MUST is appropriate.  In some cases, it is a case of describing
> how the protocol works and certain elements being chosen, for no clear
> reason, to be described by a MUST.  In other cases, it isn't clear exactly
> who is being told he/she/it MUST do a particular thing or in which a
> statement of fact that X may occur is phrased as ".. MUST NOT assume" X
> will not occur.
>
> > This may be an issue in during IESG review.  In the past, the focus was
> on asking people why RFC2119 terms were not used.  This may have happened
> >       • replacing "NFSv4-related issues are discussed" by "some
> RDMA-related matters with regard to NFSv4 are dealt with"
> >       • Replacing the first sentence by "There is not an adequate
> definition of RDMA_MSGP in [RFC5666], although some incomplete
> implementation work has been done".
> >       • Replacing the the first clause of the second sentence by "even
> if an adequate definition were provided and implementation produced"
> > The receiver considers the data items conveyed via RDMA operations as
> part of the Payload stream.  Generally, because these data items are opaque
> and have been directly placed into appropriate locations, they are skipped
> in the process of decoding the Payload stream.
> >
> > The first sentence of the second paragraph appears misplaced  The
> previous paragraph has just told you what is done.  It doesn't make sense
> to now tell someone that what is done can be done.
> >
> > The piece of memory containing the portion of the data stream that is
> split out and directly placed is referred  to as a "chunk".  In some
> contexts, data in the RPC-over-RDMA header that describes such pieces of
> memory is also referred to as a "chunk".
> >
> > As is the case with the diagram on section 4.4.3, I don't see the point
> of this diagram.  Unlike that case, this diagram adds confusion in that,
> while it accurately shows what is on the wire, it obscures the XDR
> structure (i.e, that a plain segment is a substructure of a read segment).
> >
> > It appears to me that there is an ambiguity regarding the term
> position.  Basically, with all the reduction/re-insertion going on, it
> isn't clear which payload stream is being referred to and thus what
> position within it is appropriate.
> >
> > As a way of illustrating the issue consider a COMPOUND containing two
> WRITE operations and thus two read(!) chunks.  Now suppose that:
> >       • There are Y bytes following the bulk data for the first WRITE
> and preceeding the bulk data for the second WRITE.
> > Clearly the position for the first read chunk is X.  The question is
> what is the right position for the second read chunk.  Some possibilities:
> >       • X + Y
> >       • X + Y + 8K
>
> >       • The word "filled' is unclear in that data shorter that the
> segment length might be considered as not "filling" it.  Suggest replacing
> "not filled by" "receives no data from".
> >       • Although the it says "the requester is required", it is not
> clear where else in the document this requirement is actually stated.
>
> The exact text is:
>
> >       • It is particularly important to clarify this, since, if the
> precise formulation here is definitive, every COMPOUND would need write
> chunk(s) since the response to COMPOUND is a counted array where each
> element is a switched union, that will inevitably contain arms containing
> DDP-eligible items.
> > BTW, RFC5666 contains the following sentence: "Additionally, when making
> an RPC call that can result in bulk data transferred in the reply, write
> > There are a number of issues with the last sentence of the last
> paragraph, which appears to have come directly from RFC5666
> >       • If the responder tells you a write chunk is unused, and he is
> telling the truth, I can't see why it is not valid to decide/assume that
> the corresponding buffers have not been modified.
> >       • I don't see how the requester assuming this, even if this is not
> true, would affect interoperability, so "MUST" is probably not valid.
> >       • I'm not sure exactly where the boundaries of the "RPC layer"
> are, making the whole thing even more confusing
>
> > Because there is no longer a possibility of OPTIONAL extensions to
> Version One :-(, suggest that "without any OPTIONAL extensions" be deleted
> from the last sentence of this section.
> >
> > In rdma_body, if we aren't prepared to delete cases RDMA_MSGP and
> RDMA_DONE comments should indicate that these are not for general use.
> Propose "Vestigial; not to be used on encode "
> >
> > In rdma_msg,
> >       • rdma_body should have a comment indicating that the
> discriminator field is fixed for all versions.
> > 5.2.  Fixed Header Fields
> >
> > While the diagram here is accurate, it doesn't help resolve the
> fundamental problem here which is that these four field are not a single
> XDR item put rather a part of one.  Propose deleting it and putting in its
> place the following.
> >
> >               • The last of the four word is the first word of the
> rdma_body union and appears there as the discriminator for the switched
> union.  The current contents of this field are described in Section 5.2.4.
> The position of this field are fixed but the contents can changed, but only
> via extension, in subsequent RPC-over-RDMA versions.
> > In the last sentence of the last paragraph, suggest replacing
> "requirement" by "requirement, beyond the constraints imposed by the Upper
> Layer Binding,"
> >
> >
>
> --
> Chuck Lever
>
>
>
>