Subject: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

From: Karen <[email-masked]>
Date: Fri, 20 May 2016 10:11:43 -0600

Hi Chuck,

Here is some feedback on draft-ietf-nfsv4-rpcrdma-bidirection-03

Karen

Abstract section:

The use of the word "recent" seems relative to time.  5 years later, 
recent will no longer be recent.  Not sure how much of a big deal this 
is, just seems maybe instead of putting "recent minor versions of 
NFSv4", a more precise explanation would be "With the introduction of 
NFSv4.0".

2.1

/"This traditional form of ONC RPC message passing is referred to as 
operation in the "forward direction"."/

"as operation in the forward direction" seems awkward.  "is referred to 
as the "forward direction"." seems a bit more concise and clearer.

2.2

Same as in section 2.2 in /"This form of message passin is referred to 
as operation in the "backward direction."/

Also again the use of the word recently, as in "Not until recently", 
seems that "Not until NFSV4.0" would be clearer.

3.1

/"An NFSv4 "delegation" is simply a promise made by a server that it 
will notify a client before another agent is allowed access to a file"/

another agent? seems "a different client" would be clearer.

3.2

/"Without a backward direction RPC-over-RDMA capability, an NFSv4.1 
client must additionally connect using a transport with backward 
direction capability to use as a backchannel."/

I understand the point, and agree, but the sentence seems mumbled.  
Maybe its confusing to me because of the use of the word backward 
direction.  Maybe backward direction needs to be defined as a connection 
that is initiated by a requestor, but also used to accept requests from 
the respondor, in which the requestor then reponds?  Not sure if that 
wording is any clearer, but the idea that the requests are sent from the 
side that did not create the connection?

4

"/For an RDMA Send operation to work, the receiving peer must have 
posted an RDMA Receive Work Request (WR) to provide a receive buffer in 
which to land the incoming message./

why is the posted receive buffer being called a posted receive work 
request?  Seems this would be clearer

"For an RDMA Send operation to work, the receiving peer must have posted 
an RDMA receive buffer as the location in which the RDMA send request 
will land."

And similarly

/"If a receiver hasn't posted enough Receive WRs to land incoming Send 
operations..."/

If a receiver hasn't posted enough receive buffers to accept incoming 
send requests"

(Why is Receive capitalized?)

4.1

/"When message direction is not fully determined by context"/

"fully determined by context" thats confusing, what does that really 
mean?  I think this means when the rdma header does not directly 
indicate if the message is a call or reply, but the wording is confusing.

4.2

/"If a sender transmits a message for a receiver which has no prepared 
receive buffer"/

the word "posted" not "prepared" was used earlier, to be consisted 
"posted" would read more clearly than "prepared"

4.2.2

/"A forward direction RPC-over-RDMA service endpoint"/

//

did you mean "server" instead of "service"?

/"To receive incoming backward direction replies, an RPC-over-RDMA 
server endpoint must pre-post a receive buffer for each backward 
direction Call it sends"/

What prevents a buggy, or hacking client from sending more requests that 
forward credits and preventing a reply in the backward direction from 
arriving?  Could a buggy or hacky client cause a denial of service by 
sending more than credit number of requests?  Guess that could happen in 
the standard single direction and would result in the connection 
closing.  Maybe not an issue?

5.1

last sentence,/"and the header's msg_type field MUST conatin the value 
CALL",
/

for more clarification "and the RPC header's msg_type field", this is a 
nit...

5.4

/"If an ONC RPC client has no work to do, it may be some time before it 
re-establishes a transport connection. Backward direction Requesters 
must be prepared to wait indefinitely for a connection to be established 
before a pending backward direction ONC RPC Call can be retransmitted."/

I don't believe there are currently an non-idempotent backchannel 
operations, but if there ever were to be, seems that the client should 
detect if it has any drc replies and re-create the bi-directional, or 
even one way backchannel connection, right after it detects the 
connection is  gone.

Not addressed, and obvious to the implementer, but should it be 
mentioned that send buffers in the backward direction MUST be the same 
size as the posted receive buffers on the receiving side? That is the 
backward connection must use the same send/recv buffer sizes and this 
can not be negotiated?

---

From: Chuck Lever <[email-masked]>
Date: Fri, 20 May 2016 12:41:15 -0400

Hi Karen-

Thanks for the review. I'll respond below to questions
and non-editorial comments. I'll merge the other comments
before the next revision.

> On May 20, 2016, at 12:11 PM, Karen <[email-masked]> wrote:
> 
> Hi Chuck,
> 
> The use of the word "recent" seems relative to time.  5 years later, recent will no longer be recent.  Not sure how much of a big deal this is, just seems maybe instead of putting "recent minor versions of NFSv4", a more precise explanation would be "With the introduction of NFSv4.0".
> 
> 
> 2.1
> 
> "This traditional form of ONC RPC message passing is referred to as operation in the "forward direction"."
> 
> "as operation in the forward direction" seems awkward.  "is referred to as the "forward direction"." seems a bit more concise and clearer.
> 2.2 
> Same as in section 2.2 in "This form of message passin is referred to as operation in the "backward direction."
> 
> Also again the use of the word recently, as in "Not until recently", seems that "Not until NFSV4.0" would be clearer.
> 
> 3.1 
> "An NFSv4 "delegation" is simply a promise made by a server that it will notify a client before another agent is allowed access to a file"
> 

A server may also host application programs, which can
open files too. When a local application opens a file
delegated to a client, the server would recall that
delegation.

So "agent" means other clients, or programs running on
the server.

> 3.2
> 
> "Without a backward direction RPC-over-RDMA capability, an NFSv4.1 client must additionally connect using a transport with backward direction capability to use as a backchannel."
> 
> I understand the point, and agree, but the sentence seems mumbled.  Maybe its confusing to me because of the use of the word backward direction.  Maybe backward direction needs to be defined as a connection that is initiated by a requestor, but also used to accept requests from the respondor, in which the requestor then reponds?  Not sure if that wording is any clearer, but the idea that the requests are sent from the side that did not create the connection?

Isn't the definition in section 2.2 sufficient?

> 4
> 
> "For an RDMA Send operation to work, the receiving peer must have posted an RDMA Receive Work Request (WR) to provide a receive buffer in which to land the incoming message.
> 
> why is the posted receive buffer being called a posted receive work request?  Seems this would be clearer 
> "For an RDMA Send operation to work, the receiving peer must have posted an RDMA receive buffer as the location in which the RDMA send request will land."

Each receive buffer is posted by posting a Receive Work
Request on the QP's receive queue. "Posting a receive
buffer" is a more colloquial way of saying it, posting
a Receive Work Request is perhaps more precise.

An argument could be made for switching to one terminology
or the other throughout the document. But which one is
more clear? Opinions?

> And similarly
> 
> 
> "If a receiver hasn't posted enough Receive WRs to land incoming Send operations..."
> 
> If a receiver hasn't posted enough receive buffers to accept incoming send requests"
> 
> "When message direction is not fully determined by context"
> 
> "fully determined by context" thats confusing, what does that really mean?  I think this means when the rdma header does not directly indicate if the message is a call or reply, but the wording is confusing.

That means that in some cases the receiver can guess
accurately which direction the message was going, based
on the context of the operation, even without having
an RPC message payload.

I'm open to suggestions.

I think some prefer that the document simply state that
direction is always unknown in cases where an RPC
message payload is not present. That kind of opens a
can of worms with RDMA_ERROR, which is needed to report
that the client does not support backward direction
operation.

> 4.2
> 
> "If a sender transmits a message for a receiver which has no prepared receive buffer"
> 
> the word "posted" not "prepared" was used earlier, to be consisted "posted" would read more clearly than "prepared"
> 
> 4.2.2 
> "A forward direction RPC-over-RDMA service endpoint"
> 
> 
> "To receive incoming backward direction replies, an RPC-over-RDMA server endpoint must pre-post a receive buffer for each backward direction Call it sends"
> 
> What prevents a buggy, or hacking client from sending more requests that forward credits and preventing a reply in the backward direction from arriving?  Could a buggy or hacky client cause a denial of service by sending more than credit number of requests?  Guess that could happen in the standard single direction and would result in the connection closing.  Maybe not an issue?

The same issue exists in the forward direction.

> 5.1
> 
> last sentence, "and the header's msg_type field MUST conatin the value CALL", 
> 
> for more clarification "and the RPC header's msg_type field", this is a nit...
> 
> 5.4
> 
> "If an ONC RPC client has no work to do, it may be some time before it re-establishes a transport connection. Backward direction Requesters must be prepared to wait indefinitely for a connection to be established before a pending backward direction ONC RPC Call can be retransmitted."
> 
> I don't believe there are currently an non-idempotent backchannel operations, but if there ever were to be, seems that the client should detect if it has any drc replies and re-create the bi-directional, or even one way backchannel connection, right after it detects the connection is  gone.

Does backward direction RPC/TCP do this?

It certainly would be an issue for a backward-only connection.
The client would have to notice that the connection was down
and re-establish it quickly.

> Not addressed, and obvious to the implementer, but should it be mentioned that send buffers in the backward direction MUST be the same size as the posted receive buffers on the receiving side?  That is the backward connection must use the same send/recv buffer sizes and this can not be negotiated?

Well, the I-D used to say that the buffer sizes are the
same in both direction, but I guess I chopped it out at
some point. I'll have to go back and look for it.

--
Chuck Lever

---

From: Karen <[email-masked]>
Date: Fri, 20 May 2016 12:30:39 -0600

> So "agent" means other clients, or programs running on
Ah, I see, ok, thanks, the term agent in this sense was confusing to me.
> Isn't the definition in section 2.2 sufficient?
Re-read that section, yes this is sufficient.
> or the other throughout the document. But which one is
Right, both are used and should be consistent.  Speaking for me, 
"receive buffer" is clearer than "receive work request".
> that the client does not support backward direction
I don't think that it can always absolutely be clear which direction 
without the RPC header. Or am I misunderstanding RPC payload? I'm taking 
that to mean the RPC header, but maybe you are referring to the NFS data 
in the rpc payload?
> The same issue exists in the forward direction.
Right, agreed, guess its part of the nature of ib.
> Does backward direction RPC/TCP do this?
No, but only because there aren't any non-idempotent bc ops, and as 
such, I'm not sure we even had a backchannel drc.
> and re-establish it quickly.
Yes, but again, we'd need a ulp that needs drc in the backward 
direction.  Not sure there are any, but not away of any upcoming nfs bc 
changes either.

> some point. I'll have to go back and look for it.
Ok.

Karen

---

From: Chuck Lever <[email-masked]>
Date: Fri, 20 May 2016 14:44:45 -0400

> On May 20, 2016, at 2:30 PM, Karen <[email-masked]> wrote:
> 
> 
> 
> Right, both are used and should be consistent.  Speaking for me, "receive buffer" is clearer than "receive work request".
>> 
>> 
>>> And similarly
>>> 
>>> 
> I don't think that it can always absolutely be clear which direction without the RPC header. Or am I misunderstanding RPC payload? I'm taking that to mean the RPC header, but maybe you are referring to the NFS data in the rpc payload?

"Context" here means that if the receiver can tell
by other means (like, there are no other outstanding
operations). So no, it's not always going to be clear.
In those cases, direction is not known.

> No, but only because there aren't any non-idempotent bc ops, and as such, I'm not sure we even had a backchannel drc.
>> 
> Yes, but again, we'd need a ulp that needs drc in the backward direction.  Not sure there are any, but not away of any upcoming nfs bc changes either.

A DRC or session is going to be an upper layer affair;
that is, IMO not the transport's responsibility.

And for NFSv4.1, detecting loss of the backchannel is
also an upper layer thing: the server asserts
SEQ4_STATUS_CB_PATH_DOWN.

The issue in this document is that somehow, the client
has to detect the missing connection quickly in case
the server wants to send it backchannel requests. The
exact mechanism by which this occurs must be specified
somewhere else.

I'll review this text and see if it can be improved.

--
Chuck Lever

---

From: Karen <[email-masked]>
Date: Fri, 20 May 2016 12:51:35 -0600

> In those cases, direction is not known.
I don't think there is ever a 100% way to know the direction without the 
rpc header's call or reply field.  Even if there is an outstanding 
request, and it is for the xid that is defined in the rdma header, being 
that the xid is not unique between client and server, there still 
exists, though extremely small, a possiblity that this is a new request 
that just so happens to have the same xid and is not actually the reply 
to an outstanding operation.
> SEQ4_STATUS_CB_PATH_DOWN.
Agreed.
> I'll review this text and see if it can be improved.
Ok, thanks.

---

From: David Noveck <[email-masked]>
Date: Fri, 20 May 2016 14:56:35 -0400

> Not addressed, and obvious to the implementer, but should it be mentioned
that send buffers in the backward direction MUST be the same size as the
posted receive buffers on the receiving side?

It is not clear to me that you actually need send *buffers *as such.  from
an efficiency point of view, there is no point in copying what you want
into a buffer.  The send operation can simply told the memory from which
the send is to occur, including the possibility of a gather list.

What may be obvious to the implementer, and might need to be mentioned is
that any message sent MUST be no longer that the peer's receive buffer.

> That is the backward connection must use the same send/recv buffer sizes
and this can not be negotiated?

I don't see why the receive buffers on the two peers need to be the same.
As long as each side knows the other's receive buffer size, the protocol as
specified works  Right now, in Version One, each must be at least 1K and
are likely to be the same.  The situation is the same according to
draft-cel-nfsv4-rpcrdma except that the size is 4K.
 draft-dnoveck-nfsv4-rpcrdma-xcharext proposes a discovery mechanism to
allow longer receive buffer to be used and, as currently specified, each
side (independently) picks its own receive buffer size and communicates it
to the peer.  Although the word "negotiation" is often used, there is no
true negotiation anticipated.  Although there is a mechanism for one side
to state a preference for a different size. I expect implementation to take
a here's-the-size-take-it-or-leave-it approach.

On Fri, May 20, 2016 at 2:30 PM, Karen <[email-masked]> wrote:

>>> recent will no longer be recent.  Not sure how much of a big deal this is,
>>> just seems maybe instead of putting "recent minor versions of NFSv4", a
>>> more precise explanation would be "With the introduction of NFSv4.0".
>>>
>>>
>>> 2.1
>>>
>>> "An NFSv4 "delegation" is simply a promise made by a server that it will
>>> notify a client before another agent is allowed access to a file"
>>>
>>> "Without a backward direction RPC-over-RDMA capability, an NFSv4.1
>>> client must additionally connect using a transport with backward direction
>>> capability to use as a backchannel."
>>>
>>> that is initiated by a requestor, but also used to accept requests from the
>>> respondor, in which the requestor then reponds?  Not sure if that wording
>>> is any clearer, but the idea that the requests are sent from the side that
>>> did not create the connection?
>>>
>>> an RDMA Receive Work Request (WR) to provide a receive buffer in which to
>>> land the incoming message.
>>>
>>> an RDMA receive buffer as the location in which the RDMA send request will
>>> land."
>>>
> Right, both are used and should be consistent.  Speaking for me, "receive
> buffer" is clearer than "receive work request".
>
>>
>>
>> And similarly
>>>
>>>
>>> mean?  I think this means when the rdma header does not directly indicate
>>> if the message is a call or reply, but the wording is confusing.
>>>
> that to mean the RPC header, but maybe you are referring to the NFS data in
> the rpc payload?
>
>>
>>
>> 4.2
>>>
>>> "To receive incoming backward direction replies, an RPC-over-RDMA server
>>> endpoint must pre-post a receive buffer for each backward direction Call it
>>> sends"
>>>
>>> the standard single direction and would result in the connection closing.
>>> Maybe not an issue?
>>>
>>> "If an ONC RPC client has no work to do, it may be some time before it
>>> re-establishes a transport connection. Backward direction Requesters must
>>> be prepared to wait indefinitely for a connection to be established before
>>> a pending backward direction ONC RPC Call can be retransmitted."
>>>
>>> detect if it has any drc replies and re-create the bi-directional, or even
>>> one way backchannel connection, right after it detects the connection is
>>> gone.
>>>
> No, but only because there aren't any non-idempotent bc ops, and as such,
> I'm not sure we even had a backchannel drc.
>
>>
> Yes, but again, we'd need a ulp that needs drc in the backward direction.
> Not sure there are any, but not away of any upcoming nfs bc changes either.
>
>
>>
>> Not addressed, and obvious to the implementer, but should it be mentioned
>>> that send buffers in the backward direction MUST be the same size as the
>>> posted receive buffers on the receiving side?  That is the backward
>>> connection must use the same send/recv buffer sizes and this can not be
>>> negotiated?
>>>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 20 May 2016 15:15:01 -0400

> On May 20, 2016, at 2:51 PM, Karen <[email-masked]> wrote:
> 
> 
> 
> I don't think there is ever a 100% way to know the direction without the rpc header's call or reply field.

I don't think what I wrote contradicts that statement, but
it does allow latitude for innovation to close this hole
in other ways.

> Even if there is an outstanding request, and it is for the xid that is defined in the rdma header, being that the xid is not unique between client and server, there still exists, though extremely small, a possiblity that this is a new request that just so happens to have the same xid and is not actually the reply to an outstanding operation.

The only case in RPC-over-RDMA Version One where this is
a concern is RDMA_ERROR. The other two valid procs are
RDMA_NOMSG and RDMA_MSG, and both of those have RPC
message payloads.

When a client receives an RDMA_ERROR, that normally means
one of its forward requests had a problem.

Typical RPC-over-RDMA Version One client implementations
don't send an error to a server due to a problem with a
forward reply. We can probably say the same about a
server sending an RDMA_ERROR to a client in response
to a bad backward reply. (And, rfc5666bis could be
enhanced to suggest or require that requesters don't
send RDMA_ERROR in response to a bogus reply).

So the only instance where a server might receive an
RDMA_ERROR message is when it has sent a backward
request that the client did not like.

Thus: context indicates what direction that RDMA_ERROR
message is going.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Mon, 23 May 2016 10:10:51 -0400

> On May 20, 2016, at 3:15 PM, Chuck Lever <[email-masked]> wrote:
> 
> 

OK, this idea is growing on me.

What do people think of updating rfc5666bis to restrict
RDMA_ERROR to be sent only by responders? This would be
for RPC-over-RDMA V1 only, of course.

The reason to do this would be to eliminate the ambiguity
of the meaning of the XID and credit value, due to the
absence of an RPC message payload with a direction field
in it.

I'm not aware of any current requester implementation
that can send an RDMA_ERROR message from its reply
handler.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 23 May 2016 11:00:36 -0400

I think the better alternative is to document the ambiguity and ask people
to live with it.  Version One's error handling is broken and it doesn't seem
as though it can be fixed without changing the XDR, and we've decided that
all
such changes are to bump the version number.

The idea is to ask the sender not to set the credit value and the receiver
not
to interpret it.

The problem with restricting when this is sent is that it assumes, that if
you receive a message, you can tell if it is a request or a reply.  For a
well-formed
message that can be determined, but the idea here is we are dealing with a
message that is so messed up  that you can't even XDR the transport header.

In we now look at section 5.5.2 and look at the examples there:

an invalid value in the rdma_proc field

Without that, you can't even parse the header so, you don't where to look
for the call/reply inidication.

an RDMA_NOMSG message that has no chunk lists

Here, there is no payload to look at.

or the contents of the rdma_xid field might not match the contents of
the XID field in the accompanying RPC message.

In this case, you can determine which you have.

On Mon, May 23, 2016 at 10:10 AM, Chuck Lever <[email-masked]>
wrote:

> >>>>>> "fully determined by context" thats confusing, what does that
> really mean? I think this means when the rdma header does not directly
> >> Even if there is an outstanding request, and it is for the xid that is
> defined in the rdma header, being that the xid is not unique between client
> and server, there still exists, though extremely small, a possiblity that
> this is a new request that just so happens to have the same xid and is not
> actually the reply to an outstanding operation.
> >

---

From: Chuck Lever <[email-masked]>
Date: Mon, 23 May 2016 11:39:49 -0400

> On May 23, 2016, at 11:00 AM, David Noveck <[email-masked]> wrote:
> 
> as though it can be fixed without changing the XDR, and we've decided that all

I'm not proposing an XDR change, and there would be no
behavior change required of current implementations. No
current implementation sends RDMA_ERROR in response to
a bogus reply message. This is strictly documentation
of existing implementation behavior.

> The idea is to ask the sender not to set the credit value and the receiver not
> to interpret it.

The sender is going to set the credit value no matter
what. There's nothing that says otherwise. (You haven't
mentioned the sender side change before: that _would_ be
a change to existing implementations, and to rfc5666bis).

> you receive a message, you can tell if it is a request or a reply.  For a well-formed 

Not necessarily. The RDMA_ERROR message is used most
typically in two important cases where XDR is parseable:

1. To report an RPC-over-RDMA protocol version mismatch

2. When a backward direction request has non-empty chunk
lists and the responder has no support for chunks

The case where an actual parsing problem occurs is useful
mostly during early development of an implementation. It
happens in practice only when there is a catastrophic
fabric or RNIC failure that corrupts RDMA Send data
content.

> Without that, you can't even parse the header so, you don't where to look for the call/reply indication.
> or the contents of the rdma_xid field might not match the contents of the XID field in the accompanying RPC message. 

How should a requester behave when it gets such a
malformed reply message? Sending an RDMA_ERROR message
to the responder seems ineffective. The requesters I'm
familiar with drop such garbage replies and report an
error, which seems reasonable to me.

Responders are required to copy the rdma_vers field to
replies. If the requester sent version 1 but doesn't
recognize version 1 in replies, something is very wrong.

The XID field in an RDMA_ERROR message is valuable: when
sent from a responder, it tells the requester that the
responder was not able to process that XID. The requester
is then free to try again or terminate the transaction.
I believe making that field unambiguous is a good thing,
and worth making the proposed change.

Essentially I'm saying that RDMA_ERROR is always a REPLY.
That is the way current implementations treat it, thus we
should document that.

This also disambiguates the credit value, but that's
much less important. I don't have a strong opinion about
ignoring that field on receipt of an RDMA_ERROR.

> On Mon, May 23, 2016 at 10:10 AM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 23 May 2016 12:23:56 -0400

> If the requester sent version 1 but doesn't

It certainly would be, but if you want to say that you only
send this when you get a request, it is unclear how one might
determine this.  If you support Version One, and you get a
version field which is not one, you have no idea what the XDR
for that version looks like and thus you have no way to find out
whether out whether the message contains a request or reply.

In practical terms, if you are a server, then you might well assume that
the first message you receive likely to be a request, but future
versions might add initialization/setup transactions that you are not
prepared to parse.  Maybe Version two needs to be modified so
that clients always send a (possibly NULL) request before using
extensions.

> That is the way current implementations treat it, thus we

OK, but there are going to be situations in which the receiver cannot
determine whether it has received a request or a reply.  I don'y know
exactly how to deal with those situations (you might say he MAY send
RDMA_ERROR), but I have a problem with writing the spec as if
such situations don't exist.

On Mon, May 23, 2016 at 11:39 AM, Chuck Lever <[email-masked]>
wrote:

> > >>>> I don't think that it can always absolutely be clear which
> direction without the RPC header. Or am I misunderstanding RPC payload? I'm
> taking that to mean the RPC header, but maybe you are referring to the NFS
> data in the rpc payload?
> > >> Even if there is an outstanding request, and it is for the xid that
> is defined in the rdma header, being that the xid is not unique between
> client and server, there still exists, though extremely small, a possiblity
> that this is a new request that just so happens to have the same xid and is
> not actually the reply to an outstanding operation.
> > >

---

From: David Noveck <[email-masked]>
Date: Fri, 27 May 2016 08:24:56 -0400

I wrote:
> but I have a problem with writing the spec as if

I've been thinking more about this.  I think the problem lies in what I
mean by
"the spec".  If that means "rfc566bis", then such situations cannot exist.
rfc566bis seems to only deal with forward direction operation.  In such
an environment,
if you are a client and you receive a message, it has to be a reply.
Similarly, if you are
server and you receive a message, it has to be a request.  So there is no
occasion for the
kind of ambiguity I'm concerned about.

On the other hand, if "the spec" means "the specification of RPC-over-RDMA
Version One",
in which multiple directions of operation can be supported, the ambiguity
is unavoidable. If
you (client or server) receive a message you do not know a priori, whether
it is a request or
a reply and there are situations in which the receiver cannot determine,
given a malformed
message, whether it has received a malformed request or reply.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 27 May 2016 10:01:01 -0400

> On May 27, 2016, at 8:24 AM, David Noveck <[email-masked]> wrote:
> 
> I wrote:
> I've been thinking more about this.  I think the problem lies in what I mean by 
> rfc566bis seems to only deal with forward direction operation.  In such an environment, 
> if you are a client and you receive a message, it has to be a reply.  Similarly, if you are 
> server and you receive a message, it has to be a request.  So there is no occasion for the
> On the other hand, if "the spec" means "the specification of RPC-over-RDMA Version One",
> in which multiple directions of operation can be supported, the ambiguity is unavoidable. If 
> you (client or server) receive a message you do not know a priori, whether it is a request or 
> a reply and there are situations in which the receiver cannot determine, given a malformed 

I left the existing language in rpcrdma-bidirection.
So Section 4.1 still has:

>    When message direction is not fully determined by context or by an
>    accompanying RPC message with a call direction field, it is not
>    possible to tell whether the header credit value is a request or
>    grant, or whether the value applies to the forward direction or
>    backward direction.  In such cases, the receiver MUST NOT use the
>    header’s credit value.

Adding the "RDMA_ERROR is responder-to-requester only"
to rfc5666bis, plus this from Section 5.1:

>    The rdma_vers field MUST contain the same value in backward and
>    forward direction Call messages on the same connection.

And this from Section 6:

>    Therefore, an Upper Layer Protocol consumer MUST NOT perform backward
>    direction ONC RPC operations unless the peer consumer has indicated
>    it is prepared to handle them.

All serve to close the largest windows of ambiguity we
know about. But true enough, there may be others.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 27 May 2016 13:33:24 -0400

> But true enough, there may be others.

There are others but I'm coming to the conclusion that it would be too
disruptive to try to address them in Version One :-(

Fortunately, we don't have to since these are very unusual situations :-)

For example, "RDMA_ERROR is responder-to-requester only" does not admit an
ambiguity in a one-direction-only context.

The problem is that, in a multi-direction context, a given implementation
only knows whether it is a client or
a server.  It only knows if it is a requester or a responder based on the
message it is processing at any given
instant and if it can't figure that out, it is unsure whether it is a
requester or responder and may be neither at
particular times.

Sigh!