Subject: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt

From: David Noveck <[email-masked]>
Date: Sat, 3 Sep 2016 12:30:46 -0400

---------- Forwarded message ----------
From: <[email-masked]>
Date: Sat, Sep 3, 2016 at 12:28 PM
Subject: New Version Notification for
draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt
To: David Noveck <[email-masked]>

A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt
has been successfully submitted by David Noveck and posted to the
IETF repository.

Name:           draft-dnoveck-nfsv4-rpcrdma-rtissues
Revision:       01
Title:          Issues Related to RPC-over-RDMA Internode Round Trips
Document date:  2016-09-03
Group:          Individual Submission
Pages:          16
URL:            https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-
rpcrdma-rtissues-01.txt
Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-
rpcrdma-rtissues/
Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-
rtissues-01
Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-
rpcrdma-rtissues-01

Abstract:
   As currently designed and implemented, the RPC-over-RDMA protocol
   requires use of multiple internode round trips to process some common
   operations.  For example, NFS WRITE operations require use of three
   internode round trips.  This document looks at this issue and
   discusses what can and what should be done to address it, both within
   the context of an extensible version of RPC-over-RDMA and potentially
   outside that framework.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: karen deitke <[email-masked]>
Date: Mon, 12 Sep 2016 13:52:14 -0600

Hi Dave,

I'm struggling following this below:

    o  First, the memory to be accessed remotely is registered.  This is
       a local operation.

    o  Once the registration has been done, the initial send of the
       request can proceed.  Since this is in the context of connected
       operation, there is an internode round trip involved.  However,
       the next step can proceed after the initial transmission is
       received by the responder.  As a result, only the responder-bound
       side of the transmission contributes to overall operation latency.

    o  The responder, after being notified of the receipt of the request,
       uses RDMA READ to fetch the bulk data.  This involves an internode
       round-trip latency.  After the fetch of the data, the responder
       needs to be notified of the completion of the explicit RDMA
       operation

    o  The responder (after performing the requested operation) sends the
       response.  Again, as this is in the context of connected
       operation, there is an internode round trip involved.  However,
       the next step can proceed after the initial transmission is
       received by the requester.

    o  The memory registered before the request was issued needs to be
       deregistered, before the request is considered complete and the
       sending process restarted.  When remote invalidation is not
       available, the requester, after being notified of the receipt of
       the response, performs a local operation to deregister the memory
       in question.  Alternatively, the responder will use Send With
       Invalidate and the responder's RNIC will effect the deregistration
       before notifying the requester of the response which has been
       received.

    To summarize, if we exclude the actual server execution of the
    request, the latency consists of two internode round-trip latencies
    plus two-responder-side interrupt latencies plus one requester-side
    interrupt latency plus any necessary registration/de-registration
    overhead.  This is in contrast to a request not using explicit RDMA
    operations in which there is a single inter-node round-trip latency
    and one interrupt latency on the requester and the responder.

I'm confused by this summarization.  In the text above you indicate 3 
different places where "internode round trip is involved, yet in the 
summary you only mention 2.  What is the definition of an "internode 
round trip?"  Also its unclear to me what you mean my "in the context of 
a connected operation".

Also you mention that there are two-responder-side interupt latencies, 
are you referring to the notification of the RDMA_READ and the send 
completion queue for sending the response?  Does this interrupt latency 
come into play in the latency of the operation? Once the client side 
gets the response it can continue, even if the server thread is still 
waiting for notification of a successful send correct?

Also are you missing the interrupt latency of the send on the client? In 
addition to the interrupt latency of receiving the reply?

Karen

---

From: David Noveck <[email-masked]>
Date: Mon, 12 Sep 2016 16:24:05 -0400

> I'm confused by this summarization.

:-(.  Let's see what I can do to make this clearer.

> In the text above you indicate 3 different places where "internode round
trip is involved, yet in the summary you only mention 2.

The point I was trying to make was that, although there were three
round-trips, only two contribute to the request latency.  In some
cases, there is a round trip because an ack is sent, but because neither
the client nor the server is waiting for it.

> What is the definition of an "internode round trip?"

Any situation in which a message is sent in one direction and, after that,
another message is sent in the opposite direction.

> Also its unclear to me what you mean my "in the context of a connected
operation".

maybe I should have said, "Because this reliable connected operation in
which messages are acked.

> Also you mention that there are two-responder-side interrupt latencies,
are you referring to the notification of the RDMA_READ
> and the send completion queue for sending the response?

I'm referring to the notification that the request has been received and
and the notification that the RDMA_READ has comnpleted.

Does this interrupt latency come into play in the latency of the operation?

I think the two I mentioned do.

> Once the client side gets the response it can continue, even if the
server thread is still waiting for notification of a successful send
correct?

Yes.

I don't think that contributes to latency.  The request processing can
continue once the request is received on the server, even if the client
has not received notification of the completion of the send.

On Mon, Sep 12, 2016 at 3:52 PM, karen deitke <[email-masked]>
wrote:

> summary you only mention 2.  What is the definition of an "internode round
> trip?"  Also its unclear to me what you mean my "in the context of a
> connected operation".
>
> Also you mention that there are two-responder-side interupt latencies, are
> you referring to the notification of the RDMA_READ and the send completion
> queue for sending the response?  Does this interrupt latency come into play
> in the latency of the operation? Once the client side gets the response it
> can continue, even if the server thread is still waiting for notification
> of a successful send correct?
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: karen deitke <[email-masked]>
Date: Thu, 15 Sep 2016 12:58:13 -0600

Thanks,

That clears things up.  Next question, are you saying that an RDMA_WRITE 
is NOT an internode round trip and an RDMA_READ is an internode round 
trip, because we have to wait for the data from the RDMA_READ before 
proceeding, where with the RDMA_WRITE we do not need to wait? (i.e. 
sections 2.2 and 2.3)

Karen

On 9/12/2016 2:24 PM, David Noveck wrote:
> > In the text above you indicate 3 different places where "internode round trip is involved, 
> yet in the summary you only mention 2.
>
> cases, there is a round trip because an ack is sent, but because 
> neither the client nor the server is waiting for it.
>
> Any situation in which a message is sent in one direction and, after 
> that, another message is sent in the opposite direction.
>
> > Also its unclear to me what you mean my "in the context of a connected operation".
>
> maybe I should have said, "Because this reliable connected operation 
> in which messages are acked.
>
> > Also you mention that there are two-responder-side interrupt latencies, are you referring 
> to the notification of the RDMA_READ
> I'm referring to the notification that the request has been received 
> and and the notification that the RDMA_READ has comnpleted.
>
> > Once the client side gets the response it can continue, even if the server thread is still 
> > Also are you missing the interrupt latency of the send on the client? In addition to the 
> interrupt latency of receiving the reply?
>
> On Mon, Sep 12, 2016 at 3:52 PM, karen deitke <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Dave,
>
>     I'm confused by this summarization.  In the text above you
>     indicate 3 different places where "internode round trip is
>     involved, yet in the summary you only mention 2.  What is the
>     definition of an "internode round trip?"  Also its unclear to me
>     what you mean my "in the context of a connected operation".
>
>     Also you mention that there are two-responder-side interupt
>     latencies, are you referring to the notification of the RDMA_READ
>     and the send completion queue for sending the response?  Does this
>     interrupt latency come into play in the latency of the operation?
>     Once the client side gets the response it can continue, even if
>     the server thread is still waiting for notification of a
>     successful send correct?
>
>     Also are you missing the interrupt latency of the send on the
>     client? In addition to the interrupt latency of receiving the reply?
>
>     Karen
>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/nfsv4>
>
>

---

From: David Noveck <[email-masked]>
Date: Sat, 17 Sep 2016 06:15:00 -0400

> where with the RDMA_WRITE we do not need to wait? (i.e.
> sections 2.2 and 2.3)

Actually, that is why there is not an internode round trip which
contributes to latency.

I do distinguish in  this documentbetween round-trips which do and do not
contribute to latency.  For example, I mention certain acks which re part
of round trips that nobody has to wait for.

I believe that there is no ack for an RDMA write but I've never actually
looked on the wire to see that.  I imagine that there might be certain rare
cases in which some sort of separate ack were sent.  For example, if you
did multiple RDMA write in sequence without a SEND or there were a long
delay without the response being sent, a separate ACK of the RDMA WRITE
might be sent.

However, in the common case, in which a SEND immediately follows, the RDMA
WRITE, there is no need for a separate ACK and I believe the ACK of the
SEND suffices to assure the responder RNIC, that both
previous operations have completed successfully.

On Thu, Sep 15, 2016 at 2:58 PM, karen deitke <[email-masked]>
wrote:

> is NOT an internode round trip and an RDMA_READ is an internode round trip,
> because we have to wait for the data from the RDMA_READ before proceeding,
> where with the RDMA_WRITE we do not need to wait? (i.e. sections 2.2 and
> 2.3)
>
> Karen
>
>
>> come into play in the latency of the operation? Once the client side gets
>> the response it can continue, even if the server thread is still waiting
>> for notification of a successful send correct?
>>

---

From: karen deitke <[email-masked]>
Date: Tue, 20 Sep 2016 17:27:03 -0600

On 9/17/2016 4:15 AM, David Noveck wrote:
> I do distinguish in  this documentbetween round-trips which do and do 
> not contribute to latency.  For example, I mention certain acks which 
> re part of round trips that nobody has to wait for.
>
> I believe that there is no ack for an RDMA write but I've never 
> actually looked on the wire to see that.  I imagine that there might 
> be certain rare cases in which some sort of separate ack were sent.  
> For example, if you did multiple RDMA write in sequence without a SEND 
> or there were a long delay without the response being sent, 
> a separate ACK of the RDMA WRITE might be sent.
>
> However, in the common case, in which a SEND immediately follows, the 
> RDMA WRITE, there is no need for a separate ACK and I believe the ACK 
> of the SEND suffices to assure the responder RNIC, that both 
I'm not seeing that this is how it works on solaris unless I'm 
misunderstanding something.  We can potentially issue multiple 
RDMA_WRITES for one NFS READ, but we do wait for the last RDMA_WRITE to 
complete, then go on to RDMA SEND the reply.  We need to wait for the 
write to confirm it completed successfully before sending the reply.  If 
the write fails for some reason we cant send success in the reply.
Karen
>
> On Thu, Sep 15, 2016 at 2:58 PM, karen deitke <[email-masked] 
>     That clears things up.  Next question, are you saying that an
>     RDMA_WRITE is NOT an internode round trip and an RDMA_READ is an
>     internode round trip, because we have to wait for the data from
>     the RDMA_READ before proceeding, where with the RDMA_WRITE we do
>     not need to wait? (i.e. sections 2.2 and 2.3)
>
>     Karen
>
>
>>     > In the text above you indicate 3 different places where "internode
>>     round trip is involved, yet in the summary you only mention 2.
>>
>>     The point I was trying to make was that, although there were
>>     three round-trips, only two contribute to the request latency. 
>>     In some
>>     Any situation in which a message is sent in one direction and,
>>     after that, another message is sent in the opposite direction.
>>
>>     maybe I should have said, "Because this reliable connected
>>     operation in which messages are acked.
>>
>>     I'm referring to the notification that the request has been
>>     received and and the notification that the RDMA_READ has comnpleted.
>>
>>     server thread is still waiting for notification of a successful
>>     send correct?
>>
>>     Yes.
>>
>>     I don't think that contributes to latency.  The request
>>     processing can continue once the request is received on the
>>     server, even if the client
>>     On Mon, Sep 12, 2016 at 3:52 PM, karen deitke
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         Hi Dave,
>>
>>         definition of an "internode round trip?" Also its unclear to
>>         me what you mean my "in the context of a connected operation".
>>
>>         latencies, are you referring to the notification of the
>>         RDMA_READ and the send completion queue for sending the
>>         response?  Does this interrupt latency come into play in the
>>         latency of the operation? Once the client side gets the
>>         response it can continue, even if the server thread is still

---

From: David Noveck <[email-masked]>
Date: Tue, 20 Sep 2016 21:27:50 -0400

> misunderstanding something.

Let's just go forward based on your presentation of what Solaris.

I think if you that are misunderstanding something, it is how RDMA WRITE
works on the wire.  I think you are assuming it works the way that I
thought it worked when I wrote rtissues-00.

Based on some comments I received, I switched to the different approach in
rtissues-01, which I now believe to be correct.  I want to make sure my
understaning is correct.  Of course rtissues-00 and rtissues-01 can bothe
be wrong but it is impossible for the both to be correct.

>We can potentially issue multiple RDMA_WRITES for one NFS
> READ, but we do wait for the last RDMA_WRITE to complete,

The question is if you get the completion indication when do you get it?  I
(as of rtissues-01) believe that it means send of the RDMA data has
finished and it does't mean the data has actually been stored on
the destitution.  For that to happen there would have to be an additional
internode round trip which rtissues-01 says doesn't exist.  rtissues-00,
supposed that it did exist but my current understanding is that that is
wrong.

> then go on to RDMA SEND the reply.

OK.

>We need to wait for the write to confirm it completed successfully >
before sending the reply.

You don't from a sequencing point of view.  Given ordered delivery, the
reply cannot get there first.

> If the write fails for some reason we cant send success in the
> reply.

My understanding is that if the RDMA_WRITE fails, the send will not succeed
and that you could queue the request to do the RDMA_WRITE and the send at
the same time.  To me, that seems counterintuitive, but I think that is the
way RC works

On Tue, Sep 20, 2016 at 7:27 PM, karen deitke <[email-masked]>
wrote:

> misunderstanding something.  We can potentially issue multiple RDMA_WRITES
> for one NFS READ, but we do wait for the last RDMA_WRITE to complete, then
> go on to RDMA SEND the reply.  We need to wait for the write to confirm it
> completed successfully before sending the reply.  If the write fails for
> some reason we cant send success in the reply.
> Karen
>
>
>> > Also are you missing the interrupt latency of the send on the client?
>> In addition to the interrupt latency of receiving the reply?
>>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 21 Sep 2016 10:33:21 -0700

> On Sep 20, 2016, at 6:27 PM, David Noveck <[email-masked]> wrote:
> 
> I think if you that are misunderstanding something, it is how RDMA WRITE works on the wire.  I think you are assuming it works the way that I thought it worked when I wrote rtissues-00.
> 
> Based on some comments I received, I switched to the different approach in rtissues-01, which I now believe to be correct.  I want to make sure my understaning is correct.  Of course rtissues-00 and rtissues-01 can bothe be wrong but it is impossible for the both to be correct.
> 
> The question is if you get the completion indication when do you get it?  I (as of rtissues-01) believe that it means send of the RDMA data has finished and it does't mean the data has actually been stored on the destitution.  For that to happen there would have to be an additional internode round trip which rtissues-01 says doesn't exist.  rtissues-00, supposed that it did exist but my current understanding is that that is wrong.
> 
> >We need to wait for the write to confirm it completed successfully > before sending the reply.  
> 
> You don't from a sequencing point of view.  Given ordered delivery, the reply cannot get there first.
> 
> My understanding is that if the RDMA_WRITE fails, the send will not succeed and that you could queue the request to do the RDMA_WRITE and the send at the same time.  To me, that seems counterintuitive, but I think that is the way RC works

This is how I believe this to work (I think it aligns with your
responses above). I know I may have said other things before, but
my understanding has improved over time.

RDMA ordering guarantees and the RPC-over-RDMA protocol design allows
an RPC-over-RDMA responder to build each RPC Reply as a single chain
of WRs:

  Write WR (Write chunk payload)
  Write WR (Write chunk payload)
  Write WR (Reply chunk payload)
   ...
  Send WR

Which can be posted via one post_send. Implementations I'm aware of
are not yet this efficient, but do post Writes and Sends in this
order.

If the Write WRs are signaled, each completion fires when the
responder's HCA has sent the Write payload. A Write completion means
only that the responder's HCA is done with the payload buffer(s).
It says nothing about whether that payload was received by the
requester's HCA or whether the requester's HCA has placed that
payload in the requester's memory. In other words, a responder
cannot rely on Write completion to gauge the success of its Writes
on the requester.

The Send WR may also be signaled. Its completion means only that the
responder's HCA is done with the consumer's Send buffer. It also has
little relationship to requester activity (although the HCA may delay
Send completion if there are retransmits going on).

RDMA Read and Write is available only with a Reliable Connection. I
believe both HCAs rely on the RC message sequence numbers to verify
that all payloads and requests have been received in order.

The requester's HCA fires a completion when it is ready to hand the
incoming RPC Reply message to the RDMA consumer via a Receive. RDMA
ordering guarantees that all previous Write payloads are placed in
the requester's memory before that completion fires. Otherwise, the
the requester's HCA flushes the Receive in order to report a problem.

A failure on the requester is communicated back to the responder via an
RDMA ACK with a Syndrome code and a disconnect request (IB). The QP is
taken out of Ready-To-Send, in-progress and pending WRs are flushed on
both ends, and queue processing stops.

The RPC Reply is lost in the case and the content of the registered
Write chunk memory on the requester is indeterminate. The RPC client
is responsible for establishing a new connection and QP, re-registering
memory associated with the RPC request, and retransmitting that RPC
with the fresh STags.

> On Tue, Sep 20, 2016 at 7:27 PM, karen deitke <[email-masked]> wrote:
> 
> 
>> Actually, that is why there is not an internode round trip which contributes to latency.
>> 
>> I do distinguish in  this documentbetween round-trips which do and do not contribute to latency.  For example, I mention certain acks which re part of round trips that nobody has to wait for.  
>> 
>> I believe that there is no ack for an RDMA write but I've never actually looked on the wire to see that.  I imagine that there might be certain rare cases in which some sort of separate ack were sent.  For example, if you did multiple RDMA write in sequence without a SEND or there were a long delay without the response being sent, a separate ACK of the RDMA WRITE might be sent. 
>> 
>> However, in the common case, in which a SEND immediately follows, the RDMA WRITE, there is no need for a separate ACK and I believe the ACK of the SEND suffices to assure the responder RNIC, that both previous operations have completed successfully.
> I'm not seeing that this is how it works on solaris unless I'm misunderstanding something.  We can potentially issue multiple RDMA_WRITES for one NFS READ, but we do wait for the last RDMA_WRITE to complete, then go on to RDMA SEND the reply.  We need to wait for the write to confirm it completed successfully before sending the reply.  If the write fails for some reason we cant send success in the reply.  
> Karen
> 
>> 
>> On Thu, Sep 15, 2016 at 2:58 PM, karen deitke <[email-masked]> wrote:
>> Thanks,
>> 
>> That clears things up.  Next question, are you saying that an RDMA_WRITE is NOT an internode round trip and an RDMA_READ is an internode round trip, because we have to wait for the data from the RDMA_READ before proceeding, where with the RDMA_WRITE we do not need to wait? (i.e. sections 2.2 and 2.3)
>> 
>> Karen
>> 
>> 
>>> > In the text above you indicate 3 different places where "internode round trip is involved, yet in the summary you only mention 2.  
>>> 
>>> The point I was trying to make was that, although there were three round-trips, only two contribute to the request latency.  In some 
>>> cases, there is a round trip because an ack is sent, but because neither the client nor the server is waiting for it.
>>> 
>>> Any situation in which a message is sent in one direction and, after that, another message is sent in the opposite direction.
>>> 
>>> maybe I should have said, "Because this reliable connected operation in which messages are acked.
>>> 
>>> > Also you mention that there are two-responder-side interrupt latencies, are you referring to the notification of the RDMA_READ 
>>> I'm referring to the notification that the request has been received and and the notification that the RDMA_READ has comnpleted.
>>> 
>>> > Once the client side gets the response it can continue, even if the server thread is still waiting for notification of a successful send correct?
>>> 
>>> Yes.
>>> 
>>> > Also are you missing the interrupt latency of the send on the client? In addition to the interrupt latency of receiving the reply?
>>> 
>>> I don't think that contributes to latency.  The request processing can continue once the request is received on the server, even if the client
>>> On Mon, Sep 12, 2016 at 3:52 PM, karen deitke <[email-masked]> wrote:
>>> Hi Dave,
>>> 
>>> I'm confused by this summarization.  In the text above you indicate 3 different places where "internode round trip is involved, yet in the summary you only mention 2.  What is the definition of an "internode round trip?"  Also its unclear to me what you mean my "in the context of a connected operation".
>>> 
>>> Also you mention that there are two-responder-side interupt latencies, are you referring to the notification of the RDMA_READ and the send completion queue for sending the response?  Does this interrupt latency come into play in the latency of the operation? Once the client side gets the response it can continue, even if the server thread is still waiting for notification of a successful send correct?
>>> 
>>> Also are you missing the interrupt latency of the send on the client? In addition to the interrupt latency of receiving the reply?
>>> 
>>> Karen
>>> 
>>> 

--
Chuck Lever