Subject: [storm] Send vs. Immediate

From: Elena Gurevich <[email-masked]>
Date: Sun, 6 Sep 2015 09:58:42 +0000

Hello dear authors,

I am new in iWARP and need your help to clear my  misunderstanding of RFCs.
RFC5041 Section 5.4 stands that:

"At the Data Sink, DDP MUST Deliver a DDP Message if and only if all of the following are true:
* the last DDP Segment of the DDP Message had its Last flag set,
* all of the DDP Segments of the DDP Message have been Placed,
* all preceding DDP Messages have been Placed, and
* each preceding DDP Message has been Delivered to the ULP."

Let's assume that data sink receives sequence of "rdma_write" and send" messages.

According to RFC even if miss ordering happens "send" can be delivered to RDMAP only after last segment of rdma_write arrives and is placed.
So RDMAP layer completes "send" only after full rdma_write assembly.
If so why we need to generate new "Immediate request type" and cannot reuse 8 bytes length "send" ?

Best regards,
Elena Gurevich
---------------------------------------------------------
Software Architector at Toga Networks Ltd.

Email: [email-masked]
---------------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

---

From: Tom Talpey <[email-masked]>
Date: Sun, 6 Sep 2015 09:33:33 -0400

On 9/6/2015 5:58 AM, Elena Gurevich wrote:
> “At the Data Sink, DDP MUST Deliver a DDP Message if and only if all of
> the following are true:
>
> * each preceding DDP Message has been Delivered to the ULP.”
>
> Let’s assume that data sink receives sequence of “rdma_write” and send”
> messages.
>
> According to RFC even if miss ordering happens “send” can be delivered
> to RDMAP only after last segment of rdma_write arrives and is placed.
>
> So RDMAP layer completes “send” only after full rdma_write assembly.

These statements are correct, the Send is delivered only after Placing
all the previous RDMA Write segments. This is a key behavior upon which
most upper layers depend.

>
> If so why we need to generate new “Immediate request type” and cannot
> reuse 8 bytes length “send” ?

I don't understand the question. An Immediate is another behavior,
and which is used by upper layers rather differently. Also, what
"8 bytes" are you referring to?

Perhaps you mean, why doesn't the upper layer use an RDMA Write with
Immediate? Some upper layers do, I believe Lustre does, for example,
but the original iWARP RDMAP (RFC5040) protocol did not support that
operation. The RFC7306 extension, published last year, adds it.

Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need
a larger, separate message to complete an operation, and so use a full
Send. These protocols use iWARP, without Immediates.

---

From: Elena Gurevich <[email-masked]>
Date: Mon, 7 Sep 2015 09:48:38 +0000

Hello Tom, thanks for a prompt response, my question was little bit different.

I failed to found differences in behavior between RDMA Write followed by Immediate Data and RDMA Write followed by Send.
Both consume untagged buffer of queue #0 - so at data sink this buffer should be preposted in both cases.

According to RFCs except DDP message format both operation has to be processed similarly as at data Source as at data Sink.

According to ordering and completion rules, specified in RFC 5040 and  7306:

First          |Second       | Placement         | Placement       | Ordering
Operation| Operation | Guarantee at     | Guarantee at   | Guarantee at
                 |                  | Remote Peer     | Local Peer        | Remote Peer
-------------+---------------+----------------------+--------------------+----------------
RDMA      | Send          | No placement   | Not applicable | Not applicable
Write       |                   | guarantee. If     |                         |
                |                   | guarantee is      |                         |
                |                   | necessary, see   |                         |
                |                   | footnote 1.         |                         |
-------------+---------------+----------------------+--------------------+------------------
RDMA     | Immediate| No Placement   | Not                   | Immediate Data
Write      | Data          | Guarantee         | Applicable        | is Completed
               |                   |                           |                          | after RDMA
               |                   |                           |                          | Write is Placed
               |                   |                           |                          | and Delivered

But even if the cell  "Ordering Guarantee at Remote Peer" is different for RDMA Write followed by Immediate Data and RDMA Write followed by Send.
At the end behavior has to be the same, because of specification of RFC5041 Section 5.4. ( see details in my original e-mail)

So why we need Immediate Data operation and cannot just reuse Send operation ?

Best regards,
Lena

-----Original Message-----
From: storm [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Sunday, September 06, 2015 4:34 PM
To: [email-masked]
Subject: Re: [storm] Send vs. Immediate

On 9/6/2015 5:58 AM, Elena Gurevich wrote:
> According to RFC even if miss ordering happens "send" can be delivered

These statements are correct, the Send is delivered only after Placing all the previous RDMA Write segments. This is a key behavior upon which most upper layers depend.

> reuse 8 bytes length "send" ?

I don't understand the question. An Immediate is another behavior, and which is used by upper layers rather differently. Also, what
"8 bytes" are you referring to?

Perhaps you mean, why doesn't the upper layer use an RDMA Write with Immediate? Some upper layers do, I believe Lustre does, for example, but the original iWARP RDMAP (RFC5040) protocol did not support that operation. The RFC7306 extension, published last year, adds it.

Other upper layers, for example, iSER, NFS/RDMA, SMB Direct, etc, need a larger, separate message to complete an operation, and so use a full Send. These protocols use iWARP, without Immediates.

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: Tom Talpey <[email-masked]>
Date: Mon, 7 Sep 2015 11:25:25 -0400

> So why we need Immediate Data operation and cannot just reuse Send 
operation ?

Using either one is a choice by the upper layer (RDMA consumer), and
it's absolutely possible to just reuse a Send. As mentioned, many upper
layers simply use Send. Others, for various reasons, use Immediate
data. DDP, and the extended RDMAP (RFC7306) support either one.

---

From: Elena Gurevich <[email-masked]>
Date: Tue, 8 Sep 2015 05:55:54 +0000

Thanks you again,

If so, the statement "Not applicable" in cell "Ordering at Remote Peer" for RDMA Write followed by Send operation in Appendix B of RFC5040 is inaccurate
And should be changed as "Send is completed after RDMA Write placed and delivered".

Best regards,
Elena

-----Original Message-----
From: Tom Talpey [mailto:[email-masked]]
Sent: Monday, September 07, 2015 6:25 PM
To: Elena Gurevich; [email-masked]
Subject: Re: [storm] Send vs. Immediate

 > So why we need Immediate Data operation and cannot just reuse Send operation ?

Using either one is a choice by the upper layer (RDMA consumer), and it's absolutely possible to just reuse a Send. As mentioned, many upper layers simply use Send. Others, for various reasons, use Immediate data. DDP, and the extended RDMAP (RFC7306) support either one.

On 9/7/2015 5:48 AM, Elena Gurevich wrote:
> At the end behavior has to be the same, because of specification of
> RFC5041 Section 5.4. ( see details in my original e-mail)
>
>> According to RFC even if miss ordering happens "send" can be
>> delivered to RDMAP only after last segment of rdma_write arrives and is placed.
>>
> I don't understand the question. An Immediate is another behavior, and
> which is used by upper layers rather differently. Also, what

---

From: Tom Talpey <[email-masked]>
Date: Tue, 8 Sep 2015 10:08:05 -0400

> And should be changed as "Send is completed after RDMA Write placed and delivered".

I disagree, it is accurate because the table is defining the behavior
as visible from the *local* peer, i.e. the initiator. You will note
that the only messages with anything but "not applicable" are the
RDMA Read responses.

Note the last words of the first sentence:

    Appendix B.  Ordering and Completion Table

    The following table summarizes the ordering relationships that are
    defined in Section 5.5, "Ordering and Completions", from the
    standpoint of the local peer issuing the two Operations.

The state of the *remote* peer depends upon it taking completions,
which is not something the local peer can determine without an
upper layer exchange.

Tom.