Subject: [storm] DDP messages ordering

From: Elena Gurevich <[email-masked]>
Date: Tue, 8 Sep 2015 10:02:28 +0000

Hello,

RFC 5041 section 5.3 stands that

----------------------------------
5.3 Ordering Among DDP Messages

Messages passed through the DDP MUST conform to the ordering rules
defined in this section.
At the Data Source, DDP:
* MUST transmit DDP Messages in the order they were submitted to
the DDP layer,
* SHOULD transmit DDP Segments within a DDP Message in increasing
MO order for Untagged DDP Messages, and in increasing TO order
for Tagged DDP Messages.
----------------------------------

Does this mean that transmitter MUST not interleave DDM segments related to consequent DDP messages ?

Best regards,
Elena Gurevich
---------------------------------------------------------
Software Architector at Toga Networks Ltd.

Email: [email-masked]
---------------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

---

From: Tom Talpey <[email-masked]>
Date: Tue, 8 Sep 2015 10:15:29 -0400

> RFC 5041 section 5.3 stands that

It depends on what you mean by "the transmitter". It's certainly
possible that network delivery and TCP retransmission can reorder
segments. But the first rule requires that DDP not interleave two
separate operations when passing them to the TCP transport.

Tom.

---

From: <[email-masked]>
Date: Tue, 8 Sep 2015 14:40:04 +0000

> On Sep 8, 2015, at 10:15 AM, Tom Talpey <[email-masked]> wrote:
> 
>> ...
> segments.

No, it isn't.  TCP provides two services: (1) ordered delivery, and (2) guaranteed delivery or report of failure to do so.

Network delivery and TCP retransmission will NOT reorder the data delivered by TCP to the layer above.

	paul

---

From: Elena Gurevich <[email-masked]>
Date: Wed, 9 Sep 2015 13:07:15 +0000

Hello,

During my testing of some iWARP adaptor I discovered that RNIC interleaves DDP segments of Send and Read Response messages.
According to your previous response this behavior is forbidden, is it ?

Best regards,
Lena

-----Original Message-----
From: storm [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, September 08, 2015 5:15 PM
To: [email-masked]
Subject: Re: [storm] DDP messages ordering

> RFC 5041 section 5.3 stands that

It depends on what you mean by "the transmitter". It's certainly possible that network delivery and TCP retransmission can reorder segments. But the first rule requires that DDP not interleave two separate operations when passing them to the TCP transport.

Tom.

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: Tom Talpey <[email-masked]>
Date: Wed, 9 Sep 2015 11:53:35 -0400

> According to your previous response this behavior is forbidden, is it ?

Not necessarily, because an RDMA Read response is special. It's
injected into the DDP stream within the RNIC by RDMAP, and therefore
it follows the RDMAP rules. See section 5.5 of RFC5040.

Tom.

---

From: Elena Gurevich <[email-masked]>
Date: Thu, 10 Sep 2015 12:02:32 +0000

Hello Tom,

Section 5.5 just specifies ordering rules of RDMAP requester and responder.
According to such rules RDMAP generates RDMA messages and passes them to DDP layer one by one for future processing.
RDMAP's granularity and visibility is a particular RDMA message.
DDP layer gets RDMA messages from RDMAP layer maps them one-by-one to DDP messages performs segmentation and passed them to transport layer.
So DDP segments interleaving/rules cannot be part of RDMAP , it is clearly DDP layer behavior.
And according to RFC 5041 Section 5.3 and already mentioned by you "first rule" cannot interleave DDP segments of different operations.

Best regards,
Lena

-----Original Message-----
From: Tom Talpey [mailto:[email-masked]]
Sent: Wednesday, September 09, 2015 6:54 PM
To: Elena Gurevich; [email-masked]
Subject: Re: [storm] DDP messages ordering

> According to your previous response this behavior is forbidden, is it ?

Not necessarily, because an RDMA Read response is special. It's injected into the DDP stream within the RNIC by RDMAP, and therefore it follows the RDMAP rules. See section 5.5 of RFC5040.

Tom.

> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> ----- This email and any files transmitted and/or attachments with it
> are confidential and proprietary information of Toga Networks Ltd.,
> and intended solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system
> manager. This message contains confidential information of Toga
> Networks Ltd., and is intended only for the individual named. If you
> are not the named addressee you should not disseminate, distribute or
> copy this e-mail. Please notify the sender immediately by e-mail if
> you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> ----
>
>
>

---

From: Elena Gurevich <[email-masked]>
Date: Thu, 17 Sep 2015 11:53:01 +0000

Hello Tom,

Please reference exact place that explicitly or implicitly allows/forbids the behavior in question.

Best regards,
Lena

-----Original Message-----
From: storm [mailto:[email-masked]] On Behalf Of Elena Gurevich
Sent: Thursday, September 10, 2015 3:03 PM
To: Tom Talpey; [email-masked]
Subject: Re: [storm] DDP messages ordering

Hello Tom,

Best regards,
Lena

-----Original Message-----
From: Tom Talpey [mailto:[email-masked]]
Sent: Wednesday, September 09, 2015 6:54 PM
To: Elena Gurevich; [email-masked]
Subject: Re: [storm] DDP messages ordering

Tom.

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

---

From: Tom Talpey <[email-masked]>
Date: Mon, 28 Sep 2015 12:16:32 -0400

I was traveling last week so this is a delayed response. I want
to say that I am not answering in the role of STORM WG co-chair.
This is simply my interpretation of the specifications.

The behavior you observed is allowed by the RDMA Read processing
rules. In RFC5040 section 4.5, the RDMA Read Response is defined to
be simply a DDP message. The key is that the sender (i.e. the RDMA
Read responder) can and does respond to a single RDMA Read with
multiple Responses at the RDMAP layer. The "Last" flag, defined in
RFC5041 section 4.1, is used to indicate the final Response in such a
sequence. The RDMA Read Requestor simply processes the responses until
the Last flag is received, at which point it completes the RDMA Read
operation back to the application.

Because each RDMA Read Response is a separate DDP message, it is
perfectly legal for the Responder to interleave other DDP messages
while processing the RDMA Read. There is no requirement that DDP pause
the stream until the Last flag is sent.

I'd actually expect most RNICs to do this for large RDMA Reads,
for example if they are larger than a single Ethernet frame, or
any other size if desired. Because the responding RNIC needs to
perform multiple PCI Reads, and because RNICs don't usually buffer
tagged data, it will be very convenient for them to issue interim
RDMA Read Responses.

Does this help?

Tom.

---

From: Elena Gurevich <[email-masked]>
Date: Tue, 29 Sep 2015 06:18:11 +0000

Hi Tom,  that you for your responses you are the only person that reacts to my question :(.

If we back to my question:
according to RFC 5040
-------------------------------------------------------------
5.2 RDMA Read Operation
The RDMA Read operation MUST consist of a single RDMA Read Request Message and a single RDMA Read Response Message.
-------------------------------------------------------------

And "the last flag" you mentioned relates to DDP layer to sign last segment of the same DDP message ( RDMA message is mapped one-by-one to DDP message).

Personally I think that for transmitter it will be more simple to allow interleaving of DDP segments of different RDMA messages related to different transmit queues: SQ and IRRQ, but for receiver this behavior complicates the processing.

So if we allow this behavior RFC 5041 must more accurate define this rule, or I still miss something.

Best regards,
Elena

-----Original Message-----
From: Tom Talpey [mailto:[email-masked]]
Sent: Monday, September 28, 2015 7:17 PM
To: Elena Gurevich; [email-masked]
Subject: Re: [storm] DDP messages ordering

I was traveling last week so this is a delayed response. I want to say that I am not answering in the role of STORM WG co-chair.
This is simply my interpretation of the specifications.

The behavior you observed is allowed by the RDMA Read processing rules. In RFC5040 section 4.5, the RDMA Read Response is defined to be simply a DDP message. The key is that the sender (i.e. the RDMA Read responder) can and does respond to a single RDMA Read with multiple Responses at the RDMAP layer. The "Last" flag, defined in
RFC5041 section 4.1, is used to indicate the final Response in such a sequence. The RDMA Read Requestor simply processes the responses until the Last flag is received, at which point it completes the RDMA Read operation back to the application.

Because each RDMA Read Response is a separate DDP message, it is perfectly legal for the Responder to interleave other DDP messages while processing the RDMA Read. There is no requirement that DDP pause the stream until the Last flag is sent.

I'd actually expect most RNICs to do this for large RDMA Reads, for example if they are larger than a single Ethernet frame, or any other size if desired. Because the responding RNIC needs to perform multiple PCI Reads, and because RNICs don't usually buffer tagged data, it will be very convenient for them to issue interim RDMA Read Responses.

Does this help?

Tom.