Subject: [rddp] Another DDP segment question

From: "Larsen, Roy K" <[email-masked]>
Date: Fri, 18 Oct 2002 13:02:35 -0700

There seems to be consensus that a Data Source is free to send segments of
an ULP message in any order and overlap provided the segment with the L bit
set is sent last.  However, I need a further clarification.  Does this
reordering at the Data Source apply across ULP messages.  For example, for
untagged messages, is a Data Source free to send segments from MSN n+1
before segments from MSN n provided that the n segment with the L bit set is
sent before the n+1 segment with the L bit set?

Section 7.3 states that Data Source MUST transmit DDP Messages in the order
they were submitted to the DDP layer.  Is the "order" dictated by when the
segment with the L bit is sent (since that is the prerequisite event for
indicating message reception to the ULP) or the first segment of an ULP
message?

Certainly a segment from MSN n+1 can arrive at the Data Sink before a
segment from n, but is it valid for the n+1 segment to have a larger TCP
sequence number than n?

Although there seems to be a reluctance to do so, I strongly urge that all
message segmentation ordering rules be explicitly stated in the draft.

Roy
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Shah, Hemal" <[email-masked]>
Date: Fri, 18 Oct 2002 14:27:53 -0700

Roy,

See my answers below embedded within <hvs>.

Hemal

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]] 
Sent: Friday, October 18, 2002 3:03 PM
To: [email-masked]
Subject: [rddp] Another DDP segment question

There seems to be consensus that a Data Source is free to send segments of
an ULP message in any order and overlap provided the segment with the L bit
set is sent last.  However, I need a further clarification.  Does this
reordering at the Data Source apply across ULP messages.  For example, for
untagged messages, is a Data Source free to send segments from MSN n+1
before segments from MSN n provided that the n segment with the L bit set is
sent before the n+1 segment with the L bit set?
<hvs>All the DDP Segments of an Untagged Message with MSN n are sent before
any of the DDP Segments of an Untagged Message with MSN n+1. In other words,
in the TCP stream in the sequence space, all the DDP Segments of MSN n will
appear before any of the DDP Segments of MSN n+1.<hvs> 

Section 7.3 states that Data Source MUST transmit DDP Messages in the order
they were submitted to the DDP layer.  Is the "order" dictated by when the
segment with the L bit is sent (since that is the prerequisite event for
indicating message reception to the ULP) or the first segment of an ULP
message?
<hvs> This is probably answered by the first comment. Let me try it again
with an example to aid the previous comment. If the ULP has submitted two
messages Msg 1 and Msg 2 (in that order) to DDP, then, all DDP segments of
Msg 1 will be sent before any of the DDP Segments of Msg 2. Out of all the
DDP Segments of a DDP Message, DDP will send the DDP Segment with L bit set
to 1 last. <hvs>

Certainly a segment from MSN n+1 can arrive at the Data Sink before a
segment from n, but is it valid for the n+1 segment to have a larger TCP
sequence number than n?
<hvs>Yes. In TCP arithmetic, a TCP segment containing a DDP Segment of an
Untagged DDP Message with MSN n+1 will carry larger sequence number than the
sequence number carried by a TCP segment containing a DDP Segment of an
Untagged DDP Message with MSN n.
<hvs>

Although there seems to be a reluctance to do so, I strongly urge that all
message segmentation ordering rules be explicitly stated in the draft.
<hvs> There will be more clarification in the next draft.<hvs>

Roy

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Mon, 21 Oct 2002 21:29:23 +0200

Transmitting Tagged DDP messages in-order does not say anything about the
order of the data placement on the receiver. The only way to make sure that
the second Write is what we would see in the memory is to use FENCE
semantics defined by RDMA spec.

Vadim.

P.S. By the way, I absolutely agree with you regarding the DDP segments
order in the DDP message. To be more precise, I think that allowing
Untagged DDP segments be transmitted out-of-order put unnecessary
complexity on receiver implementation, and also potentially leads to
performance hit.

|---------+---------------------------->
|         |           "Larsen, Roy K"  |
|         |           <roy.k.larsen@int|
|         |           el.com>          |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           21/10/02 09:18 PM|
|         |                            |
|---------+---------------------------->
  >------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                              |
  |       To:       "'Caitlin Bestler'" <[email-masked]>, "[email-masked]'" <[email-masked]>                               |
  |       cc:       "Shah, Hemal" <[email-masked]>                                                                         |
  |       Subject:  RE: [rddp]  Another DDP segment question                                                                     |
  |                                                                                                                              |
  |                                                                                                                              |
  >------------------------------------------------------------------------------------------------------------------------------|

Comments in line...

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, October 18, 2002 3:08 PM
> To: [email-masked]'
> Cc: Shah, Hemal; Larsen, Roy K
> Subject: RE: [rddp] Another DDP segment question
>
>
> On 10/18/02, Larsen, Roy K wrote:
>
> >Hi Hemal,
> >
> >Although they seem perfectly reasonable to me, I can't
> >find anything in the DDP draft that supports your
> >assertions.  However, your last comment suggests such text
> >will be added to the draft which has always been my only
> >concern.
> >
>
> It is the combination of section 7.3's requirement that
> DDP "MUST transmit DDP Messages in the order they were
> submitted to the DDP layer" and section 5's requirement
> that LLP delivery (as opposed to placement) must preserve
> the ordering that messages were submitted by the DDP layer.
>
Humm.  It was these passages that led me to infer that segments within a
message should be sent in order.  I have been convinced by the arguments on
the list that one should be able to infer from the draft that segments
(sans
the 'L' bit segment) can be sent in any order and conform.

> DDP segments are part of the same message, and hence were
> submitted to the DDP layer at the same time.  Therefore the
> only constraint is that the untagged segment with the 'L'
> bit must be the last submitted to the LLP.
>
Indeed.  It is the 'L' bit segment that indicates a complete message has
been sent.  Therefore, I contend one could infer that a Data Source could
interleave segments of multiple messages and still comply provided an 'L'
bit segment is held back until all preceding 'L' bit segments have been
sent.  This will satisfy the requirement that the DDP MUST transmit DDP
*Messages* in the order they were submitted. This send behavior would
satisfy the DDP message ordering rules for the Data Sink specified in 7.3
because it is silent on the use of LLP sender order information (bullet 5
of
5.1).  Since the draft is silent on this relationship, one can infer that
reordering segments between messages is just as valid as reording segments
within a message provided 'L' bit semantics are retained.

> Even if  a local interface COULD accept multiple tagged
> writes at the same time, and therefore the sending DDP layer
> was allowed  to interleave them, it would not interfere with
> the Data Sink's ability to process tagged writes.
>
I'm not sure that's true.  If an ULP at the Data Source does two RDMA
writes
with the same STag to the same memory space, the target memory should
reflect the contents of the last write posted.  If segments of the second
message were reorded ahead of the first (by the fabric or Data Source),
this
would not be the case.  Again, the LLP segment ordering information can
mitigate this but I don't see it applied in the draft.

> Untagged messages, by contrast, are inherently ordered. So
> if a local interface allowed multiple untagged messages to
> be submitted "at one time" it would have to accept them in
> a ULP specified "order". Hence interleaving of untaggeed
> messages on transmit would not be allowed.
>
Well, as long a message delivery was done in order (i.e. 'L' bit segments
sent in order), it shouldn't matter when segments of an untagged message
get
placed and in fact must be allowed for to deal with segments from different
messages that have been reorded by the fabric.

Please don't misunderstand me.  I don't think that sending interleaved
message segments is a good idea. As I've said before, I would advocate for
strict Data Source ordering rules that state all segments within a message
be sent in order and not overlapped.  Nevertheless, I take it on faith that
allowing a more relaxed message ordering semantic will provide efficiencies
for someone.  If there are efficiencies for reordering segments within a
message, I would assume these same efficiencies would be present across
messages and I believe the current draft can be read to support such a send
method.

The bottom line is that having message segment ordering rules inferred
instead of explicitly stated could lead to interoperability issues down the
road.  If we were talking about pure software stacks, these can be easily
adjusted.  However, all of the interesting implementation will be done in
silicon/microcode.  Adjustments in this space are expensive and time
consuming.

Roy

> That leaves the loophole, that probably should be closed,
> that a local interface that accepted one or more untagged
> writes and an untagged message "at the same time" MUST
> consider the untagged message to have been provided later.
>
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
>