Subject: RE: [rddp] Another DDP segment question

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 21 Oct 2002 12:18:25 -0700

Comments in line...

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, October 18, 2002 3:08 PM
> To: [email-masked]'
> Cc: Shah, Hemal; Larsen, Roy K
> Subject: RE: [rddp] Another DDP segment question
> 
> 
> On 10/18/02, Larsen, Roy K wrote:
> 
> >Hi Hemal,
> >
> >Although they seem perfectly reasonable to me, I can't
> >find anything in the DDP draft that supports your
> >assertions.  However, your last comment suggests such text
> >will be added to the draft which has always been my only
> >concern.
> >
> 
> It is the combination of section 7.3's requirement that
> DDP "MUST transmit DDP Messages in the order they were
> submitted to the DDP layer" and section 5's requirement
> that LLP delivery (as opposed to placement) must preserve
> the ordering that messages were submitted by the DDP layer.
> 
Humm.  It was these passages that led me to infer that segments within a
message should be sent in order.  I have been convinced by the arguments on
the list that one should be able to infer from the draft that segments (sans
the 'L' bit segment) can be sent in any order and conform.

> DDP segments are part of the same message, and hence were
> submitted to the DDP layer at the same time.  Therefore the
> only constraint is that the untagged segment with the 'L'
> bit must be the last submitted to the LLP.
> 
Indeed.  It is the 'L' bit segment that indicates a complete message has
been sent.  Therefore, I contend one could infer that a Data Source could
interleave segments of multiple messages and still comply provided an 'L'
bit segment is held back until all preceding 'L' bit segments have been
sent.  This will satisfy the requirement that the DDP MUST transmit DDP
*Messages* in the order they were submitted. This send behavior would
satisfy the DDP message ordering rules for the Data Sink specified in 7.3
because it is silent on the use of LLP sender order information (bullet 5 of
5.1).  Since the draft is silent on this relationship, one can infer that
reordering segments between messages is just as valid as reording segments
within a message provided 'L' bit semantics are retained.

> Even if  a local interface COULD accept multiple tagged
> writes at the same time, and therefore the sending DDP layer
> was allowed  to interleave them, it would not interfere with
> the Data Sink's ability to process tagged writes.
> 
I'm not sure that's true.  If an ULP at the Data Source does two RDMA writes
with the same STag to the same memory space, the target memory should
reflect the contents of the last write posted.  If segments of the second
message were reorded ahead of the first (by the fabric or Data Source), this
would not be the case.  Again, the LLP segment ordering information can
mitigate this but I don't see it applied in the draft.

> Untagged messages, by contrast, are inherently ordered. So
> if a local interface allowed multiple untagged messages to
> be submitted "at one time" it would have to accept them in
> a ULP specified "order". Hence interleaving of untaggeed
> messages on transmit would not be allowed.
> 
Well, as long a message delivery was done in order (i.e. 'L' bit segments
sent in order), it shouldn't matter when segments of an untagged message get
placed and in fact must be allowed for to deal with segments from different
messages that have been reorded by the fabric.

Please don't misunderstand me.  I don't think that sending interleaved
message segments is a good idea. As I've said before, I would advocate for
strict Data Source ordering rules that state all segments within a message
be sent in order and not overlapped.  Nevertheless, I take it on faith that
allowing a more relaxed message ordering semantic will provide efficiencies
for someone.  If there are efficiencies for reordering segments within a
message, I would assume these same efficiencies would be present across
messages and I believe the current draft can be read to support such a send
method.

The bottom line is that having message segment ordering rules inferred
instead of explicitly stated could lead to interoperability issues down the
road.  If we were talking about pure software stacks, these can be easily
adjusted.  However, all of the interesting implementation will be done in
silicon/microcode.  Adjustments in this space are expensive and time
consuming.

Roy

> That leaves the loophole, that probably should be closed,
> that a local interface that accepted one or more untagged
> writes and an untagged message "at the same time" MUST
> consider the untagged message to have been provided later.
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 21 Oct 2002 17:49:57 -0500

On 10/21/02, Larsen, Roy K wrote:

>Well, as long a message delivery was done in order (i.e.
>'L' bit segments sent in order), it shouldn't matter when
>segments of an untagged message get placed and in fact
>must be allowed for to deal with segments from different
>Please don't misunderstand me.  I don't think that sending
>interleaved message segments is a good idea. As I've said
>before, I would advocate for strict Data Source ordering
>rules that state all segments within a message be sent in
>order and not overlapped.  Nevertheless, I take it on
>faith that allowing a more relaxed message ordering
>semantic will provide efficiencies for someone.  If there
>are efficiencies for reordering segments within a message,
>I would assume these same efficiencies would be present
>across messages and I believe the current draft can be
>read to support such a send method.
>

This would indeed be valid. Further, it does not have any
adverse effect on the receiver. The receiver already has to
deal with *receiving* segments out of order, anyway.

Specifically, processing an untagged segment requires two
steps: placement and optional completion.

Placement requires matching the message number with the
appropriate queue of anonymous buffers (presumably a
"receive queue", although other compliant options could be
found). The target address range must be validated, and then
the payload is eligible for placement.

This is virtually identical to a tagged placement, except
for the method by which the target buffer is specified.

The additional step is that the DDP receiver must remember
when an 'L' bit was set, and the LLP location-marker where
it was found. When the LLP ack-window reaches this point the
completion must be reported.

Nothing in that required processing is in any way affected
by creative ordering of the segments within a message.

As that it is not a requirement for interoperability, the
draft should retain the absolute minimalism of the current
specification.

As for their being no reason for this flexibility, I think
it is best to just stick with the general mantra to accept
as much as possible. The looser receive specification does
not add work for the receiver, or cause any problems.

It may seem 'obvious' that there is no reason not to send
the segments of a message in order, but that assumes the
source of the message is pre-assembled in a sequence of
bytes on the source. If the message is being assembled on
the fly (with a scatter-gather list) then there could be
differences in the availability of elements of the message.
There is no reason to limit future implementations if the
only benefit to the protocol is simpler examples.

>The bottom line is that having message segment ordering
>rules inferred instead of explicitly stated could lead to
>interoperability issues down the road.  If we were talking
>about pure software stacks, these can be easily adjusted. 
>However, all of the interesting implementation will be
>done in silicon/microcode.  Adjustments in this space are
>expensive and time consuming.
>

Again, there is a simple receiver algorithm that will not
even notice reordering of non-"L" untagged segments. This
algorithm is clearly implementable in hardware or software.
Therefore the only danger to interoperability would be
non-compliant implementations that imposed invalid, and
unneeded, false validity checks merely because they worked
for most peers.

Caitlin Bestler