Subject: [rddp] legality of interleaving untagged message segments

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 17 May 2004 00:28:22 -0500 (CDT)

The current RDDP draft requires that messages be submitted
to the LLP in order.

It also requires that when a message is segmented, that the
final segment MUST be the last one submitted to the LLP.

It does not, however, define what is meant by submitting
"a message" when in fact the DDP layer submits *segments*
to the LLP.

I believe this could allow a very strained interpretation
that would allow interleaving of untagged segments to the
LLP, as long as the *final* segments were submitted in order.
For example, it might allow the following submission order:

     Untagged MSN 7 segment 0 (not final)
     Untagged MSN 8 segment 0 (not final)
     Untagged MSN 7 segment 1 (final)
     Untagged MSN 8 segment 1 (final)

A typical out-of-order receiver would not even notice
anything unusual about this sequence of untagged segments.
However, it has a major impact on in-order receivers. It
requires that multiple RQ (or S-RQ) entries be 'current'
for the packet processor. If the in-order packet-processor
is on the RNIC and the RQ/S-RQ is in host memory this is
a totally unnecessary complication.

I still believe that the sender should be free to re-order
segments within a message (but SHOULD NOT unless it gains
some advantage in doing so, such as sending data that is
already cached), but I cannot come up with any benefit
for the sender to interleave message segments. The complexity
of doing so is simply too great for any potential additional
caching gains.

The fact that such sequences are so unlikely means that it
is highly likely that there will be implementations that
will not properly handle this case, whether or not the
specification explicitly forbids it. This makes any theoretical
advantages even more remote. Customers seldom care that
your esoteric optimization is "legal", they just know that
*your* RNIC is the one that isn't working with everyone else.

So I believe the draft should be clarified to require that
all segments for untagged message N MUST be submitted to
the LLP before any segments for untagged message N+1 are
submitted to the LLP.

Doing so will have no impact on unordered receivers. I doubt
it will have any impact on any existing senders, and it
eliminates one corner condition that would require extra
resources and testing for in-order receivers.

>>
>> Several months ago I raised a query as to whether DDP could/should
>> mandate that the process of segmenting DDP Messages MUST NOT
>> result in interleaving of messages. That is *all* segments for DDP
>> Message N must be submitted to the transmitting LLP: before any
>> segment of DDP Message N+1 is submitted.
>>
>> I believe the current draft could, in a very strained malicious way,
>> be interpreted to only require:
>>
>>     that the segment with the L bit for each message be the last
>>     segment for that message.
>>
>>     that the segments with L bits are themselves submitted in order.
>>
>> The only real response on the list, as I recall, was from Mike Krause
>> who pointed out that such interleaving would not even be noticed
>> by an out-of-order receiver. And that is completely correct.
>>
>> It does have an impact on in-order receivers, and particular on
>> how many untagged buffers an in-order RNIC has to have looked
>> up from the host-side receive queues (or worse SRQs).
>>
>> I doubt very much whether anyone will ever produce a sender
>> with a segmentation algorithm that will violate this rule, and I
>> am one of the strong advocates of allowing senders to use
>> creative ordering *within* a message to optimize for sending
>> data that is already in the RNICs cache.
>>
>> But it would be handy to have something firm to justify coding
>> an RNIC based on this assumption.
>>
>> Do you feel that the current language already forbids interleaving
>> of message segments (in terms of the order they are submitted
>> to the LLP, and hence their LLP sequence numbers)?
>>
>> If so, it is probably enough to just have that stated on the
>> reflector.
>>
>> Otherwise, as one of the principal advocates of having the protocols
>> be friendly to *both* ordered and unordered implementations, would
>> you support such a change?
>>
>> In my evaluation, the drafters are firmly in the "out of order" camp
>> and hence do not see any need for this clarification.
>>
>

--
Caitlin Bestler
http://asomi.com/

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Mon, 17 May 2004 09:47:55 -0700

Within a given LLP stream, you are correct that the optimization benefits 
may be minimal if I/O device is somewhat "distant" (read as relatively high 
latency) from the memory controller.  However, as we see the industry move 
towards PCI Express (PCIe) and with the potential to create more integrated 
memory controller chipsets, there may be greater benefit over time 
(especially with some architectures moving to be more NUMA oriented).

Given this may take time to occur and validate, would you be willing to 
make this a SHOULD NOT instead of a MUST NOT thus allowing leeway for 
future development while providing strong guidance that such interleaving 
is strongly discouraged?

Mike

---

From: "Jeff Dwork" <[email-masked]>
Date: Mon, 17 May 2004 11:16:51 -0700

A receiver capable of tracking out-of-order messages is more complex
than one which requres in-order submission.  Support of out-of-order
message submission must be negotiated prior to stream initialization.  

A possible implementation: The receiver declares the maximum number of
simultaneous messages it will accept, possible values being an integer
greater than 0 or the special value "NO_LIMIT" (perhaps represented as
0).

A value of "1" means that all segments of message N must be submitted
before any segments of message N+1.  A value of "NO_LIMIT" means that
the receiver will accept any number of out-of-order messages up to the
size of the RQ/S_RQ.

Jeff

Michael Krause writes:
 > To: [email-masked]
 > Subject: Re: [rddp] legality of interleaving untagged message segments
 > Date: Mon, 17 May 2004 09:47:55 -0700
 > 
 > 

-- 
Jeff Dwork                     | [email-masked]  408-307-2496 (cell)
Advanced Micro Devices, M/S 362| 408-749-5216 (desk) 408-774-7811 (fax)
PO Box 3453                    |----------------------------------------
Sunnyvale, Ca 94088-3453       |

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 18 May 2004 01:24:34 -0500 (CDT)

> future development while providing strong guidance that such interleaving

There is already a SHOULD that segment SHOULD be submitted in ascending
order. So no further SHOULD clause is required.

Or for that matter useful.

 A "SHOULD NOT 'X'" clause does not truly allow the receiver to optimize on
the assumption that senders will not do X. It just discourages testing of
'X',
so in a sense it's worse than doing nothing. Receivers are not free to
optimize,
and senders are discouraged from whatever optimization there might be.

My assumption is that the MUST NOT I am proposing will not preclude any
achievable optimization on the sender side.  If that assumption is correct
then benefit to the receiver is clearly preferable. But if anyone had an
algorithm that really would optimize across message boundaries and gain
actual benefits from it then the tradeoff is indeed more complex.

The benefit at the receiver side, for an implementation that defers placement
processing of DDP segments until they can be processed in order, is that
there would be at most one active untagged buffer per DDP Stream. This
is a major benefit, which I believe outweighs the theoretical benefits that
some unidentified sender algorithm *might* achieve in the future.

Of course I am willing to be convinced to the contrary if someone were to
present a contemplated optimization that this restriction would interfere
with.
Or for that matter, even assert that they are researching such an algorithm
that they are unwilling to disclose.

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 18 May 2004 01:30:54 -0500 (CDT)

> the receiver will accept any number of out-of-order messages up to the

The potential optimizations to an "in-order receiver" are real, but they
definitely would not justify more session parameters negotiated on the wire.

What is proposed is far simpler, that untagged messages behave as though
their segments are submitted to the LLP atomically. For example, SCTP
has such a restriction on SCTP layer segmentation of user messages.
>From the RDMAP layer this is entirely transparent. It is only an issue
to the implementation of the DDP layer.

---

From: Michael Krause <[email-masked]>
Date: Tue, 18 May 2004 11:34:24 -0700

>that they are unwilling to disclose.

I willing to make this a MUST NOT and address this in a future version of 
the spec should the optimizations be publicized.

Mike