Subject: [rddp] DDP: Ordering Among DDP Messages (7.3) Data Source rules

From: Caitlin Bestler <[email-masked]>
Date: Tue, 5 Nov 2002 13:32:48 -0600

The following rule was added:

> At the Data Source, DDP:
>
> ...
> 
> * SHOULD transmit DDP Segments within a DDP Message in
> increasing MO order for Untagged DDP Messages and in
> increasing TO order for Tagged DDP Messages.

I believe this topic was thoroughly discussed, and that was
NOT the consensus. While it may seem innocuous, I do not
believe that a SHOULD clause is appropriate when there are
known valid reasons for at least some implementations not to
do so.

Further, in this case, there is absolutely no benefit to
this clause. It will only encourage sloppy implementations
to add extraneous "error checking" code looking for
"incorrectly" sequences segments.

As previously covered, a simple compliant receiver will not
even *notice* when segments are not in a natural order. It
has no *need* to examine their relative offsets at all.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Wed, 6 Nov 2002 22:55:21 +0200

MSN determines a WQE. Each WQE may have many virtually contiguous buffers
associated with (SGL). So first thing that should be done is to find a
right buffer from SGL, and this is just because of ooo transmit of DDP
segments in the same DDP message. We may talk about thousands of segments
and many buffers in SGL of the given WQE. It's not clear that complicating
RX implementation, and loosing  performance on Rx would worth possible gain
of performance on transmit.

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |                            |
|         |           06/11/02 07:34 PM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       Vadim Makhervaks/Haifa/IBM@IBMIL, [email-masked]                                                               |
  |       cc:                                                                                                                     |
  |       Subject:  Re: [rddp] DDP: Ordering Among DDP Messages (7.3) Data Source rules                                           |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

On 11/6/02, Vadim Makhervaks wrote:

>I see problem with efficient location of target buffers
>for untagged segments in case of ooo generated DDP
>segments. Let's discuss for example a case of transmitter
>which generates DDP segments in the following order of
>MOs: MO=4000, MO=3000, MO=2000, MO=1000, MO=0, M=5000. How
>receiver is supposed to find a proper location in the
>posted WQE for each one of those segments?
>
As each is recieved, the MSN determines which untagged
buffer is the target.

Each segment is checked that MO + segement payload size
will not exceed the size of the target buffer.

Then the payload is placed at target buffer base + MO.

This is exactly how tagged placements work, except for the
method for identifying the target buffer.

Paul Culley's response alluded to the possibility of
flushing the first part of the target buffer to the host
memory. But if untagged buffers are small (Ethernet size or
less) then it wouldn't make sense to flush before the entire
message was collected.

If the message *is* larger than an Ethernet frame, then:

a) it probably should have been sent tagged anyway.

b) the arguments on why having to collect it as a single
   image on the sender side also become signifigant.

Basically, if the size of the buffer is a justification as
to why this is not a problem on the sender side, I contend
it is just as valid as a reason why it is not a problem on
the receiver side.

Caitlin Bestler

---

From: "Culley, Paul" <[email-masked]>
Date: Wed, 6 Nov 2002 20:36:46 -0600

Receiver optimizations include the following:
* Cache (and even Prefetch) the next expected Receive buffer STag context (for Sends), to avoid lookup latencies when the segment actually arrives.  If segments are mostly in order, a single cache entry in the Stream context (or somewhere) will save time.
* Pre-know which "segment" (a la VIA style) of the destination buffer the DDP untagged (Receive queue) segment arrives in.  This is even more critical where there may be multiple "segments" describing the destination buffer.  If segments arrive in order, the implementation does not have to "scan" through a segment list, it simple fetches the next one.  If you always get segments out of order, you have to (for every segment) scan the segment list, summing up the segment sizes until the incoming DDP segment falls in the correct spot in the buffer.
* Prefetch and cache the next expected DDP segment "Page translation"; this avoids having to have a large TLB (Translation Lookaside Buffer) cache, which would be needed for good performance if completely random segment arrivals were the normal case.

These optimizations only save time and context memory bandwidth for large multisegment messages; however, if we cannot save some of the time/bandwidth here, the small messages in the elasticity buffer may be difficult to drain.

Again, receivers MUST be able do deal with out of order segments, but these optimizations help quite a bit when most segments arrive in order.

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Wednesday, November 06, 2002 10:46 AM
To: Culley, Paul; [email-masked]
Subject: RE: [rddp] DDP: Ordering Among DDP Messages (7.3) Data Source
rules

On 11/6/02, Culley, Paul wrote:

>The authors believe that implementations can get some
>significant optimizations at the receiver if the segments
>arrive (mostly) in order.  So that's the reason for adding
>an increasing MO/TO order rule to the sender.  The idea is
>that certain types of context caching operations at the
>receiver will perform better if the segments are in
>sequentially increasing order.
>

What are those optimizations?

An additional requirement should be justified by algorithms
presented in public *before* a requirement is added.

As for caching of the receive buffer. The receiver can
easily just assume that the entire untagged buffer will be
updated. And flush it after it is complete.

Unless you are optimizing for extremely large untagged
buffers, I do not see any benefit. And optimizign for large
untagged buffers would be a mistake, untagged buffers SHOULD
be small, we should not complicate the specification for the
benefit of a practice that is contrary to the intent of the
entire RDMA model.

>As you say, the receivers must handle out of order
>segments, but that may take a bit more time, context
>lookup bandwidth etc.  Since it was not demonstrated that
>a sender could really get any advantage when filling
>reasonable sized (1500 byte or so) segments (which would
>occur when sending messages large enough to require
>multiple segments), the authors went in favor of the
>receiver.
>

This is based upon the assumption that the source of a Send
is a contiguous in-memory buffer. If it comes from a
scatter-gather list, or is a virtual constructor (a "NIC
Side Include"), then the segments can easily be "available"
at different times. Hence sending them "out of order" can
make considerable sense.

Further, the bulk of the data transfers are tagged, for
which the receiver can make no orderign assumptions at all.

Again, I do not believe this requirement has been justified
by any publicly presented algorithm or scenario. It should
not be added.

Caitlin Bestler