Subject: Re: [rddp] DDP: Ordering Among DDP Messages (7.3) Data Source rules

From: "Vadim Makhervaks" <[email-masked]>
Date: Wed, 6 Nov 2002 18:58:06 +0200

I see problem with efficient location of target buffers for untagged
segments in case of ooo generated DDP segments. Let's discuss for example a
case of transmitter which generates DDP segments in the following order of
MOs: MO=4000, MO=3000, MO=2000, MO=1000, MO=0, M=5000. How receiver is
supposed to find a proper location in the posted WQE for each one of those
segments?

Once the location is found, everything becomes very simple ...

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |                            |
|         |           06/11/02 06:12 PM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       Vadim Makhervaks/Haifa/IBM@IBMIL                                                                              |
  |       cc:       [email-masked], [email-masked]                                                                            |
  |       Subject:  Re: [rddp] DDP: Ordering Among DDP Messages (7.3) Data Source rules                                           |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

On 11/6/02, Vadim Makhervaks wrote:

>
><VM> disagree. It much more difficult to receiver to
>efficiently handle reordered DDP segments of Untagged DDP
>message. TCP ooo can be easily handled, because the
>information stored by receiver to allow efficient
>implementation is proportional to the number of supported
>holes. Regarding the DDP re-ordering, the information
>required for the similar-efficient implementation is
>proportional to the number of the segments in the DDP
>message, which can be pretty unlimited from the
>implementation perspective (say 2GB message, with 1K MSS,
>gives 2M of segments). This problem is relevant to
>Untagged DDP messages only<VM>.
>3

Could you provide some substantiation for the claim that
unordered message offsets creates ANY difficulty for the
receiver.

As I see it, the receiver logic is simple:

    Place the data at the message offset within the
    target buffer (whether untagged or tagged).

    Segments with the 'L' bit are subject to eventual
    completion processing. The completion reported will
    take the size of the untagged message based upon the
    message offset and size of the last segment.

Note, there is no impact to the receiver on the ordering of
any segment other than the last one.

Do you have a contrary algorithm with some citable benefit?

Caitlin Bestler
http://asomi.com/CaitlinBestler/

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 6 Nov 2002 11:34:47 -0600

On 11/6/02, Vadim Makhervaks wrote:

>I see problem with efficient location of target buffers
>for untagged segments in case of ooo generated DDP
>segments. Let's discuss for example a case of transmitter
>which generates DDP segments in the following order of
>MOs: MO=4000, MO=3000, MO=2000, MO=1000, MO=0, M=5000. How
>receiver is supposed to find a proper location in the
>posted WQE for each one of those segments?
>
As each is recieved, the MSN determines which untagged
buffer is the target.

Each segment is checked that MO + segement payload size 
will not exceed the size of the target buffer.

Then the payload is placed at target buffer base + MO.

This is exactly how tagged placements work, except for the
method for identifying the target buffer.

Paul Culley's response alluded to the possibility of
flushing the first part of the target buffer to the host
memory. But if untagged buffers are small (Ethernet size or
less) then it wouldn't make sense to flush before the entire
message was collected.

If the message *is* larger than an Ethernet frame, then:

a) it probably should have been sent tagged anyway.

b) the arguments on why having to collect it as a single
   image on the sender side also become signifigant.

Basically, if the size of the buffer is a justification as
to why this is not a problem on the sender side, I contend
it is just as valid as a reason why it is not a problem on
the receiver side.

Caitlin Bestler

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 06 Nov 2002 16:35:28 -0500

>a) it probably should have been sent tagged anyway.

Strongly disagree. The size of the link or transport layer frame
is not the right metric, and is certainly not the one an application
will use.

The first and most obvious metric is the "bcopy threshold",
the byte count at which the cost of copying is less than the
cost of tagged buffer setup. On many machines this is larger
than the Ethernet frame size.

The second metric is what I/O model the target application
wishes to use. Application designers don't write code differently
in the expectation that it will run over a different transport
framing. 

For example, DAFS does not expose tagged buffers on the
server. A DAFS write operation from client to server, then,
must choose between untagged Send-style transfers from
client to server, or tagged RDMA Reads issued from server
back to client. There is a significant difference between these,
and the decision which one to use has little to do with the
frame size. Typical settings for this threshold are between
4096 and 65536, all of course mapping to several Ethernet
frames.

So, optimizations which allow the efficient placement of
multisegment untagged messages are worthwhile. At that
point, whether such an optimization warrants a MAY or a
SHOULD in the spec is a minor point.

Tom.

---

From: Michael Krause <[email-masked]>
Date: Sun, 10 Nov 2002 15:26:26 -0800

>less) then it wouldn't make sense to flush before the entire

Sure it would.  It is a function of the local I/O DMA policies including 
buffer resource management. This is strictly an implementation area to 
innovate but one can envision operations whereby one does not present large 
DMA to the local I/O fabric in order to minimize HOL blocking in that 
fabric thus one can may see a SAR occur in the local I/O (implicitly or 
explicitly managed by the I/O controller).

>a) it probably should have been sent tagged anyway.

Clearly not true else we would not have unsolicited data in iSCSI for all 
of the reasons cited there.  I can think of more but the use of tagged vs. 
untagged is both a function of the bcopy threshold as well as the ULP 
requirements.  There are many components to the bcopy threshold with the 
basics being the impact of Moore's Law and the RNIC implementation 
capabilities which will vary between vendors.

>b) the arguments on why having to collect it as a single image on the 
>sender side also become signifigant.

The depth of the SGL is a component in the decision.

>Basically, if the size of the buffer is a justification as to why this is 
>not a problem on the sender side, I contend it is just as valid as a 
>reason why it is not a problem on the receiver side.

Mike