Subject: RE: [rddp] Clarification on DDP segmentation

From: "Larsen, Roy K" <[email-masked]>
Date: Thu, 10 Oct 2002 15:32:59 -0700

See embedded comments...

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Thursday, October 10, 2002 2:26 PM
> To: [email-masked]
> Cc: Larsen, Roy K
> Subject: Re: [rddp] Clarification on DDP segmentation
> 
> 
> On 10/10/02, Larsen, Roy K wrote:
> 
> >The draft is ambiguous as to whether the DDP layer may
> >send DDP message segments out of order to the LLP.  The
> >draft does explicitly deal with OOO segments on receive
> >but only as it applies to a "rototilled" implementation
> >with the implication being that reordering has been done
> >by the fabric. However, in describing the Last Flag bit in
> >section 6.1, the sentence:
> >
> >"The DDP Segment with the L bit set to 1 MUST be posted to
> >the LLP after all other DDP Segments...."
> >
> >This implies that DDP may send segments OOO to the LLP. 
> >In any case, I would like to see one of the following
> >statements added to section 7.2:
> >
> >"The Data Source MUST deliver segments of a DDP message to
> >the LLP in order."
> >
> >or
> >
> >"The Data Source MAY deliver segments of a DDP message to
> >the LLP in any order with the exception of the DDP segment
> >with the L bit set which MUST be the last segment sent to
> >the LLP."
> >
> >Non-rototilled implementations (particularly hybrid RNIC
> >solutions) can take advantage of an in-order DDP segment
> >semantic so it is important that the draft be clear on
> >this subject.
> >
> >Roy
> 
> The phrase "in order" here begs the question. Normally, 
> DDP segment ordering is *defined* by the order in which
> they were presented to the LLP.
> 
> If you want to specify the first option, it would probably
> make more sense to require that the Message Offsets (MO)
> must increase with each segment.
> 
> The draft already requires that the entire message be
> delivered. About the only other possible requirement
> would be to mandate that each byte of the message be
> placed in *exactly* one DDP segment (as opposed to
> at least one).
> 

Wow, I hadn't even considered that it could be valid to send overlapping DDP
segments from a single ULP message.

> It shouldn't make a difference to the Data Sink anyway.
> If its checking the offsets on successive messages then
> its wasting cycles. There is a valid reason why the highest
> offset must be in the last segment, it makes calculation
> of the message size very simple.
> 

You lost me there.  If DDP segments of an ULP message can be sent in
non-assending MO order, the Data Sink MUST be checking the offsets of
successive messages to know where to place them.  And it shouldn't matter
when the last segment is sent because it is indicated with the L bit.  Why
would it matter if you knew the length before you received all the segments?
In a rototilled implementation, it could still be processed OOO regardless
of when it was sent by the Data Source to the LLP.

> The only benefit I can think of for mandating the others 
> to be in sort ascending order is to make things easier
> for packet sniffers.
> 

There are RNIC environments that will do DDP/MPA processing employing an
in-order TCP service (non-rototilled).  If the DDP protocol constrained the
Data Source to sending non-overlapping, ascending-only message offset DDP
segments to the LLP, MPA header/marker and DDP header processing can be
simplified.

> But I agree, it might be nice to highlight that warning
> so that nobody builds an RNIC that rejects out-of-order
> segments that were otherwise valid. Requiring that they
> be sent in order is also equally valid. Unlike tagged
> messages, there is no need to generate untagged message
> segments out of order.

Although I lean toward an "in-order" send constraint to the LLP (only
because it is how I presume virtually all DDP/RDMA layer will implement it
anyway), it is not a strong conviction.  I do believe, however, that the
draft MUST be explicit on what is and is not allowable with respect to DDP
segmentation.  It is these sorts of subtle ambiguities that cause erroneous
assumptions resulting in interoperability problems.

Roy

> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 10 Oct 2002 18:00:19 -0500

>> If you want to specify the first option, it would
>> probably make more sense to require that the Message
>> Offsets (MO) must increase with each segment.
>> 
>Wow, I hadn't even considered that it could be valid to
>send overlapping DDP segments from a single ULP message.
>

The one scenario I can think of would actually involve
redundant transmissions. It is far more likely with tagged
transfers, but concievably could happen with untagged as
well as that some protocols allow for both RDMA and
'in-line' delivery of data.

Basically, in an embedded device level environment, the
ULP may have a custom local interface to 'send' not from
RAM but directly from some form of persistent storage.
In such an environment it might decide that it needed
to retry a read after shipping the first segments.

I did work on a project that interfaced ATA drives
directly to a switch fabric in a single FPGA. Even
in *that* minimal environment the entire sector was
read, and validated, before the cells were sent.
But we *could* have sent the first cell earlier.
So it is a possible scenario.

Obscure, yes. It's more likely that it would be a test
specifically designed to emberrass a competitor's RNIC.

>> its wasting cycles. There is a valid reason why the
>> highest offset must be in the last segment, it makes
>> calculation of the message size very simple.
>> 
>
>You lost me there.  If DDP segments of an ULP message can
>be sent in non-assending MO order, the Data Sink MUST be
>checking the offsets of successive messages to know where
>to place them.  And it shouldn't matter when the last
>segment is sent because it is indicated with the L bit. 
>Why would it matter if you knew the length before you
>received all the segments? In a rototilled implementation,
>it could still be processed OOO regardless of when it was
>sent by the Data Source to the LLP.
>
Each Message Offset must be used in placing the packet.
However there is no need for the RNIC to remember what has
been placed previously. The sender *must* send the entire
message, therefore there can be no gaps. The sender *must*
place the highest MO in the final segment, therefore the
receiver can determine the size of the entire message from
the last segment alone.

There is no need to checkerboard or otherwise validate the
incoming segments, other than the previously mentioned
analyzing sniffer.

>There are RNIC environments that will do DDP/MPA
>processing employing an in-order TCP service
>(non-rototilled).  If the DDP protocol constrained the
>Data Source to sending non-overlapping, ascending-only
>message offset DDP segments to the LLP, MPA header/marker
>and DDP header processing can be simplified.
>

Each message received from the TCP layer can still be
processed, no matter what order the Message Offsets are in.
The receiver should not be using the MOs as some sort of
additional validity/sequencing check on the framing. That
layer is supposed to be part of the LLP.

>Although I lean toward an "in-order" send constraint to
>the LLP (only because it is how I presume virtually all
>DDP/RDMA layer will implement it anyway), it is not a
>strong conviction.  I do believe, however, that the draft
>MUST be explicit on what is and is not allowable with
>respect to DDP segmentation.  It is these sorts of subtle
>ambiguities that cause erroneous assumptions resulting in
>interoperability problems.
>

Philosophically I prefer absolute minimums in specs. However
this is a no-lose option. Whether it is permissable to send
them out of order or not, sending segments in Message Offset
order is the easiest most natural implementation anyway. And
the simplest most natural implementation of the receiver
won't care what the message offset of  any segment is in
other than the last one. It will simply place the message
at that offset (validating only that it fits within the
bounds of the target buffer).

Caitlin Bestler

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 11 Oct 2002 14:15:03 +0200

You won't have to check addresses and you must rely on the sender (data 
source) to have all the data (no scoreborading needed). But you will have 
to check carefully all the offsets and lengths to make sure that they are 
within the bounds of the memory associated with the data sink STag.

The results in case of overlapping may be somewhat unpredictable is the 
data changes.

Julo

Caitlin Bestler <[email-masked]>
Sent by: [email-masked]
11/10/02 01:00
 
        To:     [email-masked]
        cc:     "Larsen, Roy K" <[email-masked]>
        Subject:        RE: [rddp]  Clarification on DDP segmentation

Caitlin Bestler

---

From: "Caitlin Bestler" <[email-masked]>
Date: Fri, 11 Oct 2002 09:55:25 -0500

----- Original Message -----
From: Julian Satran
To: Caitlin Bestler
Cc: [email-masked] ; [email-masked] ; Larsen, Roy K
Sent: Friday, October 11, 2002 7:15 AM
Subject: RE: [rddp] Clarification on DDP segmentation

You won't have to check addresses and you must rely on the sender (data
source) to have all the data (no scoreborading needed). But you will have to
check carefully all the offsets and lengths to make sure that they are
within the bounds of the memory associated with the data sink STag.

The results in case of overlapping may be somewhat unpredictable is the data
changes.

-----------------------------------

Correct, redundant placements within an unfenced sequence will be
non-deterministic
as to which value will end up in the target memory. (And of course there is
no way to
fence within a single untagged message). The Data Source SHOULD NOT send
contradictory data, and it MUST NOT rely on the latter data superceding the
data it had sent earlier.

Which, now that I think about it more, means that there is absolutely no
valid reason
for sending the same data twice. The "corrected" data placement cannot be
relied
upon to be placed later than the original placement. Therefore, there would
be no
harm in elevating "should not" send contradictory redundant placements to
"must not".
In either case, the receiver does not need to be doing score-boarding, but
since it
would merely be wasting its own CPU cycles there is no need to include even
a
"should not" about it.