Subject: Re: [rddp] ddp architectural issue

From: "Vadim Makhervaks" <[email-masked]>
Date: Fri, 8 Nov 2002 11:29:16 +0200

Vadim Makhervaks
Haifa Development Lab, IBM Israel
phone: +972-4-829-6537, mobile: +972-52-97-6458
Lotus: Vadim Makhervaks/Haifa/IBM@IBMIL
e-mail:  [email-masked]

|---------+---------------------------->
|         |           "Williams, Jim"  |
|         |           <Jim.Williams@emu|
|         |           lex.com>         |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           06/11/02 07:49 PM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       [email-masked]                                                                                                 |
  |       cc:                                                                                                                     |
  |       Subject:  [rddp] ddp architectural issue                                                                                |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

Sorry if this has been discussed before.

Section 8.2.2 of draft-shah-iwarp-ddp-01.txt
discusses DDP Abortive Teardown.

A possible cause of abortive teardown is that
the transport had a gap in the received data
and timed out before being able to fill that
gap.

Assuming DDP layer is allowed to do out of order
placement, some of the data following the gap
may have already been placed.  This is a problem
if the transport used is either TCP or SCTP ordered
delivery.  Under the semantics of these transports,
that post gap data is not valid and should never have
been delivered.

I believe DDP can work around this by making a
statement something like the following.

    In the event DDP indicates to the ULP that
    a fatal error has occurred, the ULP MUST
    assume that invalid data may have been written
    any buffers for which that DDP stream had
    write access.  Any DDP writes for which
    the ULP has received completion from DDP
    can be assumed to have been correctly written,
    however there is no guarantee that they were not
    subsequently overwritten with invalid data.

<VM> Both statements are true not just for the case of abortive tear down,
and similar disclaimer should be added to the spec (like no-peeking-rule in
RDMA spec).
I would propose to state that: "ULP can consider buffers as valid, only if
DDP has DELIVERED corresponding DDP messages." I would not include the
"however" part of your proposal, because I think that overwriting of the
same buffer by several DDP messages is not a problem of DDP definition, or
ooo data placement. This is a misleading behavior of ULP. Even in case of
in-order placement, ULP transmitter cannot use the same buffer unless it
gets explicit status from the receiver that the previous access has been
seen by receiver. In that case even FENCE won't help too much. <VM>

It is tempting to do some hand waving as say that
since out of order data must have originated from
the remote end, and has been validated by a CRC,
it can be assumed to valid from a DDP perspective.
My concern is that this is an architectural rat hole,
and a clear violation of transport semantics.

There may be other creative solutions to this
problem.  I would like to see any other
thoughts on this.

 - Jim
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Williams, Jim" <[email-masked]>
Date: Fri, 8 Nov 2002 10:48:15 -0500

> -----Original Message-----
> From: Vadim Makhervaks [mailto:[email-masked]]
> Sent: Friday, November 08, 2002 4:29 AM
> To: Williams, Jim
> Cc: [email-masked]; [email-masked]
> Subject: Re: [rddp] ddp architectural issue
> 
> [...] 
> seen by receiver.

I think you are depending on the remote ULP sender
not to improperly overwrite buffers that have been
delivered to the ULP.
This is reasonable, but you are making an implicit
assumption that the "post gap" data received and
placed did in fact come from the remote ULP.
In a de facto sense it did.  But to rely on this
is to change the semantics of the transport layer.
The "post gap" data is data that the transport
definition requires not be delivered.

So I think you need either to admit that the
transport semantics are changed, or else
regard the data as invalid and not assume
it came from the remote ULP.  In the latter
case, it is hard to argue that it won't
improperly overwrite buffers that have already
be delivered to the ULP.  I don't see any
other alternative.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Uri Elzur" <[email-masked]>
Date: Mon, 18 Nov 2002 16:48:37 -0800

pls see [UE] comments below

Validating the DDP header itself is not enough. consider the following case, a DDP message has been segmented into 2 TCP segments. IFT receives the first one with a cleanly validated "DDP header" using the special header CRC. The NIC doesn't keep this data buffered locally and allows DDP to PLACE it (this is the proposed advantage of this additional CRC). Now the second part of the DDP message is received, but it has a CRC error. What is the proposed behavior here?

In this case, a ULP buffer is filled with data that might be incorrect! The ULP now needs to deal with recovery, not a fun task. If the solution is tearing the connection down, then some buffers may be filled with data that has not been Delivered. Isn't that the scenario you were trying to avoid below?

Note: MPA only places data that has been tested to be CRC clean, as the whole DDP segment is placed in one time!

Uri

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Friday, November 08, 2002 3:45 PM
To: Williams, Jim
Cc: [email-masked]
Subject: RE: [rddp] ddp architectural issue

On 11/8/02, Williams, Jim wrote:

>> From: Caitlin Bestler [mailto:[email-masked]]
>> Sent: Friday, November 08, 2002 11:33 AM
>> To: [email-masked]
>> Cc: Williams, Jim
>> On 11/6/02, Williams, Jim wrote:
>> 
>> It was not delivered, it was placed.
>
>I'm OK with that distinction, but how do you know
>that it was not placed incorrectly in a buffer
>which was previously completed by DDP and delivered
>to the ULP?  In order to know that, you must
>make some minimal assumptions about the correctness
>of the data, at least the DDP header.  But
>to assume anything about the data one must
>make some assumptions about how the transport
>behaves which are inconsistent with how the 
>transport is currently defined.
>
[UE] out-of-order data can not be completed/delivered to the ULP before all gaps have been "closed". No change to transport definitions. 

Data is placed in a buffer as directed by the ULP. DDP (using MPA) would place it after the TCP segment has been verified and MPA CRC checks ok (and some other checks). How is that inconsistent with any definition? IFT is using the same exact semantics for locating the buffer for data placement.
[UE]
Agreed, at the minimum the DDP Header must
have been validated. That can be done by
validating the entire DDP segment, or as
you have proposed with a seperate header CRC.

For TCP this would mean that the entire TCP
segment must have been received, so that its
CRC can be checked. Only then can the appropriate
DDP CRC be checked.

For SCTP this would mean that the SCTP chunk
must have been validated.

The assumption, which I believe is compatible
with both SCTP and TCP, is that the content
is only presented to the transport level once.

Hence the need to retransmit LLP segment N-1
can in no way invalidate an already received
LLP segment N.

In fact, If I understood the issue correctly,
it is this semantic which forbids resegmenting
a DDP Message to a new PMTU on retry.

I don't have all of the relevant TCP and SCTP
RFCs memorized by heart. But if those semantics
are not already on the record, I could see
adding that clarification to the DDP spec.

>I am not arguing that such changes to the
>transport definition are necessarily bad,
>just that they are outside the scope
>of this group, and therefore not acceptable
>in this context or in the DDP draft.
>
As aggravating as it is, the cost of using
a ubiquitous transport is that you have to use
it as is. Besides, there already is an RDMA
friendly transport defined, InfiniBand. The
whole point is deploy RDMA over the network
that is already in place.

>Alternatively, DDP must make a contract with the 
>ULP that it will not overwrite a buffer that
>has been completed and delivered to the ULP.
>So effectively all windows are write once
>and auto close.  This, however, is contrary
>to what is proposed in draft-shah-iwarp-ddp-01.txt,
>specifically section 7.1.1.          
>
Once we have verified that the target specification
was not subject to corruption, then we can only
presume that the ULP is aware that the target memory
is eligible for update. You can concentrate on the
race condition, but it is really part of a broader
problem of authorizing two concurrent upates to
the same shared resource.

>From a user guide's point of view, it should be
emphasized that the ULP should not assume that
it will *process* the results of message N
before placement of message N+M is in flight.
A prudent ULP will not re-advertise a buffer
until there has been some form of ULP round
trip (even if it is implicit by send credits,
i.e. the ULP will not restock the read queue
until it has processed the input buffer).

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 18 Nov 2002 22:19:46 -0500

> From: Uri Elzur [mailto:[email-masked]]
> Sent: Monday, November 18, 2002 7:49 PM
> To: Caitlin Bestler; Williams, Jim
> Cc: [email-masked]
> Validating the DDP header itself is not enough. consider the 
> following case, a DDP message has been segmented into 2 TCP 
> segments. IFT receives the first one with a cleanly validated 
> "DDP header" using the special header CRC. The NIC doesn't 
> keep this data buffered locally and allows DDP to PLACE it 
> (this is the proposed advantage of this additional CRC). Now 
> the second part of the DDP message is received, but it has a 
> CRC error. What is the proposed behavior here?

The connection would be closed with error, and the bad data
never delivered to the application.  This is 
consistent with DDP definition that if a connection closes
with error, the ULP may not assume anything about buffers
written but not delivered.  Any other assumption leads
to problems.

> [...]
>
> Note: MPA only places data that has been tested to be CRC 
> clean, as the whole DDP segment is placed in one time!

If the data is never delivered to the ULP, its correctness
is moot.  The ULP should not written in a way that depends
on the correctness of undelivered data.

---

From: "Uri Elzur" <[email-masked]>
Date: Mon, 18 Nov 2002 20:35:37 -0800

the same response  is applicable to the question you've raised before of MPA placing data out of order and then connection is tear down. ALL data that has been placed but NOT DELIVERED is ok as "ULP may not assume anything about buffers
written but not delivered.  Any other assumption leads
to problems."

Uri

---

From: "Garcia, Dave" <[email-masked]>
Date: Fri, 22 Nov 2002 10:14:31 -0800

> From: Jerry Chu [mailto:[email-masked]]
> Sent: Thursday, November 21, 2002 1:40 AM
> To: Garcia, Dave
> Cc: Caitlin Bestler; [email-masked]
> Dave,
> Garcia, Dave wrote:

snipped

> >
> >11. The LLP MUST NOT pass a duplicate DDP Segment to 
> >    the DDP Layer after it has passed all the previous DDP 
> >    Segments to the DDP Layer and the associated ordering 
> >    information for the previous DDP Segments and the 
> >    current DDP Segment. 
> >
> Yes I saw that paragraph but failed to parse the sentences following
> "after it ...".
> 
> >
> >You mentioned that this write-once rule may introduce
> >a slew of other problems. What might that be?
> >
> If a STag's scope (TBD) is defined by the RDMA layer or ULP, adding
> a constraint such as a write-once rule to the STag by the LLP 
> will likely
> to interfere with the scope/semantics of the STag defined by the ULP.
> 
> An in-order, reliable LLP's job is to deliver each byte of data of the
> same "sequence #", not the same "STag", exactly once.
> 
> Jerry
> 
> >
> >Dave Garcia
> >HP

Jerry, item 11 has nothing to do with an Stag's scope. It's a 
requirement on the LLP that each segment be passed up to DDP 
exactly once. It's just as you state in your last paragraph, the
LLP's job is to deliver each byte of data for the same "seq #"
exactly once. A buffer is completed only after the LLP has ensured
that it won't be written again. Once the ULP sees a completion no stale or
duplicate packets will overwrite the completed buffer. This is
exactly the semantics you're looking for.

Dave Garcia
HP