Subject: RE: [rddp] RDDP Buffer Issues

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sun, 1 Dec 2002 21:03:53 -0800 (PST)

>On 11/15/02, Williams, Jim wrote:
>
>>
>>
>>> -----Original Message-----
>>> From: Caitlin Bestler [mailto:[email-masked]]
>>> Sent: Thursday, November 14, 2002 7:52 PM
>>> To: [email-masked]'
>>> Cc: Williams, Jim
>>> Subject: Re: [rddp] RDDP Buffer Issues
>>>
>>> [...]
>>>
>>> So what you are proposing is a ULP restriction, or a
>>> requirement that the DDP actively prevent the ULP from
>being
>>> stupid. With limited exceptions, I do not believe this is
>an
>>> appropriate requirement. 
>>
>>I won't go through your entire response point by point. 
>>But I think there is a philosophical difference between
>>your approach to protocol design and what I am proposing. 
>>I also think that many will agree with your philosophy, so
>>I won't try to convert you.  But I will at least try to
>>make clear my motivation here.
>>
>>The primary goal is not to prevent the ULP from being
>>"stupid", although that may be a beneficial side effect.
>>The primary goal is to prevent the ULP from using 
>>other protocol layers in ways for which they were not
>>architected, particularly the transport layer.
>>
>>If anything the goal is to prevent the ULP from being "too
>>clever".  If the ULP is designed so that its proper
>>operations depends on transport properties that are not
>>architected, The ULP may work fine and be quite "happy". 
>>The problem is that non-architectural dependencies are
>>created between layers.  The result of this is that the

TO avoid this problem the required semantics of the transport
where DDP sits on top must be clearly stated. E.g. for DDP
over an ordered transport, it is the job of the transport alone,
not DDP, nor ULP, DDP+LLP or any other combination that must
ensure the in-order exactly once data delivery. The only exception
clause is the allowing of out-of-order placement as long as the
target seq# hasn't been delivered, i.e. the said data segment is
not a duplicate from the transport's perspective, and all the
other transport checks have passed.

Any DDP/LLP implementations that expose a different semantic
can be declared non-compliant and outlawed.

If we agree that a clear semantic separation between layers is
desireable, then the job becomes finding a flexible architecture
that can enable a high-performance implementation. This may turn
out to be difficult.

>>overall internet architecture becomes very brittle and
>>difficult to evolve.  Architects and implementors working
>>on a specific protocol layer do things that seem perfectly
>>reasonable from their perspective, but end up breaking
>>some pre-existing ULP they never even heard of.
>>
>>The whole internet is currently suffering 
>>greatly from this type of problem.
>>
>>So the ULP actually may have little incentive
>>to avoid being "stupid".  But the IETF can and
>>should have significant incentive to make sure
>>the "stupid" ULPs don't proliferate.
>>Figuring out a mechanism to prevent this is
>>not easy and requires work.  Understandably, 
>>this work may seem a bit silly and 
>>overly nit picking to those whose primary
>>interest is in getting an "ad hoc" solution
>>in place that works and doing it quickly.
>>
>>
>
>I also agree that applications should be discouraged from
>relying upon implementation quirks.
>
>However, I view the fact that a ULP buffer advertisement is
>ULP sliceable as a required feature, and not as an
>implementation quirk.
>
>I think a case can be made that the local ULP MUST be given
>the ability to invalidate an STag synchronously with an
>untagged received. This would have the benefit of avoiding
>reliance upon a potentially suspect peer to avoid the
>post-receive vunerability window. However, for reasons I

If the vulnerability is at the ULP level, then it is an ULP
issue that is best dealt with separately from the discussion of
the DDP/LLP transport to cause less confusion. (Some of the
discussions in the reflector so far have confused a hell of me
:-(.)

>have covered earlier, I do not view it as a major
>vunerability.
>
>If the Data Sink ULP lacks trust in its peer, then it SHOULD
>use one-shot STags that are enabled ONLY for the target
>buffer. It SHOULD invalidate them manually before examining
>the content of the received buffer, or disconnect if the
>remote end failed to request the invalidation.

Is this vulnerability at ULP level or at the transport level?
If it is at the transport level, then it violates the big rule
stated above. That is, a transport layer issue must NOT be
addressed at the ULP layer!

Also the one-shot STag doesn't go well with a traditional
transport like TCP. In a TCP implementation an OOO placement of
the same seq# can take place more than once before the final in-
order delivery (considering all the complexity in its retransmission
algorithm). Also TCP only guarantees the exact once delivery of
data based on seq#, not STag (so if ULP sends two set of data
towards the same remote memory location it'd cause the remote TCP
to choke.)

Jerry
 
>
>Those are not onerous restrictions, but mandating the
>existence of a local invalidate option would improve the
>situation. It certainly feels cleaner.
>
>Beyond that, however, I do not see how any of your
>suggestions can be made to work without requiring that all
>STags be made one-shot transfer buffers. This is far from
>optimum for a large number of applications.
>
>I remain firmly convinced that management of shared buffers
>is a ULP issue, and that the hardware MUST NOT impose any
>standard solutions that all applications must follow.
>
>Caitlin Bestler
>http://asomi.com/CaitlinBestler/
>_______________________________________________
>rddp mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 2 Dec 2002 10:10:20 -0600

On 12/1/02, H.K. Jerry Chu wrote:

>
>TO avoid this problem the required semantics of the
>transport where DDP sits on top must be clearly stated.
>E.g. for DDP over an ordered transport, it is the job of
>the transport alone, not DDP, nor ULP, DDP+LLP or any
>other combination that must ensure the in-order exactly
>once data delivery. The only exception clause is the
>allowing of out-of-order placement as long as the target
>seq# hasn't been delivered, i.e. the said data segment is
>not a duplicate from the transport's perspective, and all
>the other transport checks have passed.
>
>Any DDP/LLP implementations that expose a different
>semantic can be declared non-compliant and outlawed.
>
>If we agree that a clear semantic separation between
>layers is desireable, then the job becomes finding a
>flexible architecture that can enable a high-performance
>implementation. This may turn out to be difficult.
>

There is "out-of-order placement" meaning that placement is
enabled even when a packet arrives out-of-transport order.

Equally important there is the option of the Data Source to
send data in a data-source convenient order, rather than in
the "expected" order. The most obvious example being sending
the data sectors requested by a read in the order that the
drive head(s) reached them, rather than in their logical
order. Hence block 7 could be sent "first" (LLP sense),
followed by blocks 3,4,0,1,2,5 and 6.

>Is this vulnerability at ULP level or at the transport
>level? If it is at the transport level, then it violates
>the big rule stated above. That is, a transport layer
>issue must NOT be addressed at the ULP layer!
>
>transport like TCP. In a TCP implementation an OOO
>placement of the same seq# can take place more than once
>before the final in- order delivery (considering all the
>complexity in its retransmission algorithm). Also TCP only
>guarantees the exact once delivery of data based on seq#,
>not STag (so if ULP sends two set of data towards the same
>remote memory location it'd cause the remote TCP to choke.)
>
That is why STag invalidation is correctly tied to the
completion of an *untagged* message. If completion can be
signalled on an untagged message, then the LLP seq# is
already passed and is no longer eligible for reception.

This would apply for both remote STag invalidation (as in
the wire protocol) and local STag invalidation (as in the
local interface option that has been suggested, and nobody
seems to really object to other than the obvious fact that
it is not *required* for interoperability and hence best
left out of an IETF standard).

Caitlin Bestler