Subject: RE: [rddp] ddp architectural issue

From: "Williams, Jim" <[email-masked]>
Date: Fri, 8 Nov 2002 14:17:58 -0500

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, November 08, 2002 11:33 AM
> To: [email-masked]
> Cc: Williams, Jim
> Subject: Re: [rddp] ddp architectural issue
> 
> 
> On 11/6/02, Williams, Jim wrote:
> 
> >A possible cause of abortive teardown is that
> >the transport had a gap in the received data
> >and timed out before being able to fill that
> >gap.
> >
> >Assuming DDP layer is allowed to do out of order
> >placement, some of the data following the gap
> >may have already been placed.  This is a problem
> >if the transport used is either TCP or SCTP ordered
> >delivery.  Under the semantics of these transports,
> >that post gap data is not valid and should never have
> >been delivered.
> >
> 
> It was not delivered, it was placed.

I'm OK with that distinction, but how do you know
that it was not placed incorrectly in a buffer
which was previously completed by DDP and delivered
to the ULP?  In order to know that, you must
make some minimal assumptions about the correctness
of the data, at least the DDP header.  But
to assume anything about the data one must
make some assumptions about how the transport
behaves which are inconsistent with how the 
transport is currently defined.

I am not arguing that such changes to the
transport definition are necessarily bad,
just that they are outside the scope
of this group, and therefore not acceptable
in this context or in the DDP draft.

Alternatively, DDP must make a contract with the 
ULP that it will not overwrite a buffer that
has been completed and delivered to the ULP.
So effectively all windows are write once
and auto close.  This, however, is contrary
to what is proposed in draft-shah-iwarp-ddp-01.txt,
specifically section 7.1.1.          

> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 8 Nov 2002 17:45:04 -0600

>transport is currently defined.

Agreed, at the minimum the DDP Header must
have been validated. That can be done by
validating the entire DDP segment, or as
you have proposed with a seperate header CRC.

For TCP this would mean that the entire TCP
segment must have been received, so that its
CRC can be checked. Only then can the appropriate
DDP CRC be checked.

For SCTP this would mean that the SCTP chunk
must have been validated.

The assumption, which I believe is compatible
with both SCTP and TCP, is that the content
is only presented to the transport level once.

Hence the need to retransmit LLP segment N-1
can in no way invalidate an already received
LLP segment N.

In fact, If I understood the issue correctly,
it is this semantic which forbids resegmenting
a DDP Message to a new PMTU on retry.

I don't have all of the relevant TCP and SCTP
RFCs memorized by heart. But if those semantics
are not already on the record, I could see
adding that clarification to the DDP spec.

>in this context or in the DDP draft.
As aggravating as it is, the cost of using
a ubiquitous transport is that you have to use
it as is. Besides, there already is an RDMA
friendly transport defined, InfiniBand. The
whole point is deploy RDMA over the network
that is already in place.

>specifically section 7.1.1.          
Once we have verified that the target specification
was not subject to corruption, then we can only
presume that the ULP is aware that the target memory
is eligible for update. You can concentrate on the
race condition, but it is really part of a broader
problem of authorizing two concurrent upates to
the same shared resource.

>From a user guide's point of view, it should be
emphasized that the ULP should not assume that
it will *process* the results of message N
before placement of message N+M is in flight.
A prudent ULP will not re-advertise a buffer
until there has been some form of ULP round
trip (even if it is implicit by send credits,
i.e. the ULP will not restock the read queue
until it has processed the input buffer).

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Sun, 10 Nov 2002 15:20:39 -0800

> >> It was not delivered, it was placed.

Correct.  Delivered implies a completed message was received and if there 
is a gap, then it is not complete thus the buffers are undefined.  All ULP 
must be designed to account for this which is clearly a reasonable request 
and one requirement that has been used in other technologies for quite some 
time.

>CRC can be checked. Only then can the appropriate

DDP is defined to be capable of operating using a well-defined layered 
architecture. Thus the transport must validate the packet before it is 
handed to the DDP layer (from the transport's view this is just a ULP - 
really no different than what is done for iSCSI or other ULPs operating 
over the Internet today).  As such, one must receive the TCP segment in its 
entirety before processing any ULP headers / data.  This is one of the 
reasons why there is no DDP header CRC as one must receive the entire 
segment to validate it from the transport layer first in order to maintain 
a well-layered architecture approach.

>can in no way invalidate an already received

Correct.  The transport is responsible for only delivering valid data (from 
its perspective) to the ULP  - no duplicates, etc. allowed.

>As aggravating as it is, the cost of using a ubiquitous transport is that 
>you have to use it as is. Besides, there already is an RDMA friendly 
>transport defined, InfiniBand. The whole point is deploy RDMA over the 
>network that is already in place.

There are no transport changes being proposed by the use of DDP / RDMA 
technology.  All of these are just payload from the transport's perspective 
thus these specifications can be implemented over any compliant transport 
which meets the defined LLP requirements (clearly TCP meets these 
requirements).

>Once we have verified that the target specification was not subject to 
>corruption, then we can only presume that the ULP is aware that the target 
>memory is eligible for update. You can concentrate on the race condition, 
>but it is really part of a broader problem of authorizing two concurrent 
>upates to the same shared resource.

This has always been the case for any ULP.  It must be aware of the memory 
that it is updating either explicitly or implicitly by the data being 
exchanged.  Thus it is not a DDP/RDMA problem to prevent a ULP from 
overwriting its buffers aside from the enforcement of STags and base / len 
checks.

> From a user guide's point of view, it should be emphasized that the ULP 
> should not assume that
>it will *process* the results of message N before placement of message N+M 
>is in flight. A prudent ULP will not re-advertise a buffer until there has 
>been some form of ULP round trip (even if it is implicit by send credits, 
>i.e. the ULP will not restock the read queue until it has processed the 
>input buffer).

Strictly a ULP issue.

Mike

---

From: Jerry Chu <[email-masked]>
Date: Tue, 19 Nov 2002 14:18:45 -0800

>> On 11/8/02, Williams, Jim wrote:
<snip>

> handed to the DDP layer (from the transport's view this is just a ULP 
> - really no different than what is done for iSCSI or other ULPs 
> operating over the Internet today).  As such, one must receive the TCP 
> segment in its entirety before processing any ULP headers / data.  
> This is one of the reasons why there is no DDP header CRC as one must 
> receive the entire segment to validate it from the transport layer 
> first in order to maintain a well-layered architecture approach.

Are you saying DDP must not perform any placement until after LLP 
processing?
(This seems to be implied from the architecture draft but this thread of 
discussions
seems to have allowed placement before a complete LLP processing.)

> Correct.  The transport is responsible for only delivering valid data 
> (from its perspective) to the ULP  - no duplicates, etc. allowed.
>
>
>> As aggravating as it is, the cost of using a ubiquitous transport is 
>> that you have to use it as is. Besides, there already is an RDMA 
>> friendly transport defined, InfiniBand. The whole point is deploy 
>> RDMA over the network that is already in place.
>
>
> technology.  All of these are just payload from the transport's 
> perspective thus these specifications can be implemented over any 
> compliant transport which meets the defined LLP requirements (clearly 
> TCP meets these requirements).
>
>> Once we have verified that the target specification was not subject 
>> to corruption, then we can only presume that the ULP is aware that 
>> the target memory is eligible for update. You can concentrate on the 
>> race condition, but it is really part of a broader problem of 
>> authorizing two concurrent upates to the same shared resource.
>
>
> This has always been the case for any ULP.  It must be aware of the 
> memory that it is updating either explicitly or implicitly by the data 
> being exchanged.  Thus it is not a DDP/RDMA problem to prevent a ULP 
> from overwriting its buffers aside from the enforcement of STags and 
> base / len checks. 

WHile I  tend to agree that changes to the content of a buffer that has 
been posted to LLP
regardless of whether successful transport operations have been 
performed is not a big
violation of the existing transport semantics (the semantic in this case 
has not been well defined
IMHO), it is clearly a major vioaltion of the current transport 
semantics if a buffer that has been
returned to ULP is overwritten by the LLP later.

>
>
>> From a user guide's point of view, it should be emphasized that the 
>> ULP should not assume that
>> it will *process* the results of message N before placement of 
>> message N+M is in flight. A prudent ULP will not re-advertise a 
>> buffer until there has been some form of ULP round trip (even if it 
>> is implicit by send credits, i.e. the ULP will not restock the read 
>> queue until it has processed the input buffer).
>
Not sure I understand the arguement here (still playing catch up), if a 
buffer posted by NFS received
good data and the data has been committed to the disk, can the VM system 
freely reuse the buffer
without worrying about buffer getting corrupted from DDP of duplicate 
pkts? I hope the answer
is a resounding YES.

Rgds,

Jerry

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 19 Nov 2002 17:05:05 -0600

On 11/19/02, Jerry Chu wrote:

>Not sure I understand the arguement here (still playing
>catch up), if a buffer posted by NFS received good data
>and the data has been committed to the disk, can the VM
>system freely reuse the buffer without worrying about
>buffer getting corrupted from DDP of duplicate pkts? I
>hope the answer is a resounding YES.
>

The Data Source ULP MUST NOT use an advertised buffer in a
way that contradicts the ULP specific understanding. Hence,
if the understanding is that the buffer is written  only
until the untagged response completes the transaction then
you will have no problem.

You can go further, by having the Data Source invoke the
STag Invalidation option, so as to ensure that the STag
is invalidated precisely when the untagged buffer completes.

However, there are a few gotchas if you do NOT trust the
Data Source.

-- The Data Source could simply not set the STag Invalidate
    option, and/or could otherwise continue to use the
    STag in ways contrary to the ULP-specific agreements.
    
-- The Data Source could even 'hide' its deception by
    actually setting the STag Invalidate, and then
    deliberately ensuring that 'later' tagged puts
    arrive ahead of it. The receiver, not having seen
    the invalidate yet, will process these requests.
    Because they are *after* the untagged completion,
    nothing will ensure that the cache flush will
    complete before the Data Sink ULP receives the
    completion notification.
    
There are generally two solutions to dealing with peers
that cannot be trusted to follow the ULP protocol:

-- Do not expose your own buffers. This is the solution
    that DAFS (Direct Access File System) uses for servers.
    Only client memory is exposed. There are other valid
    reasons for this, dealing with the overhead of
    committing server memory for thousands of concurrent
    clients who are under no obligation to be prompt.
    
-- Explicitly invalidate the STag locally, and *then*
    commit the buffer to disk.

As a general principle, DDP does not expose buffers to
remote access beyond the degree authorized by the ULP.
However, designers must remember that the DDP layer can only
enforce at the STag layer. Using a single STag for a
session, and then having multiple buffers exchanged over the
life of the session can reduce overhead. However, it
requires that the peer be somewhat trusted.

Caitlin Bestler

---

From: Jerry Chu <[email-masked]>
Date: Tue, 19 Nov 2002 17:46:12 -0800

This may be a confusion of my part. If the ooo placement issue the original
thread brought up is concerning ooo at the ULP level, then I agree it's 
really a ULP
issue. Somehow I got the impression that it was talking about DDP placing
data that is ooo from an ordered LLP perspective. (Or  was it?)

Anyway I've been wondering if it is possible to allow DDP placement to 
be done
before LLP processing. I understand that processing pkts in this order 
violates the
layering of DDP over LLP. But it may much simplify the RNIC design. Sipping
through various drafts I don't seem to find a clear answer. (Many paragraphs
seem equally confusing to me as to what layer(s) the context of the 
discussions are.)

Rgds,

Jerry

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 19 Nov 2002 21:19:34 -0600

On 11/19/02, Jerry Chu wrote:

>This may be a confusion of my part. If the ooo placement
>issue the original thread brought up is concerning ooo at
>the ULP level, then I agree it's really a ULP issue.
>Somehow I got the impression that it was talking about DDP
>placing data that is ooo from an ordered LLP perspective.
>(Or  was it?)

The questions of RNIC mechanics are not really being
challenged. It is more a question of semantics, and the
tradeoff between flexibiilty and safety.

As currently defined, each completion of an untagged message
to the Data Sink ULP implies the delivery of one more more
tagged transfers as well -- BUT which tagged transfers were
associated with than untagged transfer is not validated by
the DDP layer. It is something that is only known, and
understood at the ULP layer.

If the Data Source ULP is trustworthy and reliable there is
absolutely no issue. The question is whether the protocol,
as currently defined, presents opportunities for accidental
and/or malicious breakings of the presumed relationship.

What Jim was suggesting was providing a tighter coupling
between the tagged transfers and the untagged transfer. For
example, each STag would be valid for exactly one transfer.

The counter-argument is two-fold.

First, the Data Source already has sufficient tools to
protect itself. Further, this could be improved in the local
interface without requiring any changes to the protocol.

Secondly, the single tagged buffer per untagged message is a
very useful and powerful ULP strategy, but it is far from
the only one. There are numerous valid examples where ULP
peers can legitimately engage in reliable complex data
transfers, and no feasible way for hardware filters to
validate that the actual transfers conform to a complex
protocol.

>
>Anyway I've been wondering if it is possible to allow DDP
>placement to be done before LLP processing. I understand
>that processing pkts in this order violates the layering
>of DDP over LLP. But it may much simplify the RNIC design.
>Sipping through various drafts I don't seem to find a
>clear answer. (Many paragraphs seem equally confusing to
>me as to what layer(s) the context of the discussions are.)

The RNIC can place before LLP ordered delivery because the
target buffer is undefined until completion. It is exactly
this semantic that provides concern if the peer is not
trusted. Suppose the Data Sink ULP is merely trusting the
Data Source ULP to only write a given advertised buffer
before sending the untagged message, and not enforcing this
with per-buffer STags. Then buffers the Data Sink ULP thinks
are deterministic are in fact undefined, leading to the risk
of applications that stop working because of timing flukes
or reconfigured networks.

In any event, nothing proposed would limit the ability of
the RNIC to directly place each segment as it arrives. The
question is whether to more tightly control the enabling of
local buffers so that the RNIC can slam the door shut
reliably and promptly, or whether the RNIC must avoid
placing unnecessary restrictions on the ULP.

I favor the latter position, especially because the Data
Sink already has the tools to protect itself when it feels
the need to do so.

Caitlin Bestler

---

From: Jerry Chu <[email-masked]>
Date: Wed, 20 Nov 2002 10:18:22 -0800

>absolutely no issue. The question is whether the protocol,
I'm not sure about the "absolutely no issue" part. Many ULPs also perform
retransimission at the session layer even when running on top of a 
reliable transport
due to the different requirement in the semantics of the reliability 
from different layers.
A spurious retransmission at the ULP layer will be caught and discarded 
by the
receiving end ULP when running on top of a traditional, non-DDP transport.
It casuses no damage because data placement was made to an anonymous buffer.
But with DDP and the fact that placement occurs before complete ULP
processing, it seems to open up many holes even for the regular, 
non-malicious
cases.

>target buffer is undefined until completion. It is exactly
This is not what I'm concerned about. I'm willing to accept an added 
semantic
to the existing transports that once a buffer is posted to receive data, 
its content
may change regardless of whether legitimate data have landed in the 
buffer or not

But my concern is with duplicate pkts. If DDP placement happens before LLP
processing, a dup pkt may corrupt a buffer that has been delivered to ULP
unless its Stag has been invalidated somehow.

To prevent the above from happening I think some minimal LLP processing is
required, at least to catch and discard dup pkts. Tom Talpey mentioned 
to me a
rule of no more than one update per buffer needs to be enforced. Is that 
what
people agree upon? (I don't see it written down anywhere.) Also that 
write-once
rule may introduce a slew of other problems too.

>or reconfigured networks.
I'm not worried about any of the trust issues yet cuz I haven't been 
able to convince
myself all the basic cases are well covered.

Thanks,

Jerry

---

From: Stephen Bailey <[email-masked]>
Date: Thu, 28 Nov 2002 13:36:14 -0500

Jerry,

> processing?  (This seems to be implied from the architecture draft
> but this thread of discussions seems to have allowed placement
> before a complete LLP processing.)

Yes, that is what the architecture draft says.  Obviously (well,
apparently not), you can not specify a protocol on top of another
without respecting the lower-layer's exported interface and
requirements.

In the case of TCP, DDP can not know anything about `LLP processing'.
The LLP (TCP) delivers a stream of bytes, and status indications.  LLP
processing is a purely internal function of an implementation of TCP.

However, an IMPLEMENTATION of DDP+TCP can certainly take advantage of
characteristics of the combined pair of protocols as long as the
resulting implementation behaves exactly as specified by the two
individual protocol definitions.  TCP permits substantial choices
(nondeterminism in the specification) to the implementation.  I hope
(expect) DDP will as well.

The DDP(+RDMA) architecture provides (tries to) for optimized
implementations without violating strict layering in the
specification.  A DDP implementation is allowed substantial freedom in
edits (data placement, or set() operations in the lingo of the
architecture) it can perform on registered buffer.  In fact, the
architecture draft itself doesn't specify any constraints on the
set()s performed on a registered buffer.  Constrainting set()s is
delegated to the protocol specifications. If you read my old
*-ddpp-core-*, it defined set() ordering on a sequenced, reliable
transport in a sufficiently flexible way that you COULD (probably,
obviously nobody had yet), implement a combined TCP+DDP that placed
data directly from TCP segments.  However, you still couldn't violate
any `laws of physics' of the transport.  You couldn't place data
before you had performed transport layer validation (checksum,
windows, etc.).

Put another way, pretty much any behavior of an optimized, combined
implementation could also be seen in a strictly layered, though, you
might also argue, stupid, and certainly not efficient, implementation.

We really shouldn't be talking about `partial LLP processing' except
as a means of sieving our architecture and protocol to ensure it CAN
be implemented efficiently in a combined way (or not, if the consensus
of the group is that combined implementations are uninteresting).
Neither the protocol nor the architecture should say anything about
it.  In fact, it seems historically, that it has been best to leave as
much meaningful implementation choice as you can in the protocol and
architecture, while still getting the job done.

Steph