Subject: RE: [rddp] Review comment on DDP untagged buffer model

From: "Williams, Jim" <[email-masked]>
Date: Mon, 14 Jul 2003 12:15:15 -0400

The standard use of TCP allows that the 
application (in this case RDDP) may stop accepting
data from TCP at any point, and TCP will manage
flow control, and buffer any interim data, until
the application is able to accept more data.

The definition of RDDP was intended to relieve
TCP of this responsibility, and therefore the
need to buffer a windows worth of data for an
arbitrary period of time, by requiring RDDP
is always able to accept data from TCP.  So
all received data will have a destination
when it arrives, or if not, an error condition
has occurred.

So the question is we have relieved TCP from
the responsibility of buffering data for
application level flow control, but should
we forbid it from doing so to help out RDDP?

In the context of the above, Mallikarjun's 
proposal seems reasonable.

I would raise one caution, though.  My experience
from VI is that failure to correctly handle
flow control at the application level was
one of the most common bugs and support issues
we needed to deal with.  The need to insure
that receive buffers were always pre-posted
before data was sent seemed to be a foreign
concept to a lot of application programmers,
and even those who fully understood it, often
got it wrong.

The technical arguments for doing receive buffer
flow control at the ULP level rather than the
RDMA level seem to have some merit, but the
decision could come back to haunt us.  In particular
if a significant subset of RDMA implementations
are able to handle flow control without help
from the ULP, I would expect a proliferation
of application code that requires this help
in order to run correctly.  The ultimate result
would be a de facto requirement for RDMA
to do flow control (or else forgo a large
percentage of application code that requires it).

So my order of preference might be something
like the following.

   1.  Provide a clean architectural solution to
       allow RDMA to do receive buffer flow control.
       Applications would be allowed to send data
       before the receive buffers were available.
       RDMA layer would not allow data to be sent
       on the wire until receive buffers were posted
       and available.

   2.  Require ULP to do receive buffer flow control.
       In the event that ULP violates this requirement
       and sends data before receive buffer is available,
       the RDMA receiver MUST indicate a fatal error,
       and the ULP bug must be fixed.

   3.  Require ULP to do receive buffer flow control,
       but if it fails to do this properly, things will
       still work most of the time.  Things may still
       work all the time with some RDMA implementations.

Perhaps it's just a personal bias, but I hate things 
that work most of the time.  I would much rather see
things either work always, or fail quickly so they 
can be fixed. 

> -----Original Message-----
> From: Mallikarjun C. [mailto:[email-masked]]
> Sent: Thursday, July 10, 2003 10:45 PM
> To: [email-masked]
> Subject: [rddp] Review comment on DDP untagged buffer model
> 
> 
> I  have a comment on the DDP untagged buffer model
> of the current DDP spec.
> 
> The DDP spec implies that the DDP Stream is to be
> dropped by implementations when there is no receive
> buffer to place an inbound untagged message.  This
> is mandating a specific type of implementation, and IMHO,
> is needless.  The DDP spec should instead allow for
> implementation flexibility in this area.  Those RNICs
> that can gracefully deal with lack of receive buffers
> via LLP means (such as TCP retransmit) should be
> allowed as well.  I can agree with the authors' original intent
> that ULP-level flow control and/or adequate buffer provisioning
> must be done to avoid needless retransmits, but mandating
> a connection drop on the first instance of hitting a (most
> likely a very transient) no-buffer window is severe.
> 
> I believe the DDP spec can allow this RNIC implementation
> flexibility with minimal text changes.  My review uncovered
> only two areas of text that mandated the connection drop
> requirement.
> 
> section 3.2 (Protocol Overview),
>     " Note, however, that
>         the ULP needs to address flow control issues because if a DDP
>         Message arrives for an Untagged Buffer without an associated
>         receive ULP buffer, the DDP Message is dropped, the DDP Stream
>         is disabled for reception, and an error is reported to the ULP
>         at the Data Sink."
> 
> and section 9.1  (Errors detected at the Data Sink).
>     "   2.  The QN and MSN have an associated buffer that 
> allows Placement
>        of the payload."
> 
> My suggestion is to change the text along the following lines:
> Section 3.2 to:
>     " Note, however, that the ULP needs to address flow 
> control issues."
> and section 9.1 to:
>     " 2. The QN and MSN have an associated buffer that allows 
> Placement of the payload. DDP implementations
> MAY optionally consider lack of an associated buffer to be 
> not an error, but such implementations MUST ensure
> that the DDP Segment is eventually recovered using LLP means 
> in a ULP-transparent way."
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions
> Hewlett-Packard MS 5668
> Roseville CA 95747
> [email-masked]
> 
> 
> 
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 12:20:11 -0500

>things either work always, or fail quickly so they 

I have an identical bias against things that work most
of the time. I prefer my bugs to be blatant and painful.

But, bugs and faults are inevitable. The question is
how to deal with them.

To my tastes, one of the least useful terms in use within
software engineering is "fault tolerant". I am never tolerant
of faults. I'll try to design my system to keep running in
the face of faults, to contain any damage caused by a fault,
to be robust in the face of non-compliance by the peer, etc.
But it does not mean I am *tolerant* of those problems.

So, with my draconian design-review hat on I can very simply
state the local ULP MUST provide sufficient buffers to cover
all authorized untagged messages that may arrive. Period.
If you want to do statistical under-provisioning, fine, just
be certain that you are providing enough buffering.

But if you shift perspective from the design of a single ULP
to the robustness of the entire system, different concerns 
leap to the forefront. Taking a zero tolerance approach has
the danger of creating a cascading fault.

Consider a ULP that was responsibly about to post extra
buffers to a buffer pool, but delivery of the message
has been delayed because of a system response to another
unrelated fault. Perhaps a disk drive is down and the extra
traffic to recreate the spare has clogged the IPC mechanisms.
Or perhaps the portion of the ULP responsible is just being
delayed in being swapped in.

I prefer designing high volume servers as embedded systems
with deterministic response times. But I would not limit RDMA
servers to such environments. And if you require a buffer pool
to be adequate under *all* circumstances, even in the face of
other unrelated system faults, then you are essentially saying
you cannot use buffer under-provisioning on a Unix or Windows
server unless you are the *only* daemon running on it.

An implementation preventing cascading of faults strikes me as
not only permissible, but even commendable.

An implementation that covers up sloppy ULP practices may be
hard to forbid, but it certainly not commendable. It undermines
interoperability. If you want to help the ULP, provide great
diagnostics, not coddling. If the implementatino is covering
up ULP flaws and not even logging a complaint then, in my
opinion, it is going to far.

Perhaps that is the real criteria in the back of my mind, its
ok to fix a fault as long as it is infrequent enough that the
log entries reporting it don't become an issue. If the user needs
to suppress the log entries, then something has gone wrong.

But back on your first point. There is no mechanism that will
allow the sending side to defer sending a message. That would
require an RDMA layer ack. I believe the decision not to piggy-
back send-credits on untagged messages was the correct one.

If the receiving side explicitly knows how many untagged messages
the ULP is expecting, then there is a natural enforcemnt of flow
control at the receiving side. If 'n' have been enabled, then that
is the range of valid MSNs. An MSN out of that range is an invalid
message. Period. It is a ULP fault at one end or the other. Tearing
down the DDP Stream is the correct response.

Valid MSN without a buffer can be a different situation, though.
Unless we can determine that the absence of a buffer is *always*
the fault of the receiving ULP I think we need to allow the
stack to be robust.

But the rule for ULP developers should remain:
-- enable the next receive (typically by posting a buffer)
-- THEN send the response.

My experience has also been that lots of app programmers keep
wanting to reverse those two. You can find rants about the same
essential bug in Jackson System Design back in the 70s. The fact
that this is a persistent application design mistake is exactly
why we need to be very clear that the ULP *is* responsible for
providing buffers *before* the messages arrive.