Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required

From: John Hufferd <[email-masked]>
Date: Tue, 15 Jul 2003 09:07:48 -0700

I believe we are moving towards general agreement on not being hard on the 
ULP that uses Buffers in a buffer pool or Shared RQ type of an operation 
(in other words, we seem to be agreeing that we may find words to permit 
special handling on a, hopefully rare missing buffer/WQE event.)

So with that assumption,  I think we might still need to be careful as we 
talk through the words that imply a connection must be drop for non shared 
environments when a buffer (or WQE) is missing.

As a point of discussion let me give you an example of an application 
which in normal situations always have its buffers posted for the remote 
node to use when issuing RDMA "Send" type messages, but can have problems 
in special cases.  An example of the special case may be an 
Error/Exception handling async type message.  Generally these types of 
messages are completely unpredictable about when they may occur, and they 
also may have an unpredictable number of such message.  (Many times when 
things go wrong, lots of things also seem to go wrong.)

Though this is a very rare situation, the important point is that often 
the timing and number of messages are not well predictable.  Also, it is 
not reasonable for the failing node to negotiate for more credits as it is 
attempting to send messages about its current trauma.

So the point I want to make that many times it is important for there to 
be a graceful handling of a missing buffer, even in the case of non shared 
buffers/queues.  Based on this I am aware of system vendors that would 
like to build their systems from RNICs that would permit graceful handling 
of missing buffers even in the non shared case.  And I think we should 
give careful consideration to this desire.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 15 Jul 2003 11:43:22 -0500

On Tuesday, July 15, 2003, [email-masked] wrote:

>I believe we are moving towards general agreement on not being
>hard on the ULP that uses Buffers in a buffer pool or Shared RQ
>type of an operation (in other words, we seem to be agreeing
>that we may find words to permit special handling on a,
>hopefully rare missing buffer/WQE event.)
>
>So with that assumption,  I think we might still need to be
>careful as we talk through the words that imply a connection
>must be drop for non shared environments when a buffer (or WQE)
>is missing.
>
Agreed. The distinction I prefer is that the credit MUST be there
(i.e., the WQE) but that the buffer itself can be under-provisioned
and therefore on rare occassions might be absent. The first is
a violation by the peer, the latter is a local fault. Recovery
from local faults are not only permissable, but commendable.

>As a point of discussion let me give you an example of an
>application which in normal situations always have its buffers
>posted for the remote node to use when issuing RDMA "Send" type
>messages, but can have problems in special cases.  An example
>of the special case may be an Error/Exception handling async
>type message.  Generally these types of messages are completely
>unpredictable about when they may occur, and they also may have
>an unpredictable number of such message.  (Many times when
The error/exception handling is a double-edged sword here.

If the server under-provisions buffers, and there is a rare
surge of outside traffice that depletes the buffer pool and
triggers a "low buffer" exception.

Under normal circumtances this exception would be processed
in time for the buffer pool to be replenished. But because
of another operation on the same OS being at its peak activity
level the exception is not processed quickly enough. Recovering
from such a fault is a very good idea. As I commeneted previously,
it is unreasonable to expect a daemon on a Unix/Windows type system
to take responsibility for worst case performance. You cannot
build absolute guarantees when your OS only gives you statistical
guarantees.

But I believe that reliance on an Error/Exception handler to
respond to what may be a Denial of Service attack is a broken
interface with severe security implications.

>Though this is a very rare situation, the important point is
>that often the timing and number of messages are not well
>predictable.  Also, it is not reasonable for the failing node
>to negotiate for more credits as it is attempting to send
>messages about its current trauma.
>
If the ULP is responsible for pacing, then the number of untagged
messages is not unpredictable. It is known.

If it is known to the ULP it can easily be communicated to
the DDP stack. This is traditionally done by posting receive
operations. A minimal change is to simply allow posting of
receives that specify a buffer pool rather than a buffer.
Further optimizations can be made by piggy-backing this
operation on sending the reply.

>So the point I want to make that many times it is important for
>there to be a graceful handling of a missing buffer, even in
>the case of non shared buffers/queues.  Based on this I am
>aware of system vendors that would like to build their systems
>from RNICs that would permit graceful handling of missing
>buffers even in the non shared case.  And I think we should
My fear is similar to one that Jim Williams expressed. I do
not want RNIC vendors taking on duties that belong to the ULP
and presenting them as though they were RDMAP/DDP functions.

If someone were to build a DAFS server on top of an RNIC that
did not enforce untagged buffer limits then the DAFS server
would have no reason to seriously negotiate the per-session
in-flight limits. Such a DAFS server would fail to work on a
compliant RNIC.

Additionally, on a pure-network basis, ULP level flow control
is vital to being network friendly.

If the ULP *wants* to issue more credits than it advertises,
then it should be allowed to do so. But having the iWARP stack
do so for it be default is dangerous and should be avoided.

Even so, I would not forbid it. The rule I proposed is that
accepting an unauthorized untagged message MUST NOT have an
adverse impact on the ability to accept an authorized message
on any other stream. This is equivalent to a common rate-
enforcement policy, that an over-rate cell/packet must never
be forwarded in preference to a rate-compliant cell/packet.
Arguments can be made either way as to when forwarding
over-rate cells/packets that do not block legitimate
traffic becomes a bad idea. Most will agree that there is
a point that the sender is deliberately abusing the system,
and the cell/packet will be dropped even if the network
is otherwise completely idle. But that can vary, what's
important is that a cheater must not be able to drain
resources from a non-cheater. Accepting non-authorized
untagged messages from a buffer pool or shared receive
queue runs *exactly* that risk.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 15 Jul 2003 09:50:50 -0700

I agree with John that the core issue is the same regardless of S-RQ.  

In my opinion, the sharing/non-sharing of buffers is an orthogonal 
issue to graceful handling.  While the scope of sharing (and whether to share)
across remote peers is determined by a ULP based on trust/DoS considerations
wrt the peers, the need for graceful handling is an inherent characteristic of
the ULP/UULP's design.  Even in the absence of any sharing, no ULP 
would like a severe stream termination approach on hitting this exception.

As I said at the beginning of the untagged model discussion -

    > The Data Sink ULP would still be required to authorize each reception

    Generally, I agree.  It may however be the case for some ULPs (such as
    iSCSI) that they may not be able to precisely predict every possible 
    ULP PDU though the peers know about >95%  of the PDUs with certainty
    (and can provision for them).  The ULP in this case can post a few extra
    "slack" buffers, but it still is theoretically possible in some very corner cases 
    that a local buffer pool may transiently run out.

    John Hufferd gave an example of processing delays that could also lead to
    this same scenario.

John is elaborating this with iSCSI-specific examples (a bunch of immediate
Abort Task requests is another example), and these should give an idea of
the issue that some of us are seeing with the current mandate of the DDP spec.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 15 Jul 2003 12:47:39 -0500

>I agree with John that the core issue is the same regardless of S-RQ.  
Agreed. A "non-sharing" endpoint is simply sharing over a pool of one.

So, regardless of how big the pool is, I believe there is full agreement
that a) under-provisioning that pool is a valid local option, b) this
should not be abused, that is you should not expect the pool to go dry
and c) graceful recovery from miscalculations on the pool size are
permissable and even commendable.

The remaining issue is ULP flow control.

Traditionally, RDMA protocols have enforced ULP flow control by
requiring each untagged message (or Send) to be explicitly authorized
at the receiving end.

Buffer pooling / underprovisioning can be achieved without eliminating
this fundamental end-to-end safety check.

Further, because other modes of flow control are *less* effective
once RDMA is enabled it is important that we leave ULP flow control
fully intact.

>In my opinion, the sharing/non-sharing of buffers is an
>orthogonal issue to graceful handling.  While the scope of
>sharing (and whether to share) across remote peers is
>determined by a ULP based on trust/DoS considerations wrt the
>peers, the need for graceful handling is an inherent
>characteristic of the ULP/UULP's design.  Even in the absence
>of any sharing, no ULP would like a severe stream termination
>approach on hitting this exception.
>
Failure to bound the range of valid MSNs creates a penalty
on implementations. Further, it is a type of slippery slope
that should be avoided. If *some* RNICs allow servers to
ignore their responsibility for end-to-end flow control
then applications will be developed that fail to perform
this role -- and it is the compliant NICs that will be
blamed when those applications fail.

We don't allow the RNIC to create a spare buffer for an
invalid STag and keep the data somewhere until the application
figures out where it was really meant to go. We expect the
peer to follow the protocol and send the correct STag.

I fail to see why we should allow the RNIC to accept an
invalid MSN, consume buffers that might be needed for legitimate
messages, and hold it until it is time to deliver it on the
assumption that it was really a valid message. Accepting
"out of window" TCP segments or SCTP data chunks is considered
to be a security hole, not a feature. Why are invalid untagged
messages so deserving of coddling?

> Generally, I agree.  It may however be the case for some ULPs
> (such as iSCSI) that they may not be able to precisely predict
> every possible ULP PDU though the peers know about >95%  of
> the PDUs with certainty (and can provision for them).  The ULP
> in this case can post a few extra "slack" buffers, but it
> still is theoretically possible in some very corner cases that
> a local buffer pool may transiently run out.
>
If the ULP does not know how many untagged messages its peer is
authorized to send then it is not doing flow-control. It is probably
relying on LLP flow control to keep it out of trouble. That does
not work with RDMA because it has been designed to keep the LLP
from being a bottleneck. We don't want to add additional network
messages to allow iWARP acks and credit transmissions, therefore
we need to rely on the ULPs to send credits.

That is not some vague philosophical assertion, it is an important
design point that must be taken into account by anyone who designs
an upper layer protocol to run over RDMAP/DDP.

Negotiate a limit on "asynchronous" messages, and wait for a ULP
ack before sending more. It's very simple, and very secure.

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 15 Jul 2003 17:56:50 -0500

> a local buffer pool may transiently run out.

Some further thoughts on the above.

An implementation MUST be able to take a strict interpretation
on valid MSN windows and buffers being pre-supplied by the ULP.

An implementation also MUST be able to layer cleanly over the
LLP, where the DDP layer does not even receive the data until
the LLP has decided that the segment/chunk is ackable.

-- This is even a *reasonable* implementation over SCTP or
   when a general purpose TCP optimizing capability is in place.
   
Therefore, a ULP MUST be defined such that each end can know the
maximum number of untagged messages that could be in-flight.

If this is not the case, then the Upper Layer Protocol requires
more adaptation to use RDMAP/DDP. Any ULP that implicitly relied
upon transport layer buffers to act as a flow control will require
some adjustment -- but it is required. We cannot accept as a design
goal an optimization for ULP behavior that would *break* a compliant
implementation.

Layered implemetnations are legal. Refusing to provide system buffering
is legal. Therefore all ULPs MUST be defined to comply with these
requirements -- which means they MUST include a method by which
the Data Sink knows the maximum number of valid untagged messages
that could be in flight.