Subject: [rddp] Re: Untagged Message Validation Recap

From: John Hufferd <[email-masked]>
Date: Wed, 16 Jul 2003 13:41:03 -0700

Caitlin,
I do not think your characterization of the proposal is correct or fair.
No one has said what you are trying to read into our messages!

The point is that the various protocols we are talking about, actually have
what most folks think are an adequate number of pre-authorized buffers.
But just as we talked about system impacts that were out of the control of
the application when shared Buffers/RQs are used, this same situation
occurs for Single RQs.  That is, an application might have a contingency of
say 4 extra buffers, which work in all the testing and a number of various
systems.  As long as the local system is acting "normal", then even if the
remote system has a trauma, there will be enough buffers queued so that the
ULP can receive the message and then queue the next request.  But sometimes
that will not always be the case.  This is not due to any fault of the ULP.
It is often caused by freakish events happening in the local system as the
remote system is having a trauma which it dearly needs to report (hence why
the ULP had sum extra buffers defined, just in case).  Also sometimes, the
thing that is causing the problem may be also happening when, or even
causing a lot of out-of-order arrival of a number of these messages to the
ULP.  The out-of-order arrival causes many of the WQE buffers to be tied up
but not completed.

As pointed out by Mallikarjun and Pat, these corner cases exist, and
through no fault of the ULP.  System vendors would like to shield their
customers from these types of disappointments.  As I have stated before a
number of different RNIC vendors are planning on doing a number of things
which might give the system some latitude.  However, that is all it is,
some latitude.  These various implementations still require the ULP to
function as we have been stating but they can do so based on "ULP time" (as
opposed to "System time"), with the RI and the RNIC making up the
difference.

There has been no statements made by myself, Mallikarjun, or Pat that said
that they want the ULP not to have to post the approprate number of buffers
and replenish them as approprate.  We have just said that we need the words
in the Draft to permit agreements between the System vendors and the RNIC
vendors for what we call graceful handling of these Rare situations.

We all understand that this flexibility in the system and RNIC, is in fact
a very rare and seldom used capability, but as it turns out, the
flexibility permits the system vendors to cover over, what in a narrow view
may seem like, their various erratic behaviors, and permit the ULP to
actually be more portable across different systems.

In the various implementations that the vendors have proposed, all sooner
or later ended up killing the QP, if the situation was not adequately
handled by the ULP.  And in no implementation that I saw, was any other QP
in jeopardy.

These are system issues, NOT ULP issues!  And the system needs ways to
handle it.  Hence the request for loser language.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

                      Caitlin Bestler                                                                                                 
                      <[email-masked]>         To:       [email-masked]                                                                
                                               cc:       John Hufferd/San Jose/IBM@IBMUS, [email-masked], "Mallikarjun C."    
                      07/16/2003 09:08          <[email-masked]>                                                                     
                      AM                       Subject:  Untagged Message Validation Recap                                            

Initial drafts required that the local ULP pre-post a buffer on
a stream specific basis for each authorized untagged message.
This is an unreasonable requirement that can have a serious
impact on servers that support a large number of clients
concurrently.

    There is full agreement on this point.

    I'll also point out for the record that I raised the first
    objection on this point when David issued a last call on
    the architecture document.

The current drafts still imply that the abence of a buffer
requires that the stream be terminated. This is excessive.

    There is full agreement on this point.

    At the minimum it must be clear that receipt of a valid
    MSN for which there is no buffer is a local fault. Not
    only is the local implementatin *allowed* to recover from
    such a fault, it is in fact commendable to do so.

That leaves the question of what to do with Invalid MSNs.
What I propose here are two rules:

1) ULP design MUTS NOT rely upon delivery of invalid MSNs.
   Through *some* mechanism, the Data Sink ULP must be able
   to predict the maximum number of untagged messages that
   may be in flight at any given instant.

   Note that this is true even if the DDP layer is ignorant
   of this limitation. The ULP is responsible for properly
   sizing the Completion Queue. Untagged messages generate
   completions. You cannot reliably size the Completion Queue
   if you do not know how many messages may be in flight.

   In terms of the verbs draft - I am stating that it is
   incorrect and unacceptable for the Upper Layer Protocol
   to assume that a Shared Receive Queue, or shared Buffers
   are in use. The Upper Layer Protocol MUST be designed so
   that a compliant implementation is possible with simple
   Receive Queues.

   That requires a shared understanding between the ULP peers
   as to the maximum number of in-flight untagged messages.
   There is no escaping that. How can the ULP be responsible
   for flow control if it isn't paying any attention?

2) Accepting an untagged message with an invalid MSN MUST NOT
   place at risk the reception of any untagged message with
   a valid MSN.

   Allowing Client X's violation of the ULP to disrupt Client Y
   is clearly a major security vunerability.

    The verbs draft makes this a serious vunerability by actually
    requiring the DDP layer to continue processing MSNs until the
    ULP pulls the plug.

    A properly designed attack could generate a burst of untagged
    message completions which would overwhelm the Completion Queue
    and drain the buffer pool.

Is it reasonable for the DDP layer to enforce validity of MSNs?

    I believe that it is easy and natural for the local ULP to convey
    the current range of valid MSNs to the Endpoint. In fact in the
    initial drafts this was the assumption. I find claims that this
    is now somehow overly burdensome to be unconvincing. But specifically:

        The same "post receive operation" used with simple Receive Queues
        could be used to grant the credit and then populate a shared
        Buffer Pool. The link between Endpoint and Buffer Pool must
        exist upon message arrival, so it is there when Work Requests
        are processed as well.

        Typically credits are communiciated to the peer via an untagged
        message acting as a reply (imlicitly, as with DAFS, or explicitly
        as with NFS over RDMA). An option to increment the untagged
        receive credits attached to an untagged send would allow per
        Endpoint/QP credit tracking wihtout tying it to buffer
replenishment
        or requiring any additional crossing of the system bus.

    There are reasons why the ULP is in a very poor position to enforce
    MSN validation:

        Messages with invalid MSNs must be accepted provisionally.
        Provisionally they consume resources, both buffers and CQEs.
        This may result in CQ overflow and/or buffer exhaustion *before*
        the ULP has a chance to respond.

        ULP diagnosis of MSN validity is asynchronous. The ULP cannot tell
        if an MSN that is valid by the time the ULP examines it had in fact
        "jumped the gun" and was sent before it had been authorized. If the
        ULP has decided to limit a client to 4 in-flight messages, it is
        important that they not be able to send 5. But it is next to
impossible
        for the ULP to determine whether then nth request was sent before
        or after the n-5th reply had been sent. Having every client round
        up the number of in-flight messages allowed by one would have a
        serious impact on a multi-client server.

In summary, underprovisioning is fine. It is even a good idea when done in
a way that buffers shortfalls occur very rarely. ULPs that want to abdicate
their responsibility to implement flow control are not fine. They cannot
rely upon the local implementations to sweep such an abdication under the
rug.

Many non-RDMA ULPs rely upon the LLP to provide flow control for them. It
is
a beneficial side-effect of the undersirable bottleneck of traditional
LLPs.
RDMA eliminates that bottleneck. That means that ULPs will have to actually
think about how they are flow controlled. Review the applicability draft
and raise questions about the analysis there if you disagree.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 16:32:10 -0500

>No one has said what you are trying to read into our messages!

Actually, I believe you are the one mistating your position. And the
problem is that the mistatement creates a major problem.

You want to be able to underprovision buffers. Fine. There is no
disagreement with underprovisioning buffers, as long as it is done
responsibly.

But you are proposing doing so by *eliminating* the concept of
an invalid or out-of-range MSN. That I object to.

The minimal rules I proposed allow for under-provisioning of buffers.
The only requirement is that the receiving side SHOULD stock an
adequate supply, which I believe is totally consistent with the
intent. I do not believe anyone is planning on severely under-provisioning
the buffers. (Of course I would say that 95% confidence is *way* too
low, but I don't think the IETF is the proper forum to debate how
many nines are really required).

The hard requirement that you objected to was that the data sink
SHOULD reject an untagged message with an invalid MSN, and that it
MUST reject an untagged message with an invalid MSN if accepting
it would place reception of any untagged message with a valid MSN
at risk.

The criteria is whether or not the local ULP has authorized the
reception -- *not* whether the local ULP has supplied a buffer.

I do not expect a ULP like iSCSI over iWARP to pre-post a buffer
for *every* possible untagged message. I do expect that the ULP
will include some mechanism by which the Data Sink knows the
maximum number of in-flight untagged messages, so that it can
control whether or not those untagged messages will be recognized
as valid or not.

Even if you accept Pat's argument that it is unreasonable to require
the DDP layer to accept this data, it would not mean that the data
is not required.

If you implemented using simple receive queues, that data would be
needed. The ULP definition cannot *mandate* the use of under-provisioning.

Foremost on the list of reasons is that an implementatin that was
cleanly layered over the LLP has  no choice but to terminate the
stream when there are no available buffers. And we simply cannot
require the Data Sink to provide buffer space for *invalid* messages.

You are proposing a change to a protocol. Even if the intenet of 
your wording is not to do anything that would be abusive, if the
language *allows* for implementations that will cause problems
then there is a problem with the proposed language.

And allowing an invalid untagged message to consume a buffer in
*preference* to a valid untagged message is indeed a problem.
It is a recipe for abusing network and server resources, as
well as for denial of service attacks.