Subject: [rddp] Untagged Message Validation Recap

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 11:08:22 -0500

Initial drafts required that the local ULP pre-post a buffer on
a stream specific basis for each authorized untagged message.
This is an unreasonable requirement that can have a serious
impact on servers that support a large number of clients
concurrently.

    There is full agreement on this point.
    
    I'll also point out for the record that I raised the first
    objection on this point when David issued a last call on
    the architecture document.
    
The current drafts still imply that the abence of a buffer
requires that the stream be terminated. This is excessive.

    There is full agreement on this point.
    
    At the minimum it must be clear that receipt of a valid
    MSN for which there is no buffer is a local fault. Not
    only is the local implementatin *allowed* to recover from
    such a fault, it is in fact commendable to do so.
    
That leaves the question of what to do with Invalid MSNs.
What I propose here are two rules:

1) ULP design MUTS NOT rely upon delivery of invalid MSNs.
   Through *some* mechanism, the Data Sink ULP must be able
   to predict the maximum number of untagged messages that
   may be in flight at any given instant.
   
   Note that this is true even if the DDP layer is ignorant
   of this limitation. The ULP is responsible for properly
   sizing the Completion Queue. Untagged messages generate
   completions. You cannot reliably size the Completion Queue
   if you do not know how many messages may be in flight.
   
   In terms of the verbs draft - I am stating that it is
   incorrect and unacceptable for the Upper Layer Protocol
   to assume that a Shared Receive Queue, or shared Buffers
   are in use. The Upper Layer Protocol MUST be designed so
   that a compliant implementation is possible with simple
   Receive Queues.
  
   That requires a shared understanding between the ULP peers
   as to the maximum number of in-flight untagged messages.
   There is no escaping that. How can the ULP be responsible
   for flow control if it isn't paying any attention?
   
2) Accepting an untagged message with an invalid MSN MUST NOT
   place at risk the reception of any untagged message with
   a valid MSN.
   
   Allowing Client X's violation of the ULP to disrupt Client Y
   is clearly a major security vunerability.
   
    The verbs draft makes this a serious vunerability by actually
    requiring the DDP layer to continue processing MSNs until the
    ULP pulls the plug.

    A properly designed attack could generate a burst of untagged
    message completions which would overwhelm the Completion Queue
    and drain the buffer pool.
    
Is it reasonable for the DDP layer to enforce validity of MSNs?

    I believe that it is easy and natural for the local ULP to convey
    the current range of valid MSNs to the Endpoint. In fact in the
    initial drafts this was the assumption. I find claims that this
    is now somehow overly burdensome to be unconvincing. But specifically:
    
        The same "post receive operation" used with simple Receive Queues
        could be used to grant the credit and then populate a shared
        Buffer Pool. The link between Endpoint and Buffer Pool must
        exist upon message arrival, so it is there when Work Requests
        are processed as well.
        
        Typically credits are communiciated to the peer via an untagged
        message acting as a reply (imlicitly, as with DAFS, or explicitly
        as with NFS over RDMA). An option to increment the untagged
        receive credits attached to an untagged send would allow per
        Endpoint/QP credit tracking wihtout tying it to buffer replenishment
        or requiring any additional crossing of the system bus.
        
    There are reasons why the ULP is in a very poor position to enforce
    MSN validation:
    
        Messages with invalid MSNs must be accepted provisionally.
        Provisionally they consume resources, both buffers and CQEs.
        This may result in CQ overflow and/or buffer exhaustion *before*
        the ULP has a chance to respond.
        
        ULP diagnosis of MSN validity is asynchronous. The ULP cannot tell
        if an MSN that is valid by the time the ULP examines it had in fact
        "jumped the gun" and was sent before it had been authorized. If the
        ULP has decided to limit a client to 4 in-flight messages, it is 
        important that they not be able to send 5. But it is next to impossible
        for the ULP to determine whether then nth request was sent before
        or after the n-5th reply had been sent. Having every client round
        up the number of in-flight messages allowed by one would have a
        serious impact on a multi-client server.
        
In summary, underprovisioning is fine. It is even a good idea when done in
a way that buffers shortfalls occur very rarely. ULPs that want to abdicate
their responsibility to implement flow control are not fine. They cannot
rely upon the local implementations to sweep such an abdication under the rug.

Many non-RDMA ULPs rely upon the LLP to provide flow control for them. It is
a beneficial side-effect of the undersirable bottleneck of traditional LLPs.
RDMA eliminates that bottleneck. That means that ULPs will have to actually
think about how they are flow controlled. Review the applicability draft
and raise questions about the analysis there if you disagree.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp