Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required

From: [email-masked]
Date: Wed, 16 Jul 2003 04:02:30 -0600

Caitlin,

Your suggestion presumes that buffers are returned as they are emptied by a process that knows which connection they were used by. In most cases and especially in cases where one is designing for high utilizations (and if one wasn't doing that one wouldn't need to use DDP), that isn't likely to be the implementation. It would be common for there to be some bulk process that gets freed buffers and restores them to the shared queue or pool in bunches. It doesn't know or care which connection the buffers were freed by. Also, this process is talking to the shared queue manager part of the DDP implementation. It isn't talking to the individual connections.

A post buffer and grant credit operation would have to operate on both the shared queue manager context and on the connection context which is neither necessary or desireable. With bulk returns of buffers, the ULP's buffer recycler would have to maintain a list of credits for connections and give that list with the buffers to the DDP implementation which would have to update multiple connection contexts. 

It should be up to the application using the shared buffer queue to manage the buffer pool for over usage. If higher independence between connections is desired, then the application can use dedicated receiver queues. Shared buffer queues are only one of several resources shared between connections.

I don't see any justification for your statement about virtually all ULPs having limits on the number of untagged messages. In any case, if the ULP does have such a limit, it can certainly see when its peer is violating the limit and can disconnect. It doesn't need DDP to do that.

Pat

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Monday, July 14, 2003 9:45 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required

On Monday, July 14, 2003, [email-masked] wrote:

>This proposed rule below is not practical as the DDP implementation has 
>no information on how many untagged messages the ULP has authorized. To 
>add that information would require constant communication between the 
>ULP and the DDP implementation to relay information about the credits 
>that had been passed to the peer. Such communication would be a 
>performance problem on the backplane. The ULP and not DDP should be 
>responsible for detecting that the peer has overrun its credits. 
>
There are no more interactions than are required when fully provisioning
the buffers. There is an inherent need to a) provide buffers and b)
notify the ULP of completed reads.

An example local interface would have the following capabilities:

-- post buffer and credit to Endpoint

    Number of allowed untagged messages for Endpoint increments.
    Buffer is placed in receive queue for Endpoint (normally)
        or in an associated Shared Buffer Pool / Shared Receive Queue 
        (if the Endpoint is so configured).
        
-- grant credits to Endpoint

    without actually providing any buffers -- increasing the amount
    of underprovisioning.
    
-- post buffer without credit to Endpoint

    To decrease the number of buffers.
    
Only the first method/verb would be used during the processing of
a typical transaction. A typical session would post some buffers,
and if underprovisioning is to be used, some additional credits.
As each completino is processed, a replacement buffer/credit is
provided.

Now the ULP is free to bundle these steps. If it wants to process
three transactions and then post a single replacement with three
buffers and credits. A more promising optimization might be to allow
the replacement buffer and credit to be posted in the same operation
as the reply message.

I am not proposing a mandated procedure on *how* the ULP meters
receives. Merely that it remain part of the loop.

But given that the credit can easily be piggy-backed on re-enabling
the buffer and/or sending the reply, I do not see how this can be
a performance issue.

>What is necessary to protect other connections in a shared buffer pool 
>situation is to deal with the problem of a peer that has tied up 
>excessive numbers of partially filled buffers. The ULP can only see 
>completions, so an attack is possible by sending segments for many 
>messages without completing any message. For instance, in the verbs 
>document, we put an alarm (with threshold settable by the ULP) for 
>excessive incomplete buffers in a receive queue.

An Alarm that requires a ULP response is inherently suspect as a
defense againt a Denial of Service attack.

A static limit also does nothing to prevent an over-eager peer that
always puts the next request on the wire too early. A static limit
is fixed at N, it does not realize that it should be N-m because
there are already m completed messages being processed.

Virtually every ULP protocol seen to data limits the number of
untagged messages that have not been responded to. Expanding the
range of Valid MSN by posting a nrw receive operation (whether
to the specific EP or to its pool) accomplishes this because it
is done *after* the ULP hass finished processing a message.

This sets the valid MSN window to the correct value, and can do so
as part of inherent per-transaction interactions.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 09:18:09 -0500

Answering different portions in an arbitrary order...

On Wednesday, July 16, 2003, [email-masked] wrote:

>I don't see any justification for your statement about virtually all 
>ULPs having limits on the number of untagged messages.

Can you cite a published protocol that works over RDMA that does *not*
have a concept of message credits? Every protocol over IB has credits.
DAFS has credits. NFS over RDMA has credits.

If the ULP does not have limits on the number of untagged messages, how
is it doing flow control? Keep in mind that the ULP does not see 
tagged messages, and has no way of knowing how much tagged traffic
was actually done associated with any untagged message (other than
the presumption being that it was what was allowed, but technically
the peer is allowed to send redundant tagged writes as long as it
doesn't expect them to be processed in a specific order).

The implications of RDMA on application flow control are covered
in the applicability draft. If you have questions, why not review that?

My position remains that you cannot correctly specify an Upper Layer
Protocol to run over iWARP without explicit control over the number
of in-flight untagged messages.

Therefore the local consumer *knows* how many untagged messages
to allow. It is only a matter of how to share that data.

> In any case, if the ULP does have such a limit, it can
> certainly see when its peer is violating the limit and can
> disconnect. It doesn't need DDP to do that.

That was already covered in prior postings on this topic, to
which nobody has offered a counter-argument. To recap, the
local ULP is not in a position to deal with either of the
extremes:

-- A typical design will place completions on an asynchronous
   and opaque queue, which will frequently be shared across
   multiple streams. The only way for the application to know
   that there are N messages from stream X is to completely
   drain the queue. That takes time. The probability that a
   peer could consistently send its Nth reply *before* being
   authorized to do so is very high. "Off by one" errors are
   very common. "Off by one" errors that *benefit* the client
   are not likely to be caught. Once this edge is discovered,
   all clients will feel competitive pressure to emultate this
   "error", some of those emulations will have "off by one"
   errors and actually be *two* over the limit...
   
-- A Denial of Service attack succeeds if it requires the ULP
   to take action.
   
But these both miss the critical point. If the MSN is invalid
it should be rejected as early as possible. The DDP layer rejects
invalid STags and invalid TOs. It does not wait for the STag to
become valid again (even though it typically will). Why should
it wait for the MSN to become valid and then deliver it?

As I pointed out, before Shared Receive Queues were proposed,
there was no doubt in any discusions or any documents that the
MSN would be validated by the Endpoint/QP on arriving messages.

>Your suggestion presumes that buffers are returned as they are emptied 
>by a process that knows which connection they were used by. In most 
>cases and especially in cases where one is designing for high 
>utilizations (and if one wasn't doing that one wouldn't need to use 
>DDP), that isn't likely to be the implementation.

The software component that processes the contents of a buffer is not
aware of which connection the data was delivered on?

What does it think the data is? Who will it tell that the data was
good / bad?

Is it accepting an indefinite/infinite supply of such messages from
anybody? That sounds like an unreliable protocol. DDP messages are
delivered over a reliable transport, and rely upon ULP flow control.
How can the ULP do flow control if it has forgotten who it is talking
with?

All RDMA-specific APIs that I am aware of preserve this information.
Even when using a central completion queue, each delivered untagged
message will note the Endpoint/QP that it came from.

>It would be common for there to be some bulk process that gets
>freed buffers and restores them to the shared queue or pool in
>bunches. It doesn't know or care which connection the buffers
>were freed by. Also, this process is talking to the shared
>queue manager part of the DDP implementation. It isn't talking
>to the individual connections.
>
If you are running a shared pool the buffers will be a uniform size.
Otherwise you are running a true heap, something that I have trouble
imagining being stable under a continuous multi-GByte allocation demand.
Even simple real-time system avoid dynamic variable allocation, I would
assume that virtually all RNIC designs will be fetching a buffer of
a fixed size (whether from a pool or an ordered receive queue) and
then expecting the message to match. The fanciest I can imagine is
that there might be multiple size dependent pools.

In any event, the most common technique will be to pre-stock the pool
and then enter a loop to:
    - receive a request as a completion of an untagged receive.
    - process the request, sending any tagged messages.
    - post the replacement receive buffer (i.e. the same one)
    - post the untagged message that is the reply.
    
There is no real benefit to coalescing them before placing them in
the common pool. And the natural time to do so would be when completing
the handling of each untagged receive completion.

In any event, I did not propose that the receive credit had to be
supplied with a buffer. Those operations can be seperated. It could
just as easily be attached to sending the reply.

>A post buffer and grant credit operation would have to operate on both 
>the shared queue manager context and on the connection context which is 
>neither necessary or desireable.

A Work Request is posted to the Endpoint that optionally supplies a
buffer. The Endpoint increments its credits, and then posts the buffer
either to its own receive queue or to a shared buffer pool (according
to the Endpoint's configuration).

The link between the endpoint and the shared buffer pool already
exists. It is used to allow the Endpoint to *allocate* from the
pool when an untagged message is detected. The allocation must
be performed at peak rates to match the link on demand, the
restore operation can be done in slack time at a sustained rate.
It is *easier* than allocating on demand.

The mechanisms are obvously in place to deal with serializing
allocation requests? Why does this become undesirable when restocking
the buffer pool?

Keep in mind that the shared buffer pool algorithm I have described
allows the ULP to be designed without knowing whether buffer-pooling
is in use or not, and keeps the per-transactin processing loop
virtually identical whether using a buffer-pool or a receive queue.

>With bulk returns of buffers, the ULP's 
>buffer recycler would have to maintain a list of credits for connections 
>and give that list with the buffers to the DDP implementation which 
>would have to update multiple connection contexts. 
>
No, it just has to remember which Endpoint/QP is having the credit
replenished. The Endpoint/QP already knows what pool it belongs to.

It is never more than a single Endpoint/QP that is being credited,
and never more than a single buffer pool or receive queue that is
being restocked.

>It should be up to the application using the shared buffer queue to 
>manage the buffer pool for over usage. If higher independence between 
>connections is desired, then the application can use dedicated receiver 
>queues. Shared buffer queues are only one of several resources shared 
>between connections.
>
Which is why accepting invalid MSNs is particularly dangerous. If the
completion queue is defined more narrowly than the buffer pool, the
excess completions from invalid MSNs could cause a CQ overflow.
I have not seen a local interface that can gracefully recover from
a CQ overflow. One or more streams will have to be aborted.

It is factors like that which triggered my concerns. Accepting invalid
MSNs opens multiple vunerabilities to DoS attacks, errors and client
abuse of server resources.

Buffer pooling is desirable, but can be achieved *without* changing
the end-to-end credit semantics or creating any additional security
vunerabilities.