Subject: RE: [rddp] Review comment on DDP untagged buffer model

From: John Hufferd <[email-masked]>
Date: Mon, 14 Jul 2003 02:56:10 -0700

David,
In your response that said: "I agree that the current requirement "is severe", but OTOH implementations 
cannot rely on LLP retransmit or the use of "hidden" buffers of some form 
to recover from this entire class of situations.".  This response caused me to pause and re-read it a number of times.  At 
the conclusion of the re-reading I am not sure that I agree or disagree. 
This is because the missing buffer situation can be caused by things out 
of the control of the ULP.   And therefore not be a ULP error.

Suppose we have a application that manages many (perhaps thousands of 
connections) and therefore needs to manage a buffer pool and queue the 
approprate number of buffers (WQEs etc.) at various times.  There are 
clearly ways to determine, based on the link speed, the maximum number of 
buffers to queue, and at what interval to replenish the queue.  However, 
it is also possible that the OS many have some high priority system task 
which gets invoked just as the ULP is about to post additional buffers. In 
this case it is possible that authorized responses from the various remote 
nodes  may use up the posted buffers, before the ULP can get control and 
post additional ones.

This is an important situation since even though the various remote nodes 
are given authorization (credits) to send in a message they may wait for 
some unknown time before sending the message and thus using the credit. 
Since we can not determine when the message will be sent, the ULP will 
need to manage a buffer pool.  And the only thing it knows is the all the 
combined remote nodes can not exceed the bandwidth of the connection, and 
therefore upper buffer limit can be established.   When you take this 
upper limit and add some number of extra buffers, to handle the most 
probable cases of high priority system task getting in the way, you have a 
reasonable approach for managing a buffer pool.  However, some system 
vendors would like to protect customers from what ever impact  might be 
caused by the system itself or other higher priority tasks getting in the 
way of the ULP's buffer replenishment.  (Especially since the connection, 
which may be struck by this problem is not predictable since it may be one 
of thousands that are sharing the buffer pool.)  Therefore, these systems 
vendors would like to pick a RNIC vendor that did not cause ULP 
connections to be ended because of some higher priority tasks getting in 
the way of the buffer replenishment.  I feel therefore, that having a HARD 
requirement that the connection MUST be terminated if a buffer is not 
available, is not the right thing to do. 

So I am not OK with treating a buffer shortage for receiving an "RDMA 
Send" messages as an error that must terminate the connection. 

 But I can not determine from your response whether you agree or not.  I 
am just not sure what words you are suggesting that we include.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

[email-masked]
Sent by: [email-masked]
07/14/2003 12:20 AM

        To:     [email-masked], [email-masked]
        cc: 
        Subject:        RE: [rddp] Review comment on DDP untagged buffer model

On this topic, starting with Mallikarjun's original message:

> The DDP spec implies that the DDP Stream is to be
> dropped by implementations when there is no receive
> buffer to place an inbound untagged message.  This
> is mandating a specific type of implementation, and IMHO,
> is needless.  The DDP spec should instead allow for
> implementation flexibility in this area.  Those RNICs
> that can gracefully deal with lack of receive buffers
> via LLP means (such as TCP retransmit) should be
> allowed as well.  I can agree with the authors' original intent
> that ULP-level flow control and/or adequate buffer provisioning
> must be done to avoid needless retransmits, but mandating
> a connection drop on the first instance of hitting a (most
> likely a very transient) no-buffer window is severe.

plus one of Caitlin's comments that is very much on target:

> It make senese for the spec to allow the receiving stack some
> flexibility in handling a *fault* situation. Not having a buffer
> that should have been present for an authorized untagged message
> can indeed qualify as a fault.

I suggest that treating this sort of situation as a "fault" is a
good way to think about it - among the reasons for this is that
if TCP retransmit is the recovery mechanism, frequent and/or
routine usage of this sort of recovery has the potential to
run afoul of the "SHOULD be capable of queuing out-of-order
TCP segments" requirement in Section 4.2.2.20 of RFC 1122.

Mallikarjun's proposed text revisions don't seem to capture
the sense that this situation is a "fault" that SHOULD NOT be
permitted to occur repeatedly or frequently.  I agree that
the current requirement "is severe", but OTOH implementations
cannot rely on LLP retransmit or the use of "hidden" buffers
of some form to recover from this entire class of situations.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 07:45:52 -0500

On Monday, July 14, 2003, [email-masked] wrote:

>Suppose we have a application that manages many (perhaps
>thousands of connections) and therefore needs to manage a
>buffer pool and queue the approprate number of buffers (WQEs
>etc.) at various times.  There are clearly ways to determine,
>based on the link speed, the maximum number of buffers to
>queue, and at what interval to replenish the queue.  However,
>it is also possible that the OS many have some high priority
>system task which gets invoked just as the ULP is about to post
>additional buffers. In this case it is possible that authorized
>responses from the various remote nodes  may use up the posted
>buffers, before the ULP can get control and post additional
>ones.

Assuming that the ULP had a reasonable expectation that the
posted buffers would become available, their failure to do
so promptly would certainly qualify as a "fault" to me.

On the other hand, an application that ran with only a few
spare buffers, and counted on restocking them "just in time"
without making any serious effort to deal with latency
would be abusing the system. 

Having 100% predictable real-time behavior requires specialized
execution environments. We certainly do not want to restrict
RDMA deployment to such environments. But that does not mean
the ULP can be sloppy and/or deliberately push the boundaries
on providing an adequate buffer supply.

In financing, it is acceptable to keep some of your funds in interest
bearing accounts and only moving them into your checking account
just-in-time.  But it is not acceptable to routinely ignore all
first billings and not even working on getting the funds to pay
them when they are threatening to go to court.

We need a similiar statement of expectations here. Prudent under-
provisioning of buffers is simply wise resource management.
When something goes wrong with it, it is acceptable to have a
fault recovery strategy. What is not acceptable is to have any
strategy that routinely *relies* upon retransmits, any more than
it is acceptable to have a business plan that relies upon never
paying an account until it is at least 90 days past due.

Obviously, enforcement of such matters can never be better than
SHOULD or SHOULD NOTs. But at least the intent needs to be clear.