Subject: RE: [rddp] Review comment on DDP untagged buffer model

From: [email-masked]
Date: Mon, 14 Jul 2003 03:20:18 -0400

On this topic, starting with Mallikarjun's original message:

> The DDP spec implies that the DDP Stream is to be
> dropped by implementations when there is no receive
> buffer to place an inbound untagged message.  This
> is mandating a specific type of implementation, and IMHO,
> is needless.  The DDP spec should instead allow for
> implementation flexibility in this area.  Those RNICs
> that can gracefully deal with lack of receive buffers
> via LLP means (such as TCP retransmit) should be
> allowed as well.  I can agree with the authors' original intent
> that ULP-level flow control and/or adequate buffer provisioning
> must be done to avoid needless retransmits, but mandating
> a connection drop on the first instance of hitting a (most
> likely a very transient) no-buffer window is severe.

plus one of Caitlin's comments that is very much on target:

> It make senese for the spec to allow the receiving stack some
> flexibility in handling a *fault* situation. Not having a buffer
> that should have been present for an authorized untagged message
> can indeed qualify as a fault.

I suggest that treating this sort of situation as a "fault" is a
good way to think about it - among the reasons for this is that
if TCP retransmit is the recovery mechanism, frequent and/or
routine usage of this sort of recovery has the potential to
run afoul of the "SHOULD be capable of queuing out-of-order
TCP segments" requirement in Section 4.2.2.20 of RFC 1122.

Mallikarjun's proposed text revisions don't seem to capture
the sense that this situation is a "fault" that SHOULD NOT be
permitted to occur repeatedly or frequently.  I agree that
the current requirement "is severe", but OTOH implementations
cannot rely on LLP retransmit or the use of "hidden" buffers
of some form to recover from this entire class of situations.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Mon, 14 Jul 2003 06:07:30 -0400

John,

> In your response that said: "I agree that the current requirement
> "is severe", but OTOH implementations cannot rely on LLP retransmit
> or the use of "hidden" buffers of some form to recover from this
> entire class of situations.".  This response caused me to pause and
> re-read it a number of times.  At the conclusion of the re-reading
> I am not sure that I agree or disagree.  This is because the missing
> buffer situation can be caused by things out of the control of the
> ULP.   And therefore not be a ULP error.

[ ... long explanation of statistical provisioning and OS interference
	in buffer pool replenishing snipped ... ]

I think I'm in basic agreement.  I believe that it is acceptable
to use LLP recovery techniques for ULP-level problems such as not enough
buffers due to a transient OS condition (a ULP-level problem, but not a
ULP-level error) as long as such recovery occurs very infrequently.
There are a number of reasons for this, starting with the performance
and delivery consequences of an unnecessary retransmit.  While I don't
have any specific wording to propose, I think some well-placed
SHOULDs/SHOULD NOTs plus explanation along the line that this sort of
situation is a "fault" for which reliance on LLP retransmit or the
like may be made in extremis (and infrequently) should suffice.  I
think the performance consequences will encourage a robustly-designed
implementation to do the "right thing" here, but some direction needs
to be given to implementers to tell them what that "right thing" is
- when over-committing local resources, make sure that the situations
in which otherwise legitimate traffic has to be dropped due to
unavailable resources are very rare (SHOULD NOT happen, including
the part of the "SHOULD NOT" about understanding the consequences).

Thanks,
--David

-----Original Message-----
From: John Hufferd [mailto:[email-masked]] 
Sent: Monday, July 14, 2003 5:56 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [rddp] Review comment on DDP untagged buffer model

David, 
In your response that said: "I agree that the current requirement "is
severe", but OTOH implementations cannot rely on LLP retransmit or the use
of "hidden" buffers of some form to recover from this entire class of
situations.".  This response caused me to pause and re-read it a number of
times.  At the conclusion of the re-reading I am not sure that I agree or
disagree.  This is because the missing buffer situation can be caused by
things out of the control of the ULP.   And therefore not be a ULP error. 

Suppose we have a application that manages many (perhaps thousands of
connections) and therefore needs to manage a buffer pool and queue the
approprate number of buffers (WQEs etc.) at various times.  There are
clearly ways to determine, based on the link speed, the maximum number of
buffers to queue, and at what interval to replenish the queue.  However, it
is also possible that the OS many have some high priority system task which
gets invoked just as the ULP is about to post additional buffers.  In this
case it is possible that authorized responses from the various remote nodes
may use up the posted buffers, before the ULP can get control and post
additional ones. 

This is an important situation since even though the various remote nodes
are given authorization (credits) to send in a message they may wait for
some unknown time before sending the message and thus using the credit.
Since we can not determine when the message will be sent, the ULP will need
to manage a buffer pool.  And the only thing it knows is the all the
combined remote nodes can not exceed the bandwidth of the connection, and
therefore upper buffer limit can be established.   When you take this upper
limit and add some number of extra buffers, to handle the most probable
cases of high priority system task getting in the way, you have a reasonable
approach for managing a buffer pool.  However, some system vendors would
like to protect customers from what ever impact  might be caused by the
system itself or other higher priority tasks getting in the way of the ULP's
buffer replenishment.  (Especially since the connection, which may be struck
by this problem is not predictable since it may be one of thousands that are
sharing the buffer pool.)  Therefore, these systems vendors would like to
pick a RNIC vendor that did not cause ULP connections to be ended because of
some higher priority tasks getting in the way of the buffer replenishment.
I feel therefore, that having a HARD requirement that the connection MUST be
terminated if a buffer is not available, is not the right thing to do.   

So I am not OK with treating a buffer shortage for receiving an "RDMA Send"
messages as an error that must terminate the connection.   

 But I can not determine from your response whether you agree or not.  I am
just not sure what words you are suggesting that we include. 

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked] 

[email-masked] 
Sent by: [email-masked] 
07/14/2003 12:20 AM 
        
        To:        [email-masked], [email-masked] 
        cc:         
        Subject:        RE: [rddp] Review comment on DDP untagged buffer
model 

Thanks,
--David