Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required

From: John Hufferd <[email-masked]>
Date: Wed, 16 Jul 2003 07:35:55 -0700

Responses in line [Huff/] ... [/Huff]
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Caitlin Bestler <[email-masked]>
07/16/2003 07:10 AM

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     [email-masked]
        Subject:        Re: [rddp] Some Minimal end-to-end Buffer Rules Required

On Wednesday, July 16, 2003, [email-masked] wrote:

>Again, I think that Mallikarjun stated things correctly.  There are 
>important applications which can only correctly predict the number of 
>buffers needed in non error/special/emergency conditions.  When 
>error/special/emergency conditions occur, the number and urgency of the 
>handling is problematical.
>
>So rather than going back through all the arguments yet again,
>please accept that these are real and important applications
>that want to exploit RDMA.

Then these applications do not work over InfiniBand? 

[Huff/] 
These application have not yet been shown to operate on InfiniBand, for a 
number of reasons that are independent of what we have been talking about 
here.  On the other hand, there are currently extensions for InfiniBand 
being proposed which may remove these other restrictions.  So we will have 
to wait and see about that.
[\Huff]

You have
a credit limit over IB RC, and it is even less flexible than
what you can do with iWARP.

If these applications MUST send an unregulated number of messages
on demand then they should be using an unreliable transport.

[Huff/]
Now that is not a helpful response.
[\Huff]

More likely, the applications as they currently exist are used to
being throttled transparently by LLP flow control. (or built into
the RDMA layer itself, as with InfiniBand).

That does not work for iWARP. This was one of the early design
decisions. Because middleboxes can generate LLP Acks that do not
actually imply DDP delivery the only way to have InfiniBand style
credits would be to generate *additional* network traffic at
the RDMAP or DDP layer. This was properly and resoundingly
rejected.

[Huff/]
I think we are drifting off topic here.
[\Huff]

But there was never a concept of *eliminating* credits, merely
making their enforcement local. Rather than receiving credits
from your peer, you receive credits when the local ULP posts
a receive operation (which also acts as a throttle under IB).

[Huff/] Again off topic [\Huff]

Bottom line: there are valid implementations of RDMA which
*cannot* forgive invalid MSNs. The ULPs MUST be designed to
work under these circumtances. If that condition is met, then
individual implemenations may choose to be more fault tolerant.
But it is unacceptable to consider as a factor ULP behavior
that will break a *compliant* RNIC.

[Huff/]
I have not, in any of my messages suggested that we need to have a 
solution that breaks compliant RNICs.  The point was always to define what 
"compliant" meant.  You have been building up a strawman and knocking it 
down.  We should get back to the original points detailed by Mallikarjun 
about ensuring that the language in the specification is such that the 
requirements of the various System Vendors can be met. 
[\Huff]

Which means that invalid MSNs MUST be rejected. How to handle
valid MSNs without a buffer is a local issue.