Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required

From: John Hufferd <[email-masked]>
Date: Wed, 16 Jul 2003 06:52:58 -0700

Again, I think that Mallikarjun stated things correctly.  There are 
important applications which can only correctly predict the number of 
buffers needed in non error/special/emergency conditions.  When 
error/special/emergency conditions occur, the number and urgency of the 
handling is problematical.

So rather than going back through all the arguments yet again, please 
accept that these are real and important applications that want to exploit 
RDMA.

Both the shared buffer and non shared buffer case can use some sort of 
graceful handling technique.  When we talk through this set of issues, in 
the past, with several different RNIC vendors, the various RNIC vendors 
had their own specific solutions (each one different from the others). The 
common aspect was that their solutions did not reflect through to the ULP 
and were good enough to permit system glitches, or special dispatching of 
higher level tasks etc. to occur, yet finally permitting the ULP to get 
around to posting the needed buffers.

Now there is another point that I have been having a problem with in the 
various reflector messages, and that is: I do not think it is useful to 
call the types of message that can use special handling, to be 
unauthorized messages.  They are not.  They are only unpredictable.  If 
the ULP does not, in due time, post the approprate buffers, the connection 
will of course be brought down.  That should not be an issue here. 

I also do not think it is approprate to compare the advertisement of 
buffers (STags) for use with RDMA Reads and Writes with the authorization 
of untagged messages.  the RDMA Reads/Writes are generally used for 
sending agreed upon data buffers, however, RDMA Send (untagged) messages 
is often used for control, and special handling messages.  And it is the 
handling of unexpected, but permitted "special handling" messages which we 
have been attempting to  address.

So the only thing the system vendors are asking for is NOT to include 
language in the Draft that would prevent them from being able to select a 
RNIC vendor which can meet their systems' needs.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 09:10:10 -0500

On Wednesday, July 16, 2003, [email-masked] wrote:

>So rather than going back through all the arguments yet again,
>please accept that these are real and important applications
>that want to exploit RDMA.

Then these applications do not work over InfiniBand? You have
a credit limit over IB RC, and it is even less flexible than
what you can do with iWARP.

If these applications MUST send an unregulated number of messages
on demand then they should be using an unreliable transport.

More likely, the applications as they currently exist are used to
being throttled transparently by LLP flow control. (or built into
the RDMA layer itself, as with InfiniBand).

That does not work for iWARP. This was one of the early design
decisions. Because middleboxes can generate LLP Acks that do not
actually imply DDP delivery the only way to have InfiniBand style
credits would be to generate *additional* network traffic at
the RDMAP or DDP layer. This was properly and resoundingly
rejected.

But there was never a concept of *eliminating* credits, merely
making their enforcement local. Rather than receiving credits
from your peer, you receive credits when the local ULP posts
a receive operation (which also acts as a throttle under IB).

Bottom line: there are valid implementations of RDMA which
*cannot* forgive invalid MSNs. The ULPs MUST be designed to
work under these circumtances. If that condition is met, then
individual implemenations may choose to be more fault tolerant.
But it is unacceptable to consider as a factor ULP behavior
that will break a *compliant* RNIC.

Which means that invalid MSNs MUST be rejected. How to handle
valid MSNs without a buffer is a local issue.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 09:33:35 -0500

On Wednesday, July 16, 2003, [email-masked] wrote:

>So the only thing the system vendors are asking for is NOT to
>include language in the Draft that would prevent them from
>being able to select a RNIC vendor which can meet their
>systems' needs.

But the draft MUST include language that allows the design
of upper layer protocols in an implementation independent fashion.

I do not see how that is possible if one peer is allowed to send
an "unpredictable" number of untagged messages.

The receiving side MUST be able to bound this problem. The availability
of buffers is not the critical issue, but rather sizing of completion
queues and the resources required to track in-process untagged messages.

You cannot have ULP flow control if one of the sides is ignorant on
how many untagged messages can be in-flight.

LLP flow control does not work. It would be seriously incorrect to use
LLP buffer flow control to hold off *all* DDP Segments because there
is no buffer space for an *untagged* message. That gets even worse
under SCTP where you would have to hold off *all* messages on *all*
streams because *one* stream did not not have an available untagged
buffer.

It's even more complex: you could accept *some* untagged segments
because they are completing in-process messages, but not others.
For Endpoints with a simple receive queue, buffers are posted for
specific MSNs. Clearly LLP flow control cannot allow only those
packets with Tagged segments or Untagged segments for one of the
following MSNs.

Can you propose what it is that you expect of an RNIC that would
enable a ULP to send an unpredictable number of untagged messages
in a reliable, implementation independent, fashion?

If there is a draft of a ULP that makes such assumptions, it needs
to be corrected.