Subject: RE: [Ipoverib] A Couple of IPoIB Questions

From: Dror Goldenberg <[email-masked]>
Date: Mon, 22 Nov 2004 23:36:25 +0200

Hi Mike,
 
Please see below.
 
Thanks
Dror

-----Original Message-----
From: Michael Krause [mailto:[email-masked]] 
Sent: Monday, November 22, 2004 8:37 PM
To: IPoverIB
Subject: RE: [Ipoverib] A Couple of IPoIB Questions

At 08:49 AM 11/20/2004, Dror Goldenberg wrote:

-----Original Message-----

From: Michael Krause [mailto:[email-masked] <mailto:[email-masked]> ] 

Sent: Friday, November 19, 2004 3:45 AM

To: Vivek Kashyap

Cc: IPoverIB

Subject: Re: [Ipoverib] A Couple of IPoIB Questions

At 05:14 PM 11/18/2004, Vivek Kashyap wrote:

RC and UC both have benefits. There is almost no difference other than

the connection flag between the two. 

Many host OS implementations do not support UC as RC and UD are all that is
really required within the industry.  The ACK overhead associated with RC is
truly noise and the end-to-end credits are very nice as IB now supports
three signaling rates combined with 4 link widths (though only three are
really being implemented).  Such a permutation in bandwidth capability makes
RC a more tenable / good citizen as we designed it to be so I'd prefer RC. 

[DG] Mike,

 A few reasons I think that the end to end credits / RNR  in an RC
connection is a problem.

It may be worth discussing it:

1) Lack of receive WQEs in the responder implies a slow responder. Getting
the messaged dropped 

    in this case is desirable from protocols that have injection control
such as TCP.  In this case it

    is supposed to back off and restart slowlier. While UC/UD result  in a
similar behavior of messages

    being dropped at the receiver when it's slow, RC does not. Instead,
there is persistence in getting the

    message transmitted and the receiver won't be able to tell the requester
that it's being slow.

TCP on the sending side will regulate due to lack of update window  credits.
Hence, there is no need to restart the large messages that are put forth as
the reason for using *C instead of UD. 
 
[dg] I think it'll be common to find very large TCP windows being
advertised. Therefore, when you work against a very slow receiver, I think
that it makes sense to activate the TCP congestion mechanism rather than to
rely on the TCP window which is not intended to take care of congestion.
Typically, the overall advertised TCP windows (from all connections
together) is much more than actually being posted on the IPoIB QP receive
queue. In a slow receiver, the replenishment pace on receive WQEs is slow,
and you'd want remote senders to slow down when trying to fill its TCP
Windows.

2) How would you configure the RNR retry counters. Would they be configured
to infinity ? Doesn't sound

    good. Would they be configured to a finite value (should be <7), in
which case, in the case of a slow

    receiver you'd end up recreating connections that had end to end credits
problem, which is a real 

    overhead on the protocol.

RNR would be no different for IP over IB than for any other IB RC instance.

[dg] Example ULPs such as SDP and SRP use SW level flow control and do not
rely on RNR NAKs. What I am trying to say is if you configure your QP for
finite retries and a reasonable timeout, then when the receiver is slow,
you'd often get the QP into the error state, after RNR retries are
exhausted. The overhead of reestablishing a new connection each time the QP
gets into the error state is high.  If you use UC, then this is not a
problem, because none of this happens.

3) What happens with implementations that don't support RNR Nak generation ?
That poses more

    difficulties on (2).

A HCA is required to support RNR NAK.  A TCA has the option.  If you don't
support RC, then use UD.  Where is the real problem as nothing shown here on
either side is more than speculation?
[dg] Agree, there isn't a real problem here.

Mike

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 22 Nov 2004 14:26:36 -0800

At 01:36 PM 11/22/2004, Dror Goldenberg wrote:
>>Many host OS implementations do not support UC as RC and UD are all that 
>>is really required within the industry.  The ACK overhead associated with 
>>RC is truly noise and the end-to-end credits are very nice as IB now 
>>supports three signaling rates combined with 4 link widths (though only 
>>three are really being implemented).  Such a permutation in bandwidth 
>>capability makes RC a more tenable / good citizen as we designed it to be 
>>so I'd prefer RC.
>>[DG] Mike,
>>1) Lack of receive WQEs in the responder implies a slow responder. 
>>Getting the messaged dropped
>>     is supposed to back off and restart slowlier. While UC/UD result  in 
>> a similar behavior of messages
>>     message transmitted and the receiver won't be able to tell the 
>> requester that it's being slow.
>
>TCP on the sending side will regulate due to lack of update 
>window  credits.   Hence, there is no need to restart the large messages 
>that are put forth as the reason for using *C instead of UD.
>
>[dg] I think it'll be common to find very large TCP windows being advertised.

A TCP window that is advertised is required to have the associated 
buffering available.  While some implementations assume statistical 
provisioning in the kernel, they assume that the application buffers are 
available and the only problem is being able to move kernel buffers quickly 
enough to application buffers which is a transient issue.

>  Therefore, when you work against a very slow receiver, I think that it 
> makes sense to activate the TCP congestion mechanism rather than to rely 
> on the TCP window which is not intended to take care of congestion. 
> queue. In a slow receiver, the replenishment pace on receive WQEs is 
> slow, and you'd want remote senders to slow down when trying to fill its 
> TCP Windows.

Dropping a buffer is fine but that should be at the TCP/IP level and not a 
driver decision.  A driver should have sufficient buffers to avoid having 
wasted the network bandwidth.  Hence, the driver should be posting 
sufficient buffers to keep up with the workload which may span multiple 
connections / datagrams.  Use of UC or RC does not change anything in this 
regard.  A drop using UC would simply waste IB network bandwidth, consume 
HCA resources flushing the work (the transmitter would continue to transmit 
so nothing is saved there), etc. and only impact one connection at a 
time.  It does nothing for the rest of the connections.  So while one might 
get a bit of benefit akin to a RED scheme, if the endnode pairs are 
operating at a high workload, all one gets with UC is the ability of one 
endnode to flood another with no push back except on random 
connections.  This would lead to bursty behavior and unpredictable 
application responsiveness.  RC leads to smoother performance between the 
endnode pair and with the use of multiple RC QP, one can differentiate 
traffic for QoS purposes which is something that will benefit applications.

>>2) How would you configure the RNR retry counters. Would they be 
>>configured to infinity ? Doesn't sound
>>     receiver you'd end up recreating connections that had end to end 
>> credits problem, which is a real
>rely on RNR NAKs.

These are also not IP based ULP.

>  What I am trying to say is if you configure your QP for finite retries 
> and a reasonable timeout, then when the receiver is slow, you'd often get 
> the QP into the error state, after RNR retries are exhausted. The 
> overhead of reestablishing a new connection each time the QP gets into 
> the error state is high.  If you use UC, then this is not a problem, 
> because none of this happens.

Given RC uses send credits and therefore should not see a new message 
unless there is an associated buffer available which increments the credit 
count, one should not get a RNR NAK ever.  The reason for RNR NAK was to 
deal with a resource other than a receive buffer missing, e.g. QP context 
or V-to-P translation or whatever not being chip resident and some time 
would be required to refresh without going into the error state.  Given RC 
is still send-receive based, there should not be any reason for a RNR NAK 
and no SEND will occur unless a credit is provided.

>
>>3) What happens with implementations that don't support RNR Nak 
>>generation ? That poses more
>>     difficulties on (2).
>
>support RC, then use UD.  Where is the real problem as nothing shown here 
>on either side is more than speculation?

Mike