Subject: Re: [Ipoverib] small change in the connected mode draft

From: Harald Alvestrand <[email-masked]>
Date: Wed, 05 Apr 2006 17:41:44 +0200

Michael Krause wrote:
> At 02:51 PM 3/23/2006, Dror Goldenberg wrote:
>>
>> The case that you mentioned, RNR driven retransmission, is another 
>> case of retransmission. It has nothing to do with congestion or 
>> packet drop in the fabric. It is just being impacted by the ability 
>> of the receiver to post receive buffers on the RQ/SRQ. Here the 
>> timeouts are application based and as you wrote can be configured to 
>> infinity. I agree that infinit number of retries would be a bad 
>> choice of RNR retry count. So, maybe we should recommend on selecting 
>> also the RNR timeout and retry count to be low too. I am also 
>> wondering what happens when there is a slow receiver, e.g. posting 
>> too slow on the RQ/SRQ, in which case the RNR Nak will happen very 
>> frequently and might cause the QP to get into error state because of 
>> RNR timeout exahusted.
>
> The question raised was what impact will retransmission have on a ULP 
> such as TCP which would treat timeouts as congestion events.  I raised 
> two cases which are not easy to control at the IB protocol level since 
> they are admin managed:
>         - RNR which can lead to very long delays in the connection 
> transmission completion
>         - Arbitration which can lead to very slow forward progress on 
> a given QoS channel
>
> Both of these should be comprehended minimally as informative text to 
> guide developers and management solutions to do the right thing.  They 
> also need to be aware that in the case of RNR, there is no single 
> right answer and its usage may entail long delays while the OS does 
> whatever it needs that triggered the RNR operation.
My naive understanding of how IB stacks operate would be to give STRONG 
advice to TCP/IP/IB-CM implementors that the IB interface should be fed 
enough buffers to fill all the open TCP connection windows. If that is 
done, the only time RNR could occur is when the TCP window is closed - 
and in that situation, the application is in trouble anyway.

Of course, this approach has its own set of cross-layer issues (and 
doesn't work for UDP-based applications). But they are all about buffer 
management within a single host, which makes layering violations 
slightly less egregrious.

                            Harald

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Thu, 06 Apr 2006 10:54:28 -0700

At 08:41 AM 4/5/2006, Harald Alvestrand wrote:
>>>The case that you mentioned, RNR driven retransmission, is another case 
>>>of retransmission. It has nothing to do with congestion or packet drop 
>>>in the fabric. It is just being impacted by the ability of the receiver 
>>>to post receive buffers on the RQ/SRQ. Here the timeouts are application 
>>>based and as you wrote can be configured to infinity. I agree that 
>>>infinit number of retries would be a bad choice of RNR retry count. So, 
>>>maybe we should recommend on selecting also the RNR timeout and retry 
>>>count to be low too. I am also wondering what happens when there is a 
>>>slow receiver, e.g. posting too slow on the RQ/SRQ, in which case the 
>>>RNR Nak will happen very frequently and might cause the QP to get into 
>>>error state because of RNR timeout exahusted.
>>
>>         - Arbitration which can lead to very slow forward progress on a 
>> given QoS channel
>>
>>also need to be aware that in the case of RNR, there is no single right 
>>answer and its usage may entail long delays while the OS does whatever it 
>>needs that triggered the RNR operation.
>done, the only time RNR could occur is when the TCP window is closed - and 
>in that situation, the application is in trouble anyway.

I am not aware of any OS or network stack that tracks the total available 
window space of all TCP connections.  Most implementations use statistical 
provisioning and hope for the best that few packets will be dropped as a 
result of a lack of buffer availability.  In practice depending upon the 
service rate and inbound packet delivery, this has not always been possible 
to do and customers can see, especially with high-speed links, packet loss 
at the device level.

One difference here for IB / RC is that there is a connection per endnode 
pair so the problem can be mitigated.  This does not mean that RNR will not 
occur or its occurrence will be rare.

>management within a single host, which makes layering violations slightly 
>less egregrious.

IB / RC is for all IP so it does not differentiate between UDP, TCP, 
whatever is at IP and above.  This makes it a bit more difficult to create 
even a reasonable view of the amount of receive buffers that need to be 
posted per RC connection.  Further, given the IP / above layers can now 
post very large buffers, all endpoints will see increased HOL blocking and 
thus QoS issues as a result.  This is where local management will need to 
be involved to determine what exactly should flow over a RC connection or 
perhaps one of several RC connections or just over the existing IP over IB 
UD QP in order to meet QoS requirements.

Mike

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 10 Apr 2006 12:09:50 -0700

[Note my new email addr is [email-masked]. The old
[email-masked] is no longer valid.]

On 4/6/06, Michael Krause <[email-masked]> wrote:
> The case that you mentioned, RNR driven retransmission, is another case of
> retransmission. It has nothing to do with congestion or packet drop in the
> fabric. It is just being impacted by the ability of the receiver to post
> receive buffers on the RQ/SRQ. Here the timeouts are application based and
> as you wrote can be configured to infinity. I agree that infinit number of
> retries would be a bad choice of RNR retry count. So, maybe we should
> recommend on selecting also the RNR timeout and retry count to be low too. I
> am also wondering what happens when there is a slow receiver, e.g. posting
> frequently and might cause the QP to get into error state because of RNR
> timeout exahusted.
>
>
> The question raised was what impact will retransmission have on a ULP such
> as TCP which would treat timeouts as congestion events.  I raised two cases
> which are not easy to control at the IB protocol level since they are admin
> managed:
> guide developers and management solutions to do the right thing.  They also
> need to be aware that in the case of RNR, there is no single right answer
> and its usage may entail long delays while the OS does whatever it needs
> that triggered the RNR operation.
>
> enough buffers to fill all the open TCP connection windows. If that is done,
> the only time RNR could occur is when the TCP window is closed - and in that
> situation, the application is in trouble anyway.
>
>

In my mind the receive buffer allocation in the regular IPoIB/RC where all
ULP/TCP/UDP sharing the same RC is much like IPoIB/UD in that the receive
side
link layer software will allocate and replenish the receive queue with
anonymous memory as fast as possible. In that case the occurence of RNR
should be like
receive buffer underflow for UD; it should be very rare. When that happens
the
system is running out of any useable memory that all TCP connections should
be advertising zero-window and ULP struggles to find memory too.

If one wants to tie the receive buffer mgmt in RC to a particular set of
ULP/TCP
connections, possibly for better performance (e.g., avoiding copy), it's a
different
problem all together. First one must solve the problem of how to multiplex
traffic
into multiple RCs. Then presumably the transport should go into zero-window
before running out of receive buffers to post.

> posted per RC connection.  Further, given the IP / above layers can now post
> very large buffers, all endpoints will see increased HOL blocking and thus
> QoS issues as a result.  This is where local management will need to be
> involved to determine what exactly should flow over a RC connection or

IPoIB/CM still requires a "MTU" size to be negotiated. Presumably there is a
tradeoff on the MTU size. A large size can help bulk data but may create HOL
blocking and waste a lot of memory for small msgs.

Jerry

Mike