Subject: [Ipoverib] Connected mode draft and retransmit implosion

From: "bill" <[email-masked]>
Date: Fri, 13 Aug 2004 16:56:31 -0700

Vivek,

So I have been concerned about the Connected mode draft and I run into a
simple problem here...

Imagine I have a TCP socket running over a 64K RC MTU link.

I transmit the IP packet (all 64K of it)

Infiniband breaks the packet up into a bunch of 2K Infiniband MTU
packets (all 32 of them) - lets say I loose the second packet.

No problem IB will retransmit the packet over the link and it gets to
the other side...

The problem that I want to see - is what happens if the TCP retransmit
timer goes off - causing the whole TCP segment to be retransmitted.

Is there anyway that IPoIB-RC can handle this.  I am assuming the first
pass is "There are no IB link errors".

The second pass will be "IB Link timers are orders of magnitude faster
than TCP timers" (order millisecond rather than order second)

The third (And I hope correct) answer will be "We have done the analysis
and we can show that there is such a low probability of 1 happening,
that 2 will cover all but some rediculously small ammount that the
retransmission problem doesn't happen"

Any input ?

Am I worried over nothing (because of 1 and 2)

Bill

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Fab Tillier" <[email-masked]>
Date: Fri, 13 Aug 2004 17:41:39 -0700

> From: bill [mailto:[email-masked]]
> Sent: Friday, August 13, 2004 4:57 PM
> 
> Vivek,
> 

Wouldn't you end up with the same packet getting sent twice?  If so, why is
this problematic (aside from the inefficiency)?

- Fab

---

From: "bill" <[email-masked]>
Date: Fri, 13 Aug 2004 17:59:17 -0700

Yes, that is exactly the problem... Both layers retransmitting at the
same time

And possibly not recovering for quite a while
Bill

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Fab Tillier
> Sent: Friday, August 13, 2004 5:42 PM
> To: 'bill'; [email-masked]
> Subject: RE: [Ipoverib] Connected mode draft and retransmit implosion
> 
> 
> > So I have been concerned about the Connected mode draft and 
> I run into 
> > a simple problem here...
> > 
> Wouldn't you end up with the same packet getting sent twice?  
> If so, why is this problematic (aside from the inefficiency)?
> 
> - Fab
> 
> 
> 
> [email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
>

---

From: Ajoy Siddabathuni <[email-masked]>
Date: Fri, 13 Aug 2004 18:04:42 -0700 (PDT)

Isnt tcp timers in the order of msec and ib timers in the order of usecs?
I think its just a problem of tuning the stack for the application in
question.

~Ajoy

---

From: "Fab Tillier" <[email-masked]>
Date: Fri, 13 Aug 2004 18:25:41 -0700

> Sent: Friday, August 13, 2004 5:59 PM
> 

While both layers issue retransmits at the same time, the packets on the
wire maintain their order.  The first packet sent will be delivered first,
with the second queued up behind.  The data will not get interleaved.  There
is no way for the retransmitted packet to get delivered first.

> And possibly not recovering for quite a while

IB recovery is pretty much binary - if an RC QP exceeds its retry limit, it
will transition to the error state.  At that point, all processing of the
send queue stops and the consumer gets an error completion and an async
error notification for that QP.

I still don't see the harm in TCP retransmitting while IB is also
retransmitting aside from inefficiencies.

- Fab

---

From: Michael Krause <[email-masked]>
Date: Mon, 23 Aug 2004 07:43:32 -0700

>retransmitting aside from inefficiencies.

Given this is a RC QP, the WR are processed in order and must complete in 
order to make forward progress.  As such, the consumer which is posting WR 
can post new or duplicate packets and the underlying QP will not see these 
until it completes the current WR.  As such, IB retransmission and consumer 
retransmission algorithms are orthogonal to one another in all respects 
sans perhaps doing a poor job configuring the associated IB retransmission 
timers as well as the arbitration policies within the fabric such that the 
consumer falls into its retransmission algorithm too often.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 25 Aug 2004 00:30:21 -0700 (PDT)

> consumer falls into its retransmission algorithm too often.

Yes, TCP will not see out of order packets since the underlying layer delivers
in order, reliable data. Also, TCP timers are likely to be much longer than 
the RC timers. Therefore, it is quite likely that RC will retransmit multiple 
times and recover by the time TCP timers expire.

As an aside, IPoIB over connected mode is not limited to RC, it includes UC 
too. Also, it is IP (which implies UDP, SCTP etc.) over RC or UC. 

Vivek

---

From: Michael Krause <[email-masked]>
Date: Wed, 25 Aug 2004 11:33:03 -0700

>too. Also, it is IP (which implies UDP, SCTP etc.) over RC or UC.

Hence why I did not state a particular consumer type only that it is 
posting to a connected QP.  Whether RC or UC, the processing order remains 
the same.  The only difference is the lack of IB ACK as well as Send Credit 
management for UC but for this discussion, they are essentially the same 
from the consumer's perspective.

Mike