Subject: Re: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt

From: Margaret Wasserman <[email-masked]>
Date: Sat, 11 Dec 2004 07:47:16 -0500

Michael Krause wrote:
>No.  It uses InfiniBand RC or UC to communicate IP datagrams (v4 / 
>v6) between connected endnodes.

And what is InfiniBand RC, under the covers?

>Rest is based on a misconception as there is only IB below IP and no 
>tunneling of TCP over TCP occurs.

Well, I know that there is another IP(v6) in there, as IB is very 
closely based on IPv6.  Is RC based on TCP? SCTP?  Or did IB define a 
different reliable, connection-oriented IP protocol?  If the latter, 
what type of retransmission and congestion control algorithms are 
used by IB RC?  And, has there been any study regarding how they 
would interact with the retransmission and congestion control 
algorithms of TCP or other reliable upper-layer transports?

Even if this isn't TCP, per se, there may still be issues, but it 
will take more careful research to determine what they are.

Margaret

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Strahm" <[email-masked]>
Date: Sat, 11 Dec 2004 06:53:15 -0800

OK - a couple of misconceptions

IB Layer 3 is NOT IPv6.  It has this thing that looks ALMOST like an
IPv6 header - but it does routing completely differently.  Infact it
isn't until you do IB Routing that you would even put a GRH (Global
Route Header) that is what ends up looking like a IPv6 header.  If you
are staying on the local IB subnet - you are only required to put a LRH
(Local Route Header) that looks nothing like IPv6.

The reliability mechanism is not TCP therefore.  I will agree with
Michael on this one.  That said - the same problems that you discuss can
happen between the layer 4 IB and the Layer 4 TCP/IP.

I do not believe there will be a problem however because the timers are
on such a different scale.  Let me try and give an example... There are
retries in Ethernet (I am talking about collision detection and back
off) - What if we decided to worry about TCP retransmitting because the
Layer 2 packet hasn't been able to get on the wire and the TCP timers
went off and started retransmitting ?  People don't worry because the
timers are SO different - this will be even more pronounced in the IB
world.

I don't think we will be able to move the MTU above 64K - just because I
believe the IP packet size is (was ??? Has the size parameter gone up ?)
64K so don't worry about someone trying to stick a 2G IP packet onto the
wire.  At the same time if it is possible to specify a single IP packet
that is 2G, maybe we need some text staying the MTU should be limited to
something sane

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Margaret Wasserman
Sent: Saturday, December 11, 2004 4:47 AM
To: [email-masked]
Subject: Re: [Ipoverib] comments on
draft-kashyap-ipoib-connected-mode-02.txt

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Sat, 11 Dec 2004 20:54:11 -0500

Hi Bill,

>OK - a couple of misconceptions

I am sorry for my limited understanding of Infiniband...  Hopefully 
we all have enough information between us to reach the right answers 
here...

>timers are SO different - this will be even more pronounced in the IB

I think I understand your point about timers, but I don't really 
understand how this works...  Does IB RC actually retransmit lost 
packets?  Or does it just have some type of collision-detection & 
back-off mechanism, like Ethernet?  If it retransmits, how can it do 
that without waiting for at least one round-trip-time to know that 
data has been lost?  And, if it doesn't retranmit, how can it 
guarantee reliability if data is lost?

Ethernet does not actually retransmit packets... an Ethernet device 
will detect a collision during transmission of the packet preamble 
and wait a very short period before trying again.  This time is much 
shorter than the Ethernet round-trip-time, so it is guaranteed (if I 
understand this correctly) to be shorter than the smallest possible 
TCP retransmission timer (which will continuously decrease if things 
are going well until it gets quite close to the round trip time for 
the TCP connection).

The fact that Ethernet does not actually retransmit is also 
important, as you don't need to worry about retransmissions at the 
Ethernet layer exacerbating a congestion condition.

>that is 2G, maybe we need some text staying the MTU should be limited to

IPv6 includes a Jumbo Payload hop-by-hop option which allows the 
transmission of IPv6 packets with payloads of >64K bytes.  RFC 2147 
(TCP and UDP over IPv6 Jumbograms) defines how to use TCP and UDP 
with IPv6 Jumbograms.  I don't now if either of these mechanisms is 
widely implemented, though.

Margaret

---

From: Michael Krause <[email-masked]>
Date: Mon, 13 Dec 2004 10:07:02 -0800

>timers are SO different - this will be even more pronounced in the IB

True.  One can configure the IB timers to be rather small or even 
infinite.  In practice, the timers will be no more than a couple of seconds 
worst case but had been envisioned as measured in milli-seconds in practice 
as the bandwidths combined with the switch latencies are such that large 
timers do not make a lot of sense.  When we were developing the IB 
transports, a lot of discussion went into how things change when moving 
from a Gbps world to N GBps world.  Things like congestion management 
become more complicated as attempting to detect and adjust injection rates 
for something that may be momentary burst can cause such oscillations in 
the performance that one needs to consider two approaches.  Detect and 
monitor for N events in a given period of time.  If sustained, then examine 
injection rates or adjust path selection to reduce the number of 
events.   This is done on more of a global basis as one may not want to 
limit this on a per endnode basis depending upon what type and priority a 
given endnode operates.  Or, detect and monitor and adjust on a per endnode 
basis assuming that all endnodes are of equal priority.  One can also 
adjust the VL arbitrations, etc. as well to effect change without impacting 
injection rate simply because of the service rates for the VL and the 
implicit flow control that occurs which will cause appropriate 
back-pressure.  The IBTA completed the congestion spec a couple of months 
ago and it is worth a read if people have interest.

>that is 2G, maybe we need some text staying the MTU should be limited to

TCP Jumbo I thought could go to 256K.

Mike

---

From: Michael Krause <[email-masked]>
Date: Thu, 16 Dec 2004 14:00:46 -0800

At 05:54 PM 12/11/2004, Margaret Wasserman wrote:

>I am sorry for my limited understanding of Infiniband...  Hopefully we all 
>have enough information between us to reach the right answers here...
>
>understand how this works...  Does IB RC actually retransmit lost packets?

Yes.  It uses a strongly ordered protocol to recover all lost packets 
between endnodes.

>Or does it just have some type of collision-detection & back-off 
>mechanism, like Ethernet?

Congestion management is optional but can be applied.

>  If it retransmits, how can it do that without waiting for at least one 
> round-trip-time to know that data has been lost?

Detection is either explicit through the receipt of a NAK or through a 
timer expiring.

>And, if it doesn't retranmit, how can it guarantee reliability if data is 
>lost?

It does retransmit or the connection is terminated with an error.

>Ethernet does not actually retransmit packets... an Ethernet device will 
>detect a collision during transmission of the packet preamble and wait a 
>very short period before trying again.  This time is much shorter than the 
>Ethernet round-trip-time, so it is guaranteed (if I understand this 
>correctly) to be shorter than the smallest possible TCP retransmission 
>timer (which will continuously decrease if things are going well until it 
>gets quite close to the round trip time for the TCP connection).
>
>The fact that Ethernet does not actually retransmit is also important, as 
>you don't need to worry about retransmissions at the Ethernet layer 
>exacerbating a congestion condition.

There are pros / cons to each scheme but at this point the focus is not on 
why IB does what it does but whether this works well for an IP over IB 
solution.

>transmission of IPv6 packets with payloads of >64K bytes.  RFC 2147 (TCP 
>and UDP over IPv6 Jumbograms) defines how to use TCP and UDP with IPv6 
>Jumbograms.  I don't now if either of these mechanisms is widely 
>implemented, though.

Whether widely or not, they should not be precluded from being supported in 
an implementation.

Mike