Subject: Re: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt

From: Margaret Wasserman <[email-masked]>
Date: Fri, 10 Dec 2004 12:19:41 -0500

Hi All,

I also have some comments on draft-kashyap-ipoib-connected-mode-02.txt.

I haven't thoroughly reviewed the document, so I may have some 
further comments later, but in my initial read, one major concern 
jumped out at me.

If I understand correctly, the reliable connected mode of IPOIB is 
TCP/IPv6.  Is that correct?

If so, I think that you need to carefully consider the issues 
associated with tunneling TCP inside of TCP.  In particular, I am 
concerned about this section:

        The default MTU of the IPoIB-CM interface is 2044 octets i.e.
         2048 octet IPoIB-link MTU minus the 4 octet encapsulation
         header.

         The connected modes of InfiniBand allow message sizes up to 2^31
         octets.  Therefore, IPoIB-CM can use a much larger MTU for
         unicast communication between any two endpoints. At the same
         time the maximum and/or optimal payload that can be received or
         sent over an InfiniBand connection is dependent on the
         implementation, HCA and the resources configured.

Having a larger MTU at the higher-level TCP layer than at the 
lower-level TCP layer may lead to situations where the upper and 
lower layers' retransmit timers will both fire at the same time, 
causing real problems in the case of congestion.

This may happen because bothTCP layers will use the same algorithm 
(and maybe the same code) for calculating the retransmission time. 
When a packet is lost, the lower layer's TCP retransmission timer 
will fire, causing a retransmission of one lower-layer MTU of data 
(2048 bytes), AND the upper layer's TCP retransmission timer will 
also fire, causing a retransmission of one upper-layer MTU of data 
(up to 2^31 bytes).  The lower layer will not recognize that the 
upper layer's retransmission is duplicate data, so this will result 
in the transmission of many 2K packets when a single 2K packet is 
lost.

We may want to talk to the Transport area to determine if there are 
other issues associated with tunneling TCP inside of TCP/IP.

Thoughts?

Margaret

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Fri, 10 Dec 2004 12:54:42 -0800

At 09:19 AM 12/10/2004, Margaret Wasserman wrote:

>I haven't thoroughly reviewed the document, so I may have some further 
>comments later, but in my initial read, one major concern jumped out at me.
>

No.  It uses InfiniBand RC or UC to communicate IP datagrams (v4 / v6) 
between connected endnodes.

>If so, I think that you need to carefully consider the issues associated 
>with tunneling TCP inside of TCP.  In particular, I am concerned about 
>this section:
>
>Having a larger MTU at the higher-level TCP layer than at the lower-level 
>TCP layer may lead to situations where the upper and lower layers' 
>retransmit timers will both fire at the same time, causing real problems 
>in the case of congestion.
>
>This may happen because bothTCP layers will use the same algorithm (and 
>maybe the same code) for calculating the retransmission time. When a 
>packet is lost, the lower layer's TCP retransmission timer will fire, 
>causing a retransmission of one lower-layer MTU of data (2048 bytes), AND 
>the upper layer's TCP retransmission timer will also fire, causing a 
>retransmission of one upper-layer MTU of data (up to 2^31 bytes).  The 
>lower layer will not recognize that the upper layer's retransmission is 
>duplicate data, so this will result in the transmission of many 2K packets 
>when a single 2K packet is lost.
>
>We may want to talk to the Transport area to determine if there are other 
>issues associated with tunneling TCP inside of TCP/IP.

Rest is based on a misconception as there is only IB below IP and no 
tunneling of TCP over TCP occurs.

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 13 Dec 2004 09:57:28 -0800

At 04:42 AM 12/11/2004, Margaret Wasserman wrote:
>And what is InfiniBand RC, under the covers?

An InfiniBand reliable connection which is an IB transport type.  It isn't 
TCP or SCTP.

>Well, I know that there is another IP(v6) in there, as IB is very closely 
>based on IPv6.

I co-developed / led IB and aside from trying to align the addressing to 
IPv6, it isn't IPv6.  Never was my intention to have it be IPv6.

>   Is RC based on TCP? SCTP?

No.

>  Or did IB define a different reliable, connection-oriented IP protocol?

It is a connected oriented protocol that is not IP based.

>  If the latter, what type of retransmission and congestion control 
> algorithms are used by IB RC?

Retransmission is via a strongly ordered scheme thus there isn't the 
concept of SACK, etc.  This keeps it rather thin and easy to implement in 
hardware.  Congestion control was recently added and uses a variation of 
notification as well as selective back-off algorithms.

>   And, has there been any study regarding how they would interact with 
> the retransmission and congestion control algorithms of TCP or other 
> reliable upper-layer transports?

They are orthogonal as IB only guarantees a packet is delivered to an 
endnode.  It does not state anything about what occurs above.  So, in the 
case of IP over IB, if IP drops the datagram for whatever purpose, the 
associated transport or ULP will take appropriate recovery actions if 
desired.

>Even if this isn't TCP, per se, there may still be issues, but it will 
>take more careful research to determine what they are.

There are a variety of universities, etc. who are doing such research.  Try 
Ohio State to see what they have completed to date.

Mike