Subject: Re: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt

From: "David M. Brean" <[email-masked]>
Date: Sun, 12 Dec 2004 15:55:22 -0500

Hello,

Vivek Kashyap wrote:

snip ...

>
>ok..let me posit what seems to be the summary to me (looking for more comments
>from WG members here). I'm more or less reverting to the earlier version of 
>the draft.
>
>In an IPoIB subnet:
>
>	- Every interface MUST support IPoIB-UD
>
>	- An interface MAY optionally also support IPoIB-CM (one or both)
>		i.e. removing the mutually exclusive restriction on rc/uc
>		Note: IIRC, the same serviceID can be used for both RC/UC.
>		If not then they have to stay mutually exclusive.
>
>	- Interoperability is maintained by all nodes supporting IPoIB-UD. 
>	  Any two interfaces that do not have a connection mode in common will
>	  fall back to IPoIB-UD. 
>
>	- The support of any particular IB mode is indicated by the flags
> 	  in the link layer address. Note: IPoIB-UD is always supported and
>	  hence there are no flags to indicate UD support.
>
>	- An interface completes the IPoIB-UD address resolution and then
>          optionally MAY set up RC/UC connections based on the local support
>	  and received flags.
>
>	- A pure IPoIB-UD implementation ignores the RC/UC flags in link layer
>	  address in received packets. It zeroes them on transmit.
>
>	- Every implementation MUST accept all unicast transmissions received
>	  over any of the IPoIB modes it supports. Multicast/Broadcast by 
>	  their nature will be transmitted and received over the IPoIB-UD only.
>
>	***This implies that an interface MAY transmit/receive a packet
>		over any of RC or UC or UD depending on the modes supported
>	        between the peer IP and itself.***
>		
>	- It is an implementation's decision to connect or retry a connect on
>  	  failure on the CM modes. This decision is independently made per
>	  transmission or reception of a connection request.
>
>	- An implementation MAY make multiple connections to a peer. This
>          is a local decision. So is the decision of the peer to refuse
>          such a connection. 
>	
>	The serviceID, link setup, the link address flags, MTU negotiation etc.
>	are covered in the draft. 
>
>  
>

Wasn't there a suggested change to the alignment of the QPN in the 
service ID?  (See 
http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01158.html).

>	- MTU -- we need to discuss more as below.
>
>  
>
snip ...

>
>The interface MTUs at the peers need not be the same at IP or IB layers.
>
>I agree with the concept of just exchanging the max receive MTU at the IB 
>connection setup. 
>
>  
>
In addition to all the above, there may still be some interest from 
folks in having the specification choose one of the *C transport types 
and a fixed logical MTU.

Also, the I-D mentions ARP and RARP as "protocol types" in the frame 
format.  Seems like these would not apply to connected modes.

-David

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 13 Dec 2004 11:37:15 -0800

Margaret,

As noted, the draft is describing transmission of IP (v4 and v6) over
InfiniBand's connected modes. The underlying layer is not TCP/v6.

Your concern about retransmissions across the layers is a valid. Some
background:

The IB level MTU maybe be theoretically set as high as 2^31 in the
connected modes. However, it will likely be limited to a much smaller size
as set by the peers - the method is described in the draft. The data will
however be broken down to the physical link MTU (2K or 4K). The packets may
be dropped  in 'unreliable connected' mode and no retransmission attempted
at the IB layer. The reliable connected (RC) mode guarantees in-order
delivery. In case of any packet being dropped the RC layer will retransmit.
My discussions with the various IBTA folks indicate that they expect the
retransmission timouts in milliseconds.

Is a discussion of the possible interaction and the requirement for
relevant tuning of the retransmission timers at RC  layer sufficient in the
draft? Afterall, such issues should be common with any implementation of
TCP over a reliable protocol.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Margaret                                                                                                         
                      Wasserman                To:       <[email-masked]>                                                             
                      <margaret@thingma        cc:       [email-masked]                                                             
                      gic.com>                 Subject:  RE: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt          
                      Sent by:                                                                                                         
                      ipoverib-bounces@                                                                                                
                      ietf.org                                                                                                         

                      12/11/2004 05:54                                                                                                 
                      PM                                                                                                               

Hi Bill,

At 6:53 AM -0800 12/11/04, Bill Strahm wrote:
>OK - a couple of misconceptions

I am sorry for my limited understanding of Infiniband...  Hopefully
we all have enough information between us to reach the right answers
here...

>I do not believe there will be a problem however because the timers are
>on such a different scale.  Let me try and give an example... There are
>retries in Ethernet (I am talking about collision detection and back
>off) - What if we decided to worry about TCP retransmitting because the
>Layer 2 packet hasn't been able to get on the wire and the TCP timers
>went off and started retransmitting ?  People don't worry because the
>timers are SO different - this will be even more pronounced in the IB
>world.

I think I understand your point about timers, but I don't really
understand how this works...  Does IB RC actually retransmit lost
packets?  Or does it just have some type of collision-detection &
back-off mechanism, like Ethernet?  If it retransmits, how can it do
that without waiting for at least one round-trip-time to know that
data has been lost?  And, if it doesn't retranmit, how can it
guarantee reliability if data is lost?

Ethernet does not actually retransmit packets... an Ethernet device
will detect a collision during transmission of the packet preamble
and wait a very short period before trying again.  This time is much
shorter than the Ethernet round-trip-time, so it is guaranteed (if I
understand this correctly) to be shorter than the smallest possible
TCP retransmission timer (which will continuously decrease if things
are going well until it gets quite close to the round trip time for
the TCP connection).

The fact that Ethernet does not actually retransmit is also
important, as you don't need to worry about retransmissions at the
Ethernet layer exacerbating a congestion condition.

>I don't think we will be able to move the MTU above 64K - just because I
>believe the IP packet size is (was ??? Has the size parameter gone up ?)
>64K so don't worry about someone trying to stick a 2G IP packet onto the
>wire.  At the same time if it is possible to specify a single IP packet
>that is 2G, maybe we need some text staying the MTU should be limited to
>something sane

IPv6 includes a Jumbo Payload hop-by-hop option which allows the
transmission of IPv6 packets with payloads of >64K bytes.  RFC 2147
(TCP and UDP over IPv6 Jumbograms) defines how to use TCP and UDP
with IPv6 Jumbograms.  I don't now if either of these mechanisms is
widely implemented, though.

Margaret

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 13 Dec 2004 12:00:16 -0800

David,

Addressing your comments here rather than inline since this mailer doesn't
do that well:

1. QPN alingnment:
Yes, there was a suggestion on QPN alignment. I'll address it in the next
draft. Please send in any comments if you have any.

2. "In addition to all the above, there may still be some interest from
folks in having the specification choose one of the *C transport types
and a fixed logical MTU."

Is it not an implementation issue? One can choose to enable only the
desired mode and always use the fixed MTU in all negotiations.

3.  ARP in IPoIB-CM.

It is still part of the definiton of IPoIB-CM.  It is just that it won't be
over the Connected modes.We do ARP/RARP over the UD QP.

Vivek

                                                                                                                                       
                      "David M. Brean"                                                                                                 
                      <David.Brean@Sun.        To:       [email-masked]                                                             
                      COM>                     cc:                                                                                     
                      Sent by:                 Subject:  Re: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt          
                      ipoverib-bounces@                                                                                                
                      ietf.org                                                                                                         

                      12/12/2004 12:55                                                                                                 
                      PM                                                                                                               

Hello,

Vivek Kashyap wrote:

snip ...

snip ...

-David