Subject: Re: [Ipoverib] recharter consideration

From: "Vivek Kashyap" <[email-masked]>
Date: Fri, 6 Sep 2002 11:36:44 -0700

OK. Thanks. That clarified the matter to me. I'll address these items.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      [email-masked]                                                                                                
                      igh.ibm.com               To:       Vivek Kashyap/Beaverton/IBM@IBMUS                                             
                                                cc:       "H.K. Jerry Chu" <[email-masked]>, [email-masked], [email-masked], 
                      09/06/2002 11:31           [email-masked]                                                                
                      AM                        Subject:  Re: [Ipoverib] recharter consideration                                        

> 1) encapsulating IP datagrams using a non-UD transport (e.g.
> UC, RC)

> question - scope is not clear

> <VK> What is meant by 'scope' in this regard? Does
> draft-kashyap-ipoib-advanced-capabilities-00.txt provide some of the
> answers?
> <VK>

There are many non-UD transport modes. Is the proposal to define an
ecapsulation for each of other other modes? Or Just one (and if so,
which one)?

> 2) demux IP datagrams over multiple QPs

> issue - dubious benefit; it may introduce routing and session
> establishment problems?

> <VK> To discuss this issue better I would like to know what are the
> routing/session establishment problems that have been brought forward?
> <VK>

Let me turn the question around and ask: what problem does 2) fix?
Would this be a mandatory-to-implement extension for all nodes?

Thomas

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 6 Sep 2002 12:18:20 -0700 (PDT)

Thomas, Jerry, Bill,

	InfiniBand provides multiple transport types:

	1) Unreliable Datagram (UD)
	2) Reliable Datagram (RD)
	3) Unreliable Connected (UC)
	4) Reliable Connected (RC)

        Of these the UD mode is the best fit for IPoIB since it is
        supported by all IB channel adapters(CA) and supports
        multicast. At the same time the other modes, especially RC,
        offers some advantages that are not to be had through
        IPoIB-UD. It goes without saying that all the IPoIB
        implementations must be able to interoperate.

        I list some of the advantages below:

	i) Large MTU
                The Reliable connected mode supports message sizes
                upto 2^31 bytes. That is too large a number but it can
                allow point to point IPoIB links to support reasonably
                large MTUs (say) 8, 16 or 64K or as desired by the
                implementation. The maximum MTU over UD is 4096
                bytes. 

	ii) Reliability
		The RC/RD modes is reliable in more than one way. 

                It provides for 'Automatic path migration' (APM) so
                that the two endpoints may continue their
                communication over alternate paths due to errors
                experience over the current path or for other reasons
                such as load-balancing. All this happens at the IB
                layer below the IP communication layer.

                The reliable modes also guarantees that the message is
                delivered between the peers at most once, in order and
                without corruption. A reliable connection will cause
                lesser number of ULP error paths for
                retransmission/recovery to be traversed. One could
                also, in some cases, do with the less robust UDP based
                solutions rather than use TCP solutions.

	iii) De-multiplexing 

                The underlying IB connections can be mapped to
                different upper layer protocols or even ports. E.g. an
                RC connection for TCP another for UDP. Or, one for TCP
                port 5000, another for 5500 and another for the rest.
                This can be accomplished over RC/UD/UC modes.

                In large systems it is often that the incoming streams
                are put on different queues to be handled by different
                threads based on certain parameters. By associating
                different QPNs with different streams this sorting is
                done in hardware itself rather than in the interrupt
                context in the host. E.g. A UDP connection over RC
                connection tied to the specific port will provide
                reliable, in order delivery with low overhead.

                Nothing special has to be done to achieve the above,
                it is a side-effect of the IB level connection
                establishment that will be required to support any of
                these modes. See section 4.0 in
                draft-kashyap-ipoib-advanced-capabilities-00.txt. 

What would be requried?

        The requirement on supporting any IPoIB implementation over
        any of the other modes should be:

	1) Interoperablity

                The IPoIB-UD is mandatory. The rest of the extensions
                are optional. The presence of data streams belonging
                to various IPoIB modes on the same IB subnet must not
                disrupt one another.

                Not all the extensions need be implemented. Rather, as
                noted above, an implementation may choose to not
                implement any of the extensions. By the same token, an
                implementation could choose to implement all the
                modes. Each of the extensions is independent of the
                basic IPoIB-UD mode and one another.

	2) Describe the setting up of IB level connections

	3) Provide a method of address resolution in non-UD modes
                The point to point links could be setup 'by hand' but
                one could use proxy-ARP on IPoIB-UD to resolve other
                modes. 

        4) Describe the set up of non-UD and QPN-based links

Vivek

503 578 3422 (o)