Subject: Re: [Ipoverib] recharter consideration

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 12 Sep 2002 19:12:25 -0700 (PDT)

Is the scope of IPoIB over non-UD limited to nodes within the
same IP subnet, or it will allow nodes from different subnets to
communicate directly?

Jerry

>Thomas, Jerry, Bill,
>
>	
>	InfiniBand provides multiple transport types:
>
>	1) Unreliable Datagram (UD)
>	2) Reliable Datagram (RD)
>	3) Unreliable Connected (UC)
>	4) Reliable Connected (RC)
>
>	
>        Of these the UD mode is the best fit for IPoIB since it is
>        supported by all IB channel adapters(CA) and supports
>        multicast. At the same time the other modes, especially RC,
>        offers some advantages that are not to be had through
>        IPoIB-UD. It goes without saying that all the IPoIB
>        implementations must be able to interoperate.
>
>        I list some of the advantages below:
>
>	i) Large MTU
>                The Reliable connected mode supports message sizes
>                upto 2^31 bytes. That is too large a number but it can
>                allow point to point IPoIB links to support reasonably
>                large MTUs (say) 8, 16 or 64K or as desired by the
>                implementation. The maximum MTU over UD is 4096
>                bytes. 
>
>
>	ii) Reliability
>		The RC/RD modes is reliable in more than one way. 
>
>                It provides for 'Automatic path migration' (APM) so
>                that the two endpoints may continue their
>                communication over alternate paths due to errors
>                experience over the current path or for other reasons
>                such as load-balancing. All this happens at the IB
>                layer below the IP communication layer.
>
>                The reliable modes also guarantees that the message is
>                delivered between the peers at most once, in order and
>                without corruption. A reliable connection will cause
>                lesser number of ULP error paths for
>                retransmission/recovery to be traversed. One could
>                also, in some cases, do with the less robust UDP based
>                solutions rather than use TCP solutions.
>
>	iii) De-multiplexing 
>
>                The underlying IB connections can be mapped to
>                different upper layer protocols or even ports. E.g. an
>                RC connection for TCP another for UDP. Or, one for TCP
>                port 5000, another for 5500 and another for the rest.
>                This can be accomplished over RC/UD/UC modes.
>
>                In large systems it is often that the incoming streams
>                are put on different queues to be handled by different
>                threads based on certain parameters. By associating
>                different QPNs with different streams this sorting is
>                done in hardware itself rather than in the interrupt
>                context in the host. E.g. A UDP connection over RC
>                connection tied to the specific port will provide
>                reliable, in order delivery with low overhead.
>
>                Nothing special has to be done to achieve the above,
>                it is a side-effect of the IB level connection
>                establishment that will be required to support any of
>                these modes. See section 4.0 in
>                draft-kashyap-ipoib-advanced-capabilities-00.txt. 
>
>
>What would be requried?
>
>        The requirement on supporting any IPoIB implementation over
>        any of the other modes should be:
>
>	1) Interoperablity
>
>                The IPoIB-UD is mandatory. The rest of the extensions
>                are optional. The presence of data streams belonging
>                to various IPoIB modes on the same IB subnet must not
>                disrupt one another.
>
>                Not all the extensions need be implemented. Rather, as
>                noted above, an implementation may choose to not
>                implement any of the extensions. By the same token, an
>                implementation could choose to implement all the
>                modes. Each of the extensions is independent of the
>                basic IPoIB-UD mode and one another.
>
>
>	2) Describe the setting up of IB level connections
>
>	3) Provide a method of address resolution in non-UD modes
>                The point to point links could be setup 'by hand' but
>                one could use proxy-ARP on IPoIB-UD to resolve other
>                modes. 
>
>        4) Describe the set up of non-UD and QPN-based links
>
>
>Vivek
>
>
>
>> 
>> 
>> OK. Thanks. That clarified the matter to me. I'll address these items.
>> 
>> Vivek
>> --
>> Vivek Kashyap
>> Linux Technology Center, IBM
>> [email-masked]
>> [email-masked]
>> Ph: 503 578 3422 T/L: 775 3422
>> 
>> 
>> 
>>                                                                                                                                         
>>                       [email-masked]                                                                                                
>>                       igh.ibm.com               To:       Vivek 
>Kashyap/Beaverton/IBM@IBMUS                                             
>>                                                 cc:       "H.K. Jerry Chu" 
><[email-masked]>, [email-masked], [email-masked], 
>>                       09/06/2002 11:31           [email-masked]                                                                
>>                       AM                        Subject:  Re: [Ipoverib] 
>recharter consideration                                        
>>                                                                                                                                         
>>                                                                                                                                         
>>                                                                                                                                         
>> 
>> 
>> 
>> 
>> > 1) encapsulating IP datagrams using a non-UD transport (e.g.
>> > UC, RC)
>> 
>> > question - scope is not clear
>> 
>> > <VK> What is meant by 'scope' in this regard? Does
>> > draft-kashyap-ipoib-advanced-capabilities-00.txt provide some of the
>> > answers?
>> > <VK>
>> 
>> There are many non-UD transport modes. Is the proposal to define an
>> ecapsulation for each of other other modes? Or Just one (and if so,
>> which one)?
>> 
>> > 2) demux IP datagrams over multiple QPs
>> 
>> > issue - dubious benefit; it may introduce routing and session
>> > establishment problems?
>> 
>> > <VK> To discuss this issue better I would like to know what are the
>> > routing/session establishment problems that have been brought forward?
>> > <VK>
>> 
>> Let me turn the question around and ask: what problem does 2) fix?
>> Would this be a mandatory-to-implement extension for all nodes?
>> 
>> Thomas
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>
>------------- End Forwarded Message -------------
>
>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 13 Sep 2002 10:27:53 -0700 (PDT)

The non-UD nodes are within the same IP subnet.

For direct communication across IP subnets one would need something
akin to NHRPto bypass the IP routers and make an end to end IB
connection. That is a different discussion and not what I am
proposing.  

Vivek

---

From: "Bill Strahm" <[email-masked]>
Date: Sun, 15 Sep 2002 23:46:01 -0700

A persoanal note with the Chair hat sitting on the airplane seat next to
me...

I think it does imply only subnet communications...

To leave the IP subnet implies that an IP routing descision is made, and
made above the Infiniband layer.  Infiniband (in my mind) is a very
complex, and powerful layer 2 protocol.  However I am not willing to
violate time tested layering techniques that might provide a small
performance gain.

When we leave the subnet, we don't even know if we are talking to
Infiniband anymore or not.  Now that doesn't restrict us to using UD
when talking to the default gateway (the layer 3 router) I would expect
if there is an advantage to using a non-UD transport nodes setting up
this transport to the Default gateway.  

For example if it turns out that error handling/error handoff works
better for another transport (RC for this example) I could see each node
making a RC connection to the default router, that could be backed up by
an IB mechanism so when it fails, each node automatically sends to the
correct backup router for failover.

Note that I have ABSOLUTELY how such a case might work today, I am just
implying that there are reasons for doing it.

Bill Strahm
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of H.K. Jerry Chu
Sent: Thursday, September 12, 2002 7:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Is the scope of IPoIB over non-UD limited to nodes within the same IP
subnet, or it will allow nodes from different subnets to communicate
directly?

Jerry

>>                                                 cc:       "H.K. Jerry
Chu" 

>> 
>> 
>> 
>> 
>> > 1) encapsulating IP datagrams using a non-UD transport (e.g. UC, 
>> > RC)
>> 
>> > draft-kashyap-ipoib-advanced-capabilities-00.txt provide some of 
>> > the answers? <VK>
>> 

>> > routing/session establishment problems that have been brought 
>> > forward? <VK>
>> 
>> [email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>--
>Vivek Kashyap

_______________________________________________
IPoverIB mailing list
[email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib