Subject: RE: [Ipoverib] recharter consideration

From: "Rosenstock, Hal" <[email-masked]>
Date: Fri, 13 Sep 2002 11:20:34 -0400

Hi,

I would think the scope should include intersubnet non-UD transports if we
are to take this on
as that is the current scope for UD. Is there a reason it should be
different for non-UD if this
is adopted as part of the charter ?

-- Hal

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 10:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Is the scope of IPoIB over non-UD limited to nodes within the
same IP subnet, or it will allow nodes from different subnets to
communicate directly?

Jerry

>Thomas, Jerry, Bill,
>
>	
>	InfiniBand provides multiple transport types:
>
>	1) Unreliable Datagram (UD)
>	2) Reliable Datagram (RD)
>	3) Unreliable Connected (UC)
>	4) Reliable Connected (RC)
>
>	
>        Of these the UD mode is the best fit for IPoIB since it is
>        supported by all IB channel adapters(CA) and supports
>        multicast. At the same time the other modes, especially RC,
>        offers some advantages that are not to be had through
>        IPoIB-UD. It goes without saying that all the IPoIB
>        implementations must be able to interoperate.
>
>        I list some of the advantages below:
>
>	i) Large MTU
>                The Reliable connected mode supports message sizes
>                upto 2^31 bytes. That is too large a number but it can
>                allow point to point IPoIB links to support reasonably
>                large MTUs (say) 8, 16 or 64K or as desired by the
>                implementation. The maximum MTU over UD is 4096
>                bytes. 
>
>
>	ii) Reliability
>		The RC/RD modes is reliable in more than one way. 
>
>                It provides for 'Automatic path migration' (APM) so
>                that the two endpoints may continue their
>                communication over alternate paths due to errors
>                experience over the current path or for other reasons
>                such as load-balancing. All this happens at the IB
>                layer below the IP communication layer.
>
>                The reliable modes also guarantees that the message is
>                delivered between the peers at most once, in order and
>                without corruption. A reliable connection will cause
>                lesser number of ULP error paths for
>                retransmission/recovery to be traversed. One could
>                also, in some cases, do with the less robust UDP based
>                solutions rather than use TCP solutions.
>
>	iii) De-multiplexing 
>
>                The underlying IB connections can be mapped to
>                different upper layer protocols or even ports. E.g. an
>                RC connection for TCP another for UDP. Or, one for TCP
>                port 5000, another for 5500 and another for the rest.
>                This can be accomplished over RC/UD/UC modes.
>
>                In large systems it is often that the incoming streams
>                are put on different queues to be handled by different
>                threads based on certain parameters. By associating
>                different QPNs with different streams this sorting is
>                done in hardware itself rather than in the interrupt
>                context in the host. E.g. A UDP connection over RC
>                connection tied to the specific port will provide
>                reliable, in order delivery with low overhead.
>
>                Nothing special has to be done to achieve the above,
>                it is a side-effect of the IB level connection
>                establishment that will be required to support any of
>                these modes. See section 4.0 in
>                draft-kashyap-ipoib-advanced-capabilities-00.txt. 
>
>
>What would be requried?
>
>        The requirement on supporting any IPoIB implementation over
>        any of the other modes should be:
>
>	1) Interoperablity
>
>                The IPoIB-UD is mandatory. The rest of the extensions
>                are optional. The presence of data streams belonging
>                to various IPoIB modes on the same IB subnet must not
>                disrupt one another.
>
>                Not all the extensions need be implemented. Rather, as
>                noted above, an implementation may choose to not
>                implement any of the extensions. By the same token, an
>                implementation could choose to implement all the
>                modes. Each of the extensions is independent of the
>                basic IPoIB-UD mode and one another.
>
>
>	2) Describe the setting up of IB level connections
>
>	3) Provide a method of address resolution in non-UD modes
>                The point to point links could be setup 'by hand' but
>                one could use proxy-ARP on IPoIB-UD to resolve other
>                modes. 
>
>        4) Describe the set up of non-UD and QPN-based links
>
>
>Vivek
>
>
>
>> 
>> 
>> OK. Thanks. That clarified the matter to me. I'll address these items.
>> 
>> Vivek
>> --
>> Vivek Kashyap
>> Linux Technology Center, IBM
>> [email-masked]
>> [email-masked]
>> Ph: 503 578 3422 T/L: 775 3422
>> 
>> 
>> 
>>

>>                       [email-masked]

>>                       igh.ibm.com               To:       Vivek 
>Kashyap/Beaverton/IBM@IBMUS                                             
>>                                                 cc:       "H.K. Jerry
Chu" 
><[email-masked]>, [email-masked], [email-masked], 
>>                       09/06/2002 11:31           [email-masked]

>>                       AM                        Subject:  Re: [Ipoverib] 
>recharter consideration                                        
>>

>> 
>> 
>> 
>> 
>> > 1) encapsulating IP datagrams using a non-UD transport (e.g.
>> > UC, RC)
>> 
>> > question - scope is not clear
>> 
>> > <VK> What is meant by 'scope' in this regard? Does
>> > draft-kashyap-ipoib-advanced-capabilities-00.txt provide some of the
>> > answers?
>> > <VK>
>> 
>> There are many non-UD transport modes. Is the proposal to define an
>> ecapsulation for each of other other modes? Or Just one (and if so,
>> which one)?
>> 
>> > 2) demux IP datagrams over multiple QPs
>> 
>> > issue - dubious benefit; it may introduce routing and session
>> > establishment problems?
>> 
>> > <VK> To discuss this issue better I would like to know what are the
>> > routing/session establishment problems that have been brought forward?
>> > <VK>
>> 
>> Let me turn the question around and ask: what problem does 2) fix?
>> Would this be a mandatory-to-implement extension for all nodes?
>> 
>> Thomas
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>
>------------- End Forwarded Message -------------
>
>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Yaron Haviv <[email-masked]>
Date: Mon, 16 Sep 2002 10:30:06 +0300

I have been monitoring the router-RC emails recently

I believe it is obvious that the RC connection is terminated at the end of
the IP subnet, and doesn't continue to the end node (in case of
multi-subnet):
1. GRH header changes
2. In many cases different subnets imply different partitions, so P_Key
might need to change too
3. ARP is done only to the router (in multi-subnet case), RC/UD is a
parameter in the ARP capabilities, not an e2e
4. Different subnets might have different MTU's
5. Traffic might go to a different medium (e.g. GbE) from the router

I don't see any problem that a router will support RC termination and
creation of a new RC to the new subnet
It could also use RC on one side and UD on the other
In fact our product (which is a router) can support those capability
In any case where there is a router that doesn't want to, or cant support
RC, it can just disable the RC capability bit in the ARP reply, and the
client will use UD

I don't see any special consideration needs to be taken in inter-subnet case
from the client perspective
I believe the answer to do we want to enable RC in such cases is, yes

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Rosenstock, Hal
Sent: Friday, September 13, 2002 6:21 PM
To: H.K. Jerry Chu; [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] recharter consideration

Hi,

-- Hal

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 10:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Jerry

Chu"