Subject: [Ipoverib] recharter consideration

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 4 Sep 2002 16:32:06 -0700 (PDT)

In Yokohama we discussed a few new items that people thought
may be useful and have suggested to add them to the WG charter.
Note that these work items still need to be blessed by the
ADs and IESG before they can be adopted by the WG. For each
work item the ADs are looking for

a. clear benefit (over the added cost) and sufficient interest
from the WG

b. concise definition and scope

c. deliverables and milestones

The following is a summary of the work items that have been
brought up so far and the questions/issues raised by the audience
in the Yokohama meeting.

1) encapsulating IP datagrams using a non-UD transport (e.g.
UC, RC)

question - scope is not clear

2) demux IP datagrams over multiple QPs

issue - dubious benefit; it may introduce routing and session
establishment problems?

(Two opposed this item in f2f meeting.)

3) other client capability, e.g. SDP

issue - non-IP therefore out of scope

4) skipping TCP/UDP checksum when running over RC/UC.

issues - serious concern from many. difficult to ensure end2end
property. could be made to work in a limited set of cases but the
added complexity seems far outweigh the benefit.

All the above fall into the category of advanced features,
and their usage is completely optional. This way they should
not affect the basic IP over IB operation defined by the
existing IPoIB drafts.

Given the amount of concerns raised so far against 4), I'd like to
remove it from the list of consideration. If you have a problem with
that please bring it up to me and Bill. For the rest three I'll need
you to

1. show your interest and/or volunteer for the work
2. provide a concise description on what the item is, why it should
be considered (e.g. the benefit), and the deliverables

Jerry and Bill,
IPoIB co-chairs

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Sat, 7 Sep 2002 09:17:04 -0400

Hi,

At IBA 1.0.a, RMPP was only an SA protocol. At IBA 1.1, it is a general
purpose protocol. RMPP is still limited to 256 byte UD (for management
classes).

Also, there is the advantage that RC is implemented in hardware whereas RMPP
is implemented in software and hence costs additional CPU cycles.

-- Hal

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Friday, September 06, 2002 4:18 PM
To: Glaser, Steve
Cc: Thomas Narten; Vivek Kashyap; H.K. Jerry Chu; [email-masked];
[email-masked]; [email-masked]
Subject: RE: [Ipoverib] recharter consideration 

On 9/6/02, Glaser, Steve wrote:

>The only reason I can think of for doing this is to
>overcome a Path MTU limitation.
>

Why would RC be used, rather than reliable multi-packet
transation protocol used for large management datagrams
(section 15.3)?
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 9 Sep 2002 11:27:37 -0700

I've been tempted to chime in here but Caitlin seems to be covering my
position better than I could.  So I'll just say Ditto.

Roy

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Monday, September 09, 2002 8:55 AM
To: [email-masked]
Subject: RE: [Ipoverib] recharter consideration

On 9/8/02, Yaron Haviv wrote:

>Caitlin,
>
>The issue is really that TCP/IP works mostly over Ethernet
>When running over IB few limitations arouse, one is the
>inability to do MAC faking (some may consider it a
>benefit) This has nothing to do with DAT, SDP, .. it
>relates to how addresses are resolved And by the way SDP,
>DAPL relay on IPoIB to provide the address resolution
>capability
>
>Just a simple example in Ethernet a backup router sends a
>packet with the Master router MAC address in case of a
>fail-over This automatically configures the switches to
>route the traffic destined to that MAC address to the
>backup router In IB there is no way to do such a thing,
>since the HW addresses might be even fixed (base GID=GUID)
>(VRRP protocol standardize just that over Ethernet and
>other mediums) Other scenarios could be when one host
>wants to take over HW address on another this could be
>replaced by gratuitous ARP's, how ever they are limited in
>some apps (e.g. in case of multiple IP addresses,
>active/active scenarios, no client support, ..)
>
>The solution might be either change the IB L2, enable GID
>changes on the fly + immediate notifications to update the
>switches Or it could fall within the scope of IPoIB (RC)
>
>One solution that could work with IPoIB RC is for example
>in the connection setup headers pass an alternate address
>That would be used in case of failure, since RC connection
>setup headers are in the scope of IPoIB RC, this is Ok
>

I believe you are confusing an existing method (MAC
spoofing) with a requirement. The functional goal is to
allow an alternate servver to be selected automatically.

I would contend that that problem can be solved either
through the IB Communications Manager or by simply using
IPoIB to do IP to GID resolution before establishing the
connection.

MAC spoofing as a failover does not preserve an existing
connection. Path migration does. If the goal is to achieve
this sort of connection transparent as to whether an IB or
IP network is in use then SDP and DAPL would seem to be more
appropriate.

If the goal is to make a byte or message stream ULP
connection over an IB network, what benefit is there in
insisting on tunneling that connection in IP format?

If IPoIB is going to expand its charter, it should be on the
basis of solving ULP problems, not merely trying to make it
a universal solution. That becomes an effort in "fixing" IB
to be more IP like. In my opinion, the end result will be
unappealing to any target market.

One legitimate goal that has been cited was supporting a
larger MTU to allow atomic delivery of larger IP frames, at
least matching jumbo Ethernet frames. I think that RD and UC
should be considered for this goal. They meet the objective
while imposing fewer additional semantics than found in the
orignal link layer.

Caitlin Bestler
http://www.asomi.com/CaitlinBestler/
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 12 Sep 2002 18:20:45 -0700 (PDT)

When failing over IP address one either migrates the link layer addr,
or use gratuitous ARP to trigger flushes of remote ARP caches.

Compared to IEEE 802 MAC addr, components in IPoIB link layer addr
have their semantics (e.g. QPN) with the link layer thus leaving
less room for maneuver. I'd like to see more info on what problem(s)
we are solving (e.g. faster failover?) and what the ideas are
before deciding if this is interesting to pursue.

Jerry

>Larry,
>
>Obviously it is very helpful for routers (its like VRRP)
>However many applications, clusters use this mechanism to enable IP take
>over by the slave in case of fail-over
>(And clustered apps will run on IB as we know)
>
>As to bill's remark I rather first see if there is interest for it in IPoIB
>and maybe even get some ideas before I go and write a complete draft for it
>
>Yaron
>
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Larry Butler
>Sent: Friday, September 06, 2002 6:57 PM
>To: [email-masked]'; 'H.K. Jerry Chu'; [email-masked]
>Cc: [email-masked]
>Yaron,
>Are you suggesting that an RFC 2338-like Router Redundancy Protocol be
>looked at for IPoIB? Or does it go beyond routers and also apply to any IP
>end station?
>
>Larry
>
>From: Yaron Haviv [mailto:[email-masked]]
>Sent: Thursday, September 05, 2002 1:40 PM
>To: 'H.K. Jerry Chu'; [email-masked]
>Cc: [email-masked]
>Jerry,
>We can try and target fail-over with IPoIB
>In Ethernet a system can just use a MAC faking technique to migrate to a
>standby station
>In IB a client station cannot take over an HW address of a different machine
>This poses restrictions on fail-over mechanisms
>(One might suggest ARP's for it, however it doesn't solve the same problems)
>
>We can try to have a mechanism, especially in conjunction with RC
>Where we have a standby HW address configured, in case of failure we move to
>the alternate address
>(It is different from APM since the two paths direct us to two different end
>nodes, not one)
>
>Yaron
>
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>H.K. Jerry Chu
>Sent: Thursday, September 05, 2002 2:32 AM
>To: [email-masked]
>Cc: [email-masked]
>Subject: [Ipoverib] recharter consideration
>

---

From: Larry Butler <[email-masked]>
Date: Fri, 13 Sep 2002 08:05:44 -0700

Jerry,

I see the need for a seamless fail over mechanism (this implies speed so
that various layer timeouts do not occur), where the fail over is as
transparent as possible to the ULP.

I also am not sure that this this falls under the charter of IPoIB. Just
because the application is using a layer-3 IP address, it does not mean that
a transport such as SDP or DAPL cannot be extended to offer the fail over
mechanism. I could see, for instance, the IBTA adding a fail over extension
to SDP. 

I also can argue the other way (at least for SDP), in that IP address
resolution (and local IP address identity) is handled by IPoIB and that a
quick IP address migration from one IPoIB instance to another might
facilitate any future fail over mechanisms added to the transports.

Larry

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 6:21 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [Ipoverib] recharter consideration

Jerry

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Fri, 13 Sep 2002 10:19:51 -0700 (PDT)

Hal,

>I would think the scope should include intersubnet non-UD transports if we
>are to take this on
>as that is the current scope for UD. Is there a reason it should be
>different for non-UD if this
>is adopted as part of the charter ?

The draft published by Vivek proposed to use bits in the IPoIB link
layer address to support non-UD transports. Since layer-2 addresses
don't in general get past IP routers, that proposal limits itself
to nodes within the same IP subnet.

Going beyond a single IP subnet requires more work. Does the benefit
justify the cost?

Jerry

>Sent: Thursday, September 12, 2002 10:12 PM
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: [Ipoverib] recharter consideration
>
>
>Is the scope of IPoIB over non-UD limited to nodes within the
>same IP subnet, or it will allow nodes from different subnets to
>communicate directly?
>
>Jerry
>
>>Thomas, Jerry, Bill,
>>
>>	
>>	InfiniBand provides multiple transport types:
>>
>>	1) Unreliable Datagram (UD)
>>	2) Reliable Datagram (RD)
>>	3) Unreliable Connected (UC)
>>	4) Reliable Connected (RC)
>>
>>	
>>        Of these the UD mode is the best fit for IPoIB since it is
>>        supported by all IB channel adapters(CA) and supports
>>        multicast. At the same time the other modes, especially RC,
>>        offers some advantages that are not to be had through
>>        IPoIB-UD. It goes without saying that all the IPoIB
>>        implementations must be able to interoperate.
>>
>>        I list some of the advantages below:
>>
>>	i) Large MTU
>>                The Reliable connected mode supports message sizes
>>                upto 2^31 bytes. That is too large a number but it can
>>                allow point to point IPoIB links to support reasonably
>>                large MTUs (say) 8, 16 or 64K or as desired by the
>>                implementation. The maximum MTU over UD is 4096
>>                bytes. 
>>
>>
>>	ii) Reliability
>>		The RC/RD modes is reliable in more than one way. 
>>
>>                It provides for 'Automatic path migration' (APM) so
>>                that the two endpoints may continue their
>>                communication over alternate paths due to errors
>>                experience over the current path or for other reasons
>>                such as load-balancing. All this happens at the IB
>>                layer below the IP communication layer.
>>
>>                The reliable modes also guarantees that the message is
>>                delivered between the peers at most once, in order and
>>                without corruption. A reliable connection will cause
>>                lesser number of ULP error paths for
>>                retransmission/recovery to be traversed. One could
>>                also, in some cases, do with the less robust UDP based
>>                solutions rather than use TCP solutions.
>>
>>	iii) De-multiplexing 
>>
>>                The underlying IB connections can be mapped to
>>                different upper layer protocols or even ports. E.g. an
>>                RC connection for TCP another for UDP. Or, one for TCP
>>                port 5000, another for 5500 and another for the rest.
>>                This can be accomplished over RC/UD/UC modes.
>>
>>                In large systems it is often that the incoming streams
>>                are put on different queues to be handled by different
>>                threads based on certain parameters. By associating
>>                different QPNs with different streams this sorting is
>>                done in hardware itself rather than in the interrupt
>>                context in the host. E.g. A UDP connection over RC
>>                connection tied to the specific port will provide
>>                reliable, in order delivery with low overhead.
>>
>>                Nothing special has to be done to achieve the above,
>>                it is a side-effect of the IB level connection
>>                establishment that will be required to support any of
>>                these modes. See section 4.0 in
>>                draft-kashyap-ipoib-advanced-capabilities-00.txt. 
>>
>>
>>What would be requried?
>>
>>        The requirement on supporting any IPoIB implementation over
>>        any of the other modes should be:
>>
>>	1) Interoperablity
>>
>>                The IPoIB-UD is mandatory. The rest of the extensions
>>                are optional. The presence of data streams belonging
>>                to various IPoIB modes on the same IB subnet must not
>>                disrupt one another.
>>
>>                Not all the extensions need be implemented. Rather, as
>>                noted above, an implementation may choose to not
>>                implement any of the extensions. By the same token, an
>>                implementation could choose to implement all the
>>                modes. Each of the extensions is independent of the
>>                basic IPoIB-UD mode and one another.
>>
>>
>>	2) Describe the setting up of IB level connections
>>
>>	3) Provide a method of address resolution in non-UD modes
>>                The point to point links could be setup 'by hand' but
>>                one could use proxy-ARP on IPoIB-UD to resolve other
>>                modes. 
>>
>>        4) Describe the set up of non-UD and QPN-based links
>>
>>
>>Vivek
>>
>>
>>
>>> 
>>> 
>>> OK. Thanks. That clarified the matter to me. I'll address these items.
>>> 
>>> Vivek
>>> --
>>> Vivek Kashyap
>>> Linux Technology Center, IBM
>>> [email-masked]
>>> [email-masked]
>>> Ph: 503 578 3422 T/L: 775 3422
>>> 
>>> 
>>> 
>>>
>
>>>                       [email-masked]
>
>>>                       igh.ibm.com               To:       Vivek 
>>Kashyap/Beaverton/IBM@IBMUS                                             
>>>                                                 cc:       "H.K. Jerry
>Chu" 
>><[email-masked]>, [email-masked], [email-masked], 
>>>                       09/06/2002 11:31           [email-masked]
>
>>>                       AM                        Subject:  Re: [Ipoverib] 
>>recharter consideration                                        
>>>
>
>>>
>
>>>
>
>>> 
>>> 
>>> 
>>> 
>>> > <VK> What is meant by 'scope' in this regard? Does
>>> > draft-kashyap-ipoib-advanced-capabilities-00.txt provide some of the
>>> > answers?
>>> > <VK>
>>> 
>>> There are many non-UD transport modes. Is the proposal to define an
>>> ecapsulation for each of other other modes? Or Just one (and if so,
>>> which one)?
>>> 
>>> > <VK> To discuss this issue better I would like to know what are the
>>> > routing/session establishment problems that have been brought forward?
>>> > <VK>
>>> 
>>> Let me turn the question around and ask: what problem does 2) fix?
>>> Would this be a mandatory-to-implement extension for all nodes?
>>> 
>>> Thomas
>>> 
>>> 
>>> 
>>> 
>>> 
>>-- 
>>Vivek Kashyap
>>Linux Technology Center, IBM
>>[email-masked] 
>>[email-masked] 
>>503 578 3422 (o)
>>
>>------------- End Forwarded Message -------------
>>
>>
>

---

From: "Vivek Kashyap" <[email-masked]>
Date: Fri, 13 Sep 2002 10:33:42 -0700

Hal,

 the subnet being referred to by Jerry is the IP subnet. The IPoIB-UD span
across  IB subnets is achieved by spanning the same IP subnet across
mulitple IB subnets.  The direct communication therefore is within the same
IP Subnet. It is across IB subnets if the IP subnet includes them. That
would be the scope of non-UD too.   Anything beyond it, I think, is an
extension of IPoIB-UD too.

Vivek

                      "Rosenstock, Hal"                                                                                                
                      <hal@pacelinesyst        To:       "H.K. Jerry Chu" <[email-masked]>, Vivek Kashyap/Beaverton/IBM@IBMUS   
                      ems.com>                 cc:       [email-masked]                                                             
                                               Subject:  RE: [Ipoverib] recharter consideration                                        
                      09/13/2002 08:20                                                                                                 
                      AM                                                                                                               

Hi,

-- Hal

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 10:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Jerry

Chu"

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Fri, 13 Sep 2002 13:53:39 -0400

Hi Vivek and Jerry,

I misunderstood and thought that what was being said was that non UD
transports could not span IB not IP subnets
and that was what I was clarifying. IP subnet cut thru has been done with
ATM and is indeed possible but required additional protocol support (NHRP)
in IP routers to accomplish this. This could yet be another charter
enhancement as there may be advantages to this as well as there was in terms
of ATM.

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Friday, September 13, 2002 1:34 PM
To: Rosenstock, Hal
Cc: [email-masked]; H.K. Jerry Chu
Subject: RE: [Ipoverib] recharter consideration

Hal,

Vivek

                      "Rosenstock, Hal"

                      <hal@pacelinesyst        To:       "H.K. Jerry Chu"
<[email-masked]>, Vivek Kashyap/Beaverton/IBM@IBMUS   
                      ems.com>                 cc:       [email-masked]

                                               Subject:  RE: [Ipoverib]
recharter consideration                                        
                      09/13/2002 08:20

                      AM

Hi,

-- Hal

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 10:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Jerry

Chu"