Subject: RE: [Ipoverib] Please read - proposed WG termination

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 29 Aug 2005 11:48:25 -0700 (PDT)

Yaron,

If the issue and possible solutions lie completely within L2
(i.e. IB) the work doesn't seem to belong here. If it involves
layer 3 you will have to convince the ADs, who are in doubt
if there is sufficient interest on the approved charter work,
let alone any new work.

Jerry

>X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0
>Content-class: urn:content-classes:message
>MIME-Version: 1.0
>Subject: RE: [Ipoverib] Please read - proposed WG termination
>Date: Mon, 29 Aug 2005 21:02:51 +0300
>X-MS-Has-Attach: 
>X-MS-TNEF-Correlator: 
>Thread-Topic: [Ipoverib] Please read - proposed WG termination
>Thread-Index: AcWswJ1mSOybAIwqRaia2Jea2ZPrtgAAiKtg
>From: "Yaron Haviv" <[email-masked]>
>To: "H.K. Jerry Chu" <[email-masked]>, <[email-masked]>
>Cc: <[email-masked]>, <[email-masked]>
>Content-Transfer-Encoding: 8bit
>X-MIME-Autoconverted: from quoted-printable to 8bit by jurassic.eng.sun.com id 
j7TI3ElM455753
>
>Jerry,
>
>I believe there is one issue that deserves further attention
>Which is delivering highly available IPoIB solutions 
>
>In traditional GbE MAC faking techniques can be used to fail over
>between hosts or between routers (e.g. VRRP)
>Unfortunately this is not applicable to IPoIB (given the IB routing & SA
>structure)
>
>Going into more commercial/enterprise environments this becomes a real
>necessity, today some vendors (including us) use proprietary solutions
>however a standard approach is required 
>
>I proposed few directions in the pass to address the problem, e.g. using
>UNARP (like Sonet does) or using the SA/SM 
>
>Yaron
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On
>> Behalf Of H.K. Jerry Chu
>> Sent: Monday, August 29, 2005 1:38 PM
>> To: [email-masked]
>> Cc: [email-masked]; [email-masked]
>> Subject: [Ipoverib] Please read - proposed WG termination
>> 
>> Hi folks,
>> 
>> With the DHCP over IB draft in IESG's hand, we are done with all
>> the basic IPoIB work, including the architecture (informational),
>> encapsulation, and DHCP (standard tracked). Now seems a good time
>> to decide the next step for the WG.
>> 
>> The work on IPoIB connected mode and most of the MIB drafts started
>> a while back (2003/2002). Over the years the interest/participation
>> level on these work has not been high, and now the work seems to drag
>> on indefinitely. The AD has recently questioned whether the remaining
>> work should be continued or not.
>> 
>> The following is a quick update on their status:
>> 
>> 1. IPoIB connected mode
>> draft-ietf-ipoib-connected-mode-00.txt
>> updated recently
>> 
>> 2. MIB drafts
>> IPoIB Textual Conventions MIB
>> draft-ietf-ipoib-ibmib-tc-mib-06.txt
>> 
>> IPoIB InfiniBand Interfaces MIB
>> draft-ietf-ipoib-ibif-mib-07.txt
>> 
>> IPoIB subnet Management Agent MIB
>> draft-ietf-ipoib-subnet-mgmt-agent-mib-07.txt
>> 
>> IPoIB Channel Adapter MIB
>> draft-ietf-ipoib-channel-adapter-mib-06.txt
>> 
>> IPoIB Baseboard Management Agent MIB
>> draft-ietf-ipoib-baseboard-mgmt-agent-mib-02.txt
>> 
>> IPoIB performance Management Agent MIB
>> draft-ietf-ipoib-perf-mgmt-agent-mib-03.txt
>> 
>> The above 6 MIB drafts have gone through the
>> MIB doctor (Randy Presuhn) review late 2003 and
>> updated accordingly. All the updated drafts expired
>> in April 2005.
>> 
>> 3. IPoIB Subnet Manager MIB
>> draft-ietf-ipoib-subnet-manager-mib-00.txt
>> 
>> Submitted 3/2004 but has expired too.
>> 
>> If you want to see any of the above work continue in the WG,
>> AND is willing to participate to get the work completed, please
>> speak up NOW! You will have until Sep. 20 to do so. After
>> that unless there is evidence a "sufficient" interest level
>> on any of the work above still exists, the WG will be disbanded
>> (or become dormant in order to handle the on-going work on
>> the standard-tracked RFCs) and the unfinished work abandoned.
>> 
>> Note that authors of the above drafts may choose to resubmit
>> their drafts as individual submissions if they want to continue
>> to pursue their work at IETF.
>> 
>> Thank you!
>> 
>> Jerry & Bill
>> 
>> IPoIB co-chairs
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Yaron Haviv" <[email-masked]>
Date: Tue, 30 Aug 2005 01:14:52 +0300

Jerry,

You can see how it was solved in other interconnects (e.g. Sonet)
It was added to the RFC, by requesting hosts to support UNARP RFC

I don't think we can get away with a pure IB L2 implementation, unless
you have some suggestions, how can an IPoIB traffic be rerouted to a new
path/port without updating the sender tables, or somehow making sure
that the new node uses the same GID+QP+LID as the failed one ?

The UNARP option is used to remotely invalidate an ARP entry (forcing a
new addr resolution), But there can be other solutions 

Yaron

> From: H.K. Jerry Chu [mailto:[email-masked]]
> Sent: Monday, August 29, 2005 2:48 PM
> To: [email-masked]; Yaron Haviv

---

From: "Yaron Haviv" <[email-masked]>
Date: Tue, 30 Aug 2005 16:22:23 +0300

> From: Vivek Kashyap [mailto:[email-masked]]
> Sent: Tuesday, August 30, 2005 2:41 AM
> To: Yaron Haviv
> Cc: Michael Krause; [email-masked]; [email-masked];
> [email-masked]
> On Tue, 30 Aug 2005, Yaron Haviv wrote:
> 
> >
> >
> > By nature an IPoIB endpoint also contains QPs, so even theoretically
an
> > SM cannot deal with the problem
> >
> 
> Am I right in assuming that you are suggesting LID + QPN + GID
failover?
> LID is certainly beyond IPoIB. QPN + GID are in the ambit.
> 
> Vivek
> 

I'm not suggesting a solution, just pointing out the problem 
A sender sends to a LID+QP (it resolved the GID->LID before) 
When a node failed the sender keeps on sending to that LID+QP (address
vector)

If one suggests using the SM (e.g. re-route the LID) it won't solve the
problem, since the new receiver may not have the same QP
Something needs to tell the sender to redo the address resolution

It can either be an IB thing (e.g. IB GID out notifications that
requires the SM to now tell it to all the nodes using unicast, probably
not the most scalable approach), or UNARP (a simple null ARP broadcast
that deletes that entry in all nodes, require support for UNARP in IPoIB
drivers)

In any case that deserves a discussion since if we choose the later
option it relates to IPoIB and IETF 

Yaron

---

From: "Yaron Haviv" <[email-masked]>
Date: Tue, 30 Aug 2005 22:40:12 +0300

________________________________

From: Michael Krause [mailto:[email-masked]] 
Sent: Tuesday, August 30, 2005 12:47 PM
To: Vivek Kashyap; Yaron Haviv
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [Ipoverib] Please read - proposed WG termination

Am I missing something here?  I don't recall ever having to tell
everyone to unarp when I migrate an IP address from one NIC to another.
The network stack issues the new address and remote endnodes note the
change.  I don't recall ever having to tell everyone when I migrate a
MAC address from one NIC to another.  Why is IB any different in this
regard for either of these cases?  The layer 2 address which includes
all of the parameters subject to change is updated and the remote
endnodes note the change.  What is so hard about that?

Mike

<Yaron>

The scenario you present is true for a case with a node supporting
single/few IP addresses 

That is not a case for routers that relay on MAC spoofing solutions
(VRRP protocol for example)

In a case of a router there are many IP addresses that can traverse a
single MAC, in some cases the IP addresses are unknown, that's why
instead or taking over an IP address, you take over the MAC which result
in a much faster and robust take-over, this (MAC spoofing) is used in
variety of applications today that would have hard time with IB.

The problem is that in IB unlike GbE you cannot take over the MAC 

This requires to either find an IB way to take over the MAC (GID/LID+QP)
or invalidate the IP-MAC relation on the sender side 

An example can be seen in:

RFC2176 - IPv4 over MAPOS, that uses UNARP to invalidate the ARP cache
for the same reasons 

<Yaron>

At 11:40 PM 8/29/2005, Vivek Kashyap wrote:

IB availability is defined today and one can migrate between ports and
given the use of UD, can easily migrate between HCA either completely or
relatively transparent to the network stack using the IB methods or the
same IP address update using the same methods deployed today on all OS
offerings.  In the end, it really is an implementation issue and fixing
the OpenSM implementations not an IETF issue to resolve any
deficiencies.

Mike

<Yaron>

Mike, the solution that may work over IB is sending gratuitous ARPs

This doesn't cover all the cases, e.g. how would VRRP or MAC faking work
over IB ?

By nature an IPoIB endpoint also contains QPs, so even theoretically an
SM cannot deal with the problem

Am I right in assuming that you are suggesting LID + QPN + GID failover?
LID is certainly beyond IPoIB. QPN + GID are in the ambit.

Vivek

If we want to address interoperability between different vendors I
believe we need to address it through standardization

If you believe there is a solution that addresses my concerns and can be
done in a standard/interoperable way please let me know

Yaron

-----Original Message-----
From: [email-masked] [ mailto:[email-masked]

< mailto:[email-masked] <mailto:[email-masked]> >
] On

Behalf Of H.K. Jerry Chu

Hi folks,

1. IPoIB connected mode

2. MIB drafts

Thank you!

Jerry & Bill

IPoIB co-chairs