Subject: RE: [Ipoverib] What's inside an IPoIB link layer address? (SA down)

From: "Vivek Kashyap" <[email-masked]>
Date: Thu, 17 Jan 2002 21:49:46 -0800

<VK>
Jim,
      How do you propose that the SA, if it cannot preserve the LIDs, keep
the GIDs recorded rightly ? GID prefix may live but what about the many
GIDs assigned to each port ? Additionally will it selectively keep all the
MGIDs in the database so that our subnet survives but loose the LIDs ? I
would expect a backup SA to always be able to share the database. At
minimum the backup can collect the existing information and build its
database and assign to only those that are unassigned. If that can't be
provided and the crash of an SA causes all of IB activity to come crashing
down (LIDs/GIDs lost - all SRP connections/IP/SDP connections down)  no one
will buy such an SM/SA. If IB cannot stay up through the crash of an SA
then IB won't survive either.

Vivek
<VK>

Roger,

Could you walk through how the LID is preserved across failover of a SA
(i.e. it died, for whatever reason, and the backup SA takes over)? To me
this is a critically important scenario for Data Centers (we both went
through the GSN ARP fiasco, and have battle scars to prove it). It seems
to me that there is no interoperable mechanism to forward the SM/SA
state to the backup system, and thus the LIDs could be potentially
re-assigned as part of the failover?

The GID prefix though, is generally something the network administrator
has access to (or leave as the default), and thus it is probably
reasonable to expect that the GID is in fact preserved across the
failover?

Jim

> -----Original Message-----
> From: Roger Ronald [mailto:[email-masked]]
> Sent: Thursday, January 17, 2002 12:14 PM
> To: Vivek Kashyap
> Cc: [email-masked]
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
> I think it's important to note that changes in LID should be
infrequent.
> Very infrequent.
>
> VIEO is one of a small number of companies that has a SM at this time.
> We plan to maintain LID assignments whenever possible. That means
> across reboots, across power fails and across subnet merges (when
> possible).
>
> There are corner cases where LID may need to change, just like IP
> addresses occasionaly need to change. But these are VERY odd
> events...like the first time merge of fabrics. They won't happen
> hourly, daily, monthly..... They may happen once or twice in a
> fabric. Ever (unless the admin wants to make changes for the heck of
it).
>
> RR
>
>
> ----- Original Message -----
> From: "Vivek Kashyap" <[email-masked]>
> To: "Bill Strahm" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, January 17, 2002 3:27 AM
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>
> >
> > >
> > > Just free form thinking here...
> > >
> > > Could we say that GIDs are the equivalent of a Fully Qualified
Domain
> Name
> > > (www.strahm.net for example) and LIDs are the equivalent of IP
> addresses
> > > (63.105.18.42).  Now in many cases IP addresses are quite stable,
and
> do not
> > > change.  However would you ship an application with an IP address
> encoded in
> > > it, or with a FQDN.  It would be of course more efficient to use
the
> IP
> > > address and avoid the DNS lookup (that has to go VERY far away,
and
> can take
> > > quite a while for the first SYN message to get out), however if
the
> address
> > > EVER changes, things become hard.
> > >
> > > Can anyone tell me why this is a bad analogy ???
> >
> >         If it was pure IB then GID to LID would be a similar
analogy.
> >
> >         I'll try taking the your analogy further..a couple of points
> >         though - the SM corresonds to the zone primary but the nodes
> >         themselves corresond to both the secondary and the IP nodes
> >         asking for the resolution. So the analogy is not exactly apt
> >         but I'll try..
> >
> >         If the IP changes the DNS must be updated and looked up. In
> >         this case if LID changes the SM is updated (yes) but it then
> >         requires that the SM be looked up. There is no backlink here
> >         to trigger this lookup. This implies that the full lookup
all
> >         the way to the DNS root must be done - if like DNS it is
done
> >         till the local secondary server ie. referring to the local
> >         cached copy and/or merely re-receiving the peer's GID, the
> >         communication will fail. You must have seen the hell that
> >         breaks loose if any of the zone updates doesn't happen due
to
> >         some error and things are left pointing the wrong way.
> >
> >         Now, this would correspond to the internal LID cache flush -
> >         since the GID is the same but the LID has changed. There is
no
> >         hook to say, hey!, go do a zone transfer i.e. lookup the SM
> >         right now. You need a timer.
> >
> >         Now, you could make this time small - you are going to the
SM
> >         too often because things are not changing that much- make it
> >         long and LID change is surely going to happen at a critical
> >         time. Additionally this is another timer per entry for the
> >         programmers/maintainers/administrators to worry about.
> >
> >         I have all along been arguing against keeping LID out due to
> >         this reason too (code is more complex if no LID is another
> >         reason). It has been my assumption that if GID is used
without
> >         LID then one is forced to go to the SM at every address
> >         resolution. Every arp broadcast will have the whole subnet
> >         scurrying off to the SM to ask the requester's LID. Yes,
some
> >         would mark the entry to be completed but not necessarily - a
> >         thundering herd is likely.
> >
> >         This implies that if we don't use the LID we MUST ALWAYS go
to
> >         the SM for the LID - at every ARP since we don't know if LID
> >         has changed. Of course if the GID changed too then the SM
has
> >         to be consulted anyway.
> >
> >         Also note that if we did have a trap from the SM on every
LID
> >         change it still wouldn't be a good solution - too much
> >         latency; points of synchrnonisation; asynchronous updates;
and
> >         most of all the SM would end up doing nothing else but
issueing
> >         traps to every node that registered with it. IB cannot
mandate
> >         broadcasts/multicast (though making everyone look is bad
too)
> >         because multicasting is optional.
> >
> >         In the case of inclusion of the LID - some of the issues are
> >         the same but a gratuitous ARP solves the problem if the LID
> >         changed. It is an instant zone transfer. One could claim
that
> >         with GID only, on receiving any gratuitous ARP entry, the
node
> >         will go to the SM but that will require specialised coding
to
> >         recognise and handle gratuitous ARP differently than regular
> >         updates. Not a good idea.
> >
> >         Even in the absence of gratuitous ARP the changed LID will
be
> >         known when the ARP is done after a cache expiry if LID is
> >         included. If LID is not included one will get a GID back.
That
> >         will require the expiry of the GID to LID cache before one
> >         goes to the SM.
> >
> >         Inclusion of LID makes things very simple. This is because
LID
> >         is always necessary - can't escape it.
> >
> >         I also like inclusion of LID because it cuts off 2x latency
of
> >         GID to LID lookup in the SM and 4x the RTT and the realted
> >         extra interrupts. Fast failover in data centers is
necessary.
> >
> > Vivek
> >
> >
> >
> > >
> > > Bill
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On
> Behalf
> > > Of Larsen, Roy K
> > > Sent: Wednesday, January 16, 2002 6:43 PM
> > > To: 'H.K. Jerry Chu'; [email-masked];
> [email-masked]
> > > Cc: [email-masked]
> > > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
> > >
> > >
> > > > -----Original Message-----
> > > > From: H.K. Jerry Chu [mailto:[email-masked]]
> > > > Sent: Wednesday, January 16, 2002 6:13 PM
> > > > To: [email-masked]; [email-masked]
> > > > Cc: [email-masked]
> > > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer
address?
> > > >
> > > >
> > > > All the proxy ARP usages that I know of use the proxy's mac
> > > > addr in the
> > > > ARP response. I believe the flavor of the proxy ARP you
> > > > alluded below is
> > > > rare, and won't work very well anyway if the link address
includes
> any
> > > > non-persistent component like LID and QPN because it will
> > > > require another
> > > > protocol between the clients and their proxy to constantly
update
> the
> > > > proxy database containing clients mac addrs, which may change
quite
> > > > often.
> > > >
> > > > That is also why I wanted to provide an option to not include
QPN.
> > > > (But I haven't succeeded w/o requiring SIDR or something
> equivalent.)
> > > >
> > > > If LID is included as part of link addr and if it ever changes
on
> > > > a live node due to subnet merge...etc, the driver must inform
its IP
> > > > stack about such change. I believe many stacks don't have such a
> > > > facility. (FOr most IP stacks the assumption is that its own MAC
> > > > addresses just don't change on the fly.) In that case you're
totally
> > > > hosed. Can someone comment on how likely a LID assignment gets
> > > > rescinded?
> > >
> > > It is possible anytime the link goes down and comes back up again
and
> when
> > > the fabic is reconfigured.  This can happen on reboot or simply
> > > unplugging/plugging in a link.  It is ultimately a function of the
SM
> and
> > > how much trouble the implementation wants to go through to keep
LIDs
> > > constant.
> > >
> > > But of course, that has no bearing on whether the LID is contained
in
> the
> > > ARP.  You always need the LID.  Unless you want to do a pathrecord
> lookup
> > > each time your implementation is given a packet to send, you're
going
> to use
> > > a cached LID no matter if it came in the ARP or a pathrecord
lookup on
> the
> > > GID.  If it becomes stale, it may take an ARP cache flush timeout
to
> > > recover.
> > >
> > > Roy
> > >
> > > >
> > > > Jerry
> > > >
> > > > >        A device can always respond for another. The LID of
> > > > the sender
> > > > >        is not the same as the LID for which it has
> > > > responded. It also
> > > > >        allows any network monitoring/debugging device to work
> > > > >        independently. This is the reason why none of the other
> media
> > > > >        does anything like this. You don't pick ethernet source
> > > > >        address from the frame but take it from the ARP/ND
packet.
> > > > >
> > > > >        [ If it was ok we wouldn't need anything except
> QPN/cap_flags
> > > > >        - get the LID from response, consult SA using LID for
GID
> if
> > > > >        you want that, and you are done. ]
> > > > >
> > > > >Vivek
> > > > >
> > > > >>
> > > > >> Why do we need to include the LID inside the link layer
> > > > address when it
> > > > >> is already available in the SLID field? Previous
> > > > discussions indicate
> > > > >> that the SLID value works for proxy ARP/bridge
implementations.
> > > > >>
> > > > >> - Arvind
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Vivek Kashyap [mailto:[email-masked]]
> > > > >> Sent: Wednesday, January 16, 2002 10:29 AM
> > > > >> To: [email-masked]
> > > > >> Cc: [email-masked]
> > > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
> address?
> > > > >>
> > > > >> Ashwin,
> > > > >> Ashwin,
> > > > >>
> > > > >>         I agree that you get more flexibility if you
> > > > include the LID.
> > > > >>         If we include the GID along with the LID we allow all
> > > > >>         implementations the choice to create different
> > > > solutions. We
> > > > >>         don't tie them down to one thing. Choosing one or the
> other
> > > > >>         (LID or GID) blocks out various advantages that
different
> > > > >>         implementations seek. Hence I suggested that let us
> include
> > > > >>         both and move on.
> > > > >>
> > > > >> Vivek
> > > > >>
> > > > >> <snip>
> > > > >>
> > > > >> _______________________________________________
> > > > >> IPoverIB mailing list
> > > > >> [email-masked]
> > > > >> http://www.ietf.org/mailman/listinfo/ipoverib
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Vivek Kashyap
> > > > >Linux Technology Center, IBM
> > > > >[email-masked]
> > > > >[email-masked]
> > > > >503 578 3422 (o)
> > > > >
> > > > >_______________________________________________
> > > > >IPoverIB mailing list
> > > > >[email-masked]
> > > > >http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > [email-masked]
> > > > http://www.ietf.org/mailman/listinfo/ipoverib
> > > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email-masked]
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email-masked]
> > > http://www.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > [email-masked]
> > [email-masked]
> > 503 578 3422 (o)
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > http://www.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
>
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Ashwin Kamath" <[email-masked]>
Date: Fri, 18 Jan 2002 13:03:45 -0600

There are two major implementation camps, with SA and without SA. Is it the
goal of the architecture to be complete enough to deal with interoperability
between implementations? Is interop supposed to be gauranteed if two
distinct implementations adhere to the specs?
 In my opinion, it must be complete enough that two distinct
implementations, must interoperate if they adhere to IpoIB spec. Comments...

-Ashwin.