Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

From: "Larsen, Roy K" <[email-masked]>
Date: Wed, 16 Jan 2002 18:42:31 -0800

> -----Original Message-----
> From: H.K. Jerry Chu [mailto:[email-masked]]
> Sent: Wednesday, January 16, 2002 6:13 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> 
> 
> All the proxy ARP usages that I know of use the proxy's mac 
> addr in the
> ARP response. I believe the flavor of the proxy ARP you 
> alluded below is
> rare, and won't work very well anyway if the link address includes any
> non-persistent component like LID and QPN because it will 
> require another
> protocol between the clients and their proxy to constantly update the
> proxy database containing clients mac addrs, which may change quite
> often.
> 
> That is also why I wanted to provide an option to not include QPN.
> (But I haven't succeeded w/o requiring SIDR or something equivalent.)
> 
> If LID is included as part of link addr and if it ever changes on
> a live node due to subnet merge...etc, the driver must inform its IP
> stack about such change. I believe many stacks don't have such a
> facility. (FOr most IP stacks the assumption is that its own MAC
> addresses just don't change on the fly.) In that case you're totally
> hosed. Can someone comment on how likely a LID assignment gets
> rescinded?

It is possible anytime the link goes down and comes back up again and when
the fabic is reconfigured.  This can happen on reboot or simply
unplugging/plugging in a link.  It is ultimately a function of the SM and
how much trouble the implementation wants to go through to keep LIDs
constant.

But of course, that has no bearing on whether the LID is contained in the
ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
each time your implementation is given a packet to send, you're going to use
a cached LID no matter if it came in the ARP or a pathrecord lookup on the
GID.  If it becomes stale, it may take an ARP cache flush timeout to
recover.  

Roy
 
> 
> Jerry
> 
> >        A device can always respond for another. The LID of 
> the sender
> >        is not the same as the LID for which it has 
> responded. It also
> >        allows any network monitoring/debugging device to work
> >        independently. This is the reason why none of the other media
> >        does anything like this. You don't pick ethernet source
> >        address from the frame but take it from the ARP/ND packet.
> >
> >        [ If it was ok we wouldn't need anything except QPN/cap_flags
> >        - get the LID from response, consult SA using LID for GID if
> >        you want that, and you are done. ]
> >
> >Vivek
> >
> >> 
> >> Why do we need to include the LID inside the link layer 
> address when it
> >> is already available in the SLID field? Previous 
> discussions indicate
> >> that the SLID value works for proxy ARP/bridge implementations.
> >> 
> >> - Arvind
> >> 
> >> -----Original Message-----
> >> From: Vivek Kashyap [mailto:[email-masked]] 
> >> Sent: Wednesday, January 16, 2002 10:29 AM
> >> To: [email-masked]
> >> Cc: [email-masked]
> >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >> 
> >> Ashwin,
> >> Ashwin,
> >> 
> >>         I agree that you get more flexibility if you 
> include the LID.
> >>         If we include the GID along with the LID we allow all
> >>         implementations the choice to create different 
> solutions. We
> >>         don't tie them down to one thing. Choosing one or the other
> >>         (LID or GID) blocks out various advantages that different
> >>         implementations seek. Hence I suggested that let us include
> >>         both and move on.
> >> 
> >> Vivek
> >> 
> >> <snip>
> >> 
> >> _______________________________________________
> >> IPoverIB mailing list
> >> [email-masked]
> >> http://www.ietf.org/mailman/listinfo/ipoverib
> >> 
> >
> >
> >-- 
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >[email-masked] 
> >[email-masked] 
> >503 578 3422 (o)
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email-masked]
> >http://www.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/ipoverib
> 

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Strahm" <[email-masked]>
Date: Wed, 16 Jan 2002 20:06:06 -0800

Just free form thinking here...

Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
(www.strahm.net for example) and LIDs are the equivalent of IP addresses
(63.105.18.42).  Now in many cases IP addresses are quite stable, and do not
change.  However would you ship an application with an IP address encoded in
it, or with a FQDN.  It would be of course more efficient to use the IP
address and avoid the DNS lookup (that has to go VERY far away, and can take
quite a while for the first SYN message to get out), however if the address
EVER changes, things become hard.

Can anyone tell me why this is a bad analogy ???

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Larsen, Roy K
Sent: Wednesday, January 16, 2002 6:43 PM
To: 'H.K. Jerry Chu'; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Roy

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 17 Jan 2002 01:27:10 -0800 (PST)

> Can anyone tell me why this is a bad analogy ???

        If it was pure IB then GID to LID would be a similar analogy.

        I'll try taking the your analogy further..a couple of points
        though - the SM corresonds to the zone primary but the nodes
        themselves corresond to both the secondary and the IP nodes
        asking for the resolution. So the analogy is not exactly apt
        but I'll try..

        If the IP changes the DNS must be updated and looked up. In
        this case if LID changes the SM is updated (yes) but it then
        requires that the SM be looked up. There is no backlink here
        to trigger this lookup. This implies that the full lookup all
        the way to the DNS root must be done - if like DNS it is done
        till the local secondary server ie. referring to the local
        cached copy and/or merely re-receiving the peer's GID, the
        communication will fail. You must have seen the hell that
        breaks loose if any of the zone updates doesn't happen due to
        some error and things are left pointing the wrong way.

        Now, this would correspond to the internal LID cache flush -
        since the GID is the same but the LID has changed. There is no
        hook to say, hey!, go do a zone transfer i.e. lookup the SM
        right now. You need a timer.

        Now, you could make this time small - you are going to the SM
        too often because things are not changing that much- make it
        long and LID change is surely going to happen at a critical
        time. Additionally this is another timer per entry for the
        programmers/maintainers/administrators to worry about.

        I have all along been arguing against keeping LID out due to
        this reason too (code is more complex if no LID is another
        reason). It has been my assumption that if GID is used without
        LID then one is forced to go to the SM at every address
        resolution. Every arp broadcast will have the whole subnet
        scurrying off to the SM to ask the requester's LID. Yes, some
        would mark the entry to be completed but not necessarily - a
        thundering herd is likely.

        This implies that if we don't use the LID we MUST ALWAYS go to
        the SM for the LID - at every ARP since we don't know if LID
        has changed. Of course if the GID changed too then the SM has
        to be consulted anyway.

        Also note that if we did have a trap from the SM on every LID
        change it still wouldn't be a good solution - too much
        latency; points of synchrnonisation; asynchronous updates; and
        most of all the SM would end up doing nothing else but issueing
        traps to every node that registered with it. IB cannot mandate
        broadcasts/multicast (though making everyone look is bad too)
        because multicasting is optional.

        In the case of inclusion of the LID - some of the issues are
        the same but a gratuitous ARP solves the problem if the LID
        changed. It is an instant zone transfer. One could claim that
        with GID only, on receiving any gratuitous ARP entry, the node
        will go to the SM but that will require specialised coding to
        recognise and handle gratuitous ARP differently than regular
        updates. Not a good idea.

        Even in the absence of gratuitous ARP the changed LID will be
        known when the ARP is done after a cache expiry if LID is
        included. If LID is not included one will get a GID back. That
        will require the expiry of the GID to LID cache before one
        goes to the SM.

        Inclusion of LID makes things very simple. This is because LID
        is always necessary - can't escape it.

        I also like inclusion of LID because it cuts off 2x latency of
        GID to LID lookup in the SM and 4x the RTT and the realted
        extra interrupts. Fast failover in data centers is necessary.

Vivek

---

From: "Bill Strahm" <[email-masked]>
Date: Thu, 17 Jan 2002 09:18:05 -0800

Not sure I agree with a few things here...

The SM IS the ROOT DNS server, because it is the ultimate authority on what
is where in the fabric.  Is there any doubt to that.  Also the end nodes
have nothing to do with resolution, I do not believe there is any IB
mechanism to go to the end node and ask it for a Path Record is there ?

You are correct that if the LID changes there is no back path to tell
everyone that the LID has changed, this is just like DNS so the analogy
holds still.  Since the local SM is the DNS Root going "All the way to the
root" is not a problem.

You need a timer just like in DNS, so the analogy still holds.  The timer
works at the same level as the ARP timer.  When you ARP, you go get the
answer from the SM... Until then you can deal with LID changes because
either a) your communication fails - you have this problem in either
scenario b) you notice a LID change on an incoming packet so update your ARP
table

No, only the responder needs to reply to an ARP broadcast, so therefor only
one entry.  Now other nodes in the network may decide to go to the SM to
cache that particular entry as well, but then you remove latencies when you
want to talk to that node anyway... I would assume that any decent SM could
handle the traffic from 65,000 nodes asking it to do a similar lookup at the
same time...  This of course is a worst case scenario.

The problem I have is that unless you have manually configured the LID-SL
mapping for your subnet, you could deadlock the fabric this way.  Now would
you want to manually configure LID-SL lookup for those same 65000 nodes ???

Bill

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 1:27 AM
To: Bill Strahm
Cc: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 17 Jan 2002 10:59:27 -0800 (PST)

> scenario b) you notice a LID change on an incoming packet so update your ARP

In the case of only GID you don't know if the LID changed. You have to
have your secondary GID to LID timer go off too.

> same time...  This of course is a worst case scenario.

You are right that the responder needs to respond to the broadcast but
all receivers as per the ARP algorithm update their entries. That
means all rush off to the SA.

> you want to manually configure LID-SL lookup for those same 65000 nodes ???

Yes. That is a very valid point.  you have made it better than I did. 
The SL lookup, management etc. and the modes of determining it are
upto the implementation. Also note that the high spike in traffic in
itself is undesirable even if just getting the LID from the SA for the
GID just advertised.

Vivek

---

From: "Roger Ronald" <[email-masked]>
Date: Thu, 17 Jan 2002 14:13:56 -0600

I think it's important to note that changes in LID should be infrequent.
Very infrequent.

VIEO is one of a small number of companies that has a SM at this time.
We plan to maintain LID assignments whenever possible. That means
across reboots, across power fails and across subnet merges (when possible).

There are corner cases where LID may need to change, just like IP
addresses occasionaly need to change. But these are VERY odd
events...like the first time merge of fabrics. They won't happen
hourly, daily, monthly..... They may happen once or twice in a 
fabric. Ever (unless the admin wants to make changes for the heck of it).

RR

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 12:48:19 -0800

>fabric. Ever (unless the admin wants to make changes for the heck of it).

While the LID range for a given port may not change often, the LID / SL 
appropriate for a given path between two endnodes may change for a variety 
of reasons based on workload objectives / QoS requirements within the 
fabric for a given service.  The IB APM mechanisms may be used to update 
these path selections without informing IP of the change (an implementation 
option).  Thus, the rate of LID change may not be as low as you contend but 
it will also not be as high as others contend making the performance issues 
raised perhaps moot.

Mike

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Roger Ronald" <[email-masked]>
Date: Wed, 23 Jan 2002 07:47:29 -0600

I concur with your analysis. While LID assignments will change
infrequently, the proper LID/SL may change more often due
to APM.

RR

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 23 Jan 2002 14:20:46 -0800 (PST)

Is this a valid conclusion in IPoIB-UD context since APM is not
supported for Unreliable Datagram mode of IB?????

Vivek

---

From: "Roger Ronald" <[email-masked]>
Date: Wed, 23 Jan 2002 16:40:55 -0600

Doh! That's what I get for trying to be agreeable. The statements below
are true for the general case, but you are correct that APM is
not relevant for UD. 

So, I'll revert to the claim that LID changes
should be truely unusual events, at least with the SM/SA 
behavior that my company expects to provide.

RR