Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

From: "Arvind Murching" <[email-masked]>
Date: Sun, 13 Jan 2002 11:04:48 -0800

IMO, this proposal meshes well with both the IBTA spec (regarding
end-to-end communication between nodes) as well as with IP stack
implementations.

Nodes may use SIDR to resolve "more than the IP address" with partner
nodes - this is what SIDR is good for. For interop reasons, this RFC
should be either silent (preferable in the interests of completing this
RFC) regarding use of SIDR for additional resolution, or specify SIDR
usage completely. It is not a good idea to mention an optional feature
without describing it completely.

Also for interoperability reasons, this RFC should always require a
"default" QPN to be returned by a responding node. As an example,
consider gratuitous ARP packets - can these contain a 0 QPN?

Thanks
Arvind

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]] 
Sent: Saturday, January 12, 2002 9:28 PM
To: [email-masked]
Subject: [Ipoverib] What's inside an IPoIB link layer address?

While proposals beyond the basic IPoIB over UD started showing up, I
still don't think we've reached a rough consensus on some basic
questions such as what the IPoIB link layer address should contain.
Again I'd like to apologize for not being there at the SLC meeting last
month. Perhaps a consensus was reached there but i've missed the boat.
In that case please let me know what it is and I'll shut up.

What I'd like to see included in the IPoIB link layer address is

GID
QPN

and I'd like to allow QPN field to be 0.

What I don't like to see included is LID.

Discussions:

There has been a consensus of using GID, not LID as the basic IB address
identifying the port hosting an IP address. GID is sufficiently stable,
not subject to change across a node reboot or subnet merge (IB level)
like LID is. Also people don't want to exclude the possibility of IPoIB
link spanning more than one IB subnet. GID supports scope beyond local
link so it's suitable for that purpose.

Multiple GIDs with different scopes may have been assigned to a CA. It
is required that a CA returns the GID with a scope no less than the one
for the IPoIB link. E.g. if an IPoIB link covers multiple IB subnets,
but a CA uses a link-local GID as part of its IPoIB link layer address,
it's not going to work well.

I don't see any value of including LID besides GID as proposed by Vivek.
I believes a lookup of PathRecord is always needed in order for a node
to retrieve complete path info before sending unicast pkts to another
node. The destination LID is part of the PathRecord so including it in
the link layer address returned from ARP/ND doesn't save work. But you
have all the drawbacks that swayed people away from using LID - it adds
volatility to the link address. LID changing due to subnet merge or
other reasons requires potentially new mechanism being added to
IP/driver implementation to trigger ARP/ND flush. Lastly, what should be
the LID if the IP node is in a different IB subnet? Obviously it can't
be the LID of an intermediate IB router.

For QPN I'd like to suggest one thing. Since SIDR is a required feature
by IBA, the QPN field is allowed to be zero. What that means is that the
requester must use SIDR to retrieve the remote QPN, given the GID of a
remote IP address.

This feature allows the support of static/persistent "MAC addr" wrt node
reboot by removing the volatile part of the link layer address - the
QPN. In many implementations the QPN allocated for a CA may change
across node reboot. Support of static "MAC addr" has been considered
desirable by a number of people.

I don't like to add additional requirement such as SIDR to IPoIB, and
was trying to make the above feature "optional". Ideally nodes can
express whether they'd like QPN to be returned by embedding some flags,
such as "I don't talk SIDR so please return QPN" in their IPoIB link
layer addresses. But the broadcast nature of IPv4/ARP forbids the use of
different "MAC addrs" for the same IP address, so my attempt failed.

Comment?

Jerry

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Tue, 15 Jan 2002 12:26:48 -0800 (PST)

Please see comments below.

<snip>

>Vivek,
>
>I agree with your assumptions
>We need them all: LID + capability flags + GID + QPN
>Mikes remark that you could extract the LID from the incoming packet however
>looks interesting (as long as we make sure that people don't play with it,
>e.g. redirect traffic etc')
>
>As far as I understand the only open issues currently are (if someone thinks
>there are more please respond):
>LID: Yes/No
>Ethertype
>Capability flags
>
>As for the why LID:
>I think that going to the SM is not so efficient given the fact that it is a
>central resource and all the nodes in the subnet go to if we could avoid it
>why not

First the benefit is dubious. In modern TCP/IP stacks the frequency of ARP
requests is capped by the ARP cache flush internal, which is in minutes
range. (E.g. in Solaris it's five minutes.) Assuming you have 100 nodes
on a subnet, you're telling me saving the SM lookup call, which at max
rate (if a local node is activetly talking to every other node) of one
every 3 sec. is worth noting?

I'm still not sure if a trip to SM can be saved at all (need to catch
up with all the mail this morning). Even if in some cases it could, do
we believe that not doing any pathrecord lookup is going to be the norm 
of IPoIB implementation that warrents an extr field in the link addr?
How about all the issues I've pointed out before? I have another one
to add - simplicility. Unless there is sufficient justification, the more
fields you add to a protocol the more complexity (in areas you may not
even forsee now) it's going to add.

>When IP will span IB the problem becomes more serious: Unlike IP routers
>which has a default route entry, currently IB has explicit entries for each
>path this might require doing too many PathRecord operations, which are also
>currently very slow when going outside an IB subnet
>(I had posted a proposal to the MSO team trying to resolve those issues
>somehow however it is under discussion)
>I would tend to send this info anyway since the extra 16 bits wouldn't harm
>anyone,

But it does. What would you define the LID for a node on a non-local IB
subnet? It can't be a LID to a router because this is an endnode's "MAC
addr" we're defining here.

>one might choose not to use them if he wants to go to the SM, this
>way everybody will be happy and over time people will use the best solution

No we can't do handwaving here. You either define a field with clear
semantics so implementations have a chance to interoperate, or take it
out. There is no iffy here (unless you add a "LID included" flag to the
link addr).

Jerry

>
>As for the Ethertype:
>After giving Vivek a hard time on this issue I accept his proposal
>Not because of the performance issues raised (since I don't think they are
>real)
>But because it is not really needed, and if it simplifies things, why not
>
>As for the capability bits:
>Avoiding the SIDR is a smart move
>I'm not sure if the UC option will be used, if using connection why not
>reliable connection
>I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
>choose not to use it, for the one that do it really simplifies the process a
>lot
>
>I'm hoping we get a quick convergence on those issues so we would move
>forward
>The issues are not serious and I think if an alternative enable someone to
>do something he wants without violating the other's ability to do what they
>want just ignoring the extra info, we should go for it
>
>Yaron
>
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Vivek Kashyap
>Sent: Tuesday, January 15, 2002 12:46 AM
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>
>Folks,
>
>       It is close to the last call. In this regard I would like to
>       state that we have been discussing this issue for a long time
>       now. The most recent discussion is what I presented at the SLC
>       last december. (I certainly felt that the room was in agreement
>       - no queries, objection etc.)
>
>       I believe the following link-description provides the best
>       assimilation of all the choices that have been put forth. If we
>       step back and restart the whole discussion again we are likely
>       to come back to the following - so here is a brief compilation
>       of issues.
>
>
>        Suggested Link description to be used in address resolution:
>
>                LID + capability flags + GID + QPN
>
>
>
>       Why the LID?
>
>             People said they don't want to go to the SM if possible
>             (LID allows for that. As noted here and LWG the SL can be
>             determined based on QoS or other parameters. The SL may
>             also be determined from the SA/SM)
>
>             People said they want the process to end in small number
>             of messages (LID allows a query/response to settle the
>             issue. As noted one doesn't necessarily have to go to the
>             SM to get the SL).
>
>             LID is always needed. (you can't escape it).
>
>             BTW, people said they want the 'link-address' to be 48 bits.
>             (LID + QPN + cap_flags is 48 bits).
>
>
>       Why the GID ?
>
>                People want IP to span across IB subnets. We are
>                stretching stretching here since such implementations
>                are not possible yet.
>                However, we can define it such that it will work in
>                the future. Therefore include the GID.
>
>        GUID
>
>                Can be made part of the GID.
>
>
>        Why the QPN:
>
>                1) There is only one other way to get the QPN. That is
>                the SIDR protocol. However, this has a drawback.
>
>                SIDR is NOT REQUIRED by IBTA.
>
>                I had raised this point last August 'Feasibility of
>                SIDR':
>
>                ***** Section 12.11 essentially states that SIDR is
>                optional. As the specification states:
>
>               << GSAs shall support this protocol if non-management
>               services are pro-vided on the Channel Adapter at other
>               than fixed QPNs. If this protocol is *not supported*,
>               the status "The method/attribute combination is not
>               sup-ported" shall be returned, as described in section
>               Table 101 MAD Common Status Field Bit Values >>
>
>                *****
>
>                That still holds. SIDR is NOT required by IBTA. A
>                device can always choose to provide all non-management
>                services at fixed QPNs. It may just choose to provide
>                IP and nothing else at a particular QP and hence won't
>                do SIDR.
>
>                That is why I propose that the QPN be always included
>                for the default case. SIDR usage is gated through the
>                QPN flag and is in the realm of the advanced
>                capabilities draft as has been suggested by many on
>                this mailing list. And I agree.
>
>
>                2) Address resolution that returns the QPN is done in
>                one request/reply exchange. Any other method, while
>                talking to the same endpoint, requires 6 messages. 2
>                for ARP/ND and 2 each for each of the lower layer
>                request/reply. IT is wasteful, more complicated,
>                more time consuming wrt fast failover etc.
>
>
>        Capability flags :
>
>                Multiple people have opined that they would prefer in
>                certain cases to map the protocol/ULP endpoints to
>                specific QPs. People also want to use IP over RC mode
>                (and others maybe) due to the advantages of larger MTU
>                and Automatic path failover etc.
>
>                The capability flags allow for interoperation between
>                the default/dumb devices and the more sophisticated
>                devices. IT also allows the endpoints to avoid
>                unnecessary trial and error/to and fro when trying to
>                utilise these features. The flag indicates the
>                advanced capabilities supported and the receiver can
>                choose to utilies them.
>
>
>Result:
>
>        We cannot depend on SIDR. We need to include the default QPN.
>        We should not force implementations to go to the SA/SM (IBTA
>        does not) and so allow LID/GID to be determined from the
>        remote host. The default case works with :
>
>        LID+Capability flags(all set to 0)+QPN+GID
>
>
>Vivek
>
>
>
>

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 16 Jan 2002 18:13:01 -0800 (PST)

All the proxy ARP usages that I know of use the proxy's mac addr in the
ARP response. I believe the flavor of the proxy ARP you alluded below is
rare, and won't work very well anyway if the link address includes any
non-persistent component like LID and QPN because it will require another
protocol between the clients and their proxy to constantly update the
proxy database containing clients mac addrs, which may change quite
often.

That is also why I wanted to provide an option to not include QPN.
(But I haven't succeeded w/o requiring SIDR or something equivalent.)

If LID is included as part of link addr and if it ever changes on
a live node due to subnet merge...etc, the driver must inform its IP
stack about such change. I believe many stacks don't have such a
facility. (FOr most IP stacks the assumption is that its own MAC
addresses just don't change on the fly.) In that case you're totally
hosed. Can someone comment on how likely a LID assignment gets
rescinded?

Jerry

>        A device can always respond for another. The LID of the sender
>        is not the same as the LID for which it has responded. It also
>        allows any network monitoring/debugging device to work
>        independently. This is the reason why none of the other media
>        does anything like this. You don't pick ethernet source
>        address from the frame but take it from the ARP/ND packet.
>
>        [ If it was ok we wouldn't need anything except QPN/cap_flags
>        - get the LID from response, consult SA using LID for GID if
>        you want that, and you are done. ]
>
>Vivek
>
>> 
>> Why do we need to include the LID inside the link layer address when it
>> is already available in the SLID field? Previous discussions indicate
>> that the SLID value works for proxy ARP/bridge implementations.
>> 
>> - Arvind
>> 
>> From: Vivek Kashyap [mailto:[email-masked]] 
>> Sent: Wednesday, January 16, 2002 10:29 AM
>> To: [email-masked]
>> Cc: [email-masked]
>> Ashwin,
>> Ashwin,
>>         I agree that you get more flexibility if you include the LID.
>>         If we include the GID along with the LID we allow all
>>         implementations the choice to create different solutions. We
>>         don't tie them down to one thing. Choosing one or the other
>>         (LID or GID) blocks out various advantages that different
>>         implementations seek. Hence I suggested that let us include
>>         both and move on.
>> 
>> Vivek
>> 
>> <snip>
>> 
>-- 
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>

---

From: "Arvind Murching" <[email-masked]>
Date: Wed, 16 Jan 2002 19:04:48 -0800

When you have an IP subnet spanning IB subnets, the LID of a responding
node is of no use.

Consider filling in static ARP entries (IP -> Link layer address
mappings kept in persistent store) - LID fields in such entries are of
no use.

Implementations may find creative ways to work/hack around these
deficiencies, but why should we design a protocol that requires these
workarounds?

IMO Jerry's suggestion of only using the GID is the most "idealistic"
solution if not for a dependency on SIDR.

- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]] 
Sent: Wednesday, January 16, 2002 5:42 PM
To: 'Vivek Kashyap'; Arvind Murching
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Arvind, Vivek

I'm sure we must have the GID as the primary address
There is a need for IP spanning IB this is what I hear from many of the
data
centers I visit, As noted in few of the emails there are also few other
reasons for GID
lets first agree on that !,
Vivek the key for including the LID is that people will have a primary
(GID)
address that determine the route, to avoid interoperability issues
(I'm sure this was also the consensus in LWG befor we went to IETF)

We need to maintain the tables by the GID, however I support taking the
LID
to return the answer to the caller (avoiding the SM as the main reason)
I'm not sure that we could take the LID from the packet since someone
might
Forward it somehow (test tools, ARP server or something like that)
And we don't want a broken spec

No one mentioned in the arguments the additional latency caused by going
to
The SM (for the LID)
As a central resource that probably has queuing it might take time
Not to mention the latency in IP span IB case (needing to resolve all
the
Path first)

lets go for the LID in the header, and specify that the GID is the
primary
address

Yaron

[Arvind] <snip>

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 16 Jan 2002 19:27:36 -0800 (PST)

Roy,

Thanks for the info. A comment on your statement of volatility of LID having
no bearing on whether LID should be included in ARP reply. Yes it does. One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The difference
is whether one must modify the existing stack code to add a facility to support
mac addr change, or do the necessary chore in a complete new IPoIB driver.
The former requires an upgrade of OS release that can take a long time. The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable than
gratuitous ARP. But I don't know the SM archtecture enough to make such a claim.

To be fair I don't really know how many OSes lack support for mac add
change. The two that I know, one supports it, one doesn't.

I'm assuming there are solid facility in IBA to support the notification of
attribute changes to interested parties. Otherwise not only we should include
LID, but also all the rest in the link addr.

Jerry

>From: "Larsen, Roy K" <[email-masked]>
>To: "'H.K. Jerry Chu'" <[email-masked]>, [email-masked], [email-masked]
>Cc: [email-masked]
>Date: Wed, 16 Jan 2002 18:42:31 -0800
>MIME-Version: 1.0
>
>> Sent: Wednesday, January 16, 2002 6:13 PM
>> To: [email-masked]; [email-masked]
>> Cc: [email-masked]
>It is possible anytime the link goes down and comes back up again and when
>the fabic is reconfigured.  This can happen on reboot or simply
>unplugging/plugging in a link.  It is ultimately a function of the SM and
>how much trouble the implementation wants to go through to keep LIDs
>constant.
>
>But of course, that has no bearing on whether the LID is contained in the
>ARP.  You always need the LID.  Unless you want to do a pathrecord lookup
>each time your implementation is given a packet to send, you're going to use
>a cached LID no matter if it came in the ARP or a pathrecord lookup on the
>GID.  If it becomes stale, it may take an ARP cache flush timeout to
>recover.  
>
>Roy
> 
>> 
>> Jerry
>>

---

From: "Larsen, Roy K" <[email-masked]>
Date: Thu, 17 Jan 2002 09:56:37 -0800

Bill's last paragraph gets to the crux of the issue on pathrecord looks for
ARP.  There seems to be a camp that will deploy their solution in a
controlled environment where they can dictate the fabric topology and it's
configuration.  A cluster product would fit this model.  There is another
camp that needs to supply a generalized IPoverIB solution that will work in
any fabric topology and configuration such as a very large data center with
a mixture of vendor products.

I believe both of these models are valid.  For the fixed cluster type of
environment, I believe it is reasonable to add things like the LID to the
ARP if that will allow foregoing the pathrecord lookup.  I say this with the
proviso that there are no other constraints on ARP contents and semantics
that would prevent those who simply want to use the GID to consult the SM
for correct address vector information.

If we can agree that these models are equally valid, perhaps we can spend
less bandwidth trying to sway others to a model that will not meet their
market requirements and more on finalizing our work.

Roy

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 9:18 AM
To: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Not sure I agree with a few things here...

The SM IS the ROOT DNS server, because it is the ultimate authority on what
is where in the fabric.  Is there any doubt to that.  Also the end nodes
have nothing to do with resolution, I do not believe there is any IB
mechanism to go to the end node and ask it for a Path Record is there ?

You are correct that if the LID changes there is no back path to tell
everyone that the LID has changed, this is just like DNS so the analogy
holds still.  Since the local SM is the DNS Root going "All the way to the
root" is not a problem.

You need a timer just like in DNS, so the analogy still holds.  The timer
works at the same level as the ARP timer.  When you ARP, you go get the
answer from the SM... Until then you can deal with LID changes because
either a) your communication fails - you have this problem in either
scenario b) you notice a LID change on an incoming packet so update your ARP
table

No, only the responder needs to reply to an ARP broadcast, so therefor only
one entry.  Now other nodes in the network may decide to go to the SM to
cache that particular entry as well, but then you remove latencies when you
want to talk to that node anyway... I would assume that any decent SM could
handle the traffic from 65,000 nodes asking it to do a similar lookup at the
same time...  This of course is a worst case scenario.

The problem I have is that unless you have manually configured the LID-SL
mapping for your subnet, you could deadlock the fabric this way.  Now would
you want to manually configure LID-SL lookup for those same 65000 nodes ???

Bill

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 1:27 AM
To: Bill Strahm
Cc: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

>
> Just free form thinking here...
>
> Could we say that GIDs are the equivalent of a Fully Qualified Domain Name
> (www.strahm.net for example) and LIDs are the equivalent of IP addresses
> (63.105.18.42).  Now in many cases IP addresses are quite stable, and do
not
> change.  However would you ship an application with an IP address encoded
in
> it, or with a FQDN.  It would be of course more efficient to use the IP
> address and avoid the DNS lookup (that has to go VERY far away, and can
take
> quite a while for the first SYN message to get out), however if the
address
> EVER changes, things become hard.
>
> Can anyone tell me why this is a bad analogy ???

        If it was pure IB then GID to LID would be a similar analogy.

        I'll try taking the your analogy further..a couple of points
        though - the SM corresonds to the zone primary but the nodes
        themselves corresond to both the secondary and the IP nodes
        asking for the resolution. So the analogy is not exactly apt
        but I'll try..

        If the IP changes the DNS must be updated and looked up. In
        this case if LID changes the SM is updated (yes) but it then
        requires that the SM be looked up. There is no backlink here
        to trigger this lookup. This implies that the full lookup all
        the way to the DNS root must be done - if like DNS it is done
        till the local secondary server ie. referring to the local
        cached copy and/or merely re-receiving the peer's GID, the
        communication will fail. You must have seen the hell that
        breaks loose if any of the zone updates doesn't happen due to
        some error and things are left pointing the wrong way.

        Now, this would correspond to the internal LID cache flush -
        since the GID is the same but the LID has changed. There is no
        hook to say, hey!, go do a zone transfer i.e. lookup the SM
        right now. You need a timer.

        Now, you could make this time small - you are going to the SM
        too often because things are not changing that much- make it
        long and LID change is surely going to happen at a critical
        time. Additionally this is another timer per entry for the
        programmers/maintainers/administrators to worry about.

        I have all along been arguing against keeping LID out due to
        this reason too (code is more complex if no LID is another
        reason). It has been my assumption that if GID is used without
        LID then one is forced to go to the SM at every address
        resolution. Every arp broadcast will have the whole subnet
        scurrying off to the SM to ask the requester's LID. Yes, some
        would mark the entry to be completed but not necessarily - a
        thundering herd is likely.

        This implies that if we don't use the LID we MUST ALWAYS go to
        the SM for the LID - at every ARP since we don't know if LID
        has changed. Of course if the GID changed too then the SM has
        to be consulted anyway.

        Also note that if we did have a trap from the SM on every LID
        change it still wouldn't be a good solution - too much
        latency; points of synchrnonisation; asynchronous updates; and
        most of all the SM would end up doing nothing else but issueing
        traps to every node that registered with it. IB cannot mandate
        broadcasts/multicast (though making everyone look is bad too)
        because multicasting is optional.

        In the case of inclusion of the LID - some of the issues are
        the same but a gratuitous ARP solves the problem if the LID
        changed. It is an instant zone transfer. One could claim that
        with GID only, on receiving any gratuitous ARP entry, the node
        will go to the SM but that will require specialised coding to
        recognise and handle gratuitous ARP differently than regular
        updates. Not a good idea.

        Even in the absence of gratuitous ARP the changed LID will be
        known when the ARP is done after a cache expiry if LID is
        included. If LID is not included one will get a GID back. That
        will require the expiry of the GID to LID cache before one
        goes to the SM.

        Inclusion of LID makes things very simple. This is because LID
        is always necessary - can't escape it.

        I also like inclusion of LID because it cuts off 2x latency of
        GID to LID lookup in the SM and 4x the RTT and the realted
        extra interrupts. Fast failover in data centers is necessary.

Vivek

> Bill
> From: [email-masked] [mailto:[email-masked]]On Behalf
> Of Larsen, Roy K
> Sent: Wednesday, January 16, 2002 6:43 PM
> To: 'H.K. Jerry Chu'; [email-masked]; [email-masked]
> Cc: [email-masked]

---

From: "Larsen, Roy K" <[email-masked]>
Date: Thu, 17 Jan 2002 11:42:20 -0800

Jerry,

Ok, your concern is a mechanism for notifying a node that a LID change has
occurred regardless of how the LID was initially obtained.  I'm no SM expert
but seem to recall asking my resident expert about this a while back and he
said there wasn't.  However, if the LID was included in the ARP, wouldn't
sending a gratuitous ARP when the link comes up solve the problem? 

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Wednesday, January 16, 2002 7:28 PM
To: [email-masked]; [email-masked];
[email-masked]
Cc: [email-masked]

Roy,

Yes no matter what, LID is cached somewhere and must be flushed. The
difference
is whether one must modify the existing stack code to add a facility to
support
mac addr change, or do the necessary chore in a complete new IPoIB driver.
The former requires an upgrade of OS release that can take a long time. The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable
than
gratuitous ARP. But I don't know the SM archtecture enough to make such a
claim.

I'm assuming there are solid facility in IBA to support the notification of
attribute changes to interested parties. Otherwise not only we should
include
LID, but also all the rest in the link addr.

Jerry

>To: "'H.K. Jerry Chu'" <[email-masked]>,
[email-masked], [email-masked]

---

From: "Arvind Murching" <[email-masked]>
Date: Thu, 17 Jan 2002 12:16:58 -0800

I'm not an IB expert, but I would think that -if- we decided to use
gratuitious ARP to notify remote nodes of changes, then those remote
nodes would consult with the SA and re-create their address handles
using the indicated GID.

"Optimized/simplified" implementations would pick up new SLIDs from the
headers of these gratuitous ARP packets.

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]] 
Sent: Thursday, January 17, 2002 11:42 AM
To: 'H.K. Jerry Chu'; Arvind Murching; [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry,

Ok, your concern is a mechanism for notifying a node that a LID change
has occurred regardless of how the LID was initially obtained.  I'm no
SM expert but seem to recall asking my resident expert about this a
while back and he said there wasn't.  However, if the LID was included
in the ARP, wouldn't sending a gratuitous ARP when the link comes up
solve the problem? 

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Wednesday, January 16, 2002 7:28 PM
To: [email-masked]; [email-masked];
[email-masked]
Cc: [email-masked]

Roy,

Thanks for the info. A comment on your statement of volatility of LID
having
no bearing on whether LID should be included in ARP reply. Yes it does.
One
of the goal of IPoIB is to minimize the changes to the existing IP stack
implementations. Note that I stated it as a goal, not a constraint. If
we
must do it then we should.

Yes no matter what, LID is cached somewhere and must be flushed. The
difference
is whether one must modify the existing stack code to add a facility to
support
mac addr change, or do the necessary chore in a complete new IPoIB
driver.
The former requires an upgrade of OS release that can take a long time.
The
latter doesn't have such a problem since it's brand new.

One can also hope that LID flushing through SM can be made more reliable
than
gratuitous ARP. But I don't know the SM archtecture enough to make such
a claim.

I'm assuming there are solid facility in IBA to support the notification
of
attribute changes to interested parties. Otherwise not only we should
include
LID, but also all the rest in the link addr.

Jerry

---

From: "Arvind Murching" <[email-masked]>
Date: Thu, 17 Jan 2002 12:22:16 -0800

Also note that you can't send your LID to a node on a different IB
subnet anyway.

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]] 
Sent: Thursday, January 17, 2002 11:42 AM
To: 'H.K. Jerry Chu'; Arvind Murching; [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry,

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Wednesday, January 16, 2002 7:28 PM
To: [email-masked]; [email-masked];
[email-masked]
Cc: [email-masked]

Roy,

Jerry

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 17 Jan 2002 12:49:09 -0800 (PST)

Roy,

Sorry I didn't explain it well. The potential issue I was trying to
point out is inside a local node that has its LID change. The question
is how will the LID change be propergated up to the IP stack. Many IP 
implementations never expect a local NIC's  MAC address can change on
the fly, all by itself. This poses a potential implementation problem.

Jerry

>Date: Thu, 17 Jan 2002 11:42:20 -0800
>MIME-Version: 1.0
>
>Jerry,
>

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 17 Jan 2002 13:25:44 -0800 (PST)

>Jerry,
>
>Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
>is integral to IBA, they are just going to have to find a way to deal with

And I've been saying this is an IBA issue in general (not just for IPoIB).
So IBA should find a solution for it.

Including the LID in the link addr forces this issue into the IP stack.

>it.   As I suggested, perhaps with gratuitous ARP.

No, forget about gratuitous ARP. The issue I brought up is purely a
issue within the local node that has its own LID changed.

>Anyway, would you agree
>it is outside the scope of IPoIB.

It's an iffy (IMHO). As I've pointed out before, I believe there is a
concensus that IPoIB should fit into the existing IP stack implementations
But that is a "goal", not a constraint. Should we make an exception for
LID?

Not including LID in link addr punts the changing LID problem back to
IB, which I think is the right place for it.

Jerry

>Sent: Thursday, January 17, 2002 12:49 PM
>To: [email-masked]
>Cc: [email-masked]

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Vivek Kashyap" <[email-masked]>
Date: Thu, 17 Jan 2002 14:43:18 -0800

Jerry,

I believe Roy has a point. We are dealing with a new technology and need to
work with its features. If the MAC changes the OS has to find ways to deal
with it.

Are you saying that the MAC will change, the OS will not know of it, will
return the GID only and the receiver must therefore go to the SM/SA to get
the LID ? That seems all wrong. If the OS doesn't know what the new LID is
it cannot send the packet at all. An implementation not recognising this
will fail - not just IPoIB but any IB application. The OS knows the LID
changed and all it does is update the necessary table.

BTW,  Roger  has pointed out that the LIDs will hardly ever change so the
concern regarding LID in this case is unfounded. On the contrary the GID
used may change more often based on the proposals seen on this mailing list
- a) use GIDs based on scope b) use non-GUID based GIDs for various reasons
etc. c)  One might also use the chassis' GID or the GID of a group etc. As
far as I can tell a port can have more than one GUID too.  These are more
variable. Any change here is more likely. Since the LID is uniquely tied to
a port but not the GID, return it in the ARP packet and the resolution is
complete.

 LID is the real address uniquely assigned to a port. Not the GID.  You
don't need a GID to talk to another port. You need the LID only. GID is
just an optional item.

[ Now coming to the gratuitous ARP case. If the LID had changed but not the
GID - a gratuitous ARP will succeed only if the receiver actually went to
the SA. This implies that one has to go to the SA on every ARP request one
gets not just when the cache expires. Otherwise such LID changes would
never get through to the peer in time. ]

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      "H.K. Jerry Chu"                                                                                            
                      <[email-masked]        To:       [email-masked]                                                   
                      n.com>                   cc:       [email-masked]                                                        
                      Sent by:                 Subject:  RE: [Ipoverib] What's inside an IPoIB link layer address?                
                      ipoverib-admin@ie                                                                                           
                      tf.org                                                                                                      

                      01/17/2002 01:25                                                                                            
                      PM                                                                                                          
                      Please respond to                                                                                           
                      "H.K. Jerry Chu"                                                                                            

Jerry

>>Ok, your concern is a mechanism for notifying a node that a LID change
has
One
driver.
The
of

---

From: "Larsen, Roy K" <[email-masked]>
Date: Thu, 17 Jan 2002 15:09:22 -0800

Bill and Jerry,

I think I see were we are misunderstand each other.  It is my assumption
that mapping IP address to MAC address from within the OS will be
implementation specific.  That is, what my IPoIB ARP module and link driver
registers as a MAC address to the OS is my private concern.

For example, in Linux, there is a 8 bytes limit to the hardware address.
Since I don't want to be in the OS distribution business, my driver must
live with that.  Needless to say, an IBA address vector aint going to fit in
8 bytes.  All that is required is for me to place something in that field
that will lead me to an internal structure that contains what I need.  It is
at that point I need to make sure I am using a current LID.

The point is, I am not going to be using the LID in registering hardware
address to the OS.  My own or those of remote nodes.  So, it doesn't matter
to the OS if they change.  It will probably be the GUID but that is an
implementation choice for my driver.  Since I need to supply an IPoIB ARP
module, it is easy to inspect all source LIDs received to see if they still
match what I have in my private address vector.  If not, I update it.  The
OS was never involved.

Roy 

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 1:41 PM
To: Larsen, Roy K; 'H.K. Jerry Chu'
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Roy,

Gratuitous ARP does not help here... I do I send a gratuitous ARP if I can
not change my MAC address in my ARP module...

The easiest solution is to not put LID into the MAC address, then you don't
have to worry about the problem at all on the sender side.  On the receiver
side I believe you have to do the same work regardless...

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Larsen, Roy K
Sent: Thursday, January 17, 2002 1:00 PM
To: 'H.K. Jerry Chu'
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry,

Yes, this could be a issue for the OS.  But since the dynamic nature of LIDs
is integral to IBA, they are just going to have to find a way to deal with
it.   As I suggested, perhaps with gratuitous ARP.  Anyway, would you agree
it is outside the scope of IPoIB.

Roy

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 12:49 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Roy,

Jerry

>To: "'H.K. Jerry Chu'" <[email-masked]>,
[email-masked],
[email-masked]
having

---

From: "Jim Pinkerton" <[email-masked]>
Date: Thu, 17 Jan 2002 17:12:20 -0800

I think it is important to note that the SLID can be retrieved on
receipt of a UD message. 

Please see IB spec 1.0.a, section 11.4.2.1, for the arguments returned
on the "poll for completion" verb. They include (from the list of output
modifiers starting on page 533) a "remote node address and QP". Per
Table 85, for unreliable datagram, the addressing information includes
the SLID, SL, SQP, and DLID path bits.

I'm not a proponent of using the LID, but I did want to point out that
the SLID can be retrieved without putting it into the ARP packet format.

Jim

> From: Arvind Murching [mailto:[email-masked]]
> >a cached LID no matter if it came in the ARP or a pathrecord lookup
on

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 12:51:46 -0800

>nodes would consult with the SA and re-create their address handles

It is an implementation choice.  However, one should note that for 
Ethernet, this notification method is often used and for IB to use another 
mechanism would require a change in the endnode's implementation.  Given 
many have a strong desire to not change the upper layer / services 
implementations, we would most likely use such a mechanism.  Depending upon 
the amount of cached objects maintained for each service, the SA may not 
need to be consulted for these subsequent updates though it would be for 
the initial set-up.  It really is implementation specific.

>headers of these gratuitous ARP packets.

As noted by Paul, one cannot trust this to always work.  One would have to 
set up all of the routes between all endnodes using IP to allow the LID / 
SL pair to always work.

Mike