Subject: RE: [Ipoverib] What's inside an IPoIB link layer address? (SAdown)

From: "H.K. Jerry Chu" <[email-masked]>
Date: Fri, 8 Mar 2002 04:46:17 -0800 (PST)

It looks like "192.168.126.37" puked out bunch of old email recently:

Received: from 192.168.126.37 by wsecure.Brocade.COM with SMTP ( 
 Tumbleweed MMS SMTP Relay (MMS v4.7)); Tue, 05 Mar 2002 14:41:03 -0800    
X-Server-Uuid: 9f2286a2-9f82-11d3-b258-00805f85976b
 Received: from 132.151.1.19 by wsecure.Brocade.COM with ESMTP (
  Tumbleweed MMS SMTP Relay (MMS v4.7)); Thu, 17 Jan 2002 14:42:48 -0800
  
I can no longer ping or nslookup "192.168.126.37".

Jerry

>X-MimeOLE: Produced By Microsoft Exchange V6.0.6157.0
>Content-Class: urn:content-classes:message
>MIME-Version: 1.0
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address? (SAdown)
>Date: Fri, 8 Mar 2002 01:21:59 -0800
>Thread-Topic: [Ipoverib] What's inside an IPoIB link layer address? (SAdown)
>thread-index: AcHEl24J28zq273hTsGgIH5PS46l6QAd84Yw
>From: "Jim Pinkerton" <[email-masked]>
>To: "Vivek Kashyap" <[email-masked]>
>Cc: <[email-masked]>
>X-OriginalArrivalTime: 08 Mar 2002 09:22:01.0161 (UTC) 
FILETIME=[B408D390:01C1C682]
>Content-Transfer-Encoding: 8bit
>X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id 
EAA14903
>X-Mailman-Version: 1.0
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>X-BeenThere: [email-masked]
>
>
>Vivek,
>
>First, my apologies but somehow an email that I sent about 2 months ago
>popped out on the reflector. I'm unclear on how this happened. The
>original e-mail was part of the discussion on GID vs. LID, which has now
>been closed. So I won't go there.
>
>To address your question though, I'm not trying to propose anything -
>just trying to get an accurate assessment of the limitations of the
>existing IB spec. SM failover of database entries is not supported in
>the existing IB 1.0.a specification (confirmed by Roger's reply to my
>question) - thus you either have to:
>
>	- have a private protocol to fail-over the information, 
>	- apply some heuristic to attempt to retain as much 
>		information as possible
>	- or start from scratch
>
>I'm not an advocate of any of the above, but did want to point out (2
>months ago) that it is quite possible the LID won't be preserved. If a
>node has a single GID, derived from the GUID, then preserving GIDs is
>straightforward. As to multiple GIDs per port, that is a separate issue,
>which equates to why you would want multiple MAC addresses on a single
>port. I can't speak to that issue (although machines that support
>multiple virtual OS's on a single machine might :-)
>
>
>Jim
>
>
>
>
>
>
>-----Original Message-----
>From: Vivek Kashyap [mailto:[email-masked]] 
>Sent: Thursday, January 17, 2002 9:50 PM
>To: Jim Pinkerton
>Cc: [email-masked]
>Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>(SAdown)
>
>
><VK>
>Jim,
>      How do you propose that the SA, if it cannot preserve the LIDs,
>keep
>the GIDs recorded rightly ? GID prefix may live but what about the many
>GIDs assigned to each port ? Additionally will it selectively keep all
>the
>MGIDs in the database so that our subnet survives but loose the LIDs ? I
>would expect a backup SA to always be able to share the database. At
>minimum the backup can collect the existing information and build its
>database and assign to only those that are unassigned. If that can't be
>provided and the crash of an SA causes all of IB activity to come
>crashing
>down (LIDs/GIDs lost - all SRP connections/IP/SDP connections down)  no
>one
>will buy such an SM/SA. If IB cannot stay up through the crash of an SA
>then IB won't survive either.
>
>Vivek
><VK>
>
>Roger,
>
>Could you walk through how the LID is preserved across failover of a SA
>(i.e. it died, for whatever reason, and the backup SA takes over)? To me
>this is a critically important scenario for Data Centers (we both went
>through the GSN ARP fiasco, and have battle scars to prove it). It seems
>to me that there is no interoperable mechanism to forward the SM/SA
>state to the backup system, and thus the LIDs could be potentially
>re-assigned as part of the failover?
>
>The GID prefix though, is generally something the network administrator
>has access to (or leave as the default), and thus it is probably
>reasonable to expect that the GID is in fact preserved across the
>failover?
>
>
>
>Jim
>
>
>
>
>> -----Original Message-----
>> From: Roger Ronald [mailto:[email-masked]]
>> Sent: Thursday, January 17, 2002 12:14 PM
>> To: Vivek Kashyap
>> Cc: [email-masked]
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>
>> I think it's important to note that changes in LID should be
>infrequent.
>> Very infrequent.
>>
>> VIEO is one of a small number of companies that has a SM at this time.
>> We plan to maintain LID assignments whenever possible. That means
>> across reboots, across power fails and across subnet merges (when
>> possible).
>>
>> There are corner cases where LID may need to change, just like IP
>> addresses occasionaly need to change. But these are VERY odd
>> events...like the first time merge of fabrics. They won't happen
>> hourly, daily, monthly..... They may happen once or twice in a
>> fabric. Ever (unless the admin wants to make changes for the heck of
>it).
>>
>> RR
>>
>>
>> ----- Original Message -----
>> From: "Vivek Kashyap" <[email-masked]>
>> To: "Bill Strahm" <[email-masked]>
>> Cc: <[email-masked]>
>> Sent: Thursday, January 17, 2002 3:27 AM
>> Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
>>
>>
>> >
>> > >
>> > > Just free form thinking here...
>> > >
>> > > Could we say that GIDs are the equivalent of a Fully Qualified
>Domain
>> Name
>> > > (www.strahm.net for example) and LIDs are the equivalent of IP
>> addresses
>> > > (63.105.18.42).  Now in many cases IP addresses are quite stable,
>and
>> do not
>> > > change.  However would you ship an application with an IP address
>> encoded in
>> > > it, or with a FQDN.  It would be of course more efficient to use
>the
>> IP
>> > > address and avoid the DNS lookup (that has to go VERY far away,
>and
>> can take
>> > > quite a while for the first SYN message to get out), however if
>the
>> address
>> > > EVER changes, things become hard.
>> > >
>> > > Can anyone tell me why this is a bad analogy ???
>> >
>> >         If it was pure IB then GID to LID would be a similar
>analogy.
>> >
>> >         I'll try taking the your analogy further..a couple of points
>> >         though - the SM corresonds to the zone primary but the nodes
>> >         themselves corresond to both the secondary and the IP nodes
>> >         asking for the resolution. So the analogy is not exactly apt
>> >         but I'll try..
>> >
>> >         If the IP changes the DNS must be updated and looked up. In
>> >         this case if LID changes the SM is updated (yes) but it then
>> >         requires that the SM be looked up. There is no backlink here
>> >         to trigger this lookup. This implies that the full lookup
>all
>> >         the way to the DNS root must be done - if like DNS it is
>done
>> >         till the local secondary server ie. referring to the local
>> >         cached copy and/or merely re-receiving the peer's GID, the
>> >         communication will fail. You must have seen the hell that
>> >         breaks loose if any of the zone updates doesn't happen due
>to
>> >         some error and things are left pointing the wrong way.
>> >
>> >         Now, this would correspond to the internal LID cache flush -
>> >         since the GID is the same but the LID has changed. There is
>no
>> >         hook to say, hey!, go do a zone transfer i.e. lookup the SM
>> >         right now. You need a timer.
>> >
>> >         Now, you could make this time small - you are going to the
>SM
>> >         too often because things are not changing that much- make it
>> >         long and LID change is surely going to happen at a critical
>> >         time. Additionally this is another timer per entry for the
>> >         programmers/maintainers/administrators to worry about.
>> >
>> >         I have all along been arguing against keeping LID out due to
>> >         this reason too (code is more complex if no LID is another
>> >         reason). It has been my assumption that if GID is used
>without
>> >         LID then one is forced to go to the SM at every address
>> >         resolution. Every arp broadcast will have the whole subnet
>> >         scurrying off to the SM to ask the requester's LID. Yes,
>some
>> >         would mark the entry to be completed but not necessarily - a
>> >         thundering herd is likely.
>> >
>> >         This implies that if we don't use the LID we MUST ALWAYS go
>to
>> >         the SM for the LID - at every ARP since we don't know if LID
>> >         has changed. Of course if the GID changed too then the SM
>has
>> >         to be consulted anyway.
>> >
>> >         Also note that if we did have a trap from the SM on every
>LID
>> >         change it still wouldn't be a good solution - too much
>> >         latency; points of synchrnonisation; asynchronous updates;
>and
>> >         most of all the SM would end up doing nothing else but
>issueing
>> >         traps to every node that registered with it. IB cannot
>mandate
>> >         broadcasts/multicast (though making everyone look is bad
>too)
>> >         because multicasting is optional.
>> >
>> >         In the case of inclusion of the LID - some of the issues are
>> >         the same but a gratuitous ARP solves the problem if the LID
>> >         changed. It is an instant zone transfer. One could claim
>that
>> >         with GID only, on receiving any gratuitous ARP entry, the
>node
>> >         will go to the SM but that will require specialised coding
>to
>> >         recognise and handle gratuitous ARP differently than regular
>> >         updates. Not a good idea.
>> >
>> >         Even in the absence of gratuitous ARP the changed LID will
>be
>> >         known when the ARP is done after a cache expiry if LID is
>> >         included. If LID is not included one will get a GID back.
>That
>> >         will require the expiry of the GID to LID cache before one
>> >         goes to the SM.
>> >
>> >         Inclusion of LID makes things very simple. This is because
>LID
>> >         is always necessary - can't escape it.
>> >
>> >         I also like inclusion of LID because it cuts off 2x latency
>of
>> >         GID to LID lookup in the SM and 4x the RTT and the realted
>> >         extra interrupts. Fast failover in data centers is
>necessary.
>> >
>> > Vivek
>> >
>> >
>> >
>> > >
>> > > Bill
>> > >
>> > > -----Original Message-----
>> > > From: [email-masked] [mailto:[email-masked]]On
>> Behalf
>> > > Of Larsen, Roy K
>> > > Sent: Wednesday, January 16, 2002 6:43 PM
>> > > To: 'H.K. Jerry Chu'; [email-masked];
>> [email-masked]
>> > > Cc: [email-masked]
>> > > Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: H.K. Jerry Chu [mailto:[email-masked]]
>> > > > Sent: Wednesday, January 16, 2002 6:13 PM
>> > > > To: [email-masked]; [email-masked]
>> > > > Cc: [email-masked]
>> > > > Subject: Re: [Ipoverib] What's inside an IPoIB link layer
>address?
>> > > >
>> > > >
>> > > > All the proxy ARP usages that I know of use the proxy's mac
>> > > > addr in the
>> > > > ARP response. I believe the flavor of the proxy ARP you
>> > > > alluded below is
>> > > > rare, and won't work very well anyway if the link address
>includes
>> any
>> > > > non-persistent component like LID and QPN because it will
>> > > > require another
>> > > > protocol between the clients and their proxy to constantly
>update
>> the
>> > > > proxy database containing clients mac addrs, which may change
>quite
>> > > > often.
>> > > >
>> > > > That is also why I wanted to provide an option to not include
>QPN.
>> > > > (But I haven't succeeded w/o requiring SIDR or something
>> equivalent.)
>> > > >
>> > > > If LID is included as part of link addr and if it ever changes
>on
>> > > > a live node due to subnet merge...etc, the driver must inform
>its IP
>> > > > stack about such change. I believe many stacks don't have such a
>> > > > facility. (FOr most IP stacks the assumption is that its own MAC
>> > > > addresses just don't change on the fly.) In that case you're
>totally
>> > > > hosed. Can someone comment on how likely a LID assignment gets
>> > > > rescinded?
>> > >
>> > > It is possible anytime the link goes down and comes back up again
>and
>> when
>> > > the fabic is reconfigured.  This can happen on reboot or simply
>> > > unplugging/plugging in a link.  It is ultimately a function of the
>SM
>> and
>> > > how much trouble the implementation wants to go through to keep
>LIDs
>> > > constant.
>> > >
>> > > But of course, that has no bearing on whether the LID is contained
>in
>> the
>> > > ARP.  You always need the LID.  Unless you want to do a pathrecord
>> lookup
>> > > each time your implementation is given a packet to send, you're
>going
>> to use
>> > > a cached LID no matter if it came in the ARP or a pathrecord
>lookup on
>> the
>> > > GID.  If it becomes stale, it may take an ARP cache flush timeout
>to
>> > > recover.
>> > >
>> > > Roy
>> > >
>> > > >
>> > > > Jerry
>> > > >
>> > > > >        A device can always respond for another. The LID of
>> > > > the sender
>> > > > >        is not the same as the LID for which it has
>> > > > responded. It also
>> > > > >        allows any network monitoring/debugging device to work
>> > > > >        independently. This is the reason why none of the other
>> media
>> > > > >        does anything like this. You don't pick ethernet source
>> > > > >        address from the frame but take it from the ARP/ND
>packet.
>> > > > >
>> > > > >        [ If it was ok we wouldn't need anything except
>> QPN/cap_flags
>> > > > >        - get the LID from response, consult SA using LID for
>GID
>> if
>> > > > >        you want that, and you are done. ]
>> > > > >
>> > > > >Vivek
>> > > > >
>> > > > >>
>> > > > >> Why do we need to include the LID inside the link layer
>> > > > address when it
>> > > > >> is already available in the SLID field? Previous
>> > > > discussions indicate
>> > > > >> that the SLID value works for proxy ARP/bridge
>implementations.
>> > > > >>
>> > > > >> - Arvind
>> > > > >>
>> > > > >> -----Original Message-----
>> > > > >> From: Vivek Kashyap [mailto:[email-masked]]
>> > > > >> Sent: Wednesday, January 16, 2002 10:29 AM
>> > > > >> To: [email-masked]
>> > > > >> Cc: [email-masked]
>> > > > >> Subject: Re: [Ipoverib] What's inside an IPoIB link layer
>> address?
>> > > > >>
>> > > > >> Ashwin,
>> > > > >> Ashwin,
>> > > > >>
>> > > > >>         I agree that you get more flexibility if you
>> > > > include the LID.
>> > > > >>         If we include the GID along with the LID we allow all
>> > > > >>         implementations the choice to create different
>> > > > solutions. We
>> > > > >>         don't tie them down to one thing. Choosing one or the
>> other
>> > > > >>         (LID or GID) blocks out various advantages that
>different
>> > > > >>         implementations seek. Hence I suggested that let us
>> include
>> > > > >>         both and move on.
>> > > > >>
>> > > > >> Vivek
>> > > > >>
>> > > > >> <snip>
>> > > > >>
>> > > > >> _______________________________________________
>> > > > >> IPoverIB mailing list
>> > > > >> [email-masked]
>> > > > >> http://www.ietf.org/mailman/listinfo/ipoverib
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Vivek Kashyap
>> > > > >Linux Technology Center, IBM
>> > > > >[email-masked]
>> > > > >[email-masked]
>> > > > >503 578 3422 (o)
>> > > > >
>> > > > >_______________________________________________
>> > > > >IPoverIB mailing list
>> > > > >[email-masked]
>> > > > >http://www.ietf.org/mailman/listinfo/ipoverib
>> > > >
>> > > >
>> > > > _______________________________________________
>> > > > IPoverIB mailing list
>> > > > [email-masked]
>> > > > http://www.ietf.org/mailman/listinfo/ipoverib
>> > > >
>> > >
>> > > _______________________________________________
>> > > IPoverIB mailing list
>> > > [email-masked]
>> > > http://www.ietf.org/mailman/listinfo/ipoverib
>> > >
>> > >
>> > > _______________________________________________
>> > > IPoverIB mailing list
>> > > [email-masked]
>> > > http://www.ietf.org/mailman/listinfo/ipoverib
>> > >
>> >
>> >
>> > --
>> > Vivek Kashyap
>> > Linux Technology Center, IBM
>> > [email-masked]
>> > [email-masked]
>> > 503 578 3422 (o)
>> >
>> > _______________________________________________
>> > IPoverIB mailing list
>> > [email-masked]
>> > http://www.ietf.org/mailman/listinfo/ipoverib
>> >
>> >
>>
>>
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> http://www.ietf.org/mailman/listinfo/ipoverib
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Bill Strahm <[email-masked]>
Date: Fri, 8 Mar 2002 09:52:31 -0500 (EST)

I noticed that :(

I have also been watching the bounce messages from qlogic.  When I get
Internet at home on Monday, I am going to look into some of this (Maybe
unsubscribe the unlucky folks who are generating the continuous failures)

Bill

---

From: Roland Dreier <[email-masked]>
Date: 08 Mar 2002 07:49:18 -0800

>>>>> "Jerry" == H K Jerry Chu <[email-masked]> writes:

    Jerry> It looks like "192.168.126.37" puked out bunch of old email
    Jerry> recently: Received: from 192.168.126.37 by
    Jerry> wsecure.Brocade.COM with SMTP ( Tumbleweed MMS SMTP Relay
    Jerry> (MMS v4.7)); Tue, 05 Mar 2002 14:41:03 -0800 X-Server-Uuid:
    Jerry> 9f2286a2-9f82-11d3-b258-00805f85976b Received: from
    Jerry> 132.151.1.19 by wsecure.Brocade.COM with ESMTP ( Tumbleweed
    Jerry> MMS SMTP Relay (MMS v4.7)); Thu, 17 Jan 2002 14:42:48 -0800
  
    Jerry> I can no longer ping or nslookup "192.168.126.37".

192.168.126.37 is part of the private, non-routable 192.168.0.0/16
netblock (see e.g. RFC 1597).  Unless your local network is using the
address 192.168.126.37, you should not be able to ping or nslookup the
address -- and even if you could, no one outside your organization
would be able to reach the same machine.  Based on the headers above
the system sending out old mails would seem to be inside Brocade's
firewall.

Roland