Subject: RE: [ipoverib] reasons for not using LID ?

From: "Culley, Paul" <[email-masked]>
Date: Fri, 18 Jan 2002 08:37:12 -0600

1) a LID in the body of the of the ARP is insufficient for IB routing,
you need the LID/SL pair in order to actually get a packet anywhere.  IB
provides NO WAY to select an SL, based only on the LID, unless the ARP
requestor already has that LID and an SL cached or is willing to look it
up with the SA.  Since one of the reasons for including the LID at all
is to avoid this lookup, the solution is not complete without SL.
[Please don't respond that some implementation specific mechanism can be
chosen to select SL; there is no way for an endnode implementation to
determine the SA policies for SL usage at this time, so it is not
possible for the endnode to reliably chose an SL without asking the SA
(or caching a LID/SL derived from the SA).

2) LID in the body of the ARP will be invalid if the ARP crosses IB
subnets.  This has been discussed a lot already, but the following
observations seem pertinent:
 a) IB routers will happen someday, it would be nice to not have to
  go back to this process to fix ARP when it does happen.
 b) The IB Management group currently intends to support cross
  subnet operation for IB via use of the same SA PathRecord Lookup
  mechanism as is used for local operations.  This is primarily so
  that early endnode implementations can operate without changes.
  It is possible that other mechanisms may be defined as
  optimizations for routers, but that is at best an early concept. If
  we support a GID lookup for the ARP packets (and either drop body
  LIDs, use IB Header SLIDs, or indicate when to ignore them) ARP and
  IP can work across IB subnet boundaries.
 c) It is certain that IB routers will modify IB LRH headers;
  choosing new values for SLID/DLID/SL at subnet boundaries.  And a
  requirement on IB subnets and routers is that a unicast packet's
  SLID/SL can be used to return a packet to the source allows using
  SLID/SL in any case where the source of the ARP is also the place
  comm packets go.
 d) It seems a significant violation of layering to require an IB
  router to understand an IP ARP, just so that it can modify a LID in
  the body of the packet.

3) As it happens, it is perfectly legal for an IB fabric to use
different LID addresses to talk to the same device from different other
devices, that is one of the reasons that we allow up to 2^LMC addresses
for each device.  (That's right, there are potentially up to 128 LID
addresses for each port!)  And not all LID/SL addresses need be
reachable from all other devices.  This means that taking a LID from the
ARP Request as a way to communicate, may not work.  And it is not
currently possible for an endnode to determine in a defined way that the
SA uses this type of policy, so endnodes have NO architected WAY to tell
when to ignore the LID and go for GID on their own.

As an aside, this means that gratuitous multicast ARP and snooping the
multicast ARP requests is only good for the GID.  An endnode might
perform the snoop, but would have to keep track of the fact that the
LID/SL were not valid, probably somewhere in the ARP module.

My preference is to include only the GID in the body of the ARP request
or response, along with a bit that indicates when the IB Header's
LRH:SLID/SL is known to be invalid by the sender.  This bit would always
be set for multicast ARP requests, and usually reset for unicast ARP
responses.  It can be set in the ARP response if the responder KNOWs
that it is responding for some other port on the IB fabric.

Given this method, the majority of the time the endnodes could get
valid, best effort, LID/SL combinations directly from the ARP response
packets, and cache them.  The endnode could also snoop the ARP requests
or gratuitous ARPs for GIDs, and perform SA lookups when it needed to
send to one of the devices which only had the GID in the table.

Paul R. Culley
Compaq Sr. Fellow
281-514-5543

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 6:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like “you don't need it”, “we want to save 16-bit”, etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 18 Jan 2002 10:02:43 -0800 (PST)

> (or caching a LID/SL derived from the SA).

        SL can be determined based on workload characteristics. It is
        not tied to LID. An implementation is free to go to the SA or
        determine it on such a mapping. The IB spec leaves it upto the
        imlementation to make the choice - and some of us have. 

>   go back to this process to fix ARP when it does happen.

        As noted in replies to the above 'reason' there is nothing to
        fix. Still nothing has been specified when this will happen
        and why will it even be needed.

	ARP across IB subnet:

        On receiving an ARP request the receiving node will check the
        GID-prefix. If it is of the remote node it asks the routing
        infrastructure for the LID. If the GID is local it just uses
        it. It is very simple.

>   IP can work across IB subnet boundaries.

        Lot of speculation. Out of scope. We need to work with what is
        there and is likely to be present for a long time. A short
        incomplete list of IBTA to do:

		First routers need to be fully defined. 
		Then multicast needs to fleshed out.
        	Partitioning across subnets has to be figured out. 
		SA/SM coordination has to be specified. 
		Routing protocol has to be specified.
		Node to Routing infrastructure interaction has to be specified
		SA to Router interaction

	Then the usefulness of such a setup has to be justified. 

	[ Note that ARP including LID will work very simply as noted above.]

>   router to understand an IP ARP, just so that it can modify a LID in

        On receiving an ARP request the node will check the
        GID-prefix. If it is of the remote node it asks the routing
        infrastructure for the LID. If the GID is local it just uses
        it. It is very simple.

> when to ignore the LID and go for GID on their own.

There is no architected way to choose which of the LMC bits to pick.
If multiple paths are registered at the SA a GID lookup will return a
set of them. It is then upto the responder to pick the path it
chooses. which one will it choose ? how? It is a lot better for the
server, which setup the paths for a reason, specifies the one it wants
used. It is *better* to use LIDs in this case.

Include both GID and LID. It allows all implementations to work.

Vivek
-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 18 Jan 2002 13:53:07 -0800 (PST)

Yaron,

	personal mail. 

        what? not the clean approach ? how ? I thought I presented how
        the inclusion of LID is cleaner.

Vivek

Vivek

> 
> This is a multi-part message in MIME format.
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
> Content-type: text/plain; charset=windows-1255
> Content-Transfer-Encoding: 7BIT
> 
> Bill, Paul
> 
> I would like to conclude that discussion with out adding any counter
> arguments
> I agree that going the path of LID in the payload is not the clean approach,
> as both Paul & Bill presented 
> My understanding is that It could be made to work and solve other
> multi-subnet issues that exists today
> 
> I'm personally willing to go for Paul's suggestion and use SLID, plus add a
> way to indicate when to ignore it 
> I hope that Vivek would agree to it too, and we could go to the next topic
> on the agenda 
> 
> Paul, we need to see what we need to do in MSO to improve the multi-subnet
> path discovery mechanism 
> 
> Yaron
> 
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)
> Content-type: application/ms-tnef; name=winmail.dat
> Content-disposition: attachment; filename=winmail.dat
> Content-Transfer-Encoding: base64
> 
> eJ8+IiEVAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5wQAAAAAAADrAAEIgAcAGAAAAElQTS5NaWNy
> b3NvZnQgTWFpbC5Ob3RlADEIAQ2ABAACAAAAAgACAAEGgAMADgAAANIHAQASABcAEwAAAAUAGwEB
> A5AGAAgIAAAsAAAACwACAAEAAAALACMAAAAAAAMAJgAAAAAACwApAAAAAAADAC4AAAAAAAMANgAA
> AAAAHgBwAAEAAAAnAAAAW2lwb3ZlcmliXSByZWFzb25zIGZvciBub3QgdXNpbmcgTElEID8AAAIB
> cQABAAAAFgAAAAHBoGXP84B+bada8klnrhit06R7hEMAAAIBHQwBAAAAGQAAAFNNVFA6WUFST05I
> QFZPTFRBSVJFLkNPTQAAAAALAAEOAAAAAEAABg4A+gS/ZaDBAQIBCg4BAAAALgAAAAAAAADWDNpo
> g4/REbTSAIBfixTzAQARbIDaX4/REbTSAIBfixTzAAAAAatjAAAAAAMAFA4BAAAAAgEJEAEAAABi
> AgAAXgIAAGYDAABMWkZ1/brFCocACgENA0N0ZXh0Aff/AqQD5AXrAoMAUALzBrQCgyYyA8UCAGNo
> CsBzZWB0MTc3IAcTAoB9/wqACM8J2QKACoQLNxLCAdAEIEIDEGwsIFBhHnUVQAqyCoUKgEkgdwEI
> YGxkIGxpa2WQIHRvIAWgbmMKQC8BABowE3AFQGQEAGN1rQQQaQIgGYBpGwAgCGBJBUBhZBtQbmcc
> gG46eRphdQIwBJAcgHJn3nUHgAIwEzAZFmEJwhr0fGdvHMIbABogCrAcImZgIExJRCALgB/lec8U
> 8ByQIOAEIG5vBUAf8sUaoGUDkWFwcANgANB6aBgwYQQgBuAcIRhSIP4mF+MgIBVgE6AdcRnQGJXu
> TR0gHWAEgXMBkCYgHMK7IeEbA0kFQB1BGcFiGiDPAMAa0hpQGZByaxzxGdC6cwbwdhogI9EdkW0Y
> cIB0aS1zdWJuE7D/IdEqYAeRGwMPAAQAHjAaMeZkIXAYnSdtICAmQQIg/QdAbB0gA/AYEB+zGlAf
> kG4gAhAFwBhSJwQgKmBnfmcHkCowG9EpAhuQGiBT/yCxGDALUBuQHIIcgBmAIXC3GjILgBtQYxsg
> GiB3IAD7IQEyQWciEBVgIOAFQRkWzGhvLXAa9FZpKXAo4M8ZlB7lMkEiMW9vI3EpEf53InEZoy7R
> GkEf8iqQDxD9GjFwMpAcQCEEHuAJ8Cwg/yVmGJUYUhgwN1EqkCVBGkFvE6Ay0hshO1lkMkIF0U/9
> MiNtIxEpcR/yKgsgMxtSvT5Rch0gB4ATYQMAcy1QXRicWQrAAiAYnH1DUAAAHgBCEAEAAABHAAAA
> PDREMDI3OTg2MzUzRDEzNDFBREE0RjJBNEY4QTE3MDc2MEUzMTc1QGNjZWV4YzE4LmFtZXJpY2Fz
> LmNwcWNvcnAubmV0PgAAAwAJWQEAAAACAeJlAQAAABQAAAAyMlrrW6fzS5Zn4CmlyGckAABYyAIB
> 42UBAAAAFQAAABQyMlrrW6fzS5Zn4CmlyGckAABYyAAAAAMAA4AIIAYAAAAAAMAAAAAAAABGAAAA
> AFKFAACOagEAHgAEgAggBgAAAAAAwAAAAAAAAEYAAAAAVIUAAAEAAAAEAAAAOS4wAAsABoAIIAYA
> AAAAAMAAAAAAAABGAAAAAAaFAAAAAAAAAwAHgAggBgAAAAAAwAAAAAAAAEYAAAAAAYUAAAAAAAAL
> AAiACCAGAAAAAADAAAAAAAAARgAAAAADhQAAAAAAAAsACYAIIAYAAAAAAMAAAAAAAABGAAAAAA6F
> AAAAAAAAAwAKgAggBgAAAAAAwAAAAAAAAEYAAAAAEIUAAAAAAAADAAuACCAGAAAAAADAAAAAAAAA
> RgAAAAARhQAAAAAAAAMADIAIIAYAAAAAAMAAAAAAAABGAAAAABiFAAAAAAAAHgANgAggBgAAAAAA
> wAAAAAAAAEYAAAAANoUAAAEAAAABAAAAAAAAAB4ADoAIIAYAAAAAAMAAAAAAAABGAAAAADeFAAAB
> AAAAAQAAAAAAAAAeAA+ACCAGAAAAAADAAAAAAAAARgAAAAA4hQAAAQAAAAEAAAAAAAAACwAfDgEA
> AAACAfgPAQAAABAAAABZyzDRsdPRToXqq96IqtnMAgH6DwEAAAAQAAAA1gzaaIOP0RG00gCAX4sU
> 8wIB+w8BAAAAjAAAAAAAAAA4obsQBeUQGqG7CAArKlbCAABFTVNNREIuRExMAAAAAAAAAAAbVfog
> qmYRzZvIAKoAL8RaDAAAAE5UU0VSVkVSAC9vPVZvbHRhaXJlIEFkdmFuY2VkIERhdGEgU2VjdXJp
> dHkvb3U9Vk9MVEFJUkUvY249UmVjaXBpZW50cy9jbj1ZQVJPTkgAAwD+DwUAAAADAA00/T8AAAIB
> FDQBAAAAEAAAAFSUocApfxAbpYcIACsqJRceAPo/AQAAAAwAAABZYXJvbiBIYXZpdgACAX8AAQAA
> AI0AAAAwMDAwMDAwMEQ2MENEQTY4ODM4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDcwMDExNkM4MERB
> NUY4RkQxMTFCNEQyMDA4MDVGOEIxNEYzMDAwMDAwMDFBQjYyMDAwMDEyNkM4MERBNUY4RkQxMTFC
> NEQyMDA4MDVGOEIxNEYzMDAwMDBBQzkwQTU1MDAwMAAAAAADAAYQw5fPEgMABxDoAQAAAwAQEAAA
> AAADABEQAAAAAB4ACBABAAAAZQAAAEJJTEwsUEFVTElXT1VMRExJS0VUT0NPTkNMVURFVEhBVERJ
> U0NVU1NJT05XSVRIT1VUQURESU5HQU5ZQ09VTlRFUkFSR1VNRU5UU0lBR1JFRVRIQVRHT0lOR1RI
> RVBBVEhPRkwAAAAAv8o=
> 
> --Boundary_(ID_FXr1pBXMduEyHZhYqhe+Zw)--
>

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Jan 2002 23:19:28 +0200

Bill, Paul

Yaron

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 18 Jan 2002 14:25:12 -0800 (PST)

Oops! personal? I ended up sending to everyone :).

The way the router implementation was discussed with the router
looking into the packet was not clean.

But the router is not involved at all. See my earlier mail in this regard.

Consider:

* ARP packet:   LRH+GRH+Payload(LID+GID)

* The router forwards using the GRH.

* The final destination is the ARP receiver - the destined endnode

* The ARP module picks up the packet payload

	- if GID-prefix local
		copy LID + GID as is
	  else if GID-prefix not local
		use GID-prefix to get the LID from the routing infrastructure

The IB router is not involved at all.

Of course for the above to be setup the IPoIB subnets must span IB
subnets first. We can do it only when they do.

Now consider the case which can be implemented. The IPoIB module
associated with the IB router responds (proxy-ARP) for a particular
node on the other IB subnet that it knows about.

If LID is not included you cannot access it directly at all. Only if
the LID is included can you point at the IB router (LID) and the
remote node (GID).

On the whole I didn't come across any reason that blocks any of the
solutions the no-LID camp wants. Additionally, if they are not
satisfied they can always ignore the LID and always go get it again
from the SA.

Vivek

---

From: Libor Michalek <[email-masked]>
Date: Fri, 18 Jan 2002 15:12:10 -0800

> 		use GID-prefix to get the LID from the routing infrastructure

  Are you proposing the addition of this LID+GID usage algorithm, or
something similar, to the IPoIB specification? I'm concerned about 
interoperability across multiple implementations. If it is decided
that both LID and GID are included then there needs to be a requirment
for their usage by a compliant implementation, to insure interoperability.

  The fear is that as soon as a major implementation starts using the LID 
without looking at the GID, all IPoIB subnets will be restricted to a 
single IB subnet to maintain compatability.

  If LID+GID can be used anyway an implementation wishes, then the
spec might as well come out and say that an IP subnet cannot span
multiple IB subnets. I'm not saying that this is a bad thing, only
that the intention should be made clear to avoid future confusion. 
  
-Libor

--
Libor Michalek
[email-masked]
Topspin Communications.

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 18 Jan 2002 16:44:15 -0800 (PST)

> for their usage by a compliant implementation, to insure interoperability.

Things need to interoperate, I am fully behind that. Whenever the
IPoIB subnets get to span the IB subnets this algorithm would become
the requirement. 

> single IB subnet to maintain compatability.

        I don't think so. LID+GID are both used in the ARP packet. At
        present the implementations won't do cross-subnet packets but
        when they do the above algorithm will have to be followed.

> that the intention should be made clear to avoid future confusion. 

        Do you mean that the spec should say that at present the IPoIB
        subnets don't span IB subnets? The architecture draft says
        that much(I have received some comments on fine-tuning the
        words but the meaning remains the same). The encapsulation
        draft can indicate that the GID-prefix needs to be checked.

        I am not in favour of solving the span IB subnet problem in
        general but the above doesn't block that.

Vivek

---

From: Libor Michalek <[email-masked]>
Date: Mon, 21 Jan 2002 15:04:38 -0800

>         when they do the above algorithm will have to be followed.

  If the algorithm is something that will have to be followed, then
shouldn't it be a part of the specification? If it is not part of
the specification, then why assume that all implementations would
work this way.

  In fact I would assume that if something is not stated it will be
implemented in two, potentially incompatable, ways. So if the spec
can result in two incompatable implementations, at least across IB
subnets, it should be stated that the scope of the spec is only
within a single IB subnet.
 
>         draft can indicate that the GID-prefix needs to be checked.

  The architecture draft states a number of IB problems that prevent
cross subnet packets. I'm concerned about problems with how the spec
uses IB which would introduce incompatability once/if the IB problems
are resolved.

>         general but the above doesn't block that.

  There are problems with IB and multiple subnets, and I have no wish
to tackle those problems. However, this is more of a problem with how
IB is used. Even if all the IB specific problems are resolved, an IP
subnet will not work across two IB subnets if the GID is not used
correctly, the correct usage should be in the spec. If the correct
usage is not in the spec, it needs to be stated that two compliant
implementations can only be guaranteed to interoperate in a single
IB subnet. 

-Libor

--
Libor Michalek
[email-masked]
Topspin Communications.

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 21 Jan 2002 18:22:21 -0800 (PST)

>   If the algorithm is something that will have to be followed, then
Libor,
        Maybe I wasn't as clear as you wanted it but I was agreeing
        with you that we need to make it part of the spec. No argument
        there. 

> within a single IB subnet.

Yes, it needs to be stated in the spec.

We also need to, independent of the above, answer the question - why
do we need to span IP subnet across IB subnets ? It has been stated
often enough that IP subnets will span IB subnets without any
justification. The maximum I have heard is that it is possible to do
when IBTA solves the IB level issues. That is neither here nor there.

Why do we need IP subnets spanning IB subnets ?

Vivek

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 12:31:53 -0800

>         imlementation to make the choice - and some of us have.

The SL is an opaque identifier derived from the path record. The meaning of 
this identifier is local to each IB subnet, i.e. it is not a global or 
well-defined value.  Therefore there is no workload characteristics one may 
use to determine what is the appropriate value for the SL.. Further, given 
the SL is used to map VLs within a switch and thus may be used to break 
deadlock loops within an IB subnet, there can be an implicit relationship 
between the SL and the SLID in terms of the appropriate LID to use to 
return a response.  The only way I can see to take the SL and SLID and 
re-use them for a unicast ARP response would be to have a homogeneous 
implementation, i.e. single vendor driven - I see nothing to preclude 
someone from developing such a solution and nothing within the current IP 
over IB architecture per se.

>         infrastructure for the LID. If the GID is local it just uses

The GID within the ARP request payload is used to determine whether an 
endnode is on a local or remote IB subnet.  The responder will need to 
consult either it local cache of address vectors for this endnode or 
consult the SA to acquire a path record to determine the appropriate path 
(IB LRH parameters) to return a response.  If the response does not cross 
an IB subnet, the responder does not need to include a GRH within the 
response thus the GID is not necessarily used / returned on the wire itself 
(though it will be used to update the responder's ARP cache).

>                 First routers need to be fully defined.

IB routers are defined in terms of the SM / SA interactions.  This is 
documented in the current IB specification.  While there are on-going 
errata to update the current specification, there is nothing that directly 
impacts IP operation over IB routers as they are currently defined.

>                 Then multicast needs to fleshed out.

IB multicast currently only lacks the multicast group management 
protocol.  Within the industry, there are many technologies that have 
allowed multiple group management protocols to be implemented on top of a 
single fabric type.  There is currently nothing to preclude IB management 
vendors from providing multicast using existing technologies and perhaps 
the IBTA should take notice of how it might benefit from a similar strategy.

>         Then the usefulness of such a setup has to be justified.

The fact that some of these items have yet to be worked out does not 
translate into the requirement to support IP across multiple IB subnets 
cannot be justified.  Requirements are business and technology driven and 
many have concluded that IP over multiple IB subnets is a MUST requirement 
for this technology and have people working within the IETF and the IBTA to 
ensure this requirement can be met as products become ready.  Understanding 
the issues up front provides both organizations with the needed guidance to 
create something viable and interoperable.

>used. It is *better* to use LIDs in this case.

Hence why one should use the address vector that corresponds to a given 
service.  The address vector includes a number of parameters beyond just 
the GID and LID.   The endnode should examine the service being established 
and consult whatever policies service management that exists (either local 
or remote) to determine the appropriate vector and from that derive the LRH 
/ GRH fields to use.

Mike