Subject: RE: [Ipoverib] GID clarifications

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sun, 7 Dec 2003 20:22:33 -0800 (PST)

Mike,

>At 06:04 PM 12/4/2003 -0800, bill wrote:
>>Ok, let me ask a few questions to lead us toward concensus
>
>As the author of the IB spec for chapter 4, I'll attempt to provide some 
>clarifications.
>
>>1) Does anyone give a compelling reason to use anything but GID0 for IPoIB 
>>- what is the benefit of doing this ?
>
>Multiple GIDs were provided as a method to:
>
>- Provide traffic segregation between subnets conceptually similar to 
>multi-LID usage within a single subnet.

According to IBA 1.0a spec (i don't have 1.1 spec at hand), "A subnet shall
have only one assigned GID prefix at any given time (at GID index 0)."
It seems to me unless a CA port can be simultaneously connected to more
than one IB subnet, the unique subnet prefix plus the default GID prefix
(the link-local 0xFF80 prefix) are the only two possible values to have
for the upper 64 bits of a GID. I wonder what the traffic seggregation
between subnets you have in mind.

>
>- Provide a method to support software / OS virtualization and device 
>sharing via using a unique global identifier independent of the number of 
>LIDs assigned (though one can take a similar approach if the LMC value is 
>greater than 0.
>
>I will note that there are many OSVs offering or starting to offer 
>virtualization services and while some initial implementations may be 
>focused on GID 0, this may not hold true in the coming few years.

Not sure what flavor of "virtualization" you're alluding to. If the
virtualization is achieved through the use of multiple IB partitions,
different IB partitions can safely share the same GID much like
different Ethernet VLAN sharing the same MAC address. I don't see the
need for multiple GIDs.

Perhaps you meant virtualization like the hosting of multiple IP
addressses with the same subnet prefix but different host numbers
on the same NIC, this can be achieved through the use of multiple
locally-assigned EUI-64s in IB, and I agree it is a useful feature
to support.

>
>- Provide a method to constrain out some data flows are routed - local vs. 
>non-local subnet access given a GID can use either the default or the SM 
>assigned subnet prefix.

Note that the draft currently recommends the use of the IB subnet
prefix for the GID in IPoIB. Although the real requirement is that the
GID must be reachable (i.e. routable) at the IB layer from any node
within the scope of the IP link it belongs to, recommending the IB
subnet prefix is less error prone as it continues to work even after
the underlying IP link has been expanded to cover multiple IB subnets. 

>
>>2) Does anyone have an implementation that uses anything but GID0 for IPoIB ?
>
>I believe it would be a mistake to limit support to just GID 0.  IB 
>technology was designed to support a diverse set of usage models that will 
>evolve over time.  The only potential problem was a DLPI application raw 
>level usage model which is (a) somewhat outside of the scope of the IP over 
>IB specification and (b) rather limited in terms of usage within the 
>industry.  I don't really want to debate the merits of raw DLPI but do 
>acknowledge it is in use today on Ethernet and other layer 2 interconnects.
>
>>If I do not hear any counter arguments by next Friday - I would like to 
>>ask for concensus on MUST use GID0
>
>Please consider this a counter argument and a vote against such a movement.

You are opposing requiring GID0, but are you ok with the current wording
of RECOMMENDING the GID formed by the IB subnet prefix and the portGUID
(the one assigned by the manufacturer)?

BTW, I'm confused by the use of GID0 as I thought in IBA spec. it means
the one with the link-local prefix but I thought people wanted to use
the one with the IB subnet prefix.

Jerry

>
>Mike
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 08 Dec 2003 06:37:53 -0800

At 08:22 PM 12/7/2003 -0800, H.K. Jerry Chu wrote:
>for the upper 64 bits of a GID.

Correct.  One can have either the default or an assigned thus all GID share 
one or the other.

>  I wonder what the traffic seggregation between subnets you have in mind.

The basic idea was one could encode in the lower 64 bits (these are admin 
assigned if not the port EUI64 value) or through a combination of other GRH 
attributes (e.g. TClass, Flow Label, etc.) one could segregate traffic to 
take different paths when passing through a router. The path selection 
would be implemented via selecting an egress port and assigning an 
appropriate next router hop / endnode LID.

>different Ethernet VLAN sharing the same MAC address. I don't see the

Virtualization is nominally a function of address while partitions are a 
function of segregation enforcement ala 802.1Q.  While one can use 
partitions, they are not seen by the underlying software while a GRH is 
seen thus depending upon how it is implemented, how the remote peer "sees" 
each target peer, etc. a GID is more flexible and may align better with 
application usage.  Please keep in mind I'm explaining how the motivation 
for multiple GID from an IBA perspective.  Now, can one use a single GID 
with multiple P_Key and then have a unique IP-to-QP mapping per resulting 
partition to achieve something similar?  The answer is yes, it is possible 
but that was not the intent of the IBTA when this was first being debated.

>the underlying IP link has been expanded to cover multiple IB subnets.

Agreed.

>(the one assigned by the manufacturer)?

I can live with RECOMMENDING not with MUST.

>the one with the IB subnet prefix.

GID 0 is just the guaranteed GID to be created via the combination of the 
port GUID (manufacturer assigned GUID) and the subnet prefix.

Mike

---

From: Kanoj Sarcar <[email-masked]>
Date: Mon, 08 Dec 2003 19:05:52 -0800

> At 08:22 PM 12/7/2003 -0800, H.K. Jerry Chu wrote:

Hi,

Michael, thanks for providing the IBTA context about multiple GIDs and path selection.

> appropriate next router hop / endnode LID.

It seems to me that for endports to influence which path to
use for each UD packet, the SM and the endnodes need to exchange 
some information. I was under the impression that SL/VL information 
within a IBA subnet, and additionally Tclass etc between IBA 
subnets, would be sufficient; ie, segregation could be achieved
with just these parameters, and would not need to extend to source
GID. I am not sure whether your use of the word "or" above means
that this is still a viable option.

> >locally-assigned EUI-64s in IB, and I agree it is a useful feature

At least from the ipoib perspective, it is also possible to 
virtualize based on the QPN (using same Pkey), since that is 
part of the mac address. I do agree that virtualization based on GIDs
is attractive.

> > >acknowledge it is in use today on Ethernet and other layer 2 interconnects.

Note that as has been pointed out, it is also possible for non DLPI apps to
want to know the sender's L2 address (trivially possible on ethernet by looking
at the received packet and picking out the source ethernet address). As you point
out, this might be slightly out of scope, but here's an example that is within
scope: imagine a node A with mac address M1 wants to know what IP address the 
RARP server will assign for a different mac address M2; it can fabricate a RARP 
message and broadcast it out; the RARP server must send the RARP response to M1
and not to M2. For this, the RARP server must be able to determine that the
sender's L2 address is M1. Similarly, an ARP implementation might want to 
determine the sender of an ARP request and validate against the "sender hardware
address" in the ARP message. I can't make a call on how extensive this type
of usage is though.

Thanks.

Kanoj