Subject: Re: [Ipoverib] GID clarifications

From: Ajoy C Siddabathuni <[email-masked]>
Date: Thu, 04 Dec 2003 14:04:25 -0800 (PST)

! > As far as RECOMMENDED vs. MUST, going by the IETF standards, RECOMMENDED is
! > compelling enough for implementors to follow it blindly, still leaving room
! > for some clever implementations.
! > 
! > <snip>
! > 3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
! >    may exist valid reasons in particular circumstances to ignore a
! >    particular item, but the full implications must be understood and
! >    carefully weighed before choosing a different course.
! 
! Just to be careful, if a non DLPI system (eg Linux) uses GID1 for the mac
! address, the Linux system itself will function perfectly (ignoring whatever
! uniqueness problems there might be in choosing GID1 over GID0). The problem
! will be that any _other_ DLPI based system on the subnet will possibly not be
! able to interact 100% with the Linux box, even though the DLPI system itself 
! uses GID0 per the draft recommendations. IE, for the Linux box, the "full
! implications" are not local, but have to do with interop.
! 
! It seems a bit much to expect non DLPI systems (like Linux) to understand
! what impact their using GID1 might have on _other_ DLPI based system. That
! is why this issue should be addressed in the draft/rfc.

The analysis provided seems to rely on two assumptions -

1) Linux/non-dlpi systems are free to use GID1 and thus will not interoperate
   with Solaris
2) We expect implementations to interoperate in spite of the fact
   that they willfully deviated from the spec.

Given the current definition of the ipoib pkt format, the issue on hand does 
not necessarily only apply to DLPI systems, any application talking to the ipoib
driver directly (for whatever purposes) would have to deal with src/dst(IB)
address lookup problems; when that information is not provided explicitly by the
driver when passing up pkts. Thus, even Linux and other OSes should be
implementing the rfc by its letter and intent.

Finally, if we were to make a case with such assumptions as above, do we then
reconsider all uses of "RECOMMENDED/SHOULD" in the rfc.

thx,
~Ajoy

! 
! Thanks.
! 
! Kanoj
!  
! > 
! > thx,
! > ~Ajoy
! > 
! > !
! > ! Vivek
! > !
! > ! On Wed, 3 Dec 2003, Kanoj Sarcar wrote:
! > !
! > ! > Vivek Kashyap wrote:
! > ! > >
! > ! > > Kanoj,
! > ! > >
! > ! > > The port's GUID is it's EUI-64 identifier. A GID, in turn can be 
formed
! > using this GUID, the CA's GUID or some other value. Any GID is unique in the 
IB
! > fabric however only by using the port's GUID/EUI-64 identifier can one 
uniquely
! > identify the port. The other GID's, as noted, can be applied to other ports 
too
! > - though not concurrently. The GID formed using the EUI-64 identifier is the
! > 'GID at index 0'. I've made note of your comment and can clarify further as
! > needed.
! > ! > >
! > ! > > wrt, using only GID at index 0 as in the MAC address, there have been
! > proponents of using the other GIDs (non EUI-64 GID).
! > ! > >
! > ! > > Vivek
! > ! >
! > ! > Vivek,
! > ! >
! > ! > Per IBA 1.1 section 4.1 EUI-64 subsection and 4.1.1 item 2, manufacturer
! > assigned and
! > ! > SM assigned values are required to be EUI-64 compliant (they differ only 
in
! > u/l bits).
! > ! > Thus, all GIDs contain an EUI-64 in the lower 64 bits; thus saying "GUID 
is
! > its EUI-64
! > ! > identifier" might be confusing. Saying "GUID using manufacturer assigned
! > EUI-64 identifier"
! > ! > or better "GID index 0" removes all confusion. I think we agree on the
! > intent, I just
! > ! > want to make sure if you use the term GUID, you do specifically define 
what
! > you mean
! > ! > by it.
! > ! >
! > ! > Regarding MUST vs RECOMMENDED, is it worthwhile reconsidering given that
! > ! >
! > ! > 1. SM assigned GIDs are not that popular (anymore?)
! > ! > 2. it allows determination of sender's ipoib address, thus allowing
! > (receiver side)
! > ! > implementations to be dlpiv2 conformant?
! > ! >
! > ! > Thanks.
! > ! >
! > ! > Kanoj
! > ! >
! > ! >
! > ! >
! > ! > > --
! > ! > > Vivek Kashyap
! > ! > > Linux Technology Center, IBM
! > ! > > [email-masked]
! > ! > > [email-masked]
! > ! > > Ph: 503 578 3422 T/L: 775 3422
! > ! > >
! > ! > >     Kanoj Sarcar <[email-masked]>
! > ! > >     Sent by: [email-masked]
! > ! > >
! > ! > >     12/02/2003 03:54 PM
! > ! > >     Please respond to Kanoj.Sarcar
! > ! > >
! > ! > > Hi,
! > ! > >
! > ! > > It seems that certain parts of the
! > ! > > draft-ietf-ipoib-ip-over-infiniband-05.txt draft might
! > ! > > require clarification. For this discussion, please refer to
! > ! > >
! > ! > > 1. section 9.1 Link Information says:
! > ! > > "The GID, if formed using the GUID, can be used to unambiguously
! > ! > > identify an endpoint."
! > ! > >
! > ! > > 2. section 9.1.1 Link Layer Address/Hardware Address, part c) says:
! > ! > > "It is RECOMMENDED that the GID formed by the combination of the
! > ! > > IB subnet prefix and the port's GUID be included in the
! > ! > > link-layer/hardware address."
! > ! > >
! > ! > > Can comment #2 be safely taken to mean that GID index 0 is
! > ! > > being RECOMMENDED? Port GUID seems somewhat ambiguous, since I
! > ! > > am not sure (after reading IBA 1.1 section 4.1) whether the
! > ! > > term GUID refers only to the manufacturer assigned value that appears
! > ! > > in lower 64 bits of GID index 0, or whether it also includes
! > ! > > lower 64 bits of SM assigned GIDs at higher indices. If GID index 0
! > ! > > was really intended, the comment should explicitly state so to prevent
! > ! > > confusion.
! > ! > >
! > ! > > Also, to make things simple, is it reasonable to REQUIRE 
implementations
! > ! > > to always use GID index 0 (maybe this depends on whether existing
! > ! > > implementations all use GID0 only)? The reason this might be
! > ! > > interesting is that when a node receives a packet without a GRH
! > ! > > (whereby it already knows the sender's LID and QPN), it can possibly
! > ! > > reverse lookup the sender's LID to identify the sender's GID index
! > ! > > 0 and concatenate the sender's QPN to figure out the sender's link
! > ! > > address. Thus, a receiver can always figure out the sender's address,
! > ! > > even when a GRH is not present. This would be similar to ethernet,
! > ! > > where a receiver can always determine a sender's address, and thus
! > ! > > is more DLPIv2 (unitdata_indication) friendly.
! > ! > >
! > ! > > Regarding comment #1, what does "unambiguously identify an endpoint"
! > ! > > mean? GIDs are unique in the subnet; and if an IP link contains
! > ! > > members in multiple IBA subnets, the subnet prefixes will disambiguate
! > ! > > which endport is being referred.
! > ! > >
! > ! > > Thanks.
! > ! > >
! > ! > > Kanoj
! > ! > >
! > ! > > _______________________________________________
! > ! > > IPoverIB mailing list
! > ! > > [email-masked]
! > ! > > https://www1.ietf.org/mailman/listinfo/ipoverib
! > ! >
! > ! > _______________________________________________
! > ! > IPoverIB mailing list
! > ! > [email-masked]
! > ! > https://www1.ietf.org/mailman/listinfo/ipoverib
! > ! >
! > ! >
! > !
! > ! __
! > !
! > ! Vivek Kashyap
! > ! Linux Technology Center, IBM
! > !
! > !
! > ! _______________________________________________
! > ! IPoverIB mailing list
! > ! [email-masked]
! > ! https://www1.ietf.org/mailman/listinfo/ipoverib
! 
! _______________________________________________
! IPoverIB mailing list
! [email-masked]
! https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Kanoj Sarcar <[email-masked]>
Date: Thu, 04 Dec 2003 16:43:54 -0800

> The analysis provided seems to rely on two assumptions -

Hi,

>    that they willfully deviated from the spec.

"Willfully deviated" might be too strong, given that the working group currently
chooses to allow the flexibility of using other GIDs; what seems to be under
discussion is the relative benefit of allowing that flexibility, if there is a
cost to be paid in terms of interop.

> implementing the rfc by its letter and intent.

Here is the difference between DLPI and nonDLPI systems: on non DLPI systems, if
the os allows raw access to the interface, then programs utilizing the raw
interface must neccesarily be aware of the underlying packet format, and either
deal with sender address not being present, or know how to decipher it from the
packet header. If you took a program that runs on raw Ethernet, there is a very
good chance you will need to modify the program to run on raw IPoIB.

With DLPI, there is an attempt to reduce the media awareness of programs, by
requiring the os to provide some meta information for every packet received
(sender's mac address in this case). In practice, this does not guarantee that 
a program that runs on ethernet will run unmodified on IPoIB, but there is a 
good chance that it might. 

As you point out, determining the sender's address is problematic in either
case (dlpi or non dlpi): in the case of dlpi though, the onus is on the driver
to provide the information (dlpi apps expect it), whereas in non dlpi case, 
programs can choose to solve the problem in their own unique ways.
Using GID0 always is one way to solve the problem for everyone, the other is to require 
using GRH for every packet transmitted, but that has a performance impact and
is thus not desirable.

> reconsider all uses of "RECOMMENDED/SHOULD" in the rfc.

If there is an interop issue, or a case where some established practice or
software breaks (due to an implementation not following recommendation), in my 
opinion, we should think about the item seriously. 

Is there a reason you are opposed to changing the RECOMMENDED to a REQUIRED
in this specific case? Vivek mentioned there was inclination towards allowing
the flexibility, and I generally agree to that; but in this case, curtailing
the flexibility is solving a problem, without creating any new ones I am aware 
of.

Thanks.

Kanoj

---

From: "bill" <[email-masked]>
Date: Thu, 4 Dec 2003 18:04:16 -0800

Ok, let me ask a few questions to lead us toward concensus

1) Does anyone give a compelling reason to use anything but GID0 for
IPoIB - what is the benefit of doing this ?

2) Does anyone have an implementation that uses anything but GID0 for
IPoIB ?

If I do not hear any counter arguments by next Friday - I would like to
ask for concensus on MUST use GID0

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Kanoj Sarcar
Sent: Thursday, December 04, 2003 4:44 PM
To: Ajoy C Siddabathuni
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] GID clarifications

Ajoy C Siddabathuni wrote:
> 
> ! > As far as RECOMMENDED vs. MUST, going by the IETF standards, 
> RECOMMENDED is ! > compelling enough for implementors to follow it 
> blindly, still leaving room ! > for some clever implementations. ! >
> ! > <snip>

Hi,

>    that they willfully deviated from the spec.

"Willfully deviated" might be too strong, given that the working group
currently chooses to allow the flexibility of using other GIDs; what
seems to be under discussion is the relative benefit of allowing that
flexibility, if there is a cost to be paid in terms of interop.

> 
> Given the current definition of the ipoib pkt format, the issue on 
> hand does not necessarily only apply to DLPI systems, any application 
> talking to the ipoib driver directly (for whatever purposes) would 
> have to deal with src/dst(IB) address lookup problems; when that 
> information is not provided explicitly by the driver when passing up 
> pkts. Thus, even Linux and other OSes should be implementing the rfc 
> by its letter and intent.

Here is the difference between DLPI and nonDLPI systems: on non DLPI
systems, if the os allows raw access to the interface, then programs
utilizing the raw interface must neccesarily be aware of the underlying
packet format, and either deal with sender address not being present, or
know how to decipher it from the packet header. If you took a program
that runs on raw Ethernet, there is a very good chance you will need to
modify the program to run on raw IPoIB.

With DLPI, there is an attempt to reduce the media awareness of
programs, by requiring the os to provide some meta information for every
packet received (sender's mac address in this case). In practice, this
does not guarantee that 
a program that runs on ethernet will run unmodified on IPoIB, but there
is a 
good chance that it might. 

As you point out, determining the sender's address is problematic in
either case (dlpi or non dlpi): in the case of dlpi though, the onus is
on the driver to provide the information (dlpi apps expect it), whereas
in non dlpi case, 
programs can choose to solve the problem in their own unique ways. Using
GID0 always is one way to solve the problem for everyone, the other is
to require 
using GRH for every packet transmitted, but that has a performance
impact and is thus not desirable.

> 
> Finally, if we were to make a case with such assumptions as above, do 
> we then reconsider all uses of "RECOMMENDED/SHOULD" in the rfc.
>

If there is an interop issue, or a case where some established practice
or software breaks (due to an implementation not following
recommendation), in my 
opinion, we should think about the item seriously. 

Is there a reason you are opposed to changing the RECOMMENDED to a
REQUIRED in this specific case? Vivek mentioned there was inclination
towards allowing the flexibility, and I generally agree to that; but in
this case, curtailing the flexibility is solving a problem, without
creating any new ones I am aware 
of.

Thanks.

Kanoj

> ! > ! > > The port's GUID is it's EUI-64 identifier. A GID, in turn 
> can be formed ! > using this GUID, the CA's GUID or some other value. 
> Any GID is unique in the IB

_______________________________________________
IPoverIB mailing list
[email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Fri, 05 Dec 2003 09:16:07 -0800

>Ok, let me ask a few questions to lead us toward concensus

As the author of the IB spec for chapter 4, I'll attempt to provide some 
clarifications.

>1) Does anyone give a compelling reason to use anything but GID0 for IPoIB 
>- what is the benefit of doing this ?

Multiple GIDs were provided as a method to:

- Provide traffic segregation between subnets conceptually similar to 
multi-LID usage within a single subnet.

- Provide a method to support software / OS virtualization and device 
sharing via using a unique global identifier independent of the number of 
LIDs assigned (though one can take a similar approach if the LMC value is 
greater than 0.

I will note that there are many OSVs offering or starting to offer 
virtualization services and while some initial implementations may be 
focused on GID 0, this may not hold true in the coming few years.

- Provide a method to constrain out some data flows are routed - local vs. 
non-local subnet access given a GID can use either the default or the SM 
assigned subnet prefix.

>2) Does anyone have an implementation that uses anything but GID0 for IPoIB ?

I believe it would be a mistake to limit support to just GID 0.  IB 
technology was designed to support a diverse set of usage models that will 
evolve over time.  The only potential problem was a DLPI application raw 
level usage model which is (a) somewhat outside of the scope of the IP over 
IB specification and (b) rather limited in terms of usage within the 
industry.  I don't really want to debate the merits of raw DLPI but do 
acknowledge it is in use today on Ethernet and other layer 2 interconnects.

>ask for concensus on MUST use GID0

Please consider this a counter argument and a vote against such a movement.

Mike