Subject: [Ipoverib] Broadcast GID creation and Q-Key resolution

From: Barton Shields <[email-masked]>
Date: Fri, 1 Nov 2002 12:58:08 -0800

All,

   In trying to practically implement IPoIB, I have
found a number of areas that need to be clarified.
As I have had to find solutions, which at present I
consider to be proprietary (inferring I fear we
won't work with other vendors' equipment).

   While I am sure most of us who have to implement
IPoIB have found similar solutions for the particular
problems addressed in this e-mail, these solutions
should be part of the spec so that our equipment can
interoperate.

   The first area I have had difficulty with is defining
exactly who it is that "creates" the Broadcast GID
multicast group.  I know the "IPoIB Architecture" draft
says, "The fabric administrator creates the IB multicast
group corresponding to the all-IP nodes/IPv4 broadcast
(henceforth called 'broadcast group') when the IPv6/IPv4
subnet is setup.  The method by which the broadcast group
is setup is not defined by IPoIB."

   While the aforementioned paragraph infers this will be
done on the Subnet Manager, it doesn't explicitly do so.
Additionally, even if it did, what happens in a point-to
point network where there is no subnet manager.  Who
creates it then, what is the mechanism?  I currently have
a proprietary protocol for this situation that uses the MI
to co-register my LIDs/GIDs with both sides of the link,
but that is strictly proprietary and will not work with
another vendor's equipment.

  Last, if the method of how to set this group up is not
defined by IPoIB, does this imply that a particular
implementer of IPoIB may choose to allow the user to
configure the Q-Key on individual nodes, so that if the
fabric administrator did not define the multicast group
representing the IPoIB broadcast group, that the node
can create the Broadcast Group itself via a MCGroupRecord
create?

   The second area I found particularly disconcerting was
in obtaining the Q-Key.  The "IP encapsulation and address
resolution over InfiniBand networks" draft states the
following:

"Every IPoIB interface MUST join the InfiniBand multicast group
 defined by the broadcast-GID. This operation returns the MTU
 and the Q_Key associated with the IPoIB link. Thus the IPoIB
 subnet (and the link) is formed by the IPoIB nodes joining the
 broadcast GID."

  This may lead one to believe that I can get the Q-Key from the
header.  However, in reality, I cannot get to this since it is
the DETH and the DETH is stripped off by the particular IB chip
I am using (I can't speak for other chips, but know I can't on
this one).  The obvious solution is to obtain the Q-Key from the
subnet manager via a MCMemberRecord "join" (assuming the
broadcast group already exists).  However, nowhere is this
mechanism specified  Additionally, as before, no mention is made
as to what to do for point-to-point networks with no SM (although
this is a trivial problem, it should be addressed).

                            Respectfully,

                            Bart Shields
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Barton Shields <[email-masked]>
Date: Fri, 1 Nov 2002 17:35:03 -0800

Vivek,

  Your point with respect to how IB does the
MCMemberRecord Join is obvious given the constraints
of the chip not providing the DETH.  As it is the
ONLY way to do it and I had already figured that out.
However, you are implying that it is the ONLY way to
accomplish the discovery of the Q-Key (i.e., via the
MCMemberRecord join).

  Is there anything in IB that specifies a chip
has to strip off the DETH?  What if an IB chip
maker gave the user the DETH and/or gave the
Q-Key in its descriptor information when presenting
the buffer to the user?  Then, it would be unnecessary
to do the MCMemberRecord Join.

   My point is by assuming it is obvious, you
are leaving a glaring hole.  If the later case
occurs and it has not been addressed, then we
could very well end up with the lack of
interoperability as I mentioned before.

   In an emerging technology like IB, especially
one that is to some degree struggling, we can not
leave things like this vague.  They must be
specified clearly and not be left up to interpretation.

			Respectfully,

			     Bart

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Friday, November 01, 2002 3:04 PM
To: Barton Shields
Cc: [email-masked]
Subject: Re: [Ipoverib] Broadcast GID creation and Q-Key resolution

> another vendor's equipment.

Barton,

The IB fabric is brought-up by the SM - as per the IB specification it
discovers and then assigns the LIDs to the ports. Thus whichever
entity is assigning the LIDs, and initialising the P_Key tables is
fulfilling the role of the subnet-manager. That entity should
therefore fulfill other roles of the subnet manager too.

> can create the Broadcast Group itself via a MCGroupRecord

Any node can setup the broadcast-group but the fabric-administrator
will likely assign the task to a particular node/utility or set it on
the SM/SA directly. As per InfiniBand the creator needs to set the
following for the broadcast-group:

	- partition
	- Q_Key
	- MTU
	- SL
	- Scope
	- Hop-limit

     and various other values. 
     
A random setup has the potential to disrupt things. 

So, it needs to be left to the administrator to decide who sets the
group up and how. An implementation might provide a method by which an
administrator can assign it the task of creating the broadcast-group
along with a mechanism by which the administrator can tell it the
parameters to be used for creating the broadcast group.

In other words, a node can surely create the Broadcast-group but it is
better if the decision on which node and the parameters to be used is
thought out carefully by an administrator. The actual creation of the
broadcast-group is secondary.

> this is a trivial problem, it should be addressed).

The Q_Key is retrieved from the MCMemberRecord like you say above.
That is what was meant to be conveyed by '....operation returns the
MTU and Q_Key'. The architecture draft will provide the description of
the version 1.1 changes regarding multicast join and creation. We can
modify the encapsulation draft to explictly mention the member
record. 

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Barton Shields <[email-masked]>
Date: Mon, 4 Nov 2002 08:26:38 -0800

Thank you Vivek for that explanation.  I never knew
that that (that it was not the same Q-Key).

                    Bart

-----Original Message-----
From: vivek [mailto:[email-masked]]
Sent: Sunday, November 03, 2002 3:03 PM
To: Barton Shields
Cc: [email-masked]
Subject: RE: [Ipoverib] Broadcast GID creation and Q-Key resolution

Barton,

The Q_Key in the DETH of the multicast join request/reply message is not
the same, as the Q_Key set for IPoIB communication in the MCMemberRecord.
So tapping the DETH will get you the GSI Q_Key.

Secondly, a port needs to join the Broadcast-group. Only then does the SM
program the switches to route the relevant  multicast packets to the
port. Therefore the join operation is essential.

I'll make this clearer in the drafts.

Vivek

---

From: Barton Shields <[email-masked]>
Date: Mon, 4 Nov 2002 08:26:38 -0800

Bart

-----Original Message-----
From: vivek [mailto:[email-masked]]
Sent: Sunday, November 03, 2002 3:03 PM
To: Barton Shields
Cc: [email-masked]
Subject: RE: [Ipoverib] Broadcast GID creation and Q-Key resolution

Barton,

Vivek

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 4 Nov 2002 10:15:45 -0800 (PST)

>to do the MCMemberRecord Join.

Don't you need the right Q_Key to program the receiving
QP before the latter can receive anything? So retreiving
Q_Key from the DETH is out of the question. You'll have to
communicate Q_Key out-of-band. The best way to think of
Q_Key (along with other attributes such as MTU, TClass...)
is that it is an attribute of the IPoIB link that needs
to be defined when the link is being set up. (See the link
and multicast draft.) All these link attributes must be
retrieved from the respective multicast group record when an
IP stack on an end node configures an IPoIB link.

Jerry

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 4 Nov 2002 10:15:45 -0800 (PST)

Jerry