Subject: RE: [Ipoverib] Broadcast GID creation and Q-Key resolution

From: Barton Shields <[email-masked]>
Date: Fri, 1 Nov 2002 17:35:03 -0800

Vivek,

  Your point with respect to how IB does the
MCMemberRecord Join is obvious given the constraints
of the chip not providing the DETH.  As it is the
ONLY way to do it and I had already figured that out.
However, you are implying that it is the ONLY way to
accomplish the discovery of the Q-Key (i.e., via the
MCMemberRecord join).

  Is there anything in IB that specifies a chip
has to strip off the DETH?  What if an IB chip
maker gave the user the DETH and/or gave the
Q-Key in its descriptor information when presenting
the buffer to the user?  Then, it would be unnecessary
to do the MCMemberRecord Join.

   My point is by assuming it is obvious, you
are leaving a glaring hole.  If the later case
occurs and it has not been addressed, then we
could very well end up with the lack of
interoperability as I mentioned before.

   In an emerging technology like IB, especially
one that is to some degree struggling, we can not
leave things like this vague.  They must be
specified clearly and not be left up to interpretation.

			Respectfully,

			     Bart

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Friday, November 01, 2002 3:04 PM
To: Barton Shields
Cc: [email-masked]
Subject: Re: [Ipoverib] Broadcast GID creation and Q-Key resolution

> 
> All,
> 
>    In trying to practically implement IPoIB, I have
> found a number of areas that need to be clarified.
> As I have had to find solutions, which at present I
> consider to be proprietary (inferring I fear we
> won't work with other vendors' equipment).
> 
>    While I am sure most of us who have to implement
> IPoIB have found similar solutions for the particular
> problems addressed in this e-mail, these solutions
> should be part of the spec so that our equipment can
> interoperate.
> 
>    The first area I have had difficulty with is defining
> exactly who it is that "creates" the Broadcast GID
> multicast group.  I know the "IPoIB Architecture" draft
> says, "The fabric administrator creates the IB multicast
> group corresponding to the all-IP nodes/IPv4 broadcast
> (henceforth called 'broadcast group') when the IPv6/IPv4
> subnet is setup.  The method by which the broadcast group
> is setup is not defined by IPoIB."
> 
>    While the aforementioned paragraph infers this will be
> done on the Subnet Manager, it doesn't explicitly do so.
> Additionally, even if it did, what happens in a point-to
> point network where there is no subnet manager.  Who
> creates it then, what is the mechanism?  I currently have
> a proprietary protocol for this situation that uses the MI
> to co-register my LIDs/GIDs with both sides of the link,
> but that is strictly proprietary and will not work with
> another vendor's equipment.

Barton,

The IB fabric is brought-up by the SM - as per the IB specification it
discovers and then assigns the LIDs to the ports. Thus whichever
entity is assigning the LIDs, and initialising the P_Key tables is
fulfilling the role of the subnet-manager. That entity should
therefore fulfill other roles of the subnet manager too.

> 
>   Last, if the method of how to set this group up is not
> defined by IPoIB, does this imply that a particular
> implementer of IPoIB may choose to allow the user to
> configure the Q-Key on individual nodes, so that if the
> fabric administrator did not define the multicast group
> representing the IPoIB broadcast group, that the node
> can create the Broadcast Group itself via a MCGroupRecord
> create?

Any node can setup the broadcast-group but the fabric-administrator
will likely assign the task to a particular node/utility or set it on
the SM/SA directly. As per InfiniBand the creator needs to set the
following for the broadcast-group:

	- partition
	- Q_Key
	- MTU
	- SL
	- Scope
	- Hop-limit

     and various other values. 
     
A random setup has the potential to disrupt things. 

So, it needs to be left to the administrator to decide who sets the
group up and how. An implementation might provide a method by which an
administrator can assign it the task of creating the broadcast-group
along with a mechanism by which the administrator can tell it the
parameters to be used for creating the broadcast group.

In other words, a node can surely create the Broadcast-group but it is
better if the decision on which node and the parameters to be used is
thought out carefully by an administrator. The actual creation of the
broadcast-group is secondary.

> 
>    The second area I found particularly disconcerting was
> in obtaining the Q-Key.  The "IP encapsulation and address
> resolution over InfiniBand networks" draft states the
> following:
> 
> "Every IPoIB interface MUST join the InfiniBand multicast group
>  defined by the broadcast-GID. This operation returns the MTU
>  and the Q_Key associated with the IPoIB link. Thus the IPoIB
>  subnet (and the link) is formed by the IPoIB nodes joining the
>  broadcast GID."
> 
>   This may lead one to believe that I can get the Q-Key from the
> header.  However, in reality, I cannot get to this since it is
> the DETH and the DETH is stripped off by the particular IB chip
> I am using (I can't speak for other chips, but know I can't on
> this one).  The obvious solution is to obtain the Q-Key from the
> subnet manager via a MCMemberRecord "join" (assuming the
> broadcast group already exists).  However, nowhere is this
> mechanism specified  Additionally, as before, no mention is made
> as to what to do for point-to-point networks with no SM (although
> this is a trivial problem, it should be addressed).

The Q_Key is retrieved from the MCMemberRecord like you say above.
That is what was meant to be conveyed by '....operation returns the
MTU and Q_Key'. The architecture draft will provide the description of
the version 1.1 changes regarding multicast join and creation. We can
modify the encapsulation draft to explictly mention the member
record. 

Vivek

> 
>                             Respectfully,
> 
>                             Bart Shields
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: vivek <[email-masked]>
Date: Sun, 3 Nov 2002 15:03:21 -0800 (Pacific Standard Time)

Barton,

The Q_Key in the DETH of the multicast join request/reply message is not
the same, as the Q_Key set for IPoIB communication in the MCMemberRecord.
So tapping the DETH will get you the GSI Q_Key.

Secondly, a port needs to join the Broadcast-group. Only then does the SM
program the switches to route the relevant  multicast packets to the
port. Therefore the join operation is essential.

I'll make this clearer in the drafts.

Vivek

---

From: vivek <[email-masked]>
Date: Sun, 3 Nov 2002 15:03:21 -0800 (Pacific Standard Time)

Barton,

Vivek