Subject: Re: [Ipoverib] IP encapsulation Design Team

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 20 Oct 2001 15:04:02 -0700 (PDT)

Vivek,

>> >1) Define the link boundary for an IPoIB link
>> >2) Determine the per-link IB attributes
>> >3) Determine how to map IP multicast to IB multicast on each IPoIB link
>> >4) Determine the IPoIB link-layer address (or "hardware address" in ARP's
>> >term) format
>>
>> The I-D "draft-hkchu-ipoib-ipv6oib-00.txt" I submitted in July
>> contains all the details. Note that although the I-D focuses on
>> IPv6, most of the content can be applied equally to IPv4. Also
>> the I-D covers unicast as well as multicast. (I didn't think
>> it was necessary to separate them into different I-Ds.)
>
>For an alternatve view the above issues are discussed in detail in
>draft-kashyap-ipoib-ipv4-and-multicast-00.txt (submitted in April) and
>-01.txt (submitted in May) after receiving feedback on this mailing

I reread your I-Ds again and frankly it is still not clear to me what
you proposed to use to define IPoIB link boundary. Do you use IB
subnet along, or IB partitions, or together, or P_Key/Q_Key to
delimit an IPoIB link? You only mentioned something like "It is a MUST
that the administrator setup the IB multicast group corresponding to
IPv4 broadcast address".

I'm not sure what feedback from this mailing list you're alluding to.
There have been a long discussion of IPoIB since late last year, first
in IBTA, then in IETF that both of us were involved. But on the subject
of multicast and IPoIB link boundary, I don't remember any public,
detailed discussion so far. It may have been due to the incompleteness
of the IBA spec. in this area (as I pointed out in London IETF).

>list. draft-kashyap-ipoib-ipv4-and-arp-01.txt builds on the multicast
>document. The same ideas were applied to the ipv6 case in the
>draft-kashyap-ipoib-ipv6-over-infiniband-01.txt. Please see the above
>drafts for details.
>
>I'll concentrate on the first three issues raised in your mail and
>list the common grounds between our drafts and subsequent suggestions.
>It seems that we agree on all but for minor differences. The following
>discussion applies to IPv4 and IPv6 equally. The differences are
>called out specifically.
>
>> 
>> 1) link boundary for an IPoIB link
>> 
>> A link boundary is defined by using MCGroupRecords through Subnet
>> Administration (SA). P_Key and Q_Key combined serve as a "link ID".
>> All the CAs belonging to the same all-node multicast group (FF02::1)
>> defined through a MCGroupRecord with a given link ID form an IPoIB
>> link. (See more detailed discussion below.)
>>
>> 2) per-link IB attributes
>> 
>> MTU, P_Key, Q_Key
>> 
>> 3) IP multicast address to IB multicast address mapping
>> 
>> Identity function for IPv6 with possible adjustment on the scope bits.
>
>1. Link : distribution of attributes
>
>        The MCGroup does define the link boundary. It also serves the
>        purpose of distributing the link attributes to the hosts that
>        join the MCGroup.

Hmmm, MCGroupRecords in IBA spec. defines IB multicast groups, not
link boundaries. An IB link boundary is defined by an IB subnet. I
don't know what you mean by "MCGroup does define the link boundary".

I'm proposing to use P_Key plus (optional) Q_Key to define an IPoIB
link boundary. All the entities belonging to the same IPoIB link
can discover each other through a well-defined (e.g. all-node)
MCGroup. What do you propose to define an IPoIB link?

>
>        Thus any of the nodes that joins the MGID corresponding to
>        all-nodes IP address receives the three important link
>        attributes: MTU, P_Key and Q_Key. There are other link
>        attributes but the process for their determination may be
>        different.

You didn't say what if there are multiple MCGroups with the same MGID,
which one a node should choose? Or perhaps you allow only one all-node
MGID and thus one IPoIB link per IB subnet?

>
>        I don't think we can use the MGID FF02::1 in the mappings.

This actually depends on how IBTA decides on what defines an IB MC
group. Is it MGID along, or MGID plus P_Key, or MGID plus P_Key plus
Q_Key, or something else. IBA spec. designated FF02::1 as the ALL
CHANNEL ADAPTERS MULTICAST GROUP, but neglect to worry about the
ramification involved P_Key and Q_Key matching.

>        Furthermore we can use Q_Key along with P_Key as the link ID
>        but Q_Key is not useful when used in IP multicast to MGID
>        mappings. See section 3d for a detailed discussion.
>
>2. Link: IP multicast to IB MGID mapping
>	(See below for detailed discussion.)
>
>2a. IPv4 mapping
>
>	Broadcast address maps to:
>
>	FF12:3334:<16-bit P_Key>::255.255.255.255
>	FF12:3334:<16-bit P_Key>::<IP multicast address>
>
>                * usage of P_Key provides separation of broadcast
>                domains even if IB doesn't allow multiple
>                MCGroupRecords with same MGIDs.
>
>		* usage of separate MGIDs aids in administration.
>
>                * 0x3334 clearly indicates and IPv4 MGID. Will help
>                with administration, trouble-shooting.
>
>
>2c. IPv6 mapping 
>
>	FF12:3333:<P_Key>:<80 Low order bits of IPv6 multicast addresses>
>
>
>Detailed discussion:
>--------------------
>
>3: Link : IP multicast to IB MGID mapping
>
>3a. Scope bits
>
>        The solution in this regard is effected by the answer to the
>        following questions :
>
>        Should IP subnets span IB subnets ? What are the benefits ?
>        Why is IP subnetting not enough ?
>
>        There was a discussion on this issue in August but in my view
>        no strong benefit was put forward. So I lean toward keeping IP
>        subnets within IB subnets. Thoughts ?
>
>        Even if we can identify the benefits a final solution cannot
>        be implemented unless IB routing is defined, P_Keys across IB
>        subnets are well-defined, and setting of IB multicast groups
>        across IB subnets is well defined.
>
>	So I agree with the following scheme:
>		
>                Define IPoIB subnets to be within IB subnets at this
>                time. When the IB infrastructure is complete the
>                implementations can be upgraded along the lines you
>                suggest. 
>
>                Let the interface try and join the multicast address
>                with the local scope. If it doesn't succeed in finding
>                the MGID then try the global scope.
>
>                The implementations can have a local variable that
>                can be used to turn on the above behaviour when the IB
>                routing is fully defined.

Back a few months ago I felt the same as you. At that time IBTA wasn't
even sure if P_Key should be preserved across IB routers.

But now I hear that P_Key will be preserved across IB routing. There
doesn't seem to be anything else in the IBA TBD list that is required
before we can define an IPoIB link spanning multiple IB subnets.
Whether An IPoIB link crosses IB subnets or not will simply be
determined by the scope bits of the corresponding all-node MCGroup.
(Actually if IB multicast routing works as expected, this may be
unnecessary. The scope bits will always be site-local. An IP subnet
falling completely within an IB subnet is just a degenerate case. In
that case IPoIB link boundary is completely defined by P_Key/Q_Key
only, regardless of IB subnet boundary.)

It's up to the network admin to set up MCGroups with the correct
parameters (e.g. all the MCGroups belonging to same IPoIB link must
be given the same MTU...) in a group of connected IB subnets. Do you
see anything missing that will prevent the above from working?

>
>	** Thus the use of local scope in the mappings 2a/2b above **
>
>
>3b. Link: IP multicast to MGID valid range
>
>        There are some well known IP (v4/v6) multicast groups.
>        Additionally multicast groups are dynamically created in IP
>        networks. An algorithmic mapping for the same is needed. We
>        must keep the following constraints in mind when defining the
>        mapping: 
>
>        * FF02::1 is defined as the all IB nodes multicast address.
>          Thus it is not limited to IP but includes all IB nodes in
>          the IB subnet. Also note that this address is defined as a
>          well-known (transient bit 0) IB multicast address.

I proposed to carve out a chunk from the Q_Key space (0x9B1B0000 to
0x9B1BFFFF) for IPoIB QPs uses. And if you agree to make Q_Key part
of IB MCGroup id, then there is no conflict.

Some of us are trying to get things clarified or amended in this area
through IBTA.

>
>        * All the well-known multicast IPv6 multicast addresses as
>          defined in RFC2375 are defined as the well-known IB Raw mode
>          addresses by the IB specification.
>
>          The MCGroupRecord has a field that indicates whether the
>          group is IB Raw mode group or otherwise.

Yes I know. BUt IBTA/LWG already talks about deprecating the raw
datagram.  Raw IPv6 datagram was invented solely for IPoIB. If we all
agree that UD is a better choice, why keep it?

>
>          The IB specification doesn't indicate anywhere the
>          possibility of having the Raw and Unreliable datagram (UD)
>          share the same MGID. As far as I can tell packets sent to
>          raw datagram MGIDs cannot be received by UD QPs since the
>          packet format is different. The reverse is also true.

Do you still propose to support IPoIB through raw datagram?

>
>        * The IB specification, see the multicast join algorithm,
>          gives the the impression that multiple MGIDs do not exist in
>          a SA. As noted by you, a discussion in IBTA's working group
>          on allowing multiple MCGroupRecords for the same MGID has
>          not progressed. It doesn't seem likely that IBTA will agree
>          to multiple MGID entries with different P_Key or Q_Key
>          entries. 
>
>          [ If such a change is accepted then things will be simpler.
>          This change must also solve the duplication of raw mode/UD
>          mode IB MGIDs. ]

So hopefully you'll support the motion by Sun in IBTA, proposed
originally by me for the ease of the IPoIB MC mapping. But like I
indicated in my previous mail, we can still embed P_Key/Q_Key in MC
addr the proposal is rejected. (There is another issue with the usage
of MC addr itself though.)

>
>        The IB MCGroupRecords can be created by any IB member. Thus it
>        is prudent to avoid any MGIDs, especially since multiple
>        multicast MGIDs may not be allowed, that may be mapped by
>        generic IB nodes. This removes the Raw IPv6 groups and the
>        FF02::1 group from the possible IP to IB mappings.
>
>	We therefore need a different set of MGIDs.
>

Fundamentally to avoid MGID allocation conflicts with other, non-IP
protocols, we need to carve out a chunk from the MCGroup ID space for
IPoIB use only. If IBTA decides that the "MCGroup ID space" includes
Q_Key, we can simply reserve a chunk from the Q_Key space for IPoIB
use only, like I suggested.

Ethernet avoids this problem all together by defining an "EtherType"
field in every frame. The proposal you made to IBTA is along the same
line. If it is adopted by IBTA, perhaps we don't need to worry about
MGID conflict any more.

>
>3c. Link: use of transient bit in MGIDs
>
>        The IB specification borrows the definition of multicast
>        addresses from the IPv6 specification. It therefore states
>        that that MGID is of the format:
>
>	FFxy:<112 bits>
>
>        The flag bits, denoted by x above, are the flag bits 000T. T=0
>        denotes a permanently assigned MGID. The IB specification
>        further clarifies that the well known MGIDs are IB's raw IPv6
>        well known addresses (and FF02::1). It refers to RFCs2373/2375
>        and incorporates all the listed addresses as raw IB's well
>        known addresses.
>
>        This implies that if IPoIB sets T=0 in its mappings it is in
>        contravention of the IB specification.
>
>	** Thus the use of transient bit in MGID mappings ** 
>
>        We could go ahead and use well known addresses but then we are
>        essentially saying that raw IPv6 mode cannot be implemented
>        over the IB subnet on which IPoIB is present.
>
>
>3d: IP multicast to IB MGID mapping
>
>        We are in good agreement here. I had suggested the following
>        mapping for IPv4 addresses at the London meeting (see the
>        presentation slides) :
>
>        FF12::<16-bit unique IP link ID>:<IP multicast/broadcast address> 
>        with the possibility of the 16-bit ID being the P_Key itself. 
>
>> 3) My original proposal didn't cover IPv4 MC mapping. For completion
>> I'll propose one here.
>> 
>> Given an IPv4 multicast address, replace the lower 24 bits of
>> FF02:0:0:401B:: with the lower 24 bits of the IPv4 address. Then fix
>> the scope bits, and embed P_Key/Q_Key if necessary as described above.
>> 
>> E.g. the IB MGID for the all-node IPv4 multicast address in the IPoIB
>> link of the last example above will be FF02:8:CDEF:401B::1
>
>
>        I agree with the use of P_Keys in the mapping and moving the
>        P_Key to the location you indicate rather than to the right
>        edge as I had suggested.
>
>        Introducing the Q_Key in the MGID mapping is not useful
>        though. It adds another startup time configuration parameter.
>        The interface that is coming up can look into its P_Key table
>        and map to the MGID corresponding to the broadcast MGID. It
>        cannot, however, map to the MGID if Q_Key is added to the
>        mapping. This makes the Q_Key another configuration parameter.
>        So my suggestion is to leave Q_Key out of the IP multicast
>        address to IB MGID mappings.

Yes, embedding the Q_Key in MGID makes the lookup more difficult w/o
the knowledge of Q_Key beforehand. If IBTA adopts my proposal of
allowing two MCGroups to use the same MGID, P_Key, but different Q_Keys,
then this problem doesn't exist. When an IPoIB interface is being
configured, it can wildcard the Q_Key field and look for all the
MCGroups with specific MGID and P_Key. If there is only one match, use
the Q_Key from the found MCGroupRecord. If there are multiple matches,
it means that multiple IPoIB links can be connected from the local
interface. It's up to local administration procedure to determine how
many virtual IPoIB interfaces to creat.

Including the Q_Key as part of IPoIB "link" makes it very easy to
reconfigure IP subnet boundary. P_Key provides a "hard" partitioning
mechanism. This makes rekeying both difficult and risky. Q_Key only
provide a "soft" protection and is very easy to reprogram. (It can
all be done by individual host in the OS privilege mode.) E.g. if
servers are grouped according to departmental functions into different
IB partitions that are centrally controlled, each department can still
freely setup/reconfigure its IP subnets using Q_Key.

>
>        Given the above discussion on scope and transient bits the
>        mapping for IPv4 multicast would be:
>
>	FF12:<16-bit P_Key>::< IPv4 multicast address>
>
>        Question: Why do you suggest using only 24 bits of the address
>        when we can use the full 32 bits? 24 bits were used in etheret
>        mapping due to other compulsions.

Somehow I got the impression from rfc3171 that only 224/8 is being
actively used for IPv4 multicast. This leaves only 24 (32-8) bits
unique. I need to recheck.

>
>	Similarly why not use all 1's for broadcast mapping i.e.
>
>	FF12:<16 bit P_Key>:: 255.255.255.255 
>
>	rather than  ::1 as you suggest. Why make it different ?

The concept of a special "broadcast" address is being phased out and
replaced by multicast addresses. ::1 corresponds to the lower 24 bits
of the all-node IPv4 multicast address (224.0.0.1).

We should pick one IPv4 all-node (local broadcast) address for mapping
into the special MGID. Either 255.255.255.255 or 224.0.0.1. I picked
224.0.0.1.

>
>        In the case of IPv6 it is important to avoid the IB's Raw IPv6
>        reserved addresses.
>
>	FF12:3333:<P_Key>:<80 Low order bits of IPv6 multicast addresses>
>
>        The use of 0x3333 instead of a random value such as the Q_Key
>        allows one to immediately know that the MGID is an IP MGID.
>        This should aid in administration of the subnets. Secondly if
>        the conflict with Raw IPv6 MGIDs or other non-IP MGIDs is to
>        be avoided it is easier if a specific range can be set aside
>        for IP.
>
>	Therefore, we could modify the IPv4 mapping to be :
>
>	FF12:3334:<P_Key>::<IPv4 multicast/broadcast address>

I need to think about this.

Jerry

>
>
>Note:   Even if the IBTA/WG is confident that the conflict with raw
>        IPv6 will not become an issue (raw IPv6 is not outlawed by
>        IBTA) the mapping may not be 1 to 1 due to the issues of i)
>        multiple MCGroupRecords with same MGID ii) scope bits.
>
>
>	Thoughts ?	 
>
>Vivek
>
>
>> Detailed discussion - (can be skipped)
>> 
>> 1) The use of both P_Key and Q_Key as a link ID gives the network
>> administrators ample flexibility in setting up arbitrary IPoIB link
>> boundary. My original proposal places a restriction on an IP subnet
>> not spanning more than one IB subnet. This restriction is due to the
>> lack of IB routing definition in IBA 1.0 spec. Now if IBA defines the
>> validity of P_Key as fabric wide, rather just limited to a given IB
>> subnet (i.e. IB routing preserves the P_Key value when forwarding an
>> IB packet), the restriction can be removed with minimal modification
>> as follows.
>> 
>> When an IPoIB link is set up to span multiple IB subnets, the network
>> admin must create an all-node multicast group with a site-local scope
>> (FF05::1 instead of FF02::1) in each of the IB subnet that the IPoIB
>> link covers. When a CA tries to discover a locally attached IPoIB link,
>> it should first search SA for a link-local, all-node MC group with a
>> matching P_Key and Q_Key in the MCGroupRecord database before looking
>> for a site-local one.
>> 
>> The IP to IB multicast mapping must also set the scope bits correctly.
>> That is, if the IPoIB link fits completely in a single IB subnet,
>> the scope bits of a derived IB MC address must be set to link-local.
>> Otherwise they must be set to site-local. (BTW, I'm assuming we don't
>> need to go beyond site-local scope for an IB fabric. But this is
>> debatable.)
>> 
>> 2) For an IPv6 MC address, the I-D proposes to use the same address
>> value (with the scope bits adjusted as described above) as the
>> corresponding IB MGID. The proposal made an assumption that different
>> IB multicast groups identified by the same MGID but different P_Key
>> and/or Q_Key can co-exist in different "IPoIB links" and operate
>> independently from one another.
>> 
>> This is analogous to VLANs in Ethernet where multicast groups by the
>> same Ethernet multicast address are used and operate independently in
>> different VLANs.
>> 
>> It is not clear whether my proposed use of MGIDs described above
>> complies with or violates the current IBA spec. IMO the IB spec does not
>> address the use of MGIDs at all. In a recent IBTA meeting I heard that
>> the majority of members prefer to restrict the use of MGID to be one
>> per IB subnet regardless of the P_Key or Q_Key. Although I don't see
>> any value of such a restriction, the jurisdiction of this belongs to
>> IBTA. If the restriction is approved by the majority, in order to come
>> up with a unique MGID for each IPoIB link, my proposed mapping can
>> be modified simply by embedding the P_Key and Q_Key when creating a
>> MGID. The following is one possibility:
>> 
>> Given an IPv6 MC address of the form FFXX:x1:x2:x3:x4:x5:x6:x7, fix the
>> scope bits first. Then replace x1 with the value of the P_Key, and
>> x2 with the lower 16-bit of the IPoIB Q_Key.
>> 
>> E.g. the IB MGID for the IPv6 all node multicast address FF02::1 of
>> an IPoIB link that spans more than one IB subnet with P_Key = 6,
>> Q_Key = 0x9B1B1234 will be FF05:6:1234::1
>> 
>> The IB MGID for the IPv6 all router multicast address FF02::2 of an
>> IPoIB link that falls within a single IB subnet with P_Key = 8,
>> Q_Key = 0x9B1BCDEF is FF02:8:CDEF::2
>> 
>> 3) My original proposal didn't cover IPv4 MC mapping. For completion
>> I'll propose one here.
>> 
>> Given an IPv4 multicast address, replace the lower 24 bits of
>> FF02:0:0:401B:: with the lower 24 bits of the IPv4 address. Then fix
>> the scope bits, and embed P_Key/Q_Key if necessary as described above.
>> 
>> E.g. the IB MGID for the all-node IPv4 multicast address in the IPoIB
>> link of the last example above will be FF02:8:CDEF:401B::1
>> 
>> Comments?
>> 
>> Jerry
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 24 Oct 2001 00:22:41 -0700 (PDT)

Jerry,

> that the administrator setup the IB multicast group corresponding to

        The drafts define the following as the basis of the IPoIB setup:

		- an all-nodes multicast group must be setup
			 - translates to broadcast group in the case of IPv4
		- all nodes join the all-nodes multicast group
                         - link attributes (Path MTU, P_Key, Q_Key) are
                           thereby distributed to the members 
		- the broadcast domain or the span of the IP subnet
			- constrained by the membership of the broadcast group.
			  the members must have common link attributes
	
	The setup would therefore be:

                A multicast group is setup by the administrator. The
                MCGroup for IPv4 would be (updated mapping from my
                previous mail) FF12:3334:<P_Key>::255.255.255.255.

                All nodes MUST join this. By doing so they get the
                P_Key, Q_Key, and Path MTU among other things (section
                5.0 in ...ipv4-multicast-01.txt and section 3.3 of
                ...ipv4-and-arp-01.txt). 

        I didn't use the term link boundary but is it different from
        defining the link attributes and the broadcast domain ?

        I have, as described above, used the MCGroup, its attributes
        and members to define an abstract link and its span. You are
        focussing on P_Key/Q_Key as the distinguishing Link
        identifiers. This may be because you are focussed on a common
        MGID for all of the possible links e.g. FF02::1 for all
        broadcast/all-nodes multicast traffic. Thus the distinguishing
        factor becomes the P_Key/Q_Key.

        This is no different than what I have said. The MGID and its
        attributes define the link. Thus if the MGID is the same for
        different broadcast domains(I don't think we will succeed in
        that) the link attributes of P_Key/MTU etc. become the primary
        entities that distinguish the link. If the broadcast/all-nodes
        multicast MGIDs themselves are different then the MGID itself
        is a distinguishing factor (along with link attributes).

        I see us in total agreement on this issue. Hopefully I have
        explained the slight distinction that might exist not confused
        the matter.

> of the IBA spec. in this area (as I pointed out in London IETF).

        I meant feedback on my draft (See ipoib archives of
        april/may). I received some interesting feedback and hence
        updated my draft.

> don't know what you mean by "MCGroup does define the link boundary".

        I mean that the MCGroup ie. the all-nodes multicast group that
        would be setup and used to distribute the link attrributes
        constitutes the definition of the broadcast domain. And hence
        the link boundary. In other words the members of the MCGroup
        define the link since they agree on the IP broadcast/all-nodes
        multicast address to MGID mapping; they agree on the MTU; they
        agree on the P_Key and the Q_Key. Or the MGID defines the link
        boundary - since it defines the link attributes such as the
        MTU, P_Key, Q_Key.

> MCGroup. What do you propose to define an IPoIB link?

	Would you agree with the following:

                Link : the abstract 'link' joining the members of the
                       MCGroup corresponding to:

		       FF12:3334:<P_Key>::255.255.255.255 (IPv4)
		       FF12:3333:<P_Key>::1               (IPv6)

                       The P_Key is the Link ID since it
                       differentiates one abstract link from the
                       other. 

		       The link boundary or the broadcast domain is limited by:

			* MCGroup
			* P_Key 

	( see discussion below on Q_Key)

> MGID and thus one IPoIB link per IB subnet?

        My original hope was: A join request includes the P_Key in the
        request. The SM will match the P_Key and return the right MGID
        if there were multiple entries.

        However, the aspect of multiple entries is unclear and hence I
        then proposed (in London, IETF) to use the P_Key as part of
        the multicast mapping. This makes the resolution of multiple
        MGIDs issue in IBTA irrelevant to IPoIB. Additionally, I favor
        this approach since it sounds administratively simpler to me.

        Rather than seeing multiple MGIDs of the same type and then
        deteriming the differences in their attributes it is simpler
        to have a limiting attribute such as the P_Key be part of the
        MGID itself. It will be quite helpful in keeping the SA
        records straight and in troubleshooting.

        One can also special-case the value of 0 in place of P_Key in
        the mappings when there is no need to define multiple IPoIB
        links. In this case the nodes take the P_Key from the
        all-nodes multicast group as the P_Key to use for all IPoIB
        communication. This could be the default case.

> ramification involved P_Key and Q_Key matching.

	That is exactly my  concern too. 

> see anything missing that will prevent the above from working?

        I don't think we are disagreeing here. But we can only work
        with the published specification. If the IBA changes its
        specification in the future, which can be a long drawn
        process, then we can leverage the changes. At present there is
        no IB routing, no definition of IB multicast groups across IB
        subnets, no specification on P_Key preservation across
        subnets, no clear specification on the meaning of
        site/organisation scope bits etc. It may all come to pass some
        day. 

        The IP node when it comes up must determine the MGID to join.
        It needs to map the all-nodes IP multicast to a valid MGID.
        For now it can default to using the local scope. In the future
        if and when IB defines the scope bits that need to be used,
        the mapping to the MGID can use those bits. A local
        configuration parameter can let the IP node know the scope
        bits to use if the default is not the right choice. An
        implementation may choose to by default try all the choices
        and join all the multicast groups it can.

        [ Another issue that needs an answer is : What is the
        advantage that we will gain by having IP subnets span IB
        subnets ? ]

> of IB MCGroup id, then there is no conflict.

        I agree with using the P_Key as part of the mapping. See
        earlier mail. Am not convinced on the use of Q_Key. I don't
        see any additional advantage as yet.

> agree that UD is a better choice, why keep it?

        But it is not an IPoIB issue. If the IBTA says Raw IPv6 will
        not be used and the conflict is allowed then fine. I don't
        think IPoIB can say that Raw IPv6 mode not be used. IB was
        requested to remove Raw IPv6 mode from IB specification if it
        is not be used but the request didn't fly. No one agreed with
        the proposal of removing IB's Raw IPv6 mode. Note that IB's
        Raw IPv6 mode is not IPv6; it is a transfer mode of IB. It is
        part of the IB specification.

> Do you still propose to support IPoIB through raw datagram?

        It has nothing to do with IPoIB. As per IB specification one
        can send IB packets using IB's Raw IPv6 mode.

> of MC addr itself though.)

        I think we are both saying the same thing. I lean toward using
        the P_Key into the MGID mapping. You would go with a similar
        solution if your proposal in IBTA fails.

        So how about defining the rule to include the P_Key in the
        MGID always. If your proposal in the IBTA is accepted the
        P_Key can still be part of the MGID. Nothing is lost since the
        P_Key must be known by the sender anyway. If the proposal is
        rejected then the inclusion of P_Key is necessary anyway.

> use only, like I suggested.

        Anyone can join the MCGroup with the right P_Key. The Q_Key
        will be returned to them. So the Q_Key may offer no
        protection. 

        If the IBTA agrees with your proposal, within the IPoIB
        schedule then we could consider Q_Key. Though as you say below
        the Ethertype problem is more easily solved by the propsal of
        adding the type field in the IB packets. Let us hope that both
        of our proposals are agreed to by the IBTA.

> line. If it is adopted by IBTA, perhaps we don't need to worry about

        Yes, hopefully this proposal goes forward too since it is an
        exact parallel to Ethernet's Ethertype.

> >        over the IB subnet on which IPoIB is present.

        Yes, but since we can avoid such a conflict why not avoid it?
        Secondly, the issue is a little deeper than that. We are in
        effect saying that though the IBTA may have specified and may
        specify some well-known addresses for its use IPoIB will
        override them. After all IPoIB MGIDs are not well-known IB
        MGIDs. This is especially true of IPoIB MGIDs that correspond
        to IP multicast addresses that themselves are not well-known.

> many virtual IPoIB interfaces to creat.

        I don't like the idea of knowing the Q_Key beforehand. Your
        suggestion about using a wild-card Q_Key is preferable.
        However it implies that the IBTA must define a wild-card
        value. The current specification doesn't allow any way for one
        to pass this information to the SM/SA. The Q_Key in the packet
        (which must be a priviliged GSI Q_Key and hence one that the
        node will always use) is not checked by the SA except to
        verify that it is a priviliged Q_Key. The creation of
        MCGroupRecord doesn't allow one to specify any paramter but to
        specify the MGID. I don't know if IB relaxes this restriction
        anywhere else.

	See Pg 726 in release 1.0a.

                "A multicast group can be joined using the SunAdmSet()
                method by specifying Add Record in the edit modifier
                and the MGID (all other fields are zero)".

> freely setup/reconfigure its IP subnets using Q_Key.

        That sounds interesting but do you need such level of
        separation ? Also, as I noted above allowing Q_Key
        differentiation for MGIDs in IB may be problematic.

        Overall, I don't want Q_Key to become another configuration
        parameter. The final solution should include a workable
        wild-card scheme as you have suggested.

> unique. I need to recheck.

No, I don't think that is the case. The address space is still from
224/8 to 239.255.255.255.  Will check though. 

> into the special MGID. Either 255.255.255.255 or 224.0.0.1. I picked

        Choosing the MGID to correspond to broadcast address makes the
        rule simple. The rule states that given an IP address the
        corresponding MGID is:

	FF12:3334:<P_Key>::<IP multicast address>

        In contrast mapping broadcast and all-nodes multicast to the
        same MGID introduces an exception to the rule. The exception
        says that when mapping the IPv4 broadcast address use the IPv4
        all-nodes multicast address for the mapping. At all other
        times use the IP address as is.

        Also, one doesn't need to join the broadcast address but does
        need an explicit join for receiving IPv4 all-nodes multicast
        packets. Why not keep separate MGIDs and keep the separation
        rather than mix it all up?

Vivek
-- 
Vivek Kashyap
IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)