Subject: Re: [Ipoverib] IP encapsulation Design Team

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 13 Oct 2001 19:22:57 -0700 (PDT)

>I would love to see the answers to the following questions (these are Jerry
>Chu's questions so maybe he would like to lead the discussion)
>1) Define the link boundary for an IPoIB link
>2) Determine the per-link IB attributes
>3) Determine how to map IP multicast to IB multicast on each IPoIB link
>4) Determine the IPoIB link-layer address (or "hardware address" in ARP's
>term) format

Let me start first with my answers to the above questions. I encourage
others to do the same. This way we can zoom in on the key differences
among different proposals/ideas, and hopefully resolve them quickly.

The I-D "draft-hkchu-ipoib-ipv6oib-00.txt" I submitted in July
contains all the details. Note that although the I-D focuses on
IPv6, most of the content can be applied equally to IPv4. Also
the I-D covers unicast as well as multicast. (I didn't think
it was necessary to separate them into different I-Ds.)

1) link boundary for an IPoIB link

A link boundary is defined by using MCGroupRecords through Subnet
Administration (SA). P_Key and Q_Key combined serve as a "link ID".
All the CAs belonging to the same all-node multicast group (FF02::1)
defined through a MCGroupRecord with a given link ID form an IPoIB
link. (See more detailed discussion below.)

2) per-link IB attributes

MTU, P_Key, Q_Key

3) IP multicast address to IB multicast address mapping

Identity function for IPv6 with possible adjustment on the scope bits.
(See more discussion below.)

4) IPoIB link-layer address (or "hardware address" in ARP's term) format

The link-layer address will contain QPN and GUID.

----------------------------------------------------------------
Detailed discussion - (can be skipped)

1) The use of both P_Key and Q_Key as a link ID gives the network
administrators ample flexibility in setting up arbitrary IPoIB link
boundary. My original proposal places a restriction on an IP subnet
not spanning more than one IB subnet. This restriction is due to the
lack of IB routing definition in IBA 1.0 spec. Now if IBA defines the
validity of P_Key as fabric wide, rather just limited to a given IB
subnet (i.e. IB routing preserves the P_Key value when forwarding an
IB packet), the restriction can be removed with minimal modification
as follows.

When an IPoIB link is set up to span multiple IB subnets, the network
admin must create an all-node multicast group with a site-local scope
(FF05::1 instead of FF02::1) in each of the IB subnet that the IPoIB
link covers. When a CA tries to discover a locally attached IPoIB link,
it should first search SA for a link-local, all-node MC group with a
matching P_Key and Q_Key in the MCGroupRecord database before looking
for a site-local one.

The IP to IB multicast mapping must also set the scope bits correctly.
That is, if the IPoIB link fits completely in a single IB subnet,
the scope bits of a derived IB MC address must be set to link-local.
Otherwise they must be set to site-local. (BTW, I'm assuming we don't
need to go beyond site-local scope for an IB fabric. But this is
debatable.)

2) For an IPv6 MC address, the I-D proposes to use the same address
value (with the scope bits adjusted as described above) as the
corresponding IB MGID. The proposal made an assumption that different
IB multicast groups identified by the same MGID but different P_Key
and/or Q_Key can co-exist in different "IPoIB links" and operate
independently from one another.

This is analogous to VLANs in Ethernet where multicast groups by the
same Ethernet multicast address are used and operate independently in
different VLANs.

It is not clear whether my proposed use of MGIDs described above
complies with or violates the current IBA spec. IMO the IB spec does not
address the use of MGIDs at all. In a recent IBTA meeting I heard that
the majority of members prefer to restrict the use of MGID to be one
per IB subnet regardless of the P_Key or Q_Key. Although I don't see
any value of such a restriction, the jurisdiction of this belongs to
IBTA. If the restriction is approved by the majority, in order to come
up with a unique MGID for each IPoIB link, my proposed mapping can
be modified simply by embedding the P_Key and Q_Key when creating a
MGID. The following is one possibility:

Given an IPv6 MC address of the form FFXX:x1:x2:x3:x4:x5:x6:x7, fix the
scope bits first. Then replace x1 with the value of the P_Key, and
x2 with the lower 16-bit of the IPoIB Q_Key.

E.g. the IB MGID for the IPv6 all node multicast address FF02::1 of
an IPoIB link that spans more than one IB subnet with P_Key = 6,
Q_Key = 0x9B1B1234 will be FF05:6:1234::1

The IB MGID for the IPv6 all router multicast address FF02::2 of an
IPoIB link that falls within a single IB subnet with P_Key = 8,
Q_Key = 0x9B1BCDEF is FF02:8:CDEF::2

3) My original proposal didn't cover IPv4 MC mapping. For completion
I'll propose one here.

Given an IPv4 multicast address, replace the lower 24 bits of
FF02:0:0:401B:: with the lower 24 bits of the IPv4 address. Then fix
the scope bits, and embed P_Key/Q_Key if necessary as described above.

E.g. the IB MGID for the all-node IPv4 multicast address in the IPoIB
link of the last example above will be FF02:8:CDEF:401B::1

Comments?

Jerry

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 18 Oct 2001 11:38:56 -0700 (PDT)

> it was necessary to separate them into different I-Ds.)

For an alternatve view the above issues are discussed in detail in
draft-kashyap-ipoib-ipv4-and-multicast-00.txt (submitted in April) and
-01.txt (submitted in May) after receiving feedback on this mailing
list. draft-kashyap-ipoib-ipv4-and-arp-01.txt builds on the multicast
document. The same ideas were applied to the ipv6 case in the
draft-kashyap-ipoib-ipv6-over-infiniband-01.txt. Please see the above
drafts for details.

I'll concentrate on the first three issues raised in your mail and
list the common grounds between our drafts and subsequent suggestions.
It seems that we agree on all but for minor differences. The following
discussion applies to IPv4 and IPv6 equally. The differences are
called out specifically.

> Identity function for IPv6 with possible adjustment on the scope bits.

1. Link : distribution of attributes

        The MCGroup does define the link boundary. It also serves the
        purpose of distributing the link attributes to the hosts that
        join the MCGroup.

        Thus any of the nodes that joins the MGID corresponding to
        all-nodes IP address receives the three important link
        attributes: MTU, P_Key and Q_Key. There are other link
        attributes but the process for their determination may be
        different. 

        I don't think we can use the MGID FF02::1 in the mappings.
        Furthermore we can use Q_Key along with P_Key as the link ID
        but Q_Key is not useful when used in IP multicast to MGID
        mappings. See section 3d for a detailed discussion.

2. Link: IP multicast to IB MGID mapping
	(See below for detailed discussion.)

2a. IPv4 mapping

	Broadcast address maps to:

	FF12:3334:<16-bit P_Key>::255.255.255.255
	FF12:3334:<16-bit P_Key>::<IP multicast address>

                * usage of P_Key provides separation of broadcast
                domains even if IB doesn't allow multiple
                MCGroupRecords with same MGIDs.

		* usage of separate MGIDs aids in administration.

                * 0x3334 clearly indicates and IPv4 MGID. Will help
                with administration, trouble-shooting.

2c. IPv6 mapping 

	FF12:3333:<P_Key>:<80 Low order bits of IPv6 multicast addresses>

Detailed discussion:
--------------------

3: Link : IP multicast to IB MGID mapping

3a. Scope bits

        The solution in this regard is effected by the answer to the
        following questions :

        Should IP subnets span IB subnets ? What are the benefits ?
        Why is IP subnetting not enough ?

        There was a discussion on this issue in August but in my view
        no strong benefit was put forward. So I lean toward keeping IP
        subnets within IB subnets. Thoughts ?

        Even if we can identify the benefits a final solution cannot
        be implemented unless IB routing is defined, P_Keys across IB
        subnets are well-defined, and setting of IB multicast groups
        across IB subnets is well defined.

	So I agree with the following scheme:
		
                Define IPoIB subnets to be within IB subnets at this
                time. When the IB infrastructure is complete the
                implementations can be upgraded along the lines you
                suggest. 

                Let the interface try and join the multicast address
                with the local scope. If it doesn't succeed in finding
                the MGID then try the global scope.

                The implementations can have a local variable that
                can be used to turn on the above behaviour when the IB
                routing is fully defined.

	** Thus the use of local scope in the mappings 2a/2b above **

3b. Link: IP multicast to MGID valid range

        There are some well known IP (v4/v6) multicast groups.
        Additionally multicast groups are dynamically created in IP
        networks. An algorithmic mapping for the same is needed. We
        must keep the following constraints in mind when defining the
        mapping: 

        * FF02::1 is defined as the all IB nodes multicast address.
          Thus it is not limited to IP but includes all IB nodes in
          the IB subnet. Also note that this address is defined as a
          well-known (transient bit 0) IB multicast address.

        * All the well-known multicast IPv6 multicast addresses as
          defined in RFC2375 are defined as the well-known IB Raw mode
          addresses by the IB specification.

          The MCGroupRecord has a field that indicates whether the
          group is IB Raw mode group or otherwise.

          The IB specification doesn't indicate anywhere the
          possibility of having the Raw and Unreliable datagram (UD)
          share the same MGID. As far as I can tell packets sent to
          raw datagram MGIDs cannot be received by UD QPs since the
          packet format is different. The reverse is also true.

        * The IB specification, see the multicast join algorithm,
          gives the the impression that multiple MGIDs do not exist in
          a SA. As noted by you, a discussion in IBTA's working group
          on allowing multiple MCGroupRecords for the same MGID has
          not progressed. It doesn't seem likely that IBTA will agree
          to multiple MGID entries with different P_Key or Q_Key
          entries. 

          [ If such a change is accepted then things will be simpler.
          This change must also solve the duplication of raw mode/UD
          mode IB MGIDs. ]

        The IB MCGroupRecords can be created by any IB member. Thus it
        is prudent to avoid any MGIDs, especially since multiple
        multicast MGIDs may not be allowed, that may be mapped by
        generic IB nodes. This removes the Raw IPv6 groups and the
        FF02::1 group from the possible IP to IB mappings.

	We therefore need a different set of MGIDs.

3c. Link: use of transient bit in MGIDs

        The IB specification borrows the definition of multicast
        addresses from the IPv6 specification. It therefore states
        that that MGID is of the format:

	FFxy:<112 bits>

        The flag bits, denoted by x above, are the flag bits 000T. T=0
        denotes a permanently assigned MGID. The IB specification
        further clarifies that the well known MGIDs are IB's raw IPv6
        well known addresses (and FF02::1). It refers to RFCs2373/2375
        and incorporates all the listed addresses as raw IB's well
        known addresses.

        This implies that if IPoIB sets T=0 in its mappings it is in
        contravention of the IB specification.

	** Thus the use of transient bit in MGID mappings ** 

        We could go ahead and use well known addresses but then we are
        essentially saying that raw IPv6 mode cannot be implemented
        over the IB subnet on which IPoIB is present.

3d: IP multicast to IB MGID mapping

        We are in good agreement here. I had suggested the following
        mapping for IPv4 addresses at the London meeting (see the
        presentation slides) :

        FF12::<16-bit unique IP link ID>:<IP multicast/broadcast address> 
        with the possibility of the 16-bit ID being the P_Key itself. 

> link of the last example above will be FF02:8:CDEF:401B::1

        I agree with the use of P_Keys in the mapping and moving the
        P_Key to the location you indicate rather than to the right
        edge as I had suggested.

        Introducing the Q_Key in the MGID mapping is not useful
        though. It adds another startup time configuration parameter.
        The interface that is coming up can look into its P_Key table
        and map to the MGID corresponding to the broadcast MGID. It
        cannot, however, map to the MGID if Q_Key is added to the
        mapping. This makes the Q_Key another configuration parameter.
        So my suggestion is to leave Q_Key out of the IP multicast
        address to IB MGID mappings.

        Given the above discussion on scope and transient bits the
        mapping for IPv4 multicast would be:

	FF12:<16-bit P_Key>::< IPv4 multicast address>

        Question: Why do you suggest using only 24 bits of the address
        when we can use the full 32 bits? 24 bits were used in etheret
        mapping due to other compulsions.

	Similarly why not use all 1's for broadcast mapping i.e.

	FF12:<16 bit P_Key>:: 255.255.255.255 

	rather than  ::1 as you suggest. Why make it different ?

        In the case of IPv6 it is important to avoid the IB's Raw IPv6
        reserved addresses.

	FF12:3333:<P_Key>:<80 Low order bits of IPv6 multicast addresses>

        The use of 0x3333 instead of a random value such as the Q_Key
        allows one to immediately know that the MGID is an IP MGID.
        This should aid in administration of the subnets. Secondly if
        the conflict with Raw IPv6 MGIDs or other non-IP MGIDs is to
        be avoided it is easier if a specific range can be set aside
        for IP.

	Therefore, we could modify the IPv4 mapping to be :

	FF12:3334:<P_Key>::<IPv4 multicast/broadcast address>

Note:   Even if the IBTA/WG is confident that the conflict with raw
        IPv6 will not become an issue (raw IPv6 is not outlawed by
        IBTA) the mapping may not be 1 to 1 due to the issues of i)
        multiple MCGroupRecords with same MGID ii) scope bits.

	Thoughts ?	 

Vivek

-- 
Vivek Kashyap
IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)