Subject: [Ipoverib] Bridging of IP over IB

From: "Roger Ronald" <[email-masked]>
Date: Fri, 11 Jan 2002 14:27:20 -0600

Howdy,

I'll apologize for subscribing and immediately diving in to a heavy
duty topic. I've noticed text in the IP over IB Architecture draft and 
IPv4 over IB drafts that imply that bridging/IB routing of IP over IB is 
stuck until the IBTA makes changes. I don't believe that this needs to be true.

In "IB routing" between IB subnets, the same issues arise as with 
bridging IP over IB to Ethernet. While it's certainly not easy to do
this IB routing or Ethernet bridging, I don't believe that it's all that
different than other bridging problem. There are a few spec issues
that need resolution, but these aren't too tough. I'm convinced that
there is a need to allow such bridging.

To start, I will postulate a bridge that joins an Ethernet and an IB
fabric. To simplify, my hypothetical bridge will only be concerned with
IP traffic. To move traffic, it will rely on MAC layer addresses. 

Each packet that is forwarded through the bridge must have source 
and destination addresses replaced with addresses appropriate to
the destination side's address format (Ethernet devices cannot
be changed to recognize anything but 48 bit ULAs and IB devices
probably cannot be persuaded to deal with addresses other than
LIDs or GIDs. These addresses must be assigned by the bridge and 
remembered for subsequent use. They must be unique and have 
no possibility of colliding with addresses already in the address domain.

The fact that each side of the bridge has a well known broadcast 
(or multicast) address and a protocol for learning addresses 
allows resolving addresses so that the address resolution is known
to the requestor AND an address mapping can be established in the bridge.

Given address translation and packet format translation in the 
bridge for the MAC header, the balance of the bridging problem
is pretty well covered in the IEEE 802.1d spec.
Despite the address swizzling, the Spanning Tree Algorithm and
Protocol seems applicable and usable to prevent topology
loops (although there might need to be an extension to prevent
multiple IB subnets from using the same 48 bit ULAs and 
GIDs for the proxied addresses).

I have some additional input on the direct text of the two affected 
IP over IB internet drafts, but this email is already too long. I'll save them
for a bit. I will say that the only protocol changes to these documents
that I envision involves increasing the requirement for use of the GRH 
to cover bridged messages as well as the multicast address.

Roger Ronald
VIEO, Inc.

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 11 Jan 2002 13:03:43 -0800 (PST)

Roger,
        Bridging/proxying for the subnet beyond will work but that is
        not the same as stating that IP subnets span multiple IB
        subnets. It is a specialised solution.

        To accommodate exactly what you have just said I have included
        the GID in the link layer address (see the slides I presented
        at IETF, http://www.geocities.com/vivk36/IETF_files/ipoib_52_ietf.ppt) 
        or see the first message of the thread 'IPoIB
        architecture/encapsulation'. 

        The 'IPoIB-bridge' responds to ARP requests with its own LID
        but the remote node's GID. Note that it is not an IB router
        that is responding though you can always put IPoIB component
        in your device. The IB router is the 'official' bridge between
        the IB subnets.

        The source can then send the packet out normally with the
        assuarance that the packet will be received by the peer after
        tunneling through this bridge. The use of GRH thus becomes a
        configuration parameter, set by the administrator, to ensure
        that the IPoIB stacks use the full GRH. This bridge will have
        to keep track of the multicast groups across the two subnets
        and pass the packets through on either side. The details are
        not in the domain of the IPoIB specificaiton. It all is an
        implementation and subnet setup choice of the administrator.

        Such a solution allows those that want across the IB subnet
        solutions without making things difficult for those that would
        just stay within an IB subnet. Of course, we could always
        define the MAC addresses as having a variable length - if 6
        bytes(48 bits) then it is LID,capability flags and the QPN. If
        22 bytes then it is LID+capability flags+QPN+GID.

        For a generic implementation we do need the necessary IB
        infrastructure for multicast support. Frankly though, I have
        not seen any compelling reason for IP to span multiple
        subnets. However, if it can be done we can do it when IB
        provides the necessary hooks. 

	Please do send your comments on the drafts.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: "Roger Ronald" <[email-masked]>
Date: Fri, 11 Jan 2002 15:55:33 -0600

----- Original Message ----- 
From: "Vivek Kashyap" <[email-masked]>
To: "Roger Ronald" <[email-masked]>

<snip> Please do send your comments on the drafts.
> Vivek

1) In your IPv4 over IB draft, section 3.1, I suggest deleting the last two paragraphs:

       "The process is complex and unnecessary. The additional
        complexity introduced by IPv4 subnets spanning IB subnets
        gains very little if anything.

        IPv4 subnets therefore MUST be fully contained within an IB
        subnet."

2) In the ARP hardware address format, (section 3.2.2), I would prefer GID to GUID.
This would change 3.2.2.4 as well (the description of GUID). GID is more
complete, if you are crossing IB subnet boundaries the additional data is
useful (but not essential since you can capture the inbound port #).

3) In section 4.0, the text that reads:
        "The InfiniBand specification requires the use of GRH when
        multicasting. A unicast packet need not include the GRH. It is
        RECOMMENDED that a GRH be included in the interest of symmetry.
        It is RECOMMENDED that for unicast packets the destination GID
        be formed from the GUID in the ARP cache and the local IB
        subnet prefix. The GID at GID index 0 MUST be used as the
        source GID.
should be changed to either require the GRH at all times (my favorite) 
or changed to read:
       "The InfiniBand specification requires the use of GRH when
        multicasting. Any unicast packet that will be bridged or IB
        routed also requires the GRH. A unicast packet destined for the local
        IB fabric need not include the GRH. In the interest of symmetry, 
        it is RECOMMENDED that a GRH be included for unicast packets 
        destined for the local  IB fabric by forming the destination GID 
        from the GUID in  the ARP cache and the local IB subnet prefix. The 
        GID at  GID index 0 MUST be used as the source GID."

4) In the IP over IB Architecture document, section 3.3 reads:
       "3.3 IP subnets across IB subnets ?

        Some implementations may desire to support multiple clusters
        of machines in their own IB subnets but otherwise part of a
        common IP subnet. For such a solution the IB specification
        needs multiple upgrades:

        1) A method for creating IB multicast GIDs that span multiple
           IB subntes. The partition keys and other parameters need to
           be consistent across IB subnets.

        2) Develop IB routing protocol to determine the IB topology
           across IB subnets.

        3) Define the process and protocols needed between IB nodes
           and IB routers

        Until the above conditions are met it is not possible to
        define IPoIB subnets that span IB subnets. The IPoIB
        architecture however is capable of providing IP subnets across
        IB subnets if the underlying IB fabric provides the
        infrastructure.

I would delete the above section or (at least) remove the text that
claims the solution must be provided by changing the IBTA
specification. I don't believe that the IB specification
provides the answers, but I don't think they prevent a solution
either. The IETF or the IEEE could choose to take this on or a
proprietary solution could work. 

As to list item #1, there needs to be a method for supporting multicast 
across IB subnets, but partition keys and other parameters might 
NOT be consistent. Instead, they might be substituted as appropriate 
by a bridging device.

The first sentence of that last paragraph sounds like a challenge :-)

RR

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 11 Jan 2002 14:06:41 -0800 (PST)

>         IPv4 subnets therefore MUST be fully contained within an IB

        The draft is no longer under consideration. However, the
        process would be quite a bit difficult. In the newer draft
        there in no discussion of this issue except in the
        architecture draft that we just discussed.

> 2) In the ARP hardware address format, (section 3.2.2), I would prefer GID to GUID.

	See my previous mail. I have done that.

>         GID at  GID index 0 MUST be used as the source GID."

        IT has been decided to leave it to the implementation to use
        or not use GRH. However, I would like to recommend using the
        GID formed by the IB subnet prefix and the GUID in the ARP/ND
        packets. 

> proprietary solution could work. 

	See my earlier response to your mail.

> NOT be consistent. Instead, they might be substituted as appropriate 

        A bridging device is a non-IB solution. IP over IB is
        concerned with the IB specification. In IB the P_Keys etc.
        have to be kept consistent.

Vivek