Subject: RE: [Ipoverib] IPoIB multicast forwarding

From: "Larsen, Roy K" <[email-masked]>
Date: Wed, 10 Apr 2002 16:28:09 -0700

<snip>

>         It is compelling to you to broadcast everything and yet you
>         are bothered that a sender will broadcast(though it won't) its
>         packets if it can't IB_join the relevant group ???
> /rkl
> I am bothered by complexity for arguably marginal benefit.  I am bothered
> that the IPoIB encapsulation protocol is requiring existing OS and user
mode
> protocol implementations to be modified to fit our needs (DHCP, now IGMP,
> and Yaron has brought up VRRP).  We should be striving to keep our impact
to
> link encapsulation.  When given the choice between a compromise on the
usage
> of some link capabilities and requiring changes in existing OS stacks and
> network applications, I will always come down on the side of compromise.
> For companies that don't supply a complete vertical solution, supplying
> IPoIB related products that require modifications to the core OS and
> application environment leave them at a distinct disadvantage.  This type
of
> environment does not lend itself to the wide adoption of IBA for general
> networking.
> 
> Roy
> rkl/

        Can you point to the exact 'complexity' that bothers you in
        the proposed solution? What 'core' OS or application setup in
        your implementation is adversely effected ?

/rkl
Certainly there is complexity in managing IBA multicast groups.  It requires
statefull transmits.  This will be amplified if the LWG makes it necessary
to do "join to send" operations.  But, if we could faithfully support
multicast, I guess that's the price we pay.  But then we add the requirement
that says transform a multicast request from layer 3 into an all-routers
multicast if the group doesn't exist and, on a device type basis, have the
interface join/leave multicast groups on multicast routers when the IGMP
implementation receives IGMP/MLD reports.  If it could all go away with
broadcast-always, I'd tend to lean that way.
rkl/

        The proposal is taking care to keep the impact minimal while
        utilising the benefit offered. It effects only those that want
        to implement IPoIB and not others. It does not effect the
        protocols themselves (DHCP/IGMP are left untouched).

/rkl
When layer 3 transmit requests are not performed consistently, we are
technically changing the wire protocol and something I think we should not
disregarded lightly.  But for the sake of this discussion, my concern is not
about changing the wire protocols, it is requiring changes to their
implementation.  As you stated in a previous reply:

        "The IGMP/MLD report lets the router know the IP multicast
         groups that exist on an interface. I am suggesting that the
         router then give a command to the interface to add the
         corresponding IB group to its list. This command, at the IPoIB
         driver translates into an IB_join for the corresponding IB
         group."

So, we need IPoIB aware DHCP clients and now IPoIB aware IGMP
implementations.  We'll see what happens with VRRP.  It's not that any one
thing is a big deal, its just that the slope is getting pretty slippery.  My
vision is one that does not require a vertically aware IPoIB operating
environment.  Just an OS-standard network link driver that encapsulated IP
frames.  I am a realist and know that it is not quite that simple.  But I'll
tend to be a voice that pushes the group in that direction.
rkl/

        By dumbing down the solution to a broadcast link we will loose
        all the benefit that could have accrued to us. I would rather
        not have unneccesary broadcasts take away the bandwidth from
        my storage data. I also would rather that the unnecessary
        broadcasts do not fill my receive buffers and take up CPU time
        in filtering the irrelevant packets. I would rather that the
        IB switches don't get overwhelmed replicating packets out
        every port for broadcast but rather do it only for the links
        that really need such replication - implying use separate
        multicast groups as far as possible. Not all considerations
        need to be software specific; we also need to look at other
        issues.

/rkl
Technically, those are all valid concerns.  I don't have a good feel for how
heavily multicast is used in the data center, but of course it will all be
moot if the data center admin finds it far simpler, from an OS integration
and compatibility standpoint, to deploy emerging Ethernet technologies
instead of IBA.  This is my paramount concern.

Roy
rkl/
        As I said, I did ask about the impact on the routers and was
        told it was an acceptable requirement. 

Vivek
<snip>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 11 Apr 2002 00:31:40 -0700 (PDT)

> to do "join to send" operations.  

        Every media driver(including Ethernet) keeps a list of
        multicast groups if the device can't handle it all by itself.
        There is nothing special about IPoIB driver keeping track of
        the groups.

        No complexity is added by 'join to send' or no 'join to send'
        since the driver still needs to know the MLID to use for a
        particular MGID. It needs to keep track of the MLIDs.
        We write IPoIB to the specification that is available. If it
        says join to send is required then we do that, if it says join
        is not required then we don't. It in no way effects the
        discussion.   

But, if we could faithfully support
> multicast, I guess that's the price we pay.  
 
        We are supporting multicast faithfully if my proposal is
        implemented. That is the idea. If we use a comon broadcast
        group - we no longer support multicast.

But then we add the requirement
> multicast if the group doesn't exist and, on a device type basis, have the

        That is an incorrect statement. 

        There are two layers IP and IB. Please note - the IP packets
        are not modified. When the IP group is joined the IPoIB driver
        needs to create the corresponding IB group or join the
        corresponding IB group. This mapping is specified in
        draft-ietf-ipoib-link-and-multicast-00.txt. 

        The same thing happens in Ethernet. The ethernet driver is
        asked to join the corresponding ethernet group. It programmes
        the interface accordingly. In the case of InfiniBand the SM/SA
        is contacted.

        If the IB group is created then the IGMP packet is also sent
        to *IB group* corresponding to the IP all-routers group. 

> disregarded lightly.  

	See above.

> implementations.  We'll see what happens with VRRP.  It's not that any one

        We have gone ahead and chosen a radically different technology
        to support IP on. So it is unreasonable to expect media aware
        components to work unchanged.

> instead of IBA.  This is my paramount concern.

        It is better to support these technical reasons for they are
        more likely to ward off any assault by other technologies than
        a dumbed down solution.

        I don't think the complexity of InfiniBand is in any way
        affected by reducing the multicast funtion. InfiniBand is
        complex. Reducing some function is likely to have more adverse
        an effect on market/administrator sentiment than a delta
        change in 'complexity'.

Vivek 

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Thu, 11 Apr 2002 07:19:38 -0700

Let's make this simple:

- IP is one type of non-IB based layer 3 functionality that may operate 
over an IB fabric.  Given the need to support Ethertype, the solutions 
proposed within this workgroup should be applicable to all layer 3 
protocols to avoid having to rework everything anew for each layer 3 
protocol that is desired to be supported.

- If a non-IB layer 3 router is attached to an IB subnet, then it must 
communicate to the subnet's SA that it is present.  Note: a layer 3 router 
is not an IB router; it is an IB TCA that provides layer 3-focused 
connectivity to the non-IB based fabric attached to the other side (this is 
no different than an Ethernet or Fibre Channel attached TCA other than it 
is IB multicast capable as a base requirement).

- The TCA uses the IB multicast management services to determine all of the 
IB multicast groups that are operational within the fabric.  While it is 
outside the current IB specification, the IB management software 
implementation will have to include a field or fields to identify what the 
layer 3 or application service is associated with a given IB multicast 
group (e.g. for a group supporting IP protocols, it will need to also 
supply the list of IP multicast groups that are bound to this IB multicast 
group and some method must be defined to create such bindings - this could 
be done within this workgroup, within the IBTA, or left as a value add for 
IB management suppliers).

- Upon determining what multicast groups support its layer 3 services, the 
TCA will contact the SA to have the following occur:

(1) For each partition supporting the stated layer 3 traffic, a multicast 
tree is expanded to include this router.  This includes programming the 
switch P_Keys, the TCA P_Key table, and all relevant TCA attributes.

(2) The layer 3 TCA always performs a full join of the layer 3 focused IB 
multicast groups (i.e. to send and receive packets).  If there is no one 
within the IB fabric that wishes to receive packets, then the IB switch 
acts as a filter mechanism since the layer 3 TCA just forwards all unicast 
/ multicast traffic from the external fabric to the appropriate IB set of 
endnodes based on its decoding and mapping of the layer 3 headers.  Note: 
The layer 3 TCA operates in promiscuous mode for all of its communication.

(3) For Unicast traffic, the TCA participate in the appropriate layer 3 
protocols to advertise its services as a forwarding agent for layer 3 (this 
requires no additional specification from this workgroup to enable).

- All of the discussion of the layer 3 protocols and how they discover 
things from the outside world is rather irrelevant as this is the TCAs job 
to deal with and not the IBTA or this workgroup's responsibility.

Mike