Subject: RE: [Ipoverib] IPoIB multicast

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 22 Apr 2002 01:45:53 -0700 (PDT)

>At 12:05 PM 4/19/2002 -0700, Larsen, Roy K wrote:
>>Jerry and Bill,
>>
>>As Vivek alluded to when he published his latest thinking on multicast and
>>again here, the IBTA is currently considering making some significant
>>changes to multicast semantics and facilities.  I echo Vivek's sentiments on
>>not taking the multicast design any further until the IBTA makes public
>>their multicast updates.

These multicast related issues have been brought to IBTA by a number
of us (including both me and Vivek) for some time, and have been
actively debated upon over there. Since it was Vivek who initiated
this round of discussion here before IBTA closes on their part of
the debate, I certainly have no problem if Vivek and you want to
suspend the discussion. But remember IBTA also looks to IETF for
input on a sensible mutlicast desgin. So the discussion here is
not completely wasted.

>>Otherwise, we are going to have to revisit many of
>>these issues or worse, accept non-optimal solutions because no one wants to
>>rehash this again.  Therefore, I am requesting that this topic be tabled
>>until we have a stable IBA multicast service to design against.
>
>You have a good idea based on what I wrote and Vivek wrote to understand 
>that there will be minimal impact to IP over IB specifications and this can 
>go forward.  All that needs to be done is just describe the IP multicast 
>address encapsulation and the work is essentially done.  The IP router 
>interactions are also a non-issue.  Keep in mind that the IP over IB 
>implementation is what is impacted by any of these IB changes and not the 
>IP over IB protocol so let's finish encapsulation for unicast and multicast 
>and be done with it.

Disagreed. These multicast issues, if not resolved, will affect the
proper function of IP multicast over IB. They are not mere
implementation issues, but they are interoperability concerns.

Let me attempt to summarize the differences between IB multicast
(as defined in 1.0) and IP multicast that cause all the grieves.
Similar discussions have also been documented in the link and
multicast draft.

Note that sometimes when I say "IP multicast" I really mean
"IP multicast over Ethernet". I use the two interchangably because,
although IP is supposed to be link agnostic, Ethernet has been the
dominant link type, and has exerted a great deal of influence over
IP design. The reality is that in order for other link types to
work seamlessly with IP, the easiest way often is to emulate the
model used in Ethernet.

The two main differences between IP/Ethernet and IB multicast are

1. In Ethernet multicast groups don't need to be created before
use, or at least this is the illusion made to the hosts by the
Ether switches. An IP multicast sender conveniently dump pkts
into the corresponding Ethernet multicast address without worrying
whether such address "exists" or not. All the right things happen
automagically behind the switches.

But in IB, a MC group must be created before use. This is not a
biggie for a multicast receiver, as a group can be created by
the first join, and deleted by the last member leaving. But it
causes some grief for the multicast sender, as IP doesn't require
a sender to be a member. Anyone can freely send MC pkts to any
MC group. In that case, should an IB MC group be created for
send only? If yes, how does one determine when to delete a
"send-only" MC group, since IP doesn't give any hint to the
interface driver when a sender is done sending MC pkts? (For
receivers there is explict join/leave call.)

The proposal I made in the link-multicast draft when the IB MC
group corresponding to the target IP MC address doesn't already
exist, is to have the interface driver forward the pkt to the all-
router MC group. If the MC group comes into existence later
(i.e. someone joins as a member), the driver needs to be notified
so it can forward pkts back to the right MC group. This requires
some support from SA. I believe a proposal has been made recently
in IBTA to add this support to SM/SA.

2. Ethernet NICs can be programmed into promiscuous multicast
mode when the NIC will send a copy of every MC pkt to IP.
IGMP/MLD uses this feature conveniently to pick up a copy of
every MC pkt to track MC membership, and to route MC pkts
correctly to other subnets.

Since IB doesn't support a promiscuous multicast mode. The
workaround I proposed is to have the interface driver make a copy
of every MC pkt and send it to the all-router MC group. This is
pretty straightfoward with a small performance penalty. But is
still much more efficient than resorting to broadcast. There is
a small problem in case the router itself also joins a MC group
and will then receive two copies. This case can be adressed with
some more check in the interface driver of a router but since
the case is rare, I'm not sure it's worth the effort and additional
complexity.

Other alternative includes Vivek's first proposal to snoop IGMP/MLD
pkts in the interface driver. This requires driver to understand
IGMP/MLD, and will make it very difficult to maintain (with newer
version of IGMPv1,2,3.../MLD).

Vivek's recent proposal fixed the above problem by moving the work
from the driver to IGMP/MLD themselves. Unfortunately IMO the
chance of requesting link specific changes in a link-independent
module like IGMP is not very good.

Perhaps if IBTA can supply us an equivalent of a promiscuous
multicast mode without bending IBA backwards too much that would
be ideal.  I've heard some discussion along that line is going on.
Let's see what comes out of it.

Jerry

>Mike

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 22 Apr 2002 06:34:18 -0700

>in IBTA to add this support to SM/SA.

To make it quite clear - one cannot send on IB without providing a MLID 
that has been configured into switches.  It does not matter whether this is 
for an all routers or a specific multicast group - this level of 
configuration must be performed.  Hence, trying to target an all routers 
multicast group provides zero benefit since the same work needs to be 
performed for both types so why not do it right the first time.  This is 
why I am strongly opposed to the use of such a technique and view this as 
unnecessary complexity.

I've stated the simplest algorithm:

- Join the IB multicast group
- Get the MLID configured
- Send the packet
- If any one is listening, they will receive it
- If they are not listening at the time of the send, then they will get a 
subsequent send but no attempt should be made to turn this into a reliable 
service as suggested in some e-mails.
- Group changes are tracked by the SA
- The SA will inform a group when it is no longer available

The new IBTA method will be a form of async notification service that will 
allow endnodes to detect when groups are added / deleted which is something 
a couple of us have proposed for quite some time and is now finding 
traction.  If the workgroup focuses on the join process and recognizes that 
a notification method will exist (which is entirely an IB issue with an 
implementation's LLP masking any of this complexity - this is clearly not 
an IP over IB issue), then we can complete this set of specifications.

>the case is rare, I'm not sure it's worth the effort and additional

IB can be configured to see all multicast packets within a given partition 
- this requires one to join all multicast groups.  This is a policy 
decision.  The fact that it is different than Ethernet is irrelevant quite 
frankly.  This is an IP over IB not IP over Ethernet workgroup.  What you 
are describing is in many cases really no different than the class of 
problems associated with LANE.

>module like IGMP is not very good.

There is no reason to add this complexity at all which is why I continue to 
oppose this line of logic.

>Let's see what comes out of it.

One has an promiscuous solution by employing the notification method and 
performing a join operation for all multicast groups associated with a 
particular partition.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 22 Apr 2002 11:13:47 -0700 (PDT)

> input on a sensible mutlicast desgin. So the discussion here is

        I did not ask for the debate to be closed. Roy mistook my
        message. 
  
<snip>

> a sender to be a member. Anyone can freely send MC pkts to any

        IP still is not involved. All the issues of group
        existence/join or not etc. exist in the driver layer. The
        groups in question are the IB level groups.

> receivers there is explict join/leave call.)

        Also, no one is informed beforehand of any intention to
        transmit multicast packets. For receive there is the IGMP/MLD
        report mechanism.

        That is the reason, in the algorithm that I presented, to
        leave the creation with the receivers and use an existing
        mechanism (IGMP/MLD update) to trigger router's discovery of
        the groups. Senders don't create groups and are asked to use
        timers. 

> in IBTA to add this support to SM/SA.

        That won't gel with 1.0 spec and we didn't provide any method
        in link-multicast draft for the necessary notification. Now
        that IBTA is moving to modify the IBTA setup we can certainly
        make use of such a mechanism.

> the case is rare, I'm not sure it's worth the effort and additional

	One of the issues I have with this proposal is that it 
        deliberately introduces duplicates in the fabric. The solution
        also doesn't provide the method by which a sender gets to know
        the MGIDs nor does it say how the router comes to know of the
        new IB multicast Groups.

> module like IGMP is not very good.

        I disagree here. I checked with the routing area AD before
        continuing with the discussion in IPoIB WG. Given the current
        IBA multicast infrastructure this is a good solution.

        Now, if IBTA does bring out the right mechanism we won't have
        to do this.

> Let's see what comes out of it.

        I am not sure how IB architecture can accomodate a promiscuous
        mode - it will require including the router(s) interfaces in
        every multicast tree setup. But other solutions are being
        discussed in IBTA that hopefully will address IPoIB WG's
        concerns. 

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Mon, 22 Apr 2002 11:28:58 -0700

>         groups in question are the IB level groups.

Agreed.

>         the groups. Senders don't create groups and are asked to use

We don't need this involvement as it is the IB LLP driver implementation 
that deals with this and not IP.

>         the MGIDs nor does it say how the router comes to know of the

These are the exact issues I raised a couple of weeks ago and am strongly 
opposed to pursuing such a strategy.

>         Now, if IBTA does bring out the right mechanism we won't have

Exactly.

>         discussed in IBTA that hopefully will address IPoIB WG's

See my e-mail.  Any CA can join all multicast groups for a given partition 
(akin to a specified VLAN) and thus can operate in promiscuous mode.  This 
is all hidden within the IB LLP driver implementation.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 22 Apr 2002 11:30:42 -0700 (PDT)

> I've stated the simplest algorithm:

This is pretty straightforward alright -

	IP level send => IB level (create?) join/send 

	IP level receive(join) => IB level (create?)/join/send/receive

        But it leaves some questiosn - when do we create ? when does a
        sender leave ?

> - Join the IB multicast group

        Only if it exists. We need to define a method for its
        creation. Based on the new IBTA mechanims we will have to
        decide whether anyone can create the IB group or we leave it
        to the receivers and routers.

> service as suggested in some e-mails.

        They need to join first. Secondly, do we allow the group to
        stay around - we still need to suggest a timer (for some idle
        time) at which a sender should leave the group.

> - The SA will inform a group when it is no longer available

        Inform whom ? Do we need to define a method that lets the SA
        know about the nodes to be informed or is it implicit in some
        way? We need to answer these questions in the IBA though.

> problems associated with LANE.

        I agree with Mike here - this is IPoIB and irrelevant to IPoE.
        We need to ensure that the IP interface doesn't have to change
        but don't have to ensure that the IB driver looks the same as
        ethernet driver.

> oppose this line of logic.

        Only if the new IBTA changes are brought in. Without any IBTA
        changes there is no alternative other than use broadcast
        always or the dual-transmit of multicast packets suggestion.