Subject: Re: [Ipoverib] remaining multicast issues - one more try

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 6 Jun 2002 20:09:05 -0700 (PDT)

>> A couple of weeks ago I asked IBTA for a summary to the IPoIB
>> reflector on the changes likely to come as part of IBA 1.1
>> spec later this year that will affect IPoIB. Unfortunately,
>> the request was declined because IBTA rules forbid any 1.1
>> info to go outside of member companies at this point. But
>> based on my knowledge of their progress I felt comfortable to
>> move the current drafts forward without risking incompatibility
>> with IBA 1.1. If the latter does happen, we can always amend
>> the drafts then.
>> 
>> But yesterday I heard from David Brean of IBTA/MGTWG that the
>> item 3
>> 
>> a facility to allow the software emulation of the promiscuous
>> multicast mode
>> 
>> remains unresolved within IBTA. Vivek also raised some concern
>> too. IMO it is probably not worth holding the drafts for a single
>> issue. But I decided to give it another try to resolve this issue
>> in the next few days. If we can't reach a consensus before next
>> Friday, I'd suggest we move the rest of the drafts forward
>> leaving this issue behind to be addressed later.
>
>        Are you suggesting that we define the algorithm based on
>        points (1) and (2) ignoring (3) or are you suggesting that we
>        define IPoIB multicast based on the current IB specification ? 

Confused. I laid down three known and mutually exclusively proposals
and asked for people's opinion. Personally I prefer (3) because
I heard from some members of IBTA that (3) is not difficult at all,
and it makes promiscuous MC mode a non-issue for IPoIB. If we reach
consensus on (3) then we need to apply persuasion power to IBTA.

>
>
>> 
>> Let me restate that the promiscuous multicast mode is not only
>> required for the proper operation of IP group mamagement protocol,
>> it is also how a sender's multicast pkt gets picked up by an IP
>> multicast router. The details are described in
>> 
>> rfc1112 -
>> "Note that the multicast routers receive all IP multicast
>> datagrams, and therefore need not be addressed explicitly..."
>> 
>> rfc2710 -
>> "For each interface over which the router is operating the MLD
>> protocol, the router must configure that interface to listen to all
>> link-layer multicast address that can be generated by IPv6
>> multicasts..."
>> 
>> IBA 1.0 spec. does not provide a promiscuous multicast mode in
>> hardware or allow one to be emulated in software. So far there
>> have been three proposals to address this issue:
>> 
>> 1. Require the IPoIB driver to send a copy of every outgoing
>> multicast pkt to the all router multicast group. The algorithm
>> looks like this: for a pkt destined to a multicast group M
>> 
>> 	if (M doesn't exist)
>> 		send pkt to all_router
>> 	else if (M != all_router)
>> 		send pkt to all_router;
>> 		send pkt to M;
>> 	else send pkt to M;
>> 		
>> 2. Modify IP group management protocol module to do some extra
>> work in case the link type is IB. An IP router will explicitly
>> join any new multicast group that comes into existence. The
>> router needs to figure out when to leave too (details from
>> Vivek).
>> 
>> 3. Ask IBTA to add a facility to allow promiscuous multicast
>> mode to be emulated in software.
>
>        And a 4th would be to use only one multicast group at the IB
>        layer for all IP multicast packets such as use broadcast_GID
>        for all IP multicast. Not a good solution but an option.

I don't think it is adequate for a multicast draft to advocate
something anti-multicast.

>
>> 
>> 1. above seems to be the least painful. But it requires every
>> node to be capable of sending pkts to all_router multicast
>> group.
>
>        The most bothersome issue of this option is the deliberate
>        insertion of duplicates into the network. A big no-no in my
>        opinion. 

Still don't understand why this is a big no-no. In a subnet of 100
nodes and 2 listeners of MC group A, a multicast to A of 2 nodes
plus a separate send to router R is still a vast improvement over
broadcast over 100 nodes.

But like I said I prefer (3) if it doens't cost much to IBA.

>
>> 
>> The biggest problem with 2. IMO is that it requires changes to
>> existing IP code in every version of group management code
>> including IGMPv1, IGMPv2, MLD..., etc, and the change doesn't
>> seem trivial. It may present a big obstacle for many OSes to
>> request IB only changes in generic IP code.
>
>        That is not the case.

What is not the case? "Requiring changes to existing IP code" or
"the change doesn't seem trivial" or "a big obstacle for OSes"?
The latter two can be very subjective.

>
>        In a generic implementation, the routing daemon (mrouted/gated
>        for example) updates the kernel routing and multicast group
>        tables. This is done since the IP forwarding code looks into
>        this table when making its decisions.
>
>        The multicast group information is always linked to the
>        interface it is relevant to (ie. the subnet the multicast
>        group is on). All one has to do is to add code that looks at
>        the interface type and takes interface specific action(s). In
>        the case of IPoIB it would join the corresponding IB MGIDs.
>
>        Such a change is no different than adding an IB specific
>        module. It does not effect the IGMPv1/v2/MLD code.
>
>
>> >From what I've heard 3 should be simple to add. The SA has all
>> the MC info already. It simply needs to supply the caller a
>> list of all the existing MC groups, and notify the caller of
>> any new group coming into existence so that the caller (driver
>> of an IP router) can ensure it "join to observe" every
>> multicast group.
>
>        I am unclear on the meaning of 'join to observe'. A router may
>        have to send packets out on the MGID - so it will have to
>        'join' for receive and send.

For each active MC group (with > 0 member) the router needs to
join in order to receive. But the regular "join to receive" will
count the router as a real member, which must not be the case (otherwise
the MC group won't ever be deleted because its member count will
never drop to 0.) Thus we'll need a special "join to observe" that
still adds the caller to the list of listeners but doesn't cause the
member count maintained by SA to increment.

>
>	
>> 
>> Please express your opinion if you have one in the next few
>> days. If we can't get to a consensus I'd say we shelf it and
>> proceed with the rest of stuff. Note that since the
>> encapsulation draft has a dependency on the link/multicast
>> draft, it can't move forward alone. Also I don't want to wait
>
>	The link-multicast draft defines the :
>
>		- mapping from IP multicast to IB MGID
>		- the definition of the IPoIB link
>
>        We need to define the multicast forwarding methodology. In the
>        interest of forward movement I suggest we do one of the
>        following: 
>
>        a) Add the mapping and link-definition pieces to the
>        encapsulation draft. These two items are quite straigtforward
>        and simple enough. Define the multicast forwarding later when
>        IBTA publishes its specification.
>
>        b) Publish the link-multicast draft now without multicast
>        forwarding. It will define only the mapping and the notion of
>        IPoIB link. The mutlicast forwarding solution is published
>        later when IBTA specification comes out.
>
>        We can live without having IPoIB multicast routing defined
>        until september. 
>
>> until September for the IBA 1.1 spec. We have other work to do.
>
>        Does the other work include advanced capabilities (and
>        DHCP/MIB)? 

Yep.

Jerry

>
>Vivek
>
>> 
>> Jerry
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 7 Jun 2002 13:25:58 -0700 (PDT)

<snip>
> >> But yesterday I heard from David Brean of IBTA/MGTWG that the

<snip>

> consensus on (3) then we need to apply persuasion power to IBTA.

        I was referring to the 3 changes to IB multicast definition
        you mentioned. So I was asking, since it is not clear from
        your comments, whether you meant 'ignore item 3 but use 1 & 2'
        or did you mean ignore them all and work as per the current
        specification. 

<snip>

> something anti-multicast.

	Totally agree that this is not a good option - as I noted
	earlier. However, for the sake of completion, there are 4
	options (good or bad).
> broadcast over 100 nodes.

        I am not comparing it to the broadcast case. I was wondering
        about the following situation:

        How do one ensure that the router is not a member of any IB
        group ? What mechanism stops it from being a member ? Many
        nodes do double duty - they host applications and also act as
        routers - how do they decide?

        As per this solution a router will receive two packets and if
        the IP subnet has multiple routers then multiple routers get
        two packets. If the corresponding IP group has a scope larger
        than the local subnet - each of these routers will now forward
        multiple packets outward. If one or more of these neighbouring
        subnets are IPoIB subnets we can further multiply the number
        of copies by a factor of two at the least. The result would be
        that IPoIB subnets would end up creating multicast storms
        quite often.

        I have not given it more thought than this but I can imagine
        multicast services receiving multiple queries for the same
        request. I have no idea how that would work - probably just
        add more load and nothing else but that is bad too.

        The chances of such errors are greater when one inserts
        duplicate packets into the network expecting the routers to
        forward only one copy.

> The latter two can be very subjective.

        Take a look at the description below. The change is a simple
        function in the driver with a line or two added (considering
        BSD code) to the 'add multicast group' socket option.

        The suggestion works within the existing infrastructure
        with a minor addition.

        In existing implmementation the routing daemon on receiving an
        IGMP report adds the IP multicast group to the interface list.
        It tells the kernel to add the group by a socket option (in
        BSD/LInux implementations). The function after it adds the IP
        group to the interface can call the interface's device
        specific function - for IPoIB this function will ask the
        driver to join the IB group.

	(IGMP_report)--- n/w -> driver -> IGMP -> mrouted/gated
							|
						    setsockopt(ADD_LGRP)
							|
							:
							:
							add_grp()
							|
						** call device_add_grp() **
						** join relevant IB group **
						** if needed            **

        Only the call marked with '**' is new. Nothing else. Note that
        the same function or similar function will have to exist on
        hosts for IP_ADD_MEMBERSHIP directive when a host application
        joins a particular IP group. In this case the router does the
        same thing on receiving an IGMP_HOST_MEMBERSHIP_REPORT.

        The change is simple and does not require changes to 'every
        version of group mgmt code'. It does not require changes to
        routing daemons, nor does it imply large, complicated changes
        in OSes.

> member count maintained by SA to increment.

	That is a thought but it has some complications. This would be
	yet another kind of 'join' - not just 
		'send-only-but-not-receive' or 'send-and-receive' 
		but 
		'send-and-receive-but-not-be-counted-a-member'.

	Well it is upto IBTA to define all this but it is getting
        messy. In my view, as far as IPoIB is concerned, the routers
        can leave when the routing daemon indicates that the kernel
        can delete the IP group. It becomes a simple side-effect in
        the kernel. I feel that is a much simpler approach.

        The alternative requires the MADs to be handled; then passed
        to the driver QP or passed to an IPoIB-specific IB agent that
        then (exactly like the scheme with mrouted daemon) asks the
        driver to join/leave the IB groups. It also requires changes
        to IBTA that may or may not make into the 1.1 specification in
        september. 

> >        We can live without having IPoIB multicast routing defined

        Don't you agree that option a) or b) above is the best way to
        move forward?

> >        Does the other work include advanced capabilities (and

	OK. In that case I'll update the advanced capabilities draft
	and also plan to give a presentation on it to move the
	discussion forward. 

Vivek

---

From: Michael Krause <[email-masked]>
Date: Mon, 10 Jun 2002 06:59:05 -0700

>member count maintained by SA to increment.

Routers send / receive multicast packets thus are real members just like 
any other endnode within an IB subnet especially if they are using the 
subnet as an intermediate hop along the way to other subnets and the 
multicast group spans all subnets even if not locally used by any other 
endnode.  The SA would need to identify the router and determine whether 
there is anyone using this service to determine whether the router should 
be told to go away.

Mike