Subject: [Ipoverib] IPoIB multicast forwarding

From: Vivek Kashyap <[email-masked]>
Date: Tue, 9 Apr 2002 01:31:29 -0700 (PDT)

Folks,

        We have defined a mapping from the IP addresses to the IB
        multicast groups. Use of different groups causes minimal
        replication at the switches and less load at the receivers.

        But there is a quirk in the way the groups are setup in
        InfiniBand - there is no way for any of the nodes to know if
        another has created an InfiniBand group. This is easy to fix
        in most cases - one checks with the SM. But it doesn't work
        out well for IPoIB packet forwarding.

        The router needs to forward the multicast packets beyond the
        local IP subnet if it knows there are recipients for it or in
        the case of some routing protocols it must do that unless it
        gets an update stating not to forward. Similarly, it needs to
        forward external packets to the local receivers if there are
        any receiver present. 

        Thus if there are IB multicast groups associated with an IP
        multicast group the router must know about them or the router
        must receive all multicast packets in the subnet via some
        other construct.

        Below are the three alternatives that I mentioned at the
        53rd IETF meet. 
	
        i) Define only one IB multicast group i.e. IB broadcast MGID.
        All IP multicast packets use this group.

        ii) The multicast IP packet is sent to both the associated IB
        multicast group and the IB mutlicast group corresponding to
        the all-IP routers IP group.

        iii) Sender always sends to corresonding IB group if it exists
        else it sends to the broadcast group.

             IP receivers create IB groups and send the IGMP/MLD
             groups on the broadcast MGID.

             Routers join relevant groups based on the IGMP report
             and routing protocol updates.

        I also put this problem to Bill Fenner (AD of routing area)
        and asked his advice on the alternatives. Of the three options
        he favored option (iii) above.

        Option (i) doesn't take advantage fo IB multicast at all and
        so should be looked at as a last resort solution only.

        Option (ii) requires the sender to send two copies of the same
        packet and 'deliberately' increases the chance of duplicates
        spreading across the network.

Suggested solution  in detail:

        1. Pure senders - at IB level check for the relevant IB group.
           If it exists then send to the IB group else send to the
           'all-routers' IB group. If there is no IB group
           corresponding to a particular IP group then it implies that
           there are no IP receivers on the subnet for that IP
           multicast group. 

           Note: a) Pure senders don't create IB groups. This is
           because there is no existing mechanism for the senders to
           let the routers know of the new groups.

        2. IP receivers: As per IGMP/MLD the first packet sent by a
           host on joining a multicast group for receiving
           (IP_ADD_MEMBERSHIP) is the multicast report.

	   The hosts, as a result of IP_ADD_MEMBERSHIP, do the following:

	   	check if the corresponding IB group exists. 

		if (IB group exists)
			IB_join the group and send the IGMP/MLD report

		else
			IB_Create the group
			IB_join the group
                        Send IGMP/MLD report to IB group corresponding
                        to all-routers (all-hosts or broadcast) IP
                        group (Note: only done when creating the IB
                        group) 

	3. IP routers IB_join MGIDs on:

		a) Finding a new IP group on the subnet in the IGMP/MLD report
                b) Determining a need to forward IP multicast packets
                   through a routing protocol

Comments ?

Vivek
			
-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 9 Apr 2002 11:54:36 -0700

I was also wondering why the multicast sender needed to do anything more
that send to the multicast group (if it existed).  Unless a multicast router
is in promiscuous mode, wouldn't it have the same issues for an Ethernet
link?  Either it has set a MAC filter to receive the IP multicast addresses
or it will miss them.  Why is an IB link different?  If a multicast router
knew to set its MAC filter for a particular IP multicast address, then in an
IB link, wouldn't it have joined the appropriate IB multicast group?

Roy

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Tuesday, April 09, 2002 11:41 AM
To: Sukanta ganguly
Cc: Bill Strahm; Culley, Paul; Vivek Kashyap; [email-masked];
[email-masked]
Subject: Re: [Ipoverib] IPoIB multicast forwarding

Question then...

As an IP Multicast sender, I am not required to join 
the Multicast group... So now there has not been an
IGMP message telling the first hop router to join the
group either.  So the router has not told its layer
2 to join the broadcast address for the multicast group.

How does the first hop Multicast router know to join
this group and forward the traffic on to the multicast
tree (either RP, or Prune/Flood) if it can't receive
the initial packet...

This is the magic that I was wanting to know, I know how
I would make it work, but I am hoping Bill F. can tell me
the correct way to do it <G>

Bill Strahm

On Tue, Apr 09, 2002 at 11:36:22AM -0700, Sukanta ganguly wrote:
> Bill,
>   When you send in the Multicast packet (as a sender),
> at layer 3 you send it to the multicast IP address
> that the sender decided on. Your layer 2 module will
> decide on the next hop for that Multicast IP address.
> And a similar functioning will be build up as a chain.
> Note in this pacth while the senders original packet
> is moving to its destination which is the IP multicast
> address, the intermediate routers (which are multicast
> aware) will check to see if they have any IGMP joins
> they have received from any other subnets. If they
> have any such mapping they have built due to receive
> of IGMP joins then they will oblige that join by
> copying the packet to that joined destination. That's
> how the copy of the packet is enroute to its rightful
> receiver.
> 
> SG
> 
> P.S :: Dr Fenner is an authority on this and can
> provide some special comments 
> 
> 
> --- Bill Strahm <[email-masked]> wrote:
> > Ok, I am going to express a scary amount of
> > ignorance here...
> > 
> > I know how multicast works on receive, I know how
> > IGMP works to build the forwarding tree,
> > I know how flood/prune, RP Multicast Routing
> > protocols work...
> > 
> > What I would like a pointer to is a simple this is
> > how a first hop multicast router 
> > behaves.  An example on Ethernet would be
> > acceptable, a generic example would be better.
> > 
> > The question I have is as a Multicast sender, I send
> > a multicast packet to the first
> > hop router that is multicast aware.  Now, do I send
> > that packet to its unicast MAC 
> > address, or do I send that to the mapped layer 2
> > broadcast address and let the IP 
> > layer rebroadcast to the broadcast address if it is
> > needed on the local subnet.
> > 
> > Once I have that, everything else I understand
> > 
> > Bill
> > PS Bill Fenner, the IPoIB list is subscriber posting
> > only.  If you haven't subscribed, please do 
> > so we can get the benefit of your wisdom...
> > 
> > On Tue, Apr 09, 2002 at 08:53:57AM -0500, Culley,
> > Paul wrote:
> > > I suspect that the problem can be made simpler; IP
> > routers should always IB_join ALL IPoIB Multicast
> > groups for a given interface.  The router will
> > simply ask to join any groups in the same
> > partition(s) it is assigned to.  Whether the IP
> > router forwards the multicast packets will then
> > depend on the IP routing protocol.
> > > 
> > > IB routers have the same problem, and may use the
> > same solution...
> > > 
> > > Paul R. Culley
> > > Compaq Sr. Fellow
> > > 281-514-5543
> > > 
> > > 
> > > -----Original Message-----
> > > From: Vivek Kashyap [mailto:[email-masked]]
> > > Sent: Tuesday, April 09, 2002 3:31 AM
> > > To: [email-masked]
> > > Cc: [email-masked]
> > > Subject: [Ipoverib] IPoIB multicast forwarding
> > > 
> > > 
> > > 
> > > Folks,
> > > 
> === message truncated ===
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Tax Center - online filing with TurboTax
> http://taxes.yahoo.com/

---

From: "Aditya Dube" <[email-masked]>
Date: Thu, 11 Apr 2002 21:32:46 -0700

Hello Vivek

I have a question. Could you clarify what will happen in the following
scenario in the third scheme that was proposed below:

Pure senders - at IB level check for the relevant IB group.
           If it exists then send to the IB group else send to the
           'all-routers' IB group.

What happens if a Receiver creates the 'relevant IB group' after the
sender has made the check? 

Aditya Dube
-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]] 
Sent: Tuesday, April 09, 2002 1:31 AM
To: [email-masked]
Cc: [email-masked]
Subject: [Ipoverib] IPoIB multicast forwarding

Folks,

	

	   The hosts, as a result of IP_ADD_MEMBERSHIP, do the
following:

		if (IB group exists)

		else
			IB_Create the group
			IB_join the group

		a) Finding a new IP group on the subnet in the IGMP/MLD
report
                b) Determining a need to forward IP multicast packets
                   through a routing protocol

Comments ?

Vivek

---

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 15 Apr 2002 12:54:02 -0700

Can someone clarify for me the semantics of a join and delete operation with
respect to the following:

If a group has been created by node X and pure multicast sender Y has
detected that fact and recorded the MGID/MLID without joining, what is the
state of the MGID/MLID after X deletes the group?  If the group is then
created again, by X or a different node, must the SM assign the same MLID?
If not, node Y is holding a stale MLID which would argue that it must always
consult the SM before sending a multicast frame.

Along those same lines, if node Y had joined the multicast group X created
and then X deletes it, does the SM delete the multicast group and update IB
switches at that time or is the group reference counted such that as long as
a node has joined, switches are not reprogrammed until the last reference
does a leave?  If the former, we would have the same stale MLID problem.  If
the latter, doesn't a node crash in this situation preserve an "open"
multicast group indefinitely? 

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, April 15, 2002 11:39 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] IPoIB multicast forwarding

> 
> 
> Vivek
> 
> As the sender will only send packets to the all-router MGID after doing
> the check, how will a receiver who creates the MGroup on the same subnet
> as the sender ever receive the multicast packets that it should get?

        The suggested solution is as follows. The basic idea is
        certainly workable. Hopefully, after some discussion we can
        refine it to cover all the issues that crop up.

	Sender:
		got IP multicast packet to send
			determine the corresponding IB group
			if already 'IB_joined' or 'recorded'
				IB_send to it

			else if IB group exists
				IB_join/record MGID locally
				IB_send to it
			else
				
				IB_send to all-routers MGID

	Receiver:
		want to receive IP multicast beloging to an IP group 
			check for corresponding IB group
			if IB group exists 
				IB_join it
				IB_send report to group
			else
				IB_create IB group
				IB_send report to all-routers MGID
				(IB_send report to group)

	Router:
		always receive on broadcast/all-hosts and all-routers
		if IGMP/MLD indicates new group
			IB_join corresponding IB group
		if routing protocol indicates new group
			IB_create corresponding IB group 

        Now, a sender must check for the IB group everytime it is sending
        to an IB group it doesn't know about (no prior IB_join/record
        with it). But once the group exists and the sender has a
        packet to send, it will detect the presence and join the
        group. From then on it will always send to that MGID. The
        receiver will get the packets since it (or another receiver)
        created the IB group.

        In other situation where the router determines that there are
        external receivers it creates the group too. The sender will
        therefore find this and join the group.

        The sender needs to continuously check for the IB group only
        in case is where the router doesn't need to forward the packet
        and there are no receivers in the local subnet. I don't think
        this will be too often. Is this case of concern ?

Vivek

<snip>

---

From: Michael Krause <[email-masked]>
Date: Mon, 15 Apr 2002 13:10:31 -0700

At 12:54 PM 4/15/2002 -0700, Aditya Dube wrote:
>Hi Vivek,
>
>After reading through your response, I realized that there is one more
>case to consider:
>
>Sender does a check for IB Mcast Group...check succeeds because a
>receiver has created the group.
>
>Now, the receiver decides to leave the group and the IB MGroup gets
>deleted.
>
>The Sender needs detect this and revert back to sending the multicast
>packets to the All-router Address.
>
>I think in the scheme we are discussing, the sender will always have to
>check for the existence of the IB Group just because of the edge cases.

Yet another example of why use of an all router address is a bad idea.  The 
IB multicast group can be created and any node can be added / deleted 
(including a TCA providing IP routing services) without requiring any 
notification of address change, etc.

Mike

---

From: Brian Forbes <[email-masked]>
Date: Wed, 17 Apr 2002 11:02:54 -0700

Bill, you are correct that router interaction with MC groups has not been
fully defined.  I think it's safe to assume that anything that is useful can
be done, but you can't assume that the IB and IP semantics will/can be
identical.

Brian Forbes
Brocade Communications 

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Tuesday, April 16, 2002 4:02 PM
To: Vivek Kashyap
Cc: Michael Krause; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] IPoIB multicast forwarding

I don't think it is that easy... How will the router know to leave ? It
can't
be when there are no receivers, because it has to forward any traffic from
the senders to other locations on the network...  This is especially true
for
flood/prune algorithms (see PIM-DM)

I'm also not sure that this would work in an intermitant sender environment.

I may have a sender that only wants to multicast trigger events, lets say
an SNMP TRAP message that happens once in a blue moon...  This means that
there is the overhead of group creating and deletion with every send (and
you still have the problem with the router not being on the line)

I am very tempted to do some deep thinking about having ALL multicast
packets
go to a default multicast traffic IB address, and have the IP portion echo
the packets back to the IB subnet on the correct IB MC Group if there are 
konwn subscribers (through IGMP etc) and forward on to other ports depending
on the IP multicast forwarding tree...

This means that all locally sent MC traffic goes onto the fabric twice, once
to get to the router on a "All multicast" traffic channel, and once to the
correct multicast channel...

Does IB provide a mechanism to allow the Router to detect multicast group
creations... if it does, then we can use the existance of the group rather
than needing IGMP, I don't think this has been defined in IBTA fully however

Bill
On Tue, Apr 16, 2002 at 10:58:43AM -0700, Vivek Kashyap wrote:
> Great! So the deletion of the group is not a difficult issue at all.
> The IPoIB nodes only need to leave the group.
> 
>         The receiver leaves when it doesn't need to receive any more.
>         Router when there are no receivers and it knows it doesn't
>         need to forward. sender after some idle time i.e. no activity
>         on the group. My suggestion is to have a minimum timeout of 2
>         minutes - avoids group join/leave happening too often.
> 
> Vivek
> 
> > 
> > At 04:31 PM 4/15/2002 -0700, Vivek Kashyap wrote:
> > >         a) No delete is done by the IPoIB interfaces, it is an SM
issue
> > >
> > >         This I took from the IB architecture document in chapter 3. It
> > >         talks of the SM deleting unused groups. ( Now, there are a few
> > >         who say that this is not going to be the case -- in which case
> > >         a different solution is needed.)
> > >
> > >         The receiver leaves when it no longer wants to receive. The
> > >         router when there are no more receivers. The sender after some
> > >         idle time.
> > 
> > The specification states that a multicast management entity not the SM
is 
> > responsible for multicast group management.  This entity implement IB 
> > multicast group management and could also be responsible for resolving
IP 
> > multicast addressing to IB multicast groups.  It may also be responsible

> > for informing / updating participating IB routers of any changes (and
vice 
> > versa for the IB routers informing the entity of changes within its view
of 
> > the router cloud).  This entity could also be used to provide a similar 
> > service for CAs that act as multi-protocol router endnodes.
> > 
> > >         b) Any entity wanting to leave an IB group attempts to delete
> > >         it. If there are members in the group then the deletion should
> > >         fail. The last one will automatically delete the group.
> > >
> > >         [Depends on delete semantics]
> > 
> > If an endnode wants to leave an IB multicast group, it informs the 
> > management entity of this event and the management entity working with
the 
> > SM removes this endnode from the IB multicast tree.  If the entity 
> > determines that there are no longer any endnodes participating in the 
> > multicast group, it may delete the entire group from the fabric.  This
does 
> > not require the endnode to delete the group and deal with any failure 
> > semantics.  An endnode should simply delete itself and let the
management 
> > entity deal with the problem.
> > 
> > >         c) Leave is fine. No need to bother about delete.
> > >
> > >         let the group entries stay around in the database. They don't
> > >         bother anyone. The admin can periodically, once in a few
> > >         months may be, delete unused ones. The entries will get reused
> > >         if others join. The multicast tree is affected by joins and
> > >         leaves and not by deletes.
> > 
> > Delete should be performed automatically by the management entity; this
is 
> > a policy decision that has no bearing on the IP over IB specification.
> > 
> > >         d) {Don't know about this }
> > >
> > >         The setup is such that if the last member leaves the group the
> > >         group is automatically removed.
> > 
> > This is how IB specifies it should work.
> > 
> > Mike
> > 
> > 
> > 
> 
> 
> -- 
> Vivek Kashyap