Subject: RE: [Ipoverib] IPoIB multicast forwarding

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 16 Apr 2002 16:54:54 -0700

Given all the uncertainty of multicast group semantics and some of the
algorithms we're coming up with for presumed correctness, I still lean
toward multicast = broadcast.  It is guaranteed to be correct and straight
forward to implement.  We can always come back to it once we get some
experience with how multicast is going to be used in the data center and
time for the IBTA to fix/clarify group semantics.  The way it stands now,
the SM implementations would seem to have a lot of flexibility which will
lead to interoperability problems.  IBTA needs to fix this before we can
reliably use this in IPoIB.

Roy

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Tuesday, April 16, 2002 4:02 PM
To: Vivek Kashyap
Cc: Michael Krause; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] IPoIB multicast forwarding

I don't think it is that easy... How will the router know to leave ? It
can't
be when there are no receivers, because it has to forward any traffic from
the senders to other locations on the network...  This is especially true
for
flood/prune algorithms (see PIM-DM)

I'm also not sure that this would work in an intermitant sender environment.

I may have a sender that only wants to multicast trigger events, lets say
an SNMP TRAP message that happens once in a blue moon...  This means that
there is the overhead of group creating and deletion with every send (and
you still have the problem with the router not being on the line)

I am very tempted to do some deep thinking about having ALL multicast
packets
go to a default multicast traffic IB address, and have the IP portion echo
the packets back to the IB subnet on the correct IB MC Group if there are 
konwn subscribers (through IGMP etc) and forward on to other ports depending
on the IP multicast forwarding tree...

This means that all locally sent MC traffic goes onto the fabric twice, once
to get to the router on a "All multicast" traffic channel, and once to the
correct multicast channel...

Does IB provide a mechanism to allow the Router to detect multicast group
creations... if it does, then we can use the existance of the group rather
than needing IGMP, I don't think this has been defined in IBTA fully however

Bill
On Tue, Apr 16, 2002 at 10:58:43AM -0700, Vivek Kashyap wrote:
> Great! So the deletion of the group is not a difficult issue at all.
> The IPoIB nodes only need to leave the group.
> 
>         The receiver leaves when it doesn't need to receive any more.
>         Router when there are no receivers and it knows it doesn't
>         need to forward. sender after some idle time i.e. no activity
>         on the group. My suggestion is to have a minimum timeout of 2
>         minutes - avoids group join/leave happening too often.
> 
> Vivek
> 
> > 
> > At 04:31 PM 4/15/2002 -0700, Vivek Kashyap wrote:
> > >         a) No delete is done by the IPoIB interfaces, it is an SM
issue
> > >
> > >         This I took from the IB architecture document in chapter 3. It
> > >         talks of the SM deleting unused groups. ( Now, there are a few
> > >         who say that this is not going to be the case -- in which case
> > >         a different solution is needed.)
> > >
> > >         The receiver leaves when it no longer wants to receive. The
> > >         router when there are no more receivers. The sender after some
> > >         idle time.
> > 
> > The specification states that a multicast management entity not the SM
is 
> > responsible for multicast group management.  This entity implement IB 
> > multicast group management and could also be responsible for resolving
IP 
> > multicast addressing to IB multicast groups.  It may also be responsible

> > for informing / updating participating IB routers of any changes (and
vice 
> > versa for the IB routers informing the entity of changes within its view
of 
> > the router cloud).  This entity could also be used to provide a similar 
> > service for CAs that act as multi-protocol router endnodes.
> > 
> > >         b) Any entity wanting to leave an IB group attempts to delete
> > >         it. If there are members in the group then the deletion should
> > >         fail. The last one will automatically delete the group.
> > >
> > >         [Depends on delete semantics]
> > 
> > If an endnode wants to leave an IB multicast group, it informs the 
> > management entity of this event and the management entity working with
the 
> > SM removes this endnode from the IB multicast tree.  If the entity 
> > determines that there are no longer any endnodes participating in the 
> > multicast group, it may delete the entire group from the fabric.  This
does 
> > not require the endnode to delete the group and deal with any failure 
> > semantics.  An endnode should simply delete itself and let the
management 
> > entity deal with the problem.
> > 
> > >         c) Leave is fine. No need to bother about delete.
> > >
> > >         let the group entries stay around in the database. They don't
> > >         bother anyone. The admin can periodically, once in a few
> > >         months may be, delete unused ones. The entries will get reused
> > >         if others join. The multicast tree is affected by joins and
> > >         leaves and not by deletes.
> > 
> > Delete should be performed automatically by the management entity; this
is 
> > a policy decision that has no bearing on the IP over IB specification.
> > 
> > >         d) {Don't know about this }
> > >
> > >         The setup is such that if the last member leaves the group the
> > >         group is automatically removed.
> > 
> > This is how IB specifies it should work.
> > 
> > Mike
> > 
> > 
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> [email-masked] 
> [email-masked] 
> 503 578 3422 (o)
> 
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 16 Apr 2002 17:52:42 -0700 (PDT)

> algorithms we're coming up with for presumed correctness, I still lean

        What is not 'correct' in the solution that I have presented? I
        feel that the solution hangs together pretty well. I'll
        present it in a consolidated form in the next few days. Let us
        then see if there is anything missing.

> toward multicast = broadcast.  It is guaranteed to be correct and straight

        As we discussed earlier in this thread, the broadcast option
        is not optimal and short changes InfiniBand.

> reliably use this in IPoIB.

        The solutions presented are in line with the specification as
        published by IBTA. As far as I know the basic assumptions
        specified in IBA version 1.0a are not being overridden in IBTA
        even if they are working on rounding off some of the
        unresolved issues of IB multicast. 

Vivek

---

From: Yaron Haviv <[email-masked]>
Date: Wed, 17 Apr 2002 10:28:32 +0200

Roy, in general I agree with you, since we have to have something running
this year
However we need to have some flexibility for future expansion

Another problem we haven't discussed is that Multicast LID's number is
limited, and in each CA it is a different number
Not to mention that the MLID's are used for other things beside IPoIB
This might create a problem where you can fail when trying to join a group,
or configure the CA
In Ethernet a NIC can just pass all the multicast traffic to the stack if it
needs more groups

We need to enable both the sender and receiver to use either broadcast or
multicast in an interoperable manner
The problem occurs when the sender sends to a multicast group and the
receiver listens only on the broadcast group
Since it failed joining the multicast group, or it even didn't try doing so
We need to have a mechanism telling the sender it can't use multicast for
that group, and it should use broadcast

This could be achieved is several ways:
- Manually: forcing all sender traffic to use broadcast (used initially,
answers Roy's request)
- Limiting the number of IPoIB MCGroups at the SM (where zero is also a
valid value), Whenever we exceed this number we can use broadcast for the
rest
- Any thing you can also think of

This approach also address the router issues, as well as the leave issues
(by deleting the group at the SM)

Example: limiting number of IPoIB MCGroups, Allocated MLID's=3
(Initially we can set the number of allocated MLID's to zero, and get all
broadcasted, in future tune the Multicast to use a non zero
This way we can both maintain interoperability with future implementations,
and solve the MLID limitation)

Receiver:
	Try to create/join the group
		SM: If less & eq 3 return info, if more than 3 return fail
		(if MLID's=zero fail all the time, or if manually configured avoid doing
the SM join request)
	If failed use broadcast else use mcast

Sender
	Want to send to mcast and not already a member
	Try to join, if failed use broadcast
		** to solve the issue where the multicast is created later and we want to
use it we can:
 1. Check with the SM periodically or every few packets
 2. Or register for change notifications on the MCGroup

This way we can accommodate both approaches, and help the early adopters
come up with implementations
Even before the IB MngWG and SM vendors finish defining and implementing the
mcast mechanisms
Otherwise I agree with Roy we wont manage to have a working solution this
year

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Larsen, Roy K
Sent: Wednesday, April 17, 2002 1:55 AM
To: 'Bill Strahm'; Vivek Kashyap
Cc: Michael Krause; [email-masked]; [email-masked]
Subject: RE: [Ipoverib] IPoIB multicast forwarding

Roy

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Tuesday, April 16, 2002 4:02 PM
To: Vivek Kashyap
Cc: Michael Krause; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] IPoIB multicast forwarding

Bill
issue
is
IP

vice
of
the
does
management
is

---

From: Michael Krause <[email-masked]>
Date: Wed, 17 Apr 2002 07:13:27 -0700

At 10:28 AM 4/17/2002 +0200, Yaron Haviv wrote:
>this year However we need to have some flexibility for future expansion

The IBTA is trying to address this problem as we speak which will allow one 
to ship product this year....something a bit similar to what I described in 
a previous e-mail is being proposed and this could be used to simplify this 
whole discussion and eliminate this ambiguity being debated here.  One does 
not need to go into a broadcast solution.  One does not need to flip 
between known multicast group to all routers.  One only needs to obtain / 
track the multicast group identifiers that one is using and leave the rest 
of these problems to IB to resolve.

Again, a multiprotocol router is nothing more than an IB CA (nominally a 
TCA but could be a HCA) that joins an IB multicast group just like a normal 
IB endnode.  It is the one responsible for interpreting the layer 3 headers 
and determining how to forward the packet.  It is the one responsible for 
layer 3 multicast group management not IB and not IP over IB in particular.

If IP over IB were to focus on how to join an IB multicast group and how to 
encapsulate IP multicast addresses within IB MGIDs, then this work would be 
greatly accelerated.   Leave the IP router issues to their appropriate 
workgroups which I do not believe is this workgroup in particular.

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Wed, 17 Apr 2002 21:13:15 +0200

Mike,

My email did not refer to IP or IB routers specifically, you didn't refer to
the problem I mentioned in the email
What happens if you run out of MLID resources, or fail to join?
Not to mention that I don't see how SM guys finish dealing with all the MC
side issues soon this leaves us with no IPoIB this year

If you mention IB routers, going back to your suggestion,
Lets say the sender sends the packet to the mcast group,
How can you make sure that the IB router (not an IP router) joins that group
before the sender issued the send ?
Do you want the sender to request a join, than the SM make sure that the
routers joined, only than the SM returns the response ?
Can you shed some light on how you see a complete sender operation ?

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Michael Krause
Sent: Wednesday, April 17, 2002 4:13 PM
To: [email-masked]
Subject: RE: [Ipoverib] IPoIB multicast forwarding

Mike

---

From: Michael Krause <[email-masked]>
Date: Wed, 17 Apr 2002 12:48:20 -0700

>What happens if you run out of MLID resources, or fail to join?

Either free up resources or fail to join. If people are good in their 
policies for assigning MLIDs then this will not be a problem for all 
reasonable implementations within switches (I've reviewed nearly every 
design in flight and their associated resources).

>side issues soon this leaves us with no IPoIB this year

It does not prevent shipping this year at all.

>before the sender issued the send ?

Multicast is unreliable by definition so the race condition is not an 
issue.  If the application cares about always getting through then this an 
application issue not an IB issue (no different than IP today).

>routers joined, only than the SM returns the response ?

No. It requests to join and the SM configures the endnode so that its 
packets can be routed through the switch and life proceeds.  If the router 
is up, then it is already part of the tree and receives the packet should 
it make it there.  If the router is not part of the tree, then it does not 
receive anything.  Multicast group membership is simply unknown so why are 
we trying to make this so hard to know that all parties are up and accessible.

>Can you shed some light on how you see a complete sender operation ?

(1) Join multicast group (send only or send/receive)
   - Obtain appropriate IB parameters to fill out a UD packet

(2) Fill out a UD packet as appropriate with headers and data

(3) Inject packet into fabric

(4) Repeat as needed

Some time in the future, if sender does not care about multicast any 
longer, leave the multicast group.

If endnodes join (independent of whether they are multi-protocol capable or 
not), add them into the multicast tree so they may receive packets if they 
have requested a send/receive join capability else allow them to send 
packets as desired.

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Wed, 17 Apr 2002 22:35:43 +0200

Mike,

How does the IB or IP router know that it needs to listen on a specific
group (in the send case) ?
The sender joined/created the group, how does the router know about it ?
Is the SM going to configure it like it does to switches ?
What happens until everything is fully configured in the fabric, do we wait
? do we drop ?

Why not allow the sender to broadcast as an optional method ?
We can also use your method, and it will probably prevail in the future (not
now)
It will certainly help many of us to build our products sooner than later

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Michael Krause
Sent: Wednesday, April 17, 2002 9:48 PM
To: [email-masked]
Subject: RE: [Ipoverib] IPoIB multicast forwarding

we trying to make this so hard to know that all parties are up and
accessible.

(4) Repeat as needed

Mike

---

From: Michael Krause <[email-masked]>
Date: Wed, 17 Apr 2002 15:49:15 -0700

>group (in the send case) ?

It is informed of when there is a change in the multicast groups within the 
subnet it is attached to just like one does for other change notifications.

>The sender joined/created the group, how does the router know about it ?

See above.

>Is the SM going to configure it like it does to switches ?

An IB router is treated as a leaf node from a SM perspective (same would 
apply to an IP router via a CA). As such, the SM does not configure it but 
the leaf switch it is attached to.  The IB router protocol deals with cross 
subnet routing and thus there is no SM involvement in such actions.

>What happens until everything is fully configured in the fabric, do we wait

If a multicast packet cannot make forward progress, the CA (IB router or 
IP) can drop it if desired.

>Why not allow the sender to broadcast as an optional method ?

Unnecessary complexity, too many interactions for update purposes, 
unnecessary performance loss, etc.

>now) It will certainly help many of us to build our products sooner than later

It can be used today via a management entity providing such multicast group 
management services (something not defined today within the IBTA) or one 
can wait until the IBTA releases a normative method.  I don't think the 
wait is a major issue and the IBTA will solve the update problem thereby 
eliminating having to specify such complexity within the IP over IB 
work.  It will also eliminate some interoperability problems because there 
is a single method to solve this problem and not two which is completely 
unnecessary.

Mike