Subject: [Ipoverib] Re: MCGroup Attach Failure --- IPoIB link and multicast draft -

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 16 Jan 2002 19:40:52 -0800 (PST)

Hello Aditya,

Do you have any reason to believe this case should be handled
differently by IPoIB stack than other IB multicast applications?

If the join operation is unreliable or intrinsically weak by IBA
design, we can always recommend a backup algorithm. (Any idea why
a join may fail?)

Jerry

>Hello Jerry
>
>I have a question from the Multicast draft that you posted a few days
>ago.
>
>What should a node that wishes to receive IP traffic on a MCast GID do
>if the 'Attach' (SubnAdmSet MCMemberRecord) operation on that IB MCast
>GID fails?
>
>
>Aditya Dube
>
>-----Original Message-----
>From: H.K. Jerry Chu [mailto:[email-masked]] 
>Sent: Monday, January 14, 2002 12:20 PM
>To: [email-masked]
>Subject: [Ipoverib] IPoIB link and multicast draft
>
>
>...<deleted>

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Aditya Dube" <[email-masked]>
Date: Wed, 16 Jan 2002 19:17:19 -0800

Hello Vivek

Which document is the Architecture draft referring to in the relevant
section (4.2.2.1, IB join of MGIDs by a listener):
           
		"The exact parameters and timers for such retries are
           specified in another document."

Aditya Dube

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]] 
Sent: Tuesday, January 15, 2002 10:16 PM
To: Aditya Dube
Cc: H.K. Jerry Chu; [email-masked]
Subject: Re: [Ipoverib] MCGroup Attach Failure --- IPoIB link and
multicast draft -

Aditya,
        The failure cases at the IB level are covered in the
        architecture draft.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: "Aditya Dube" <[email-masked]>
Date: Sun, 20 Jan 2002 21:54:12 -0800

Hello Jerry

I looked into this. An HCA can only join a limited number of MCGroups.
This limit is specified in the "Query HCA" Verb 11.2.1.2 (on pg 452). If
this number is exceeded then the "Attach QP to MCGroup" verb will fail
(11.3.1 page 496).

Whenever this number is exceeded the HCA will not be able to listen to
any MCast traffic on a new MCast GID because the sender of the data will
still continue to send data on these MCast GIDs unaware that the
recipient is unable to get the data.

Aditya Dube

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]] 
Sent: Wednesday, January 16, 2002 7:41 PM
To: Aditya Dube
Cc: [email-masked]
Subject: Re: MCGroup Attach Failure --- IPoIB link and multicast draft -

Hello Aditya,

Jerry

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 10:40:16 -0800

>recipient is unable to get the data.

There is no promiscuous mode within IB that would allow all joins to always 
succeed. This stems in part from allowing SMs to layout the any logical 
routing topology that they desire.  In addition, HCAs are restricted to how 
many MGIDs they can multiplex on a given port which is an 
implementation-specific issue.  For IP over IB, I do not view any of these 
issues as a problem or requiring additional work from the IBTA as these are 
not architectural limitations.  The IP over IB effort should simply require 
that for each partition that the endnode wants to provide IP service, there 
should be at a minimum a corresponding set of multicast resources 
available.  HCA vendors will then rely upon market input to guide them in 
updating their requirements.

In May 2001, HP made available a set of IB HCA, TCA, Switch, and Router 
requirements and motivations to help vendors decide what features and 
associated resources to implement.  This material is available to all and 
within one will find the multicast requirements.  The minimum acceptable 
for a HCA was 512 per port; optimal was 2048.  Additional values were 
provided for TCA, Switch, and Routers.  In discussions with many IHVs who 
have received this material, it is fair to say that the majority will 
implement generally more than the minimums and quite often near the 
optimals with some exceeding even these.  As such, the market will be able 
to support IP over IB requirements for the vast majority of the customer 
requirements thereby eliminating the need for IP over IB to do anything 
special to recover from such a group join problem.

Mike

---

From: "Aditya Dube" <[email-masked]>
Date: Tue, 22 Jan 2002 14:49:33 -0800

I agree that this is an edge case, but it is one that should be
addressed by our IP over Infiniband multicast proposal.

As a result of the HCA constraint and the IP over Infiniband - MCast GID
mapping scheme, we have limited the number of IP MCast addresses an
Interface can listen on. Ethernet has no such limitation.

On another note, the current multicast draft should say that that the
current scheme has this limitation. 

Aditya Dube
-----Original Message-----
From: Michael Krause [mailto:[email-masked]] 
Sent: Monday, January 21, 2002 10:40 AM
To: [email-masked]
Subject: Re: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and
multicast draft -

>recipient is unable to get the data.

There is no promiscuous mode within IB that would allow all joins to
always 
succeed. This stems in part from allowing SMs to layout the any logical 
routing topology that they desire.  In addition, HCAs are restricted to
how 
many MGIDs they can multiplex on a given port which is an 
implementation-specific issue.  For IP over IB, I do not view any of
these 
issues as a problem or requiring additional work from the IBTA as these
are 
not architectural limitations.  The IP over IB effort should simply
require 
that for each partition that the endnode wants to provide IP service,
there 
should be at a minimum a corresponding set of multicast resources 
available.  HCA vendors will then rely upon market input to guide them
in 
updating their requirements.

In May 2001, HP made available a set of IB HCA, TCA, Switch, and Router 
requirements and motivations to help vendors decide what features and 
associated resources to implement.  This material is available to all
and 
within one will find the multicast requirements.  The minimum acceptable

for a HCA was 512 per port; optimal was 2048.  Additional values were 
provided for TCA, Switch, and Routers.  In discussions with many IHVs
who 
have received this material, it is fair to say that the majority will 
implement generally more than the minimums and quite often near the 
optimals with some exceeding even these.  As such, the market will be
able 
to support IP over IB requirements for the vast majority of the customer

Mike

---

From: Michael Krause <[email-masked]>
Date: Wed, 23 Jan 2002 07:34:33 -0800

>Interface can listen on. Ethernet has no such limitation.

To clarify:

A MGID is a logical handle of an IB multicast group.  MGIDs are validated 
within the CA channel interface which may be implemented either in hardware 
or software (MGIDs are also processed by IB routers who forward to the 
appropriate next hop based on the multicast routing protocol).  For each 
MGID, there is associated a QP (for IP, one would most likely have one QP 
per IP multicast group).

A MLID is a local subnet IB address handle that defines the multicast group 
within the subnet.  MLIDs are typically implemented in hardware - e.g. the 
switch chapter defines how the multicast routing table is updated.  The 
number of MLIDs may be limited within a given hardware implementation - the 
maximum is ~16K as defined within the IB specification.

It is allowed to map multiple MGIDs to a single MLID thereby allowing an 
implementation to support many more IB multicast groups than the physical 
hardware may define.  As such, there really isn't any major limitation that 
IP over IB must focus upon in terms of specification language.  There might 
be some benefit derived from having some informative text to explain some 
of this material but one can just use the IB specification as a reference 
on these issues.

Mike

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 24 Jan 2002 18:48:51 -0800 (PST)

So it looks like the contraint on the number of MGIDs is much less
stringent than MLIDs. (I suspect many implementations will simply
implement MGIDs in software so the real contraint for it is how
much memory.) Since IB allows MLIDs sharing among different MGIDs,
I'd say the chance of running out of MGIDs in the real world is very
small. If it does happen, so be it. Note that this could happen on
IPoverEther as well.

If the failure occurs when an end node tries to attach to an IPoIB
link, e.g. the attach to the all-node or all-router mutlicast group
fails, it seems reasonable to expect the node can't join the link
until it frees up some MC resources.

Jerry

>X-Sender: [email-masked]
>Date: Wed, 23 Jan 2002 07:34:33 -0800
>To: <[email-masked]>
>From: Michael Krause <[email-masked]>
>Subject: RE: [Ipoverib] RE: MCGroup Attach Failure --- IPoIB link and multicast 
draft -
>Mime-Version: 1.0
>X-Mailman-Version: 1.0
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>X-BeenThere: [email-masked]
>