Subject: Re: [Ipoverib] IPoIB link address

From: Vivek Kashyap <[email-masked]>
Date: Tue, 30 Sep 2003 10:44:58 -0700

See below in <VK>

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Kanoj Sarcar                                                                                                     
                      <Kanoj.Sarcar@Sun        To:       "H.K. Jerry Chu" <[email-masked]>                                          
                      .COM>                    cc:       Vivek Kashyap/Beaverton/IBM@IBMUS, [email-masked], [email-masked]      
                      Sent by:                 Subject:  Re: [Ipoverib] IPoIB link address                                             
                      Kanoj.Sarcar@Sun.                                                                                                
                      COM                                                                                                              

                      09/30/2003 09:56                                                                                                 
                      AM                                                                                                               
                      Please respond to                                                                                                
                      Kanoj.Sarcar                                                                                                     

"H.K. Jerry Chu" wrote:
>
> Vivek & Vandana,
>
> My comments are embedded in the text below (with co-chair hat OFF).
>
> >> >         1. Define a broadcast-GID defined as follows (one of -
please
> >> >         comment):
> >> >
> >> >         a) FF1x:401B::255.255.255.255
> >> >         b) FF1x:001B::255.255.255.255
> >> >         c) FF1x:0000::255.255.255.255
> >> >         d) FF1x:401B::1
> >> >
> >> >     My preference is b) or c) since it removes the IP version
> >> >     completely from the mapping. a) is what we have today and
requires
> >> >     no change and so could be used. d) is derived from the multicast
> >> >     mapping and so I'd like to leave it separate.
> >>
> >>I would go for a) since it is already there; if we do change, I would
> >>go for d, but can live with b or c.
> >>
> >><VK> OK. I find (a), (b) and (c) acceptable. My order of preference
(c),
> >>then (b) then (a). No (d). <VK>
> >
> >
> >My order of preference is (a) then (b). No (c) or (d). The reason I do
not
> >think (c) is a good idea
> >is because having a signature defined is useful to mark these MGIDs as
> >IPoIB MGIDs. There is less possibility of MGID collision with non-IPoIB
> >applications
> >Also, (a) makes the most sense as a broadcast group applies only to v4,
so
> >why not leave things as is.
>
> I'd suggest "191B" (for "IPIB") as the signature
(FF1x:191B::255.255.255.255)
> to reflect the fact that the broadcast group is for both v4 and v6.
>

> But if people prefer any of the above it's fine with me too, as long as
> we can get a closure on this sooner.
>

Hi,

Summarization of what I have seen so far seems to indicate that people (who
are expressing an opinio) are converging on option a
(FF1x:401B::255.255.255.255)
since it has been the first preference of two people (Vandana/Kanoj) and
okay
with Vivek/Jerry.

<VK> OK. The single broadcast-GID is : FF1x:401B::255.255.255.255. <VK>

> >
> >> >
> >> >         2. The multicast mappings stay as currently specified.
> >> >
> >> >         3. The broadcast-GID (one of)
> >> >
> >> >         a) MUST be FullMember IB_joined
> >> >
> >> >         b) MUST be used to derive the attributes such as the MTU,
Q_Key.
> >> >         The attributes MAY be obtained by FullMember IB_Join of the
> >> >         broadcast-GID or by some other operation supported by IBA.
The
> >> >         node MUST fail all IP operations on the interface if the
> >> >         Broadcast-GID ceases to exist since that will correspond to
a
> >> >         downed link. It is upto the implementation to track the
existence
> >> >         fo the Broadcast-GID in a suitable way. Note: If the
> >>broadcast-GID
> >> >         is FullMember IB_joined then its existence/deletion is
> >> >         automatically tracked.
> >>
> >>I would go with option b, but without the requirement "MUST fail all IP
> >>operations ..." for multiple reasons; firstly, it requires at least a
> >>memory check in the send/receive path, and secondly, if other L2's
> >>(ethernet)
> >>aren't requring this, why should IB? (Seems like mingling an L2 issue
with
> >>L3,
> >>the same logic that was used against seperate groups).
> >>
> >><VK> On further thinking tracking with a requirement to re-acquire
modified
> >>attributes is needed. 'must fail' is not needed though for reasons
below
> >>(analogous to why we want to track other multicast GIDs):
> >>
> >>Assume that the broadcast-GID was not FullMember joined and so ends up
> >>getting deleted. There are 3 possibilities (not counting a recreate
with
> >>the same attributes and MLID):
> >>a) Broadcast-GID is not recreated.
> >>b) Broadcast-GID is recreated with different attributes but same MLID
> >>c) Broadcast-GID is recreated with different/same attributes but
different
> >>MLID
> >>
> >>
> >>Now, if (a) occurs - the nodes still have agreed upon MTU, Q_Key etc.
and
> >>can continue to communicate with peers. No new nodes can join the link
> >>though. No broadcasts can occur if the MLID was removed from the switch
> >>tables => no ARP. One  could let it all go on till nodes drop off(ARP
> >>timeouts for example) or declare interface is down.
> >>
> >>Now, if (b) occurs the link's attributes have changed and communication
> >>with nodes that were on the link before, and are still using the old
> >>information, can occur however communication with other nodes using new
> >>attributes on the link may fail. Broadcast however will probably
succeed
> >>with some set of nodes (dependent on how the fabric is setup for this
> >>MLID).
> >>
> >>if (c) occurs then broadcast with nodes that were part of the link
before,
> >>and are still using the old information, cannot occur (don't know the
new
> >>MLID) though the node can still communicate with the nodes using older
> >>attributes.
> >>
> >>Therefore, the requriement really needs to be:
> >>       - track broadcast-GID
> >>       - reacquire attributes if broadcast-GID is recreated
> >>       - (possibly) if no broadcast-GID down the interface
> >>
> >>Thoughts?, especially for the case where the broadcast-GID is deleted
and
> >>not brought back.
> >><VK>
> >
> >I agree. If end-nodes do not want to join the broadcast GID, they need
to
> >deal with all the requirements that
> >you have detailed above.
>
> Vivek brought up a good point that the empty broadcast group risks being
> deleted by SM. But unfortunately this problem already exists in the
current
> draft regardless of one or two broadcast groups.
>
> In my mind the broadcast groups for IPoIB are to be treated differently
by IB
> fabric adminstrator/management software compared with other IB MC groups.
> They are to be created as part of configuring an IPoIB link, and their
> livelihood should not depend on the condition of nodes running an IP
stack.
> (E.g. your Ethernet VLAN doesn't disappear when you shutdown all the
nodes.)
> This reasoning is compatible with IBA 1.0 spec when there was
"MCGroupRecord"
> to hold MC groups separate from "MCGroupRecord" that tracks membership.
> Unfortunately "MCGroupRecord" got removed in 1.0a spec. Now it seems that
one
> must join a MCMemberRecord before an IB MC group can be successfully
created.
>
> Instead of adding more complexity to the draft, can we leave this to the
> implementation, e.g., the fabric software to ensure the continuous
existence of
> the special broadcast group as part of IPoIB link? I've checked with some
people
> from IBTA and this seems to be a reasonable requirement that is very easy
to
> meet.
>
> We can simply add a clause requiring implementations to ensure that the
special
> broadcast group won't be deleted other than when the IPoIB link is torn
down.
> Then we don't need to require v6 node to JOIN the broadcast group. (Note
that
> even if we require v6 to JOIN, it doesn't solve the problem as I stated
ealier.)

Agreed. This problem will not be solved easily by requring end-nodes to
ensure
existance of broadcast group (because if all end nodes deconfigure their
devices,
they most probably will LEAVE the group, effectively deleting it; and after
that,
they can not reconfigure anymore without administrator recreating the
group).

<VK>However, the group won't go away, at least not without a complaint,
while at least one interface is full-joined to it. <VK>

Its probably best to require the administrator to ensure continued
existance of
the group, using some scheme outside IPoIB's scope. I already see the
following
in section 7.0:

" ... the network administrator must create a special IB all-node multicast

group (henceforth referred to as the broadcast group) with these link
attributes
for every node on the IPoIB link to join."

Changing to something like:

" ... the network administrator must create and guarantee existance of a
special
IB all-node multicast group (henceforth referred to as the broadcast group)
with
these link attributes for the entire period that the IPoIB link is in
existance;
every node on the link can join or query the group to find these
attributes."

might encompass what we have been discussing.

<VK>..however, the issue I'm concerned with is what if the admin wants to
modify the link - bring it down and reconfigure with different values.
Unless the nodes are tracking the broadcast-GID we could end up with a
partitioned IP subnet as I pointed out earlier. The  admin has no record of
existing members of the IPoIB link (since they left the GID already but are
actually still using it) and so can make the mistake of reconfiguring the
link while there are existing members.

Vivek

<VK>

Thanks.

Kanoj

>
> Jerry
>
> >
> >-Vandana
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email-masked]
> >https://www1.ietf.org/mailman/listinfo/ipoverib
>
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Kanoj Sarcar <[email-masked]>
Date: Tue, 30 Sep 2003 18:44:23 -0700

Hi,

> Vivek

I think it depends; if the admin presses a button on a GUI, and that
translates to the SM doing Proxy LEAVEs of all nodes (ie without the
nodes themselves doing the LEAVEs) so that the count drops to 0 and 
the group gets deleted, there will be no complaints.
 
> link while there are existing members.

Okay, I did not think you were talking about reconfiging the mcg. The admin 
can notice that there are non zero members in multicast groups that map the 
IPv4/v6 groups and infer that there are active nodes on the link.

Notice that whatever is done, the solution of when and how to change the mcg
properties will probably be quite vendor specific, and will depend on how 
the mcg was created. Even if the admin checks that are no members using the 
group and proceeds to delete the group, that is at best a racy solution, 
since it is not an atomic operation and a node can JOIN between the admin's 
check and deletion of the group (unless a vendor specific atomic operation
exists).

Using traps to do this isn't straightforward either, because the administrator
can not just get by with changing the properties of the broadcast gid. What
about the other IP mcg's that are being used? Even if you require that all
nodes LEAVE all the IP mcg's, it would seem to me that the admin must wait
till all IP mcg's have been deleted, before going ahead and recreating the
broadcast gid with the new properties.

One way to change the mcg properties safely is to make sure that no
nodes are using the link (deconfigure the network device on all nodes), then 
change the properties of the mcg, and then allow the nodes to reconfigure 
their network device. Vendor specific mechanisms are also possible.

Thanks.

Kanoj