Subject: Re: [Ipoverib] IPoIB link address

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 1 Oct 2003 22:30:22 -0700 (PDT)

<snip>

(co-chair hat off!)

>>a) make it a requirement that the broadcast-GID won't be torn down unless
>>the link is torn down.
>>
>>
>>This is somewhat circular since we define the link using the broadcast-GID.
>>Additonally, the admin will have to find some method to know that the link
>>can be brought down.
>>
>>b) Always have to full-member join broadcast-GID.
>>
>>Now, the admin will always know if someone is using the link and can do a
>>managed shutdown.
>>
>>
>>c) Always derive the data from the broadcast-GID and track it either by
>>full-member joining or reports.
>>
>>Now, the admin can delete/modify as needed. the nodes will respond 
>>accordingly.
>>
>
>I would prefer (c) too. This allows a deployment to not have any admin 
>involvement if necessary. Again, as I said before, if the implementation 
>decides to join full-member, there is no issue.

Not sure I understand what exactly c) is. It refers to v6 only so v4 MUST
always join the broadcast group, right?

I prefer not to require v6 to join the broadcast group. This is compatible
with c). But I prefer to continue to leave the creation/maintainence of
the broadcast group out of scope because the broadcast group should really
be managed together with the IPoIB link as part of the fabric management
function. (Note that this doesn't necessarily imply admin involvement.
If implementations can somehow find a way to embed this function in IPoIB
drivers that is fine too.) If we start getting into problems that really
belong to fabric management, we may never find a bullet-proof solution within
IPoIB itself. E.g., if we don't require the continuous existence of the
broadcast group as part of the IPoIB link, but we merely rely on IP nodes
joining so the broadcast group won't go away, what happen if for some reason
all the IP nodes went down, and the broadcast group got deleted as a result
of zero member. On a subsequent boot, what should IPoIB driver do? Create
the broadcast group? But how would the driver know what link attributes
to use? Implementations may supply link attributes to the IPoIB driver
through some implementation-specific mechanism. But we can't make this a
requirement in IPoIB spec, right?

Similarly link attributes shouldn't be modified without some coordination
between fabric mgmt and IP software and that coordination shouldn't be part
of IPoIB spec just like IP over Ethernet doesn't talk about how to cope
with VLAN reconfiguration or changes in VLAN properties.

So my preference remains to be a).

Jerry

>-Vandana
>
>>We can avoid going into the details in the specification since the details
>>listed above are largely informational. However, I believe that the case
>>of 'no-broadcast GID' needs to be specified. Is it not similar to the case
>>of a downed interface?
>>
>>
>>
>>My preference is for (c).
>>
>>Vivek
>>
>>
>>
>>Vivek
>>
>> >
>> > Jerry
>> >
>> > >
>> > >-Vandana
>> > >
>> > >
>> > >_______________________________________________
>> > >IPoverIB mailing list
>> > >[email-masked]
>> > >https://www1.ietf.org/mailman/listinfo/ipoverib
>> >
>> >
>> > _______________________________________________
>> > IPoverIB mailing list
>> > [email-masked]
>> > https://www1.ietf.org/mailman/listinfo/ipoverib
>> >
>> >
>>
>>__
>>
>>Vivek Kashyap
>>Linux Technology Center, IBM
>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 2 Oct 2003 01:58:37 -0700 (Pacific Daylight Time)

> always join the broadcast group, right?

The suggestion is in general for an IPoIB link. The link attributes are
derived from the IPoIB broadcast-GID. In the case of v4 it just so happens
that IP broadcast maps to this and so a node will surely join it. We have
earlier always talked of full-join since the node will send and receive
packets on the broadcast GID. However, with the notion that one need not
join the broadcast GID I believe we need to, irrespective of the IP layer,
state that :

a) either full-join b) or track the broadcast-GID using traps/reports.
(the GIDs could be joined sendonly or nonmember joined or as some have
suggested not joined at all).

> with c). But I prefer to continue to leave the creation/maintainence of

Yes, it is in conformance with (c).

> requirement in IPoIB spec, right?

It is not with fabric management that I have any concerns.. it is with
nodes that are up and running and with the suggestion that they not be
full-members of the broadcast-GID. If there are no members then the
broadcast-GID can be easily deleted - I'm sure that every impelementation
will have safeguards against it. Option (a) merely is stating that - have
safeguards and keep it up. But it is not comprehensive and leaves some
aspects to individual impelementations which could lead to
interoperability issues.

What happens if, for some reason, the link needs to be modified. The admin
has to bring the link down -- with no members the admin has to resort to
other methods (and it is not an IBA issue) to determine if there are IPoIB
members on the link. It cannot be determined trivially, one needs to look
for other MGIDs and guess or find some other method. If there are members
in the broadcast-GID then it is known easily that the link is in use.

Now, let us go with (a) and choose to bring the link down. What do we do
now? As per (a) - say it is out of scope of our specification which really
means there could be interoperability issues since implementations could
do things differently. Instead why not have it that if the broadcast-GID
is deleted or modified then the nodes declare the link down or modfied
respectively. It allows for a commonly agreed method of downing the link
and all the nodes will respond the same way.

If we go with (a) and if there is an error and the GID does get deleted.
What happens? As per (a) -- out of scope. Instead we can say that if the
link is deleted then the node declares the link down - we therefore have a
common agreed response across all implementations.

If all nodes go down and the link/GID stays there are no issues. The node
coming up will join the link and all is fine. If the nodes go down and the
GID is not there then the nodes will complain. The situation is the same
whether we choose (a) or (c). It is equivalent to first bringup of the
link.

In summary it appears to me that we can combine the two. (A) describes the
requirement on the administrator to keep the link consistent. (B)
describes the requirement on the hosts.

A) Broadcast-GID defines the link. It MUST not be deleted/modified if the
IPoIB subnet is in use.

[Just (A) has the following issues issues:
	what about managed shutdown of the link?
	what about erroneous deletion/modification of broadcast-GID?]

B) Nodes always derive link attributes from the broadcast-GID. If a node
discovers that the broadcast-GID is no longer in existence it MUST declare
the interface 'down'. If the node discovers that the link-attributes are
different than the ones that it had discovered earlier then it SHOULD
reconfigure itself with the new attributes. A node may detect the
existence/disappearance of the broadcast-GID by using traps/reports or the
node might choose to full-member join the broadcast-GID.

> with VLAN reconfiguration or changes in VLAN properties.

In Ethernet we also do not talk about the link-broadcast address
determining the existence of the link - in IPoIB it is the basic
requirement. However, ethernet nodes to respond to a downed link. In IPoIB
the disappearance of a broadcast-GID is equivalent to a downed link: no
one can broadcast and no one new can join. It is a downed link for
implementations that do not actively join the broadcast-GID also, since we
require that all implementations derive their attributes from the
broadcast-GID.

Vivek

__

Vivek Kashyap

---

From: Kanoj Sarcar <[email-masked]>
Date: Thu, 02 Oct 2003 10:54:43 -0700

> >decides to join full-member, there is no issue.

Hi,

> always join the broadcast group, right?

Isn't it true that in limited v4 configurations, you can use static ARP
tables on the hosts to eliminate broadcast traffic? If you want to optimally
support this, I think even the requirement to JOIN for v4 has to be
rethought.

I think whatever decision we take (MUST JOIN, should JOIN, MUST JOIN or
MUST track) can apply to both v4 and v6.

Thanks.

Kanoj