Subject: Re: [Ipoverib] IPoIB link address

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 2 Oct 2003 06:46:25 -0700 (PDT)

<snip>

>> Not sure I understand what exactly c) is. It refers to v6 only so v4 MUST
>> always join the broadcast group, right?
>
>The suggestion is in general for an IPoIB link. The link attributes are
>derived from the IPoIB broadcast-GID. In the case of v4 it just so happens
>that IP broadcast maps to this and so a node will surely join it. We have
>earlier always talked of full-join since the node will send and receive
>packets on the broadcast GID. However, with the notion that one need not
>join the broadcast GID I believe we need to, irrespective of the IP layer,
>state that :

I am confused. What is the notion that "one need not join the broadcast
GID" for v4? There isn't any leeway for v4 - it must join the broadcast
group for ARP to work.

>
>a) either full-join b) or track the broadcast-GID using traps/reports.
>(the GIDs could be joined sendonly or nonmember joined or as some have
>suggested not joined at all).
>
>
>>
>> I prefer not to require v6 to join the broadcast group. This is compatible
>> with c). But I prefer to continue to leave the creation/maintainence of
>
>Yes, it is in conformance with (c).
>
>> the broadcast group out of scope because the broadcast group should really
>> be managed together with the IPoIB link as part of the fabric management
>> function. (Note that this doesn't necessarily imply admin involvement.
>> If implementations can somehow find a way to embed this function in IPoIB
>> drivers that is fine too.) If we start getting into problems that really
>> belong to fabric management, we may never find a bullet-proof solution within
>> IPoIB itself. E.g., if we don't require the continuous existence of the
>> broadcast group as part of the IPoIB link, but we merely rely on IP nodes
>> joining so the broadcast group won't go away, what happen if for some reason
>> all the IP nodes went down, and the broadcast group got deleted as a result
>> of zero member. On a subsequent boot, what should IPoIB driver do? Create
>> the broadcast group? But how would the driver know what link attributes
>> to use? Implementations may supply link attributes to the IPoIB driver
>> through some implementation-specific mechanism. But we can't make this a
>> requirement in IPoIB spec, right?
>
>It is not with fabric management that I have any concerns.. it is with
>nodes that are up and running and with the suggestion that they not be
>full-members of the broadcast-GID. If there are no members then the
>broadcast-GID can be easily deleted - I'm sure that every impelementation
>will have safeguards against it. Option (a) merely is stating that - have
>safeguards and keep it up. But it is not comprehensive and leaves some
>aspects to individual impelementations which could lead to
>interoperability issues.
>
>What happens if, for some reason, the link needs to be modified. The admin
>has to bring the link down -- with no members the admin has to resort to
>other methods (and it is not an IBA issue) to determine if there are IPoIB
>members on the link. It cannot be determined trivially, one needs to look
>for other MGIDs and guess or find some other method. If there are members
>in the broadcast-GID then it is known easily that the link is in use.
>
>Now, let us go with (a) and choose to bring the link down. What do we do
>now? As per (a) - say it is out of scope of our specification which really
>means there could be interoperability issues since implementations could
>do things differently.

We shouldn't have to deal with link reconfiguration any more than IP over
Ethernet dealing with VLAN. IPoE doesn't ever try to deal with any L2
configuration issue. The admin had better bring down the IP stack on
nodes before bringing down a VLAN and reassign its id. Any
interoperability issue resulted from adminstrator misstep IMO is out
of scope.

>Instead why not have it that if the broadcast-GID
>is deleted or modified then the nodes declare the link down or modfied
>respectively. It allows for a commonly agreed method of downing the link
>and all the nodes will respond the same way.
>
>If we go with (a) and if there is an error and the GID does get deleted.
>What happens? As per (a) -- out of scope. Instead we can say that if the
>link is deleted then the node declares the link down - we therefore have a
>common agreed response across all implementations.

Ok.

>
>If all nodes go down and the link/GID stays there are no issues. The node
>coming up will join the link and all is fine. If the nodes go down and the
>GID is not there then the nodes will complain. The situation is the same
>whether we choose (a) or (c). It is equivalent to first bringup of the
>link.

No, this is not an issue if we require a). If we don't require a), but
instead relying on non-empty broadcast membership to persist the broadcast
group as suggested by c), it will be an issue when all nodes go down.

>
>In summary it appears to me that we can combine the two. (A) describes the
>requirement on the administrator to keep the link consistent. (B)
>describes the requirement on the hosts.
>
>A) Broadcast-GID defines the link. It MUST not be deleted/modified if the
>IPoIB subnet is in use.
>
>[Just (A) has the following issues issues:
>	what about managed shutdown of the link?
>	what about erroneous deletion/modification of broadcast-GID?]

I'd say both are out-of-scope. See my IPoE analogy above.

>
>B) Nodes always derive link attributes from the broadcast-GID. If a node
>discovers that the broadcast-GID is no longer in existence it MUST declare
>the interface 'down'. If the node discovers that the link-attributes are
>different than the ones that it had discovered earlier then it SHOULD
>reconfigure itself with the new attributes. A node may detect the
>existence/disappearance of the broadcast-GID by using traps/reports or the
>node might choose to full-member join the broadcast-GID.

Although I still don't see the necessity of including the above paragraph,
it does look much better and more acceptable.

>
>>
>> Similarly link attributes shouldn't be modified without some coordination
>> between fabric mgmt and IP software and that coordination shouldn't be part
>> of IPoIB spec just like IP over Ethernet doesn't talk about how to cope
>> with VLAN reconfiguration or changes in VLAN properties.
>
>In Ethernet we also do not talk about the link-broadcast address
>determining the existence of the link - in IPoIB it is the basic
>requirement. However, ethernet nodes to respond to a downed link. In IPoIB
>the disappearance of a broadcast-GID is equivalent to a downed link: no

I guess this is the main difference between our view points. You have
the notion of a "downed link" while in my mind the broadcast group is
an integral part of the IPoIB link (because it defines the link attributes
such as MTU, Q_Key...). Without the braodcast group, we don't have a link
(not ever a "downed" one).

Jerry

>one can broadcast and no one new can join. It is a downed link for
>implementations that do not actively join the broadcast-GID also, since we
>require that all implementations derive their attributes from the
>broadcast-GID.
>
>Vivek
>
>> So my preference remains to be a).
>>
>> Jerry
>>
>> >-Vandana
>> >
>> >>We can avoid going into the details in the specification since the details
>> >>listed above are largely informational. However, I believe that the case
>> >>of 'no-broadcast GID' needs to be specified. Is it not similar to the case
>> >>of a downed interface?
>> >>
>> >>
>> >>
>> >>My preference is for (c).
>> >>
>> >>Vivek
>> >>
>> >>
>> >>
>> >>Vivek
>> >>
>> >> >
>> >> > Jerry
>> >> >
>> >> > >
>> >> > >-Vandana
>> >> > >
>> >> > >
>> >> > >_______________________________________________
>> >> > >IPoverIB mailing list
>> >> > >[email-masked]
>> >> > >https://www1.ietf.org/mailman/listinfo/ipoverib
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > IPoverIB mailing list
>> >> > [email-masked]
>> >> > https://www1.ietf.org/mailman/listinfo/ipoverib
>> >> >
>> >> >
>> >>
>> >>__
>> >>
>> >>Vivek Kashyap
>> >>Linux Technology Center, IBM
>> >
>>
>>
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>>
>>
>
>__
>
>Vivek Kashyap
>Linux Technology Center, IBM
>
>

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 2 Oct 2003 09:59:07 -0700 (Pacific Daylight Time)

> GID" for v4? There isn't any leeway for v4 - it must join the broadcast

Jerry, I think we are all confused :).

Join is of three types: Full member, Nonmember, Sendonlynonmember.

As far as I know one will have to join (implying one of the above three) to
retrieve the multicast information. In the case of v4, for the broadcast-GID
one will have to either full-member join or non-member join. Both allow for
sends and receives except the latter does not count the requester for
deletions(at the SM).

In my previous note I meant 'need not full-join'. Until now we have always
talked in context of full-join, however if we are now talking about not
full-joining I would rather have a common specification whether v4 or v6
and not have an exception for v6 only.

> interoperability issue resulted from adminstrator misstep IMO is out

Not all correlations hold between the two layers.  As I noted, IPoE also
does not state that the link is defined only by joining the broadcast link
address. IPoIB does and therefore is much more closely intertwined with what
happens to the broadcast-GID.

> group as suggested by c), it will be an issue when all nodes go down.

We are also considering possible error cases. Note that (a) is  essentially
asking the layer 2 to ensure that the interface always stays up but due to
some reason it could go down. That is what I've attempted to address in
(A) and (B) below -- (A) as a requirement on the administrator and (B) as
a requirement on the hosts. It also takes care of managed shutdown of the
link.

> it does look much better and more acceptable.

OK. Is the above (A) and (B) acceptable to all?
Note: read 'down' above as 'no link'.

> (not ever a "downed" one).

No, that is not the difference. We are talking the same thing except for the
use of different words and more importantly the reaction to the possibility
of the broadcast-GID disappearing.

I used the words 'downed link' to imply the link is gone(no link since no
broadcast-GID) from the point of view of the IPoIB interface. Therefore since
the link is non-existent it is illegal for the interfaces to continue to be
functioning. You prefer that we make it illegal for the link to
go away  by mandating that the link always be administratively kept up/in
existence.

I actually think that we need both as addressed in (A) and (B) above.

Vivek

__

Vivek Kashyap

---

From: Vandana Rao <[email-masked]>
Date: Thu, 02 Oct 2003 12:17:26 -0700

Hi Vivek,

>sends and receives except the latter does not count the requester for

A node does not have to join the group to retrieve the multicast information.

<snip>

>Note: read 'down' above as 'no link'.

It is fine by me except that I would like stronger language on (B) 
indicating that all those requirements
in (B) apply only if the node did not full-member join the broadcast-GID.
-Vandana

---

From: Kanoj Sarcar <[email-masked]>
Date: Thu, 02 Oct 2003 12:24:38 -0700

> On Thu, 2 Oct 2003, H.K. Jerry Chu wrote:

Hi,

> retrieve the multicast information. In the case of v4, for the broadcast-GID

I believe you can query (without doing any joins) for specific IBA groups. That
way, you can retrieve the mtu/sl/qkey etc. At least, that is what I meant when
I proposed to remove the requirement for doing a JOIN.

If any host does a JOIN followed by a QUERY, it is always safe/racefree in
the face of admin changing the group's properties; if the group gets deleted,
the JOIN will fail obviously; if the group's property gets changed, again the
JOIN with the properties from the QUERY will fail. In any case, you do not
have to analyze this case if you require the admin to have the group stable
when hosts are QUERY/JOINing. 

> and not have an exception for v6 only.

Agreed.

> > >in the broadcast-GID then it is known easily that the link is in use.

Please see my previous mail about why the admin has to look at the other
groups anyway before he can recreate the broadcast group (unless maybe you put
more requirements on the hosts). Also see the logic about why this is 
racy. IE if the admin looks at the group, it appears empty, does not mean that 
someone could not JOIN an instant later. Thus, it appears to me that 
recreation needs vendor specific solutions, and new requirements are not
going to help. 

If you are trying to solve the issue of how to change link parameters safely
in a racefree manner, please list the steps that the admin/hosts have to do;
I don't believe JOIN or traps solve all aspects of the problem, unless the
admin iterates thru all the groups; contrary to what you are stating above.

If you are trying to solve the problem of how to track all IP nodes on the
fabric, thats a different problem, and yes, I agree that requring all nodes
to JOIN will give you an instantaneous snapshot. I don't know though how this 
information can be used meaningfully.

> > >common agreed response across all implementations.

But to what good though? If the MGID gets deleted, how does it matter how
the hosts behave? We agree that the MGID getting deleted when hosts are
active is erroneous; assuring the admin that the end nodes will "do the
right thing" (declaring link down et al) is dangerous; because he might 
turn around and recreate the MGID with new properties, and face all the race 
problems with other existing IP multicast groups still using old properties.

Much better to require the admin to manually go and clean up the fabric
(and nodes) to come out of this erroneous deletion of the MGID. Or use
some other vendor specific cleanup solution.
 
> a requirement on the hosts. It also takes care of managed shutdown of the

Are you also talking about "managed link shutdown"? Is that somehow different
from "changing link properties by deleting and recreating group"? If you are
somehow trying to destroy the link without intending to recreate it, wouldn't
it make more sense to delete the partition (this is again going out-of-scope
though)?

Maybe we first need to be clear on the problems we are trying to solve.
 
> OK. Is the above (A) and (B) acceptable to all?

Agree to A. Don't see the neccesity of adding B (in fact might give a false
sense of assurance to admin when broadcast GID has been deleted in error in
violation to A).

Thanks.

Kanoj

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 2 Oct 2003 23:45:27 -0700 (Pacific Daylight Time)

> I proposed to remove the requirement for doing a JOIN.

OK.

> when hosts are QUERY/JOINing.

Yes, I agree however one of the issues is erroneous deletion or
disappearance of the broadcast GID for some reason.

> > and not have an exception for v6 only.

OK good. So, we have an agreement on two issues so far:

1. There is only a single broadcast GID. It is FF1x:401B::255.255.255.255.
2. A common specification for both v4 and v6.

> recreation needs vendor specific solutions, and new requirements are not

I believe that we all agree that the broadcast-GID defines the link. If
there is no broadcast-GID there is no link.

The question to look into is: If there is no link then shouldn't the hosts
react by reflecting this condition to their stacks? What should the
reaction be?

> admin iterates thru all the groups; contrary to what you are stating above.

I think we are looking at the issue differently. One way is to say that
the MGIDs have to be looked at by the admin. The other way is to say that
if the broadcast-GID is removed, the host brings the interfaces down. All
packet transfer automatically stops. The other is to also imply that since
there is no broadcast-GID the derived MGIDs do not make any sense any more
and hence the hosts need to purge them. The admin doesn't do anything in
this case -- I believe this has been Vandana's point too.

> information can be used meaningfully.

At the begining of the this thread the issues with broadcast-GID
removal/modification were listed. Those issues are adequately addressed by
using both (A) and (B). (B) appears to be simple enough to implement --
just a report; same as the reports used with other MGIDs.

> problems with other existing IP multicast groups still using old properties.

When the link is down -- it follows that no packets leave or are received
on the interface. So the existing MGIDs cannot be used either.

> it make more sense to delete the partition (this is again going out-of-scope

I don't know if it is easier to destroy the partition --  IPoIB may not
be the only thing in the partition. Anyway, that I agree is out of scope.

My comment about managed shutdown is just a comment - if (B) is a
requirement on the hosts then it is easier to bring down the link without
inventing some other method to stop the interfaces. It really is a
side-effect of downing the link.

> sense of assurance to admin when broadcast GID has been deleted in error in

What if one views it as - no broadcast-GID is same as no link. In IPoE it
could be cable pulled or a downed switch. It atleast means that the
interface is down.

Modification of broadcast-GID is same as moving the NIC to a different
ethernet segment -- things might not work any more.

These are the issues covered by (B).

Vivek

__

Vivek Kashyap

---

From: Kanoj Sarcar <[email-masked]>
Date: Fri, 03 Oct 2003 15:24:26 -0700

Hi,

> react by reflecting this condition to their stacks? What should the

Two things here: in an ethernet fabric, if the link between two switches in
the fabric break in a way that broadcast traffic from one node can not reach
some others (maybe causing segregated nodes as discussed previously), do the
end node stacks get told about this? In fact, if the ethernet link is completely
broken, maybe by pulling out all the wires, do any of the stacks get told
about this? Of course, ethernet does not provide a way to detect this condition
possibly, whereas IB does (via the GID deletion trap), so it might make 
sense to decide whether IB should do something that ethernet does not.

Also, I think we should be clear: I believe link conditions should not be 
reflected to the IP stack (aren't we mixing L2 and L3 if we do that?), but 
as you mention at other places, the L2 driver can stop tx/rx of packets. But
definitely the driver can not (for example) "inform" the stack that some 
IP multicast membership that was previously granted has been revoked;
although it can silently drop those mcg packets.

> this case -- I believe this has been Vandana's point too.

So, are you saying that the hosts MUST drop IBA memberships to other
groups too when it receives the broadcast-GID deleted trap? This was
what I hinted in my previous mail.  Are you suggesting nodes that
JOINed the broadcast GID also have to do this (otherwise, how would 
you do any recovery actions on erroneous deletions of the bcast-GID?) I don't 
believe Vandana mentioned this, I would like to hear her comment specifically 
if its okay to put more requirements for nodes that do JOIN. 

As I mentioned, if you believe a certain handling of bcast-GID
deleted trap will help solve some problem(s), (when a node JOINs 
or doesn't JOIN the broadcast GID) please list the problem(s) and how 
using the traps help, specifically by listing the steps the admin 
and hosts must take. I have tried to portray how it doesn't help, at 
least for the problem(s) we have discussed.

On the other hand, if you believe trap usage requirement is due to
some different view point, or some architecture or cleanliness issue,
and really does not contribute to solving or creating any more problems,
please state that. I would be very disinclined to putting more requirements
based on this argument, but at least I (and maybe others) will clearly
understand the basis of the requirement.
 
> removal/modification were listed. Those issues are adequately addressed by

I disagree; I don't believe having nodes EITHER join bcast-GID OR track
bcast-GID solves any problems for the host or the fabric; specially if
I don't see the rules of what a host that subscribes to the trap must do
(and secondarily, how those actions might help the host itself or the
fabric administrator).

> just a report; same as the reports used with other MGIDs.

I am not sure what you mean by "same as reports used with other MGIDs".
I thought we are discussing what the host SHOULD/MUST do when it discovers
the bcast-GID deleted erroneously or recreated.

> on the interface. So the existing MGIDs cannot be used either.

So, do the nodes LEAVE these other IP groups (and if so, how many seconds
does the admin wait to make sure all nodes have done the LEAVEs and thus
all old groups are gone, before he can safely recreate)? 

Or should the admin just delete these other groups? I think this is what
you were trying to prevent in the first place, ie having the admin iterate
over the groups.

> could be cable pulled or a downed switch. It atleast means that the

The distinction is that IP still sends packets to the driver, the driver might
not send it out (or the transmit fails). IP's memberships to groups are not
affected. The ipoib driver can drop IBA memberships without telling IP, but
are you suggesting that?

To conclude, I still can not see that option B is required; I will refrain 
from reiterating that opinion, unless I see more logic to change it.

Thanks.

Kanoj

---

From: Vivek Kashyap <[email-masked]>
Date: Sat, 4 Oct 2003 01:53:58 -0700 (Pacific Daylight Time)

Kanoj,

In summary,

    - we have had the notion of full-joining (and I don't include query in
    joining) the broadcast GID.

    - now it has been proposed that one may not join the broadcast-group
    and get the attributes using a query or may join but not full join.

    - the IPoIB link is defined by creating a broadcast-GID. The
    broadcast-GID is the source of the link's attributes.

    - if there is no broadcast GID then there is no link.

Now, given the above we have the follwoing suggestion:

<snip..>
> about this? Of course, ethernet does not provide a way to detect this condition

By the word 'down' above (and we can define the word more or substitute
another word) I do mean that the packets are no longer transmitted or
received. Same as the case of cables pulled you mention. Depending on the
implementation 'telling the stack', can mean various things -- I just mean
that it has an effect such that no packets can get transmitted or
received.

> sense to decide whether IB should do something that ethernet does not.

We have done IB specific things -- such as a more elaborate multicast
mapping with indications for the protocol being mapped.

> although it can silently drop those mcg packets.

That is something I've said before -- no link implies cannot send packets.
That is even if the IB cable is connected and possibly one could transmit
unicast/mcast packets to peers, but the broadcast-GID is gone, then: do
not transmit/receive packets. I agree with that. 'Inform' stack as you
mention above is not what I had in mind.

> if its okay to put more requirements for nodes that do JOIN.

I was, probably not very clearly, giving options: if the link is being downed
then the rules that can help are -

one way -->
    Admin must look up the related MGIDs and nodes and down/delete them

2nd way -->
    if no broadcast-GID then the nodes stop transmission/reception.

3rd way (or extension to 2nd way) -->

    if no broadcast-GID should we also drop all MGIDs. If this was done
then the admin doesn't have to manually or through some other means have
to get the nodes to leave the membership.

The MUST clause in (B) above is the '2nd way'.

> understand the basis of the requirement.

The above is not clear to me -- the traps/reports are suggested in (B)
above. The suggestion is to track the disappearance of broadcast-GID so
that the interface may be 'downed' (we can define downed suitably).

> (and secondarily, how those actions might help the host itself or the

The host (in the driver or anywhere) must stop transmission or reception
of packets on the IPoIB interface(s) associated with that broadcast-GID.
That is the MUST clause in (B).

> I am not sure what you mean by "same as reports used with other MGIDs".

I mean implmenting reception of reports for broadcast-GID is not difficult
afterall the reports are probably requestd for MGIDs to which one might
joing send-only or non-member or 'create MGID' events.

> the bcast-GID deleted erroneously or recreated.

<snip...>

> you were trying to prevent in the first place, ie having the admin iterate

IP layer is not affected by the MGIDs going away.

<snip..>

> affected. The ipoib driver can drop IBA memberships without telling IP, but

OK, the crux of the matter...

Yes, by 'down' the interface I meant that the link is down. Whether the
down indication causes the IP interface to be marked down or whether the
driver just queues packets/drops packets is upto the implementation.

IP's membership is not affected, that is correct. That is certainly not
what I suggested.

I did ask if we could, because all MGIDs are derived from broadcast-GID
and if the broadcast-GID is down drop the MGID membership? Yes, that
amounts to dropping IBA membership without telling IP.

> from reiterating that opinion, unless I see more logic to change it.

Well, reading the rest of the discussion above I was getting the
impression that you are not exactly disagreeing.....

Vivek

__

Vivek Kashyap

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 6 Oct 2003 22:24:03 -0700 (Pacific Daylight Time)

Kanoj and I discussed the issues raised in this thread offline. After some
discussion we came to the conclusion that it is better to have the nodes
always full-join the broadcast-GID and not query or non/sendonlynon member
join the broadcast GID. If that is done then the implementations are much
simpler and cleaner.

Therefore, we suggest that the proposal of 'one may receive the
broadcast-group attributes using a query/other join modes' be dropped.
Note this does not mean that every QPN (if multiple are used) on a given
port MUST join the broadcast-GID but that at least one of them MUST join.

I'd like to suggest the following two as the conclusion of this thread.

i) There is a single broadcast-GID associated with any IPoIB link. This
broadcast-GID is -> FF1x:401B:<P_Key>::255.255.255.255.

ii) The broadcast-GID MUST be full-member joined by the IPoIB interface.

Thanks
	Vivek

__

Vivek Kashyap

---

From: "bill" <[email-masked]>
Date: Mon, 6 Oct 2003 23:59:40 -0700

Well this seems to change the concensus I was leaning toward last
week... ( part ii, part i is still correct )

Ok, baring anyone expressing bad things happening stating that the IPoIB
interface MUST full member join the broadcast GID - that will be
concensus Friday.

Speak up now - or forever hold your peace

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Vivek Kashyap
Sent: Monday, October 06, 2003 10:24 PM
To: Kanoj Sarcar
Cc: H.K. Jerry Chu; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] IPoIB link address

Kanoj and I discussed the issues raised in this thread offline. After
some discussion we came to the conclusion that it is better to have the
nodes always full-join the broadcast-GID and not query or
non/sendonlynon member join the broadcast GID. If that is done then the
implementations are much simpler and cleaner.

Therefore, we suggest that the proposal of 'one may receive the
broadcast-group attributes using a query/other join modes' be dropped.
Note this does not mean that every QPN (if multiple are used) on a given
port MUST join the broadcast-GID but that at least one of them MUST
join.

Thanks
	Vivek

On Sat, 4 Oct 2003, Vivek Kashyap wrote:

> A) Broadcast-GID defines the link. It MUST not be deleted/modified if 
> the IPoIB subnet is in use.
>
> B) Nodes always derive link attributes from the broadcast-GID. If a 
> node discovers that the broadcast-GID is no longer in existence it 
> MUST declare the interface 'down'. If the node discovers that the 
> link-attributes are different than the ones that it had discovered 
> earlier then it SHOULD reconfigure itself with the new attributes. A 
> node may detect the existence/disappearance of the broadcast-GID by 
> using traps/reports or the node might choose to full-member join the 
> broadcast-GID.
>
>
> <snip..>
> > >
> > > I believe that we all agree that the broadcast-GID defines the 
> > > link. If there is no broadcast-GID there is no link.
> > >
> > > The question to look into is: If there is no link then shouldn't 
> > > the hosts react by reflecting this condition to their stacks? What

> > > should the reaction be?
> >
> > Two things here: in an ethernet fabric, if the link between two 
> > switches in the fabric break in a way that broadcast traffic from 
> > one node can not reach some others (maybe causing segregated nodes 
> > as discussed previously), do the end node stacks get told about 
> > this? In fact, if the ethernet link is completely broken, maybe by 
> > pulling out all the wires, do any of the stacks get told about this?

> > Of course, ethernet does not provide a way to detect this condition
>
>
> By the word 'down' above (and we can define the word more or 
> substitute another word) I do mean that the packets are no longer 
> transmitted or received. Same as the case of cables pulled you 
> mention. Depending on the implementation 'telling the stack', can mean

> various things -- I just mean that it has an effect such that no 
> packets can get transmitted or received.
>
> > possibly, whereas IB does (via the GID deletion trap), so it might 
> > make sense to decide whether IB should do something that ethernet 
> > does not.
>
> > Also, I think we should be clear: I believe link conditions should 
> > not be reflected to the IP stack (aren't we mixing L2 and L3 if we 
> > do that?), but as you mention at other places, the L2 driver can 
> > stop tx/rx of packets. But definitely the driver can not (for 
> > example) "inform" the stack that some IP multicast membership that 
> > was previously granted has been revoked; although it can silently 
> > drop those mcg packets.
>
>
> That is something I've said before -- no link implies cannot send 
> packets. That is even if the IB cable is connected and possibly one 
> could transmit unicast/mcast packets to peers, but the broadcast-GID 
> is gone, then: do not transmit/receive packets. I agree with that. 
> 'Inform' stack as you mention above is not what I had in mind.
>
>
>
> >
> > >
> > > >
> > > > If you are trying to solve the issue of how to change link 
> > > > parameters safely in a racefree manner, please list the steps 
> > > > that the admin/hosts have to do; I don't believe JOIN or traps 
> > > > solve all aspects of the problem, unless the admin iterates thru

> > > > all the groups; contrary to what you are stating above.
> > >
> > > I think we are looking at the issue differently. One way is to say

> > > that the MGIDs have to be looked at by the admin. The other way is

> > > to say that if the broadcast-GID is removed, the host brings the 
> > > interfaces down. All packet transfer automatically stops. The 
> > > other is to also imply that since there is no broadcast-GID the 
> > > derived MGIDs do not make any sense any more and hence the hosts 
> > > need to purge them. The admin doesn't do anything in this case -- 
> > > I believe this has been Vandana's point too.
> >

> > you do any recovery actions on erroneous deletions of the 
> > bcast-GID?) I don't believe Vandana mentioned this, I would like to 
> > hear her comment specifically if its okay to put more requirements 
> > for nodes that do JOIN.
>
> I was, probably not very clearly, giving options: if the link is being

> downed then the rules that can help are -
>
> one way -->
>     if no broadcast-GID should we also drop all MGIDs. If this was 
> done then the admin doesn't have to manually or through some other 
> means have to get the nodes to leave the membership.
>
> > deleted trap will help solve some problem(s), (when a node JOINs or 
> > doesn't JOIN the broadcast GID) please list the problem(s) and how 

> > some different view point, or some architecture or cleanliness 
> > issue, and really does not contribute to solving or creating any 
> > more problems, please state that. I would be very disinclined to 
> > putting more requirements based on this argument, but at least I 
> > (and maybe others) will clearly understand the basis of the 
> > requirement.
>

> above. The suggestion is to track the disappearance of broadcast-GID 
> so that the interface may be 'downed' (we can define downed suitably).
>
>
> >
> > >
> > >  >
> > > > If you are trying to solve the problem of how to track all IP 
> > > > nodes on the fabric, thats a different problem, and yes, I agree

> > > > that requring all nodes to JOIN will give you an instantaneous 
> > > > snapshot. I don't know though how this information can be used 
> > > > meaningfully.
> > >
> > I disagree; I don't believe having nodes EITHER join bcast-GID OR 
> > track bcast-GID solves any problems for the host or the fabric; 
> > specially if I don't see the rules of what a host that subscribes to

> > the trap must do (and secondarily, how those actions might help the 
> > host itself or the fabric administrator).
>
> The host (in the driver or anywhere) must stop transmission or 
> reception of packets on the IPoIB interface(s) associated with that 
> broadcast-GID. That is the MUST clause in (B).
>
>
> >
> > > using both (A) and (B). (B) appears to be simple enough to 
> > > implement -- just a report; same as the reports used with other 
> > > MGIDs.
> >
> I mean implmenting reception of reports for broadcast-GID is not 
> difficult afterall the reports are probably requestd for MGIDs to 
> which one might joing send-only or non-member or 'create MGID' events.
>
> > I thought we are discussing what the host SHOULD/MUST do when it 
> > discovers the bcast-GID deleted erroneously or recreated.
> >
> > >
> > >  > > > > >
>
> <snip...>
>
>
> > > > But to what good though? If the MGID gets deleted, how does it 
> > > > matter how the hosts behave? We agree that the MGID getting 
> > > > deleted when hosts are active is erroneous; assuring the admin 
> > > > that the end nodes will "do the right thing" (declaring link 
> > > > down et al) is dangerous; because he might turn around and 
> > > > recreate the MGID with new properties, and face all the race 
> > > > problems with other existing IP multicast groups still using old

> > > > properties.
> > >
> > > When the link is down -- it follows that no packets leave or are 
> > > received on the interface. So the existing MGIDs cannot be used 
> > > either.
> >
> > So, do the nodes LEAVE these other IP groups (and if so, how many 
> > seconds does the admin wait to make sure all nodes have done the 
> > LEAVEs and thus all old groups are gone, before he can safely 
> > recreate)?
> >
> > Or should the admin just delete these other groups? I think this is 
> > what you were trying to prevent in the first place, ie having the 
> > admin iterate over the groups.
>
> > > What if one views it as - no broadcast-GID is same as no link. In 
> > > IPoE it could be cable pulled or a downed switch. It atleast means

> > > that the interface is down.
> >
> > The distinction is that IP still sends packets to the driver, the 
> > driver might not send it out (or the transmit fails). IP's 
> > memberships to groups are not affected. The ipoib driver can drop 
> > IBA memberships without telling IP, but are you suggesting that?
>
> Yes, by 'down' the interface I meant that the link is down. Whether 
> the down indication causes the IP interface to be marked down or 
> whether the driver just queues packets/drops packets is upto the 
> implementation.
>
> IP's membership is not affected, that is correct. That is certainly 
> not what I suggested.
>
> I did ask if we could, because all MGIDs are derived from 
> broadcast-GID and if the broadcast-GID is down drop the MGID 
> membership? Yes, that amounts to dropping IBA membership without 
> telling IP.
>
> >
> > To conclude, I still can not see that option B is required; I will 
> > refrain from reiterating that opinion, unless I see more logic to 
> > change it.
>
> > > Modification of broadcast-GID is same as moving the NIC to a 
> > > different ethernet segment -- things might not work any more.
> > >
> > > > > > >> Similarly link attributes shouldn't be modified without 
> > > > > > >> some coordination between fabric mgmt and IP software and

> > > > > > >> that coordination shouldn't be part of IPoIB spec just 
> > > > > > >> like IP over Ethernet doesn't talk about how to cope with

> > > > > > >> VLAN reconfiguration or changes in VLAN properties.
> > > > > > >
> > > > > > >In Ethernet we also do not talk about the link-broadcast 
> > > > > > >address determining the existence of the link - in IPoIB it

> > > > > > >is the basic requirement. However, ethernet nodes to 
> > > > > > >respond to a downed link. In IPoIB the disappearance of a 
> > > > > > >broadcast-GID is equivalent to a downed link: no
> > > > > >
> > > > > > I guess this is the main difference between our view points.

> > > > > > You have the notion of a "downed link" while in my mind the 
> > > > > > broadcast group is an integral part of the IPoIB link 
> > > > > > (because it defines the link attributes such as MTU, 
> > > > > > Q_Key...). Without the braodcast group, we don't have a link

> > > > > No, that is not the difference. We are talking the same thing 
> > > > > except for the use of different words and more importantly the

> > > > > reaction to the possibility of the broadcast-GID disappearing.
> > > > >
> > > > > I actually think that we need both as addressed in (A) and (B)

> > > > > above.
> > > > >
> > > > > Vivek
> > > > >
> > > > > >
> > > > > > Jerry
> > > > > >
> > > > > > >one can broadcast and no one new can join. It is a downed 
> > > > > > >link for implementations that do not actively join the 
> > > > > > >broadcast-GID also, since we require that all 
> > > > > > >implementations derive their attributes from the 
> > > > > > >broadcast-GID.
> > > > > > >
> > > > > > >Vivek
> > > > > > >
> > > > > > >> >>We can avoid going into the details in the 
> > > > > > >> >>specification since the details listed above are 
> > > > > > >> >>largely informational. However, I believe that the case

> > > > > > >> >>of 'no-broadcast GID' needs to be specified. Is it not 
> > > > > > >> >>similar to the case of a downed interface?
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > [email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> > [email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
> __
>
> Vivek Kashyap

__

Vivek Kashyap

---

From: Vandana Rao <[email-masked]>
Date: Tue, 07 Oct 2003 08:32:26 -0700

>ii) The broadcast-GID MUST be full-member joined by the IPoIB interface.

This is good news from the perspective of keeping things simple.  I agree 
with the above conclusions completely.
-Vandana

> the hosts
> switches in
> parameters safely
> nodes on the
> discovers
> matter how
> received
> IPoE it
> driver might
> refrain
> some coordination
> You have
> except for the
> link for

---

From: "bill" <[email-masked]>
Date: Wed, 8 Oct 2003 14:49:35 -0700

From your working group co-chair,

Seems everyone fully agrees.  I have Vivek, Kanoj, Vandana, and Jerry -
with no objecting opinions...

Sounds like consensus to me

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of bill
Sent: Tuesday, October 07, 2003 12:00 AM
To: 'Vivek Kashyap'
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB link address

Bill

-----Original Message-----

Thanks
	Vivek

__

Vivek Kashyap