Subject: Re: [Ipoverib] IPoIB link address

From: "H.K. Jerry Chu" <[email-masked]>
Date: Wed, 24 Sep 2003 14:35:55 -0700 (PDT)

Following are my comments (as a co-author not a co-chair).

From: Michael Krause <[email-masked]>

>At 01:25 AM 9/24/2003 -0700, Vivek Kashyap wrote:
>>On Tue, 23 Sep 2003, Kanoj Sarcar wrote:
>>
>> > Vivek Kashyap wrote:
>> > >
>> > > Bill,
>> > >
>> > > [Vandana had asked me this question and so I brought it up with you and
>> > > Jerry. I then requested Vandana to send it to the list. So not a
>> > > coincidence :)].
>> > >
>> > > I discussed it with Jerry and it is clear that the drafts are not 
>> explicit
>> > > in making it a requirement that the attributes have to be the same. There
>> > > are implementations that can work with different Q_Keys (since they 
>> choose
>> > > to support different QPs), however an implementation that cannot/does not
>> > > choose different QPs for the two protocols will have to 'work harder'
>> > > within the driver to make it work. Therefore I agree that a common set of
>> > > Q_Keys (and other attributes) needs to be an explicit requirement.
>> > > Architecturally too, a link shouldn't depend on the upper layer 
>> protocol --
>> > > whether it is v4 or v6 -- it is just a link.
>> > >
>> > > How we do it has two options:
>> >
>> > Hi,
>>
>>Hello,
>>
>> >
>> > >
>> > > 1) Two Broadcast-GIDs
>> > >
>> > >    a) Explicitly require that the two broadcast-GIDs have same attribute
>> > >    values
>> > >    b) Explicitly require that both the broadcast-GIDs are always created
>> > >
>> > >    This is closer to the current specification except we are adding some
>> > >    requirements.
>> >
>> > Note that this scheme allows the fabric administrator (via the creation of
>> > v4 and/or v6 groups) to indicate to the nodes whether v4 and/or v6 
>> operation
>> > is intended on the fabric. Smart nodes can then decide to fail or succeed
>> > setting up IPv4 or IPv6 protocol on the interface (during the DLPI bind
>> > operation or similar). Whether such a way of controlling operation of the
>> > nodes by the fabric administrator is desirable or not is another issue.
>>
>>Agreed that it allows such a function. However, I also believe that such a
>>notion must be rare.  When a user sets up an ip interface he/she expects to
>>just assign IP addresses and send the data out. It would be odd if the user
>>is only able to assign one set of addresses but not the other set or transmit
>>v4 packets but not v6 packets.
>
>Agreed.  Layer 2 should stay as ignorant about Layer 3 and above thus 
>applying KISS, a single QP should support both v4 / v6.

Actually the notion of v4-only interface or v6-only interface is not
rare. Whether or not it's useful or even appropriate to allow a L2
configuration process to influence a L3 decision is a different matter.

Wrt a single QP supporting both v4/v6, two broadcast groups with the same
attribute set allow that too.

>
>> > There is also the issue of broadcast traffic isolation that has been
>> > pointed out; any port which has a only a single qpn accepting v6 traffic
>> > currently does not need to receive broadcast traffic, saving both fabric
>> > resources and host interruption.
>>
>>The v6 qpn need not join the broadcast-GID if option 2) below is chosen. The
>>driver can, on startup, join the broadcast-GID on the qpn associated with v4
>>broadcasts. The attributes can then be borrowed for the v6 qpn without it
>>having to actually join the broadcast-GID. The isolation can still be
>>maintained in those implementations that prefer it.
>
>The expectation was that this broadcast GID was primarily "management" 
>focused (e.g. ARP / ND) thus the overhead / interrupt rate / etc. is 
>expected to be relatively low.  If we all agree that the separation does 
>not impact interoperability, then it is a local issue.

Actually the broadcast traffic is not rare either, not until we move
all the apps off broadcast to SLP/DHCP or well-known multicast.
IPv6 has got it right by basing ND on multicast addresses instead of
the broadcast address.

>
>>I'm unclear on the port supporting only one of the ip protocols and not 
>>the other.
>
>Don't do this with other layer 2 so why do this now?

Ok.

From the responses so far it looks like the majority prefer a single
broadcast group. Therefore I'd like to propose the following amendment
to the draft (similar to what Kanoj has proposed):

1. a single broadcast group will be defined and shared by v4 and v6.
Both v4 and v6 will use the single broadcast group to retrieve link
attributes, thus no room for errors.

2. the above allows the single-QP implementation, which is deemed by
many as a requirement.

3. the draft continues to be silent on the use of multiple QPs. It is
completely left to the implementations.

4. IPv4 broadcast address (255.255.255.255), as well as all the
net-directed, subnet-directed and all-subnets-directed broadcast
addresses will map to this combined broadcast group.

5. The current multicast mapping stays unchanged.

6. The broadcast group takes the following format:

FF1X:401B:<P_Key>:0::1

This allows the IPv4 all-node multicast address (224.0.0.1) to map
naturally to this broadcast group too, saving one IB MC group.

If people don't like mixing L3 all-node multicast with L2 broadcast,
we can use the existing v4 broadcast group, or even something else.

7. So long as we don't pick FF1X:601B:<P_Key>:0::1 as the combined
broadcast group, the IPv6 all-node multicast address will not share
broadcast traffic with IPv4. Thus we continue to maintain some
level of traffic separation. This addresses my previous concern.

Comments are welcomed.

Jerry

>
>
>> >
>> > >
>> > > 2) One common broadcast-GID
>> > >
>> > >    a) One broadcast-GID that the driver joins. [the protocol 
>> signatures can
>> > >    still be included in the multicast mappings].
>> > >
>> > >    This is architecturally cleaner and may be for implementation too --
>> > >    only one broadcast-GID to be created and joined.
>> >
>> > Yes, its does seem cleaner and simpler, and makes fabric configuration
>> > errors less likely (note that the fabric administrator still can create
>> > a multicast group corresponding to one of the v4/v6 IP multicast addresses
>> > with the wrong qkey).
>>
>>Yes that is possible. However, in general I would expect the setup to be such
>>that the multicast GIDs are created by the nodes based on the broadcast-GID
>>attributes. Since first full-join is a create this shouldn't be much of a
>>problem for creating the all-nodes v6 multicast GID.
>>
>> >
>> > If this common group is adopted, one possibility to think about is making
>> > the link broadcast address the same as the link multicast address for
>> > 224.0.0.1 or FF02:0:0:0:0:0:0:1, resulting in a reduction of one
>> > multicast group required for v4 and/or v6 operation on the fabric.
>>
>>I prefer your alternative suggestion. I am viewing the common broadcast-GID
>>to be the same as the all-1's link address in Ethernet. The multicast
>>mapping is only for IP multicast addresses same as the ethernet
>>model (the mapping itself is different from ethernet of course).
>>[nit: it is FF12::1 and not FF02::1].
>>
>> >
>> > Alternatively, what might work best (ie requires minimal changes
>> > to existing implementations) is naming the current v4 broadcast group as
>> > the common group, and eliminating any implications that the v6 all-node
>> > group should be created by the fabric administrator.
>>
>>Yes, this works well. I'm ok with that though would still prefer that, if
>>this option is chosen, we drop the protocol signature from the
>>broadcast GID.
>
>Mike
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 24 Sep 2003 15:01:26 -0700 (PDT)

> configuration process to influence a L3 decision is a different matter.

Interface at the IP layer is fine..however, in this case the lower
layer will dictate/deny the IP  layer from even defining an IP
interface supporting the other version. 

> IPv6 has got it right by basing ND on multicast addresses instead of

Yes, however, as I noted earlier an implementation can always join the
broadcast-GID on one QP and then borrow its attributes for v6
multicast related QPs; it does not have to join the broadcast-GID
itself. Therefore an implementation can choose to keep the traffic
separate.

> 6. The broadcast group takes the following format:

My suggestion is to allow the broadcast-GID to be independent of
multicast mappings. Therefore, it would take the form:

FF1x:0000:<P_Key>::255.255.255.255.

Or, if a signature is preferred:

FF1x:001B:<P_Key>::255.255.255.255

The multicast mapping can have the protocol signature in it. As I
noted earlier we can view this as equal to the all 1's ethernet
address. The multicast mapping can continue to, again analogous to
Ethernet, as :

FF1x:401B:<P_Key>::<28 bits>
FF1x:601B:<P_Key>: <80 bits>

Vivek

---

From: Kanoj Sarcar <[email-masked]>
Date: Wed, 24 Sep 2003 16:02:18 -0700

> interface supporting the other version.

I think we have already decided on the other solution, but maybe I can
clarify my comment here. 

Notice that whatever the IP layer does, finally device creation, 
and/or possibly IP configuration, is dependent on an L2 property for 
IPoIB, namely the existance of broadcast group(s). I am not suggesting 
everyone's implementation should error out V4 and/or V6 configuration
depending on the group's existance, but pointing out that is a possibility
for certain implementations (in lieu of a central administrative tool 
that can control v4/v6 plumbing on a node basis). 

Why I think it might be more important to control v4 and/or v6 protocol
plumbing in an IBA fabric (vs ethernet) is that each v4 and v6 configuration 
on a node adds routing overhead to the fabric for the various multicast
groups. This is absent in ethernet (or at least the switches' spanning
trees know how to gracefully handle too many multicast groups). Section
11 of the multicast draft discusses some of these limitations.

> itself. Therefore an implementation can choose to keep the traffic

Agreed. By extension, I think we should be careful not to require a 
node to JOIN the combined broadcast group if it can determine that 
only v6 will be used on the device, and leave that as an implementation 
decision.

Thanks.

Kanoj

---

From: Vandana Rao <[email-masked]>
Date: Wed, 24 Sep 2003 22:07:52 -0700

At 04:02 PM 9/24/2003 -0700, Kanoj Sarcar wrote:

<snip>

>11 of the multicast draft discusses some of these limitations.

The issue with IB multicast resource limitation has nothing to do with 
allowing v4 and/or v6 configuration.
It has to do with using too many  (v4 or v6) IP multicast groups. I do not 
understand what you mean by each v4/v6 configuration requiring additional 
routing overhead to the fabric for the various multicast groups. I do not 
believe configuring a new IP address will add any extra overhead to the 
fabric for multicast groups.

>only v6 will be used on the device, and leave that as an implementation

Again, could you clarify what you mean here? The node HAS to join the 
combined broadcast group
if it has to support either or both protocols. In the example given above, 
the v4 QP  (if different for v4 and v6) does not have to attach to the 
broadcast MGID.
-Vandana

---

From: Kanoj Sarcar <[email-masked]>
Date: Thu, 25 Sep 2003 11:35:59 -0700

Hi,

> fabric for multicast groups.

This is academic at this point, but when v4 or v6 is configured on a device, 
normally IP tries to JOIN to a few multicast groups (224.0.0.1 et al). These 
map to IBA JOINs, which are messages to the SM to arrange the switching tables 
in all the IBA switches so that the multicast packets are appropriately 
forwarded. Thats fabric overhead, and also consumes switching resources at 
the IBA switches.

> the v4 QP  (if different for v4 and v6) does not have to attach to the

If a node only ever intends to support v6 traffic, it might have no reason 
to join the combined broadcast group; rather it can just QUERY the attributes
for the combined group, and then just JOIN the appropriate V6 multicast 
groups as instructed by IP.

Thanks.

Kanoj

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 25 Sep 2003 11:56:45 -0700 (PDT)

Kanoj/Vandana,

	Could we say that it is upto the implementation on how the
	attributes are received? If there is a query attributes call as
	Kanoj says then it could be used - my concern is that the
	broadcast-GID cannot be deleted if someone is using it. It can be
	if no one is -- that is one of the reasons for joining it.
	
	I'll attempt a summation of the changes. Please comment:

	1. Define a broadcast-GID defined as follows (one of - please
	comment):

	a) FF1x:401B::255.255.255.255 
	b) FF1x:001B::255.255.255.255
	c) FF1x:0000::255.255.255.255
	d) FF1x:401B::1

    My preference is b) or c) since it removes the IP version
    completely from the mapping. a) is what we have today and requires
    no change and so could be used. d) is derived from the multicast
    mapping and so I'd like to leave it separate.

	2. The multicast mappings stay as currently specified.

	3. The broadcast-GID (one of)

	a) MUST be FullMember IB_joined 
		
	b) MUST be used to derive the attributes such as the MTU, Q_Key.
	The attributes MAY be obtained by FullMember IB_Join of the
	broadcast-GID or by some other operation supported by IBA. The
	node MUST fail all IP operations on the interface if the
	Broadcast-GID ceases to exist since that will correspond to a
	downed link. It is upto the implementation to track the existence
	fo the Broadcast-GID in a suitable way. Note: If the broadcast-GID 
	is FullMember IB_joined then its existence/deletion is
	automatically tracked. 

> forwarded. Thats fabric overhead, and also consumes switching resources at 

Overloading of MLIDs can mitigate some of it.

---

From: Kanoj Sarcar <[email-masked]>
Date: Thu, 25 Sep 2003 13:49:26 -0700

Hi,

>     mapping and so I'd like to leave it separate.

I would go for a) since it is already there; if we do change, I would
go for d, but can live with b or c.
 
>         is FullMember IB_joined then its existence/deletion is

I would go with option b, but without the requirement "MUST fail all IP
operations ..." for multiple reasons; firstly, it requires at least a
memory check in the send/receive path, and secondly, if other L2's (ethernet)
aren't requring this, why should IB? (Seems like mingling an L2 issue with L3, 
the same logic that was used against seperate groups). 

Also, the Note: seems informational, but it does not mention that an 
implementation can subscribe to group creation/deletion traps to become 
aware of the broadcast-GID coming and going.

Thanks.

Kanoj