Subject: RE: [ipoverib] moving forward with the open Issues - end node configuration

From: "Aditya Dube" <[email-masked]>
Date: Fri, 25 Jan 2002 17:17:58 -0800

Hi folks

The Software Transport Verbs have this limitation:

1) To Attach to an MCGroup, one needs a Queue Pair
2) To create a Queue Pair, one needs a P_Key.

Therefore, a node needs a P_Key before it can join any MCGroup or create
a Queue Pair. I propose that there be a well-known default ServiceRecord
that we use to configure the P_Key and MTU.

On another note, the Administrator is welcome to override our defaults
and set up as many subnets, VLANs or P_Keys as he or she desires. That
is outside the scope of my proposal.

Aditya Dube

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]] 
Sent: Friday, January 25, 2002 12:04 PM
To: [email-masked]
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration

<VK> I am not sure what you gain by that. The P_Key is programmed into
the
port by the SM on startup. You already have it in the P_Key tables.  How
you determine the P_Key to use is upto your administrator.

If there are multiple IPoIB subnets over the fabric - how do you know
which
serviceID to choose ? These will have to be specified by the
administrator.
So the administrator might as well specify the P_Key per IP interface.
You
may automate the process by ensuring that your interface tries all the
P_Keys in the relevant port.

The P_Key then allows you to determine the all-nodes multicast/broadcast
address (as defined in the multicast draft). This address that has to be
setup by the administrator (first node/other method). Every interface is
required to join this address. The MTU returned on the join request is
the
MTU of the interface. So the MTU is automatically configured. The join
request is less than 256 bytes and hence will be received by the SM
whatever be the link MTU/path MTU.

Zero-conf can be done in the following way though:

      The P_Keys are in the P_Key tables. Let, by default, there exist
an
MGID formed by zeroing out the link-id (P_Key) field in the MGID. So for
IPv4 it will be FF12::255.255.255.255. This group is joined by the
interfaces. If they are in the partition of this MGID they will receive
the
right P_Key to use for the subnet. They will also get the subnet MTU.
There
can be only one such subnet in the IB subnet. If multiple IPoIB links
are
defined then this scheme won't be universal but it allows complete
automation in the default case.

For multiple IPoIB links:

      If the IPoIB ports can support at least two P_Keys (since we are
looking at multiple IPoIB links) we can always define a default subnet
(as
above). A DHCP server can be put on this IPoIB link. A node then
enquires
this DHCP server for the P_Key. I can suggest the DHCP
extensions/process
required if needed.

Comments ?

Vivek
<VK>

The current IPOverIB spec requires that the P_Key and MTU be configured
on each node.

The IPOverIB spec should provide a default method for an end-node to
come online with no IPoIB-specific configuration.

During initialization, a node could query an IPOverIB specific
ServiceRecord (Table 164, pg 685). This query will return a P_Key and
ServiceSpecificFlags. The ServiceSpecificFlags can specify whether 2048
or 4096 is the link MTU.

The ServiceRecord's ServiceID should have a default value specific to
IPOverIB. This number could be 0xFFFF:4000 in accordance with vol3
section on "ServiceID Namespace" (Table 128, pg 215).

The ServiceRecord needs to be created just as the all-node or broadcast
MCGroup is created, i.e. by the first node on the network.

If the Network Admin wishes to put the end-node on a different P_Key,
then the Admin may override the values returned by the ServiceRecord.
That is outside the scope of my proposal.

Aditya Dube

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 26 Jan 2002 14:43:18 -0800 (PST)

There are two levels of configuration tasks needed for IPoIB. One at the
link level the other at IP level. The link and multicast draft addresses
the configuration steps required to set up IPoIB "links". The equivalent
in Ethenet network is setting up VLANs. In Ethernet, configuring VLANs
to run IP is much simpler because there is no MTU issue, no special
steps required to enable broadcast/multicast... In a way you can use
the link and multicast draft to apply it to other, non-IP networks
too. It contains all the generic steps for setting up a link for
a broadcast/multicast capable network.

You should consider P_Keys of a CA port as defining "virtual" network
interfaces for the local node. E.g. if an HCA port has been assigned
ten P_keys, when starting up the network subsystem, the OS must lookup
from SA for the special all-node MC group with MGIDs defined in the
draft. If only three are found, then only three "virtual" network
interfaces will be enabled to run IP.

Once the link layer is fully set up, the configuration of IP subnets
on top of these IPoIB links are really no different from other IPoverFoo
(e.g. Ethernet). It is my hope that all the nice work IETF has done in
the area of autoconfiguration/zeroconf can be applied to IPoIB with
no (or minimal) changes.

Do you still see a need for special ServiceRecords, given all the info
readily available from MCGroupRecords?

Jerry 

>
>Quick comment,
>
>I thought Vol 3 was never an approved IBTA specification and was dropped...
>
>I definately agree with you that a zero-config type environment would be
>nice for IPoIB traffic, however there might be some other alternatives.
>
>I can see DHCP extensions
>I can see Service Records (can the IETF define these Mike Krause ?) like you
>suggested
>I can see manual configuration - technically not zero config at this point
>I can see defaults in the protocol - Though SHOULD use 2K MTU and a P_Key of
>mikimitdigoat or I'll even let you use a ascii representation of your
>initials if you are vain <G> (Just don't tell anyone)
>
>Do you think you could write a proposal describing how any/all of these
>methods that you think will work and any others you might come across...
>
>Thank you
>Bill
>
>From: [email-masked] [mailto:[email-masked]]On Behalf
>Of Aditya Dube
>Sent: Friday, January 25, 2002 11:17 AM
>To: [email-masked]

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 26 Jan 2002 15:25:20 -0800 (PST)

Yaron,

>I agree that some method should be specified in the future
>However it is not so trivial to do it right now
>a. There are issues between IBTA & IETF on things like service records etc'
>b. You could have few VLAN's on the same IB subnet, how would you
>distinguish between them, etc' (maybe a service ID range)

For each P_Key in a port's P_Key table, simply do a lookup to SA for
an IPoIB MGID defined in the link and multicast draft. If the lookup
succeeds, the IB partition defined by the P_Key will be used to host
IPoIB, otherwise it won't. Isn't that pretty simple?

>
>I would tend to go for an RFC with a static configuration, with known
>defaults like:
>The default P_Key value, 1500 MTU
>Work on a follow on proposal either in IBTA or IETF to address the
>automation

Static configuration is a thing of the past. Perhaps the motivation here
is to bypass SM/SA again (like the LID proposal)? I can sympathize with
the concern that SM/SA may be so complex that having a dependency on
it is a bit risky. But let's see how feasible a bypass is. You can
hardwire MTU, Q_Key, TClass... But you really need a broadcast mechanism
(simulating broadcast by unicast is real messy) so you've got to have a
MLID. Are you going to hardwire that too?

If the concern is about the complexity of SM/SA, why not just dummy up
a SM/SA implementation and hardwire all your favorite parameters there?
This way we don't need to muck with IPoIB standard.

>
>You probably wonder why 1500, some of the reasons originally for determining
>that IPoIB supporting devices will have minimum MTU of 2K was to maintain
>similarity with Ethernet
>Since in IB you have either 1K or 2K, 1500 falls in the 2K domain, so the
>device must support 2K, not to say the MTU should by 2K
>
>Why 1500 again, lets say you have an Ethernet to IB bridge/router it will
>have to fragment/reassemble the packets
>This will create a lot of overhead on both the hosts and the routers that
>will significantly affect performance

There are two very different cases that you mentioned here, bridge and
router. For the latter, no sweat, PMTU discovery takes care of it so
you don't need to muck with individual link MTU. (That's the whole
purpose of PMTU.) For bridges, no problem, just put your favorite
MTU value in the MCGroupRecord when you set up an IPoIB link to bridge
with an Ether segment.

Jerry

>
>(it might be 1400 & something)
>
>Yaron
>
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Aditya Dube
>Sent: Friday, January 25, 2002 9:17 PM
>To: [email-masked]

---

From: Yaron Haviv <[email-masked]>
Date: Sun, 27 Jan 2002 22:08:11 +0200

Jerry,

I guess you just missed my point; it was not to say that manual
configuration is good, and we need to avoid the SM
I just said that the process of agreeing on an automatic solution is not
simple, and I would like to get an RFC first, than quarrel over this
mechanism
Not to mention that it might involve additional standardization from IBTA
(e.g. determining ServiceID's, etc') which wont make it faster
This is the same reasoning like abandoning the capability bits issue today

As for your proposal I'm not so sure that it is so efficient going over all
the P_Key's defined, in some cases there could be more than a few
I don't have strong feelings against it, just rather see something more
efficient, if possible
On the same query to the SM we might also get more info on that IP subnet
that will ease the configuration, and shorten the booting process

Regarding the 1500 MTU, I know all about PMTU
The 2K adds some load on the network, which also depends on the topology and
traffic characteristics
It could be acceptable by many people
I would just want to enable setting the values to a number lower than 2K, if
the admin requires so
I'm not against setting 2K values for anyone that wants it, or even as the
default magic number for IPoIB as long as smaller numbers are also valid
And from your answer it looks like you agree to that too

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
H.K. Jerry Chu
Sent: Sunday, January 27, 2002 1:25 AM
To: [email-masked]; Yaron Haviv
Subject: RE: [ipoverib] moving forward with the open Issues - end node
configuration

Yaron,

Jerry

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sun, 27 Jan 2002 12:57:02 -0800 (PST)

Yaron,

>simple, and I would like to get an RFC first, than quarrel over this

Disagree. See my other mail. IETF already provides a number of mechanisms
for autoconfiguration that apply to IPoIB as well as any IPoverFoo. All
we need to solve here is to autoconfigure IPoIB links.

>I don't have strong feelings against it, just rather see something more

Note first that this link configuration process only occur once per node
reboot. The efficiency concern IMHO is even less convincing here than the
one that argues for inlcuding LID in ARP. Second, there is nothing that
prevents implementations from supplementing additional configuration
files on an endnode that cache/specify IPoIB P_Keys, for example. These
local configuration practices don't present any interoperability problem,
therefore does not need to be standardized.

>And from your answer it looks like you agree to that too

Yup. I've been against some earlier proposals to designate a fixed MTU size
(e.g. 2K) for IPoIB because I don't think that is necessary. This is covered
in the link and multicast draft by using the MTU field in the MCGroupRecord.
But an issue just came up to me if one wants to support 1500 MTU. Contrary
to the "IP MTU" that allows one to specify an arbitrary size as long as
it's not greater than the link MTU, the "IB MTU" in the MCGroupRecord
field only allows fives choices from 256 to 4096.

We can either
1. punt it, since other IPoverFoo doesn't support the use of arbitrarily
configured MTU size either. (How does bridge vendors solve this problem when
bridging FDDI with Ethernet?)

or
2. find some place to store the IP MTU (ServiceRecord? I prefer not.)

Any suggestion?

Jerry

---

From: "Aditya Dube" <[email-masked]>
Date: Tue, 29 Jan 2002 22:42:23 -0800

I would like to withdraw my proposal for zero config as we will need the
help of the IBTA to reserve special values for the Service Record.

Configuration of the P_Key and the MTU should be an end-node issue in
the IpOverIB spec. 

Aditya Dube