Subject: RE: [Ipoverib] More musing on address format

From: "Larsen, Roy K" <[email-masked]>
Date: Thu, 21 Mar 2002 13:51:16 -0800

/rkl
embedded
rkl/

-----Original Message-----
From: Jim Pinkerton [mailto:[email-masked]]
Sent: Thursday, March 21, 2002 11:19 AM
To: Larsen, Roy K; Michael Krause; [email-masked]
Subject: RE: [Ipoverib] More musing on address format

Roy,

A quick comment on how I see SDP being plumbed, using IPoIB for address
resolution. 

Since SDP really sits at the transport layer, I'm assuming there is a
structure somewhere which maps the destination IP address to a link
address. This might be the ARP table entry, it might not. In addition,
you need the equivalent of a route table entry to select the local
interface to go out of.

/rkl
Yep, and this is at the crux of the problem.  Those data structures, in the
OSs I'm interested in, have a fixed length array to store the hardware
address and it is less than 20 bytes.  I am not an OS vendor or supplier, so
I have little say in this.  The OS defines the device driver interface, and
I comply.  This means that my driver will perform a mapping function that
will convert IPoIB 20 byte addresses in ARP/ND frames to something that will
fit and convert them back for outgoing ARP/ND frames.  The OS network
infrastructure will never see the "true" device hardware address because it
simply can't deal with it.  I freely admit that this is a *personal*
problem.  But it is a problem every HCA vendor providing an IPoIB driver is
going to have to deal with on those OSs they can't control.  The
ramification of this is that the existing OS tools that display device
hardware information will display the re-mapped address - not the "true"
one.  The discussion I was trying to spur was; if this is undesirable,
should we try to come to an agreement on a re-mapping function so all
implementations requiring re-mapping produce the same result?
rkl/
                         Note that this has nothing to do with P_Key - it
is probably a table created through normal routing algorithms. Once the
local interface is know, a variety of host specific methods can be used
to find out fun information about the local interface (P_Key, APM type
issues, etc, etc).

/rkl
If you have control over the OS, perhaps you can provide a host specific
method.  At the very least, one can use a device driver specific ioctl(2)
but that feeds into device driver implementation dependencies for
applications such as SDP and uDAPL.  Not a promising start for IBA
networking.
rkl/

Thus I don't see any of this being either visible to the user or visible
on the wire - particularly P_Key.

/rkl
I hope no one on this list believes that the P_Key is going to provide any
security for IPoIB.  As we agree above, there needs to be a host interface
to get all the fun information which includes the P_Key.
rkl/

I believe pretty strongly that we haven't seen a compelling reason to
re-open the GID/LID debate - I'm hoping we can put this one to bed. I
agree though that the architecture we have come up with means you can't
simply re-use the existing Ethernet software infrastructure - it was a
pill we had to swallow quite some time ago though.

/rkl
I agree there is no single killer reason - just a death of a "thousands"
cuts.  (OK, maybe only three for four.)  To say we can't simply re-use the
existing [network] software infrastructure implies a power that I, and many
HCA vendors, don't have.  It was my understanding that one of the basic
tenets of our work was to not impact existing IP protocols, applications, or
core OS network infrastructure.  We have failed on all but the former.  The
long address issue is a personal problem I and others can work around.  It's
just that the result will be less than desireable.  Given that 10Gbit
Ethernet is right around the corner, I'd like to make the choice for using
IBA for general networking in the data center as compelling as possible.

Consider my side of the shortened IPoIB address debate closed.  I'd still
like to hear thoughts on common re-mapping algorithms and standardized
driver interfaces to obtain address vector information.

Roy
rkl/

Jim

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]] 
Sent: Thursday, March 21, 2002 9:26 AM
To: 'Michael Krause'; [email-masked]
Subject: RE: [Ipoverib] More musing on address format

/rkl
Comments embedded
rkl/

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Wednesday, March 20, 2002 10:59 PM
To: [email-masked]
Subject: RE: [Ipoverib] More musing on address format

At 02:13 PM 3/20/2002 -0800, Larsen, Roy K wrote:
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:[email-masked]]
> > Sent: Saturday, March 16, 2002 4:22 AM
> > To: [email-masked]
> > Cc: [email-masked]; [email-masked]
> > Subject: Re: [Ipoverib] BOOTP/DHCP chaddr field
>
><snip>
>
> >
> > > >Alas, a LID-based hardware address would be my first choice too.
> > >
> > > Or keep GID but drop QPN and use SIDR as you've mentioned too.
> > >
> >
> > I am more open to using LID + QPN rather than GID + SIDR.
>
>Our SDP and uDAPL folks have just realized that the P_Key is not a
formal
>part of the IPoIB hardware address.  Both of these protocols are
relying on
>IPoIB for address resolution to obtain the GID and they are going to
need
>side/proprietary interfaces to the IPoIB driver to retrieve the P_Key
>associated with an IP address.
>
>Although I realize this is a personal problem for them, a hardware
address
>that consisted of P_Key + LID + QPN would be ideal from my point of
view.
>It is 6 bytes (how convenient), fully qualified within the scope of the
IB
>subnet, and would be a consistent representation of the hardware
address at
>the administrative level (ipconfig(8), arp(8), etc.).

The P_Key is derived from the multicast group that one must join to get
the 
GID so there is no reason to use it as part of the link address.

/rkl
I am aware of that.  The reason for adding the P_Key was to help support
SDP
and uDAPL implementations that have a dependency on IPoIB for address
resolution.  These folks are going to need the P_Key for the subnet and
I
was trying to get away from having dozens of IPoIB driver proprietary
interfaces to get it.  I know these protocols are not our concern.
However,
I tend to take a global view in making IBA a success.  The better we all
play together, the more compelling it becomes to install IBA in the data
center.
rkl/

>The latter part of that benefit list is something that may warrant more
>discussion.  That is, is there a desire to have a universal mapping of
IPoIB
>hardware address at the administrative interface?  For Ethernet, all
>administrative tools in all OSs present the hardware address in a
universal
>and consistent fashion.  If we have a 20 byte IPoIB address, there are
going
>to be several OSs that will need to re-map that to a smaller value for
>internal network stack consumption (this is how I stumbled onto the
DHCP
>issue).  If my re-mapping algorithm is different from others,
administrators
>will not see the same hardware address listed for the same node from
system
>to system.

This is why we have ARP helper modules in some implementations so that
the 
format isn't as critical any longer.

/rkl
I'm not sure how this addresses my question.  How important is it that
an
administrator see the same IPoIB hardware address displayed when she
does an
ifconfig(8) on the host system and an arp(8) on all the others?  Does it
matter that they don't match or even show the same number of bytes in
the
address from system to system?  If your suggesting that there would be
IPoIB
specific tools that would display all IPoIB information in it's true
form,
that's an approach.  It's just a little awkward especially on data
center
systems that have both IBA and Ethernet connections.
rkl/

>I know there are some on the list that believe the data center will
employ
>lots of small-grain IB subnets interconnected by [relatively] expensive
IB
>routers.  I've never found that compelling.  Given all the ambiguity
there
>currently is with IBA routers, the considerations above just becomes
more
>food for thought on address format....

I doubt there will be many routers used - I also doubt they will be very

expensive.  I tend to view things as a series of partition domains that
may 
span a portion of a subnet or across subnets.  These domains are
centrally 
managed and the IP communication is per partition domain.

/rkl
This is a reasonable view and one I would like to support.  I'm just
having
trouble implementing it without hacks to the OS and some protocols like
DHCP.  So the cost/benefit analysis puts me on the other side.
Certainly
network administrators are comfortable with multiple subnets in the
Ethernet
world and I think they would be just as comfortable with it in the IBA
world.  They could still make their IPoIB subnets partition based - they
just wouldn't configure the partition to cross an IB subnet boundary.
rkl/ 

Mike

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Yaron Haviv <[email-masked]>
Date: Sat, 23 Mar 2002 16:48:25 +0200

Roy,

I have hard time understanding what it all got to do with P_Key ?
I also think (Know) that there are ways to implement SDP in ways that will
not create the limitations you mention

The only problem I take from here is the address been longer than what the
standard stack supports
This problem doesn't relate to SDP or P_Key, it is a general problem

If the HCA vendors want to accelerate the IB industry, and help it to
compete with 10GbE
They need to provide real HCA HW sooner, and not confuse the market with
contradicting messages

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Larsen, Roy K
Sent: Thursday, March 21, 2002 11:51 PM
To: 'Jim Pinkerton'; Michael Krause; [email-masked]
Subject: RE: [Ipoverib] More musing on address format

/rkl
embedded
rkl/

-----Original Message-----
From: Jim Pinkerton [mailto:[email-masked]]
Sent: Thursday, March 21, 2002 11:19 AM
To: Larsen, Roy K; Michael Krause; [email-masked]
Subject: RE: [Ipoverib] More musing on address format

Roy,

/rkl

/rkl

/rkl

/rkl

Roy
rkl/

Jim

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]]
Sent: Thursday, March 21, 2002 9:26 AM
To: 'Michael Krause'; [email-masked]
Subject: RE: [Ipoverib] More musing on address format

/rkl
Comments embedded
rkl/

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Wednesday, March 20, 2002 10:59 PM
To: [email-masked]
Subject: RE: [Ipoverib] More musing on address format

formal
relying on
need
address
view.
IB
address at

/rkl

IPoIB
universal
going
DHCP
administrators
system

/rkl

employ
IB
there
more

/rkl

Mike