Subject: RE: [Ipoverib] IPoIB/ifconfig question

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 16:33:59 -0800

/rkl
coments in-line
rkl/

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 3:22 PM
To: Larsen, Roy K
Cc: [email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] IPoIB/ifconfig question

"Larsen, Roy K" wrote:
> 
> The more I think about it, the more I'm in favor of the condensed/IPv6
> format.  I really would like to consider the aesthetic of the format.
> Although my mailer is probably going to wrap these lines (a hint at what
it
> would look like on an 80 character xterm), consider the following displays
> from a Linux system.  The first is the standard output (80 character line
> length) followed by option #1(120 chars), #3(88 chars) and #4(107 chars).
> Note that the arp header is constant meaning that it would take the new
> field widths for all network devices including Ethernet.  Also keep in
mind
> that the primary reason to display this info is for a human to read.  Of
> these, I believe the admin would want to use option #3 for input and
output.
> It's easy enough to change/add a MIB definition for a IPoIB hardware
address
> (as opposed to the GUID display hint that folks have alluded to).
>

Roy,

You might also want to try a non-default GID prefix, specially one that does
not have too many runs of 0s. I am not sure if the Linux implementation will
allow that, but it is probably possible on other systems. In that case, the
benefits of #3 might be limited (whereas 1, 2 and 4 are independent of
actual
values).

/rkl
I take your point but my assumptions is that there would always be display
economy using the condensed format
rkl/

Another suggestion is to eliminate leading 0's for all options (that can
only shorten the output) when you compare.

Comments below.
 
> Other comments in-line /rkl
> Roy
> 

I am including output from a Solaris system here, similar to what Roy did
for Linux:

hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.7.198.20 netmask ffffff00 broadcast 10.7.198.255
        ether 8:0:20:d9:4:97
ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 504 index 3
        inet 192.129.100.100 netmask ffffff00 broadcast 192.129.100.255
        ib 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81
root@pap# arp pap
pap (10.7.198.20) at 8:0:20:d9:4:97 permanent published
root@pap# arp pap-ibd
pap-ibd (192.129.100.100) at 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81
permanent established

/rkl
Clearly the Solaris output is more suited for long hardware address display.
This is not the case for Linux....
rkl/

<snip>

> I think we are down to #1 and #4, right? (Another point against #3 is that
> it *might* be confusing to have a v6 protocol layer addressing scheme
> applied
> to a link layer address; specially since the link layer address is not
> 128bits
> long).
> 
> /rkl
> As Roland points out, GIDs are IPv6 addresses so it doesn't bother me that
> they follow an IPv6 format.  The admin tools make it very clear where the
hw
> address is displayed
> rkl/

But we are talking about GID+QPN here, and since thats 20b instead of v6
16 bytes, I don't know whether this is a good idea. The other part is, the
GID is similar to v6 in the context of IBA network layer, but that same
GID is being used as (part of) a link layer address for IPoIB. 

/rkl
I'm not sure I appreciate why this would matter.  Although the componets of
the hw address are generally not a concern to the admin, the GID is a
component.  I'm just looking for a way to make them more user/display
friendly in the same way the IPv6 folks made their address more friendly.
rkl/

> 
> Here they are (with QPN=0x123456):
> 
> 1. 00:12:34:56:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
> (or 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81)
> 4. 123456:FE80:0000:0000:0000:0002:c900:011c:9781
> (or 123456:FE80:0:0:0:2:c900:11c:9781)
> 
> Given the irregularity (length wise) in the first few digits in 4,
> I would think #2 would be better. Here it is:
> 
> 2. 12:3456:FE80:0000:0000:0000:0002:c900:011c:9781
> (or 12:3456:FE80:0:0:0:2:c900:11c:9781)
> 
> I do have a preference for #1 still, primarily because
> 
> a. it is the most easily adapatable to the "arp" command (ie, when you
> are doing a "set" operation, it is very easy to teach "arp", which on
> most systems currently only expect ethernet addresses, how to parse a
> #1 style ib command, in contrast to #2 or #4). Any other format picked
> can of course be handled, I have seen the least amount of work with #1
> while prototyping (I am dealing with Solaris, but I expect that to be
> true on most other systems like BSD too; Linux has a -H option, so this
> is not an issue).
> 
> /rkl
> When you say adaptable to the arp(8) command, are you taking into account
> the display headaches I try to illustrate above?  Given that the IPoIB
> hardware address is virtual and the pathrecord is always required, do you
> think "set" is viable?  Even so, do you think the admin would want to
enter
> 60 characters as opposed to something around 27?
> rkl/

We have to allow that "set"s might be meaningful in certain implementations.
Does the Linux implementation allow it? If so, what is the syntax of the
command, and how was "arp" modified for accepting IPoIB link address input?
(For Solaris, which is similar to BSD, I can do
"arp -s hostname 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81"
I had to teach arp to accept not only a:b:c:d:e:f ethernet format, but also
the 20b format).

/rkl
Although I have taken a look at the Linux kernel and network admin tools
that are affected, the SourceForge project has not jet tackled these.
Currently they truncate the IPoIB hardware address.  I believe the Linux
maintainers have just accepted the changes needed to deal with the
kernel/user interface (the sock_addr size issue).  Yes, Linux arp(8), as a
facility, allows entries to be set.
rkl/

Yes, I am aware there are display issues, but I think they might be
different
issues on different platforms (eg Solaris prints the link address on a
seperate
line, so its lesser of an issue).

/rkl
This is my primary concern since changes here will affect the display format
for all network devices including Ethernet on Linux.  It appears the Solaris
format makes long hw addresses painless.  This is not true for Linux and
must be taken into consideration.  The customer experience is more important
to me that a standardized format.  If changing the core Linux display format
for ifconfig(8) and arp(8) is acceptable, that may be a way to satisfy both
interests.
rkl/

<snip>

> What about (sysadmin) scripts that parse output from ifconfig (ie look at
> HWaddr field), which currently assume ethernet format?
> 
> /rkl
> But I assume they already assume a fixed length representing 6 octets so
> they are going to have to change anyway.  At most, they are going to
> comparisons so I don't think this a compelling argument for #1
> rkl/
> 
> I assume you are talking about Linux (and thus you have the -H arp option
> to help)? Notice, you could still build in automatic parsing in Linux
"arp",
> 
> which on seeing format #1 will recognize it as an IPoIB address (ie,
> the default type for 20b addresses can be IPoIB, similar to the default
> type for 6b addresses is ethernet; this would allow admins not to have
> to specify -H for each IPoIB operation, thus making IPoIB just a little
> more similar to ethernet); I imagine it will be just a little harder to
> do #2 or #4.
> 
> /rkl
> I believe the -H option only acts as a filter, not a display formatter.
> rkl/

Okay. I am still curious about how Linux/IPoIB does a arp set operation.

/rkl
When implemented, it would be set in the same way as any non-default (ether)
network device:
arp -H infini -s <hostname> <hw address>

The format of the address is network device dependent so any format would be
acceptable.

Roy
rkl/

Thanks.

Kanoj

<snip>
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Kanoj Sarcar <[email-masked]>
Date: Tue, 14 Jan 2003 18:08:28 -0800

> economy using the condensed format

True. But see below what the (at least perceived) reasons are against this format.
I think the main problem is in formatting the ifconfig/arp output; adoption of
a v6-like format does not guarantee you solve that problem for non default GIDs
(on Linux, that is).

> This is not the case for Linux....

True, it can accommodate longer addresses by virtue of being on a seperate
line (and I am not claiming anyone did this by design). Notice that both
Linux and Solaris places the inet6 address on a seperate line. I don't think 
I would suggest trying to change Linux's ifconfig to print the HWaddr on a 
seperate line.

> friendly in the same way the IPv6 folks made their address more friendly.

Agreed; I am just pointing out ([in]valid?) reasons why this might be confusing. 

> facility, allows entries to be set.

Off topic, but I am very curious to find out exactly what sock_addr changes
were accepted. Can you provide pointers (names of ioctls, or structures, in
2.4.X or 2.5.X)?

Also, I am not sure of the ifconfig output that you provided: was that based
on hacking up ifconfig/arp to demonstrate what might happen with longer
addresses? 
 
> for ifconfig(8) and arp(8) is acceptable, that may be a way to satisfy both

I am not sure what you are saying: in all your Linux/ifconfig examples, you did
not change the core display format, and as you point out, option #3 seemed
to experience least line wrap issues (at least with the address that was tried).
If you change the ifconfig/arp output format (for Ethernet and all else, probably
by putting the HWaddr on a seperate line), you might be able to solve the problem 
(and in which case, you can adopt option #1 too). Option #3 is good in default 
GID cases, but is not guaranteed to work always. 

> The format of the address is network device dependent so any format would be

Also, note the suggestion I made, that -H might not be needed if arp can be
taught that the default media for option #1 (or any other formatting option
that you/we finally pick) is for IPoIB. If any other current/future media
happens to use the same formatting option, they will be constrained to using
-H.

Thanks.

Kanoj

---

From: Kanoj Sarcar <[email-masked]>
Date: Tue, 14 Jan 2003 18:08:28 -0800

Thanks.

Kanoj

---

From: Roland Dreier <[email-masked]>
Date: 14 Jan 2003 18:24:49 -0800

Kanoj> Off topic, but I am very curious to find out exactly what
    Kanoj> sock_addr changes were accepted. Can you provide pointers
    Kanoj> (names of ioctls, or structures, in 2.4.X or 2.5.X)?

I was the author of the patch that went into the 2.5.55 Linux kernel.
It increased Linux's MAX_ADDR_LEN from 8 to 32, and extended the
rtnetlink facility to allow setting the hardware and broadcast address
of an interface.  rtnetlink already handled arbitrary lengths
properly, and provided mechanisms for querying hardware addresses and
managing ARP entries.

struct sock_addr did not change, so SIOCGIFHWADDR et al will not work
correctly for IPoIB interfaces.  Users will probably have to use the
ip command (from the iproute2 package) instead of ifconfig.

If you would like a copy of the actual patch, I would be happy to
forward it to you.

Best,
  Roland

---

From: Roland Dreier <[email-masked]>
Date: 14 Jan 2003 18:24:49 -0800

Best,
  Roland

---

From: Roland Dreier <[email-masked]>
Date: 15 Jan 2003 09:29:25 -0800

By the way, here's the output format used today by the Linux iproute2
package.  Note that it already handles (though not gracefully) long
hardware addresses.

The "link/[32]" is how ip currently displays an IPoIB hardware
address.  I would expect that to be replaced by "link/ipoib", and it
would be easy to use any format we decide on for the actual address.

(ib0 is a dummy driver I use for testing with a 20 byte hardware
address; the specific hardware addresses shown are just random junk)

# ip address
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:30:48:23:3c:f3 brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.84/16 brd 10.3.255.255 scope global eth0
3: ib0: <BROADCAST,UP> mtu 2044 qdisc pfifo_fast qlen 128
    link/[32] bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc:bc brd ff:fe:fd:fc:fb:fa:f9:f8:f7:f6:f5:f4:f3:f2:f1:f0:ef:ee:ed:ec
    inet 12.0.0.1/8 brd 12.255.255.255 scope global ib0
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Roland Dreier <[email-masked]>
Date: 15 Jan 2003 09:29:25 -0800

# ip address

---

From: Kanoj Sarcar <[email-masked]>
Date: Wed, 15 Jan 2003 09:38:52 -0800

> If you would like a copy of the actual patch, I would be happy to

Roland,

Sure, I would like a copy of the patch (feel free to send it to me
in private mail, if you think its appropriate). I imagine from your
description though that you have enabled the Linux kernel (internally)
to support 32b mac addresses, using existing 
RTM_NEWADDR/RTM_DELADDR/RTM_GETADDR messages to manipulate the ARP 
cache (using which command? GNU "arp" uses the SIOC*ARP calls, right?), 
and using new rtnetlink messages to set the hw/bcast address via a 
new ipoib-specific command (or will that be driven via the ip command
for IPoIB interfaces?).

Thanks.

Kanoj

---

From: Kanoj Sarcar <[email-masked]>
Date: Wed, 15 Jan 2003 09:38:52 -0800

Roland,

Thanks.

Kanoj

---

From: Kanoj Sarcar <[email-masked]>
Date: Wed, 15 Jan 2003 13:58:47 -0800

Folks,

While I have your attention, there is another issue I
would like to get feedback on from implementors.

On BSD, Linux, Solaris, AIX  etc, there are a set of
calls (maybe with minor variations):

#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <netinet/in.h>
#include <netinet/if_ether.h>

char *ether_ntoa(struct ether_addr *e);
struct ether_addr *ether_aton(const char *asc);

It seems to me that most systems that support IPoIB
will need something like:

#include <net/if_dl.h>

char *link_ntoa(char *addr, int length, int type);
unsigned char *link_aton(const char *asc, int &size);

to handle network to ascii formatting and vice-versa.

(Note: ether_line is not relevant since IPoIB link
address is not persistent and can not be put into
ethers databases; similar logic applies to ether_ntohost()
and ether_hostton(); link_aton() also sounds quite
optional to me).

The "type" field in link_ntoa() is to force media
specific formatting behavior; its value should
identify the media type (or 0 for default behavior).
Thus for Linux, ARPHRD_ETHER, ARPHRD_INFINIBAND etc
would be possible values, whereas for Solaris and
other DLPI based systems, it would be DL_ETHER etc.
link_ntoa() should by default format the xx:yy:...
(sized according to the length argument) string,
but its formatting action can be altered depending
on the type input (eg a compressed v6 format for
IPoIB, if that is the way Linux should go).

Are there any plans on these lines by IPoIB
implementors on various systems?

Thanks.

Kanoj

---

From: Kanoj Sarcar <[email-masked]>
Date: Wed, 15 Jan 2003 13:58:47 -0800

Folks,

#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <netinet/in.h>

#include <net/if_dl.h>

Thanks.

Kanoj