Subject: [Ipoverib] IPoIB/ifconfig question

From: Kanoj Sarcar <[email-masked]>
Date: Mon, 13 Jan 2003 14:34:30 -0800

Hi,

Here's an IPoIB implementation issue that I would
like to get feedback on from implementors. 

What should the ifconfig/arp/packet-sniffer output look like for 
the mac address part of an IPoIB interface? In the examples below, 
assume QPN=0x14, GID prefix is FE80::0 and GID suffix is 
0002c900011c9781, so that the mac address in hex is 
14FE800000000000000002c900011c9781.

Some options might be

1. 00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
(or 0:0:0:14:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81)
2. 0:14:FE80:0000:0000:0000:0002:c900:011c:9781
(or 0:14:FE80:0:0:0:2:c900:011c:9781)
3: 0:14:FE80::2:c900:011c:9781

I understand this is probably not something that should be
put into any of the I/Ds, but it would be nice if various
vendors' outputs matched. My choice is option 1, since it
is the closest to how most systems represent ethernet
addresses (except the length of course). Most systems'
"arp" command is also easily converted to handle this 
format and obviates the need for complicated parsing.

Note that the I/D 

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-link-multicast-02.txt

on page 8 section 8.0 represents IBA multicast GIDs in v6 
format, I am not sure whether to suggest that the link layer 
address should also be represented similarly by network tools.

For reference, here's the ifconfig output line that prints 
mac addresses on Solaris and Linux:
Link encap:Ethernet  HWaddr 00:06:5B:4E:E4:6E
ether 8:0:20:d9:4:97

Thanks.

Kanoj
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 10:41:23 -0800

Kanoj,

While not a draft issue, I agree that it would be nice to use this forum to
set a de facto standard for IPoIB hardware address display format.  Given
the long length, I'm inclined to go with the most condensed representation
which would be #3.  However, this does not provide a fixed length display
which is useful for scripting applications so I'm thinking a modified #2
with all fixed length numbers and a single field QPN of 0014 instead of
0:14).

Roy

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Monday, January 13, 2003 2:35 PM
To: [email-masked]; Kanoj Sarcar
Subject: [Ipoverib] IPoIB/ifconfig question

Hi,

Some options might be

Note that the I/D 

Thanks.

Kanoj

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 11:25:53 -0800

Ah yes, 24 bits....  I had a brain fart and was thinking the QPN was 16.
Then if fixed length is desirable, what you show for #4 is better than the
60 digit #1 (regardless of what the mib draft proposes).  It just isn't
practical to display 60 characters (although 47 characters is pretty much in
the same camp) and is not very parsable by a human.  Remember, this sequence
would be displayed for things like ifconfig(8) and arp(8) which are fairly
frequent administrative operations.  I still like #3 but since I don't do
much network administration, I don't know what the impact of a variable
length hardware address would have.

The network admin tools I work with already use a hardware type based
library for displaying interface specific information so there is no need to
follow Ethernet convention from my point of view.

Roy

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 11:00 AM
To: Larsen, Roy K
Cc: [email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] IPoIB/ifconfig question

> with all fixed length numbers and a single field QPN of 0014 instead of

Roy,

It was pointed out to me that 

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ibmib-tc-mib-02.txt

has

IbGuid ::= TEXTUAL-CONVENTION
     DISPLAY-HINT "1x:"
     STATUS      current
     DESCRIPTION

It would seem to me that unless there is good reason for deviation,
tcpdump/snoop/ifconfig/arp et al could just use this format (maps to
option 1).

Another reason I prefer 1 is because scripts written with the ethernet
format in mind would mostly work (if the length issues were fixed in
the script). Option 2 packs 16 bits in between each :, whereas these
ethernet based scripts pack 8 bits. 

What do you think?

The reason I went with 0:14 for the QPN (in #3) is because the QPN is 
actually 4 bytes, with the top byte set to 0. So QPN = 0x00123456 
could be a possible value (theoretically). To represent that, I would 
need to say 12:3456:... I wanted this to look the same (in terms of
number of :s) as in the case of QPN=0x00000014. Or are you suggesting
something like

4. 123456:FE80:0000:0000:0000:0002:c900:011c:9781

in this case?

Thanks.

Kanoj

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 13:50:35 -0800

The more I think about it, the more I'm in favor of the condensed/IPv6
format.  I really would like to consider the aesthetic of the format.
Although my mailer is probably going to wrap these lines (a hint at what it
would look like on an 80 character xterm), consider the following displays
from a Linux system.  The first is the standard output (80 character line
length) followed by option #1(120 chars), #3(88 chars) and #4(107 chars).
Note that the arp header is constant meaning that it would take the new
field widths for all network devices including Ethernet.  Also keep in mind
that the primary reason to display this info is for a human to read.  Of
these, I believe the admin would want to use option #3 for input and output.
It's easy enough to change/add a MIB definition for a IPoIB hardware address
(as opposed to the GUID display hint that folks have alluded to).

Other comments in-line /rkl
Roy

Standard      #3                 #4           #1
 
V       V                  V            V
         1         2         3         4         5         6         7
8         9         100       110       120
1234567890123456789012345678901234567890123456789012345678901234567890123456
78901234567890123456789012345678901234567890
--------------------------------------------------------
Standard format

eth0      Link encap:IPoIB  HWaddr 00:A0:C9:FC:15:4A  
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5398865 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2396617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148296 txqueuelen:100 
          RX bytes:810245774 (772.7 Mb)  TX bytes:1587921980 (1514.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress           Flags Mask
Iface
10.7.159.251             ether   00:D0:D3:39:18:A8   C
eth0

--------------------------------------------------------
option #1

ib01      Link encap:Infina  HWaddr
00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress
Flags Mask            Iface
10.7.159.251             infina
00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81 C
ib01

-------------------------------------------------------
option #3

ib01      Link encap:IPoIB  HWaddr 0:14:FE80::2:c900:011c:9781
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress                   Flags Mask
Iface
10.7.159.251             infina  0:14:FE80::2:c900:011c:9781 C
ib01

-------------------------------------------------------
option #4

ib01      Link encap:Infina  HWaddr
123456:FE80:0000:0000:0000:0002:c900:011c:9781
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress
Flags Mask            Iface
10.7.159.251             infina
123456:FE80:0000:0000:0000:0002:c900:011c:9781 C                     ib01

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 12:45 PM
To: Larsen, Roy K
Cc: [email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] IPoIB/ifconfig question

> length hardware address would have.

I think we are down to #1 and #4, right? (Another point against #3 is that
it *might* be confusing to have a v6 protocol layer addressing scheme
applied
to a link layer address; specially since the link layer address is not
128bits 
long). 

/rkl
As Roland points out, GIDs are IPv6 addresses so it doesn't bother me that
they follow an IPv6 format.  The admin tools make it very clear where the hw
address is displayed
rkl/
 
Here they are (with QPN=0x123456):

1. 00:12:34:56:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
(or 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81)
4. 123456:FE80:0000:0000:0000:0002:c900:011c:9781
(or 123456:FE80:0:0:0:2:c900:11c:9781)

Given the irregularity (length wise) in the first few digits in 4, 
I would think #2 would be better. Here it is:

2. 12:3456:FE80:0000:0000:0000:0002:c900:011c:9781
(or 12:3456:FE80:0:0:0:2:c900:11c:9781)

I do have a preference for #1 still, primarily because 

a. it is the most easily adapatable to the "arp" command (ie, when you
are doing a "set" operation, it is very easy to teach "arp", which on
most systems currently only expect ethernet addresses, how to parse a
#1 style ib command, in contrast to #2 or #4). Any other format picked
can of course be handled, I have seen the least amount of work with #1 
while prototyping (I am dealing with Solaris, but I expect that to be 
true on most other systems like BSD too; Linux has a -H option, so this
is not an issue).

/rkl
When you say adaptable to the arp(8) command, are you taking into account
the display headaches I try to illustrate above?  Given that the IPoIB
hardware address is virtual and the pathrecord is always required, do you
think "set" is viable?  Even so, do you think the admin would want to enter
60 characters as opposed to something around 27?
rkl/

b. I do believe there is value in matching the MIB format, specially since
admins might want to correlate the outputs from various commands (ifconfig,
arp, snoop and SNMP).

/rkl
I don't disagree.  But if the MIB format is not practical, we can change the
draft.  Anyway, it doesn't have a display hint for IPoIB hardware address so
we can invent one.
rkl/

c. Any current scripts that parse ethernet addresses (HWaddr output in
Linux etc) if applicable on IPoIB would need to be converted; I am not
aware of any, but if there are some, logic in a. again applies.

/rkl
I'm not aware of any need to parse a hardware address except perhaps to
compare them.  Format in that case is not as important.
rkl/

The advantage of 2 and 4 are their somewhat shorter lengths.
/rkl
#3 is even shorter :^)
rkl/

> follow Ethernet convention from my point of view.

What about (sysadmin) scripts that parse output from ifconfig (ie look at 
HWaddr field), which currently assume ethernet format? 

/rkl
But I assume they already assume a fixed length representing 6 octets so
they are going to have to change anyway.  At most, they are going to
comparisons so I don't think this a compelling argument for #1
rkl/

I assume you are talking about Linux (and thus you have the -H arp option 
to help)? Notice, you could still build in automatic parsing in Linux "arp",

which on seeing format #1 will recognize it as an IPoIB address (ie,
the default type for 20b addresses can be IPoIB, similar to the default
type for 6b addresses is ethernet; this would allow admins not to have
to specify -H for each IPoIB operation, thus making IPoIB just a little
more similar to ethernet); I imagine it will be just a little harder to
do #2 or #4.

/rkl
I believe the -H option only acts as a filter, not a display formatter.
rkl/

Thanks.

Kanoj

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 15:56:44 -0800

James,

That's and easy one.  The hw address is listed in the column labeled
"Physical Address" and the IP under "Destination/Mask".  :^)

Roy

-----Original Message-----
From: James Carlson [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 2:32 PM
To: Larsen, Roy K
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB/ifconfig question

> format.  I really would like to consider the aesthetic of the format.

I think that's going to be exciting when you're using IPv6 over IB.

% netstat -pnf inet6

Net to Media Table: IPv6
 If   Physical Address                                 Type      State
Destination/Mask
----- ----------------------------------------------  ------- ------------
---------------------------
ib01  123456:fe80:0000:0000:0000:0002:c900:011c:9781  dynamic REACHABLE
2002:8194:aeaa:1:203:baff:fe00:de2f

... yuck.  Which one is which?

-- 
James Carlson, Solaris Networking         <[email-masked]>
Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 15:56:44 -0800

James,

Roy

-----Original Message-----
From: James Carlson [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 2:32 PM
To: Larsen, Roy K
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB/ifconfig question

% netstat -pnf inet6

Net to Media Table: IPv6

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 16:33:59 -0800

/rkl
coments in-line
rkl/

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 3:22 PM
To: Larsen, Roy K
Cc: [email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] IPoIB/ifconfig question

Roy,

You might also want to try a non-default GID prefix, specially one that does
not have too many runs of 0s. I am not sure if the Linux implementation will
allow that, but it is probably possible on other systems. In that case, the
benefits of #3 might be limited (whereas 1, 2 and 4 are independent of
actual
values).

/rkl
I take your point but my assumptions is that there would always be display
economy using the condensed format
rkl/

Another suggestion is to eliminate leading 0's for all options (that can
only shorten the output) when you compare.

Comments below.
 
> Other comments in-line /rkl

I am including output from a Solaris system here, similar to what Roy did
for Linux:

hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.7.198.20 netmask ffffff00 broadcast 10.7.198.255
        ether 8:0:20:d9:4:97
ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 504 index 3
        inet 192.129.100.100 netmask ffffff00 broadcast 192.129.100.255
        ib 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81
root@pap# arp pap
pap (10.7.198.20) at 8:0:20:d9:4:97 permanent published
root@pap# arp pap-ibd
pap-ibd (192.129.100.100) at 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81
permanent established

/rkl
Clearly the Solaris output is more suited for long hardware address display.
This is not the case for Linux....
rkl/

<snip>

> they follow an IPv6 format.  The admin tools make it very clear where the
hw

But we are talking about GID+QPN here, and since thats 20b instead of v6
16 bytes, I don't know whether this is a good idea. The other part is, the
GID is similar to v6 in the context of IBA network layer, but that same
GID is being used as (part of) a link layer address for IPoIB. 

/rkl
I'm not sure I appreciate why this would matter.  Although the componets of
the hw address are generally not a concern to the admin, the GID is a
component.  I'm just looking for a way to make them more user/display
friendly in the same way the IPv6 folks made their address more friendly.
rkl/

> 60 characters as opposed to something around 27?

We have to allow that "set"s might be meaningful in certain implementations.
Does the Linux implementation allow it? If so, what is the syntax of the
command, and how was "arp" modified for accepting IPoIB link address input?
(For Solaris, which is similar to BSD, I can do
"arp -s hostname 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81"
I had to teach arp to accept not only a:b:c:d:e:f ethernet format, but also
the 20b format).

/rkl
Although I have taken a look at the Linux kernel and network admin tools
that are affected, the SourceForge project has not jet tackled these.
Currently they truncate the IPoIB hardware address.  I believe the Linux
maintainers have just accepted the changes needed to deal with the
kernel/user interface (the sock_addr size issue).  Yes, Linux arp(8), as a
facility, allows entries to be set.
rkl/

Yes, I am aware there are display issues, but I think they might be
different
issues on different platforms (eg Solaris prints the link address on a
seperate
line, so its lesser of an issue).

/rkl
This is my primary concern since changes here will affect the display format
for all network devices including Ethernet on Linux.  It appears the Solaris
format makes long hw addresses painless.  This is not true for Linux and
must be taken into consideration.  The customer experience is more important
to me that a standardized format.  If changing the core Linux display format
for ifconfig(8) and arp(8) is acceptable, that may be a way to satisfy both
interests.
rkl/

<snip>

> I believe the -H option only acts as a filter, not a display formatter.

Okay. I am still curious about how Linux/IPoIB does a arp set operation.

/rkl
When implemented, it would be set in the same way as any non-default (ether)
network device:
arp -H infini -s <hostname> <hw address>

The format of the address is network device dependent so any format would be
acceptable.

Roy
rkl/

Thanks.

Kanoj

<snip>

---

From: "Larsen, Roy K" <[email-masked]>
Date: Wed, 15 Jan 2003 09:24:22 -0800

<snip>

Off topic, but I am very curious to find out exactly what sock_addr changes
were accepted. Can you provide pointers (names of ioctls, or structures, in
2.4.X or 2.5.X)?

Also, I am not sure of the ifconfig output that you provided: was that based
on hacking up ifconfig/arp to demonstrate what might happen with longer
addresses? 

/rkl
Roland's mail addressed the changes (of which I was a little off base).
Yes, I edited the output of the ifconfig/arp commands.  I have not done any
work yet toward implementing support IPoIB support for admin commands.
rkl/
 
> for ifconfig(8) and arp(8) is acceptable, that may be a way to satisfy
both

I am not sure what you are saying: in all your Linux/ifconfig examples, you
did
not change the core display format, and as you point out, option #3 seemed
to experience least line wrap issues (at least with the address that was
tried).
If you change the ifconfig/arp output format (for Ethernet and all else,
probably
by putting the HWaddr on a seperate line), you might be able to solve the
problem 
(and in which case, you can adopt option #1 too). Option #3 is good in
default 
GID cases, but is not guaranteed to work always. 

/rkl
I am trying to say that the goal is to come up with a common hw address
display format that provides a pleasing customer experience.  But, the
customer experience is paramount.  I don't have any problem with option #1
or #4.  However, I think there will be a lot of resistance by the Linux
community to rearranging the current admin format but I am willing to
explore that option.  If I fail at making that change, the #3 format would
be the most accommodating to the current ifconfig/arp commands - and I
accept that it may be different than on other platforms (and may be
unpleasing under certain GID values).
   
Roy

rkl/

<snip>

Thanks.

Kanoj

---

From: "Larsen, Roy K" <[email-masked]>
Date: Wed, 15 Jan 2003 09:24:22 -0800

<snip>

/rkl

/rkl
   
Roy

rkl/

<snip>

Thanks.

Kanoj