Subject: RE: [Ipoverib] IPoIB/ifconfig question

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 14 Jan 2003 13:50:35 -0800

The more I think about it, the more I'm in favor of the condensed/IPv6
format.  I really would like to consider the aesthetic of the format.
Although my mailer is probably going to wrap these lines (a hint at what it
would look like on an 80 character xterm), consider the following displays
from a Linux system.  The first is the standard output (80 character line
length) followed by option #1(120 chars), #3(88 chars) and #4(107 chars).
Note that the arp header is constant meaning that it would take the new
field widths for all network devices including Ethernet.  Also keep in mind
that the primary reason to display this info is for a human to read.  Of
these, I believe the admin would want to use option #3 for input and output.
It's easy enough to change/add a MIB definition for a IPoIB hardware address
(as opposed to the GUID display hint that folks have alluded to).

Other comments in-line /rkl
Roy

Standard      #3                 #4           #1
 
V       V                  V            V
         1         2         3         4         5         6         7
8         9         100       110       120
1234567890123456789012345678901234567890123456789012345678901234567890123456
78901234567890123456789012345678901234567890
--------------------------------------------------------
Standard format

eth0      Link encap:IPoIB  HWaddr 00:A0:C9:FC:15:4A  
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5398865 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2396617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148296 txqueuelen:100 
          RX bytes:810245774 (772.7 Mb)  TX bytes:1587921980 (1514.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress           Flags Mask
Iface
10.7.159.251             ether   00:D0:D3:39:18:A8   C
eth0

--------------------------------------------------------
option #1

ib01      Link encap:Infina  HWaddr
00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress
Flags Mask            Iface
10.7.159.251             infina
00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81 C
ib01

-------------------------------------------------------
option #3

ib01      Link encap:IPoIB  HWaddr 0:14:FE80::2:c900:011c:9781
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress                   Flags Mask
Iface
10.7.159.251             infina  0:14:FE80::2:c900:011c:9781 C
ib01

-------------------------------------------------------
option #4

ib01      Link encap:Infina  HWaddr
123456:FE80:0000:0000:0000:0002:c900:011c:9781
          inet addr:10.7.159.98  Bcast:10.7.159.255  Mask:255.255.255.0
          UP BROADCAST NOTRAILERS RUNNING  MTU:1500  Metric:1
          RX packets:5390566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2388650 errors:0 dropped:0 overruns:0 carrier:0
          collisions:148295 txqueuelen:100 
          RX bytes:809726070 (772.2 Mb)  TX bytes:1584781754 (1511.3 Mb)
          Interrupt:21 Base address:0xa000 

Address                  HWtype  HWaddress
Flags Mask            Iface
10.7.159.251             infina
123456:FE80:0000:0000:0000:0002:c900:011c:9781 C                     ib01

-----Original Message-----
From: Kanoj Sarcar [mailto:[email-masked]]
Sent: Tuesday, January 14, 2003 12:45 PM
To: Larsen, Roy K
Cc: [email-masked]; Kanoj Sarcar
Subject: Re: [Ipoverib] IPoIB/ifconfig question

"Larsen, Roy K" wrote:
> 
> Ah yes, 24 bits....  I had a brain fart and was thinking the QPN was 16.
> Then if fixed length is desirable, what you show for #4 is better than the
> 60 digit #1 (regardless of what the mib draft proposes).  It just isn't
> practical to display 60 characters (although 47 characters is pretty much
in
> the same camp) and is not very parsable by a human.  Remember, this
sequence
> would be displayed for things like ifconfig(8) and arp(8) which are fairly
> frequent administrative operations.  I still like #3 but since I don't do
> much network administration, I don't know what the impact of a variable
> length hardware address would have.

I think we are down to #1 and #4, right? (Another point against #3 is that
it *might* be confusing to have a v6 protocol layer addressing scheme
applied
to a link layer address; specially since the link layer address is not
128bits 
long). 

/rkl
As Roland points out, GIDs are IPv6 addresses so it doesn't bother me that
they follow an IPv6 format.  The admin tools make it very clear where the hw
address is displayed
rkl/
 
Here they are (with QPN=0x123456):

1. 00:12:34:56:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
(or 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81)
4. 123456:FE80:0000:0000:0000:0002:c900:011c:9781
(or 123456:FE80:0:0:0:2:c900:11c:9781)

Given the irregularity (length wise) in the first few digits in 4, 
I would think #2 would be better. Here it is:

2. 12:3456:FE80:0000:0000:0000:0002:c900:011c:9781
(or 12:3456:FE80:0:0:0:2:c900:11c:9781)

I do have a preference for #1 still, primarily because 

a. it is the most easily adapatable to the "arp" command (ie, when you
are doing a "set" operation, it is very easy to teach "arp", which on
most systems currently only expect ethernet addresses, how to parse a
#1 style ib command, in contrast to #2 or #4). Any other format picked
can of course be handled, I have seen the least amount of work with #1 
while prototyping (I am dealing with Solaris, but I expect that to be 
true on most other systems like BSD too; Linux has a -H option, so this
is not an issue).

/rkl
When you say adaptable to the arp(8) command, are you taking into account
the display headaches I try to illustrate above?  Given that the IPoIB
hardware address is virtual and the pathrecord is always required, do you
think "set" is viable?  Even so, do you think the admin would want to enter
60 characters as opposed to something around 27?
rkl/

b. I do believe there is value in matching the MIB format, specially since
admins might want to correlate the outputs from various commands (ifconfig,
arp, snoop and SNMP).

/rkl
I don't disagree.  But if the MIB format is not practical, we can change the
draft.  Anyway, it doesn't have a display hint for IPoIB hardware address so
we can invent one.
rkl/

c. Any current scripts that parse ethernet addresses (HWaddr output in
Linux etc) if applicable on IPoIB would need to be converted; I am not
aware of any, but if there are some, logic in a. again applies.

/rkl
I'm not aware of any need to parse a hardware address except perhaps to
compare them.  Format in that case is not as important.
rkl/

The advantage of 2 and 4 are their somewhat shorter lengths.
/rkl
#3 is even shorter :^)
rkl/

> 
> The network admin tools I work with already use a hardware type based
> library for displaying interface specific information so there is no need
to
> follow Ethernet convention from my point of view.
>

What about (sysadmin) scripts that parse output from ifconfig (ie look at 
HWaddr field), which currently assume ethernet format? 

/rkl
But I assume they already assume a fixed length representing 6 octets so
they are going to have to change anyway.  At most, they are going to
comparisons so I don't think this a compelling argument for #1
rkl/

I assume you are talking about Linux (and thus you have the -H arp option 
to help)? Notice, you could still build in automatic parsing in Linux "arp",

which on seeing format #1 will recognize it as an IPoIB address (ie,
the default type for 20b addresses can be IPoIB, similar to the default
type for 6b addresses is ethernet; this would allow admins not to have
to specify -H for each IPoIB operation, thus making IPoIB just a little
more similar to ethernet); I imagine it will be just a little harder to
do #2 or #4.

/rkl
I believe the -H option only acts as a filter, not a display formatter.
rkl/

Thanks.

Kanoj

> Roy
> 
> -----Original Message-----
> From: Kanoj Sarcar [mailto:[email-masked]]
> Sent: Tuesday, January 14, 2003 11:00 AM
> To: Larsen, Roy K
> Cc: [email-masked]; Kanoj Sarcar
> Subject: Re: [Ipoverib] IPoIB/ifconfig question
> 
> "Larsen, Roy K" wrote:
> >
> > Kanoj,
> >
> > While not a draft issue, I agree that it would be nice to use this forum
> to
> > set a de facto standard for IPoIB hardware address display format.
Given
> > the long length, I'm inclined to go with the most condensed
representation
> > which would be #3.  However, this does not provide a fixed length
display
> > which is useful for scripting applications so I'm thinking a modified #2
> > with all fixed length numbers and a single field QPN of 0014 instead of
> > 0:14).
> >
> > Roy
> >
> 
> Roy,
> 
> It was pointed out to me that
> 
> http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ibmib-tc-mib-02.txt
> 
> has
> 
> IbGuid ::= TEXTUAL-CONVENTION
>      DISPLAY-HINT "1x:"
>      STATUS      current
>      DESCRIPTION
> 
> It would seem to me that unless there is good reason for deviation,
> tcpdump/snoop/ifconfig/arp et al could just use this format (maps to
> option 1).
> 
> Another reason I prefer 1 is because scripts written with the ethernet
> format in mind would mostly work (if the length issues were fixed in
> the script). Option 2 packs 16 bits in between each :, whereas these
> ethernet based scripts pack 8 bits.
> 
> What do you think?
> 
> The reason I went with 0:14 for the QPN (in #3) is because the QPN is
> actually 4 bytes, with the top byte set to 0. So QPN = 0x00123456
> could be a possible value (theoretically). To represent that, I would
> need to say 12:3456:... I wanted this to look the same (in terms of
> number of :s) as in the case of QPN=0x00000014. Or are you suggesting
> something like
> 
> 4. 123456:FE80:0000:0000:0000:0002:c900:011c:9781
> 
> in this case?
> 
> Thanks.
> 
> Kanoj
> 
> > -----Original Message-----
> > From: Kanoj Sarcar [mailto:[email-masked]]
> > Sent: Monday, January 13, 2003 2:35 PM
> > To: [email-masked]; Kanoj Sarcar
> > Subject: [Ipoverib] IPoIB/ifconfig question
> >
> > Hi,
> >
> > Here's an IPoIB implementation issue that I would
> > like to get feedback on from implementors.
> >
> > What should the ifconfig/arp/packet-sniffer output look like for
> > the mac address part of an IPoIB interface? In the examples below,
> > assume QPN=0x14, GID prefix is FE80::0 and GID suffix is
> > 0002c900011c9781, so that the mac address in hex is
> > 14FE800000000000000002c900011c9781.
> >
> > Some options might be
> >
> > 1. 00:00:00:14:FE:80:00:00:00:00:00:00:00:02:c9:00:01:1c:97:81
> > (or 0:0:0:14:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81)
> > 2. 0:14:FE80:0000:0000:0000:0002:c900:011c:9781
> > (or 0:14:FE80:0:0:0:2:c900:011c:9781)
> > 3: 0:14:FE80::2:c900:011c:9781
> >
> > I understand this is probably not something that should be
> > put into any of the I/Ds, but it would be nice if various
> > vendors' outputs matched. My choice is option 1, since it
> > is the closest to how most systems represent ethernet
> > addresses (except the length of course). Most systems'
> > "arp" command is also easily converted to handle this
> > format and obviates the need for complicated parsing.
> >
> > Note that the I/D
> >
> >
http://www.ietf.org/internet-drafts/draft-ietf-ipoib-link-multicast-02.txt
> >
> > on page 8 section 8.0 represents IBA multicast GIDs in v6
> > format, I am not sure whether to suggest that the link layer
> > address should also be represented similarly by network tools.
> >
> > For reference, here's the ifconfig output line that prints
> > mac addresses on Solaris and Linux:
> > Link encap:Ethernet  HWaddr 00:06:5B:4E:E4:6E
> > ether 8:0:20:d9:4:97
> >
> > Thanks.
> >
> > Kanoj
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: James Carlson <[email-masked]>
Date: Tue, 14 Jan 2003 17:32:05 -0500

> format.  I really would like to consider the aesthetic of the format.

I think that's going to be exciting when you're using IPv6 over IB.

% netstat -pnf inet6

Net to Media Table: IPv6
 If   Physical Address                                 Type      State      Destination/Mask
----- ----------------------------------------------  ------- ------------ ---------------------------
ib01  123456:fe80:0000:0000:0000:0002:c900:011c:9781  dynamic REACHABLE    2002:8194:aeaa:1:203:baff:fe00:de2f

... yuck.  Which one is which?

-- 
James Carlson, Solaris Networking         <[email-masked]>
Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: James Carlson <[email-masked]>
Date: Tue, 14 Jan 2003 17:32:05 -0500

% netstat -pnf inet6

Net to Media Table: IPv6

---

From: Kanoj Sarcar <[email-masked]>
Date: Tue, 14 Jan 2003 15:21:47 -0800

Roy,

You might also want to try a non-default GID prefix, specially one that does
not have too many runs of 0s. I am not sure if the Linux implementation will
allow that, but it is probably possible on other systems. In that case, the
benefits of #3 might be limited (whereas 1, 2 and 4 are independent of actual
values).

Another suggestion is to eliminate leading 0's for all options (that can
only shorten the output) when you compare.

Comments below.
 
> Other comments in-line /rkl

I am including output from a Solaris system here, similar to what Roy did
for Linux:

hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.7.198.20 netmask ffffff00 broadcast 10.7.198.255
        ether 8:0:20:d9:4:97
ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 504 index 3
        inet 192.129.100.100 netmask ffffff00 broadcast 192.129.100.255
        ib 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81
root@pap# arp pap
pap (10.7.198.20) at 8:0:20:d9:4:97 permanent published
root@pap# arp pap-ibd
pap-ibd (192.129.100.100) at 0:0:0:14:0:0:0:0:0:0:0:4:0:2:C9:0:1:1C:97:81 permanent established

> V       V                  V            V

But we are talking about GID+QPN here, and since thats 20b instead of v6
16 bytes, I don't know whether this is a good idea. The other part is, the
GID is similar to v6 in the context of IBA network layer, but that same
GID is being used as (part of) a link layer address for IPoIB. 

> 60 characters as opposed to something around 27?

We have to allow that "set"s might be meaningful in certain implementations.
Does the Linux implementation allow it? If so, what is the syntax of the
command, and how was "arp" modified for accepting IPoIB link address input?
(For Solaris, which is similar to BSD, I can do
"arp -s hostname 0:12:34:56:FE:80:0:0:0:0:0:0:0:2:c9:0:1:1c:97:81"
I had to teach arp to accept not only a:b:c:d:e:f ethernet format, but also
the 20b format).

Yes, I am aware there are display issues, but I think they might be different
issues on different platforms (eg Solaris prints the link address on a seperate
line, so its lesser of an issue).

> I believe the -H option only acts as a filter, not a display formatter.

Okay. I am still curious about how Linux/IPoIB does a arp set operation.

Thanks.

Kanoj

---

From: Kanoj Sarcar <[email-masked]>
Date: Tue, 14 Jan 2003 15:21:47 -0800

Roy,

Comments below.

Thanks.

Kanoj