Subject: [Ipoverib] IPoIB architecture/encapsulation

From: Vivek Kashyap <[email-masked]>
Date: Fri, 7 Dec 2001 13:34:30 -0800 (PST)

Folks,
	
        This is a request to the WG to comment on IPoIB architecture
        and encapsulation issues. Attached is, based on the documents
        submitted to the WG and discussions on this malining list,
        what I believe to be the best solution. Please comment.

Ref: a) http://www.ietf.org/mail-archive/working-groups/ipoverib/current/maill 
     b) draft-kashyap-ipoib-requirements-01.txt (IPoIB Architecture)

	1) IPoIB Link

        The IPoIB link is described by the multicast GID corresponding
        to the IPoIB broadcast/all-IP nodes multicast group. The
        defining link characteristics are:

		a) Members of the IPoIB subnet
		b) Partition Key
		c) Q_Key
		d) MTU

        Thus the abstract IPoIB link is defined by the administrator
        when the MGID is created. The link gets formed as IPoIB nodes
        join (at the IB level) the multicast GID.

	2) IPoIB subnet setup

        The IPoIB subnet comes into being with the nodes joining the
        multicast GID. It is MUST that all IPoIB interfaces join (this
        is the IB_Join) the multicast GID.

	3) Mutlicast GID mapping

        The multicast GIDs are derived from the IP addresses as per
        the following rules:

	a) IPv4 address to IB MGID

	FF12:3334:<Link_ID>::<IPv4 address>

        Thus the IPv4 over IB multicast GID that defines the IPoIB
        link is :

	FF12:3334:<Link_ID>::255.255.255.255

	b) IPv6 address to IB MGID

	FF12:3333:<Link_ID>:<low order 80 bits of IPv6 address>

        Thus the IPv6 over IB multicast GID that defines the IPoIB
        link is :

	FF12:3333:<Link_ID>::1

	c) Link_ID

        The link ID needs to be a unique value. As discussed in
        ipoverib mailing list thread "IP Encapsulation Design Team"
        the link ID is the P_Key. Using the P_Key as the Link_ID
        implies that the P_Key is a configuration parameter. The IPoIB
        node must know the P_Key to use before it can join the IPoIB
        'link'. 

        Jerry has suggested incorporating the Q_Key in link ID too as
        noted in the above thread. We left the discussion at the point
        where it was mentioned that any solution including Q_Key in
        the mapping MUST not make the Q_Key another configuration
        parameter. I for one don't favor including the Q_Key in the
        Link_ID. 

        NOTE: IBTA has clarified that multiple MGIDs of the same scope
              cannot exist in the same subnet. Thus the link_ID must
              be incorporated to differentiate the multiple groups.
	
	Link_ID of zero:

        A non-zero link id is needed to allow for isolating multiple
        broadcast domains over the same IB subnet the way VLANs
        isolate broadcast domains over LAN fabrics. However, one may
        choose to have only a single broadcast domain ie. not use
        VLANs.  Thus a link_ID of zero is a valid option.

        NOTE: The P_Key returned from the joining of the broadcast
              MGID MUST match the P_Key used as the link_ID except
              when the link_ID is zero.

        4) Transient bit in the IPoIB MGIDs

        The IBTA defines the raw datagram mode and assigns all the
        well known IPv6 addresses to it. The IPoIB mapping therefore
        always uses the transient bit to avoid any conflicts with IB's
        Raw datagram IPv6 mode.

	5) Use of scope bits

        The use of scope bits is linked to the question of IPoIB
        subnets spanning multiple IB subnets. As has been discussed in
        this mailing list this requires IB to develop & define IB
        routing. Therefore at this time it is proposed that the scope
        bits will always be set to local scope. Once IB provides the
        necessary infrastructure larger scopes may be used in the
        MGIDs. This makes the scope bit a configuration parameter.

        An IPoIB host may choose to use local scope and if
        unsuccessful try with a larger scope until successful or the
        administrator may specify the scope to be used. The default
        scope is always the local scope.

	6) Link local address/autoconfiguration in IPv6

        Every IB port is assigned a globally unique, EUI-64 complaint
        ID (GUID). This value may be used to create the link local
        address or the auto-configured IPv6 address.

        The above is the default method. If concerned about
        privacy (RFC 3041) one may choose to utilise the LID assigned
        to the port for creating these addresses. Note that the LID is
        unique in the local IB subnet only. If the IPoIB subnet spans
        multiple IB subnets or the LID actually stays constant(depends
        on the SM and fabric configuration) then one of the methods
        suggested in RFC may be used to form the auto-configuration
        address. 

	7) IP/ARP encapsulation

	IP packets are encapsulated in the payload of the IB packet. 

        +-------+------+---------+---------+-------+---------+-------+
        |Local  |      |Base     |Datagram |       |Invariant|Variant|
        |Routing| GRH  |Transport|Extended |Payload| CRC     | CRC   |
        |Header |Header|Header   |Transport|       |         |       |
        |       |      |         |Header   |       |         |       |
        +-------+------+---------+---------+-------+---------+-------+

        The payload will include the IPv4/IPv6 or ARP packet. An
        'ethertype' i.e. a 4 byte value giving the payload type will
        not be used. It is unnecessary and doesn't provide any benefit
        because: 

        1. The 'ethertype' was suggested to help filter/demultiplex
           the packet streams received by the interface. But the
           interface can receive a random set of packets and this
           random set of packets can have any payload. A random packet
           is as likely to alias the 'ethertype' as it is to alias the
           IPv4/IPv6 or ARP headers. Therefore one cannot depend on
           'ethertype' to filter/demultiplex the packes.

        2. The demultiplexing of the valid packets can be done using
           the protocol header signatures with equal ease as with
           introducing an artificial ethertype in the payload. 

        3. The MTU of the payload is a power of two value. The
           multicast group (and the links) take the values 256, 512,
           1024, 2048, or 4096 bytes. If 4 bytes are taken up by the
           'ethertype' we endup with MTUs of 252, 508, 1020, 2044,
           4092. Not the best of the situations.

        4. There is a proposal in IBTA aiming to include a payload
           type(IP/ARP etc.) field in the IB header. This would make
           'ethertype' ie. the type field external to the payload.

	8) Address resolution

        For an IB packet to be successfully sent to another IPoIB node
        the sender needs to know:

        LID, GID(if sending across an IB subnet), 
	P_Key, Q_Key (for UD), 
	QPN, SL

	It also needs to know the MTU to use.

        To allow for non-UD IPoIB implementations the peers must know
        whether the other node implements IP over these modes. Thus
        the nodes need to exchange the capability flags.

	These parameters/values are determined as follows:

        a) P_Key 
                P_Key is a configuration parameter except when the
                Link_ID is set to 0. In that case the P_Key returned
                by the join operation is the P_Key to be used for all
                IP communication in the IPoIB subnet.

	b) Q_Key
                Q_Key returned by the join operation is the Q_Key used
                for all IPoIB communication in the subnet.

	c) MTU
                The interface MTU is set to the value returned when
                the interfae joins the MGID corresponding to the
                all-IP nodes multicast/broadcast address.

	d) Service Level (SL)
                The service level in IB is used to satisfy many
                functions viz. load balancing, QoS, deadlock 
                avoidance. There is no 'standard' method defined by
                IBTA to determine SL. The determining of the SL to use
                in any packet is left to the implementation. The
                implementation may choose to derive the SL from a QoS
                mapping, by some other indirect means or by querying
                the SA.

        The rest of the parameters are exchanged using the address
        resolution protocol. The parameters used in the address
        resolution are: LID, capability flags, QPN and the GID.

        The 'link-layer address' used in the address resolution
        process (ARP/ND) will be:

		LID + capability_flags + QPN + GID 

        Note: This is the address included in the address
              resolution(ARP/ND) packets only. It does not imply the
              values or structure of the IB packets themselves.

	e) LID
                LID is the 'real' address. It is always needed and
                uniquely identifies a port in the IB fabric. There is
                only one LID per IB interface. 

	f) GID
                Many members desire to eventually support IPoIB over
                multiple IB subnets. Such a solution has to await
                IBTA's definition of the necessary routing
                infrastructure but the necessary hooks to enable the
                feature must be included now. 
		
                To reach a remote port one always needs to know its
                GID. Therefore the GID will be included in the link
                layer address exchanged during address resolution.

                Knowing the GID is also useful in supporting IPoIB
                nodes beyond a single IB subnet by way of using proxy
                ARP. An IPoIB router/bridge (or special device) can
                return its own LID but the GID of the IP node in
                another subnet.

	g) Capability flags
                Many members would like to take adavantage of a higher
                MTU, robustness, failover functions offered by non-UD
                transport modes of IB. The capability flags advertise
                whether the node implements IP over the different
                modes of IB.

	h) QPN
                IP over UD mode is mandatory. The QPN exchanged during
                address resolution is the QPN corresponding to the UD
                mode. 

	9) IPoIB - non-UD modes

        It is mandatory for all IPoIB stacks to support the UD mode.
        Some stacks may choose to implement IPoIB over non-UD modes
        too. Such a capability is indicated by a set of flags that are
        made part of the link-layer address that is exchanged between
        the nodes. The link layer address is defined as:

	LID(16 bits) + capability flags(8 bits) + QPN(24 bits)+ GID(128 bits)

        The QPN is the QPN to be used for IPoIB-UD mode.

	The capability_flags are defined as follows:

		+---+---+---+---+---+---+---+---+
		|RC |UC |RD |QPN| 0 | 0 | 0 | 0 |
		+---+---+---+---+---+---+---+---+
	
        The flags indicate the IB transport mode that is supported.
        There is no flag for the UD mode since that is always
        supported. The QPN flag indicates whether the node supports
        multiple QPNs with the UD mode. These QPNs may be associated
        with different ports and protocols. Note that multiple QPs
        implies multiple MAC addresses for the same IP address
        (something that is not commonly supported). 

	Rules :

		1) An IPoIB interface *always* supports UD mode.
                2) The QPN exchanged during address resolution
                   (ARP/ND) is always the QPN corresponding to the UD
                3) The flags are ignored on reception by the nodes
                   that don't support IPoIB with RC/UC/RD/QPN features.
                   A node is only required to recognise the flag(s) it
                   supports. It ignores the others. 
                4) A node always sets the flags it doesn't support to
                   zero. 

        Thus any two IPoIB nodes can always communicate since they
        always support UD mode. There is no interoperability issue. An
        edge device may choose not to support anything but the default
        UD mode but it can still exist in the same subnet as high-end
        servers that may implement IPoIB-UD/RC/UC etc.

 	Consider two nodes: A (supports IPoIB-UD/RC) and B (IPoIB-UD only)

	B wants to talk to A:

                It sends an ARP request. It ignores the RC flag in the
                reply. To it host A has answered and it will
                communicate on the UD QPN using UD mode.
	
	Impact on IPoIB node A:

                A receives the ARP request. This request lets it know
                that B doesn't support RC. It will therefore only
                communicate to B using UD mode.

        Note that such a scheme allows all nodes to talk to each
        other. The complexity is with the nodes that implement the
        non-default features. A simple implementation is not affected
        and can ignore the capability flags at all times with all the
        members of the subnet.

        Consider a third IPoIB node C which suports IPoIB-UD/RC

                It knows that A supports RC when it resolves the IP
                address belonging to A. Similarly A knows C's
                capability when they resolve the addresses. The
                implementation can therefore choose to communicate
                using RC mode between the two endpoints.

	MTU of the IPoIB-RC interface:

        The RC connection is setup as defined in IB specification. The
        private data field is utilised to exchange the MTU to be used
        for the interface.

        Each administrator decides the maximum MTU that its RC
        interface can accept. This is limited by the maximum MTU the
        HCA can accept. This value is exchanged during the RC
        connection setup as part of the private data field. The lower
        of the two values is taken to be the MTU of the IPoIB
        interface by both the ends.

The QPN flag:
------------

        This flag is used in conjunction with the UD mode. This
        implies if the UD mode supports multiple QPNs. The SIDR
        protocol is used by the interface after address resolution to
        make newer 'connections'.

        The communications on the default UD QPN is always allowed. A
        receiver MUST accept all packets on the default UD QPN even
        when a 'special' QPN has been associated with a particular
        service ID. 

UD/RD modes
-----------

	Similar solutions are possible with these modes. 

Vivek
-- 
Vivek Kashyap
Linux Technology Center,IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Jim Pinkerton" <[email-masked]>
Date: Mon, 10 Dec 2001 22:14:32 -0800

I agree. I also believe we need ethertype. The overhead is small, and it
enables existing naming infrastructure to be re-used. That is a huge
win.

Jim

> -----Original Message-----
> From: Michael Krause [mailto:[email-masked]]
> Sent: Monday, December 10, 2001 6:04 PM
> To: Larsen, Roy K
> Cc: 'Vivek Kashyap'; [email-masked]
> Subject: RE: [Ipoverib] IPoIB architecture/encapsulation
> 
> At 05:05 PM 12/10/2001 -0800, Larsen, Roy K wrote:
> >Vivek,
> >
> >I'd like to express my strong desire to have IPoverIB packets contain
> >explicit "ethertype" information.  Although I don't believe the
rationale
> >given to remove an ethertype from the payload is compelling (see
in-line
> >comments below), I believe a compromise can be reached by sending the
> >ethertype as immediate data. This keeps it out of the payload and
uses
> >existing IBA packet format standards.  I realize immediate data could
not
> be
> >used with raw data packets, but capabilities outside of UD for
IPoverIB
> are
> >beyond the scope of what we're trying to get drafted here.
> 
> In general, use of immediate data was intended strictly for backward
> compatibility of existing VIA implementations and nearly all of the
people
> involved in InfiniBand wanted it removed.  Strongly suggest not
relying
> upon this field for anything and not encouraging it use for IPoIB.
> 
> Mike
> 
>

---

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 11 Dec 2001 00:08:41 -0800

See below between <VK>

--
Vivek Kashyap
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                    Michael Krause                                                                                
                    <[email-masked]       To:     "H.K. Jerry Chu" <[email-masked]>                         
                    .com>                cc:     [email-masked], [email-masked]                         
                    Sent by:             Subject:     Re: [Ipoverib] IPoIB architecture/encapsulation             
                    ipoverib-admin                                                                                
                    @ietf.org                                                                                     

                    12/10/2001                                                                                    
                    06:56 PM                                                                                      

Agreed.  All implementations must support multicast to avoid a repeat of
the problems found with other technologies that did not support native
multicast.

<VK>
     I do want multicast supported at the IB layer. It is best supported
using multicast switches. As far as I can tell multicast by the SM is not
optional; it is optional at the switches only. So if the switches don't
support multicast the IB setup must find some solution - may be using an IB
level multicast server.

<VK>

>Section 3.4 Multicast address to LID mapping, IMO not much of
>explanation is needed here because according to IB spec., LID is
>allocated and managed by SM, period. (Clients can't specify a MLID when
>creating a MCGroupRecord.)

If one were using only Raw Multicast, they could provide such a capability
but given the use of IB transports for this effort, the multicast group is
identified by its MGID and not by the MLID (especially since one can map
multiple MGIDs to a single MLID making de-multiplexing in software even
more cumbersome to implement / support).

<VK>
     As noted in my earlier mail the IB specification in section 3.5.11.2.1
says that the MLID may be specified when creating an MGID. Was this
intended for Raw only ? I have been reading it to apply to both raw and UD
multicast and I am sure others have done so too.

<VK>

>Subsection 1) Sending/receiving IP multicast, I don't think a
>sender of a mulicast packet must be a member of that multicast group.
>This is similar to IP over Ethernet. The only trouble for IB is that
>there may not be a link mutlicast group to send this packet out. I have
>suggested for such a case to forward the packet to a local multicast
>router.

An endnode is allowed to inject multicast packets without being a
member.  As you note, there is an issue in that the IB SM may not have set
up the MLIDs to switch the packet into the fabric.  A solution would be to
always require the SM to use the default multicast output port per switch
to forward the packet into the multicast-enabled portion the
fabric.  However, even this is not really that practical as there can be
many multicast groups with different endnodes participating so a proxy
router would be required as you suggest or it may require the SM to set up
a switch that acts as a root for all multicast groups thereby allowing the
default to direct all packets to this switch who then forwards the packets
on the endnode's behalf.

<VK>
     An endnode is allowed to inject multicast packets without being a
member. It may, as noted, not reach the intended recepients becuase the SM
may not have setup the necessary MLIDs. Therefore the IPoIB solution
becomes:

     - all endnodes become members of the broadcast MGID.
     - if a multicast group doesn't exist
          - if level 2 compliance or IPv6 create it/join it and send
packets
               - if create fails
                    - send packet on broadcast MGID (guaranteed to reach
all recepients)

          - else send on broadcast MGID

          (in the meantime, depending on the error retry to create/join the
MGID)

<VK>

Vivek

---

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 11 Dec 2001 00:05:11 -0800

Roy,

     See below between <VK>

Vivek

---------------------

> From: Vivek Kashyap [mailto:[email-masked]]
> Sent: Friday, December 07, 2001 1:35 PM
> To: [email-masked]
> Subject: [Ipoverib] IPoIB architecture/encapsulation
>
>
> Folks,
>

<snip>

>          IP packets are encapsulated in the payload of the IB packet.

The QP should only receive packets with the correct Q_Key.  In your
proposal, this would be derived from all-nodes multicast/broadcast join
operation for the configured P_Key.  So I don't see how this will solicit
random packets.  Nevertheless, an Ethernet interface does receive random
packets and seems to cope with this very well using an ethertype.

<VK>
Yes, but any random user can use that Q_Key. Secondly, you will receive
ARP/IPv4 packets on the broadcast MGID by definition. There could be others
if that MGID got used by another IB application.

<VK>

>            introducing an artificial ethertype in the payload.

I am unaware of a deterministic ARP or IP packet signature that comes close
to the speed and simplicity of an ethertype compare.  If there is, wouldn't
it fall into the same aliasing problems you outlined in 1)?

<VK>
     Yes it would. Hence we can't solve the aliasing problem. Ethertype is
not useful for it.

     That leaves demultiplexing ARP/IPv4 (and IPv6). In the first nibble
IPv4 has (version) 4; IPv6 has 6; and ARP's hardware type has 0 (it is a 16
bit entity with allocations within 5 bits at present). So why can't we use
that ?

     In the case of 'ethertype' you are parsing 4 bytes of the header; in
the case of 'signature' you are looking at the first 4 bits.

<VK>

>            4092. Not the best of the situations.

There is nothing magical about a power of 2 MTU.  Ethernet seems to get by
pretty well on 1514.

<VK>
     There is nothing magical but it certainly is more covenient. Not the
best of reasons but nice.

<VK>

>            'ethertype' ie. the type field external to the payload.

This is the most compelling argument.  However, this will most likely
affect
CA design and certainly existing software interfaces.  Because of this, I
seriously doubt that this will make it into existing or imminent profile B
silicon designs and IBA software libraries.  If the IBTA adopts this
proposal without making the utilization of this field optional, and its use
is required by IPoverIB, it could mean a limited deployment and therefore
slow adoption of IPoverIB.

<VK>
      May be I should have listed this first :). I got a good response to
the suggestion in IBTA.

      I am not sure what you mean by making the utilisation of the field
optional. I don't know if you have looked at the proposal; may be we could
discuss it offline (IBTA issues). I'll be restarting the process in LWG  in
the coming days.
<VK>

< snip >

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 11 Dec 2001 07:54:16 -0800

Vivek,

You were silent on the immediate data proposal.  I was hoping this would
meet both of our needs.  Do you have an opinion on using this existing
standard interface for carring the ethertype?

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 12:05 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Roy,

     See below between <VK>

Vivek

---------------------

<snip>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

< snip >

---

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 11 Dec 2001 10:00:17 -0800

Roy,
     sorry about that. You noted that it cannot be used for UD. That sort
of removes it from the problem. I would be ok with using it if it was
possible..

Vivek

                    "Larsen, Roy                                                                                  
                    K"                   To:     Vivek Kashyap/Beaverton/IBM@IBMUS                                
                    <roy.k.larsen@       cc:     [email-masked]                                                
                    intel.com>           Subject:     RE: [Ipoverib] IPoIB architecture/encapsulation             
                                                                                                                  
                    12/11/2001                                                                                    
                    07:54 AM                                                                                      

Vivek,

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 12:05 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Roy,

     See below between <VK>

Vivek

---------------------

<snip>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

< snip >

---

From: "Grun, Paul" <[email-masked]>
Date: Tue, 11 Dec 2001 10:05:30 -0800

I absolutely disagree with Mike's characterization of immediate data being
"intended strictly for backward compatibility of existing VIA
implementations..."  That is simply, absolutely and categorically not true
and is totally outside the bounds of this discussion.  The fact is that
immediate data is part of the IBA architecture.  I have seen or heard
*nothing* that suggests it won't be part of the architecture going forward.

Furthermore, I totally reject Mike's claim that "...nearly all of the people
involved in InfiniBand wanted it removed."  Mike is certainly entitled to
his opinion and that of HP.  He is not entitled to characterize the sense of
"most of the people."

Let's be clear here:  As is *always* the case in the IETF, the individual,
including Mike, speaks for himself.  I strongly resent *anyone* representing
my views. 

Roy has made a perfectly reasonable suggestion.  Let's give the idea its due
consideration.  If, at the end, we find that immediate data is not the right
way to carry the ethertype field, so be it.  But let's not doom it to
oblivion prematurely because somebody doesn't like the concept of immediate
data.

-paul

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Monday, December 10, 2001 6:04 PM
To: Larsen, Roy K
Cc: 'Vivek Kashyap'; [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

At 05:05 PM 12/10/2001 -0800, Larsen, Roy K wrote:
>explicit "ethertype" information.  Although I don't believe the rationale
>given to remove an ethertype from the payload is compelling (see in-line
>ethertype as immediate data. This keeps it out of the payload and uses
>existing IBA packet format standards.  I realize immediate data could not
be
>used with raw data packets, but capabilities outside of UD for IPoverIB are

In general, use of immediate data was intended strictly for backward 
compatibility of existing VIA implementations and nearly all of the people 
involved in InfiniBand wanted it removed.  Strongly suggest not relying 
upon this field for anything and not encouraging it use for IPoIB.

Mike

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 11 Dec 2001 10:28:56 -0800

Vivek,

You misread what I wrote, which was:

>I realize immediate data could not be

Immediate data is available for all message types EXCEPT raw packets.
(Refer to table 60 on page 388 of the IBA spec.)  Using raw packets as a
transport mechanism is outside the scope of the IPoverIB spec as is using
RC, UC, and RD services.  Given this, can I assume you support immediate
data for conveying the ethertype?

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 10:00 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Roy,

Vivek

                    "Larsen, Roy

                    K"                   To:     Vivek
Kashyap/Beaverton/IBM@IBMUS                                
                    <roy.k.larsen@       cc:     [email-masked]

                    intel.com>           Subject:     RE: [Ipoverib] IPoIB
architecture/encapsulation             

                    12/11/2001

                    07:54 AM

Vivek,

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 12:05 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Roy,

     See below between <VK>

Vivek

---------------------

<snip>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

<VK>

< snip >

---

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 11 Dec 2001 11:25:54 -0800

I am ok with LID not being specified by the application but noted as an
aside that an interface exists. It is just that it has been closed off for
LID specification. If LID was allowed to be specified then one would use
the same interface as is used for queries or P_Keys.   Not contending the
MGTWG decision on disallowing such specification or what the intent of the
IB spec is. It was just a comment.

Vivek

                    "Bill                                                                                         
                    Swortwood"           To:     Vivek Kashyap/Beaverton/IBM@IBMUS, <[email-masked]>          
                    <billiam33@att       cc:     <[email-masked]>                                              
                    bi.com>              Subject:     RE: [Ipoverib] IPoIB architecture/encapsulation             
                    Sent by:                                                                                      
                    ipoverib-admin                                                                                
                    @ietf.org                                                                                     

                    12/11/2001                                                                                    
                    11:09 AM                                                                                      

Vivek -

 The compliance statement on 724-725 (Vol 1.a) is a bit confusing, to be
sure. But the
intent was to allow specification of the Q_Key, MTU and PKey for a
multicast
group, with some allowance to set the MGID. The MLID usage in the attribute
is for later query or deletion, but not to set. See specification text
below:

A multicast group can be created by the SubnAdmSet() method, specifying
Add Record in the edit modifier, the Q_Key, MTU and the P_Key (all
other fields are zero). If a particular MGID is required, it can be
specified
in the SubnAdmSet() as well. When a multicast group is deleted, it can be
done by the SubnAdmSet() method, specifying the Delete Record in the
edit modifier and the McGroupRID. Alternatively, it can specify the MGID,
Q_Key, MLID, MTU and P_Key. If a field is and can be set to zero for a
delete, it will result in a match for that field for all MCGroupRecords by
Subnet administration.

Vandana is correct, and has just earned her black belt in MGTWG spec
folklore. :)

- Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Vivek Kashyap
Sent: Tuesday, December 11, 2001 10:15 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

OK. That settles it then.

LID is always chosen by the SM.  Therefore  IP address to the MLID  mapping
or MGID to MLID mappings are beyond the control of IPoIB and hence beyond
its scope.

BTW:  Wouldn't it be possible to set the value MLID in the MCGroupRecord
create request  ? That would be an interface since the same method is
suggested for specifying the P_Key in the create request.

Vivek

                    Vandana Rao
                    <[email-masked]       To:     Vivek
Kashyap/Beaverton/IBM@IBMUS, [email-masked]
                    p.com>               cc:     [email-masked]
                                         Subject:     Re: [Ipoverib] IPoIB
architecture/encapsulation
                    12/11/2001
                    08:31 AM

At 11:59 PM 12/10/2001 -0800, Vivek Kashyap wrote:
<snip>

>Yes, they can. See section 3.5.11.2.1.
>
>  " 1) An (administrative) application defines (or determines) a target
>multi-cast
>    group address (GID). It specifies particular group characteristics
>    (PMTU, raw, P-Key, etc.) and creates the multicast group by invoking
>    a multicast group create to the multicast management entity. This
>ap-plication
>    may request a specific multicast LID or have one allocated
>    for it. "

Section 3.5.11.2 is incorrect and an erratum against this has already been
filed by the IB MgtWG as it
is inconsistent with what is stated in the SA chapter (which is where the
the compliance statements regarding multicast group create are defined).
The MLID is always created by the SM/SA. The application has no interface
available to specify MLIDs.

-Vandana

---

From: Michael Krause <[email-masked]>
Date: Tue, 11 Dec 2001 11:20:09 -0800

>*nothing* that suggests it won't be part of the architecture going forward.

I do not want to violate the IBTA confidential requirements, but the only 
reason it is present at all in the architecture is backward compatibility 
support for VIA and there was strong support for its removal by nearly all 
companies in the IBTA LWG at the time.  I do not believe I am alone in the 
belief and recommendation that immediate data usage should be strongly 
discouraged.  I did not state that it was going to be removed from the 
architecture.

>his opinion and that of HP.  He is not entitled to characterize the sense of

If you want to go back and review the LWG meeting minutes (see IBTA LWG 
meeting, August 1999 in Palm Springs), I believe you will find the exact 
issue and how this was resolved.  My statement of nearly all people is 
completely accurate.

>including Mike, speaks for himself.  I strongly resent *anyone* representing

Not representing your views and did not state it as all but nearly all 
which allows for your views to be different than my views.

>oblivion prematurely because somebody doesn't like the concept of immediate

I don't disagree that it is a reasonable suggestion but stated that it is 
not an optimal solution and uses a feature that many would prefer to go 
away and never be used.

Mike

---

From: Michael Krause <[email-masked]>
Date: Tue, 11 Dec 2001 13:51:12 -0800

So, let's get back to the proposal.  There are three methods being proposed 
to identify the subsequent byte stream:

(1) Encode the Ethertype within the immediate data portion of the IBTA message

(2) Encode the Ethertype within the data payload which contains the IP / 
etc. headers.

(3) Encode the Ethertype (to a limited extent) in some newly defined bits.

Option 1 is possible because every implementation is supposed to support 
immediate data in their HCAs.   There are some that do not like immediate 
data for a variety of reasons with some being driven by 
implementation-specific issues.

Option 2 is also possible and being in-line with the rest of the headers 
provides some benefits (potentially performance related depending upon 
implementation).  I believe this was in the original proposal at the 
previous IETF meeting and was generally well received.

Option 3 presents a number of problems stemming from it being new 
functionality.  If we believe the current implementations are just 
development vehicles and all next gen implementations would include this, 
then this might be practical.  However, option 3 uses optional 
functionality and therefore may not be universal.  Anything that is not 
universal presents both economic and technical problems to solve that in 
this case, may make it impractical pursue.

My personal preference is option 2 as it will work everywhere and it 
simpler to implement from a software perspective.

Mike

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 11 Dec 2001 14:28:15 -0800

To be complete, there was a 4th option which was to imply the Ethertype type
by looking at the four most significant bits of the first byte where a value
of 4=ipv4, 6=ipv6, and 0=arp.

For the record, my strong preference is option (2) - the Ethertype as the
first field in the payload.  The immediate data proposal was only offered as
an alternative to option (3) as it uses existing IBA mechanism.

Roy

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 1:51 PM
To: Grun, Paul
Cc: Larsen, Roy K; [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

(1) Encode the Ethertype within the immediate data portion of the IBTA
message

Mike

---

From: "Vivek Kashyap" <[email-masked]>
Date: Wed, 12 Dec 2001 00:04:42 -0800

Roy, you didn't answer the question I asked .. see below.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

<VK>

<VK>

---

From: Yaron Haviv <[email-masked]>
Date: Wed, 12 Dec 2001 16:43:35 +0200

Agreed that UD is first priority, just need to make sure we don't preclude
the use of RC/RD in the future

I don't see any reason why not to go for ethertype in payload
It is probably better than immediate data, bit option alternatives
I think the TCP/IP stack issues could be easily overcome 
 
As for checksum since it has a significant overhead due to caching issues,
etc' it could be interesting
You could send a bit field with each message, notifying the other side if
checksum is needed for transport & network
Very similar to MS TCP/IP task offloading stuff
This way a node/router that doesn't do the checksum wont set those bits, it
will only set it if it either originated from him, or he did the checksum
verification for the packet 

Yaron
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Michael Krause
Sent: Wednesday, December 12, 2001 4:05 PM
To: Yaron Haviv
Cc: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

At 11:05 AM 12/12/2001 +0200, Yaron Haviv wrote:
>Bill,
>
>My personal experience is with BSD & Linux stacks since they are public
>domain The network adapter drivers specifically put the entire Ethernet+IP
>headers
>in the sk_buf/m_buf From my little experience with NDIS drivers, I assume
>it is also the case in
>Windows (See TCP/IP Illustrated Vol 2, page 35) You could use both
>offsetting, and the 2 data segments approach I mentioned to resolve issues
>around it in other cases I understand from the responses in the group that
>this option is gaining wide acceptance, more than the other alternatives

In general, most networking stacks post receive buffers that contain
sufficient length to accommodate all headers from all layers allowing one
to implement a single DMA for the packet when it arrives - performance win
for most implementations.  Each layer within the stack then parses their
component of the packet and advances the pointer so that the next layer
only views their headers and nothing before - agree with Bill that it would
be unwise to look earlier through pointer arithmetic as you never know if
the packet was part of a scatter operation (should header / data split be
used).  Given this is a well-known paradigm, the advantages of using the
Ethertype right after the IB headers I hope are apparent.

>I could try and write a draft for IPoIB over RC/RD, any one who likes to
>help is welcomed

Would like to see the workgroup first get through the requirements and a
common UD draft so we can get it implemented and start interop testing as
many of us require IP over IB to deliver solutions to our customers in the
coming months.

>Another thing that was mentioned in the recent emails was that IPoIB is
>defined just for UD In the discussion in LWG few months ago I recall few
>capability bits in one
>of the suggestions I personally think we should not preclude the use of
>RC/RD for IPoIB unicast
>traffic and leave it as an option This could be used in future
>enhancements to increase MTU to "Jumbo Frames", add help avoiding the
>Checksum calculations

Unless you are absolutely sure that the packet was originated and I mean
absolutely originated within an IB subnet, one cannot just turn off
checksum calculations just because IB provides strong end-to-end data
integrity.  Given that endnodes may act as routers between the Ethernet /
etc. fabrics and IB and there is no way to signal this has occurred at the
IP layer, checksums must be calculated by all endnodes.  A smart CA vendor
might come up with some technique to accelerate this calculation for
software to then fix-up (much like we do checksum off-load today with
Ethernet) but that is not part of the specification (IP or IB).

>In the near future RC might assist in supporting profile A chips that
>doesn't have large MTU's, and will be deployed in environments where they
>provide enough value (I would have hoped to have just Profile B chips,
>however they are here, with us, and probably for some time)
>For some of those limited chips that also doesn't support Multicast we
could
>implement a multicast server that will either convert the traffic to
>multicast or send it to the multiple clients, it might also proxy the ARP's
>and answer based on his internal table & SA information
>To be clear I don't like it !!! (see ATM), but it could do the work as an
>interim solution that will help accelerate IB adoption with the given HW
>I assume that someone building a multimedia solution relying on UDP
>multicast will probably wont use such HW, however it might be sufficient
for
>ARP's, DHCP's, etc'

The requirements document is focused on what would be called profile B
endnodes.  Given just the sample of the problems you note here, most people
appear to be comfortable with staying with a profile B focus and getting
the requirements and associated specification implemented first.  Several
of us will then be willing to help develop the connection oriented variants
should there be sufficient interest.

Mike

---

From: "Ashwin Kamath" <[email-masked]>
Date: Thu, 13 Dec 2001 09:54:52 -0600

Vivek,
 A third protocol using IPoIB UD queue pair may be assisted using the ether-type field. Questions I want to ask are 
1. Is there a possibility of addition of another protocol besides IPV4/IPV6 and ARP?
2. If there is a possibility of adding other protocols, is there a way to assist addition of more protocols?
3. Do we want to throttle all possibility of adding another protocol at this time?
4. If we do have an ether-type field, isn't it possible to dictate to all protocols using IpoIB resources to use this field in a way that IPoIB still functions?
-Ashwin.

---------- Original Message ----------------------------------
From: Vivek Kashyap <[email-masked]>
Date:  Wed, 12 Dec 2001 15:40:49 -0800 (PST)

>Yaron,
>        I am afraid we have sidetracked. The comment about performance
>        was different. The point I was making was that generally you
>        would get better performance if you had less to do. It doesn't
>        mean that one cannot by specialisation mitigate the negatives
>        of a more complicated task.
>
>	  delete/add header vs. no such work. I like no work. 
>
>        Here is the problem and the reasoning:
>
>        The network stack doesn't change you merely add a network
>        interface driver. This driver has the task of determining the
>        upper layer to call. It can call it directly or it can slot
>        the packet in some type based queue. The interface driver will
>        determine the type from the packet stream. It can do so in
>        many ways. One of the ways is by using the protocol signature.
>        Another is by way of 'ethertype' and a third is by mandating a
>        QP per protocol.
>
>        The use of 'ethertype' is not universal. A choice by IPoIB to
>        use ethertype does not mean 'protocol X' over IB will use the
>        same encapsulation and therefore you cannot rely on
>        'ethertype' differentiating your packets. 
>
>        So the 'ethertype' is of use in differentiating IPoIB
>        protocols only (IPv4, IPv6 and ARP). For this task we have two
>        other methods that require no change in the on-wire
>        encapsulation: use the protocol signatures (4 bits in the
>        begining) and use separate QPs per protocol. Of the two : use
>        protocol signatures doesn't require any on-wire change,
>        doesn't require the driver to strip headers, doesn't require
>        the TCP layer to keep space for the header. That is why it is
>        preferable.  
>
>        In essence the protocol signature solution gives us the same
>        returns with the additional advantage of :
>
>		- no need to reflect it on the wire
>		- no removal/addition of headers
>	
>
>Vivek
>
>
>> 
>> Vivek,
>> 
>> I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on mem access as well as CPU cycles
>> My assumption is that OSV?s will not change the way their TCP/IP stack is built and separate ARP?s from IP?s, they will just add IPoIB as another net interface driver
>> The demultiplexing doesn?t add much value since it is done in the same kernel context, there is no urgency in dealing with the ARP?s, there aren?t many ARP?s, it also relates to the stack work (ARP response might release pending IP packets)
>> 
>> it looks to me like the 4 bit option is not the straightforward approach 
>> 
>> If you want to improve performance significantly, without affecting the stack you should do things similar to what the new NIC?s do:
>> -  Offload checksum (avoid cache trashing)
>> -  Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
>> -  Suppress Interrupts 
>> -  Improve memory management 
>> These things could be done with RC/RD and using some IB tricks 
>> 
>> Yaron
>> 
>> Original Message:
>> -----------------
>> From: Vivek Kashyap [email-masked]
>> Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
>> To: [email-masked], [email-masked]
>> Yaron,
>> > Few questions & suggestions
>> > 
>> > I'm trying to understand way not to take the straightforward approach for
>> > the ethrtype field
>> > And just add that before the IP header in the payload area, if alignment is
>> > needed so align it
>> 
>>         Since we already filter the recepients by way of QP to one
>>         (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
>>         work is the same whether you see 4 bits or 32 bits. But it is a
>>         lot less with 4 bits. 
>> 
>>         The interface driver will distinguish the types and call the
>>         upper layer accordingly.
>> 
>>                 It is easier with 4 bits than with 4 bytes. Some people
>>                 have mentioned performance - I would say that if you
>>                 have less to do you are more performant. With 4 bits
>>                 you have less to do. Secondly, if you are in
>>                 'performance' mode switching in hardware then it is
>>                 even better since you don't have to strip the header
>>                 but merely have to pass the buffer to the relevant
>>                 upper layer directly.
>> 
>>         The interface driver does not have to do the followig when
>>         using protocol signatures:
>> 
>>     - It doen't have to add a header when sending
>>     - It doesn't have to receive header when receiving
>> 
>>   These tasks are required for if one invents an 'ethertype'.
>> 
>>         The upper layer doesn't have to keep space for the header, the
>>         ethertype, when allocating data for sending if the protocol
>>         signatures are to be used.
>> 
>>         I don't see any gain if an artificial header is included to
>>         distinguish protocols in IPoIB. I see more work. We already
>>         know, by definition, it is IP or ARP. Just test which one it
>>         is by looking at the first 4 bits.
>> 
>> 
>> > 
>> > I might be missing something here so forgive me if I do
>> > It looks to me like TCP stacks are used to get a packet with everything
>> > including Ethernet header, it is not like we are doing full header/data
>> > separation that could help the stack
>> > On the worst case if the TCP implementation has problems with it we could
>> > have 2 data segments in the receiving side, one for the first word, second
>> > for the rest
>> 
>> Things are generally done in simpler ways. The IP stack gets the IP
>> packet. The lower headers are stripped away. If you don't have the
>> any header to strip in your interface driver all the better.
>> 
>> > 
>> > defined just for UD
>>         If you look at the beginning of this thread (the mail that
>>         started it all) I have discussed capability flags and how
>>         RC/RD/UC connections can be setup and work interoperably with
>>         UD only implementations. IPoIB over UD is the default and
>>         madatory. 
>> 
>> 
>> Vivek
>> 
>> --------------------------------------------------------------------
>> mail2web - Check your email from the web at
>> http://mail2web.com/ .
>> 
>> 
>
>
>-- 
>Vivek Kashyap
>IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>

---

From: "Ashwin Kamath" <[email-masked]>
Date: Sun, 16 Dec 2001 11:25:33 -0600

I am sorry, I do not know what happened here. This was an earlier post, which ended up with the moderater for some reason. Now it is out of context.

-Ashwin