Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or not)

From: "Arvind Murching" <[email-masked]>
Date: Sun, 13 Jan 2002 11:11:59 -0800

FWIW, other IP over FOO RFCs do take care to allow protocols other than
IP to share the link layer connection, typically using an EtherType.

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]] 
Sent: Saturday, January 12, 2002 4:52 PM
To: [email-masked]
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

Sorry I am late to the party (was stuck on something else most of Dec.),
but like to make some comments regarding EtherType.

While I don't buy the performance arguement, I sympathize with Vivek on
his feeling toward EtherType. Architecturally it is ugly to force one
into the payload. The right approach would have the framing layer
carrying a field identifying the type of data being encapsulated. But
BTH wasn't designed for encapusulation purpose. That is supposed to be
the job and the only purpose of raw datagrams. 
Unfortunately due to various reasons the latter has been considered by
many as not fit for the job of IPoIB encapsulation.

Aside from the architectural correctness, I only see two valid concerns
against not including EtherType:

1. Compatibility issue - how many existing stack implementations expect
EtherType, or part of framing header to be present in the datastream?
Although the driver can always fabricate a pseudo header if one is
needed but is missing, it may incur some performance penalty since it
won't be in-line.

2. Not including Ethertype forbids one from sharing the same QP with
other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
Note that this concerns only applies unicast. For multicast, by
embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use only,
demuxing can be based on MGID so sharing QPs won't be an issue.

Before the two questions are fully understood, I'm leaning slightly
toward including the EtherType, just to be on the safe side.

Jerry

<snip>

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Strahm" <[email-masked]>
Date: Sun, 13 Jan 2002 12:27:13 -0800

I am very much talking about a Bill Strahm personal opinion here, and not as
WG Chair...

You can consider the "ethertype" to be something akin to a port, saying
please deliver this to the protocol above with this ethertype.  The
equivalent to "ethertype" for IB is QPN in my eyes.  I can put as many
"Layer 3" protocols on top of IB by having each Layer 3 protocol bind to a
QPN, then have packets to that QPN show up to the bottom of the Layer 3
protocol...

I understand that QPN are a scarce resource, but attaching multiple
protocols on top of a single QPN and having an upper layer demux does not
seem like the answer, if I do this, then I do not know what memory address
to deliver the packets to, so I might incur a copy... At a minimum I incur a
page flip.

WG Chair hat on ...
My understanding is that we have 2 major points to consider before we can
get to WG last call on the Encapsulation draft they are
1) Use of Ethertype, or not in the encapsulation
2) What to do about IP subnets crossing IB subnets, allow, architect for,
don't consider

I know that there are some link issues that Jerry has brought up, hopefully
his draft on the topic will come up early next week and people can tell me
what there opinions are.

The goal is to have a draft that is VERY close to WG Last call for the
Minnesota IETF, so we can tweak it and ask for Last Call after the meeting.
Please comment on these topics and I will try to direct the conversation,
and see if we have an answer.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Arvind Murching
Sent: Sunday, January 13, 2002 11:12 AM
To: H.K. Jerry Chu; [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation (EtherType or
not)

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: Saturday, January 12, 2002 4:52 PM
To: [email-masked]
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

Jerry

<snip>

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Jan 2002 16:37:08 -0800

Two cents on a variety of subjects concerning what fields to exchange 
between endnodes.

Mike

Link address fields:

P_Key - For each partition that an endnode participates and requires IP 
over IB services, it will send an ARP request to identify other nodes on 
the IP subnet.  Only those endnodes that share a common P_Key will respond 
and thus be discovered.  Hence, all discussion of P_Key is irrelevant from 
the IP over IB perspective and should not be included within the IP over IB 
specification.

Q_Key - Q_Key provides no value to IP over IB solutions and should be 
treated as a well-defined value by the specification.  Recommend a value of 
0xFFFFFFFE be used.

GUID - The value of a GUID is primarily being ascertain how things are 
co-located on a given CA.  At first glance this seems to have some value 
but most people no longer think it is required to establish a communication 
channel between endnodes, in other words, use the GID to define the port 
and let software figure out any co-location using local means if 
required.  Hence, this field should not be included for IP over IB operations.

GID - In reality there are two ways of creating GIDs within IB (the 
original IP over IB architecture discussion isn't quite accurate). The 
first (called GID 0) and is created by concatenating the port's EUI64 with 
the GID prefix to create a 128-bit value.  The second by concatenating a 
SM-assigned 64-bit value with the GID prefix.  This second method provides 
a way for endnodes to have a variety of link addresses (e.g. one per OS 
instance, per set of IP addresses per endnode, etc.).  In general, given an 
IP subnet may span multiple IB subnets, this is the mandatory value to 
exchange to uniquely identify an endnode for address resolution purposes.

QP - There is some benefit to exchanging the QP as part of address 
resolution.  For example, an endnode may have different policies in terms 
of how a QP is mapped per IP subnet.  One might have one QP per IP address 
or may have one QP per multiple IP addresses (potentially for all IP 
communications).  As such, the ARP process should exchange the QP where it 
is known and should use QP 0xFFFFFF when it is unknown, e.g. an IP router 
at the edge of an IB fabric instance.

LID - There has been general discussion that IP subnets should be able to 
span multiple IB subnets.  As such, the LID is invisible to the address 
resolution process since this is an IB subnet only issue (whether it refers 
to an IB router or an IB endnode is irrelevant).  The counter arguments 
that one could optimize the address resolution process slightly by avoiding 
a trip to the SA / SM by including the LID as a visible exchanged item (it 
is always available since this can be derived from the IB packet in the 
first place) really don't provide much substance.  The resolution process 
should be able to resolve an address independent of the underlying IB 
fabric using a single algorithm.  Implementations can take appropriate 
steps to optimize their communications through smart caching and 
examination of the provided IB headers.

GRH - There has been some discussion of whether the GRH should be included 
for all ARP operations or not.  For an ARP request, one injects a multicast 
packet per partition.  The receiving endnode will receive the packet and 
can determine whether the source is on the same subnet or not and respond 
with either a GRH or a LRH-only packet.   As such, it seems GRH usage is 
implicit and the current IB specifications on usage do not require any 
additional work within the IP over IB specification.

Capability bits - If an IP router is allowed to exist between two endnodes, 
then the specification must include a method to exchange capabilities (e.g. 
additional IB transport services that may be used to for IP over IB 
communications).   Since this topology is allowed, the specification must 
provide a method to exchange capabilities through some form of standalone 
operation, i.e. an endnode must be able to resolve an address to an IP 
router. The endnode will then inject a capabilities enumeration message to 
the destination endnode.   Given the endnode must be able to perform this 
operation to guarantee interoperability separate from the address 
resolution process, it would be preferred if the capability discovery (only 
required if an endnode provides additional services beyond IP over IB which 
is not necessarily universal) be kept a separate operation.  Once the 
capabilities are determined, these can be cached for a fairly substantial 
amount of time such that there is no performance loss for the majority of 
the operations that may occur between endnodes.  In addition, the 
capabilities exchange may be used to provide a target QP to be used for 
each capability thus allowing one to optimize the communications to 
initiate / implementations of these capabilities.

Ethertype - Quite a bit of discussion has occurred but it may be fair to 
state that the address resolution provides a first level method to obtain 
the associated QP for a given IP address and that the capabilities exchange 
may be used to provide additional QP for a given capability (e.g. perform a 
RC establishment using QP X), it seems the Ethertype is no longer necessary 
and can be removed from the specification.

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 13:34:47 -0800 (PST)

> the IP over IB perspective and should not be included within the IP over IB 

        Agreed except to ensure unique MGIDs within partitions we have
        had to include the P_Key into the IP multicast to MGID
        mapping. 

> treated as a well-defined value by the specification.  Recommend a value of 

        This is left to the admin. The value set in the broadcast-GID
        is used for all communication.

> instance, per set of IP addresses per endnode, etc.).  

        You can create any form of interface within your machine and
        tie it to any number of link addresses; these can be LIDs from
        different ports. Whatever GID is chosen it is finally tied to
        a port and the LID needs to be determined. Having different
        link addresses don't provide any gain in themselves if they ar
        tied to the same port. You can differentiate more easily by
        using QPs. If the ports are different then use the LIDs
        anyway. 

> In general, given an 

        Why do we need to span multiple IB subnets? The fabric is
        broken into IB subnets for isolation. Why should we want to
        rejoin them when they are separated by design ? 

> at the edge of an IB fabric instance.

        Any endpoint that will talk IP will have a default QP. It
        returns that. I am not sure how it cannot know its own QP.

> examination of the provided IB headers.

        As discussed in the earlier mails these issues are important.
        The argument is that the method to determine any other
        information is upto the end node's implementation. Determining
        LID is disjoint from that. Those that don't want to use the
        LID in the adress resolution packets can always ignore the
        value on input and consult the SA.

        The process when determining the LID when across a subnet *is*
        different by definition. In the case of resolving the LID
        within the subnet the LID is received from the peer/SA but
        when communicating across IB subnets the LID of the first hop
        router is received from the routing infrastructure. A common
        method is not possible.

> additional work within the IP over IB specification.

	Agreed.

> initiate / implementations of these capabilities.

        Please see the
        draft-kashyap-ipoib-advanced-capabilities-00.txt. I think I
        have taken good care of interoperability. The default case
        which doesn't support RC/QPN modes work seamlessly with nodes
        that do.

> and can be removed from the specification.  

	Agreed.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 15:39:10 -0800

>         had to include the P_Key into the IP multicast to MGID

MGID is an IB MGID that is obtained by an endnode joining an IB multicast 
group.  When it joins such a group, it be definition is provided the P_Key, 
etc. parameters.  As such, there is no need to have P_Key used for the IP 
multicast to MGID mapping - the identification of the IP service implicitly 
will provide the requisite values.

>         is used for all communication.

I believe the simplification to a well-known value will provide better 
interoperability as Q_Key in this context provides no benefit and having it 
administered seems counter productive.

>         using QPs. If the ports are different then use the LIDs

The example provided illustrates why multiple GIDs and the use of GID as 
part of address resolution has value.  As such, different link address has 
value and can be used as part of a QoS strategy for different IP workloads 
as well as multi-OS instances per port.

>         rejoin them when they are separated by design ?

See another response but the essence is IB is a layer 2 from an IP 
perspective and there is no value in not supporting a transparent IP subnet 
across multiple IB subnets.  People are already managing IP subnets and 
their associated infrastructures so why not allow this to be supported from 
day one.

>         returns that. I am not sure how it cannot know its own QP.

Of course it can know the individual QPs - the method is implementation 
specific as to how.  The spec need only state that a QP will be 
returned.  The local meaning of that QP within the endnode is outside the 
scope of the spec.

>         router is received from the routing infrastructure. A common

One can create a common method by simply asking the SA.  Anything beyond is 
an optimization and implementation specific.  Focusing on the architectural 
minimums will lead to a more interoperable solution from the start.

Mike