Subject: Re: [Ipoverib] Encapsulation Issues

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 5 Nov 2001 12:42:51 -0800 (PST)

Attached are my preferences and some comments.

Regards,

Jerry Chu
Solaris Clustering
Sun Microsystems, Inc.

--------------------------------------------------------------------
>1. Determination of service level
>Option a: Use SL from all hosts multicast group
>Option 1: Use SL from path record

Does the choice of SL by individual node affects interoprability?
If not, I contend that this doesn't belong here (IPoIB standard). We
should leave this completely to implementations. The same arguement
applies to other IB attributes as well. IPoIB spec should focus on
aspects of IB that are required in order for the IP level interoperability.
It should not talk about generic IB rules. IPoIB clients are themselves
IB clients. Their compliance with IB spec is always assumed.

>
>1a. Determination of service level
>Option a: Include a flag provided by the subnet administrator that
>indicates the SL of the all hosts multicast group may be used for IP
>unicast communication.
>Option 1: Do not include such a flag.

See 1 above.

>
>2. IB Transport Service
>Option a: Require support of unreliable datagram; permit use of others
>Option 1: Title RFC something like “Encapsulation of IP in IB Unreliable
>Datagram”, require use of unreliable datagram, and within this RFC
>remain silent with respect to other transport methods (neither implies
>nor prohibits use of other IB transports).

Option 1 with one important addition - a mechanism for an IPoIB node to
determine upfront what (if any) non-default transport is supported on a
remote node.

>
>3a. Relationship between IP and IB subnets
>Option a: An IP subnet is contained within a single IB subnet.
>Option 1: An IP subnet spans all IB subnets interconnected with IB
>routers.

If the current IBA spec has all the needed information for us to support
1. without introducing much more complexity, then I'd say why not?
Otherwise a.

>
>3b. Mapping IPv6 scope bits to IB GID scope bits (the answer to this is
>implied by the decision made in 3a).
>Option a: Use local scope in IB GID for all IP packets
>Option 1: Use global scope in IB GID for all IP packets

This really depends on the scheme we come up with from 3a. Also don't
forget the site-local scope, which may be more appropriate than global.

>
>4a. Mapping of IPv4 multicast address to IB multicast GID
>Option a: Define an algorithmic mapping
>Option 1: Define a method to dynamically map as required

Option a. (Haven't seen any concrete proposal on option 1 yet.)

>
>4b. Mapping of IPv6 multicast address to IB multicast GID
>Option a: Define an algorithmic mapping
>Option 1: Define a method to dynamically map as required

Option a. (Haven't seen any concrete proposal on option 1 yet.)

>
>5. Mapping of multicast GID to multicast LID
>Option a: Define a mapping
>Option 1: It’s the IBTA’s problem, let them fix it

Option 1.

>
>6a. Contents of the hardware address for IBARP (in addition to QPN,
>GUID, and QKey)
>Option a: Include LID
>Option 1: Do not include LID

The question assumes the inclusion of QKey and GUID already. Why?
It looks like the only concensus we have so far is to include QPN.

A better question would be:
"Contents of the hardware address for IBARP besides QPN"

>
>6b. Contents of the hardware address for IBARP (in addition to QPN,
>GUID, and QKey)
>Option a: Include GID
>Option 1: Do not include GID
>
>6c. Contents of the hardware address for IBARP (in addition to QPN,
>GUID, and QKey)
>Option a: Include Capability Flags
>Option 1: Do not include Capability Flags (but possibly include a
>reserved field that could be used for capability flags in the future as
>RFCs are developed to define the capabilities)

I'd like to propose Option 2: 
Do not embed any capability flags in IBARP. The capability is a per-node
property. It's quite a waste to include that in every ARP handshake.
Instead, embed the capability flags somewhere else in the SA database
(e.g. the CapabilityMask of PortInfo) for which any node on the subnet can
query.

>
>7. Inclusion of GRH in IB subnet local unicast IP over IB packets (GRH
>required in multicast packets and inter-IB subnet packets in accordance
>with the InfiniBand specification)
>Option a: Recommend inclusion of GRH in all IP over IB packets
>Option 1: Remain silent (which leaves inclusion of GRH in unicast
>packets optional as specified by the InfiniBand specification)

Option 1.

>
>8. Inclusion of a field to identify the protocol (like an “Ethertype”).
>Option a: Do not include a protocol identification field
>Option 1: Do include an identification field

Another option is to use a flag during address resolution to indicate
whether EtherType is needed or not. E.g. if a QP is dedicated by the
the local node for a specific protocol, a protocol-type field is not
needed any more. (We need to evaluate some performance/complexity tradeoffs.)

>
>9.  Determination of multicast parameters for the multicast group used
>for IBARP. (Note: there was a proposal to use the InfiniBand SIDR
>mechanism; at this point there appears to be no one that believes this
>is a workable solution).
>Option a: Use a well-known MGID and do a MCGroup lookup to get the rest
>of the parameters.
>Option 1: Use a well-known ServiceName, do a ServiceRecord lookup to get
>the MGID, and then do a MCGroup lookup for the rest of the parameters.

Option a.

IMHO Option 1 suffers two drawbacks. While it may be feasible for one
special case, the all-node MC group to support IBARP, it doesn't address
the general multicast mapping well. IPoIB may use many multicast groups,
some are statically created, other are dynamically created/destroyed.

Another drawback is that it makes supporting 3a option 1 difficult. In order
to span an IP subnet over multiple IB subnets, each IP multicast group will
need a single GID to go across all the IB subnets.

>
>
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 7 Nov 2001 00:56:37 -0800 (PST)

Ellen writes:
> Preference for option a: use the SL from the all hosts multicast group. The
> reason this was preferred over the option to use the SL from the path record
> is that getting the SL from the path record requires the implementation to
> either cache the SL or go to the SM to get the path record. 
> 

	What does InfiniBand say about SL ?

        * SL is meant for QoS. SL is meant for load balancing. SL is
          meant for avoiding deadlocks.

        * It is a must that the SLs in a fabric are setup to avoid a
          deadlock. This is ensured by the subnet manager (SM).

        * There may be multiple SLs between any two endpoints but IB
          provides no mechanism to choose from the multiple SLs.

        * For cross IB subnet traffic the SLs are derived based on the
          GRH:TClass value as per the IB routing chapter with the
          TClass of 0 mapping to the best effort SL. 
		
        Thus there is an assumption that there is a QoS framework in
        place. The IB specification leaves the usage and mode of
        determination of the SL for the application to decide. The SL
        and list of SLs are available in the SA but it is upto the
        endnode's application to choose the 'right' value.

        IP is one such IB application and so IPoIB needs to define a
        set of rules on the choice of the SL. Furthermore it would be
        nice, some may say required, to have all the nodes agree on a
        common set of rules so that they come up with the same
        answers. 

	How about the following :

        The node is free to determine the SL to be used in any
        implementation specific way. A configuration parameter must be
        provided that when set implies that the all-IP nodes multicast
        GID SL is suitable to be used for all unicast traffic in the
        IPoIB subnet. Thus who value the benefits outlined by Ellen
        may set their subnets to use the common SL others can
        determine the SL by accessing the SA.

Vivek

-- 
Vivek Kashyap
IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)