Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sun, 17 Feb 2002 13:53:32 -0800 (PST)

Mike,

Thanks for reviewing the draft. First a general comment on your
points regarding IB partitions, multicast, link MTU..., etc. Those
topics really belong to the "IPoIB link and multicast" draft. Vivek
and I divided the IPoIB task into two drafts. The "link and
multicast" draft covers IPoB link setup, the configuration of
various link parameters, and multicast mapping. The "encap and addr
resolution" draft builds on that foundation to further cover unicast
and encapsulation to complete the task. The coverage of IB partitions,
link attributes and multicast in the following draft should be kept
to a minimum to avoid duplication.

So I'd like to invite you to use your comments on the "link and
multicast" draft to see it they are applicable there.

More responses below.

>>
>>         An IB subnet is formed by a network of IB nodes interconnected
>>         either directly or via IB switches. IB subnets may be
>>         connected using IB routers to form a fabric made of multiple
>>         IB subnets. Multiple IP subnets may be overlaid over this IB
>>         cloud. The boundary of this IP subnet is arbitrary and not
>>         associated with a physical demarcation. The IPoIB nodes that
>>         are members of this subnet are interconnected by an abstract
>>         'link'. The link is defined by its members and common
>>         characteristics such as the P_Key, link MTU and Q_Key that are
>>         defined per 'link'.
>
>It would be simpler and more accurate to state that:
>
>An IP subnet overlays an IB partition.  An IB partition defines a virtual 
>IB fabric instance which may be composed of any number of physical IB 
>endnodes, switches, and routers.  To implement the IP service, this virtual 
>IB fabric must also share a common set of characteristics: IB multicast 
>functionality, minimum IB PMTU capable of supporting the requisite IP 
>protocol suite, and a common Q_Key per IP subnet instance.
>
>>         IPv4 defines a limited-broadcast address over the link. All
>>         IPv4 hosts that are members of the IPv4 subnet are members of
>>         this address. IPv6 defines a multicast address referred to as
>>         the all-IP hosts address. IPoIB associates a multicast GID
>>         with these addresses[IPoIB_MCAST]. This multicast GID will
>>         henceforth be referred to as the broadcast-GID. The
>>         broadcast-GID is required to be setup for an IPoIB subnet to
>>         be formed.
>>
>>         Every IPoIB interface MUST join the broadcast-GID. This
>>         operation returns the MTU and the Q_Key associated with the
>>         IPoIB link. Thus the IPoIB subnet(and the link) is formed by
>>         the IPoIB nodes joining the broadcast GID.
>
>One does not need to join the IB multicast group to support IP 
>services.  As such, this should not be a MUST requirement.

This is made a requirement in order to better support IP multicast and
ARP broadcast. Again this requirement really belongs to the "link and
multicast" draft.

>
>>         The P_Key is a configuration parameter that must be known
>>         before the broadcast-GID can be formed[IPoIB_MCAST].
>>
>>2.1 IP Support on IPoIB Link
>>
>>         The unreliable datagram(UD) mode of communication is supported
>>         by all IB elements be they IB routers, HCAs or TCAs. In
>>         addition to being the only universal transmission method it
>>         supports multicasting, partitioning and a 32-bit CRC[IB_ARCH].
>>         Though multicasting support is optional in IB fabrics, IPoIB
>>         architecture requires the participating components to support
>>         it [IPoIB_MCAST].
>
>Technically, the IB specification only requires the IB partition and 
>associated multicast sub-tree to be such that any endnode can inject 
>multicast packets and may use the default multicast forwarding capability 
>within a switch such that one can forward the multicast packet to the 
>participating endnode.  While this may not be a common implementation, it 
>should not be precluded by this specification.  In general, this 
>specification should simply state that an IB subnet must be laid out from a 
>routing topology / multicast capability such that an endnode may 
>participate without requiring the endnode IP stack to be modified to 
>understand this the specifics of IB fabric capabilities.
>
>>         All IPoIB implementations MUST support IP over the unreliable
>>         datagram(UD) transport mode of IBA.
>
>Should specifications be written to support other IB transport types, there 
>is nothing to preclude an implementation from only supporting the discovery 
>services defined within this specification and only implement these other 
>transport services.  I do not see why this is a MUST or it even needs 
>mentioning in any specification.   These types of requirements are 
>unnecessary and should be removed.

We need to make some transport type to be mandatory for IPoIB in order
to ensure interoperability. In that regard UD seems a much better choice.

>
>>3.0 Maximum Transmission Unit
>>
>>         The IB architecture supports multiple MTU values: 256, 512,
>>         1024, 2048, 4096 bytes. An implementation determines the IPoIB
>>         link MTU from the MTU listed in the MCGroupRecord of the
>>         broadcast GID[IPoIB_MCAST]. The IPoIB link does not have a
>>         default MTU.
>
>In general, the whole MTU is made way too complicated.  Just state:
>
>- The IB MTU must be sufficiently large to support the minimum required for 
>the associated IP version being used.
>
>- The IB MTU is set by the IB management services and is derived from the 
>MCGroupRecord.
>
>- The IP MTU must not exceed the IB MTU.
>
>- The IP MTU may be manually configured or be derived from either the IB 
>MTU or through a PMTU update notification.
>
>The rest of this discussion is at best informational and as demonstrated on 
>the reflector seems to just cause confusion.  Nice, concise normative 
>statements would be appreciated with informative material moved to the 
>architecture overview document.
>
>

<snip>

>>         The InfiniBand specification requires the use of Global
>>         Routing Header(GRH)[IPoIB_ARCH] when multicasting or when an
>>         InfiniBand packet traverses from one IB subnet to another
>>         through an IB router. Its use is optional when used for
>>         unicast transmission between nodes within an IB subnet. The
>>         IPoIB implementation MUST be able to handle packets received
>>         with or without the use of GRH.
>>
>>         The IP/ARP datagrams SHALL be encapsulated in IB unreliable
>>         datagrams in the payload.
>
>Somewhat redundant as this was already stated as the purpose of this 
>specification.
>
>>The QPs advertised for IP
>>         communication MUST NOT be used for other protocols.
>
>It is unclear if this can be enforced or worthwhile to restrict so why 
>state it.  If used for anything else, this is a local implementation 
>problem to resolve.

Agree.

>
>
>>5.0 IPv6 Stateless Autoconfiguration
>>
>>         IB architecture associates an EUI-64 identifier termed the
>>         GUID (Globally Unique Identifier) [IPoIB_ARCH, IB_ARCH] with
>>         each port. The LID (16 bits) is unique within an IB subnet
>>         only.
>>
>>         The interface identifier may be chosen from:
>
>The interface identifier may be any of the EUI-64 (manufacturer or SM 
>assigned) for this port.  I see no reason to support the use of a LID in 
>this regard and view this as unnecessary options.

The main requirement for an interface identifier in IPv6 is its
"uniqueness". It looks like implementations can pick any scheme as
long as "uniqueness" is satisfied. Another requirement may be
"persistency" in order to better support the notion of client IDs
in DHCP.

The bottom line is, does it add more value to allow flexibility, or
does it add more confusion and complexity?

>
>
>
>>         The parameters that need to be determined are:
>>
>>         a) LID (local identifier)
>>
>>                 The LID is always needed. A packet always includes the
>>                 LRH that is targeted at the remote node's LID, or an
>>                 IB router's LID to get to the remote node in another
>>                 IB subnet.
>>
>>         b) GID (global identifier)
>>
>>                 The GID is not needed when exchanging information
>>                 within an IB subnet though it may be included in any
>>                 packet. It is an absolute necessity when transmitting
>>                 across the IB subnet since the IB routers use the GID
>>                 to correctly forward the packets. The source and
>>                 destination GIDs are fields included in the GRH.
>>
>>                 The GID, if formed using the GUID, can be used to
>>                 unambiguously identify an endpoint.
>
>There may be multiple GIDs per port.   Each GID, in essence, is required to 
>be unique within an IB partition which does not care about what GUID is 
>used to create this GID.
>
>Most of these IB parameters are defined within the IB spec and do not need 
>to be repeated here.  Recommend they be removed from the specification.
>
>
>>         f) SL (service level)
>>
>>                 Every IBA packet contains an SL value. A path in IBA
>>                 is defined by the three-tuple (source LID, destination
>>                 LID, SL). The SL in turns is mapped to a virtual
>>                 lane(VL) at every xCA, switch that sends/forwards the
>>                 packet [IPoIB_ARCH]. Multiple SLs may be used between
>>                 two endpoints to provide for load-balancing, SLs may
>>                 be used for providing a QoS infrastructure, or may be
>>                 used to avoid deadlocks in the IBA fabric.
>
>SL is subnet local and has nothing to do with the definition of a path. A 
>path within a subnet is defined by the LID.  The SL is used to select the 
>VL and thus may be used to prevent deadlock via a loop but it does not 
>define the path.
>
>
>>         b) Queue Pair Number (QPN)
>>
>>                 Every unicast communication in IB architecture is
>>                 directed to a specific queue pair(QP)[IB_ARCH]. This
>>                 QP number is included in the link description. All IP
>>                 communication to the relevant IPoIB interface MUST be
>>                 directed to this QPN. In the case of IPv4 subnets the
>>                 address resolution protocol(ARP) reply packets are
>>                 also directed to the same QPN.
>>
>>                 The choice of the QPN for IP/ARP communication is up
>>                 to the implementation.
>>
>>         c) Global Identifier (GID)
>>
>>                 This is one of the Global Identifiers(GIDs)[IB_ARCH]
>>                 of the port associated with the IPoIB interface. IB
>>                 associates multiple GIDs with a port. It is
>>                 RECOMMENDED that the GID formed by the combination of
>>                 the IB subnet prefix and the port's GUID be included
>>                 in the link-layer/hardware address.
>
>This recommendation is unnecessary.  A local implementation selects the GID 
>and what policy it uses is outside the scope of the specification.

We should add a contraint in the draft something like "Local node must
choose a GID with a scope at least as large as the scope of the IPoIB
link it belongs" (covered in the "link and multicast" draft).
E.g. if a port belongs to an IPoIB link that spans more than one IB subnet,
one can not use a link-local GID to construct one's IPoIB link layer addr.

Jerry

>
>This section should just list all of the IB parameters and the source of 
>the value and nothing more.
>
>
>>         e) Path rate
>>
>>                 The implementation must leverage IB methods to
>>                 determine the path rate as required.
>
>While static rate is defined today, it is no the only method to be used in 
>the future.  All that should be stated is that the implementation should 
>not exceed a given bandwidth value to reach a specific endnode thus 
>allowing one to build an implicit bandwidth management solution without 
>requiring a change within the IP over IB spec / implementation.
>
>Mike
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib