Subject: [Ipoverib] comments on draft-kashyap-ipoib-connected-mode-02.txt

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 18 Nov 2004 11:27:41 -0800 (PST)

In the last IETF61 IPoIB meeting I made several comments on the
connected mode draft. I'm sending them to the list for a general
discussion. (Yes I saw some disucssion on the connected mode
draft already. I'll try to catch up with the thread after this
mail.)

1. The draft makes a distinction between IPoIB-CM interfaces
and IPoIB-UD interfaces, and portrays IPoIB-UC or IPoIB-RC as
separate subnets superimposed on top of an IPoIB-UD subnet.

For the above to work, due to a lack of multicast support, a fully
connected network by itself can't meet the requirement of an IP
link unless multicast is fully emulated through the use of
multiple unicasts. The latter is complex and cumbersome.

A much simpler model, which I think was presented in earlier
drafts, is to fold the use of IB connections fully into a
regular IPoIB-UD subnet, allowing any two IPoIB nodes to
optionally negotiate the use of IB connection between themselves.

This much simplified model is not without its drawback. Some
nice IP link attributes are no longer unique within a link.
E.g., the link MTU now becomes per-node-pair MTU. Moreover,
the MTU size for multicast will be different from the MTU size
for unicast if IB connections are used. IB UC/RC may exhibit
different RAS, flow control, QoS or other link characteristics
than UD. But I consider these problems a reasonable price to
pay for a seamless support of UC/RC mode in an IPoIB link
defined by UD.

2. The negotiation of the per-connection MTU seems more
complicated than necessary. I think all is needed is for a
node to advertise its own "receive MTU". That is, the MTU
size its peer should never go over when sending packets
to the local interface. Yes this may break the traditional
concept of "symmetric" MTUs. But we're already breaking the
notion of per-link MTU, requring a lot of changes in the host
stack anyway. This additonal breakage doesn't seem much.

I haven't verified if this asymmetric MTU matches well with
IBA connections though.

3. Regarding allowing multiple IB connections between a node
pair, since given an IP address there is only one link-address
for it implying one QPN, hence one service-ID, if a single
service-ID can be used to create multiple IB connections
then this can happen transparently. Otherwise we've got a
problem.

Jerry

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 18 Nov 2004 15:59:50 -0800 (PST)

> multiple unicasts. The latter is complex and cumbersome.

Exactly. The current draft also continues to use UD for 
multicast.

> optionally negotiate the use of IB connection between themselves.

The difference in the earlier draft and this one is that
I modified the requirement on the UD QP. That is, it need not be that
IPoIB-CM and IPoIB-UD share a QP but that any UD QP will do for IPoIB-CM. 
In effect an implementation can still share the UD QP.

The only issue is whether the same IP subnet can contain pure 
IPoIB-UD mixed in with IPoIB-CM nodes or, all nodes must be of the same type.
	- all IPoIB-UD
or
	- all IPoIB-RC

or 	-- all IPoIB-UC

I beleive all of the same type is a good option to choose.

> I haven't verified if this asymmetric MTU matches well with

How about:

The MTU I would think is exchanged at the IB level during the
IPoIB-CM connection setup. The IP layer at both ends keeps a per connection
MTU if the implementation permits it. At the link layer the connection will
not send messages larger than that requested by the peer.