Subject: [Ipoverib] Comments on IPoIB Connected Mode Connection Establishment

From: Dror Goldenberg <[email-masked]>
Date: Wed, 10 Aug 2005 13:33:18 +0300

Reviewing the ipoib-cm draft
(http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01372.html
<http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01372.html> )
I have three comments about the connection establishment of IPoIB-CM.
 
(1) Service ID Reserved Bits
 
Section 3.3. defines the Service-ID as having reserved bits which must be
transmitted
as zeroes and ignored on receive. 
It doesn't sound right to ignore those bits on the receive side. Ignoring
those bits on
receive essentially means that the IPoIB-CM has to listen on a range of
service IDs.
I believe that it's better to define a service ID without reserved bits such
that the 
passive side listens on a single service ID.
For example, service IDs defined for SDP, don't have any reserved bits.

(2) Active/Active Model Support
 
2.3 implies that it is desirable to have one connection between peers, i.e.
one QP
on each side. I believe that active-active connection establishment model is
more
appropriate here. Otherwise, there may be cases of simultaneous access that
will
end up having two connections between the same two nodes (one for each
direction).
The current Service ID assignment, makes it impossible to use the
active-active 
model. The main reason is that the service ID has to be identical at both
ends. In
other words, active-active requires that both REQ messages be used with the
same
Service ID. The current Service ID is different (it has the QP number in
it).
 
A model that enables active-active, for example, is using a Service ID which
is a derivative
of the PKey (no QP# in the Service ID). However, it will not support the
model of more than
one UD QP on the same HCA on the same PKey. This model is important.
 
Another model that enables active-active is to have both local and remote QP
numbers
in the service ID. However, it will require both ends to listen on many
Service IDs.
 
Another alternative is to stay with active-passive connection establishment,
but add
an interface through which one of the connection can be quiesced and closed.
In the 
case where you happened to open two connections between the hosts, then you 
can close one of them afterwards.

(3) Identification of Remote Peer in Passive Side
 
It is unclear to me how can the passive side tell which is the remote peer.
It must be 
able to tell that, so that it can use the other half of the connection to
send messages
back to the active side (instead of creating a new connection).
The identification of the peer is based on Link-Layer Address, which is
{GID,QPN}. The
CM REQ message only includes GID. Maybe the QPN (the UD QP) should be passed

through the REQ private data ?
 
-Dror
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Sun, 14 Aug 2005 23:41:28 -0700 (PDT)

> For example, service IDs defined for SDP, don't have any reserved bits.

Is it not possible to mask the bits marked as reserved (or unused)? The draft
requires that the reserved bits must be zeroes on send. If it is a limitation 
of the connection manager that it cannot mask the bits and will have to listen
to the 'range' then I agree that the service-ID will have to be specified 
received as zeroes instead of ignoring them.

> through the REQ private data ?

ok..thinking about the above points, how about:

1. Include the local QPN in the request
2. The receiver of CM REQ will have:
 	- local QPN (by way of ServiceID requested)
 	- remote QPN (in private data as above)
 	- remote GID (in REQ)
3. In case of the active-active connection the request from the numerically 
smaller MAC Address (QPN+GID or probably just the GID) is terminated.
The error code 'state connection' is a possibility here.

The active-active situation is known when it is determined that
there is an outstanding request to the remote peer (as determined by 
GID+QPN) while a request from the same is received on local ServiceID.

4. However, there might be a need to establish multiple connections
between the same two peers. Therefore, if a connection already exits
between the remote GID+QPN on the local ServiceID, it is up to the 
receiver to accept this connection. The implementation accepting
multiple connections and the one requesting multiple connections on
the same ServiceID must manage the load; it is beyond the scope of
this spec.

If two connection requests cross then case 3. above applies.

thoughts?

Vivek