Subject: [Ipoverib] Re: ARP on IPoIB.

From: Ashwin Kamath <[email-masked]>
Date: Thu, 8 Nov 2001 13:59:46 -0500 (EST)

Vivek,
 Yes it is true that if the type field is introduced or the you multiplex
on MGID, you do not have to use ServiceID Based resolution. (By multiple
consumers, I meant multiple consumers on the same IB Node). I will try and
explain it below. Forgive me for wasting your time, if I am
all wrong. Also I might have left some parameters out since my intention
is to communicate the idea and not the details (There could be problems
with the details that I don't know about.)

Terminology
Layer-2 => Infiniband Link Layer
Layer-3 => Infiniband Network Layer
Layer-4 => Infiniband Transport Layer
Layer-5 => Application Specific Layer

My intention is to try and resolve the impact of using queue pair
0xffffff for ARP. In this context ARP resolves addressing between Layer-3 and Layer-2
protocols, for unicast Layer-3 addresses. I would try to do this without
modifying the current Infiniband specs. All my discussion is based on the
UD Service.

Problem Statement
There are some unique Layer-4 addresses which are used by multiple
consumers on a single IB Node. The use of these unique addresses by
multiple IB Consumers is resolved by introducing a Layer-5 protocol which
demuxes the messages.

QP 0 and Redirected Queue Pairs => Layer-5 protocol is handled and demuxed
by the SMI
QP 1 and Redirected Queue Pairs => Layer-5 protocol is handled and demuxed
by the GSI
QP 0xffffff => No Layer 5 protocols defined.

All other queue pairs are assigned to individual consumers by way of
Unique QPN and QKEY per consumer per QueuePair on a single IB Node (I
will call these queue pairs "consumer queue pairs" ). So no
Layer-5 based demuxing is required because the individual consumers are
responsible for driving the appropriate Layer-5 protocol over these queue
pairs.

The use of QP 0xffffff warrants a definition of a Layer-5 protocol (by way
of the Type field.). The current IB Spec verbs do not define a mechanism
to demux messages arriving at a Queue Pair to different
consumers based on the DGID in the GRH. Such a definition will have
problems because UDs are not required to carry a GRH.

The ARP request needs to be initiated in the absence of the knowledge of a  
consumer queue pair at the responder end. This is because a consumer queue pair defines its own layer-5
protocol. Since the initiator of the ARP request requires that a responder
queue pair exists, it needs to know the parameters of the responder queue
pair. The way to do this is to use the well defined queue pairs on the
responder. Hence a need arises to use either
GSI
SMI
QP 0xffffff

It would be ideal if the ARP requestor uses the existing infiniband
mechanisms to find out the responder queue pair parameters and at the same
time also resolve the addressing. The following discussion intends to do
this.

Discussion
The SMI is specific to Subnet Management and so it cannot be used.
Within the constraints of the current Infiniband Specs the QP 0xffffff
cannot be used because there can be multiple consumers on the responder
node which might be dequeueing from this queue pair. So the remaining
choice is the GSI. The GSI demuxes the Communication management messages.

The usage of Service ID resolution parallels the intention of
ARPs. Typically one would use a unicast lid in SIDR to determine if a
particular service exists on the destination target. One could also use a
multicast lid to determine if a particular service exists on a set of
nodes. Having done this, one can obtain the complete set of parameters 
that are required by the initiator to use the particular service
end-to-end.

A ServiceID can be assigned for an IPoIB service within the constraints of
the current specs. (Service IDs are being used for even higher layer
protcols like port numbers for SDP, and IPoIB seems to be more important
than SDP because SDP relies on IPoIB for its address resolution.) I could
be wrong about this but it seems to be much more of a problem to change
the IB Specs for ARP, especially since one can rely on already existing
mechanisms to achieve the purpose (although the changes to the IB Specs
could be imminent because they solve other problems too). 

Also, it is mandatory to have Communication Management agents on each of
the IB nodes. So the ARP requestor/responder can use the existing
mechanisms for Address resolution.

Solution
The following actions could be the steps required for any host wanting to
send out an ARP. This assumes that the node has already joined the
multicast group FF12::255.255.255.255 and has obtained the MLID.

All nodes intending to use IPoIB will also
Register with the Communication manager to receive all service requests
for Service ID 0x<assigned-service-id-for-ipoib>

To send an ARP one will do the following
Initiate a Service ID resolution Request at the Communication manager
LRH is set using the Group parameters for the service. (DLID=MLID)
GRH DGID=Multicast GID, SGID=source GID.
BTH is set up by the Communication manager.
SIDR RequestID is set by the communication manager
     Service ID = <assigned-service-id-for-ipoib>
     Private Data = Unicast GID which needs to be resolved + any extra
       data that needs to be sent.

Since the Service ID is unique to Ipoib only a registered ARP responder
will respond to it. This agent will know to interpret Private Data to be
the Unicast GID that needs to be resolved. The responder will check if the
Unicast GID is its own GID and respond. If the Unicast GID is not its own
it will drop the request (it should not respond at all.). Else it will
reply as follows
LRH is set using the Group parameters for the Service except (DLID=LID)
GRH DGID=(SGID From the request), SGID=Unicast GID which was requested in
    the SIDR.
BTH is setup by the Communication manager
SIDREP RequestID is set by the communication manager
       QPN, Status, ServiceID, QKey is setup by the Arp Responder using
    the consumer queue pair that IPoIB has created for its operation on
    the responder node.
PrivateData Any extra data that needs to be sent.

-Thanks,
Ashwin.

On Thu, 8 Nov 2001, Vivek Kashyap wrote:

> > 
> > Sorry I did not send this to a mailing list, couldn't find out which
> > one. If you think it is appropriate can you post it on my behalf, or you
> > could give me the information so that I can post it.
> > 
> > ARP mechanism requires that the ARP Request be broadcast to the broadcast
> > QP 0xffffffff. The problem with this is that there can be more than one
> > consumer using this queue pair, and it will be hard to demux the messages
> > to the different entities using it without a wire protocol over the IB
> > transport protocol, which can be used for the purpose. The problem is that
> > this wire protocol has not been defined yet.
> 
>         The packet is forwarded to the QPs based on the MGID
>         associated with them. In some cases it might be so that there
>         are duplicates to be made. However, if only the IP stack
>         associates with the group FF12:3334:<P_Key>::255.255.255.255
>         (I have updated the mapping) then only that QP will get the
>         packet. 
> 	
>         It is true that there could be multiple consumers associated
>         with the same multicast address but that is what multicasting
>         means. If there is only one multicast QP then it must demux
>         based on the MGID. By a wire protocol I suppose you mean
>         something like a type field in the IB packet that can let the
>         receiver know the consumer to forward the packet to. If you
>         are a member of IBTA you can join a discussion on this issue
>         in the link working group. If you have the type or you
>         multiplex on MGID (largely) you don't need a special serviceID
>         based resolution.
> 
> 
> >  I was toying around with the following mechanism that every host can
> > incorporate to participate in the IPoIB subnet. I am sorry if I wasted
> > your time by throwing in a flawed setup.
> > 
> > The following steps need to be taken for a host to join the IPoIB Subnet
> > (as in draft-kashyap-ipoib-ipv4-multicast-01.txt)
> > 1. Host cold starts.
> > 2. Host gets configured by the Subnet Manager(SM).
> > 3. Host uses SubnAdmGet (MCGroupRecord, FF12::255.255.255.255)
> > 4. If SubnAdmGet says that there is no such entry, host uses
> > 
> > SubnAdmSet(MCGroupRecord,FF12::255.255.255.255,QKEY,MTU,TClass,PKEY,FlowLabel,HopLimit)
> > 
> > 5. Host uses SubnAdmSet (MCMemberRecord, FF12::255.255.255.255) to join
> > the multicast group.
> > 
> > At this point, the Mulitcast LID (MLID) can be used by the host, the only
> > problem being that there is no receiving end for ARP.
> > An ARP can have a direct correspondence with the SIDR CommMgmt
> > message. Advantage is taken of the fact that for a GSI UD, the DLID is
> > compared with the lids configured for the link and an MLID is a lid
> > configured for the link.
> > 
> > Sending an ARP Request.
> > ( The fields indicated here are the ones that can be set by the request
> > generating entity. Fields not listed are generated by the Communication
> > Management entity in a way it sees fit.)
> > 
> > * LRH.DLID will be equal to MLID, and so the message will be received on
> > all nodes which are members of this multicast group.
> > * The SIDR Request must contain a GRH, where the DGID
> > will be FF12::255.255.255.255 and the SGID will contain the GID of the
> > requester. The other fields in the GRH will be set in according to the
> > fields indicated in the Group Membership information.
> > * The Service ID will be assigned by <somebody-who-does-it> and will be
> > the Service ID used for the IPV4 datagram service.
> > 
> > Responding to an ARP Request.
> > The serviceid in the arp request is demuxed to the ARP responder from the
> > Communication Management entity.
> > The responder must create a new UD queue pair if one does not exist, to
> > handle IP datagrams.
> > The responder must then generate a SIDR_REP if everything is ok.
> > * QPN is the Queuepair number of the UD Queuepair which handles IP
> > datagrams
> > * Status will contain the status of the request.
> > * QKey will contain the qkey used for the UD Queue pair.
> > 
> > The private data field can be used if any extra data needs to be sent with
> > this request. The LID of the target host can be obtained from the SLID of
> > the LRH.
> > 
> > One specific advantage of this mechanism is that the ARP request
> > (SIDR) can be forwarded by a router to another IB subnet, so an IPV4
> > subnet can encompass multiple IB Subnets. ( This advantage is specific to
> > the ARP request, I haven't thought about other problems that it might
> > create. )
> 
>         This can be done without SIDR too. The ARP request is received
>         by an IB node which answers for an IP node on another IB
>         subnet. 
> 
> 	
>         It might be that I have not understood your scheme correctly.
>         Please let me know if you think so. The ipoverib mailing list
>         is [email-masked].
> 
> Vivek
> 
> 
> 
> 	
> > 
> > 
> > -Thanks,
> > Ashwin.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 

-- 

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Ashwin Kamath <[email-masked]>
Date: Thu, 8 Nov 2001 23:08:54 -0500 (EST)

There have been a number of mistakes in my previous mail because I
 glossed over compliance o10-35. So SIDR cannot have a multicast
 destination, because QP1 is a real QP and not a multicast QP, and QPN
 for a multicast UD QP is not used in the BTH.DESTQP. I am very
 sorry for the deviation, am taking back my statements.
 
 -Ashwin

>  On Thu,8 Nov 2001, H.K. Jerry Chu wrote:
> 
> > I'm confused. GRH is required for any multicast packet. QP=0xffffff
> > implies multicast.
> 
> 
> > 
> > Jerry
> > 
> > 
> > 
> 
> 

--