Subject: RE: [ipoverib] reasons for not using LID ?

From: "Culley, Paul" <[email-masked]>
Date: Wed, 23 Jan 2002 08:35:43 -0600

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543
P.S. are we having fun yet?

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, January 22, 2002 5:15 PM
To: Michael Krause
Cc: [email-masked]
Subject: Re: [ipoverib] reasons for not using LID ?

>>Bill, Yaron,
>>
>>         There are two situations
>>                 - proxy-ARP for a node on another subnet
>>                 - span IP subnets across IB subnets
>>
>>1.      proxy-ARP for a node on another subnet works very easily.
>>         You return the LID of the local IB router and the GID of the
>>         remote IB node. Note this is possible with the current
>>         specification. If only the GID is included excluding the LID
>>         this is not possible since you can't lookup the remote node's
>>         GID with the SA. You need to include both. [ SA to SA
>>         coordination within an IB subnet isn't specified what of
>>         across the subnet. ]
>>
>>         Note that the ARP packet is fielded by the IPoIB device
>>         logically/physically associated with the IB router. The ARP
>>         packet itself does not travel beyond the IB subnet.
>
>Within the requester's IB subnet, there will exist at least one endnode
>that will respond on behalf of any request that targets an endnode that
>exists on another IB or IP subnet.   The responder will work with the
SA to
>
>determine the IB router; an IP router would respond on behalf of its
>responsibility "cloud" and it too would work with the SA to determine
the
>IB LRH values to use to establish appropriate communication between the
>requester and the IP router.  In both cases, the responder is
responsible
>for determining the appropriate LRH / GRH (where applicable) header
>parameters to establish the communication channel.
>

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?
<prc>
 Yes, one way or another the SA/SM must know at least which router port
the GID prefix needs to route to.  We may make it a requirement (to
simplify SA's) that the route cloud be able to deal with anything
arriving at any router port on the subnet; then a simple SA need only
route to a single router port.  More complex SA's should still be able
to chose an optimum port. 
<endprc>

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.
<prc>
By only looking up the path to the router, the SA can only report on
that path.  Since we expect to support SA lookups for the end to end
path, including the full router cloud, it will be possible to get good
information on the full path.  This is needed for backwards
compatibility with devices that don't understand anything but IB 1.x
interfaces.
<endprc>

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet
and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?
<prc>
At this time, there is no method defined for the SA to interact with the
routers, or for endnodes to get router information.  As stated before,
the current plan is to use the PathRecord query to report on the full
path, as is necessary for 1.x compatibility.  The returned path
information would include LID/SL for the appropriate router port.  When
the IB specs are updated to support routers, I expect the SA/router
interface will be included, requiring some additional support from SAs.
If you wish to submit a proposal for additional interfaces that support
the more traditional IP like methods (where endnodes must chose the
router ports for non-local work), you are free to do so.
<endprc>

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.
<prc> As long as we fully support the 1.x IB spec in the routed fabric,
the PathRecord query will work, so we have a solution.
<endprc>

Question 2: Is the LRH:LID always available for all UD communication ?
Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I
do
expect it to be available in all cases but want to be sure).
<prc> The values are in the headers for all UD packets; HCAs have an
interface for getting this information, switches should not ever need
this, routers are yet to be fully defined, and TCAs can provide the
header info if their designer's care about cross subnet support...
<endprc>

>>2.      For IP subnets to span IB subnets there is too much
>>         speculation and no justification. First of all we don't know
>>         why we need it. We don't know when it will be possible. We
>>         don't know if it will be possible. It is a complicated issue
-
>>         too many things have to be updated in IB architecutre.
>
>The justification is that vendors and customers believe it is the right
>thing to do and there is both technical and economic feasibility to
achieve
>
>this objective.  Unless there is strong consensus to not support this
type
>of functionality, the workgroup must take steps to enable it or at
least
>not preclude this from being implemented at some future date once the
IBTA
>has provided the appropriate specification updates.
>

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?
<prc>
It has been stated before, but here goes again:
1) the customers want it
2) We expect that IB routers are coming, probably about the time IPoIB
really gets going...
3) We think we have a workable, if not optimum, method of supporting the
routers in mind right now (1.x PathRecord support).  So you don't really
have to wait for anything new to complete the ARP RFC.
4) The alternative for customers (to IP across IB subnets) is to install
extra IP routing functionality across any IB router; it seems that this
would slow the adoption rate of either IB routers, or IPoIB...  as well
as cost more for the customers.
5) I personally (and some others as well) would like to avoid the NEED
to return to the ietf to fix up the IPoIB RFCs when routers do become
available.  That creates too tight a tie on the RFC process to releasing
useful IB routers.  And creates chicken/egg problems if every endnode
needs to be fixed as well. That does not preclude providing some
optimizations through new RFCs as IB fabrics grow and the need for those
optimizations becomes apparent.
<endprc>

Vivek

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Ashwin Kamath" <[email-masked]>
Date: Wed, 23 Jan 2002 13:49:46 -0600

It is possible that you will not find a valid path to a LID in the LRH of
an ARP using any given SL. You can ( and will, if any kind of traffic
management is done ) have different SL/LID (including LMC) combinations in
the paths to any given end node, such that any given LID will not have paths
for all possible SLs.

 The response path can use a different LID ( even within an IB subnet ) so
that the response reaches the same end node. When the ARP request is
generated, the requestor will not know about the existing paths to the end
node for Unicast packets. So it is bound to use a local LID which should be
used to send packets to the multicast group. The responding end node can use
the following lids in its response
 1. The LID from the LRH
    The problem with this is that a path to this LID may not exist for any
given SL (and possibly for all SLs, depending on how traffic management is
done). A path to a different LID on the same end-node/port might have to be
used for things to work.
 2. The LID from the ARP request
    The problem with this is that the requestor cannot know this value in
advance. It will use a lid but what value of LMC will it choose to guarantee
the existence of the path to the ARP responder.

 In the interop case where different implementations are working together,
the compliances on the responder must be stricter. The responder has to know
what fields to use and what fields to ignore, to guarantee proper operation.

 As is the case with ARPs today, even if an end node has a persistent ARP
cache entry, it is still bound to respond to ARP requests and it does not
make any assumptions like - the ARP requestor has a persistent ARP cache
entry/static tables.

 Question is, as a responder what should I be doing so that I can
interoperate with all implementations, because as a responder I would not
know how the requestor is implemented? The draft does not lay down any
compliance for this. Is the response gauranteed to work if I use the fields
in the ARP response, or is it gauranteed to work if I use the SA for
obtaining path information, or is it gauranteed to work in both cases.

-Ashwin.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Culley, Paul
Sent: Wednesday, January 23, 2002 8:36 AM
To: Vivek Kashyap; Michael Krause
Cc: [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Tuesday, January 22, 2002 5:15 PM
To: Michael Krause
Cc: [email-masked]
Subject: Re: [ipoverib] reasons for not using LID ?

SA to
the
responsible

-
achieve
type
least
IBTA

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 23 Jan 2002 14:44:28 -0800 (PST)

Paul,
        I was asking about the present status. Based on your answers
        it seems that proxy-ARP for a node on anotehr subnet won't
        work at present. THus such setups will have to wait for IB
        routing to be developed.

> the PathRecord query will work, so we have a solution.

I did not understand - unless you meant in the future you expect the
pathrecord query to work with the GID whether with local/remote
prefix.  Am I right if I say there is no solution at present?

> header info if their designer's care about cross subnet support...

So, this is not a universal interface. 

You say routers are not defined yet - will they include this information in the future.

Switches receive MADs which are sent over the UD mode - aren't they
going to receive the information? 

I didn't realise that TCA's are not bound by these rules. What about
multicasting then - are TCAs bound by the same rules as the HCAs?

This also springs a doubt - is it mandatory for the HCAs to provide
this interface?

> It has been stated before, but here goes again:

	To do what  ?

> optimizations becomes apparent.

I am sorry if you had to repeat these but this is the first time I
have seen a list of reasons.

Agree with 5, but wanted to do the extra dotting of i's only if
needed. Still not sure if IPoIB multi-IB subnets are needed -
basically need to understand what use could they be put to. However,
am dropping this line of discussion - wouldn't mind any 'customer' to
list their reasons for asking this function though.

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 24 Jan 2002 15:31:11 -0800 (PST)

Ashwin,
        That is a good point but connectivity is not guaranteed across
        the IB fabric even if you exchange the GIDs. You may enquire
        on a GID, received in a multicast (ARP/ND), but that does not
        mean there is *any* path back to the sender.

        This therefore translates into a requirement for IPoIB - the
        administrator must ensure connectivity within the partition.

        An IP interface essentially can be tied to any particular
        value - the GID (at certain level - port/HCA etc.), the port
        (GID or LID), the LID (no LMC).

        The leftmost in the list provides for most flexibility - can
        choose from many paths, even ports. The rightmost provides for
        a single path but certainty that all nodes (senders/receivers)
        come to the same conclusion.

        The choice of the link-descriptor, in general, need not be
        directly mapped to the IP interface's associations since the
        internal storage of an implementation can always be different.
        We have chosen to return the GID and the QPN :

	QPN+ Flags+ GID

        A GID resolution from the SA can return a multiple DLIDs, 1
        DLID or nothing. 
	
	If only 1 path : things are simple and fine.

        If nothing - we have the requirement for connectivity and a
        non-working connection for the administrator to debug.

        If multiple paths the choice is more difficult. The SA doesn't
        provide many hints - it gives the MTU, SL etc. For things to
        be effective we need the SA to provide or reflect a policy so
        that the sender and a multiple set of receivers reach the same
        conclusion.  

        At present there is no general way for the sender/receiver to
        agree on the paths to be followed back to the sender.
        Load-balancing will not work if all responders just choose the
        same DLID for example. It might work in a limited setup where
        different receivers in the subnet all can reach the sender
        only on a single path - this has to be ensured by the
        administrator. If there are multiple paths there is no way of
        ensuring that the receivers follow a particular path. That can
        be accomplished by letting the receiver know the exact LID to
        use either directly or setting the SA with only one path.

        Given the above I guess a way out - though not guaranteeing
        connectivity - is to say that the IB subnet will decide based
        on IB methods. These methods include - getting the LIDs from
        the SA, getting the LID from the LRH, getting the parameters
        from the SA, getting the parameters in some other way etc. The
        choice of the specific path based on these parameters is upto
        the implementations. They will leverage any IB solution
        available whenever it is.

        The other end is to state that the subnet has to be more
        tightly controlled - the administrator sets up the
        connectivity and every LID that may be used in the IPoIB
        subnet must have connectivity to every other node.

Vivek	

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Sun, 27 Jan 2002 08:14:26 -0800

>         mean there is *any* path back to the sender.

 From a partition perspective, if one receives an ARP packet, the endnodes 
must be in the same partition.  Ideally, SAs / router protocols, etc. would 
be set up such that if the endnodes participate in such a multicast group 
that there indeed exists a unicast path between the endnodes else why link 
them together in a multicast group in the first place.  I see nothing wrong 
with IP assuming that at least one unicast path must exist between any two 
endnodes that can see each other from an ARP perspective.

Mike