Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 14:12:07 -0800 (PST)

Jerry,

> >As for the why LID:
> >I think that going to the SM is not so efficient given the fact that it is a
> >central resource and all the nodes in the subnet go to if we could avoid it
> >why not
> 
> First the benefit is dubious. In modern TCP/IP stacks the frequency of ARP
> requests is capped by the ARP cache flush internal, which is in minutes
> range. (E.g. in Solaris it's five minutes.) Assuming you have 100 nodes
> on a subnet, you're telling me saving the SM lookup call, which at max
> rate (if a local node is activetly talking to every other node) of one
> every 3 sec. is worth noting?

        What if there are 500/1000 or 3000 nodes ? Besides, as I have
        noted in the last mail the process is more complicated not
        only internally but also requires 6 messages where only 2
        would have sufficed. More complication means more maintenance,
        slower failover, more bugs etc. We also know that centralised
        systems don't scale well. That is why we are using all-nodes
        multicast and reply rather than go to some central ARP
        server. 

> 
> I'm still not sure if a trip to SM can be saved at all (need to catch
> up with all the mail this morning). Even if in some cases it could, do
> we believe that not doing any pathrecord lookup is going to be the norm 
> of IPoIB implementation that warrents an extr field in the link addr?
> How about all the issues I've pointed out before? I have another one
> to add - simplicility. Unless there is sufficient justification, the more
> fields you add to a protocol the more complexity (in areas you may not
> even forsee now) it's going to add.

        See my note I sent you describing the process. Not including
        the LID is more complicated. Including it allows you to be
        complicated or simple as you wish. Using the LID your ARP
        module can be coded in a common way - as explained in the
        previous mail - without it you need two disjoint proceses with
        timeouts, multiple messages interspersed within the whole
        process. 

> 
> >When IP will span IB the problem becomes more serious: Unlike IP routers
> >which has a default route entry, currently IB has explicit entries for each
> >path this might require doing too many PathRecord operations, which are also
> >currently very slow when going outside an IB subnet
> >(I had posted a proposal to the MSO team trying to resolve those issues
> >somehow however it is under discussion)
> >I would tend to send this info anyway since the extra 16 bits wouldn't harm
> >anyone,
> 
> But it does. What would you define the LID for a node on a non-local IB
> subnet? It can't be a LID to a router because this is an endnode's "MAC
> addr" we're defining here.

	See my previous mail.

> 
> >one might choose not to use them if he wants to go to the SM, this
> >way everybody will be happy and over time people will use the best solution
> 
> No we can't do handwaving here. You either define a field with clear
> semantics so implementations have a chance to interoperate, or take it
> out. There is no iffy here (unless you add a "LID included" flag to the
> link addr).
> 
> >
> >As for the Ethertype:
> >After giving Vivek a hard time on this issue I accept his proposal
> >Not because of the performance issues raised (since I don't think they are
> >real)
> >But because it is not really needed, and if it simplifies things, why not
> >
> >As for the capability bits:
> >Avoiding the SIDR is a smart move
> >I'm not sure if the UC option will be used, if using connection why not
> >reliable connection
> >I just wanted to add SDP (Socket Direct Protocol) bit, Again ones might
> >choose not to use it, for the one that do it really simplifies the process a
> >lot
> >
> >I'm hoping we get a quick convergence on those issues so we would move
> >forward
> >The issues are not serious and I think if an alternative enable someone to
> >do something he wants without violating the other's ability to do what they
> >want just ignoring the extra info, we should go for it
> >
> >Yaron
> >
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]]On Behalf Of
> >Vivek Kashyap
> >Sent: Tuesday, January 15, 2002 12:46 AM
> >To: [email-masked]
> >Cc: [email-masked]
> >Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?
> >
> >Folks,
> >
> >       It is close to the last call. In this regard I would like to
> >       state that we have been discussing this issue for a long time
> >       now. The most recent discussion is what I presented at the SLC
> >       last december. (I certainly felt that the room was in agreement
> >       - no queries, objection etc.)
> >
> >       I believe the following link-description provides the best
> >       assimilation of all the choices that have been put forth. If we
> >       step back and restart the whole discussion again we are likely
> >       to come back to the following - so here is a brief compilation
> >       of issues.
> >
> >
> >        Suggested Link description to be used in address resolution:
> >
> >                LID + capability flags + GID + QPN
> >
> >
> >
> >       Why the LID?
> >
> >             People said they don't want to go to the SM if possible
> >             (LID allows for that. As noted here and LWG the SL can be
> >             determined based on QoS or other parameters. The SL may
> >             also be determined from the SA/SM)
> >
> >             People said they want the process to end in small number
> >             of messages (LID allows a query/response to settle the
> >             issue. As noted one doesn't necessarily have to go to the
> >             SM to get the SL).
> >
> >             LID is always needed. (you can't escape it).
> >
> >             BTW, people said they want the 'link-address' to be 48 bits.
> >             (LID + QPN + cap_flags is 48 bits).
> >
> >
> >       Why the GID ?
> >
> >                People want IP to span across IB subnets. We are
> >                stretching stretching here since such implementations
> >                are not possible yet.
> >                However, we can define it such that it will work in
> >                the future. Therefore include the GID.
> >
> >        GUID
> >
> >                Can be made part of the GID.
> >
> >
> >        Why the QPN:
> >
> >                1) There is only one other way to get the QPN. That is
> >                the SIDR protocol. However, this has a drawback.
> >
> >                SIDR is NOT REQUIRED by IBTA.
> >
> >                I had raised this point last August 'Feasibility of
> >                SIDR':
> >
> >                ***** Section 12.11 essentially states that SIDR is
> >                optional. As the specification states:
> >
> >               << GSAs shall support this protocol if non-management
> >               services are pro-vided on the Channel Adapter at other
> >               than fixed QPNs. If this protocol is *not supported*,
> >               the status "The method/attribute combination is not
> >               sup-ported" shall be returned, as described in section
> >               Table 101 MAD Common Status Field Bit Values >>
> >
> >                *****
> >
> >                That still holds. SIDR is NOT required by IBTA. A
> >                device can always choose to provide all non-management
> >                services at fixed QPNs. It may just choose to provide
> >                IP and nothing else at a particular QP and hence won't
> >                do SIDR.
> >
> >                That is why I propose that the QPN be always included
> >                for the default case. SIDR usage is gated through the
> >                QPN flag and is in the realm of the advanced
> >                capabilities draft as has been suggested by many on
> >                this mailing list. And I agree.
> >
> >
> >                2) Address resolution that returns the QPN is done in
> >                one request/reply exchange. Any other method, while
> >                talking to the same endpoint, requires 6 messages. 2
> >                for ARP/ND and 2 each for each of the lower layer
> >                request/reply. IT is wasteful, more complicated,
> >                more time consuming wrt fast failover etc.
> >
> >
> >        Capability flags :
> >
> >                Multiple people have opined that they would prefer in
> >                certain cases to map the protocol/ULP endpoints to
> >                specific QPs. People also want to use IP over RC mode
> >                (and others maybe) due to the advantages of larger MTU
> >                and Automatic path failover etc.
> >
> >                The capability flags allow for interoperation between
> >                the default/dumb devices and the more sophisticated
> >                devices. IT also allows the endpoints to avoid
> >                unnecessary trial and error/to and fro when trying to
> >                utilise these features. The flag indicates the
> >                advanced capabilities supported and the receiver can
> >                choose to utilies them.
> >
> >
> >Result:
> >
> >        We cannot depend on SIDR. We need to include the default QPN.
> >        We should not force implementations to go to the SA/SM (IBTA
> >        does not) and so allow LID/GID to be determined from the
> >        remote host. The default case works with :
> >
> >        LID+Capability flags(all set to 0)+QPN+GID
> >
> >
> >Vivek
> >
> >
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email-masked]
> >http://www.ietf.org/mailman/listinfo/ipoverib
> >
> 
> 
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/ipoverib
> 

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)
-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 15:29:17 -0800

At 02:12 PM 1/15/2002 -0800, Vivek Kashyap wrote:
>         What if there are 500/1000 or 3000 nodes ?

Let's see, we are all complaining about an operation that may occur some N 
operations unit of time where N may be somewhat large but the end result is 
still something measured in 10's if not 100's of milliseconds in terms of 
interval.  Given the type of platforms delivered in such a time frame, it 
seems inconceivable that such a workload would impose any performance hit 
on either the fabric or the endnode.

>         Besides, as I have

I don't agree.  It is an implementation issue as to whether one implements 
the additional messages (which may already be required for other purposes 
between endnodes such that there is no real net savings).   One can also 
not flush the ARP cache so often given the relative stability of the IB 
fabric elements thus making the workload potentially much larger.

As for whether something central scales well or not that is really a 
function of the type of service being delivered.  A very complicated IB 
fabric will use various techniques to simplify and create scalability for 
management operations.  I can easily see a value-add management service 
that replicates the SA database across a set of nodes thus allowing the 
workload to be horizontally scaled.  One can create various policies as to 
how this data is accessed (there is nothing that precludes a solution 
provider from delivering the data through a multicast-based request / 
unicast response given these are primarily on a LID-based communications QP 
which would be available given the requirement to have IP run over a 
multicast switch environment).

Mike