Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

From: "Ashwin Kamath" <[email-masked]>
Date: Wed, 16 Jan 2002 08:50:54 -0600

Vivek, 
 This is a pure response to the statements below, not to be regarded as an opinion on inclusion/exclusion of the LID in ARPs.

 a. You asked for a counter example for simplicity of the including the LID. There is no example provided to counter on a simple way to use SLs and pathrates. The one example given is that there can be static tables, but the simplicity on administering these tables depends on the complexity of the IB Fabric topology. Why is an SA lookup more difficult? It would be if there were no SA and in many cases that could be a valid reason today.
 b. Another reason is to save the round trip for SA lookup. A round trip needs to be saved if the latency is significant to override the benefits. Current implentations have the following latencies for any given ARP and its response
  * 4 PCI burst transactions ( + some normal transactions )
  * 2 trips on Ethernet links + switches
 ARP with SA lookup will have (assuming HCA proximity with Memory and CPU, DMAing data will be atleast an order of magnitude smaller than PCI transactions and thus negligible)
  * 4 trips on Infiniband links + switches
 One can factor in the bus rates + switch latencies, and see that latency of an extra Infiniband round trip is not an issue. If there is a problem with the assumption of proximity, one will have to add 8 PCI burst transactions to the above math. I might be wrong about this but ARP latencies were never an issue even on 100Mbps Ethernet links and you will still come up with a number that is better than atleast 100Mbps Ethernet links with the above math.
 c. Data load on SAs. This has already been discussed. Data load on ethernet controllers today can add significant latencies in the ARP round trip, because of backing up of hardware queues at the end nodes. This fact was never used to promote or demote usage of static tables. 

 It is quite okay to have an architecture which can potentially use static tables, but in this case the justification for this choice may not be Simplicity, latency or data load because these parameters are relative to an implementation. Flexibility is the only justification albiet a very important one.

-Ashwin.

---------- Original Message ----------------------------------
From: Vivek Kashyap <[email-masked]>
Date:  Tue, 15 Jan 2002 23:10:15 -0800 (PST)

>
>> 
>> Congestion management would be optional functionality given the V1.0 
>> specification has been public for over a year now.  As such, it cannot be 
>> relied upon to solve any problems only to improve a solution over 
>> time.  Further discussion into the details is not permitted due to the IBTA 
>> confidentiality rules but it is fair to state that injection rate control 
>> is one lever used by congestion management but congestion management does 
>> not lead to injection rate control which may be driven more by QoS 
>> requirements which go beyond the scope of congestion management.
>> 
>> The IP over IB spec should define the architectural minimums needed to 
>> implement a credible IP over IB solution and leave optimizations such as 
>> what have been debated these last few days to vendors to solve.  If the 
>> optimizations prove worthwhile after experience has been gained in real 
>> customer deployment situations such that a subsequent spec should be 
>> generated, then the workgroup can provide a new spec that provides an 
>> interoperable solution using these new techniques.
>
>        The only issue I am pointing out is that there are multiple
>        ways to solve the problem - as you yourself point out. The
>        architectural minimum should not be defined such that it ties
>        down every body to the minimum leaving the optimisations out
>        of scope in the future.
>
>        Therefore include LID/GID in the address resolution packets
>        and all sorts of implementations can coexist. Remove GID and
>        only one set is possible, remove LID and only another set is
>        possible. Include both, both are possible and there are no
>        negatives for either.
>
>> 
>> > > And what's the real benefit of saving just a lookup out of an already
>> > > infrequent operation like ARP due to cacheing?
>> >
>> >         I would like to know what is the drawback that you see that
>> >         you don't want one to go ahead with it? ARP is repeated every
>> >         so often and happens quite often. It also happens on failover
>> >         - it is preferable that these operations be simple, fast and
>> >         error-free.
>> 
>> No one is arguing with any of these assertions.   The issue is strictly 
>> what is the architectural minimum the spec should define and what should be 
>> left for future spec or vendor implementation to solve.
>> 
>> 
>> >         Secondly, things are more complicated the way you propose.
>> >         Note that the first level address would be first determined.
>> >         This completes ARP/ND. However, address resolution is not
>> >         complete. The packet is sent as far as IP is concerned and now
>> >         the SA/SM is consulted to get the LID etc. This has very
>> >         strong shades of ATM VCs. IP thought it had sent the packet
>> >         since the ARP/ND was complete - it took a while before the
>> >         lower layer completed the determination. The driver of course
>> >         keeps a GID to LID table too since the LID is a separate
>> >         entity. More complication.
>> 
>> This is one implementation choice and while it may be used by some, it does 
>> not require the spec to make this a requirement.
>
>        The assertion against LIDs was that they cause complication. I
>        am showing that it is the other way down. This description is
>        not part of the spec. but shows that the argument that
>        inclusion of LIDs causes more complexity is incorrect. It is
>        the GID to LID conversion that is more complicated. Including
>        LIDs in address resolution allows for a simple, 2 message,
>        fast failover solution.
>
>        I have also noted that the process, in another mail, of
>        resolving the IP address is simpler and fits in more with a
>        'single method' when LIDs are included than does the process
>        when the LID is not. I have not seen a counter example.
>
>
>> 
>> 
>> >         Include LID in the resolution - fast/simple/error-free.
>> 
>> It isn't that simple either
>
>        As shown above it is simpler than getting the GID and the
>        answer. I request a corresponding scenario that shows that the
>        GID to LID conversion is a better solution. I am not against
>        anyone implementing it differently. Allow those that want ot
>        implenet it in this way to do so.
>
>
>> 
>> >         I was pointing out another issue that many have raised.
>> >         In general I am ok with dropping GID since it is for the
>> >         future. IT buys us nothing as of now. So if we dropped GID we
>> >         can work with these 48 bits. When IPoIB in reality starts
>> >         spanning multiple IPoIB subnets (not for a long time) we can
>> >         include GID. Then the hardware length will become 22 bytes as
>> >         against 6 bytes. One can still interoperate quite easily.
>> 
>> GID is required because it is the primary way and application determines 
>> where a service resides on a given port as such not including it makes life 
>> quite difficult to implement.  GIDs are also used by applications within 
>> the address vector to identify a given vector to use for a given service 
>> such as IP.  GIDs cannot be dropped and have value.
>> 
>> LIDs are strictly a local issue that will be learned for all CA / router 
>> ports participating within a given partition that support IP services.  As 
>> such, their assignment, management, transitory nature, etc. make them 
>> unsuited as an architectural minimum to include within the spec.  I see no 
>> problem with people ascertaining the value and using it for optimizing 
>> their implementation but this is outside the scope of the specification.
>
>        I don't understand. If those that don't want this information
>        just ignore it what is the loss? We can format the link
>        description such that the LID is the last element. Those that
>        don't want it can drop this piece of information and ignore
>        it. Of course those that don't want to use GID can ignore it
>        too. 
>
>> 
>> >         Wouldn't that work for all ?
>> >
>> >         I included the GID because many want to put in the hooks for
>> >         the future case of IPoIB spanning IB subnets. I would though
>> >         like a valid reason specified for why do we need IP subnets to
>> >         span IB subnets. Isn't IP subnetting enough?
>> 
>> IB is viewed as a layer 2 fabric w.r.t. IP so the fact that IB has multiple 
>> subnets is independent of IP operation.  I see no point in dealing with the 
>> IB subnet issues within this specification other than to state that a LID 
>> is not an end-to-end address between two endnode ports and thus should not 
>> be mandated within the IP over IB spec for address resolution.
>> 
>> Mike
>> 
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>http://www.ietf.org/mailman/listinfo/ipoverib
>

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 16 Jan 2002 10:28:38 -0800 (PST)

Ashwin,
Ashwin,

        I agree that you get more flexibility if you include the LID.
        If we include the GID along with the LID we allow all
        implementations the choice to create different solutions. We
        don't tie them down to one thing. Choosing one or the other
        (LID or GID) blocks out various advantages that different
        implementations seek. Hence I suggested that let us include
        both and move on.

Vivek