Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 15 Jan 2002 00:12:16 -0800

Greetings Joe,

      A couple of clarifications are in order.

      The idea of avoiding the SM is a very valid one - I wonder what all
this SM/SA will do ? Everything on the CA will go over the same QPs(0/1)
and will go over VL15 (which doesn't have congestion control, looses
packets etc.), will go the SM/SA at its QP0/1 in a large SM. The SM will be
consulted for path records, for service id resolutions, for this and that
and that also. It seems a bit overmuch. That is an aside anyway for one to
think about.

      There are other reasons for avoiding the SM/SA that I  have mentioned
previously. However, nowhere has it been suggested that SM/SA cannot be
consulted. The counter-suggestion from you though says that one must choose
to go to  SM/SA - that is something that IBTA itself doesn't agree with.
Those that want to consult the SM/SA can continue to do so but those that
don't want to because they certainly see advantages in avoiding SM/SA when
possible should be allowed to do so; exactly as IBTA does.

      Regarding your comments on SL - I am a member of IBTA. IBTA rules do
not allow to one to give out the statements and the voting results  to
non-IBTA forums, so I'll only state that IBTA did not change the
specification - it is available to anyone on www.infinibandta.org  and did
not modify the rules regarding SL determination as Mike Krause in the mail
before yours has clarified. Mike is also a member(chair) of LWG.  Others
who are members of the IBTA can verify for themselves the final decision of
LWG. It did not preclude determining SL from sources other than the SA.

      You are right in stating that protection from a random SL is required
but no where is it being suggested that the SL used by the endnodes will be
random one. It will be a well thought of one and a correct one except it
will be determined without consulting the SA/SM. Nothing in IBTA precludes
it. Rather IBA specification promotes it.  BTW, the use SL from multicast
or determination of it  in any other way has been left to the choice of the
implementor. Not part of IPoIB architecture. Also, again, IBTA does not
specify a particular method either.

      All nodes if working greater than 1x are required to support static
rate iff any destination in the fabric works at < the node's rate.  It
merely says that the values needs to be determined but does not make the
method for determining it mandatory. Consulting the SA to get the path
record is one of the methods. Additionally IBTA is defining a congestion
control mechanism disjoint from injection control.

      An example:  do you mean to say that if I know that my subnet is made
of 4x through and through I need to consult the SA to get the information
always ? I don't think so.

      The suggestion here that some implementations will avoid the SA/SM in
a particular subnet if they think it is right in no way precludes
consulting the SA/SM or any other entity when required. That is not the
contention at all.  It is not being mandated on everyone. Similarly there
is no need to require every implementation to throw a look toward the SA/SM
for SL/path rate or any other information.

Vivek

--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Joe Pelissier                                                                                               
                      <[email-masked]>        To:       [email-masked]                                                        
                      Sent by:                 cc:                                                                                
                      ipoverib-admin@ie        Subject:  Re: [Ipoverib] What's inside an IPoIB link layer address?                
                      tf.org                                                                                                      

                      01/14/2002 08:35                                                                                            
                      PM                                                                                                          

Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to
whether ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it
just to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple
paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not,
deadlock of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.
The IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately,
the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should
not be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current
IBTA
spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

Michael Krause wrote:

> At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
> >There is a general thought that ARP is a one step solution for obtaining
the
> >parameters required for establishing IPoIB over UD. But I think that
even
> >for using ARP, one will require to get the Path record information. I do
not
> >think that ARP is a one stop solution. What happens if both nodes
determine
> >that their MTU is 2K without considering that a switch component in
between
> >might restrict the MTU to 256 bytes?
>
> The minimum supported MTU for UD traffic is 2KB so a path with a 256
bytes
> MTU should not be used by the IP over IB service.  The SM working with
the
> partition manager will establish a set of paths that can be used for IP
> over IB traffic which will include the setting up of an all nodes
multicast
> group per partition that support IP traffic.
>
> >Such information can be obtained only
> >from the central SM via a path record. Am I missing something here? If
> >address resolution is all that ARP is required to do, without trying to
> >establish parameters for communication one may not want to know the MTU,
but
> >from the discussions I see that the Address resolution is being used for
> >establishing communication parameters (like the QPN)
>
> QP identification is an optimization that allows one to eliminate the
> Ethertype field.  It also allows an implementation to map different IP
> addresses to different QP thereby providing an implementation
optimization
> through more fine grain hardware-based de-multiplexing.  An example use
> might be to assign an IP address per OS instance which shares a HCA port
> and then allow all traffic to target a corresponding QP without having to
> flow through an intermediate hypervisor that would have to de-multiplex
the
> traffic.  This can provide both performance and scalability improvements
> for vendor offerings.
>
> >So the only real problem with having the LID in the ARP response, that I
see
> >is when the ARP response needs to cross IB Routers. Can I assume that
> >changes in LID assignments are not a big problem because such changes
are
> >not intended to be frequent and current ARP cache implementations
already
> >factor in the possibility of a link level address change. Changing LIDs
> >would have a bigger impact on shared devices using RC than on IPoIB
using
> >UD. Is it ok if I consider a change in LID assignment to be equivalent
of a
> >server reboot?
>
> No.  Applications should not be impacted by a change in LID assignment as
> there are defined methods to migrate applications from one LID or GID or
> port to another that are completely transparent to the application.  For
> UD-based services, one could simply re-ARP the endnode via the all-nodes
> multicast and provide a simple update of the LID mapping as this can be
> derived from the IB headers if desired.
>
> >If I understood what Jerry, one will have a problem if the ARP responder
> >lies in another IB Subnet, in which case the LID in the response will
not be
> >relavant to the originator of the ARP request, in-fact the LID will have
to
> >be the LID of the router. Was there any resolution reached for this
problem?
> >
> >In short you do need an ARP + Path record diagnosis to determine all the
> >parameters for communication. What would happen if we bring an IB router
in
> >between? One cannot obtain a path record to a node in another subnet.
One
> >can obtain the path record to the IB Router. Should we be talking about
the
> >behaviour of the IB router in case of mismatch of MTUs or should we say
that
> >we will always use the smallest mtu for communicating across an IB
router.
> >Or should we say that this could fall on the plate of the Infiniband
WGs.
>
> The IB router protocol will define how one routes across IB subnets - an
> IBTA issue.  An IP router may be implemented along side an IB router or
it
> may be an edge device such as a multi-protocol router.  The fact that IP
> subnets may span multiple IB subnets isn't really that big of an issue
per
> se.  By keeping the LID not part of the link address structure, the ARP
> should be able to transparently flow across multiple IB subnets.
>
> Mike
>
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Swortwood" <[email-masked]>
Date: Tue, 15 Jan 2002 09:41:50 -0800

Greetings IPoverIB subscribers -

A few conceptual clarifications. There have been a number of
misnomers here, not central to the main discussion, but these do
muddy the waters of understanding. These are marked with [whs] below:

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Vivek Kashyap
Sent: Tuesday, January 15, 2002 12:12 AM
To: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Greetings Joe,

this SM/SA will do ?

[whs] Set up the topology of the IB subnet, keep the subnet topology intact,
collect and disperse subnet event information, set up partitions within the
subnet (part of topology), provide path information within the subnet, act
as the subnet multicast setup and join agency, and act as a low level
service
registry.

Everything on the CA will go over the same QPs(0/1)
and will go over VL15 (which doesn't have congestion control, looses
packets etc.), will go the SM/SA at its QP0/1 in a large SM.

[whs] Only SMPs (Subnet Management Packets) go on VL15. GMPS (General
Management Packets) go on any other assigned VL that is not VL15. The
SM communications are those that set up and keep the topology within
a subnet intact, and are the controlled domain of the SM, for the most
part. GMP communications are used by entities wanting information --
like topology or path record information, from the SA. The SA can also
redirect its communications to any other QP/Port within the subnet, so
communications can be balanced or serviced at other locales within the
subnet. Point here - it is NOT all VL15, and NOT all QP0/1. Further,
SL and static rate information are specific to the path in question.
--

The SM will be

[whs] Suggest you chat with Dr. Pfister [IBM] on this. PathRecords need to
come
from somewhere. Paths can change. SLs and static rate are particular to a
Path.

Service records
exist at the SA as it is the only well known address at the time any node
comes
up on an IB subnet. Note: if you hang a name service or directory service
as a service record with the SA, and that service should also contain all
information (like SL and static rate data), you would only have to
communicate
within the subnet to the SA when you need to establish a communication link
to that service.
--

[whs] I agree with you on keeping your SA communications to a minimum. Joe's
point is that the SM/SA is the origination point of path data. Some other
service can obtain that data from the SM/SA and outwardly provide it. I
am not, and would not think that Joe would be (correct me if I am wrong)
adverse to someone postulating such a service. But as things stand today
with the 1.0a spec, the SA now is the only defined interface.
--

---[snip]---

regards all -

-------------------------------
Bill Swortwood
(Office)(503) 690 8578
(CELL)  (503) 819-5281
--------------------------------

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 10:15:30 -0800 (PST)

> SL and static rate information are specific to the path in question.

        OK so there is some respite. However, how many paths lead to
        SA/SM in a subnet ? It still stands to reason if one wants to
        avoid it and can avoid it why force them ? 

        As noted earlier, SLs need not come from the SA. They of
        course will be recorded there in the path record but that does
        not imply that they can't be determined differently e.g. QoS
        mappings. 

        Similarly the path rate needs to be determined and will be
        recorded at the SA/SM. That does not mean that one has to go
        to the SA at the time of the address resolution to determine
        it. 
	
        I am arguing against the notion that because SL or path rate
        need to be known the LID need not be included in the address
        resolution. I don't see why even though based on your
        communication. 

> from somewhere. Paths can change. SLs and static rate are particular to a

        Please re-read what I have said. No where do I say that path
        records won't be recorded at the SA. Nor is it suggested that
        you preclude it. The discussion is different. It is not
        correct to use the argument that the path record is at the
        SA/SM and hence we need to get the LID from there.

        Inclusion of LID is an IP requirement. It needs to be
        determined at every address resolution. SL/path rate can be
        static in a subnet, can be determined differently including
        indirectly/directly from the SA.

> within the subnet to the SA when you need to establish a communication link

        SA is the repository of the information. How you get it, when
        you get it is the implementations choice. It is not tied to
        address resolution. So it is not an argument for getting the
        LID from the SA at all times.

Vivek

503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 11:20:00 -0800

>         avoid it and can avoid it why force them ?

Interoperability by providing only one method to obtain a particular data 
object.  This does not preclude additional methods from being used but the 
spec will define one method guaranteed to always work for all.

>         not imply that they can't be determined differently e.g. QoS

This would be an implementation issue as all implementations can rely upon 
the SA for this information.

>         to the SA at the time of the address resolution to determine

Again an implementation issue.  The SA provides the one source that can be 
relied upon.  All other methods are potential optimizations that can be 
defined other adjunct documentation.

>         resolution. I don't see why even though based on your

I do not believe there is any benefit to having the LID be a mandatory 
element for the base specification.  Define one method that provides 
guaranteed interoperability for all implementations and then come back and 
define optimizations as experienced is gained from real 
deployments.  Keeping this spec simple to implement is the first step to 
getting a solution deployed from which experience can be gained for future 
enhancements.  It also will allow new material to be added once the IBTA 
has completed its router protocol definition which may present additional 
issues yet to be raised within this workgroup.

>         indirectly/directly from the SA.

The LID, SL, rate control, P_Key, Q_Key, etc. all need to be determined for 
address resolution.  All that is being recommended by some is to use the 
KISS principle for the base spec which is quite a reasonable request.

Mike

---

From: Joe Pelissier <[email-masked]>
Date: Tue, 15 Jan 2002 11:47:47 -0800

Comments embedded marked <JEP>
Regards,
Joe Pelissier

> and that also. It seems a bit overmuch. That is an aside anyway for one to

<JEP>  A very small amount of traffic goes over QP0/VL15.  None of it is relevant to the discussion here.  We are
concerned with communication with the SA over QP1 which does not use VL15 and is subject to flow control.  These
packets are not dropped due to congestion.

> possible should be allowed to do so; exactly as IBTA does.

<JEP>  I agree that no one is suggesting prohibiting consultation with the SM/SA.  What I object to is the
recommendation in the draft against such consultation.  If this recommendation was simply removed and the draft was
silent on the topic, then I would have no objection.  I think I am correct in stating the you believe it works and
interoperates either way (which, btw, I do not believe), so why push so hard for the recommendation?  It seems clear
the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future
RFC.

> LWG. It did not preclude determining SL from sources other than the SA.

<JEP>  You are correct about the rules, my apologies.  Yes, the specification does not require that one obtain the SL
from the SA and no public statement has been made by the IBTA regarding whether or not this is likely to change in the
future.

> specify a particular method either.

<JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.

> control mechanism disjoint from injection control.

<JEP> A minor nit - all nodes that are operating at a rate greater than 1x are required to support static rate.  I
have not checked the spec in detail, but my guess is your probably right - it probably does not specify that one gets
the static rate from the SA.  However, as in the SL case, a mechanism is provided to get this information from the
SA.  So again, I grant you that you are not technically violating the specification if you use a static rate other
than that provided by the SA.  My only objection would be any language in a draft or RFC that specifically required or
recommended that one not consult the SA for this information.  Likewise, I am not proposing that the the draft or RFC
specifically prohibit or recommend against it.

> always ? I don't think so.

<JEP> The only mechanism provided by IB to make such a determination is the SA.  If you have some other way of knowing
that there are no 1x links and there never will be (say, e.g., by an administrative setting), then the fabric would
operate as intended if 4x nodes did not consult with the SA.  Such operation, however, is beyond the scope of the
specification.  And as I stated previously, I have no desire to prohibit such operation within the drafts or RFCs.  I
simply do not wish to promote it either.  I just do not believe that consensus is obtainable for a recommendation
either way so lets move on.

> for SL/path rate or any other information.

<JEP>  I am not asking that we require or recommend that every implementation throw a look the the SA/SM.  I am asking
that we do not require or recommend that they do not.

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 11:58:23 -0800 (PST)

> the consensus is unlikely, so lets compromise by not promoting nor prohibiting either approach in the draft and future

<VK> Joe, you have not been looking at the drafts latest drafts. There
are two working group drafts out - IPoIB architecture and on
multicast. Things have moved forward. Skip the old draft. It is no
longer in consideration.
<VK>

> <JEP>  Hmmm, I'm not sure that the fact that the spec is silent on the topic means that it is promoting it.

<VK> It refers to service levels under QoS and talks a lot about that. It hardly mentions deadlocks for example.
<VK>

> specifically prohibit or recommend against it.

<VK> Nothing is suggested to be added to the draft. All I am trying to
point out is that other solutions are possible. So let us not put it
into the draft that it must be through a particular method i.e.
consult the SA. I think we are in agreement to say that the method of
determinig the values is upto the implementations.

I am also suggesting that due to this reason and the reason that this
information may be received at a different 'time period' one must not
make it a reason to say that the LID must be got from the SA. That is
my main point.

<VK>

> that we do not require or recommend that they do not.

<VK> We are in agreement then.  

        Determining such values is upto the implementation.
        Determining the LID is a separate disjoint issue.

Vivek

---

From: Joe Pelissier <[email-masked]>
Date: Tue, 15 Jan 2002 12:57:31 -0800

Sumarizing, yes I think we are in agreement.  Stating that the IB spec does not specify how one gets SL or static rate is a
simple statement of fact - I have no objection to this.  As you state in the architecture draft, "Every IPoIB implementation
will determine the relevant SL value based on its own policy. No method or process for choosing the SL will be defined by
the IPoIB standards."  I completely agree with this philosophy.  I think this makes any further discussion on whether the IB
specification or its creators intended one mode or the other moot.

Regards,
Joe Pelissier

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 13:03:13 -0800 (PST)

I would like to add to the summary that since the mode of determinig
these values is upto the implementation there is no link between
stating that these values have to be got from SA and therefore LID can
be got from there too. The two are disjoint issues.

Vivek