Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 14 Jan 2002 14:00:35 -0800 (PST)

My comments are embeded in the following.

>I agree with using GID both for longevity and scope. What about using
>SIDR(Private Data) for communicating all the capability bits and any other
>extras? If it is not too late for this, I have some new feedback.
>
>For ARP on an IB subet one would require that
>a. IB Switches support multicast. (This is true in most cases today.)
>b. IP addresses used in an infiniband subnet are unique for use by IPoIB.
>(This is an important constraint which gets rid of the ethertype field)
>c. Some issues related to Multicast over IB need to be resolved (like
>multiple consumers using multicast resources.)

This one is really simple. A MGID contains 128 bits. Even IPv6 recognizes
that it only needs the lower 32bits for the forseeable future. So the
probability of MGID collisions is negligible. If one really wants
something bullet-proof, we can ask IBTA to reserve a chunk (or endorse
the IPoIB signature covered in the link and multicast draft) for IPoIB.

>
>Requirement (b) was addressed by defining a multicast group and the IP
>address format. One would still have to deal with the GID, LID and the IP
>address. Including the LID in the ARP Response will be difficult to deal
>with because
> a. LID Assignments change.
> b. ARP might have to traverse a router, for IP Subnets spanning IB Subnets
>(a) might not really be an issue because ARP entries are aged. In ethernet,
>the ethernet topology might change and MAC Addresses can change on server
>reboots (by way of using a different MAC or by assigning a MAC Address.) The
>current ARP implementations do take these scenarios into account. Moreover
>Path record lookups alone would not help resolve the issue of changing LIDs.
>The locally cached path record will not be valid if the LID changes. For a
>true solution, there has to be some way of using IB Traps to get information
>about topology changes as and when it happens. An end node can subscribe for

This is a generic, not IPoIB specific issue. So the work belongs to IBTA
to ensure some mechanism exists for flushing stale path records.

>a trap, but the end node will have to do something like,
> * Tell me when all the switches on the path to this node report a state
>change on such and such a port.
> This will not work very well right now because one will have to subscribe
>for traps on all switches leading to the end node, which implies that an end
>node interested in the existence of another end node will need to have
>knowledge of the complete IB topology. This is probably something worth
>considering in the Infiniband WGs.

Ok, we're saying the same thing.

>
>On the same token, if you use the globally unique GID, information about the
>GID will not be maintained by the local subnet's subnet administration and
>hence a path record lookup based on the GID would fail.

In this case you follow the IBA rule. The rule says that you don't even
attempt to lookup a pathrecord for a remote GID in a different IB subnet.
You need to find an IB route to it first. This is similar to IP.

> So if the GID lookup
>fails, one would send the ipoib packet to the designated IB router (as is
>done with ethernet today, albiet based on netmasks). But what if you have an
>IP subnet spanning IB Subnets. The designated router exists for an IP Subnet
>and the local subnet IB administration will not have information about all
>the IP nodes, because they could exist on another IB Subnet.

Hmmm, you seem to confuse routing at the IB level with that at the IP
level. (Or am I the one who is confused?)

>
> The following IB transactions are required before sending any IPoIB data
>over the subnet (assuming the multicast join is already done and we use GIDs
>as link layer addresses)
> 1. IpoIB ARP.
> 2. Path record lookup.
> 3. SIDR
>
>In all this, the existence of the ARP is questionable. The same
>functionality can be achieved using 2 and 3 above.
>If you use GIDs as the IP addresses ,you can use a Path record lookup and
>then use SIDR to get all the information to use IPoIB. The advantages are
>evident, for eg,

I think similar idea has been brought up before. One major obstacle here
is that a given interface may host many (e.g. a few thousand) IP addresses
simultaneously (e.g. virtual hosting). You just don't have space for that
many GIDs on a HCA.

Jerry

> a. Traversing a router. Assuming that the path record lookup fails, one
>will redirect all the related ipoib packets to the Gateway/router. Since ARP
>does not exist, one does not have to worry about the issues involved by ARP
>traversing a router, or the router proxying for end nodes on the same IP
>Subnet but different IB Subnets. IP and IB Subnets will be one and the same.
> b. Debate on using the ether-type field vanishes. The Service ID will be
>unique to IPoIB and will be assigned by some WG. You could potentially use
>two Service IDs (might be an overkill), one for IPv6 packets and another for
>IPv4.
> c. No special format for IP addresses is required (I have not thought this
>through very well.)
> d. Switches are not required to support Multicast.
> e. There are no issues with multiple consumers using the same multicast
>resources.
> f. Multicast could be used by an IB Gateways only to avoid multi-unicast.
>
>In this scenario the IP subnet is the same as an IB Subnet, since the same
>network addresses are used.
>
>-Ashwin
>
>
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf
>Of H.K. Jerry Chu
>Sent: None
>To: [email-masked]
>Subject: [Ipoverib] What's inside an IPoIB link layer address?
>
>
>While proposals beyond the basic IPoIB over UD started showing up,
>I still don't think we've reached a rough consensus on some basic
>questions such as what the IPoIB link layer address should contain.
>Again I'd like to apologize for not being there at the SLC meeting
>last month. Perhaps a consensus was reached there but i've missed
>the boat. In that case please let me know what it is and I'll shut
>up.
>
>What I'd like to see included in the IPoIB link layer address is
>
>GID
>QPN
>
>and I'd like to allow QPN field to be 0.
>
>What I don't like to see included is LID.
>
>Discussions:
>
>There has been a consensus of using GID, not LID as the basic
>IB address identifying the port hosting an IP address. GID is
>sufficiently stable, not subject to change across a node reboot
>or subnet merge (IB level) like LID is. Also people don't want
>to exclude the possibility of IPoIB link spanning more than
>one IB subnet. GID supports scope beyond local link so it's
>suitable for that purpose.
>
>Multiple GIDs with different scopes may have been assigned to a CA.
>It is required that a CA returns the GID with a scope no less than
>the one for the IPoIB link. E.g. if an IPoIB link covers multiple
>IB subnets, but a CA uses a link-local GID as part of its IPoIB
>link layer address, it's not going to work well.
>
>I don't see any value of including LID besides GID as proposed
>by Vivek. I believes a lookup of PathRecord is always needed
>in order for a node to retrieve complete path info before sending
>unicast pkts to another node. The destination LID is part of
>the PathRecord so including it in the link layer address returned
>from ARP/ND doesn't save work. But you have all the drawbacks that
>swayed people away from using LID - it adds volatility to the link
>address. LID changing due to subnet merge or other reasons requires
>potentially new mechanism being added to IP/driver implementation
>to trigger ARP/ND flush. Lastly, what should be the LID if the IP
>node is in a different IB subnet? Obviously it can't be the LID of
>an intermediate IB router.
>
>For QPN I'd like to suggest one thing. Since SIDR is a required
>feature by IBA, the QPN field is allowed to be zero. What that
>means is that the requester must use SIDR to retrieve the remote
>QPN, given the GID of a remote IP address.
>
>This feature allows the support of static/persistent "MAC addr"
>wrt node reboot by removing the volatile part of the link layer
>address - the QPN. In many implementations the QPN allocated for
>a CA may change across node reboot. Support of static "MAC addr"
>has been considered desirable by a number of people.
>
>I don't like to add additional requirement such as SIDR to IPoIB,
>and was trying to make the above feature "optional". Ideally nodes
>can express whether they'd like QPN to be returned by embedding
>some flags, such as "I don't talk SIDR so please return QPN" in
>their IPoIB link layer addresses. But the broadcast nature of
>IPv4/ARP forbids the use of different "MAC addrs" for the same IP
>address, so my attempt failed.
>
>Comment?
>
>Jerry
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>http://www.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Ashwin Kamath" <[email-masked]>
Date: Mon, 14 Jan 2002 17:10:11 -0600

There is a general thought that ARP is a one step solution for obtaining the
parameters required for establishing IPoIB over UD. But I think that even
for using ARP, one will require to get the Path record information. I do not
think that ARP is a one stop solution. What happens if both nodes determine
that their MTU is 2K without considering that a switch component in between
might restrict the MTU to 256 bytes? Such information can be obtained only
from the central SM via a path record. Am I missing something here? If
address resolution is all that ARP is required to do, without trying to
establish parameters for communication one may not want to know the MTU, but
from the discussions I see that the Address resolution is being used for
establishing communication parameters (like the QPN)

So the only real problem with having the LID in the ARP response, that I see
is when the ARP response needs to cross IB Routers. Can I assume that
changes in LID assignments are not a big problem because such changes are
not intended to be frequent and current ARP cache implementations already
factor in the possibility of a link level address change. Changing LIDs
would have a bigger impact on shared devices using RC than on IPoIB using
UD. Is it ok if I consider a change in LID assignment to be equivalent of a
server reboot?

If I understood what Jerry, one will have a problem if the ARP responder
lies in another IB Subnet, in which case the LID in the response will not be
relavant to the originator of the ARP request, in-fact the LID will have to
be the LID of the router. Was there any resolution reached for this problem?

In short you do need an ARP + Path record diagnosis to determine all the
parameters for communication. What would happen if we bring an IB router in
between? One cannot obtain a path record to a node in another subnet. One
can obtain the path record to the IB Router. Should we be talking about the
behaviour of the IB router in case of mismatch of MTUs or should we say that
we will always use the smallest mtu for communicating across an IB router.
Or should we say that this could fall on the plate of the Infiniband WGs.

-Ashwin.

-----Original Message-----
From: H.K. Jerry Chu [mailto:[email-masked]]
Sent: None
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Jerry

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 14 Jan 2002 15:47:59 -0800 (PST)

> from the central SM via a path record. Am I missing something here? If

        The all-nodes multicast(IPv6)/broadcast(IPv4) addresses are
        mapped to a multicast GID. This multicast GID is setup with
        the MTU for the subnet/link. All nodes always IB_join this
        MGID. Thus the MTU is always determined from the multicast
        GID. So is the Q_Key to be used. See the ipoib architecture
        and multicast drafts that have been posted. You can get them
        from the ipoverib pages at IETF site.

        Other parameters LID/GID/QPN are received via ARP. SL is
        determined based on the implementation. So, ARP is and
        certainly can be a one-step solution like you mention in the
        begining of your note.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: "Ashwin Kamath" <[email-masked]>
Date: Mon, 14 Jan 2002 18:45:51 -0600

Vivek,

Is the assumption true that the MTUs established for switching multicast
packets will be the same as that used for Unicast switching? ie will
multicast packets from one end node to another traverse the same path as a
unicast packet? I personally would not try to configure a path which uses a
higher MTU for a multicast group vs a lower MTU for other path records, but
in general the SM is not restricted by this requirement. ( It is possible
that the SM chooses 4x 1K MTU links for normal data and 1x 4K mtu links for
Multicast data, if by some chance, the topology forces it to do so. If some
switches do not support multicast and atleast one path per node exists such
that multicast can be supported for the fabric as a whole, the end-to-end
paths for the unicast packets may differ from that of multicast packets for
any given pair of nodes and hence the mtus can be different, more
specifically, the mtus can be larger for multicast than for unicast. ) If it
is true that multicast and unicast packets traverse the same paths
end-to-end or if it is reasonable to assume the MTU to be that of the
Multicast group because that is the best possible configuration, I do not
see any issue, except that such a statement of restriction be made on the
subnet management for use with ipoib. I think such a restriction is
reasonable for operation of IPoIB though I do not know if it is essential,
neither can I offer an alternative at this point. Do you think that such a
statement of restriction is required?

 What about the maximum static rate of injection which is part of the path
record? This parameter is not part of the Multicast Group record and is a
function of the DLID for any node injecting packets into the fabric. The
compliance statments for static rate injection are called optional. But from
the compliance statements I could not determine whether they were optional
contingent of the CA not supporting rates higher than 2.5 GBPs or they are
truly optional.

 PS: You can ask me to shut up if it is kind of late for such questions? :)

-thanks,
Ashwin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Vivek Kashyap
Sent: None
To: Ashwin Kamath
Cc: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Vivek

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Jan 2002 16:57:13 -0800

At 05:10 PM 1/14/2002 -0600, Ashwin Kamath wrote:
>might restrict the MTU to 256 bytes?

The minimum supported MTU for UD traffic is 2KB so a path with a 256 bytes 
MTU should not be used by the IP over IB service.  The SM working with the 
partition manager will establish a set of paths that can be used for IP 
over IB traffic which will include the setting up of an all nodes multicast 
group per partition that support IP traffic.

>Such information can be obtained only

QP identification is an optimization that allows one to eliminate the 
Ethertype field.  It also allows an implementation to map different IP 
addresses to different QP thereby providing an implementation optimization 
through more fine grain hardware-based de-multiplexing.  An example use 
might be to assign an IP address per OS instance which shares a HCA port 
and then allow all traffic to target a corresponding QP without having to 
flow through an intermediate hypervisor that would have to de-multiplex the 
traffic.  This can provide both performance and scalability improvements 
for vendor offerings.

>UD. Is it ok if I consider a change in LID assignment to be equivalent of a

No.  Applications should not be impacted by a change in LID assignment as 
there are defined methods to migrate applications from one LID or GID or 
port to another that are completely transparent to the application.  For 
UD-based services, one could simply re-ARP the endnode via the all-nodes 
multicast and provide a simple update of the LID mapping as this can be 
derived from the IB headers if desired.

>Or should we say that this could fall on the plate of the Infiniband WGs.

The IB router protocol will define how one routes across IB subnets - an 
IBTA issue.  An IP router may be implemented along side an IB router or it 
may be an edge device such as a multi-protocol router.  The fact that IP 
subnets may span multiple IB subnets isn't really that big of an issue per 
se.  By keeping the LID not part of the link address structure, the ARP 
should be able to transparently flow across multiple IB subnets.

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Jan 2002 17:19:30 -0800

At 06:45 PM 1/14/2002 -0600, Ashwin Kamath wrote:

>Multicast data, if by some chance, the topology forces it to do so.

The requirement should be to use a 2KB MTU as the default for all UD-based 
communications.  If the fabric can support a 4KB MTU, then it may also be 
used between endnodes.

>If some
>specifically, the mtus can be larger for multicast than for unicast. )

All switches should be required to support multicast so this class of 
problems does not exist. Given that most implementations appear to support 
multicast in order to support IP over IB, I don't see why we need to add 
any complexity to accommodate switches that do not support multicast.

>If it is true that multicast and unicast packets traverse the same paths

The only restrictions for IP over IB in terms of what the IB fabric 
provides are 2KB PMTU  between participating endnodes and switch-based 
multicast (will apply to routers) support.

>contingent of the CA not supporting rates higher than 2.5 GBPs or they are

Static rate control is not optional for any port injecting packets into the 
fabric that operates at greater than 2.5 Gbps (current 1x 
bandwidth).   This is clearly indicated in both Chapter 9 compliance 
statements as well as chapter 17 table 254 (volume 1, release 1.0.a).

Mike

---

From: "Ashwin Kamath" <[email-masked]>
Date: Mon, 14 Jan 2002 19:39:24 -0600

Mike,
 So it would be mandatory for the end nodes to determine the static rate
before injecting packets targetted for a particular DLID. The path record is
the only mechanism for an end node to determine the static rate for a path.
Does that make the path record query mandatory for using a UD queue pair for
IPoIB? If not then the suggested ARP mechanism for ipoib is truly
self-sufficient.
 Also since 2K minimum MTU is required, should this be stated as REQUIRED of
the ipoib requirements draft, or is it outside its scope?
-Ashwin.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Michael Krause
Sent: Monday, January 14, 2002 7:20 PM
To: [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Jan 2002 19:26:09 -0800

At 07:39 PM 1/14/2002 -0600, Ashwin Kamath wrote:

>before injecting packets targetted for a particular DLID.

Yes.

>The path record is the only mechanism for an end node to determine the 
>static rate for a path.
>Does that make the path record query mandatory for using a UD queue pair 
>for IPoIB?

For the initial unicast communication, one must know what the static rate 
is for a given path within an IB subnet.  I would also contend that one 
will need to know this when crossing IB subnets as well should an IP subnet 
span multiple IB subnets.  This is currently under investigation for the IB 
router specification.  A multi-protocol IP router at the edge of an IB 
fabric (of any diameter including a single link if desired) would act as an 
injection point (CA) and would be required to support static rate control 
if operating at greater than 2.5 Gbps.

>If not then the suggested ARP mechanism for ipoib is truly self-sufficient.

Yes. Required.

Mike

---

From: Joe Pelissier <[email-masked]>
Date: Mon, 14 Jan 2002 20:35:19 -0800

Greetings:
I unfortunately have had to reduce my participation in this effort for a while,
but have now been studying the email threads attempting to get back up to speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether ARP
can be accomplished without consulting the SA (or at least cached information
from a previous consultation).  There seems to be two major issues: static rate
and service level.

All nodes that operate at greater than x1 speed are required to support static
rate.  The static rate setting is a function of the path between two endpoints.
It is important to note that the path multicast uses is very likely to be quite
different than the path that unicast traffic uses.  Therefore, the static rate
may well be different.  The only way to get this information is to consult the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not require
getting the SL from the SA.  This is true.  This has been recognized by the IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls which
SLs a node may use based on path.  The SA may provide a set of SLs from which a
node may choose, but the SA needs to control this.  If it does not, deadlock of
the fabric may occur.  To prevent deadlock, the SA can load certain IB tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The IB
spec already talks about how to determine both static rate and SL.  Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these are
determined IAW the InfiniBand standard).

wrt/ SL, this is pretty much what section 6.0 of
draft-kashyap-ipoib-ipv4-and-arp-01.txt says, except for the last paragraph
(which recommends using the SL from the broadcast group).  Unfortunately, the
recommendation is directly contrary to the some intended usage of IB and is
likely to either be prohibited or at least strongly discouraged by a future
version of the specification.

The LWG voted unanimously to recommend the addition of language that strongly
discourages the use of parameters that were not obtained by an applicable path
record and stating that the parameters from multicast and unicast should not be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move forward
without further discussion.  It just seems bazaar to continue to push something
as "recommended" that causes great concern to many of the architects that created
IB.  All we need to do is leave out the recommendation and we are done with this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA
spec.  There seems to be little downside to this.

Besides, you need to get static rate anyway....

Another $0.02 worth...
Regards,
Joe Pelissier

---

From: Yaron Haviv <[email-masked]>
Date: Tue, 15 Jan 2002 07:13:20 +0200

Joe, welcome back to the discussion
Regarding the IPD (static rate control) issue you mentioned
I don't see any host managing to fill a 1X pipe doing IPoIB (the TCP stack
would die much sooner); those bandwidths are for native IB protocols such as
SDP and SRP, setting it to 1X as a default will probably wont create any
problem

Another issue around it, currently the LWG is discussing using the IPD for
congestion avoidance with a thinner granularity that will be dynamically
adjusted (through FECN's, MAD's)

Yaron
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Joe Pelissier
Sent: Tuesday, January 15, 2002 6:35 AM
To: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Greetings:
I unfortunately have had to reduce my participation in this effort for a
while,
but have now been studying the email threads attempting to get back up to
speed.
My apologies in advance if my comments below are missing some context from
previous disucssions.

It seems that there continues to be a great deal of discussion as to whether
ARP
can be accomplished without consulting the SA (or at least cached
information
from a previous consultation).  There seems to be two major issues: static
rate
and service level.

All nodes that operate at greater than x1 speed are required to support
static
rate.  The static rate setting is a function of the path between two
endpoints.
It is important to note that the path multicast uses is very likely to be
quite
different than the path that unicast traffic uses.  Therefore, the static
rate
may well be different.  The only way to get this information is to consult
the SA
(actually, one could always set their static rate to x1 speed and avoid the
consultation, but if you got a x12 link, it seems a shame not to use it just
to
avoid talking to your friendly SA ;-).

The other is SL.  It has repeatedly been argued that the spec does not
require
getting the SL from the SA.  This is true.  This has been recognized by the
IBTA
and steps are being taken to correct it.  In order to support multiple paths
through the IB fabric, it was a fundamental assumption that the SA controls
which
SLs a node may use based on path.  The SA may provide a set of SLs from
which a
node may choose, but the SA needs to control this.  If it does not, deadlock
of
the fabric may occur.  To prevent deadlock, the SA can load certain IB
tables
that would cause the offending packets to be discarded.  This protects the
fabric, but would cause communication to fail between the offending nodes.

So we could argue all this forever, or maybe there is another solution.  The
IB
spec already talks about how to determine both static rate and SL.
Therefore, we
don't need to rehash it in the RFC (other than maybe a note that says these
are
determined IAW the InfiniBand standard).

(which recommends using the SL from the broadcast group).  Unfortunately,
the

The LWG voted unanimously to recommend the addition of language that
strongly
discourages the use of parameters that were not obtained by an applicable
path
record and stating that the parameters from multicast and unicast should not
be
mixed (see the LWG meeting minutes dated 8/16/01).

If we just remove the recommendation from the draft, then we can move
forward
without further discussion.  It just seems bazaar to continue to push
something
as "recommended" that causes great concern to many of the architects that
created
IB.  All we need to do is leave out the recommendation and we are done with
this
issue.  Those that wish to ignore the SL assignment from the SA and use the
multicast one can still do so without technically violating the current IBTA

Another $0.02 worth...
Regards,
Joe Pelissier

---

From: "Bill Strahm" <[email-masked]>
Date: Mon, 14 Jan 2002 22:30:15 -0800

Why do you say that... Current stacks exist that can push TCP traffic well
above 1 Gbit, even with standard hardware (1Ghz proc, standard nic etc)
Intel was showing TCP running at around 800 Mbit.  I go with a TOE and all
the sudden my throughput becomes limited by the bus bandwidth, guess what
happens when someone puts a 12x connection in the chipset running at full
throttle...

I don't think we want to limit ourselves to 1x just because that is all
people are thinking about today.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Yaron Haviv
Sent: Monday, January 14, 2002 9:13 PM
To: 'Joe Pelissier'; [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Yaron
-----Original Message-----

Greetings:

Another $0.02 worth...
Regards,
Joe Pelissier

---

From: Yaron Haviv <[email-masked]>
Date: Tue, 15 Jan 2002 08:35:01 +0200

I would be happy to see any benchmark running TCP in 1Gbit without any form
of offload, and regular MTU
The ones that you see are using special techniques, and are in ideal cases
And they consume the entire CPU processing power
(1X is also 2Gbit)

I heard a wise man saying once that if eventually TCP/IP traffic will
dominant IB fabric we probably did something wrong here
And could have stayed with Ethernet

If you would insist on killing your CPU exceeding the 2Gbit, you could still
do the PathRecord query
My notion is that the IPoIB will be used to initialize the fabric, low
bandwidth applications
The data movement applications will be using RDMA (the IP guys already got
it and now imitating IB stuff, see iWarp)

Yaron

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Tuesday, January 15, 2002 8:30 AM
To: [email-masked]; 'Joe Pelissier'; [email-masked]
Subject: RE: [Ipoverib] What's inside an IPoIB link layer address?

Bill

-----Original Message-----

Yaron
-----Original Message-----

Greetings:

Another $0.02 worth...
Regards,
Joe Pelissier

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 09:16:53 -0800

>SDP and SRP, setting it to 1X as a default will probably wont create any

I strongly disagree with the above assertion.  There are many servers that 
today can drive 1x at link rate; in fact, many that can drive 4x using TCP 
/ IP implemented in software - keep in mind that Moore's Law combined with 
new I/O chip interconnects such as PCI-X, PCI-X 2.0, and 3GIO one will have 
the processing power and I/O bandwidth to drive the very high bandwidths 
that are supported by IB fabrics.

>adjusted (through FECN's, MAD's)

What the LWG is doing is separate from this effort as it is in another 
standards group.  The IB spec states any link that supports greater than 
2.5 Gbps must implement static rate control. This means that any IP over IB 
specification must not preclude or prescribe any implementation from using 
static rate control in compliance with the current IB specification.

Mike

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 09:28:16 -0800

>happens when someone puts a 12x connection in the chipset running at full

As another data point, HP has a PCI GbE NIC running full-duplex at link 
rate (i.e. over 1800 Mbps aggregate of application data - user-space to 
user-space) and over 200,000 packets per second both measured using netperf 
TCP_STREAM and TCP_RR respectively.  This bandwidth was achieved using 
standard Ethernet MTU and not protocol off-load sans checksum 
off-load.  Less than 1 CPU was used to drive this bandwidth.  BTW, we can 
drive 6 of these links on a 8-way SMP without consuming all of the CPU 
power.  Scaling is fairly linear within the TCP implementation and with the 
point-to-point non-shared PCI implementation, we don't have any of the I/O 
scaling problems found in many PCI implementations.

>people are thinking about today.

Agreed.

Mike

---

From: [email-masked]
Date: Tue, 15 Jan 2002 13:25:44 -0500 (EST)

> avoid talking to your friendly SA ;-).

Actually always using a static rate of 1x for ARP multicast may be a
good idea.  All this does is limit source multicast traffic to not
exceed ~200MB/s (surely this is enough).  I see no downside to this.

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 10:50:24 -0800 (PST)

> exceed ~200MB/s (surely this is enough).  I see no downside to this.

        Exactly, it is an implementation's choice on how to use path
        rates/ setup the subnets etc. It is not tied to determining
        the link address.

Vivek

---

From: Joe Pelissier <[email-masked]>
Date: Tue, 15 Jan 2002 11:09:25 -0800

I was not referring to the static rate for multicast.  I agree there is no downside to
limiting ARP traffic to 1x speeds.  I was referring to real IP traffic.  If I want to talk
IP from a to b at rates greater than 1x, I believe it was the intent that I must get a
static rate value from the SA.  Probably this is another case where the IBTA went through
all the trouble of providing a mechanism to determine a parameter but did not mandate its
use.

Cheers,
Joe Pelissier

---

From: Michael Krause <[email-masked]>
Date: Tue, 15 Jan 2002 11:11:42 -0800

>         rates/ setup the subnets etc. It is not tied to determining

I do not believe anyone is contending that an implementation may not 
default to a static rate of 1x for any IP service; all that is being 
contended is that the IP over IB spec be required to specify that one must 
use static rate in compliance with the IB specification.  This will allow 
implementations to achieve interoperability independent of the physical IB 
attributes for all types of IP traffic.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 11:32:29 -0800 (PST)

> attributes for all types of IP traffic.

No argument with using a specific or any IBTA method. Am only stating
that it is upto the implementation to follow any method to determine
the values. That still doesn't tie it to determining the LID. If one
has found the values due to any method - SL through QoS method, path
rate due to knowing the topology is homogenous in
speed/speeds/whatever - one may choose not to go to SA for those
values. This has nothing to do with the LID.

My objection is to the claim that the LID must be determined from the
SA/SM because some may choose to get the path-rate/SL from the SM/SA.
These are disjoint issues.

Vivek

---

From: "Roger Ronald" <[email-masked]>
Date: Tue, 15 Jan 2002 13:43:14 -0600

There are benchmarked IP implementations that already exceed 
1x IB rates (work done by Jim Pinkerton at SGI using GSN). 
These are few and far between, but speeds of this sort
will be more common as time goes by. The reason that 
IP over IB is important is the opportunity to "go fast".

Any solution for IP over IB should NOT assume that progress
will not continue and that  requirements will not grow!

I will agree that the requirements for ARP multicast 
bandwidth are much lower.

RR

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Jan 2002 13:37:49 -0800 (PST)

Actually I'll ask you to read the ipoib-architecture and multicast
drafts that hve been posted. They should answer your questions. If not
then ask them and we will update the drafts accordingly.

Vivek

> contingent of the CA not supporting rates higher than 2.5 GBPs or they are

You might have already got the answer based on the recent discussions
between me, Joe and others.

---

From: "Ashwin Kamath" <[email-masked]>
Date: Tue, 15 Jan 2002 17:03:40 -0600

Vivek,

I am agreeable to the results of the discussion you had on the reflector. ie
end to end communication parameters are implementation specific, and the ARP
mechanism tries its best to get all information which may or may not be
enough for end-to-end communication. It is upto the implementation to use
the ARP information in an IB compliant fashion. As such, all end-to-end
communication parameters which cannot be determined from the configuration
cached on any given end-node, within the perimeter of the IB Specifications'
definition for end nodes, are omitted from ARP. This is to keep the
implementations free to choose from any method to obtain such information as
they deem fit. The method could be static configuration or consulting the
SA, or anything else. More specifically the definition does not exclude the
possibility of static configuration or othwerwise, of parameters like MTU,
SL, static rate. The intention of such possibilities could be simplicity,
conformance or anything else.
  I agree that this is reasonable.

-Ashwin.

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: None
To: Ashwin Kamath
Cc: [email-masked]
Subject: Re: [Ipoverib] What's inside an IPoIB link layer address?

Vivek

> > UD. Is it ok if I consider a change in LID assignment to be equivalent
of
> a
> > lies in another IB Subnet, in which case the LID in the response will
not
> > >reboots (by way of using a different MAC or by assigning a MAC
Address.)
> > The
> > >The locally cached path record will not be valid if the LID changes.
For
> a
> > >about topology changes as and when it happens. An end node can
subscribe
> > for
> > >for traps on all switches leading to the end node, which implies that
an
> > end
> > >On the same token, if you use the globally unique GID, information
about
> > >done with ethernet today, albiet based on netmasks). But what if you
have
> > an
> > >will redirect all the related ipoib packets to the Gateway/router.
Since
> > >two Service IDs (might be an overkill), one for IPv6 packets and
another
> > for