Subject: RE: [Ipoverib] Please read - proposed WG termination

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 01 Sep 2005 11:12:23 +0200

--On 31. august 2005 20:10 -0400 Bernard King-Smith <[email-masked]> 
wrote:

> Having IPoIB-CM is a very important feature to make IB a viable
> interconnect in clustered systems. Without IPoIB-CM for HPC clusters, and
> commercial clusters using IB to SAN, you need two networks for good total
> cluster performance, one for IB ( non-IP traffic ) and an IP performance
> network like GigE.  This means that IB is not cost effective as the GigE (
> or 10 GigE) network which handles both types of traffic reasonably.

Actually the numbers I've seen show that you can get more than 1 Gbit/sec 
(the test I was shown had 2 Gbit/sec) out of TCP/IP over Infiniband.
Still pitiful for an 8 Gbit/sec network, but faster than GigE.
(Limitation seems to be completely in the host stacks, btw)

So while parallell 10GigE and IB networks will probably give you more 
performance than an IB network alone, parallell 1GigE and IB networks will 
probably not.

Your mileage may vary, of course.

Apart from that, I agree totally with your logic - with the small provisos 
that 1) we do not know what the performance will be until we code it, and 
that 2) with smarter adapters and better stacks, performance of TCP/IP over 
DG is likely to improve too.

                          Harald

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Thu, 01 Sep 2005 10:38:50 -0700

At 05:10 PM 8/31/2005, Bernard King-Smith wrote:

>In the HPC world most clusters use the cluster fabric ( and IB is the
>future direction ) for both MPI and IP traffic. The IP traffic is usually
>for parallel file systems and system management and control. This high
>bandwidth IP network is required in most production HPC clusters.  With the
>current IPoIB only using UD, the performance is dismal. Our simulations
>using the small packet MTU of IB says that the parallel file systems (
>GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link today and at 12X
>it will be about 10%. The problem is that the IP drivers are single
>threaded per adapter. Also the CPU utilization of TCP/IP at a MTU of the IB
>link very high because of the per packet stack processing. Going to
>IPoIB-CM means we can cut down the number of TCP/IP stack traversals from
>32 to 1 for a 60K IP packet. This means that you have 30 times as much data
>transmitted per device driver call. This will enable IP to show similar
>bandwidth with multiple sockets as other protocols that can use RC or
>fragment within the device driver.

These performance problems are primarily implementation-specific and have 
little to do with IB technology itself.  In addition, nearly all IB 
solutions use a 2KB not the smallest MTU to transfer data - no different 
than Ethernet.   As I and others have raised over the years, the enablement 
of IP over IB to perform well is a local HCA issue not a standards 
issue.  Addition of checksum off-load support to the HCA is rather trivial 
and does not require standardization (this is what is done for Ethernet 
today and is non-standard).  Addition of large send off-load support is a 
local HCA issue not a standards issue and effectively provides the same 
benefit as connected mode.  The use of multiple QP to spread work across 
CPU for both send / receive ala the multi-queue support I've worked with 
various Ethernet IHV to get in place is again a local HCA issue (does not 
have to be visible as part of the layer 2 address resolution).  One can 
construct a very nice performing IP over IB solution but there hasn't been 
much public progress to implement these de facto capabilities found in 
Ethernet solutions on IB.  Getting these into a HCA implementation is a 
heck of a lot easier and faster to do than to develop a standard and 
getting all of the OS changes made (the HCA implementation issues can all 
be done underneath the IP stack just like with Ethernet so no real OS impacts).

>For commercial clusters, if IB is used for storage, then you save a network
>by having fast IP performance and can use the IB network for both. Why use
>IB and another network for the commercial cluster, when the other network
>supports similar bandwidth for storage and IP.

There will always be Ethernet in any cluster so the fabric is there.  The 
question is whether it is just for low-bandwidth / management services or 
for applications.  For storage, need to separate the discussion into 
whether it is block or file.  For block, IB gateways to Fibre Channel, etc. 
can and are being used today quite nicely.  Performance is reasonable and 
the ecosystem costs, target availability, customer "pain", etc. are much 
lower than attempting to move to native IB storage.  The same applies to 
file based where IB gateways to Ethernet which then attaches to file 
servers works quite nicely.  In fact, the original vision of IB was that of 
an I/O fabric to create modular server solutions.  The addition of IPC came 
later in the process when it was found to be relatively low cost to 
define.  So, IB is successful in the HPC world and slowly entering some 
commercial solutions.  To state that its future relies on getting an IP 
over IB RC solution is perhaps blowing it a bit out of proportion.   The 
easier path for all is to simply use the techniques I and others have 
advocated for years now and solve the problems within the HCA 
implementation.  Much lower costs and will result in delivering a good 
performance solution.

BTW, RNIC / Ethernet solutions implement these techniques today.  With the 
arrival of 10 GbE and the lower prices of RNIC and 10 GbE switch ports, 
lower latency switches (competitive enough with IB for commercial and many 
HPC clusters), etc. the success of IB must lie elsewhere and not on an IETF 
spec.   This was noted at the recent IEEE Hot Interconnects conference as 
well so isn't just my opinion.

Mike

>Implementing IPoIB-CM makes IB viable in the HPC cluster and some
>commercial clusters. Otherwise I don't think it competes economically with
>other network technologies.
>
>Regards.
>
>Bernie King-Smith
>IBM Corporation
>Server Group
>Cluster System Performance
>[email-masked]    (845)433-8483
>Tie. 293-8483 or wombat2 on NOTES
>
>"We are not responsible for the world we are born into, only for the world
>we leave when we die.
>So we have to accept what has gone before us and work to change the only
>thing we can,
>-- The Future." William Shatner
>
>
>
>              Dror Goldenberg
>              <[email-masked]
>              o.il>                                                      To
>              Sent by:                  [email-masked],
>              ipoverib-bounces@         "H.K. Jerry Chu"
>              ietf.org                  <[email-masked]>
>                                                                         cc
>                                        [email-masked],
>              08/30/2005 09:32          [email-masked],
>              AM                        [email-masked]
>                                                                    Subject
>                                        RE: [Ipoverib] Please read -
>                                        proposed WG termination
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> > From: Vivek Kashyap [mailto:[email-masked]]
> > Sent: Tuesday, August 30, 2005 8:39 AM
> >
> > On Mon, 29 Aug 2005, H.K. Jerry Chu wrote:
> >
>
>
><snip>
>
>
> > > 1. IPoIB connected mode draft-ietf-ipoib-connected-mode-00.txt
> > > updated recently
> >
> > Well, in recent days there has been a discussion going on
> > based on Dror's input. I also made some updates after some
> > discussion on OpenIB (not on
> > IETF though).  This draft itself became a working group draft
> > this february
> > after some lively discussion just before that.  It appears to
> > me that we
> > should be possible to finalise this draft soon enough.
> >
> > 20th sept. might be long enough to know one way or the other...
> >
> > vivek
> >
>
>
>We would like to see IPoIB-CM being finalized in IETF. We see
>great value in having a standard for connected mode which effectively
>increases the MTU. We are willing to contribute to the standardization
>effort. We're also looking at the implementation of IPoIB-CM in Linux.
>
>
>-Dror _______________________________________________
>IPoverIB mailing list
>[email-masked]

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 1 Sep 2005 10:51:34 -0700 (PDT)

On Thu, 1 Sep 2005, Roland Dreier wrote:

>    Bernard> In the HPC world most clusters use the cluster fabric (
>    Bernard> and IB is the future direction ) for both MPI and IP
>    Bernard> traffic. The IP traffic is usually for parallel file
>    Bernard> systems and system management and control. This high
>    Bernard> bandwidth IP network is required in most production HPC
>    Bernard> clusters.  With the current IPoIB only using UD, the
>    Bernard> performance is dismal. Our simulations using the small
>    Bernard> packet MTU of IB says that the parallel file systems (
>    Bernard> GPFS, PVFS, Lustre etc ) can only get 25% of a 4X IB link
>    Bernard> today and at 12X it will be about 10%.
>
> It's not clear to me that IPoIB-CM is really the answer here.  First
> of all, for Linux to take advantage of a 64K MTU, major surgery to the
> network stack will be required.  Production systems have trouble even
> enabling 8K jumbo frames, because the kernel can't allocate two
> physically contiguous buffers to use for receive.  Given that no

That is an implementation issue in Linux and shouldn't effect the protocol.

> current IB hardware can offload TCP/IP checksums for multi-packet RC
> messages, it's not clear how feasible this network stack surgery is.

Checksum offload is not part of the specification but an HCA feature.

>
> Second, if the justification for IPoIB-CM is really for parallel file

The justification is better performance through larger MTU, and leveraging 
connectd modes's apm capability.  I'd view parallel fs is just one application 
that can use it.

Vivek

> systems, then it seems much more promising to run the file system
> natively on InfiniBand.  For example, since Lustre is implemented on
> top of the Portals abstraction, native implementations of Lustre have
> been in use on IB, Myrinet, Quadrics, etc. for quite some time.
>
> - R.
>
>

---

From: Roland Dreier <[email-masked]>
Date: Thu, 01 Sep 2005 10:45:12 -0700

- R.

---

From: Michael Krause <[email-masked]>
Date: Thu, 01 Sep 2005 12:08:59 -0700

>That is an implementation issue in Linux and shouldn't effect the protocol.

Linux does support large send off-load today so it is not clear what the 
issue is you are speaking of above.

>Checksum offload is not part of the specification but an HCA feature.

It is also not that hard to implement.

>connectd modes's apm capability.  I'd view parallel fs is just one 
>application that can use it.

Just to clarify, there is no change in physical MTU.  The IB hardware is 
unchanged w.r.t. the MTU.  All that is done is the same concept as for 
large send off-load - tell the network stack there is a larger MTU and let 
the hardware SAR the segment / datagram into physical units of transfer.

Mike

---

From: Bernard King-Smith <[email-masked]>
Date: Thu, 1 Sep 2005 15:43:26 -0400

>  These performance problems are primarily implementation-specific and
have little to do with IB technology itself.  In  >  >  addition, nearly
all IB solutions use a 2KB not the smallest MTU to transfer data - no
different than Ethernet.   As I and >  others have raised over the years,
the enablement of IP over IB to perform well is a local HCA issue not a
standards >  >  issue.  Addition of checksum off-load support to the HCA is
rather trivial and does not require standardization (this is>   >  what is
done for Ethernet today and is non-standard).  Addition of large send
off-load support is a local HCA issue not >  a standards issue and
effectively provides the same benefit as connected mode.  The use of
multiple QP to spread work >  across CPU for both send / receive ala the
multi-queue support I've worked with various Ethernet IHV to get in place
is >  again a local HCA issue (does not have to be visible as part of the
layer 2 address resolution).  One can construct a very >  nice performing
IP over IB solution but there hasn't been much public progress to implement
these de facto capabilities >  >  found in Ethernet solutions on IB.
Getting these into a HCA implementation is a heck of a lot easier and
faster to do >  >  than to develop a standard and getting all of the OS
changes made (the HCA implementation issues can all be done >  >  >
underneath the IP stack just like with Ethernet so no real OS impacts).

I understand the point of putting expansions into the HCA or stack that
don't already exist, just like jumboframes and large_send were added by
vendors for GigE. However, where I lost your logic was when you recommend
IP not to use the RC function, already implemented in all adapters, but
instead ask vendors to add an additional (duplicate) function to IB
adapters to provide RC equivalence specifically for IP.

>  There will always be Ethernet in any cluster so the fabric is there.
The question is whether it is just for low-bandwidth /

 >  management services or for applications.  For storage, need to separate
the discussion into whether it is block or file.  For   >  block, IB
gateways to Fibre Channel, etc. can and are being used today quite nicely.
Performance is reasonable and the >  >  ecosystem costs, target
availability, customer "pain", etc. are much lower than attempting to move
to native IB storage.  The >  same applies to file based where IB gateways
to Ethernet which then attaches to file servers works quite nicely.  In
fact, the >  original vision of IB was that of an I/O fabric to create
modular server solutions.  The addition of IPC came later in the >  >  >  >
process when it was found to be relatively low cost to define.  So, IB is
successful in the HPC world and slowly entering >  >  >  some commercial
solutions.  To state that its future relies on getting an IP over IB RC
solution is perhaps blowing it a bit out >  of proportion.   The easier
path for all is to simply use the techniques I and others have advocated
for years now and solve >  the problems within the HCA implementation.
Much lower costs and will result in delivering a good performance solution.
>
>  BTW, RNIC / Ethernet solutions implement these techniques today.  With
the arrival of 10 GbE and the lower prices of RNIC >  and 10 GbE switch
ports, lower latency switches (competitive enough with IB for commercial
and many HPC clusters), etc. >  >  the success of IB must lie elsewhere and
not on an IETF spec.   This was noted at the recent IEEE Hot Interconnects
>  >  >  >  conference as well so isn't just my opinion.
>
Mike

Regards.

Bernie King-Smith
IBM Corporation
Server Group

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 1 Sep 2005 14:42:27 -0700 (PDT)

> issue is you are speaking of above.

Let us say that even if there is an issue with 8K frames/contiguous memory
allocation in Linux, it is a Linux specific issue.

> It is also not that hard to implement.

agreed and is not relevant to developing the protocol.

> unchanged w.r.t. the MTU.  All that is done is the same concept as for large 
> send off-load - tell the network stack there is a larger MTU and let the 
> hardware SAR the segment / datagram into physical units of transfer.

yes, it is for a higher IP MTU and brings large sendoff benefits to IPoIB.

Vivek