Subject: RE: Ecosystems cost of additional specs (RE: [Ipoverib] Please re ad - proposed WG termination)

From: Carl Hensler <[email-masked]>
Date: Sat, 3 Sep 2005 12:39:54 -0700 (PDT)

> Date: Fri, 02 Sep 2005 15:45:30 -0700
> From: Michael Krause <[email-masked]>
> Subject: RE: Ecosystems cost of additional specs (RE: [Ipoverib] Please	
re ad - proposed WG termination)
> To: Dror Goldenberg <[email-masked]>, "'Harald Tveit Alvestrand'" 
<[email-masked]>
> Cc: [email-masked]

> Somewhat an aside, but it is ironic that standard APIs are often ignored / 
> deemed "crap" by some even if many systems can benefit from them yet this 
> spec which is likely to be implemented by a single OS and used only in 
> perhaps a subset of deployments is deemed worthy. Apologies for the tangent.

Mike,

IPoIB-CM is an optional protocol extension accessed through the 
standard networking API, Sockets.

If standardized, it will be implemented for all three major OSes,
Linux and Windows in OpenIB, and Solaris at Sun.

I think the WG should complete work on IPoIB-CM expeditiously, 
and determine whether there is community support for completing
the MIBs.

Carl

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 5 Sep 2005 23:56:08 -0700 (PDT)

> standard networking API, Sockets.

Yes, and I'd like to add that IPoIB-CM does not imply any change to IPoIB-UD.
The draft takes care of ensuring there is no impact on the existing 
implementations.

> Linux and Windows in OpenIB, and Solaris at Sun.

and a few other major OSes :).

> I think the WG should complete work on IPoIB-CM expeditiously,

I'd posted the draft of -01.txt to the list earlier. I should be able to
update it with Dror's suggestions soon enough. We should be able to move
fast on it from there.

Vivek

---

From: Michael Krause <[email-masked]>
Date: Tue, 06 Sep 2005 08:29:59 -0700

>Linux and Windows in OpenIB, and Solaris at Sun.

Didn't know Solaris was a major OS from a volume perspective (might be said 
for any Unix these days as there really are only two OS of any real volume 
any longer).  :)  I will state that at this point in time, I am not 
recommending to my company to do any work on IPoIB-CM and I plan to push on 
spending resources on where there is significant value and ROI 
opportunities on the RDMA side of things not on making IP-based stacks work 
better (we are already getting more than good enough performance with the 
existing specs implemented).

In any case, Windows is really a third-party provider doing the work not 
MS.  Does not really matter I suppose in the end.

>and determine whether there is community support for completing

I don't see the work as compelling but people are free to ignore me as I'm 
just one voice within the IETF here.  I personally believe the emphasis 
should be on getting a solid RDMA infrastructure in place and using that 
for ULP communication than on making what is already a reasonable IP over 
IB network stack more optimal.  If you go back through the history of 
interconnects, when one cannot demonstrate significant benefit difference 
to customers or take too long to get the value-add right, then customers 
are fine with going with the evolutionary technology.  10 GbE RNIC prices 
are dropping as are switch port costs.  While they may not be on par with 
IB today, by the time it really matters to customers, the total solution 
costs which go well beyond the hardware may result in 10 GbE being just 
fine.  10 GbE will have LSO, LRO, whatever as well as RDMA and a much lower 
cost to integrate management and associated ecosystems into the data 
center.  Spending resources on making IP communication better on IB is 
simply a poor use of resources with minimum end visible customer benefit 
especially if some of the OS such as Windows already have the hooks in 
place to support RDMA ULP via Winsocks Direct.

Mike

---

From: Michael Krause <[email-masked]>
Date: Wed, 07 Sep 2005 07:12:55 -0700

BTW, as a data point, testing with a PCI-X 266 10 GbE NIC hits link rate 
with using regular 1500B Ethernet frames and de facto techniques such as 
checksum off-load (link rate for Ethernet is higher than link rate for IB 
since one delivers a 10 Gbps payload rate and the other delivers a 10 Gbps 
raw rate which is typically 7-8 Gbps depending upon if the 2KB MTU is used 
something that is also larger than standard Ethernet's frame).  This has 
been measured on Itanium platforms running Unix so people's mileage will 
vary but clearly there are implementations that are able to deliver 
performance without requiring jumbo / large MSS.   It may take a bit more 
CPU but then no real server network workload runs requires link rate all of 
the time - statistically, there are bursts when applications complete 
various stages of the work.

In any case, I sense the hum is to support the work but the reality is if 
one wants performance, the work isn't that compelling and there is that 
other value-add that IB brings called RDMA that was intended to be used to 
deliver high-performance workloads that many of us believe is the proper 
and more valuable technology to make happen for IB than IP 
connectivity.  Existing IP over IB implementations measured on low-end 
Itanium running Linux is at 3-3.5 Gbps which is very respectable and meets 
nearly all workload needs. Heck, real world workloads like NFS do not even 
generate that much network bandwidth utilization so it is not clear where 
exactly the problem lies sans perhaps micro benchmarks which will be 
impossible for IB to beat Ethernet without jumping to wider links or higher 
signaling rates (the minimum required given the differences in the physical 
layers).

Mike

---

From: Bernard King-Smith <[email-masked]>
Date: Wed, 7 Sep 2005 11:59:44 -0400

> Didn't know Solaris was a major OS from a volume perspective (might be
said for any Unix these days as there really > > are only two OS of any
real volume any longer).  :)  I will state that at this point in time, I am
not recommending to my > > company to do any work on IPoIB-CM and I plan to
push on spending resources on where there is significant value and > ROI
opportunities on the RDMA side of things not on making IP-based stacks work
better (we are already getting more > than good enough performance with the
existing specs implemented).
> In any case, Windows is really a third-party provider doing the work not
MS.  Does not really matter I suppose in the > end.

I don't include Windows as a user of IB because it is pretty much a
Ethernet only OS looking at its current market.

> I don't see the work as compelling but people are free to ignore me as
I'm just one voice within the IETF here.  I > personally believe the
emphasis should be on getting a solid RDMA infrastructure in place and
using that for ULP > communication than on making what is already a
reasonable IP over IB network stack more optimal.  If you go back > through
the history of interconnects, when one cannot demonstrate significant
benefit difference to customers or take too > long to get the value-add
right, then customers are fine with going with the evolutionary technology.
10 GbE RNIC prices > are dropping as are switch port costs.  While they may
not be on par with IB today, by the time it really matters to > customers,
the total solution costs which go well beyond the hardware may result in 10
GbE being just fine.

This is where technology and marketing get into trouble. If IB is not on
par, then 10 GigE is all people will use. Even though IB has other non-IP
uses where it excells, when you look at total cost, two networks are more
expensive than one. Hence, IPoIB just good enough means no IB for most
customers. I don't agree that 10 GigE can be everything to everyone, and IB
has a place. Make it cost effective, and they will use it. IP is not going
away in the foreseeable future, and many a person/company has proclaimed
the death of IP over some newer technology, only to end up in the memories
of time. The only networks that survived for any length of time supported
IP well (Token ring, FDDI, ATM), even if not always fully utilized by an
application set.

  10 GbE will > have LSO, LRO, whatever as well as RDMA and a much lower
cost to integrate management and associated ecosystems > into the data
center.  Spending resources on making IP communication better on IB is
simply a poor use of resources > with minimum end visible customer benefit
especially if some of the OS such as Windows already have the hooks in
place > to support RDMA ULP via Winsocks Direct.
>
> Mike_______________________________________________
> IPoverIB mailing list
> [email-masked]

Bernie King-Smith
IBM Corporation
Server Group
Cluster System Performance
[email-masked]    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES

"We are not responsible for the world we are born into, only for the world
we leave when we die.
So we have to accept what has gone before us and work to change the only
thing we can,
-- The Future." William Shatner

---

From: Michael Krause <[email-masked]>
Date: Wed, 07 Sep 2005 10:45:35 -0700

>Ethernet only OS looking at its current market.

The approach appears to be IB is provided via a third-party NDIS based 
driver.  Perfectly reasonable which has been used for many technologies 
including Ethernet.  This is what is delivered today by some companies.

>IP well (Token ring, FDDI, ATM), even if not always fully utilized by an

I don't see where IB is not on par.  On the contrary, many will contend 
that IB is superior in many ways.  The issue is whether there is a 
compelling reason to have a new spec defined for IP over IB so that it can 
compensate for the differences in performance between 10 GbE and a x4 IB 
2.5 GT/s raw.  Some contend this is required; perhaps I am alone in 
thinking it isn't.  There is a complete IP over IB spec suite today and 
people have implemented a large percentage of it and demonstrated it is 
interoperable.  Sounds like IB is therefore on par with Ethernet.  Given 10 
GbE Ethernet using PCI-X 265 can hit link rate without resorting to jumbo 
frames, there really isn't any reason why IB cannot hit its lower link rate 
using its slightly larger 2K MTU.  The issue is in the implementation of 
HCA which is where I contend is the better solution approach to the 
problem.  However, others contend they would rather spend resources 
defining / implementing a new spec rather than fixing the problem using / 
leveraging the approach already proven successful with Ethernet.  This is 
really ROI issue.  If people believe there is sufficient ROI, then they 
will implement it.  Others, and again perhaps I'm the only one, would 
rather see resources spent in getting the RDMA infrastructure completed and 
view the IP over IB work as complete, i.e. it met the goals set forth by 
the workgroup since the first BOF occurred several years ago.

As for Ethernet being everything to everyone, I think you have can go back 
to any number of pundits who will state that no matter what comes about, in 
the end it will be called Ethernet.  So, yes today's Ethernet isn't your 
"father's Ethernet" but it sill inter-operates, still ships hundreds of 
millions of units per year, will see 10 GbE ship in the 100K's in the next 
year and ramp to millions in the next 3-4 years (even by conservative 
estimates), and will still have the lowest cost of entry from a total 
customer cost perspective for many years to come.  This is why if IB is to 
make in roads, the focus needs to be on the RDMA ecosystem and not on 
tweaking some corner aspect of IP over IB.  10 GbE RNIC are real. They will 
ship next year with PCI-X 266 and PCIe points of attach (IB is only PCIe) 
so there isn't any advantage to IB any longer in that regard.  200ns 10 GbE 
with good port counts switch chips are real and at very reasonable 
prices.  Companies who know how to deliver very low-latency, high-bandwidth 
switch and NICs such as Quadrics and Myricom are creating dual-hardware 
models that will deliver 10 GbE with full support for RDMA and link-rate 
solutions.  These companies are hardly new to the HPC market - being the 
performance and volume leaders - and they have the ability to quickly 
translate their implementation prowess into credible offerings. Given their 
track records, they have immediate customer trust that will be easily 
leveraged for pilot and production deployments.  Given their known quality, 
this trust can be easily leveraged into the commercial market as they join 
established players like Cisco, Broadcom, etc.

In any case, I think the debate has played itself out and the virtual hum 
says to proceed.  So, I'll go quiet unless someone has a specific issue or 
wants to continue the debate for some other reason.  Mark me down as not 
supporting the hum which is likely just me in this regard.

Mike