Subject: RE: [Ipoverib] Please read - proposed WG termination

From: Dror Goldenberg <[email-masked]>
Date: Fri, 2 Sep 2005 23:18:13 +0300

> From: Vivek Kashyap [mailto:[email-masked]]
> Sent: Friday, September 02, 2005 3:59 AM
> 
> On Thu, 1 Sep 2005, Michael Krause wrote:
> 
> > At 11:19 AM 9/1/2005, H.K. Jerry Chu wrote:
> >> [co-chair hat off]
> >> 
> >> ...
> >> <snip>

> >
> > Keep in mind that Ethernet is also doing other activities 
> to solve various 
> > issues - multi-path support to compensate for the spanning 
> tree limitations, 
> > conceptually similar virtual lanes are being investigated, 
> improved security, 
> > link-level congestion management, etc.  All of these 
> changes combined with de 
> > facto use of large send / receive-side scaling, etc. will 
> likely make the 
> > issue of 9KB rather moot except for bulk data transfers in 
> the WAN (even then 
> > the QoS issues of multiple flows across a common link will 
> likely limit the 
> > use of 9KB).
> >
> > In the end, IB HCA implementations can adopt the same 
> techniques as Ethernet 
> > and deliver the same performance as a connected model 
> without the overhead /

The features that you mentioned being added to Ethernet sound to me like
things that exist in IB for couple of years now. BTW, fast link rates weaken
the QoS argument regarding jumbo frames. Furthermore, in ipoib-cm,
jumboframes won't look like 9KB or 64KB frames on the wire. They will look
like 2KB or smaller packets interleaved with the rest of the traffic. So, I
don't see any QoS problem for ipoib-cm.
Reusing the existing Ethernet techniques, e.g. LSO and large receive. Maybe
possible, but it's going to take way more than just using the existing
ipoib-cm mechanisms in a pure software implementation.

> 
> however connected mode is already available to us in all 
> HCAs. Why not use it as it is ?

I strongly agree. Redeveloping HCA silicon takes a long time. Why
make people wait for that long period of time, if they can have it
TODAY ?

-Dror
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Fri, 02 Sep 2005 15:24:11 -0700

At 01:18 PM 9/2/2005, Dror Goldenberg wrote:

>things that exist in IB for couple of years now.

As one of the architects of IB, I'm well aware of what IB offers as well as 
Ethernet and where there are technology as well as business issues with 
both.  I'm also aware, as are many other people, that technology 
transitions can take much longer than most predict (I'm perhaps a bit more 
realistic in my time lines than other marketroids / analysts) thus 
customers view a technology problem from a different perspective including 
when it will really be a major issue as opposed to a minor 
deficiency.   Just because Ethernet lacks something today does not mean it 
will not have something when it is required.  Similarly, just because IB 
has something today does not mean that it will be a point of 
differentiation in the long-run.  All technologies live in this paradigm in 
the end.

>BTW, fast link rates weaken the QoS argument regarding jumbo frames. 
>Furthermore, in ipoib-cm, jumboframes won't look like 9KB or 64KB frames 
>on the wire. They will look like 2KB or smaller packets interleaved with 
>the rest of the traffic. So, I don't see any QoS problem for ipoib-cm.

If the implementation is brain dead and arbitration is only done at the 
message level rather than the packet level on a given VL, then there is a 
QoS issue.  As I noted before, jumbo frames have micro benchmark or 
selective workload benefits but due to their higher link residency can 
cause QoS concerns.  Large send off-load as well as IB connected do not 
suffer from such issues as they perform SAR at the link level.

>Reusing the existing Ethernet techniques, e.g. LSO and large receive. 
>Maybe possible, but it's going to take way more than just using the 
>existing ipoib-cm mechanisms in a pure software implementation.

Large send can be implemented entirely in software and local to the 
injection point making it rather trivial to accomplish whille reaping the 
performance benefits.  Large receive generally needs some hardware assist 
to work well but given a large percentage of the protocol off-load 
implementations rely upon firmware for various operations, it will vary as 
to whether there is a legitimate execution problem or not.  The same cannot 
be said for the OS infrastructure changes which is what I and a couple of 
others have touched upon.

>make people wait for that long period of time, if they can have it

The existence of a spec does not mean that people can have it today.  They 
might have a first step but that is not a solution that is viable.  It is 
also not clear whether there is more than one OS that really cares about 
this capability as well.  Certainly I/O gateways can effectively use the 
existing specs just fine.  Are there multiple OS that can support IB in 
connected mode that have network stacks that will work in a sufficiently 
general purpose way that will enable iWARP and IB?  Are there even multiple 
OS interested in this capability at all?  People have already noted that 
Linux is not set up to support all of this today.  Is there a time line 
when people could execute the spec in the OS - not just an add-in 
non-kernel.org module - that would demonstrate it delivers superior value 
compared with just using existing technologies?

Mike