Subject: RE: [rddp] The Case Against Bufferless NICs

From: "Herbert, Howard C" <[email-masked]>
Date: Wed, 27 Nov 2002 17:52:26 -0800

I agree with Mike.  Jim's email seems to be trying to defend a particular (current?) implementation approach.  In the end, the marketplace will decide the success of bufferless NICs or any other implementation.  It seems we should focus our time on getting the "wire" protocols done rather than debating how difficult/complex a given implementation will be.  And we should not be using this process to legislate what implementations can or can not be built. 

Howard

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Wednesday, November 27, 2002 12:09 PM
To: [email-masked]'
Subject: Re: [rddp] The Case Against Bufferless NICs

At 12:31 PM 11/27/2002 -0500, Williams, Jim wrote:
>By "bufferless", I am actually referring to NIC designs
>with small packet buffers that are practical to integrate
>on chip.  This is not a case against bufferless, or small buffer,
>NICs per se, but a concern about overly specializing the protocol
>definitions for this particular design point.  For a number
>of reasons the bufferless NIC is a particular design point
>of limited scope and one should be cautious about using
>that as the driver for defining the RDDP protocol suite.
>
>There are some fundamental problems and limitations
>with the "bufferless" concept.  I will list a number
>of these.
>
>1.  TCP Compliance
>
>       This applies only to TCP and not SCTP.
>       The problem is with respect to how the NIC will
>       handle non-aligned PDUs.  These become much
>       more difficult to handle than the aligned case.
>       In the particular case where second half of
>       a PDU is receive before the first half, it must
>       be buffered in the mean time.
>
>       When buffering is exhausted, the NIC must either
>       drop packets, or else pass unprocessed packets
>       to the host and defer all processing to the
>       host.

The other option is to use host memory as a buffer pool for such cases and 
thus keep everything under the control of the NIC.  This applies to QP 
context, transport context, work queue elements, etc. which for some 
implementations may not exist entirely in the NIC all at the same 
time.  Many NIC designs (even for other transports) support such caching 
algorithms.  Why wouldn't these same techniques be used for the buffering 
issue?  It avoids all of this complexity with host interaction and only 
requires the NIC driver to allocate private memory which is something all 
drivers do today.

>       Passing packets to the host for processing, while
>       theoretically possible, is a real nightmare for
>       vendors.  OS kernel space code is particularly difficult
>       to write and debug.  It also requires significant
>       porting effort for each OS to be supported.  And
>       it tends to break with each new revision of the OS,
>       requiring updates and modifications.
>
>       Bottom line is that vendors selling bufferless
>       NIC technology will most likely for valid
>       practical and business reasons, choose not to
>       support the unaligned PDUs.  Vendors are quite
>       reluctant to invest millions of dollars in
>       developing and supporting capabilities that will
>       never actually be used by customers.
>       Although technically it may be a violation of
>       some RFCs to only support the aligned case, it will
>       most likely meet customer requirements.
>
>       Although it may not be a bad thing to see
>       TCP evolve (in a de facto sense) into an
>       aligned protocol, it is an important architectural
>       change that should be given due consideration.

Alignment is a nice thing but not a requirement.  Most vendors understand 
these issues and will comply with the TCP specifications and thus will 
interoperate in the desired fashion.  I don't see why this is an issue - 
clearly not architecture w.r.t. these protocols.

>2. Host Bus Speed
>
>       In order for a bufferless NIC to be practical,
>       it is necessary that the host bus (or more generally
>       that path to host memory) must be able to keep up
>       with the link.  This appears to be practical
>       given current host IO bus technology and
>       1Gb Ethernet links.  At 10Gb link speeds this
>       becomes highly questionable.  It is possible
>       that a specific host system with a robust
>       memory system and IO system might be able
>       to keep up.  However, a NIC vendor attempting
>       to develop a more generic NIC that can plug
>       into any number of different hosts and expect
>       to work can't realistically depend on the host
>       IO bus to be able to keep up with a 10Gb link.
>
>       The more difficult case is not raw bandwidth, but
>       rather the case with a succession of small packets.
>
>       If the host bus can't keep up with the link, the
>       it will become necessary for the NIC to drop
>       large numbers of packets.  Insufficient buffering
>       exists to allow the NIC to flow control the source
>       and continue accepting packet in the mean time.

This is not an architecture issue but a deployment and to some extent an 
implementation issue.  The local I/O technology is evolving to support 10 
Gbps NICs and by the time they come into existence, there will be multiple 
server, storage, etc. providers with sufficient local I/O bandwidth as well 
as memory bandwidth to keep up.  As we progress out in time, we see 
increased I/O bandwidth available - take a look at the simplest progression 
- PCI-X 1.0 (1 GB/s), PCI-X 266 (2 GB/s), PCI-X 533 (4 GB/s), PCI-X 1066 (8 
GB/s).  All of this will be available by the time 10 Gbps technology 
reaches volume thus allowing many implementations to keep up quite 
nicely.  As for the memory subsystems, these are improving as well and most 
solutions will again be able to keep up in feeding these slots / chips.

BTW, an examination of process technology as a whole might also be 
worthwhile: 300 mm fabs (250% larger than 200 mm wafers today) coming on 
line today with 130nm process.  There will be a relatively quick 
progression to 90nm, then 65nm, and then 45nm over the next 5-7 
years.  Given the process / wafer improvements that will occur, the cost of 
increasing the buffering on the NIC will rapidly drop in cost as this 
technology ramps through high volume thus it is not clear that the NIC's 
will be as bufferless as some have projected in the coming years.

>3.  Context Memory Bandwidth
>
>       Assuming full offload of transport and RDDP protocols,
>       the processing of each incoming packet require
>       access to context associated with that connection.
>       The exact size of the context is difficult to
>       predict as it is dependent on a number of design
>       tradeoffs.  However experience has shown 256 Bytes
>       to be a reasonable number.  Even if a smaller 128
>       bytes is assumed, it can be seen that the
>       context is considerably larger than a minimum
>       size packet.

One can do this in a variety of ways with even 96 bytes being possible with 
some intelligence.

>       If one assumes that for each received packet
>       the associated context must be read from context
>       memory, and later written back, then the context
>       memory bandwidth is considerably larger than
>       the payload memory bandwidth.
>
>       If one assumes a modest 4096 supported connections,
>       and 256 bytes context per connection, then a 1MB context
>       memory is required.  For a more scalable NIC, these
>       numbers could get much larger.  This precludes
>       storing context on chip.

That is an implementation choice and again is impacted directly by the 
process technology being used.

>       In many cases, bandwidth to the context memory
>       can be minimized by caching on chip, however this
>       only works well when the number of simultaneous
>       active connections is small.  It is easy to create
>       a test case where the number of simultaneous active
>       connections is large, and the effective context cache
>       hit rate is near zero.

True - there are many test cases under which this might occur.  However, 
there are many workloads and design points that are not that connection 
intensive thus many designs will have all of this on-chip and will then 
extend to provide host memory caching for those elements that periodically 
go off-chip.  There is always the option of having side memory to the chip 
as well.

>       Given these considerations, the NIC designer is left
>       with a choice:
>
>       A:  Build a NIC that only works efficiently when the
>           number of simultaneous active connections is small,
>           and drops a large percentage of incoming packets
>           when the number of simultaneous active connections
>           gets large, and the average packet size is small.

Don't see why dropping is required here.

>       B:  Build a NIC with a large, high bandwidth, off chip
>           context memory.  The bandwidth of this context memory
>           is considerably larger than the bandwidth of an off chip
>           payload buffer memory would have been.

This is a function of the caching algorithms, resource layouts, and 
workloads - all an implementation choice in the end.

>       C:  Build a NIC with a large, high bandwidth, off chip
>           packet buffer memory, as well as an off chip context
>           memory.  In this case the combined bandwidth of
>           of the payload and context memory can be lower than
>           the bandwidth of the context memory alone in case B.
>           This is because the NIC need not support worst case
>           line rate for minimum sized packets.  Instead, end
>           to end flow control can be used to prevent or minimize
>           packet buffer overflow and resulting dropped packets.
>
>       All three design points, as well as compromises between
>       than, may makes sense for certain applications.
>
>Conclusion
>
>       Bufferless NICs may make sense for certain applications.
>       But only if the number of simultaneous active connections
>       is small, or the average packet size is always much larger
>       than the minimum packet size.
>
>       Bufferless NICs require that the host bus can keep up
>       with the link.  For the next few years, this will limit
>       bufferless NICs to 1Gb link speeds and preclude 10Gb
>       implementations.
>
>       Bufferless NICs running on a TCP transport will cause
>       TCP to evolve into an PDU aligned transport, and
>       cause interoperability problems with non-aligning
>       TCPs.  In theory there should be no interoperability
>       problems, but in practice there will be.

I don't agree with any of your conclusions or the assumptions you have 
made.  I don't see what any of this discussion has in terms of bearing on 
the RDDP architecture.  All of this is implementation in nature and does 
not point to a need to change anything in the current drafts.

Mike

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 27 Nov 2002 19:27:32 -0800

I agree that the wire protocols should be defined in such a way
that either implementation is feasible.

Jim's note appears to advocate a certain implementation approach.
Advocating is fine, but I fail to see if any changes are being proposed to
the WG drafts.  If so, please clarify.

It is also unclear to me if Jim is advocating that all data should be
routed through the NIC buffers simply because NIC memory is needed
for other reasons (which itself is open to question of course).  I think
cut-through routing is still a desirable goal for the protocol specs regardless
of the amount of NIC memory being present.  At a minimum, I'd assume
copying through the NIC memory increases power consumption among
other things.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
[email-masked]

----- Original Message -----
From: "Herbert, Howard C" <[email-masked]>
To: "Michael Krause" <[email-masked]>; <[email-masked]>
Sent: Wednesday, November 27, 2002 5:52 PM
Subject: RE: [rddp] The Case Against Bufferless NICs

>
> I agree with Mike.  Jim's email seems to be trying to defend a particular (current?) implementation
approach.  In the end, the marketplace will decide the success of bufferless NICs or any other implementation.
It seems we should focus our time on getting the "wire" protocols done rather than debating how
difficult/complex a given implementation will be.  And we should not be using this process to legislate what
implementations can or can not be built.