Subject: [rddp] The Case Against Bufferless NICs

From: "Williams, Jim" <[email-masked]>
Date: Wed, 27 Nov 2002 12:31:18 -0500

By "bufferless", I am actually referring to NIC designs
with small packet buffers that are practical to integrate
on chip.  This is not a case against bufferless, or small buffer,
NICs per se, but a concern about overly specializing the protocol
definitions for this particular design point.  For a number
of reasons the bufferless NIC is a particular design point
of limited scope and one should be cautious about using
that as the driver for defining the RDDP protocol suite.

There are some fundamental problems and limitations
with the "bufferless" concept.  I will list a number
of these.

1.  TCP Compliance

      This applies only to TCP and not SCTP.
      The problem is with respect to how the NIC will
      handle non-aligned PDUs.  These become much
      more difficult to handle than the aligned case.
      In the particular case where second half of
      a PDU is receive before the first half, it must
      be buffered in the mean time.

      When buffering is exhausted, the NIC must either
      drop packets, or else pass unprocessed packets
      to the host and defer all processing to the
      host.

      Passing packets to the host for processing, while
      theoretically possible, is a real nightmare for
      vendors.  OS kernel space code is particularly difficult
      to write and debug.  It also requires significant
      porting effort for each OS to be supported.  And
      it tends to break with each new revision of the OS,
      requiring updates and modifications.

      Bottom line is that vendors selling bufferless
      NIC technology will most likely for valid 
      practical and business reasons, choose not to
      support the unaligned PDUs.  Vendors are quite
      reluctant to invest millions of dollars in
      developing and supporting capabilities that will
      never actually be used by customers.
      Although technically it may be a violation of
      some RFCs to only support the aligned case, it will
      most likely meet customer requirements.

      Although it may not be a bad thing to see
      TCP evolve (in a de facto sense) into an
      aligned protocol, it is an important architectural
      change that should be given due consideration.

2. Host Bus Speed

      In order for a bufferless NIC to be practical,
      it is necessary that the host bus (or more generally
      that path to host memory) must be able to keep up
      with the link.  This appears to be practical
      given current host IO bus technology and
      1Gb Ethernet links.  At 10Gb link speeds this
      becomes highly questionable.  It is possible
      that a specific host system with a robust
      memory system and IO system might be able
      to keep up.  However, a NIC vendor attempting
      to develop a more generic NIC that can plug
      into any number of different hosts and expect
      to work can't realistically depend on the host
      IO bus to be able to keep up with a 10Gb link.

      The more difficult case is not raw bandwidth, but
      rather the case with a succession of small packets.

      If the host bus can't keep up with the link, the
      it will become necessary for the NIC to drop
      large numbers of packets.  Insufficient buffering
      exists to allow the NIC to flow control the source
      and continue accepting packet in the mean time.

3.  Context Memory Bandwidth

      Assuming full offload of transport and RDDP protocols,
      the processing of each incoming packet require
      access to context associated with that connection.
      The exact size of the context is difficult to 
      predict as it is dependent on a number of design
      tradeoffs.  However experience has shown 256 Bytes
      to be a reasonable number.  Even if a smaller 128
      bytes is assumed, it can be seen that the
      context is considerably larger than a minimum
      size packet.

      If one assumes that for each received packet
      the associated context must be read from context
      memory, and later written back, then the context
      memory bandwidth is considerably larger than
      the payload memory bandwidth.

      If one assumes a modest 4096 supported connections,
      and 256 bytes context per connection, then a 1MB context
      memory is required.  For a more scalable NIC, these
      numbers could get much larger.  This precludes
      storing context on chip.

      In many cases, bandwidth to the context memory
      can be minimized by caching on chip, however this
      only works well when the number of simultaneous 
      active connections is small.  It is easy to create
      a test case where the number of simultaneous active
      connections is large, and the effective context cache
      hit rate is near zero.

      Given these considerations, the NIC designer is left
      with a choice:

      A:  Build a NIC that only works efficiently when the
          number of simultaneous active connections is small,
          and drops a large percentage of incoming packets
          when the number of simultaneous active connections
          gets large, and the average packet size is small.

      B:  Build a NIC with a large, high bandwidth, off chip
          context memory.  The bandwidth of this context memory
          is considerably larger than the bandwidth of an off chip
          payload buffer memory would have been.

      C:  Build a NIC with a large, high bandwidth, off chip
          packet buffer memory, as well as an off chip context
          memory.  In this case the combined bandwidth of 
          of the payload and context memory can be lower than
          the bandwidth of the context memory alone in case B.
          This is because the NIC need not support worst case
          line rate for minimum sized packets.  Instead, end
          to end flow control can be used to prevent or minimize
          packet buffer overflow and resulting dropped packets.

      All three design points, as well as compromises between
      than, may makes sense for certain applications.

Conclusion

      Bufferless NICs may make sense for certain applications.
      But only if the number of simultaneous active connections
      is small, or the average packet size is always much larger
      than the minimum packet size.

      Bufferless NICs require that the host bus can keep up
      with the link.  For the next few years, this will limit
      bufferless NICs to 1Gb link speeds and preclude 10Gb 
      implementations.

      Bufferless NICs running on a TCP transport will cause
      TCP to evolve into an PDU aligned transport, and 
      cause interoperability problems with non-aligning
      TCPs.  In theory there should be no interoperability
      problems, but in practice there will be. 

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Wed, 27 Nov 2002 11:08:40 -0800

>       to the host and defer all processing to the

The other option is to use host memory as a buffer pool for such cases and 
thus keep everything under the control of the NIC.  This applies to QP 
context, transport context, work queue elements, etc. which for some 
implementations may not exist entirely in the NIC all at the same 
time.  Many NIC designs (even for other transports) support such caching 
algorithms.  Why wouldn't these same techniques be used for the buffering 
issue?  It avoids all of this complexity with host interaction and only 
requires the NIC driver to allocate private memory which is something all 
drivers do today.

>       change that should be given due consideration.

Alignment is a nice thing but not a requirement.  Most vendors understand 
these issues and will comply with the TCP specifications and thus will 
interoperate in the desired fashion.  I don't see why this is an issue - 
clearly not architecture w.r.t. these protocols.

>       and continue accepting packet in the mean time.

This is not an architecture issue but a deployment and to some extent an 
implementation issue.  The local I/O technology is evolving to support 10 
Gbps NICs and by the time they come into existence, there will be multiple 
server, storage, etc. providers with sufficient local I/O bandwidth as well 
as memory bandwidth to keep up.  As we progress out in time, we see 
increased I/O bandwidth available - take a look at the simplest progression 
- PCI-X 1.0 (1 GB/s), PCI-X 266 (2 GB/s), PCI-X 533 (4 GB/s), PCI-X 1066 (8 
GB/s).  All of this will be available by the time 10 Gbps technology 
reaches volume thus allowing many implementations to keep up quite 
nicely.  As for the memory subsystems, these are improving as well and most 
solutions will again be able to keep up in feeding these slots / chips.

BTW, an examination of process technology as a whole might also be 
worthwhile: 300 mm fabs (250% larger than 200 mm wafers today) coming on 
line today with 130nm process.  There will be a relatively quick 
progression to 90nm, then 65nm, and then 45nm over the next 5-7 
years.  Given the process / wafer improvements that will occur, the cost of 
increasing the buffering on the NIC will rapidly drop in cost as this 
technology ramps through high volume thus it is not clear that the NIC's 
will be as bufferless as some have projected in the coming years.

>       context is considerably larger than a minimum

One can do this in a variety of ways with even 96 bytes being possible with 
some intelligence.

>       numbers could get much larger.  This precludes

That is an implementation choice and again is impacted directly by the 
process technology being used.

>       connections is large, and the effective context cache

True - there are many test cases under which this might occur.  However, 
there are many workloads and design points that are not that connection 
intensive thus many designs will have all of this on-chip and will then 
extend to provide host memory caching for those elements that periodically 
go off-chip.  There is always the option of having side memory to the chip 
as well.

>           gets large, and the average packet size is small.

Don't see why dropping is required here.

>           payload buffer memory would have been.

This is a function of the caching algorithms, resource layouts, and 
workloads - all an implementation choice in the end.

>       problems, but in practice there will be.

I don't agree with any of your conclusions or the assumptions you have 
made.  I don't see what any of this discussion has in terms of bearing on 
the RDDP architecture.  All of this is implementation in nature and does 
not point to a need to change anything in the current drafts.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 27 Nov 2002 15:06:32 -0600

Buffer-rich RNICs are a valid design option. This is
especially true if the RNIC is also supplying TOE
functionality. We should avoid solutions that penalize these
designs, but that is not the same as designing the protocol
on the assumption that all RNICs are buffer-rich.

For example, we do not require that the RNIC place payload
immediately (even though that is what a buffer-sparse RNIC
would do). We merely state that it is placeable, and define
when it must be placed (before any trailing completion).

Buffer-sparse RNICs are a valid design as well. There are
numerous deployments where a more minimal RNIC that provided
hardware optimization of aligned connections would be
adequate, and considerably more cost effective.

Keep in mind that unlike dropped packets, there will be many
connections that *never* lose alignment. A hardware solution
that "only" optimizes for this case will be of great value
for many, perhaps most, deployments.

A few other points on context bandwidth versus payload
bandwidth:

-   Only a small portion of the context is updated on a per
    packet basis. Far larger portions of the context are
    only required to identify the connection quickly.
    These do not update on a per-packet basis. This could
    cut your context bandwidth nearly in half.
    
-   In the highest bandwidth environments the Maximum
    Segment Size will be Ethernet-sized, rather than
    the theoretical minimum. This reduces the context
    to bandwidth ratio still further.

Taking both of those into account, I could see as little as
128 bytes of context memory bandwidth per Ethernet Frame.
The bandwith required for on-chip buffering would scale up
considerably from there.

Pre-buffering host bus access will be a requirement with
many designs, but not all. More importantly, it is a factor
of the relative rates of the external interface and the
internal host memory bus. It is not affected by the number
of connections. The amount of memory required to support
per-connection buffering is dependent on the number of
connections.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp