Subject: Re: [rddp] The Case Against Bufferless NICs

From: "John Hufferd" <[email-masked]>
Date: Wed, 27 Nov 2002 12:11:06 -0800

Jim,
In what way would you adjust your comments if you also took into account,
that these same RNICs are suppose to be able to use their TOEs for normal
NIC processing?  That is, not only can the RNIC support RDMA, it is also
suppose to handle regular TCP/IP processing via its TOE, and perform the
normal acceleration function such as TCP checksum processing, etc.  I would
expect in that case that on chip or on card reordering buffers would be
required.

What I have been attempting to determine is if an RNIC is going to support
this normal (accelerated) NIC operation, will there be any savings in
chip/card buffers especially in the early days when there is few RDMA
applications and many normal socket applications using the RNIC and its
TOE.  What are your thoughts?

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Williams, Jim" <[email-masked]>@ietf.org on 11/27/2002 09:31:18
AM

Sent by:    [email-masked]

To:    "[email-masked]'" <[email-masked]>
cc:
Subject:    [rddp] The Case Against Bufferless NICs

By "bufferless", I am actually referring to NIC designs
with small packet buffers that are practical to integrate
on chip.  This is not a case against bufferless, or small buffer,
NICs per se, but a concern about overly specializing the protocol
definitions for this particular design point.  For a number
of reasons the bufferless NIC is a particular design point
of limited scope and one should be cautious about using
that as the driver for defining the RDDP protocol suite.

There are some fundamental problems and limitations
with the "bufferless" concept.  I will list a number
of these.

1.  TCP Compliance

      This applies only to TCP and not SCTP.
      The problem is with respect to how the NIC will
      handle non-aligned PDUs.  These become much
      more difficult to handle than the aligned case.
      In the particular case where second half of
      a PDU is receive before the first half, it must
      be buffered in the mean time.

      When buffering is exhausted, the NIC must either
      drop packets, or else pass unprocessed packets
      to the host and defer all processing to the
      host.

      Passing packets to the host for processing, while
      theoretically possible, is a real nightmare for
      vendors.  OS kernel space code is particularly difficult
      to write and debug.  It also requires significant
      porting effort for each OS to be supported.  And
      it tends to break with each new revision of the OS,
      requiring updates and modifications.

      Bottom line is that vendors selling bufferless
      NIC technology will most likely for valid
      practical and business reasons, choose not to
      support the unaligned PDUs.  Vendors are quite
      reluctant to invest millions of dollars in
      developing and supporting capabilities that will
      never actually be used by customers.
      Although technically it may be a violation of
      some RFCs to only support the aligned case, it will
      most likely meet customer requirements.

      Although it may not be a bad thing to see
      TCP evolve (in a de facto sense) into an
      aligned protocol, it is an important architectural
      change that should be given due consideration.

2. Host Bus Speed

      In order for a bufferless NIC to be practical,
      it is necessary that the host bus (or more generally
      that path to host memory) must be able to keep up
      with the link.  This appears to be practical
      given current host IO bus technology and
      1Gb Ethernet links.  At 10Gb link speeds this
      becomes highly questionable.  It is possible
      that a specific host system with a robust
      memory system and IO system might be able
      to keep up.  However, a NIC vendor attempting
      to develop a more generic NIC that can plug
      into any number of different hosts and expect
      to work can't realistically depend on the host
      IO bus to be able to keep up with a 10Gb link.

      The more difficult case is not raw bandwidth, but
      rather the case with a succession of small packets.

      If the host bus can't keep up with the link, the
      it will become necessary for the NIC to drop
      large numbers of packets.  Insufficient buffering
      exists to allow the NIC to flow control the source
      and continue accepting packet in the mean time.

3.  Context Memory Bandwidth

      Assuming full offload of transport and RDDP protocols,
      the processing of each incoming packet require
      access to context associated with that connection.
      The exact size of the context is difficult to
      predict as it is dependent on a number of design
      tradeoffs.  However experience has shown 256 Bytes
      to be a reasonable number.  Even if a smaller 128
      bytes is assumed, it can be seen that the
      context is considerably larger than a minimum
      size packet.

      If one assumes that for each received packet
      the associated context must be read from context
      memory, and later written back, then the context
      memory bandwidth is considerably larger than
      the payload memory bandwidth.

      If one assumes a modest 4096 supported connections,
      and 256 bytes context per connection, then a 1MB context
      memory is required.  For a more scalable NIC, these
      numbers could get much larger.  This precludes
      storing context on chip.

      In many cases, bandwidth to the context memory
      can be minimized by caching on chip, however this
      only works well when the number of simultaneous
      active connections is small.  It is easy to create
      a test case where the number of simultaneous active
      connections is large, and the effective context cache
      hit rate is near zero.

      Given these considerations, the NIC designer is left
      with a choice:

      A:  Build a NIC that only works efficiently when the
          number of simultaneous active connections is small,
          and drops a large percentage of incoming packets
          when the number of simultaneous active connections
          gets large, and the average packet size is small.

      B:  Build a NIC with a large, high bandwidth, off chip
          context memory.  The bandwidth of this context memory
          is considerably larger than the bandwidth of an off chip
          payload buffer memory would have been.

      C:  Build a NIC with a large, high bandwidth, off chip
          packet buffer memory, as well as an off chip context
          memory.  In this case the combined bandwidth of
          of the payload and context memory can be lower than
          the bandwidth of the context memory alone in case B.
          This is because the NIC need not support worst case
          line rate for minimum sized packets.  Instead, end
          to end flow control can be used to prevent or minimize
          packet buffer overflow and resulting dropped packets.

      All three design points, as well as compromises between
      than, may makes sense for certain applications.

Conclusion

      Bufferless NICs may make sense for certain applications.
      But only if the number of simultaneous active connections
      is small, or the average packet size is always much larger
      than the minimum packet size.

      Bufferless NICs require that the host bus can keep up
      with the link.  For the next few years, this will limit
      bufferless NICs to 1Gb link speeds and preclude 10Gb
      implementations.

      Bufferless NICs running on a TCP transport will cause
      TCP to evolve into an PDU aligned transport, and
      cause interoperability problems with non-aligning
      TCPs.  In theory there should be no interoperability
      problems, but in practice there will be.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Uri Elzur" <[email-masked]>
Date: Wed, 27 Nov 2002 19:34:43 -0800

Regardless of the design point every NIC vendor chooses, I don't see any architectural reason to limit the use of so-called bufferless NICs. As a matter of fact, I suspect you were not spending that much energy trying to shut it down, unless you thought this is actually POSSIBLE (and compliant)!

The drafts on the table (mpa included), thus far, have not explicitly assumed anything about the NIC buffer level. The only relevant assumption made is that alignment helps the receiver - I hope you agree and that as the speed of the network goes higher the importance of handling o-o-o grows. Actually if you're not ready to make theses assumptions, the cost and complexity of 10G NICs or faster grows significantly. Maybe to the point that it becomes  hard to envision wide deployment.

You keep bringing buffering up, as if this is an architectural issue where in reality it is an implementation issue that changes with time. All parameters are changing with time as higher density silicon is coming on line, memory technology morphs, network speeds change and IO bus technologies climb a notch every 2 years. 

more comments below [UE]

Uri
-----Original Message-----
From: Williams, Jim [mailto:[email-masked]]
Sent: Wednesday, November 27, 2002 9:31 AM
To: [email-masked]'
Subject: [rddp] The Case Against Bufferless NICs

1.  TCP Compliance

[UE] But what if  actually you get alignment most of the time? and why is this different from a buffered NIC who has to buffer everything? One can dedicate some amount of buffering for this case. This buffer can be on-chip or off-chip. The NIC needs to handle both cases.
[UE]

[UE] There are more implementation choices, but the discussion here ought to focus on complying to a wire protocol only. 
[UE]

[UE] good case why not do it. But why is that related to a wire protocol discussion?

[UE] the test for this is wire protocol compatibility. As long as the implementation is compliant, it should be allowed. The business related discussion has no place on this reflector. 
[UE]

[UE] MPA is an Upper Layer Protocol on top of TCP. It has nothing to do with any TCP change you are trying again and again to elude to. 
[UE]
2. Host Bus Speed

[UE] this analysis seems to deal with today's IO bus and doesn't cover in enough details the imminent and inevitable migration of these technologies. PCI-X2 266 provides ~17Gbps and is avail now. It is easy to envision that PCI-X 533MHz be there in time for 10 Gbps NICs....
[UE]

[UE] don't buy into the assumption that the whole world is moving and the host bus is still 16 bit ISA
[UE]
3.  Context Memory Bandwidth

Conclusion

      
[UE] I disagree with any of the conclusions you draw. The goal of this WG is to design a protocol that can last for several technology generations. I find your conclusions as significantly limiting the applicability and the realm of implementations. 
[UE]