Subject: RE: [rddp] Option or mandatory CRC in MPA

From: "Barron, Dwight" <[email-masked]>
Date: Wed, 9 Apr 2003 12:09:40 -0500

I think Jim's argument boils down to:

- There are scenarios where a software CRC may be deemed unnecessary overhead and should be optionally eliminated at both the RDDP and ULP layers.

To achieve this efficiency one must accept the following in their implementation:

- Discrimination of the appropriate scenario, eg. identify highly reliable private networks and then determine which traffic solely traverses those highly reliable private networks vs. that which doesn't
- Design the ULP error handling scenarios to deal with the subtle differences in TCP checksum vs. CRC-32

I contrast this with:

- The opportunity to provide all ULPs designed for RDDP the same data integrity for RDDP/TCP and RDDP/SCTP, with roughly equal performance impact for both implementations
- The architectural encouragement to move CRC-32 into hardware at one layer in the protocol stack and be done with it
- Many engineers are uncomfortable operating an RDDP/RDMA protocol without the strength of CRC-32
- The fact that when existing ULPs are recast for RDDP, they can define a negotiation to stay in their current sockets mode and operate without CRC overhead (per John Hufferd's comment)
- The fact that software CRC, while not trivially cheap, is not the performance killer it is perceived to be. Some internal tests indicate that systems of a vintage that are, ahem, challenged to keep up with wire speed on GbE can perform CRC at rates that would support wire speed. Modern systems can perform CRC at rates that would support multiple GbE interfaces

I believe the RDDP group highly values the input of the NFS team's requirements and look forward to reviewing them. However, I think the field experience will have to be tempered with the following forward-looking fudge factors; increase in frequency of errors due to bandwidth increases and quantification of the number of NFS errors that routinely escape detection in the field. Otherwise, how do we know they are not hovering just below some acceptable threshold today (similar to the TCP error escapes), only waiting to surface once better detection is available and/or higher bandwidths push the error rate.

Regards,
Dwight

> -----Original Message-----
> From: Williams, Jim [mailto:[email-masked]]
> Sent: Tuesday, April 08, 2003 7:17 PM
> To: [email-masked]'
> Subject: RE: [rddp] Option or mandatory CRC in MPA
> 
> 
> The argument for making the CRC optional
> in the RDDP over TCP mapping is the following.
> 
>   A.  A large percentage of usage will be on
>       highly reliable private networks where
>       the TCP checksum is more that sufficient
>       the achieve the desired level or reliability.
> 
>   B.  Optimizing the performance of software 
>       implementations is important, and computing
>       an unneeded CRC is too high a price to pay
>       for avoiding the need to negotiate an option.
> 
> 
> The input from NFS is important because it takes
> into account years of field experience.  If they
> determine that for most all usage, even on private
> networks, the TCP checksum is NOT adequate, then
> I would regard that as a refutation of point A.
> If they determine that for a large enough percentage
> of implementations the TCP checksum IS adequate,
> then I would regard that as support of point A.
> 
> Regarding point B, if one envisions that all 
> implementations of interest are hardware customized
> for RDDP, then the point does not carry much weight.
> However, I would argue that this may be too narrow
> a view for the IETF to take.
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Wed, 9 Apr 2003 18:04:57 -0600

Jim,

One of the reasons for having end-to-end error detection is to protect against errors that occur in intermediate devices (middle boxes and switches) on the network where the data is not protected by the layer 2 CRC. Such errors can occur on private networks as well as on public networks. Unfortunately, we've seen intermediate devices that introduce errors into the data stream.

Pat

---

From: Michael Krause <[email-masked]>
Date: Wed, 09 Apr 2003 17:44:09 -0700

At 06:04 PM 4/9/2003 -0600, [email-masked] wrote:
>Jim,
>
>One of the reasons for having end-to-end error detection is to protect 
>against errors that occur in intermediate devices (middle boxes and 
>switches) on the network where the data is not protected by the layer 2 
>CRC. Such errors can occur on private networks as well as on public 
>networks. Unfortunately, we've seen intermediate devices that introduce 
>errors into the data stream.

Another data point to consider is local I/O technology which is a much 
smaller fabric diameter than a data center and yet still has strong data 
integrity requirements:

- PCI-X 2.0 requires ECC protection

- PCI Express provides an end-to-end CRC which is optional - primary focus 
for optional was graphics which generally do not care about data integrity; 
for servers, we it is basically a MUST to remain consistent with PCI-X 2.0 
ECC protection coverage.

There was analysis done during the creation of these technologies focused 
on preventing intermediate device failures / corruption being undetected 
(as Pat points out) and we felt this was a worthwhile thing to include even 
though the fabric diameter is very small (to us, it isn't a function of 
fabric diameter but one of process technology getting so small that atomic 
hits are more of a problem than people realize).  In any case, the whole 
point was to plug as many potential areas of problems as possible without 
creating undue performance / cost burdens and these were the paths we chose 
and there is strong industry support about having this capability present.

Just something to consider as to what is being provided in new local I/O 
technology with the objective of preventing problems before they occur in 
customer solutions.  Having a mandatory ECRC via MPA or SCTP seems like the 
right thing to do and the software burden to support isn't that high to 
make this impractical - there are plenty of ways to implement this and 
deliver good performance independent of the type of endnode being used to 
generate the RDMA message.

Mike

---

From: John Hufferd <[email-masked]>
Date: Thu, 10 Apr 2003 17:59:04 -0700

As you probably know by now, I believe that software implementations are 
not what  we defined as a problem which needed to be solved.  Having said 
that, lets see where the software implementations might make sense.  First 
off, I categorically reject, out of hand, that a Storage Server 
(controller) will be implementing RDMA in software.  I know of no one that 
is even thinking about that.  So lets move off that topic, and look at the 
Host systems.

The point has been made, and I guess I can accept, that some Client 
systems will be "required" by an "IT Staff" to help the Server System that 
has HW RDMA.  That is, the Clients (desktops) will have to use RDMA mode 
software so that the load on the server is kept low.  I do not think one 
will see this as a large situation, but it may exist.

I do not believe that it is reasonable to talk about Servers running a 
software version of RDMA for all the reasons I stated in my last note.

So lets revisit the Client systems.  Project if you will, to the point in 
time that there are servers that have RDMA chips.  Lets say servers will 
start to have meaningful numbers of systems with RDMA chip (RNIC) sometime 
in 2004 (probably late 2004, but pick your own date since that will not 
change the argument much).  Now, pick the time that you think there will 
be OSes that will support  the RNIC via Socket Direct Protocol (SDP) -- 
which is a way to make RDMA transparent to socket users -- in numbers that 
matter.  Or pick the time that applications will be rewritten to use the 
RDMA features (without SDP),  with or without the Host OS directly 
supporting them (could be supported via the RNIC vendor's special SW). 

Now most of the servers application vendors that I know will not rewrite 
their software to work with RDMA, unless they have a very large market 
that supports the required RDMA software.  There are, of course, some 
bleeding edge vendors and end-users which will, but ask your self when 
your favorite application will do that.

So now lets look at the desktop again, and ask yourself when you think 
those OSes will support (in meaningful numbers) a software version of 
RDMA.

OK, now with what ever dates you have come up with, consider the 
following:

Desktop are shipping with 2-3Mhz this year, probably 1.5 - 2 times that 
next year or so, and even more than that the following year.  We are 
talking about 3-6 Mhz next year, and maybe 6 - 10 Mhz the following year. 
All this at your personal desktop.

For us to be debating the overhead of such a desktop when it is computing 
the CRC is, in my opinion, not reasonable. 

This situation is quite different on the servers, which want every MIP 
they can get in order to serve the desktops; they will need and require HW 
RDMA implementations.

So again, lets not attempt to optimize for the software version, which 
will only be on desktops which do not need the optimization.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Somesh Gupta" <[email-masked]>
Sent by: [email-masked]
04/10/2003 03:07 PM

        To:     <[email-masked]>
        cc: 

Mike,

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Michael Krause
> Sent: Thursday, April 10, 2003 1:43 PM
> To: [email-masked]
> At 12:55 PM 4/10/2003 -0700, Somesh Gupta wrote:
> >It is difficult for people to definitively say what will or
> >what will not happen in the marketplace. As an example, we
> >went thru all of these debates during iSCSI protocol development.
> >
> >As it turns out, bulk of the products are and may continue
> >to be "software based".
> >
> >1. Most initiators at this time are in the category and may
> >continue to be software based
> >
> >2. Many (if not majority) of targets are "software based" and
> >may continue to be.
>
>
> Given the iSCSI spec just became a standard, it is understandable that
> there is a predominance of soft implementations.  However, not
> that it is a
> standard, there are people finalizing their hardware implementations 
thus
> the current state is clearly subject to change.

  I am fairly certain there will continue to be a range of 
implementations.

>
>
> >Interesting, on the target side the processors used are not
> >these high end P-4s, but slower embedded processors with
> >smaller caches (32KB or whatever).
> >
> >In these instances, everything that is done (CRC and mandatory
> >markers even on receive side where they provide no benefit)
> >solely from the perspective of a protocol optimized for hardware
> >implementations is likely to have a significant performance
> >impact.
>
> Not clear.  Whether one walks memory to do a checksum calculation, a CRC
> calculation, or a copy operation, the net effect is the processor data
> cache is made hot with these cache lines.  Given applications
> typically ask
> for data once it has arrived, the fact that is in cache mitigates
> the cache
> load latency thus the subsequent performance impact.   I do not
> see this as
> creating a significant performance impact.  Do you have data to show 
that
> there is such an impact?

  I think this brings up a number of interesting points:

1. We should assume checksum is offloaded as fairly all NICs do
   it now.
2. There are a certain class of apps that do not touch the data
   but just shunt it from one place to another. A classic example
   is any kind of storage server. These are actually the ones
   that benefit from zero copy the most and may use means other
   than RDDP to get the benefit (i.e. handle SGLs)
3. Obviously there are many other apps that chew on the data.
   In those cases, there is just a whole range of factors that
   come into play. The rate at which data can be burst vs the
   apps consumption of data, the size of the data caches etc
   etc. In a number of cases, it does not matter how much
   network acceleration you provide - the processing of the data
   consumes bulk of the cycles anyway.

>
> >I think we should be very clear and explicit about whether
> >we allow software based products to play in this market on
> >an equal footing or not.
>
> I thought there was general consensus that the focus of RDMA is hardware
> acceleration and that it must be able to be implemented in
> software.  There
> were no requirements put forth that I'm aware of that require
> software and
> hardware to have an equal footing especially as some operations such as 
a
> RDMA Read will always outperform a software implementation as no
> processor
> interactions are required when implemented in hardware.

  The statement of requirements should be very explicit on this.
  We can all agree that hardware acceleration will provide higher
  performance under all cases, but that is not the point.

>
> >On one hand a protocol enabling software based products will
> >have a wider acceptability and shorter time to market/revenue.
> >It may also have a greater chance as a standards track document.
> >
> >On the other hand, a protocol focused on hardware based solns
> >will allow for greater optimization in silicon. There also seems
> >to be a greater meeting of minds in a certain group on this model.
>
> Mike

  Somesh