Subject: RE: [rddp] A different thought on CRCs

From: [email-masked]
Date: Fri, 13 Dec 2002 16:40:24 -0700

Jim,

I reviewed the paper. It has taken a bit for me to post my thoughts on it, partly because my hard drive crashed Thanksgiving morning and by the time my system got fixed I had lots of catch-up work to do.

First some comments about the particular data in the Stone-Partridge paper followed by comments about the general problem of protecting data on RNICs and similar hardware.

Note that the second paragraph in prior work makes an inaccurate statement about CRC error protection. The 32-bit CRCs protect against all 2-bit errors up to the cycle length of the CRC which is 2^32 bits. The CRC used by most link layers (including Ethernet) was also chosen to protect against all 3-bit errors up to well above the packet sizes used in Ethernet, Fibre Channel, etc. A CRC that failed to detect a 2 bit error 2048 bits apart as the paper suggests would have a pretty poor undetected error rate. (This isn't particularly relevant to this discussion but I hate to see misinformation spread.)

The data in the paper on errors created by host adapters is all based on measurements of a 10BASE2 based dorm network. This is a true CSMA/CD Ethernet (that is, it is shared media where collisions are occurring). Today, a pretty small portion of the nodes are running on coaxial Ethernets and switches can be bought for the same price as hubs. Therefore, much of the network is running full duplex.  When running on a shared media Ethernet, there is interaction between the transmit and receive sides of the chip. If the design is not done very carefully, there can be times when the transmit or receive FIFO gets confused and corrupts part of the packet while it is not protected by the link CRC. This isn't inevitable; I know of designs where the clock domains are carefully thought out and proper synchronization techniques are used between the domains, that don't have this problem. But I have also seen designs that weren't as careful that had problems similar to what was described in
  the paper.  

Note the tendency in the paper for the end-host hardware flaws in 4.3.2 to be specific to certain hardware implementations. At the risk of being politically incorrect, I'll also point out that the failures were on adapters at the very low end of the market - not the class of equipment I would expect for RDDP.

Much of the Ethernet today runs full duplex and doesn't have as much exposure to poor design. As far as I know, no one is making a 1 Gbit/s Ethernet repeater (though the standard allows for CSMA/CD operation at this speed). The 10 Gbit/s Ethernet standard only supports full duplex. Full duplex designs don't have the interaction between the transmit and receive paths so it is easier to avoid FIFO problems. 

What we are facing today is increased backplane speeds, lower voltage levels and shrinking memory cell size. Adding host to host CRC is not necessarily a suitable way to deal with these problems. Increased backplane speed and lower voltage levels increase the suceptability to noise hits. Both PCI-X 2.0 and PCI Express deal with this by adding error protection to the bus. PCI Express adds CRC protection and PCI-X 2.0 uses an error correcting code. Reduced memory cell size means that soft memory error probability increases - an alpha partical hits one or two cells and changes the contents. In memory, one copes with this by adding parity or error correction bits and by laying out the memory so that the error protection still works when two adjacent cells are corrupted. 

Reasons CRC is not necessarily the right answer for protection against backplane or RNIC errors:
CRC is very low cost to implement in hardware processing data as it is transfered. It just takes a 32 bit register plus a few handfuls of XOR gates. It takes a lot of processor cycles to run CRCs on data in host memory so it isn't likely to get used for that application. There are some applications where it is worthwhile to agree on chunk size and to pay the processing cost for host memory CRC calculation, but it isn't the general case.

To protect data with CRC end-to-end, we have to agree ahead of time on what chunks will be transfered because the CRC has to be computed over a set of bytes. That limits its flexibility for full end-to-end protection of the data.

Even if the data chunks are protected with end-to-end CRC, there could be an error in passing address information or in control/context information in RNIC memory that causes the wrong chunk of data to be accessed. The chunk will have a valid CRC but will be the wrong data.

If an alpha particle corrupts data in host memory, what one needs is error correction rather than error detection to recover. CRC error detection is good for places where the data can be resent to recover from an error.

Therefore, it is better to use CRC RNIC to RNIC and to use other appropriate mechanisms to protect the data from host memory to the point where the DDP CRC is applied. 

Regards,
Pat

-----Original Message-----
From: Williams, Jim [mailto:[email-masked]]
Sent: Friday, November 22, 2002 12:49 PM
To: [email-masked]'
Subject: [rddp] A different thought on CRCs

The paper at http://citeseer.nj.nec.com/stone00when.html
is pretty sobering.  Worth reading with respect to 
better understanding CRC design tradeoffs.

It raises the question in my mind of how best to 
define the CRC to get maximum possible coverage
of RNIC internal failures as well as RNIC firmware
and hardware logical bugs.

Is there a way to move CRC computation and checking
as close to the host side as possible so as to get
maximum coverage of NIC internals?

I don't have a proposal or answer, just raising
a question.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 13 Dec 2002 15:59:10 -0800

Pat,

One of the conclusion I would draw from your analysis is that
we do not need CRCs at the DDP layer. The links are protected
by the ethernet CRC. And if routers, switches and NICs are
well engineered (and I hope so), then this whole issue goes away.

But wait, there are those messy middle-boxes which not only mess
up alignment in unpredictable ways, but also have a high probability
of introducing data corruption.

On the other hand, having two CRCs (header & data instead of one),
does not add any complexity at all. It does enable significant
implementation flexibility, and can add additional protection
(again depending on the implementation). And it does not discriminate
against any particular implementation.

Somesh

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> [email-masked]
> Sent: Friday, December 13, 2002 3:40 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [rddp] A different thought on CRCs
> 
> 
> Jim,
> 
> I reviewed the paper. It has taken a bit for me to post my 
> thoughts on it, partly because my hard drive crashed Thanksgiving 
> morning and by the time my system got fixed I had lots of 
> catch-up work to do.
> 
> First some comments about the particular data in the 
> Stone-Partridge paper followed by comments about the general 
> problem of protecting data on RNICs and similar hardware.
> 
> Note that the second paragraph in prior work makes an inaccurate 
> statement about CRC error protection. The 32-bit CRCs protect 
> against all 2-bit errors up to the cycle length of the CRC which 
> is 2^32 bits. The CRC used by most link layers (including 
> Ethernet) was also chosen to protect against all 3-bit errors up 
> to well above the packet sizes used in Ethernet, Fibre Channel, 
> etc. A CRC that failed to detect a 2 bit error 2048 bits apart as 
> the paper suggests would have a pretty poor undetected error 
> rate. (This isn't particularly relevant to this discussion but I 
> hate to see misinformation spread.)
> 
> The data in the paper on errors created by host adapters is all 
> based on measurements of a 10BASE2 based dorm network. This is a 
> true CSMA/CD Ethernet (that is, it is shared media where 
> collisions are occurring). Today, a pretty small portion of the 
> nodes are running on coaxial Ethernets and switches can be bought 
> for the same price as hubs. Therefore, much of the network is 
> running full duplex.  When running on a shared media Ethernet, 
> there is interaction between the transmit and receive sides of 
> the chip. If the design is not done very carefully, there can be 
> times when the transmit or receive FIFO gets confused and 
> corrupts part of the packet while it is not protected by the link 
> CRC. This isn't inevitable; I know of designs where the clock 
> domains are carefully thought out and proper synchronization 
> techniques are used between the domains, that don't have this 
> problem. But I have also seen designs that weren't as careful 
> that had problems similar to what was described in
>   the paper.  
> 
> Note the tendency in the paper for the end-host hardware flaws in 
> 4.3.2 to be specific to certain hardware implementations. At the 
> risk of being politically incorrect, I'll also point out that the 
> failures were on adapters at the very low end of the market - not 
> the class of equipment I would expect for RDDP.
> 
> Much of the Ethernet today runs full duplex and doesn't have as 
> much exposure to poor design. As far as I know, no one is making 
> a 1 Gbit/s Ethernet repeater (though the standard allows for 
> CSMA/CD operation at this speed). The 10 Gbit/s Ethernet standard 
> only supports full duplex. Full duplex designs don't have the 
> interaction between the transmit and receive paths so it is 
> easier to avoid FIFO problems. 
> 
> What we are facing today is increased backplane speeds, lower 
> voltage levels and shrinking memory cell size. Adding host to 
> host CRC is not necessarily a suitable way to deal with these 
> problems. Increased backplane speed and lower voltage levels 
> increase the suceptability to noise hits. Both PCI-X 2.0 and PCI 
> Express deal with this by adding error protection to the bus. PCI 
> Express adds CRC protection and PCI-X 2.0 uses an error 
> correcting code. Reduced memory cell size means that soft memory 
> error probability increases - an alpha partical hits one or two 
> cells and changes the contents. In memory, one copes with this by 
> adding parity or error correction bits and by laying out the 
> memory so that the error protection still works when two adjacent 
> cells are corrupted. 
> 
> CRC is very low cost to implement in hardware processing data as 
> it is transfered. It just takes a 32 bit register plus a few 
> handfuls of XOR gates. It takes a lot of processor cycles to run 
> CRCs on data in host memory so it isn't likely to get used for 
> that application. There are some applications where it is 
> worthwhile to agree on chunk size and to pay the processing cost 
> for host memory CRC calculation, but it isn't the general case.
> 
> To protect data with CRC end-to-end, we have to agree ahead of 
> time on what chunks will be transfered because the CRC has to be 
> computed over a set of bytes. That limits its flexibility for 
> full end-to-end protection of the data.
> 
> Even if the data chunks are protected with end-to-end CRC, there 
> could be an error in passing address information or in 
> control/context information in RNIC memory that causes the wrong 
> chunk of data to be accessed. The chunk will have a valid CRC but 
> will be the wrong data.
> 
> If an alpha particle corrupts data in host memory, what one needs 
> is error correction rather than error detection to recover. CRC 
> error detection is good for places where the data can be resent 
> to recover from an error.
> 
> Therefore, it is better to use CRC RNIC to RNIC and to use other 
> appropriate mechanisms to protect the data from host memory to 
> the point where the DDP CRC is applied. 
> 
> Regards,
> Pat
>

---

From: [email-masked]
Date: Tue, 17 Dec 2002 12:43:00 -0500

WG chair hat off ...

> -   Most DDP senders will send originate the data aligned. It is
>     easily implemented in any custom stack and/or RNIC. The only
>     exceptions will be user-mode code written on top of pre-existing
>     TCP/IP stacks. These will mostly be clients. The fact that the
>     TCP/IP stack has not been modified to support alignment would
>     be indicative that *most* of the traffic for this host in *not*
>     DDP enabled. If there is a substantial amount of DDP traffic
>     the DDP/TCP interface will be optimized and/or an RNIC will
>     be installed.

That assumption may be questionable in an environment with a lot of
small clients talking to a large server (e.g., a fileserver).  While
each client may only use DDP for a small portion of its outbound traffic,
most of the traffic inbound to the fileserver may be DDP and unaligned,
based on the clients doing the fileserver a favor by inserting the DDP
commands, but being unable to adjust the alignment behavior of their
existing TCP stacks.  In this environment, the clients don't care
whether the fileserver sends aligned or not (or even DDP or not), as
the existing stacks will probably force them to pay a copy.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Tue, 17 Dec 2002 16:48:43 -0800

Caitlin,

Seems like you have the answer to the 1CRC vs. 2CRC 
question yourself.

RDDP is chartered(I believe) to provide equal opportunity 
to TCP and SCTP mapping. Distill out your bias for SCTP
from your arguments (temporarily :-)). Since we are 
talking about TCP mapping, *pretend* that SCTP does not
exist and will not be the next world order.

Don't drag me into SCTP vs. TCP arguments :-)

-Shridhar Mukund

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Monday, December 16, 2002 9:56 PM
> To: [email-masked]
> Cc: Somesh Gupta; Mukund, Shridhar
> Shridhar Mukund wrote:
> > Caitlin,
> > > If you have a scenario where DDP would be deployed and
> > > alignment destroying middle boxes would be common, whether
> > > immediate or delayed, then I could see adding the second
> > > CRC. Otherwise, why not stick with what is already proposed
> > > and *is* the best solution for virtually every envisioned
> > > deployment?
> > 
> > Well, there you go. Other than the fact that the 1CRC 
> > argument is really a link-level digest(implementation wise) 
> > pretending to be a DDP+(MPA) level digest, it is further
> > predicated on the *alignment* requirement!
> > 
> > Do you really buy the alignment argument? I may not have a
> > problem with *some implementations* choosing to optimize 
> > around that assumption for *some specific* applications. 
> > But to base an IETF RDMA-over-TCP protocol spec. around 
> > that assumption is preposterous.
> > 
> Even looking ahead, I see non-allignment to be an exception.
> One that must be dealt with for completeness, but not an issue
> that will materially affect performance.
> 
> This is based upon two assumptions. If either is false, or even
> suspect, I would fully support using split CRCs (because unlike
> some of the critics, I agree that it is *very* feasible).
> 
> Those assumptions:
> 
> -   Terminating and re-originating a TCP stream is work. Doing it
>     for an unknown application will not be done routinely. It will
>     happen, but the only common reason for doing so will be
>     to gateway between IP networks with radically different
>     communications characteristics (round trip time, MTU,
>     latency or some combination thereof). In other words,
>     re-originating an unknown TCP stream is most likely when
>     normal end-to-end flow control would get confused.
>     
>     Most connections will *not* cross such a barrier, although
>     some clients will cross it virtually all of the time.
>     
>     No server would place itself on the other side of such
>     a barrier from any substantial portion of its clientele.
>     
> The most common meddlesome middlebox once we break out of the pure
> data-center environment will be a firewall/NAT. The IP and TCP headers
> get munged, but IP datagrams are passed through on a one-to-one basis.
> Alignment will not be broken (applications that pass IP addresses in
> the payload, are a different but very well known problem).
> 
> If I were making a data-center *only* assumption, I would be totally
> opposed to markers. In a data-center-only deployment there is no
> justification for using TCP rather than SCTP. A data center has
> no excuse for not fixing middleboxes without SCTP support.
> 
> It is the "second generation" of RDMA applications that justify the
> use of TCP: those where home clients access content from 
> remote servers
> (or use remote storage services). While these will be slow to develop,
> there can be major benefits of RDMA for these applications. 
> Having worked
> consumer electronics, I am quite confident in predicting that 
> many of the
> SCTP unfriendly middleboxes that are in place today will 
> still be there
> in ten years. Lack of OS support doesn't strike me as much of 
> issue, by
> the time the applications are ready to RDMA, SCTP will be supported in
> all OSs. A turnkey box that already does all that is required - those
> are left in place until they stop working. Which can be a 
> very long time.
>