Subject: RE: [rddp] A different thought on CRCs

From: [email-masked]
Date: Fri, 13 Dec 2002 18:30:01 -0700

Somesh,

I expect RDMA NICs to be relatively high-end well-engineered devices. Once the packet is out on the network, it is pretty hard to control which routers, switches and middle boxes it goes through. That is why we need CRC protection from RNIC to RNIC.

Two CRCs does add complexity and it doesn't provide any significant gain. It doesn't provide for host to host protection - one would need message CRC rather than segment CRC to do that. (See the message from Mike Krause.)

Also, since the CRC is provided by MPA, it would also be pretty strange layering as it requires MPA to make an assumption about what part of the segment it is passed is DDP header and what part is DDP data. 

Pat

-----Original Message-----
From: Somesh Gupta [mailto:[email-masked]]
Sent: Friday, December 13, 2002 3:59 PM
To: Rddp
Subject: RE: [rddp] A different thought on CRCs

Pat,

One of the conclusion I would draw from your analysis is that
we do not need CRCs at the DDP layer. The links are protected
by the ethernet CRC. And if routers, switches and NICs are
well engineered (and I hope so), then this whole issue goes away.

But wait, there are those messy middle-boxes which not only mess
up alignment in unpredictable ways, but also have a high probability
of introducing data corruption.

On the other hand, having two CRCs (header & data instead of one),
does not add any complexity at all. It does enable significant
implementation flexibility, and can add additional protection
(again depending on the implementation). And it does not discriminate
against any particular implementation.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> [email-masked]
> Sent: Friday, December 13, 2002 3:40 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [rddp] A different thought on CRCs
> 
> 
> Jim,
> 
> I reviewed the paper. It has taken a bit for me to post my 
> thoughts on it, partly because my hard drive crashed Thanksgiving 
> morning and by the time my system got fixed I had lots of 
> catch-up work to do.
> 
> First some comments about the particular data in the 
> Stone-Partridge paper followed by comments about the general 
> problem of protecting data on RNICs and similar hardware.
> 
> Note that the second paragraph in prior work makes an inaccurate 
> statement about CRC error protection. The 32-bit CRCs protect 
> against all 2-bit errors up to the cycle length of the CRC which 
> is 2^32 bits. The CRC used by most link layers (including 
> Ethernet) was also chosen to protect against all 3-bit errors up 
> to well above the packet sizes used in Ethernet, Fibre Channel, 
> etc. A CRC that failed to detect a 2 bit error 2048 bits apart as 
> the paper suggests would have a pretty poor undetected error 
> rate. (This isn't particularly relevant to this discussion but I 
> hate to see misinformation spread.)
> 
> The data in the paper on errors created by host adapters is all 
> based on measurements of a 10BASE2 based dorm network. This is a 
> true CSMA/CD Ethernet (that is, it is shared media where 
> collisions are occurring). Today, a pretty small portion of the 
> nodes are running on coaxial Ethernets and switches can be bought 
> for the same price as hubs. Therefore, much of the network is 
> running full duplex.  When running on a shared media Ethernet, 
> there is interaction between the transmit and receive sides of 
> the chip. If the design is not done very carefully, there can be 
> times when the transmit or receive FIFO gets confused and 
> corrupts part of the packet while it is not protected by the link 
> CRC. This isn't inevitable; I know of designs where the clock 
> domains are carefully thought out and proper synchronization 
> techniques are used between the domains, that don't have this 
> problem. But I have also seen designs that weren't as careful 
> that had problems similar to what was described in
>   the paper.  
> 
> Note the tendency in the paper for the end-host hardware flaws in 
> 4.3.2 to be specific to certain hardware implementations. At the 
> risk of being politically incorrect, I'll also point out that the 
> failures were on adapters at the very low end of the market - not 
> the class of equipment I would expect for RDDP.
> 
> Much of the Ethernet today runs full duplex and doesn't have as 
> much exposure to poor design. As far as I know, no one is making 
> a 1 Gbit/s Ethernet repeater (though the standard allows for 
> CSMA/CD operation at this speed). The 10 Gbit/s Ethernet standard 
> only supports full duplex. Full duplex designs don't have the 
> interaction between the transmit and receive paths so it is 
> easier to avoid FIFO problems. 
> 
> What we are facing today is increased backplane speeds, lower 
> voltage levels and shrinking memory cell size. Adding host to 
> host CRC is not necessarily a suitable way to deal with these 
> problems. Increased backplane speed and lower voltage levels 
> increase the suceptability to noise hits. Both PCI-X 2.0 and PCI 
> Express deal with this by adding error protection to the bus. PCI 
> Express adds CRC protection and PCI-X 2.0 uses an error 
> correcting code. Reduced memory cell size means that soft memory 
> error probability increases - an alpha partical hits one or two 
> cells and changes the contents. In memory, one copes with this by 
> adding parity or error correction bits and by laying out the 
> memory so that the error protection still works when two adjacent 
> cells are corrupted. 
> 
> Reasons CRC is not necessarily the right answer for protection 
> against backplane or RNIC errors:
> CRC is very low cost to implement in hardware processing data as 
> it is transfered. It just takes a 32 bit register plus a few 
> handfuls of XOR gates. It takes a lot of processor cycles to run 
> CRCs on data in host memory so it isn't likely to get used for 
> that application. There are some applications where it is 
> worthwhile to agree on chunk size and to pay the processing cost 
> for host memory CRC calculation, but it isn't the general case.
> 
> To protect data with CRC end-to-end, we have to agree ahead of 
> time on what chunks will be transfered because the CRC has to be 
> computed over a set of bytes. That limits its flexibility for 
> full end-to-end protection of the data.
> 
> Even if the data chunks are protected with end-to-end CRC, there 
> could be an error in passing address information or in 
> control/context information in RNIC memory that causes the wrong 
> chunk of data to be accessed. The chunk will have a valid CRC but 
> will be the wrong data.
> 
> If an alpha particle corrupts data in host memory, what one needs 
> is error correction rather than error detection to recover. CRC 
> error detection is good for places where the data can be resent 
> to recover from an error.
> 
> Therefore, it is better to use CRC RNIC to RNIC and to use other 
> appropriate mechanisms to protect the data from host memory to 
> the point where the DDP CRC is applied. 
> 
> Regards,
> Pat
> 
> -----Original Message-----
> From: Williams, Jim [mailto:[email-masked]]
> Sent: Friday, November 22, 2002 12:49 PM
> To: [email-masked]'
> Subject: [rddp] A different thought on CRCs
> 
> 
> The paper at http://citeseer.nj.nec.com/stone00when.html
> is pretty sobering.  Worth reading with respect to 
> better understanding CRC design tradeoffs.
> 
> It raises the question in my mind of how best to 
> define the CRC to get maximum possible coverage
> of RNIC internal failures as well as RNIC firmware
> and hardware logical bugs.
> 
> Is there a way to move CRC computation and checking
> as close to the host side as possible so as to get
> maximum coverage of NIC internals?
> 
> I don't have a proposal or answer, just raising
> a question.
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 13 Dec 2002 18:05:11 -0800

Pat,

First of all, it is not clear what complexity is added.
Please demonstrate where the complexity is. (the
benefits are flexibility and coverage from a number
of type of stomping errors on the device). Once
you can locate where the header starts, there is no
issue anymore.

Secondly, the CRC calculations are done only when we
can locate the ULP (MPA or whatever) header, and can
look into it to understand how long the message is
(isn't there a layering issue here from TCP <-> TCP-user
perspective if this is done on out of order segments?)
and then the proposal is to look even further into it
to see if it tagged or not, and then use the information
to place the data. If you can determine from the TCP context
that this is a DDP stream, and how to place the date,
inferring that the stream is running a particular ULP protocol
and determining the header size is trivial.

A number of alternatives are possible to get around
any issues.

1. Fixed sized headers as are in proposal today.
2. Adding a header length field (one way would be
   to still 2 bits from the length field and use
   the numbers to indicate 4 different header sizes)

I am sure more inventive minds would come up with
other proposals.

Somesh

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, December 13, 2002 5:30 PM
> To: [email-masked]; [email-masked]
> I expect RDMA NICs to be relatively high-end well-engineered 
> devices. Once the packet is out on the network, it is pretty hard 
> to control which routers, switches and middle boxes it goes 
> through. That is why we need CRC protection from RNIC to RNIC.
> 
> Two CRCs does add complexity and it doesn't provide any 
> significant gain. It doesn't provide for host to host protection 
> - one would need message CRC rather than segment CRC to do that. 
> (See the message from Mike Krause.)
> 
> Also, since the CRC is provided by MPA, it would also be pretty 
> strange layering as it requires MPA to make an assumption about 
> what part of the segment it is passed is DDP header and what part 
> is DDP data. 
> 
> Pat
>

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 13 Dec 2002 22:15:23 -0600

>you can locate where the header starts, there is no
Correct, the header CRC is really an additional CRC
comparison point. Only one CRC accumulator is needed
whether there is a single CRC or a header and payload CRC.

>I am sure more inventive minds would come up with
A ULP specific fixed header size could be specified via the
local interface when MPA is enabled.

That said, the second CRC is *somewhat* more complex and it
does steal space from ULP payload.

Those are admittedly minor costs.

The second CRC provides no benefit when alignment has been
maintained. No placement can take place until the whole TCP
segment has been validated anyway. When alignment is
maintained each TCP segment will have a complete MPA CRC.

What justification do you have for an optimization that
costs *every* connection effective bandwidth but is only of
benefit for exceptional connections under network conditions
that *should* be avoided?

If you have a scenario where DDP would be deployed and
alignment destroying middleboxes would be common, whether
immediate or delayed, then I could see adding the second
CRC. Otherwise, why not stick with what is alredy proposed
and *is* the best solution for virtually every envisioned
deployment?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 16 Dec 2002 14:25:42 -0800

There are perhaps differences in the way various people see
the protocol being deployed. The charter does not provide any
restrictions in this area.

I think one of the main applications for an RDMA protocol will
be storage. One of the main advantages of IP based storage
protocols is that every (almost) computer in the world is
capable of taking advantage of this protocol without additional
cost.

Assuming that there will be software based clients (unless we
want to exclude them), the fastest way they will be do this is
by running on top of vanilla stack. At least my reading of the
BSD stack is that it does not in any way or shape guarantee
alignment, especially when the app queues up more data than
the tcp congestion management allows to send. Since the BSD
stack is the ancestor for many (most?) networking stacks, I assume
quite a few stacks behave that way.

There are a number of tracks we can go on from here. It can
be argued that software based clients responding to rdma
read requests from the storage server can go on some sub-optimal
path. However, the protocol should allow a particular
implementation to accelerate both a hardware based implementation
and a software based implementation.

Or we could specifically indicate that RDMA is envisioned for data-center,
hardware accelerated environments.

Somesh

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, December 13, 2002 8:15 PM
> To: [email-masked]
> Cc: [email-masked]; Somesh Gupta