Subject: RE: [rddp] CRC mandatory?

From: "Williams, Jim" <[email-masked]>
Date: Mon, 5 May 2003 17:44:03 -0400

I don't have a major problem with mandatory
CRCs, even if my reasons are a bit different.
But I do want to push back on some of the arguments,
though, to see if they hold up.

Jeff Hilland wrote:
> Caitlin brings out an excellent point below.  
> How is it that CRC32C is acceptable overhead for SCTP,
> but it is onerous for MPA?  

Dwight Barron wrote:
> Why shouldn't this place the burden of proof
> on those who oppose mandatory CRCs?

The Internet has grown from a laboritory curiosity
to a world wide phenominon based on using 16 bit
checksums.  SCTP, however, is not far beyond the
labority curiosity stage at the present time.
Is the world clamoring that TCP be replaced by
SCTP because TCP is corrupting too much data?
If so, SCTP should quickly become widely deployed,
and RDDP should simply ride this transition, and
use SCTP.

On the other hand, is TCP fine for its current
uses, but RDDP if fundamentally different and
requires a higher level of data integrity?
This seems like a difficult case to make given
the wide spread use of NFS and CIFS over TCP.
Is silent data corruption over RDDP worse than
silent data corruption over NFS or CIFS?
Is there a push to add more data integrity to
NFS?  If not, then this seems to add credibility
to the case that TCP as is provides sufficient
data integrity in many cases.

Sould RDDP to take the position that TCP is 
so deficient in data integrity that to NOT
use a CRC is irresponsible and therefore 
the CRC must be mandated?  Are there any
other protocols layered on TCP that
are a precedent for this?   The CRC for
iSCSI is optional.  Given that RDDP
proposes something new, I would submit
that the burden or proof would be on those
supporting mandatory CRC.

Because SCTP uses a CRC, it is suggested
that that proves a CRC is not too onerous
to add to a protocol layered on TCP.
This seems a stretch in a couple respects.
One being that two data checks are more
expensive than one, and not justified if
the first (TCP checksum) is sufficient.
Second, the jury is still out on SCTP
in a number of respects.  And RDDP is 
arguable targetted towards performance
critical applications more so than
SCTP.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Tue, 06 May 2003 07:32:20 -0700

>labority curiosity stage at the present time.

I believe there has been a clear message delivered on the value of SCTP and 
this workgroup has stated it will define specifications to support SCTP as 
evidenced by the work done by Randall, et. al.  While I strongly support 
the TCP effort from a TTM perspective, SCTP is in the charter and our 
efforts to develop specifications to accommodate both technologies using a 
well layered architecture have been in support of that charter.

>Is the world clamoring that TCP be replaced by SCTP because TCP is 
>corrupting too much data?

The value props provided by SCTP go beyond data corruption as stated 
numerous times within the TSVWG - I don't think we need to repeat them all 
here.  The 32-bit data integrity was viewed as an important improvement 
over the existing 16-bit checksum so it should not be discounted.

>If so, SCTP should quickly become widely deployed, and RDDP should simply 
>ride this transition, and
>use SCTP.

As it has been stated during the entire 32-bit CRC debate in IPS and in 
TSVWG, there is strong motivation to providing better data integrity by 
many companies and their customers.   I have helped create new technologies 
where while the probability of data corruption is relatively small, the 
data rates combined with the impact of just one instance of data corruption 
motivated all to make data integrity (32-bit CRC) a requirement.   This is 
true for local I/O technologies (in the box) such as PCI-X 2.0 which 
provides strong ECC protection and PCI Express which has a 32-bit CRC for 
link-level (hop-by-hop) data integrity and a 32-bit CRC for end-to-end data 
integrity.  Same applies to InfiniBand where we had a 16-bit link-level and 
a 32-bit end-to-end CRCs.  All of the designers I work with across many 
companies understand the value of strong data integrity and have worked to 
make sure this is available from the start.  In one sense, the proof of 
these other specifications shows the value of data integrity in the market 
place and that the cost to implement is not viewed as onerous (PCI* 
technology will be implemented across the entire spectrum of design and 
cost points).

All of our experience to date with high-speed I/O has driven many of us to 
support a mandatory 32-bit CRC for MPA which, as John points out, enables a 
well-layered architecture with the same level of semantics delivered over 
either of the RDDP chartered transports.

>requires a higher level of data integrity?

Yes.  TCP implementations target anonymous buffers using local memory 
translations which are not known in the wire.  RDDP uses TO and STag to 
determine the target buffering thus there is the potential for silent data 
corruption from a corruption in the header.  The argument for having a 
single CRC for header / data is that one MUST validate the transport 
checksums which covers the entire payload (RDDP headers and data) prior to 
placement thus a single CRC can be done in parallel and validated at the 
end of the chunk / frame.

>This seems like a difficult case to make given the wide spread use of NFS 
>and CIFS over TCP. Is silent data corruption over RDDP worse than silent 
>data corruption over NFS or CIFS?

Yes.  As noted above.  NFS / CIFS all use anonymous buffers thus there data 
integrity is only impacting their own buffers which isn't guaranteed to be 
true for all RDDP operations.

>Is there a push to add more data integrity to NFS?  If not, then this 
>seems to add credibility to the case that TCP as is provides sufficient 

NFS has additional data integrity as an option.  Whether people push to 
deploy is an orthogonal issue.  For many years people did not use the 
existing NFS data integrity methods which were just a checksum using 
similar logic as you but now they always use the base integrity methods.

>Sould RDDP to take the position that TCP is so deficient in data integrity 
>that to NOT use a CRC is irresponsible and therefore the CRC must be 
>mandated?

It is irresponsible for the reasons I've listed above and as noted in the 
other workgroups where this was heavily debated before.

>Are there any other protocols layered on TCP that are a precedent for 
>this?   The CRC for iSCSI is optional.  Given that RDDP proposes something 
>new, I would submit that the burden or proof would be on those supporting 
>mandatory CRC.

The iSCSI CRC being optional was based on discussions that a lower layer 
could provide data integrity, e.g. IPSec can be used to provide strong 
end-to-end data integrity thus many felt they'd rather spend their 
resources on a single pass of the data rather than multiple passes in order 
to deliver good performance (applies to both software and hardware 
implementations).  I don't recall anyone on IPS stating that they would be 
happy using just checksums or that they would be happy for all time without 
providing the ability to provide strong data integrity.

>Because SCTP uses a CRC, it is suggested that that proves a CRC is not too 
>onerous to add to a protocol layered on TCP.

It is not from an implementation perspective.

>This seems a stretch in a couple respects. One being that two data checks 
>are more expensive than one

A software or hardware implementation may make a single pass of the data 
with minimal overhead to calculate (TCP case) the checksum and CRC.  This 
is not viewed as onerous or that expensive to implement though there is a 
slight cost.

>  and not justified if the first (TCP checksum) is sufficient.

It is not viewed as sufficient by many people in this workgroup as well as 
IPS and TSVWG from my recollection of the previous debates on the need for 
strong data integrity.

>Second, the jury is still out on SCTP in a number of respects.  And RDDP 
>is  arguable targetted towards performance critical applications more so 
>than SCTP.

I will not attempt to categorize SCTP usage as focused on performance or 
not.  There is nothing that I have read that would preclude performance 
focused solutions thus I do not concur that RDDP (which has a clear charter 
to support SCTP) should have what is currently a cleanly, well-layered 
architecture permuted to be transport specific when it is not called for by 
any argument presented to date.  RDMAP and DDP layer on top of SCTP or MPA 
in a very clean and precise manner thus allowing simpler implementations 
and future opportunities to leverage all of the work done above the 
transport / framing mechanisms.  I view this as just as important a benefit 
as anything from a technical perspective.  Creating additional options that 
have very little if any value to the protocol suite only creates legacy 
inertia and will slow adoption of this technology.   Given the economic 
conditions, the costs of adding options into solutions (we have already 
debated this), etc. I do not agree with any of your assertions or your 
request to place "the burden of proof" on those that support mandatory CRC 
as though the case of mandatory has not already been made in other 
workgroups within the IETF.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 6 May 2003 11:42:48 -0500

I believe there are some fundamental issues regarding making
CRC coverage optional.

The first is exactly what is our starting point?

My understanding is that we are defining RDMA over reliable
IP transports and that we should be cleanly layered.

So, which interpretation to we make of "cleanly layered":

a) We define uniform expectations of the transport layer,
   and provide adaptation layers to fill the gap between
   those expectations and the existing IP transports.
   
b) We add adaptation layer features *only* as required
   to make DDP work, leaving each transport mapping as
   close to the original transport as possible.
   
The "uniform transport layer requirements" basically match
SCTP. Basically because RDDP requires unordered delivery
while preserving the order of submission, this required
the addition of the DDP-SSN in a very thin adaptation layer.
It has been agreed that MPA adds framing and markers if the
receiver wants them.

So the debate is whether MPA should provide a standard level
of protection (i.e. end-to-end CRC32C on the entire DDP
Segment), or whether we have to justify *each* addition
to TCP as being specifically required by RDDP.

I am a firm advocate of the "uniform expectations" approach.
This is mostly because I believe that applications
developers believe even more firmly in uniform expectations,
and will achieve it by lowering their expectations to match
the lowest common denominator.

There are two guides as to what constitutes a reasonable
expectation for reliable transport for RDMA protocols: SCTP
and InfiniBand. SCTP is relevant because it is the reliable
transport most recently defined by the IETF. InfiniBand is
relevant because many of the ULPs that RDMAP/RDDP plans on
supporting will also run over InfiniBand.

Having the adaptation layer raise all transports to a
uniform layer that matches the needs of RDDP makes much
more sense to me than having all applications conform
to the worst transport over which RDDP will run.

The LCD approach *will* waste effort, because ULP integrity
checks will be applied redundantly to SCTP's CRC32c.

Further, even if we accept the need to justify each feature
within MPA on an RDDP-specific basis, there is definitely a
need for coverage of the DDP and RDMAP headers.

I have not seen a single refutation to the following:

-- There are applications where there is trust between
    the ULP peers. That is, the Data Sink ULP feels no
    need to protect itself from maliscious attacks by
    its peer. There is sufficient containment of damage
    from other clients and/or sufficient authentication
    of the peer.
    
    Whether a given ULP should or should not work from
    this model is irrelevant here. Clearly there are
    are applications where this is true, and they should
    be supported.
    
--  There are applications that require a higher level
    of protection than afforded by TCP's 16 bit checksum.
    
--  There are applications where the server wishes to allow
    the client to have multiple requests in-flight.
    
There are even applications where all of the above apply.
A DAFS Client requires more than a TCP 16 bit checksum,
trusts the DAFS server and can have a sequence of read
requests pending at the same instant.

Without adequate protection for the DDP header, the ULP
would be forced to have only a single STag open on the
connection at any given time.

That would mean that a DAFS client running over MPA/TCP
would be less capable than one running over SCTP or IB.
All three objectives cited above could be met via either
IB or SCTP.

I believe that clearly establishes an RDDP specific need
to cover at least the headers. The cost of covering the
entire DDP Segment is trivial compared to that.

But making the length of the coverage variable is an option,
which means it adds cost. See earlier posts on markers.

Just because we decided that one option was justified does
not make the comments on why options are expensive any less
valid or relevant. Options are an ongoing complexity that
must be continuously retested. They represent a perpetual
drain of QA resources.

A properly designed receiver can evaluate multiple CRCs
and/or checksums in parallel during a single scan over
memory. The calculations required for a CRC32C are minor,
and no additional memory touches are required.

Even if each addition to MPA must be justified, the
simplicity of covering the complete DDP Segment would
outweigh the benefits of allowing a handful of applications
to omit some trivial calculations over the payload-only
portions.

If we accept the goal of providing uniform transport
expectations to the ULP across all forms of reliable
RDMA networks then matching both SCTP and IB with
CRC coverage of the entire payload is clearly the only
choice.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Tue, 06 May 2003 10:09:18 -0700

Strongly concur.

Mike

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 6 May 2003 11:11:57 -0700

If SCTP is the most desirable transport, the SCTP mapping
should be used. There are many features of SCTP that are
thankfully not attempted to be grafted on to TCP. But
they could have been and the same justification could have
been used. And it will be great when everybody adopts SCTP.

Similar arguement is for IB. I don't think there would be
any of these debates for use of RDMA over IB.

At the same time, if people want to use TCP congestion control
and other features, + a few other options, then the best thing
to use another transport number.

However the desire is to use TCP and fix the percieved shortcomings
that it has.

This is where the debate is.

The best way to handle this is options and not making new
percieved high-value features as mandatory. This way the market
can sort out which are useful and which are not.

1. I use CIFS all the time and my bits are not randomly being
corrupted (i.e. TCP checksum is sufficient).
2. CRC calculation in software does have additional overhead.
Theoretically the CRC calculation could be combined in some
other part of the processing, but practically it will be
a separate computation.
3. It is quite likely that IPSec offload will become
prevalant, thus eliminating any
need to have CRC as the app level.
4. If applications want even higher level of protection, then
they will best not off-load integrity checks.
5. A separate header and data CRC allows two things - one is
to overcome the bugaboo about direct data placement, the second
about allowing data protection to be stripped while data is
DMAed to the host and eliminating another potential source of
corruption (in the chip buffers).
6. Perceived reliability is a feature of deployments and not
of applications. That is why HA features are optional and
deployed as needed (in systems, configurations, network
equipment ...), and people buy different types of equipment
at different price points. Something as simple as ECC memory
is not uniformly found in systems.

---

From: Michael Krause <[email-masked]>
Date: Tue, 06 May 2003 11:47:37 -0700

>corrupted (i.e. TCP checksum is sufficient).

RDMA = Remote DMA which is in essence the same as local DMA from an 
exposure perspective but now controlled by a remote.   For local DMA, we 
are adding in strong data integrity as a requirement, e.g. ECC or 
CRC-32.  I assert that the same logic for local DMA having strong data 
integrity is no different than for Remote DMA.

CIFS / NFS / etc. use anonymous buffers and thus are exposed in the same 
way as one is with Remote DMA from corruption of the buffers or targeting 
incorrect memory.

>other part of the processing, but practically it will be

That is an implementation choice and not one a high performance solution 
would pursue, IMO.

>3. It is quite likely that IPSec offload will become prevalant, thus 
>eliminating any need to have CRC as the app level.

This isn't at the app level; it is at the RDMA level.  Also, IPSec is often 
implemented in gateway devices and people desire to have end-to-end data 
integrity between the NICs (rely upon local I/O data integrity to insure 
final delivery is correct).

>corruption (in the chip buffers).

Two CRC serves no purpose as the transport protection checks must be done 
in conjunction prior to validating the packet for placement purposes thus a 
single CRC provides the same level of protection at a lower cost.

>is not uniformly found in systems.

ECC memory is becoming the standard for servers and storage these days.  It 
may eventually work its way into clients.  In any case, the debate is on 
the value of having a mandatory CRC not on HA which is a separate topic.

Mike