Subject: RE: [rddp] CRC mandatory?

From: "Carrier, John" <[email-masked]>
Date: Tue, 6 May 2003 10:55:28 -0700

I also am a strong advocate of "uniform expectations" 

--jc

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Tuesday, May 06, 2003 9:43 AM
> To: [email-masked]
> Subject: RE: [rddp] CRC mandatory?
> 
> 
> I believe there are some fundamental issues regarding making
> CRC coverage optional.
> 
> The first is exactly what is our starting point?
> 
> My understanding is that we are defining RDMA over reliable
> IP transports and that we should be cleanly layered.
> 
> So, which interpretation to we make of "cleanly layered":
> 
> a) We define uniform expectations of the transport layer,
>    and provide adaptation layers to fill the gap between
>    those expectations and the existing IP transports.
>    
> b) We add adaptation layer features *only* as required
>    to make DDP work, leaving each transport mapping as
>    close to the original transport as possible.
>    
> The "uniform transport layer requirements" basically match
> SCTP. Basically because RDDP requires unordered delivery
> while preserving the order of submission, this required
> the addition of the DDP-SSN in a very thin adaptation layer.
> It has been agreed that MPA adds framing and markers if the
> receiver wants them.
> 
> So the debate is whether MPA should provide a standard level
> of protection (i.e. end-to-end CRC32C on the entire DDP
> Segment), or whether we have to justify *each* addition
> to TCP as being specifically required by RDDP.
> 
> I am a firm advocate of the "uniform expectations" approach.
> This is mostly because I believe that applications
> developers believe even more firmly in uniform expectations,
> and will achieve it by lowering their expectations to match
> the lowest common denominator.
> 
> There are two guides as to what constitutes a reasonable
> expectation for reliable transport for RDMA protocols: SCTP
> and InfiniBand. SCTP is relevant because it is the reliable
> transport most recently defined by the IETF. InfiniBand is
> relevant because many of the ULPs that RDMAP/RDDP plans on
> supporting will also run over InfiniBand.
> 
> Having the adaptation layer raise all transports to a
> uniform layer that matches the needs of RDDP makes much
> more sense to me than having all applications conform
> to the worst transport over which RDDP will run.
> 
> The LCD approach *will* waste effort, because ULP integrity
> checks will be applied redundantly to SCTP's CRC32c.
> 
> Further, even if we accept the need to justify each feature
> within MPA on an RDDP-specific basis, there is definitely a
> need for coverage of the DDP and RDMAP headers.
> 
> I have not seen a single refutation to the following:
> 
> -- There are applications where there is trust between
>     the ULP peers. That is, the Data Sink ULP feels no
>     need to protect itself from maliscious attacks by
>     its peer. There is sufficient containment of damage
>     from other clients and/or sufficient authentication
>     of the peer.
>     
>     Whether a given ULP should or should not work from
>     this model is irrelevant here. Clearly there are
>     are applications where this is true, and they should
>     be supported.
>     
> --  There are applications that require a higher level
>     of protection than afforded by TCP's 16 bit checksum.
>     
> --  There are applications where the server wishes to allow
>     the client to have multiple requests in-flight.
>     
> There are even applications where all of the above apply.
> A DAFS Client requires more than a TCP 16 bit checksum,
> trusts the DAFS server and can have a sequence of read
> requests pending at the same instant.
> 
> Without adequate protection for the DDP header, the ULP
> would be forced to have only a single STag open on the
> connection at any given time.
> 
> That would mean that a DAFS client running over MPA/TCP
> would be less capable than one running over SCTP or IB.
> All three objectives cited above could be met via either
> IB or SCTP.
> 
> I believe that clearly establishes an RDDP specific need
> to cover at least the headers. The cost of covering the
> entire DDP Segment is trivial compared to that.
> 
> But making the length of the coverage variable is an option,
> which means it adds cost. See earlier posts on markers.
> 
> Just because we decided that one option was justified does
> not make the comments on why options are expensive any less
> valid or relevant. Options are an ongoing complexity that
> must be continuously retested. They represent a perpetual
> drain of QA resources.
> 
> A properly designed receiver can evaluate multiple CRCs
> and/or checksums in parallel during a single scan over
> memory. The calculations required for a CRC32C are minor,
> and no additional memory touches are required.
> 
> Even if each addition to MPA must be justified, the
> simplicity of covering the complete DDP Segment would
> outweigh the benefits of allowing a handful of applications
> to omit some trivial calculations over the payload-only
> portions.
> 
> If we accept the goal of providing uniform transport
> expectations to the ULP across all forms of reliable
> RDMA networks then matching both SCTP and IB with
> CRC coverage of the entire payload is clearly the only
> choice.
> 
>     
> 
> 
> 
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 06 May 2003 10:57:58 -0700

It seems to me that one (but NOT the only) consideration in this
discussion is whether CRC calculation is really a burden for a
software implementation.  I've seen qualitative assertions
(some vaguer than others) on both sides, but nothing quantitative.

RFC 3309 ("Stream Control Transmission Protocol (SCTP) Checksum Change")
has an appendix about CRC implementations, and briefly discusses
the issue.  It cites 

	David C. Feldmeier
	Fast software implementation of error detection codes
	IEEE Transactions on Networking, vol 3 no 6, pp 640-651,
		December, 1995. 

Unfortunately, measurements made in 1995 on a SPARCstation LX
(27 SPECint932) probably won't mean much in 2003.

I spent a little time with CiteSeer and Google trying to find
a more recent study of CRC performance, but I couldn't find
anything.

Perhaps someone with a good understanding of software CRC
implementation could do some measurements on modern hardware
that would give a solid quantitative basis for our debate?
For example, one could hack the Linux or *BSD stack to
do a CRC computation on each TCP segment (sent or received),
then throw away the results.  Measuring the performance change
on bulk throughput (say, using netperf) seems like a reasonable
approximation of software CRC costs.

-Jeff

P.S.: Don't look at me!  I don't have any spare time this month,
nor do I have a "good understanding of software CRC implementation".
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: John Hufferd <[email-masked]>
Date: Tue, 6 May 2003 16:27:55 -0700

Caitlin,
I also believe strongly in what you have stated. 
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Caitlin Bestler <[email-masked]>
Sent by: [email-masked]
05/06/2003 09:42 AM

        To:     [email-masked]
        cc: 

 

 

 

Caitlin Bestler

---

From: "Hilland, Jeff" <[email-masked]>
Date: Wed, 7 May 2003 07:49:33 -0500

I strongly agree with Caitlin as well.  Uniform transport layer
requirements.

Her point about at least requiring CRC32 for the headers is something I
thought this group was headed toward by earlier emails.  The
consequences of landing data in an unintended location are severe, so we
should take all reasonable steps to ensure that this doesn't happen.
CRC32 of the header is one of these reasonable steps.

If we are expecting an application to cover the data with it's own CRC,
then this would mean the application would have to add another "touch"
to the data, so we would just be pushing the memory bandwidth issue up
one level.  I would prefer a uniform transport layer requirement to
prevent upper layer protocols from being required to touch the data
again.

Regards,
Jeff

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Tuesday, May 06, 2003 11:43 AM
To: [email-masked]
Subject: RE: [rddp] CRC mandatory?

I believe there are some fundamental issues regarding making CRC
coverage optional.

   
   
The "uniform transport layer requirements" basically match SCTP.
Basically because RDDP requires unordered delivery while preserving the
order of submission, this required the addition of the DDP-SSN in a very
thin adaptation layer. It has been agreed that MPA adds framing and
markers if the receiver wants them.

So the debate is whether MPA should provide a standard level
of protection (i.e. end-to-end CRC32C on the entire DDP Segment), or
whether we have to justify *each* addition to TCP as being specifically
required by RDDP.

I am a firm advocate of the "uniform expectations" approach. This is
mostly because I believe that applications developers believe even more
firmly in uniform expectations, and will achieve it by lowering their
expectations to match the lowest common denominator.

There are two guides as to what constitutes a reasonable expectation for
reliable transport for RDMA protocols: SCTP and InfiniBand. SCTP is
relevant because it is the reliable transport most recently defined by
the IETF. InfiniBand is relevant because many of the ULPs that
RDMAP/RDDP plans on supporting will also run over InfiniBand.

The LCD approach *will* waste effort, because ULP integrity checks will
be applied redundantly to SCTP's CRC32c.

Further, even if we accept the need to justify each feature within MPA
on an RDDP-specific basis, there is definitely a need for coverage of
the DDP and RDMAP headers.

    

    

Without adequate protection for the DDP header, the ULP
would be forced to have only a single STag open on the connection at any
given time.

But making the length of the coverage variable is an option, which means
it adds cost. See earlier posts on markers.

Just because we decided that one option was justified does
not make the comments on why options are expensive any less valid or
relevant. Options are an ongoing complexity that must be continuously
retested. They represent a perpetual drain of QA resources.

Even if each addition to MPA must be justified, the
simplicity of covering the complete DDP Segment would
outweigh the benefits of allowing a handful of applications
to omit some trivial calculations over the payload-only portions.

If we accept the goal of providing uniform transport expectations to the
ULP across all forms of reliable RDMA networks then matching both SCTP
and IB with CRC coverage of the entire payload is clearly the only
choice.

Caitlin Bestler

---

From: "Barron, Dwight" <[email-masked]>
Date: Wed, 7 May 2003 08:27:59 -0500

I agree with Caitlin also, uniform transport requirements with roughly uniform overhead for equivalent implementations. I also agree with Jeff, if we don't solve it consistently at the RDDP layer it will pop back up to the app layer, where we lose the memory bandwidth savings of RDDP and a really good chance to move the CRC overhead into hardware once and for all. 

John, hopefully everyone is on the tack of mandatory to implement, whatever the outcome of the optional debate.

Regards,
Dwight 

> From: Hilland, Jeff 
> Sent: Wednesday, May 07, 2003 7:50 AM
> To: [email-masked]
> expectation for
> which means
> expectations to the

---

From: "Williams, Jim" <[email-masked]>
Date: Wed, 7 May 2003 10:50:24 -0400

Jeff Hilland wrote:
> The consequences of landing data in an unintended
> location are severe, so we should take all reasonable
> steps to ensure that this doesn't happen.

I agree that the consequences are severe.  However
in most cases the consequences of putting bad data
in the correct location are equally severe.  (There
are counter examples, but they are the exception 
rather than the rule.)

Given an unreliable transport, undetected payload corruption
is far more likely than undetected header corruption.
This is both because there are a lot more bits of
payload than header, and because a large percentage
of header corruption will be caught by means other
than a CRC, for instance illegal format, buffer ID,
or offset.  Payload corruption will often be totally
unchecked at the ULP level.

So I am led to a couple opinions.

  1.  I do not see sufficient justification for a mode
      including a header CRC but not a payload CRC.
      This does represent a change of opinion on my
      part from some previous statements.

  2.  The only justification for opting not to use
      RDDP layer CRCs is confidence that the
      underlying transport, TCP, will deliver
      correct data.  I am reluctant to
      go down the path of saying bad headers
      or payloads are "acceptable" under any
      circumstances.

> the lowest common denominator.

I certainly support the idea of uniform expectations.
I would just like to avoid confusing expectations with
mechanisms.  The expectation is reliable correct data
delivery by the transport.  The mechanism is some number
of bits of checksum and/or CRC.

I prefer to frame the debate as to whether or not TCP
achieves the expectation of reliable and correct
data delivery.  Clearly different transports use 
different mechanisms to achieve each of their expectations.
I don't believe equivalence of mechanisms is or should
be an objective.