Subject: RE: [rddp] CRC mandatory?

From: Carl Hensler <[email-masked]>
Date: Tue, 6 May 2003 17:49:38 -0700 (PDT)

I strongly support Caitlin's position.

Carl Hensler
Distinguished Engineer
Sun Microsystems

> Date: Tue,  6 May 2003 11:42:48 -0500
> From: Caitlin Bestler <[email-masked]>
> Subject: RE: [rddp]  CRC mandatory?
> To: [email-masked]
> 
> I believe there are some fundamental issues regarding making
> CRC coverage optional.
> 
> The first is exactly what is our starting point?
> 
> My understanding is that we are defining RDMA over reliable
> IP transports and that we should be cleanly layered.
> 
> So, which interpretation to we make of "cleanly layered":
> 
> a) We define uniform expectations of the transport layer,
>    and provide adaptation layers to fill the gap between
>    those expectations and the existing IP transports.
>    
> b) We add adaptation layer features *only* as required
>    to make DDP work, leaving each transport mapping as
>    close to the original transport as possible.
>    
> The "uniform transport layer requirements" basically match
> SCTP. Basically because RDDP requires unordered delivery
> while preserving the order of submission, this required
> the addition of the DDP-SSN in a very thin adaptation layer.
> It has been agreed that MPA adds framing and markers if the
> receiver wants them.
> 
> So the debate is whether MPA should provide a standard level
> of protection (i.e. end-to-end CRC32C on the entire DDP
> Segment), or whether we have to justify *each* addition
> to TCP as being specifically required by RDDP.
> 
> I am a firm advocate of the "uniform expectations" approach.
> This is mostly because I believe that applications
> developers believe even more firmly in uniform expectations,
> and will achieve it by lowering their expectations to match
> the lowest common denominator.
> 
> There are two guides as to what constitutes a reasonable
> expectation for reliable transport for RDMA protocols: SCTP
> and InfiniBand. SCTP is relevant because it is the reliable
> transport most recently defined by the IETF. InfiniBand is
> relevant because many of the ULPs that RDMAP/RDDP plans on
> supporting will also run over InfiniBand.
> 
> Having the adaptation layer raise all transports to a
> uniform layer that matches the needs of RDDP makes much
> more sense to me than having all applications conform
> to the worst transport over which RDDP will run.
> 
> The LCD approach *will* waste effort, because ULP integrity
> checks will be applied redundantly to SCTP's CRC32c.
> 
> Further, even if we accept the need to justify each feature
> within MPA on an RDDP-specific basis, there is definitely a
> need for coverage of the DDP and RDMAP headers.
> 
> I have not seen a single refutation to the following:
> 
> -- There are applications where there is trust between
>     the ULP peers. That is, the Data Sink ULP feels no
>     need to protect itself from maliscious attacks by
>     its peer. There is sufficient containment of damage
>     from other clients and/or sufficient authentication
>     of the peer.
>     
>     Whether a given ULP should or should not work from
>     this model is irrelevant here. Clearly there are
>     are applications where this is true, and they should
>     be supported.
>     
> --  There are applications that require a higher level
>     of protection than afforded by TCP's 16 bit checksum.
>     
> --  There are applications where the server wishes to allow
>     the client to have multiple requests in-flight.
>     
> There are even applications where all of the above apply.
> A DAFS Client requires more than a TCP 16 bit checksum,
> trusts the DAFS server and can have a sequence of read
> requests pending at the same instant.
> 
> Without adequate protection for the DDP header, the ULP
> would be forced to have only a single STag open on the
> connection at any given time.
> 
> That would mean that a DAFS client running over MPA/TCP
> would be less capable than one running over SCTP or IB.
> All three objectives cited above could be met via either
> IB or SCTP.
> 
> I believe that clearly establishes an RDDP specific need
> to cover at least the headers. The cost of covering the
> entire DDP Segment is trivial compared to that.
> 
> But making the length of the coverage variable is an option,
> which means it adds cost. See earlier posts on markers.
> 
> Just because we decided that one option was justified does
> not make the comments on why options are expensive any less
> valid or relevant. Options are an ongoing complexity that
> must be continuously retested. They represent a perpetual
> drain of QA resources.
> 
> A properly designed receiver can evaluate multiple CRCs
> and/or checksums in parallel during a single scan over
> memory. The calculations required for a CRC32C are minor,
> and no additional memory touches are required.
> 
> Even if each addition to MPA must be justified, the
> simplicity of covering the complete DDP Segment would
> outweigh the benefits of allowing a handful of applications
> to omit some trivial calculations over the payload-only
> portions.
> 
> If we accept the goal of providing uniform transport
> expectations to the ULP across all forms of reliable
> RDMA networks then matching both SCTP and IB with
> CRC coverage of the entire payload is clearly the only
> choice.
> 
>     
> 
> 
> 
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: David Ford <[email-masked]>
Date: Tue, 06 May 2003 21:07:05 -0400

>I strongly support Caitlin's position.

What he said.

David Ford
Network Appliance

---

From: Stephen Bailey <[email-masked]>
Date: Wed, 07 May 2003 10:48:19 -0400

> > >    close to the original transport as possible.

You've outlined two extremes.  There's an intermediate position.

If applications (rather than RDDP), SPECIFICALLY want a capability,
the adaptation layer can provide it.  CRC protection is an example.

I'm personally opposed to the `uniform expectations' model for its own
sake.  This is much more broad than to CRC or not to CRC.  RDDP solves
an orthogonal problem to over-the-wire data delivery, and should not
be constrained to a single set of transport characteristics.

It may well be the case (is the case, apparently) that a) we don't
understand how to build an RDDP that operates orthogonally from the
transport b) we don't see a clear and present NEED for RDDP on
transports other than connected, sequenced and reliable.

I actually think a) may not be as hard as people think (I believe the
RDDP architecture in the arch ID IS transport agnostic), but b) makes
it relatively unlikely that anybody other than a hobbyist (i.e. me)
would be interested.

Along these lines, I'm somewhat disappointed that (if I understand
correctly) the SCTP mapping seems to build sequencing on top of
unsequenced SCTP.  Seems more natural to allow the application to
chose one or the other transport mode.

Perhaps my concern with `nailing down every loose detail' in the
protocol architecture comes from my history with processor design.
You really want to have the architecture specify enough behavior to be
useful, and leave as much as possible to implementation or application
choice.  Ok, so I worked at DEC.  Alpha got the coherency model right
(but failed for other reasons).  Alpha never really took this
position, but clearly, if you want a strong coherency model, e.g. for
an embedded application, you can buy an implementation which provides
superset (stronger coherency) capabilities.  Your code won't be 100%
portable, but that's the choice you make.

I think you've gone too far if you say CRC protection is a specific
instance of a general principle we should apply.  CRC protection is
CRC protection.  The general principle is to leave as much flexibility
to the implementations while still providing a useful service.  Saying
that RDDP MUST run on a sequenced, reliable, CRC protected transport
seems ridiculous.

Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 7 May 2003 10:14:46 -0500

On 5/7/03, Stephen Bailey wrote:

>I think you've gone too far if you say CRC protection is a
>specific instance of a general principle we should apply. 
>CRC protection is CRC protection.  The general principle
>is to leave as much flexibility to the implementations
>while still providing a useful service.  Saying that RDDP
>MUST run on a sequenced, reliable, CRC protected transport
>seems ridiculous.

Agreed.

To be more precise, RDDP should have uniform expectations
for sequenced/reliable transports.

It has been my assumption all along that RDDP maps very
well to unreliable transports, but that the use of an
unreliable transport is something that the ULP would
request to meet specific requirements.

The specific issue that will have to be dealt with when
we are ready to address unreliable transports is how the
set of DDP Segments that form an atomic made it / lost it
decision are identified. When we discussed this briefly
on the old yahoo groups you seemed to favor having the
receiving ULP make such determinations, I favored 
explicitly labeling the RDMA Writes and Send that
formed a cohesive accept it/drop it unit.

But I think we both agree that unreliable RDDP is a
specialized application that can be dealt with after the
reliable version has been fully dealt with. [And even if
we don't, David would probably agree for us more vehemently,
;-) ].

Caitlin Bestler