Subject: RE: [rddp] CRC - rough consensus update

From: "Carrier, John" <[email-masked]>
Date: Wed, 28 May 2003 17:27:24 -0700

I am sorry, but I think this discussion is meandering way out of scope:
remember that DDP/RDMAP will be layered over SCTP as well as TCP.  There is
no way to disable the CRC in SCTP; therefore, there really should be no way
to disable the CRC for MPA/TCP.  

I would prefer to think that the RDMAP consumer can be indifferent to the
underlying transport.   Otherwise, ULPs using DDP/RDMAP would have to be
written with two different expectations of the RDMA transport depending on
whether it uses SCTP or MPA/TCP. 

If we keep the overall architecture in mind, then I don't think the options
discussed in this thread belong in the protocol.  

--jc

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, May 28, 2003 4:48 PM
> To: Caitlin Bestler
> Cc: [email-masked]
> Subject: RE: [rddp] CRC - rough consensus update
> 
> 
> On Wed, 28 May 2003, Caitlin Bestler wrote:
> 
> First off, thank you for the thoughtful note.
> 
> > On 5/28/03, [email-masked] wrote:
> >
> > >On Tue, 27 May 2003, Caitlin Bestler wrote:
> > >
> > >> On 5/27/03, Michael Krause wrote:
> > >>
> > >> >>My point is that the data belongs to the application,
> > >> >>not to the system administrator.
> > >> >
> > >> >The system belongs to the administrator / customer not
> > >> >the application designer or the ULP.
> > >>
> > >> On the contrary, it is the ULP.
> > >
> > >I'm sorry. I'm missing something. I go out and buy ten
> > >computers. One of them will be a server to the other nine.
> > >I pay for them myself. I install the OS myself. I buy &
> > >install the applications myself. I seed the data myself.
> > >At what point did the computers, software, and data start
> > >belonging to themselves and not me?
> > >
> > And if you want to use email on those computers that you
> > just bought, do you sit down and decide to do it on your
> > own? Of course not, you use an existing Upper Layer
> > Protocol.
> 
> True, but the data still belong to me, don't they?
> 
> I'm not sure if EMail is the best example here, for three 
> reasons. First,
> EMail is an application, not just a ULP. Second, Email gets transfered
> over a number of different protocols. While SMTP is the most 
> common, POP
> and IMAP and other protocols get used. Third, EMail is, well, 
> complicated.
> :-) It's a mess. Configuring sendmail or postfix is something 
> I'd rather
> not do. I'd rather build a new kernel file system or iSCSI 
> stack or RDMA
> implementation instead. ;-)
> 
> > Or, occassionally, you design your own ULP to meet your
> > own custom needs.
> >
> > The point is that it is the ULP that decides if additional
> > data integrity is required, because it is the ULP that
> > must specify what form (if any) that additional integrity
> > will take.
> 
> I definitely agree that the ULP knows better than the RDMA 
> layer how risk-
> sensitive it is.
> 
> My whole point though is that the ULP should listen to the admin about
> relaxing requirements on CRCs. i.e. for just the ULP & RDMA, 
> the ULP knows
> better. But when we add the admin to things, the admin should 
> be further
> up the authority chain than either the ULP or RDMA.
> 
> > For example, RDDP offers no services to authenticate the
> > end user. A ULP that requires those services must add
> > them on their own. Virtually everyone agrees that this
> > is a good thing because the needs for user authentication
> > vary so widely amongst different ULPs. It is not a natural
> > service for a transport layer.
> >
> > So when I design a ULP using RDMA, can I rely upon
> > end-to-end CRC32c (or equivalent) data protection or not?
> 
> Yes.
> 
> I think it is reasonable to assume that you will have CRC32C 
> or equivalent
> data protection. The only time you wouldn't have CRCs in 
> active use are
> times when the admin(s) have decided that the links a connection will
> travel over will be safe enough with the TCP checksum.
> 
> Given how contentious this point is, I think it would be 
> reasonable to add
> text indicating that SHOULD not disable RDMA CRCs (well 
> indicate a desire
> to ignore them) without assessing both the links such a 
> connection will
> traverse and the error sensitivity of the ULP. Yes, it's not 
> enforceable by
> RDMA, but it can serve as an agreement between the ULPs and 
> the admins as
> to when CRCs will be ignorable.
> 
> > If I have to actively *check* something that I really do
> > not. An active control is an additional failure point.
> > It is one more thing that has to be tested and monitored
> > forever. It is a permanent ongoing cost for all
> > applications, well at least those that do QA.
> >
> > And for what benefit? So *some* implementations can
> > safe a very small amount of CPU power? When *most*
> > will either perform the check in hardware and/or
> > have adequate CPU power?
> >
> > Reduced data-integrity checking is a specialized
> > need. I have no problem allowing it, provided that
> > it costs the general application nothing.
> 
> And I don't think this will really cost the general 
> application anything.
> 
> Everything we're saying (or everything I've been hearing) 
> indicates that
> everyone is fine with CRCs defaulting to enabled, and having 
> either side
> being able to require them. Thus for them to NOT be active 
> would mean that
> a deliberate policy decision by the admin(s) covering the 
> connection has
> been made.
> 
> > That means that an affirmative step should be required,
> > *for each ULP* and *for each path* to enable voiding CRCs.
> >
> > Anything that resembles a "big red switch" to disable CRCs,
> > that applications must check to make sure it wasn't thrown
> > on accident, is imposing an unacceptable cost on all
> > applications for the benefit of a small handful.
> 
> Then let's make that part of the spec.
> 
> I think it is appropriate for CRC disabling to be performable 
> on a port by
> port basis. We can make it a SHOULD in the RFC. In fact, it 
> sounds like we
> should. I've certainly had "it's not a big red switch" in my 
> mind as to
> how things should be.
> 
> Actually, something closer to how IPsec policy is set up 
> would be best.
> Then you can throw address ranges into the policy. So you can 
> permit not
> using CRCs over the direct connection you have to a box, but 
> not permit
> them being skipped over the connection that goes through the wiring
> closet. If you want.
> 
> It's sounding like an informational guide for administrators 
> RFC may be in
> order for the working group to create.
> 
> > That means the ULP must take an affirmative act to indicate
> > a willingness to accept less than end-to-end CRC32c
> > protection. Avoiding taking an affirmative act is not
> > something that you have to test for with each release
> > (after all, there an an infinite number of affirmative
> > actions that any application has not taken, they obviously
> > cannot be tested for).
> 
> If we do it like that (say something like the TCP_NDELAY 
> option), then the
> ULP or its implementing application really MUST be 
> configurable to forgo
> requiring CRCs.
> 
> Otherwise the ULP is making a policy decision for the admin, when it
> should be the other way around.
> 
> > While I cannot think of a way to make it an enforceable MUST
> > or SHOULD, any determination by a network administrator that
> > a given path offers the equivalent of end-to-end CRC32c
> > protection should last only as long as that path is stable.
> 
> At some point we have to say, "Yes, there's the rope. If you hang
> yourself, that's your own foolishness." Because every time we prevent
> admins from doing something stupid, we also prevent them from doing
> something clever.
> 
> > Further, the network administrator MUST NOT make predictions
> > for any path that goes through any middle-box not under
> > their exclusive control. To do otherwise creates an
> > obligation on a middlebox to continue to function as
> > it has in the past. Any such requirement would be contrary
> > to the fundamental nature of an IP network.
> >
> > Ultimately, the futility of CRC suppression based on
> > "network quality" is that the environments where that
> > assertion can be made are exactly those where software
> > implementations are the least likely.
> >
> > Any real value will more likely be found with software
> > stacks running on remote clients for those applications that
> > can actively assert that an end-to-end 16-bit TCP checksum
> > is adequate for their needs. The gotcha there is that
> > calculating a CRC32c for even T1 speeds is trivial.
> 
> Since we're talking hypotheticals, it's not to hard to find a counter
> example. :-)
> 
> I think a common and reasonable case would be one where we 
> have end-to-end
> IPsec. It is my understanding that the cryptographic hashes 
> in IPsec are
> stronger than CRC32C, so the CRC will, in a well-crafted 
> system, add no
> value. So there's a case that will be using high-performance hardware
> where an application/ULP will see CRC32C equivalent 
> protection w/o a CRC.
> 
> I agree that IPsec in and of itself isn't the cure-all, since 
> there may be
> perfectly-valid systems that have the IPsec calculation far 
> enough away
> that the TCP checksum isn't good enough. But there can (and I 
> expect will)
> be times when it is.
> 
> Also, haven't we been talking about (as one installation 
> scenario) the case
> of fan-in systems, where a number of clients fan into one 
> system? While
> it's not hard to calculate the CRCs at T1 rates, if you have a few
> thousand simultaneous feeds (or ten thousand feeds), you 
> start to notice.
> 
> Take care,
> 
> Bill
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 31 May 2003 10:58:55 -0400

> to disable the CRC for MPA/TCP.  

The IP tradition seems to suggest you use different transports to get
different service.  Just because SCTP provides CRC doesn't mean
MPA/TCP must also provide CRC.  If you turn CRC off in SCTP, you would
have no integrity checking at all.  If you turn CRC off in MPA, you
still have the TCP checksum.

I agree (with Bill & Caitlin) that the issue IS software overhead.
Software implementations are most likely to be `cleanly layered' which
means TCP and MPA (and IPSec) will probably all have different touches
of the data.  A user mode implementation of RDDP+MPA on a stock TCP
socket is a key early implementation approach.

Even though we shouldn't plan on aggressively optimized (rototilled)
software implementations that reduce the data touching,
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 31 May 2003 12:12:57 -0500

On 5/31/03, Stephen Bailey wrote:

>> I am sorry, but I think this discussion is meandering
>> way out of scope: remember that DDP/RDMAP will be
>> layered over SCTP as well as TCP.  There is no way to
>> disable the CRC in SCTP; therefore, there really should
>> be no way to disable the CRC for MPA/TCP.  
>
>The IP tradition seems to suggest you use different
>transports to get different service.  Just because SCTP
>provides CRC doesn't mean MPA/TCP must also provide CRC. 
>If you turn CRC off in SCTP, you would have no integrity
>checking at all.  If you turn CRC off in MPA, you still
>have the TCP checksum.
>
>I agree (with Bill & Caitlin) that the issue IS software
>overhead. Software implementations are most likely to be
>`cleanly layered' which means TCP and MPA (and IPSec) will
>probably all have different touches of the data.  A user
>mode implementation of RDDP+MPA on a stock TCP socket is a
>key early implementation approach.
>
>Even though we shouldn't plan on aggressively optimized
>(rototilled) software implementations that reduce the data
>touching,

And if they code it in Java it will be even slower. And
while we don't want to forbid Java implementations, I
doubt anyone would propose a change to the protocol for
the benefit of Java implementations.

There are several reality-checks needed here.

There will indeed be software stacks that are cleanly
layered on top of existing TCP stacks.  But they will
mostly be used for remote clients at low speeds.
    
An implementation can be cleanly layered if it wants,
but the extra memory touches that clean layering causes
are its own problem. The protocol works, even with the
extra protocols. It also allows a rototilled implementation
to optimize to reduce the number of memory touches required.

Software implementations of the entire stack are indeed
feasible (and frequently what "hardware" solutions are
really doing). But whatever the speed limits for such
implementations are, the CRC32c is scarcely a factor.
The "extra" memory touch is only an artifact of building
upon existing pieces, not of the inherent problem. The
computation required is trivial.

Keep in mind that any platform (software and/or hardware)
must have the raw processing power (in software and/or
hardware) to handle all MPA/TCP connections with CRCs
enabled. The point of the option is allow some
implementations to redeploy some of that processing
power when L2 protections have provided equivalent
end-to-end coverage.

The benefit is slight, and only available to some
implementations. But under Jim's proposal it has virtually
zero impact on the other implementations. From their point
of view it "wastes" some bits in the MPA mode frame.

For that low of a cost I can see no reason not to allow
the handful of implmentations that would benefit from
this from doing so.

But we have to be very careful to make certain that the
costs do not grow. Most importantly, ULP designers must
have no doubts that they are guaranteed end-to-end CRC32c
protection.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp