Subject: RE: [rddp] CRC - rough consensus update

From: "Uri Elzur" <[email-masked]>
Date: Wed, 4 Jun 2003 14:19:55 -0700

Maybe most important comment is that, if CRC is optional in any way,
there is no way (or it becomes very hard) for the ULP to know it is
there. It takes a mandatory API to make this info avail on every OS to
every ULP. RDDP has no control over the APIs. It means the ULP has to
design its own error detection and correction scheme. This might be the
most important motivation to not only have CRC as MUST IMPLEMENT but
also MUST USE.

More comments embedded [UE]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, June 03, 2003 10:30 AM
To: Uri Elzur
Cc: Barron, Dwight; Stephen Bailey; Caitlin Bestler; [email-masked]
Subject: RE: [rddp] CRC - rough consensus update

On Mon, 2 Jun 2003, Uri Elzur wrote:

> No one suggests to turn off Ethernet CRC even if you can show the
> local short link from a host to the next switch is error free. Why? It

> is in the hardware and it is free, so why bother. It was probably as 
> difficult 20 years ago running @10Mb/S.

So? I don't see how the fact that no one can nor would turn off the
Ethernet CRC matters here.

[UE]
There was a suggestion on this mailing list that you don't need CRC on
local high qual links. My point is that no one really wants to worry
about managing CRC on link by link basis because 
	A. it is an additional headache for the admin 
	B. Admin normally has no way to know link qualities 
	C. Admin normally will leave default as-is

Second point: much like Ethernet at the time, once the decision to
mandate CRC is taken, one can expect HW to support it. No one said we
will not mandate CRC on a local link because it will be too much to ask
of a 8086 or 80186, before HW supports it... [UE]

> If we make it mandatory it will be implemented in products supporting
> the outcome of this group's work. After some time, the impact for 
> those who do it in SW, will go away as more will use HW acceleration 
> or simply have faster CPU. On the other hand, we need be careful not 
> to impose a too strong of an impact for the transition period. That's 
> why a performance argument is interesting.

Note: everyone is talking about CRCs being MUST implement, so they will
be implemented.

[UE]See above Comment [UE]

I disagree that the "impact" of a software CRC will ever go away. Yes,
CPUs will get faster & faster. But as they get faster & faster, people
will want them to do more; expectations will scale with ability. So
whatever impact the CRC has will still be there.

[UE] Most likely data is touched at least once by SW, i.e.
middleware/applications will consume the data. At that point the
additional cost for CRC compute goes down significantly. But, as you
say, there will always be additional "impact".  This is why it is
important to deliver a clear direction to the HW community - it is
mandatory, put CRC in HW and this is how the "impact" goes away
eventually - no need to worry about it in SW anymore [UE]

Also, we shouldn't talk about turning off the CRCs because of a
transition impact. We should talk about doing it because it's safe. 

[UE] I guess my message wasn't clear enough. I'm not suggesting to turn
it off. I strongly believe CRC needs to be on[UE]

RDMA is a lot more sensitive to errors than tcp or sctp, mainly since we
use in-stream data to decide where to write the packet contents. And
some ULPs are more sensitive than others; ones with multiple buffers
have to worry about data going into the wrong buffer. ;-)

That's why we think disabling CRCs should be a per-port (ULP) kinda
thing. An admin can know, "Oh, this server only has one open buffer at a
time, turning off CRCs will be ok given our current network. That server
has multiple buffers open, better keep on CRCs," and set policy
accordingly.

[UE] with tcp you still have the risk of corrupted data landing in the
app buffer, so apps (e.g.. iSCSI) have their own CRC mechanism. With
RDMA whether there is one of few buffers, data can only land in buffers
belonging to that "flow". Again, I don't see an admin turning CRC off.
If CRC is avail it will be always on. Admin will never be 100% sure
whether the app posts just 1 buffer, mult buffers etc. It can change
from one OS to another and from one release to another.[UE]

> If the _option_ to turn it off is there, it will forever add a
> negotiation which slows down the initialization and adds complexity to

> implementations. It will be yet another knob the administrator has to 
> deal with and no one showed there is a HUGE performance hit

Now I'm confused. You talk about not using CRCs above, yet you don't
want an option in-band to turn them off? How else would you turn them
off? Just arbitrarily flipping them off? That's scary...
[UE] no intention to recommend NO CRC. On the contrary, I'm for CRC and
always ON. [UE]

We want a way in-band for either side to say, "We MUST use markers", so
we need a form of negotiation. We have it down to a statement-of-desire
negotiation. It's very simple, and closes in one round-trip.

Each side says, "I want this," in the first PDU, and then everyone knows
what's happening. We also have it down to where this won't slow down
initial data, either (to the extent data can flow before the client has
heard from the server). Since we will need an initial PDU anyway, using
one flag in it for CRCs-can-turn-off isn't a loss at all.

So I don't see what we loose from your discussion above by permitting
CRCs to be negotiated to off?

Take care,

Bill

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 4 Jun 2003 16:51:07 -0500

> most important motivation to not only have CRC as MUST IMPLEMENT but

I agree and disagree.

An API that the ULP MUST use to guarantee that CRCs are used is clearly
unacceptable.

However, if there MAY be an API, which IF used MAY suppress CRCs, but
if not available or not used then CRCs MUST be used.

That is acceptable to me.

If you want CRCs used all of the time, simply set the bit so indicating.
In fact we could even agree that "zero" indicates that CRCs are mandatory,
so in fact you have to do nothing. The field must be set with the same
zero that it would have been set with if it had been "reserved - must be zero".

If we were talking about *anything* that an implementation would have
to *actively* do I would be in full agreement with you.

As it is though, even though I am confident that I would never turn
on this option, the cost of allowing others to have it seems to be as
close to zero as is humanly possible.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Thu, 5 Jun 2003 00:22:47 +0000 (UTC)

> most important motivation to not only have CRC as MUST IMPLEMENT but

No. We specifically don't want such an API as it's not needed. That's the
whole point of setting guidelines as to when CRCs may be disabled. And
that's why I suggested using the word, "promise," in there. Specifically
so ULPs _don't_ feel the need to add their own error detection schemes.

That's the reason for requiring that the ULP always have a CRC-32C-
comparable protection level. Either it's the CRC, or, for some reason that
the admin knows, there is a comparable level of protection. End-to-end
IPsec is one obvious case.

> 	C. Admin normally will leave default as-is

The suggestion wasn't that CRCs SHOULD be turned off on high-quality
links, but that sites/ULPs will be content/safe if they are.

An admin who leaves things as the default will be leaving CRCs enabled.
I've heard a number of folks state CRCs should default to on, and I agree
with them.

> of a 8086 or 80186, before HW supports it... [UE]

Comparing this CRC with Ethernet's is really clouding the issue. First
off, Ethernet _needs_ the CRC. Ethernet was designed as a distributed
party line where, among other things, you could have two transmitters
ending up sending overlapping transmissions. If the packets were sized
right, a receiver could see one entirely contained in the other.  Without
the CRC, there is no way to detect this error. It has nothing to do with
line quality but how ethernet actually works.

Second, Ethernet's CRC happens at the very front end of processing.
Nothing flows farther up the processing chain if the CRC's bad. RDMA's
CRC, though, hapens after a bit of processing. So even if you have
hardware assist, chances are you'll have a noticable processing point
where the CRC happens. It won't be invisable.

> from one OS to another and from one release to another.[UE]

I know admins who I think can get this question right. Yes, they are not
the majority, but they do exist. I think the majority will just leave CRCs
on (especially if we word the spec right), so ULPs will be safe.

Also, the iSCSI CRC is a bit different. It has both header and data
options, which are independant. The reason for that is so that the data
CRC can be end-to-end, even through different iSCSI gateways. For
instance, an iSCSI/SCPT <-> iSCSI/TCP gateway, or an iSCSI/TCP <->
iSCSI/RDMA gateway.

For an iSCSI/RDMA connection that doesn't get translated, there's really
no need for either the header or data CRCs (AFAICT).

> [UE] no intention to recommend NO CRC. On the contrary, I'm for CRC and

Well, there are folks on the list who see cases where they want CRCs off.
The WG has decided to accomidate them as long as we don't end up shooting
ourselves in the foot. Making ULPs think they need to add their own CRC
'cause ours might be off is what we consider, "shooting ourselves in the
foot."

My understanding of the budding concensus is that that's ok to request
disabling CRCs as long as the admin is able to determine that even w/o the
CRC, the connection will be as reliable as if it had the CRC enabled. Thus
ULPs are ASSURED that they have at least CRC-32C reliability.

Take care,

Bill