Subject: RE: Optional CRCs ( was [rddp] Marker Interval )

From: "Barron, Dwight" <[email-masked]>
Date: Mon, 24 Feb 2003 17:15:06 -0600

> -----Original Message-----
> From: Williams, Jim [mailto:[email-masked]]
> Sent: Monday, February 24, 2003 4:23 PM
> To: [email-masked]
> Subject: Optional CRCs ( was [rddp] Marker Interval )
> 
> 
> 
> 
> > -----Original Message-----
> > From: Barron, Dwight [mailto:[email-masked]]
> > Sent: Monday, February 24, 2003 4:28 PM
> > To: Somesh Gupta; [email-masked]
> > Subject: RE: [rddp] Marker Interval
> > 
> > 
> > There are clearly a wide spectrum of implementation 
> > possibilities, but I like to focus on the endpoints first, 
> > then determine how many additional click-stops are necessary 
> > to adequately support the middle of the spectrum. Clearly, 
> > support of an ULP that wants to invoke RDDP over an 
> > unmodified TCP stack is an important goal, but optionality 
> > isn't free, so let's examine what we are really buying with 
> > negotiable marker intervals. I'll assume that the ULP using 
> > RDDP wants to see RDDP semantics (whatever that may be) and 
> > not interim RDDP headers and components. If you assume this 
> > RDDP "library" lives on top of unmodified sockets/TCP, then I 
> > believe there is no choice but for a receive stream to a) 
> > sequentially read every byte for purposes of CRC calculation 
> > and b) copy incoming stream data to separate the RDDP headers 
> > from ULP payload. This is especially true of tagged messages 
> > where the ULP is using a contiguous memory address range to 
> > reference data. On the transmit side, the R!
> > DDP library will still have to touch every byte for CRC 
> > calculation purposes, 
> 
> I think if we are serious about supporting unmodified TCP, then
> the CRC should be optional.  NFS typically runs with only TCP/UDP
>
I think there is a significant delta between "serious about supporting" and "serious about optimizing for". Quoting from the RDDP charter (http://www.ietf.org/html.charters/rddp-charter.html) "RDDP capabilities refer to the ability to place data directly from the NIC into application buffers, without intensive CPU usage. This strategy avoids the costs of data copying and enables using IP as a method for high speed buffer to buffer transfer, allowing IP to replace special purpose networks currently in use." I believe we know how to optimize to deliver this behavior at one end of the implementation spectrum, but have no illusions of delivering this behavior with a pure software implementation. 

> checksum for protection.  Not a good idea to do this over the 
> public internet, but then critical data traveling through
> the public internet will (or should) use IPSec protection.
> 
And where does the IPSEC processing power come from? I would be willing to bet there is a large overlap of candidate environments that need to run RDDP in unmodified software stacks that will need to run IPSEC in that same unmodified software stack. IPSEC is great in its own right, but is certainly not a good tradeoff when a CRC would suffice. 

> iSCSI is a precedent for optional CRCs.  Data errors in iSCSI
> or NFS are equally catastrophic as RDDP.  Tradeoff can be
> made by user as to required protection, quality and reliability
> of network infrastructure, etc., to decide whether to use CRCs.
> 
> > but may try to avoid copying by using 
> > multiple send calls. But at what point does copying become 
> > cheaper than multiple send calls? Especially if the sender is 
> > at least *attempting* to provide header alignment, which 
> > would mean adding RDDP headers for each TCP_MAXSEG interval. 
> > While we all like to avoid data copies (the whole reason for 
> > RDDP to begin with), once we bite the bullet for software 
> > CRCs (which is clearly the case with standard sockets), the 
> > copy really isn't that painful. 
> > 
> > Regards,
> > Dwight 
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Tue, 25 Feb 2003 11:38:25 -0500

Dwight,

WG chair hat off ...
 
> > > I think there is a significant delta between "serious about 
> > > supporting" and "serious about optimizing for".
> > 
> > An alternate viewpoint is that "serious about supporting" includes
> > "not placing unreasonable implementation requirements upon". 
> 
> But unreasonable from which perspective. If we were assuming 
> symmetric implementations at each end, with roughly equal 
> costs for performing a function, then optionality can be 
> evaluated from end user trade-offs, usually functionality 
> (like CRC) vs. performance or cost to implement or whatever. 
> Asymmetric implementations mean that one end will (willingly) 
> bear more of a burden for the benefit of the other. I have 
> not heard nor seen any evidence that a CRC is an unreasonable 
> burden to implement in hardware. It is standard practice for 
> modern IETF protocols (SCTP), other modern storage and 
> datacenter interconnect technologies (FC, IB).

I generally agree with much all of the above discussion, but the
NFS example is conspicuous by its absence, as NFS is definitely
a candidate ULP for RDDP.

> Why do we want to burden over-burdened administrators with
> yet another option (would you like data integrity with that?).

That doesn't follow.  A "mandatory to implement" requirement plus
an implementation policy of "will not communicate with a peer who
does not use CRCs" would avoid the administrative burden issue,
and would be a good thing to describe as an acceptable policy.

> The pervasive OSV support was exactly my point, unmodified 
> existing software stacks. However, I have been told, but have 
> not personally confirmed, that there is no easy/standard way 
> for a ULP above TCP to discover if IPSEC is actually being 
> invoked on a given connection with these existing stacks. If 
> this is actually the case, then exploiting overlapping 
> IPSEC/CRC coverage would seem to be left as yet another 
> exercise for the administrator.

Or the configuration software that sets out to make the
administrator's life easier by coordinating IPsec and RDDP
configuration.

> > And there are two scenarios discussed above where the CRC does not
> > appear to be required.
> 
> Again, these were/are very valid architectural decisions when 
> software implementations were heavily weighted in the 
> decision process. Hardware equivalent architectural decisions 
> (e.g. FC) that carry the exact same class of storage traffic 
> always have CRCs enabled.

WG chair hat on for the following two paragraphs:

RDDP is about more than just SCSI storage traffic.  If RDDP were
scoped only to IP Storage (and was in the IP Storage WG), this
argument would carry much more significance.

In case it's not clear, "mandatory to use" is a stiff requirement.
The burden of demonstrating the need for such a requirement is
generally on those advocating it.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Tue, 25 Feb 2003 07:03:07 -0800

At 09:43 AM 2/25/2003 -0500, [email-masked] wrote:
>"not placing unreasonable implementation requirements upon".  NFS
>as currently used does not need CRCs (although NFS did learn the
>hard way that turning on checksums was a real good idea), and ...
>
> > And where does the IPSEC processing power come from? I would be
> > willing to bet there is a large overlap of candidate environments
> > that need to run RDDP in unmodified software stacks that will
> > need to run IPSEC in that same unmodified software stack. IPSEC
> > is great in its own right, but is certainly not a good tradeoff
> > when a CRC would suffice.
>
>To begin with, there's quite a bit of OSV support for IPsec in
>existing stacks today.  The point here is not that IPsec is
>computationally cheap (it's not), but rather that in an environment
>where IPsec is required for security policy reasons and the IPsec
>coverage is similar to the CRC coverage, the CRC may not be needed
>as the 96-bit IPsec MAC provides a considerably stronger integrity
>check - see Section 3.6 of draft-ietf-ips-security-19.txt.  As
>Jim Williams wrote:

The reason I have heard quoted by many is the IPSec is often implemented in 
a gateway device and is therefore not end-to-end which is why end-to-end 
CRC is still required by many.  I know from discussions with customers on 
the use of end-to-end CRC that none view it is as optional for iSCSI.  Yes, 
for initial evaluation purposes with early technology they are willing to 
live with it but for actual production deployment, they are not willing to 
live without it on and IPSec is not a substitute as they are using VPN 
gateways which are relatively cheap to doing this in an end device.

Mike

> > > iSCSI is a precedent for optional CRCs.
>

---

From: [email-masked]
Date: Tue, 25 Feb 2003 11:03:46 -0700

David,

The TCP alternative checksum of 1146 (Fletcher)would not have been an adequate alternate solution because it provides much weaker protection against undetected errors than CRC. It does provide some improvement over the protection of the normal TCP checksum but the improvement is only modest. I provided the analysis of Fletcher and Adler (another checksum alternative) to IPS reflector at the time this was discussed. The impact of the alternative checksums are summarized and compared to the protection of a CRC in RFC 3385.

Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, February 25, 2003 6:26 AM
To: [email-masked]; [email-masked]
Subject: RE: Optional CRCs ( was [rddp] Marker Interval )

> Isn't SCTP CRC protection required per chunk?

Actually, it's per packet (in the SCTP common header), as Caitlin has
pointed out.  SCTP did not have a checksum that needed to be preserved,
in contrast to TCP.  The IPS WG considered the TCP alternate checksum
mechanism in RFC 1146 as a means of using a CRC to replace the TCP
checksum, but that mechanism is undeployable in practice due to what
it breaks, even though it would have been a cleaner solution in principle.

> The reason to have a CRC for 
> MPA is to provide data integrity below DDP/RDMA  / ULP which has been 
> demonstrated as a concern during the development of iSCSI.  Do we really 
> need to go back through the entire IPS debate as to why CRCs are good 
> things here or accept that a CRC at or above the transport but below 
> DDP/RDMA  / ULP is a good thing?

We don't need to go back through that debate, because the question
isn't about "good to implement" - it's about "mandatory to use", which
is a very different issue.  CRCs are optional for iSCSI, and NFS (another
important client for RDDP) has never needed CRCs, as the TCP checksum
has been adequate for its purposes (yes, a CRC would be an improvement).

Thanks,
--David

---

From: [email-masked]
Date: Tue, 25 Feb 2003 13:22:42 -0500

Pat,

I completely agree about the strength of Fletcher checksums -
what I was intending to refer to was a potential use of RFC 1146's
mechanisms to replace the TCP checksum with a real CRC.  This would
have involved defining a new chksum value for the CRC, but that
isn't practical because when there isn't a TCP checksum in the
TCP checksum field, too much stuff breaks.  This idea was rejected
out-of-hand after being briefly considered.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, February 25, 2003 1:04 PM
> To: [email-masked]; [email-masked]; [email-masked]
> The TCP alternative checksum of 1146 (Fletcher)would not have 
> been an adequate alternate solution because it provides much 
> weaker protection against undetected errors than CRC. It does 
> provide some improvement over the protection of the normal 
> TCP checksum but the improvement is only modest. I provided 
> the analysis of Fletcher and Adler (another checksum 
> alternative) to IPS reflector at the time this was discussed. 
> The impact of the alternative checksums are summarized and 
> compared to the protection of a CRC in RFC 3385.
> 
> Pat
>

---

From: "Barron, Dwight" <[email-masked]>
Date: Tue, 25 Feb 2003 14:09:43 -0600

Somesh,
I am not discounting the importance of software implementations to enable RDMA, even in the datacenter. What I am questioning is the supposition that CRC is so performance prohibitive that it is "placing unreasonable implementation requirements upon" software stack implementations and therefore needs to be optional. The SCTP working group apparently came to the conclusion it does not and therefore made CRCs mandatory. To my knowledge there have not been any complaints on this reflector that SCTP's CRC makes RDDP/SCTP unreasonable to implement in software. While one may claim that an RDDP/TCP CRC is superfluous or unnecessary, I contend that the SCTP decision is still precedent for the unreasonable test. 

My personal observations/experience [your mileage may vary caveat goes here] with iSCSI suggests that the optional CRC didn't buy the "unmodified software stack" implementations as much benefit as might be expected. The software implementations ended up doing buffer copies after all (which means computing a CRC wouldn't have cost that much more) and they are being targeted at environments where there are either excess CPU cycles available or the storage traffic overhead is only accounting for 10-20% of the CPU. Clearly there are software implementations that are using SGLs to avoid copying, but I don't think they fall into the "unmodified software stack" category (either user space or kernel mode access to the network stack), which I believe was a fundamental tenet of this thread. 

Regards,
Dwight

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Tuesday, February 25, 2003 12:13 PM
> To: Barron, Dwight; [email-masked]; [email-masked]
> We went through some similar debates in iSCSI with one camp
> assuming primarily hardware based initiators and targets,
> and the other group assuming a number of software based
> implementations (I started off assuming hardware based but
> switched later).
> 
> As it turns out, the adoption is being driven by software
> based initiators and it is a good thing that the optionality
> allowed software based initiators to avoid the features 
> that would have made this avenue futile.
> 
> Perhaps there would not be so much debate if RDMA was focused
> on IPC in the data center. But with storage being a primary apps,
> it is important to enable this adoption route. Negotiated
> options seem like a good engineering choice.
> 
> Somesh
> 
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Barron, Dwight
> > Sent: Tuesday, February 25, 2003 7:58 AM
> > To: [email-masked]; [email-masked]
> > > Sent: Tuesday, February 25, 2003 8:44 AM
> > > To: Barron, Dwight; [email-masked]
> > symmetric implementations at each end, with roughly equal costs 
> > for performing a function, then optionality can be evaluated from 
> > end user trade-offs, usually functionality (like CRC) vs. 
> > performance or cost to implement or whatever. Asymmetric 
> > implementations mean that one end will (willingly) bear more of a 
> > burden for the benefit of the other. I have not heard nor seen 
> > any evidence that a CRC is an unreasonable burden to implement in 
> > hardware. It is standard practice for modern IETF protocols 
> > (SCTP), other modern storage and datacenter interconnect 
> > technologies (FC, IB). Why do we want to burden over-burdened 
> > administrators with yet another option (would you like data 
> > integrity with that?). Note that I am not trying to be 
> > unsympathetic to the needs and desires of efficient software 
> > implementations, I am trying to balance the realities asymmetric 
> > implementations and the need for optionality, both from an 
> implement!
> > ation/test perspective and the end user administrator. 
> > 
> > 
> > > NFS
> > existing software stacks. However, I have been told, but have not 
> > personally confirmed, that there is no easy/standard way for a 
> > ULP above TCP to discover if IPSEC is actually being invoked on a 
> > given connection with these existing stacks. If this is actually 
> > the case, then exploiting overlapping IPSEC/CRC coverage would 
> > seem to be left as yet another exercise for the administrator. 
> > 
> > 
> > > Jim Williams wrote:
> > > 
> > software implementations were heavily weighted in the decision 
> > process. Hardware equivalent architectural decisions (e.g. FC) 
> > that carry the exact same class of storage traffic always have 
> > CRCs enabled. 
> > 
> > > 
> > > Thanks,
> > > --David

---

From: Mike Eisler <[email-masked]>
Date: Wed, 26 Feb 2003 07:17:49 -0800

>Just so no one is confused, while NFS typically runs over
 >TCP or UDP using their default checksums and most find that
 >an adequate solution, NFS also can optionally use RPCSEC_GSS
 >to add integrity checking if desired. So NFS has belts and
 >suspenders, although most just wear a belt. Adding yet another
 >integrity check is not required for NFS.

Violent agreement here. It is the duty of the transport
to ensure reliable delivery, not NFS.

However, while RPCSEC_GSS indeed provides
integrity checking, if the server fails to verify the
arguments of the remote procedure call, an error is
returned to the caller. The caller normally treats
this error as a server side "xdr decode error",
and the error is propogated to the application. The
rationale is if a message passed the transport level
checksum, but failed the crypto-level checksum,
then an attacker is clearly trying to tamper with the
traffic.

One could certainly extend the semantics of RPCSEC_GSS
to CRC-style checksumming that would result in the
RPC-level of the stack re-transmitting. But that semantic is
not currently specified (and as I've noted above, a
layering violation, imho).

    -mre