Subject: [rddp] Optimized Processing not necessarily Hardware Optimized

From: Caitlin Bestler <[email-masked]>
Date: Fri, 11 Apr 2003 12:07:15 -0500

David chose the following paragraph from John Hufferd.

> I find some of these arguments somewhat strange.  We have
> all entered into this effort to bring optimized
> processing with Remote Direct Data Placement, yet we
> continue to talk about, and give serious weight to, the
> argument concerning an all software implementation.  It
> seems that we might think about a complete software
> implementation, in passing, but this is NOT the goal of
> RDMA or this RDDP workgroup (in my opinion).

Read carefully, the statement correctly states that the
purpose of RDDP is optimized processing. It does not state
that the goal is hardware optimized processing, allthough I
think most people read it that way.

This is a very critical distinction.

In my opinion, the cause of RDMA has been hindered by
arguments made on its behalf that assume it will be
implemented in hardware. Indeed, that its implementability
in hardware is a justification for RDMA.

Ultimately, a justification for RDMA that current stacks
result in too many context switches do not justify the
development of a new transport protocol. They justify the
development of new motherboards.

But RDMA is not merely a hardware optimization. It
fundamentally improves the collaboration between. It
allows two hosts to achieve the same results with less
work, no matter how the hosts are organized/built.

With RDMA a host (through whatver combination of hardware,
firmware, kernel drivers, co-processors and software
libraries):

    is able to directly place the payload from received DDP     
    Segments into user memory, even if the DDP segments are 
    received out of order.
    
    has fewer required interactions with ULP software.
    
    is able to send content in the order most convenient to
    the Data Source, not in an arbitrary order that the
    protocol requires the Data Sink to dictate.
    
It is these fundamental optimizations that justify RDMA.
The fact that they are simple powerful algorithms is *why*
they can be implemented in hardware. But their hardware
implementability is the result of their strength, not the
other way around.

I would like to examine the question of "optional" CRCs in a
similar way. What is the fundamental difference in the work
that is required with or without mandatory CRCs? To what
extent does the proposed protocol require work that is
an undue burden for some applications?

As many have already argued, the need for payload validity
checking increases with network speed. While there are a
handlful of applications that do not require protection
against payload corruption (including many that I have
worked on such as streaming media), this is clearly an
exception. 

At the speeds we are envisioning supporting, payload
validity checking is not a specialized requirement of
storage applications being imposed upon everyone else.
The question is more whether the handful of appliations that
do not need it should be able to impose it being variable.

I believe the impact on the applications that can live
with data loss is very slight, in addition to those
applications representing a small portion of total
network usage.

First, the term "optional" CRC coverage is misleading.
CRC coverage for the DDP header is mandatory. RDDP cannot
responsibly place data directly into user buffers based
soley upon TCP validation.

That would require the application to confine the Protection
Domain to only those buffers that could be corrupted. With
only TCP protection, all buffers exposed through the
Protection Domain would be subject to being falsely
overwritten by any flawed TCP segment that escaped
detection.

Working around this would require additional Protection
Domains, and hence additional connections. Clearly this is
unacceptable.

So the "optional" CRC is really a *shortened* CRC. It covers
only the header length, rather than the full length.

This is additional complexity, and requires a mode-setup
message exchange at the start of a connection (although
one-way messages in each direction are adequate, so at least
no additional round trip is mandated).

All receivers, still must be able to calculate CRCs.

The only difference is that some of them may be able to stop
after the DDP header, while most application will want to
cover the entire FULPDU.

What is the benefit in stopping, however? What is the undue
burden being imposed on a "software implementation"?

The basic argument is that calculating the full CRC will
result in *additional* memory touches.

But examine that more closely. Obviously the memory is
touched. It has to be copied to the user buffer.

The reason that this is an "additional" touch is that on
many systems there is already an *existing* hardware
solution to perform that memory copy, but that *existing*
hardware solution does not know how to perform a CRC on an
MPA FULPDU.

Further, that existing hardware *does* know how to validate
a TCP segment. If it didn't the pass to validate the TCP
segment could also validate the MPA FULPDU, so there would
be no "additional" memory touch required at all.

As I stated earlier, if the only problem with TCP/SCTP and
other existing transports is that they required too many
conext switches, then the correct response would be that new
motherboards needed to be revised, not new transports.

Is the only justification for a shortened CRC that many
motherboards have dedicated co-processors ("DMA Engines")
that understand TCP validation but not MPA?

If so, I would argue that we are complicating the protocol
for the benefit of *existing* hardware.

Clearly, calculating the CRCs is possible because it will
have to be done for the duration of the header and for the
vast majority of payload anyway.

The required processing is very slight, so the only real
issue is the "extra" memory touch. As Mike Krause pointed
out earlier, in many cases cache dynamic will eliminate that
overhead anyway. And as just covered, it is only
"additional" because of the limitations of *current*
hardware designs.

The overhead is minor *and* transitional. It provides no
justification for the complexity of variable CRC coverage.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp