Subject: RE: [rddp] Option or mandatory CRC in MPA

From: John Hufferd <[email-masked]>
Date: Tue, 8 Apr 2003 18:04:17 -0700

I find some of these arguments somewhat strange.  We have all entered into 
this effort to bring optimized processing with Remote Direct Data 
Placement, yet we continue to talk about, and give serious weight to, the 
argument concerning an all software implementation.  It seems that we 
might think about a complete software implementation, in passing, but this 
is NOT the goal of RDMA or this RDDP workgroup (in my opinion). Therefore, 
I do not think it is important to do things that will make the protocol 
more complicated for this small segment of the market, assuming that it 
exist at all other then for a testing platform. 

Most of the applications that will be using the RDMA protocols already 
have existing TCP/IP implementations, and they are not enhanced or 
accelerated by a software implementation of RDMA.  If systems only have a 
standard NIC, then the applications should continue to use TCP/IP. 

Expanding on this point further, the work being done in various places on 
Socket Direct Protocols, will permit the application to continue to use 
the Socket APIs, and the OS will convert that to RDMA where possible and 
reasonable.  I submit that will be when there is an RDMA chip set 
available.

There are other applications that I am aware of that are "adding"  the 
RDMA capability, not taking away the TCP/IP  implementation.  They check 
to see which one they can use and act according. 

So again, being overly concerned about a software implementation of RDMA 
is not something on which we should spend much time. 
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Williams, Jim" <[email-masked]>
Sent by: [email-masked]
04/08/2003 05:17 PM

        To:     "[email-masked]'" <[email-masked]>
        cc: 
        Subject:        RE: [rddp] Option or mandatory CRC in MPA

The argument for making the CRC optional
in the RDDP over TCP mapping is the following.

  A.  A large percentage of usage will be on
      highly reliable private networks where
      the TCP checksum is more that sufficient
      the achieve the desired level or reliability.

  B.  Optimizing the performance of software 
      implementations is important, and computing
      an unneeded CRC is too high a price to pay
      for avoiding the need to negotiate an option.

The input from NFS is important because it takes
into account years of field experience.  If they
determine that for most all usage, even on private
networks, the TCP checksum is NOT adequate, then
I would regard that as a refutation of point A.
If they determine that for a large enough percentage
of implementations the TCP checksum IS adequate,
then I would regard that as support of point A.

Regarding point B, if one envisions that all 
implementations of interest are hardware customized
for RDDP, then the point does not carry much weight.
However, I would argue that this may be too narrow
a view for the IETF to take.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 9 Apr 2003 10:53:19 -0500

On 4/8/03, John Hufferd wrote:

>Most of the applications that will be using the RDMA
>protocols already have existing TCP/IP implementations,
>and they are not enhanced or accelerated by a software
>implementation of RDMA.  If systems only have a standard
>NIC, then the applications should continue to use TCP/IP. 

There are two sides to every client/server relationship.

A software implementation of RDDP on the client side allows
the server side to have fully optimized RDDP. Presumably
with hardware optimization, but even without hardware
optimization there are major advantages to storage servers
in the freedom to deliver material in any order convenient.

Review the draft applicability statement, many are covered
there. These are important. If RDDP was solely of use to
enable a hardware justification then it would not merit
IETF consideration. Motherboards should be changed before
the Internet. The more important point is that the protocol
enables more efficient collaboration, especially for storage
applications, no matter how either end is implemented.

That said, I don't think we need to make the protocol more
complex to save CPU cycles for software implementations.
Calculating CRCs is not difficult. The fact that it is more
easily done in hardware does not mean that it is hard to do
in software.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Brent Callaghan <[email-masked]>
Date: Tue, 08 Apr 2003 18:25:19 -0700

John Hufferd wrote:
> :
> accelerated by a software implementation of RDMA.  If systems only have 
> a standard NIC, then the applications should continue to use TCP/IP. 

Perhaps.  Though a credible scenario is a large number of
cheap and/or legacy clients doing RDMA to a large, heavily
loaded server (not uncommon).

In that environment, it might be attractive to have the clients
implement RDMA in software, and equip the server with the fancy
RDMA hardware to relieve its suffering.

	Brent

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 10 Apr 2003 10:54:49 -0700

I can see that the software-based client scenario is perhaps
credible for the first-generation of products.  But let me note
that the performance % delta incurred by supporting a mandatory
CRC is likely to be insignificant even for that first generation,
given all the RDMAP/DDP/MPA/TCP/IP protocol processing in 
software that must happen.

From the first-hand experience of being very closely involved in 
iSCSI error recovery design largely predicated on iSCSI digest
failures (which in turn are dependent on TCP checksum "escapes"), 
the single biggest attraction for me in a mandatory MPA CRC is
the deterministic QoS of the transport substrate from the ULP 
perspective.  That leads to obvious ULP protocol optimizations.

As someone has earlier noted, data integrity at high speeds is
an extremely important consideration for storage (and FC satisfies
this today).  IMHO, we would be doing a disservice to the RDDP
technology by designing it to be "optionally deterministic" wrt QoS.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Apr 2003 12:55:24 -0700

It is difficult for people to definitively say what will or
what will not happen in the marketplace. As an example, we
went thru all of these debates during iSCSI protocol development.

As it turns out, bulk of the products are and may continue
to be "software based".

1. Most initiators at this time are in the category and may
continue to be software based

2. Many (if not majority) of targets are "software based" and
may continue to be.

Interesting, on the target side the processors used are not
these high end P-4s, but slower embedded processors with
smaller caches (32KB or whatever).

In these instances, everything that is done (CRC and mandatory
markers even on receive side where they provide no benefit)
solely from the perspective of a protocol optimized for hardware
implementations is likely to have a significant performance
impact.

I think we should be very clear and explicit about whether
we allow software based products to play in this market on
an equal footing or not.

On one hand a protocol enabling software based products will
have a wider acceptability and shorter time to market/revenue.
It may also have a greater chance as a standards track document.

On the other hand, a protocol focused on hardware based solns
will allow for greater optimization in silicon. There also seems
to be a greater meeting of minds in a certain group on this model.

Somesh

---

From: Michael Krause <[email-masked]>
Date: Thu, 10 Apr 2003 13:42:34 -0700

>2. Many (if not majority) of targets are "software based" and

Given the iSCSI spec just became a standard, it is understandable that 
there is a predominance of soft implementations.  However, not that it is a 
standard, there are people finalizing their hardware implementations thus 
the current state is clearly subject to change.

>implementations is likely to have a significant performance

Not clear.  Whether one walks memory to do a checksum calculation, a CRC 
calculation, or a copy operation, the net effect is the processor data 
cache is made hot with these cache lines.  Given applications typically ask 
for data once it has arrived, the fact that is in cache mitigates the cache 
load latency thus the subsequent performance impact.   I do not see this as 
creating a significant performance impact.  Do you have data to show that 
there is such an impact?

>we allow software based products to play in this market on

I thought there was general consensus that the focus of RDMA is hardware 
acceleration and that it must be able to be implemented in software.  There 
were no requirements put forth that I'm aware of that require software and 
hardware to have an equal footing especially as some operations such as a 
RDMA Read will always outperform a software implementation as no processor 
interactions are required when implemented in hardware.

>to be a greater meeting of minds in a certain group on this model.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 10 Apr 2003 16:03:45 -0500

On 4/10/03, Michael Krause wrote:

>Not clear.  Whether one walks memory to do a checksum
>calculation, a CRC calculation, or a copy operation, the
>net effect is the processor data cache is made hot with
>these cache lines.  Given applications typically ask for
>data once it has arrived, the fact that is in cache
>mitigates the cache load latency thus the subsequent
>performance impact.   I do not see this as creating a
>significant performance impact.  Do you have data to show
>that there is such an impact?
>
More importantly, there are no additional memory accesses
required that would not have been required to remove
markers.

Admittedly, you could have an adequate DMA engine to avoid
this problem. But then we're worrying about making things
too hard on implementations that have half of the problem
solved in hardware.

Efficient CRC algorithms are well known, even for software.
As you point out, it is *at worst* a pass through the
buffer. If that is unacceptable, then buy an RNIC.

Meanwhile, optionally performing the check will be a
complexity that *all* implementations will have to bear.

Keep in mind that the header MUST be checked. So an
"optional" CRC is really just a *shortened* CRC that
only covers the header. Absolutely no code savings,
only a reduction in memory touches for *some*
implementations.

>I thought there was general consensus that the focus of
>RDMA is hardware acceleration and that it must be able to
>be implemented in software.  There were no requirements
>put forth that I'm aware of that require software and
>hardware to have an equal footing especially as some
>operations such as a RDMA Read will always outperform a
>software implementation as no processor interactions are
>required when implemented in hardware.

It would also be impossible. Any fixed algorithm can
ultimately be performed with at least a slight edge in
hardware.

The goal is to enable software stacks mostly for quick
adoption by clients. I haven't seen arguments for mandating
that software be able to handle 10 Gb NICs on its own.
I think most people would be very happy with a software
stack that handled a GigE NIC.

Caitlin Bestler

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Apr 2003 15:07:45 -0700

Mike,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Michael Krause
> Sent: Thursday, April 10, 2003 1:43 PM
> To: [email-masked]

  I am fairly certain there will continue to be a range of implementations.

> creating a significant performance impact.  Do you have data to show that

  I think this brings up a number of interesting points:

1. We should assume checksum is offloaded as fairly all NICs do
   it now.
2. There are a certain class of apps that do not touch the data
   but just shunt it from one place to another. A classic example
   is any kind of storage server. These are actually the ones
   that benefit from zero copy the most and may use means other
   than RDDP to get the benefit (i.e. handle SGLs)
3. Obviously there are many other apps that chew on the data.
   In those cases, there is just a whole range of factors that
   come into play. The rate at which data can be burst vs the
   apps consumption of data, the size of the data caches etc
   etc. In a number of cases, it does not matter how much
   network acceleration you provide - the processing of the data
   consumes bulk of the cycles anyway.

> interactions are required when implemented in hardware.

  The statement of requirements should be very explicit on this.
  We can all agree that hardware acceleration will provide higher
  performance under all cases, but that is not the point.

  Somesh

---

From: Michael Krause <[email-masked]>
Date: Thu, 10 Apr 2003 21:46:45 -0700

>1. We should assume checksum is offloaded as fairly all NICs do

It is common for single segment, IPv4 packets.  It is not common for IPv4 
fragments or for IPv6 (IPv6 is complicated by the next header field making 
it more difficult to support checksum off-load as the number of headers, 
etc. is somewhat unpredictable).

>    than RDDP to get the benefit (i.e. handle SGLs)

This primarily is true for the outbound path where one encapsulates the 
file system or storage block.  The inbound path usually requires a copy 
operation from the mbuf to the storage block / file system due to the 
deltas in size between the layer 2 packet and the blocks up above.

>    consumes bulk of the cycles anyway.

Acceleration does play a role especially when it comes to RDMA because the 
data is placed in the right place from the start thus the correct thread / 
processor is always chosen to process the data.  This is very helpful for 
inbound traffic.  Outbound just takes advantage of copy avoidance if there 
is OS bypass or special mappings to allow a kernel network stack to see 
into user-space (not always true or simple to support on many OS offerings).

> > acceleration and that it must be able to be implemented in
> > software.  There were no requirements put forth that I'm aware of that 
> require
> > software and  hardware to have an equal footing especially as some 
> operations such as a
> > RDMA Read will always outperform a software implementation as no
> > processor interactions are required when implemented in hardware.
>

I believe there was a question of whether the two types of implementations 
were on equal footing - if one performs potentially radically differently 
than the other and has additional functionality that while possible to 
emulate in software is clearly not the same in terms of remote endnode 
interaction / solution cost, then it is not possible to be on equal footing 
and thus the motivation to compromise on functionality may be quite a bit 
less given this delta.

Mike

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 11 Apr 2003 09:54:18 -0700

> Sent: Thursday, April 10, 2003 9:47 PM
> To: [email-masked]
> calculation, a CRC

  The IPV4 single segment continues to be > 99% of the traffic over TCP.
  And many of the NICs do support checksum for IPV6 (the driver may not).
  The host may be required to fix some stuff.

> deltas in size between the layer 2 packet and the blocks up above.

  Since the SCSI/FC controllers do support scatter-gather lists, it
  is a buffer management issue. With TOE and header-data split, those
  that do not want to complicate their buffer management can get the
  benefits without a change to the wire protocol.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 11 Apr 2003 14:52:04 -0500

On 4/11/03, Somesh Gupta wrote:

>> It is common for single segment, IPv4 packets.  It is
>> not common for IPv4 fragments or for IPv6 (IPv6 is
>> complicated by the next header field making it more
>> difficult to support checksum off-load as the number of
>> headers, etc. is somewhat unpredictable).
>
> The IPV4 single segment continues to be > 99% of the
> traffic over TCP. And many of the NICs do support
> checksum for IPV6 (the driver may not). The host may be
> required to fix some stuff.

Making protocol design decisions on the presumption that
future Internet traffic will remain IPv4 is unlikely to be
persuasive with the Area Directors.

The fact that some *current* hardware knows how to handle
IPv4 validation, but not IPv6, merely re-inforces my earlier
point that these are not intrinisic flaws with the protocol.
They are only transitional issues. Unless there is a severe
penalty with the interim solutions, I would be very
reluctant to impose a *permanent* penalty (in the form of
option tracking and negotiation) to solve an *interim*
problem.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Sat, 12 Apr 2003 23:48:30 -0700

>   The host may be required to fix some stuff.

Some NICs support IPv6 checksum off-load if the next header is the 
transport header.  If the next header is something else, then checksum 
off-load is relatively rare.  There is also the lack of support in many 
NICs when things get fragmented.

>   benefits without a change to the wire protocol.

I thought we were discussing a software implementation and not a protocol 
off-load in hardware solution?  The performance issues that were raised 
were about the impact to the software implementation thus jumping into a 
hybrid isn't quite an apples-to-apples comparison.

Mike