Subject: Re: [rddp] Marker Interval

From: Michael Krause <[email-masked]>
Date: Mon, 24 Feb 2003 07:06:44 -0800

At 09:29 AM 2/24/2003 -0500, [email-masked] wrote:
>Somesh Gupta wrote:
>
> > >As David pointed out, a smaller marker
> > >interval puts an unacceptable burden on a software client
> > >(I thing the example was of a 64KB read/write requiring
> > >140+ markers - each of which causes creation of that many
> > >segments, plus creating and inserting markers and marker
> > >vectors etc) - worse than simply using a NIC doing a large
> > >TCP send or TOE offload.
>
>That's definitely a problem.
>
>Caitlin Bestler wrote:
>
> > The pattern where the true payload is a repeating pattern.
> > 140 markers does not require 140 distinct descriptors.
> > If you look up old mini-computers you'll find many examples
> > of repeating groups in DMA control.
>
>That's mostly a statement about building new hardware.  A lot of
>existing DMA hardware, OS syscall interfaces, etc. don't support
>repeating descriptors.
>
>Michael Krause wrote:
>
> > The optimal design is a hardware implementation that provides marker
> > insertion / removal.  The objective is to insure that one can implement
> > and interoperate with a software-based solution without creating a
> > complete performance nightmare but also with the understanding that a
> > software-based solution will not perform as well as a hardware-based
> > solution.
>
>The 512-byte fixed marker interval appears to have been over-optimized
>for optimal (new hardware) designs to the detriment of (re)use of
>existing software stacks.  The design rationale that I've seen based
>on minimum required IP payload sizes and the desire for a "flow-through"
>implementation is not very convincing, especially as the current MPA
>design can't achieve "flow-through" due to the inability for a receiver
>to determine whether an aligned stream will stay aligned (yes, one
>could build a "flow-through" implementation, but it will have robustness
>problems  akin to a TCP implementation with inadequate buffering).  In
>addition, we can go back through the "there's no such thing as a buffer-
>less [R]NIC discussion" if that's necessary, but more to the point is
>that the number of descriptors for reasonable-sized operations
>(e.g., 16 or more for a single 8KB transfer) is hard to justify.

The 512-byte fixed marker interval proposed was based on requirement to be 
compliant with existing RFCs, analysis of packet sizes relative to targeted 
workloads, ability to efficiently implement in hardware, etc.

Depending upon the software stack implementation, the insertion of markers 
can be accomplished at the same time one is copying / calculating the 
checksum for the packet thus it is relatively isolated within the 
stack.   This was verified by examination of various OSV  offerings 
including active support by multiple OSVs who stated this was not an issue 
after they had examined their implementations to validate the proposal and 
its cost of implementation.

The consensus of many involved in the draft development was that due 
diligence was done in examining the impacts to software and make the right 
trade-off between the two types of implementations.

Mike

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 24 Feb 2003 09:42:05 -0800

The discussion is related to existing software stacks and
interfaces. For storage protocols (as is for iSCSI), this
forms the first wave of client side implementations.

Case in point, even though the knowledge of doing checksum
while copying has been around for a long time, how many
actually implement it.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Michael Krause
> Sent: Monday, February 24, 2003 7:07 AM
> To: [email-masked]
> Subject: Re: [rddp] Marker Interval
> 
>

---

From: Michael Krause <[email-masked]>
Date: Mon, 24 Feb 2003 10:04:00 -0800

>forms the first wave of client side implementations.

Again this is a function of implementation and a well-layered 
implementation can be delivered without requiring any network stack 
modifications if desired.  Same applies to software based iSCSI which has 
been demonstrated by multiple vendors without network stack 
modifications.  Some of these implementations now deliver sufficient 
performance that current customers using them are not raising any major 
objections.  Given this can be done with iSCSI and markers are clearly less 
complex to implement / support in software than iSCSI, one might argue that 
the complexity / cost issues are not as high as some assert.

>while copying has been around for a long time, how many

A lot of the Unix implementations I'm familiar with clearly do.  I'll let 
Windows speak for itself but it isn't hard and has been done for quite some 
time and I don't see any issue with adding in a marker bit of code to the 
mix.  Same will apply to adding in SCTP's CRC for those implementations 
using SCTP today.  The key is that this is all implementation specific and 
there are clear demonstrations of shipping product showing that the 
complexity / performance cost is not high and thus a reasonable balance is 
available with MPA / DDP / RDMA as currently proposed in the draft 
specifications.

Mike

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 24 Feb 2003 11:10:25 -0800

> Sent: Monday, February 24, 2003 10:04 AM
> To: [email-masked]
> Subject: RE: [rddp] Marker Interval
>
>

  I don't know what the point here. You can run a distributed
  database over a software implementation which is clearly less complex
  than markers. How does that imply that the cost of markers is not
  high? In terms of complexity, it is close to 0 complexity to
  implement markers in software (especially when you don't have to
  modify the stack to do so).

---

From: Michael Krause <[email-masked]>
Date: Mon, 24 Feb 2003 12:07:36 -0800

At 11:10 AM 2/24/2003 -0800, Somesh Gupta wrote:

>   I don't know what the point here.

- That markers are not complex to implement nor are they that major of a 
performance loss as the data is already being moved from one buffer to 
another since the hardware could not perform the direct placement when 
using a software stack.

- That markers can be implemented on top of an existing network stack 
keeping things well layered which is one of the objectives being stated on 
this e-mail string.

>You can run a distributed database over a software implementation which is 
>clearly less complex
>   than markers. How does that imply that the cost of markers is not high?

I've requested a definition of "high" as the discussion to date is perhaps 
too subjective.  I described a potential implementation to illustrate what 
can be done that IMO is not complex or a major performance inhibitor.

>In terms of complexity, it is close to 0 complexity to implement markers 
>in software (especially when you don't have to modify the stack to do so).

Then we are in agreement that the complexity is not high in software for a 
well-layered implementation. Is there any remaining disagreement in this 
regard?

Mike

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 24 Feb 2003 16:46:45 -0800

Software implementation should not be considered the same as
user space apps.

Many of the iSCSI implementations are implemented in the kernel and
so is NFS. I assume that if one of main apps for RDMA is storage,
then kernel based implementations (iSCSI and some NFS variant) will
be common. In such as case there is no data copy needed and other
costs will continue to dominate.

Regarding the CRC debate, I don't believe that NAS or iSCSI is
broken because CRC is not part of the spec or not implemented
by software based iSCSI stacks.

It is very useful to look to take a deeper look at the paper
behind this whole debate. Most of the bugs were traced to bad
implementations at the end systems. A NIC that corrupts data
before performing checksum calculations will corrupt data
before CRC calculation.

The CRC will provide additional protection as the data passes
through switches but there are no studies focused on those
errors and their escape rates from TCP checksum.

Somesh

> Sent: Monday, February 24, 2003 12:08 PM
> To: [email-masked]

---

From: Michael Krause <[email-masked]>
Date: Tue, 25 Feb 2003 06:38:52 -0800

>costs will continue to dominate.

There may not be a data copy on outbound but certainly there will be on 
in-bound for many implementations of iSCSI, NFS, etc. as the operations do 
not necessarily align with the target buffers.

>by software based iSCSI stacks.

No one stated they are broken nor that neither of these types of solutions 
do not use CRC in their software based stacks.  Same argument applies to 
solutions implemented on top of SCTP which provides a CRC I believe in 
place of the Addler checksum (need to go back and re-read that RFC again to 
verify).

>before performing checksum calculations will corrupt data

CRCs also exist to mitigate unprotected intermediated devices which may not 
provide ECC protection of their associated memories.  As hardware moves to 
ever smaller processes, end-to-end protection requirements increase.  I've 
heard many a customer state that they will not accept technology that does 
not provide end-to-end protection as silent data corruption is not 
acceptable - it does not matter whether it is in someone's implementation 
as they are examining the solution as a whole and don't want to have to 
take into account every physical device along the way.