Subject: [rddp] Marker Interval

From: [email-masked]
Date: Mon, 24 Feb 2003 09:29:39 -0500

Somesh Gupta wrote:

> >As David pointed out, a smaller marker
> >interval puts an unacceptable burden on a software client
> >(I thing the example was of a 64KB read/write requiring
> >140+ markers - each of which causes creation of that many
> >segments, plus creating and inserting markers and marker
> >vectors etc) - worse than simply using a NIC doing a large
> >TCP send or TOE offload.

That's definitely a problem.

Caitlin Bestler wrote:

> The pattern where the true payload is a repeating pattern.
> 140 markers does not require 140 distinct descriptors.
> If you look up old mini-computers you'll find many examples
> of repeating groups in DMA control.

That's mostly a statement about building new hardware.  A lot of
existing DMA hardware, OS syscall interfaces, etc. don't support
repeating descriptors.

Michael Krause wrote:

> The optimal design is a hardware implementation that provides marker 
> insertion / removal.  The objective is to insure that one can implement
> and interoperate with a software-based solution without creating a
> complete performance nightmare but also with the understanding that a
> software-based solution will not perform as well as a hardware-based
> solution.

The 512-byte fixed marker interval appears to have been over-optimized
for optimal (new hardware) designs to the detriment of (re)use of
existing software stacks.  The design rationale that I've seen based
on minimum required IP payload sizes and the desire for a "flow-through"
implementation is not very convincing, especially as the current MPA
design can't achieve "flow-through" due to the inability for a receiver
to determine whether an aligned stream will stay aligned (yes, one
could build a "flow-through" implementation, but it will have robustness
problems  akin to a TCP implementation with inadequate buffering).  In
addition, we can go back through the "there's no such thing as a buffer-
less [R]NIC discussion" if that's necessary, but more to the point is
that the number of descriptors for reasonable-sized operations
(e.g., 16 or more for a single 8KB transfer) is hard to justify.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Mon, 24 Feb 2003 13:46:22 -0500

> Again this is a function of implementation and a well-layered 
> implementation can be delivered without requiring any network stack 
> modifications if desired.  Same applies to software based iSCSI which has 
> been demonstrated by multiple vendors without network stack 
> modifications.  Some of these implementations now deliver sufficient 
> performance that current customers using them are not raising any major 
> objections.  Given this can be done with iSCSI and markers are clearly
less 
> complex to implement / support in software than iSCSI, one might argue
that 
> the complexity / cost issues are not as high as some assert.

Unless those iSCSI implementations include iSCSI markers at a 512-byte
interval (which I very much doubt), this sort of comparison is apples
vs. oranges, as in the absence of markers, those implementations can
send a 4K, 8K or larger data block in one operation.

Thanks,
--David

---

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 24 Feb 2003 11:37:06 -0800

> -----Original Message-----
> From: Michael Krause [mailto:[email-masked]]
>
> At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote:
> >The discussion is related to existing software stacks and
> >interfaces. For storage protocols (as is for iSCSI), this
> >forms the first wave of client side implementations.
> 
> objections.  Given this can be done with iSCSI and markers 
> are clearly less 
> complex to implement / support in software than iSCSI, one 
> might argue that 

Mike,

Marker use in iSCSI is a negotiated option.  Do you know if these successful
software iSCSI implementations are running with markers enabled?

Roy

<snip>

---

From: "Barron, Dwight" <[email-masked]>
Date: Mon, 24 Feb 2003 15:28:20 -0600

There are clearly a wide spectrum of implementation possibilities, but I like to focus on the endpoints first, then determine how many additional click-stops are necessary to adequately support the middle of the spectrum. Clearly, support of an ULP that wants to invoke RDDP over an unmodified TCP stack is an important goal, but optionality isn't free, so let's examine what we are really buying with negotiable marker intervals. I'll assume that the ULP using RDDP wants to see RDDP semantics (whatever that may be) and not interim RDDP headers and components. If you assume this RDDP "library" lives on top of unmodified sockets/TCP, then I believe there is no choice but for a receive stream to a) sequentially read every byte for purposes of CRC calculation and b) copy incoming stream data to separate the RDDP headers from ULP payload. This is especially true of tagged messages where the ULP is using a contiguous memory address range to reference data. On the transmit side, the R!
DDP library will still have to touch every byte for CRC calculation purposes, but may try to avoid copying by using multiple send calls. But at what point does copying become cheaper than multiple send calls? Especially if the sender is at least *attempting* to provide header alignment, which would mean adding RDDP headers for each TCP_MAXSEG interval. While we all like to avoid data copies (the whole reason for RDDP to begin with), once we bite the bullet for software CRCs (which is clearly the case with standard sockets), the copy really isn't that painful. 

Regards,
Dwight 

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Monday, February 24, 2003 11:47 AM
> To: Barron, Dwight; [email-masked]; [email-masked]
> Subject: RE: [rddp] Marker Interval
> 
> 
> Dwight,
> 
> The point is of allowing the marker interval to be flexible
> so that hardware implementations that want to work well with
> software based senders may allow a larger marker interval
> or not require it at all.
> 
> Somesh
> 
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Barron, Dwight
> > Sent: Monday, February 24, 2003 7:56 AM
> > To: [email-masked]; [email-masked]
> > Subject: RE: [rddp] Marker Interval
> > 
> > 
> > Assuming we had a large number of negotiable marker intervals, 
> > what would the optimum negotiation strategy be for a software-MPA 
> > to connect to a hardware-MPA implementation? Clearly, the SW-MPA 
> > would want larger intervals to minimize overhead, while 
> > presumably the HW-MPA would want smaller intervals to minimize 
> > buffering. While I see a lot of value in having mixed mode 
> > connections, I keep coming back the notion that the SW 
> > implementation is willingly taking on additional overhead for the 
> > benefit of the HW implementation, in which case the SW end would 
> > accept the HW's recommended marker interval value. 
> > 
> > Regards,
> > Dwight
> > 
> > > From: [email-masked] [mailto:[email-masked]]
> > > Sent: Monday, February 24, 2003 8:30 AM
> > > To: [email-masked]
> > > Subject: [rddp] Marker Interval
> > > 
> > > 
> > > Somesh Gupta wrote:
> > >