Subject: RE: [rddp] Options and Markers

From: "Barron, Dwight" <[email-masked]>
Date: Wed, 16 Apr 2003 13:02:11 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, April 16, 2003 10:51 AM
> To: Barron, Dwight; [email-masked]
> Subject: RE: [rddp] Options and Markers
> 
> 
> > > The discussion in San Francisco concerned a receive side 
> full-software
> > > implementation.  It did not cover a "full-offload with full TCP
> buffering"
> > > class of implementation, as the discussion would up 
> considering only
> > > full-software and "flow-through" RNIC implementations.  
> There's a third
> > > class that is relevant.
> > 
> > I haven't heard anyone assert that this relevant third class 
> > implementation does not fall into one of the two previously 
> > mentioned catagories.
> 
> Here's the assertion:
> 
> A full-offload full-buffering implementation (1) can achieve copy
> avoidance via direct data placement between the NIC and host memory
> (2) without using markers.  Item (1) distinguishes this class from
> full-software implementations, and item (2) distinguishes it 
> from RNICs.

My comments on the third class (how about BNIC, for buffered RDDP+TCP?) was (perhaps incorrectly) based on the following previous discussions/data points:
- The performance overhead disadvantage of markers primarily impacts the transmitter (per the San Francisco discussion)
- There are two basic ways to insert markers in a transmit data stream in the two classes I was envisioning: gather lists or copying (e.g. using explicit BNIC or host buffer pool. Just for completeness, RNICs are assumed to use inline insertion, which is a third category assumed beyond the reach of BNICs and hosts)
- At the time, reference was made to software upgrade of an existing product to incorporate RDDP. I will focus solely on the architectural connotation of *software* in that context; underneath the covers of a BNIC are still one of two basic marker insertion mechanisms discussed in the host context, gather lists or copying. 
- Which comes back to my previous question, which one of these methods is marker insertion overloading?

> 
> > > The assertion that there is a single "best architectural practice"
> > > for RDDP and it is a "flow-through RNIC" with minimal buffering
> > > is part of the problem.  
> > 
> > I'm sorry, but I disagree. This is the Remote Direct Data 
> > Placement WG, not the Remote Store and Forward WG. From the 
> > WG Description at 
> http://www.ietf.org/html.charters/rddp-charter.html
> >  ..RDDP capabilities refer to the ability to place data 
> directly from
> >	the NIC into application buffers, without intensive CPU usage.
> >	This strategy avoids the costs of data copying and enables using
> >	IP as a method for high speed buffer to buffer transfer....
> 
> That text deliberately does not specify what is implemented in the NIC
> or how the NIC does its buffering, so a "full-offload full-buffering"
> NIC is within the scope of the description, and this was intended when
> the charter was written.

Again, I did not suggest the charter mandated a specific design. I suggested it points to an architectural goal against which one may evaluate, ahem, their options.

> 
> > I have no quarrel with anyone that wants to use assembly buffers in
> > their design, every design will need some amount of buffering. I do
> > take exception that we mandate an option that essentially diminishes
> > or precludes a flow-through design
> 
> Could we see "valid engineering justification" for the latter 
> statement,
> especially the "precludes" assertion?  Keep in mind that the "option"
> under discussion allows a receiver who benefits from markers 
> to require
> their use.

You are correct, the "preclude" assertion does not apply if we get to the point that markers are mandatory to implement/optional to use at all receivers (are you putting your WG Chair hat on?). However, this means that markers are mandatory to implement at the transmitter also, which seems to be where the argument for the most marker pain has manifested itself for those that are opposed to markers. Once implemented, is there sufficient delta to a) warrant an option and b) operate the connection in an asymmetric mode? I apologize in advance for slipping into "commercial merits diatribe" but options cost real money once the ink is dry on the spec. I like to reserve their use/cost for things that are to the left of the decimal point, rather than the right. 

> 
> I snipped material on commercial success of implementations as being
> basically irrelevant to the scope issues under discussion.  I would
> caution against further raising of this sort of issue, as I suspect
> an IESG discussion of the commercial merits of InfiniBand would not
> bode well for the future of this WG, and I'd just as soon not
> encourage the IESG to go there.

My previous comment was in the context of a software upgrade for an existing product. I think the word on VI and IB has already leaked out, but just in case, I will be happy to constain further comments to architectural lessons learned from those experiences. 

Regards,
Dwight

> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Apr 2003 11:48:28 -0500

On 4/16/03, Barron, Dwight wrote:

>You are correct, the "preclude" assertion does not apply
>if we get to the point that markers are mandatory to
>implement/optional to use at all receivers (are you
>putting your WG Chair hat on?). However, this means that
>markers are mandatory to implement at the transmitter
>also, which seems to be where the argument for the most
>marker pain has manifested itself for those that are
>opposed to markers. Once implemented, is there sufficient
>delta to a) warrant an option and b) operate the
>connection in an asymmetric mode? I apologize in advance
>for slipping into "commercial merits diatribe" but options
>cost real money once the ink is dry on the spec. I like to
>reserve their use/cost for things that are to the left of
>the decimal point, rather than the right.

If we agree that a sender MUST have the capability to send
with markers, then the only question is whether there is a
cost in disabling that capability.

Given that the same stack will need to send unmodified TCP
traffic it obviously needs that capability anyway.

Other than QA testing, I don't see the cost to a sender
of making this receiver specifiable. QA testing *is* a
serious cost, but the benefit to some receivers, IMHO,
clearly justifies it.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Thu, 17 Apr 2003 12:36:32 -0700

>traffic it obviously needs that capability anyway.

There may be two stacks: a host-based software stack for non-RDMA traffic 
and a hardware-based RDMA stack.  There is no requirement that the 
hardware-based solution support raw access to the transport layers for 
non-RDMA access just as there is no requirement that the software-stack 
support RDMA - this is all in the implementation space and outside the 
scope of the RDDP charter.

>Other than QA testing, I don't see the cost to a sender of making this 
>receiver specifiable. QA testing *is* a serious cost, but the benefit to 
>some receivers, IMHO, clearly justifies it.

QA often becomes the dominant cost in delivery technology - it isn't just a 
one-time thing but is on-going and applies to multiple permutations thus 
options have a multiplying effect.  There is also the cost issue on the 
transmitter to support both options - its design may be such that removing 
markers increases complexity (I believe this was noted by several people 
prior to the recent IETF event) and thus cost.  Development, QA, etc. costs 
all impact the market adoption rate, TTM for solutions, etc.  Each option 
also comes at an opportunity cost as gates, engineers, etc. spent on the 
option are not free and delay or prevent other technologies from being 
incorporated into design points (numerous examples of this in the marketplace).

Mike