Subject: RE: [rddp] Options and Markers

From: "Barron, Dwight" <[email-masked]>
Date: Fri, 11 Apr 2003 16:17:44 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, April 11, 2003 12:41 PM
> To: [email-masked]
> Subject: [rddp] Options and Markers
> 
> 
> Pulling another topic out of Paul Culley's post:
> 
> > In my mind "enabling" these scenarios, while important, 
> should not mean
> > that we go to great lengths to optimize or reduce overhead on the
> > software side at the expense of making the overall protocol 
> less robust,
> > or adding multiple options that applications will have to 
> sort through
> > and all concerned will have to test.
> 
> I have no problem with robustness considerations, but the vehemence
> of opposition to options in any form from a significant portion of
> this WG has been among the things that have hindered progress.

I do not believe it is a "vehemence of opposition to options" per se, but rather a strong desire to see valid engineering justification for such options. I believe we established, per your comments at the last F2F, that receive side copying is not likely to be avoided, regardless of marker interval and we went on to focus on transmit side impact of marker interval options. I recall that there were two likely implementation scenarios on the transmit side, single buffer pointer per transmit/system call and gather lists per transmit/system call. My personal assertion was that environments with single buffer pointers per transmit/system call would quickly resort to copying to coalesce the RDDP header and payload to avoid system call overhead and marker interval wasn't significant in the copy loop (1/512 is 0.2%, taking it 0.02%, while numerically a 10X improvement, is not a valid justification for an option, IMHO). Others asserted that gather lists were sufficiently large as to !
not warrant additional marker intervals. Neither of these scenarios seem to have been challenged at all, much less to an extent to convince me I am being unreasonable and vehement. 

> 
> I think some form of option is going to be hard to avoid for
> markers, as there is "running code" out there for a class of
> implementations (RDDP + TCP with full buffering offloaded to
> a souped up NIC) that do not need markers.  Requiring that this
> class of implementation use markers solely to avoid putting an
> option in the protocol appears to be very hard to justify.

How does this argument not cross the line from what I would consider debate in search of best architectural practice for RDDP to that of defending an existing RDDP implementation of "running code". What RDDP specification was this running code conformant to such that it should impact the design of future RDDP specifications? 

> 
> One alternate possibility would be to make marker usage optional,
> but put control of this "option" entirely in the hands of
> the receiver - senders MUST implement both markers and not-markers,
> receivers MUST implement one OR the other, and each receiver
> gets to pick which one is used as the benefits of markers are
> primarily on the receiver side.
> 
> Comments?
> 
> Thanks,
> --David
> 
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Sun, 13 Apr 2003 15:51:56 -0400

Dwight,

> I do not believe it is a "vehemence of opposition to options" 
> per se, but rather a strong desire to see valid engineering 
> justification for such options. I believe we established, per 
> your comments at the last F2F, that receive side copying is 
> not likely to be avoided, regardless of marker interval and 
> we went on to focus on transmit side impact of marker 
> interval options.

The discussion in San Francisco concerned a receive side full-software
implementation.  It did not cover a "full-offload with full TCP buffering"
class of implementation, as the discussion would up considering only
full-software and "flow-through" RNIC implementations.  There's a third
class that is relevant.

[... material on full-software implementations snipped ... ]

> How does this argument not cross the line from what I would 
> consider debate in search of best architectural practice for 
> RDDP to that of defending an existing RDDP implementation of 
> "running code". What RDDP specification was this running code 
> conformant to such that it should impact the design of future 
> RDDP specifications? 

The assertion that there is a single "best architectural practice"
for RDDP and it is a "flow-through RNIC" with minimal buffering
is part of the problem.  The implementation I referred to above
is based on the VI architecture and runs it over TCP/IP with
all RDMA processing done outboard.  AFAIK, the protocol used
is specific to that implementation; among the reasons for this is
that VI is an interface, not a protocol.

The reason that this discussion doesn't cross the line, is that
the "running code" serves to demonstrate the viability of the
third class of implementations, speaking to your above desire for
"valid engineering justification" for protocol design concerns
raised by that class.  While I can't speak for the implementers,
I think they plan to update the implementation in question to
whatever specifications the RDDP WG adopts.  Given the viability
of that class of implementation (FWIW, it was extensively discussed
prior to formation of the RDDP WG), design decisions which exclude
that class will tend to run afoul of the admonition not to
impose requirements on implementors that are not required for
interoperability (see Section 6 of RFC 2119).

Thanks,
--David

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 14 Apr 2003 10:28:06 -0700

My take-away from the San Francisco RDDP meeting was that there
was rough consensus for mandating markers, though there was quite a 
bit of discussion around marker interval (I just went back to the minutes
and they seem to confirm this recollection).  I am somewhat surprised 
to see this discussion brought up again.

>design decisions which exclude

This WG is trying to solve some architectural problems around 
performance/scalability bottlenecks.  Markers is one of the architectural 
elements designed to solve those bottlenecks and once we accept it's a 
desired element of the architecture, it is of course *required* for 
interoperability to implement and use it.  

So I guess you are questioning the value of the markers as a desired
element of RDDP/TCP architecture.  Again, I thought we're past that 
question.

>The implementation I referred to above
> all RDMA processing done outboard.

Let's not assume that a commercially available VI-based adapter running
at relatively low speeds, compared to what this WG is trying to design for, 
defines the "architectural best practices" for RDDP/TCP.  I would be much
more convinced about this VI-based design if there is "running code" for 10G.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 14 Apr 2003 15:27:40 -0400

> From: Mallikarjun C. [mailto:[email-masked]]
> Sent: Monday, April 14, 2003 1:28 PM
> To: [email-masked]
> Subject: Re: [rddp] Options and Markers
> 
> 
> was rough consensus for mandating markers,

There was certainly not unanimous consensus.  I guess it's 
David's call as to what constitutes rough consensus.

> element of RDDP/TCP architecture.  Again, I thought we're past that 

I have certainly questioned the value of markers and continue
to question it.  Nothing has transpired that led me to 
believe we were "past that question".

> > is based on the VI architecture and runs it over
> > TCP/IP with all RDMA processing done outboard.
> 

I don't presume that this VI-based design
resolves the issue one way or the other,
but it suggests that there is at least one
design point that works better without markers.
I personally don't see any fundamental reason
that this design approach won't scale to 
much higher speeds, but as you point out
this is not convincing until it is actually
done.

Much more to the point, though, is that there
are no working examples (correct me if I am wrong)
of designs that successfully exploit markers.
iSCSI defines optional markers, and iSCSI
implementations that use these to advantage
are conspicuous by their absence.

There are a couple different reasons to add
options to a protocol.  One is that it is 
sometimes beneficial for two different
modes to co-exist.

A second important reason is that during
the initial development of a protocol, it
is often not clear what will and will not
be desirable features long term.  This is
particularly true when there are not yet
any interoperable implementations.  If the
option is successful, it will become a de
facto standard and always be implemented.
If it is not successful, it will be dropped
and forgotten.

Others can comment on the IETF standards
process,  but perhaps it makes sense that
options be defined as the "proposed standard"
level, and then become either mandated or
eliminated as the standard progresses along
the standards track, and experience is gained.

Markers are a feature which is not required
for functional correctness of the protocol,
adds complexity to the protocol, is claimed
(but not demonstrated) to improve the
cost/performance of a subset of the possible
implementations, and hurts the cost/performance
of a different subset of the possible 
implementations.  I would urge that 
significant caution be used when
considering mandating such a feature.

---

From: "Barron, Dwight" <[email-masked]>
Date: Tue, 15 Apr 2003 14:36:16 -0500

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Tuesday, April 15, 2003 12:40 PM
> To: [email-masked]
> My reading of the charter of the workgroup is that
> the rddp protocol address the problem of data copying
> in the host (from the NIC into application buffers).
> 
> The charter does not indicate how the NIC is implemented.

As I stated in my posting, I have no quarrel over implementations with buffering, nor am I advocating this WG dictate implementation practice. It should not, and cannot. My point was that the WG charter suggests that there is an architectural goal which in turn suggests best engineering practices to reach that goal, which in turn suggests an appropriate weighting when forced to make the inevitable trade offs between a rock and hard place. I believe the absence of markers to be a much more serious impediment to the viability of a flow-through implementation than any additional overhead they impose on the other classes of design. The proponents of markers presented a case for such impact to memory in the elzur paper and quantified the impact as something that statistically scales with on the order of per connection overhead, which is three orders of magnitude for 1000 connections and another order of magnitude for 10Gig as a very conservative design point. I have not seen a si!
milar attempt to quantify the overhead of markers and personally find it very hard to believe they will be more than 10% in either class of software based implementation. 

> 
> When the Work Group was being chartered, there was a
> fairly big debate about whether there should even be
> a TCP mapping, since TCP would always require some
> buffering or roto-tilling of TCP (the argument is over
> how much buffering). Those of us who wanted
> a mapping over TCP pushed for it on the ground that
> we could achieve the mapping without trying to create a
> different version of TCP - and it did not matter from
> the perspective of the problem being solved what we did in
> the NIC.
> 
> Maybe I was being overly optimistic at that time, not
> in any way about the engineering challenges, but about
> the process. Seems like there is a strong desire to get
> the legitimacy provided by using the TCP protocol, but
> at the same time by significant modifications to it
> (or by creating a new supertransport to overcome the
> limitations of TCP).

Does the existence of iSCSI markers and CRC create any more of a new supertransport protocol by overcoming the same limitations of TCP? If any consumer is to be aware of a new transport protocol, it will the ULPs that are specifically defined to use RDDP, in which case I would much rather provide them with the same RDDP behaviors and implementation impact for both SCTP and TCP mappings. 

Regards,
Dwight

> Regards,
> Somesh
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Barron, Dwight
> > Sent: Tuesday, April 15, 2003 9:12 AM
> > To: [email-masked]; [email-masked]
> > > Sent: Sunday, April 13, 2003 2:52 PM
> > > To: Barron, Dwight; [email-masked]
> > I haven't heard anyone assert that this relevant third class 
> > implementation does not fall into one of the two previously 
> > mentioned catagories. I would suspect, given the embedded nature 
> > of such designs, that they have relatively unencumbered access to 
> > a NIC with scatter/gather capability. 
> > 
> > > 
> > > is part of the problem.  
> > I'm sorry, but I disagree. This is the Remote Direct Data 
> > Placement WG, not the Remote Store and Forward WG. From the WG 
> > Description at 
> > https://www1.ietf.org/html.charters/rddp-charter.html ..RDDP 
> > capabilities refer to the ability to place data directly from the 
> > NIC into application buffers, without intensive CPU usage. This 
> > strategy avoids the costs of data copying and enables using IP as 
> > a method for high speed buffer to buffer transfer....
> > I have no quarrel with anyone that wants to use assembly buffers 
> > in their design, every design will need some amount of buffering. 
> > I do take exception that we mandate an option that essentially 
> > diminishes or precludes a flow-through design for those of us 
> > that are taking the *Direct* part seriously. 
> > 
> > I cannot find said product available for sale today, I have been 
> > told it is withdrawn from the market and it is not on the 
> > vendor's website (if this comment seems untowardly callous, the 
> > same is true for my own company's VI product). That said, I have 
> > not heard that the product is incapable of supporting markers. 
> > Given the general catagory of a dedicated CPU and souped up NIC 
> > capable of being upgraded by firmware, does that product become 
> > (more) commercially unviable due to the overhead of inserting 
> > markers? As for excluding a class as a whole, I think we are down 
> > to quibbling whether an implementation should use an XXX or YYY 
> > MHz embedded processor to hit ZZZ% of wire speed. 
> > 
> > > 
> > > Thanks,
> > > --David

---

From: "Williams, Jim" <[email-masked]>
Date: Wed, 16 Apr 2003 08:39:52 -0400

> Second, my experience from several past IETF standards (Path MTU
> Discovery, HTTP/1.1) is that interoperability suffers unless the
> specification leaves very little wiggle room.  Unlike Jim, I am
> not optimistic that the marketplace will save us from products
> that don't support "valuable" features whose implementation is
> optional.  (For example, in a corporate network with 50,000 nodes
> and hundreds or thousands of routers, it only takes one or two
> routers that black-hole PMTUD messages to force nearly everyone
> to turn off their end-host PMTUD support.)

Jeff,

I think your point has a lot of merit.  I would suggest, though, 
that markers are not QUITE as bad as the PMTU example you give.
Specifically, if markers are optional to implement, and an endpoint
chooses not to implement them, the worst case is that the performance
of connections to THAT endpoint may be slower.  There are no
functional or interoperability problems, an other connections are
not penalized.

 - Jim

---

From: [email-masked]
Date: Wed, 16 Apr 2003 11:50:32 -0400

> mentioned catagories.

Here's the assertion:

A full-offload full-buffering implementation (1) can achieve copy
avoidance via direct data placement between the NIC and host memory
(2) without using markers.  Item (1) distinguishes this class from
full-software implementations, and item (2) distinguishes it from RNICs.

> Placement WG, not the Remote Store and Forward WG. From the 
> WG Description at http://www.ietf.org/html.charters/rddp-charter.html
>  ..RDDP capabilities refer to the ability to place data directly from
>	the NIC into application buffers, without intensive CPU usage.
>	This strategy avoids the costs of data copying and enables using
>	IP as a method for high speed buffer to buffer transfer....

That text deliberately does not specify what is implemented in the NIC
or how the NIC does its buffering, so a "full-offload full-buffering"
NIC is within the scope of the description, and this was intended when
the charter was written.

> I have no quarrel with anyone that wants to use assembly buffers in
> their design, every design will need some amount of buffering. I do
> take exception that we mandate an option that essentially diminishes
> or precludes a flow-through design

Could we see "valid engineering justification" for the latter statement,
especially the "precludes" assertion?  Keep in mind that the "option"
under discussion allows a receiver who benefits from markers to require
their use.

I snipped material on commercial success of implementations as being
basically irrelevant to the scope issues under discussion.  I would
caution against further raising of this sort of issue, as I suspect
an IESG discussion of the commercial merits of InfiniBand would not
bode well for the future of this WG, and I'd just as soon not
encourage the IESG to go there.

Thanks,
--David

---

From: Dave Sheehy <[email-masked]>
Date: Wed, 16 Apr 2003 12:36:10 PDT

Jim Williams wrote:
> > From: Caitlin Bestler [mailto:[email-masked]]
> > [...]
> > That leave the real issue being assessing which is the 
> > greater imposition:
> > 
> > -- forcing senders to support both marker and no-marker mode.
> > -- forcing memory-rich receivers to implement markers.
> > 
> > The primary virtue of the latter is that it avoids options. 
> > Allowing recievers
> > to mandate "no markers" is clearly superior for pure-software 
> > RDDP stacks
> > written on top of sockets.
> 
> If you will forgive a bit of oversimplification, I view
> the problem as follows.
> 
> If markers are optional to implement, then there are two
> possible outcomes:
> 
>     A:  Markers turn out to be valuable, and all implementations
>         will either support them or be non-competitive.
> 
>     B:  Markers turn out NOT to be valuable, and will not
>         be implemented.

Neither A or B is the most likely outcome. The most likely outcome is 
that markers will not be implemented at all since they are useless if 
optional.  iSCSI is a perfect example of this.

Dave Sheehy