Subject: RE: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

From: "Culley, Paul" <[email-masked]>
Date: Sun, 22 Sep 2002 16:42:31 -0500

A few comments:

The MPA, DDP and RDMA drafts (the DDP and RDMA drafts are further discussed on the RDDP reflector) are intended to be used together, although there may be some who would find MPA or DDP useful on there own.  They are separate drafts, rather than a single draft, because it was thought that this might make mapping to SCTP easier (at the bottom of the protocol stack), and because discussions in the past indicated that DDP might be regarded as simpler or more desirable in some other fashion than a full blown RDMA system.

That said, the authors believe that likely implementations of these drafts will be done together.  In fact we believe that the drafts will often be implemented in some sort of hardware NIC, along with the lowest layers of the network (TCP, IP, Ethernet etc.).  When this is done, some of the arguments against the MPA draft become less of an issue.  For example, 

"TCP cannot pass data to the ULP until the right window edge has passed, precisely because it cannot ensure that the ULP won't use information in that data until the right window edge has passed.  To use that information prior to that event would constitute a change in semantics, period, whether by an ULP or an application."

The MPA draft, when read with the DDP draft, (or implemented together) carefully insures that the "application" in this case the DDP user, will not be told that the data is ready until it is all in place and all prior data is also in place.  The ULP uses DDP's semantics, rather than TCP's or even MPA's and gets to take advantage of those semantics.  In addition, if the implementation does get done in hardware, the overall system can see useful reductions in system memory bus bandwidth, and some other reductions in processing overhead.
We tried, in the MPA draft, to separate the "passing" of data from when it is valid to use, without saying just how the ULP would use that data.  Perhaps we should add some guidelines for MPA's ULP that say more about how to use this?

One question that comes up time and again is "Why not use SCTP?"  I am sure that there are other answers, but one significant one is that the infrastructure is not quite there yet for SCTP use.  Infrastructure includes many things; SCTP hardware accelerators, analysis and management tools, verification tools, widespread industry experience, firewall support etc.  Many folks want to see RDMA (or DDP) become standardized on some kind of IP, but are worried that coupling RDMA with SCTP may slow down the adoption rate of RDMA to that of SCTP.  Also, we believe that available TCP implementations, with the MPA "enhancements" will do the job just fine, and with less total work than creating the necessary SCTP implementations.

The RDDP workgroup is chartered to develop a mapping of DDP onto SCTP, and I am sure that that group will also provide a draft into that area.  If that work is completed quickly enough, and SCTP is adopted more easily and faster than expected, DDP on MPA/TCP can die as the market allows.

Another question recently asked was "(why is it) that RDMA does not want to use unordered delivery in SCTP even though it may be desired"?  We certainly expect that when DDP is layered on top of SCTP that DDP will support multiple streams.  We only defined what the semantics of DDP when run on a single stream was, since that what TCP provides.

That said, we may not have gotten the text of the MPA draft written with enough of the right language to satisfy everyone (and especially not those who don't read the DDP and RDMA drafts).  So we will be monitoring this list to see what the technical and other issues are and trying to correct them with future drafts.  We hope that enough of the idea is already there to start that process. 

Paul Culley
HP Sr. Fellow
281-514-5543

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Sun, 22 Sep 2002 17:15:03 -0700

Culley, Paul wrote:
> A few comments:
> 
> The MPA, DDP and RDMA drafts (the DDP and RDMA drafts are further
> discussed on the RDDP reflector) are intended to be used together,
> although there may be some who would find MPA or DDP useful on there
> own.  They are separate drafts, rather than a single draft, because
> it was thought that this might make mapping to SCTP easier (at the
> bottom of the protocol stack), and because discussions in the past
> indicated that DDP might be regarded as simpler or more desirable in
> some other fashion than a full blown RDMA system.
> 
> That said, the authors believe that likely implementations of these
> drafts will be done together.  In fact we believe that the drafts
> will often be implemented in some sort of hardware NIC, along with
> the lowest layers of the network (TCP, IP, Ethernet etc.).  When this
> is done, some of the arguments against the MPA draft become less of
> an issue.  For example,
> 
> "TCP cannot pass data to the ULP until the right window edge has
> passed, precisely because it cannot ensure that the ULP won't use
> information in that data until the right window edge has passed.  To
> use that information prior to that event would constitute a change in
> semantics, period, whether by an ULP or an application."
> 
> The MPA draft, when read with the DDP draft, (or implemented
> together) carefully insures that the "application" in this case the
> DDP user, will not be told that the data is ready until it is all in
> place and all prior data is also in place.  

This would need to be a requirement of the MPA draft, or the multiple 
component drafts would need to require concurrent use.

It clearly is inappropriate to equate TCP with this protocol if its 
semantics cannot be equated stand-alone.

> The ULP uses DDP's
> semantics, rather than TCP's or even MPA's and gets to take advantage
> of those semantics.  In addition, if the implementation does get done
> in hardware, the overall system can see useful reductions in system
> memory bus bandwidth, and some other reductions in processing
> overhead. We tried, in the MPA draft, to separate the "passing" of
> data from when it is valid to use, without saying just how the ULP
> would use that data.  Perhaps we should add some guidelines for MPA's
> ULP that say more about how to use this?

Requirements, not guidelines. And the correctness of the solution would 
need to be analyzed as a set, not separately, in that case.

> One question that comes up time and again is "Why not use SCTP?"  I
> am sure that there are other answers, but one significant one is that
> the infrastructure is not quite there yet for SCTP use.
 >
> Infrastructure includes many things; SCTP hardware accelerators,
> analysis and management tools, verification tools, widespread
> industry experience, firewall support etc.

MPA is sufficiently different from TCP that none of this applies. All 
the analysis and management tools are going to need to peek into the 
packets to look at the data to see if MPA is working correctly; they all 
need to be rewritten.

How much of that is currently available for MPA? How much for SCTP?

It's not a matter of leveraging off of TCP; very little, if any, of TCP 
tools or experience applies here.

> Many folks want to see
> RDMA (or DDP) become standardized on some kind of IP, but are worried
> that coupling RDMA with SCTP may slow down the adoption rate of RDMA
> to that of SCTP.  Also, we believe that available TCP
> implementations, with the MPA "enhancements" will do the job just
> fine, and with less total work than creating the necessary SCTP
> implementations.

SCTP exists. What work?

SCTP isn't as widely deployed as TCP, but MPA isn't TCP either. It needs 
to be deployed on both ends to work, so there's no leveraging off an 
installed base.

> The RDDP workgroup is chartered to develop a mapping of DDP onto
> SCTP, and I am sure that that group will also provide a draft into
> that area.  If that work is completed quickly enough, and SCTP is
> adopted more easily and faster than expected, DDP on MPA/TCP can die
> as the market allows.
> 
> Another question recently asked was "(why is it) that RDMA does not
> want to use unordered delivery in SCTP even though it may be
> desired"?  We certainly expect that when DDP is layered on top of
> SCTP that DDP will support multiple streams.  We only defined what
> the semantics of DDP when run on a single stream was, since that what
> TCP provides.

TCP provides a single, ordered stream

MPA provides multiple streams that are unordered, just like SCTP. DDP 
isn't running on a single stream when it runs on MPA.

> That said, we may not have gotten the text of the MPA draft written
> with enough of the right language to satisfy everyone (and especially
> not those who don't read the DDP and RDMA drafts).

Again, if this is supposed to be TCP-equivalent, reading the DDP and 
RDMA drafts _MUST_ not be necessary. If it MUST be coupled with DDP and 
RDMA to make sense, then the draft MUST explicitly state that.

In which case this is a stovepipe protocol, not a modification of TCP.

> So we will be
> monitoring this list to see what the technical and other issues are
> and trying to correct them with future drafts.  

Please also review the previous discussions related to this type of 
modification. It has been proposed before, and this attempt does not 
successfully address many of the shortcomings identified earlier.

Joe