Subject: RE: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

From: "Culley, Paul" <[email-masked]>
Date: Mon, 23 Sep 2002 09:17:48 -0500

embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Joe Touch [mailto:[email-masked]]
Sent: Sunday, September 22, 2002 7:15 PM
To: Culley, Paul
Cc: [email-masked]
Subject: Re: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

Culley, Paul wrote:
> A few comments:
> 
> The MPA, DDP and RDMA drafts (the DDP and RDMA drafts are further
> discussed on the RDDP reflector) are intended to be used together,
> although there may be some who would find MPA or DDP useful on there
> own.  They are separate drafts, rather than a single draft, because
> it was thought that this might make mapping to SCTP easier (at the
> bottom of the protocol stack), and because discussions in the past
> indicated that DDP might be regarded as simpler or more desirable in
> some other fashion than a full blown RDMA system.
> 
> That said, the authors believe that likely implementations of these
> drafts will be done together.  In fact we believe that the drafts
> will often be implemented in some sort of hardware NIC, along with
> the lowest layers of the network (TCP, IP, Ethernet etc.).  When this
> is done, some of the arguments against the MPA draft become less of
> an issue.  For example,
> 
> "TCP cannot pass data to the ULP until the right window edge has
> passed, precisely because it cannot ensure that the ULP won't use
> information in that data until the right window edge has passed.  To
> use that information prior to that event would constitute a change in
> semantics, period, whether by an ULP or an application."
> 
> The MPA draft, when read with the DDP draft, (or implemented
> together) carefully insures that the "application" in this case the
> DDP user, will not be told that the data is ready until it is all in
> place and all prior data is also in place.  

This would need to be a requirement of the MPA draft, or the multiple 
component drafts would need to require concurrent use.
<prc> I happen to agree with you, but some of the other authors believe that it is inappropriate to -ever- place a hard requirement on a ULP.
<prc>

It clearly is inappropriate to equate TCP with this protocol if its 
semantics cannot be equated stand-alone.
<prc> I am not sure what you mean when you say we "equate" TCP with MPA+TCP.  We clearly do not say they are equal, otherwise there would be no need for the MPA draft.  What we do say is that MPA -uses- TCP to maintain a ULP's record boundaries while preserving reliable in order delivery.  It happens that we also allow ULP's to get their hands on the data before we tell them that it is valid and ordered, which seems to be your major concern.  But I suspect that there are well respected TCP implementations that do the same (what happens if my app peeks into it's receive buffer before Sockets returns it?).  As noted, we can (and probably will) add something like "If an ULP requires data to be delivered in the order that it was sent by the remote peer, then the ULP must use the MPA provided ordering information to re-establish that order."

In addition, we do fully interoperate with TCP.  If one endpoint chooses to implement MPA on top of sockets, it will still talk to an endpoint that more tightly integrates with TCP.  It just loses some of the advantages of the more tightly integrated version.
<prc>

> The ULP uses DDP's
> semantics, rather than TCP's or even MPA's and gets to take advantage
> of those semantics.  In addition, if the implementation does get done
> in hardware, the overall system can see useful reductions in system
> memory bus bandwidth, and some other reductions in processing
> overhead. We tried, in the MPA draft, to separate the "passing" of
> data from when it is valid to use, without saying just how the ULP
> would use that data.  Perhaps we should add some guidelines for MPA's
> ULP that say more about how to use this?

Requirements, not guidelines. And the correctness of the solution would 
need to be analyzed as a set, not separately, in that case.

> One question that comes up time and again is "Why not use SCTP?"  I
> am sure that there are other answers, but one significant one is that
> the infrastructure is not quite there yet for SCTP use.
 >
> Infrastructure includes many things; SCTP hardware accelerators,
> analysis and management tools, verification tools, widespread
> industry experience, firewall support etc.

MPA is sufficiently different from TCP that none of this applies. All 
the analysis and management tools are going to need to peek into the 
packets to look at the data to see if MPA is working correctly; they all 
need to be rewritten.

How much of that is currently available for MPA? How much for SCTP?

It's not a matter of leveraging off of TCP; very little, if any, of TCP 
tools or experience applies here.
<prc> I'm afraid we just have different opinions about this... I noticed that you did not mention firewall support, does your home router do NAT for SCTP (as an example)?
<prc>

> Many folks want to see
> RDMA (or DDP) become standardized on some kind of IP, but are worried
> that coupling RDMA with SCTP may slow down the adoption rate of RDMA
> to that of SCTP.  Also, we believe that available TCP
> implementations, with the MPA "enhancements" will do the job just
> fine, and with less total work than creating the necessary SCTP
> implementations.

SCTP exists. What work?

SCTP isn't as widely deployed as TCP, but MPA isn't TCP either. It needs 
to be deployed on both ends to work, so there's no leveraging off an 
installed base.
<prc> Of course it needs to be deployed on both ends to work (don't most non-trivial ULP or application?), again I think we disagree on the amount of leverage off of the installed base.
<prc>

> The RDDP workgroup is chartered to develop a mapping of DDP onto
> SCTP, and I am sure that that group will also provide a draft into
> that area.  If that work is completed quickly enough, and SCTP is
> adopted more easily and faster than expected, DDP on MPA/TCP can die
> as the market allows.
> 
> Another question recently asked was "(why is it) that RDMA does not
> want to use unordered delivery in SCTP even though it may be
> desired"?  We certainly expect that when DDP is layered on top of
> SCTP that DDP will support multiple streams.  We only defined what
> the semantics of DDP when run on a single stream was, since that what
> TCP provides.

TCP provides a single, ordered stream

MPA provides multiple streams that are unordered, just like SCTP. DDP 
isn't running on a single stream when it runs on MPA.
<prc>This depends on a point of view; our view is that MPA -does-, when all the provided information is utilized, provide for a single ordered stream.  It only allows the ULP to see the data records before ordering has been established in case that is useful (as it is in DDP) before re-establishing the order.
<prc>

> That said, we may not have gotten the text of the MPA draft written
> with enough of the right language to satisfy everyone (and especially
> not those who don't read the DDP and RDMA drafts).

Again, if this is supposed to be TCP-equivalent, reading the DDP and 
RDMA drafts _MUST_ not be necessary. If it MUST be coupled with DDP and 
RDMA to make sense, then the draft MUST explicitly state that.

In which case this is a stovepipe protocol, not a modification of TCP.
<prc> If tying the drafts together and calling it a "stovepipe protocol" (whatever that means) is what it takes to satisfy the majority of the group, we can surely do that...
<prc>

> So we will be
> monitoring this list to see what the technical and other issues are
> and trying to correct them with future drafts.  

Please also review the previous discussions related to this type of 
modification. It has been proposed before, and this attempt does not 
successfully address many of the shortcomings identified earlier.
<prc> Of course we thought we had done that; since the authors had been well aware of previous attempts to specify a record oriented protocol for TCP (TUF, COWS, iSCSI Markers etc.).  If you can -help- us by identifying additional specific problems with this proposal, we will surely appreciate it.
<prc>

Joe

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 Sep 2002 08:11:42 -0700

Culley, Paul wrote:
> <prc> I happen to agree with you, but some of the other authors
> believe that it is inappropriate to -ever- place a hard requirement on a
> ULP.
> <prc>

The problem is that, in the absence of such hard requirements, MPA
changes the semantics of TCP to the application above the ULP.

As far as TCP semantics are concerned, the ULP _IS_ the 'application',
i.e., as 'application' is used in TCP RFCs.

> <prc> I am not sure what you mean when you say we "equate" TCP with
> MPA+TCP.  We clearly do not say they are equal, otherwise there would
> be no need for the MPA draft.

Equal semantics. Equal enough to still call it TCP.

> What we do say is that MPA -uses- TCP
> to maintain a ULP's record boundaries while preserving reliable in
> order delivery.  It happens that we also allow ULP's to get their
> hands on the data before we tell them that it is valid and ordered,
> which seems to be your major concern.

One of them. Yes - this, in the absence of ULP constraints, allows the 
semantics of TCP to change.

> But I suspect that there are
> well respected TCP implementations that do the same (what happens if
> my app peeks into it's receive buffer before Sockets returns it?).

The application should be blocked until TCP releases the buffer. That's 
what locks are for. To do otherwise would be to violate TCP semantics as 
explicitly stated in the RFCs.

> As noted, we can (and probably will) add something like "If an ULP
> requires data to be delivered in the order that it was sent by the
> remote peer, then the ULP must use the MPA provided ordering
> information to re-establish that order."

If MPA allows data to be used by a ULP out of order, and does not 
require EVERY ULP to prevent that data from being accessed by the 
application, then it's not enforcing TCP semantics.

Even if it does, it's not MPA that would have semantics equal to TCP, it 
would be MPA + ULP. That would mean that MPA is NOT equal in semantics 
to TCP.

> In addition, we do fully interoperate with TCP. If one endpoint
> chooses to implement MPA on top of sockets, it will still talk to an
> endpoint that more tightly integrates with TCP. It just loses some of
> the advantages of the more tightly integrated version.
> <prc>

You run two stacks, one TCP, one MPA. When the other side runs TCP, you 
speak TCP. When the other side runs MPA, you cease to speak TCP. That is 
dual-stack, not "a protocol interoperable with TCP".

I.e., replace MPA with SCTP (clearly not TCP), and you'll see that it 
works just as well, but SCTP clearly isn't "interoperable" with TCP.

> <prc> I'm afraid we just have different opinions about this... I
> noticed that you did not mention firewall support, does your home router
> do NAT for SCTP (as an example)?
> <prc>

My home router is, well, pardon, it's a ROUTER.

That aside, the problems with NATs are well known, and break TCP as well 
as SCTP, MPA, and just about everything else. Further, if you're so 
worried about NATs, you can't have 'yet another' layer of protocol 
inside TCP and expect interoperability. E.g., what if you run FTP over 
MPA? If the NAT can't recalculate your checksums, it won't be able to 
modify the IP addresses in-band.

Furthermore, what happens if 6.4.3.2 is translated to 194.121.201.105 in 
an ASCII stream? Won't that break alignment?

I don't see MPA as helping the situation with NATs. I don't see anything 
as helping them, per se: see 
http://dsonline.computer.org/0207/departments/w4icon.htm That's why we 
developed a way to undo their damage: see http://www.isi.edu/xbone/tethernet

> <prc> Of course it needs to be deployed on both ends to work (don't
most non-trivial ULP or application?), again I think we disagree on the
amount of leverage off of the installed base.
> <prc>

SCTP that backs off and uses TCP is just as interoperable as MPA in that 
case.

> <prc>This depends on a point of view; our view is that MPA -does-,
when all the provided information is utilized, provide for a single
ordered stream. It only allows the ULP to see the data records before
ordering has been established in case that is useful (as it is in DDP)
before re-establishing the order.
> <prc>

MPA does NOT provide a single ordered stream when there are no rules on 
the ULP. It could provide incorrect data, as per my first example. 
That's more than "only allowing [visibility of] data records before 
ordering has been established".

> <prc> If tying the drafts together and calling it a "stovepipe
> protocol" (whatever that means)

A quick check on Google shows:
http://cio.doe.gov/iap/publications/vol1_foundations/glossary.html :

stovepipe (system)  A system procured and developed to solve a specific 
problem, characterized by a limited focus and functionality, and 
containing data that cannot be easily shared with other systems.

(this is a common term in systems design, combining it with protocol 
should yield the obvious connotations)

> is what it takes to satisfy the majority
> of the group, we can surely do that...
> <prc>

It may be necessary, but not sufficient to still call it TCP. Again, 
this still looks like a non-TCP protocol that backs off and talks TCP.

> <prc> Of course we thought we had done that; since the authors had
been well aware of previous attempts to specify a record oriented
protocol for TCP (TUF, COWS, iSCSI Markers etc.). If you can -help- us
by identifying additional specific problems with this proposal, we will
surely appreciate it.
> <prc>

That's what I've _been_ doing. The point about the previous drafts as 
well is that one of the counterpoints was "what about SCTP". That, among 
some of the other points I've raised (why is this still TCP, vs. a 
two-stack system) all were raised before, but none is directly addressed 
in the draft, and the discussions on email have still not been convincing.

I have to admit, from the outside (non-RDMA/DDR camp), this all looks 
like the insistance on the use of three letters (TCP) vs four (SCTP) is 
being driven by product marketing, not protocol design. Offline 
discussions I've had have hinted at same; addressing that would further 
be useful.

Joe

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 24 Sep 2002 16:02:49 -0400

>explicitly stated in the RFCs.

That is implementation, not protocol. Certainly incoming data is not valid
until the buffer is released ("delivered") by TCP. Whether the buffer is
validly within the receiver's address space, or how it is populated, is
quite another story and one which the protocol by itself cannot dictate.

Would a TCP be non-compliant simply because it might happen to store
data in a buffer accessible to an application? That would rule out most
embedded stacks, as well as optimized direct I/O models such as storage
or shared memory. Is that acceptable?

Tom.

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Sep 2002 14:55:04 -0700

Talpey, Thomas wrote:
> That is implementation, not protocol.

See RFC793, P47 (or 48 - I can't see ^L's), Sec 3.8:
     Receive

         Format:  RECEIVE (local connection name, buffer address, byte
         count) -> byte count, urgent flag, push flag

         This command allocates a receiving buffer associated with the
         specified connection.  If no OPEN precedes this command or the
         calling process is not authorized to use this connection, an
         error is returned.

         In the simplest implementation, control would not return to the
         calling program until either the buffer was filled, or some
         error occurred, but this scheme is highly subject to deadlocks.
         A more sophisticated implementation would permit several
         RECEIVEs to be outstanding at once.  These would be filled as
         segments arrive.  This strategy permits increased throughput at
         the cost of a more elaborate scheme (possibly asynchronous) to
         notify the calling program that a PUSH has been seen or a buffer
         filled.

 > Certainly incoming data is not valid

See above. It is explicitly required by the protocol.

> data in a buffer accessible to an application?

Yes. TCP that allows writes to non-locked buffers is non-compliant, 
notably because the semantics is visibly different from the application.

> That would rule out most
> or shared memory.

I'd be surprised if embedded stacks didn't have lock bits on their 
buffers; even embedded systems have pseudo-OS protection capabilities. 
Direct I/O/shared memory doesn't preclude locks either.

Otherwise, it depends on whether the application can be changed; if the 
system is bundled, then the violations in semantics aren't tickled by 
the application, so the system is OK. If the application can be 
arbitrarily rewritten, then no, it would not be a compliant TCP.

(that doesn't mean anyone has necessarily noticed yet - there are plenty 
of TCPs thought to be compliant that later were found to have 
non-compliant bugs)

Joe