Subject: RE: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

From: "Uri Elzur" <[email-masked]>
Date: Sun, 22 Sep 2002 14:27:10 -0700

Joe

next round marked by [UE1]

Uri

-----Original Message-----
From: Joe Touch [mailto:[email-masked]]
Sent: Friday, September 20, 2002 8:00 AM
To: Uri Elzur
Cc: allyn; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

Uri Elzur wrote:
> pls see comments below [UE]
> 
> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Thursday, September 19, 2002 11:18 AM
> To: allyn
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt
> 
> 
> allyn wrote:
> 
>>No reason not to use SCTP, to the contrary, every reason to use it. The 
>>RDMA protocols - DDP and RDMA protocols are to be transport independent 
>>and there is to be a mapping to both SCTP and to TCP. The MPA draft is 
>>for framing to allow mapping to TCP.
>>What do you think about the details of the proposal?
> 
> 
> I have the same concerns the last time I saw this sort of thing:
> 
> - TCP semantics change:
> 	consider a sequence of FPDUs (or in TCP, segments) sent:
> 	1,2,3,4
> 	and let '2' be dropped by the channel.
> 
> 	now close the connection.
> 
> 	what does the receiver pass to the application?
> 	- in TCP, only 1
> 
> 	- in this (not-TCP), 1,3,4
> 
> [UE] reading the draft carefully would suggest there is a distinction
> between PLACEMENT and "Delivery". In the corner case described above,
> MPA+TCP behaves exactly like TCP. It will only DELIVER to the application
> all the data that has been received  to the left of the TCP window edge.
In
> this case "only 1".
> [UE]

The word "placement" occurs exactly twice in the ID, in reference to 
DDR. Please clarify.

I think you mean that TCP passes the data to the ULP as soon as 
possible, and that there is an additional mechanism for TCP to indicate 
that the right window edge has passed. HOWEVER, the ID is very clear to 
avoid specifying properties of the ULP.

As far as TCP is concerned, the ULP _IS_ the application. TCP cannot 
pass data to the ULP until the right window edge has passed, precisely 
because it cannot ensure that the ULP won't use information in that data 
until the right window edge has passed.

To use that information prior to that event would constitute a change in 
semantics, period, whether by an ULP or an application.

[UE1] you might recall that MPA does have a set of requirements from the ULP
(see section 5.2). A good example of such ULP, might be Direct Data
Placement (DDP, see draft-shah-iwarp-ddp-00.txt submitted to the RDDP WG).
DDP defines Placement (which I've enclosed below for your convenience). If a
ULP conforms by this set of rules, it gets the benefits of saving all of the
intermediate buffers without violating the TCP semantics for delivery. I.e.
there will be NO ACTION taken on this data until such time that the right
edge of the window has passed that point.

Data Placement (Placement, Placed, Places) - For DDP, this term is
specifically used to indicate the process of writing to a data buffer by a
DDP implementation.  DDP Segments carry Placement information, which may be
used by the receiving DDP implementation to perform Data Placement of the
DDP Segment ULP Payload. See "Data Delivery".
[UE1]

> - once the semantics change, it's not TCP
> 	having dual stacks (TCP and not-TCP), and using
> 	a handshake to pick between them doesn't make
> 	the protocol "interoperable"
> 
> 	if you want different semantics, use a different
> 	transport protocol number.
> [UE] The number of stacks on the end station or how they are picked is
> beyond the scope of a wire protocol specification.  MPA is a ULP that uses
> TCP and preserves the TCP interoperability on the wire. And therefore,
like
> any new ULP on top of TCP it does not requires "using a handshake to pick
> between them". 
> Hence, if the term "non-TCP" is used, one should point out in what way
this
> is not TCP from the wire stand point. 
> [UE]

TCP very specifically indicates when data is to be passed to the 
application. FWIW, many protocols specify both on-the-wire and events at 
the endpoints. On-the-wire equivalent isn't equivalence.

MPA doesn't preserve TCP interoperability on the wire, either - it has 
two completely separate protocols, one that it uses when the other end 
has MPA, one when it doesn't. That's two stacks.

[UE1] Let's address this two separate topics (on-the-wire, endpoint)
separately.  
So let me ask this again, how does MPA+TCP change anything for the wire
behavior?

As for the endpoint, I think your point is implementation specific on how
one would do it: in one stack or two or potentially as Jim Williams
indicated in his mail.

[UE1] 
> - not-TCP specifies PDU sizes that cause problems
> 	in particular, TCP uses the interface MTU or path MTU
> 	as the unit of both transmission and congestion control.
> 
> 	this uses the application PDU as the unit in both cases.
> [UE] it is the other way actually, the FPDU is determined by the supported
> path MTU. Which makes sure there is no IP fragmentation issue.
> [UE]

 From the draft:
    The sending ULP MUST NOT post a ULPDU larger than 64768 octets to
    MPA. The ULP MAY post a ULPDU of any size between one and 64768
    octets, however MPA is NOT REQUIRED to support a ULPDU length that
    is greater than the current MULPDU.

    MULPDU - Maximum ULPDU. The current maximum size of the record that
        is acceptable for the ULP to pass to MPA for transmission.

It doesn't say "MUST NOT" send MULPDUs larger than EMSS.

    MPA combined with an MPA-aware TCP can only ensure FPDU Alignment
    with the TCP Header if the FPDU is less than or equal to TCP's EMSS.
    Thus if FPDU alignment is desired by the ULP, the ULP must cooperate
    with MPA to ensure FPDUs lengths do not exceed the EMSS under normal
    conditions.
...
    To provide optimum performance, a transmit side TCP implementation
    SHOULD:

    *   With an EMSS large enough to contain the FPDU, segment the
        outgoing TCP stream such that the first octet of every FPDU is
        aligned with the beginning of a TCP segment, and is entirely
        contained in the TCP segment.
...
    A possible, but less desirable, alternative is to use IP
    fragmentation on accepted FPDUs to deal with MTU reductions or
    extremely small EMSS.

QED - there is no requirement to avoid FPDUS > EMSS, or therefore to 
avoid fragmentation. Once fragmentation occurs, copy avoidance is ruined.

IMO, this protocol will be significantly less effective if it enforces 
that FPDUs <= EMSS; either it will violate TCP semantics by rejecting 
socket sends, or it MUST disable MPA. Notably because the efficiency 
gained by MPA will be stripped by the context switching overheads of 
requiring socket sends for each FPDU, where these FPDUs will largely be 
512-1500 bytes (this is the IETF, not the LAN-ETF).

[UE1] It is not a functionality issue, hence no need to use "MUST NOT". It
is merely a receiver buffer issue. In case the ULPDU is larger than EMSS,
the ULPDU may occupy more than one TCP segment and therefore the receiver
will need to buffer it before it can recover the FPDU boundaries or not
enjoy the benefits of FPDU alignment. Hence, the MPA draft provides a good
advice: a receiver optimized sender (ULP) will not post ULPDU larger than
MULPDU. One could use ANY ULP, but reap more benefits from a ULP complying
with this approach. There is no scenario of rejecting socket sends or
causing a socket send for each FPDU. It is envisioned that on top of MPA one
could use DDP or DPP/RDMAP. DDP provides a segmentation service that
eliminates any need to change socket sends or to have any negative
performance implications

Note also that MPA provides a significant benefit to the receiver by
allowing zero copy with no buffering! Not only is the receiver simpler and
enjoys lower cost it can also scale to higher speeds. Without cutting back
on the number of copies the memory subsystem is maxed out before you get to
10Gbps....Therefore, in any analysis, you may want to look at both the
sender and receiver and not just at the sender.
[UE1]
> 	this implies that:
> 	- IP fragmentation will be more likely to occur, and all
> 	the copy avoidance and direct data placement desired at the
> 	TCP layer will be obviated by the copying now occuring during
> 	IP reassembly.
> 
> 	- not-TCP won't be able to retransmit the portion actually lost;
> 	instead, it'll retransmit the entire FPDU.
> 
> [UE] The FPDU matches the path MTU and is sent in one TCP segment that is
> therefore not fragmented at the IP layer. Now, the chance that a portion
of
> it is lost while the rest of the segment made it to the receiver is very
> small. So it is not clear why this can make any difference, really
> [UE]

See above. Furthermore, MPA either increases fragmentation (where apps 
write large chunks), or whatever benefits it gets from direct placement 
will be wiped out by context switch overheads.

[UE1]
Again, DDP as the ULP on top of MPA complies with this requirements and
therefore no additional fragmentation takes place. For any other ULP that
wishes to use MPA directly it is advised to comply with this requirement. If
for some reason the ULP is non compliant with this requirement MPA can use
larger FPDU with the loss of some of the receive side benefits.

Context switch - the use of MPA/DDP/RDMAP on top of TCP is DESIGNED
primarily to reduce context switches. These savings are not theoretical and
can be measured with VIA or Inifiniband systems that use similar techniques
to avoid the context switch. It is easy to show that WITHOUT this
technology, the CPU will run out of gas due to context switch way before
multi gig per second can be achieved. 
[UE1]

> 	- not-TCP will adjust its congestion window in larger chunks
> 	than TCP, potentially swamping competing TCP flows.
> [UE] so a realistic scenario for a gigabit+ TCP is to adjust the
congestion
> window by say 40 bytes at the time? How long will it take for such a TCP
to
> ever get to wire speed?
> [UE]

This isn't the Gigabit-ETF either. TCP has known problems in extreme 
environments; the IETF choses interoperability over efficiency. If this 
version will interfere with TCP flows that don't use MPA, then that's a 
show-stopper.

There are other proposals in this WG to address high bandwidth-delay 
environments in interoperable ways. This ID will either need to be 
interoperable with that sort of solution, or develop a solution which is 
equally interoperable.
[UE1] we are on the same page. It is about interoperability and efficiency.
How about an example to show your point? 
Joe

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Sun, 22 Sep 2002 16:54:42 -0700

...
...
> (see section 5.2).
...

 From that section:
    If an MPA implementation supports passing out of order ULPDUs to the
    ULP, the MPA implementation SHOULD:

SHOULD != MUST.

An ULP _MUST NOT_ use the data out of order unless intervening data has 
been received by TCP.

This protocol is not equivalent to TCP until that can not only be 
asserted, but enforced.

The pair of MPA+DDR _might_ satisfy this constraint, but that would need 
to be specified as a requirement of the pair.

This protocol can't be equivalent to TCP since it can't enforce that 
assertion.

...
> So let me ask this again, how does MPA+TCP change anything for the wire

On the wire, this won't speak with a non-MPA TCP. On the wire, it's a 
different packet, with different rules (MTUs dictated by the application 
not the interface, emission dictated by application socket writes rather 
than TCP timers and rules, etc.). On the wire, this isn't TCP.

I've already addressed why semantically, at the endpoint, it's not TCP.

> one would do it: in one stack or two or potentially as Jim Williams

At the endpoint, when MPA is enabled, the semantics are not that of TCP.

> sender and receiver and not just at the sender.

What I've discussed above is that:

	1) if you don't use the EMSS, you won't get zero-copy
	benefits, and so using MPA is a waste of time

	2) if you do use EMSS, you're stuck with packets so
	small that your application will thrash with the OS
	doing too small write calls, so using MPA is a waste
	of time

If MPA doesn't provide a clear performance benefit, what's the gain?

> therefore no additional fragmentation takes place. 

DDP means there will be a large number of write calls to TCP.

> For any other ULP that

If this is all that specific, then it's not an IETF issue, period. 
Furthermore, if it's that specific, it's not sufficient to call it TCP 
or spec it solely as a mod to TCP; you need to spec it together with all 
the components required to ensure correct semantics and utility.

> How about an example to show your point? 

How about, well, TCP. The point of TCP is that it's interoperable. It's 
not efficient at gigabit rates, but it's certainly interoperable.

As to other possible solutions, see Floyd's recent proposals. Those 
certainly address interoperability; MPA does not.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 22 Sep 2002 18:11:17 -0600

Uri,

Your statement below confuses me.  It is quite possible to achieve
zero copy (and has already been achieved in implementation) with
existing TCP.  One does *not* need MPA for a zero copy implementation.
Each TCP header contains sufficient information about the segment
to directly place it in a stream buffer.

Is it not true that MPA simply tries to achieve this an framed PDUs
at the same time?  It is not so much direct placement and zero copy
which causes me problems with this draft, it is the fact that it is
trying to superimpose PDU delimitation over TCP.  This has already
been better and more suitably accomplished by SCTP (which is also
capable of direct placement and zero copy with framed PDUs).

So where is the need for this MPA draft at all?

--brian

On Sun, 22 Sep 2002, Uri Elzur wrote:

<snip>
> sender and receiver and not just at the sender.
<snip>

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg