Subject: RE: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

From: "Uri Elzur" <[email-masked]>
Date: Thu, 19 Sep 2002 17:57:59 -0700

pls see comments below [UE]

-----Original Message-----
From: Joe Touch [mailto:[email-masked]]
Sent: Thursday, September 19, 2002 11:18 AM
To: allyn
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Tsvwg] FW: I-D ACTION:draft-culley-iwarp-mpa-00.txt

allyn wrote:
> No reason not to use SCTP, to the contrary, every reason to use it. The 
> RDMA protocols - DDP and RDMA protocols are to be transport independent 
> and there is to be a mapping to both SCTP and to TCP. The MPA draft is 
> for framing to allow mapping to TCP.
> What do you think about the details of the proposal?

I have the same concerns the last time I saw this sort of thing:

- TCP semantics change:
	consider a sequence of FPDUs (or in TCP, segments) sent:
	1,2,3,4
	and let '2' be dropped by the channel.

	now close the connection.

	what does the receiver pass to the application?
	- in TCP, only 1

	- in this (not-TCP), 1,3,4

[UE] reading the draft carefully would suggest there is a distinction
between PLACEMENT and "Delivery". In the corner case described above,
MPA+TCP behaves exactly like TCP. It will only DELIVER to the application
all the data that has been received  to the left of the TCP window edge. In
this case "only 1".
[UE] 

- once the semantics change, it's not TCP
	having dual stacks (TCP and not-TCP), and using
	a handshake to pick between them doesn't make
	the protocol "interoperable"

	if you want different semantics, use a different
	transport protocol number.
[UE] The number of stacks on the end station or how they are picked is
beyond the scope of a wire protocol specification.  MPA is a ULP that uses
TCP and preserves the TCP interoperability on the wire. And therefore, like
any new ULP on top of TCP it does not requires "using a handshake to pick
between them". 
Hence, if the term "non-TCP" is used, one should point out in what way this
is not TCP from the wire stand point. 
[UE]

- not-TCP specifies PDU sizes that cause problems
	in particular, TCP uses the interface MTU or path MTU
	as the unit of both transmission and congestion control.

	this uses the application PDU as the unit in both cases.
[UE] it is the other way actually, the FPDU is determined by the supported
path MTU. Which makes sure there is no IP fragmentation issue.
[UE]
	this implies that:
	- IP fragmentation will be more likely to occur, and all
	the copy avoidance and direct data placement desired at the
	TCP layer will be obviated by the copying now occuring during
	IP reassembly.

	- not-TCP won't be able to retransmit the portion actually lost;
	instead, it'll retransmit the entire FPDU.

[UE] The FPDU matches the path MTU and is sent in one TCP segment that is
therefore not fragmented at the IP layer. Now, the chance that a portion of
it is lost while the rest of the segment made it to the receiver is very
small. So it is not clear why this can make any difference, really
[UE]

	- not-TCP will adjust its congestion window in larger chunks
	than TCP, potentially swamping competing TCP flows.
[UE] so a realistic scenario for a gigabit+ TCP is to adjust the congestion
window by say 40 bytes at the time? How long will it take for such a TCP to
ever get to wire speed?
[UE]
(not intended to be a complete list, but a start)

Joe

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Fri, 20 Sep 2002 07:59:47 -0700

> all the data that has been received  to the left of the TCP window edge. In

The word "placement" occurs exactly twice in the ID, in reference to 
DDR. Please clarify.

I think you mean that TCP passes the data to the ULP as soon as 
possible, and that there is an additional mechanism for TCP to indicate 
that the right window edge has passed. HOWEVER, the ID is very clear to 
avoid specifying properties of the ULP.

As far as TCP is concerned, the ULP _IS_ the application. TCP cannot 
pass data to the ULP until the right window edge has passed, precisely 
because it cannot ensure that the ULP won't use information in that data 
until the right window edge has passed.

To use that information prior to that event would constitute a change in 
semantics, period, whether by an ULP or an application.

> is not TCP from the wire stand point. 

TCP very specifically indicates when data is to be passed to the 
application. FWIW, many protocols specify both on-the-wire and events at 
the endpoints. On-the-wire equivalent isn't equivalence.

MPA doesn't preserve TCP interoperability on the wire, either - it has 
two completely separate protocols, one that it uses when the other end 
has MPA, one when it doesn't. That's two stacks.

> path MTU. Which makes sure there is no IP fragmentation issue.

 From the draft:
    The sending ULP MUST NOT post a ULPDU larger than 64768 octets to
    MPA. The ULP MAY post a ULPDU of any size between one and 64768
    octets, however MPA is NOT REQUIRED to support a ULPDU length that
    is greater than the current MULPDU.

    MULPDU - Maximum ULPDU. The current maximum size of the record that
        is acceptable for the ULP to pass to MPA for transmission.

It doesn't say "MUST NOT" send MULPDUs larger than EMSS.

    MPA combined with an MPA-aware TCP can only ensure FPDU Alignment
    with the TCP Header if the FPDU is less than or equal to TCP's EMSS.
    Thus if FPDU alignment is desired by the ULP, the ULP must cooperate
    with MPA to ensure FPDUs lengths do not exceed the EMSS under normal
    conditions.
...
    To provide optimum performance, a transmit side TCP implementation
    SHOULD:

    *   With an EMSS large enough to contain the FPDU, segment the
        outgoing TCP stream such that the first octet of every FPDU is
        aligned with the beginning of a TCP segment, and is entirely
        contained in the TCP segment.
...
    A possible, but less desirable, alternative is to use IP
    fragmentation on accepted FPDUs to deal with MTU reductions or
    extremely small EMSS.

QED - there is no requirement to avoid FPDUS > EMSS, or therefore to 
avoid fragmentation. Once fragmentation occurs, copy avoidance is ruined.

IMO, this protocol will be significantly less effective if it enforces 
that FPDUs <= EMSS; either it will violate TCP semantics by rejecting 
socket sends, or it MUST disable MPA. Notably because the efficiency 
gained by MPA will be stripped by the context switching overheads of 
requiring socket sends for each FPDU, where these FPDUs will largely be 
512-1500 bytes (this is the IETF, not the LAN-ETF).

> small. So it is not clear why this can make any difference, really

See above. Furthermore, MPA either increases fragmentation (where apps 
write large chunks), or whatever benefits it gets from direct placement 
will be wiped out by context switch overheads.

> window by say 40 bytes at the time? How long will it take for such a TCP to

This isn't the Gigabit-ETF either. TCP has known problems in extreme 
environments; the IETF choses interoperability over efficiency. If this 
version will interfere with TCP flows that don't use MPA, then that's a 
show-stopper.

There are other proposals in this WG to address high bandwidth-delay 
environments in interoperable ways. This ID will either need to be 
interoperable with that sort of solution, or develop a solution which is 
equally interoperable.

Joe