Subject: [rddp] MPA TCP analysis draft

From: [email-masked]
Date: Tue, 18 Feb 2003 11:41:37 -0500

This draft (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) has been
out for over a week and generated no discussion.  Let me see if
I can engender some discussion by offering some observations in
my role as WG chair.  All of these are preliminary observations
for further discussion, but I would ask that folks pay particular
attention to the "prove" issue in (3) to avoid the WG chair having
to get out a clue-by-four.

The Elzur MPA TCP analysis draft covers 3 topics:
(1) Packing ULPDUs into a TCP segment
(2) Sender alignment of ULPDUs to the start of TCP segments
(3) Avoiding endpoint confusion about whether MPA is enabled
Discussion of each follows:

-- (1) Packing

The draft recommends that packing of multiple ULPDUs into a
single TCP segment be supported (sender MAY pack, receiver
MUST support receipt of packed segments).  This strikes me
as a good thing and a welcome change.

-- (2) Sender ULPDU alignment to TCP segments

Some of the buffer size calculations seem questionable given the
prior WG consensus that senders using unmodified TCP stacks (which
cannot support alignment in all circumstances) be supported -
a sender running over an unmodified stack will lose alignment
despite its best efforts, so the claimed benefits of alignment
are probably overstated, BUT - see the end of the discussion of
(3) below on identifying sender behavior.

OTOH, the calculation of buffering required for a non-aligned MPA
sender seems to assume no drops or reordering, although considering
those would only increase the advantage of reliable sender alignment.

Overall, I tend to agree that there are useful receiver
implementation optimizations enabled by a sender who can
align ULPDUs to TCP segments.  One that I didn't see mentioned
is that checking the MPA CRC when each packet is received allows
an MPA CRC failure to be dealt with by dropping the offending
packet and using TCP retransmission - an approach that is not
as readily usable if a CRC spans multiple packets that do not
arrive back-to-back.  While this is a layer violation (CRC check
"above" TCP triggers TCP drop), I would be prepared to argue
with the "layering police" that this is acceptable and desirable
because it strengthens an existing TCP function (detection and
rejection of packets corrupted in transmission), and the pure TCP
alternative (TCP Alternate Checksum - RFC 1146) is undeployable
in practice due to the changes it requires to both TCP stacks
and the wire format (lots of stuff will break).

A key design challenge/goal is to ensure that receivers can
take advantage of receiver alignment performance optimizations 
*without* locking out senders that use unmodified TCP stacks
due to their inherent ability to lose alignment.

-- (3) Avoiding Endpoint Confusion about MPA use

I want to remind everyone that our AD asked the WG to "prove" that
senders and receivers cannot be confused about whether MPA is in use;
I assure everyone our AD chose the word "prove" deliberately.  With
suitable design attention to the MPA Hello/Hello ACK/ACK details,
I believe such a mechanism has a good possibility of providing a
suitable "proof" - Figure 5 looks like a good starting point, and
its conciseness is a positive indication.

OTOH, I have serious concerns about the protocol analysis required
(Section 8.2) for transition via a well known port or service
location protocol (Figure 4).  With apologies to the authors (whose
hard work on this draft is acknowledged and appreciated), that
analysis reminds me of an old joke about how to "prove" that
all odd numbers > 1 are prime:
	3 is prime
	5 is prime
	7 is prime
	9 is ... experimental error	
	11 is prime
	13 is prime
	lacking a counterexample, they must all be prime ... :-) :-)
[for "experimental error", see the DNS discussion in Section 8.2.4.3]

In all seriousness, the inability to find a counterexample does
not prove anything, as there are many more protocols that run
over TCP than those considered in Section 8.2, and who knows whether
32 bits of zero is a command code in some private protocol that
hasn't been considered.

It seems to me that efforts might be better directed towards using
an MPA Hello/Hello ACK/ACK or similar mechanism in all cases,
even when a well-known port or service location protocol is used for
the transition into MPA mode.  Such a design would also provide an
opportunity to address a nagging buffering issue - a receiver that
can't distinguish an MPA-modified aligning TCP sender from an
unmodified TCP sender who is nonetheless using MPA (e.g., for the CRC)
will have to pessimistically assume that all senders are unmodified TCPs
and allocate the larger buffering resources discussed in Section 7.1.

In the current MPA design, I don't believe a receiver can make
that distinction.  In contrast, if different MPA Hello codes were
used for the two classes of senders, the receiver could more reliably
optimize the buffering for the MPA-modified aligning TCP that should
require less buffering.  A receiver will still have to watch out for
unexpected alignment failures even from MPA-modified aligning
TCP senders (e.g., courtesy of Caitlin's resegmenting middleboxes
that delay making themselves known).

This is all for further discussion ... I would suggest that replies
to this message pick out one of the three topics above and change
the message subject accordingly.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 18 Feb 2003 15:55:24 -0600

The MPA analysis draft offers the following formula starting
on page 10:

> For the case when the FPDU is not aligned with the TCP
> segment, it may take, on average, 2 TCP segments to
> assemble one FPDU. Therefore, the receiver has to
> allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU)
> octets: 
>
>       BufferSizeNAF = K1* EMSS * number_of_connections +
>           K2 * EMSS 
>
>
> Where K1 and K2 are implementation dependent constants
> and EMSS is the effective maximum segment size.  
>
> For example, a 1 Gbps link with 10,000 connections and an
> EMSS of 1500B would require 15 MB of memory. Often the
> number of connections used scales with the network speed,
> aggravating the situation for higher speeds.  

The derivation of this formula does not seem to be supported
anywhere in the text.

In fact I can't figure out where it came from, or what
assumptions are being swept under the rug in K1 and K2.

As pointed out, 2 adjacent TCP segments should allow the
assembly of one FPDU. (In fact it is only the strangest
of corner cases where this is not true). So the question
becomes, how many TCP segments might have to be buffered
before an FPDU is assembled and can be placed?

Phrased that way, the answer is obviously half of the
TCP segments that can be in flight. The worst case scenario
is clearly dropping every other TCP segment. (And as
previously noted, if a load-balancing router dispatches
successive segments by alternate path that is *exactly* what
could happen if one path is delayed).

I believe that is a simpler explanation, and formula.
It does not change any of the conclusions, however.

If the formula presented actually represent a *lower*
estimate of the required buffering then the rationale
should be explained.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 19 Feb 2003 11:36:26 -0600

On 2/18/03, [email-masked] wrote:

>This draft (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) has
>been out for over a week and generated no discussion.  Let
>me see if I can engender some discussion by offering some
>observations in my role as WG chair.  All of these are
>preliminary observations for further discussion, but I
>would ask that folks pay particular attention to the
>"prove" issue in (3) to avoid the WG chair having to get
>out a clue-by-four.
>

This nearly makes each ULPDU the equivalent of an SCTP Data
Chunk. Aside from suggesting that the feature be renamed 
"bundling", to match SCTP, I believe we can take the lead
from SCTP to identify relevant issues.

SCTP allows the ULP to request that a Data Chunk not be
delayed for purposes of bundling, The DDP/SCTP draft
specifies invoking that option for untagged messages.
A similar requirement/recomendation is probably appropriate
for SCTP.

SCTP allows re-bundling on retransmission, and explicitly
states that the DO_NOT_BUNDLE option is to be ignored on
retransmission.

While SCTP's rationale is fully applicable to MPA/TCP, there
is a major difference. Retry may be implemented entirely at
the TCP layer. Indeed it may be handled by a middle-box
unaware of MPA. More importantly, the data bytes for a TCP
stream should be immutable, not merely equivalent on a
retransmit.

I believe the draft should be explicit that MPA presents
segments to TCP *once* and that retransmission MUST NOT
result in re-bundling/re-packing.

Caitlin Bestler

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 21 Feb 2003 16:20:29 -0800

I wanted to explore the assumption behind the memory sizes somewhat
more.

The formulae given are

BufferSizeNAF = K1 * EMSS * number_of_connections + K2 * EMSS

and

BufferSizeAF = K2 * EMSS

There is no question that overall there is a lot of simplification
if there is header alignment, and there is only upper layer PDU
per TCP segment (BTW by having multiple Upper Layer PDUs per
segment, a lot of the simplification has been lost).

However, it is important that the protocol not be optimized only
around this, since alignment cannot be maintained by software
based senders without modifications to the TCP stack. I pointed
out references to the BSD TCP software in one of my previous emails,
and can repeat it if needed.

Now to discussing the formuale:

1. There is an assumption that the DDP & RDMA headers are not
needed or buffered elsewhere? For an implementation that
wants to check for and protect against violation of the DDP/RDMA
protocols, the headers will be needed later when the gaps have
been filled in (an example is that untagged MSN must be in
increasing order). The oft repeated rule is out of order placement
and not out of order delivery.

We can make a number of assumptions here.

a. It is not necessary as we don't expect this to happen.
b. Our NIC is only a TCP Direct Data Placement NIC and defers
   all DDP/RDMA processing to the host
c. We have to buffer the headers. In such a case, the BufferSize
   becomes a function of the number of connections regardless of
   the method chosen. (see discussion of number of connections below)

2. There is an assumption of markers being smaller than the EMSS. 
Since markers are essential to locating the header (either in
MPA or in IFT), you have to first find a marker, and only then
can you place the data if segments are out of order. As Divid
pointed out, a smaller marker interval puts an unacceptable
burden on a software client (I thing the example was of a
64KB read/write requiring 140+ markers - each of which causes
creation of that many segments, plus creating and inserting
markers and marker vectors etc) - worse than simply using
a NIC doing a large TCP send or TOE offload.

a. Either severely penalize software based senders (at hopefully
your own risk rather than institutionalizing the risk via the
protocol).

b. or alternatively add a factor of
    K1 * Marker Interval/EMSS * number_of_connections to the
buffering needs.

3. The number of connections in the above equations is really
the number of connections with out of order segments - and not
the total number of connections. Of course, for this you would
have to have the separate header and data CRC as proposed by Jim.
Hopefully this is a much smaller number than the number of
connections.

Of course, if every other segment is out of order, one does not have
worry about buffering or memory speeds etc because TCP algorithms
will force you to a rate in the range of KBytes and not Gbits.
TCP is not a credit based mechanism but uses the AIMD algorithms to
"detect" network congestion. It will assume that it needs to back
off significantly to prevent a congestion collapse of the network.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Caitlin Bestler
> Sent: Tuesday, February 18, 2003 1:55 PM
> To: [email-masked]
> Subject: Re: [rddp] MPA TCP analysis draft
> 
>

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 21 Feb 2003 20:25:26 -0600

On 2/21/03, Somesh Gupta wrote:

>c. We have to buffer the headers. In such a case, the
> BufferSize becomes a function of the number of
> connections regardless of the method chosen. (see
> discussion of number of connections below)
>
My working assumption is that placement is done by the RNIC.
The data that controls the placement may be edited on the
RNIC based on queued requests, or directly edited by host
based software.

The RDMA Read Engine could be implemented either as another
block on the RNIC or on the host.

There is no need to buffer received segments after
placement.

>2. There is an assumption of markers being smaller than
>the EMSS. Since markers are essential to locating the
>header (either in MPA or in IFT), you have to first find a
>marker, and only then can you place the data if segments
>are out of order. As David pointed out, a smaller marker
>interval puts an unacceptable burden on a software client
>(I thing the example was of a 64KB read/write requiring
>140+ markers - each of which causes creation of that many
>segments, plus creating and inserting markers and marker
>vectors etc) - worse than simply using a NIC doing a large
>TCP send or TOE offload.
>
The pattern where the true payload is a repeating pattern.
140 markers does not require 140 distinct descriptors.
If you look up old mini-computers you'll find many examples
of repeating groups in DMA control.

>
>Of course, if every other segment is out of order, one
>does not have worry about buffering or memory speeds etc
>because TCP algorithms will force you to a rate in the
>range of KBytes and not Gbits. TCP is not a credit based
>mechanism but uses the AIMD algorithms to "detect" network
>congestion. It will assume that it needs to back off
>significantly to prevent a congestion collapse of the
>network.
>
The primary scenario that I can come up with is when one of
two load balanced paths fails. That will indeed result in a
rapid drop in the memory speed, but that won't help you
*this* msec.

Caitlin Bestler

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 22 Feb 2003 12:02:42 -0600

On 2/21/03, Somesh Gupta wrote:

>1. There is an assumption that the DDP & RDMA headers are
>not needed or buffered elsewhere? For an implementation
>that wants to check for and protect against violation of
>the DDP/RDMA protocols, the headers will be needed later
>when the gaps have been filled in (an example is that
>untagged MSN must be in increasing order). The oft
>repeated rule is out of order placement and not out of
>order delivery.

I presume you are talking about something like receiving
two DDP segments which are back-to-back in LLP ordering
for the DDP stream, but claim to finish MSN: N and MSN: N+2.

The receiving DDP MAY detect that MSN N+1 can never arrive
and therefore the stream is in error and should be
terminated.

I would suggest that it SHOULD have better uses for its
processing capacity.

Rules requiring MSNs to be in sequence are intended to allow
the receiver to optimize its code, not to add to a list of 
validity checks that it MUST perform.

So if the receiving DDP notes that it cannot properly
generate a completion to the ULP because there is
a gap in the MSNs, it should definitely terminate
the connection and report an error. But that's detecting
the error as a by-product of its normal operations.
Similarly, A duplicate MSN should be detected, because the
first time the final segment of MSN N is received it should
result in completion of the pre-posted receive operation.
The second one would have nothing to complete, hence BOING.

The receiver is only obligated to validate its peer run-time
compliance with the spec when out-of-spec behaviour could
be used to compromise system security. But Out-of-sequence
MSNs do not place the receiver's resources at risk. It
cannot be used to obtain access to any ULP buffers that
could not have been accessed in order.

LLP checks against improper sequencing are common because
they are a method of spotting forged packets. But I would
submit that this is an LLP responsibility, DDP should
presume that the packet passed to it is a valid part
of the DDP stream and not attempt to do the LLP's job.

Caitlin Bestler

---

From: "Somesh Gupta" <[email-masked]>
Date: Sat, 22 Feb 2003 12:37:52 -0800

I wonder if we are down a slippery slope on this one
where the behavior of the ULP (and its externally
visible behavior) starts to depend on whether a TCP
packet was dropped or not. 

Even if this was acceptable from the protocol (TCP based
ULP at that) perpectice,
all headers that did have completion information in
them have to be buffered (or some reduction of them).

Also perhaps all ULP rules that
may not be enforced under packet drop scenarios vs
when everything is in order should be removed from the
specification and converted into loose guidelines.

Somesh

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Saturday, February 22, 2003 10:03 AM
> To: Somesh Gupta; [email-masked]
> Subject: RE: [rddp] MPA TCP analysis draft
> 
>

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 22 Feb 2003 15:29:30 -0600

>packet was dropped or not. 
DDP runs over a reliable IP-based LLP. If an LLP packet
has been lost, the connection will be terminated.

As with any LLP error, the timing of when that error will
be reported to the local ULP is not determined. In that
sense the ULP behavior is "dependent" on the handling.
But in the sense that the connection will be terminated
the behavior is totally consistent.

LLP enforcement of LLP rules is not an issue for RDMAP/DDP.

Further the receiving RDAMP/DDP is not required to
dynamically check that its peer is compliant with the
protocol's specifications unless failure to do so exposes
the host to an attack.

I cannot come up with any scenario where playing games
with DDP segments can compromise memory that had not already
been exposed by the ULP and was known by the ULP to be
exposed to the remote peer.

Similarly, how one ULP peer protects itself against its
alleged collaborator is a ULP issue, not an RDMAP/DDP one.

There are rules that MUST be enforced. For example, after
delivering the untagged message X which invalidated STag Y
any use of STag Y MUST be rejected. Event if it pretends to
have a lower MSN.

But it is also mostly irrelevant. Each untagged message will
have a unique MSN (within a wrap-around cycle), and it will
generate exactly one completion to the ULP. Further,
reception of all untagged messages MUST be pre-authorized by
the ULP. So there will only be a small range of valid MSNs
at any given time, and inherently each one can only have a
single final segment.

Those are all by-products of maintaining a ReceiveQ (or some
equivalent thereof). There is no need to retain headers of
DDP Segments after their data has been placed. That applies
to untagged segments, not just tagged ones.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Sun, 23 Feb 2003 07:02:09 -0800

>and not out of order delivery.

Implementations clearly do not have to buffer any headers - they only need 
to understand the number of in-flight messages and know when the LLP window 
has been filled in such that a given message can be delivered.  Tracking of 
such information can be done, for example, within the posted work request 
which is used by the implementation to describe the associated physical 
buffer if desired.  This is again an implementation option, and as one can 
see, it is fairly straight forward to implement in hardware - there are 
many other implementation choices one could use to solve this type of 
problem without requiring the headers to be maintained.

>    all DDP/RDMA processing to the host

One of the objectives of the RDMA / DDP protocols is to allow a source 
endnode to interact with remote memory without requiring any remote 
interaction.  While an implementation is not precluded from having remote 
interaction (cannot be seen at the wire protocol level), one does take away 
some of the customer visible value propositions if the remote processor is 
used in any way to perform a RDMA Write, RDMA Read, or SEND operation as 
this reduces the value of the functionality provided by RDMA/DDP as well as 
the potential system / application performance.

>a NIC doing a large TCP send or TOE offload.

The optimal design is a hardware implementation that provides marker 
insertion / removal.  The objective is to insure that one can implement and 
interoperate with a software-based solution without creating a complete 
performance nightmare but also with the understanding that a software-based 
solution will not perform as well as a hardware-based solution.  The 
current draft specifications attempt to strike a good balance in allowing 
high-performance hardware implementations while insuring software-based can 
be constructed.  I know of at least a couple of software implementations 
that are in-flight today and no one is stating the use of markers or its 
proposed interval are a problem.

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 24 Feb 2003 09:24:11 -0800

> Sent: Saturday, February 22, 2003 1:30 PM
> To: Somesh Gupta
> Cc: [email-masked]

  Dropped not lost. Dropped (and subsequently recovered)
  packets should not be visible to the ULP, much less have
  an impact on its behavior.
> the behavior is totally consistent.

  I am somewhat confused over here. For TCP dropped and
  subsequently recovered packets are normal and should not
  have any impact on ULP behavior. Obviously I am missing
  a point here.

> LLP enforcement of LLP rules is not an issue for RDMAP/DDP.

  I was talking about enforcing DDP rules here.
> protocol's specifications unless failure to do so exposes

  That means that it is not important to have those rules
  in the protocol specification.

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 24 Feb 2003 09:24:10 -0800

> Michael Krause
> Sent: Sunday, February 23, 2003 7:02 AM
> To: [email-masked]

  Clearly some form of information has to be kept (does not have to
  be headers). If the DDP rules are to be verified, then more information
  has to be kept. Even if rules are not verified in out of order
  segments case, the headers or variation thereof that generate completion
  must be kept. The DDP specification should be changed if rules are
  not to be verified - otherwise the information to be maintained is
  of the order of number of connections. Or scoreboarding can be done,
  which is not likely to be easier.

> proposed interval are a problem.

  I have to disagree with both points over here. Markers do pose a
  burden on software implementations (vectors have to be split,
  memory for markers has to be allocated and vectors created for it,
  number of calls to system have to increase, a number of completions
  from NIC will increase. The penalty is quite significant.

  Similarly, the current draft does not allow optimization of RNICs
  that work well with software based senders (or penalizes it to
  the point that it is no longer a very viable choice). The IFT draft
  strikes a much better balance for both environments.