Subject: RE: [rddp] MPA TCP analysis draft

From: Michael Krause <[email-masked]>
Date: Mon, 24 Feb 2003 09:53:54 -0800

At 09:24 AM 2/24/2003 -0800, Somesh Gupta wrote:

> > Implementations clearly do not have to buffer any headers - they
> > only need
> > to understand the number of in-flight messages and know when the
> > LLP window
> > has been filled in such that a given message can be delivered.
> > Tracking of
> > such information can be done, for example, within the posted work request
> > which is used by the implementation to describe the associated physical
> > buffer if desired.  This is again an implementation option, and
> > as one can
> > see, it is fairly straight forward to implement in hardware - there are
> > many other implementation choices one could use to solve this type of
> > problem without requiring the headers to be maintained.
>
>   Clearly some form of information has to be kept (does not have to
>   be headers). If the DDP rules are to be verified, then more information
>   has to be kept. Even if rules are not verified in out of order
>   segments case, the headers or variation thereof that generate completion
>   must be kept. The DDP specification should be changed if rules are
>   not to be verified - otherwise the information to be maintained is
>   of the order of number of connections. Or scoreboarding can be done,
>   which is not likely to be easier.

The DDP specification is quite clear on how to validate an operation and 
delineating the responsibilities of the LLP and the DDP layers.  How an 
implementation chooses to enforce these rules is up to it but I do not see 
anything stated as yet to indicate that the rules are incorrect or that it 
is not possible to implement a compliant solution (either in software or 
hardware).

> > The optimal design is a hardware implementation that provides marker
> > insertion / removal.  The objective is to insure that one can
> > implement and
> > interoperate with a software-based solution without creating a complete
> > performance nightmare but also with the understanding that a
> > software-based
> > solution will not perform as well as a hardware-based solution.  The
> > current draft specifications attempt to strike a good balance in allowing
> > high-performance hardware implementations while insuring
> > software-based can
> > be constructed.  I know of at least a couple of software implementations
> > that are in-flight today and no one is stating the use of markers or its
> > proposed interval are a problem.
>
>   I have to disagree with both points over here. Markers do pose a
>   burden on software implementations (vectors have to be split,
>   memory for markers has to be allocated and vectors created for it,
>   number of calls to system have to increase, a number of completions
>   from NIC will increase. The penalty is quite significant.

I have a different implementation in mind here so obviously costs will 
vary.  A minimum implementation in software will have:

         - If the data + markers fits into a single segment, then there is 
no additional memory costs as most implementations use power of 2 memory 
buffers to simplify the design so finding the right size with some 
additional room may be nothing more than moving up to the next power of 2 
buffer size.

         - If the data + markers does not fit into a single segment, then 
it will require an additional segment for the overflow caused by 
markers.  However, most software implementations already deal with packet 
trains today so the overhead is not significant (function of implementation).

         - The number of calls to insert / remove markers is worse case 1 
additional call but most implementations will perform these operations as 
part of their copy or checksum calculations.

         - There is no increase in the number of completion events from the 
NIC except perhaps when there are additional packets received. Most good 
NIC implementations already contain coalescence techniques to reduce / 
eliminate completion events wherever possible so again this may be 
implementation specific.

 From our work to date, the implementation complexity is not high nor 
excessively burdensome; mileage will vary.

>   Similarly, the current draft does not allow optimization of RNICs
>   that work well with software based senders (or penalizes it to
>   the point that it is no longer a very viable choice). The IFT draft
>   strikes a much better balance for both environments.

Please specify.  I have yet to find a RNIC that creates any type of penalty 
regarding a software-based implementation as the RNIC is implementing the 
RDMA / DDP components already.  Now, if you are referring to a TOE 
interface, then this is a function of the TOE implementation as to what it 
provides.  A well-layered implementation would simply hand a buffer with 
the markers already included to the TOE interface who would transmit the 
buffer just like any other traffic.  For inbound, the TOE would deliver a 
buffer normally and it would be up to the

 From our prototype work to date with TOE, we have not found any undue 
complexity / penalties and this is what I expect given the MPA / DDP / RDMA 
specifications are capable of being implemented over a well-layered 
implementation.

Mike

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp