Subject: RE: [rddp] RE: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

From: Parag Bhide <[email-masked]>
Date: Fri, 25 Oct 2002 10:13:22 -0700

I agree that out of order processing will help reducing the buffering
requirements. But it is important to validate both the DDP header (which
contains the memory address information) *and* the fpdu length provided by
MPA header before data can be placed into memory.

This requires buffering of a complete MPA pdu for CRC validation before it
can be placed. This is a bit of a problem especially when MPA pdus got
fragmented due to MPA un-aware sender and/or middle boxes. Buffering
requirements become worse with jumbo frames (9KB MPA PDU).

It would be nice if there is a way to validate the MPA and DDP headers
(Header CRC that would cover MPA header and DDP header) without waiting for
complete MPA pdu. This will allow NIC to perform CRC check on data while it
is being placed into the host memory.

It was suggested on the reflector that this issue should be addressed by
DDP. But since fpdu length is provided by MPA and it should be the LLP which
should provide for data integrity, this issue should be addressed by MPA.

Thanks.

Parag
Aarohi Communications, Inc.

-----Original Message-----
From: Williams, Jim [mailto:[email-masked]] 
Sent: Friday, October 25, 2002 8:17 AM
To: [email-masked]; [email-masked]
Subject: [rddp] RE: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

> -----Original Message-----
> From: Culley, Paul [mailto:[email-masked]]
> 
> Interestingly enough, during our framing discussions, both 
> the added header CRC and Markers were seriously discussed 
> before we came to our conclusions.  We had several NIC 
> vendors present who presented private analysis as well as 
> (too many) opinions based on various potential NIC models.  
> We came to several conclusions:
> 
> 1) You are right that packet processing rates require an 
> elasticity buffer.
> 
> 2) When many connections are running at one time, the NIC 
> memory needed for dealing with the  out-of-order packets 
> scales by the number of connections.  We intend to support 
> large numbers of connections (100s to 10000s).  The memory 
> needed rapidly goes out of range of what is needed for 
> elasticity.  The amount of memory also scales with latency 
> bandwidth product, at 1-10 gigabit this gets nasty very fast. 
> Thus out of order processing and markers to absolutely 
> identify them...

This is really the critical core issue we need to discuss 
in more depth.

My views tend to be as follows.

  A.  I think you are underestimating the amount of memory
      required for elasticity buffering, and overestimating
      the memory required for out of order reassembly.

  B.  Once the cost is incurred of adding external (off ASIC)
      memory of sufficient bandwidth for elasticity, the
      incremental cost of additional megabytes is quite
      low.

  C.  I question whether memory required for out of order
      reassembly scales with number of connections.  On a
      particular connection, the required memory should 
      never be larger than the bandwidth delay product
      of that connection.  The more active connections
      there are, the less bandwidth is available for each
      connection.

  D.  A product design needs to set performance goals at two
      (at least) different traffic mixes, large
      messages, and small messages.  An accelerated NIC
      which is able to support line rate with small messages
      is seriously over designed and non cost effective.
      Supporting line rate at large and medium size messages
      is reasonable, and requires significantly lower peak
      packet processing rates.

      The result of this is that for small messages, one
      needs to rely on elasticity buffering and end to 
      end flow control to avoid large scale dropping of
      incoming packets.  With insufficient buffering,
      the supported rate for small messages will be 
      terrible due to constant dropped packets and 
      retransmission timeouts. 

Unfortunately, at the moment I don't have quantitative
analysis sufficiently clear and documented to share.
I will try to work on that, but it is tricky due the
need to make a number of somewhat arbitrary assumptions.

In the mean time it would be helpful for the RDMAC to share
what ever quantitative analysis they have done to reach 
their conclusions.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 25 Oct 2002 20:38:58 -0500

On 10/25/02, Parag Bhide wrote:

>I agree that out of order processing will help reducing
>the buffering requirements. But it is important to
>validate both the DDP header (which contains the memory
>address information) *and* the fpdu length provided by MPA
>header before data can be placed into memory.
>
>This requires buffering of a complete MPA pdu for CRC
>validation before it can be placed. This is a bit of a
>problem especially when MPA pdus got fragmented due to MPA
>un-aware sender and/or middle boxes. Buffering
Wouldn't the extra buffering required because of possible
resegmentation be a constant ratio, no matter what the
normal size of the TCP segments were?

>It would be nice if there is a way to validate the MPA and
>DDP headers (Header CRC that would cover MPA header and
>DDP header) without waiting for complete MPA pdu. This
>will allow NIC to perform CRC check on data while it is
>being placed into the host memory.
>
>It was suggested on the reflector that this issue should
>be addressed by DDP. But since fpdu length is provided by
>MPA and it should be the LLP which should provide for data
>integrity, this issue should be addressed by MPA.
>
>
That's essentially fancy talk for asking the DDP layer to
re-issue the DDP segment with the new PMTU.

That isn't allowed because it would be a severe layering
violation. DDP must submit the DDP segment to the LLP
exactly once. No do-overs.

Similarly, having the MPA or LLP understand DDP sufficiently
to restructure a DDP segment properly isn't solving the
problem, it's just pretending that it isn't the first
solution by falsely labeling the agent doing the
resegmenting. If "MPA" understands what a DDP segment is,
then it isn't the MPA layer.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp