Subject: [rddp] RE: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

From: "Culley, Paul" <[email-masked]>
Date: Wed, 23 Oct 2002 09:47:30 -0500

Interestingly enough, during our framing discussions, both the added header CRC and Markers were seriously discussed before we came to our conclusions.  We had several NIC vendors present who presented private analysis as well as (too many) opinions based on various potential NIC models.  We came to several conclusions:

1) You are right that packet processing rates require an elasticity buffer.

2) When many connections are running at one time, the NIC memory needed for dealing with the  out-of-order packets scales by the number of connections.  We intend to support large numbers of connections (100s to 10000s).  The memory needed rapidly goes out of range of what is needed for elasticity.  The amount of memory also scales with latency bandwidth product, at 1-10 gigabit this gets nasty very fast.  Thus out of order processing and markers to absolutely identify them...

3) It was observed that a NIC must buffer the entire packet before it can be used in any way if the Ethernet CRC and IP/TCP checksums are checked before use.  Most NICs have macros that check this stuff before any layer 4 or higher processing is started. The NIC vendors found that adding CRC on the whole packet to this scheme offered no disadvantage over separate CRCs.  They would not be able to do a "flow-through" design anyway.

All of this discussion applies to "rototilled" NICs which freely work across layer boundaries.

While strictly layered NICs (and software stacks) will undoubtedly be used, we focused on enabling inexpensive but performant devices.

Paul R. Culley (Sr. Fellow)
Hewlett-Packard Company
281-514-5543

-----Original Message-----
From: Williams, Jim [mailto:[email-masked]]
Sent: Friday, October 18, 2002 3:13 PM
To: [email-masked]'; [email-masked]'
Subject: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title		: iWARP Framing for TCP
	Author(s)	: J. Williams
	Filename	: draft-williams-iwarp-ift-00.txt
	Pages		: 10
	Date		: 2002-10-10
	
A framing protocol is defined for DDP over TCP that is fully
compliant with applicable TCP RFCs and fully interoperable with
existing TCP implementations.  This protocol is being offered 
for discussion and consideration as an alternative to the
proposal offered in "draft-culley-iwarp-mpa-00.txt".

The protocol offers to two things, definition of the DDP record
boundaries within the TCP stream, and added data integrity 
by means of added CRC protection.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-williams-iwarp-ift-00.txt
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Fri, 25 Oct 2002 13:41:23 -0700

Jim,

... inline

> This is really the critical core issue we need to discuss 
> in more depth.
> 
> My views tend to be as follows.
> 
>   A.  I think you are underestimating the amount of memory
>       required for elasticity buffering, and overestimating
>       the memory required for out of order reassembly.

The first part of your statement though true in several scenarios
does not hold in others. Since it is circumstantial, the argument
should not be used in driving a standard. Vendors will find 
applications where that is currently not the case and/or in time as
IO technologies evolve, is going to move around anyway.

> 
>   B.  Once the cost is incurred of adding external (off ASIC)
>       memory of sufficient bandwidth for elasticity, the
>       incremental cost of additional megabytes is quite
>       low.
> 

Again in the first part of the above statement you are making
implementation assumptions. There are other ways of doing ASIC
and infact there are packaging technologies other than ASIC.
Pardon me for being vague here, I am required to :-)

>   C.  I question whether memory required for out of order
>       reassembly scales with number of connections.  On a
>       particular connection, the required memory should 
>       never be larger than the bandwidth delay product
>       of that connection.  The more active connections
>       there are, the less bandwidth is available for each
>       connection.
> 

Agree with you 100%. (For some, it may be an artifcat of an
implementation) However, I don't think Paul meant that. I do 
agree with the main points he is making in 2.

-Shridhar Mukund

---

From: "Williams, Jim" <[email-masked]>
Date: Fri, 25 Oct 2002 11:16:49 -0400

> From: Culley, Paul [mailto:[email-masked]]
> 
> Interestingly enough, during our framing discussions, both 
> the added header CRC and Markers were seriously discussed 
> before we came to our conclusions.  We had several NIC 
> vendors present who presented private analysis as well as 
> (too many) opinions based on various potential NIC models.  
> We came to several conclusions:
> 
> 2) When many connections are running at one time, the NIC 
> memory needed for dealing with the  out-of-order packets 
> scales by the number of connections.  We intend to support 
> large numbers of connections (100s to 10000s).  The memory 
> needed rapidly goes out of range of what is needed for 
> elasticity.  The amount of memory also scales with latency 
> bandwidth product, at 1-10 gigabit this gets nasty very fast. 
> Thus out of order processing and markers to absolutely 
> identify them...

  D.  A product design needs to set performance goals at two
      (at least) different traffic mixes, large
      messages, and small messages.  An accelerated NIC
      which is able to support line rate with small messages
      is seriously over designed and non cost effective.
      Supporting line rate at large and medium size messages
      is reasonable, and requires significantly lower peak
      packet processing rates.

      The result of this is that for small messages, one
      needs to rely on elasticity buffering and end to 
      end flow control to avoid large scale dropping of
      incoming packets.  With insufficient buffering,
      the supported rate for small messages will be 
      terrible due to constant dropped packets and 
      retransmission timeouts. 

Unfortunately, at the moment I don't have quantitative
analysis sufficiently clear and documented to share.
I will try to work on that, but it is tricky due the
need to make a number of somewhat arbitrary assumptions.

In the mean time it would be helpful for the RDMAC to share
what ever quantitative analysis they have done to reach 
their conclusions.

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 28 Oct 2002 11:48:12 -0500

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Friday, October 25, 2002 11:47 AM
To: Williams, Jim
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [rddp] RE: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

Jim, 

Do you have some data about the cost ratio for your point B (initial cost
versus cost/MB)? 

 [...]  

Julo  

I don't know that I could shed any more light on memory pricing than any
number of others on these lists.  Also getting too detailed on price
information
is not helpful for designing a protocol expected to outlive any current
pricing.
 
However I would point out that to support 10G with current DDR SDRAM
technology, one would probably need to go at least 64 bits wide on the 
interface.  At this width, once you pay for the first megabyte, you get
the next 31 or 63 megabytes for free.  (I.e. it does not make sense
to buy less than 32 or 64 megabytes given the needed
bandwidth and required bus width to support this.)
 
I would expect that the ratio of minimum practical memory 
size to required bandwidth will only get larger over time as memory
technology advances.
 
The numbers would of course be different if you look at QDR SRAM
or other technologies.  But the basic principle is the same. 

 - Jim 

	"Williams, Jim" <[email-masked]> 
Sent by: [email-masked] 

25/10/02 17:16   

[...] 

        To:        [email-masked], [email-masked] 
        cc:         
        Subject:        [rddp] RE: [Tsvwg] I-D
ACTION:draft-williams-iwarp-ift-00.txt 

[...]

---

From: "Culley, Paul" <[email-masked]>
Date: Wed, 30 Oct 2002 21:11:47 -0600

embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Williams, Jim [mailto:[email-masked]]
Sent: Wednesday, October 30, 2002 4:16 PM
To: 'Caitlin Bestler'
Cc: [email-masked]; [email-masked]
Subject: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Monday, October 28, 2002 10:28 PM
> To: Williams, Jim
> Cc: [email-masked]; [email-masked]
> Subject: Re: [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt
>
> [...]
> 
> The elasticity buffer you are citing is an artifact of *one*
> type of implementation with one type of NIC / host
> interface.
> 
> Protocol decisions should still be valid even when DDP
> support is designed into the motherboard, or is part of a
> distributed cache system, etc.
> 
> The point is that each DDP segment is placeable as soon
> as it is received, but the flushing of that placement to
> host memory is subject to a great deal of implementation
> optimization (and hence variation). The only requirement is
> that it be placed *before* the following completion is
> reported to the consumer.
> 
> Because the mechanics of placement is so flexible, and so
> open to optimization, I do not think we should be making
> other protocol design decisions based on the artifacts of
> one way of interfacing the DDP engine and the host.

I agree with your last sentence.  I disagree that it applies
in this case.  Implementation concerns should not drive
protocol definitions if they are an artifact of one
type of implementation.  Implementation concerns
are an important driving factor if they are inherent
in the protocol itself and most reasonable implementations
of that protocol.

When a DDP packet is received, there are some fundamental
things that need to be done before the data can be placed.
These things need to be done independent of the type
of implementation.

  1.  The IP addresses and port numbers must be looked
      up in an associative array to locate the connection
      context.

  2.  The connection context must be referenced.

  3.  PAWS check must be done to determine is packet is valid,
      and sequence number must be checked to insure it is
      within the valid window.  This is done using information
      contained in the context.

  4.  The memory buffer keys must be checked for validity
      and permission, and the physical destination address
      determined.

Some of these involve round trips to large memory structures.
If the number of active connections is large, one can't
necessarily depend on caches.

Is it theoretically possible to do all this and still process
packets at a rate of one every 100ns?  Absolutely.  Is it 
more economical to do this than to add a reasonable amount
of buffer memory and process packets at a rate of one
every 500ns or 1000ns?  My experience would indicate not.
Not even close.

<prc> There are vendors of NICs that do not agree with you...  And if you can only process packets every 1000 ns, a 10Gigabit NIC could only sustain about 4 gigabits throughput, not a very good NIC.  (using 80%bw for large 1500 byte packets, 20%bw for small 100 byte packets)
<prc>

I believe the MPA draft as currently proposed adds considerable
complexity to the protocol, and is considerably more intrusive
on how TCP should be implemented.  It also is quite non-optimal
for implementation that do not attempt to do out of order
data placement.  It therefore requires significant justification
with respect to how it can enable more cost effective 
implementations.

<prc> In what way is MPA non-optimal for implementations that choose to do in-order?  The only things that come to mind are the segmentation alignment requirements at the transmitter, (weren't you a proponent of this type of alignment in prior framing drafts?) and a requirement to remove markers at the receiver.  Neither seems to be particularly difficult to do in an in-order NIC, but without these features, it is not possible to reliably do an out of order NIC.  Seems like we should enable as wide a range of NIC implementations as possible...
<prc>

I look forward to hearing more detailed arguments from MPA
proponents describing how MPA will enable more cost
effective implementations.

 - Jim