Subject: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

From: "Williams, Jim" <[email-masked]>
Date: Mon, 4 Nov 2002 11:50:41 -0500

> > -----Original Message-----
> > From: Culley, Paul [mailto:[email-masked]]
> > Sent: Wednesday, October 30, 2002 10:12 PM
> > 
> > Is it theoretically possible to do all this and still process
> > packets at a rate of one every 100ns?  Absolutely.  Is it 
> > more economical to do this than to add a reasonable amount
> > of buffer memory and process packets at a rate of one
> > every 500ns or 1000ns?  My experience would indicate not.
> > Not even close.
> 
> <prc> There are vendors of NICs that do not agree with you...

I am sure you are right.  And those NIC vendors seem to 
have prevailed behind the closed doors of the RDMA consortium.
I am doing my best, though, to draw out their arguments
within this forum.
 
> And if you can only process packets every 1000 ns, a 
> 10Gigabit NIC could only sustain about 4 gigabits throughput, 
> not a very good NIC.  (using 80%bw for large 1500 byte 
> packets, 20%bw for small 100 byte packets)
> <prc>

Whether or not it's a good NIC depends on the cost performance
point that it hits and how that corresponds to customer
requirements.  However the numbers I picked are arbitrary,
and the point holds for different numbers.

> > I believe the MPA draft as currently proposed adds considerable
> > complexity to the protocol, and is considerably more intrusive
> > on how TCP should be implemented.  It also is quite non-optimal
> > for implementation that do not attempt to do out of order
> > data placement.  It therefore requires significant justification
> > with respect to how it can enable more cost effective 
> > implementations.
>  
> <prc> In what way is MPA non-optimal for implementations that 
> choose to do in-order?  The only things that come to mind are 
> the segmentation alignment requirements at the transmitter, 
> (weren't you a proponent of this type of alignment in prior
> framing drafts?) and a requirement to remove markers at the 
> receiver.

If you accusing me of changing my mind, I may be guilty.
I have spent the last couple years architecting and designing
accelerated NICs for RDMA and similar applications.  So
possibly I learned something in the process. :)

However, I regard alignment as one of the good things
about the MPA proposal.  It is perhaps not justified
based only on the need for out of order data placement for
reasons I have stated, but I think alignment is generally
a good thing, and has other benefits as well in terms
of overall efficiency of processing.

I do have concerns about requiring alignment, or encouraging
receiver designs become grossly inefficient when transmitters
fail to align, or when middle boxes trash alignment.  But 
I still think opportunistic alignment may be a good thing
if an appropriate mechanism can be constructed.

Having said that, though, I believe the MPA mechanism for
alignment is ugly, complicated, and an abuse of layering.
Admittedly this is a subjective reaction, but I don't
think it's irrational.  Clean layering is best done
by taking the PDU of the next upper layer, leaving it
unchanged and adding a header, and maybe a trailer.
Segmentation and reassembly are operations that have
significant associated complexity for both hardware
and software implementations, and the number of times
and the number of layers where this must be done should
be kept to an absolute minimum.  Markers effectively
segments the payload into pieces and wedges layer 
specific data between each of these pieces.

I think (and hope) that no past proposals from me
have involved slicing and dicing the payload as
a mechanism to do alignment.

I also have concerns about the IETF process for getting
an alignment solution approved.  I think going this
route will significantly slow down the process for
RDDP overall.  It is much preferable to define
a simple more conservative solution, get it approved
quickly and gain some deployment experience.
Future optimization based on such experience
are preferable to adding every bell and whistle up
front.  Especially when we are talking about
a large heavy bell that will be viewed in years
to come as most unfortunate baggage.

 - Jim
 
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Wed, 6 Nov 2002 10:14:41 +0200

I probably late on this discussion, but I just recently went thru the
differences between both drafts.
It seems (to me) that the changes proposed by Jim are not tightly coupled,
and I would propose to discuss them separately:

ddp header CRC: Sounds as a reasonable approach to me. I can imagine why
such CRC would help to RNIC implementation. On the other side, I don't see
a damage of adding such CRC to the DDP segment,  beside another word of
overhead per DDP segment. I also think that such CRC should protect the DDP
header only, and RDMA extensions (for Read Request and Terminate) should be
considered as a payload. Anyway this disclaimer is out-of-the-scope of MPA
definition, and probably is not relevant too much for this discussion.

padding: It's definitely easier to implement CRC logic when everything is
word aligned. It's not too much overhead - upto 3 bytes per DDP segment. I
also agree that this is not a MUST requirement, and RNIC vendor should be
able to handle not-aligned CRC generation/validation.

packing: I think that this is an important feature to allow packing of
several DDP segments to the same TCP segment, unless it makes Rx
implementation much more complex. I know that this feature was recently
removed from the MPA specification, and not sure that understand why is
that.

markers: We may argue whether this feature is helpful or not, and both side
could give plenty examples showing cons and pros of this feature. I think
that if this feature could be used by some RNIC vendors to make a
better/different RNIC we should leave it. If some vendor thinks that this
is not helpful too much, then just don't use it. Generate markers on Tx,
remove them on Rx, I don't see too much overhead even for software
implementation of such feature. By the way, I think that the iWARP
definition is intended to allow efficient hardware implementation, and not
oriented only for software implementations using existing technology.

segmentation/alignment: It's clear that some NIC vendors (those use the
markers option) may take advantage of having this feature, and thus may
want to align DDP segments to the TCP segments (I don't think that packing
violates this feature). However, since ,due to resegmenting box, RNIC needs
to know how to handle not aligned DDP segments, the transmitter should not
be required to align DDP and TCP segments, it should be recommended but not
required.

So to summarize I would propose to add changes to the recent MPA
specification to allow both implementation approaches:

Add ddp header CRC. Allow packing. Leave markers as defined today. Make
segmentation/alignment optional. Not sure about padding, but this does not
seem to be a critical issue ...

Vadim

|---------+---------------------------->
|         |           "Williams, Jim"  |
|         |           <Jim.Williams@emu|
|         |           lex.com>         |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           04/11/02 06:50 PM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       [email-masked], [email-masked]                                                                                 |
  |       cc:                                                                                                                     |
  |       Subject:  [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt                                                          |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

 - Jim

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Culley, Paul" <[email-masked]>
Date: Wed, 6 Nov 2002 13:40:37 -0600

You have done an excellent analysis of the differences and issues at hand.  Here are a few comments, embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Vadim Makhervaks [mailto:[email-masked]]
Sent: Wednesday, November 06, 2002 2:15 AM
To: Williams, Jim; Culley, Paul
Cc: [email-masked]; [email-masked]; Giora Biran; Julian Satran
Subject: Re: [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

<prc> As discussed in prior emails, the MPA authors (after much discussion) felt that this is not necessary.  
  You need a separate header CRC if designing a "Flow through NIC":
	* But Need for elasticity buffer anyway makes "flow through" an
	  un-needed extra complexity.
	* But NIC designers want to check L2 CRC and IP checksum first
	  anyway, requires whole packet.
  You need a separate header CRC if Placing partial FPDUs (can place 1st
  part without whole FPDU):
	* As long as alignment is correct, this is not needed.
	  o Alignment is Expected to be the usual case in the data center,
	    at least, and common elsewhere.
	  o If lost alignment is an uncommon case, extra wire overhead and
	    extra complexity of saving headers, while placing data doesn't
	    seem worth the small buffer saving
<prc>

<prc> Padding also eliminates the problem of how to deal with markers that fall in the middle of the CRC.  If everything is aligned on 4 byte breaks, then markers also end up on the same breaks (tied into the 512 byte marker interval).
<prc>

<prc> This created some real discussion in writing the MPA spec.  The upshot was that NIC vendors did not like to have to process multiple FPDUs in a TCP segment as a common case.  It was perceived to add complexity at both the receiver and sender.  In addition, reasonably likely data streams would not often have multiple small messages (converted to FPDUs) close enough together in time to make packing a big wire savings.  I was personally of the opinion that we should allow or even require packing, but was in a minority opinion ;-(.
<prc>

<prc> Good argument, and right about the hardware...

<prc> That is why we used the word "SHOULD" in the MPA doc, rather than a "MUST".  The intent was that this be a strong recommendation.
<prc>

Vadim

  |                                                                                                                               |

 - Jim

---

From: "Williams, Jim" <[email-masked]>
Date: Wed, 6 Nov 2002 16:07:31 -0500

> You have done an excellent analysis of the differences and 
> issues at hand. 

I would echo that sentiment.

> and I would propose to discuss them separately:

Generally true.  There is some coupling, though, which I note.

> <prc> As discussed in prior emails, the MPA authors (after 
> much discussion) felt that this is not necessary.  

I agree that a single CRC makes sense if alignment is always
guaranteed.  The IFT proposal does not guarantee alignment,
therefore two CRCs make sense.  SCTP does guarantee alignment,
so its single CRC makes sense.

Standard TCP does not provide alignment.  TSVWG is
considering an experimental variant of TCP that 
does provide alignment.  But the RDDP work group 
charter is clear that RDDP must be standardized on 
standard TCP.  Hence the motivation for the IFT proposal.

Also, the benefit of the second CRC for the unaligned
case seems much greater than the cost of an extra
CRC in the aligned case.

> word aligned.

This is not an issue for which I have a whole lot of energy.
But my experience and those of the other hardware designers
I have worked with directly contradict the above.
Unaligned CRCs are trivial to implement.  Logic necessary
to insert padding is not trivial.  Either way works.
My vote is no padding, but I'm not terribly concerned
about conceding this point if a majority feel otherwise.

> It's not too much overhead - upto 3 bytes per 
> DDP segment. I
> <prc> Padding also eliminates the problem of how to deal with 
> markers that fall in the middle of the CRC.  If everything is 
> aligned on 4 byte breaks, then markers also end up on the 
> same breaks (tied into the 512 byte marker interval).
> <prc>

True.  IFT does not propose markers, so this point is moot
for IFT.

> <prc> [...] I was personally of the opinion that we should 
> allow or even require packing, but was in a minority opinion ;-(.
> <prc>

I suspect that packing may ultimately be required by the IETF.
Either that, or some detailed explaining
as to why network traffic patterns are not negatively affected.

> could give plenty examples showing cons and pros of this 
> feature.

I volunteer for the cons. :)

There are some basic architectural issues needing
to be resolved as to whether it is permissible for 
DDP to do out of order placement when running over
an ordered transport.  I personally am not religious
about this either way.  But unless this can be resolved,
markers make no sense whatsoever. 

I do believe the benefits of out of order placement 
are sufficiently questionable that the simplicity
of the in order model should not be compromised
if at all avoidable.

I am a bit religious in feeling that markers are
an ugly, complicated, and technically embarrassing
way to solve the problem of alignment.

It has been clearly stated that primary problem
the RDDP group needs to solve is layering RDDP
on standard TCP, and layering on an experimental
aligned version of TCP is secondary.  Markers
appears to be an effort to optimize for 
the secondary case at the expense of the primary
case.

I believe that alignment can be done without
markers and without complicating or burdening
the primary objective of standard TCP.  I will
attempt to explore this in future drafts.

draft-ietf-tsvwg-tcp-ulp-frame-01.txt (expired),
is an example of such an approach.  This type
of approach is far more consistent with the 
RDDP group charter of treating standard TCP
as the primary objective, and an experimental
aligning version of TCP as a secondary objective.

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Thu, 7 Nov 2002 00:52:10 +0200

I'll try to put some pros for markers.

First of all I need to admit, that I'm not fun of markers, it is not an
easy thing to implement, but right now I do not see any other solution for
the problem that I'm going to present below.

First few words about host interface, and it's ability to handle bursts
from the wire. I believe that DDP concept is 10gb oriented, and thus a host
interface that should be considered is not PCI-X, but QDR-PCIX, RapidIO,
direct interface to memory controller, etc. It's not clear that such
interface would have any problem to handle inbound traffic on wire-speed.
It should be mostly write operations on the bus, and assuming efficient
bridge/MC implementation, we should not have big problems with bursts of
writes.

Even assuming that RNIC vendor would decide to pass the received stream via
on-board/on-chip memory before placing to the host memory, markers still
would be very helpful. I'll give just one example to emphasize this:
Without markers the location of the next DDP segment is identified by the
Length field of the header of previous DDP segment. If DDP header was lost,
then without use of markers, RNIC vendor would need to wait for receiving
this header, in order to place the data to the destination buffers on the
host. However, if markers are supplied by transmitter, receiver may several
choices: a) use a fast-path (cut-thru) and place received segments directly
to the destination buffers; b) pass received stream thru the memory, but
still be able to place the payload to the destination buffers out-of-order;
c) use a classic reassembly buffers, and place the data to the destination
buffers in-order.

So why should we force RNIC vendor to choose option c), and not leave this
as another feature open for competition?

Vadim

|         |           06/11/02 11:07 PM|
  |                                                                                                                               |
  |       To:       "'Culley, Paul'" <[email-masked]>, Vadim Makhervaks/Haifa/IBM@IBMIL                                       |
  |       cc:       [email-masked], [email-masked], Giora Biran/Haifa/IBM@IBMIL, Julian Satran/Haifa/IBM@IBMIL                     |
  |       Subject:  [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt                                                          |
  |                                                                                                                               |
  |                                                                                                                               |

---

From: "Giora Biran" <[email-masked]>
Date: Fri, 8 Nov 2002 18:41:29 +0200

Let me add some comments regarding the complexity in a hardware
implementation as I see it:

- It seems that the checking of the CRC is separate from the checking of
the TCP/IP checksum and requires additional buffering. It is true that in
the case of a single (and full) DDP segment per TCP segment both checks can
be done on the same time using the same buffer. But, it doesn't work in the
case of resegmentation middle-box.
The bottom line is that CRC on the DDP header makes the hardware
implementation more efficient by saving buffer size and simplified the flow
of direct placement.

- Allowing resegmentation middle-box caused much more complicated cases
then the packing itself. All the cases even those which seems to be rare,
have to be handle by the RNIC (preferably hardware). Therefore I do not see
a big impact for the implementation to allow packing. Packing might add
complexity on the transmit side, but the receive side have to be capable to
handle packing (caused by a resegmentation middle-box.

Giora

Vadim Makhervaks   07/11/2002 00:52

To:    "Williams, Jim" <[email-masked]>
cc:    Giora Biran/Haifa/IBM@IBMIL, Julian Satran/Haifa/IBM@IBMIL,
       "'Culley, Paul'" <[email-masked]>, [email-masked],
       [email-masked], [email-masked]
From:  Vadim Makhervaks/Haifa/IBM@IBMIL
Subject:    Re: [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt
       (Document link: Giora Biran)

Vadim

                      "Williams, Jim"                                                                                             
                      <Jim.Williams@emu        To:       "'Culley, Paul'" <[email-masked]>, Vadim Makhervaks/Haifa/IBM@IBMIL  
                      lex.com>                 cc:       [email-masked], [email-masked], Giora Biran/Haifa/IBM@IBMIL, Julian       
                      Sent by:                  Satran/Haifa/IBM@IBMIL                                                            
                      [email-masked]        Subject:  [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt                     
                      rg                                                                                                          

                      06/11/02 11:07 PM

---

From: "Uri Elzur" <[email-masked]>
Date: Mon, 18 Nov 2002 15:59:12 -0800

>on 10/28/02, Jim Williams wrote:
>
>I believe an important objective in the protocol design
>should be avoiding the need to re-assemble split
>DDP segments.  This is an important motivating factor for
>a separate header CRC.
>
>The idea is that the first part of the split DDP segment can
>be placed without the need to reassemble it.  When the
>second part of the split segment arrives, payload CRC
>computation resumes where it left off on the first part.
>All that needs to be saved in the interim are the header
>info and the CRC seed.

One key issue that is glossed over, is how does the receiver find the beginning of the first or second part of that split DDP segment? In presence of middle box or TCP sender that provides no alignment to DDP, this can be any place in the received TCP segment. The draft suggests that IFT targets in-order only ("Assuming in order processing of received TCP data, 
   this is fully sufficient to define the DDP PDU boundaries"). How does it deal with out-of-order? Does it suggest to drop it? Buffer it? Has it looked into the performance implication of doing that on a high speed network? Several people on the list commented out-of-order is an expected behavior of the network

Now under the in-order only assumption, the argument for a separate "header CRC" is much weaker. Note also, that If the receiver gets the first segment of the DDP segment, there is no guarantee the whole "DDP header" is present in one TCP segment. In that case the receiver has to buffer this fraction

hence, the IFT algorithm requires the receiver to: 

a. find the beginning of the IFT in the received TCP segment and 
b. Dealing with the following cases: a partial "DDP header", complete DDP header reception, complete DDP header with part of the data and complete DDP header with complete data.
c. run CRC on a selected number of bytes located anywhere in that segment  
d. Run a separate CRC on the data portion of the segment
e. Keep state for the case partial segment has been received

This requires much more flexible HW (assuming the goal is to perform CRC in HW) than what MPA requires, while IFT offers a subset of the functionality. Another assumption IFT makes is that the NIC uses a buffer anyhow. In that case then, why not do away with its separate header CRC and use one CRC for the whole message after its boundaries has  been found? This simplifies the hardware and exposes the same exact delivery semantics to DDP as the two separate CRC provides.

Uri
-----Original Message-----
From: Williams, Jim [mailto:[email-masked]]
Sent: Monday, October 28, 2002 9:27 AM
To: 'Caitlin Bestler'; [email-masked]; [email-masked]
Subject: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, October 25, 2002 12:10 PM
> To: [email-masked]; [email-masked]
> Cc: Williams, Jim
> Subject: Re: [rddp][Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt
> 
> 
> On 10/25/02, Williams, Jim wrote:
> 
> 
> >
> >This is really the critical core issue we need to discuss 
> >in more depth.
> >
> >My views tend to be as follows.
> >
> >  A.  I think you are underestimating the amount of memory
> >      required for elasticity buffering, and overestimating
> >      the memory required for out of order reassembly.
> >
> >  B.  Once the cost is incurred of adding external (off 
> >       ASIC) memory of sufficient bandwidth for elasticity, 
> >       the incremental cost of additional megabytes is 
> >       quite low.
> 
> True. But particularly at the lower bit rates, on-chip
> memory could easily be adequate for re-assembling split DDP
> segments and buffering host bus access.

I believe an important objective in the protocol design
should be avoiding the need to re-assemble split
DDP segments.  This is an important motivating factor for
a separate header CRC.

The idea is that the first part of the split DDP segment can
be placed without the need to reassemble it.  When the
second part of the split segment arrives, payload CRC
computation resumes where it left off on the first part.
All that needs to be saved in the interim are the header
info and the CRC seed.

> Also consider that during the life of the protocol, the
> scope of on-chip memory is only going to improve.

So will the required memory as bandwidths increase.
It's not clear which will increase faster, but I think
I put my money on network bandwidths.

> >  D.  A product design needs to set performance goals at 
> >       two (at least) different traffic mixes, large
> >      messages, and small messages.  An accelerated NIC
> >      which is able to support line rate with small 
> >       messages is seriously over designed and non cost 
> >       effective.
> >
> >      Supporting line rate at large and medium size 
> >       messages is reasonable, and requires significantly 
> >       lower peak packet processing rates.
> >
> >      The result of this is that for small messages, one
> >      needs to rely on elasticity buffering and end to 
> >      end flow control to avoid large scale dropping of
> >      incoming packets.  With insufficient buffering,
> >      the supported rate for small messages will be 
> >      terrible due to constant dropped packets and 
> >      retransmission timeouts. 
> >
> 
> That is a characteristic of a general TCP optimizer. I do
> not believe that it applies to framing DDP segments. The
> point of the MPA proposal is that it allows each DDP segment
> to be processed on it own as it arrives, and further that in
> virtually every case there will be exactly one DDP segment
> per TCP segment / IP datagram / Ethernet Frame.
>
> Under that scenario the size of the tagged or untagged
> message is totally irrelevant, each segment is placed
> as it arrives.

The size is relevant because it is inversely proportional
to the segment arrival rate. On a 10Gb link, 100 byte
DDP segments can arrive every 100ns (approx).
1000 byte DDP segments can arrive every microsecond.
Doing a direct placement every 100ns is
impractical.  It's probably impossible with
current host bus designs.

Therefore the segments must be held in the 
elasticity buffer, and end to end flow control
must be used to avoid overflowing the buffer.
It's this requirement that drives the size of
the needed buffering.

> [...]
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
>

---

From: "Uri Elzur" <[email-masked]>
Date: Mon, 18 Nov 2002 15:59:15 -0800

- A separate CRC on the header adds complexity without any noticeable performance gains as the scenario IFT supports is in-order only. (see my other mail)

-  IFT offers no support for the detection of a middle box, while MPA allows middle-box detection. The implementation has lots of wiggle room with how it wants to deal with it.

thx

Uri

-----Original Message-----
From: Giora Biran [mailto:[email-masked]]
Sent: Friday, November 08, 2002 8:41 AM
To: Vadim Makhervaks
Cc: Williams, Jim; Julian Satran; 'Culley, Paul'; [email-masked];
[email-masked]; [email-masked]
Subject: Re: [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

Giora

Vadim

                      "Williams, Jim"                                                                                             

                      06/11/02 11:07 PM

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 18 Nov 2002 22:08:11 -0500

> From: Uri Elzur [mailto:[email-masked]]
> Sent: Monday, November 18, 2002 6:59 PM
> To: Williams, Jim; Caitlin Bestler; [email-masked]; [email-masked]
> Subject: RE: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt
> 
> 
> One key issue that is glossed over, is how does the receiver 
> find the beginning of the first or second part of that split 
> DDP segment? In presence of middle box or TCP sender that 
> provides no alignment to DDP, this can be any place in the 
> received TCP segment. The draft suggests that IFT targets 
> in-order only ("Assuming in order processing of received TCP data, 
>    this is fully sufficient to define the DDP PDU 
> boundaries"). How does it deal with out-of-order? Does it 
> suggest to drop it? Buffer it?

IFT does not need to deal with out of order segments because
TCP delivers them in order.  The combined TCP/IFT implementation
of course must buffer out of order segments until the gap 
is filled.  However the vast majority of segments arrive in
order.

> b. Dealing with the following cases: a partial "DDP header", 
> complete DDP header reception, complete DDP header with part 
> of the data and complete DDP header with complete data.

This sounds about right, and is not problematic.

MPA assumes alignment.  With guaranteed alignment, a single
CRC makes some sense.  IFT does not assume alignment (as
standard TCP doesn'r provide it) so a separate header CRC
makes sense.
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg