Subject: RE: [rddp] Let's discuss markers [Tunable Marker Intervals]

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 05 Dec 2002 01:18:02 -0500

At 06:45 PM 12/4/2002, Uri Elzur wrote:
>The goal should be "Optimize for HW and Allow SW". This is interesting for 
>hi-speed networking only. The vast majority of say 10GbE will be HW and SW 
>will not keep up. Adding options and sacrificing the no-negotiation-required 
>at connection setup, seems a high price to pay.

I have to disagree with this. While I expect hardware assisted
implementations to be required for full utilization of 10Gb, I do
not believe that such an environment dictates the protocol
requirement. As the Problem Statement goes to great length
to explain, RDMA is important at lower speeds as well.

In certain host environments such as embedded systems
with a single address space, STags and TOs can be used to
enumerate received segments in place, without mapping them
to their "address". This is easily accomplished in software on
such platforms. When markers intervene, the 18 or so "breaks"
in the received segment of a jumbo frame work heavily against
this.

Importantly, my proposal only affects the transmit side of
the hardware implementation. Its receive side can have
markers at whatever fixed interval it chooses. The software
peer's transmitter will of course need to insert them, as
plainly as the hardware peer's transmitter must leave them
out.

Tom.

>Again, pls note that CRC in SW place a high toll on SW. Adding markers is 
>not the killer at this point
>
>Uri
>
>-----Original Message-----
>From: Talpey, Thomas [mailto:[email-masked]]
>Sent: Tuesday, December 03, 2002 6:40 AM
>To: [email-masked]
>Cc: Giora Biran
>Subject: Re: [rddp] Let's discuss markers
>
>
>Vadim thanks for the insightful summary. I agree that
>Alignment and Markers are useful to distinguish.
>
>What if the marker interval were negotiable on each
>side of the connection? Software implementations
>would request zero (no markers) on their receive side,
>but "consenting adults" could agree otherwise?
>
>The obvious advantage for tuning the marker interval
>relative to the MSS, as well as enabling software
>implementations on highly optimized peers (e.g.
>embedded) would seem to far outweigh the addition
>of optional behavior.
>
>Tom.
>
>At 02:43 AM 12/3/2002, Vadim Makhervaks wrote:
>>
>>Sorry for previous note, please discard it.
>>
>>I would propose to separate Alignment and Markers.
>>
>>If one could assume a perfect Alignment, no Markers are needed.
>>
>>Markers are needed to handle a case when the TCP segment carrying DDP
>>header is lost,
>>and no alignment assumption can be made.
>>
>>I do not think that receiver is allowed to make any assumption regarding
>>alignment,
>>at the same time I think that Alignment suggestion (should and not must)
>>and Marker do help to make receiver implementation efficient (not
>>necessarily simple).
>>Therefore no negotiation process is needed.
>>
>>Let me describe my view on the receiver decision-flow:
>>
>>1. New TCP segment arrives.
>>
>>2. This segment passes all TCP validations.
>>
>>3. If the segment arrives in-order, then the location of DDP header is
>>known
>>(the length field of the previous DDP segment can be used for this
>>purpose).
>>Alignment would help in this case, because the receiver DDP segment can
>>be
>>directly placed to the memory.
>>Without alignment, if only part of DDP segment was received, this part
>>would need to be stored locally
>>in the RNIC memory.
>>
>>4. If the segment arrives out-of-order, we need Markers to find the
>>nearest
>>DDP header.
>>In out-of-order case, Markers allows reduce the memory resources of
>>RNIC.
>>Without Markers, to recover from segment lost, RNIC would need either
>>store
>>in all arriving TCP segments
>>in its internal buffers, till it won't receive the missing TCP segment,
>>and only then proceed DDP segments processing.
>>Markers limit pretty well the memory resources needed to recover from
>>TCP
>>segment lost:
>>MSS*number-of-holes, and number of supported holes is anyway limited by
>>TCP
>>implementation.
>>
>>5. Note that once DDP header is found, all following DDP segments are
>>treated as received in-order,
>>till the next hole.
>>
>>
>>To conclude: on my view Markers is not a first thing that I would wish
>>to
>>implement,
>>but by now I do not see any other solution for efficient recovery from
>>TCP
>>segment lost.
>>
>>Vadim
>>
>>
>>
>>|---------+---------------------------->
>>|         |           [email-masked]|
>>|         |           om               |
>>|         |           Sent by:         |
>>|         |           [email-masked]|
>>|         |           rg               |
>>|         |                            |
>>|         |                            |
>>|         |           03/12/02 08:26 AM|
>>|         |                            |
>>|---------+---------------------------->
>> 
>>>-----------------------------------------------------------------------
>>--------------------------------------------------------|
>>  |
>>|
>>  |       To:       [email-masked]
>>|
>>  |       cc:
>>|
>>  |       Subject:  [rddp] Let's discuss markers
>>|
>>  |
>>|
>>  |
>>|
>> 
>>>-----------------------------------------------------------------------
>>--------------------------------------------------------|
>>
>>
>>
>>Since an important technical issue underlying this debate
>>about not-quite-bufferless NICs is markers, let's take a
>>closer look at markers.  This entire message is with my WG
>>Chair hat off for the purpose of provoking discussion ...
>>
>>... so, for the sake of discussion, let's assume that we can
>>run a framing-modified TCP transmitter in aligned mode - the TCP
>>ULP framing draft that was ready to go experimental not that
>>long ago proposed to do this, so this is not an entirely
>>unreasonable assumption.
>>
>>Under this assumption, what's the purpose of markers?  MPA was
>>apparently designed under this assumption, and the MPA draft says:
>>
>>             7.4.1  Re-segmenting Middle boxes and non-conforming
>>senders
>>
>>Non-conforming senders sure sounds like a matter for connection setup
>>negotiation - get the other side to say "yes, I do aligned FPDUs"
>>on setup, and then close the connection in error if the receiver
>>ever detects that the sender broke this promise (e.g., as if a CRC
>>error not recoverable via retransmits occurred).
>>
>>That leaves re-segmenting middle boxes.  First of all, it appears
>>to be clear from list discussion that the designers of (and advocates
>>for) MPA view them as a rare or error case.  Even so, many statically
>>resegmenting middleboxes ought to be detectable via some sort of
>>"training sequence" test run at connection setup designed to cause
>>resegmentation - if resegmentation happens, disable framing before
>>sending any data that depends on it.
>>
>>This would appear to reduce the scenarios of concern to a subset
>>of a rare error case.  Given this, the impact of markers on the
>>operation of framing seems to be rather disproportionate -
>>e.g., in their current "every 512 bytes" form, they all but
>>exclude software implementations.
>>
>>Nonetheless, there's still a problem here.  What if TCP drops a packet
>>and at that precise point in the bytestream, a dynamic resegmenting
>>middlebox decides to kick in and screw up everything?  That should be
>>sufficiently rare that it may be ok to just close the connection ...
>>and tell the poor user to get a real network ... ;-).
>>
>>... but a whole bunch of data is going to come in before the retransmit
>>arrives that indicates the receiver's been fooled by the middlebox -
>>what
>>gets done with it?  One possibility is to process it on the assumption
>>that it's all aligned frames, and close the connection in error when
>>the retransmit shows up indicating that they're not, and the receiver
>>was fooled by what looked like frames in data (in most cases, errors
>>will happen very quickly, but the "frame in data" case has to work
>>correctly) ...
>>
>>... so what about the frames that were processed incorrectly between the
>>original drop and the retransmit?  Well, they could only have been
>>placed,
>>because the missing retransmit prevents delivery to the ULP (TCP
>>delivers
>>in order, and framing can't change that).  So, when the retransmit
>>arrives,
>>close the connection, and complete all the pending receives with errors
>>indicating that the buffers are invalid.  This has to be implemented
>>anyway to deal with a CRC error that can't be recovered via retransmits,
>>so it's almost "free" :-) ...
>>
>>... but what if the frames that were processed incorrectly affected
>>memory that wasn't supposed to be affected by any pending receive?  Any
>>memory that it could have modified has to be declared invalid - this is
>>somewhat worse than the CRC error as any exposed memory in the
>>protection
>>domain gets invalidated ... but I think I've seen folks headed towards
>>a design approach of being rather conservative in how much memory is
>>exposed when (e.g., to provide application assurances that the memory
>>from completed receives can't be overwritten).  This might not be an
>>important difference, but it is clearly fodder for discussion, and
>>reinforces the importance of precisely specifying protection domain
>>access control functionality.
>>
>>One more thing - anytime one comes up with an error mechanism that
>>closes a connection, denial of service becomes a possibility via
>>the attacker causing the error.  In this case it's a definite
>>possibility,
>>as a clever attacker could corrupt a TCP packet and then send
>>a fake one that looks like a retransmit that's been resegmented by a
>>middlebox, causing the connection to close.  This doesn't look like
>>a major increase in risk, as an attacker who can send the latter packet
>>can probably also send a TCP RST which will close the connection far
>>more effectively (yes, one could in theory do TCP RST filtering at
>>[some] routers, but in practice this is not done, and hence can't be
>>relied upon).  IPsec is needed to defend against this class of attacks.
>>
>>Ok, that should be more than enough to provoke discussion ...
>>... comments?
>>
>>Thanks,
>>--David
>>----------------------------------------------------
>>David L. Black, Senior Technologist
>>EMC Corporation, 176 South St., Hopkinton, MA  01748
>>+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
>>[email-masked]        Mobile: +1 (978) 394-7754
>>----------------------------------------------------
>>_______________________________________________
>>rddp mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/rddp
>>
>>
>>
>>
>>_______________________________________________
>>rddp mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/rddp
>_______________________________________________
>rddp mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Dec 2002 01:06:20 -0600

On 12/5/02, Talpey, Thomas wrote:

>>The goal should be "Optimize for HW and Allow SW". This
>>is interesting for hi-speed networking only. The vast
>>majority of say 10GbE will be HW and SW will not keep up.
>>Adding options and sacrificing the
>>no-negotiation-required at connection setup, seems a high
>>price to pay.
>
>I have to disagree with this. While I expect hardware
>assisted implementations to be required for full
>utilization of 10Gb, I do not believe that such an
>environment dictates the protocol requirement. As the
>Problem Statement goes to great length to explain, RDMA is
>important at lower speeds as well.
>
>In certain host environments such as embedded systems with
>a single address space, STags and TOs can be used to
>to their "address". This is easily accomplished in
>software on such platforms. When markers intervene, the 18
>or so "breaks" in the received segment of a jumbo frame
>work heavily against this.
>
With mandatory markers many (or perhaps most) ULPs will be
able to initiate DDP/MPA mode immediately after TCP
connection establishment. No knowledge of the peer is
required.

For ULPs that exchange some private data first, negotiating
marker intervals sounds like a great idea. But would it ever
make sense to add a preliminary exchange solely for that
purpose?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp