Subject: Re: [rddp] Let's discuss markers

From: "Vadim Makhervaks" <[email-masked]>
Date: Tue, 3 Dec 2002 09:02:46 +0200

I would propose to discuss Markers and Alighment separately.

Assumption of perfect alignment  makes Markers redundent. If each received
TCP segment would start with DDP header, no markers are needed.

Markers are needed to help to recover in case when alignment is lost. Even
more, we need to have not-aligned segments together with loosing the packet
that carries DDP header.

There are two reasons why DDP segment can be not aligned to the TCP
segment: resegmenting middle-box and non-conforming senders. I do not think
that resegmenting  middle-box and non-conforming senders should be treated
as an error-scenario. They still may cause to the performance degradation.

Let me present my view of possible receiver behavior.

First - No negotiation between sender and receiver is needed.

Receiver has two

|---------+---------------------------->
|         |           [email-masked]|
|         |           om               |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           03/12/02 08:26 AM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       [email-masked]                                                                                                 |
  |       cc:                                                                                                                     |
  |       Subject:  [rddp] Let's discuss markers                                                                                  |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

Since an important technical issue underlying this debate
about not-quite-bufferless NICs is markers, let's take a
closer look at markers.  This entire message is with my WG
Chair hat off for the purpose of provoking discussion ...

... so, for the sake of discussion, let's assume that we can
run a framing-modified TCP transmitter in aligned mode - the TCP
ULP framing draft that was ready to go experimental not that
long ago proposed to do this, so this is not an entirely
unreasonable assumption.

Under this assumption, what's the purpose of markers?  MPA was
apparently designed under this assumption, and the MPA draft says:

             7.4.1  Re-segmenting Middle boxes and non-conforming senders

Non-conforming senders sure sounds like a matter for connection setup
negotiation - get the other side to say "yes, I do aligned FPDUs"
on setup, and then close the connection in error if the receiver
ever detects that the sender broke this promise (e.g., as if a CRC
error not recoverable via retransmits occurred).

That leaves re-segmenting middle boxes.  First of all, it appears
to be clear from list discussion that the designers of (and advocates
for) MPA view them as a rare or error case.  Even so, many statically
resegmenting middleboxes ought to be detectable via some sort of
"training sequence" test run at connection setup designed to cause
resegmentation - if resegmentation happens, disable framing before
sending any data that depends on it.

This would appear to reduce the scenarios of concern to a subset
of a rare error case.  Given this, the impact of markers on the
operation of framing seems to be rather disproportionate -
e.g., in their current "every 512 bytes" form, they all but
exclude software implementations.

Nonetheless, there's still a problem here.  What if TCP drops a packet
and at that precise point in the bytestream, a dynamic resegmenting
middlebox decides to kick in and screw up everything?  That should be
sufficiently rare that it may be ok to just close the connection ...
and tell the poor user to get a real network ... ;-).

... but a whole bunch of data is going to come in before the retransmit
arrives that indicates the receiver's been fooled by the middlebox - what
gets done with it?  One possibility is to process it on the assumption
that it's all aligned frames, and close the connection in error when
the retransmit shows up indicating that they're not, and the receiver
was fooled by what looked like frames in data (in most cases, errors
will happen very quickly, but the "frame in data" case has to work
correctly) ...

... so what about the frames that were processed incorrectly between the
original drop and the retransmit?  Well, they could only have been placed,
because the missing retransmit prevents delivery to the ULP (TCP delivers
in order, and framing can't change that).  So, when the retransmit arrives,
close the connection, and complete all the pending receives with errors
indicating that the buffers are invalid.  This has to be implemented
anyway to deal with a CRC error that can't be recovered via retransmits,
so it's almost "free" :-) ...

... but what if the frames that were processed incorrectly affected
memory that wasn't supposed to be affected by any pending receive?  Any
memory that it could have modified has to be declared invalid - this is
somewhat worse than the CRC error as any exposed memory in the protection
domain gets invalidated ... but I think I've seen folks headed towards
a design approach of being rather conservative in how much memory is
exposed when (e.g., to provide application assurances that the memory
from completed receives can't be overwritten).  This might not be an
important difference, but it is clearly fodder for discussion, and
reinforces the importance of precisely specifying protection domain
access control functionality.

One more thing - anytime one comes up with an error mechanism that
closes a connection, denial of service becomes a possibility via
the attacker causing the error.  In this case it's a definite possibility,
as a clever attacker could corrupt a TCP packet and then send
a fake one that looks like a retransmit that's been resegmented by a
middlebox, causing the connection to close.  This doesn't look like
a major increase in risk, as an attacker who can send the latter packet
can probably also send a TCP RST which will close the connection far
more effectively (yes, one could in theory do TCP RST filtering at
[some] routers, but in practice this is not done, and hence can't be
relied upon).  IPsec is needed to defend against this class of attacks.

Ok, that should be more than enough to provoke discussion ...
... comments?

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Culley, Paul" <[email-masked]>
Date: Tue, 3 Dec 2002 07:25:27 -0600

The main reason we chose markers was to properly identify the start of out of order segments.  The perception was that using a fixed pattern, with a CRC or even only with the CRC to identify a properly framed segment involved the risk that an improperly framed segment MIGHT be detected as properly framed.  This risk was very small (10E-24 or so) but concerns were raised that the user data might contain sequences (even deliberately) that could cause such confusion.  A deliberate case was when the data stream contained a packet dump of another data stream...

Also, we really do want to continue operation despite loss of framing;  This can happen on MTU changes and other events that we don't want to have the ULP doing retries for.

More embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, December 03, 2002 12:26 AM
To: [email-masked]
Subject: [rddp] Let's discuss markers

<prc> that assumes that you are willing to stop using anything framing depends on (DDP).

<prc> Might be good enough if you could be sure to detect it (see above)...

<prc> For DDP, you cannot safely "process it on the assumption that it's all aligned frames" since that could place data where it was not wanted, particularly if the DDP header is misidentified.

<prc> While this sounds ok assuming that the DDP user had only opened up the exact memory regions allowed at that time, as indicated in other threads, that seemed to be too restrictive an operating model.  We want to avoid protocol created data contamination for those apps that are willing to accept a more trusted remote peer.

<prc> see above

Thanks,
--David

---

From: "Williams, Jim" <[email-masked]>
Date: Tue, 3 Dec 2002 10:20:24 -0500

A lot of good thoughts, David.

A proposal that provides alignment without using
markers is an appealing compromise.  It allows
out of order placement and experimentation with
bufferless (not-quite) NICs.  At the same time
it creates no significant overhead for more 
conventional designs.

I have for a while been toying with writing
up a proposal for such a scheme, but have
been scared off by the political implications.

It seems there are two groups with fiercely
held convictions.  One that MPA should not 
be modified, and the other that TCP should
not be modified.  Clearly both groups would
oppose such a proposal.  So not modifying
TCP seems like the preferable of the
alternatives that actually stands a chance.

Is my assessment of the situation correct?
Some guidance with your WG chair hat back
on might be helpful here.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: David Borman <[email-masked]>
Date: Tue, 3 Dec 2002 10:44:32 -0600 (CST)

Hi,

I've been thinking about the marker issue with TCP.  My understanding
of the issue is that it boils down to this question:

	If there is a hole in the TCP packet stream, is it possible
	to identify later DDP headers in the stream so that their
	data may be placed into the correct ULP buffers rather than
	forcing the NIC to buffer up the data, waiting for the missing
	TCP packet?

The first observation that I'll make is that we are dealing with
an error case, not the normal case.  That is important to remember.
In the normal case, while processing the TCP data stream in order
you can reliably identify all DDP headers.  No additional framing
or markers are needed.  Now, in the error case, we really don't need
to reliably identify all DDP headers, we are just interested in
identifying at least one DDP header.  That's where the interesting
data is.  TCP already has a mechanism for identifying interesting
points in the data stream, called the Urgent Pointer.

If each TCP packet that was sent contained an Urgent pointer indicating
where the DDP header is located, then when the out-of-order packets
arrived the Urgent pointer could be used to identify if and/or where
a DDP header begins in that chunk of data.  Then simply verifying the
CRC should suffice as a sanity check to make sure it is a DDP header.
You shouldn't have to worry about the data containing embedded DDP
headers (like a file that contains a packet dump from a DDP session...)
because the Urgent pointer wouldn't ever be set to point to that data.

So, let me answer a few questions that I envision people asking
about this:

1) Urgent mode?  Doesn't that mean you are supposed to throw away
   all the data up to the Urgent pointer?

	No.  It is totally up to the application to decide what
	it means to have an Urgent pointer, and what is to be done
	with the data before and after the Urgent pointer.  All
	the data is still presented in sequence to the ULP, the
	only difference is that TCP first indicates to the ULP that
	an urgent pointer has been received, and then it also
	indicates to the ULP when it has read data past the
	Urgent pointer.  The ULP can totally ignore received
	information about the Urgent pointer if it wants to.

2) But the Urgent pointer is not a record marker...

	I'm not saying to try to use it as a record marker.  In
	normal in-line processing the Urgent pointer won't matter
	and can be effectivly ignored by the ULP, because the
	boundries of DDP headers are well defined.

	The Urgent pointer allows a way to indicate an interesting
	point in the data stream, where "interesting" is defined by
	the ULP.  I'd consider the start of a DDP header as qualifying
	as interesting data.

3) What about middle boxes?

	The Urgent pointer is part of the TCP spec.  A middle box
	should maintain any Urgent pointers in the TCP stream; if
	it doesn't, then the box is broken.

4) What if the CRC check fails?

	No matter how it is being done, we're talking about optimizing
	the processing for an error case.  So if something goes wrong
	while trying to do direct placement of out-of-order data, such
	as the CRC failing, then the optimization should be aborted,
	and just wait until the missing data arrives and then process
	everything in sequence.

		-David Borman, [email-masked]
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: David Borman <[email-masked]>
Date: Tue, 3 Dec 2002 15:00:36 -0600 (CST)

Jim,

> From: "Williams, Jim" <[email-masked]>
> Subject: RE: [rddp] Let's discuss markers
> Date: Tue, 3 Dec 2002 13:26:31 -0500 
>
> This was discussed for iSCSI.
>
> See the following:
>    http://www.pdl.cmu.edu/mailinglists/ips/mail/msg02410.html

I found lots of discussion, but I couldn't find the actual
proposal... :-(

> Looks like modifying TCP was off the table at that point.
> If proposed TCP modifications can now be considered, perhaps
> this is worth a second look.

Strictly speaking, DDP is the layer directly above TCP, so it would
not be hard to argue that any processing of out-of-order TCP segments
is a violation of TCP in-order delivery, since though the data is
not being deliverd to the ULP above DDP, it is being delivered to
the DDP layer.  So from a strict sense of not changing TCP, even the
DDP layer would not be allowed to deal with any out-of-order TCP packets.

However, if there is going to be any processing of out-of-order
TCP packets, then any method for doing direct placment of data from
out-of-order TCP packets requires modification to the TCP processing
code.  It doesn't matter whether the code is using Markers, the TCP
Urgent pointer, or any other heuristic to find the DDP header.  You
are still crossing the boundry and modifying the TCP processing code.

But within that context, I'm not suggesting changing the definition of
the TCP Urgent Pointer.  I'm saying that the Urgent pointer should be
able to be used as-is to provide information about where a DDP header
may be located in out-of-order TCP segments.

I see getting agreement from the ADs or IESG to allow out-of-order
TCP packets to be processed as a much bigger issue than using the
TCP Urgent pointer as a hint in that processing.  So if you have
the former, then the latter shouldn't be a big deal.

Remember, TCP does not define the semantics of processing data
relative to the Urgent pointer.  That is left up to the ULP,
in this case that would be the DDP layer.  TCP provides a
mechanism for the application to mark a point in the data stream
as "interesting", it is up to the application to define what
"interesting" means and how to process the data.

---

From: "Williams, Jim" <[email-masked]>
Date: Tue, 3 Dec 2002 16:41:06 -0500

Try the following:

http://www.watersprings.org/pub/id/draft-wakeley-iscsi-msgbndry-00.txt

> From: David Borman [mailto:[email-masked]]
> Sent: Tuesday, December 03, 2002 4:01 PM
> To: [email-masked]; Williams, Jim; [email-masked]

---

From: Parag Bhide <[email-masked]>
Date: Tue, 3 Dec 2002 13:44:54 -0800

URG pointer seems like a good idea except,

1. (I agree with Somesh) TCP receives keeps only one URG pointer value
("latest as defined by RFC 793). While what we need is URG pointer
associated with each packet.

2. When DDP segments are carried in different packets (alignment problem at
the sender or middle-box) it could result in TCP packets without any URG
pointer. These are the packet that will carry the second half of the DDP
segment. (Note that URG pointer can point only to future seq# and when the
packet was created there may not be another DDP segment available following
this segment submitted to TCP layer).

These will not be possible w/o changes in TCP.

Thanks

Parag.
Aarohi Communications, Inc.

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Tuesday, December 03, 2002 10:47 AM
> To: [email-masked]
> If you look at the urgent pointer related portions of RFC 793
> (and if I recall correctly), the sender and reciever are required
> to manage only one "urgent pointer variable". This means that
> urgent pointers work as indicated by the user only when TCP can
> keep sending data - and breaks down when the window is not open
> or retransmission occurs. The window may not be open for many
> reasons including slow start, congestion avoidance or other end
> not acking. The behavior of middle boxes could be unpredictable
> (BTW does anyone have an example of a resegmenting middle box - is
> a proxy server considered one?).
> 
> BTW, some people have indicated that PSH flag could be used for
> creating aligned segments. I believe that some
> implementations do not follow this when window is closed or
> on retransmission.
> 
> Somesh
> 
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Williams, Jim
> > Sent: Tuesday, December 03, 2002 10:27 AM
> > To: 'David Borman'; [email-masked]; [email-masked]
> > I don't fully recall, but I think there may have been other
> > reasons this was rejected around potential non-interoperability.
> > I can't reconstruct this argument, however.
> >
> >
> > > From: David Borman [mailto:[email-masked]]
> > > Sent: Tuesday, December 03, 2002 11:45 AM
> > > To: [email-masked]; [email-masked]
> > > Subject: Re: [rddp] Let's discuss markers
> > >
> > >
> > > Hi,
> > >
> > > [...]  TCP already has a mechanism for identifying interesting
> > > [...]

---

From: David Borman <[email-masked]>
Date: Tue, 3 Dec 2002 15:59:52 -0600 (CST)

> Date: Tue, 3 Dec 2002 16:41:06 -0500 
>
> Try the following:
>

Thanks.  That document proposes the same thing (marking the first
byte of each message as "urgent"), except that it has some additional
song-and-dance about whether the urgent pointer points to the last
byte of urgent data or the first byte of non-urgent data.  You shouldn't
have to do that, because you're modifying TCP at both the sender
and receiver, so you can just state that it points to the last byte
of urgent data, (as clarified by RFC1122).  So, the ULP can chose to
define it as the first byte of the DDP header (or the last byte of
the header, or anyplace inbetween for that matter, just as long as
it is constent...).

I find it strange that the ADs/IESG would sanction the processing
of out-of-order TCP segments, but then balk at using the TCP Urgent
pointer as a hint to aid in that processing.  I don't know the specs,
does iSCSI allow for processing out-of-order TCP segments?

---

From: "Teisberg, Robert" <[email-masked]>
Date: Tue, 3 Dec 2002 16:51:25 -0600

> TCP is already being changed if out-of-order processing of 
> TCP segments
> is being allowed.
> 

That depends on the meaning of the word, "processing."

Certainly we're talking about a TCP implementation that's different
from the existing TCP implementations most of us are familiar with.
However, there are lots of ways one can change
TCP-the-implementation without changing TCP-the-protocol.

In this case, the debate involves interpretation of the part of
RFC793 that describes TCP's upper interface.  One camp says it's
a change to TCP if TCP can signal a ULP indicating, "this payload
has arrived but may be out of order, so take care not to use it
inappropriately."  The other believes such a signal is a local
implementation feature not affecting the protocol.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Dave Sheehy <[email-masked]>
Date: Tue, 03 Dec 2002 15:08:02 PST

IIRC, another argument against the use of the Urgent pointer is that routers
will process traffic for those connections with outstanding urgent data in 
their slow path resulting in a negative performance impact.

Dave Sheehy

---

From: Dave Sheehy <[email-masked]>
Date: Tue, 03 Dec 2002 19:02:04 PST

Somesh,

> Dave,
> 
> Being ignorant of how routers actually work, could
> you shed some more light on why this is the case.

I was recalling a point raised by somebody (presumably a router dude) on the 
ips reflector so I can't shed more light on this. 

After some spelunking on the ips archive the only comment I could find was:

> To:   [email-masked]
> cc:   [email-masked]
> Subject:  iSCSI: Urgent Pointer Negotation
>
> If the URG pointer feature is to be used, it should be negotiated.
>
> The reason it should be negotiated is that the presence of the URG
> pointer takes many TCP receive stacks off of their fast path. I am
> not familiar with whether it significantly slows current TCP send stacks.

which is about end points and not routers. I can't find the reference to
routers so perhaps my recollection is incorrect.

Dave Sheehy

> > From: [email-masked] [mailto:[email-masked]]On Behalf Of Dave
> > Sheehy
> > Sent: Tuesday, December 03, 2002 3:08 PM
> > To: [email-masked]

---

From: "Uri Elzur" <[email-masked]>
Date: Wed, 4 Dec 2002 15:45:51 -0800

Lots have been said on this in the last 2 days. This tries to address the key discussion topics raised by David Black.

Markers allow for 

1) Detection of FPDU boundaries
2) Direct Data Placement for out-of-order FPDU
3) Detection of intermediate boxes or misaligned FPDU senders 

Detection is required to allow taking action. 

Note that action taken upon detection is completely up to the implementation. It allows optimizing for cases where congestion can lead to more drops or to cases where it is clear no middle box is present. ANY IMPLEMENTATION should be fully TCP compliant and interoperable on the wire. The performance for each mode can change. 

The key attribute is that boundary detection for out-of-order is DETERMINISTIC (was not possible with TUF). I.e. there is no statistical argument for wrongly interpreting non aligned data as an aligned FPDU. The only way I know of, to let a Receiver know the original Sender alignment without a mechanism external to TCP (assuming the URG, TCP Options etc were banned), is by planting hints (i.e. markers) at locations known to the receiver a' priori. BTW: TUF promotes alignment as well ("The TUF-aware TCP sender preserves boundaries of shimmed ULPDUs (TUF FPDUs) as much as possible when delivering segments to the IP layer.") and this wasn't deemed a "TCP Change", at the time (at least I've seen it discussed that way).

Declaration or negotiation at connection set-up time, adds unnecessary latency to getting into DDP mode and doesn't solve the problem of middle-box. MPA allows middle-box detection (I share the view that this is NOT an error case). As the MPA draft says the implementation can choose the buffering level to decide how to handle this case. I'm not sure how a "training sequence" can guarantee it handles all the middle boxes and future middle-boxes with different algorithms. And again it adds un-desired latency to run this "training sequence".

512B as Marker interval - keeps the wire overhead at about the same level TUF had it. At high data rates the receiver efficiency, reducing RCV memory bandwidth and size, is much more important.

SW implementation - when comparing the overhead associated with CRC in SW, it is surprisingly irrelevant to add Markers handling as well.

Markers have been endorsed as a way for steering data by iSCSI 1.0. MPA uses a more efficient Markers approach, and should enjoy same endorsement as did the iSCSI Markers. 

MPA handles all of these cases (aligned or misaligned sender, Middle box etc) with no need to assume anything about static or dynamic middle box. No need to drop any data or find out later it was un-aligned and misplaced. AND it doesn't add latency to connection setup. AND it is no worse in SW than CRC in SW. AND it saves precious HW resources on the receive if you care to implement it.

Uri

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Monday, December 02, 2002 10:26 PM
To: [email-masked]
Subject: [rddp] Let's discuss markers

Thanks,
--David