Subject: RE: [Tsvwg] RE: ULP Framing for TCP

From: [email-masked]
Date: Sun, 12 Aug 2001 20:54:15 -0400

One big item and a bunch of small ones:

-- Middleboxes and PDU Alignment

Middlebox damage repair should happen automatically when the missing
packets show up and TCP delivers in the correct order -- the ULP
discovers that the data hasn't been placed in the fashion it expected,
and has to clean up (e.g., copy data to where it should have been
placed).  The draft needs a statement to the effect that
PDU alignment framing is a hint-based optimization that won't always
work - if the framing header with the right values occurs at exactly
the point in the data where the middlebox resegments, the receiver
will get confused and place the data improperly.  Hence, ULPs MUST
check that any data delivered to them via PDU alignment framing is
correctly placed -- this puts some teeth into the requirement that
TCP deliver framed data in order, because a ULP generally can't
check for this sort of incorrect data placement unless it has all
the intervening data.  A ULP that assumes correct placement is at
serious risk of doing something very wrong in the presence of
resegmentation.

But it gets worse ... when a middlebox resegments, a framing header that
used to be at the front of a TCP segment is now in the middle somewhere.
Unlike markers, where the position is always known and hence the marker
can always be removed, there doesn't seem to be any way to take the PDU
alignment framing headers out after resegmentation.  The problem is two-
fold - not only are the framing headers no longer aligned, but using the
length of the previous header to reliably find the next one won't work
if any of the exception conditions in Section 5.2 occurred, and the
sender chose not to use a framing header in order to deal with the
exception.  Delivering a framing header to the ULP as if it were data
is clearly unacceptable, so something has to be done.  

I can see two approaches to addressing this:
- Move ownership of the PDU Alignment framing header from TCP to the ULP,
	and use it as a PDU prefix rather than a TCP segment prefix - since
	the header then precedes every PDU, it can always be removed.  This
	puts yet more teeth into the requirement that TCP deliver in order.
- Get rid of the framing header entirely.  The sending TCP does its best to
	align PDUs with segments, the receiver does its best to place data,
	and the ULP checks for things landing in unexpected locations and
	cleans up anything that results.
The second approach eliminates any risk of a buggy implementation mistaking
a framing header for something else because there are no framing headers.

In any case, the draft needs some serious revision to deal with these
issues.

-- Fragmentation

Doug's right about 2 A/B/C in the description of what a sender does
when it has to fragment - the draft needs to say how a sender does this.
I presume a sender looks at the first 8 octets in this situation,
and if they look like a framing header, it chooses a TCP segment
size so that this confusion can't happen.  OTOH, the above approaches
to deal with resegmenting middleboxes would also get rid of this
fragmentation problem because the ULP doesn't care about fragmentation
and has to check data placement for other reasons.

-- ULP Requirements

As I noted previously, the requirements text at the end of Section 2
is a potential issue - the fastest way to address this may be to remove
this text and allow the ULP to specify its requirements for use of these
mechanisms, as different ULPs may reach different conclusions.

-- Packet size

The packet size limitation to less that 64k may be an issue for iSCSI.
SCSI data transfers often occur in power of 2 sizes, so any power of 2
above 32k would require iSCSI segmentation.  While iSCSI is capable of
doing such segmentation, it's not free (TCP is already segmenting, so
this is pure overhead), and 64k (and larger) SCSI data transfers do 
occur regularly in practice.

-- Negotiation

Section 4 addresses negotiation but the wording strikes me as a bit
loose.  It ought to place requirements on the negotiation mechanism
(what must be agreed to by the endpoints and communicated between
them in order for the framing mechanisms to work).

-- Interfaces

The software interface appears to be socket options (example
given in the draft's Appendix), plus some sort of packet filter
(highly platform-dependent) on the receiving side. I don't
think it's appropriate to standardize the interfaces to either
hardware or packet filters here.

-- Layering

The basic mechanism for inserting the code required to handle
the  receiver-side data placement already exists -  it's called
a packet filter.  Packet filters have an additional virtue -
it tends to be unbelievably difficult to implement any serious
protocol processing in them.  I won't defend packet filters on
elegance grounds, but they're accepted technology that already
make low-level decisions based on ULP data, although I'd guess
that they probably need some additional enhancement to do the
sort of placement that iSCSI would want.

-- Summary

The middlebox issues require some serious revision to the draft,
and the result looks like it'll be something other than a datagram
protocol because out of order delivery carries the risk that
the PDU boundaries will be incorrect and hence in order delivery
is a MUST.

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Stephen Bailey <[email-masked]>
Date: Wed, 22 Aug 2001 21:18:11 -0400

David,

Thank you for your scrutiny and feedback on the draft.

> exception.

Not sending a framing header is not an option for the sender.  As long
as the the framing protocol is being used in PDU alignment mode, a
framing header is sent.

If a received TCP segment does not pass the validation criteria for a
PDU-aligned segment (as a result of middlebox resegmentation or the
sender being induced to send non PDU-unaligned TCP segments, e.g. from
PMTU shrinkage), the data in the segment must be held awaiting
reception of all previous data.  Once all preceeding data is received,
the location of PDUs in the TCP stream becomes unambiguous.

The behavior when a TCP segment is not PDU aligned is equivalent to
current (unframed) TCP processing.

There are a range of algorithms for implementing this behavior.  One
candidate, which you seem to have alluded is that TCP segments which
pass the PDU alignment check are passed to a hardware direct placement
dodad, and TCP segments which fail the PDU alignment check are passed
to a (the) processor which performs traditional reassembly and then
placement.  This is a fast path/slow path solution.

All the draft is really saying is that when the PDU alignment check
passes, you have one (or more if PDU packing is enabled) complete ULP
PDU(s) in the TCP segment.  What an implementation choses to do with
this information, within the semantics required by TCP and the ULP, is
its own business.

The sender rules ensure that PDU alignment is restored after temporary
interruptions, such as result from PMTU reduction, or receiver window
reduction.  The draft also indicates that when PDU alignment is
expected and not seen for a LONG period of time, use of PDU alignment
mode should be discontinued.  This occurence likely indicates a
resegmenting middlebox.  Nothing fatal happens if use of PDU alignment
mode is not discontinued, but direct data placement may not be
possible.

> Delivering a framing header to the ULP as if it were data is clearly
> unacceptable, so something has to be done.

The framing header is not delivered to the ULP.  Complete ULP PDUs are
delivered to the ULP.  Doing so requires only knowledge of the framing
protocol, not the ULP, so layering is preserved.

We will try to improve the draft to reinforce the point that ULP PDUs
are what are delivered from the framing protocol layer.  The framing
headers or markers are added and stripped within the framing
protocol---the ULP never knows about or sees them.

> when it has to fragment the draft needs to say how a sender does
> this.

We felt that this declarative specification (what the sender must
ensure) is the best way to be precise here.  There are various
implementation alternatives which satisfy these critiera and we are
allowing implementors the latitude to find the the one(s) that best
trades off implementation complexity and performance.  The correctness
of the protocol does not depend upon the choice in any case.

Based upon Doug's and your comments, we will expand this section with
a discussion of some implementation alternatives for the TCP
resegmentation choice ('2').

> The packet size limitation to less that 64k may be an issue for

A ULP that is a candidate for using the framing protocol must have the
ability to tile its PDU stream in response to the PMTU.  iSCSI already
requires the ability to tile data PDUs in to chunks as small as 512
bytes, so the mechanics are there.

As for the requirement that sizeof(PDU) < 2^16-8, for direct placement
(and hence the framing protocol) to be effective sizeof(PDU) must be <
max TCP segment payload, which is, itself < 2^16-8.  In other words,
the limit of 2^16-8 doesn't actually restrict sizeof(PDU), max TCP
segment payload does.

> It ought to place requirements on the negotiation mechanism (what
> must be agreed to by the endpoints and communicated between them in
> order for the framing mechanisms to work).

We'll try to clean up the wording.  What makes the present wording
seem weak is that `probably' actually modifies `exchange'.  Before the
framing protocol is enabled, each end point must know all the
parameters required to operate.  However, these parameters need not be
exchanged in any particular way.

For example, connecting to an distinguished port might a priori
indicate a particular ULP with a particular framing mode, and
particular parameters.  No `negotiation' or `exchange' is required in
this case at all.

Thanks again for your constructive comments.  We hope to have an
updated draft shortly which incorporates the feedback we've received.

Steph