Subject: [Tsvwg] RE: ULP Framing for TCP

From: "Douglas Otis" <[email-masked]>
Date: Wed, 8 Aug 2001 08:21:53 -0700

All,

My concerns about introducing DCP and ULP Framing for TCP speaks to a
concern regarding the blurring of the message to ensure software and
equipment manufactures see a clear road map for implementing a Protocol Data
Unit sensitive protocol.  Before SCTP has gained momentum, introduction of
these alternatives negatively impact adoption of SCTP and in the case of ULP
Framing for TCP will hurt the stability of TCP.  DCP offers congestion
control variants less streams.  Such congestion variants could be added to
SCTP by means of initialization if required.  TCP should remain unchanged
and if there is a new protocol behavior desired, this should be a new
protocol!

Contrary to the views of Allison Mankin, the principle intent for ULP
Framing for TCP is to align and frame ULP PDUs within the TCP frame.  The
other modes offered by this framing document are sub-optimal fall back
modes.  There is NO difference between this ULP PDU and an SCTP data chunk
and with the alignment requirement, there is little difference between this
and DCP except for the desire to overlay the protocol designator with these
new behaviors.  Why is this overlay a problem?  It requires that middle
boxes such as security gateways to now maintaining the original TCP
segmentation while there is NO method to recognize this requirement or their
equipment will negatively impact the performance of this new TCP!

http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-00.txt

Pg 4:

"2.  Theory Of Operation

  One very efficient way to guarantee that subsequent PDUs can always
  be found when a previous PDU header has been lost is to ensure each
  TCP segment begins with a PDU and contains an integral number of
  PDUs.  In this case, the data in each TCP segment may be placed
  independently of all other segments."

  "This document defines the behavior of a TCP with a modified sender
   behavior, called a `framing-aware TCP'.  A framing-aware TCP allows
   a ULP implementation to ensure that each TCP segment begins with a
   PDU."

  "PDU alignment mode is more powerful, and is preferable when
   available.  Marker mode still requires some high-speed reassembly
   memory, whose size is a linear function of the number of active TCP
   connections."

The ULP PDU alignment is the SAME as an SCTP data chunk!

  "The TCP maximum segment size (MSS) is defined in RFC 793 [TCP] as
   the segment size exchanged on TCP connection establishment.  In
   addition, there is the segment size presently used by TCP which is
   less than or equal to the exchanged MSS, adjusted by the current
   path MTU [PathMTU].  This document calls the MSS presently in use
   the `effective maximum segment size' (EMSS).  The EMSS is of
   primary concern to the framing protocol in PDU alignment mode.

   The TCP EMSS can shrink to 8 octets [PathMTU] which leaves no room
   for a PDU in PDU alignment mode. If the EMSS goes below 512 octets,
   the ULP MAY instruct the framing protocol to enter an "emergency
   mode."  In this mode, the framing module MUST accept PDUs up to 512
   octets and MAY fragment a PDU across TCP segments."

This definition is very close to that of an SCTP data chunk except for the
minimum maximal provisions.  SCTP went to great lengths to deal with
problems related to changes in the MTU and even allows return of data to the
application which is not covered by this specification.  Why would such
provisions be required for SCTP and not this new version of TCP?

"4.  Negotiating Use Of The Framing Protocol

   Negotiating use of the framing protocol is the responsibility of
   the ULP.  The use of the framing protocol MAY be negotiated
   separately for each direction on a particular connection.  The
   negotiation procedure MUST ensure that when receive framing is
   enabled, the remote peer will not transmit the first TCP segment
   with framed data until it is certain that the local peer has
   actually enabled receive framing.

   If a receiver requests PDU alignment mode, and the sender supports
   PDU alignment mode, then the sender MUST enable PDU alignment mode.
   This ensures that PDU alignment mode, with its favorable hardware
   characteristics, is used when possible."

There is no definition of how this negotiation is to happen as this is left
to the ULP which means this will impact all applications that use this
extension of TCP.  In addition, there needs to be a new API defined between
the application and the network adapter driver to adopt the use of this new
style of TCP sending and receiving.  This API has not been defined by this
draft nor have the modifications to the TCP stack interfaces for marking
these PDU boundaries.  I would expect the function of a standards body is to
ensure a modification of a protocol does not invite a proliferation of
variant implementations.  At the very least, these open issues should be
addressed.

 "If the framing protocol in PDU alignment mode must send an
  unaligned framing PDU, it SHALL take one of the following actions.

  1.   Send the framing PDU as a single TCP segment using IP
       fragmentation.  While this behavior is discouraged, it is not
       prohibited by the framing protocol, or any other applicable
       RFCs.

  2.   Send the framing PDU as several TCP segments, with each
       segment guaranteed not to appear as a well-formed, complete
       framing PDU on its own, at the time the segment is sent.  That
       is, the sender SHALL ensure that one of the following is true
       for every segment with a partial framing PDU:

       A.   octets 0-1 do not equal the segment length minus 8
       B.   octets 2-8 do not match the framing key value
       C.   the total segment length is less than the framing PDU
            header of 8 octets."

Even for fragmentation there are four equally valid methods of indicating?

"5.3.  Validity Of Framing-aware TCP Segmentation

  A framing-aware TCP normally sends exactly one framing PDU per TCP
  segment.  This may therefore result in more segments being sent
  than would occur in a traditional TCP.  However, the framing module
  is allowed to pack multiple ULP PDUs into a single framing PDU if
  ULP packing is enabled, which will give behavior approaching that
  of a traditional TCP.  Even with ULP packing disabled, the behavior
  of a framing-aware TCP effectively corresponds to that of a
  traditional TCP sender with the Nagle algorithm disabled (i.e.
  TCP_NODELAY), and this is considered acceptable behavior.

  Framing-aware TCPs still respect congestion control windows, which
  are maintained as a octet count not as a segment count.
  On retransmission, a framing-aware TCP respects the original stream
  segmentation.  This is allowed by RFC1122 [RFC1122], section
  4.2.2.15."

Here again, you see the chunking behavior of SCTP now introduced into TCP.
My concern is that this invites the sub-optimal adoption of this technique
over the superior behavior offered by SCTP.  I see no justification to
endorse this modification of TCP even with the excuse that SCTP is not yet
ready.  Even the intended application for this variant TCP is not complete.
This is nothing more than a significant hack to TCP that should not happen
although I would agree that a protocol that provides PDU alignment is
needed.  I think that requirement has been adequately been addressed by
SCTP.  Either you support a convergence on this concept or you risk the loss
of momentum.  Cobbling a change onto TCP to make it operate akin to SCTP
ensures a degradation of both TCP and SCTP.

Doug

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Allison Mankin <[email-masked]>
Date: Wed, 08 Aug 2001 23:25:28 -0400

> Framing for TCP is to align and frame ULP PDUs within the TCP frame. 

It certainly isn't contrary to my views to define TCP ULP framing
as you do here.  But calling TCP ULP framing a datagram protocol is
contrary to transport concepts, because a datagram protocol in transport
is unreliable.

> Cobbling a change onto TCP to make it operate akin to SCTP 

There is much more to SCTP than the service afforded here.

I believe the authors of the draft should address your specific
technical issues excerpted below.

Allison 

> Date:    Wed, 08 Aug 2001 08:21:53 PDT
> Subject: [Tsvwg] RE: ULP Framing for TCP
> From:    "Douglas Otis" <[email-masked]>
> To:      "Tsvwg" <[email-masked]>
> --------
> 
> Why is this overlay a problem?  It requires that middle
> segmentation while there is NO method to recognize this requirement

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 9 Aug 2001 05:42:20 -0700

Allison,

I said provide the same services as Datagram... but I was cut off.  I did
not want to use the mnemonic DCP because it sounds too close to TCP.  Never
the less, I will happily argue against even your erroneous statement about a
datagram.
A datagram is, to quote the Internet's RFC 1594, "a self-contained,
independent entity of data carrying sufficient information to be routed from
the source to the destination computer without reliance on earlier exchanges
between this source and destination computer and the transporting network."
You will note this does NOT define datagram in terms of reliability.  The
reliability of any datagram is dependent upon a higher level retry mechanism
and is no more or less reliable as a result of this reliance.  In the case
of the ULP Framing for TCP, the TCP transport itself is relegated to this
higher level retry mechanism as segment processing and placement takes place
prior to this reliability mechanism once known as TCP that, until this
Framing scheme, TCP was also responsible for placement of data that also
held off processing of the packet until a sequential stream of bytes were
received.  For ULP Framing for TCP, it changes the TCP segment into a form
properly defined as a datagram where this new TCP is just an independent
datagram retry mechanism.  Instead of passing bytes for processing, this new
TCP simply signals the sequential boundary for datagrams.

The ULP Framing for TCP comes extremely close to this RFC definition as the
intent of the Framing is to allow processing and placement of the TCP
segment payload independent of earlier exchanges regardless of delivery
order.  Those advocating this protocol hack deny this is their intent, but
that is only subterfuge.  It is likely the processing of this segment is
unaware of the state of the TCP transport that now resides above this
process erroneously described as packet filtering.  A filter that is unique
to the application that also examines structures and places control and data
chunks accordingly is NOT a simple packet filter.

Both ULP Framing for TCP and SCTP produce segment aligned chunks, but lets
look at the differences
of ULP Framing for TCP and SCTP.
-------------------------------------|ULPfTCP| SCTP |
Defines the Application Interface    | No    | Yes  |
Defines MTU Application interaction  | No    | Yes  |
Changes the NIC Driver Interface     | Yes   | No   |
Improves Error detection             | No    | Yes  |
Changes optimal gateway behavior     | Yes   | No   |
Modifies existing Protocol Stacks    | Yes   | No   |
Extensively Modifies Layer Boundaries| Yes   | No   |
Adds multi-homing (in iSCSI)         | No    | Yes  |
Aggregation of streams (in iSCSI)    | No    | Yes  |
Protection from DoS attacks          | No    | Yes  |
Protection from Blind Spoofing       | No    | Yes  |
ULP Negotiation with NIC Driver      | Yes   | No   |
Provides data recovery on failure    | No    | Yes  |
Optionally allows chunk fragmentation| No    | Yes  |
-----------------------------------------------------

Yes, ULPfTCP provides a solution but SCTP provides the SAME solution even
better.  I find it hard to believe that it is preferable to redefine TCP so
dramatically just to avoid use of SCTP just because they can not wait for
SCTP.  I have yet to hear a clear explanation as to why SCTP can not be used
in this case to raise this hack of TCP as a preferred solution.

  TCP layers

   [APP]
    ^ |  (byte exchanges)
    | v
   [TCP]
    ^ |  (segment exchanges)
    | v
   [NIC]

  ULPfTCP layers

   [upper APP]
    |  ^ |  (completion & chunks)
    |  | v
    |[new TCP]
    |  ^ |  (mode, completion, datagram send & retry)
    v  | v
   [lower APP]
      ^ |  (datagram exchanges)
      | v
     [NIC]

Although there may have been work prior to placement into tsvwg, the amount
of exchanges within this wg have been rather limited.  I suspect few within
this wg were even aware of this activity as this occurred largely to satisfy
the desires of ips and was only discussed in ips.  Placing an unannounced
consensus call on the reflector where you even managed to skip announcement
of the status of this draft within the wg raises doubts about a desire for a
fair consensus as it is difficult to determine such over a reflector.  It
would appear your passion for making this change has clouded your judgment.

You claim that allowing this modification will not impact the roll-out of
SCTP, but with Microsoft as one of the major proponents of this
modification, once this framing protocol is in place, their desire to
implement SCTP will likely wane.  Much as the profusion of variant
interfaces for WinModems has greatly benefited Microsoft, a profusion of
variant network adapters will likely produce a similar result again much to
the benefit Microsoft.  A clean interface, layering and standardization is
NOT to Microsoft's benefit.  You can make a significant difference here by
simply saying NO we will not change TCP.  If you want a frame aligned
protocol, use SCTP!  TCP is not nor should ever be frame aligned.  To do so
now will be to the detriment of many appliances and applications that will
then be deemed "broken" as a result of sub-optimal performance because they
do not track and restore TCP segment boundaries.  In some case, blind
segment restoration is likely not the benefit of the network but as this
scheme provides no inband clue, doing so on every TCP stream will become the
required behavior.

Doug

---

From: Kacheong Poon <"kcpoon"@eng.sun.com>
Date: Thu, 09 Aug 2001 10:54:11 -0700

Douglas Otis wrote:
> DCP offers congestion
> SCTP by means of initialization if required.

As pointed by Randy at the tsvwg meeting, U-SCTP and DCP are very
different.  It is not just about congestion control algorithms
selection.  And I don't think making SCTP the "ultimate" transport
protocol by forever adding new features to it is a very good idea.
It will become a monster very quickly...  SCTP has its own design
requirements which lead to its current form.  And I think those 
requirements are very different from that of DCP.

							K. Poon.
							[email-masked]

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 9 Aug 2001 11:08:12 -0700

Kacheong,

SCTP does have a fair amount of flexibility that should not greatly encumber
its use.  The major difference is the flow control as SCTP streams are
likely beneficial for most DCP application.  Regulating the send function
can be done a number of ways where you want the sender to respond to
congestion as it does now.  Anything done differently will represent a
fairness problem.  I also think you will shortly hear about some flow
adaptation shortly for SCTP but I will let Randall lead that discussion.
Output pacing can be done a number of ways using the present schemes with
the proper recommendations.  It takes a great deal of effort to introduce a
new protocol especially one as significant as SCTP.  DCP appears to
trivialize that effort will little benefit except for its ability to open
Pandora's box with respect to flow control.

Doug

---

From: Kacheong Poon <"kcpoon"@eng.sun.com>
Date: Thu, 09 Aug 2001 11:01:55 -0700

> technical issues excerpted below.

Two non-technical questions for the ADs...  If this ULP framing
for TCP becomes an RFC, what is the classification of that RFC?
Informational?

I was surprised to learn that the last call for this draft will
happen at the end of this month.  I beleive the discussion of
the previous version of this draft (which was not a tsvwg document)
in the end2end list clearly shows that a lot of people don't think
it is a good idea.  And it seems to me that there is no fundamental
difference in the current draft and the last one.  So can we do
a last call for this draft at all?

							K. Poon.
							[email-masked]

---

From: Allison Mankin <[email-masked]>
Date: Sat, 11 Aug 2001 13:08:35 -0400

Kacheong,

We did not state plans to IETF Last Call the ULP framing
document at the end of the month, but rather to working group
last call it.  As you say, there was a discussion of ULP framing on
the end2end-interest list, but while that was valuable, it didn't
have many participants who had read the current draft.

TSVWG had two working group sessions about ULP framing at IETFs -
San Diego and Minneapolis.  In San Diego, we had much contention.
The new draft came from that, I believe dealing with many of the
contentious issues.

In Minneapolis, Jim Pinkerton presented the new specifics.  When we
asked for the room's sense of whether to make the new draft a working
group draft in the standards track, we received a quite strong support
for doing so.  

The next steps are to have a rev of the current draft (AD comments)
and then working group last call.  Everyone should read the next draft
and participate in the detailed discussion during the working group
last call.  It may be at the end of the month or else a few weeks
after, because some (including me) are taking a post-IETF vacation.

Allison

---

From: Allison Mankin <[email-masked]>
Date: Sat, 11 Aug 2001 13:24:45 -0400

Doug,

The ULP framing document perhaps does not say strongly enough
(but will after the upcoming rev) that there is no delivery
of the data before it is acknowledged - placement is not the same
as provision to the application.  I have a comment inline in
your message below as well.

I would again invite the authors to take up the specific
points you have raised about the draft, in your previous message.

I'm about to depart on a vacation, during which I will read and
answer mail pretty sparsely, but I hope that your comments can enter
into the new rev.  I apologize for addressing at best only one
of your lengthy email's points (though I think it's the major one), but
there is this imminent departure for another airplane... :)

Allison

> Date:    Thu, 09 Aug 2001 05:42:20 PDT
> Subject: RE: [Tsvwg] RE: ULP Framing for TCP 
> To:      <[email-masked]>
> Cc:      "Tsvwg" <[email-masked]>
> --------
> 
#### (excerpt)
> In the case
> that is only subterfuge.  

#### if the iSCSI document of how to use ULP Framing prohibits delivery
#### of unacked data, and the ULP Framing spec does so?

---

From: "Douglas Otis" <[email-masked]>
Date: Sun, 12 Aug 2001 13:18:10 -0700

Allison,

I am aware of the semantics used to cloak the level of processing proposed
to occur below TCP.  Rather than TCP placing the data as normal
(sequentially within buffers), now TCP is only a signal path to the
application as it sorts these signals within its new role as a datagram
manager.  This lower layer process that understands the frame aligned header
residing adjacent to the framing marker is unique to the application.  In
the case of iSCSI as example, there is no means for the application to
anticipate what data is being received which already has defined buffers
allocated.  It would be the process residing below TCP that makes the needed
association to direct this data into this disparate array.  This puts the
application both above and below TCP and violates the present layering
structure.

SCTP can provide the same function using streams to direct this information
into the correct buffer without the application being placed both above and
below the transport as with the case of ULP Framing for TCP.  There would
not be a need for making thousands of vendor and application unique drivers
but then again if you are a company that wants this level complexity at the
OS, then you would not want a protocol that preserves the traditional
networking layers.

I would encourage those interested to read this draft
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-00.txt

ULP Framing for TCP:
"Abstract

 The framing protocol accepts PDUs from a ULP (upper level protocol)
 and transports them over a TCP connection.  This is done in such a
 way that the PDUs can be recovered at the receiver even if
 preceding TCP segments have not yet been received.  This is useful
 when the PDUs are self describing within the context of a protocol
 TCP connection.  In this case, the framing protocol allows incoming
 packets to be parsed (but not processed) in the order received and
 their data to be placed directly in the ultimate destination memory
 instead of TCP reassembly buffers."

It is interesting their insistence on "not processing" while buffer
association needed to ensure the "ultimate destination" is application
unique with RDMA and iSCSI used as example applications within their draft.
While insisting there is a philosophy granted in RFC 879 (that discusses
inter-layer communication for determining MSS) for this TCP hack, the
layering is dramatically violated and worse yet, these signals are
undefined.  I can not think of a better way to make a grand mess.

Here is where TCP becomes a datagram protocol.
ULP Framing for TCP:
"2.  Theory Of Operation

  One very efficient way to guarantee that subsequent PDUs can always
  be found when a previous PDU header has been lost is to ensure each
  TCP segment begins with a PDU and contains an integral number of
  PDUs.  In this case, the data in each TCP segment may be placed
  independently of all other segments.  No reassembly buffer is
  required at all.  Guaranteeing a TCP segment begins with a PDU
  requires a modification to TCP's sending behavior.  This document
  defines the behavior of a TCP with a modified sender behavior,
  called a `framing-aware TCP'.  A framing-aware TCP allows a ULP
  implementation to ensure that each TCP segment begins with a PDU."

If one were to squint, you would think they were discussing SCTP.  One
important aspect left out of their scheme is the concept of streams.
Without this concept, they then require that there is an application unique
routine below their FATCP (Framing Aware TCP).  The placement of an
application both above and below FATCP is a violation of layering!  This
also has a negative impact on current equipment.  This is not in harmony
with any current architecture and makes a mockery of any grand statements
about taking the high road at the Plenary meeting.  There is nothing
provided by this hack that is not also provided by SCTP with a tremendous
difference with respect to architecture or its destruction.  Your
recollections of these meeting were different than mine, but then we seem to
be viewing things differently.  Apart from the ips group, you will find few
eager to embark on this destruction of TCP.  It was outside of the ips
charter to make these changes to TCP, where I was about to be censured by
David Black when I suggested repeatedly that modifying TCP was not allowed
nor reasonable (A goal stated at the ips pre-BOF).  It would appear they
have found a wg without a charter limitation.  Does tsvwg have a charter?

Doug

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Allison Mankin
> Sent: Saturday, August 11, 2001 10:25 AM
> To: Douglas Otis
> Cc: Tsvwg
> Subject: Re: [Tsvwg] RE: ULP Framing for TCP
>
>
> Doug,
>

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 18 Aug 2001 21:54:12 -0400

Doug,

> ULP Negotiation with NIC Driver      | Yes   | No   |

Neither mode of the framing protocol requires change to the NIC driver
interface, nor any direct communication with the NIC driver.  PDU
alignment mode uses the existing path MTU interface with the IP layer.
Note that RFC 1191 (Path MTU) makes a strong case that such
inter-layer communication does not `break' layering.

> SCTP [...] allows return of data to the application which is not
> covered by this specification.  Why would such provisions be
> required for SCTP and not this new version of TCP?

The framing protocol chose one particular solution to this problem.
SCTP chose another.  It's not a matter of externally imposed
requirement.

> Even for fragmentation there are four equally valid methods of

There are actually only 2.  The first is to use IP fragmentation.  The
use of IP fragmentation can not be prevented by the framing protocol
in this case.  Its mention here is not specifying a new behavior, but
simplying pointing out that this existing behavior choice solves the
problem.  Some TCP stacks already employ this exact behavior on PMTU
change.

The second alternative, ensuring that when a framing PDU is split
across several TCP segments, none of the segments fail the PDU
alignment check false positively when they are sent, is specified
because of concerns about the risk of wrapping the IP fragment ID
value on very high speed links.  The fragment ID wrapping problem
exists independently of any transport protocol.  The framing protocol
simply provides a mechanism to avoid exacerbating the problem.

Steph

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 20 Aug 2001 15:11:40 -0700

Stephen,

> inter-layer communication does not `break' layering.

Your support for this new inter-layer communication to occur
bi-directionally for nearly every packet is premised on a change to an ICMP
response to include within a Datagram Too Big message a report of the MTU of
the constricting hop.  In this change, they included a suggestion for the IP
layer to store PMTU information where the ICMP layer still processes
Datagram Too Big messages.  Here inter-layer communication between the
transport and IP layer was based on shared PMTU information where the IP
layer response to a SEND datagram with the DF bit set that is larger than
the PMTU estimate is to fail and return a suitable error indication.  IP
uses tables generated from other related protocols where PMTU is added to
these existing structures.  From this simple change, you see endorsement for
this aggressive inter-layer communication?

The aggressive inter-layer communication that will be generated by this
draft is to allow the "Packet Filter" to process incoming packets in the
order received where their contained data is to be placed directly within
the "ultimate destination memory" instead of within buffers.  This placement
is done at the system interface for there to be avoidance of buffers as this
draft indicates.  Taking iSCSI as an example, these memory locations are
assigned by the application but are received in any order within the stream
where, in addition, each packet to be handled independently of the next
during this placement process.

The "Packet Filter" process runs somewhere between the Network Interface
Card and the IP layer and will be expected to interchange this new
information of ultimate destinations together with related identifications
matched within the application's data structures.  Whether you call this
parsing, error checking, identifier matching with buffers, and actual
extraction and placement of control, sense, and data a process or not does
not change the application unique nature of this "below the IP layer"
function.  This application specific process below IP is not analogous to
the inter-layer communication suggested in RFC1191 which was not application
unique as it would be for this draft.  With this draft, the condition of the
network interface would never be stable as there will be perhaps hundreds of
these unique application widgets running below IP and capable of doing
anything.

> SCTP chose another.  It's not a matter of externally imposed

The point I was making by creating this list was to illustrate the inferior
solution created by this draft.  As SCTP is further along in both
development and testing, I find it hard to argue that this draft's
modification should be done pending the adoption of SCTP.  This pre-supposes
the immediate inculcations resulting from this draft together with flawless
implementation.  In addition, this leap into immediacy is not made possible
through documentation.  Unlike SCTP, there is no information as how these
inter-layer signals are to be exchanged, what they are, or how existing
interfaces are to be modified.  This draft seems to be blank check for
anyone to do anything with respect to altering networking layers.

> simply provides a mechanism to avoid exacerbating the problem.

My comment was in regarding the many options allowed this process of
fragmentation.

Pg 9&10:

            header of 8 octets"

Doug

---

From: Stephen Bailey <[email-masked]>
Date: Tue, 21 Aug 2001 18:43:51 -0400

Doug,

> bi-directionally for nearly every packet is premised on a change to
> an ICMP response to include within a Datagram Too Big message a
> report of the MTU of the constricting hop.

There's no new lower-layer interfaces required.  The PMTU information
that TCP uses, collected in whatever form it is presently collected,
works fine for the framing protocol.

> The aggressive inter-layer communication that will be generated by
> this draft is to allow the "Packet Filter" to process incoming
> packets in the order received where their contained data is to be
> placed directly within the "ultimate destination memory" instead of
> within buffers.

A packet filter model was not mentioned in the draft nor specifically
suggested by any of the authors.  The framing protocol provides cues
for locating ULP PDU boundaries in the TCP byte stream.  What an
implementation choses to do with these queues is, well, a matter of
implementation.  A packet filter might be a good implementation model,
but it's not the only one.

> The point I was making by creating this list was to illustrate the
> inferior solution created by this draft.

Pure technical superiority, is not the sole basis for chosing an
approach:

> > the approach the hardware vendors are comfortable with is to start
> > on unmodified TCP, migrate to TCP + the framing protocol when
> > framing-aware (hardware) receivers become available, and finally,
> > move to a new transport (e.g. SCTP) when it becomes widely
> > deployed.

> My comment was in regarding the many options allowed this process of

Many being two (2).  One is an behavior which could not be excluded,
and the second choice is for when the first choice is not appropriate.
Most protocols offer a practically infinite range of observable
behaviors, and the framing protocol is no exception.

Steph

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 21 Aug 2001 17:31:56 -0700

Stephen,

> works fine for the framing protocol.

Sorry, but I was simply describing the contents of the RFC that you used to
justify the inter-layer communication that would be required of your draft.
I did not intend to suggest that this particular RFC1191 information was the
nature of the communication that was to be used.  I went to some length to
indicate the difference in the nature of this communication. :(

> implementation chooses to do with these queues is, well, a matter of

Although a packet filter was not mentioned specifically, there would be
needed some form of application specific process that runs somewhere below
the TCP transport to allow the features being advocated by this draft or be
forced to use buffering which the draft indicates is not used.  The packet
filter concept was forwarded by David Black where I think that packet filter
is too nice a term for this function as I think it would be better to call
it an Application Layer Director.   David describes the application
signaling as happening through the receive function and that too is not
mentioned in this draft.  It would almost seem you do not wish anyone to
understand how to make use of this packetized version of TCP.  Perhaps such
use is proprietary or simply being obscured.  Perhaps you can explain this
quote from the draft in terms that does not include some additional process
below the transport.

"In this case, the framing protocol allows incoming packets to be parsed
(but not processed) in the order received and their data to be placed
directly in the ultimate destination memory instead of TCP reassembly
buffers."

What is the difference is between an application specific parsing and
placement and a process?

> Pure technical superiority, is not the sole basis for choosing an
> approach:
>

Again, you are making an assumption as to what should be sanctioned by the
IETF.  For the IETF to advocate the adherence to network layering, I find it
difficult to see this technique being endorsed when one considers the level
of destruction this has with respect to layering.  Worse yet, there are no
standard signaling methods created as a result, as this draft has selected
silence instead of being explicit.  I can only see this silence a benefit to
the major proponent of this draft and not to the community at large as they
will be forced to grope for these interfaces used in various applications
running on a proprietary operating system.

Having the application sit at the network adapter is not without its
pitfalls.  In addition, the poor technique used to determine the integrity
of framing makes this scheme weak at best.  Even David Black objected to
this technique as being weak and suggested the complete removal of the
framing header.  This draft disregarded an important feature found in SCTP.
Use the frame error checking scheme of SCTP to have a much more secure
method of determining the state of the segmentation for any particular
frame.  This would be important as the draft intends to parse, match,
extract and place the frame's content when delivered out of sequence as so
indicated.  Even David's solution would not be complete.  Again, this draft
should not be endorsed by the IETF nor should it be seen as an evolutionary
extension of TCP.  The use of this technique will be the demise of TCP as
there are no standards placed on the application specific processing objects
below IP and the TCP interface will rapidly become depreciated with all the
undefined signaling added.

SCTP is a much safer route than this draft.  SCTP does not require such high
levels of non-standard inter-layer signaling and provides known application
interfaces.  Yes, such TCP tweaks are possible, but that does not make them
a safer choice nor one that merits being placed into even an informational
RFC.  I would not be surprised to find that SCTP will be stable long before
such quick fixes.  There are many that make changes to networking outside
such endorsement, but few such changes are to the benefit of the networking
community.  Opening this door where applications sit both above and below
the networking transport is not an improved design nor a secure one.  It
won't be long after that before both IP and TCP layers become historical
artifacts.

> behaviors, and the framing protocol is no exception.

Sorry, I still count 4 ways.  You have the IP fragmentation and then three
methods to mark such fragmentation within your header.  Why all of the
marking options?

Doug

---

From: Stephen Bailey <[email-masked]>
Date: Wed, 22 Aug 2001 21:47:00 -0400

> I went to some length to indicate the difference in the nature of
> this communication. :(

The nature of the communication is the same.  The PMTU already known
by a PMTU-aware TCP may be used directly by an implementation of the
framing protocol.

> What is the difference is between an application specific parsing and

It is an implementation-specific choice where data is placed before it
is delivered to the ULP.  Traditional TCP implementations place it in
a reassembly buffer.  Implementations that leverage information
provided by the framing protocol may be able to implement a different
placement behavior.

> IETF.

No assumption.  We're making a proposal for the IETF to consider.

> For the IETF to advocate the adherence to network layering, I find
> it difficult to see this technique being endorsed when one considers
> the level of destruction this has with respect to layering.

The case for layering distruction resulting from path MTU (RFC 1191)
discovery applies identically to the framing protocol.  In fact, the
framing protocol needs only the existing inter-layer information which
is already collected to implement path MTU for TCP.

> methods to mark such fragmentation within your header.

The protocol does not specify an implementation, it specifies
properties any implementation must ensure.  Either the TCP segment
must contain the entire framing PDU, in which case it will be
IP-fragmented, or each segment of the segmented framing PDU must not
pass the checks which indicate that a TCP segment is PDU aligned.  A,
B & C are a restatement of the properties checked by the PDU
alignment-mode receiver.

Steph

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 23 Aug 2001 12:26:46 -0700

Stephen,

There is a large networking community that could benefit from the details of
this new networking stack being proposed in this draft.  Please stop the
pretense there are no changes of concern or as you said "is its own
business."  Perhaps you represent a segment of the industry that tends to
feel such details are to be shared at their prerogative.  It also raises
concerns as to which of these details do you consider proprietary.  It is
hard to say, if you fail to discuss them.

> by a PMTU-aware TCP may be used directly by an implementation of the

But of course you will be using existing elements found in the current
stacks.  But you will be doing much more than that.

In your response to David you indicated:
"The framing header is not delivered to the ULP.  Complete ULP PDUs are
 delivered to the ULP.  Doing so requires only knowledge of the framing
 protocol, not the ULP, so layering is preserved.

 We will try to improve the draft to reinforce the point that ULP PDUs
 are what are delivered from the framing protocol layer.  The framing
 headers or markers are added and stripped within the framing
 protocol---the ULP never knows about or sees them."

Well yes, there may be a simple case where the next available buffer is used
and PDUs are blindly passed to the ULP.  But this is NOT the case for iSCSI
as example and ignores the negotiations needed to do even the stripping of
the framing header or how the TCP sender is to behave when it switches to
the framing mode of operation which have not been covered in this draft.

Here is another quote from the draft:
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-00.txt

Pg 3:
"Both specific application ULPs, such as iSCSI, and generic hardware
 acceleration ULPs, such as an RDMA protocol, offer the potential
 for direct data placement.  The advantage of using a generic
 acceleration ULP for direct data placement is that the same direct
 placement network interface can be used to accelerate many
 different application protocols (e.g. iSCSI on RDMA)."

Here you have indicated the mode possible but completely left out any
details relating to the inter-layer signaling needed to achieve this "direct
data placement."  SCTP explains the means for accomplishing this technique
which is done through the use of streams.  SCTP has offered documents to
provide a common set of interfaces and can be used for many different
applications and on many different operating systems.  SCTP does not require
special application specific network adapters to be built with proprietary
interfaces to accomplish this feat.  Although this draft wishes to claim a
common solution, in reality, each application must develop their own
Application Layer Director.

Pg 1:
"1.  Introduction
  Many upper layer protocols (ULP)s, particularly those which perform
  bulk data transfer, permit the final location of transferred data
  (e.g. a ULP client buffer) to be known when the data is received.
  The information required to compute the final location of such data
  is contained in local protocol state and ULP protocol data unit
  (PDU) headers.  In this case, ULP data can be placed directly at
  its final destination by a network interface with knowledge of the
  ULP.  A direct placement network interface can offer extremely high
  performance since the host CPU does not copy the data at all, and
  the data only crosses system buses once."

This is a noble goal, but one easily accomplished by SCTP without the
application specific layer placed within the network interface.

With your draft the following layers are in play:
          Application
    [Upper Layer Protocol]
             TCP
             IP
  [Application Layer Director]
           [IPSEC]
        Physical Layer

The Application Layer Director places PDUs into [designated] receive
buffers.  This parsing and placement is application specific as in the case
of iSCSI, RDMA, etc.  You suggest that no one has any business knowing about
these additional signals used to indicate when it okay to start stripping
the framing header or how designated buffers are identified or for that
matter how PDUs are delineated except through contiguous bytes within sends
and when rules change that violate normal TCP send behavior.  You have
suggested that specific ports may be designated for this special behavior to
eliminate the need to explain even an inter-layer communication to support
even this negotiation.

There will be a rich set of signals that will be happening for almost every
exchange between the ULP and the ALD.

Pg 12:
"The framing protocol passes the contained ULP PDUs to a ULP parser.
 The ULP parser performs direct placement for the PDUs.  The ULP
 parser MUST NOT execute the ULP protocol (i.e. none of the ULP
 protocol state variables change), until all preceding octets in the
 TCP stream have also been received."

Here is where I understand what is considered processing.  In reality, there
will be state within the Application Layer Director that will impact the
Upper Layer Protocol as this layer must behave in an application specific
fashion for the ULP to operate correctly.  The ALD acts as a partial
pipeline of the process happening in the ULP.  You insist that the partial
processing be statewise independent of the ULP.  This implies all errors are
to be handled by the ALD and yet you also discuss a fast/slow path design as
a legitimate implementation of this draft.  It would be good to include the
details of this interaction for general review.

> provided by the framing protocol may be able to implement a different

I have no doubt that this is your intent.  The question remains as to how
many of these details remain proprietary and what will become part of the
public domain.  I am not in favor of creating these application specific
interfaces at physical devices nor do I think this is the only solution.
Again, SCTP does offer a better solution here.  If you wish to further this
effort in transforming TCP into an SCTP lite product, then at least share
the details of the inter-layer signaling together with their related
interfaces to allow the community at large to also benefit.

> > Again, you are making an assumption as to what should be
> > sanctioned by the IETF.
>

That is not true as you will be wishing to change the behavior of the TCP
send as well as stripping out the framing header.  That demands additional
signaling.  In addition, the intended applications will do much much more
than this.

> > Sorry, I still count 4 ways.  You have the IP fragmentation and
> > then three methods to mark such fragmentation within your header.
>

I must note here that SCTP uses both a 32 bit pattern together with a 32 bit
frame check to verify the validity of the frame, yet here there is not
additional error check beyond the 16 bit checksum within TCP.  That appears
to be a serious weakness in this design especially if this moves forward
into higher bandwidth networks.  But I suppose you already have a phalanx of
proprietary solutions to handle this situation.  You may wish to be more
explicit as how a fragment is to be marked to allow greater future
flexibility while at the same time indicating which fields must be checked.

Doug

---

From: Stephen Bailey <[email-masked]>
Date: Thu, 23 Aug 2001 18:18:14 -0400

Doug,

There are many interesting details to making a hardware accelerated
implementation of a ULP running on the framing protocol.  As you (and
the draft) point those details are a function of the specific ULP.  If
you're trying to beat them out of me for a specific ULP, you're trying
to get blood out of a turnip.  Perhaps you already know more about the
details than I do.

The framing protocol by itself communicates one small piece of
information.

If you believe the mechanics of accelerating a particular ULP using
the cues provided by the framing protocol should be presented, you
could always write an ID on the subject.

The ID specifies a protocol, rather than specifying an implementation
of it.

> With your draft the following layers are in play:

It is actually:

          Application
      Upper Layer Protocol
        Framing Protocol
              TCP
              IP
             IPSEC
         Physical Layer

Steph

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 23 Aug 2001 15:50:07 -0700

Stephen,

So you just don't know how this draft is to be used.  That would seem to
include everyone at this point with the exception of David Black.  You wish
to suggest there is no lower layer, required additional inter-layer
signaling, or additional information to be included in this draft.  It is
non-sense to suggest no extra layer is to be the case as you indicated
clearly that this lower layer will be extracting the framing header before
passing it to the ULP.  What magical layer is doing this stripping?  In
addition, the intended applications or ULP repeated throughout this draft
requires considerably more detail than provided here.  I do not wish to beat
any details out of you, however, you should at least admit there is a
substantial omission of information from this draft.  This omission is
alarming for many reasons.

Doug