Subject: RE: [Tsvwg] ULP Framing and IETF process

From: [email-masked]
Date: Wed, 15 Aug 2001 20:04:35 -0400

Doug,

This is becoming pointless, as you keep inventing changes that
aren't required by the draft, and continue to make blatantly
false assertions:

- Data placement is not "iSCSI processing", and it's a very small
piece of an iSCSI implementation.  I challenge you to back your assertion
that placement is "easily a third" of an iSCSI implementation with
a code line count from an open source iSCSI driver.

- TCP/IP would continue to see complete inbound packets as it always
has, they just wouldn't be contiguous in memory.  Nothing novel is needed
to deal with this because TCP/IP stacks dating back to the Berkeley Unix
implementation have been capable of processing data that is discontiguous
in memory and making arbitrary divisions of it to deliver data to the
application or network in the quantities required (e.g., by the size
of application recv/recvfrom calls or network packets).  Different
OSs use different internal data structures to do this - for example,
Berkeley Unix used mbufs, and the IETF has never had any interest in
standardizing this sort of internal OS data structure.  Please take
the time to understand how mbufs and mbuf chains work before making
further erroneous assertions on this topic.

- Raw interfaces to network devices are at least two decades old in
Unix, but not a lot of applications are written to them.  This
draft won't change that situation.

- I stand by the assertion that this draft is not about placing an
RDMA implementation beneath TCP.  The RDMA reference in the draft
indicates that RDMA is a ULP (U = Upper).  Much of your confusion is
based on making the erroneous assumption that RDMA would be placed
below TCP and then inventing all sorts of things that aren't in
the draft to try to clean up the mess that your erroneous assumption
creates.

- I never requested that "the marker section of this draft be removed".
I suggested removing the framing headers from the PDU alignment
specification as a possible way to correct the problem that the draft
currently has with detecting segmentation violations (e.g., courtesy of a
resegmenting middlebox).

- TCP requires only ordered delivery to the application through the
associated API.  Even with a sockets interface, placement of inbound
data in application memory can occur in an order other than the one in
which the bytes came off the wire - the only requirement is that receive
calls to the socket interface complete in the correct order after the
correct data is in memory.

As I've said earlier, this draft is fundamentally about changing the
organization in memory of the data processed by TCP, a matter on which
the TCP RFC is silent, and hence wild assertions of this constituting
a substantial change to TCP are simply not credible.

--David

> -----Original Message-----
> From: Douglas Otis [mailto:[email-masked]]
> Sent: Wednesday, August 15, 2001 4:47 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [Tsvwg] ULP Framing and IETF process
> 
> 
> David,
> 
> Let me try and recap.  Although you admit that there will be additional
> capabilities added to the process you euphemistically call a "packet
> filter", and liken an iSCSI processing of the payload to a simple pattern
> scan, you feel moving this portion of the application below the IP layer
is
> not a concern as it is just an extension of the use of packet filters.
You
> feel the extensive parsing and placement into buffers identified by means
of
> indexes within the ULP PDU does not in itself impact TCP, as TCP will be
> forced to wait until all intervening segments have arrived before
returning
> the receive thread back to the application and hence your assurances of
> sequential delivery.
> 
> One does wonder what TCP will be seeing from this Packet Filter process as
> it will not be intact segments nor even complete sections of segments.
You
> also feel this aspect of the draft is deemed OS dependent and thus not
worth
> describing.  The aspects as to how TCP will track or even note the
location
> of these PDUs is as well OS dependent and thus too warrants no mention.
> Clearly, Microsoft needs no help in this area as you have indicated they
> have already developed these signaling mechanisms.
> 
> In considering the freedom this offers, even multiple connections can be
> combined into a single connection within this packet filter where any
number
> of games can be played.  There are NO rules for packet filters as you have
> conveyed.  If I desired, I could transform these connections into any
> protocol.  As TCP is clumsy using this "packet filter" approach, it is
very
> likely a new OS interface will be developed where capabilities will be
added
> on a daily basis much to be benefit of the illustrious provider.  Here the
> IETF can now safely stand aside and watch the flashing lights as this
packet
> filter technique removes all barriers as the applications sit directly on
> the physical interface.
> 
> The layers are changed as you place easily a third of your iSCSI
application
> into the "packet filter" section of this "TCP stack" that now includes a
> packet filter as an elemental component to its operation.  This packet
> filter must also communicate the buffer identities in an application
> specific manner and find a means of interoperating with a TCP stack that
now
> must both receive the remnants of this highly processed information, but
> must also send at specific boundaries to keep this mess functional.  All
of
> these changes go undocumented.
> 
> As you have also indicated, this packet filter process can go very wrong
and
> have requested that even the marker section of this draft be removed.
Here
> is where you tend to indicate a greater reliance on the ips group to
> correctly write a packet filter process than those writing the one to
> extract just the marker.  What is clearly missing is a means to detect a
> segmentation problem.  In addition, there is no standardization offered by
> any of this activity.  Each application will be writing their own packet
> filter, each will be looking for solutions for this messy error problem
> differently, and worse yet, there will be no common interface exposed to
the
> application for doing this on different operating systems.  It seems like
> the perfect scheme for a company that wishes to leverage their dominance
in
> the market place at the expense of others.
> 
> You have indicated it is okay provided the "wire" specification of TCP is
> not changed.  You have also indicated that direct use of RDMA is not
likely,
> but yet an application indicated by this draft.  You wish to find solace
in
> a claim of sequential delivery, but the application will not see anything
> that remotely looks sequential.  To quote from the draft "While
notification
> from TCP is required to be in-order, this does not prohibit arbitrary
> placement of TCP data received in any order."  This means a new paradigm
of
> handling this disjointed array of receive requests will need special
> handling by the application and is not business as usual.
> 
> http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-00.txt
> 
> TCP is only a "wire" specification?
> 
> Doug
> 
> 
> 
> > > Thank you for the clarification.  Although most packet filters
> > implement a
> > > rather static set of  procedures and do not extensively parse then
> > > separately place data within the packet, this Packet 
> Filter, of which I
> > > think should be called an Application Layer Director to better
> > characterize
> > > its function, brings a tremendous amount of the 
> application into this
> > layer
> > > below IP.
> >
> > That's seriously exaggerated.  Take the Berkeley Packet Filter -
> > one of the older and simpler ones.  It's already capable of 
> doing all of
> > the "extensive parsing" required by the PDU alignment 
> approach (start
> > from front of packet, look for the right pattern of 
> headers, check packet
> > length), as well as copying a packet.  The truncation support would
> > need extensions to do separate placement (in essence the 
> TCP/IP headers
> > go in one buffer and the payload, which is the ULP PDU, goes in the
> > next one).  I wouldn't want to do this with the BPF, and 
> TCP/IP isn't
> > normally stacked on top of a BPF filter, but all of this is 
> possible.
> > The point of this is that the software equivalent of what 
> an optimized
> > implementation will do is consistent with current layering (e.g., an
> > optimized implementation is unlikely to use the BPF 
> instruction formats,
> > and could well put the entire match function into hardware gates).
> >
> > > So much so, that with this dynamic interaction between received
> > > data application specific parsing and placement, 
> transport function
> > > modification, and PDU tracking within the send buffer, 
> this changes TCP
> > > dramatically and obliterates any concept of layering.
> >
> > >From an "on the wire" perspective, this use of a Packet 
> Filter doesn't
> > change
> > TCP, it just changes the way that the data that TCP 
> processes is organized
> > in
> > memory.  As for "obliteration of layering", most of the 
> protocol-specific
> > knowledge (e.g., of iSCSI if one wants to split iSCSI data from
> > its headers)
> > can already be injected using existing packet filters.  Not much
> > is changing
> > from the existing state of things wrt layering.
> >
> > > Whether it is done using a non-blocking technique or with multiple
> > threads,
> > > it heralds network programming beneath IP where data is to be
> > directed to
> > > the associated buffers.
> >
> > Nonsense.  The non-blocking technique or multiple threads get used
> > above the sockets interface.  The packet filter routines 
> are well below
> > the sockets interface and continue to be executed in the 
> fashion they
> > always have where there are neither threads nor calls that could be
> > blocking or non-blocking.  There's no question that the buffer usage
> > below TCP and IP is being changed, but that doesn't change 
> the protocols.
> >
> > > The concern I had with the buffer handling that now takes 
> place between
> > the
> > > ALD/TCP/Application was concerned with an RDMA application.
> >
> > Bad assumption - this is not about doing full-blown RDMA.
> >
> > > I am aware that
> > > SCSI has more limited uses of buffers and that limitation 
> seemed to be
> > > reflected in your response.  As TCP will be blocking the 
> release of a
> > > buffer, the ability to regain access is then beyond the 
> control of the
> > > application.
> >
> > In other words, TCP continues to deliver in-order, contrary 
> to what you've
> > been claiming was the inherent purpose of this draft.  Are 
> we done yet?
> >
> > --David
> >
> > ---------------------------------------------------
> > David L. Black, Senior Technologist
> > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> > [email-masked]       Mobile: +1 (978) 394-7754
> > ---------------------------------------------------
> 

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 15 Aug 2001 18:18:06 -0700

David,

I am well aware of the various buffer techniques.  Some schemes place the
TCP payload as delivered using various understood techniques for handling
differences between the virtual perspective of the processor to that of the
physical perspective of the adapter to allow a zero-copy of TCP as example.
These are not a difficult concepts, but every iota of complexity within this
process is now being used by you to justify or masquerade changes being made
by this draft while you continue to insist this is business as usual.

This process below IP used to examine each PDU, to determine the buffer
related by the application tag together with possible relevant data payloads
placed separately, and to perform the related consistency checks in the
process of populating these myriad of receivers also bearing their
respective tags is not business as usual.  The time spent in this process
sitting between the adapter and IP is very likely to envelope much of the
time and resources spent by an application but why quibble over the exact
percentages.  I will be happy to use your estimates.  Once you relinquish
such a fundamental aspect of an application into a separate complex process,
you need to develop an interface to handle errors that will occur.  The
complexity of this intra-stack communication is non-trivial and not
documented even in a general sense.

Sorry for not using the correct term for the header as I had overlaid the
use of a marker scheme alternative while I was attempting to describing
these headers.  Your attempt to downplay the level of impact this will have
on TCP is understandable as ips is likely to insist on making these changes
as it has done from the very beginning.

Doug