Subject: [rddp] RE: [Tsvwg] Re: "DO IT RIGHT", or "DON'T DO IT AT ALL"

From: [email-masked]
Date: Thu, 12 Dec 2002 10:44:39 -0700

Mark,

Firewalls and middleboxes are certainly not the only reasons. 

Our customers want a card that can connect to existing TCP nodes that don't support RDMA and connect to nodes using RDMA. With an RDMA that is able to run over TCP, we can do that with one TCP offload engine. Having to support two transports to talk to legacy devices and new RDMA devices would be prohibitive. 

There is nothing sleazy about building architectures for graceful migration to new technologies. Those technologies that have ignored migration needs often have gone nowhere.

Regards,
Pat

-----Original Message-----
From: Mark Allman [mailto:[email-masked]]
Sent: Thursday, December 12, 2002 6:15 AM
To: Caitlin Bestler
Cc: [email-masked]; [email-masked]; Joe Touch
Subject: Re: [Tsvwg] Re: "DO IT RIGHT", or "DON'T DO IT AT ALL" 

> Sticking with existing transports is not a sleazy trick to
> evade the oversight of network administrators, rather it is
> avoiding creating unncessary new work for them.

I'd vote for "sleazy trick" myself.  It is nice that you are
thinking of firewall admins needs -- but, I think Joe is right and
the bottom line is that the reason you want to use TCP and not SCTP
for DDP and the reason my IM client uses our HTTP proxy are the
same.  That is, they are rational responses to getting traffic
through firewalls/middleboxes.  I can understand the rational, and
can even sympathize with it on a good day.  But, I still think both
of these examples are architecturally broken.  Both of these tactics
hide one thing in another and obfuscate what is really going
on.

allman

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Lars Eggert <[email-masked]>
Date: Thu, 12 Dec 2002 09:56:46 -0800

[email-masked] wrote:

> Our customers want a card that can connect to existing TCP nodes that 
> don't support RDMA and connect to nodes using RDMA.

Make the new box dual-stack, so old boxes can talk to it using the old 
protocol.

Lars
-- 
Lars Eggert <[email-masked]>           USC Information Sciences Institute

---

From: Joe Touch <[email-masked]>
Date: Thu, 12 Dec 2002 09:57:19 -0800

> Make the new box dual-stack, so old boxes can talk to it using the old 

FWIW, these proposals are basically "dual-stack in wolves clothing". 
They're dual stack buried deeply inside a single stack, IMO, for purely 
marketing purposes.

Joe

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 12 Dec 2002 12:22:44 -0600

On 12/12/02, [email-masked] wrote:

>Our customers want a card that can connect to existing TCP
>nodes that don't support RDMA and connect to nodes using
>RDMA. With an RDMA that is able to run over TCP, we can do
>that with one TCP offload engine. Having to support two
>transports to talk to legacy devices and new RDMA devices
>would be prohibitive. 
>
I think you are overestimating the work involved in
supporting SCTP. It co-exists with TCP and UDP, and the bulk
of any DDP solution has nothing to do with the LLP.

Valid reasons beyond firewalls and middleboxes do include
lack of tools, including testing, to fully support SCTP.
That is a legitimate factor for both development and
deployment. But I wouldn't endorse any characterization that
SCTP support is a prohibitively expensive option for an RDDP
NIC. In fact, I can think of a few low-end designs that rely
more heavily on the host software stack that are possible
with SCTP that are not possible with TCP.

>There is nothing sleazy about building architectures for
>graceful migration to new technologies. Those technologies
>that have ignored migration needs often have gone nowhere.
>
Exactly.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 12 Dec 2002 21:41:07 -0600

On 12/12/02, Brian F. G. Bidulock wrote:

>Sam,
>
>Because TCP can't be extended to do the things that SCTP
>does and still remain TCP.
>
When RPC is run over TCP, you can look at the TCP stream and
understand it in TCP terms without understanding RPC. It is
a valid TCP stream. But RPC does things that TCP cannot.

All of that applies to MPA as well.

Does that mean that RPC should have been its own transport?

How is DDP over MPA any more of an "assault" against TCP
than DDP is to SCTP?

Is this really all because the data is placed in the correct
before "early"? How does the receiver "owe" the sender the
knowledge that the data is copied at least once?

Do zero-copy stacks that deliver pointers to the data where
it was received violate these alledged TCP semantics?

Or is it only a violation because we only have the ULP's
word that it isn't looking at the data early? In that case,
are all kernel mode daemons in non-compliance?

Hint: if it is impossible to detect that there is anything
invalid about the connection on the wire then just maybe
there isn't anything wrong it.

Caitlin Bestler

---

From: Joe Touch <[email-masked]>
Date: Thu, 12 Dec 2002 21:03:20 -0800

Brian F. G. Bidulock wrote:
> Sam,
> 
> On Thu, 12 Dec 2002, Sam Liang wrote:
> 
> 
>>>Sam,
>>>
>>>Because TCP can't be extended to do the things that SCTP does and
>>>still remain TCP.
>>
>>Brian,
>>
>>   So what are the things TCP can't do, can't be easily added to TCP, and
>>majority of the applications are dying for?
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> - Multihoming.

Which applications are dying for this?

> - Partially ordered data.

Ditto? This is often an artifact of running multiple streams over a 
single TCP connection unnecessarily, and often inappropriately.

> - Record delimitation

See partial ordering.

> - SACK (but that was shoe-horned into TCP with much effort)

Hmm. I.e., it's done.

> - Partial Reliabiity

Exactly what applications want security (below) and partial reliability? 
What is the utility of getting data that's busted?

> - 4-way connection establishment for security

Perhaps... - though we do already have SSL, so what's the problem? What 
apps aren't happy with SSL, and need this?

> - Rapid orderly shutdown and restart without timed-wait or half-open socket hurdles

TIME-WAIT doesn't affect new connections, only the reuse of a single 
pair of port numbers.

> - Multiple streams without head-of-line blocking

Open one TCP per stream. Let IP do the muxing it was designed to do, and 
use TCB sharing ;-) to coordinate the state of the streams. All within 
the definition of existing TCP.

Most of these are contrived problems, or misapplication of TCP. That 
said, as before, there at least _is_ already an alternative - SCTP, that 
examines this space, and is just as easy to deploy and port than the 
numerous private stacks masquerading as 'TCP'.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 12 Dec 2002 14:19:22 -0800

[email-masked] wrote:
 > don't support RDMA and connect to nodes using RDMA. With an RDMA
 > that is able to run over TCP, we can do that with one TCP
 > offload engine. Having to support two transports to talk
 > to legacy devices and new RDMA devices would be prohibitive.

Good reasons why your box should, and in fact, must speak existing TCP 
anyway. Thus the two stacks - one for RDMA, one for TCP.

You still need to negotiate whether you're using this additional layer 
anyway - a legacy box won't insert or delete the extra headers your 
using, so you still need to speak vanilla TCP at the end of the day.

That's two stacks - one with the layer, one without. Given the still 
plummeting cost of RAM, are you saying this is such an urgent interim 
solution (until it fits in RAM available next week) that the IETF needs 
to work on it?

If so, then by definition, it is a stop-gap measure...

 > that have ignored migration needs often have gone nowhere.

On that we hopefully all agree. What we disagree on is whether wedging a 
new protocol on top of TCP in ways that emulate having a separate stack 
are graceful, or migratable in ways that using a separate protocol are not.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 12 Dec 2002 21:22:08 -0700

Caitlin,

> All of that applies to MPA as well.

It think it is the case of contention that it does not.  That,
unlike RPC, MPA reaches down into TCP and requires a different
semantic from it: reordering how TCP delivers segments to MPA.

RPC, doing no such thing, runs equally well over UDP, TCP
and, yes, even SCTP, because it does not break this semantic.
CORBA orbs are now running over SCTP, when they ran over UDP
and TCP because they do not break this semantic.

> Does that mean that RPC should have been its own transport?

The argument is that because MPA changes TCP, that it is
ipso facto its own transport.

> How is DDP over MPA any more of an "assault" against TCP

Because TCP knows nothing of records, and must be forced to
to handle DDP.  SCTP already has a concept of records and
performs reassembly of records.  In fact DDP does not have
to run over SCTP: SCTP can do what DDP users expect (direct
placement into user memory), particularly as SCTP does not
constrain the ULP interface in the same manner that the TCP
specifications do.

> it was received violate these alledged TCP semantics?

The only zero-copy (no copy from kernel to user) stacks that
I have seen are for ATM and Alan Cox's recent send_pages
addition to Linux NET4 TCP (which, by the way, uses a
non-standard API and system call).

> there isn't anything wrong it.

I'm afraid you'll have to take that up with those opposing
it on the basis of breaking TCP API semantics.  It seems from
previous discussions in the WG that zero-copy and asynchronous
I/O are supported, but that order of delivery is strict.  One
might have to lock the memory away from the user before
notification of a complete buffer to avoid violating the RFC.
(Actually the Linux send_pages does this, because it transfers
one VM page at a time and can lock the page away from the
user before SIGIO.)

Another approach might be to have a new sendmsg (like sendpages)
return the pointer to a user-space buffer that than taking a
pointer to a user-space buffer.  Then it is not possible for
the user to access the buffer area before the reassembly is
complete.

I don't think this was the only problem, however, the problem
how to do realignment when a header is missing, and "changing"
TCP (the internals, not just the API) to accomplish that.

SCTP already does record realignment internally, and as long
as a DDP frame is a SCTP record you should be O.K. to start.
A minor modification (discussed much about a year ago) which
indicates a fragement's position in the overall record added
to the DATA chunk in SCTP would not only completely support
DDP, but would provide SCTP with some other efficiencies in
handling large records that have gaps.

SCTP is already super-TCP, has more opensource implementations
than TCP, and runs on all manner of architectures, and even
has TCP-compatibility to the ULP built in.  There's nothing that
you can run over TCP that you can't run over SCTP (although
the reverse is not true, there are applications that can only
run over SCTP).

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Sam Liang <[email-masked]>
Date: Thu, 12 Dec 2002 19:11:53 -0800 (PST)

Brian,

Sam

> >   It sounds like that eventually we'll all migrate to SCTP.  Maybe SCTP
> > can do a couple things TCP doesn't do currently. But when TCP can already do
> > 95% of the job, what's the compelling reason for everybody to move to SCTP?
> > 
> >   To me, for the really appealling functionalities, we can extend TCP to
> > support them. It's much more deployable than attempting to implement a
> > brand new protocol, which basically does the same thing as TCP except for
> > being much more cumbersome, in every operating system. There is no need to
> > reinvent the wheel, when the current wheel rolls fine.
> > 
> > Sam
> > 
> > > 
> > > 
> > > > There is nothing sleazy about building architectures for graceful
> > > > migration to new technologies. Those technologies that have
> > > > ignored migration needs often have gone nowhere.
> > > 
> > > I agree.  Poor copying of words, maybe.  I completely agree with
> > > your statement above. 
> > > 
> > > However, that does not necessarily mean that I agree with the
> > > approach to DDP / TCP.  My high order bit is that migration is
> > > certainly one aspect of engineering that one would do good to think
> > > about.  But, it is just one item.  Considering the architectural
> > > impact of the solution is also quite important.  
> > > 
> > > That is, in a very general sense there are lots of really bad ideas
> > > running around out there that come from people trying to solve a
> > > problem with a point engineering solution.  One of the functions of
> > > this body (the IETF) is to make sure we're doing things in a sound
> > > way.
> > > 
> > > My preference is SCTP.  I sympathize with the problem of timliness,
> > > though.  But, we have sort of a chicken and egg problem here.  If we
> > > hack at TCP to make it do DDP will we ever migrate to SCTP?  If we
> > > do that with all sorts of things that SCTP might be a good match for
> > > will SCTP ever actually mature so that we can migrate to it?  I.e.,
> > > why push it along and mature it if folks just hack around it because
> > > it hasn't been pushed along?  These are not easy questions...
> > > 
> > > allman
> > > 
> > > 
> > > --

---

From: Joe Touch <[email-masked]>
Date: Thu, 12 Dec 2002 21:12:36 -0800

> Does that mean that RPC should have been its own transport?

RPC never called itself TCP.

> knowledge that the data is copied at least once?

Zero copy isn't the issue; availability of the data to the layer above 
TCP is. Delivering data to _ANY_ layer above TCP before all data before 
it has been received correctly is a violation of TCP semantics.

> it was received violate these alledged TCP semantics?

No. And the semantics are spelled out in RFC793. Perhaps they're being 
ignored and misunderstood (in some of these discussions), but they are 
far from alleged.

> word that it isn't looking at the data early? 

Yes.

 > In that case,

No. They don't get the data from TCP - 'released' by TCP - or whatever 
calls itself 'TCP' - before all data before it has been received in order.

> there isn't anything wrong it.
Hint: please read RFC793. Please check, especially, the rules for SEND 
and RECEIVE, and rules for when data can be made available.

If there's something wrong with _that_, it still isn't TCP.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 12 Dec 2002 13:37:31 -0700

Caitlin,

> lack of tools, including testing, to fully support SCTP.

Bull.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 12 Dec 2002 20:31:48 -0700

Sam,

- Multihoming.

Perhaps you should read the SCTP applicability statement? ;)

--brian

---

From: Mark Allman <[email-masked]>
Date: Thu, 12 Dec 2002 16:52:00 -0500

allman

---

From: Sam Liang <[email-masked]>
Date: Thu, 12 Dec 2002 15:05:57 -0800 (PST)

Sam

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 12 Dec 2002 16:12:30 -0700

Sam,

--brian

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 13 Dec 2002 11:17:33 -0600

On 12/12/02, Joe Touch wrote:

>> Hint: if it is impossible to detect that there is
>> anything invalid about the connection on the wire then
>> just maybe there isn't anything wrong it.

>Hint: please read RFC793. Please check, especially, the
>rules for SEND and RECEIVE, and rules for when data can be
>made available.
>

An optimal interface supporting MPA will make validated TCP
segments available for *placement* to MPA out-of-order, but
it does not *deliver* them early. The data is not
"available".

This is merely co-operative buffer management -- something
already done in existing stacks.

Yes, the placement is based upon the contents of the packet.
But if that is a violation of the rules, consider that the
following would also be illegal:

- placing the data from TCP segment into a reassembly buffer
  at the correct offset for the specific connection.
  
- Redirecting an IP packet based upon a TCP or UDP port
  even when the IP packet is received out of order.
 
Beyond the semantics games, it is important to remember
the following important things about MPA:

- It is compatible with *ALL* existing network equipment.

- It is compatible with *ALL* existing TCP stacks. The
  optimizing "placement" interface is an optional
  enhancement. The protocol works without it.
  
- It is compatible with *ALL* network traffic shaping,
  monitoring, firewalling and policy controls.
  
Maintaining compatability with existing networking equipment
and software is a vital goal. Maintaining compatability with
some interpretations of prior documents is not a goal that
justifies forbidding optimizations that let processors
accomplish more work with the same resources.

Caitlin Bestler

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 13 Dec 2002 01:44:12 -0700

Joe,

It sounds like you think that SCTP was (is) unnecessary.

That could not be farther from the truth.  I suggest you research
the topic a bit more.  You will find that SCTP addresses many
deficiencies in TCP that make TCP unsuitable.

Check out the sctp.org or sctp.de sites.  You will find there
a number or papers that do not agree with you.  In fact the TSVWG
does not agree with you, otherwise it would not waste its time
on SCTP.

I hear Randall's book is a good buy.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 13 Dec 2002 11:13:26 -0600

...

Joe, 

This is off-topic for this thread. But as a quick example, multi-homing is a
critical feature needed by sigtran. Without it, people there would have a hard
time to have any confidence in achieving high system availability, which is
fundamental to most telecom applications.

-Qiaobing

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 07:52:41 -0800

> It sounds like you think that SCTP was (is) unnecessary.

I still do. The problem is that - in the debate on telling RDDP why not 
to muck with TCP, the question of _need_ arose.

I was pointing out that - as I have observed years ago when SCTP began, 
the majority of reasons are based on misapplication of TCP, rather than 
real need. Observing that TSVWG doesn't agree with me doesn't cut it - 
the IAB also acknowledged NATs, and as per the last plenary, some are 
starting to finally recognize the impact of that decision.

Sure, there's a 'field of dreams' approach, but that wasn't the topic.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 09:51:26 -0800

> fundamental to most telecom applications.

I showed how to do this years ago, using exiting routing protocols over 
a set of tunnels. That decouples the functions of tunneling and routing 
from the transport protocol.

See:

  "Dynamic Host Routing for Production Use of Developmental Networks"

J. Touch and T. Faber, Proc. ICNP '97, Atlanta, Oct. 1997, pp. 285-292.
http://www/touch/pubs/icnp97/

This is an example where existing stuff already works; there are others 
where there are a lack of apps. As you observed, this is off-topic for 
this thread, and FWIW was raised many years ago.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 13 Dec 2002 12:48:25 -0700

Joe,

> this thread, and FWIW was raised many years ago.

Saw that years ago too, and dismissed it largely out of hand because
it was too difficult to manage.  I don't recall anyone but you supporting
the approach.  Also, it did not solve HOL blocking (without having
to open 16000 TCP connections, which is also unmanageable).  Using TCP
it could not quickly detect loss of the connection (minutes in TCP),
also for fixed port numbers on either host, timed wait prevents connections
from restarting quickly (SCTP can restart a connection even as it is
shutting down), nor did it address record delimitation.

So I suppose the answer is that not only did applications want multihoming,
they wanted other features of which TCP was not capable because of its
legacy.  Gee, kinda what DDP/MPA is faced with now. ;)

--brian

---

From: Randall Stewart <[email-masked]>
Date: Fri, 13 Dec 2002 13:59:22 -0600

Joe:

I have attempted to not get sucked into this thread.. it seems
like a rat-hole but alas I feel I must comment on this... I realize
that you and I "agree to disagree" on the value of SCTP.. but it
is sad in my mind that you can't see any value in the
protocol...

comments inter-spersed below...

>>> majority of the applications are dying for?

I think there are several applications that want this.. signaling for
one.. and yes you can setup a "local" router and use routing
protocols to do this..as you have shown in past work.. The thing
the routing protocol is NOT going to get for you is a faster
failover... if I have a primary and a backup in SCTP at the
first RTO I retransmit to the alternate... for a call control
application (among, I am sure, others) this multi-second delay
for the routing protocols to converge is not acceptable...

> single TCP connection unnecessarily, and often inappropriately.
But there are still many applications that want this. HTTP itself
likes this type of behavior.. aka I have N pictures and X text to
send down to a client.. I don't want to create N+X connections but
I sure would like to send things in a round-robin fashion amongst
the images to let the user preceive things are happening...

And of course call control itself.. you would not want N tcp connections
for all the IAM messages progressing between two call control
gateways... partial ordering is a very important feature..

> See partial ordering.
But this very record deliniation is one of the big things TCP lacks for
the RDDP folks... I don't understand your argument here...

> What is the utility of getting data that's busted?

It is not so much data that is busted... if Qiaobing and I would have
had this idea earlier the PR draft would be in the base spec... we
tried several times to get this behavior during SCTP's developement.

In terms of signaling.. think of a call setup message traversing
through a call control MG.. the IAM message is sent from the
source (where the call is started) and around 3 seconds (I
can't remember the exact number) if no response comes back the
call is torn down. If I remember correctly a intervening node
that the IAM transits through has a strict time budget that
at a certian point the message just needs to be discarded...

PR-SCTP fits this model and as I said would have been in the
base spec if we had had the idea about 4 weeks before we
did.. sigh. This is where the u-sctp/pr-sctp drafts came
from...

Another application of the above is in ipfix. Here we have
a router sending a mix of peg counts and possibly other
billing information to a collector. During non-busy times
I would like everything to get through... but if the router
is busy maybe I only want to make limited attempts at sending
the peg counts but I want the other information to go completly
reliable.. all of this based on how much buffer space I have
exhausted... PR-SCTP fits this bill perfectly...

There are other applications in the gaming area where I might want
to send a "character stat" but then at some point I, the user, change
the character... If I have not completed sending the first c-stat I
would just prefer to skip by it and replace it with my new send..
again PR-SCTP fits this model.. It is not a question of broken data
but instead data that is no longer useful since things have changed..

The apps out there are currently doing funny things to get around these
shortcomings... and yet just like RDDP could use SCTP so could
these apps use PR-SCTP...

> apps aren't happy with SSL, and need this?

I think apps do like SSL but does SSL help you with the SYN attack? Yes
we have a limited cookie mechanism we can use that some have
implemented for TCP but it is not quite the same thing...

> TIME-WAIT doesn't affect new connections, only the reuse of a single 

I can't tell you how many times the above has impacted me during demo's
and other fun times when I needed to get that server restarted but it
could not due to the TIME-WAIT issue. I can even remember rebooting
a machine to get around this issue... it is a problem.. maybe not a
glaring problem.. but it does at times impact folks..

> the definition of existing TCP.

Hmm.. it seems we used to do that with HTTP .. crank up the number of
connections that are allowed... but this caused problems with all
the tcp connections that do NOT and probably will not (anytime soon)
have shared congestion state...

> Most of these are contrived problems, or misapplication of TCP. 

No I don't think the problems are contrived.. it could be you have
never been impacted by these issues.. I among alot of other folks
have... this is the basis for SCTP... I am quite sorry you
feel that the problems are "contrived".

>That 
I agree with you on this last point... SCTP can be deployed as easy as
hacking up a "second" TCP stack pretending to be TCP...

It is just a shame that you can't see the other values of SCTP..

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 13 Dec 2002 15:23:51 -0700

Joe,

On Fri, 13 Dec 2002, Joe Touch wrote:

> Qiaobing Xie wrote:
> > Joe, 
> > 
> > So, you did see the need for multihoming years ago. You only disagree that SCTP is
> > the right solution.
> 
> Yup.
> 
> > BTW, the URL you provided has an illegal syntax :-)
> 
> Drop the trailing slash: http://www/touch/pubs/icnp97
                                  ^^^

Aren't you missing a usable host name here?

> (that's my typing habit)

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 13:42:51 -0800

> is sad in my mind that you can't see any value in the

Let me be clear on my position on SCTP, FWIW:

- most of the protocol tries to solve problems that I believe result 
from misuse of TCP, or desired misuse of a single connection to do more 
than provide for a single end-to-end channel.

- I'm not clear that the other issues it tries to solve are indeed 
driven by application need. More like "build it first, then find a use", 
IMO.

That said, I certainly think that the IETF _should_ be the place where 
new protocols are developed, and that new protocols _should_ have new 
transport identifiers. In both those ways, SCTP is the right thing to do.

...
> for the routing protocols to converge is not acceptable...

Routing protocols get the failover you configure them for. Given 
existing RIP, sure, it's 6 seconds. however, since it's just running 
between the two endpoints, that constant can be modified as desired.

We're not talking about widescale convergence; just the two endpoints.

> the images to let the user preceive things are happening...

HTTP wants this because it wants to use 1 connection for multiple 
streams. Not "wanting" to create those connections, we end up with 
complex consequences, for the sake of merely 'want'.

> gateways... partial ordering is a very important feature..

There are two cases:
	- use total order, and show that the reordering delay
	is unacceptable

	- use separate connections

Most of the arguments I've heard here and thus far are for using IP for 
"fast connect call management" - since nothing in IP has timing 
guarantees, I'm not clear why TCP needs to be fixed _first_.

> the RDDP folks... I don't understand your argument here...

Show me the application that needs this, in ways that aren't provided by 
a layer on top of TCP. Record delineation is neat, and works just fine 
over TCP. TCP itself need not provide it.

>> Exactly what applications want security (below) and partial 
>> reliability? What is the utility of getting data that's busted?
> 

Again, we're back to the fact that IP is 'best effort', not realtime.

...
> exhausted... PR-SCTP fits this bill perfectly...

Just make the peg count TCP backoff. Unreliable information isn't going 
to be useful, notably for accounting.

> but instead data that is no longer useful since things have changed..

How does delivering broken data help in that case?

>> Perhaps... - though we do already have SSL, so what's the problem? 
>> What apps aren't happy with SSL, and need this?
> 

Not quite, but might be close enough.

> glaring problem.. but it does at times impact folks..

Fix your server and/or OS. TIMEWAIT prevents talking to people reusing 
old ports; your OS is preventing you from listening because you have 
time-waits. I.e., port use fixes this; part of your system is cutting 
corners to not have to keep track of the ports properly.

>> Open one TCP per stream. Let IP do the muxing it was designed to do, 
>> and use TCB sharing ;-) to coordinate the state of the streams. All 
>> within the definition of existing TCP.
> 

That's an implementation issue; bad implementations do bad things. A bad 
SCTP implementation can too.

> feel that the problems are "contrived".

I've been impacted by the issues, and solved them other ways, or 
realized that IP wasn't realtime and thus was the wrong hammer for the 
nail in question.

>> That said, as before, there at least _is_ already an alternative - 
>> SCTP, that examines this space, and is just as easy to deploy and port 
>> than the numerous private stacks masquerading as 'TCP'.
>>

Whether I see the values or not (and I still don't) doesn't make me 
thing SCTP should go away. It's fine to have other protocol designs, 
some win, some lose, and some never convince anybody.

ATM convinced a lot of people, including a lot of consortia and 
standards bodies, and a lot of papers were written on it. Others kept 
asking what problem it solved, and why it was needed. And it's waning.

I stand by my assessment of SCTP, and maybe it'll be wrong. Maybe it'll 
be right. But either way, I respect doing SCTP as a separate stack.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 13:45:33 -0800

Yup.

(that's my typing habit)

Joe

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 13 Dec 2002 15:25:12 -0600

Joe, 

-Qiaobing

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 15:05:36 -0800

> Aren't you missing a usable host name here?

http://www.isi.edu/touch/pubs/icnp97

Yikes - cut and paste, even from existing apps, dropped the domain ;-(

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 17:05:10 -0800

...
>>>I don't recall anyone but you supporting the approach.  
>>
>>Yeah, well, if popular votes governed things, we'd still be teaching 
>>that the earth was the center of the universe. I.e., majority rule 
>>doesn't always indicate correctness.
> 
> But concensus is the IETF way.  (But not necessarily mine: Check my sig.)

Agreed. But consensus doesn't mean we all have to have a party and like 
it either. I remain opposed, but that doesn't mean (as I stated before) 
that I think this is a bad 'experiment' per se.

>>>Also, it did not solve HOL blocking (without having
>>>to open 16000 TCP connections, which is also unmanageable). 
>>
>>Why? That's just fine, IMO. Again, doing otherwise buries most of the 
>>same 'unmanageablility' inside the transport protocol, in ways that are 
>>very difficult to resolve with other schedulers in the endsystem. I.e., 
>>doing muxing inside TCP is layered muxing, since we already mux in IP. 
> 
> Because a TCP socket is heavy-weigth structure, particularly to support the
> semantics of the RFC that you cherish so dearly.  I think Randall said that he
> had stream overhead down to about 8 bytes, I have about 16 or so.  The
> overheads for a full TCP socket are much larger by comparison, particularly
> when one considers that each requires an inode, process file descriptor,
> asynchronous notification structures, etc.  Not to mention the overheads of
> poll() or select() with descriptor lists that are 16000 file descriptors long.
> SCTP is a far more practical approach.

Memory is cheap. Complex protocols are not, but again, that's part of 
the experiment...

> Also, there is the fact that Randall mentioned that in SCTP all 16000 streams
> would share a common instance of the congestion algorithm, which is far more
> stable that 16000 separate instances.

That's an implementation issue. Either 2140-style sharing or CM can 
avoid that, depending on the implementation.

>>And I thought we all knew why layered muxing was bad.
> 
> Please educate me.

D. Tennenhouse, "Layered Multiplexing Considered Harmful", Protocols for
High-Speed Networks, 1989.

>>>Using TCP
>>1) connection loss hides at the transport level what you really want, 
>>IMO - link loss detection. That can happen at whatever timescale you 
>>want in the tunnels.
> 
> But was not presented in a framework manageable to typical transport
> applications, without yet another layer above TCP (of course making it
> Un-TCP).

It's below TCP, not above, in our paper.

>>2) timewait prevents reuse of existing ports, not new connections.
> 
> Yes.  Applications using SCTP (and is allowed for TCP too: for example, the
> FTP port command) often use fixed port allocations for connections.

Allowed, yes. And you get what you'd expect.

> Timed
> wait prevents these fixed port allocation from being reused because TCP is
> incapable of handling restarting connections.  SCTP is capable of doing
> because of its use of verification tags with conections and the tie-tag
> mechanism in the cookie (and its 4-way handshake).  Therfore, SCTP fixed port
> connections can restart immediately after failure due to short-lived
> annomalies.

I.e., a complex protocol mechanism to solve a user error, IMO.

>>Record delineation is handled in other systems at a layer above TCP.
> 
> Gee, Joe, I thought that was the whole debate about DDP and framing.  That is,
> that it can't be done at a layer above TCP without breaking the semantics of
> the TCP API, and can't be done inside TCP without making it Un-TCP.

Actually, framing is just fine. Delivering the data out of order is the 
problem. And calling it TCP. TCP is an ordered byte stream.

 > Well,
> because SCTP has record delimitation inside SCTP, it can do it quite nicely.
> 
> And I suppose that's the point here: the very application which is demanding
> capabilities of which TCP is not capable has a home in SCTP.

If it just wants records it could have a home in any protocol above TCP 
that provides record delineation above TCP. That's fairly simple, and 
need not require transport-layer support.

> Gee, I suppose there are applications after all that need SCTP due to
> deficiencies/limitations/legacy in TCP that have nothing to due with misusing
> it.

They may need more than TCP, but not necessarily at the transport layer.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 13 Dec 2002 18:31:43 -0700

Joe,

> Memory is cheap. Complex protocols are not, but again, that's part of 

Actually we have found with SCTP that the approach result in a reduction
in both complexity and memory.  An this is no so suprising: the most complex
of protocols consume vast amounts of memory.  The same is true of other
procecessing power.

> avoid that, depending on the implementation.

Valuable learnings adopted from SCTP.

> High-Speed Networks, 1989.

I suppose you figure the whole ATM network is a big mistake then.  Too bad
there was such inadequate and uneducated engineers that built it. ;)

> It's below TCP, not above, in our paper.

And that's precisely why a typical application cannot manage it, because
TCP provides no mechanism for its management.  SCTP, on the other hand,
does provide mechanisms for management.  Also, in SCTP, these functions
sit under SCTP and are mostly at the IP level.  The difficulty in
implementing SCTP with raw sockets, or over UDP, exemplified this.

> Allowed, yes. And you get what you'd expect.

Yes, in SCTP you get what you expect: immediate restart.  In TCP you don't.

> I.e., a complex protocol mechanism to solve a user error, IMO.

It is a user error to close and reopen a connection?  Where in the TCP
RFC is that discouraged?

> problem. And calling it TCP. TCP is an ordered byte stream.

I understand it is not.  Framing information cannot be placed in every TCP
segment without changing TCP.  If framing information is not placed in every
segment, loss of the segment contiaining the framing information defeats
the purpose of DDP because buffering must be used for reassembly to recover
the framing information.

This is by very nature of the fact that TCP has no concept of records.
Because SCTP does, and does not have a legacy preventing changes to things
like TCP segmentation, the it is easy to place framing information in every
DATA chunk.  The receiver then never has to buffer DATA chunks for reassembly
and can direct place every DATA chunk.

Also SCTP has no restriction on the API about "order of delivery" to the ULP.
There is nothing in RFC 2960 that requires that the data be strictly in order
before it is made available to the ULP.  In fact, in SCTP the ULP interface is
more of an example, and not a requirement.  Again, SCTP does not suffer TCP's
legacy.

> need not require transport-layer support.

It cannot when the purpose of record delimitation is direct data placement.

> They may need more than TCP, but not necessarily at the transport layer.

Yes, when the application layer cannot suffice as described above.

--brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 15:19:58 -0800

...
> it was too difficult to manage. 

Less difficult than burying the same stuff inside a transport protocol.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 13 Dec 2002 17:54:47 -0700

Joe,

Please educate me.

FTP port command) often use fixed port allocations for connections.  Timed

Gee, Joe, I thought that was the whole debate about DDP and framing.  That is,
that it can't be done at a layer above TCP without breaking the semantics of
the TCP API, and can't be done inside TCP without making it Un-TCP.  Well,
because SCTP has record delimitation inside SCTP, it can do it quite nicely.

--brian

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 14 Dec 2002 10:30:34 -0600

On 12/13/02, Joe Touch wrote:

>If the data exits TCP, it is delivered. If it is delivered
>out of order, it is a violation of the spec.
>

No TCP sub-system that I am aware of has control over the
processor's mapping registers. Therefore TCP is always
placing data into buffers which others have potential
control over.

The ULP has no authority to examine or manipulate the
buffers until DDP has delivered them. DDP cannot deliver
them until the LLP does so.

>Inside the stack, yes. But not between TCP and something
outside TCP.
>
>> Yes, the placement is based upon the contents of the
>> packet. But if that is a violation of the rules,
>> consider that the following would also be illegal:
>> 
>> - placing the data from TCP segment into a reassembly
>> buffer at the correct offset for the specific connection.
>
>No - because it hasn't left TCP yet.
>
But under your logic it would be illegal for IP to perform
its buffering based upon the TCP header.

Compressing layers within an implementation has *always*
been legal, as long as it had no effect on the peer or the
network. Why is this different?

>Redirection is an IP processing step; that's routing, not
TCP processing.
But the redirection is based upon TCP data, under the
standards you are suggesting performing L3 routing based
upon L4 data should be illegal. Given that mosts hosts that
use the Internet are behind some form of firewall and/or NAT
that would strike me as a rather extreme position.

>What about all APPS?
Huh???

Neither DDP nor MPA can be slipped in under an App without
its permission. Anymore than RPC or HTTP could just suddenly
be slipped in without the app looking.

Both DDP and MPA allow existing IP traffic to be routed to
existing applications without any interruption.

Caitlin Bestler

---

From: Randall Stewart <[email-masked]>
Date: Sun, 15 Dec 2002 05:59:22 -0600

Caitlin:

I will let Joe comment on most of this but I do
have one little comment to this :->

Caitlin Bestler wrote:

> TCP.
> 
>>>An optimal interface supporting MPA will make validated
>>
> TCP
> 
>>>segments available for *placement* to MPA out-of-order,
>>
> but
> 

I am going to speak from a decidedly BSD slant (since that is what
I have been working with for quite some time now) but I do
think it applys to other O/S's...

1) I don't see your statement about mapping registers as relevant or
    even correct for that matter. TCP (or SCTP for that matter) allocates
    mbufs to hold data (skbuf's in linux and I am sure similar things
    in other OS's). The mapping registers themselves are only setup once
    at the very initialization of the O/S aka allocate bytes to be used
    by the mbufs.. and we are done (some o/s's may do a lazy allocation
    of this and allocate on demand .. it probably vary's). Once the
    memory is allocated (either lazy or at init) it stays in the form of
    mbufs until shutdown... A transport stack allocates a mbuf uses it
    with no one else knowing what it is doing with it. I don't see how
    mapping registers are even germain to this... I will say that since
    the stack is "in control" it COULD fiddle with the mapping registers
    for its memory (if it wanted) but I would consider this more buggy
    behavior.

2) At any one time only one entity, aka the transport stack, knows about
    the mbuf's it is fiddling with. Yes, I suppose some other sub-system
    could have an old pointer to a mbuf.. but this is a buf not a
    feature In BSD they hang around on a socket
    buf and/or ordering queues...It is only when the transport stack is
    truely done with them does it shove them up the socket buffer and
    make them available....This I think is the key point Joe is getting
    at... there is no need for another layer (MPA) to know about the
    data until the TCP processing is complete and hand it off to MPA.
    (Joe am I right here?)...

R

---

From: Randall Stewart <[email-masked]>
Date: Sun, 15 Dec 2002 20:04:03 -0600

Brian F. G. Bidulock wrote:
> Randall,
> 
> Linux has had a pskbuf (paged skbuf) for some time now (2.3, 2.4).  It handles
> sending virtual memory pages which are mapped into user space, using sendpage.
> 2.4 linux only uses this technique for zero-copy send (not receive, as DDP
> would do).
> 
> In principle there is no reason why TCP cannot peform VM page locking.
> 
> I seem to recall that mbuf clusters performs a similar function in BSD, where
> the pages in the cluster are allocated external to the mbuf.  At least a
> simiar zero-copy approach was used in FreeBSD ATM stack to move user-space
> buffers directly to and from the SAR cell processor.
> 

Yep... the mbuf is a seperate allocation external to the mbuf.. but it
does normally come from a pool of clusters... there is no reason
why this pool could not be some sort of VM/zero copy technique... and
of course one would need to do VM page locking.. but it can be done..
Not that it normally is ...

R

> --brian
> 
> 
> On Sun, 15 Dec 2002, Randall Stewart wrote:
> 
> 
>>Caitlin:
>>

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 23:43:44 -0800

> Valuable learnings adopted from SCTP.

Sigtran's first meeting: Dec 98
SCTP first mentioned: Jul 99

RFC2140 date of publication: April 1997

Nothing about 2140 had anything to do with SCTP, which didn't exist at 
the time.

> there was such inadequate and uneducated engineers that built it. ;)

ATM has basically disappeared, except for provisioning. It was, at the 
beginning, sold as a replacement to IP. Whether it was an error or just 
a failed attempt is a matter of interpretation.

> does provide mechanisms for management. 

Just because it's below TCP doesn't mean it's not controllable.

While I'll certainly concede that SCTP has already developed an API to 
do this, there's no reason a similar API can't configure the tunnel or 
routing protocol over which TCP run. Just because TCP runs over it 
doesn't mean it cannot be configured or managed by the app.

> Also, in SCTP, these functions

In our system they sat at IP as well, and no raw sockets were required.

> Yes, in SCTP you get what you expect: immediate restart.  In TCP you don't.

OK - so we have achieved a great result. A huge, complex system designed 
to relieve the programmer from making a _known_ mistake. Hardly a sales 
point.

> It is a user error to close and reopen a connection?  Where in the TCP

It is a user error - or at least outside convention - to set the source 
port number. Convention is to set the destination. Had the user done 
just that, it'd work just fine.

> segment without changing TCP.

I never said placing it in every segment. You wanted framing; there's no 
reason that can't occur.

> If framing information is not placed in every

You wanted framing. DDP wants framing AND to place the data based on 
that framing; i.e., it wants (needs) per segment framing. That's not the 
only reason for wanting framing. I said you can have framing. I didn't 
say it'd help DDP.

> It cannot when the purpose of record delimitation is direct data placement.

Agreed. As per above, framing per se has nothing to do with DDR.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 Dec 2002 23:49:39 -0800

> it does not *deliver* them early. The data is not

If the data exits TCP, it is delivered. If it is delivered out of order, 
it is a violation of the spec.

> already done in existing stacks.

Inside the stack, yes. But not between TCP and something outside TCP.

>   even when the IP packet is received out of order.

Redirection is an IP processing step; that's routing, not TCP processing.

What about all APPS?

> accomplish more work with the same resources.

What if there are apps that depend on those interpretations? That's what 
we're talking about here - the apps that will break.

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 15 Dec 2002 10:54:51 -0700

Randall,

--brian

---

From: "Talpey, Thomas" <[email-masked]>
Date: Mon, 16 Dec 2002 09:00:28 -0500

While it's possible to page flip into mbuf clusters, it is
definitely not inexpensive. The VM operation itself is not
free, but the killer comes in with TLB shootdowns. VM
remaps for I/O become prohibitively expensive on
multiprocessors.

For receive, the problem is worse, as applications rarely
perform I/O in page-sized, page-aligned operations. And
even when they do, rule 1 applies.

It becomes "pick your poison", bcopy or remap. The
proposed antidote is RDMA :-)

Tom.