Subject: RE: An alternative to TCP (part 1)

From: "Christian Huitema" <[email-masked]>
Date: Tue, 6 Feb 2001 13:29:36 -0800

> One could argue that it's a responsibility of wireless 
> link-layer protocols to ensure that random loss is rare (say, 
> by employing an ECC).

For slow links, maybe. But TCP's congestion avoidance algorithm does
make the assumption that 90%+ of the losses are caused by congestion,
not link errors. Plug in the asymptotic behavior that tells you that the
average data rate scales as 0(1/sqrt(loss-rate)). For a 1 Mbps
data-rate, with 512 bytes packets and 100 ms RTT, that means a loss rate
of at most 0.34%, of which 0.034% at most should come from transmission
errors. That means a bit-error rate after ECC of less that 1.E-7; OK, we
can probably engineer most links to do that.

Now, consider a high speed network, with connections running at 1Gbps,
with 8 Kbytes packets and still about 100 ms RTT. To not be slowed down
by the congestion avoidance algorithm, we need a packet loss rate better
that 1.E-6. If less than 10% of the losses are due to congestion, that
means a bit error rate better of about 1.E-12. Well, I am not at all
sure that we can hit that one.

It only gets better with speed, by the way. To reach 10 Gbps, you will
need a BER of 1.E-14. That amounts to missing at most one bit every 2
hours. There are very many ways to not achieve that...

-- Christian Huitema

---

From: "Jun'an Gao" <[email-masked]>
Date: Wed, 07 Feb 2001 10:17:46 +0800

>> 6.  Doing traffic shaping at the network edge is better than on the host node for
> The host *is* the edge of the network.

I'm sorry to have not mentioned that I consider the host nodes, or the end nodes, are not edges but instead something attaching on network edges. I consider the very last hub, or the access router which the end nodes connected to as the 'network edge'. And I'm sorry not to mention that practically  it is probably better to do traffic shaping at the service provider network edge, for some reason of accouting, congestion control, or required by service-level-agreement (I don't think throwing bandwidth at the QoS issue is a good choice in the near foreseeable future, especially for the enterprise users.).

---

From: [email-masked] (Jun'an Gao)
Date: Wed, 07 Feb 2001 10:32:52 +0800 (CST)

Well, I believe I have read rfc1323:-) I think I should give a short comparison between ATP and the enhancements of TCP (The enhancements include ECN.). I'll make the comparison(it may take me quite a few days for I 'm not a fluent English writer:-( ) after I have fully translated and posted the essay on ATP, which is originally written in Chinese. 

As to ECN, I believe it is a good idea. But ECN requires a somewhat fundamental change in the routing system. And it consumes two bits in the IP header (in the diffserv/traffic class field). I don't think it is feasible to implement ECN in an IPv4 network. I think designing a new transport protocol optimizing for IPv6 may be a better choice for the next generation Internet. In the key point 6, I specify that (trivially-updated) ECN is 'now' mandatory.

As to the sequence number, I have to confess that a 32 bit word counting byte is definitely abundant at least in the near future. But suppose some day TCP is applied in an ultra-high-performance distributed computing misson. Suppose two ultra-high-performance computers(or, some future version iSCSI devices) are linked with a 100Gbps network. I don't think it is difficult to design  I/O ports for two SMP computers each with four 500Mhz 64-bit CPUs to consume the bandwidth. It is quite possible that the full sequence number space is depleted in less than 345ms(The Twap should be less than about 170ms  in 100Gps network  for TCP to work correctly, according to rfc1323). Maybe the fiber network would never carry 100Gbps in a single lamda. Maybe never two computer joining in a ultra-high-performance distributed computing mission would be partitioned by a 100Gbps network with delay longer than 170ms.  Yet I want to quote the statement in rfc1323(page 6) 'Moving  towards gigabit speeds, Twrap becomes too small for 
reliable  enforcement by the Internet TTL mechanism' to call for pondering over the 32 bit TCP sequence number which counts byte.

Cheers!
Gao.

---

From: [email-masked] (Jun'an Gao)
Date: Wed, 07 Feb 2001 10:38:37 +0800 (CST)

> The host *is* the edge of the network.

I'm sorry to have not mentioned that I consider the host nodes, or the end nodes, are not network edge but instead something attached on the network edge. I consider the very last hub, or the access router which the end nodes connected to as 'the network edge'. And I'm sorry not to mention that practically  it is probably better to do traffic shaping at the service provider network edge, for some reason of accouting, congestion control, or required by service-level-agreement (I don't think throwing bandwidth at the QoS issue is a good choice in the near foreseeable future, especially for the enterprise users.).

Cheers!
Gao.

---

From: Larry Foore <[email-masked]>
Date: Tue, 6 Feb 2001 21:42:10 -0500

Jun'an Gao wrote:
>>> it is probably better to do traffic shaping at the service provider
network edge, for some reason of accouting, congestion control, or required
by service-level-agreement

Or possibly at the carrier or access network edge.  Carrier networks are not
necessarily owned by the service providers, which introduces another level
of resource consideration.

regards,
-Larry

****************************************************************************
This message is intended only for the individual or entity to whom it is
addressed and may contain information that is privileged, confidential, and
exempt from disclosure under applicable law.  If you are not the intended
recipient, or the employee or agent responsible for delivering the message
to the intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited, and
you are requested to please notify us immediately by telephone at
(321-956-8846) and return the original message to the address above.

---

From: [email-masked] (Jun'an Gao)
Date: Wed, 07 Feb 2001 15:47:15 +0800 (CST)

> I think it would be very difficult to replace TCP.  However, a new
That's true. But I don't think it's harder than replacing IPv4 with IPv6.

> transport protocol that worked better than TCP in very high bandwidth / 
> low delay conditions might be very attractive for hosts and applications
> that were able to take advantage of it.
I agree. And I believe the relatively high bit error rate and thus the high packet loss rate in the mobile environment should also be taken into account, maybe with a higher priority(for commercial reason:-))

> I don't agree that abundant IPv6 addresses remove the need for something
> akin to a port number.   They might remove the need for transport-level
> multiplexing, but only if any host could allocate a sufficiently large
> subnet, and it's not clear that this will be the case.  However port
> numbers are also used to form names of connection endpoints, and we have
> some need for well-known endpoint names to reach standard services. 
I'm sorry I have not make my opinion clear in the key point 3. By saying 'The services could be registered appropriately' is far from enough.

Knowing well of end point names does not necessarily require a port number. If there's a DNS extension (well, the DNS 'WKS' record can be extended) that support a client end query the DNS server for the IPv6 address of the application service, giving the domain name AND the service name.
Here I consider the (service) end point name consists of the domain name of the site which provided the service and the service name, formerly the service port name which is mapped to port number by function call such as 'getprotobyname'. Or the port number is directly written down in the source code if it is 'well-known'.

ATP counts on the DNS or some other widely deployed directory service to work. I assume the DNS is a directory service. The assumption itself is a problem but nevertheless ATP cannot work without a directory service.

> For example,.we could make the endpoint name a parameter of a SYN packet,
> and have the response include the actual address to be used.  This would
> be a useful facility for other reasons, including ability to load
> balance a service across several hosts without having to result to
> NAT-like kludges.
IPv6 anycast is supposed to have the same feature. Now we have another choice.
I think both of them are interesting.

---

From: [email-masked]
Date: Wed, 7 Feb 2001 08:58:08 -0500

> 1. There are two annoying incompetence of TCP. One is that 
> TCP does not distinguish packet loss caused by network 
> transmission error from that caused by network congestion. 
> The congestion control and avoidance mechanism makes TCP 
> drop its transmit window upon detecting a packet loss, 
> thus lowers the transmit rate even if the loss is caused 
> by physical link transmit error.
>    This results in an unnecessary reduction in link 
> bandwidth utilization, especially in the environment of 
> wireless physical links.

The Performance Implications of Link Characteristics (PILC) working group
has been exploring this.
(http://www.ietf.org/html.charters/pilc-charter.html) See
draft-ietf-pilc-error-06.txt and draft-ietf-pilc-link-design-04.txt for a
discussion of ways to ameliorate performance issues that don't require an
alternative to TCP. Bottom line recommendation: implement forward error
correction to improve noisy links, don't require the world to adopt a new
TCP.

---

From: [email-masked] (Jun'an Gao)
Date: Thu, 08 Feb 2001 08:58:43 +0800 (CST)

>> I'm sorry to have not mentioned that I consider the host nodes,
>> or the end nodes, are not edges but instead something attaching
>> on network edges. I consider the very last hub, or the access router
>> which the end nodes connected to as the 'network edge'.

> So there's no network between me and another computer on the
> same unswitched Ethernet?
For example the old 10Base2 ethernet links? Or a cross-link twisted-pairs?

I can say the full length of the 10Base2 link can be regarded as a (passive) repeater hub.

I think 'network edge' is somewhat a arbitrary definition. Maybe I'm not correct but I believe distinguishing 'network edge' from 'end node' may provide some convenience for further discussion on the congestion control and avoidance mechanism.

---

From: Larry Foore <[email-masked]>
Date: Wed, 7 Feb 2001 17:07:13 -0800 (PST)

>>Richard Carlson wrote:

>> You're right, the TCP peak was 1.48Gbps from the show floor in Dallas to
a
>> storage cluster at Berkeley Laboratory.  Single applications using
multiple
>> stream.  Bottleneck link was 1.5Gbps provisioned circuit on Qwest link
from
>> convention center to DARPA's HSCC pop node.
>>
>> This beat last years (SC'99) 1.2 Gbps rate Microsoft achieved running
TTCP
>> on a single PC with 2 Gbit Ethernet cards (Redmond to Portland).
>>
>> Rich
>>

>Peak rate alone is a meaningless measure of performance. The TCP session
may
>have reached peak for a small fraction of the session duration and the rest
of
>the time, the link may have been severely under-utilized.
>A meaningful performance measure here is average or percentile link
utilization
>over the tcp session duration.

I believe the inital question was answered, though.  Peak rate alone tells
me that it is _possible_ in the "near" future to exhaust the sequence number
limitation of TCP.  

Even still, is it wise to allow for such conditions?  Or would it be wiser
to influence the community use larger frames over these large, fat links?

_______________________________________________________
Send a cool gift with your E-Card
http://www.bluemountain.com/giftcenter/

---

From: [email-masked] (Jun'an Gao)
Date: Thu, 08 Feb 2001 09:20:04 +0800 (CST)

> if you are proposing that the IETF should investigate ATP, have you 
> submitted the proposal as an internet-draft?
No.

> If not, why not?
I myself have some unsureness on ATP.

1. There're too many contraints in the tranditional TCP/IPv4 internet environment. So ATP is going to be optimized for IPv6. Yet there're quite a lot debates on IPv6 itself. For example on the flow label field.
I like IPv6, Of course.

2. I think it may give the upper layer application users or programmers great convenience if key features of RSVP, ISAKMP, IPSec and IPComp are consolidated in a single set of APIs. The APIs actually represent the services provided by the transport layer.

But maybe it's more convenient to devise a session layer protocol. I want to raise the consideration.

3. ATP fundamentally changes the meaning of the diffserv/traffic-class field in the IPv4/IPv6 header.

There's a saying that IP will definitely fail if it tries to provide as complex QoS features as ATM CBR, ABR, VBR or UBR. I'm afraid I agree with the opinion. So I assume it is feasible to provide only two classes of service: real time, and best effort. The assumption itself may raise heating debate.

I'd like to defense the idea of ATP and make necessary update before it becomes a proposal (or does not become a proposal).

---

From: [email-masked] (Jun'an Gao)
Date: Thu, 08 Feb 2001 09:24:38 +0800 (CST)

> It is very hard to read your valuable contributions because you aren't
> sending pre-formatted text, rather one line per paragraph.

I'm very sorry! I'll resend part 2 after I pre-format it.

---

From: [email-masked] (Jun'an Gao)
Date: Thu, 08 Feb 2001 16:47:44 +0800 (CST)

> you should probably redirect this conversation to
> the end2end-interest list:
>	<[email-masked]>
>
> the entire IETF list is probably the wrong place for substantive technical 
> discussion of this type.

I agree.
I've subscribed to [email-masked] by visiting
  http://www.postel.org/mailman/listinfo/end2end-interest

I use another email account: Jason Gao [[email-masked]].

Further information on ATP will be posted on it. The structure of the
full essay is:

Asymmetric Transport Protocol
-- An alternative to TCP in IPv6 internetworking

Part I Motivations and Key points 
Part II Protocol Header
Part III Setup and Release of an ATP Connection
Part IV  Reliable Data Transport
Part V   Flow Control and Congestion Avoidance
Part VI  Event Handling and State Transition
Part VII Quality of Service Issues and Security Issues
Part VIII Miscellaneous
  -- ATP and ECN, ATP Congestion Control: Why not only round-trip feedback
  -- ATP and DNS, Resource Catalog or Directory Service
  -- ATP APIs and Migration Issue
  -- Possibility to consolidate the key features of RSVP, ISAKMP, IPSec/TLS
and IPComp.

Jason Gao

---

From: "Iliff, Tina" <[email-masked]>
Date: Thu, 08 Feb 2001 15:55:38 +0000

Yes, however, I think we have three terms here:
1. end-to-end:  path from host to host
2.  Network edge:  Internet access edge
3.  Network edge:  LAN edge
Maybe, start using local edge to indicate the LAN edge....

Tina Iliff

-----Original Message-----
From: John Stracke [mailto:[email-masked]]
Sent: Thursday, February 08, 2001 9:18 AM
To: [email-masked]
Subject: Re: An alternative to TCP (part 1)

Kevin Farley wrote:

> So you think your PC is the edge of the Internet?

Isn't that what the end-to-end principle means?

--
/=================================================================\
|John Stracke    | http://www.ecal.com |My opinions are my own.   |
|Chief Scientist |================================================|
|eCal Corp.      |World domination should never be left to chance.|
[email-masked]|                                                |
\=================================================================/

---

From: [email-masked] (Jun'an Gao)
Date: Sun, 11 Feb 2001 22:24:42 +0800 (CST)

> Do you really think that yet another transport protocol can be
> designed built and deployed in the world wide Internet any quicker?
> Even considering a move to IPv6?
Well, I think it may require quite a few years to migrate to IPv6. So maybe
there's time to prepare a transport protocol.

> It will require a good 2 years or so to get the specification done,
> then you have bikeoffs (with respects to pillsbury :->) and of
> course you must rally support so that it gets developed so you
> have enough bikes to race :) 
[ Well, I'm managing to rally support somewhere:-) ]

> I don't see how you can propose a new protocol and get it out
> any quicker than you will see SCTP deployed...
Mmm...
As far as I can perceive I think SCTP is by now the optimum in the IPv4
internetworking environment. But it was contrained by the incumbent
Internet. For example it didn't assume adequately deployment of ECN

But I have to admit again that no way may prevent SCTP from exploiting ECN.
[I noted that SCTP has for example reserved 'chunk ID' for 'Explicit Congestion
 Notification Echo' and 'Congestion Window Reduced'.]

As SCTP has to overcome TCP in the IPv4 internetworking environment
(well, this argument seems just an assumption) I don't think it will be widely
deployed in the incumbent Internet.

If the 'Interface with Upper Layer' were to provide downward comptibility
with TCP (where ATP is intent to), SCTP might be definitely quicker.

>> I think XTP is low efficient. SCTP is a functional superset of TCP,
>> RTP+UDP and RTSP+UDP in the unicast scenario.
> Yes, SCTP shares a lot in common with TCP but there are some
> striking differences.
I agree.

> No, the problem was that we could not mandate SCTP to use ECN because
> ECN was an Experimental RFC. Has such we had to stand on our head
> to get it in and not have a normative reference. I think you
> will find that in the BIS we will require an SCTP implementation
> to have ECN.
> Even at that you can NOT solely rely on ECN since it is NOT fully
> deployed into the Inter-net. You must keep the packet loss is a 
> signal of congestion. 
> Even if ECN were magically deployed everywhere tommorrow, you still
> have a problem. What happens when a router gets a BURST of traffic
> that exceeds its buffers. It MUST DROP packets if it runs out
> of buffer space. A lost packet needs to be intrepreted as a congestion
> event.

ATP bets on wide deployment of ECN in the next generation Internet. I uphold
ECN so strongly that I believe it should be mandatory in the IPv6 internetworking
environment. And I can't object to the opinion that the bet might fail.

If it is a burst of traffic cause a router drop packets, maybe it is better not to
feedback the congestion, for quite likely the congestion will disappear after the
burst, before the feedback makes the sender lowers its trasmit rate. The router
may choose to randomly set the ECN flag on only a small set of packets. And
the receiver may choose to ignore burst ECN signals. It is not prohibitively difficult
to detect the burstiness.

> I think there may be other ways of detecting a corrupted
> packet that may be employed... HACK TCP and similar items.. but there
> still needs to be more research ...
I think it implies that we all agree that it is important to differentiate packet loss
due to congestion from that due to link layer error.

> We discussed this in sigtran... question? How do you know (from the
> receiver side) that you are missing a packet? 
> You setup a connection with me.

> Send me something.
> I respond.

> You send somthing else, to make an additional request, one I
> am unaware you are going to make. How do
> I know to ask you for a retransmission when that packet is lost?
Send me something(with a packet sequence number N).

I respond.

You send something else, unfortunately it is lost.

I (the receiver) timeout. I send (and may resend) you NAK:
"Hi! I haven't got your message for quite a long time(since I got previous
packet sequence number N). What's the matter?"
[  In ATP it's more alike this:
  "Hi! I haven't got your message with sequence number N+1 for quite a long time.
  Is it lost?"
  And it may act as the keep-alive signal as well.]
...

I mean, no, the receiver alone never know whether a packet is lost or not.
But the cooperation between the receiver and the sender may.

But why the receiver timeout instead of the sender? The upper layer application
congestion.

In ATP the receiver may detect the upper layer application congestion (in a
common multi-thread host system) and deliberately deny a packet it has received
but not accepted yet. By sending back a explicit congestion feedback and a
negative acknowledgement the receiver makes the sender both slow down and
retransmit the unaccepted packet as soon as possible. The sender waits the
round trip time. It is expected to be faster than letting the sender timeout.

>> 4. SCTP supports multi-streams in a single connection.
>> I cannot perceive the necessity clearly.

> Look at head of line blocking. This will explain the
> item you are missing. A telephone example works best
> but it can also be applied to HTTP or many other apps.

> I have a connection carrying control information on
> 20 phone calls, Setup, teardown, addtional info etc.
> Now The first packet (a setup on call 1) is lost. But
> setup for call 2, 3 and 4 (in subsequent packets) are
> received at the server. The TCP connection will hold
> these packets awaiting the retransmission of call 1's packet.
> In SCTP you would use seperate streams for each of the 
> calls (or some subset) and thus call 2,3 and 4 would
> not be held up.
> HTTP has a similar problem when downloading jpeg's and
> other pieces.
I see.

Regards,
Gao.

---

From: Mark Allman <[email-masked]>
Date: Mon, 12 Feb 2001 10:15:58 -0500

> >As long as it is safe (i.e., follows good congestion control),
> >why should we care how many of these protocols are defined?  After
> >we ensure the protocols are safe we can just let Darwinism take its
> >course.
> 
> Because a customer with sufficient $ would inevitably request the
> Frobnitz Transport, since it garnered the IETF imprimateur and the
> code base would eventually collapse under the weight of all this
> "innovation".

Maybe I spoke too quickly above.  The points I was trying to get at:

  * TCP isn't necessarily the answer for every application.
    
  * We don't need to try to make TCP (or TCPng) the answer for every
    application.
    
  * Maybe ALF is the way to go...  But, I am not convinced that we
    cannot provide transports that go a long way towards solving a
    good chunk of the problems, such that most developers don't need
    to develop their own protocols on top of UDP due to their
    application's needs.

  * So, I don't see a need for a TCPng, per se.  But, if folks want
    to write new transports that include different mechanisms (e.g.,
    no strict ordering requirement) I think that is wonderful. 

  * I like the SCTP model (build something new) better than the
    model where we try to develop TCPng (rework something old, make
    sure it still works with the old stuff, etc.).  This just seems
    like The Way To Go at the transport layer to me.

allman

---
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/

---

From: "James P. Salsman" <[email-masked]>
Date: Wed, 14 Feb 2001 00:11:18 -0800 (PST)

>> So transport layer should somehow enhance
>>    the error check and/or correction mechanism.
>
> actually, I would put it in the application layer. I would have the 
> application include some form of checksum (PGP signature, file CRC, 
> whatever) to ensure for itself that what was sent was what was received.

If there is no error recovery at the link layer, then an erronious
set high-order bit in a packet length field might very likely leave 
input processes open listening for more data that will never come.

But I agree that redundancy is a good idea.

By the way, on the storage technology front, it seems that 3.4 
terabytes will soon be pocket-sized and cost about $50 (!)
  http://www.keele.ac.uk/research/cmrkeele.htm

Cheers,
James