Subject: Naive question on multiple TCP/IP channels

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Feb 2015 14:22:56 -0500

Today most Web browsers attempt to optimize download of images etc. by
opening multiple TCP/IP streams at the same time. This is actually done for
two reasons, first to reduce load times and second to allow the browser to
optimize page layout by getting image sizes etc up front.

This approach first appeared round about 1994. I am not sure whether anyone
actually did a study to see if multiple TCP/IP streams are faster than one
but the approach has certainly stuck.

But looking at the problem from the perspective of the network it is really
hard to see why setting up five TCP/IP streams between the same endpoints
should provide any more bandwidth than one. If the narrow waist is
observed, then the only parts of the Internet that are taking note of the
TCP part of the packet are the end points. So having five streams should
not provide any more bandwidth than one unless the bandwidth bottleneck was
at one or other endpoint.

Now there are some parts of the deployed Internet that do actually perform
statefull inspection. But I would expect increasing the number of channels
to degrade performance at a firewall or any other middle boxen.

So we have a set of behavior that seems at odd with the theory. Has anyone
done any experiments recently that would show which is right?

The reason it makes a difference is that it is becoming clear that modern
applications are not best served by an application API that is limited to
one bi-directional stream. There are two possible ways to fix this
situation. The first is to build something on top of TCP/IP the second is
to replace single stream TCP with multi-stream.

My preference and gut instinct is that the first is the proper
architectural way to go regardless of the performance benefits. When
Thompson and co were arguing that all files are flat sequences of bits,
they were saying that was the right O/S abstraction because you could build
anything you like on top.

But then I started to ask what the performance benefits to a multi-stream
TCP might be and I am pretty sure there should not be any. But the actual
Internet does not always behave like it appears it should.

I suspect that the preference for multiple streams probably comes from the
threading strategies it permits. But that is an argument about where the
boundary between the kernel and application is placed in the network stack
rather than where multiplex should live in the stack. Microsoft already
provides a network stack for .NET where the boundary is in the HTTP layer
after all.

So anyone got hard data they could share?

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Feb 2015 11:42:32 -0800

We did experimentation and real-world measurement w.r.t. HTTP2/SPDY and
HTTPS along these lines.

Multiple TCP streams are faster so long as the congestion context from one
connection isn't informed by the other connections' contexts and the
connections are using a reasonable percentage of their maximum rate.
Below the maximum rate, and there ended up being adverse effects thanks to
not triggering various TCP optimizations and requiring timeouts.

Some of the obvious effects occur at the start of the connection, with
ICWND=10, 6 connections get 60 packets worth, while one gets 10.

Unfortunately, I don't have data to give anymore, but folks at Google who
are still working on this stuff (no longer me) may have more up-to-date
information.
What is definitely true is that the fact that this is all an artifact of
the fact that congestion controllers are instantiated per connection
instead of for wider contexts, and are currently loss-based instead of
rate/timing based (which does have its own problems).
-=R

On Wed, Feb 4, 2015 at 11:22 AM, Phillip Hallam-Baker <[email-masked]
> wrote:

> This approach first appeared round about 1994. I am not sure whether
> anyone actually did a study to see if multiple TCP/IP streams are faster
> than one but the approach has certainly stuck.
>
> But looking at the problem from the perspective of the network it is
> really hard to see why setting up five TCP/IP streams between the same
> endpoints should provide any more bandwidth than one. If the narrow waist
> is observed, then the only parts of the Internet that are taking note of
> the TCP part of the packet are the end points. So having five streams
> should not provide any more bandwidth than one unless the bandwidth
> bottleneck was at one or other endpoint.
>

---

From: Donald Eastlake <[email-masked]>
Date: Wed, 4 Feb 2015 14:50:43 -0500

SCTP

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

On Wed, Feb 4, 2015 at 2:22 PM, Phillip Hallam-Baker
<[email-masked]> wrote:
> should provide any more bandwidth than one. If the narrow waist is observed,
> then the only parts of the Internet that are taking note of the TCP part of
> the packet are the end points. So having five streams should not provide any
> more bandwidth than one unless the bandwidth bottleneck was at one or other
> endpoint.
>
> situation. The first is to build something on top of TCP/IP the second is to
> replace single stream TCP with multi-stream.
>
> My preference and gut instinct is that the first is the proper architectural
> way to go regardless of the performance benefits. When Thompson and co were
> arguing that all files are flat sequences of bits, they were saying that was
> the right O/S abstraction because you could build anything you like on top.
>

---

From: David Morris <[email-masked]>
Date: Wed, 4 Feb 2015 11:54:52 -0800 (PST)

Not a statistically valid survey, but watching many last mile connections
to end user clients, I saw at least two issues that are mitigated by
multiple connections:
1) Even small pipes aren't filled by 1 connection ... for sure at
   startup
2) Error recovery from lost packets delays only one connection so
   the last mile can remain utilized.

In a context where we had a local proxy (same system), altering the
default number of connections between the browser and proxy from a few to
on the order of 16-32 improved overall page load times. (The proxy used a
single SPDY like connection to a server in all cases.)

Dave Morris

On Wed, 4 Feb 2015, Phillip Hallam-Baker wrote:

> Today most Web browsers attempt to optimize download of images etc. by opening multiple TCP/IP streams at the same time. This is actually done for two reasons, first to
> reduce load times and second to allow the browser to optimize page layout by getting image sizes etc up front.
> This approach first appeared round about 1994. I am not sure whether anyone actually did a study to see if multiple TCP/IP streams are faster than one but the approach
> has certainly stuck.
>
> But looking at the problem from the perspective of the network it is really hard to see why setting up five TCP/IP streams between the same endpoints should provide any
> more bandwidth than one. If the narrow waist is observed, then the only parts of the Internet that are taking note of the TCP part of the packet are the end points. So
> having five streams should not provide any more bandwidth than one unless the bandwidth bottleneck was at one or other endpoint.
>
> Now there are some parts of the deployed Internet that do actually perform statefull inspection. But I would expect increasing the number of channels to degrade
> performance at a firewall or any other middle boxen.
>
> So we have a set of behavior that seems at odd with the theory. Has anyone done any experiments recently that would show which is right?
>
>
> The reason it makes a difference is that it is becoming clear that modern applications are not best served by an application API that is limited to one bi-directional
> stream. There are two possible ways to fix this situation. The first is to build something on top of TCP/IP the second is to replace single stream TCP with multi-stream.
>
> My preference and gut instinct is that the first is the proper architectural way to go regardless of the performance benefits. When Thompson and co were arguing that all
> files are flat sequences of bits, they were saying that was the right O/S abstraction because you could build anything you like on top.
>
> But then I started to ask what the performance benefits to a multi-stream TCP might be and I am pretty sure there should not be any. But the actual Internet does not
> always behave like it appears it should.
>
> I suspect that the preference for multiple streams probably comes from the threading strategies it permits. But that is an argument about where the boundary between the
> kernel and application is placed in the network stack rather than where multiplex should live in the stack. Microsoft already provides a network stack for .NET where the
> boundary is in the HTTP layer after all.
>
>

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 4 Feb 2015 19:49:52 +0000

Hi,

CC'ing tsvwg, which would be a better venue for this discussion.

On 2015-2-4, at 20:22, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> Today most Web browsers attempt to optimize download of images etc. by opening multiple TCP/IP streams at the same time. This is actually done for two reasons, first to reduce load times and second to allow the browser to optimize page layout by getting image sizes etc up front.
> 
> This approach first appeared round about 1994. I am not sure whether anyone actually did a study to see if multiple TCP/IP streams are faster than one but the approach has certainly stuck.

There have been many studies; for example, http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

> But looking at the problem from the perspective of the network it is really hard to see why setting up five TCP/IP streams between the same endpoints should provide any more bandwidth than one. If the narrow waist is observed, then the only parts of the Internet that are taking note of the TCP part of the packet are the end points. So having five streams should not provide any more bandwidth than one unless the bandwidth bottleneck was at one or other endpoint.

You don't get more bandwidth in stead state (well, with old Reno stacks, you got a little more, but not much). The real win is in getting more bandwidth during the first few RTTs of TCP slow-start, which is the crucial phase when transmitting short web objects.

> Now there are some parts of the deployed Internet that do actually perform statefull inspection. But I would expect increasing the number of channels to degrade performance at a firewall or any other middle boxen.
> 

I haven't seen any performance study, but another concern is that middleboxes obviously need to maintain state per connection, and multiple parallel connections eat that binding space up more quickly. (And for a NAT, reduce the number of clients it can serve.)

> The reason it makes a difference is that it is becoming clear that modern applications are not best served by an application API that is limited to one bi-directional stream. There are two possible ways to fix this situation. The first is to build something on top of TCP/IP the second is to replace single stream TCP with multi-stream.

SCTP has what you call multiple streams in your second option, and is designed the same way.

> My preference and gut instinct is that the first is the proper architectural way to go regardless of the performance benefits. When Thompson and co were arguing that all files are flat sequences of bits, they were saying that was the right O/S abstraction because you could build anything you like on top.
> 
> But then I started to ask what the performance benefits to a multi-stream TCP might be and I am pretty sure there should not be any. But the actual Internet does not always behave like it appears it should.

See above.

Also, one motivation for SPDY/HTTP2.0 is to reduce the number of parallel connections, since web people have noticed that more is not always better here.

> I suspect that the preference for multiple streams probably comes from the threading strategies it permits. But that is an argument about where the boundary between the kernel and application is placed in the network stack rather than where multiplex should live in the stack. Microsoft already provides a network stack for .NET where the boundary is in the HTTP layer after all.
> 
> 

The TSVWG folks may have.

Lars

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Feb 2015 15:47:13 -0500

On Wed, Feb 4, 2015 at 2:49 PM, Eggert, Lars <[email-masked]> wrote:

> There have been many studies; for example,
> http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

thanks, looking at it now,

> You don't get more bandwidth in stead state (well, with old Reno stacks,
> you got a little more, but not much). The real win is in getting more
> bandwidth during the first few RTTs of TCP slow-start, which is the crucial
> phase when transmitting short web objects.

Ah yes, of course, there is a built in bandwidth throttle in the stack and
it is per TCP connection...

So basically we are looking at an end run round the TCP slow start which
works because the network stack congestion algorithm throttles bandwidth by
TCP connection and not per process.

I know that traditionally we have considered congestion control to be a per
connection thing rather than per process or per user. Those don't exactly
exist in the Internet model. But if we consider DDoS to be an extreme form
of congestion, we have to look at the O/S implementing broader rate
limiting anyway.

> SCTP has what you call multiple streams in your second option, and is

Yes, I know we have proposals for both approaches. I am trying to see if
there is a case for one over the other.

If the case for multiple streams is better performance based on friendlier
slow start parameters, maybe these should be allowed without the end run.
If the net is surviving with people starting five streams instead of one,
maybe the slow start parameters could start at five packets per destination
host instead of one per connection. It would be a little more code to
implement but speed is hardly an issue where its purpose is throttling
anyway.

Making use of SCTP means that I have to rely on O/S implementation. Which
is a really long wait. So a pragmatic approach is always going to mean a
TCP+Multiplex option. If I am going to need that anyway, SCTP has to
deliver a benefit other than simplifying my code.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 4 Feb 2015 16:00:29 -0500

On Wed, Feb 4, 2015 at 3:47 PM, Phillip Hallam-Baker
<[email-masked]> wrote:

> of congestion, we have to look at the O/S implementing broader rate limiting
> anyway.

I'm not sure this scales in a useful manner...especially if you want
(for instance) to be able to read from your cloud/nas/etc at 'line
rate' but then limit access rates to other places because of 'dos'
concerns.

Policy for that is not going to be clear, or simple, or reasonable.

> slow start parameters, maybe these should be allowed without the end run. If
> the net is surviving with people starting five streams instead of one, maybe
> the slow start parameters could start at five packets per destination host
> instead of one per connection. It would be a little more code to implement
> but speed is hardly an issue where its purpose is throttling anyway.

More connections means you may avoid the 'random early drop' problem
for some of your connections, right? Presuming somewhere between
src/dest there's a thinner pipe (or more full pipe) and RED/etc is
employed (or just queue drops in general), putting all your eggs in
one connection basket is less helpful than 5 connection baskets.

One connection (one traditional http/tcp connection) also means object
serialization gets you as well. (I think roberto alludes to this
above)

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Feb 2015 16:23:43 -0500

On Wed, Feb 4, 2015 at 4:00 PM, Christopher Morrow <[email-masked]>
wrote:

> limiting

No it is not simple at all!

At a gut level, there is something wrong if a node is generating vast
numbers of
SYN messages and never doing anything more. But only looking for SYN floods
at the network border means the attackers will attempt some Data as well.

Working at the endpoint is a lot simpler but then we rely on the endpoint
not
being compromised which is a bad assumption for a machine doing DDoS
attacks.

The reason I am doing this work is to try to work out what controls to
place
where.Traditionally a firewall has been seen as a device that protects the
local
network from attacks coming in. But preventing outbound attacks is equally
important. Nobody is going to bother attacking Comcast's customers if
all their residential gateways have egress controls that make them
useless as bots. Conversely nobody is going to pay for Comcast service
if the egress controls makes it useless for browsing etc.

> one connection basket is less helpful than 5 connection baskets.

Hmm, again sounds like gaming the congestion parameters rather than
an argument for a particular approach.

> serialization gets you as well. (I think roberto alludes to this

There are several necessary functions. One of them is MUX and another of
them is serializing data from API calls. The question being which function
each belongs at which layer.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 4 Feb 2015 17:00:04 -0500

> No it is not simple at all!

I'm glad we agree about this.

> being compromised which is a bad assumption for a machine doing DDoS

yes. there are a god number of 'not compromised' endpoints which also
are used for dos attacks. Complicit users are also a 'problem' to keep
in mind.

> The reason I am doing this work is to try to work out what controls to place

Right, this was my 'the controls are not that simple nor reasonable' comment.
ArborNetworks (for one simple example) has some decent heuristics in
place for 'anomalous traffic identification', you might ask them about
how some of that scales to 'residential gateway' sorts of locations.

Keep in mind that if you have 100,000 bots 10pps of icmp which seems
'reasonable' is 1mpps of icmp in aggregate at the far end of the
network. Individual end systems don't have to contribute much to cause
a problem.

Change icmp (becuse icmp is evil, right?) to http or https .. and
change 10pps to 1pps ... you still have a significant canon to aim
that will cause an outage for almost all websites (over 95% probably?
[note no citation] enough that 'almost all' counts). You also,
clearly, don't have to aim your http request to an actual webserver...
100kpps if syn traffic is really quite a lot for 'almost all'
endpoints on the internet to sink.

The problem of classification of traffic like this, without
coordination amongst your monitoring nodes, is ... hard.

> an argument for a particular approach.

maybe? but if you look at SPDY/http2 I think the goal there is to stop
serializing, change to a binary and compressed format for data and add
channels inside the single tcp/udp stream. Fill the pipe as quickly as
possible and keep it full for the maximal amount of time.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Feb 2015 12:54:56 +1300

> There have been many studies; for example, http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

GridFTP only exists because of lots of experience that several parallel FTP
streams achieve better throughput than a single stream, especially on paths
with a high bandwidth-delay product. I'm guessing that since buffer bloat
creates an artificially high BDP, that could apply pretty much anywhere.

SCTP is not the only one-acronym answer: try MPTCP. The interesting thing there
is that because there is explicit coupling between the streams, the throughput
increases sub-linearly with the number of streams.

    Brian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Feb 2015 09:30:52 -0500

> increases sub-linearly with the number of streams.

Buffer bloat is a great example of the consequences of the co-operative
nature
of the Internet for one of the audiences I am writing for (Folk trying to
make/understand Title II regulations).

The Internet is actually a demonstration that the commons are not such a
tragedy
after all.

 If we look at the problem of buffer bloat there are several possible
solutions
and the problem is picking one:

* Persuade manufacturers to reduce buffer sizes so the old congestion
algorithms work.

* Change the congestion algorithm.

* Break with the pure end to end principle and have the middleboxen with
the
huge buffers do some reporting back when they start to fill.

The first is difficult unless you get control of the benchmarking suites
that are
going to be used to measure performance. Which would probably mean getting
the likes of nVidia or Steam or some of the gaming companies on board.

The second is certainly possible. There is no reason that we all have to
use the
same congestion algorithm. In fact a mixture might be beneficial. Instead
of looking
for packets being dropped, the sender could look at the latency and the
difference
between the rate packets are being sent and the rate at which
acknowledgements
are being received.

Whether folk are willing to consider the third depends on how serious the
buffer bloat problem is considered to be. If the stability of the Internet
really
is at stake then everything should be on the table. But my gut instinct is
that
a middlebox does not actually have any more useful information available
to it than can be derived from the acknowledgements.

---

From: [email-masked] (Martin Rex)
Date: Thu, 5 Feb 2015 16:09:19 +0100 (CET)

> at one or other endpoint.

Your picture is completely missing a _very_ common usage scenario.
Bandwidth is often irrelevant.  It is often the "time-to-use" that counts.
And with Web Browsers, it is extremely common that users navigate to a
different page before the current page has completed loading.

Just onle single HTTP/1.0 or HTTP/1.1 pipe requires total serialization
of all objects that comprise a web page.  And it starts with the
main page.

Think of a "huge" main page (200 KByte+) that contains a long list of
things (like a catalog of an online shop or auction site), with
graphical navigation buttons at the top of the page and for every list
item and preview pictures.

When the huge main page and the embedded elements are loaded in parallel,
i.e. the browser establishes parallel connections to download embedded
small images & icons after just 10% of the main page have been loaded,
the browser can start rendering and making available to the user the
top of the page *LONG* before the page has completed loading, and the
user may actually navigate away from the page long before the loading
of the page completes.  And this is where serialized loading through
just a single pipe would result in significantly higher waiting times
for the interactive user of a browser.

-Martin

---

From: Jim Gettys <[email-masked]>
Date: Thu, 5 Feb 2015 11:14:49 -0500

> This approach first appeared round about 1994. I am not sure whether
anyone
> but the approach has certainly stuck.

They did study it; there was/is a point of diminishing returns...  In that
era, the diminishing returns were in part due to insufficient buffering at
dialup modem banks.

really
> should provide any more bandwidth than one. If the narrow waist is
observed,
> more bandwidth than one unless the bandwidth bottleneck was at one or
other

Most web objects are small.  In fact, given even IW4, much less IW10, most
objects will fit in the initial window of packets for that connection.

Note, this means that the increase in IW means that there is increasing
head of line blocking occurring at (most) bottleneck links.

Also note that the web is *not* a bulk data transfer system; it is very
latency sensitive.
If you increase the effective RTT, you kill performance.

On (almost all) TCP implementations today, those packets
​from all of those objects ​
will be transmitted at *line rate* in the data center, just to arrive
effectively simultaneously at the bottleneck near the user (typically the
broadband last mile, or in the customer's home router just before the WiFi
hop), where they cause head of line blocking on those devices, as
​almost all
 lack any sort of flow queuing.

So you get a burst of possibly up to hundreds of packets (due to web site
sharding) transmitted at the data center arriving *splat* at the bottleneck
link(s).

I've measured transients of *up to hundreds of milliseconds* on 50Mbps
links.

*So TCP congestion control/avoidance has effectively been defeated
entirely.​  It just isn't in play at all!​*

I call this "transient bufferbloat".  It is the cause of a lot of the
VOIP/teleconferencing drop outs you hear if you surf the web.

> situation. The first is to build something on top of TCP/IP the second is
to
> replace single stream TCP with multi-stream.

You generally find the size information needed for page layout in the first
*packet* of data.  Might be nice if those packets from different flows got
scheduled before the later packets, might it not?  Might be nice if other
traffic, such as other TCP opens, or DNS lookups, or your voip packets got
scheduled for transmission before those additional packets from that bunch
of flows started when you visited that crappy web page with tons of little
objects on it (median is
​of order​
10 objects, btw).

Oh, wait, you say that fq_codel does that?  Well, you might consider using
it.
On
​any​
 current Linux system, that is "one line" of configuration.  And you might
want to replace your home router too, and use firmware on it that
implements fq_codel.... Life gets much better, indeed.
​  And you might want to beat up your ISP, as the bottleneck may be your
 broadband head-end and not under your control.​ And ask Microsoft and
Apple when their OS's will do something sensible too....

https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
                                    - Jim

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Feb 2015 11:15:45 -0500

> different page before the current page has completed loading.

No, actually I am fully aware of that requirement, it is what prompted my
question. There is no question that we need a multiplexing capability. The
question is whether this should be at the Transport level by opening
multiple TCP streams or at a higher level.

I suspect that what is actually holding us back here is the lack of a good
API for any sort of multiplexed setup. Virtually all of our APIs are still
following the BSD sockets approach which in turn was developed before
threads existed.

Working at the transport level means being limited to a fairly small
number of multiplex streams. If we kicked the interaction up to the
application
level then we could make it easier for a browser to regulate the
information
flow and prioritize the information it needed.

---

From: Jim Gettys <[email-masked]>
Date: Thu, 5 Feb 2015 11:30:41 -0500

> the likes of nVidia or Steam or some of the gaming companies on board.

​The missing metric is "latency under load".  Right now, we only measure
bps.
​

> between the rate packets are being sent and the rate at which

​Doesn't matter.  TCP congestion control has effectively been defeated by
web browsers/servers.  Most of the data is transmitted in the IW's of n
connections, where n is large.  The incentives are such that applications
can/will abuse the network and not be "cooperative". I expect there will
continue to be such applications, even if we "fix" the web to be better
behaved; incentives are not aligned properly for cooperation.

So you can argue all you want about your favorite congestion control
algorithm, and it won't solve this fundamental issue.​

> to it than can be derived from the acknowledgements.

​While you need to run a mark/drop algorithm (self tuning) to keep long
lived TCP flows under control, you can't touch the transient problems we
have short of some sort of flow queuing (fair or not...).  It's head of
line blocking that is killing you.
​

​  Ergo fq_codel combining flow queuing (of a clever nature) with a
mark/drop algorithm.​  The exact details (other than being self tuning) of
the mark/drop algorithm is *much* less important than whether flow queuing
is implemented. Once you isolate the flows, life gets much easier.

​So the days of single, huge bloated FIFO queues have to go.​

Jim

​

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Feb 2015 11:51:44 -0500

On Thu, Feb 5, 2015 at 11:30 AM, Jim Gettys <[email-masked]> wrote:

>
>
> On Thu, Feb 5, 2015 at 9:30 AM, Phillip Hallam-Baker <
> [email-masked]> wrote:
>
>>
>>
>>> >> Today most Web browsers attempt to optimize download of images etc.
>>> by opening multiple TCP/IP streams at the same time. This is actually done
>>> for two reasons, first to reduce load times and second to allow the browser
>>> to optimize page layout by getting image sizes etc up front.
>>> >>
OK yes, My Netflix download is going to kill your VOIP call.

Can't fix the queuing algorithm for just one interaction...

The reason I started pushing on this is that in the wake of Title II I am
expecting a lot of people to be asking me to explain how the Internet
worketh and this is precisely the sort of example that shows how (1) things
are more complex than they appear, (2) how some sort of coordination is
needed and (3) how the coordination needs to take less than five years to
come to a decision.

---

From: Piers O'Hanlon <[email-masked]>
Date: Thu, 5 Feb 2015 10:06:42 +0000

One of the snags with using multiple [slower] TCP streams is that although you might get a faster throughput you end up losing more packets as the equivalent TCP loss rate for multiple lower rate flows is higher than for one larger stream. This can be explained by looking at the TCP [simple] equation Xrate = 1.2*PacketLen/(RTT*sqrt(loss)) where it can be seen that the lower the send rate the higher the loss needs to be on a path with the same RTT.

I guess there were also other approaches like BEEP, and Google's QUIC which multiplexed multiple connections over UDP. 

Piers

---

From: Jim Gettys <[email-masked]>
Date: Thu, 5 Feb 2015 11:59:43 -0500

On Thu, Feb 5, 2015 at 5:06 AM, Piers O'Hanlon <[email-masked]> wrote:

> One of the snags with using multiple [slower] TCP streams is that although
> you might get a faster throughput you end up losing more packets as the
> equivalent TCP loss rate for multiple lower rate flows is higher than for
> one larger stream. This can be explained by looking at the TCP [simple]
> equation Xrate = 1.2*PacketLen/(RTT*sqrt(loss)) where it can be seen that
> the lower the send rate the higher the loss needs to be on a path with the
> same RTT.
>

​There is sooo much buffering in most of the Internet edge devices, that
packet loss is mostly irrelevant for most web traffic.  There is typically
more than a page worth's of buffering everywhere, sometimes lots more. So
beware theory; it can be misleading...

The scale of buffering is sometimes truely incredible.  I measured
*interplanetary* scale buffering on GoGo on my way to Hawaii....

​See Dave Reed's comments on where it has taken them:
http://www.reed.com/blog-dpr/?p=174
​

>
> I guess there were also other approaches like BEEP, and Google's QUIC
> which multiplexed multiple connections over UDP.
>
> Piers
>
> >> You don't get more bandwidth in stead state (well, with old Reno
> stacks, you got a little more, but not much). The real win is in getting
> more bandwidth during the first few RTTs of TCP slow-start, which is the
> crucial phase when transmitting short web objects.
> >>
> >>> Now there are some parts of the deployed Internet that do actually
> perform statefull inspection. But I would expect increasing the number of
> channels to degrade performance at a firewall or any other middle boxen.
> >>>
> >>> So we have a set of behavior that seems at odd with the theory. Has
> anyone done any experiments recently that would show which is right?
> >>
> >> I haven't seen any performance study, but another concern is that
> middleboxes obviously need to maintain state per connection, and multiple
> parallel connections eat that binding space up more quickly. (And for a
> NAT, reduce the number of clients it can serve.)
> >>
> >>> The reason it makes a difference is that it is becoming clear that
> modern applications are not best served by an application API that is
> limited to one bi-directional stream. There are two possible ways to fix
> this situation. The first is to build something on top of TCP/IP the second
> is to replace single stream TCP with multi-stream.
> >>
> >>> But then I started to ask what the performance benefits to a
> multi-stream TCP might be and I am pretty sure there should not be any. But
> the actual Internet does not always behave like it appears it should.
> >>
> >> See above.
> >>
> >> Also, one motivation for SPDY/HTTP2.0 is to reduce the number of
> parallel connections, since web people have noticed that more is not always
> better here.
> >>
> >>> I suspect that the preference for multiple streams probably comes from
> the threading strategies it permits. But that is an argument about where
> the boundary between the kernel and application is placed in the network
> stack rather than where multiplex should live in the stack. Microsoft
> already provides a network stack for .NET where the boundary is in the HTTP
> layer after all.
> >>>
> >>>

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 05 Feb 2015 14:01:50 -0500

On 2/5/2015 11:51 AM, Phillip Hallam-Baker wrote:
> The reason I started pushing on this is that in the wake of Title II I
> am expecting a lot of people to be asking me to explain how the Internet
> worketh and this is precisely the sort of example that shows how (1)
> things are more complex than they appear, (2) how some sort of
> coordination is needed and (3) how the coordination needs to take less
> than five years to come to a decision.
> 

The AQM working group is attempting to do helpful things here, and
could certainly use more feedback on some of the drafts:
https://datatracker.ietf.org/wg/aqm/documents/

*Please* feel welcomed to comment on the drafts, as part of the
intention is to directly deal with the sorts of problems that have
been mentioned in this thread.

-- 
Wes Eddy
MTI Systems