Subject: HTTP/2 and TCP CWND

From: Jitu Padhye <[email-masked]>
Date: Mon, 1 Apr 2013 17:50:00 +0000

In the original SPDY draft (and the current HTTP2-01 draft), there is a TCP CWND setting exposed:

         5 - SETTINGS_CURRENT_CWND allows the sender to inform the
         remote endpoint of the current TCP CWND value.

It is not clear what the client is supposed to do with this information. There was a follow up post from Mile Belshe (https://groups.google.com/forum/?fromgroups#!topic/spdy-dev/XRkL7FlIOW4) in which he indicated that the client could persist the CWND stored by the server, and the server could later user it to "warm start" connections to the same client. This can potentially cut transfer time by several RTTs. 

I have several concerns about warm start, and if that is the only use for SETTINGS_CURRENT_CWND, then maybe we should not have the setting at all. 
 
First, if a cwnd value is persisted at the client, then, after the connection ends, it needs to be gradually decayed with time, and completely discarded if the client changes networks (e.g. from wifi to 3G etc.). This adds additional complexity to client implementation.  

Second, the server has to take any value reported by client with a large grain of salt, since it cannot trust the client to correctly decay/discard window. The client may even inflate the window in an attempt to get "better" service. So, at the very least, the server needs to check timeliness, which means it has to maintain additional state (e.g. remember when it last sent CWND to this client). This complicates server implementation.  

Third, The clients who implement this may be able to grab a higher share of bandwidth, which is unfair to legacy clients (e.g. normal HTTP clients). 

If the server wants to start with a higher initial cwnd, then that's already being discussed: http://tools.ietf.org/html/draft-ietf-tcpm-initcwnd-08.  I am not advocating for that proposal, but at least it is better than warm start, since there is no state to maintain and clients have no influence.

In general,  SETTINGS_CURRENT_CWND violates the basic layering principle. So, unless there is a really good reason to keep it, it should be removed.

Jitu

---

From: Roberto Peon <[email-masked]>
Date: Mon, 1 Apr 2013 11:31:36 -0700

Which is worse, starting with 1 connection with a cwnd of 32, or starting 6
connections each with a cwnd of 6?

-=R

On Mon, Apr 1, 2013 at 10:50 AM, Jitu Padhye <[email-masked]> wrote:

> It is not clear what the client is supposed to do with this information.
> There was a follow up post from Mile Belshe (
> https://groups.google.com/forum/?fromgroups#!topic/spdy-dev/XRkL7FlIOW4)
> in which he indicated that the client could persist the CWND stored by the
> server, and the server could later user it to "warm start" connections to
> the same client. This can potentially cut transfer time by several RTTs.
>
> I have several concerns about warm start, and if that is the only use for
> SETTINGS_CURRENT_CWND, then maybe we should not have the setting at all.
>
> First, if a cwnd value is persisted at the client, then, after the
> connection ends, it needs to be gradually decayed with time, and completely
> discarded if the client changes networks (e.g. from wifi to 3G etc.). This
> adds additional complexity to client implementation.
>
> Second, the server has to take any value reported by client with a large
> grain of salt, since it cannot trust the client to correctly decay/discard
> window. The client may even inflate the window in an attempt to get
> "better" service. So, at the very least, the server needs to check
> timeliness, which means it has to maintain additional state (e.g. remember
> when it last sent CWND to this client). This complicates server
> implementation.
>
> Third, The clients who implement this may be able to grab a higher share
> of bandwidth, which is unfair to legacy clients (e.g. normal HTTP clients).
>
> If the server wants to start with a higher initial cwnd, then that's
> already being discussed:
> http://tools.ietf.org/html/draft-ietf-tcpm-initcwnd-08.  I am not
> advocating for that proposal, but at least it is better than warm start,
> since there is no state to maintain and clients have no influence.
>
> In general,  SETTINGS_CURRENT_CWND violates the basic layering principle.
> So, unless there is a really good reason to keep it, it should be removed.
>
> Jitu
>
>
>
>

---

From: Robert Collins <[email-masked]>
Date: Tue, 2 Apr 2013 07:53:45 +1300

> connections each with a cwnd of 6?

Is this an Either-Or situation?

-Rob

---

From: Patrick McManus <[email-masked]>
Date: Mon, 1 Apr 2013 14:58:19 -0400

Hi Jitu,

Heh - I've been teasing the google team for a year over when someone
would raise a serious objection to this. I figured it just meant
nobody from TSV or sigcomm had really read the draft yet :) So welcome
and thanks!

While I can go either way, I think on balance all the persistent
settings should be removed because they provide a fairly minor benefit
at the cost of a new cookie mechanism and that trade probably isn't
worthwhile. But there are benefits here, I've seen them in practice.
My reasoning is quite different than what you presented, so I want to
explain why I have some disagreements - I think its important so we
don't close the door un-necessarily to similar approaches going
forward.

> First, if a cwnd value is persisted at the client, then, after the connection ends, it needs to be gradually decayed with time, and completely discarded if the client changes networks (e.g. from wi…

perhaps it should be time decayed for efficiency - but it doesn't have
to. This is just an initial setting - analagous as you say to TCP IW.
It doesn't change any of the AIMD aspects of TCP so we're not delving
into anything serious that would touch on congestion collapse, etc..

It's the classic startup problem - start sending too conservatively
and you waste bandwidth that you can never get back , start sending
too aggressively and you do (fixable) damage to yourself and other
flows and waste time and bandwidth making those corrections. Its not
clear to me which scenario is worse - they're both tragic :(

TCP basically just punts and uses a constant (IW). It could be too
small or too large. Its always the same. The CWND cookie tries to give
it a little context. It could still be too large or too small - but at
least its based on a historical path between the end points.

We've got tons of data that says TCP on HTTP/1 generally sends too
conservatively. That's a big driver behind the crude hammer of
parallel connections. They are, in large part, an application layer
mechanism to game IW - right? SPDY goes back to a small number of
connections (thankfully!) and provides this as a way to get back some
of that lost window. Its not especially more or less legitimate than
current practice imo.

> Second, the server has to take any value reported by client with a large grain of salt, since it cannot trust the client to correctly decay/discard window. The client may even inflate the window in…

The server has lots of options - it isn't obligated to do anything. It
could track this information locally and not send the cookie at all.
It could honor the cookie up to some kind of max. It could just leave
it all up to TCP. This is much like dos-prevention strategies - a
whole range of things are possible.

> Third, The clients who implement this may be able to grab a higher share of bandwidth, which is unfair to legacy clients (e.g. normal HTTP clients).

This is my big objection to your argument that I wanted to make.

I am opposed to making "HTTP/1 fairness" a requirement of HTTP/2.
making every change be "fair" on the network with legacy traffic both
ties the hands of new innovations and removes any incentives for
deployments of the new technology. Its a big mistake imo.

Each case has to be handled on its own merits, we obviously don't want
to be un-necessarily destructive. But perfect balance doesn't need to
be a goal.

> If the server wants to start with a higher initial cwnd, then that's already being discussed: http://tools.ietf.org/html/draft-ietf-tcpm-initcwnd-08.  I am not advocating for that proposal, but at…

IW10 is probably important - but its still a constant.. the idea of
the cwnd cookie is that it isn't a constant. Very different.

> In general,  SETTINGS_CURRENT_CWND violates the basic layering principle. So, unless there is a really good reason to keep it, it should be removed.

layers are lovely ways of thinking about things - but they aren't
goals unto themselves. Much of SPDY is about doing parts of TCP in
HTTP anyhow in order to get a performant deployable application based
on years of previous experience seeing where the bottlenecks are. CWND
is just one more piece of that and isn't, at least to me,
objectionable on a fundamental level.

-Patrick

---

From: Roberto Peon <[email-masked]>
Date: Mon, 1 Apr 2013 12:18:00 -0700

Unless we limit ourselves to a single HTTP/1 connection for HTTP/1.x , yes
(which we don't...) :)

-=R

---

From: Gabriel Montenegro <[email-masked]>
Date: Fri, 12 Apr 2013 21:52:03 +0000

I've opened issue #65 to track what we should do about SETTINGS_CURRENT_CWND:
https://github.com/http2/http2-spec/issues/65

As for my opinion about what to do: I think we should delete this TCP congestion window setting from HTTP/2.0.

This is as out of scope as I've ever seen at the IETF. Modifying TCP (by modifying its contract to upper layers such as HTTP, and by modifying its state machine) is not something that can be done outside of the Transport Area. I'm cc-ing Lars Eggert and Martin Stiemerling (former and current Transport ADs), in case they have additional comments or clarifications.

Besides, as pointed out by Lars Eggert (cc-ed, along with Martin Stiemerling) in an offline exchange, there is a perfectly reasonable alternative progressing in the Transport Area's TCPM working group, namely, the proposal to bump up the TCP's initial congestion window to 10. This has undergone the rigorous review required of such a fundamental change, and is currently in the RFC editor's queue (so it's basically done):

https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/

That exchange also confirmed my opinion that there is very little chance the IESG would allow SETTINGS_CURRENT_CWND to remain in the HTTP/2.0 spec. In the interest of optimizing community time (HTTPbis, HTTP/2.0 implementors, IESG, etc) I think eliminating this now so it does not appear in the first implementable draft makes sense.

Gabriel

---

From: Eliot Lear <[email-masked]>
Date: Mon, 15 Apr 2013 06:55:00 +0200

On 4/12/13 11:52 PM, Gabriel Montenegro wrote:
>
> I’ve opened issue #65 to track what we should do about
> SETTINGS_CURRENT_CWND:
>

+1 for all the reasons Gabriel mentioned.

Eliot

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 15 Apr 2013 16:04:48 +1000

On 13/04/2013, at 7:52 AM, Gabriel Montenegro <[email-masked]> wrote:

> I’ve opened issue #65 to track what we should do about SETTINGS_CURRENT_CWND:

Thanks.

> This is as out of scope as I’ve ever seen at the IETF. Modifying TCP (by modifying its contract to upper layers such as HTTP, and by modifying its state machine) is not something that can be done outside of the Transport Area. I’m cc-ing Lars Eggert and Martin Stiemerling (former and current Transport ADs), in case they have additional comments or clarifications.

I'd love to hear what Transport folks have to say on-list.

I've also been talking with the Martin about having more Transport cross-fertilisation; we're hoping to spend some time on discussing relevant issues in Berlin.

> Besides, as pointed out by Lars Eggert (cc-ed, along with Martin Stiemerling) in an offline exchange, there is a perfectly reasonable alternative progressing in the Transport Area’s TCPM working group, namely, the proposal to bump up the TCP’s initial congestion window to 10. This has undergone the rigorous review required of such a fundamental change, and is currently in the RFC editor’s queue (so it’s basically done):
>  

My understanding from discussion in Tokyo was that this is in the spec so that some interested parties can perform some experiments with it and figure out whether it offers any value. In as much as that will serve as input to the discussions with the transport area, it could be useful to leave it in for a bit longer.

So - are people still intending to do that? In particular, in the scope of the first few implementation drafts?

Regarding the overhead for implementers - this feature is completely optional, so it doesn't represent any waste of community time in those terms. 

That said, if we consider removing this, we may want to revisit our decision to only mark settings persistence 'at risk' for the implementation draft.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Mon, 15 Apr 2013 14:59:36 +0000

At the risk of piling on, I'd like to elaborate (some of) my concerns with SETTINGS_CURRENT_CWND:

When I saw this thread start 2 weeks ago, I thought of replying with a simple "+1 layer violation".  However, I have seen the argument that "layers are lovely ways of thinking about things - but they aren't goals unto themselves" so I took some time to think about some problems this could cause.

User-land vs. Kernel-land — HTTP (and likely HTTP/2.0) is often (at least partially) implemented in user-land.  Most TCP implementations are implemented in "kernel-land".  From a pragmatic implementation approach, this could be problematic.  I'll grant that many parameters are passed from one layer to another, but in this case, where is the history or state preserved?  This could get nasty.  Bottom line, this does cross some typical implementation boundaries.

Multiple applications, threads, or whatever — If HTTP/2.0 is going to be tracking settings from previous flows, these settings will need to persist across multiple threads, processes, or whatever (where flow properties may differ — think caches) as well as different applications (I personally use multiple HTTP/1.1 applications simultaneously).

Other applications — While I recognize that HTTP/1.1 is a large contributor to TCP traffic, it is not the only application that uses TCP.  I realize we may all be excited with the optimizations we can provide as part of HTTP/2.0, but wouldn't it be better if all applications that use TCP could utilize these improvements?

Other TCP flavors — Most of my interest in HTTP/2.0 is in the embedded space.  Sometimes in the embedded space we use other TCP flavors and do various "weird" things.  I think this group would be best-served not having to consider all of those various scenarios and the impact that SETTINGS_CURRENT_CWND may have.

I'm quite glad that folks are thinking about ways to improve TCP flows.  I really hope that continues.  However, I'd suggest that work not occur here, but within the transport area, and that SETTINGS_CURRENT_CWND be removed from HTTP/2.0.

Thanks,
Robby

Robby Simpson, PhD

System Architect

GE Energy

Digital Energy

M: +1 404 219 1851

[email-masked]

From: Eliot Lear <[email-masked]<mailto:[email-masked]>>
Date: Monday, April 15, 2013 12:55 AM
To: Gabriel Montenegro <[email-masked]<mailto:[email-masked]>>
Cc: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Robert Collins <[email-masked]<mailto:[email-masked]>>, Jitu Padhye <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Brian Raymor (MS OPEN TECH)" <[email-masked]<mailto:[email-masked]>>, Rob Trace <[email-masked]<mailto:[email-masked]>>, Dave Thaler <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>, "Eggert, Lars" <[email-masked]<mailto:[email-masked]>>, Martin Stiemerling <[email-masked]<mailto:[email-masked]>>
Subject: Re: HTTP/2 and TCP CWND
Resent-From: <[email-masked]<mailto:[email-masked]>>
Resent-Date: Monday, April 15, 2013 12:55 AM

Eliot

---

From: Roberto Peon <[email-masked]>
Date: Mon, 15 Apr 2013 08:18:28 -0700

One way or another, we will most certainly need significantly more
interaction and thought between the Transport and Application groups with
respect to this topic.
The fact that most sites end up with a CWND which is 36 times larger than
the default (by opening up that many connections on average) means that the
interaction between transport and application layer is totally broken today.
This really needs addressing, else all the wonderful pictures we have about
separation of layers are already defunct and broken...

To be absolutely clear, the problem that Gabriel stated here is far less
scary (it still converges, we rarely see large windows requested, we do see
*smaller* windows requested) given the one connection of HTTP/2 that the
current behavior is with the many connections of HTTP/1 as it exists today.
If the one connection of HTTP/2 fails to compete with the many of HTTP/1,
implementors will "work around" this limitation and make things strictly
worse for everyone.
-=R

On Mon, Apr 15, 2013 at 7:59 AM, Simpson, Robby (GE Energy Management) <
[email-masked]> wrote:

> When I saw this thread start 2 weeks ago, I thought of replying with a
> simple "+1 layer violation".  However, I have seen the argument that
> "layers are lovely ways of thinking about things - but they aren't goals
> unto themselves" so I took some time to think about some problems this
> could cause.
>
> User-land vs. Kernel-land — HTTP (and likely HTTP/2.0) is often (at least
> partially) implemented in user-land.  Most TCP implementations are
> implemented in "kernel-land".  From a pragmatic implementation approach,
> this could be problematic.  I'll grant that many parameters are passed from
> one layer to another, but in this case, where is the history or state
> preserved?  This could get nasty.  Bottom line, this does cross some
> typical implementation boundaries.
>
> Multiple applications, threads, or whatever — If HTTP/2.0 is going to be
> tracking settings from previous flows, these settings will need to persist
> across multiple threads, processes, or whatever (where flow properties may
> differ — think caches) as well as different applications (I personally use
> multiple HTTP/1.1 applications simultaneously).
>
> Other applications — While I recognize that HTTP/1.1 is a large
> contributor to TCP traffic, it is not the only application that uses TCP.
>  I realize we may all be excited with the optimizations we can provide as
> part of HTTP/2.0, but wouldn't it be better if all applications that use
> TCP could utilize these improvements?
>
> Other TCP flavors — Most of my interest in HTTP/2.0 is in the embedded
> space.  Sometimes in the embedded space we use other TCP flavors and do
> various "weird" things.  I think this group would be best-served not having
> to consider all of those various scenarios and the impact that
> SETTINGS_CURRENT_CWND may have.
>
> I'm quite glad that folks are thinking about ways to improve TCP flows.  I
> really hope that continues.  However, I'd suggest that work not occur here,
> but within the transport area, and that SETTINGS_CURRENT_CWND be removed
> from HTTP/2.0.
>
> Thanks,
> Robby
>
>
>
> Robby Simpson, PhD
>
> System Architect
>
> GE Energy
>
> Digital Energy
>
> M: +1 404 219 1851
>
> [email-masked]
>
>
> Cc: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Robert
> Collins <[email-masked]<mailto:[email-masked]>>, Jitu
> Padhye <[email-masked]<mailto:[email-masked]>>, "
> [email-masked]<mailto:[email-masked]>" <[email-masked]
> <mailto:[email-masked]>>, "Brian Raymor (MS OPEN TECH)" <
> [email-masked]<mailto:[email-masked]>>, Rob Trace
> <[email-masked]<mailto:[email-masked]>>, Dave Thaler <
> [email-masked]<mailto:[email-masked]>>, Martin Thomson <
> [email-masked]<mailto:[email-masked]>>, "Eggert,
> Lars" <[email-masked]<mailto:[email-masked]>>, Martin Stiemerling <
> [email-masked]<mailto:[email-masked]>>

---

From: Gabriel Montenegro <[email-masked]>
Date: Mon, 15 Apr 2013 18:07:23 +0000

I agree that for *existing* issues we need advice and collaboration with the transport folks, and am glad Mark is looping them in.

Issue #65, however, is a *new* potential issue with transport, so we should simply not add it to the list of transport problems to solve.  In response to Mark, yes, this feature is marked as experimental. But experimental features are there, because presumably they might have a future within the spec if we judge it works well.

There are two problems with that in this case. (1) HTTPbis does not have the expertise to decide if any modification to TCP works well or not. Beyond layering violations, I'm talking about expertise. The folks who understand those TCP dynamics do not participate in the working group. Even if we decided that the experiment goes swimmingly, I don't believe the feature has any future in the spec. IESG would likely trip on it and not allow it anyways. Again, this is from my interaction off line as I said before, but I agree that it would be best to hear from transport folks directly.

That is why I'm saying this particular feature is not even worth experimenting with. Especially, if transport already has an RFC that provides an alternative.

From: Roberto Peon [mailto:[email-masked]]
Sent: Monday, 15 April, 2013 08:18
To: Simpson, Robby (GE Energy Management)
Cc: Eliot Lear; Gabriel Montenegro; Robert Collins; Jitu Padhye; [email-masked]; Brian Raymor (MS OPEN TECH); Rob Trace; Dave Thaler; Martin Thomson; Eggert, Lars; Martin Stiemerling
Subject: Re: HTTP/2 and TCP CWND

One way or another, we will most certainly need significantly more interaction and thought between the Transport and Application groups with respect to this topic.
The fact that most sites end up with a CWND which is 36 times larger than the default (by opening up that many connections on average) means that the interaction between transport and application layer is totally broken today.
This really needs addressing, else all the wonderful pictures we have about separation of layers are already defunct and broken...

To be absolutely clear, the problem that Gabriel stated here is far less scary (it still converges, we rarely see large windows requested, we do see *smaller* windows requested) given the one connection of HTTP/2 that the current behavior is with the many connections of HTTP/1 as it exists today.
If the one connection of HTTP/2 fails to compete with the many of HTTP/1, implementors will "work around" this limitation and make things strictly worse for everyone.
-=R

On Mon, Apr 15, 2013 at 7:59 AM, Simpson, Robby (GE Energy Management) <[email-masked]<mailto:[email-masked]>> wrote:
At the risk of piling on, I'd like to elaborate (some of) my concerns with SETTINGS_CURRENT_CWND:

User-land vs. Kernel-land - HTTP (and likely HTTP/2.0) is often (at least partially) implemented in user-land.  Most TCP implementations are implemented in "kernel-land".  From a pragmatic implementation approach, this could be problematic.  I'll grant that many parameters are passed from one layer to another, but in this case, where is the history or state preserved?  This could get nasty.  Bottom line, this does cross some typical implementation boundaries.

Multiple applications, threads, or whatever - If HTTP/2.0 is going to be tracking settings from previous flows, these settings will need to persist across multiple threads, processes, or whatever (where flow properties may differ - think caches) as well as different applications (I personally use multiple HTTP/1.1 applications simultaneously).

Other applications - While I recognize that HTTP/1.1 is a large contributor to TCP traffic, it is not the only application that uses TCP.  I realize we may all be excited with the optimizations we can provide as part of HTTP/2.0, but wouldn't it be better if all applications that use TCP could utilize these improvements?

Other TCP flavors - Most of my interest in HTTP/2.0 is in the embedded space.  Sometimes in the embedded space we use other TCP flavors and do various "weird" things.  I think this group would be best-served not having to consider all of those various scenarios and the impact that SETTINGS_CURRENT_CWND may have.

Thanks,
Robby

Robby Simpson, PhD

System Architect

GE Energy

Digital Energy

M: +1 404 219 1851

[email-masked]<mailto:[email-masked]>

From: Eliot Lear <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Date: Monday, April 15, 2013 12:55 AM
To: Gabriel Montenegro <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Cc: Roberto Peon <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Robert Collins <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Jitu Padhye <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, "[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, "Brian Raymor (MS OPEN TECH)" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Rob Trace <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Dave Thaler <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Martin Thomson <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, "Eggert, Lars" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>, Martin Stiemerling <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Subject: Re: HTTP/2 and TCP CWND
Resent-From: <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Resent-Date: Monday, April 15, 2013 12:55 AM

Eliot

---

From: Roberto Peon <[email-masked]>
Date: Mon, 15 Apr 2013 15:15:30 -0700

I'll point out that it is very clear that we have (and have had) consensus
to remove this feature as it exists today.
So, we should.
It is and always has been a tool for research for the transport layer.

If it was defined as an opaque blob that the transport layer delegates to
the application layer to transmit and cache, would it seem as scary?

On Mon, Apr 15, 2013 at 2:03 PM, Eggert, Lars <[email-masked]> wrote:

> Hi,
>
> I had commented in an off-list discussion on this issue and was asked to
> summarize what I said to the list. So here we go.
>
> I fully understand why the idea to bypass slow-start and instead start
> with the window used during the last connection instantiation sounds nice.
> But: it has been thought of before a dozen times and has huge issues.
>

> This has the potential to generate large line-rate bursts into the
> network, which can can cause loss bursts and force TCP into timeout-based
> recovery, which has a huge impact on throughput. (Much more so than
> slow-starting with a smaller window.) That is, because you normally have no
> idea if the path conditions are at all comparable between when you cached
> that CWND and when you want to reuse it. So when you burst and create a
> series of losses - for yourself and other flows on the bottleneck! - they
> all go into timeout of a few hundred ms at least and then slow-start.
>
> The TCP WG has been working on the Google "IW10" proposal (allowing TCP to
> start with an initial window of 10 segments rather than 1-3). That seems to
> mitigate much of the need for caching the CWND, since new connections
> wouldn't need to start with very small windows. A large part of the
> discussion around that proposal was exactly on the question of how large
> the initial window can be without significantly increasing the danger of
> line-rate bursts. There has been a pretty in-depth analysis by multiple
> folks into whether 10 is safe or not, and the consensus seems to be that it
> should be. Just caching and reusing any arbitrarily large CWND is certainly
> not safe.
>

I've been a part of that research-- the measurement apparatus for the
proposal was done on my machines and code!
There is still an issue of how many connections get opened. Given current
implementations, with an initcwnd of 10, you are likely to get bursts of
between 10 and 60 packets, as the browser often opens up connections in
parallel, and the server responds with static content with alacrity.

>
> The issue itself has been thought about for much longer, c.f.
> http://tools.ietf.org/html/draft-hughes-restart-00 from 2002, which talks
> about the issue of what the window should be after a connection has been
> idle for a while and wants to resume sending.
>

Good stuff that is ignored when we have lots of (6-36) connections starting
up simultaneously or nearly so. :/

>
> Another related work item in TCP is
> http://tools.ietf.org/html/draft-ietf-tcpm-newcwv-00, which attempts to
> specify what TCP should do during periods where it didn't send at a rate
> that used up the current window, which can also lead to bursting when
> traffic demands increase.
>
> I'm mentioning this, because a lot of the work of the TCP WG revolves
> around mitigating these bursts in order to avoid stalls due to
> timeout-based recovery, and having HTTP go off and define knobs that would
> actively counteract that work seems, ahem, counterproductive.
>

I think you mistake the intent. The intent is to make it easy for transport
experimentation by giving a mechanism that can be implemented today of
storing transport-related data, and by giving that back to the transport
layer upon session resumption.
While an ugly thing (which should hopefully be a short-term band-aid for
the lack of this mechanism in the transport today), it works and does allow
for transport-level experimentation today. What would the transport folks
like stored for use by the transport layer? :)

>
> I'm all for making HTTP and TCP work better together. Limiting the number
> of parallel connections, seeing if we can increase the initial window
> safely, and other similar things are all great examples of what we should
> be doing more of. But the TCP and HTTP folks will need to work together on
> this - we can't afford to get this wrong.
>

Agree (and I believe that I said as much earlier), however, I can't think
of a good mechanism for limiting the number of parallel connections for
HTTP/1.1.
That is really my boogeyman.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Mon, 15 Apr 2013 15:56:19 -0700

When you have a lot of users, attempting to cache for them server-side is
not really possible, and opens the site up to some interesting DoS vectors.

There are easier ways to attack servers and clients (e.g. use DNS to
perform various attacks, simply send a SYN flood, etc.) than messing with
this, but it should always only be advisory.
The interesting thing about the client mucking with this data is that, so
long as the server's TCP implementation is smart enough not to kill itself
(and some simple limits accomplish that), the only on the client harms is
itself...
-=R

On Mon, Apr 15, 2013 at 3:26 PM, Eggert, Lars <[email-masked]> wrote:

> Hi,
>
> On Apr 15, 2013, at 15:15, Roberto Peon <[email-masked]> wrote:
> ...
> why does the app need to be involved here at all? It TCP wants to cache
> state from one connection instance to the next, it can (and does, in some
> cases) already do so.
>
> Yeah, for HTTP, you might want the HTTP client to hold that state instead
> of the HTTP server as an optimization, so you need to get it in and out of
> the server kernel. But is that really of general usefulness? There seem to
> be some significant security challenges here, such as whether a server
> would even trust this opaque information, given that the client could have
> messed with it. (Blindly using this state could have detrimental effects on
> other connections the server has open.)
>
> Lars

---

From: Roberto Peon <[email-masked]>
Date: Mon, 15 Apr 2013 17:16:23 -0700

On Mon, Apr 15, 2013 at 4:03 PM, Eggert, Lars <[email-masked]> wrote:

> Hi,
>
> On Apr 15, 2013, at 15:56, Roberto Peon <[email-masked]> wrote:
> I fail to see how you'd be able to achieve this. If the server uses a CWND
> that is too large, it will inject a burst of packets into the network that
> will overflow a queue somewhere. Unless you use WFQ or something similar on
> all bottleneck queues (not generally possible), that burst will likely
> cause packet loss to other flows, and will therefore impact them.
>

The most obvious way is that the server doesn't use a CWND which is larger
than the largest currently active window to a similar RTT. The other
obvious way is to limit it to something like 32, which is about what we'd
see with the opening of a mere 3 regular HTTP connections! This at least
makes the one connection competitive with the circumventions that HTTP/1.X
currently exhibits.

> TCP is a distributed resource sharing algorithm to allocate capacity
> throughout a network. Although the rates for all flows are computed in
> isolation, the effect of that computation is not limited to the flow in
> question, because all flows share the same queues.
>

Yes, that is what I've been arguing w.r.t. the many connections that the
application-layer currently opens :)
It becomes a question of which dragon is actually most dangerous.

-=R

---

From: Patrick McManus <[email-masked]>
Date: Mon, 15 Apr 2013 21:45:11 -0400

Hi All,

The problem being addressed here isn't a hypothetical one, nor is it
resolved separately by IW10.. but it is one that specially impacts HTTP
because of its history of parallel connection [ab]use and the goal of
HTTP/2 to transition away from that. SETTINGS may or may not be the way to
do it. Expertise is always welcome, but the topic is completely germane as
far as I am concerned.

One of the deficiencies of IW10 is derived from its adherence to layering
separation - it works only on the TCP level. So you get IW=10 whether there
is 1 connection (SPDY), 6 connections (HTTP/1), or 36 connections (6 way
sharded HTTP/1). At the TCP level of course, the stack has no idea which to
use - IW is just a default config for new connections. If it uses IW=3 and
there is 1 SPDY connection then that decision likely has a large
opportunity cost in terms of un-utilized bandwidth.. if it uses IW=10 and
there are 36 HTTP/1 connections then congestion is self-induced and a train
wreck of packet loss ensues and the cost is paid in retransmissions, out of
order deliveries, timers, etc.. (This happens today on a fairly prominent
CDN). Both scenarios are suboptimal, as there is no perfect constant, but
in neither case does the Internet collapse or the page even fail to load.

Its totally reasonable to experiment with how HTTP/2 can improve this
unfortunate situation.

As Roberto says, giving applications a knob to do this reasonably removes
the need for them to do it unreasonably and less effectively with a hammer.

If we break down the arbitrary layer barrier and inject some history into
the algorithm, in this case via SETTINGS, you get something more powerful
than a default constant. Part of what you inject is traditional L4
information (what was our CWND before) which is much more interesting than
a constant, but the other thing you implicitly inject is information that
this flow carries SPDY and therefore won't be engaging in rampant
parallelization. (This latter bit can actually be done by the
implementation without help from the wire protocol, but being comfortable
with that opens the door for the protocol to create a better implementation
via the SETTINGS frame).

-Patrick

---

From: "Martin Nilsson" <[email-masked]>
Date: Tue, 16 Apr 2013 04:12:55 +0200

On Tue, 16 Apr 2013 03:45:11 +0200, Patrick McManus <[email-masked]>  
wrote:

> than a default constant.

Shouldn't the first option be to attempt injecting this history without  
breaking layers, like adding a TCP option?

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Roberto Peon <[email-masked]>
Date: Mon, 15 Apr 2013 20:46:35 -0700

Long term, yes, absolutely.
But, realistically, today we need to deal with the HUGE set of clients who
won't be getting an update to their stack in the next 10 years...
-=R

---

From: Eliot Lear <[email-masked]>
Date: Tue, 16 Apr 2013 15:27:26 +0200

On 4/16/13 3:45 AM, Patrick McManus wrote:

> Part of what you inject is traditional L4 information (what was our
> CWND before) which is much more interesting than a constant,

Much more interesting.  That much we agree on.  Whether it is
information or misinformation is the real question.

Eliot

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Tue, 16 Apr 2013 14:59:31 +0000

It appears to me that the chief concern is in regards to HTTP/2's use of a
single flow vs. HTTP/1's use of multiple flows.  I fail to see how
changing CWND is going to make a difference (except for a beginning,
potentially nasty, burst) since, if the flows are sharing the same
end-to-end paths, they will still be competing and will steady-state to
"equal" CWND's.

Further, this is not specific to HTTP/2.  There are plenty of other
applications out there using TCP that have suffered from HTTP/1's use of
multiple simultaneous flows.  I fail to see how HTTP/2 is in a better
situation to correct that issue than any other application or, more
importantly, than the appropriate layer (4) or the offender (HTTP/1).

Am I missing something, or is the concern really that some folks are
looking for a way to throttle or compete with HTTP/1 flows?

- Robby

---

From: Roberto Peon <[email-masked]>
Date: Tue, 16 Apr 2013 09:03:29 -0700

Unfortunately, most flows with http/1 are short (and bursty) flows, so it
is incorrect to say that they have reached steady-state w.r.t. congestion
control. Many resource fetches complete in between one and three rtts, then
much of time the connection(s) sit idle for extended periods of time
(seconds to minutes).

I'm hoping Will will chime in here with data soon, but the distribution on
single connection cwnds as measured on traffic in the wild shows us that
using multiple connections puts more packets on the wire as a result of
init-cwnd (and thus not subject to congestion control, ouch) than a single
stable-state, heavily used and reused connection such as Spdy or HTTP/2
would allow.

This is happening now, the data is there now.

Multiple connections is worse, and it is the common case, and it is getting
to be even more of a problem month by month.

And I'm not even talking about how multiple http connections deal with loss
vs the one...

I agree that solving the problem for http/2 alone won't fix it for
everything. On the other hand, we also need to act swiftly to solve
problems on timescales that matter to users, and http is a fine venue for
that. The fact that http/2 could do this doesn't stop us from coming up
with an opaque blob that *any* latency sensitive application protocol
could use to communicate with the transport layer.

Really, all this is is an API which allows the transport layer to
communicate with a future instantiation of itself...
-=R

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Tue, 16 Apr 2013 17:58:39 +0000

On 4/16/13 12:03 PM, "Roberto Peon" <[email-masked]<mailto:[email-masked]>> wrote:

Unfortunately, most flows with http/1 are short (and bursty) flows, so it is incorrect to say that they have reached steady-state w.r.t. congestion control. Many resource fetches complete in between one and three rtts, then much of time the connection(s) sit idle for extended periods of time (seconds to minutes).

Good point.  Most of my work with HTTP/1 (and TCP) are with long-lived flows and I live in the long tail.  My gut tells me you are correct when it comes to traditional web usage.

I'm hoping Will will chime in here with data soon, but the distribution on single connection cwnds as measured on traffic in the wild shows us that using multiple connections puts more packets on the wire as a result of init-cwnd (and thus not subject to congestion control, ouch) than a single stable-state, heavily used and reused connection such as Spdy or HTTP/2 would allow.

Wouldn't Spdy or HTTP/2 still face the issues regarding steady-state then?

Part of my concern is that we may, once again, create an HTTP that unfairly dominates traffic due to lots of bursty flows.

<snip>

I agree that solving the problem for http/2 alone won't fix it for everything. On the other hand, we also need to act swiftly to solve problems on timescales that matter to users, and http is a fine venue for that. The fact that http/2 could do this doesn't stop us from coming up with an opaque blob that *any* latency sensitive application protocol  could use to communicate with the transport layer.

As someone stated earlier, I fear the opposite to be true.  This may end up slowing down HTTP/2 and not be swift at all..

---

From: Roberto Peon <[email-masked]>
Date: Tue, 16 Apr 2013 11:50:44 -0700

On Tue, Apr 16, 2013 at 10:58 AM, Simpson, Robby (GE Energy Management) <
[email-masked]> wrote:

> Good point.  Most of my work with HTTP/1 (and TCP) are with long-lived
> flows and I live in the long tail.  My gut tells me you are correct when it
> comes to traditional web usage.
>

No-- by muxing the many separate requests onto one connection, we are far
more likely to achieve steady-state before we stop sending bytes around.
Thusfar we've seen that the one connection thing wins (so long as it is not
handicapped by startup) so long as packet loss is less than ~1.5%. It does
better there, often with smaller aggregate window size and packets. Part of
this is that it stays along enough to actually trigger the fast rexmit
path. Above ~1.5%, and the circumvention of connections (with crap
duty-cycle) wins out, as the loss strikes fewer than all of the
connections, and thus the CWND closes more slowly than it does for the
connection with a much better proportion of time where it is actually
sending/receiving packets.

>
> Part of my concern is that we may, once again, create an HTTP that
> unfairly dominates traffic due to lots of bursty flows.
>

I understand. I worry about the same thing, just from the other end. If we
don't make a single connection competitive with many, we'll have people
using many, and we know that is a bad thing as it is a very effective
circumvention method for both initial burst size AND avoiding loss backoff
:/

> As someone stated earlier, I fear the opposite to be true.  This may end
> up slowing down HTTP/2 and not be swift at all..
>
>
This is true of any starting condition. The question becomes: Is past
experience of X time ago less wrong on average than starting with an
arbitrary constant.

-=R

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 17 Apr 2013 12:48:46 -0400

> information or misinformation is the real question.

It's definitely misinformation given the dynamic nature of the
CWND variable in TCP.  This is not a path property like MTU that
can be thought of as relatively static, and it can change on short
timescales with high granularity.

-- 
Wes Eddy
MTI Systems

---

From: Patrick McManus <[email-masked]>
Date: Wed, 17 Apr 2013 14:12:15 -0400

Hi Wes,

> timescales with high granularity.

Granted, an old CWND measurement can be inaccurate. It's an informed guess
based on path performance. I'm sure we agree the path plays a
(non-definitive) role in this.

The alternative, IW, is an inaccurate guess too. Its an uninformed guess
and I don't see why we should assume it would be more accurate.

We can't argue that IW10 is strictly more conservative because my data says
it typically isn't.  (median SPDY CWND SETTING in firefox data is 30 x 1
session.. apples to apples that compares to at least 6 parallel HTTP/1
sessions of IW 10 each). Roberto suggested he's seen something similar. I'm
not sure that more conservative is a better thing anyhow but I don't see
how it applies in this case in any event.

-Patrick

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 17 Apr 2013 15:34:33 -0700

On Wed, Apr 17, 2013 at 11:12 AM, Patrick McManus <[email-masked]> wrote:
> The alternative, IW, is an inaccurate guess too. Its an uninformed guess and
> I don't see why we should assume it would be more accurate.
>
> not sure that more conservative is a better thing anyhow but I don't see how
> it applies in this case in any event.
I'd like to give some historical perspective of IW10 and CWND-SETTING,
as I proposed both to tcpm and the SPDY team. Three years when we
started looking at the TCP startup latency problems, we wanted to
experiment both simple fixed size  solution and a dynamic
history-based one. We were aware of the pros and cons of both
approaches. But I emphasize cwnd-setting was meant to experiment the
cwnd-caching idea that's been floating around for 20? years. It was OK
to violate the layers for experimental purposes. I agree with the
consensus on the list to delegate this type of job to the network
transport.

I am happy about the discussion of this thread because it points out a
bigger issue: we need a better network transport that understands the
applications.

The situation is terrible as how much browsers need to work-around or
manage to reduce network latency: number of connections, per-opening
connections, prefer connections that have sent more data, disable
slow-start-after-idle, open two connections at the same time and use
the one that comes back first, blah blah blah. each has its merit but
together sometimes they do very bad/stupid things (e.g., opening too
many connections and hoses itself on slow networks).

I feel httpbis and tsvarea should start looking into this problem
together. The browser has good knowledge about the priorities of the
resources, HTTP/2 helps speeding up things talking to the same
host/domain, and the network/transport knows the packet delays and
bandwidth well. Together there should be a better solution. It can not
be done by any single layer/protocol.

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 17 Apr 2013 18:17:54 +0000

We need to have this discussion over on the TCPM list, where the folks hang out that have looked at these issues for decades.

-- 
Sent from a mobile device; please excuse typos.

On Apr 17, 2013, at 11:12, "Patrick McManus" <[email-masked]> wrote:

> Granted, an old CWND measurement can be inaccurate. It's an informed guess based on path performance. I'm sure we agree the path plays a (non-definitive) role in this.
> 
> The alternative, IW, is an inaccurate guess too. Its an uninformed guess and I don't see why we should assume it would be more accurate.
> 
> We can't argue that IW10 is strictly more conservative because my data says it typically isn't.  (median SPDY CWND SETTING in firefox data is 30 x 1 session.. apples to apples that compares to at least 6 parallel HTTP/1 sessions of IW 10 each). Roberto suggested he's seen something similar. I'm not sure that more conservative is a better thing anyhow but I don't see how it applies in this case in any event.
> 
> -Patrick
>

---

From: Nandita Dukkipati <[email-masked]>
Date: Thu, 18 Apr 2013 16:55:28 -0700

Nice points Patrick.

To me these are the high order bits for letting CWND cookie feature remain-

*  As you correctly note, apps get around the static value of initcwnd per
connection by using an arbitrarily large number of parallel connections --
compared to this crude hammer, CWND cookie feature is a saner way to start.

* It's always arguable on how valid the CWND is... but then, this is true
of any starting condition. So the question really is: Is past value from
some XXX time ago less/more wrong on average than starting with an
arbitrary constant?

* Lars and others make excellent points on the detrimental effects of large
bursts. The good news is that there are server side techniques to soften
bursts on the network in the absence of ACK clocking, so implementations
will need to handle this case effectively.

CWND-SETTING is an API which gives the transport access to long term
persistent information. As such its efficacy may be subjective without
data, so it only seems reasonable to me to let the feature remain. In an
ideal world, we would have a holistic design of a transport that
understands its application well, but short of this ideal, CWND-SETTING is
a way to take us forward.

Nandita

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 19 Apr 2013 00:08:44 -0400

Hi Nandita, here are some thoughts inline:

On 4/18/2013 7:55 PM, Nandita Dukkipati wrote:
> *  As you correctly note, apps get around the static value of initcwnd
> per connection by using an arbitrarily large number of parallel
> connections -- compared to this crude hammer, CWND cookie feature is a
> saner way to start.

I think that logic isn't considering other options.  I agree this is
the "less bad" of the two options you mention, but there are more than
two bad ideas (and maybe even a couple good ones) to choose from.

In this case, it doesn't appear that the story has yet been well put
together to show that this is more helpful than harmful.  There are
strawman arguments in both directions, but certainly more research
to be done before it could be suitable for a Internet standard.

> * It's always arguable on how valid the CWND is... but then, this is
> true of any starting condition. So the question really is: Is past value
> from some XXX time ago less/more wrong on average than starting with an
> arbitrary constant?

It is a great research question, and very interesting.

The things to look at are what the impact of being wrong on different
granularities is, and what the likelihood of being wrong at different
granularities is.

> understands its application well, but short of this ideal, CWND-SETTING
> is a way to take us forward.

If the prior state variables can be utilized, they should be utilized by
code that has a more global view of the set of simultaneous connections
and other things going on in the host, and provide information to TCP
for potential use.  I don't think simply exporting the variable for
some other unspecified code to control is a good design decision.

In general though, Lars made the point that TCPM would be a good place
to get rapid feedback on this ... the startup problem is well-known and
has been beat upon for many years.

-- 
Wes Eddy
MTI Systems

---

From: Eliot Lear <[email-masked]>
Date: Fri, 19 Apr 2013 08:33:27 +0200

On this one point:

> granularities is, and what the likelihood of being wrong at different

And just as a reminder, Cisco has an open Request for Proposals[1] for
all things HTTP2, of which we would consider this part.

Eliot

[1]
http://www.cisco.com/web/about/ac50/ac207/crc_new/university/RFP/rfp13077.html

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 15 Apr 2013 21:03:53 +0000

Hi,

I had commented in an off-list discussion on this issue and was asked to summarize what I said to the list. So here we go.

I fully understand why the idea to bypass slow-start and instead start with the window used during the last connection instantiation sounds nice. But: it has been thought of before a dozen times and has huge issues.

This has the potential to generate large line-rate bursts into the network, which can can cause loss bursts and force TCP into timeout-based recovery, which has a huge impact on throughput. (Much more so than slow-starting with a smaller window.) That is, because you normally have no idea if the path conditions are at all comparable between when you cached that CWND and when you want to reuse it. So when you burst and create a series of losses - for yourself and other flows on the bottleneck! - they all go into timeout of a few hundred ms at least and then slow-start.

The TCP WG has been working on the Google "IW10" proposal (allowing TCP to start with an initial window of 10 segments rather than 1-3). That seems to mitigate much of the need for caching the CWND, since new connections wouldn't need to start with very small windows. A large part of the discussion around that proposal was exactly on the question of how large the initial window can be without significantly increasing the danger of line-rate bursts. There has been a pretty in-depth analysis by multiple folks into whether 10 is safe or not, and the consensus seems to be that it should be. Just caching and reusing any arbitrarily large CWND is certainly not safe.

The issue itself has been thought about for much longer, c.f.http://tools.ietf.org/html/draft-hughes-restart-00 from 2002, which talks about the issue of what the window should be after a connection has been idle for a while and wants to resume sending.

Another related work item in TCP is http://tools.ietf.org/html/draft-ietf-tcpm-newcwv-00, which attempts to specify what TCP should do during periods where it didn't send at a rate that used up the current window, which can also lead to bursting when traffic demands increase.

I'm mentioning this, because a lot of the work of the TCP WG revolves around mitigating these bursts in order to avoid stalls due to timeout-based recovery, and having HTTP go off and define knobs that would actively counteract that work seems, ahem, counterproductive.

I'm all for making HTTP and TCP work better together. Limiting the number of parallel connections, seeing if we can increase the initial window safely, and other similar things are all great examples of what we should be doing more of. But the TCP and HTTP folks will need to work together on this - we can't afford to get this wrong. 

Lars

PS: I'm not on the WG list, so please CC me if you'd like to respond.

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 15 Apr 2013 22:27:17 +0000

Hi,

...
> layer upon session resumption.

why does the app need to be involved here at all? It TCP wants to cache state from one connection instance to the next, it can (and does, in some cases) already do so.

Yeah, for HTTP, you might want the HTTP client to hold that state instead of the HTTP server as an optimization, so you need to get it in and out of the server kernel. But is that really of general usefulness? There seem to be some significant security challenges here, such as whether a server would even trust this opaque information, given that the client could have messed with it. (Blindly using this state could have detrimental effects on other connections the server has open.)

Lars

---

From: Peter Lepeska <[email-masked]>
Date: Wed, 24 Apr 2013 11:40:27 -0400

Not sure this has been proposed before, but better than caching would be
dynamic initial CWND based on web server object size hinting.

Web servers often know the size of the object that will be sent to the
browser. The web server therefore can help the transport make smart initial
CWND decisions. For instance, if an object is less than 20KB, which is true
for the majority of objects on web pages, the web server could tell the
transport to increase the CWND to a size that would allow the object to be
sent in the initial window.

For larger objects, the benefit of a large CWND is minimal so the web
server could tell the transport to use the default and let the connection
ramp slowly.

Peter

On Mon, Apr 15, 2013 at 8:16 PM, Roberto Peon <[email-masked]> wrote:

>> I fail to see how you'd be able to achieve this. If the server uses a
>> CWND that is too large, it will inject a burst of packets into the network
>> that will overflow a queue somewhere. Unless you use WFQ or something
>> similar on all bottleneck queues (not generally possible), that burst will
>> likely cause packet loss to other flows, and will therefore impact them.
>>
>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 24 Apr 2013 09:36:09 -0700

> sent in the initial window.

In the HTTP/2 case where we often are multiplexing, this doesn't seem to
make as much sense. Also, I'm not sure that it's a reasonable argument to
select initcwnd in absence of any congestion information...or were you
suggesting merely tweaking the initcwnd a little bit if that little bit
would make a difference in terms of fitting the whole object in the
initcwnd?

Caching attempts to reuse old congestion information, although it has been
reasonably pointed out that the validity of that information is
questionable. It's an open research question as far as I'm concerned, and
I'd love to see any data people had.

> server could tell the transport to use the default and let the connection

I'm not sure this makes sense. GMail and Google+ and I'm sure other large
web apps have rather large scripts and stylesheets, but they still care
about their initial page load latency. Perhaps you're making the assumption
that large objects implies the user does not have interactivity /
low-latency expectations? If so, that's invalid. Those roundtrips still
matter and I can tell you our Google app teams work very hard to eliminate
them. Or maybe your definition is large is larger than what I'm thinking.

>> The most obvious way is that the server doesn't use a CWND which is
>> larger than the largest currently active window to a similar RTT. The other

---

From: Peter Lepeska <[email-masked]>
Date: Wed, 24 Apr 2013 14:52:51 -0400

On Apr 24, 2013, at 12:36 PM, William Chan (陈智昌) <[email-masked]> wrote:

> Not sure this has been proposed before, but better than caching would be dynamic initial CWND based on web server object size hinting.
> 
> Web servers often know the size of the object that will be sent to the browser. The web server therefore can help the transport make smart initial CWND decisions. For instance, if an object is less than 20KB, which is true for the majority of objects on web pages, the web server could tell the transport to increase the CWND to a size that would allow the object to be sent in the initial window.
> 
> In the HTTP/2 case where we often are multiplexing, this doesn't seem to make as much sense. Also, I'm not sure that it's a reasonable argument to select initcwnd in absence of any congestion information...or were you suggesting merely tweaking the initcwnd a little bit if that little bit would make a difference in terms of fitting the whole object in the initcwnd?

Right. A small number of multiplexed connections transfer less of a given page's data in slow start so this will have less impact for those connections. However it's worth nothing that often the first object requested over the multiplexed channel will be the root object alone and of course number of round trips to download the root object directly impacts page load time.

> 
> Caching attempts to reuse old congestion information, although it has been reasonably pointed out that the validity of that information is questionable. It's an open research question as far as I'm concerned, and I'd love to see any data people had.
>  
> 
> For larger objects, the benefit of a large CWND is minimal so the web server could tell the transport to use the default and let the connection ramp slowly. 
> 
> I'm not sure this makes sense. GMail and Google+ and I'm sure other large web apps have rather large scripts and stylesheets, but they still care about their initial page load latency. Perhaps you're making the assumption that large objects implies the user does not have interactivity / low-latency expectations? If so, that's invalid. Those roundtrips still matter and I can tell you our Google app teams work very hard to eliminate them. Or maybe your definition is large is larger than what I'm thinking.

The threshold is tunable. My point here is if the TCP connection is going to be used to download a 100 MB file,  or stream a video, then slow start has a negligible impact on overall download time for the file.

> I fail to see how you'd be able to achieve this. If the server uses a CWND that is too large, it will inject a burst of packets into the network that will overflow a queue somewhere. Unless you use WFQ or something similar on all bottleneck queues (not generally possible), that burst will likely cause packet loss to other flows, and will therefore impact them.
> 
> The most obvious way is that the server doesn't use a CWND which is larger than the largest currently active window to a similar RTT. The other obvious way is to limit it to something like 32, which is about what we'd see with the opening of a mere 3 regular HTTP connections! This at least makes the one connection competitive with the circumventions that HTTP/1.X currently exhibits.
>  
> TCP is a distributed resource sharing algorithm to allocate capacity throughout a network. Although the rates for all flows are computed in isolation, the effect of that computation is not limited to the flow in question, because all flows share the same queues.
> 
> Yes, that is what I've been arguing w.r.t. the many connections that the application-layer currently opens :)

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 24 Apr 2013 12:27:47 -0700

On Wed, Apr 24, 2013 at 11:52 AM, Peter Lepeska <[email-masked]>wrote:

> On Wed, Apr 24, 2013 at 8:40 AM, Peter Lepeska <[email-masked]>wrote:
>
> Right. A small number of multiplexed connections transfer less of a given
> page's data in slow start so this will have less impact for those
> connections. However it's worth nothing that often the first object
> requested over the multiplexed channel will be the root object alone and of
> course number of round trips to download the root object directly impacts
> page load time.
>

We should move away from this assumption that the first request is for the
root object. I've been advising companies on how to do SPDY deployments,
and a common scenario is origin server hosting the root doc + SPDY capable
CDN for the subresources (primarily images served on the edge). For these
CDNs, they're going to serve a burst of traffic immediately, and those
subresources often have high impact on the above the fold perceived latency
(in many of today's websites, images form a big part of the initial
viewport's content, so serving these images quickly is vital). In today's
non-SPDY / HTTP2 case, they just domain shard and do 6 * [2-4] sharded
hosts, for 12-24 connections with IW10, starting out with effective
initcwnds of 120+. They are gaming initcwnd to the benefit of their users
that don't have a congested path, and severe detriment of users that cannot
handle such high bursts. This situation sucks.

> The threshold is tunable. My point here is if the TCP connection is going
> to be used to download a 100 MB file,  or stream a video, then slow start
> has a negligible impact on overall download time for the file.
>

Sure, if you're doing non-interactive large data transfers, then the slow
start latency isn't going to matter much. I don't view that conversation as
very interesting, and no one's agitating for change there. The contentious
and more interesting discussion is how to safely, yet quickly start up TCP
connections for interactive bursty traffic like web browsing. I include
video web sites like Youtube amongst that, even if their objects are large,
since the time to start viewing the video is still important.

---

From: Peter Lepeska <[email-masked]>
Date: Thu, 25 Apr 2013 11:58:54 -0400

I have not seen initcwnds of 120+. Can you send me a URL that would have that behavior?

Thanks,

Peter
On Apr 24, 2013, at 3:27 PM, William Chan (陈智昌) <[email-masked]> wrote:

> On Wed, Apr 24, 2013 at 11:52 AM, Peter Lepeska <[email-masked]> wrote:
> 
> We should move away from this assumption that the first request is for the root object. I've been advising companies on how to do SPDY deployments, and a common scenario is origin server hosting the root doc + SPDY capable CDN for the subresources (primarily images served on the edge). For these CDNs, they're going to serve a burst of traffic immediately, and those subresources often have high impact on the above the fold perceived latency (in many of today's websites, images form a big part of the initial viewport's content, so serving these images quickly is vital). In today's non-SPDY / HTTP2 case, they just domain shard and do 6 * [2-4] sharded hosts, for 12-24 connections with IW10, starting out with effective initcwnds of 120+. They are gaming initcwnd to the benefit of their users that don't have a congested path, and severe detriment of users that cannot handle such high bursts. This situation sucks.
> 
> 
>> 
> Sure, if you're doing non-interactive large data transfers, then the slow start latency isn't going to matter much. I don't view that conversation as very interesting, and no one's agitating for change there. The contentious and more interesting discussion is how to safely, yet quickly start up TCP connections for interactive bursty traffic like web browsing. I include video web sites like Youtube amongst that, even if their objects are large, since the time to start viewing the video is still important.
>  
> 
>> 
>> 
>> Peter
>> 
>> 
>> 
>>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Thu, 25 Apr 2013 09:21:22 -0700

There's this small internet company that has a cute kitten photo search
product, check it out: https://www.google.com/search?q=kittens&tbm=isch.

I just kicked off a WebPageTest run so you can analyze it for yourself:
http://www.webpagetest.org/result/130425_7X_TQT/
It shards 4 ways, with 6 connections per host shard. That's 24 connections
with IW10. That's effective initcwnd=240. And I'm not even mentioning the
other connections it has open. Check out how freakishly long the SSL
handshakes take when you model this on a "DSL" connection with dummynet.
Ouch.
You can see how it triggers wonderful TCP level behavior in
http://cloudshark.org/captures/20818577e6b9/graphs/~?filters=tcp.analysis.retransmission%2C%21%28tcp.analysis.retransmission%29+%7Bother%7D.
That's a graph of the retransmitted packets vs non-retransmitted.

Despite doing 4 way sharding, this example isn't as bad as it could be,
because the SSL SERVER_HELLO and CERTIFICATE messages don't use the full
IW10 available cwnd. When you examine sites that don't use HTTPS, and blast
out an image object, you see this get way worse. I've got a lot of those
URLs if you want to see this too.

Patrick and I have been discussing this off and on and how to ameliorate
this in our respective browsers. We've got different takes, and I'm
watching his work closely to see how it turns out :)

On Thu, Apr 25, 2013 at 8:58 AM, Peter Lepeska <[email-masked]> wrote:

>> Caching attempts to reuse old congestion information, although it has
>> been reasonably pointed out that the validity of that information is

---

From: Peter Lepeska <[email-masked]>
Date: Thu, 25 Apr 2013 15:16:56 -0400

I don't see evidence of IW10 in these packet captures.  In fact I'm not seeing the send window increase at all -- http://cloudshark.org/captures/35ff23aa38e1?filter=tcp.stream%20eq%202 -- from the initial 2 MSS. These connections are running very slowly.

This effect might be what's killing SPDY (chrome) performance for the same site over DSL, which is more than 2x slower than IE -- http://www.webpagetest.org/result/130425_VZ_ZC1/1/details/. My guess is that because SPDY operates over a single connection, it's SOL when that connection get stuck in a low gear like these appear to be. IE9 uses too many connections and suffers from retrans but is more robust to this effect b/c it has such high TCP connection concurrency. Again, this is just a guess though since I can't look at the WPT captures from the chrome test.

Peter

On Apr 25, 2013, at 12:21 PM, William Chan (陈智昌) <[email-masked]> wrote:

> There's this small internet company that has a cute kitten photo search product, check it out: https://www.google.com/search?q=kittens&tbm=isch.
> 
> I just kicked off a WebPageTest run so you can analyze it for yourself: http://www.webpagetest.org/result/130425_7X_TQT/
> It shards 4 ways, with 6 connections per host shard. That's 24 connections with IW10. That's effective initcwnd=240. And I'm not even mentioning the other connections it has open. Check out how freakishly long the SSL handshakes take when you model this on a "DSL" connection with dummynet. Ouch.
> You can see how it triggers wonderful TCP level behavior in http://cloudshark.org/captures/20818577e6b9/graphs/~?filters=tcp.analysis.retransmission%2C%21%28tcp.analysis.retransmission%29+%7Bother%7D. That's a graph of the retransmitted packets vs non-retransmitted.
> 
> Despite doing 4 way sharding, this example isn't as bad as it could be, because the SSL SERVER_HELLO and CERTIFICATE messages don't use the full IW10 available cwnd. When you examine sites that don't use HTTPS, and blast out an image object, you see this get way worse. I've got a lot of those URLs if you want to see this too.
> 
> Patrick and I have been discussing this off and on and how to ameliorate this in our respective browsers. We've got different takes, and I'm watching his work closely to see how it turns out :)
> 
>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Thu, 25 Apr 2013 19:09:27 -0700

Cwnd is an internal implementation state variable maintained on a per
connection basis. It's never explicitly advertised on the wire. You can
only guess at it by the number of in flight packets. I've already told you
that due to how the SSL handshake does not saturate IW10 due to Google's
smallish certificate chain. Can you clarify what you mean by send window?
Did you mean receive window? And were you applying window scaling? Please
clarify further. I don't see the 2*MSS that you're talking about.

Why are you saying SPDY is slower? Please, you don't actually think SPDY is
slower here do you? :P Hehe, J/K, Chrome's got some issues here for sure.
However, what's probably confusing you is that IE9 fires the load event way
earlier, so if you are strictly evaluating by the time the load event
fires, IE9 certainly looks faster. But look at when all these resources
load. SPDY is rockin nearly a second earlier (5.8~ compared to 6.5 for
IE9). Don't let browsers fool you, look at the waterfalls yourself to see
when the resource downloads complete.

You can look at the tcpdumps if you re-run WPT with tcpdumps enabled. Check
out my run: http://www.webpagetest.org/result/130426_NX_1ZA/. I've captured
the tcpdump for you.

Let's not get sidetracked on a tangent analyzing this waterfall. IW10 is a
reality, and highly parallelized web sites are out there. Effective
initcwnd>=120 is not uncommon. If you need more examples, I can provide
more example URLs offlist (I don't want to derail this thread).

On Thu, Apr 25, 2013 at 12:16 PM, Peter Lepeska <[email-masked]>wrote:

> I don't see evidence of IW10 in these packet captures.  In fact I'm not
> seeing the send window increase at all --
> http://cloudshark.org/captures/35ff23aa38e1?filter=tcp.stream%20eq%202 --
> from the initial 2 MSS. These connections are running very slowly.
>
> This effect might be what's killing SPDY (chrome) performance for the same
> site over DSL, which is more than 2x slower than IE --
> http://www.webpagetest.org/result/130425_VZ_ZC1/1/details/. My guess is
> that because SPDY operates over a single connection, it's SOL when that
> connection get stuck in a low gear like these appear to be. IE9 uses too
> many connections and suffers from retrans but is more robust to this effect
> b/c it has such high TCP connection concurrency. Again, this is just a
> guess though since I can't look at the WPT captures from the chrome test.
>
> Peter
>
> On Thu, Apr 25, 2013 at 8:58 AM, Peter Lepeska <[email-masked]>wrote:
>
>>>> Not sure this has been proposed before, but better than caching would
>>>> be dynamic initial CWND based on web server object size hinting.
>>>>
>>> Right. A small number of multiplexed connections transfer less of a
>>> given page's data in slow start so this will have less impact for those
>> We should move away from this assumption that the first request is for
>> the root object. I've been advising companies on how to do SPDY
>> deployments, and a common scenario is origin server hosting the root doc +
>> SPDY capable CDN for the subresources (primarily images served on the
>> edge). For these CDNs, they're going to serve a burst of traffic
>> immediately, and those subresources often have high impact on the above the
>> fold perceived latency (in many of today's websites, images form a big part
>> of the initial viewport's content, so serving these images quickly is
>> vital). In today's non-SPDY / HTTP2 case, they just domain shard and do 6 *
>> [2-4] sharded hosts, for 12-24 connections with IW10, starting out with
>> effective initcwnds of 120+. They are gaming initcwnd to the benefit of
>> their users that don't have a congested path, and severe detriment of users
>> that cannot handle such high bursts. This situation sucks.
>>
>>
>>>
>>> I'm not sure this makes sense. GMail and Google+ and I'm sure other
>>> large web apps have rather large scripts and stylesheets, but they still
>>> care about their initial page load latency. Perhaps you're making the
>>> assumption that large objects implies the user does not have interactivity
>>> / low-latency expectations? If so, that's invalid. Those roundtrips still
>>> The threshold is tunable. My point here is if the TCP connection is
>>> going to be used to download a 100 MB file,  or stream a video, then slow
>>> start has a negligible impact on overall download time for the file.
>>>
>>

---

From: Peter Lepeska <[email-masked]>
Date: Fri, 26 Apr 2013 07:56:16 -0400

You are right on both counts. First I was fooled by IE9 firing the load event earlier so I see now that both are fully loaded at about 6.5 seconds. Chrome a tiny bit quicker even. 

Second I am unacostomed to looking at captures for bandwidth limited links and so was fooled by the evenly spaced acks coming up from the browser into thinking that the sender was waiting on those acks to send it's next burst of packets, but on closer inspection that's not always the case. So I do see the cwnd (what I was calling send window) increasing as expected.

Thanks for the clarifications and also the tip on how to enable tcpdumps on WPT.

Peter 
On Apr 25, 2013, at 10:09 PM, William Chan (陈智昌) <[email-masked]> wrote:

> Cwnd is an internal implementation state variable maintained on a per connection basis. It's never explicitly advertised on the wire. You can only guess at it by the number of in flight packets. I've already told you that due to how the SSL handshake does not saturate IW10 due to Google's smallish certificate chain. Can you clarify what you mean by send window? Did you mean receive window? And were you applying window scaling? Please clarify further. I don't see the 2*MSS that you're talking about.
> 
> Why are you saying SPDY is slower? Please, you don't actually think SPDY is slower here do you? :P Hehe, J/K, Chrome's got some issues here for sure. However, what's probably confusing you is that IE9 fires the load event way earlier, so if you are strictly evaluating by the time the load event fires, IE9 certainly looks faster. But look at when all these resources load. SPDY is rockin nearly a second earlier (5.8~ compared to 6.5 for IE9). Don't let browsers fool you, look at the waterfalls yourself to see when the resource downloads complete.
> 
> You can look at the tcpdumps if you re-run WPT with tcpdumps enabled. Check out my run: http://www.webpagetest.org/result/130426_NX_1ZA/. I've captured the tcpdump for you.
> 
> Let's not get sidetracked on a tangent analyzing this waterfall. IW10 is a reality, and highly parallelized web sites are out there. Effective initcwnd>=120 is not uncommon. If you need more examples, I can provide more example URLs offlist (I don't want to derail this thread).
> 
> On Thu, Apr 25, 2013 at 12:16 PM, Peter Lepeska <[email-masked]> wrote:

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 27 Apr 2013 16:12:27 +1000

On 26/04/2013, at 7:02 PM, Martin Stiemerling <[email-masked]> wrote:

> Hi all,
> 
> I'm joining in late and have to second what Lars and other pointed out on that topic in respect of letting TCP do its job.
> 
> On 04/15/2013 08:04 AM, Mark Nottingham wrote:
>>> As for my opinion about what to do: I think we should delete this
>>> TCP congestion window setting from HTTP/2.0.
>>> 
>>> This is as out of scope as I’ve ever seen at the IETF. Modifying
>>> TCP (by modifying its contract to upper layers such as HTTP, and by
>>> modifying its state machine) is not something that can be done
>>> outside of the Transport Area. I’m cc-ing Lars Eggert and Martin
>>> Stiemerling (former and current Transport ADs), in case they have
>>> additional comments or clarifications.
>> 
>> I've also been talking with the Martin about having more Transport
>> cross-fertilisation; we're hoping to spend some time on discussing
>> relevant issues in Berlin.
> 
> I wonder if we should have a HTTPbis presentation and discussion of the issue the HTTPbis WG faces to the TCPM group at the IETF in Berlin, sort of a joint session?

Makes sense to me. I'm happy to give some time in the HTTPbis session to this, or we could schedule a separate joint meeting.

Cheers,

---

From: Eliot Lear <[email-masked]>
Date: Sat, 27 Apr 2013 10:21:42 +0200

+1.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Apr 2013 09:41:36 -0700

>> I wonder if we should have a HTTPbis presentation and discussion of the issue the HTTPbis WG faces to the TCPM group at the IETF in Berlin, sort of a joint session?

As long as the transport and http-bis started talking, that'd be a
great idea. So either is fine.

I also think some problems are beyond TCPM and should be discussed in
tsv-area. Today the congestion is often caused by connection
parallelism but that's not TCP's job to handle that. In HTTP/2 such a
problem still exist if the page contains resources from different
hosts.