Subject: Pipeline hinting revisited

From: Brian Pane <[email-masked]>
Date: Thu, 11 Aug 2011 15:12:31 -0700

I've been thinking some more about request pipelining recently,
triggered by several observations:

- A significant number of real-world websites could be made faster via
widespread adoption of request pipelining (based on my study of
~15,000 sites in the httparchive.org corpus).
- A nontrivial fraction of mobile browsers are using pipelining
already, albeit not as aggressively as they could (based on Blaze's
study: http://www.blaze.io/mobile/http-pipelining-big-in-mobile/ )
- Client implementations that currently pipeline their requests are
using heuristics of varying complexity to try to decide when
pipelining is safe.  The list of conditions documented here is at the
complex end of the spectrum, and it's perhaps still incomplete:
https://bugzilla.mozilla.org/show_bug.cgi?id=599164

The key question, I think, is whether heuristics implemented on the
client side will end up being sufficient to detect safe opportunities
for pipelining.  If not, a server-driven hinting mechanism of the sort
proposed in Mark's "making pipelining usable" draft (
http://tools.ietf.org/html/draft-nottingham-http-pipeline-01 ) seems
necessary.

Anybody have additional experimental data on pipelining (including the
effectiveness of heuristics for turning pipelining on or off) that
they can share?

Thanks,
-Brian

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 12 Aug 2011 07:31:37 +0200

Hi Brian,

> effectiveness of heuristics for turning pipelining on or off) that

We've been conducting some tests for a customer working with mobile
terminals. I was very frustrated to see that pipelining did not bring
any gain there due to the first non-pipelined request to each host.
What happens is that there are many objects on a page, spread over
many hosts. The terminal opens many parallel connections to these
hosts, and as a result, there are 4-5 objects max to fetch over each
connection. All connection have a first object fetched alone, and only
once a response is received, a batch of 4 requests is sent. It is this 
pause between the first and the next request over a connection which
voids the gain. It was always faster to open more parallel connection,
despite the extra bandwidth, than to use pipeline, precisely due to
this point.

This is why I think we need to find a solution so that pipelining could
be more aggressive on riskless requests, and possibly use the server
side's hinting to safely fall back to non-pipelining ASAP if needed.
I'm well aware that the biggest issue seems to be with broken servers
getting stuck between requests. I don't know if there are many of those
or not, but maybe at one point it will become those site's problem and
not the browsers'.

Also, another important thing to note is that (especially in mobile
environments), the highest latency is between the terminal and the
ISP's infrastructure, where interception proxies are quite common.
It would be a tremendous help to be able to use pipelining by default
without hinting, because these interception proxies generally support
pipelining very well and are the only ones for whom it's important.
That way we could have a terminal always pipeline requests and the
interception proxy do whatever it likes on the last 10-millisecond
segment.

Regards,
Willy

---

From: Darin Fisher <[email-masked]>
Date: Thu, 11 Aug 2011 22:43:36 -0700

> or not, but maybe at one point it will become those site's problem and

Often it is a bad intermediary (transparent proxy).  The origin server may
be just as helpless as the client :-(

-Darin

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 12 Aug 2011 07:53:27 +0200

> be just as helpless as the client :-(

I agree, and my point is that it's the first intermediary between the
client and the origin server that counts for the client, and most often
this intermediary will support pipelining and it's too bad not to use
it with the whole world. Anyway in both cases (good or bad intermediary),
both the server and the client will have little clue and be of little
help. Ideally we should have request numbers for the connection, but as
Mark pointed it out a few months ago, those might be cached and make the
issue worse. That said, are we sure they might be cached even if advertised
in the Connection header ? I don't really think so. If we had intermediaries
or server respond with "Connection: pipeline; req=#num", or even
"Connection: req=#num", it should make it clear where it's explicitly
supported, without waiting for the whole world to adopt it on every
server. The advantage with Connection is that intermediaries will get
rid of it.

Regards,
Willy

---

From: Darin Fisher <[email-masked]>
Date: Thu, 11 Aug 2011 23:19:29 -0700

> server. The advantage with Connection is that intermediaries will get
I'm sure it is very true for a lot of mobile clients that the local
intermediary would do
the right thing.

Hmm, I'd still be worried about dumb, "transparent" intermediaries that do
not consider
themselves as a hop though.  Edge servers / virtual hosts are sometimes to
blame.

We have to consider what constraints are put on intermediaries today that
require
them to behave a certain way.  If they are not required to treat the
Connection header
as hop-by-hop, then they won't.  In some sense, it doesn't matter what the
spec says.
They will do whatever doesn't punish them.

Apologies, I'm a pipelining pessimist :-/  I think perhaps it can be viable
in a mobile
(small screen) world where users already have fairly low expectations of
their browsers.
And, if we get enough mobile clients using it, then perhaps that'll be
enough to get
servers and intermediaries to play nicely.

<aside>

I'll admit that my data is dated, but when I tried to enable pipelining in
Mozilla (around 2003), I ran into such an incredibly bazaar range of
failures.  Sure, old versions of popular servers were busted.  Apache 1.3's
CGI module did not support pipelining.  Similarly, old versions of IIS were
busted.

Apache 2.0 and the latest IIS seemed to work great, except sometimes.
 Sometimes there was a mysterious failure when communicating with one of the
"good" servers.  It seemed that some intermediary must be to blame.  The
failure modes were fascinating too.  Sometimes the response would be
nothing, sometimes it would be garbage bytes, and other times the server
would simply never reply (timeout).

Around 2005, I remember people complaining that Google maps was broken in
Firefox.  Turns out these users had enabled pipelining.  Map tiles were just
mysteriously failing to load.  Apparently, it is really easy to code a
server that doesn't handle pipelining properly, which kinda makes sense.
 Google fixed it pretty quickly, but I think that is the exception.

I guess my point is that this seems like a really tough uphill battle.  It
is a chicken-n-egg problem :-/

</aside>

-Darin

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 12 Aug 2011 18:20:13 +1200

On 12/08/11 17:53, Willy Tarreau wrote:
>> On Thu, Aug 11, 2011 at 10:31 PM, Willy Tarreau<[email-masked]>  wrote:
>>
>>> Hi Brian,
>>>

I was just thinking the 100-continue infrastructure built up over the 
last few years with some success could be leveraged here. "Expect: 
pipeline" and a 1xx status to indicate explicit support.

That appears to nicely solve the case where pipelining is default-off 
and enabled whenever possible.

  It will not solve the problem of agents being aggressive in the 
pipeline before seeing such a 1XX status. The 430 status proposed by 
Mark resolves that case and can be emitted under the same requirements 
as 417.

IMO we should add "Expect: pipeline", 1XX and 430. 
http://tools.ietf.org/html/draft-nottingham-http-pipeline-01 looks like 
the right vehicle to extend and see that implemented.

AYJ

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 12 Aug 2011 09:00:30 +0200

> pipeline" and a 1xx status to indicate explicit support.

I was thinking about something in this vein too, without very clear ideas
yet.

> pipeline before seeing such a 1XX status.

Well at least the client can start emitting the second request as soon
as it sees the 1xx, it does not have to wait for the first request to be
completed. And intermediaries could be configured to return the response
depending on what they're able to do.

Regards,
Willy

---

From: "Yngve N. Pettersen" <[email-masked]>
Date: Fri, 12 Aug 2011 09:05:10 +0200

> pipeline" and a 1xx status to indicate explicit support.

It sounds to me like you are assuming that a server that understand the  
Expect header, but does not understand the "pipeline" expectation, will  
just return a normal 200 OK response. Unfortunately, that is incorrect:  
Such servers will return a 417 "Expectation failed" error code.

I have seen servers send that code for the 100-continue expectation.

RFC 2616, sec 14.20:

    A server that does not understand or is unable to comply with any of
    the expectation values in the Expect field of a request MUST respond
    with appropriate error status. The server MUST respond with a 417
    (Expectation Failed) status if any of the expectations cannot be met
    or, if there are other problems with the request, some other 4xx
    status.

> the right vehicle to extend and see that implemented.

IMO, if the problem with bad server and intermediate handling of  
pipelining cannot be solved within the current framework, then a new  
infrastructure, either for signaling support for optional protocol  
features or upgrading the protocol, e.g. what Mark is working on, with  
proper thoughts about what an intermediate can do wrong, need to be  
considered.

-- 
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 13 Aug 2011 00:48:02 +1200

On 12/08/11 19:05, Yngve N. Pettersen wrote:
> On Fri, 12 Aug 2011 08:20:13 +0200, Amos Jeffries wrote:

I'm assuming the client which chose to send it does support it. Same 
assumptions in use for rolling out 100-continue.

pipeline as a per-hop feature.

1XX - the hop supports. Explicit support. Worst case of no change in 
overall connection lag.

417 - the hop or something downstream refuses support. Explicit failure. 
More importantly _fast_ failure.

200 (or others) is a possibility. Indicating risk.
  - From hops which do not support Expect:
  - from servers which do accept pipeline but choose not to send 1XX.
  - from hops which strip the 1XX sent by a server.

There are several cases with hops hiding themselves from Via and 
otherwise removing all traces the surrounding hops could have used to 
disable the risk. We hit the exactly same situations with 100-continue 
today.
    Does anyone have example cases of ongoing failure with that feature?

AYJ

---

From: Brian Pane <[email-masked]>
Date: Fri, 12 Aug 2011 13:51:12 -0700

On Fri, Aug 12, 2011 at 5:48 AM, Amos Jeffries <[email-masked]> wrote:
> 1XX - the hop supports. Explicit support. Worst case of no change in overall
> connection lag.
>

Two of the three cases seem to have undesirable performance characteristics:

1XX: total elapsed time is 1xRTT greater than optimal.

417: total elapsed time is 1xRTT greater than if the client hadn't
tried to pipeline at all.

200 or other: assuming the client responds by accepting the response
and disabling pipelining on the connection, the total elapsed time is
the same as it would have been if the client hadn't tried to pipeline
at all.

Ideally I'd like to avoid an extra round trip because queue depths of
only 2-4 requests on a connection are common, at least in the data set
I've studied.  Adding in a synchronous negotiation step before the
first request would eliminate much of the benefit that pipelining
would otherwise provide for these request sequences.

Here's another idea, although it's one that I expect will be
controversial: how about adding a new method?

PGET /resource HTTP/1.1
Host: www.example.com

If the server supports pipelining, it treats the request as a GET.  If
not, it replies with a 400 or 405.  (The rationale for using a new
method name instead of a new request header is to make sure the
request triggers an error response from any existing server or
intermediary implementation that doesn't recognize the new pipelining
mechanism.)

If the client receives a 400 or 405, it retries the request as a GET.
If the client receives any other response status, it pipelines
subsequent requests on that connection.  Those subsequent requests may
be sent as either GET or PGET; either will work.

If the server is a proxy that understands this new method, the
implementer can decide whether to send the request upstream to the
next hop as a PGET or convert it to a GET.

Since this approach does add an extra round trip whenever the client
sends a PGET request to a server that doesn't support it, it might
best be used in combination with an out-of-band hinting mechanism.

What do you folks think?

Thanks,
-Brian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 12 Aug 2011 14:44:53 -0700

Pipeline failures are almost never due to HTTP issues and cannot
be hinted at within HTTP.  They are due to 

  a) non-HTTP intermediaries interfering with communication, and
  b) specific resources that hog or close the connection

Hinting in HTTP (aside from "Connection: close") will not help
either one.  Hinting within the mark-up and/or links might help (b).
Using a session-level MUX below HTTP would work, of course, but
can't actually be deployed because of (a).

In short, I think it is time to stop trying to make HTTP fix all
the things wrong with TCP.  It is just as hard to deploy HTTP
changes now as it was to deploy TCP changes in 1994.  What might
work, without protocol changes, is to use a host or pathname
convention that says all of the resources on *this* site are
good for pipelining.  Ugly as heck, but it would work well enough
for the use cases where pipelining helps the most (embedded stuff).

Or simply convince the browser developers to implement visible
(transient and non-interrupting) error messages when pipelining fails.

....Roy

---

From: Brian Pane <[email-masked]>
Date: Fri, 12 Aug 2011 16:15:39 -0700

> either one.  Hinting within the mark-up and/or links might help (b).

Agreed so far; I'm thinking of out-of-band hinting, specifically
hinting within markup.

> can't actually be deployed because of (a).

You lost me on this part, though.  All that's needed for session-level
multiplexing is a stream between the client and the origin server (or
some intermediary that terminates the session) within which a framing
protocol can run.  In an environment where one can't create that
stream, CONNECT won't work either.

> the things wrong with TCP.

If I'm interpreting this right, it sounds like you're arguing in favor
of either a lighter transport layer or a session multiplexing layer
between TCP and HTTP, so that a client can issue an arbitrarily large
number of concurrent requests without the inefficiencies created by a
large number of TCP connections.  I'm of two minds.  On one hand,
lightweight multiplexing somewhere underneath layer 7 does indeed
solve the problem in a very general manner, and without the
vulnerability to head-of-line blocking that's a problem in pipelined
designs.  On the other hand, getting the world to support HTTP over
SCTP or SPDY could take a long time.  My conjecture is that a
backward-compatible change to HTTP could achieve a much faster rollout
than a relayering of HTTP on top of a new session or transport
protocol.

> It is just as hard to deploy HTTP

My preference would be a mechanism within HTML and CSS to specify URI
prefixes that are good for pipelining.  E.g.,
<link rel="quick" href="http://www.example.com/images/*">
But there's still the problem of broken intermediaries.

> (transient and non-interrupting) error messages when pipelining fails.

I'm not sure this will actually work, given Darin's observation
earlier in this thread:

    > Apache 2.0 and the latest IIS seemed to work great,
    > except sometimes.  Sometimes there was a mysterious
    > failure when communicating with one of the "good" servers.
    > It seemed that some intermediary must be to blame.  The
    > failure modes were fascinating too.  Sometimes the response
    > would be nothing, sometimes it would be garbage bytes, and
    > other times the server would simply never reply (timeout).

If the "garbage bytes" scenario involves a reasonably valid HTTP
response header followed by a corrupted response body, the client may
be helpless to detect the problem.  That's the rationale behind my
proposal of a new method name: to trigger an obvious error response
from incompatible intermediaries.

-Brian

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 13 Aug 2011 01:18:33 +0200

Hi Roy,

>   b) specific resources that hog or close the connection

I don't fully agree with you on this point. I've written myself bogus
HTTP intermediaries when I tried hard to ensure I would get pipelining
right. Silly things such as buffer wrap-arounds are not always easy to
deal with, especially when they're full. Also I remember having managed
to get a freeze when waiting for a response buffer to be released before
processing a request. What I mean is that even when you want to ensure
that your HTTP intermediary is fine, you can write bugs that make some
HTTP intermediaries buggy.

> changes now as it was to deploy TCP changes in 1994.

Speaking of TCP, another TCP-specific issue with pipelining is that
some TCP stacks send an RST and flushes any unsent data when some
data is received after one side closes. Pipelining makes this case
appear often if the server has to close the connection for whatever
reason, resulting in blank pages or truncated objects on the client
*before* the aborted requests, while the server believes it has
sent everything (and did so to the kernel). This issue is quite
hard to fix at a reasonable cost and from what I've seen, only
Apache and to some extent Squid get it right.

> What might

Well, there are sane situations (mainly between clients and proxies)
where pipelining can be reliable and efficient, and for any site. I'm
convinced we can manage to find a way to reliably make it work, even
with intercepting proxies.

> (transient and non-interrupting) error messages when pipelining fails.

Users are always afraid of errors. Turning the issue the other way around
might work though, for instance by showing a bonus for sites where pipelining
worked (eg: indicating an estimate of how much time was saved by the browser's
ability to boost the connection).

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 13 Aug 2011 17:42:05 +1000

> for the use cases where pipelining helps the most (embedded stuff).

... or perhaps a persistent, out-of-band hint, in the style of:
  http://tools.ietf.org/html/draft-nottingham-http-browser-hints

--
Mark Nottingham   http://www.mnot.net/