Subject: Re: pipelined client/server behaviour

From: Adrien de Croy <[email-masked]>
Date: Sun, 01 Apr 2007 22:25:43 +1200

Is there still some confusion about what we mean by pipelining?

In my understanding it's when multiple requests are sent down a single 
connection and subsequent requests are sent before the previous response 
is received back.

The approach we took with WinGate was to serialize the requests.  Whilst 
this effectively breaks potential pipelining performance benefits, it's 
about the only way we found to make the system stable.  We tried various 
ways where we honoured the client and server pipelining (i.e. pass 
through), but actually found the performance to be worse.  So now, we 
don't even process the FD_READ notification on the socket if it's a 
client send and we've received the entire message from the client and 
the server response hasn't been sent back, so TCP windowing etc should 
choke back the sends of subsequent requests.

>
> For Opera to use pipelining to a proxy the user must enable HTTP 1.1 
> for the proxy, and the proxy MUST respond with a HTTP 1.1 response to 
> the first request from Opera (which is sent using HTTP 1.0)
>

This behaviour we have found causes havoc with a proxy that always 
responds with the same version number as the client sent in it's 
request.  I'd recommend actually honouring the user's configuration 
wishes in this respect and if they specify using HTTP/1.1 then don't 
send any HTTP/1.0 requests.

Also I think the current Opera sends a "Connection: keep-alive" tag in 
this case instead of a "Proxy-connection: keep-alive" tag, which breaks 
NTLM through proxies (connection gets closed by proxy when it needs to 
be open because proxy doesn't recognise that the client indicated it 
wanted the connection kept alive).

Adrien

> --Sincerely,
> Yngve N. Pettersen
>  
> ********************************************************************
> Senior Developer                     Email: [email-masked]
> Opera Software ASA                   http://www.opera.com/
> Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
> ********************************************************************
>

---

From: Adrian Chadd <[email-masked]>
Date: Sun, 1 Apr 2007 19:18:44 +0800

> Is there still some confusion about what we mean by pipelining?

There's none from me. I understand that pipelining wasn't viewed as a

> connection and subsequent requests are sent before the previous response 

And the behaviour of this is what I'm going to do a little more research
into. I'm currently specifying the pipeline in this application as a property
of the connection and requests will be pipelined only to the server or
upstream over the same connection - I'm trying to avoid having to build
my own pipeline logic and would like instead to trust the user-agent's
pipeline logic.

(This means passing the invalid HTTP replies back up to the client though;
and that could be rather exciting to try and attempt..)

> choke back the sends of subsequent requests.

You found it performed worse? Did you ever investigate why it turned out to be
worse?

Adrian

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 01 Apr 2007 23:29:44 +1200

The worse aspect kicked in once servers started dropping connections 
when they got a pipelined request.  Rather a negative impact on 
performance that one, esp with connection-based authentication.

So overall statistically we saw it as worse, and the benefit on the 
sites that supported it seemed very small as well.  You're only talking 
about saving one RTTs per request to send requests serialized (after 
response received) than pipelined.  The biggest performance improvement 
we saw in testing came about from the reuse of the connection.

I guess if all servers and intermediaries supported pipelining, it would 
perform better overall, but I don't think it's widespread enough yet, so 
we decided to trade a small performance loss for an improvement in 
stability and ease of implementation.  We may still revert on this.

Adrien

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Sun, 01 Apr 2007 13:31:34 +0200

> connection and subsequent requests are sent before the previous response  

That's right

> choke back the sends of subsequent requests.

How the server/proxy handles
> send any HTTP/1.0 requests.

The server/proxy should respond with the HTTP version it supports, no  
matter what the client sends. See RFC 2145.

The reason Opera uses this approach is that we have seen too many proxies  
that break if they get HTTP/1.1 in the request.

-- 
Sincerely,
Yngve N. Pettersen

---

From: Adrian Chadd <[email-masked]>
Date: Sun, 1 Apr 2007 20:10:12 +0800

> stability and ease of implementation.  We may still revert on this.

Did you benchmark it over higher latency links? It might not give great
performance boosts for under 100ms but anecdotally it seems to load
pages in Mozilla faster when the site RTT is ~300 to ~400ms (think
.eu site accessing .au, or vice versa.)

Adrian

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 02 Apr 2007 00:01:37 +1200

>>> for the proxy, and the proxy MUST respond with a HTTP 1.1 response 
>>> to the first request from Opera (which is sent using HTTP 1.0)
>>>
>>
OK, I read the RFC.  I'm assuming it wasn't superseded by the later RFC 
2616?  Specifically para 2 of section 2 seems to be discussing the 
future existence of something like an Expects header.

I guess the approach is unique in this case because of other issues.  So 
we consider that the version tag is just an advertisement of protocol 
support of the sender of the message.  Makes sense I suppose since the 
transaction doesn't allow for pre-negotiation of anything let alone 
versions.

I'm picking the problems are more difficult upstream than downstream - 
which version to send?  Surely it's better to send an HTTP/1.0 request 
to an HTTP/1.1 server than vice versa?  RFC 2145 is pretty clear on this 
though. 

I think this could cause some issues though.  If a client knows a 
particular server is HTTP/1.0, and sends a request to it via an HTTP/1.1 
proxy, which then sends an HTTP/1.1 request to the server, then the 
server may not handle some of the tags the proxy sent, so proxies then 
need to start maintaining databases of known HTTP1.1 vs 1.0 servers.  
Seems to be a fundamental problem with the protocol if it requires 
agents to keep caches about behaviours of other agents.

How can an HTTP/1.0 client pretend to support the Connection tag?  it 
wasn't in the HTTP/1.0 spec was it?  So to the letter of the spec, the 
proxy shouldn't send a Proxy-Connection tag back to a client sending an 
HTTP/1.0 request anyway.  Which means that strictly speaking, NTLM isn't 
possible with strict HTTP/1.0.

> The reason Opera uses this approach is that we have seen too many 
> proxies that break if they get HTTP/1.1 in the request.

Surely that's why there's a setting to turn on and off http/1.1 through 
proxy?  I'm still a bit dubious about this approach.  When I saw Opera 
doing it, I have to admit to a fair amount of confusion.  I guess you 
guys are best placed to knowing what support issues it may be creating.

No other client we've tested takes the approach you've described.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 02 Apr 2007 00:09:33 +1200

> .eu site accessing .au, or vice versa.)

I'll have to check that - no is the likely answer though - good point re 
the latency. thanks.

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 1 Apr 2007 16:06:11 +0100

> .eu site accessing .au, or vice versa.)

Here's what I think, based on thinking and hand-waving (not measurement):

The main problem with pipelining is that a large or slow response
delays subsequent responses.  Sometimes the delay can be very long,
especially for a response generated on the fly.  Sometimes a response
is a stream which does not terminate.

The client cannot often predict which responses will be large or slow,
which makes it difficult to decide when to pipeline requests, quite
apart from proxy/server bugs.  (If HTTP allowed fragments of different
responses to multiplex out of order, it would be fine.)  Therefore,
there are many requests which the client should not pipeline.

The main benefits of pipelining are that: (a) multiple requests and
multiple responses can share packets; (b) latency in responses is
reduced, by approximately one RTT per response.  In principle, any
calculations the server needs to do can be initiated in parallel too,
or parallel I/O, but I've not heard of any server actually doing
these.

But the network latency, one RTT per response, might be reduced in a
different way than pipelining: just use more persistent connections,
and open them in advance of needing them.  E.g. when a browser fetches
a web page, it could open multiple connections immediately, sending
the first HTTP request over the first connection to be established.
The other connections are unused at this point, but they would
establish in a similar time scale in parallel.  Subsequent requests
could be sent in parallel over the set of (persistent but not
pipelined) connections.

That uses more network packets, but avoids the bugs which may be
triggered by pipelining, and avoids the problem of serialised
responses and unpredictable delays due to slow/large responses.  On a
link with high bandwidth-RTT product, I'm guessing that would perform
better than pipelining.  (Where there is high RTT but low bandwidth,
the additional packets may cost more than the potential benefit).

My $0.02,
-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 1 Apr 2007 10:26:11 -0700

> is a stream which does not terminate.

I would say that is one negative result of pipelining.  There can
be positive results as well, particularly when the client remembers
which servers do support pipelining.  But there is no question that
pipelining is hard to do right across all implementations.  That
doesn't mean it can't be done across some implementations, and there
really isn't much point in discussing it -- the standard will not
be changed.

<rant>The real problem with HTTP pipelining, though, is that it is
a horrible idea to try to duplicate TCP at the application layer.
If the people-in-charge hadn't been in such a hurry to blame HTTP
for everything bad happening to the Internet in 1994, they would
probably have realized that it made far more sense to fix
the semantic errors in TCP that prevent the sharing of window
sizes and setup information across multiple connections from the
same peers.  Sure, it would have taken some time to deploy such
a change across enough servers for it to make a difference, but
it would have fixed all sorts of denial-of-service vulnerabilities
at the same time as removing the need for HTTP to get into the
business of connection management, MUXing, etc.  TCP is a hard
problem that should only be solved once in the stack.</rant>

....Roy

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 1 Apr 2007 19:31:42 +0100

> problem that should only be solved once in the stack.</rant>

Isn't that now addressed by SCTP, and it's portable fudge, SCTP-over-UDP?

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 1 Apr 2007 19:44:24 +0100

Roy T. Fielding wrote:
> But there is no question that

Without changing the standard -- here's a proposed enhancement:

    "Connection: pipeline" may be included in a response to indicate
    that the agent generating the response (i.e. origin server or
    proxy) correctly handles pipelined requests on this connection.

    Both origin servers and proxies may generate this, to indicate
    their support.

    Conservative clients and proxies may refrain from sending
    pipelined requests until this has been seen in a previous
    response.  For example, a client may generate one request on a new
    connection, and upon seeing this header, send more pipelined
    requests immediately.

    Being a "Connection" token, it will be stripped by intermediate
    proxies.  (Right...?)

    (Additional enhancements to pipelining, if any, may be indicated by
    a "Pipeline" header.  For example if response multiplexing or
    reordering were to be added, support would be indicated by
    appropriate value of this header).

Just a little thought which might make pipelining more deployable.

-- Jamie

---

From: Adrian Chadd <[email-masked]>
Date: Mon, 2 Apr 2007 11:41:04 +0800

>     proxy) correctly handles pipelined requests on this connection.

I seem to recall reading an old mailing list post from a while ago where
someone noted various implementations treated Connection: <anything not
keepalive> as an implicit Connection: closed.

> Just a little thought which might make pipelining more deployable.

I thought about this a couple of days ago. You'd want to cover:

* out of order replies;
* non-pipelinable replies (eg streaming media) being a small 4xx or 5xx reply
  which is interpreted as a "No, don't do this via pipelining; resubmit on its
  own connection please" so the User-Agent doesn't have to wait for the end of
  the reply chain to discover it hasnt' received a reply for a specific request
  (And this gets worse if the last reply allows the connection to be persistent
  and unfortunately the User-Agent expects a reply that isn't coming.)
* I can see more than one situation where a pipeline request chain of A, B and C
  does look to the User-Agent like a non-idempotent sequence but does have some
  session or database expectation of request order. If you treat the requests as
  potentially concurrent requests to handle then things could get subtly confused.
  If you treat pipelining as purely a method to dodge request queue RTT and 
  have the server just do it right (ie, don't try to handle them concurrently;
  handle them one at a time and pass off to the same server) then pipelining behaviour
  suddenly looks like non-pipelining behaviour.

SCTP might not be a bad idea; there was some post to the mozilla dev list
from a party interested in implementing SCTP support in Mozilla.

Adrian

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 2 Apr 2007 13:32:40 +0100

> keepalive> as an implicit Connection: closed.

Fortunately, in this case, that does no great harm.  It's just a
performance loss for the broken implementation.

> I thought about this a couple of days ago. You'd want to cover:

Yes.  (Indeed, I have already deployed an application which uses a
protocol very like HTTP but with out of order replies.)

> * non-pipelinable replies (eg streaming media) being a small 4xx or
>   5xx reply which is interpreted as a "No, don't do this via
>   pipelining; resubmit on its own connection please" so the
>   User-Agent doesn't have to wait for the end of the reply chain to
>   discover it hasnt' received a reply for a specific request (And
>   this gets worse if the last reply allows the connection to be
>   persistent and unfortunately the User-Agent expects a reply that
>   isn't coming.)

Streaming media (and streaming Javascript!) origin servers can signal
that already with "Connection: close".  The client is informed immediately
that it must resend later requests on another connection, and that
it's safe to repeat them because they won't be processed on this
connection.

> * I can see more than one situation where a pipeline request chain
>   of A, B and C does look to the User-Agent like a non-idempotent
>   sequence but does have some session or database expectation of
>   request order. If you treat the requests as potentially concurrent
>   requests to handle then things could get subtly confused.  If you
>   treat pipelining as purely a method to dodge request queue RTT and
>   have the server just do it right (ie, don't try to handle them
>   concurrently; handle them one at a time and pass off to the same
>   server) then pipelining behaviour suddenly looks like
>   non-pipelining behaviour.

When an application on the client side submits requests A, B and C,
_usually_ the client-side HTTP implementation has the option to
use multiple connections for those requests.  So they might be
processed in parallel anyway, without pipelining.

So an application really should not submit requests concurrently to
it's HTTP client implementatiom, without waiting for the intermediate
responses, unless parallel processing is ok.

Not doing that is usually a layering violation.

The other problem with non-idempotent requests is what to do if an
intermediate response is large and/or slow.  It's not possible to
repeat the non-idempotent requests which have been transmitted later.

I think the sensible rule is: only pipeline GETs.

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 2 Apr 2007 05:53:35 -0700

> Without changing the standard -- here's a proposed enhancement:

That would obviously change the standard -- in this case, by implying
that all the existing implementations of pipelining, the vast majority
of which do work quite fine, do not work just because they lack this
new header field.

There is nothing you can add to the protocol to fix non-compliant
intermediaries and servers that are already deployed -- no matter
what you do, they will behave in some non-compliant way to screw up
the plan.  The right thing to do is let breakage occur until someone
gets off their rear end and replaces the broken software with
something not so broken, at which point the existing protocol
defined by the existing standard, which has already been proven
to be interoperable, will work fine just as it does now with the
majority of servers already deployed on the Internet.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 02 Apr 2007 15:32:07 +0200

mån 2007-04-02 klockan 05:53 -0700 skrev Roy T. Fielding:

> something not so broken

And an excellent move to get there is to work on the discussed HTTP test
suite, giving these software authors an "authoritative" stop sign
telling "you do not comply with specifications, in fact you are even
severely broken to the point that users may have trouble accessing your
service".

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 03 Apr 2007 07:11:15 +1200

> it's safe to repeat them because they won't be processed on this
some proxies in their desire to keep persistent connections with clients 
open as long as possible will convert a length-unspecified response with 
Connection: close into chunked data with Proxy-Connection: keep-alive.

so you can't rely on that getting back to the client.

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 Apr 2007 19:23:39 +1000

FYI, if you haven't seen it: <http://www2006.org/programme/files/ 
xhtml/2035/2035-natarajan-xhtml/2035-natarajan-xhtml.html>

--
Mark Nottingham     http://www.mnot.net/