Subject: Re: PROPOSAL: i99 Pipelining Problems

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Apr 2008 15:00:22 +0200

Hmm.

Any volunteers for providing spec-ready text?

BR, Julian

Mark Nottingham wrote:
> 
> Formalising the proposal discussed before;
> 
>>
>> Here's a summary of the problems people have mentioned so far;
>>
>>> - Handle the first m of n requests in order, then skip to request n 
>>> (IIRC I saw this on IIS/5)
>>>
>>> - Handle the first request in a packet, ignore the rest, then process 
>>> the first in the next, and so on. I have seen this problem in WebSeal 
>>> 5 (IBM) where it was fixed, and then recently broke in WebSeal 6, and 
>>> is apparently fixed again recently. The problem is particularly 
>>> visible for HTTPS in Opera 9, but is not limited to HTTPS.
>>>
>>> - Handle one or more requests, then stall
>>>
>>> - Handle a request, then stall when the next arrives.
>>>
>>> - Not quite pipelining: Server sending Content-Length of original 
>>> content, then apply Content-Encoding without removing or updating the 
>>> Content-Length header. I have seen this happen with PHP using global 
>>> compression when the script sets the Content-Length header.
>>>
>>> - Not knowing how many requests this connection will be able to 
>>> serve. The default in httpd is 100 requests per connection, but 
>>> svn.collab.net tuned it down to 10 - which caused all sort of fun 
>>> when testing ra_serf against Subversion's own repository.  I 
>>> eventually got them to up it back to the default.  serf tries to 
>>> figure out the limit heuristically (i.e. write as much as possible, 
>>> then count how many responses we got before the connection closed - 
>>> that'll be the limit going forward).
>>>
>>> - Lost responses are, sadly, real.  Later versions of httpd 2.0.x 
>>> re-introduced a lingering close bug where it won't wait for the last 
>>> packet to be written; this is fixed in 2.2.x, but have to be 
>>> accounted for.
>>
>> My proposal: Hand this to the editors and instruct them to include a 
>> *short* (i.e., not necessarily this level of detail) note stating that 
>> pipelining is difficult to implement for clients (both UAs and 
>> intermediaries), giving some of the underlying causes (e.g., bad 
>> server implementations, difficult runtime choices) and leave it at that.
> 
> 
> 
> -- 
> Mark Nottingham     http://www.mnot.net/
> 
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 4 Apr 2008 11:17:24 -0700

> Any volunteers for providing spec-ready text?

"Shit happens.  Deal with it."

Seriously, there is no reason to specify all the possible ways
in which messages might get lost on an unreliable connection.
The only thing I would change is to resurrect my original design
for the Keep-Alive header, which indicates how many more requests
are allowed on a given persistent connection.  The others are just
single point, non-reproducible bugs.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 05 Apr 2008 11:15:23 +0200

> are allowed on a given persistent connection.  The others are just

Are you referring to the stuff discussed in 
<http://www.hpl.hp.com/personal/ange/archives/archives-95/http-wg-archive/index.html#1661>?

> single point, non-reproducible bugs.

Right.

One point of confusion that comes up from time to time is the issue of 
ordering -- can a client that sends pipelined requests rely on the 
results arriving in the same order.

My understanding of the spec is "yes, of course", but I recall people 
arguing otherwise.

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 02:04:20 +0200

> results arriving in the same order.

Yes, it's clearly specified in a MUST level server requirement.

        A server MUST send its responses to those requests in the same
        order that the requests were received.

Implementations not following this is not HTTP compliant, only broken
and buggy.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 6 Apr 2008 15:44:48 +0100

> Implementations not following this is not HTTP compliant, only broken

Indeed, it's also obvious: there's no way (in standard HTTP) for a
client to recognise misordered responses so they _must_ be in order.

However, I think the "point of confusion" may really have been about
whether requests must be _processed_ in order.  E.g. is a pipelined
POST followed by GET on the same connection required to be processed
in that order.

It seems that proxies can forward pipelined requests onto different
connections, and then the network might reorder them, so the order of
processing is not guaranteed.  But also, in practice, most proxies
seem to keep requests from the same connection forwarded to the same
connection, perhaps to avoid those surprises ;-)

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 18:29:05 +0200

> POST followed by GET on the same connection required to be processed

Specs says nothing about how requests is processed. Only that

        "Clients SHOULD NOT pipeline requests using non-idempotent
        methods or non-idempotent sequences of methods".

> connection, perhaps to avoid those surprises ;-)

Most proxies don't do pipelining at all, but do allow clients to queue a
pipeline of requests at the proxy.

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Sun, 6 Apr 2008 16:09:36 -0700 (PDT)

I don't recall how it was a SHOULD NOT instead of MUST NOT, but there is
no way with pipelined requests for a client to besure that non-idempotent
requests are handled properly unless the client waits for the 200 OK or
equivalent response before making the next request. Even an origin server
can process two pipelined requests out of order if individual requests are
farmed out to different threads/processes.

Even idempotent requests can obtain the wrong response if processed before
completion of a prior non-idempotent request.

Dave Morris

On Sun, 6 Apr 2008, Henrik Nordstrom wrote:

>
> sÃ¶n 2008-04-06 klockan 15:44 +0100 skrev Jamie Lokier:
>

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 01:42:01 +0200

sön 2008-04-06 klockan 16:09 -0700 skrev David Morris:
> equivalent response before making the next request.

True, unless the client knows the exact behavior of the server..

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 7 Apr 2008 12:20:20 +1000

Agreed; listing the bugs present in implementations isn't useful. What  
may be useful is noting that there do seem to be a high proportion of  
bugs in deployed implementations, and so clients using pipelining need  
to exercise a fair amount of caution.

So, here's a straw-man to add to p1 7.1.2.2:

"Some early implementations of HTTP/1.1 servers and proxies have been  
noted to implement pipelining incorrectly, and some commonly-deployed  
(if not spec-compliant) devices may interfere with its correct  
operation. Clients choosing to send pipelined requests on the open  
Internet should, as a result, do so cautiously."

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 10:19:58 +0200

> Internet should, as a result, do so cautiously."

I prefer not adding this to the specs as it reduces the push on such
vendors to get their software fixed to support pipelining, opening up a
for dialogue like the following

Client vendor: Your server appears to be broken and fails when we send
pipelined requests to it.

Server vendor: Your application is the only client application we know
of having this problem, and specs says you should be cautious when
sending pipelined requests so you deal with it. 

Yes, this shows how much trust I have in the industry on getting things
right...

Comments aiding implementers in how to deal with the current mess of
clearly broken and non-compliant implementations is best placed in a
separate informal document outside the standard documenting known bugs
and how to deal with them. The focus in the standard text itself should
be longterm interoperability. These implementations is very likely to
decline over time, and having such comments in the standard text itself
only adds confusion.

So far there is no evidence that the amount of broken implementations is
so high that implementing pipelining isn't possible or useful. In fact I
would say the opposite has already been proved in the wild with several
non-browser applications making very successful use of pipelining. It's
true that enabling pipelinng is hard for the major browser vendors as
their users expect them to deal with pretty much every crappy server out
there no matter how broken it is, but thats a position they have placed
themselves in.  I would be very glad the day the major browsers started
to actually alert the user when a broken server is detected instead of
just silently work around it, placing some pressure on getting servers
fixed.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 7 Apr 2008 13:02:50 +0100

Henrik Nordstrom wrote:
> and how to deal with them.

I agree, and even better if it's written as a "Hall Of Shame Of Low
Quality Implementations".

 The focus in the standard text itself should
> decline over time, and having such comments in the standard text itself

I agree with this too.

I notice that there are some things mentioned which are like this
already, though: the recommendation to accept LF without CR for
compatibility with some (unspecified) old agents, and that some old
clients follow the request entity with a blank line.

Imho, such notes are extremely useful for new implementors, but I'd be
very happy if they were maintained in a separate document of 'Hall of
Shame: Implementation guidelines for practical compatibility with
non-complient but widely deployed implementations'.

Like the one for TCP.

> I would be very glad the day the major browsers started

The ideal evolution there might be mechanisms to detect broken
servers, alert the user, but still successfully show the right
content, for a few years.

That will be a lot easier to implement in browsers if there's a
collective document of shared knowledge of what specific types of
broken behaviour to look for.  I find that knowledge is currently far
too embedded in the source code (and only there) of major browsers -
and it's never clear which things are workarounds for servers, and
which are merely implementation bugs or laxity in the browsers
themselves.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 16:36:23 +0200

> content, for a few years.

Yes.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 8 Apr 2008 10:34:48 +1000

> decline over time, and having such comments in the standard text  

There's that document again. Any volunteers to edit it?

> non-browser applications making very successful use of pipelining.

I don't agree. AIUI those non-browser applications have been  
successful because pipelining has been used under controlled  
conditions; i.e., with a known workload, and a limited pool of  
implementations.

Perhaps my proposal wasn't focused on the right aspect of interop;  
broken implementations are part of the problem, but the other is the  
nature of pipelining itself (e.g., forcing complexity and uncertainty  
on the client WRT what the optimal use of pipelining is). Would you be  
comfortable adding text like that?

> It's

Lots of people argued for something to be said about pipelining in  
previous threads; this proposal is the end result, although it's  
pretty watered down. I'd like to hear from those who put in their .02  
earlier; does this add any value at all, or should we just close this  
issue with no action?

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Apr 2008 19:11:20 -0700

Lacking any feedback or additional proposals, I'm closing this issue  
with no spec changes. If someone can build consensus on text to add,  
we can reopen for that.

On 07/04/2008, at 5:34 PM, Mark Nottingham wrote:

> nature of pipelining itself (e.g., forcing complexity and  
> uncertainty on the client WRT what the optimal use of pipelining  
> is). Would you be comfortable adding text like that?
>
>
>> It's
> pretty watered down. I'd like to hear from those who put in their . 
> 02 earlier; does this add any value at all, or should we just close  
> this issue with no action?
>
> --