Subject: Re: Suggestion for NEW Issue: Pipelining problems

From: Mark Nottingham <[email-masked]>
Date: Mon, 3 Mar 2008 13:25:22 +1100

On 28/02/2008, at 12:53 PM, Adrian Chadd wrote:

>
> I'm interested in this, purely to try and write a "standards  
> compliant" pipeline
> support for a future Squid version rather than hacking up something  
> local
> with (lots of) heuristics.
>
> I'd really be interested in a pipeline negotiation procedure to tag  
> a peer
> or connection as correctly implementing 'standards' pipelining.

Doing that could be within our charter, certainly, but I've already  
seen some pushback against that from others (see previous thread and  
respond to those objections as appropriate).

> If thats not possible (and its certainly outside the scope of your  
> post Mark)
> then I'd again be interested in a collaborative document outlining how
> pipelining is actually currently implemented in various servers/ 
> intermediaries/UAs
> rather than each trying their own implementation.

An implementation guide (or several guides, depending on how you want  
to split the audiences up) has come up several times in discussion. As  
with many things, I think everyone would like to see the output, but  
few have the spare time to devote to producing or reviewing it.

If someone wants to author it, please say so, and we can discuss how  
it would happen. Beyond resources, the only immediate concern I have  
is that doing so shouldn't distract from our chartered work  
(especially when it comes to reviewing BIS).

> (That, and I'm interested in SCTP too..)

As am I, but that *is* outside the scope of our charter...

Cheers,

>
>
>
>
> Adrian
>
> On Thu, Feb 28, 2008, Mark Nottingham wrote:
>>
>> There hasn't been any traffic on this thread for some time, and I
>> neglected to open an issue for it. See:
>> <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/99 >.
>>
>> Based on the discussion so far, I think this is where we are WRT
>> pipelining:
>>
>> a) In the scope of HTTPBIS, the most we can do is insert some
>> implementation advice that pipelining may be incorrectly implemented
>> by origin servers and intermediaries (including "hidden"  
>> infrastructure)
>> b) However, we shouldn't deprecate it, as it is still useful (and
>> actively used) in some cases
>> c) "fixing" the pipelining problems may involve a new transport
>> protocol (see discussion of SCTP) or other mechanism, but is outside
>> the scope of HTTPBIS.
>>
>> To that end, I propose we add implementation advice as per (a) and
>> close the issue. Thoughts?
>>
>> If that sounds reasonable to people, I can trawl the archives for
>> different issues that might be encountered as fodder for
>> implementation advice.
>>
>> Cheers,
>>
>>
>> On 09/08/2007, at 5:20 AM, Justin Erenkrantz wrote:
>>
>>>
>>> On 7/23/07, Roy T. Fielding <[email-masked]> wrote:
>>>> of the network.  Likewise, pipelining has huge benefits for
>>>> specialized HTTP services, such as Subversion (when it uses it)
>>>> and content management systems.
>>>
>>> ra_serf (optional in Subversion 1.4+ and using the serf client
>>> library) uses pipelining.
>>>
>>> The pipelining problems I've seen in the real-world are:
>>>
>>> - Not knowing how many requests this connection will be able to  
>>> serve.
>>> The default in httpd is 100 requests per connection, but
>>> svn.collab.net tuned it down to 10 - which caused all sort of fun  
>>> when
>>> testing ra_serf against Subversion's own repository.  I eventually  
>>> got
>>> them to up it back to the default.  serf tries to figure out the  
>>> limit
>>> heuristically (i.e. write as much as possible, then count how many
>>> responses we got before the connection closed - that'll be the limit
>>> going forward).
>>>
>>> - Lost responses are, sadly, real.  Later versions of httpd 2.0.x
>>> re-introduced a lingering close bug where it won't wait for the last
>>> packet to be written; this is fixed in 2.2.x, but have to be  
>>> accounted
>>> for.
>>>
>>> In general, what I settled upon is that serf remembers all in-flight
>>> requests so that if we don't get a response back we can re-send them
>>> if something happened (i.e. lost response or hitting the cap on the
>>> number of requests on that connection).  It's really the only  
>>> reliable
>>> mechanism for dealing with pipelined requests.
>>>
>>> Out of order pipelining would be nice for things like WebDAV, but a
>>> bigger problem is that the WebDAV protocol requires specific methods
>>> to be executed in lock-step using the results from the prior  
>>> response
>>> as input for the next request.  This is the largest protocol  
>>> headache
>>> I have with WebDAV at the moment.  Subversion would ideally like  
>>> to be
>>> able to use pipelining commits with WebDAV, but that'll likely  
>>> require
>>> us forking the protocol in some sense to let the underlying  
>>> methods be
>>> executed out-of-order as well.  -- justin
>>>
>>
>>
>> --
>> Mark Nottingham     http://www.mnot.net/
>>
>
> -- 
> - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial  
> Squid Support -
> - $25/pm entry-level VPSes w/ capped bandwidth charges available in  
> WA -
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Robert Siemer <[email-masked]>
Date: Mon, 3 Mar 2008 06:02:43 +0100

> >intermediaries/UAs rather than each trying their own implementation.
> 

I'm writing something up that outlines how to implement 
"perfect" _intermediaries_. It should be ready in June and I will 
present it here. For flaming. (-:

Bye,
	Robert

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 11 Mar 2008 17:19:46 +1100

Here's a summary of the problems people have mentioned so far;

>  - Handle the first m of n requests in order, then skip to request n  
> (IIRC I saw this on IIS/5)
>
>  - Handle the first request in a packet, ignore the rest, then  
> process the first in the next, and so on. I have seen this problem  
> in WebSeal 5 (IBM) where it was fixed, and then recently broke in  
> WebSeal 6, and is apparently fixed again recently. The problem is  
> particularly visible for HTTPS in Opera 9, but is not limited to  
> HTTPS.
>
>  - Handle one or more requests, then stall
>
>  - Handle a request, then stall when the next arrives.
>
>  - Not quite pipelining: Server sending Content-Length of original  
> content, then apply Content-Encoding without removing or updating  
> the Content-Length header. I have seen this happen with PHP using  
> global compression when the script sets the Content-Length header.
>
> serve. The default in httpd is 100 requests per connection, but  
> when testing ra_serf against Subversion's own repository.  I  
> eventually got them to up it back to the default.  serf tries to  
> figure out the limit heuristically (i.e. write as much as possible,  
> then count how many responses we got before the connection closed -  
> that'll be the limit going forward).
>
> - Lost responses are, sadly, real.  Later versions of httpd 2.0.x re- 
> introduced a lingering close bug where it won't wait for the last  
> accounted for.

My proposal: Hand this to the editors and instruct them to include a  
*short* (i.e., not necessarily this level of detail) note stating that  
pipelining is difficult to implement for clients (both UAs and  
intermediaries), giving some of the underlying causes (e.g., bad  
server implementations, difficult runtime choices) and leave it at that.

On 28/02/2008, at 11:00 AM, Mark Nottingham wrote:

> neglected to open an issue for it. See: <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/99 
> >.
>
>> to be executed in lock-step using the results from the prior response
>> as input for the next request.  This is the largest protocol headache
>> I have with WebDAV at the moment.  Subversion would ideally like to  
>> be
>> us forking the protocol in some sense to let the underlying methods  
>> be

---

From: "Josh Cohen (MIG)" <[email-masked]>
Date: Thu, 13 Mar 2008 22:59:50 -0700

Thanks for summarizing.  I think comments like you recommend would be helpful for readers and implementers of the spec.

You're also correct that this thread hasn't had much activity.  I think it's been starved by the discussion designing a new Link feature.