Subject: Re: On the abuse of chunking for interactive usages

From: Willy Tarreau <[email-masked]>
Date: Tue, 10 May 2011 23:59:46 +0200

Hi Poul-Henning,

On Tue, May 10, 2011 at 09:45:16PM +0000, Poul-Henning Kamp wrote:
> In message <[email-masked]>, Willy Tarreau writes:
> 
> >My concern comes from this specific use of chunking. I see no reason why
> >this would be valid. I know it will not work at many places. Some proxies
> >(such as nginx IIRC) buffer the complete request before passing it on.
> 
> Well, it doesn't say anywhere in RFC2616 that "A server SHALL NOT
> reply until it has received the entire request including the body,
> if any.", so from a standards point of view, it is not "invalid".

It's not the fact that the server replies earlier which is a concern
to me, as anyway it's allowed to do so (eg: redirect, 401, etc...).
It's the fact that both ends agree that they're free to do whatever
they want on the TCP connection despite of the messaging rules.

> But we can readily agree that it will probably come as a major
> surprise to most readers of RFC2616 that anybody would do this.

As well as I'm sure that the implementers did this in good faith, they
probably didn't notice anything clearly stating it's not intended to
work. And the worst we can do is to have two incompatible designs both
justify their choices based on the same spec (though I'd argue that once
you read all RFC2616 you get the general idea that HTTP is not meant to
be used that way).

> And most HTTP proxies (as opposed to HTTP load-balancers) will
> not permit it through at all.

Indeed.

> You can configure Varnish to let this through:  Varnish has a
> "pipe" mode, where after the first request on a TCP connection has
> been passed to the backend, varnish just moves bytes forth and back.

It's the same in haproxy, the old mode which I renamed "tunnel mode"
did only analyse the first request and response of each TCP connection.
I'm pretty sure that many HTTP intermediaries started that way because
it's not too hard to do and it covers many usages. But I now want to
divert users from that mode because it's incompatible with keep-alive
and is not particularly secure.

> But that is merely a reflection of my "code, not policy" attitude
> to software tools, I agree with you that this should be discouraged
> as much as possible.

Any code more or less indicates a trend and usage. After all, many
items listed in the draft come from observations of what has been
implemented.

Thanks for your feedback, it's really valuable and useful to me, it
comforts me in the idea that I'm not the extremist guy on the subject
(or not the only one :-)) and that there are huge chances that such a
project experiences many similar or worse issues in the future. So I'm
still thinking I won't change the default behaviour but I'll add an
option to let such users have the performance they're expecting.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 10 May 2011 21:45:16 +0000

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 10 May 2011 23:26:15 +0200

Hi,

a few days ago, a user of haproxy explained to me that he was experiencing
extremely long delays between two servers when communicating through haproxy
while the two servers in direct had no issue.

He was kind enough to provide logs and network traces with a bit of explanation
on the data exchanges.

What is happening is that one server connects to the other one using a PUT
request, makes use of chunked encoding to send data and the second one sends
a chunked encoded response in turn. The protocol is specified here :

   http://php-java-bridge.sourceforge.net/doc/index.php

The issue comes from the fact that the protocol assumes that messages are
fully interactive and that whatever chunk is emitted on one side is
immediately received on the other side and conversely. So each chunk
serves as an ACK for the other one, making the workload consist in
many very small TCP segments (around 10-20 bytes of payload each).

Obviously this can only work on local networks with extremely low RTT.

The issue comes when a gateway is inserted in the middle. Haproxy was built
on the assumption that messages live their own life and that there is no
direct relation between a chunk on one side and a chunk on the other side.
And in order not to flood the receiver with TCP PUSH packets, it aggregates
as many data as possible in each segment (MSG_MORE, equivalent on TCP_CORK)
until the last chunk is seen.

What was causing the massive slowdown is that for each 10-bytes payload seen,
haproxy was telling the system "hey, please hold on for a while, something
else is coming soon". The system waits for 200ms, and seeing nothing else
come, finally sends the chunk. The same happens on the other direction,
resulting in only one req/resp being exchanged every 400 ms.

The workaround, which the user confirmed fixed the issue for him, consists
in sending all chunks as fast as possible (TCP_NODELAY). But doing this
by default makes very inefficient use of mobile networks for normal uses,
especially with compressed traffic which generally is chunked. The issue
is now that each chunk will be sent with the TCP PUSH flag which the client
has to immediately ACK, resulting in a massive slowdown due to uplink
congestion during downloads.

I can also improve the workaround so that haproxy asks the system to wait
only when there are incomplete chunks left, but still this will not cover
the mobile case in a satisfying way. So I'm now tempted to add an option
to let the user decide whether he makes (ab)use of chunking or not.

My concern comes from this specific use of chunking. I see no reason why
this would be valid. I know it will not work at many places. Some proxies
(such as nginx IIRC) buffer the complete request before passing it on. And
in fact many other ones might want to analyse the beginning of the data
before deciding to let it pass through. Also I don't see why we should
accept to turn each chunk into a TCP segment of its own, this seems
contrary to the principle of streamed messages.

My understanding has always been that the only feature that an intermediary
could guarantee is that one all the request body has been transferred, it
will let all the response body pass.

Am I wrong somewhere ? Shouldn't we try to remind implementers that there
is no guarantee of any type of interactivity between two opposite streams
being transferred over the same connection ? I'm worried by the deviations
from the original use. In fact the project above seems to have tried to
implement websocket before it was available. But the fact that some people
do this probably means the spec makes think this is something that can be
expected to work.

Any insights are much appreciated. I've not yet committed on a fix, and I'm
willing to consider opinions here to find the fairest solution for this
type of usage without unduly impacting normal users.

Thanks,
Willy

---

From: Brian Pane <[email-masked]>
Date: Tue, 10 May 2011 15:13:06 -0700

> to software tools, I agree with you that this should be discouraged

I can imagine variants of Willie's use case that happen even in the
absence of chunking: an origin server receiving a large file upload
might attempt to stream back a response "10% complete ... 20% ..."
incrementally.  Thus I think the general issue is:  Should a
proxy be allowed to (or prevented from, or perhaps even required
to) stream a request though toward the origin server before it
receives the entirety of the request?

I think the "allowed to" option is the most pragmatic approach.
There are proxies that have good reasons for buffering the entire
request body (ease of implementation, scanning of POST payload
for malicous input), and there are proxies that have good reasons
for not buffering the request body (minimization of memory usage,
denial-of-service avoidance).  It seems prudent to let each
implementor decide what tradeoffs are best for their implementation.

-Brian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 00:29:04 +0200

On Tue, May 10, 2011 at 03:13:06PM -0700, Brian Pane wrote:
> incrementally.

yes that's one valid point.

>  Thus I think the general issue is:  Should a

Well, we can't prevent a proxy from doing so because we can't require
it to buffer all the response. We can't make it mandatory to do so
either because some proxies will need to analyse everything before
letting the request pass (think webmails + anti-virii). So we must
only let them do what they feel right for what they do. In fact there
is a large spectrum of possibilities depending on the usage.

Indeed.

> implementor decide what tradeoffs are best for their implementation.

Exactly. My concern is not with proxies, as I think they all have valid
reasons to act as they do (and difficulty to implement support for either
method is a valid reason too in my opinion). My concern is with people
spending time inventing protocols that will not be able to work on top of
100%-compliant proxies. There surely is something that can be done so that
those people don't waste their (nor their users') time implementing and
deploying something which will break once widely deployed.

And since it's not the first time I read suggestions of such a way of
turning HTTP into a full duplex link, I think there is room left for
ambiguity. It has even been evocated on the hybi ML several months ago.
And a decade ago I used to abuse squid's interactive handling of POST
to telnet home from customers' so I can't blame implementors for trying
to use the protocol that way :-)

Regards,
Willy

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 11 May 2011 10:42:03 +1200

that behaviour should be strongly discouraged.

It will break on anything that

a) re-chunks the data stream (e.g. due to nagling etc)
b) spools data streams
c) has plug-in filters that do things

The client and server have to treat the entire message as the entire 
message.  There's nothing can be guaranteed about the timing of parts of 
a message.  To rely on such a thing is therefore worse than foolhardy, 
since it creates problems for intermediary implementors when the 
customer says "it works fine without the proxy", and creates pressures 
on intermediary implementors to compromise security and features to deal 
with a broken application.

just my 2c.  It will certainly break through WinGate, especially with AV 
installed.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 06:52:40 +0200

Hi Adrien,

> with a broken application.

That's exactly what I experienced and what intermediary implementors are
used to see every day :-/

> just my 2c.  It will certainly break through WinGate, especially with AV 

OK.

I'd suggest that we add something like this at the end of p1-3.3
"Message Body" :

  Implementations MUST NOT rely on the ability to receive an incomplete
  message body. Full or partial message buffering by intermediaries or
  application servers, as well as re-chunking MAY impact user experience
  (eg: progress bar sent at once, or streamed video starting when fully
  transferred), but MUST NOT break the intended use. As such, applications
  MUST NOT expect bessage bodies to be usable as a bidirectional stream
  over which conversations are interleaved between the user-agent and the
  origin server.

Any opinion ?

Thanks,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 11 May 2011 15:28:03 +1000

I think you're generally right.

See also:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/283

Suggestions for text welcome.

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 07:55:13 +0200

Hi Mark,

>   http://trac.tools.ietf.org/wg/httpbis/trac/ticket/283

I was not aware of this one, it perfectly matches the issue indeed.

> Suggestions for text welcome.

I don't know if you noticed my last reply to Adrien, I suggested a
few lines to add to the end of p1-3.3.

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 11 May 2011 16:28:21 +1000

Just saw it. Looks good, except I don't think we need to use so many RFC2119 MUST and SHOULDs; those are better to reserve for testable assertions. I'll get it into the bug so we don't forget it.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 08:42:49 +0200

> Just saw it. Looks good, except I don't think we need to use so many RFC2119 MUST and SHOULDs; those are better to reserve for testable assertions. I'll get it into the bug so we don't forget it.

OK, WFM.

Thanks,
Willy

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Wed, 11 May 2011 08:16:25 +0100

Willy,

> over which conversations are interleaved between the user-agent and the

Looks generally good but it's not clear (to me at least) what "MUST NOT break the intended use" refers to.

Ben

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 09:53:24 +0200

> break the intended use" refers to.

What I meant is that whatever service the application in intends to serve,
the effect of buffering must not stop the application from working.

Example 1:
   video streaming sent by chunks might start playing only once download
   is complete. However, once transferred, it works, so even if this causes
   discomfort to the end user, it does not break the original goal of the
   application.

Example 2:
   an application which sends exactly one image in each chunk, hoping
   for the recipient to use chunks as delimiters might break if an
   intermediary rechunks them.

Example 3:
   an application which is based on short request-response transfers
   inside a single message body has a lot of chances to break once any
   intermediary buffers.

I don't know if it's clearer now. Probably we'd find a better rewording.

Willy

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 11 May 2011 13:01:03 +0100

>    discomfort to the end user, it does not break the original goal of the

I know of several *infinitely* long video streaming applications, and
a plenty of internet radio audio streaming applications, which expect
to be able to stream continuously.

They will break if an intermediary tries to buffer it all.

>    intermediary rechunks them.

I see rechunking as similar to TCP packet boundaries.  Nobody should
depend on chunk boundaries to mean anything.

> I don't know if it's clearer now. Probably we'd find a better rewording.

I think there are 3 different issues being mixed up here:

  1. Streaming or buffering a message (in one direction only).
  2. Streaming request while streaming response.
  3. Depending on reliable chunk boundaries.

I think it's important to distinguish them, instead of declaring them
all invalid while lumped together.

1. Streaming or buffering a message (in one direction only)
-----------------------------------------------------------

1a: The ability to continuously stream a response is a fairly well
established use for some applications, such as internet radio.  Those
applications don't require devery byte to be forwarded immediately,
only that they are delivered in a reasonable time.

1b: A fair number of applications expect HTML or Javascript response
streaming to work.  HTML streaming goes back to the early days of
Netscape Navigator - when it was possible to transmit multiple HTML
documents, each replacing the previous one, and it was *expected* that
they'd be delivered in a reasonable time.  Javacript streaming is what
people do with iframes these days (still) in lieu of XmlHttpRequest
for a variety of reasons.  However there are still question marks
around how reliable streaming is in practice.

1c: It's clear that if an intermediary buffers whole messages, it will
limit the message size.  For example if you PUT or GET an 80GB file
(haven't we all), not many full-buffering proxies are going to be
happy with that.  Yet it is clearly a valid use of RFC2616 HTTP.  That
would be an example of something which works until you insert a proxy
(of a certain kind) where you can't really complain about the
application, and perhaps should complain about the proxy.

1d: For unidirectional streaming, it should be noted that CONNECT's
buffering/forwarding behaviour is not fully specified either
(e.g. forward every byte eagerly, or delay for a *limited* time to
coalesce TCP segments).  If a proxy has a sane CONNECT implementation
it's not unreasonable for it to use the same strategy for streaming
*unidirectional* requests or response, if it does not have a reason to
buffer them more.

1e: In practice, if unidirectional streaming is not reliable enough,
the common strategy is to transmit a stream of bulky HTTP entire
messages instead to accomplish the same task.  Or to try using CONNECT.

2. Streaming request while streaming response
---------------------------------------------

2a: This is certainly debatable and/or dubious.  I know from
experiments it does not work with all browser clients.  Sometimes the
client waits until it has sent the whole request before reading the
response (so deadlock is possible if the server doesn't handle this).

2b: I wouldn't expect all proxies to support this, even those which
handle CONNECT fine, which is a shame as it would be quite useful if
it worked.

2c: In practice applications can simply open two HTTP connections, and
stream one direction over each connection (at some loss of
efficiency).  This removes the question of bidirectional streaming, while
still depending on unidirectional streaming behaviours.

3. Depending on reliable chunk boundaries
-----------------------------------------

3a: Chunk boundaries are analogous to TCP segment boundaries:
Absolutely nothing should *ever* depend on their position.  If a
protocol wants to do something fancy with incremental messages, that
should be encoded in the data bytes of the messages *only*.  Even
transparent proxies may rechunk arbitrarily according to instantaneous
low-level TCP states.

3b: It isn't specified, but any behaviour that depends on a proxy
forwarding bytes in a reasonable time also requires it to forward
*parts* of chunks in a reasonable time.  That follows from the chunk
boundaries having no semantic significance, and practically the fact
an earlier sender may merge chunks arbitrarily.

3c: The sole entire purpose of chunks is to allow the sender to
terminate the data stream when it is ready without advance knowledge
of the message size.  This is why chunk boundaries don't (or
shouldn't) mean anything. It's just a mechanism for encoding an out
of band "end of message".

3d: The example which started this thread, of an application expecting
individual chunks to be forwarded through haproxy immediately, is
broken by design if it requires chunk boundaries to be preserved.
This is *separate* from whether it requires simultaneous request and
response streaming, and whether it requires its data bytes to be
forwarded in a reasonable time (and in that case it should use a
sliding window so that small delays don't stall the protocol's
progress disproportionately).

Finally: Buffering / buffer timing behaviour of CONNECT is not
specified either.  It is commonly understood that all bytes must be
forwarded in a reasonable time (otherwise HTTPS wouldn't work), but
not whether they are forwarded *eagerly*, or delayed for a short time
for TCP efficiency, as haproxy does to streamed messages in the
example which started this thread.  An application like the one which
started this thread, but running over CONNECT, may well be bitten by
the same delay RTT issue over CONNECT with a proxy in the middle, and
this is likely to affect WebSocket users who design protocols
sensitive to RTT as well.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 11 May 2011 15:14:32 +0200

Hi Jamie,

> They will break if an intermediary tries to buffer it all.

Indeed, I agree that for *infinitely* long streaming it makes a difference.
Maybe some rewording is necessary then, but the general idea is that an
application must not expect any partial or complete chunk to be delivered
ASAP.

> depend on chunk boundaries to mean anything.

I agree with your analogy.

(...)
> 1. Streaming or buffering a message (in one direction only)

I agree with your principle of "reasonable delay". A 200ms delay for
a streaming radio is not an issue at all, and at worst, all the streamcast
will be delayed by 200ms. But when expecting interactive uses, it becomes
a problem.

> around how reliable streaming is in practice.

OK, but still, you can't prevent an intermediary to require more bytes
to be sent in order to complete a processing. For instance, you advertise
that there are more data to come, so the next hop waits for those because
an intermediate encoding needs full blocks, and it will block until you
send the next chunk. For instance, if data were to be encoded in base64
in the middle, it would be reasonable to expect that any block that is
not multiple of 3 will wait for either 1 or 2 next bytes, or for an end.

> application, and perhaps should complain about the proxy.

I agree, and that's what happens with content analysis at many places.

> *unidirectional* requests or response, if it does not have a reason to

That's a good point. That said, I've never seen a proxy which delays
data in CONNECTs, probably because if they did so, they'd quickly
notice that HTTPS does not work well at all.

> messages instead to accomplish the same task.  Or to try using CONNECT.

I perfectly agree. That was really my point about the "abuse" of chunking.

> 2. Streaming request while streaming response

We could even imagine that some proxies will stop streaming the request
as soon as they see the server start responding, because it's pointless
to make the client send more bytes if the server appears to be happy
with what it has seen (think about a login redirect upon a webmail POST
involving a large attached file for instance)..

> still depending on unidirectional streaming behaviours.

Yes.

> 3. Depending on reliable chunk boundaries

I 100% agree with your point, eventhough I'm fairly sure I've observed
horrors in the past (streamed images with 1 chunk == 1 image).

> an earlier sender may merge chunks arbitrarily.

Yes, but as I explained, when you're saying someone "Hey, I'm sending
you 10 bytes" and only emit 9, it's hard to expect it to forward those
9 bytes alone without waiting for the last one. Some internal block
processing might even not easily permit to do so.

> of band "end of message".

Perfectly agree too.

> broken by design if it requires chunk boundaries to be preserved.

I don't think it requires boundaries, however since it uses only one
chunk at a time per direction, chunk merging cannot happen. Still, we
could imagine an intermediary which rechunks once in a while when a
chunk crosses an internal buffer boundary, resulting in two outgoing
chunks for an incoming one.

> progress disproportionately).

That's the problem with this protocol : it alternatively sends 10 bytes
at a time in each direction. Even a delay of one millisecond makes the
protocol stall to at most 1000 packets per second ~= 10 kB/s.

> example which started this thread.

At least if CONNECT or GET+Upgrade had been used there, no delaying would
have been performed (speaking for haproxy but other products seem OK on
this point).

> An application like the one which

The RTT issues will always impact poorly designed protocols which work
with short request/response patterns. TCP makes that doubly worse,
because short reqs means a PUSH which generally implies an immediate
ACK on the other side.

Thanks for your insights, Jamie, you are right that I probably mixed
several issues in a single report, probably because I was concerned by
all the dirty misuses of a message body.

Best regards,
Willy