Subject: Bad header syntax -- is this par for the course?

From: Jamie Lokier <[email-masked]>
Date: Wed, 23 Jun 2004 12:42:25 +0100

Take a look at the HTTP response below, from a real server.
The line beginning with "CP" is clearly not valid HTTP/1.0 or HTTP/1.1
syntax.

Is this sort of thing commonplace?

I was rather hoping to write a proxy that could at least assume the
basic lexical syntax of HTTP/1.0 and /1.1 -- so as not to forward
invalid syntax, which is a security hole -- but it appears not.

Is there a well known of server/proxy bugs, and the workarounds needed
by a robust client/proxy in the real world, so I don't have to repeat
the research people have done before?

(There's a fairly good list of known client bugs at apache.org, but
they don't document server/proxy bugs).

Thanks,
-- Jamie

[jamie@mail jamie]$ telnet www.qvcuk.com 80
Trying 167.140.19.50...
Connected to www.qvcuk.com.
Escape character is '^]'.
HEAD / HTTP/1.1
Host: www.qvcuk.com

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 23 Jun 2004 11:35:32 GMT
CP="IDC DSP COR LAW CURa ADMi DEVi TAIi PSAi PSDi OUR IND UNI": CP="IDC DSP COR LAW CURa ADMi DEVi TAIi PSAi PSDi OUR IND UNI"
X-Powered-By: ASP.NET
Connection: close
Content-Type: text/html

Connection closed by foreign host.
[jamie@mail jamie]$

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 23 Jun 2004 08:43:13 -0600 (MDT)

> Is this sort of thing commonplace?

Common or not, it does happen. When a proxy writer/administrator is
faced with the "but it works without a proxy!" or "but it works
through XYZ proxy!" pressure, there is little she can do about them,
given our overall "garbage in, compliance out" culture.

> invalid syntax, which is a security hole -- but it appears not.

A common approach is to switch to a tunnel mode for the transaction in
question and terminate the connection as soon as possible. This
approach follows the "first, do no harm" principle for intermediaries.
As any approach within the current IT culture, it may have negative
security implications.

> they don't document server/proxy bugs).

Oh, they do, but in a different place: Apache bugzilla database :-).

Alex.

---

From: Robert Olofsson <[email-masked]>
Date: Wed, 23 Jun 2004 20:02:01 +0200

>Is this sort of thing commonplace?
Sort of, many servers are broken and output:
Content-Type: text/html\n\n<resource data>

Instead of the correct one:
Content-Type: text/html\r\n\r\n<resource data>

There are _many_ perl scripts (and other) that do this.
So being strict for CRLF parsing is a bad idea.

Apart from that I dont have to worry about broken http much.
Expire dates seem to have all sorts of funny values, but that
is ok for me, only means that the cache will expire quicker.

/robo

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 23 Jun 2004 19:49:34 +0100

> So being strict for CRLF parsing is a bad idea.

That shouldn't be possible if the perl script is run through a decent
CGI-supporting HTTP server -- as the HTTP server should canonicalise
the line endings.  However, obviously many are not, not to mention the
Perl scripts which are servers in and of themselves.

Fortunately, the HTTP standards warn of this and suggest just looking
for \n and skipping a preceding \r.

That's what I'd hope for: a document which contains all of the
well-known suggestions for contemporary interoperation.

For example, I have read that Mozilla, IE and Netscape browsers will
all accept \r\n\r\r\n as the "blank line" which ends headers -- and that
they must do so.  I'm not sure if this is true, I have simply read it.

It's not hard to imagine what kind of code generates \r\n\r\r\n

I read the header parsing of Mozilla, and it accepts headers with no
colon in the line -- effectively it ignores them (though it actually
stores them verbartim).  I presume that's because of buggy IIS servers
which, according to the code in Squid, occasionally send 200 Blah Blah
lines in the middle of the headers.

See how much exciting historical information is hidden in these code bases!

I worry about little things like some clients/servers/proxies accept
lines with embedded \r's (not followed by \n), and some treat those as
line endings while others do not, while yet others reject such lines.
Some skip spaces before the header name, others do not, yet others
correctly treat them as continutation lines.  Some treat \r as a space
for this, others do not.  Some skip spaces after the header name,
others do not.  Some that do skip spaces after the header name, treat
\r as a non-space in that place but treat \r as a space in other parts
of the same line; some others treat \r as a space everywhere.  Some
treat a first header line which begins with space as an error, others
as a header begining with a space even despite concatenating later
such lines as continuations.  Some treat a line containing only spaces
as the "blank line" separator, others do not.

Think of the delightful security possibilities, knowing that some
proxies see some headers and other proxies/clients/servers see other
headers in the same text!

And that's just basic isolating of headers.  We haven't even touched
on the _values_ of headers.

-- Jamie