Subject: i19 Bodies on GET (and other) requests

From: Mark Nottingham <[email-masked]>
Date: Mon, 15 Jan 2007 17:35:48 +1100

Background at: <http://lists.w3.org/Archives/Public/ietf-http-wg/ 
2006AprJun/0103>

Does anybody have any new information / thoughts about this?

My personal take: it seems like the most expedient thing to do would  
be to go through each method defined by 2616 and explicitly state  
whether it allows a body or not, so as to remove the ambiguity. Also,  
it would be good to recommend that new method definitions (perhaps in  
an IANA registry?) also include this information.

Straw-man list:
GET - no
PUT - yes
DELETE - no
POST - yes
TRACE - yes
OPTIONS - yes
CONNECT - n/a

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 15 Jan 2007 03:18:35 -0600

Mark Nottingham wrote:
> 
> Background at:
> <http://lists.w3.org/Archives/Public/ietf-http-wg/2006AprJun/0103>
> 
> My personal take: it seems like the most expedient thing to do would be
> to go through each method defined by 2616 and explicitly state whether
> it allows a body or not, so as to remove the ambiguity. Also, it would
> be good to recommend that new method definitions (perhaps in an IANA
> registry?) also include this information.
> 
> Straw-man list:
> GET - no
> PUT - yes
> DELETE - no
> POST - yes
> OPTIONS - yes

I can put xml descriptors of the desired data to any of these requests.
Any time you can plug in ?query_args in some possibly meaningful way,
you can do the same with a post body.  But I'd have to review the spec
to determine this is absolutely true of GET, DELETE and OPTIONS.

> TRACE - yes

But TRACE explicitly disallows it (Apache supports it with an explicit
toggle, for doing echo testing and validation in an administrator
controlled manner, and we spell out the option breaks the spec.)

> CONNECT - n/a

I'd agree with you, I can't resolve a meaning under this context.

---

From: "Mark Baker" <[email-masked]>
Date: Mon, 15 Jan 2007 08:15:33 -0500

My take would be to not say anything about this because, as Roy has
pointed out before (e.g. [1]), HTTP 1.1 request messages are
self-descriptive about the presence or absence of a body independent
of the method.

 [1] http://lists.w3.org/Archives/Public/ietf-http-wg/2002JulSep/0031.html

Mark.

-- 
Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
Coactus; Web-inspired integration strategies  http://www.coactus.com

---

From: Scott Lawrence <[email-masked]>
Date: Mon, 15 Jan 2007 08:34:31 -0500

> Does anybody have any new information / thoughts about this?

It seems to me that inferring anything about the presence or absence of
a body based on the method only creates ambiguous situations when that
inference is in conflict with the explicit indications already defined
by the protocol (the Content* and Transfer* headers).  

The only exception to this should be HEAD.  In retrospect, I think that
HEAD should have defined special headers to express what would have been
in the body descriptors so that there was no ambiguity
(head-content-length would have the value that would have been in
content-length if the method had been GET, etc).

-- 
Scott Lawrence
http://skrb.org/scott/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 15 Jan 2007 14:50:44 -0800

> by the protocol (the Content* and Transfer* headers).

Right, it would just create contradictions where none are necessary.

> content-length if the method had been GET, etc).

Heh, we realized way back in 1994 that HEAD should have been defined
to return a body containing the GET-headers, but even back then it was
way too late to fix that weirdness.  At the time, content-length was
just advisory (for the progress bar) anyway, so it was actually a
significant change to the message semantics to enable persistent
connections work at all over HTTP/1.x.

It is absolutely critical that the algorithm for determining the
presence or absence of a body have no dependence whatsoever on the
semantics of the method (aside from HEAD).  Failure to sustain that
design would mean that all proxies must block all new methods.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 16 Jan 2007 10:16:06 +1100

This is fairly clear upon a careful reading of 4.3;

> The presence of a message-body in a request is signaled by the  
> inclusion of a Content-Length or Transfer-Encoding header field in  
> the request's message-headers. A message-body MUST NOT be included  
> in a request if the specification of the request method (section  
> 5.1.1) does not allow sending an entity-body in requests. A server  
> SHOULD read and forward a message-body on any request; if the  
> request method does not include defined semantics for an entity- 
> body, then the message-body SHOULD be ignored when handling the  
> request.
However, I think casual readers -- and some implementers -- have been  
tripped up because many method definitions are silent on the matter,  
and don't *explicitly* allow sending an entity-body. It might be  
clarified by changing "does not allow" to "explicitly disallows".

Also, devil's advocate;
1) What are the implications of an entity-body on GET for the caching  
model? The spec is silent on this AFAICT.
2) Ditto for content negotiation.
3) Are there any existent origin server or intermediary  
implementations that are fully conformant with this requirement?

Cheers,

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jan 2007 01:06:25 +0100

> Does anybody have any new information / thoughts about this?

The HTTP protocol message format is quite well defined in when a request
body may be allowed.. pretty much at any time except where it is
forbidden. But in many requests the meaning of said request body is
undefined by HTTP/1.1, but may well be defined by other application
protocols building on HTTP/1.1 as long as it doesn't conflict with
HTTP/1.1. Naturally such uses will be quite limited, but still..

I know the thread perhaps a bit too well, being one of the guilty ones
who instinctively blocked GET requests with a request body. But in
retrospect I did knew what the specs said, just didn't like the effects
it could have on the service provided by the software I write or having
to cover odd undefined cases rarely if ever seen in real life.. so we
blocked them to see if it would cause any problems, which it did some
many years later.. (and by which time we had almost forgot why)

But with applications already out in the field doing this kinds of
requests for various reasons it will be a bit tricky to get them covered
by the specs if straightened up to not allow request bodies where their
use is nonsense under the semantics of HTTP/1.1.

The perhaps biggest problem, apart from some implementations blocking
such requests as "nonsense use of HTTP" is that it may be used as a
covert channel to smuggle data out from a network. But there is a large
number of those in HTTP and related services so not that big of a
problem..  Hmm.. maybe there is also request smuggling attacks possible
here if there is some server/proxy software ignoring that there may be a
request body..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jan 2007 01:18:28 +0100

> clarified by changing "does not allow" to "explicitly disallows".

Agreed. It's after all what is meant.

> model? The spec is silent on this AFAICT.

There is none. Implied from

4.3 

   A server SHOULD read and forward a message-body on any request; if
   the request method does not include defined semantics for an
   entity-body, then the message-body SHOULD be ignored when handling
   the request.

> 2) Ditto for content negotiation.

See above.

But Vary covers it via the * should such uses arise.

> implementations that are fully conformant with this requirement?

Apache (by default), Squid (if enabled). Quite likely many others.

Regards
Henrik

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 15 Jan 2007 18:23:21 -0600

> model? The spec is silent on this AFAICT.

Cache-control: no-cache, unless one wants to actually cache such request
bodies and test for identity.  GET is idempotent (9.1.1).

But I don't expect most proxies today behave that way.  Therefore, the
server should consider 12.1..

  "...an
   origin server is not limited to these dimensions and MAY vary the
   response based on any aspect of the request, including information
   outside the request-header fields or within extension header fields
   not defined by this specification."

Vary: *

???

> 2) Ditto for content negotiation.

That's an implementation-dependent issue.

> 3) Are there any existent origin server or intermediary implementations
> that are fully conformant with this requirement?

Interesting question;  AIUI httpd 2 will proxy request bodies for GET etc,
I'm unsure if it's cache behavior.

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jan 2007 02:01:42 +0100

> here if there is some server/proxy software ignoring that there may be a

Hmm.. looked back a bit in my notes and find the latter to be the reason
why Squid still rejects such messages by default, protecting the cache
from request splitting attacks.. Our configuration notes even reads

request_entities

        Squid defaults to deny GET and HEAD requests with request entities,
        as the meaning of such requests are undefined in the HTTP standard
        even if not explicitly forbidden.

        Set this directive to on if you have clients which insists
        on sending request entities in GET or HEAD requests. But be warned
        that there is server software (both proxies and web servers) which
        can fail to properly process this kind of request which may make you
        vulnerable to cache pollution attacks if enabled.

Note: details on what software the above refers to have unfortunately
been lost over the years.

As a special case "Content-Length: 0" request messages is not restricted
by this directive as they are safe from the request injection problem (a
bit hard to fit a new request in 0 octets).

Regards
Henrik

---

From: Scott Lawrence <[email-masked]>
Date: Tue, 16 Jan 2007 10:56:49 -0500

> clarified by changing "does not allow" to "explicitly disallows".

This is a general problem: some people want everything spelled out in
every possible place where it might be relevant, others would prefer
that each statement be made only once so that the spec is as short as
possible but you must read all of it to understand it.  I'm strongly in
the latter camp.  From my point of view, the current spec is too long by
at least half, and the repetition of some rules in some places leads the
unwise to assume that if that rule was repeated here and wasn't repeated
there, then it must not apply there...

> model? The spec is silent on this AFAICT.

Again - the spec has clear mechanisms for providing explicit cache
controls - they should be used, and nothing should be inferred that they
do not make explicit.  IMHO, any cache should assume that any response
is not cachable unless told that it is.

> implementations that are fully conformant with this requirement?

Which requirement?

-- 
Scott Lawrence
http://skrb.org/scott/

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jan 2007 22:50:00 +0100

> is not cachable unless told that it is.

HTTP works the other way. GET responses is by default cacheable unless
told otherwise (13.4, 13.2), except for responses to query-URLs which is
a oddball special case and requires explicit freshness information
(13.9).

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jan 2007 21:17:26 +0000

Henrik Nordstrom wrote:
> Hmm.. maybe there is also request smuggling attacks possible

See also "Content-Length : 12345" (note the space).  I think that is
interpreted as a Content-Length header by some agents, and a
"Content-Length " header by others (i.e. not implying a body), and
disallowed as bad syntax by others.  Ample opportunities for request
smuggling.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jan 2007 23:45:14 +0100

> disallowed as bad syntax by others.  Ample opportunities for request

Yes... Hmm.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Jan 2007 09:49:11 +1100

On 2007/01/17, at 2:56 AM, Scott Lawrence wrote:

> the latter camp.

FWIW, I'm pretty strongly in it as well, provided that it's  
relatively easy to find the relevant portion of the spec.

>   From my point of view, the current spec is too long by

OK, but the proposal reduces the number of words in the spec by one,  
and increases the number of characters by six... surely you're not  
concerned to that degree?

> is not cachable unless told that it is.

I strongly agree with the sentiment, but heuristic freshness *is*  
explicitly allowed.

However, my question was more to how, if at all, a request entity  
body on GET affects a cache's operation when the response is clearly  
cacheable; e.g.,

C: GET http://example.com/foo HTTP/1.1
C: Content-Length: nnn
C: Content-Type: application/xml
C:
C: <?xml version="1.0" encoding="utf-8"?>
C: <thing>I am a request entity body, hear me roar.</thing>

S: HTTP/1.1 200 OK
S: Content-Length: nnn
S: Content-Type: application/xml
S: Cache-Control: max-age=600
S:
S: <?xml version="1.0" encoding="utf-8"?>
S: <thing>And I'm a response entity body.</thing>

My reading is that this would be cacheable, and that the cache key  
would be just the URI. In particular, the last sentence of the quoted  
spec text above helps clarify this, although it's talking about  
servers, not necessarily caches.

Does that make sense? I'd feel a bit better if the language were  
massaged to include not just servers, but caches as well. E.g.,

"A server SHOULD read and forward a message-body on any request. If  
the request method does not include defined semantics for an entity- 
body, or if the request method is unrecognised, then the message-body  
SHOULD be ignored by servers and caches when handling the request."

>> implementations that are fully conformant with this requirement?

... that "A server SHOULD read and forward a message-body on any  
request".

Cheers,

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jan 2007 15:20:51 -0800

On Jan 16, 2007, at 2:49 PM, Mark Nottingham wrote:
> would be just the URI. In particular, the last sentence of the  
> quoted spec text above helps clarify this, although it's talking  
> about servers, not necessarily caches.

Don't get the terminology messed up.  Caches are storage things that
could be on either clients or servers.  Servers are components that
happen to be playing the role of sending a response at this moment.
As such, the only thing it should talk about is servers -- whether
or not a cache exists in the server is irrelevant.

> body, or if the request method is unrecognised, then the message- 
> body SHOULD be ignored by servers and caches when handling the  
> request."

I disagree with both sentences.  The first one is just wrong -- it
contradicts both the rules for message parsing (fails to account for
HEAD requests) and lumps all servers into the category of proxies
and gateways.  In the second sentence, caches do not "handle the  
request",
ever.  The cache is part of the server and subject to its requirements.

When a request message contains both a message-body of non-zero
length and a method that does not define any semantics for that
request message-body, then an origin server SHOULD either ignore
the message-body or respond with an appropriate error message
(e.g., 413).  A proxy or gateway, when presented the same request,
SHOULD either forward the request inbound with the message-body or
ignore the message-body when determining a response.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 17 Jan 2007 00:45:53 +0100

> SHOULD be ignored by servers and caches when handling the request."

Looks good to me, but I don't think it really matters. The specs is
quite clear, it just takes time to grok the "ignore" aspect of GET
request message bodies when using such messages doesn't make sense. In
worst case an implementer takes the safe route and selects not to cache
the response.

But the security issues related message bodies deserves a separate
discussion in what can be done in the specs to improve the situation.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Jan 2007 13:09:56 +1100

> or not a cache exists in the server is irrelevant.

That's what I meant by "it's talking about servers, not necessarily  
caches."

>> "A server SHOULD read and forward a message-body on any request.  
>> If the request method does not include defined semantics for an  
>> entity-body, or if the request method is unrecognised, then the  
>> message-body SHOULD be ignored by servers and caches when handling  
>> the request."
>
> and gateways.

2616 already says "A server SHOULD read and forward a message-body on  
any request; if the request method does not include defined semantics  
for an entity-body, then the message-body SHOULD be ignored when  
handling the request." so it seems that this problem already exists.

> In the second sentence, caches do not "handle the request",

Are you saying that a cache on such a proxy or gateway SHOULD NOT  
return a fresh response from cache, but instead forward the request,  
if a GET request has a message-body?

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Tue, 16 Jan 2007 20:47:09 -0600

> discussion in what can be done in the specs to improve the situation.

Security issues are caused by implementors.  Please reread the Watchfire
report carefully to observe all the ways an implementor can do so.

But don't cloud the spec solving a non-issue which the spec clearly
defined for interoperability.  No conforming server or proxy agent
was subject to the HTTP Request Splitting vulnerabilities.  (Which
is to say all were, but it was very clear in each case what the
implementor had done wrong.)

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jan 2007 18:52:14 -0800

On Jan 16, 2007, at 6:09 PM, Mark Nottingham wrote:
> return a fresh response from cache, but instead forward the  
> request, if a GET request has a message-body?

No.  A cache does not "return" and does not "forward".  You aren't using
the definitions as stated in the specification.

    cache
       A program's local store of response messages and the subsystem
       that controls its message storage, retrieval, and deletion. A
       cache stores cacheable responses in order to reduce the response
       time and network bandwidth consumption on future, equivalent
       requests. Any client or server may include a cache, though a  
cache
       cannot be used by a server that is acting as a tunnel.

You are assuming that a cache is a server because that is how the term
is often used in our line of work.  That is fairly common, and you are
right that 2616 has several places where the word cache is misused.
What I suggested was:

Note that a proxy or gateway "SHOULD either forward ... or ignore".
That entirely covers all possible combinations of server, with or
without cache, and without the contradictions that were added to 2616.
When a proxy or gateway uses a cache, it is "determining a response".
We could add "determining a response though use of an internal
cache and/or configurable error mechanism", but then we would be
making a guess about the scope of ways in which a proxy or gateway
might be implemented, rather than specifying an HTTP interface
requirement.

....Roy

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Tue, 16 Jan 2007 20:57:05 -0600

> handling the request." so it seems that this problem already exists.

I misread this as Roy has; s/message-body on any request/any valid
request message-body/ has no HEAD issues, but "forward a message-body"
looked to both of us as response body, if I'm guessing right.

"unless explicitly prohibited by the method" would be worthwhile here,
since TRACE, for example, explicitly prohibits one.

> Are you saying that a cache on such a proxy or gateway SHOULD NOT return
> a fresh response from cache, but instead forward the request, if a GET
> request has a message-body?

It's how I would implement, but the spec doesn't demand it.

We also agree that the proxy may ignore the body according to spec today.
So it's up to the server to set Vary: * or the appropriate Cache Control,
if the response depends on the request body contents.

  an origin server SHOULD either ignore the message-body or respond with
  an appropriate error message (e.g., 413)

is a MUST, IIUC.  In that, it can respond with an appropriate error or
even disconnect if, for example, it considers the request-body to be a
flood of no value.  Correct?

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Jan 2007 15:25:06 +1100

> right that 2616 has several places where the word cache is misused.

I'm aware of the distinction, just lacking distinct terminology to  
describe the cache's behaviour with.

> might be implemented, rather than specifying an HTTP interface

Works for me.

---

From: Yves Lafon <[email-masked]>
Date: Wed, 17 Jan 2007 05:04:50 -0500 (EST)

> disallowed as bad syntax by others.  Ample opportunities for request

Alex Rousskov pointed out some time ago that it was covered by the spec in 
2.1, implied *LWS.
So  it should always be interpreted as "Content-Length"
Cheers,

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 17 Jan 2007 11:47:33 +0100

> report carefully to observe all the ways an implementor can do so.

The reports covers many possible permutations, but is not an
authoritative reference on what is or is not allowed by the specs. Some
of the constructs used is clearly invalid, most simply dubious
exploiting the fact that the specs is not clear.

Header names and implied LWS:
[I30] http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i30

Does the implied LWS rule apply to header names, even if it's not
allowed in MIME? Allowing LWS around the header name does not make
sense, but it is not explicitly forbidden.

I.e. is

Content-Length : 100

or if you like (equivalent)

Content-Length
 :100

valid HTTP headers, even if they are not valid MIME headers?

Double CR and end of headers:
[partly covered by I30]

Also is
<CR><CR><NL>

a blank header terminating the headers, or is it a invalid header line
consisting of a single <CR>?  Well, this one is clear (it's a <CR> line,
not blank), but unfortunately 19.3 encourages ignoring <CR> which easily
leads to implementations getting this wrong.

[no issue assigned]

And how SHOULD a recipient react if there is multiple Content-Length
headers?

These aspects is critical for defining the message delimiting within the
protocol, and still the specs is quite silent on the details which calls
for the kind of interoperability problems shown in the report.

Regards
Henrik

---

From: Travis Snoozy <[email-masked]>
Date: Wed, 17 Jan 2007 08:42:40 -0800

On Wed, Jan 17, 2007 at 11:47:33AM +0100, Henrik Nordstrom wrote:
<snip>
> sense, but it is not explicitly forbidden.

LWS is not allowed.

<snip>
> Content-Length : 100

BNF makes it clear.

token          = 1*<any CHAR except CTLs or separators>
message-header = field-name ":" [ field-value ]
field-name     = token
separator      = [...] | SP | HT

What we have there is malformed; separators (SP) are not allowed in tokens.

<snip>
> Content-Length

What we have here is malformed; CTLs (CR/LF) are not allowed in tokens.

<snip>
> And how SHOULD a recipient react if there is multiple Content-Length

The same way it MUST react whenever it gets any header that isn't allowed
multiples -- reject the message as blatantly malformed:

   Multiple message-header fields with the same field-name MAY be
   present in a message if and only if the entire field-value for that
   header field is defined as a comma-separated list [i.e., #(values)].
   It MUST be possible to combine the multiple header fields into one
   "field-name: field-value" pair, without changing the semantics of the
   message, by appending each subsequent field-value to the first, each
   separated by a comma.

(Section 4.2, page 32)

> for the kind of interoperability problems shown in the report.

If I may insert my two cents here: the spec is so fuzzy, poor, and inconsistent
in OTHER places that it affects the overall quality and ability to implement.
In my experience, it got _very_ tempting to just start ignoring the spec and do
a pigeon HTTP/1.1 implementation. Here, I think things are actually pretty
clear and obvious, except for the CR-CR-LF possibility (parsing LWS has been a
very obnoxious thorn in my side) -- but the implementer has to still be paying
attention to catch all the details.

-- 
Travis

---

From: Julian Reschke <[email-masked]>
Date: Wed, 17 Jan 2007 17:50:39 +0100

Travis Snoozy schrieb:
> ...

Are you aware of the "implied LWS" rule? 
(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.2.1.p.11>).

Best regards, Julian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Wed, 17 Jan 2007 10:55:48 -0600

> (<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.2.1.p.11>).

Precisely.  It must be accepted, because spaces are allowed between words
and between word delimiters.  But it must not be transmitted because the
spec is quite clear in it's construction.  The reporter of issue 30 seems
to think it's ambiguous, it's not.  Unless EXPLICITLY prohibited LWS is
accepted.

In the case of LWS before a header argument, that's absolutely not valid
data.  Again, the reporter of issue 30 seems to be confused about the
distinction between an implementation and a spec.

Bill

---

From: Travis Snoozy <[email-masked]>
Date: Wed, 17 Jan 2007 09:02:28 -0800

> > sense, but it is not explicitly forbidden.

... if you summarily dismiss the "implied LWS rule." Yes, LWS is allowed between
tokens and separators. My mistake; I'm used to BNF explicitly specifying exactly
what can and cannot happen, and not making implicit back-references.

LWS SHOULD NOT be allowed.

-- 
Travis

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 17 Jan 2007 19:57:47 +0100

> What we have there is malformed; separators (SP) are not allowed in tokens.

well... no. The 2616 BNF grammar is also augmented with the "implied
*LWS" rule in section 2.1.. which allows insertion of LWS between any
token and any separator unless forbidden.

> multiples -- reject the message as blatantly malformed:

Specs does not say recipients MUST reject any malformed message. It
doesn't even require recipients to detect malformed messages and the
robustness principle underlining most use of IETF protocols generally
discourages it. And everything else in software development speaks
against such rejections unless they happen "as a by product" of the
development.

And the result is as expected. Implementations swallow even the most
blatantly malformed messages, with widely varying results.

> in OTHER places that it affects the overall quality and ability to implement.

Very few of the other issues has security implications for other users
visiting other URLs..

> a pigeon HTTP/1.1 implementation.

It's what most people do, and then they maybe read the specs and try to
comply, and as result there is chaos.

Regards
Henrik

---

From: Travis Snoozy <[email-masked]>
Date: Wed, 17 Jan 2007 11:22:29 -0800

> against such rejections unless they happen "as a by product" of the

No, the principles of secure development say "fail closed." If I don't
understand it, it gets chucked in the bin. Now, the "I don't understand it" bit
kills extensibility, but that's part and parcel to security -- you punch holes
as you need them. The spec needs to be extensible, and there's a balance to be
struck between what's secure and what's utilitarian. However, being _blatently_
malformed (e.g., two of any field that's not a #list) is always grounds for
immediate rejection. Fuzzy repair work, in this case, is a Very Bad Thing.

-- 
Travis

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 17 Jan 2007 20:45:58 +0100

> to think it's ambiguous, it's not.  Unless EXPLICITLY prohibited LWS is

In this specific case I am leaning more towards that there was an
oversight in the wording of 4.2, forgetting to mention that LWS is not
allowed between the header name and the delimiter. The reasoning being

a) It's not allowed by MIME, which the HTTP header syntax is supposed to
build upon.

b) Clearly not expected to most readers.

c) And nearly all implementations gets this wrong one way or another.

As it's very doubtful there is any implementations relying on this being
allowed today, and the security implications of implementations not
properly ignoring the LWS the question is if 4.2 should be strengthened
disallowing LWS between the header name and the separator (also in line
with MIME header syntax), or if the current messy situation should be
kept relying on each implementer to figure out the implications of
implied *LWS..

> distinction between an implementation and a spec.

You mean LWS before a header name? No one has ever claimed that to be
valid, not even the reporter of I30. There is just an observation that
many implementations gets it wrong.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 17 Jan 2007 20:51:57 +0100

> immediate rejection. Fuzzy repair work, in this case, is a Very Bad Thing.

The issue being that most implementations won't check this unless it's
required. Simply chewing what they get. And depending on the
implementation this results in the implementation reading either the
first occurrence or the last of the headers.

Also with the extensibility of HTTP is hard to put as a general
requirement that recipients should check how many times a non-list
header is seen.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Wed, 17 Jan 2007 22:58:28 +0100

> against such rejections unless they happen "as a by product" of the

So in this case the robustness principle is causing some of the 
interoperability and security problems?

Sorry, couldn't resist.

Julian

(draconian error handling rulez)

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Wed, 17 Jan 2007 17:41:46 -0600

> interoperability and security problems?

Actually not-so-much.  If the middle tier properly rephrases the fields
and respects all of the guidance for building the outbound request, and either
chooses to be very liberal-yet-correct or extremely (and even overly) strict,
most of the splitting/spoofing issues would not have occurred in this specific
example.

The flaws came in where authors made assumptions (leading/trailing white
space around the header token treated as the header identifier, or
ignoring the rule to ignore C-L in the presence of T-E chunked, etc),
trusting user input without validation.  That's the root of nearly
every vulnerability in the first place.

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 18 Jan 2007 11:11:29 +0100

> interoperability and security problems?

Yes, unless one is explicitly aware of the security issues.

Regards
Henrik

---

From: [email-masked]
Date: Thu, 18 Jan 2007 11:58:27 +0000

> a) It's not allowed by MIME, which the HTTP header syntax is supposed to
It is not only supposed to build upon, but the spec explicitly
refers to it (4.2: "HTTP header fields [...] follow the same generic
format as that given in Section 3.1 of RFC 822 [9]").
If HTTP was intended to differ from RFC 822 with respect to
this, I would expect the sentence to read "HTTP header fields [...] follow
a similar format as that given [...]"

> b) Clearly not expected to most readers.
Not expected at all. Making the already awkward LWS-situation
of RFC 822 even worse by allowing it nearly everywhere deliberately,
while trying to straighten out the CRLF-lunacy of RFC 822 partially
(for the payload) really is an unforeseeable (unbelievable) pitfall.
I guess nobody would expect to encounter something like
"Host        :       foo.com      :            80"
even though the spec "allows for it", strictly speaking,
according to the implied *LWS rule.

> kept relying on each implementer to figure out the implications of
The phrase "current messy situation" gets to the point.
I would consider the "augmentation" of "implied *LWS" rather
a degradation for the following reasons:

a) any "implied" rule in a spec makes it harder to read
and more error-prone (Travis: "I'm used to BNF explicitly specifying exactly
what can and cannot happen, and not making implicit back-references")
b) implied LWS is superfluous -- nobody needs it and nobody wants
it because
c) any superfluous LWS squanders bandwidth
d) having to expect LWS nearly anywhere is a nuisance
for parsers (at least for those that go for speed)

I would not try to strengthen some minituae while pertaining
to the sentences that really impose problems reading, interpreting
and implementing the whole spec.

Even if LWS is considered a "feature" of the spec (which it is not imho)
the spec should explicitly state where it is allowed, like RFC 822 does.

Kind regards
Ingo Struck

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 18 Jan 2007 21:33:24 +0000

> So  it should always be interpreted as "Content-Length"

In fact, Alex and I read the identical text and disagreed over whether
it allows *LWS before the colon.

(Which, by the way, means that text should be clarified in any new revision).

But that's besides the point; what the spec covers is theoretical.  In
theory, there are no request smuggling attacks.

As I recall, from looking at source code, actually deployed
implementations interpret "Content-Length : 12345" in all the ways I
described.  See also " Content-Length: 12345" (space before the name)
for additional surprises.

-- Jamie

---

From: Larry Masinter <[email-masked]>
Date: Fri, 19 Jan 2007 11:05:46 -0800

In my opinion:

* The current spec is ambiguous, and needs clarification
* Although, in general, it is not appropriate to 'tighten'
  a specs requirements, in some cases (and in this case)
  it is the right choice:

  There are few clients that send LWS between header
  name and :.

  There are already many servers that reject such
  requests.

So, in my opinion, we should change the spec that
clients MUST NOT send LWS in headers before the :,
and that servers MUST reject any such message as
malformed.

This may make a few servers that were previously
compliant non-compliant, but in general it will
improve interoperability and security.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 19 Jan 2007 12:34:51 -0800

> * The current spec is ambiguous, and needs clarification

The current spec needs a complete rewrite.  Besides that, the entire
ABNF has to be rewritten in order for it to meet current IETF reqs.
I suspect that will cause a lot of implied LWS to disappear.

>   There are already many servers that reject such

I think the spec should reflect the standard, not be artificially
restricted by adherence to past revisions of itself.  By standard,
I mean the measure expected by all of the implementations that are
exchanging legitimate communication via HTTP.  AFAIK, there are no
servers or clients that send legitimate messages with anything
other than

    Field-name: field-value

so it is time for the spec to reflect that fact.  My only caveat is
that there should be an exception for the message/http media type,
such that messages received via SMTP shall allow line folding.

> improve interoperability and security.

No, I will not accept that.  MUST NOT send such LWS is fine,
including when a message is forwarded, but forbidding a server from
processing such a message is not going to happen because it would
make all implementations non-compliant.

Servers should be configurable in regards to robust or restricted
parsing behavior, and nothing we say can improve the "security"
of broken software that was deployed years ago.  Software that
correctly parses according to the RFC is not subject to those
perceived security issues.

....Roy

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 19 Jan 2007 14:52:05 -0600

Roy T. Fielding wrote:

[snip... +1 to your initial comments]

> perceived security issues.

As long as a middle tier agent doesn't forward such broken headers
as-given, this isn't an issue?  The middle-tier server may accept or
reject "Foo :bar" as forgiving as it chooses to be, as long as this
middle-tier user agent always forwards the request as "Foo:(LWS*)bar"
instead.  'MUST NOT send' as you describe would accomplish this.

In the security section, an additional observation that header field
names SHOULD be validated would further encourage implementors to
reject the case where their "Foo :bar"'s header field name is treated
as "Foo ".

These two notes would codify the core of the observation from the
Watchfire report.