Subject: Re: #409: is parsing OBS-FOLD mandatory?

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Jan 2013 21:54:22 +1100

On 19/12/2012, at 6:18 PM, Willy Tarreau <[email-masked]> wrote:

> On Wed, Dec 19, 2012 at 11:55:26AM +1100, Mark Nottingham wrote:
>> 
>> On 13/12/2012, at 8:18 AM, Willy Tarreau <[email-masked]> wrote:
>> 
>>> On Wed, Dec 12, 2012 at 10:18:28AM -0800, Roy T. Fielding wrote:
>>> (...)
>>>>> """
>>>>> If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role, and those rules whose names begin with "obs-" (e.g., obs-fold).
>>>>> """
>>>> 
>>>> Do we really want to exclude non-ASCII octets (obs-text) and older
>>>> date formats (obs-date)?  Do we demote them to SHOULD or MAY?
>>> 
>>> This is a good point. Line-folding causes security issues and does not
>>> seem to be used by senders, but I think we all regularly catch some
>>> obs-text and obs-date come from old applications or crippled devices.
>>> 
>>>> This change is fine with me, but it is a hard break from retaining
>>>> compatibility and we need to be absolutely sure we want to do that.
>>> 
>>> I'd rather not break these ones, personally.
>>> 
>>> Couldn't we settle on just stating that obs-fold is normally not used,
>>> is known to cause security issues when improperly implemented, and
>>> should either be completely supported, or rejected, but in all cases
>>> must be detected ?
>> 
>> 
>> I'm OK with it either way, as long as we're clear about what we mean, as well
>> as what's interoperable.
>> 
>> This statement is still ambiguous;
>> 
>> """
>> As a convention, ABNF rule names prefixed with "obs-" denote "obsolete"
>> grammar rules that appear for historical reasons.
>> """
>> 
>> Perhaps we could start by 
>> 
>> 1) clarifying what "obsolete" means for senders and recipients in the error
>> handling section (I think we've already done most of this, see other part of
>> thread)
>> 2) referring to that clarification in the ABNF statement quoted above (easy fix)
>> 3) re-evaluating the use of the obs- prefix in each case (???)
> 
> That sounds like a reasonable plan.
> 
>> Personally, I think obs-date and obs-text are justified in having the prefix
>> and resulting SHOULD, because they're not really interoperable.
> 
> OK. So "SHOULD" could be the default rule for obs-* and exceptions could
> be handled specifically (eg: obs-fold).

Right now, we have:
  obs-fold - senders MUST NOT generate, recipients MUST accept
  obs-text - new headers SHOULD NOT use, recipients SHOULD treat as opaque data 
  obs-date - sender MUST NOT generate (implied by MUST), recipient parsers MUST accept

obs-fold was a "recipients SHOULD accept" before Roy's change. 

I suppose we could come up with a more rigid definition of what "obs-" means here, but that seems like it's sort of diminishing returns.

The issue here, I think, is whether or not we're requiring recipients to deal with line folding.

Some quick and dirty testing shows it's not interoperable. By sending requests in this form:

GET / HTTP/1.1
Host:
    [hostname]
User-Agent: Foo/1.0

which our ABNF currently allows, I got errors on several sites. Given that, I wonder why requiring it to be accepted is necessary; making those sites non-conformant (as well as any clients that behave in a similar manner) isn't going to improve interop, and this feature isn't used in practice any way, precisely because it doesn't get interop. 

SHOULD makes sense; I could even see downgrading it to prose. I don't understand why this is a MUST, if we don't get good interop on it now, and it's not a feature that's in-demand, widely used, or important, AFAIK.

The same logic could be applied to obs-date, but I don't have hard data on how interoperable it is; AFAIK most implementations do a good job of covering the three possible formats.

The text around obs-text seems right, because it's such a different case.

How do others feel? Any additional data? I'm not adamant about any of this, I just want to a) make sure we all understand where we sit WRT obs-fold interop, and b) get this closed.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 13:37:26 +1100

Anyone? Bueller?

To make it clear -- I propose we downgrade recipient support for OBS-FOLD to a prose recommendation. 

Am happy to hear well-reasoned opinions to the contrary (but see note about interop testing below).

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 20 Jan 2013 08:44:34 +0100

Hi Mark,

Sorry for being silent so long, I was quite busy last two weeks :-/

> To make it clear -- I propose we downgrade recipient support for OBS-FOLD to

Given the security impact it can have if processed the lazy way, we must at
least have something normative. Practically, I don't know how this can be
put into valid terms in a spec, but I'd like to see something which mandates
detecting OBS-FOLD and then leaves the choice to the implementation to accept
it or to reject the message. What I don't want to see is implementations which
consider that the space is part of the next header field name, and which later
trim the spaces and take this header field for another one. And we know for
sure that lazy implementations are lazy in multiple areas (typically all
those quickly written in scripting languages).

(...)
> > which our ABNF currently allows, I got errors on several sites. Given that,
> > I wonder why requiring it to be accepted is necessary; making those sites
> > non-conformant (as well as any clients that behave in a similar manner)
> > isn't going to improve interop, and this feature isn't used in practice any
> > way, precisely because it doesn't get interop. 

I'm not so much worried by these ones which possibly have not found their Host
header field as I am by the ones which accept the following message without
yelling :

   POST / HTTP/1.1
   Host: [hostname]
   User-Agent: foo/1.0
     Transfer-encoding: chunked
   Content-Length: 10

Everything here is valid, the transfer-encoding is the continuation of the
User-Agent header which was folded. But if middle boxes don't support folding,
they'll have a "  Transfer-encoding" header field with value "chunked", which
if interpreted as "Transfer-encoding" will have precedence over Content-Length.

So I think that we have valid reasons to be concerned about just leaving a
SHOULD there.

Maybe we can have a "SHOULD" with an "IF" and a "MUST" ? Eg: "IF an
implementation decides not to implement obs-fold, then it MUST take
extreme care to reject a message which contains a line beginning with
SP/TAB" ? I know that such characters are already not allowed, but
things are spread over so many sheets that it's hard to get all the
things right at once. And here we want specific care about a real risk.

> > SHOULD makes sense; I could even see downgrading it to prose. I don't
> > understand why this is a MUST, if we don't get good interop on it now, and
> > it's not a feature that's in-demand, widely used, or important, AFAIK.

It's so much complicated to get right in line-based parsers (eg: shell
scripts) that the MUST seems excessive to me if there are no provisions
for not doing it at all.

> > The same logic could be applied to obs-date, but I don't have hard data on
> > how interoperable it is; AFAIK most implementations do a good job of
> > covering the three possible formats.

I'm fairly certain I've already seen this old format as a past expiration
date for a cookie in an old application. Rejecting such formats could have
the effect of not removing cookies upon logout, which may be of little
inconvenience to some applications.

> > The text around obs-text seems right, because it's such a different case.

Agreed, and we need to keep it.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 20:26:42 +1100

> those quickly written in scripting languages).

Understood.

I think it's a matter of either parsing obs-fold, or rejecting a message that has whitespace after a newline...

Current:

"""
Historically, HTTP header field values could be extended over multiple lines by preceding each extra line with at least one space or horizontal tab (obs-fold). This specification deprecates such line folding except within the message/http media type (Section 7.3.1). Senders MUST NOT generate messages that include line folding (i.e., that contain any field-value that matches the obs-fold rule) unless the message is intended for packaging within the message/http media type. Recipients MUST accept line folding and replace any embedded obs-fold whitespace with either a single SP or a matching number of SP octets (to avoid buffer copying) prior to interpreting the field value or forwarding the message downstream.
"""

Proposal:

"""
Historically, HTTP header field values could be extended over multiple lines by preceding each extra line with at least one space or horizontal tab (obs-fold). This specification deprecates such line folding except within the message/http media type (Section 7.3.1). Senders MUST NOT generate messages that include line folding (i.e., that contain any field-value that matches the obs-fold rule) unless the message is intended for packaging within the message/http media type. Recipients MUST either:

  - accept line folding and replace any embedded obs-fold whitespace with either a single SP or a matching number of SP octets (to avoid buffer copying) prior to interpreting the field value or forwarding the message downstream, or

  - reject a message with line folding present. Servers can do for requests by responding with 400 Bad Request and a representation explaining the condition; clients can only discard the message.

In particular, recipients who choose not to implement obs-fold processing (as described above) MUST NOT accept messages containing headers with leading whitespace, as this can expose them to attacks that exploit this difference in processing.
"""

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 20 Jan 2013 10:44:48 +0100

> In particular, recipients who choose not to implement obs-fold processing (as described above) MUST NOT accept messages containing headers with leading whitespace, as this can expose them to attack…

Seems perfect to me :-)

Cheers,
Willy