Subject: #409: is parsing OBS-FOLD mandatory?

From: Mark Nottingham <[email-masked]>
Date: Wed, 12 Dec 2012 14:48:55 +1100

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/409>

"""
p1 2.5 Conformance and Error Handling says "...recipient MUST be able to parse any value that would match the ABNF rules..." yet 3.2.2 only make parsing obs-fold a SHOULD. Which is it?
"""

Roy made a proposed edit to remove the MUST NOT generate and change the SHOULD parse to a MUST parse.
  <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2039>

However, this has the effect of un-deprecating line folding; IIRC we added those requirements because folding is not interoperable. 

My suggestion would be to change:

"""
If a received protocol element is processed, the recipient must be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.
"""

to:

"""
If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role, and those rules whose names begin with "obs-" (e.g., obs-fold).
"""

Note that we already say, in 1.2 Syntax Notation:

"""
As a convention, ABNF rule names prefixed with "obs-" denote "obsolete" grammar rules that appear for historical reasons.
"""

Comments?

--
Mark Nottingham   http://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 12 Dec 2012 19:51:18 +1300

> As a convention, ABNF rule names prefixed with "obs-" denote "obsolete" grammar rules that appear for historical reasons.

+1. Sounds good.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 12 Dec 2012 07:57:33 +0100

Hi Mark,

On Wed, Dec 12, 2012 at 02:48:55PM +1100, Mark Nottingham wrote:
> However, this has the effect of un-deprecating line folding; IIRC we added
> those requirements because folding is not interoperable. 
> 
> If a received protocol element is processed, the recipient must be able to
> parse any value that would match the ABNF rules for that protocol element,
> excluding only those rules not applicable to the recipient's role.
> """
> 
> to:
> 
> """
> If a received protocol element is processed, the recipient MUST be able to
> parse any value that would match the ABNF rules for that protocol element,
> excluding only those rules not applicable to the recipient's role, and those
> rules whose names begin with "obs-" (e.g., obs-fold).
> """

I think we may improve the situation a little bit by indicating that
recipients must *detect* these obs-* rules even if they decide to reject
them because they can't parse them. The main issue I'm seeing with relaxing
parsing too much is the risk of seeing some smuggling attacks due to lazy
implementations that claim to be mostly compliant since they're not forced
to support line folding. We've already seen implementations which accept
spaces in header field names for example. I'm a bit worried about what the
following request would cause :

     POST /foo HTTP/1.1
     Host: foo
     Content-length: 8
     X-Dummy: yes
        Transfer-encoding: chunked

     ffffffff
     POST /unfiltered HTTP/1.1
     Host: foo
     Content-length: 100

     dangerous args

If a lazy implementation decides that the "   Transfer-Encoding" field
above is just "Transfer-Encoding", it might very well skip over the whole
4GB chunk, while for another implementation it would be the continuation
of X-Dummy.

So what do you think about the following change on top of yours :

 """
 If a received protocol element is processed, the recipient MUST be able to
 parse any value that would match the ABNF rules for that protocol element,
 excluding only those rules not applicable to the recipient's role, and those
 rules whose names begin with "obs-" (e.g., obs-fold). However, the recipient
 MUST be able to detect the rules it cannot parse and MUST reject such
 messages.
 """

Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 12 Dec 2012 10:18:28 -0800

>  <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2039>

No, the text still says "Senders MUST NOT generate ...".  I only changed
the SHOULD parse to a MUST parse, because that was not part of the prior
decision and is inconsistent with the ABNF requirement below.

> However, this has the effect of un-deprecating line folding; IIRC we added those requirements because folding is not interoperable. 

No, it has the effect of requiring that line-folding be parsed, which
is required for backwards compatibility with 2616 senders.  However,
I do not personally know of any senders that send obs-fold.

> If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to th…

Do we really want to exclude non-ASCII octets (obs-text) and older
date formats (obs-date)?  Do we demote them to SHOULD or MAY?

This change is fine with me, but it is a hard break from retaining
compatibility and we need to be absolutely sure we want to do that.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 12 Dec 2012 22:18:38 +0100

On Wed, Dec 12, 2012 at 10:18:28AM -0800, Roy T. Fielding wrote:
(...)
> date formats (obs-date)?  Do we demote them to SHOULD or MAY?

This is a good point. Line-folding causes security issues and does not
seem to be used by senders, but I think we all regularly catch some
obs-text and obs-date come from old applications or crippled devices.

> compatibility and we need to be absolutely sure we want to do that.

I'd rather not break these ones, personally.

Couldn't we settle on just stating that obs-fold is normally not used,
is known to cause security issues when improperly implemented, and
should either be completely supported, or rejected, but in all cases
must be detected ?

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 13 Dec 2012 10:34:45 +1300

On 13.12.2012 07:18, Roy T. Fielding wrote:
>> p1 2.5 Conformance and Error Handling says "...recipient MUST be 
>> able to parse any value that would match the ABNF rules..." yet 3.2.2 
>> only make parsing obs-fold a SHOULD. Which is it?
>> """
>>
>> However, this has the effect of un-deprecating line folding; IIRC we 
>> added those requirements because folding is not interoperable.
>

Squid has a few bug reports with traces involving obs-fold in headers. 
Usually associated with complaints or problems handling >64KB long 
headers, and the fold is almost always occuring in custom X-* headers 
from some user generated data that should have been in the entity 
anyway.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2012 11:50:21 +1100

> MUST be able to detect the rules it cannot parse and MUST reject such

I'm OK with that, except that your MUST can be read to apply to those that aren't applicable to the recipient's role (unintentionally, I think).

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2012 11:55:26 +1100

> should either be completely supported, or rejected, but in all cases

I'm OK with it either way, as long as we're clear about what we mean, as well as what's interoperable.

This statement is still ambiguous;

"""

Perhaps we could start by 

1) clarifying what "obsolete" means for senders and recipients in the error handling section (I think we've already done most of this, see other part of thread)
2) referring to that clarification in the ABNF statement quoted above (easy fix)
3) re-evaluating the use of the obs- prefix in each case (???)

Personally, I think obs-date and obs-text are justified in having the prefix and resulting SHOULD, because they're not really interoperable.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Dec 2012 08:12:16 +0100

Hi Mark,

On Wed, Dec 19, 2012 at 11:50:21AM +1100, Mark Nottingham wrote:
> I'm OK with that, except that your MUST can be read to apply to those that
> aren't applicable to the recipient's role (unintentionally, I think).

I don't completely understand what you meant, I'm sorry :-)

Do you mean that it can be read that it is mandatory for the recipient to
understand the whole language even if it's not interested in it ?

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2012 18:16:21 +1100

> understand the whole language even if it's not interested in it ?

Pretty much. It may not be a practical concern, but it doesn't read well. I think it can be fixed editorially.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Dec 2012 08:18:58 +0100

On Wed, Dec 19, 2012 at 11:55:26AM +1100, Mark Nottingham wrote:
> As a convention, ABNF rule names prefixed with "obs-" denote "obsolete"
> grammar rules that appear for historical reasons.
> """
> 
> 1) clarifying what "obsolete" means for senders and recipients in the error
> handling section (I think we've already done most of this, see other part of
> thread)

That sounds like a reasonable plan.

> Personally, I think obs-date and obs-text are justified in having the prefix
> and resulting SHOULD, because they're not really interoperable.

OK. So "SHOULD" could be the default rule for obs-* and exceptions could
be handled specifically (eg: obs-fold).

Willy