Subject: Re: Issue 194: restricting allowed characters in quoted-pair

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Sep 2009 15:43:53 +1000

Yeah, what Julian said; I'd like to do this, but am concerned about  
what it would mean.

Can we keep it (relatively) open in BNF, and caution against quoting  
anything but DQUOTE in prose?

Not a great solution, but...

On 28/08/2009, at 7:35 PM, Julian Reschke wrote:

> Roy T. Fielding wrote:
>> On Aug 27, 2009, at 5:03 AM, Julian Reschke wrote:
>>> It appears that we *do* have consensus for disallowing controls in  
>>> quoted-pairs, thus for:
>>>
>>>  quoted-pair    = "\" ( WSP / VCHAR / obs-text )
>>>
>>> However, if that's all that we do we won't have addresses issue  
>>> #173 after all.
>>>
>>> Proposal:
>>>
>>> - add a new issue for disallowing CTLs in quoted-pair
>> I suggest we make the issue "Disallow quoted-pair productions that  
>> are
>> never used in practice nor needed for parsing", with the fix being
>>   quoted-pair    = "\" ( "\" / DQUOTE / "(" / ")" )
> > ...
>
> - I have opened issue #194 for this.
>
> - Restricting it to only these four characters is attractive, but  
> may break content where people quoted too defensively.
>
> BR, Julian
>

--
Mark Nottingham     http://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 15 Sep 2009 23:23:15 -0700

> Yeah, what Julian said; I'd like to do this, but am concerned about  

It would mean that sending other quoted characters would no
longer comply with the protocol.  Since I don't know of any
implementations that quote other characters, that's fine with me.

It also means that recipients of other quoted characters should
consider the backslash to be used in error, which is probably a
good thing since the only reason to backslash other characters
on purpose is to trigger a security hole.

> Can we keep it (relatively) open in BNF, and caution against  
> quoting anything but DQUOTE in prose?
>

We could, but how does that help implementations?  I'd rather
be more conservative in the ABNF and let Postel's Law influence
the prose.  I suspect that other uses of backslash are not
intended to be quoting in any case, so the recipient is probably
better off treating them as two characters.  e.g., DOS pathnames.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 16 Sep 2009 08:28:32 +0200

> anything but DQUOTE in prose?

Yes.

Parsing should in general accept pretty much any valid char quoted, but
when producing the \ quoting mechanism should only be used for the
special characters which need quoting in the context.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Sep 2009 16:34:52 +1000

So, strict BNF with a note in prose that old implementations may quote  
other characters? Works for me.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Sep 2009 15:08:46 +0200

> other characters? Works for me.

Not convinced yet.

1) This seems to be inconsistent with how we treated other cases, such 
as when we introduced BWS ("bad whitespace") as grammar production.

2) We currently use quoted-pair both for quoted strings and comments. So 
this would need to change, and both would have to exclude different 
characters.

3) In my experience as programmer, un-escaping usually treats all 
escaped characters the same way.

4) Also, there's a chance that producers currently use a single 
escaping/quoting implementation both for comments and quoted-string, 
which would then escape '(', ')', '\' and '"'.

Thus, my proposal is to just disallow escapes, and warn about anything 
else in prose.

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 18 Sep 2009 00:44:47 +0200

> other characters? Works for me.

Usually this kind of things is done the other way around. Open BNF with
restrictions in prose on what may be produced. Or if one wants to go the
long way by introducing "obsolete/old" BNF constructs for stuff which is
should not be produced but still accepted by parsers for legacy reasons.

Notes in prose should not require parsers to go outside the BNF.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Sep 2009 14:40:06 +0200

> Notes in prose should not require parsers to go outside the BNF.

OK, for now I have only made the change disallowing non-HTAB control 
characters in quoted-pair 
(<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/696>).

I have not yet added any prose about recommending not to use quotes when 
not needed. With respect to that, we need to decide:

- whether that's purely advisory or a requirement (SHOULD?), I lean to 
the former, and

- where exactly to state it, as quoted-pair is used both inside 
quoted-string and comment, and the characters that need escaping thus 
differ; one way to fix this would be to change the ABNF so "comment" 
gets it's own quoted-cpair rule.

BR, Julian

---

From: Sam Johnston <[email-masked]>
Date: Fri, 25 Sep 2009 13:58:54 +0100

On Fri, Sep 25, 2009 at 1:40 PM, Julian Reschke <[email-masked]>wrote:

> characters in quoted-pair (<
> http://trac.tools.ietf.org/wg/httpbis/trac/changeset/696>).
>
> - whether that's purely advisory or a requirement (SHOULD?), I lean to the
> former, and
>
> differ; one way to fix this would be to change the ABNF so "comment" gets
> it's own quoted-cpair rule.
>

I would say implementations SHOULD NOT use quotes where not required but
MUST always accept them.

Sam

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Sep 2009 15:05:43 +0200

> I would say implementations SHOULD NOT use quotes where not required but 

So what's the interop problem ([1]) that we would solve by saying 
"SHOULD NOT"?

BR, Julian

[1] <http://tools.ietf.org/html/rfc2119#section-6>

---

From: Sam Johnston <[email-masked]>
Date: Fri, 25 Sep 2009 14:16:58 +0100

On Fri, Sep 25, 2009 at 2:05 PM, Julian Reschke <[email-masked]>wrote:

> So what's the interop problem ([1]) that we would solve by saying "SHOULD
> NOT"?
>

Simple implementations (such as sysadmin scripts) will often handle quotes
poorly or not at all, particularly where they're used to dealing with single
instances of registered relations. That is to say that using quotes
unnecessarily is likely to cause interop breakage with poorer
implementations (unfortunately not everybody reads standards and of those
that do, not all of them respect them).

Sam

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 25 Sep 2009 22:58:18 +0200

> - whether that's purely advisory or a requirement (SHOULD?), I lean to 

SHOULD == RECOMMENDED.

\ quoting more than needed is not an error, but not recommended as some
applications may not understand \ quoting...

> gets it's own quoted-cpair rule.

Just a general statement that the producer SHOULD NOT \ quote other
characters than the reserved ones for the type of data produced (comment
or quoted-string).

There is no need to overdo this, and absolutely no reason to complicate
the BNF for it.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Sep 2009 23:23:31 +0200

Yes.

> applications may not understand \ quoting...

I know I'm nitpicking here, but...:

"3. SHOULD This word, or the adjective "RECOMMENDED", mean that there

    may exist valid reasons in particular circumstances to ignore a
    particular item, but the full implications must be understood and
    carefully weighed before choosing a different course." -- 
<http://tools.ietf.org/html/rfc2119#section-3>

So what would constitute a valid reason for escaping characters that do 
not need escaping? Lazyness IMHO doesn't count as "valid".

> There is no need to overdo this, and absolutely no reason to complicate

I agree with that, and I also agree it's good advice to tell people not 
to quote things that do not need quoting. Just trying to find the right 
prose...

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 26 Sep 2009 00:15:55 +0200

> not need escaping? Lazyness IMHO doesn't count as "valid".

We do not need to come up with valid reasons today. Keeping it at SHOULD
level makes sure that if/when there is a valid reason then it can be
done.

For our writing it's sufficient that there is no valid reason why this
should be a MUST level requirement.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Sun, 27 Sep 2009 13:16:06 +0200

> There is no need to overdo this, and absolutely no reason to complicate

OK, then.

So I have added the SHOULD NOT in both places, and -- to make things 
less confusing with respect to quoted-pair -- added quoted-cpair for use 
in comments (it's the same grammar, but comes with different prose).

See <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/702>

The actual text:

    A string of text is parsed as a single word if it is quoted using
    double-quote marks.

      quoted-string  = DQUOTE *( qdtext / quoted-pair ) DQUOTE
      qdtext         = OWS / %x21 / %x23-5B / %x5D-7E / obs-text
                     ; OWS / <VCHAR except DQUOTE and "\"> / obs-text
      obs-text       = %x80-FF

    The backslash character ("\") can be used as a single-character
    quoting mechanism only within quoted-string constructs:

    Producers SHOULD NOT escape characters that do not require escaping
    (DQUOTE and the backslash character).

and

    Comments can be included in some HTTP header fields by surrounding
    the comment text with parentheses.  Comments are only allowed in
    fields containing "comment" as part of their field value definition.

      comment        = "(" *( ctext / quoted-cpair / comment ) ")"
      ctext          = OWS / %x21-27 / %x2A-5B / %x5D-7E / obs-text
                     ; OWS / <VCHAR except "(", ")", and "\"> / obs-text

    The backslash character ("\") can be used as a single-character
    quoting mechanism only within comment constructs:

      quoted-cpair    = "\" ( WSP / VCHAR / obs-text )

    Producers SHOULD NOT escape characters that do not require escaping
    (the backslash character "\" and the parentheses "(" and ")").

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 27 Sep 2009 21:08:33 +0200

> in comments (it's the same grammar, but comes with different prose).

Looks great to me. Short and consise, with no complicated BNF confusing
people writing parsers.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 27 Sep 2009 15:33:11 -0700

On Sep 27, 2009, at 4:16 AM, Julian Reschke wrote:

>>> - whether that's purely advisory or a requirement (SHOULD?), I  
>>> lean to the former, and
>> SHOULD == RECOMMENDED.
>>> quoted-string and comment, and the characters that need escaping  
>>> thus differ; one way to fix this would be to change the ABNF so  
>>> "comment" gets it's own quoted-cpair rule.
> So I have added the SHOULD NOT in both places, and -- to make  
> things less confusing with respect to quoted-pair -- added quoted- 
> cpair for use in comments (it's the same grammar, but comes with  
> different prose).
>

remove "only"

>    (DQUOTE and the backslash character).

that should be "(i.e., other than DQUOTE and backslash)".

Same fix applies to the comment section.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 Sep 2009 08:24:40 +0200

Roy T. Fielding wrote:
...
> Same fix applies to the comment section.

Thanks! Applied in 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/703>.

BR, Julian