Subject: OWS, line-folding and quoted-string

From: Yutaka OIWA <[email-masked]>
Date: Tue, 06 Sep 2011 11:45:53 +0900

Dear all,

I have a question with the current definition of quoted-string in p1-16.

In section 1.2.2, OWS is defined as *( [ obs-fold ] WSP ), allowing
null-string match.  The text in the same section specifies that
non-null OWS can be transformed to single SP within field-content.

At the same time, in section 3.2.3, quoted string is defined as
DQUOTE *( qdtext / quoted-pair ) DQUOTE, where qdtext contains single OWS.
(Note that quoted-string is to be used inside field-content.)

This brings two unwanted consequences:

1) Using null-allowing OWS inside infinitely-repeating qdtext
   makes any quoted-strings to be parsed as infinitely many
   possibilities unneededly.

2) As a single OWS can eat two+ spaces at once,
   and as a side-effect of the OWS canonicalization in Sec. 1.2.2,
   continuous spaces in the quoted-string may be reduced to
   any (non-zero) number of spaces not more than the original.
   For example, five spaces may be parsed as
   ("  ": OWS) (" ": OWS) ("  ": OWS), which may be reduced to
   three spaces after "Sec 1.2.2 SHOULD rule" applied.
   This will bring unwanted ambiguity and bad interoperability
   especially with hash-or-crypto-based authentications.
   It also contradicts with what people thinks with "quoting".

So, I propose change to the definition of qdtext in section 3.2.3 as follows:
from
   qdtext = OWS / %x21 / %x23-5B / %x5D-7E / obs-text
to
   qdtext = [ obs-fold ] WSP / %x21 / %x23-5B / %x5D-7E / obs-text
(or add parentheses if clarification needed).

As Section 3.2.1 separately specifies reduction of obs-fold to either
one or two SPs, it still allows removal of line folding within quoted-string.
I think the interoperability problem with line-folded quoted-string
seems to be negligible (because line-folded quoted-string is a very bad thing
anyway).

P.S.1:
The literal reading of "3.2.1 obs-fold rule" says that a line-folding
"CR LF SP" should be reduced to either "SP SP" or "SP SP SP"
(because CRLF is reduced to single SP).  Is this correct and intended?
I guess the intention of the first alternative is "SP" instead of "SP SP".
As obs-fold is guaranteed to be followed by WSP, it can be simply removed
instead of replacing to single SP.
(This is a minor issue, however, because in many places it will be further
reduced by OWS/RWS reduction rule in Sec 1.2.2.)

# I personally prefer obs-fold to be defined as "CRLF 1*WSP" because
# it clearly says that continued-line must be started with spaces,
# but changing at this moment seems to be inadequate,
# since adoption requires rewriting of many rules using obs-fold.

P.S.2:
The proposed change above affects handling of HT in a quoted-string.
Is that better to be reduced to SP or to be kept as HT?

-- 
Yutaka OIWA, Ph.D.                                       Research Scientist
                            Research Center for Information Security (RCIS)
    National Institute of Advanced Industrial Science and Technology (AIST)
                      Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[995DD3E1] fp[3C21 17D0 D953 77D3 02D7 4FEC 4754 40C1 995D D3E1]

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 6 Sep 2011 07:49:21 +0200

> I guess the intention of the first alternative is "SP" instead of "SP SP".

I think the first case is to cover "LF SP" since implementations should be
tolerant to LF-only line delimiting. Then, the LF is replaced with an SP,
which leads to "SP SP".

> instead of replacing to single SP.

The goal was to be able to replace CRs and LFs with SPs without moving data
nor changing a packet length, so that unfolding is very cheap and leaves no
excuse to implementations for not doing it.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 06 Sep 2011 09:28:53 +0200

On 2011-09-06 04:45, Yutaka OIWA wrote:
> ...
> ...

The definition changed a few days ago, see
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#basic.rules>
(tools.ietf.org seems to be down right now).

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 06 Sep 2011 16:48:21 +0900

> <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#basic.rules>

Oh, that's good.

> (tools.ietf.org seems to be down right now).

Yes, actually I failed to check the latest version before posting,
just because of that.

OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]