Subject: Proposal for i111 / i63

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 12:29:16 +1100

This is the parts of my revised proposal for i74 that are specific to  
i111 and i63. I've mostly used HTTP-style BNF, not ABNF, for purposes  
of comparison.

* p1, 2.2:
Old:
>  The TEXT rule is only used for descriptive field contents and values
>  that are not intended to be interpreted by the message parser.  Words
>  of *TEXT MAY contain characters from character sets other than ISO-
>  8859-1 [ISO-8859-1] only when encoded according to the rules of
>  [RFC2047].
>    TEXT           = %x20-7E | %x80-FF | LWS
>                   ; any OCTET except CTLs, but including LWS
>  A CRLF is allowed in the definition of TEXT only as part of a header
>  field continuation.  It is expected that the folding LWS will be
>  replaced with a single SP before interpretation of the TEXT value.

New:
"""
Words of *TEXT MUST NOT contain characters from character sets other  
than ISO-8859-1 [ISO-8859-1].

    TEXT           = %x20-7E | %x80-FF | LWS
                   ; any OCTET except CTLs, but including LWS

A CRLF is allowed in the definition of TEXT only as part of a header  
field continuation.  It is expected that the folding LWS will be  
replaced with a single SP before interpretation of the TEXT value.

Characters outside of ISO8859-1 MAY be included where the encoded-word  
rule (as defined in RFC2047, Section 2) is specified. The encoded-word  
rule is only used for descriptive field contents and values that are  
not intended to be interpreted by the message parser. When used in  
HTTP, encoded-word has no specified length limit.
"""

Note that I've taken a minimal approach to #63 here, and that the  
outcome of i74 may change this.

* p1, 2.2:
Old:
> comment = "(" *( ctext | quoted-pair | comment ) ")"

New:
"""
comment = "(" *( ctext | quoted-pair | comment | encoded-word ) ")"
"""

* p1, 4.2:
Old:
>    field-content  = <field content>
>                     ; the OCTETs making up the field-value
>                     ; and consisting of either *TEXT or combinations
>                     ; of token, separators, and quoted-string

New:
"""
field-content = <field content>
  ; the OCTETs making up the field-value,
  ; according to the syntax specified by the field.
"""

N.B. depending on how we resolve i74, we may want to add a constraint  
regarding character encodings, so that people don't start minting  
headers in random ones.

* p3, B.1:
Old:
> filename-parm = "filename" "=" quoted-string

New:
"""
filename-parm = "filename" "=" quoted-string | encoded-word
"""

N.B.

* p6, 16.6:
Old:
> warn-text = quoted-string
New:
"""
warn-text = quoted-string | encoded-word
"""

Note that I have NOT suggested the use of encoded-word in the  
following places:

p1, 3.4 (Transfer Codings -- parameter values), p1, 6.1.1 (Reason- 
Phrase), p2, 10.2 (expect-extensions), p3, 3.3 (Media Types --  
parameter values), p3, 6.1 (accept-extension), p4, 3 (ETag opaque- 
tag), p6, 16.2 (cache-extension), p6, 16.4 (extension-pragma).

I think the *-extension and parameter value ones are straightforward;  
if a particular extension wants to specify use of encoded-word, it  
should; we shouldn't specify use of encoded-word in the generic  
extension construct, but leave it to the specific instances. I.e.,  
they still conform to TEXT, it's up to them to specify if that content  
can contain encoded-words.

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Apr 2008 19:16:19 -0700

There seemed to be implicit agreement with this approach when we were  
discussing these parts of i74, but nothing since they were split away  
from the more difficult parts of that issue.

In a nutshell, this is just replacing the blanket RFC2047 encoding  
declaration with targeted use of the encoded-word rule, while removing  
the length limit from it.

Comments?

On 03/04/2008, at 6:29 PM, Mark Nottingham wrote:

>
> This is the parts of my revised proposal for i74 that are specific  
> to i111 and i63. I've mostly used HTTP-style BNF, not ABNF, for  
> purposes of comparison.
>
> * p1, 2.2:
> Old:
> Characters outside of ISO8859-1 MAY be included where the encoded- 
> word rule (as defined in RFC2047, Section 2) is specified. The  
> encoded-word rule is only used for descriptive field contents and  
> values that are not intended to be interpreted by the message  
> parser. When used in HTTP, encoded-word has no specified length limit.
> """
>
> N.B. depending on how we resolve i74, we may want to add a  
> constraint regarding character encodings, so that people don't start  
> minting headers in random ones.
>
>
> * p3, B.1:
> Old:
> I think the *-extension and parameter value ones are  
> straightforward; if a particular extension wants to specify use of  
> encoded-word, it should; we shouldn't specify use of encoded-word in  
> the generic extension construct, but leave it to the specific  
> instances. I.e., they still conform to TEXT, it's up to them to  
> specify if that content can contain encoded-words.

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 16 Apr 2008 20:20:30 -0700

> > parser. When used in HTTP, encoded-word has no specified 

RFC2047 without the length limit is not RFC2047 any more. Besides, the
length limit is not the only RFC2047-imposed limit you would have to
remove, as explained below.

> > comment = "(" *( ctext | quoted-pair | comment | encoded-word ) ")"

ctext matches everything that encoded-word matches, so the given grammar
is ambiguous. Further, comments are not interpreted by applications, so
I see little point in defining encoding/decoding for them. In fact, the
specification should be explicitly recommending that we avoid producing
comments as they are not properly supported by many deployed
implementations. I recommend keeping the old rule as it was.

> > ; according to the syntax specified by the field.

A parser should be able to parse field-content without knowing the
syntax of any field. In particular, the field-content rule is needed to
parse unknown headers. It should be specified by real
machine-processable ABNF.

> > filename-parm = "filename" "=" quoted-string | encoded-word

RFC2047 says "An 'encoded-word' MUST NOT be used in parameter of a MIME
Content-Type or Content-Disposition field, or in any structured field
body except within a 'comment' or 'phrase'." That is what RFC2231 is
for. Also, this is not a backward-compatible change. Different products
use differing syntaxes for the filename parameter, and nobody is using
RFC2047. See
http://lists.w3.org/Archives/Public/public-html/2008Mar/0113.html. I
suggest to make a separate issue for this.

> > warn-text = quoted-string | encoded-word

Again, RFC2047 says it cannot be used in a structured field body, and
this is not a backward-compatible change anyway. Nobody has mentioned
any existing implementations that do this. 

> > specify if that content can contain encoded-words.

Again, that would not be a backward-compatible change because the
RFC2616 grammar explicitly says "token | quoted-string" for values, and
that is precisely what existing implementations expect.

Like I said before, although RFC 2616 vaguely stated that RFC 2047 could
be used, the actual BNF productions in the grammar did not allow it
anywhere. Attempting to RFC 2047 encoding anywhere now would be an
incompatible change that deployed implementations are not likely to
handle well. In order to maximize backward-compatibility, it is likely
that a new syntax for UTF-8 support is needed; this new syntax will have
to parsable as a quoted-string by applications that do not understand
it. (RFC2047 encoding does not have that property, because RFC2047
encoding is not quoted; in fact, quoting encoded-words is explicitly
forbidden by RFC2047.) 

Accordingly, any references to RFC 2047 and anything it defines should
just be removed.

- Brian

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 17 Apr 2008 05:32:10 +0200

> Comments?

You'd bet.

>> New:
>> """
>> Words of *TEXT MUST NOT contain characters from character sets  
>> other than ISO-8859-1 [ISO-8859-1].

>>                  ; any OCTET except CTLs, but including LWS

(1) If you say MUST NOT it implies "I mean it".  And if that is
    the intention permitting %x80-9F is excessively dubious. 
    Therefore you don't mean it, any you can write:

++ Words of *TEXT contain characters from the ISO-8859-1 charset.

No MUSTard, a loophole for "you can try UTF-8, and if it breaks
you own the pieces".

(2) LWS.  This horror will be limited to "FWS minus obs-FWS" in
    ABNF later, is that correct ?  No "apparently empty" lines
    only containing "significant" trailing white space, please.

>> Characters outside of ISO8859-1 MAY be included where the
>> encoded-word rule (as defined in RFC2047, Section 2) is
>> specified.

(3) How about RFC 2231, section 5 ?  It is fine if you say *NO*
    for various reasons (RFC 2231 would be a downref, and nobody
    needs language tags in <encoded-word> for HTTP), but I'd
    like to have that decision on record, just in case.

>> When used in HTTP, encoded-word has no specified length limit.

IMO you could add "and SHOULD NOT be folded".  What do the HTTP
implementation and interop reports say about 65 KB encoded-word,
1 MB, 1 GB, 1 TB ?  Does HTTP offer any header limits at all ?

>> ; according to the syntax specified by the field.

...sneaky :-)

  [no encoded word]
>> 3.3 (Media Types -- parameter values)

That okay ?  I can't tell intuitively.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Apr 2008 13:34:48 +0200

Brian Smith wrote:
> ...
> ...

I agree that we should treat this as a separate issue. It's different 
from all the other I18N use cases in that non-ISO filenames are indeed 
used in practice, and at least two UAs agree that RFC2231 is the way to go.

Mark, should I open a separate issue for that?

BR, Julian