Subject: i74 proposal take 2

From: Mark Nottingham <[email-masked]>
Date: Fri, 28 Mar 2008 12:19:14 +1100

* p1, 2.2:
Old:
>   The TEXT rule is only used for descriptive field contents and values
>   that are not intended to be interpreted by the message parser.   
> Words
>   of *TEXT MAY contain characters from character sets other than ISO-
>   8859-1 [ISO-8859-1] only when encoded according to the rules of
>   [RFC2047].
>     TEXT           = %x20-7E | %x80-FF | LWS
>                    ; any OCTET except CTLs, but including LWS
>   A CRLF is allowed in the definition of TEXT only as part of a header
>   field continuation.  It is expected that the folding LWS will be
>   replaced with a single SP before interpretation of the TEXT value.

New:
"""
Words of *TEXT MUST NOT contain characters from character sets other  
than ISO-8859-1 [ISO-8859-1].

     TEXT           = %x20-7E | %x80-FF | LWS
                    ; any OCTET except CTLs, but including LWS

A CRLF is allowed in the definition of TEXT only as part of a header  
field continuation.  It is expected that the folding LWS will be  
replaced with a single SP before interpretation of the TEXT value.

Characters outside of ISO8859-1 MAY be included where the encoded-word  
rule (as defined in RFC2047, Section 2) is specified. The encoded-word  
rule is only used for descriptive field contents and values that are  
not intended to be interpreted by the message parser. When used in  
HTTP, encoded-word has no specified length limit.
"""

One question to consider here -- should %x80-%x9F be included in TEXT?  
They don't fall into the syntactic definition of CTLs in 2616, but the  
are semantically control characters, AFAIK.

* p1, 2.2:
Old:
> comment = "(" *( ctext | quoted-pair | comment ) ")"

New:
"""
comment = "(" *( ctext | quoted-pair | comment | encoded-word ) ")"
"""

* p1, 4.2:
Old:
>     field-content  = <field content>
>                      ; the OCTETs making up the field-value
>                      ; and consisting of either *TEXT or combinations
>                      ; of token, separators, and quoted-string

New:
"""
field-content = <field content>
	; the OCTETs making up the field-value,
	; consisting of either *TEXT or combinations
	; of token, separators, quoted-string and encoded-word,
	; according to the syntax specified by the field.
"""

* p3, B.1:
Old:
> filename-parm = "filename" "=" quoted-string

New:
"""
filename-parm = "filename" "=" quoted-string | encoded-word
"""

* p6, 16.6:
Old:
> warn-text = quoted-string
New:
"""
warn-text = quoted-string | encoded-word
"""

Note that I have NOT suggested the use of encoded-word in the  
following places:

p1, 3.4 (Transfer Codings -- parameter values), p1, 6.1.1 (Reason- 
Phrase), p2, 10.2 (expect-extensions), p3, 3.3 (Media Types --  
parameter values), p3, 6.1 (accept-extension), p4, 3 (ETag opaque- 
tag), p6, 16.2 (cache-extension), p6, 16.4 (extension-pragma).

I think the *-extension and parameter value ones are straightforward;  
if a particular extension wants to specify use of encoded-word, it  
should; we shouldn't specify use of encoded-word in the generic  
extension construct, but leave it to the specific instances.

I don't see a use case for ETags being internationalised -- does  
anyone else? Reason-Phrase may be necessary, though.

Also, I haven't addressed From (p2, 10.3). Anybody want to take a stab  
at that?

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 28 Mar 2008 04:08:05 +0100

> One question to consider here -- should %x80-%x9F be included in TEXT?

Good, I was going to ask it...  

> are semantically control characters, AFAIK.

...yes, unless the wannabe-Latin-1 is in fact windows-1252, UTF-8, or
what else.  Decree that C1 controls MUST NOT be used.  I hate this
approach, but if it's the only forward solution let's at least make
sure that wannabe-Latin-1 has a decent chance to really be Latin-1.

Maybe go as far as saying that 0x80 is no Euro in Latin-1, and that
while 0x85 might be a NEL elsewhere it is C1 and verboten.

> comment = "(" *( ctext | quoted-pair | comment | encoded-word ) ")"

Adapted from 2047 => okay.

> ; according to the syntax specified by the field.

Maybe not good enough, for an unstructured *TEXT field
you need 2047 chapter 5 clause (1), no problem.

For a structured field you cannot permit encoded-word
as folks see fit, the structure must not be hidden by
encoding (parts of) it.

> filename-parm = "filename" "=" quoted-string | encoded-word

No 2231 folding here, might work.  Please note somewhere
that 2231 updated <encoded-word> adding optional language
tags.  Trying to forbid that could be bad.  

It is always tricky if A has a normative reference to B,
after B was updated or obsoleted by C, what does it mean
for A when working on A-bis ?  Eat our own dog food ?

> Also, I haven't addressed From (p2, 10.3). Anybody want
> to take a stab at that?

No Latin-1 in From.  For starters s/2822/2822upd/ minus
the obsolete syntax in chapter 4 of 2822upd (trust me,
you don't want anything obs-* ;-)

If you intend to evolve 2822upd From into some shrink
to fit 2047 MIME modulo 2231 forget it, only one person 
ever tried to combine 2882 + MIME, he is not here, and
he started with not less than four I-Ds full of hardcore
ABNF (nobody needs any prose or examples if the syntax
is "obvious")

 Frank

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 28 Mar 2008 14:47:35 +1100

> as folks see fit, the structure must not be hidden by

Maybe the safest thing to do would be:

> tags.  Trying to forbid that could be bad.

My first thought is just to add a reference to RFC2231 section 5,  
allowing that extension.

> ABNF (nobody needs any prose or examples if the syntax

Sounds like this should be punted to a separate issue...

---

From: Julian Reschke <[email-masked]>
Date: Fri, 28 Mar 2008 20:17:52 +0100

Mark Nottingham wrote:
>>   that are not intended to be interpreted by the message parser.  Words
> rule is only used for descriptive field contents and values that are not 
> intended to be interpreted by the message parser. When used in HTTP, 
> encoded-word has no specified length limit.
> """
> 

I think it would be the right thing to forbid them.

> comment = "(" *( ctext | quoted-pair | comment | encoded-word ) ")"

OK, but then we'll have to state somewhere where encoded-word comes 
from; <http://tools.ietf.org/html/rfc2047#section-2>?

Also, do we really Really REALLY want to require to support all what's 
in there?

>     ; according to the syntax specified by the field.

I prefer the simplified version you posted later:

> filename-parm = "filename" "=" quoted-string | encoded-word

I'd prefer to make C-D a special case where we specify *exactly* what's 
needed, nothing more (which means: RFC2231 encoding of utf-8, no line 
folding/contiuation lines).

> Note that I have NOT suggested the use of encoded-word in the following 
> places:
> 
> p1, 3.4 (Transfer Codings -- parameter values), p1, 6.1.1 
> (Reason-Phrase), p2, 10.2 (expect-extensions), p3, 3.3 (Media Types -- 
> parameter values), p3, 6.1 (accept-extension), p4, 3 (ETag opaque-tag), 
> p6, 16.2 (cache-extension), p6, 16.4 (extension-pragma).

OK, except probably for Reason-Phrase.

> I think the *-extension and parameter value ones are straightforward; if 
> a particular extension wants to specify use of encoded-word, it should; 
> we shouldn't specify use of encoded-word in the generic extension 
> construct, but leave it to the specific instances.

Yes.

> I don't see a use case for ETags being internationalised -- does anyone 
> else? Reason-Phrase may be necessary, though.

No. Yes.

> Also, I haven't addressed From (p2, 10.3). Anybody want to take a stab 

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 29 Mar 2008 12:47:13 +1100

On 29/03/2008, at 6:17 AM, Julian Reschke wrote:
> Also, do we really Really REALLY want to require to support all  
> what's in there?

Any specific thoughts? It's already been suggested that the charsets  
available be limited... anything else?

> I'd prefer to make C-D a special case where we specify *exactly*  
> what's needed, nothing more (which means: RFC2231 encoding of utf-8,  
> no line folding/contiuation lines).

Why not make that the case for all uses of encoded-word? The most  
effective way to clarify this might be to mint a new rule which  
defines itself by refining the definition of encoded-word, with the  
appropriate references and caveats.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 29 Mar 2008 13:34:26 +0100

> available be limited... anything else?

Section 4 defines two different encodings; are they both needed?

> effective way to clarify this might be to mint a new rule which defines 
> itself by refining the definition of encoded-word, with the appropriate 
> references and caveats.

Good point.

In my experience, the filename parameter in C-D actually is the single 
place where I18N really is used in *practice*. Unfortunately, out of the 
four major UAs, only two do it correctly.

BR, Julian