Subject: i67: quoting charsets

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 16:55:23 +1100

<http://tools.ietf.org/wg/httpbis/trac/ticket/67>

Earlier related thread at:
http://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0020.html

Roy wrote in the issue;
> There is some confusion here. First, HTTP allows both quoted and  
> unquoted forms in Content-Type, and that certainly isn't going to  
> change. However, HTTP only uses the charset ABNF production in  
> Accept-Charset, and thus is currently defined to only allow tokens  
> in Accept-Charset.
>
> Should Accept-Charset allow charset quoted strings? I don't think  
> so. Should the charset production be removed to reduce the  
> confusion? Perhaps. This is really a design issue.
>
> This would be a lot easier if IANA kept a decent registry for  
> charset that only included the "MIME preferred names". We may need  
> to request that in the IANA considerations
>

p3 3.1 already says:
> HTTP uses charset in two contexts: within an Accept-Charset request  
> header (in which the charset value is an unquoted token) and as the  
> value of a parameter in a Content-Type header (within a request or  
> response), in which case the parameter value of the charset  
> parameter may be quoted.
I can't find this text in 2616, so I'm guessing that the editors took  
a stab at resolving this before flipping it to a design issue?

At any rate, the interesting thing to me here is that the argument for  
allowing quoted charset content-type parameters seems to be that the  
BNF for params is
token | quoted-string
i.e., all parameters inherit the ability to be quoted from the generic  
BNF.

However, as part of the discussion of our favourite issue, #74, we've  
come to the place where saying that field-content is *not* subject to  
RFC2047 encoding generically, even though its BNF refers to TEXT  
(albeit in comments).

I think we need to be more explicit about when a higher-level BNF  
rule's attributes (such as encoding and quoting) are inherited. This  
will help avoid a fair amount of reader confusion.

In this case, I'm fine with the added text above, but I think we also  
need to explicitly state that quoting in media-type parameters is  
syntactic, not semantic, and so both forms are equivalent (probably in  
p2 section 3.3) for any given parameter.

As far as accept-charset goes, I'm fine with leaving it just a token,  
and don't think we need any change there.

--
Mark Nottingham     http://www.mnot.net/

---

From: Martin Duerst <[email-masked]>
Date: Fri, 04 Apr 2008 15:45:59 +0900

>> to request that in the IANA considerations

As the secondary reviewer for that registry (Ned Free is the
primary reviewer), I would like to understand what exactly
would be easier here.

Does HTTP say anything about MIME preferred names vs. other
charset names or should it?

Is this simply another call for cleanup for the charset
registry, where we all agree that it's a mess.

Do MIME preferred names have more stringent restrictions on
their syntax that is more in line with some parts of HTTP?
(e.g. they never need quotes?)

I don't think that a removal of everything not labeled
MIME preferred name in the charset registry (at
http://www.iana.org/assignments/character-sets)
would be appropiate at all. I think that doing so via
a new HTTP spec would be even less appropriate.
Proposing individual cleanup actions may be a good idea
in some cases, please always feel free to make proposals
directly along the process for that registry
(see RFC 2978). There are of course many other protocols
using this registry.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 01:59:26 +0200

> i.e., all parameters inherit the ability to be quoted from the generic  

Yes. Simplifiers parsers consiredably if they don't need to know the
parameter token to tell if it's value is allowed to be quoted or not.

> RFC2047 encoding generically, even though its BNF refers to TEXT  

Not entirely sure what you refer to.

RFC2047 is very strict on when it may be applied.

The 2616 BNF very often uses constructs where 2047 can not be applied,
either from the BNF only allowing ASCII (not TEXT) or from being outside
the scope where RFC2047 may be applied.

> and don't think we need any change there.

+1

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 8 Apr 2008 10:38:19 +1000

so, the proposal is:

- leave accept-charset as-is.
- explicitly allow all media-type parameters defined as a token to  
occur as a quoted-string, and make them equivalent (p2 section 3.3).

Thoughts?

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 14:54:11 +0200

> occur as a quoted-string, and make them equivalent (p2 section 3.3).

Sounds good to me.

But maybe with an added recommendation to only quote parameter when
needed. Possibly as a general rule regarding "token | quoted-string"
productions.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 10 Apr 2008 17:17:03 +0200

Henrik Nordstrom wrote:
> ...
> ...

I have my doubts. Why add special cases in producers when it's not 
needed? Or *are* special cases needed?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 10 Apr 2008 17:58:46 +0200

Julian Reschke wrote:

> I have my doubts. Why add special cases in producers
> when it's not needed? Or *are* special cases needed?

I'm not sure what Henrik is up to.  For my own purposes
I treat <token> vs. <quoted-string> as "always use a
<quoted-string> instead of figuring out what is and
what is not a <token>, unless it's obvious (B64, hex)".

Within a <quoted-string> it's straight forward, for \
use \\, for " use \", for RFC 2617 avoid such cases,
and for <encoded-word> forget anything I've said. ;-)

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Thu, 10 Apr 2008 17:15:15 +0200

Mark Nottingham wrote:
>> response), in which case the parameter value of the charset parameter 
>> may be quoted.
> I can't find this text in 2616, so I'm guessing that the editors took a 
> stab at resolving this before flipping it to a design issue?
> ...

No, the editors do not do things like that silently :-)

The text was added when we resolved 
<http://skrb.org/ietf/http_errata.html#charactersets>, which is now 
<http://www.tools.ietf.org/wg/httpbis/trac/ticket/14>, and which was 
resolved with <http://www3.tools.ietf.org/wg/httpbis/trac/changeset/81>.

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 11 Apr 2008 02:33:54 +0200

> needed? Or *are* special cases needed?

I wouldn't expect all parsers to deal with quoted parameters in all
cases.. the current specs is a bit fuzzy on that.

But I do expect the ones who deal with quoted parameters to also handle
the token form..

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Apr 2008 11:23:04 +1000

Hmm. That's the opposite of the direction I *thought* we were going in.

If a parameter specifies a token, e.g., charset, I read the discussion  
as saying that we should also accept a quoted-string. This allows the  
approach that Frank was talking about (quote everything, just for  
safety and simplicity's sake).

If a parameter specifies a quoted-string, there isn't any  
corresponding rationale to support also allowing a token.

That's why I wrote the proposal as I did.

---

From: David Morris <[email-masked]>
Date: Mon, 14 Apr 2008 15:15:46 -0700 (PDT)

Probably way too late in HTTP's life cycle, but the what I'd like is to
separate quoting requirements from specific situations and generically
require quoting in any case where the value isn't safe w/o quoting.
In all cases, quoting should be removed before comparison or other
processing.

Oh well ... I can dream ...

Dave Morris

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 15 Apr 2008 03:41:43 +0200

> Oh well ... I can dream ...

A related nightmare (but ignoring 2617 unrelated to http):
<http://article.gmane.org/gmane.mail.spam.spf.devel/1897>

 Frank