Subject: RE: can charsets be quoted.

From: "Melman, Howard" <[email-masked]>
Date: Thu, 8 Feb 2001 13:49:17 -0500

On Wednesday Feb 7, 2001, Paul Leach wrote:

> I believe that the word token is being used two different ways.
> This way:
> 	value                   = token | quoted-string
> and in this sentence:
> 	HTTP character sets are identified by case-insensitive tokens.
> 
> The first one is a formal ABNF definition, the second is not. I.e.,
> there was no intent to say that char-set IDs have to be "tokens" as that
> is specified in the HTTP ABNF.
> 
> I would say that it is perfectly legal for them to be quoted.

I agree.  But I think the spec could be clearer in two ways.
I think the ABNF should be removed from 3.4 (since it's not
intended) and I think the word "token" should be replaced
with the word "name" in all cases in section 3.4.  The IANA
does not refer to charset tokens, but rather charset names.
See RFC 1700 or
http://www.isi.edu/in-notes/iana/assignments/character-sets

Howard

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 9 Feb 2001 09:53:09 -0800

Hopefully this will get on the "errata" page...

> -----Original Message-----
> From: Melman, Howard [mailto:[email-masked]]
> Sent: Thursday, February 08, 2001 10:49 AM
> To: HTTP WG
> Cc: Joris Dobbelsteen; Paul Leach; Melman, Howard
> Subject: RE: can charsets be quoted.
> 
> 
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 9 Feb 2001 12:00:37 -0800

> Hopefully this will get on the "errata" page...

Why?  The spec is correct.  It takes a great deal of imagination
to believe that the use of the word token in the text should somehow imply
that the HTTP syntax excludes a quoted-string.  Any token can appear inside
a quoted string.

....Roy

---

From: "Melman, Howard" <[email-masked]>
Date: Fri, 9 Feb 2001 15:41:51 -0500

> that the HTTP syntax excludes a quoted-string.  Any token can appear inside

Perhaps, but it's not just the word "token" in text.  There
seems to be an ABNF rule in the section which as near as I
can tell adds no value to the description and does add
confusion.  Below is the text of section 3.4:

Howard

================================================================
3.4 Character Sets

   HTTP uses the same definition of the term "character set" as that
   described for MIME:

   The term "character set" is used in this document to refer to a
   method used with one or more tables to convert a sequence of octets
   into a sequence of characters. Note that unconditional conversion in
   the other direction is not required, in that not all characters may
   be available in a given character set and a character set may provide
   more than one sequence of octets to represent a particular character.
   This definition is intended to allow various kinds of character
   encoding, from simple single-table mappings such as US-ASCII to
   complex table switching methods such as those that use ISO-2022's
   techniques. However, the definition associated with a MIME character
   set name MUST fully specify the mapping to be performed from octets
   to characters. In particular, use of external profiling information
   to determine the exact mapping is not permitted.

      Note: This use of the term "character set" is more commonly
      referred to as a "character encoding." However, since HTTP and
      MIME share the same registry, it is important that the terminology
      also be shared.

   HTTP character sets are identified by case-insensitive tokens. The
   complete set of tokens is defined by the IANA Character Set registry
   [19].

       charset = token

   Although HTTP allows an arbitrary token to be used as a charset
   value, any token that has a predefined value within the IANA
   Character Set registry [19] MUST represent the character set defined
   by that registry. Applications SHOULD limit their use of character
   sets to those defined by the IANA registry.

   Implementors should be aware of IETF character set requirements [38]
   [41].

---

From: "Larry Masinter" <[email-masked]>
Date: Mon, 12 Feb 2001 11:09:02 -0800

"Although HTTP allows an arbitrary token to be used as a charset value"

It would be useful to clarify that HTTP uses charset in two contexts:
within an Accept-Charset request header (in which the charset value
is an unquoted token) and as the value of a parameter in a Content-type
header (within a request or response), in which case the parameter
value of the charset parameter may be quoted.

Larry
-- 
http://larry.masinter.net