Subject: %NN encoding, request/response headers, UTF-8 ?

From: Peter W <[email-masked]>
Date: Thu, 7 Jun 2001 19:21:45 -0400

I'm curious 
 - whether Unicode characters with values 0x100 and greater are allowed in 
   request headers (especially the request line)
 - if so, if UTF-8 encoding is allowed
 - how a client indicates to the server that it's using UTF-8
 - how an HTTP server application decides how to interpret
   hex-encoded information, e.g. is %C3%B1 encoding two characters,
   or the UTF-8 encoding for the single character "ñ"
 - how/if a server might use UTF-8 in its response headers

It looks like any content that is sent with MIME headers (e.g., an object 
sent by the HTTP server) could be announced with a charset value indicating 
UTF-8 encoding, but that headers (request or response) are only expected to 
contain characters 0x00 -> 0xFF. Yet I don't see this clearly stated.

It seems fairly clear, though, that double-byte character sets (e.g., 16
bits for each character regardless of its value) should not be used in
either request or response headers. Right?

I appreciate any light you may be able to shed on this topic.

-Peter
http://www.tux.org/~peterw/

---

From: "Joris Dobbelsteen" <[email-masked]>
Date: Mon, 11 Jun 2001 00:21:01 +0200

They are still used, I think.... (e.g. IE)

The specification expects ASCII characters, but if UTF-8 is compatible with
ASCII and the spec this can be used.

>-----Original Message-----
>From: Peter W [mailto:[email-masked]]
>Sent: Friday, 08 June 2001 1:22
>To: [email-masked]
>Subject: %NN encoding, request/response headers, UTF-8 ?
>
>
>I'm curious

No, solely ASCII, according to the spec. It states ASCII characters, with a
range from 0x00 to 0xFF. Depending on what you are entering, there may be
some more limitations (see spec).

> - if so, if UTF-8 encoding is allowed\

Don't know about UTF-8, I thought IE could use it to encode the URL.

>either request or response headers. Right?

Seems like it. It might, however, be possible to encode them so they comply
with the spec. Still, I'm not sure about this.

>http://www.tux.org/~peterw/

- Joris

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 16 Jun 2001 03:18:45 +0200

>   request headers (especially the request line)

This depends on RFC 2396 which only allows US-ASCII characters, i.e.
U+0000-U+007F minus further restrictions on URI syntax.

>either request or response headers. Right?

They cannot.

Take a look at IRIs,
http://www.ietf.org/internet-drafts/draft-masinter-url-i18n-07.txt
-- 
Björn Höhrmann { mailto:[email-masked] } http://www.bjoernsworld.de
am Badedeich 7 } Telefon: +49(0)4667/981028 { http://bjoern.hoehrmann.de
25899 Dagebüll { PGP Pub. KeyID: 0xA4357E78 } http://www.learn.to/quote/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 16 Jun 2001 03:19:47 +0200

>some more limitations (see spec).

US-ASCII doesn't define anything above 0x7F.
-- 
Björn Höhrmann { mailto:[email-masked] } http://www.bjoernsworld.de
am Badedeich 7 } Telefon: +49(0)4667/981028 { http://bjoern.hoehrmann.de
25899 Dagebüll { PGP Pub. KeyID: 0xA4357E78 } http://www.learn.to/quote/

---

From: "Larry Masinter" <[email-masked]>
Date: Sat, 16 Jun 2001 09:02:54 -0700

>  - if so, if UTF-8 encoding is allowed

The "request line" and "request headers" are different contexts.
If you're thinking about using IRIs as the request URI
  (draft-masinter-uri-i18n-07.txt)
it is still necessary to convert the IRI to a URI before using it
in the HTTP protocol, because the HTTP protocol only uses URIs
which have a restricted character repertoire.

>  - how a client indicates to the server that it's using UTF-8

depends on the context

>    or the UTF-8 encoding for the single character "ñ"

The hex-encoding is only used for URIs and not for other elements,
and what it encodes depends entirely on the server that serves it.
This is entirely server-dependent, and %C3%B1 might represent
something entirely different, not characters at all. There is
no restriction that encoding in URIs actually corresponds to
character data.

> contain characters 0x00 -> 0xFF. Yet I don't see this clearly stated.

Some response headers allow TEXT, e.g., on comments.

RFC 2616 section 14.46 (description of Warning header)

   If a character set other than ISO-8859-1 is used, it MUST be encoded
   in the warn-text using the method described in RFC 2047 [14].

RFC 2616, section 2.2, Basic Rules:
       OCTET          = <any 8-bit sequence of data>
       CHAR           = <any US-ASCII character (octets 0 - 127)>

...

   The TEXT rule is only used for descriptive field contents and values
   that are not intended to be interpreted by the message parser. Words
   of *TEXT MAY contain characters from character sets other than ISO-
   8859-1 [22] only when encoded according to the rules of RFC 2047
   [14].

       TEXT           = <any OCTET except CTLs,
                        but including LWS>

This means that you can use UTF-16 if you first base64 encode it
and then use it within the RFC 2047 method, viz
                =?UTF-8?b?<base64string>?=

> either request or response headers. Right?

Not without some kind of encoding, but the encoding rules differ
according to the context.

Larry
--
http://larry.masinter.net