Subject: i74: Encoding for non-ASCII headers

From: Mark Nottingham <[email-masked]>
Date: Mon, 17 Mar 2008 20:52:30 +1100

I think we've actually made progress on this; AFAICT, we seem to be  
moving towards removing the generic text WRT RFC2047 encoding and  
replacing it* with something that says that individual headers need to  
nominate an encoding mechanism directly, and giving guidance on when  
they should do so (roughly, wherever something is a candidate for  
display and/or user input).

Is that where we're at?

If so, the next step would be to craft recommendations / requirements  
about what that mechanism will be. Possibilities discussed;

a) RFC2047
b) UTF-8
c) Something from BCP137 section 5
d) IRI->URI

Separately, we'd need to open new issues for specifying these  
encodings for the field-values of:
   - From
   - Warning
   - Content-Location
   - Location
   - Referer
   - Content-Dispostion (?)

Am I overlooking anything?

* It isn't actually replacing it, it's moving it to something specific  
to the field-value of headers. I don't hear anyone talking about  
internationalising other protocol elements at this point...

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 17 Mar 2008 13:48:18 +0100

> display and/or user input).

Right. Let's clarify which of the HTTP/1.1 headers allow RFC2047-style 
encoding; and let's also document a sane solution for new headers.

> about what that mechanism will be. Possibilities discussed;

I haven't seen any evidence this being implemented.

> b) UTF-8

Unfortunately, RFC2616, Section 4.2 currently states:

     message-header = field-name ":" [ field-value ]
     field-name     = token
     field-value    = *( field-content | LWS )
     field-content  = <the OCTETs making up the field-value
                      and consisting of either *TEXT or combinations
                      of token, separators, and quoted-string>

Thus, if we take that as final word, we can't use anything but Latin1, 
thus need to encode non-Latin-1 characters.

> c) Something from BCP137 section 5

...which would be \u'nnnnnn' or &#xnnnnnn;...

> Separately, we'd need to open new issues for specifying these encodings 
> for the field-values of:
>   - From

...this one is currently defined in terms of RFC2822, Section 3.4...

>   - Warning

Currently explicitly refers to RFC2047.

 >   - Referer

These are URI references. No non-ASCII characters anyway.

>   - Content-Dispostion (?)

Content-Disposition uses I18N *inside* the parameters, for which there 
already is RFC2231.

> Am I overlooking anything?

Reason-Phrase, for instance. In general, we need to answer whether 
RFC2047 applies to everything using "comment" or "quoted-string".

> internationalising other protocol elements at this point...

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2008 09:41:45 +1100

My .02 -

> These are URI references. No non-ASCII characters anyway.

Right -- unless someone can come up with a case where it's important  
to be able to differentiate between IRIs and URIs in these. Since  
they're protocol elements that eventually boil down to URIs, I don't  
see it myself...

> RFC2047 applies to everything using "comment" or "quoted-string".

My inclination is that they're protocol elements, and not user- 
visible, therefore the conservative thing to do is just document  
current practice -- i.e., encoding isn't supported. I could see an  
argument for explicitly saying that RFC2047 does apply there, but  
anything beyond that seems a stretch.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2008 14:18:23 +1100

Given what's below, I wonder whether specifying a single encoding for  
new headers is a practical thing to do; we already may end up  
referring to 3987, 2822, 2047, and 2231 for existing headers, because  
there are appropriate encodings from each of those domains (URIs, e- 
mail addresses, and so on).

If that's the case, maybe we should just recommend that new headers  
use the most appropriate encoding scheme to their domain, list a few  
examples (see above), and fall back to recommending (say) \u'nnnnnn'  
from BCP137 if nothing more specific applies.

On 17/03/2008, at 11:48 PM, Julian Reschke wrote:

>> If so, the next step would be to craft recommendations /  
>> requirements about what that mechanism will be. Possibilities  
>> discussed;
>> a) RFC2047
>
> Thus, if we take that as final word, we can't use anything but  
> Latin1, thus need to encode non-Latin-1 characters.
>
> Content-Disposition uses I18N *inside* the parameters, for which  
> there already is RFC2231.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 18 Mar 2008 13:58:39 +0900

>they're protocol elements that eventually boil down to URIs, I don't  

Agree. But rather than saying "they eventually boil down to URIs",
it's better to say something like "they eventually will be used
for resolution" or some such.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 16:36:08 +0100

On Tue, 2008-03-18 at 09:41 +1100, Mark Nottingham wrote:
> <My inclination is that they're protocol elements, and not user- 

Both Reason Phrase and Warning-text is indended for the human and not
automata, but it's optional to actually display them.

 The Status-Code is intended
   for use by automata and the Reason-Phrase is intended for the human
   user. The client is not required to examine or display the Reason-
   Phrase.

   Warnings also carry a warning text. The text MAY be in any
   appropriate natural language (perhaps based on the client's Accept
   headers), and include an OPTIONAL indication of what character set is
   used.

   Multiple warnings MAY be attached to a response (either by the origin
   server or by a cache), including multiple warnings with the same code
   number. For example, a server might provide the same warning with
   texts in both English and Basque.

It escapes me however however how character set indication is supposed
to be done in warning texts as it's a quoted-string.. not token|
quoted-string.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Mar 2008 12:01:40 +1100

Warnings are definitely an issue, although I'm not aware of any UA  
that displays them.... anybody? Does Amaya?

WRT reason phrases -- although that's what the spec says, I think that  
'human user' is a stretch -- it's really for debugging / logging /  
development in common use. Whether elements with that use case should  
be able to be internationalised is a separate discussion, of course.

---

From: Robert Siemer <[email-masked]>
Date: Wed, 19 Mar 2008 12:48:33 +0100

> that displays them.... anybody? Does Amaya?

If I recall correctly what I've read on the polipo mailing list, 
Konqueror shows a "Warning: ..." in the status line until it gets 
overwritten with the final "Done." after loading completes.

Robert

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 20 Mar 2008 00:28:18 +0100

On Wed, 2008-03-19 at 12:01 +1100, Mark Nottingham wrote:
> development in common use.

Depends on where you draw the line between information and
diagnistics/debugging. Quite many command line or CLI tools do display
the reason phrase on non-200 responses for information to the user on
what is going on.

> Whether elements with that use case should  

True. And I don't think there is a need for this, but suspect some
people in areas where english isn't seen as a common language may
differ. The majority most likely do not care at all however.

Regards
Henrik

---

From: Nicolas Alvarez <[email-masked]>
Date: Tue, 25 Mar 2008 02:02:08 -0300

Mark Nottingham escribió:
> development in common use. Whether elements with that use case should be 
> able to be internationalised is a separate discussion, of course.

Apache Tomcat returns translated reason phrases:

HTTP/1.1 302 Movido temporálmente
Server: Apache-Coyote/1.1

Yep, that is an accented letter (which is even wrong, that word doesn't 
have an accent in Spanish!). Apparently ISO-8859-1.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2008 21:46:28 +1100

That's allowed;
> Reason-Phrase = *<TEXT, excluding CR, LF>
> The TEXT rule is only used for descriptive field contents and values  
> that are not intended to be interpreted by the message parser. Words  
> of *TEXT MAY contain characters from character sets other than ISO-  
> 8859-1 [22] only when encoded according to the rules of RFC 2047 [14].
> TEXT = <any OCTET except CTLs, but including LWS>
Cheers,

On 25/03/2008, at 4:02 PM, Nicolas Alvarez wrote:

>> WRT reason phrases -- although that's what the spec says, I think  
>> that 'human user' is a stretch -- it's really for debugging /  
>> logging / development in common use. Whether elements with that use  
>> case should be able to be internationalised is a separate  
>> discussion, of course.
>
> Yep, that is an accented letter (which is even wrong, that word  
> doesn't have an accent in Spanish!). Apparently ISO-8859-1.
>
>