Subject: PROPOSAL: i74: Encoding for non-ASCII headers

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2008 22:06:09 +1100

Based upon discussion, a proposal for closing i74:

* p1, section 2.2 -

> The TEXT rule is only used for descriptive field contents and values  
> that are not intended to be interpreted by the message parser. Words  
> of *TEXT MAY contain characters from character sets other than ISO-  
> 8859-1 [ISO-8859-1] only when encoded according to the rules of  
> [RFC2047].

   - remove the requirement that only RFC2047 encoding be used;  
instead, recommend that context-specific encoding rules be used  
(giving examples), and failing that, the \u'nnnnnn' form from BCP137.
   - add new issues for dealing with specific circumstances (e.g.,  
From, Content-Disposition, Warning) as necessary.

--
Mark Nottingham     http://www.mnot.net/

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Mar 2008 12:40:53 +0100

Mark Nottingham wrote:

> remove the requirement that only RFC2047 encoding be used;  
> instead, recommend that context-specific encoding rules be
> used (giving examples), and failing that, the \u'nnnnnn'
> form from BCP137.

Are you sure that you want more than one way (MIME) for this
magic, and if yes, are you sure that \u'nnnnnn' is the right
way in HTTP ?  If there is a chance that these values have to
be displayed in HTML pages or used in XML files the NCR form
&#xnnnnnn; might work "as is", for \u'nnnnnn' something needs
to determine a corresponding UTF-16, hex. NCR, or UTF-8.

 Frank

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2008 22:57:39 +1100

On 25/03/2008, at 10:40 PM, Frank Ellermann wrote:

> magic,

I think the argument here is that there's already more than one way;  
e.g., IRIs serialised as URIs, RFC2231 (for Content-Disposition), 2822  
(for From), etc.

> and if yes, are you sure that \u'nnnnnn' is the right
> way in HTTP?  If there is a chance that these values have to

I'm not particularly attached to one form or another, although BCP137  
does note the ugliness factor WRT NCRs.

Fair point WRT XML/HTML uses, although I wonder at how often that will  
happen, and it doesn't seem difficult to accommodate...

Anybody else have a preference / argument for one or the other?

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Mar 2008 13:28:33 +0100

> to determine a corresponding UTF-16, hex. NCR, or UTF-8.

Not sure I understand this.

1) Even if you want to use a value in HTML or XML, you will need to 
decode first, then re-encode, otherwise you'll end up with something 
like "&amp;xnnnnnnn;". At least if you do it properly.

2) As far as I understand, the only difference between the two formats 
(BCT137, 5.1 and 5.2) is how they are embedded.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Mar 2008 13:54:46 +0100

Mark Nottingham wrote:
>   - remove the requirement that only RFC2047 encoding be used; instead, 
> recommend that context-specific encoding rules be used (giving 
> examples), and failing that, the \u'nnnnnn' form from BCP137.
>   - add new issues for dealing with specific circumstances (e.g., From, 
> Content-Disposition, Warning) as necessary.

I do agree with the goal of allowing new headers to use a saner format; 
and also recommending a specific one (such as RFC5137, Section 5.1).

Should the ABNF reflect this? Such as:

- for TEXT, state that it allows ISO8859, plus RFC2047-encoded stuff
- introduce something like ITEXT, using RFC5137, Section 5.1 format

..and then make sure that only those headers use TEXT where this is used 
in practice...

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Mar 2008 14:01:25 +0100

Julian Reschke wrote:
 
>> If there is a chance that these values have to be displayed in
>> HTML pages or used in XML files the NCR form &#xnnnnnn; might
>> work "as is", for \u'nnnnnn' something needs to determine a
>> corresponding UTF-16, hex. NCR, or UTF-8.

> 1) Even if you want to use a value in HTML or XML, you will
> need to decode first, then re-encode, otherwise you'll end up
> with something like "&amp;xnnnnnnn;".

Not for "work as is", where decoding hex. NCRs is the job of a 
browser, or in the XML case unnecessary.  If you want something
better than "as is" for various Unicode security considerations
both notations are fine.

To protect encodings both forms allow this, your proposal &amp;
is okay for HTML and XML, maybe the &#x26; in RFC 5137 is more
general.  For the \u form RFC 5137 mentions \\u as protection,
in essence that is "double all backslashes" (for a shell prompt
when I had to do this "manually" it made me nervous... ;-)  

In both cases you'd have to explain what you want and how this
works, for NCRs that might be simpler (YMMV).

> the only difference between the two formats (BCP137, 5.1 and
> 5.2) is how they are embedded.

Yes, both specify Unicode points with hex. digits minus leading
zeros, \u uses 4..6 digits, NCRs use 2..6 digits.  Both forms
have a clear trailing terminator as recommended in Charmod.

Mark Nottingham wrote:

| BCP137 does note the ugliness factor WRT NCRs.

Yes, a matter of taste, in that case John's taste.  Pick what
you like better, for implementors both forms are trivial if the
protection is clear (if it is at all needed, I'm not sure when).

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Mar 2008 14:14:04 +0100

> better than "as is" for various Unicode security considerations

Nope. Sorry.

There are characters allowed in HTTP headers that need to be escaped 
both in HTML and XML, such as "<". So, to create HTML or XML from the 
header contents, you will need to HMTL- or XML-escape the text anyway 
(everything else is a hack).

If you do so, you can't simply include a "&#xNNNNNN;" form from the HTTP 
header, you need to decode it first.

> works, for NCRs that might be simpler (YMMV).

I'm still not sure what you're talking about. Could you please provide 
an example?

 > ...

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Mar 2008 18:52:02 +0100

Julian Reschke wrote:
 
> There are characters allowed in HTTP headers that need to be
> escaped both in HTML and XML, such as "<".

For symmetry you'd also protect ">", and of course "&" when it
doesn't introduce a hex. NCR.  At the end you can keep a hex.
NCR "as is" for the purpose of "displaying" it by insertion in
a dummy <title></title><pre> header </pre> HTML file.

Works like a charme for transformations of BCP 47 registries
to XML, admittedly that has the advantage of not using raw &
characters, any & always starts a hex. NCR, for examples see
<http://www.iana.org/assignments/language-subtag-registry>
<http://xyzzy.webhp.info/home/ltru/ltru2xml.awk> (script)
<http://xyzzy.webhp.info/home/ltru/ltru801.xml>  (output)

> Could you please provide an example?

See above.  I can't say why you mentioned &amp;#x26; output as
example for &#x26; input, it has nothing to do with displaying
hex. NCRs "as is".  But of course it is possible to do this, 
like it's possible to protect input \u'0026' by say \\u'0026' 
& => &amp; or & => &#26; is the same idea as \ => \\

 Frank

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 25 Mar 2008 11:21:56 -0700

On Mar 25, 2008, at 4:06 AM, Mark Nottingham wrote:
>> The TEXT rule is only used for descriptive field contents and  
>> values that are not intended to be interpreted by the message  
>> parser. Words of *TEXT MAY contain characters from character sets  
>> other than ISO- 8859-1 [ISO-8859-1] only when encoded according to  
>> the rules of [RFC2047].
>

I see no reason to change the existing encoding requirement unless
we are to allow raw UTF-8 in headers.  Anything else would just make
the implementations worse.  BCP137 is not mature enough to use in HTTP.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 26 Mar 2008 10:51:07 +1100

Apologies, I omitted some critical aspects. TEXT is defined as:

> values
>    8859-1 [22] only when encoded according to the rules of RFC 2047
>    [14].
>
>        TEXT           = <any OCTET except CTLs,
>                         but including LWS>

And this text defining header field-values;

>        field-content  = <the OCTETs making up the field-value
>                         and consisting of either *TEXT or combinations
>                         of token, separators, and quoted-string>

This has caused confusion, because one of the possible readings is  
that *all* header field-values have the potential for carrying RFC2047- 
encoded TEXT. This needs to be clarified.

Furthermore, quoted-string inherits TEXT;

>    A string of text is parsed as a single word if it is quoted using
>    double-quote marks.
>
>        quoted-string  = ( <"> *(qdtext | quoted-pair ) <"> )
>        qdtext         = <any TEXT except <">>

...but is used in places that clearly have the potential for being  
interpreted by the message parser (e.g., parameter values, etags,  
accept-, cache- and expectation-extensions).

E.g., a reasonable reading of the specification is that two ETags, one  
using RFC2047 encoding, and one not, are equal; I haven't checked, but  
I doubt that anyone has implemented this when they do comparison.

One way to fix both of these problems is to state that encoding is  
possible in specific use cases, rather than having a blanket statement  
about it in TEXT that is easily missed and not well-implemented.

A secondary issue is what encoding should be used in those cases were  
it is reasonable to allow it. I'm not sure what the value of requiring  
that it be the same everywhere is; some payloads (e.g., IRIs, e-mail  
addresses) have well-defined "natural" encodings into ASCII that are  
more appropriate.

Mind you, personally I'm not religious about this; I just think that  
if we mandate RFC2047 encoding be used in new headers that need an  
encoding, we're going to be ignored, for potentially good reasons.

So, revised proposal:

  1) Remove "Words of *TEXT MAY contain characters from character sets  
other than ISO-8859-1 [22] only when encoded according to the rules of  
RFC 2047 [14]."
  2) Constrain TEXT to contain only characters from iso-8859-1.
  3) Add advice that, for a particular context of use, other  
characters MAY be encoded (whether that's strictly RFC2047, or more  
fine-grained advice TBD) by specifying it in that context.
  4) Add new issues for dealing with specific circumstances (e.g.,  
From, Content-Disposition, Warning) as necessary. If the outcome of #3  
is to require RFC2047, this is relatively straightforward.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 25 Mar 2008 17:40:20 -0700

On Mar 25, 2008, at 4:51 PM, Mark Nottingham wrote:
> that *all* header field-values have the potential for carrying  
> RFC2047-encoded TEXT. This needs to be clarified.

It may need to be clarified, but that is not a possible reading.
ABNF rule field-content is only used in field-value which is only
used in message-header, which in turn is only used to represent the
superset of all header fields in the generic parser. In other words,
it is not a defining element for header generation.

> accept-, cache- and expectation-extensions).

Right, we should fix those cases that do not allow non-ASCII encodings.

> E.g., a reasonable reading of the specification is that two ETags,  
> one using RFC2047 encoding, and one not, are equal; I haven't  
> checked, but I doubt that anyone has implemented this when they do  
> comparison.
>
> possible in specific use cases, rather than having a blanket  
> statement about it in TEXT that is easily missed and not well- 
> implemented.

+1

> A secondary issue is what encoding should be used in those cases  
> were it is reasonable to allow it. I'm not sure what the value of  
> requiring that it be the same everywhere is; some payloads (e.g.,  
> IRIs, e-mail addresses) have well-defined "natural" encodings into  
> ASCII that are more appropriate.

Unless we are going to change the protocol, the answer to that question
is ISO-8859-1 or RFC2047.  If we are going to change the protocol, then
the answer would be raw UTF-8 (HTTP doesn't care about the content of
TEXT as long as the encoding is a superset of ASCII, so the only
compatibility issue here is understanding the intent of the sender).

> Mind you, personally I'm not religious about this; I just think  
> that if we mandate RFC2047 encoding be used in new headers that  
> need an encoding, we're going to be ignored, for potentially good  
> reasons.

What good reasons?  In this case, we are not mandating anything.
We are simply passing through the one and only defined i18n solution
for HTTP/1.1 because it was the only solution available in 1994.
If email clients can (and do) implement it, then so can WWW clients.

People who want to fix that should start queueing for HTTP/1.2.

> So, revised proposal:
>
>  1) Remove "Words of *TEXT MAY contain characters from character  
> sets other than ISO-8859-1 [22] only when encoded according to the  
> rules of RFC 2047 [14]."

We can just introduce a new production for quoted-tokens.

>  2) Constrain TEXT to contain only characters from iso-8859-1.

No, that breaks compliant senders.

> From, Content-Disposition, Warning) as necessary. If the outcome of  
> #3 is to require RFC2047, this is relatively straightforward.

There is no great need that has been established to support any
changes to the allowed TEXT encoding other than to separate the
rules that don't actually allow that encoding.  IMO, changes to
HTTP/1.1 must be motivated by actual implementations.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 26 Mar 2008 12:01:52 +1100

> compatibility issue here is understanding the intent of the sender).

What do you mean by ISO-8859-1 *or* RFC2047 here?

Even if RFC2047 encoding is in effect, the actual character set in use  
is a subset of ISO-8859-1; no characters outside of that are actually  
on the wire, it's just an encoding of them into ASCII.

This is why I question whether it's realistic to require RFC2047,  
given that some applications -- e.g., headers that might want to carry  
a IRI -- are already using an encoding that's not RFC2047.

Of course, you can say that they're not carrying non-ASCII characters,  
because it's just a URI, but I'd say that's just a way of squinting at  
the problem, and RFC2047 is yet another way of squinting; it looks  
like it's just ASCII as well.

> If email clients can (and do) implement it, then so can WWW clients.

See above. Specifically, what impact does the requirement to use  
RFC2047 have on other encodings -- is it saying that serialising an  
IRI as a URI in a HTTP header is non-conformant? That if another  
problem domain, for whatever reason, decides to mint a header that  
uses BCP137 instead of RFC2047, that it also violates HTTP? This seems  
a stretch to me... I'd put forth that the requirement is spurious.

> People who want to fix that should start queueing for HTTP/1.2.

Please explain how removing the requirement that only RFC2047 be used  
to encode non-ISO-8859-1 characters in new headers requires a version  
bump.

> No, that breaks compliant senders.

How? Are you saying that senders are already sending text that  
contains non-8859-1 characters (post-encoding)?

> HTTP/1.1 must be motivated by actual implementations.

Could be. Again, my main concern here is to take the blanket  
requirement away and make it more focused.

---

From: "Robert Brewer" <[email-masked]>
Date: Tue, 25 Mar 2008 18:07:10 -0700

> I doubt that anyone has implemented this when they do comparison.

CherryPy implements that (since "opaque-tag = quoted-string"). Granted,
it's a small project, but when I implemented RFC2047 decoding, that was
the only "reasonable reading" I arrived at. I must have glossed over the
"octet equality" clause for ETag at the time. :/

> about it in TEXT that is easily missed and not well-implemented.

Sounds reasonable. Perhaps the phrase "A string of text is parsed as a
single word if it is quoted using double-quote marks" needs some
elaboration as well.

Robert Brewer
[email-masked]

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 26 Mar 2008 11:32:51 -0700

On Mar 25, 2008, at 6:01 PM, Mark Nottingham wrote:
>>> IRIs, e-mail addresses) have well-defined "natural" encodings  
>>> into ASCII that are more appropriate.
>>

I meant the existing requirements as stated in 2616 for TEXT (but only
for those rules that really should be TEXT).

> Even if RFC2047 encoding is in effect, the actual character set in  
> use is a subset of ISO-8859-1; no characters outside of that are  
> actually on the wire, it's just an encoding of them into ASCII.
>
> given that some applications -- e.g., headers that might want to  
> carry a IRI -- are already using an encoding that's not RFC2047.

HTTP does not carry IRIs in the headers. There are many implementation
reasons why an IRI must be encoded as a URI before being used in a
protocol context (security, minimization of duplicates, decoding once
at the source instead of at every recipient, ...).

> Of course, you can say that they're not carrying non-ASCII  
> characters, because it's just a URI, but I'd say that's just a way  
> of squinting at the problem, and RFC2047 is yet another way of  
> squinting; it looks like it's just ASCII as well.

RFC2047 was chosen because the only fields we have that use TEXT
are actually defined by MIME and hence are already required to use
that encoding in their normal contexts.

> IRI as a URI in a HTTP header is non-conformant?

No.  A URI is not TEXT, nor is it an IRI.  An IRI can be translated
to the actual URI it represents in a protocol context.  IRI is just
one interpretation of the encoding of a URI.

> That if another problem domain, for whatever reason, decides to  
> mint a header that uses BCP137 instead of RFC2047, that it also  
> violates HTTP? This seems a stretch to me... I'd put forth that the  
> requirement is spurious.
>
> Please explain how removing the requirement that only RFC2047 be  
> used to encode non-ISO-8859-1 characters in new headers requires a  
> version bump.

Because there are no existing implementations that do what you  
suggested.
We could spend a ridiculous amount of time trying to draft a set of
rules by which both existing compliant messages and newly encoded
messages can be parsed correctly, but since we have NO NEED for such
a mechanism and NO IMPLEMENTATIONS to guide us, we might as well be
drafting HTTP/1.2.

> contains non-8859-1 characters (post-encoding)?

I thought you meant disallow 2047 encoding.

>>> characters MAY be encoded (whether that's strictly RFC2047, or  
>>> more fine-grained advice TBD) by specifying it in that context.
>>> From, Content-Disposition, Warning) as necessary. If the outcome  
>>> of #3 is to require RFC2047, this is relatively straightforward.
>>

Then we can just rephrase the requirement so that it isn't blanketed.

....Roy

---

From: Martin Duerst <[email-masked]>
Date: Thu, 27 Mar 2008 14:33:16 +0900

>the implementations worse.  BCP137 is not mature enough to use in HTTP.

I definitely agree with Roy here.

HTTP is not a context limited to 7bit bytes,
so there is no need for anything from BCP137.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Thu, 27 Mar 2008 14:28:48 +0900

Hello Mark, Roy, others,

>> actually on the wire, it's just an encoding of them into ASCII.

Mark, I think you have to be explicit on when you mean the bytes
on the wire and when you mean the characters they represent, and
when you mean the process or method of encoding. It's usually
pretty clear from context, but when reading your
earlier emails, I got as confused as Roy.

>at the source instead of at every recipient, ...).

Just an aside:
I don't really understand the "decoding once at the source instead";
that might argue for encoding IRIs to URIs before putting them e.g.
in a href attribute, but doesn't really apply that much in an HTTP
context (in many cases, the server actually decodes the URI back
to bytes, which may be looked up directly as UTF-8 in the file system).
If it applies to chains of proxies, I don't think these are usually
that long to make this a serious issue.

>that encoding in their normal contexts.

The choice of adopting the basic email/MIME mechanisms was a good one.
In many areas, that was followed up by a careful selection and some
changes (e.g. for line breaks, for entity boundaries,...). I very
much wish a little bit more restriction had been applied to the
jungle of escaping conventions in mail headers (completely independent
of internationalization).

>>> What good reasons?

Mostly, ease of implementation. See below.

>>> In this case, we are not mandating anything.

"The only solution available in 1994" is a far stretch. Unicode
was available then. But I agree that the choice, in 1994 terms,
wasn't as bad as an idea as it looks now.

>>> If email clients can (and do) implement it, then so can WWW clients.

Email clients implement RFC 2047. They do NOT implement the strange
mixture of iso-8859-1 and RFC 2047 sketched out in the HTTP spec.
A lot of libraries are available for RFC 2047. That cannot be said
for the strange mixture of iso-8859-1 and RFC 2047. Correctly
encoding or decoding an iso-8859-1/RFC 2047 mixture is not trivial,
even assuming that an RFC 2047 library is available.

>a mechanism and NO IMPLEMENTATIONS to guide us, we might as well be

There is the question of what to say about new headers.
I think it's clear that the possibility for new headers should
be left open in HTTP/1.1, that we don't want to require HTTP/1.2
for that. I think it's also clear that new headers, per definition,
are not implemented yet. So the question arises: iff somebody defines
a new header, what should they do in terms of internationalization
if they need it?

Now my understanding is that when it comes to specific (in particular
new) headers, the average HTTP/1.1 implementation doesn't do much
of anything. Indeed Roy wrote above:
   HTTP doesn't care about the content of TEXT as long as the encoding
   is a superset of ASCII,...
On the other hand, those components (plugins, modules, scripts, you
name it) that actually get written to deal with new header functionality
have to do the actual work of encoding/decoding characters. It is
very clear that for such applications, using UTF-8 is much easier
than using the ugly mixture of iso-8859-1 and RFC 2047.

Now Roy has argued, as far as I understand it, that we can't
possibly allow new headers to use UTF-8, "[u]nless we are going
to change the protocol". If this is about the protocol as written
in RFC 2616, this is probably true (although see below), but what
we are supposed to be most interested in is the protocol as implemented.
Just a few days ago, we have had what's probably the first
report of some actual iso-8859-1 data (some Spanish with
a wrong accent). But how much is that in terms of implementations?

In other words, how much is actually going to break if we allow
new headers to use UTF-8, and they go ahead and use it? How does
that minor potential breakage compare to the implementation
benefits for new headers using UTF-8? Is it worth bumping the
protocol version to 1.2 just for this minor fix? Is it a
mistake to admit that we have learned a bit between 1994
and 2008? Please answer for yourself.

On top of that, Mark was looking at whether we can prohibit
e.g. something along the lines of BCP 137. These things,
including RFC 2047, BCP 137-style stuff, IRI->URI conversion,
punycode,... are called ASCII-compatible encodings (ACE for
short), because they encode something (a large set of characters)
into something that to the lower level looks just like ASCII.
Well, it is also possible to see UTF-8 as a Latin-1-compatible
encoding (L1CE anybody). The lower layer (HTTP) may still see
it as iso-8859-1 bytes, but the higher layer will interpret it
as UTF-8.

>I thought you meant disallow 2047 encoding.

I agree with Roy here. Disallowing 2047 and leaving only iso-8859-1
would indeed be a bad idea.

Regards,    Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 27 Mar 2008 16:41:51 +1100

My reading is that HTTP is limited to iso-8859-1 *on the wire*, and  
requires RFC2047 encoding for characters outside of that range. Do you  
disagree with that?

My intent was not to disallow RFC2047, but rather to allow other  
encodings into iso-8859-1 where appropriate. Disallowing RFC2047 would  
be foolish and counter-productive; however, no one has said why  
constraining future headers from using other encodings than RFC2047 is  
a desirable -- or even realistic -- thing to do.

I'll follow up in more detail soon...

---

From: Martin Duerst <[email-masked]>
Date: Thu, 27 Mar 2008 19:17:14 +0900

>requires RFC2047 encoding for characters outside of that range. Do you  

That's what's written in RFC 2616. The question is whether and to
what extent that's (still) sensible in practice.

>encodings into iso-8859-1 where appropriate.

What do you mean by "other encodings into iso-8859-1"?
Please explain.

>Disallowing RFC2047 would  

Looking forward to it.     Regards,   Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 27 Mar 2008 22:00:19 +1100

> What do you mean by "other encodings into iso-8859-1"?

I thought I had, but obviously not well.

Roy said (to paraphrase) that IRIs do not show up in HTTP -- that  
they're just URIs. I agree with that, but only as far as you can view  
IRIs as an encoding into ASCII (albeit an imperfect one, because you  
can't round-trip them, since there's a bit of ambiguity).

RFC2047 is also an encoding into ASCII; it is not a character encoding  
in its own right. In that sense, it's a peer of BCP137 and other  
schemes that do similar things. They all end up taking characters from  
a set greater than that available to iso-8859-1 and encoding them into  
a subset of it (usually ASCII) using escape sequences.

That being the case, my question is this: is it realistic to require  
all headers to use RFC2047 encoding, to the exclusion of BCP137, etc?

I could understand such a requirement if we had a blanket requirement  
that RFC2047 encoding could occur anywhere, so that implementations  
could blindly decode/encode headers as necessary, whether they  
recognised them or not. However, we're not going in that direction,  
because it's not reasonable to implement, and in any case the encoding  
is already tied to the semantics of the headers somewhat, since you  
have to recognise the header to understand its structure enough to  
know where TEXT may appear (i.e., it's not a complete blanket, just an  
uneven one over TEXT).

That being the case, I can't help but see the RFC2047 requirement as  
spurious, and the most straightforward thing to do would seem to be to  
ditch the spurious requirement and move on -- without disallowing  
RFC2047 encoding from being specified in a particular header if that  
makes sense, but not disallowing other encodings either.

Once again, I have no desire to lie down in the road on this part of  
the issue -- I'm happy to give this up and move on, or to be told how  
wrong I am (as i18n isn't my area by any means). I'm just a bit  
surprised at how hard it is to communicate this view (which leads me  
to believe that indeed I've got something wrong here).

Cheers,

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 27 Mar 2008 16:25:37 +0000

Martin Duerst wrote:
> new headers to use UTF-8, and they go ahead and use it? 

I'm in favour of allowing UTF-8.  But we should probably also consider
what recipients should do on recieving _invalid_ UTF-8 in that case.

RFC3629 says:

   Implementations of the decoding algorithm above MUST protect against
   decoding invalid sequences.  For instance, a naive implementation may
   decode the overlong UTF-8 sequence C0 80 into the character U+0000,
   or the surrogate pair ED A1 8C ED BE B4 into U+233B4.  Decoding
   invalid sequences may have security consequences or cause other
   problems.  See Security Considerations (Section 10) below.

Firstly, which part of a recipient is expected to process the UTF-8,
or to pass it through?  And should receipt of invalid UTF-8 be an
error condition, or something which doesn't matter?  From a pure
protocol perspective, that means what should proxies do on receiving
invalid UTF-8.

Secondly, what is invalid UTF-8?  It depends if you're looking at the
ISO-10646 or Unicode definitions.  What about the Java modified UTF-8?
Are some characters disallowed?  (Similar questions apply to IRIs.)

Anything currently sending ISO-8859-1 would almost certainly be
invalid UTF-8.  This is in fact useful.  It is quite common to test
whether a byte sequence is valid UTF-8, and if not, treat it as
ISO-8859-1, because the test is quite effective at distinguishing them
in practice.

So, for informative text (i.e. non protocol) such as text after a
status code, it might be appropriate to recommend that TEXT be parsed
as UTF-8 when valid, and ISO-8859-1 otherwise.

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 27 Mar 2008 10:19:09 -0700

> wasn't as bad as an idea as it looks now.

Unicode was available but unusable as a superset of ASCII. UTF-8 was
defined but not published to the people who would actually use it.
Fonts were conceived but not distributed.  No matter how you spin it,
Unicode did not become a legitimate choice worldwide until 1995
(when both Win95 and Java started distributing the beginnings of a
usable UTF-16 support), and did not become usable in text headers
until UTF-8 was popularized by folks outside Uniforum much later.

In any case, UTF-8 would have been chosen today if we could get
four people to implement it instead of just talking on IETF lists.
The standard specifies iso-8859-1 because that is what all
implementations implemented at that time, not because it was the
most popular choice of standards mavens.  I don't care what the
encoding of TEXT is in HTTP so long as there are implementations
that are interoperable.

....Roy

---

From: "Robert Brewer" <[email-masked]>
Date: Thu, 27 Mar 2008 10:28:34 -0700

> all headers to use RFC2047 encoding, to the exclusion of BCP137, etc?

BCP137 itself says "...this specification does not recommend one
specific syntax." That is, I don't see them as peers. RFC2047 is how
HTTP "defines the syntax" for TEXT already, which means any compliant
HTTP/1.1 implementation already has code for this. How wide are we going
to open the floodgates for other encodings? As a server author, I'd
rather not have to add large chunks of code in 2008 to become "http-bis
compliant". I'm pretty happy with RFC2047.

> because it's not reasonable to implement...

I don't understand. From where I sit that sounds like not only a snap to
write from scratch, but has the potential to simplify a lot of
codebases.

> ...and in any case the encoding

Hrm. I'm not sure what "other encodings" includes. When Jamie Lokier
says "I'm in favour of allowing UTF-8," does that mean the unicode
string u'\u212bngstr\xf6m' would emit as:

    If-Match: %E2%84%ABngstr%C3%B6m

...and how is the server supposed to know how to decode that? There's
one thing that RFC2047 provides that other, more minimal, encoding
schemes do not provide: the small bit of metadata that actually declares
which encoding is being used. If you want to encode your non-ASCII
header as UTF-8, fine, that's not in opposition to RFC2047:

    If-Match: =?utf-8?q?=E2=84=ABngstr=C3=B6m?=

It's not only utf-8 but the server knows it's utf-8 without having to
sniff anything.

Robert Brewer
[email-masked]

---

From: Albert Lunde <[email-masked]>
Date: Thu, 27 Mar 2008 13:58:20 -0400

> as UTF-8 when valid, and ISO-8859-1 otherwise.

Alternate character encodings have been fruitful ground for security
attacks in the past. So I'd worry about adding too many alternate
ways to interpret header bytes, even if there is a way to distingush
them in most cases.

-- 
    Albert Lunde  [email-masked]
                  [email-masked]  (new address for personal mail)
                  [email-masked] (old address)

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 27 Mar 2008 21:07:43 +0000

> ways to interpret header bytes, even if there is a way to distingush

That's a fair point, but:

   The TEXT rule is only used for descriptive field contents and values
   that are not intended to be interpreted by the message parser.

And:

   TEXT           = <any OCTET except CTLs, but including LWS>

As a sender can choose any TEXT to send, and it has no protocol
specified meaning, it is already allowed to transmit UTF-8 encoded
character (excluding CTLs) in any TEXT production, from a perspective
of what octet sequences implementations must accept.  It's also
allowed to transmit ISO-8859-1 (excluding CTLs).  Of course, if the
recipient displays the TEXT in a message, it might not show the
characters intended by the sender in either case.

There are a _lot_ of programs which interpret TEXT in reality: the
comment in a User-Agent header is often matched for known substrings.

Imho, it makes sense for that sort of matching to operate on the
octets without attempting character decoding, rather than after
character decoding.  One reason is to avoid security holes, much as
you suggest.  Even decoding RFC2047 has the potential to introduce
surprises there.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 27 Mar 2008 21:28:18 +0000

>     If-Match: %E2%84%ABngstr%C3%B6m

Quotation marks are needed (see grammar for entity-tag and
quoted-string) and the UTF-8 octets are presented directly on the
wire, not percent-encoded as you've done.

> ...and how is the server supposed to know how to decode that?

In this case, it should match the octets within the quoted string,
which correspond with the octets it sent.

_Allowing_ UTF-8 does not mean that the client is expected to validate
or interpret UTF-8 in every place where TEXT occurs (e.g. an
entity-tag).  It means simply that it's allowed, and where
meaningfully treated as _characters_, it might be decoded that way.

It isn't meaningful for a client to decode the octets of an entity-tag
into characters, though, and much better if it doesn't.

It _is_ meaningful for a client to decode the TEXT of Reason-Phrase,
e.g. for display purposes, but _not_ for protocol reasons.

RFC2616 implies an agent displaying a Reason-Phrase should decode it
to ISO-8859-1 characters and also apply RFC2047 decoding.  Clearly,
some agents don't do this.

It seems quite unlikely to introduce any interop problems if that is
respecified to say an agent should decode it as UTF-8 (and be
leniant), or anything else similar.

Decoding to characters should _not_ happen for protocol purposes, as
"the TEXT rule is only used for descriptive field contents and values
that are not intended to be interpreted by the message parser".  So
anything which simply passes around TEXT (e.g. quoted-string,
entity-tag) should pass on the octets it receives without interpreting
or modifying them (except to scan for quotation marks etc. as required
for header parsing).

I suggest this includes _not_ attempting to decode & re-encode
protocol elements containing RFC2047 sequences, as in practice _that_
seems like it would cause interop and security issues, as most current
software using HTTP would see that as an unexpected mangling of the
text.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 27 Mar 2008 22:35:02 +0100

> "octet equality" clause for ETag at the time. :/

2047 says

   + An 'encoded-word' MUST NOT appear within a 'quoted-string'.

and 2616

      opaque-tag = quoted-string

So explain to me how seemingly 2047 encoded text in an ETag can be read
as an encoded-word?

Regards
Henrik

---

From: "Robert Brewer" <[email-masked]>
Date: Thu, 27 Mar 2008 14:52:03 -0700

Henrik Nordstrom wrote:
> > CherryPy implements that (since "opaque-tag = quoted-string").
> > Granted, it's a small project, but when I implemented RFC2047
> > decoding, that was the only "reasonable reading" I arrived at.
> > I must have glossed over the "octet equality" clause for ETag
> > at the time. :/
> 
> 2047 says
> 

It can't. I was pointing out that I got it wrong.

Robert Brewer
[email-masked]

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 28 Mar 2008 10:23:51 +1100

On 28/03/2008, at 4:19 AM, Roy T. Fielding wrote:

> I don't care what the

... and as long as existing conformant implementations don't break.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 28 Mar 2008 10:45:05 +1100

> write from scratch, but has the potential to simplify a lot of

And slow them down; most implementations (client, server and  
intermediary) are performance-sensitive, and the number of headers  
where i18n content is useful is very small. If we specified blanket  
encoding, it wouldn't get implemented.

> Hrm. I'm not sure what "other encodings" includes.

Concretely, our options at this point are:

1) Change the character encoding on the wire to UTF-8
2) Leave the character encoding on the wire at ISO-8859-1, document  
existing TEXT instances' encoding requirements on top of that, and
    a) Require new headers that need i18n content to specify RFC2047, or
    b) Require new headers that need i18n content to specify *some*  
encoding into ISO-8859-1 using character escapes (which explicitly MAY  
be RFC2047).

In either case,  i18n content isn't allowed in *any* header -- only  
those places where the header specifically allows it.

<chair_hat>
At this point, my reading is that we're leaning towards (2a); some  
people have spoken in favour of (1), but others have expressed  
concerns about backwards-compatibility, etc., but there also seems to  
be good support for, and no strong objection, to (2a) -- except for  
me, in favour of (2b), and as I said before, that's not a strong  
preference.

At this point, I think we need a more exact proposal to move forward...
</chair_hat>

... which I'll work on. If someone wants to work on text for #1 for  
comparison, feel free.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 28 Mar 2008 15:04:12 +0900

>four people to implement it instead of just talking on IETF lists.

I'd be glad to help with implementing. I'm just still not sure what
you mean by "iso-8859-1 is implemented". After repeted calls for
examples on this list, we have finally received a single example
(Spanish with a wrong accent). And you have said yourself that
HTTP mostly cares about the bytes, which means that in terms of
actual implementations, it's 99.9% or so byte-in-byte-out, and
maybe occasionally in some corner there is a tiny bit of iso-8859-1.

If it's about a server sending some iso-8859-1, it's easy to set
up a server that sends out UTF-8. If it's about a client interpreting
something as iso-8859-1, it's a bit more difficult but still possible
to change that client to use UTF-8. As my main proposal is to
allow definers of new headers to choose UTF-8, I'd probably
implement a new Foo: header or so just to prove a point.
If you can point to the location(s) in Apache httpd where
iso-8859-1 in headers makes a difference, please do so.

>encoding of TEXT is in HTTP so long as there are implementations

There is obviously quite a bit of circularity here. The first version
of a spec says it's iso-8859-1, that's what people implement
(if they do), then the second version says that's what we have to do
because that's what the implementations do, and so on. That never
allows e.g. a new header to choose something easier and simpler.
What would be your conditions to break out of this lock-in?

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 28 Mar 2008 15:14:11 +0900

>protocol perspective, that means what should proxies do on receiving

Very clear: Pass things through. As RFC 2616 allows RFC 2047 encoding,
and that already allows UTF-8, we already have that problem, and I
don't think any proxy is doing anything else than pass things through.
Checking is only necessary upon interpretation (e.g. conversion to
other encoding, use for a filename,...).

>ISO-10646 or Unicode definitions.

Different definitions in different versions of these standards
(and STD 63/RFC 3629) may be slighly out of sync at a certain time
because the different processes move at different speeds, but they
are carefully being syncronized. Also, because many of these issues
are security-related, it's not really so much an issue of what's
allowed by the standards; if you know about a security issue,
you make sure you deal with it independent of whether a given
standard has already been updated to include that issue or not.

>What about the Java modified UTF-8?

That's not UTF-8.

>Are some characters disallowed?  (Similar questions apply to IRIs.)

The IRI spec defines this quite clearly for it's purposes.

>ISO-8859-1, because the test is quite effective at distinguishing them

Yes indeed.

>as UTF-8 when valid, and ISO-8859-1 otherwise.

I agree with Albert that this would be a bad idea. I'm trying to
propose UTF-8 for new headers to get away from iso-8859-1, not to
perpetuate iso-8859-1.

Regards,    Martin.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 09:21:39 +0000

> encoding into ISO-8859-1 using character escapes (which explicitly MAY  

An issue I have with RFC2047 is it seems to imply every "proper"
implementation of a HTTP reciever, which does something with received
TEXT (such as display it), needs to have a _large_ table of known
character set names and conversion routines.

With email this is unavoidable due to history, but it seems silly for
a HTTP reciever to need it.

Since RFC2047 isn't (currently) seen in practice in HTTP, if RFC2047
continues to be recommended for TEXT, may I suggest that it be
recommended to _only_ designate the "utf-8", "iso-8859-1" and
"us-ascii" character set in RFC2047 encodings in HTTP?

That way, at least, HTTP receivers which aim for a complete,
conformant implementation and expect to do something as simple as,
e.g. decode and show received TEXT, will be complete by just decoding
those character sets.

-- Jamie

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 28 Mar 2008 10:29:48 +0100

> 1) Change the character encoding on the wire to UTF-8

-1

Some people proposed to handle headers by just looking at the octets  
alone. While that is good advice to detect line ends and such, most  
HTTP client implementations *have to* convert the header names and  
values to "characters" since those are used in their own APIs and  
host languages.

>    a) Require new headers that need i18n content to specify  
-1
ugly, no improvement

> encoding into ISO-8859-1 using character escapes (which explicitly  
> MAY be RFC2047).

+1

My personal preference is to follow the AtomPub decision on the slug  
header encoding. If memory servers me right, there was a lot of  
fruitful discussion about it and opinions asked from asians  
implementors that finally resulted in the encoding adopted.
(See RFC 5023 ch. 9.7.1)

//Stefan

--
<green/>bytes GmbH, Hafenweg 16, D-48155 Münster, Germany
Amtsgericht Münster: HRB5782

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 09:32:21 +0000

> propose UTF-8 for new headers to get away from iso-8859-1, not to

I agree with both of you, and I'm suggesting ISO-8859-1 fallback
_only_ for compatibility with currently existing headers which already
contain ISO-8859-1, and for Reason-Phrase.  Sorry for being unclear
about that.

That's because I'd expect HTTP receiver implementations to do that
anyway, if ISO-8859-1 is seen in the wild, whether the specification
recommends it or not.

If it turns out that nobody, in real life, is using ISO-8859-1 at the
moment for those things, then of course drop it.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 09:45:06 +0000

> values to "characters" since those are used in their own APIs and  

Header names must be tokens anyway, which are a subset of US-ASCII.
Nobody is proposing to change that.

Header values: HTTP implementations which must use "character strings"
tend to interpret octet as though they are ISO-8859-1, since that is
mentioned in the RFC.  So there is no actual problem with receiving
UTF-8, only that the character strings don't contain the right
characters for display and such (but can be converted to do so; the
information is present).  This is actually no different when they
receive RFC2047 encodings, as most HTTP implementations will not
decode those to the intended characters either, and those also have
validation issues.

So, in the case of receiving RFC2047 _or_ binary UTF-8, HTTP
implementations using character strings internally will actually pass
character sequences which aren't the intended "meaningful" characters,
except for those in the US-ASCII subset.

In that respect, binary UTF-8 on the wire doesn't change anything from
the present situation with RFC2047 :-)

If the wire encoding were officially UTF-8, those implementations will
tend to validate what they receive as UTF-8, with issues if they
receive something which doesn't validate.  But the same applies in the
present situation, in theory, as RFC2047 decoding can also fail to
validate.  It seems to me exactly the same issues apply with binary
UTF-8 as with RFC2047 encodings, in terms of where decoding and
validation occur (or don't), and the way mystical "I18n encoded into
ISO-8859-1" is what is really handled within implementations.

-- Jamie

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 28 Mar 2008 11:48:17 +0100

Am 28.03.2008 um 10:45 schrieb Jamie Lokier:

>> [...]

You are correct that the information would still be there. And it is  
tempting to shoot for UTF-8.

My personal feeling remains however that there is not enough to be  
gained here for introducing heuristics on char-encoding detection. If  
we follow your scenario, users of a http client API would have to  
check heuristically if the characters received as header values are  
valid UTF-8 sequence when 8859-1 converted back to octets. Next is a  
release of the HTTP client library which would need to do the same on  
the received octets. And everyone will keep their fingers crossed  
that they do not encounter an intermediary that makes some "security  
filtering" on HTTP headers and screws it up.

The atom "slug header" approach makes more sense to me. It keeps HTTP  
header handling less complicated at the cost of a few more octets on  
the wire. And yes, I am all for deprecating anything but ascii in  
header values.

//Stefan

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 28 Mar 2008 12:40:06 +0100

> character set names and conversion routines.

No, by design MIME works if the other side has no clue what it is.

It would then see gibberish like =?us-ascii*en-GB?Q?hello_world?=
and don't know that this an odd way to say "hello world".  It is
not forced to know what "encoded words" are, and if it knows this
it is not forced to support each and every charset.
 
> may I suggest that it be recommended to _only_ designate the 
> "utf-8", "iso-8859-1" and "us-ascii" character set in RFC2047
> encodings in HTTP?

It's an idea, but I think we're not forced to explicitly say this.

If some communities use koi8-r or some older charset popular in
JP this is the same issue as in ordinary Web browsers or e-mail:

I cannot read Cyrl or Jpan scripts, it is irrelevant from my POV
how that is encoded.  

 Frank

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 14:05:18 +0000

> it is not forced to support each and every charset.

You are sort of making my point for me.

If it's acceptable that a receiver sees gibberish text when it doesn't
understand the particular encoding, and it's acceptable that it
doesn't support each and every charset....

Is there a problem with transmitting binary UTF-8?  It's just an "odd
way to say" some i18n text.  Some receivers will decode it as
intended; some will show gibberish.  How is that different from your
example?

> I cannot read Cyrl or Jpan scripts, it is irrelevant from my POV

I thought the IETF was moving to recommend UTF-8 wherever possible
nowadays?

(Though I gather some people are still unhappy with Unicode, since it
doesn't distinguish some ideographs which are drawn differently in
different languages, and thus prefer not to use UTF-8.)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 14:17:18 +0000

...
> And everyone will keep their fingers crossed  

I'm thinking the same applies to RFC2047, if that becomes actually
implemented in practice (it currently isn't).

Surely the security issues with RFC2047 decoding among different
implementations are _much_ more likely than those of binary UTF-8?

E.g. =?iso-8859-1?q?=00?= will be a string terminator for some
components of some implementations, rejected by others, and passed
through as ASCII by many.

Expect "security filtering" to have opinions about such sequences for
good reasons, as soon as HTTP recipients start decoding RFC2047 in the
headers and reacting badly to such sequences.

UTF-8 has similar issues, but they are relatively well defined.  With
RFC2047, it's more open ended.

-- Jamie

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 28 Mar 2008 16:09:10 +0100

> implementations are _much_ more likely than those of binary UTF-8?

I totally agree with you here. We just seem to differ on the question  
if it is worthwhile to introduce UTF-8 headers or go back to ASCII  
and go the AtomPub "Slug" header way.

//Stefan

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 28 Mar 2008 15:27:35 +0000

> and go the AtomPub "Slug" header way.

I see the value of "slug" encoding for the AtomPub Slug header.
Percent encoded UTF-8 is useful there, because the string is meant to
be spliced directly into URIs, which use the same encoding.

However, the TEXT in HTTP headers is not meant to be used that way,
nor in any other way by the protocol.

I don't have a problem with percent encoded UTF-8, but I also don't
see any point in it, unless we decide that all HTTP header values must
be easily transcribed (same constraint as URIs), in which case
ISO-8859-1 in header values should be dropped at the same time.

-- Jamie

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 28 Mar 2008 16:33:01 +0100

> You are sort of making my point for me.

Maybe :-)  But I'd say ASCII gibberish =?...?.?...?= is one thing,
while a bunch of unidentified non-ASCII octets in header fields
are another story.  For e-mail, I know almost nothing about HTTP
apart from "please fix the syntax in RFCs 2616 and 2617".

> Is there a problem with transmitting binary UTF-8?  It's just an
> "odd way to say" some i18n text.  Some receivers will decode it
> as intended; some will show gibberish.  How is that different
> from your example?

RFC 2616 unfortunately says that it is Latin-1.  Jumping from
ASCII to UTF-8 could work if existing implementations have no
issues with non-ASCII octets.   Jumping from Latin-1 to UTF-8
or maybe not (legacy) isn't secure, implementations won't know
what it really is, UTF-8, Latin-1, windows-1252, they'd pass it
on with "guess" to applications, and I'm not confident that it
cannot cause havoc.  

For ASCII gibberish with odd =?...?.?...?= words I'm ready to
bet on "no problem", but random octets go against my instincts.
That is why I proposed to exclude 0x80..0x9F from Mark's 2(b).

> I thought the IETF was moving to recommend UTF-8 wherever 
> possible nowadays?

At least since RFC 2277 over ten years ago.  The last addition
to the club was RFC 5198 (net-utf8) yesterday.  With roots in
RFC 2068 HTTP is a "grandfathered" case, like mail and NetNews
wrt US-ASCII message headers.

Tons of legacy software where nobody can prove a negative in
the sense of "just send 8, no known problems" are tricky.
  
If we'd work on HTTP/1.2 proposing to ignore RFC 5198 would
be madness, but we are supposed to improve the HTTP/1.1 spec.

 Frank

---

From: "Jamie Lokier" <[email-masked]>
Date: Fri, 28 Mar 2008 12:42:14 -0700

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 29 Mar 2008 02:09:39 +0000

> That is why I proposed to exclude 0x80..0x9F from Mark's 2(b).

I see why your instincts go that way, but I think it's a mistake to
assume "no problem".  Those sequences, when decoded differently by
different applications, can hide a lot of nasty things.

You said about passing strings labelled "guess" to applications.

Another comment said that HTTP implementations don't have to handle
every character set they receive in =?...?=.  If they don't understand
one, they simply leave the encoding as-is.

What do you think an HTTP client/server is going to do when it
receives a string containing =?utf8?q?...?= _and_ =?koi8-r?q?....?=,
and the HTTP module doesn't handle koi8-r, and passes the string on to
the application?  How will it label that?  Is there a danger of
"double-decoding"?  (Answer: yes).  Is double-decoding a security
issue?  (Answer: yes, if any if these text strings cause security
issues.)  What about =?iso-8859-1?q?=00?=, =?iso-8859-1?q?=0a?=,
=?obscure?=00?=, other sequences which hide significant characters,
and also sequences which hide ordinary text which various code is
searching for (e.g. in User-Agent), and which different parses decode
differently?  These are analogous to the URL security bugs which
viruses took advantage of a few years ago.

It's only because nobody decodes =?...?= at the moment (in HTTP) that
their existence isn't a problem.  If the recommendation to use them
for i18n text is taken seriously, and they are decoded, there are
ambiguity (and thus potential security) issues.  This is amplified by
different implementations decoding =?...?= strings differently, which
is guaranteed due to the open-ended character set names.

At least UTF-8, or "UTF-8 with ISO-8859-1 fallback for non-parseable
sequences", or indeed _anything_ :-) does not have as much scope for
different implementations to decode the text differently.

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 28 Mar 2008 21:54:30 -0700

> different applications, can hide a lot of nasty things.

Only if those applications are ridiculously broken already, such
as by parsing the message after transcoding the encoded characters.
We don't need to care if that ship sinks.

Please, unless folks have specific implementations that have not yet
been discussed, there is no need to respond to every message as if
the weight of opinions were measured in words.  Speculation doesn't
mean diddly when we are talking about an old protocol that already
has a thousand independent implementations.  A change like this one,
that actually impacts on-the-wire parsing of compliant messages,
should be proven in the field before it is even considered for
the spec.  So, while I don't anticipate any problems with even raw
UTF-8 being placed in TEXT areas, the fact of the matter is that
the specification cannot be called HTTP/1.1 unless there are at least
two clients and two servers implemented to make use of it and NO
known breakage with previously interoperable implementations.

I'd rather defer this issue until much later in the process,
namely after the generic message parsing algorithm has been
written properly so that it will be clear that no compliant
implementation needs to care about the field-content in general
beyond it being defined (and parsed) in terms of ASCII delimiters.
If that can be shown to be implemented and interoperable, then we
need only define the parsing of individual field *content* within
those field definitions (which may or may not be within 2616bis).

....Roy

---

From: Robert Sayre <[email-masked]>
Date: Sat, 29 Mar 2008 02:46:53 -0400

> beyond it being defined (and parsed) in terms of ASCII delimiters.

I think deferral is the right way to proceed.

- Rob

---

From: Stefan Eissing <[email-masked]>
Date: Sat, 29 Mar 2008 11:35:57 +0100

+1

---

From: Julian Reschke <[email-masked]>
Date: Sat, 29 Mar 2008 13:04:53 +0100

>> I think deferral is the right way to proceed.

I do agree that if we can't make progress, we should defer. That being 
said, I think the discussion so far was very useful, if only in helping 
to understand the problem.

Also, this issue affects another issue I'd like to really make progress 
on (the BNF->ABNF switch). Is there a chance that we can decide on which 
headers currently *do* allow RFC2047 notation?

Two more thoughts:

1) Do we have *any* evidence of HTTP servers actually using RFC2047 
encoding, or clients being able to decipher it?

2) I'll also have to point out that given the state of things with HTTP 
header I18N we observe, the WebDAV WG's decision to put metadata into 
XML was absolutely a good one.

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 30 Mar 2008 00:36:24 +0100

> maybe occasionally in some corner there is a tiny bit of iso-8859-1.

The only iso-8859-1 use that is interop dependent is in authentication I
think. But there MAY Be some interop charset dependencies in cookies as
well..

For pretty much any other header defined in RFC2616 or the common
extensions the charset used for the header is not really an issue as the
use is either constrained to US-ASCII, or descriptive text that no one
we know of looks at or displays other than for diagnostic purposes.
Parsing is generally done charset neutral based on "8-bit US-ASCII octet
strings", at least in the environments using US-ASCII as the base...

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 31 Mar 2008 19:30:17 +0200

> be madness, but we are supposed to improve the HTTP/1.1 spec.

Actually there is no real difference here if we worked on HTTP/1.2. In
HTTP headers is defined to have the same meaning for as long as the
major version number is the same. If changing TEXT from ISO-8859-1 to
UTF-8 is a problem for HTTP/1.1, it's likewise a problem for HTTP/1.2 as
HTTP/1.2 still needs to deal with how the message is understood if
downgraded to earlier protocol versions by an intermediary.

My gut feeling is that the best long term move would be to move to UTF-8
and forget about 2047 and accept that some existing things MAY break,
BUT as you say it can not be proved to be completely without problems
for existing implementations. In fact it's very likely to cause problems
in some areas:
  - Authentication (RFC2617, here ISO-8859-1 is actively supported
today, but not really sufficient)
  - Cookie, for applications using/setting cookies in both client and
server contexts (not just echoing what you got)

The business of relying on RFC2047 encoding or similar "obfuscation" is
quite likely to get more bad implementations than UTF-8, and the risk of
security implications at the protocol level due to mismatches between
ISO-8859-1 / UTF-8 expectations is pretty minimal.

Related to Cookie it may be worth mentioning that RFC2965
(Cookie/Set-Cookie2) defines that the human visible attribute (Comment)
must have it's value encoded in UTF-8, within HTTP...

For now I think the only possible outcome is to keep what we have;
ISO-8859-1 as default, but clarifying that intermediaries should handle
them as 8-bit ASCII strings and consider the C0 set (0x80-0x9F) as just
another set of octets of the string (not as control characters or
invalid) and a note that future headers MAY be seen using UTF-8
encoding.

Switching to UTF-8 in general is possible, but may require a new header
declaring that this message is sent using UTF-8 and is outside the scope
of RFC2616bis until there is a strong requirement to address I18N to
advance as standard. But switching to UTF-8 is imho most likely the most
sane way of addressing I18N in both 2616 and 2617, and least likely of
causing long-term interop issues.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 31 Mar 2008 19:51:19 +0200

> headers currently *do* allow RFC2047 notation?

Difficult. No known implementations, and specifications ambigious on
when 2047 should be used, such as Warning saying 2047 should be used but
defining a syntax where 2047 says itself that it MUST NOT be used...

> encoding, or clients being able to decipher it?

I am not aware of any, but then I mostly operate in a context where
ISO-8859-1 makes sense...

But at least IE6 has optional support for sending URLs using raw UTF-8,
and it do send raw UTF-8 in the Host header in such setups..

Regards
Henrik

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 31 Mar 2008 20:12:07 +0200

Henrik Nordstrom wrote:

> My gut feeling is that the best long term move would be to move to
> UTF-8 and forget about 2047 and accept that some existing things
> MAY break, BUT as you say it can not be proved to be completely
> without problems for existing implementations. In fact it's very
> likely to cause problems in some areas:

> today, but not really sufficient)

ACK, in theory that could be fixed by adopting RFC 2831 or 2831bis
magic.  I'm not exactly sure about Unicode 3.2 SASLPREP in 2831bis,
maybe RFC 5198 NFC minus anything declared to be bad in 3987bis is
good enough for a 2617bis.

> - Cookie, for applications using/setting cookies in both client
> and server contexts (not just echoing what you got)

I cannot judge cookies.  I'm happy when I find the way to disable
double-analytics-tracker cookies from 3rd parties, for undisclosed
reasons FF2 makes that more difficult than IE6, but it is possible.

> ISO-8859-1 as default, but clarifying that intermediaries should
> handle them as 8-bit ASCII strings and consider the
C1
> set (0x80-0x9F) as just another set of octets of the string (not
> as control characters or invalid) and a note that future headers
> MAY be seen using UTF-8 encoding.

Point - I did not consider that when I proposed to exclude C1 from
Mark's 2B proposal.  Oddly we all agree that we want UTF-8 "later",
but have different ideas how to get there.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Mar 2008 20:10:12 +0200

> and it do send raw UTF-8 in the Host header in such setups..

Really?

I haven't seen that, at least not in the URI itself.

BR, Julian

---

From: David Morris <[email-masked]>
Date: Mon, 31 Mar 2008 10:21:32 -0800 (PST)

> I haven't seen that, at least not in the URI itself.

I can't certify it works, but
      Tools->Internet Options->Advanced
first option under "Browsing" is:
    Always send URLs as UTF-8 (requires restart)

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 31 Mar 2008 11:28:46 -0700

> and it do send raw UTF-8 in the Host header in such setups..

Whoa, that will open up a new can of security worms.
How is the option enabled?

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Mar 2008 21:10:23 +0200

>     Always send URLs as UTF-8 (requires restart)

As far as I remember, this means "encode non-ASCII characters in UTF-8, 
then percent-escape".

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 31 Mar 2008 21:29:57 +0200

> > and it do send raw UTF-8 in the Host header in such setups..

Yes..

> I haven't seen that, at least not in the URI itself.

Internet Options -> Advanced -> Web searches -> Always send URLs using
UTF-8  (or something like that.., my only IE6 is in Swedish)

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 31 Mar 2008 22:25:46 +0200

> As far as I remember, this means "encode non-ASCII characters in UTF-8, 

In my tests the request-URI is percent-escaped, but Host header sent as
raw UTF-8.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Mar 2008 22:33:08 +0200

> In my tests the request-URI is percent-escaped, but Host header sent as

OK.

I guess the latter can be considered a bug. Eric?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 31 Mar 2008 23:22:58 +0200

Julian Reschke wrote:
 
>> In my tests the request-URI is percent-escaped, but Host
>> header sent as raw UTF-8.
 
> OK. I guess the latter can be considered a bug.

FWIW the ICANN IDN Wiki has a link to an add-on allegedly
supporting IDN also for IE6.  I didn't test how good this
works (or if at all, but I've no reason to doubt that it
does "something"):

http://idn.icann.org/IDN-aware_software#Internet_Explorer

 Frank
-- 
Unwilling to decide 2616bis issues based on IE6 oddities.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 01 Apr 2008 14:15:52 +0900

>Whoa, that will open up a new can of security worms.

Do you mean due to the nature of UTF-8, or due to
implementations that didn't do enough defensive programming?

Some of this has been around for quite a while.
With security, there is no 100%, but the chances are that
potential security holes have already received some
scrutinity.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 01 Apr 2008 12:54:57 +0900

At 05:33 08/04/01, Julian Reschke wrote:
>
>Henrik Nordstrom wrote:
>> mテ・n 2008-03-31 klockan 21:10 +0200 skrev Julian Reschke:
>> 
>>> As far as I remember, this means "encode non-ASCII characters in UTF-8, then percent-escape".

Here is some background, as far as I understand. In the early days
of IDNs (before there was a WG), Microsoft had a proposal for using
raw UTF-8 for IDNs in the DNS. Some of that got implemented in
their servers and clients, and this is probably a leftover.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 01 Apr 2008 16:05:09 +0900

>think. But there MAY Be some interop charset dependencies in cookies as

My personal guess would be that cookies as implemented are essentially
bytes-in-bytes-out, i.e. what's sent from the server is sent back
again by the client.

Regards,    Martin.

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 01 Apr 2008 10:20:28 +0200

> bytes-in-bytes-out, i.e. what's sent from the server is sent back

Yes, which is why I said there is likely to be interop issues when the
cookies is accessed by both sides, not when it's simply echoed.

i.e. javascript/flash/whatever accessing/setting/modifying cookies also
used by the server.

But to be honest I kind of suspect such interop issues is already there.
The HTML side (and javascript running in that context) has already had
it's share of I18N support using various encodings natively without
strange encoding/recoding rules, and there most likely is some
implementations not constraining what javascript requests from http to
what http technically allows..

Regards
Henrik