Subject: Re: Straw-man charter for http-bis

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Aug 2007 13:39:55 +1000

http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i74

On 10/06/2007, at 6:05 PM, Martin Duerst wrote:

> - RFC 2616 prescribes that headers containing non-ASCII have to use
>   either iso-8859-1 or RFC 2047. This is unnecessarily complex and
>   not necessarily followed. At the least, new extensions should be
>   allowed to specify that UTF-8 is used.

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Aug 2007 13:40:36 +1000

>   allowed to specify that UTF-8 is used.

My .02;

I'm concerned about allowing UTF-8; it may break existing  
implementations.

I'd like to see the text just require that the actual character set  
be 8859-1, but to allow individual extensions to nominate encodings  
*like* 2047,without being restricted to it. For example, the encoding  
specified in 3987 is appropriate for URIs. However, it *has* to be  
explicit; I've heard some people read this requirement and think that  
they need to check *every* header for 2047 encoding.

So, I think this means;

1) Change
   "Words of *TEXT MAY contain characters from character sets other  
than ISO-8859-1 [22] only when encoded according to the rules of RFC  
2047 [14]."
to
   "Words of *TEXT MUST NOT contain characters from character sets  
other than ISO-885901 [22]."
and,

2) Identify headers that may have non-8859 content and explicitly say  
how to encode them (IRI, 2047, whatever; the existing ones will have  
to be 2047, I believe), modifying their BNF to suit.

3) When we document extensibility, require new headers to nominate  
any encoding explicitly.

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Aug 2007 02:56:54 -0400

> I'm concerned about allowing UTF-8; it may break existing
concur.  though at least it is possible to distinguish utf-8 from 8859-1. 

also, I'll note that supporting utf-8 in a way that is backward
compatible with existing implementations is almost certainly more
complex (and thus more costly, error-prone, etc) than supporting rfc 2047.
>
> I'd like to see the text just require that the actual character set be
> 8859-1, but to allow individual extensions to nominate encodings
2047 was specifically not intended for use with protocol elements that
have meaning to protocol engines.  how many HTTP headers contain text
that is intended solely for human use?

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 20 Aug 2007 08:54:46 +0100

John C Klensin said:
> If we are going to consider changes in this area, let's make
> them improvements.  Locking in 8859-1 is not an improvement: it
> would, IMO, be better to deprecate its use

+1

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Martin Duerst <[email-masked]>
Date: Mon, 20 Aug 2007 16:54:20 +0900

Hello Mark,

Thanks for giving this an issue number.

At 12:40 07/08/20, Mark Nottingham wrote:
>*like* 2047,without being restricted to it.

What do you mean by "encodings *like* 2047"? And why do you think
that UTF-8 may break existing implementations? UTF-8 has virtually
the same footprint in terms of bytes as ISO-8859-1: All bytes
above 0x7F may be used. Implementations that have to deal with
ISO-8859-1 usually do this by just being 8-bit-transparent;
that works for UTF-8, too.

If your opinion is that UTF-8 cannot be allowed at all, then
that's going to be a problem for cases where it's already in
use, see e.g. earlier posts in the http list.

It's easy to say "may break existing implementations",
but in over 10 years of being involved in the Web, I haven't
heard about that happening. If you or anybody have, please
speak up.

>For example, the encoding  
>specified in 3987 is appropriate for URIs.

As one of the authors of RFC 3987, I know what you mean, but
"the encoding specified in 3987" wouldn't be enough for a spec.
Also, it's not really very well suited to the job, because
%hh-encoding is used to escape any bytes, not only UTF-8,
and there is a considerably length increase for some scripts.

>However, it *has* to be  

I have read it that way, too. If it can be safely argued that
it was never intended that way, and that no harm is produced
if this is restricted, then I'd befine with restricting it,
because checking everything for 2047 is indeed tough, but
I'd really like to make sure that this isn't creating problems.
(i.e. that a careful examination of the various headers makes
some reasonably conservative assumptions).

>2) Identify headers that may have non-8859

There are many parts to ISO-8859, not just ISO-8859-1.

>content and explicitly say  

If that includes UTF-8, I'd be fine with it. If it excludes
UTF-8, I think that would be a problem.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Mon, 20 Aug 2007 17:10:27 +0900

>concur.  though at least it is possible to distinguish utf-8 from 8859-1. 

In practice indeed this can be done with high reliability; please
see http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf
for details. For iso-8859-1, see in particular p. 21.

>complex (and thus more costly, error-prone, etc) than supporting rfc 2047.

Well, if "backwards compatible" means also supporting RFC 2047,
then that's a tautology. If the choice is between UTF-8 and RFC 2047,
however, then I'd take UTF-8 any time, because RFC 2047 includes
UTF-8 as well as many other encodings.

Regards,    Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Aug 2007 18:55:10 +1000

The (potential) problem is that an intermediary (for example) needs  
to be able to handle headers that it doesn't understand. If it's been  
built to store headers as iso-8859-1 strings as they pass through (a  
reasonable assumption, considering 2616), an unknown header with  
another encoding -- no matter how specified or flagged -- may break it.

So, going forward, I completely agree with you, but in the case of  
HTTP, I think the horse has already bolted; it is effectively fixed  
to 8859-1, and we can't fix this in the right way without versioning  
the protocol.

Or am I missing something?

On 20/08/2007, at 5:22 PM, John C Klensin wrote:

> Sigh.  My own sense is that, going forward, we need to lose
> 8859-N, not make it the default (or only) character set for more
> protocols.  It is, to put it mildly, a little Euro-centric (and
> not even completely suitable for Europe).  Much of the advantage
> of Unicode is that one does not need to designate/ nominate a
> particular CCS or encoding and then maintain state for it... and
> that is a fairly large advantage.  See also
> draft-klensin-unicode-escapes-03.txt(probably expired, but you
> should be able to find a copy somewhere -- I'll get back to it
> sometime soon) for a discussion of issues in ASCII encoding of
> multioctet character sets.   The IRI spec may constrain things
> to encoding of octets, but that doesn't make it a good idea.
>
> would, IMO, be better to deprecate its use and require explicit
> charset designation always if that is the only choice.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 20 Aug 2007 18:18:30 +0900

>another encoding -- no matter how specified or flagged -- may break it.

I think you present a valid scenario. However, storing headers as
iso-8859-1 essentially means storing (and resending) them as bytes.
If such an implementation gets UTF-8, it will just store and
resend that as iso-8859-1, which means store and resend as bytes,
which, from the viewpoint of that implementation, will be GIGO,
but overall, will not cause any damage.

Regards,    Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Aug 2007 20:03:26 +1000

On 20/08/2007, at 5:54 PM, Martin Duerst wrote:
> that UTF-8 may break existing implementations?

See previous message.

> UTF-8 has virtually

If utf-8 is a subset of iso-8859-1, it would work; but I don't think  
that's the case (not that I'm an expert in this area, by any means).

> use, see e.g. earlier posts in the http list.

Sorry, I looked and didn't find anything. Can you give references?

> heard about that happening. If you or anybody have, please

I'm wary of changing fundamental things in HTTP like the charset for  
headers because nobody's heard of it breaking things. My experience  
has been that the people who show up in this forum and the others  
that we circulate in are the tip of a much larger iceberg of  
implementations, and we make such changes at our peril.

In particular, performance-sensitive implementations like proxies  
often rely on particular data structures that are mandated by the  
specs for optimisation. As you point out, they'll probably store them  
as bytes, but they may not. Client and server APIs have been built  
that make (again, reasonable) assumptions about headers' character sets.

In both cases, changing the spec may break those implementations.  
Yes, it's just "may", but the burden of proof should be on those who  
want to make the change. To that end, if UTF-8 were widely used in  
HTTP headers already, I wouldn't be so uncomfortable, but AFAIK  
they're few and far between.

> and there is a considerably length increase for some scripts.

I know. I didn't say it was a great solution, just one that works.

> some reasonably conservative assumptions).

I think that's a great first step.

Taking a glance at the header registry <http://www.iana.org/ 
assignments/message-headers/perm-headers.html>, there are about 30  
headers that might conceivably need non-ASCII content. Of those, the  
vast majority have URIs as a payload, which means that if you want to  
change the encoding, you'll need to change their syntax pretty  
fundamentally -- an incompatible change.

Of the remaining ones, I see (with ones that may IMO be a problem  
marked with a *):
   * WWW-Authenticate, Proxy-Authenticate -- as discussed elsewhere
   * Authorization, Proxy-Authorization -- as discussed elsewhere
   * Content-Disposition -- can contain a filename
   - Cookie, Cookie2, Set-Cookie -- can contain user data, but it's  
opaque, so servers can choose their own encoding strategy
   - Server, Via, User-Agent -- not usually presented to users
   - Warning - can contain warning text, not usually presented to users
   - From -- an e-mail address (and possibly a candidate for  
deprecation)

My earlier proposal would have the effect of clarifying the use of  
RFC2047 encoding in those marked with "*" above. It sounds like you  
want to change the syntax of those headers to allow UTF-8, no?

If so, the problem that I have with that is that no existing  
conformance-minded implementer could have looked at RFC2616 and said  
to themselves, "oh, I should use/expect UTF-8 there," while such a  
statement could quite easily have been made about RFC2047. Now, if  
everybody has gone off and used UTF-8 despite the RFC, I could see an  
argument for changing the spec to match reality, but I don't think  
that's the case here.

If there were a huge win here -- e.g., lots of improvements in i18n  
and efficiency -- again, it might be persuasive. But, looking at the  
list above, the benefits of such a change don't seem overwhelming.

Again, I'm not an i18n expert, so please educate me.

> There are many parts to ISO-8859, not just ISO-8859-1.

Sorry, just using shorthand.

> UTF-8, I think that would be a problem.

In my view, the proxy case and the API case above both rule that out;  
we can't assume that deployed software is set up to produce, accept  
or process UTF-8.

> Regards,    Martin.

Cheers,

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 20 Aug 2007 11:24:06 +0100

> that's the case (not that I'm an expert in this area, by any means).

It's not.

Printable text in ISO-8859-n (for all n) consists of a sequence of
characters, each of which is either:

    one octet in the range 20 to 7E
    one octet in the range A0 to FF

Printable text in UTF-8 consists of a sequence of characters, each of which
is either:

    one octet in the range 20 to 7E
    one octet in the range C2 to E4 followed by between 1 and 3 octets
              in the range 80 to BF (the first octet tells you how many [*])

In both cases, 20 to 7E are the ASCII characters. In both cases, codes like
09 (HTAB) and 0A (LF) have the same meaning. In ISO-8859-n the meaning of
codes A0 to FF depends on the value of n. In UTF-8 each sequence has a
unique meaning that never changes.

The syntax in 2616 allows any octet in the range 20 to FF except 7F; both
of these are subsets of that.

(*) To be precise:
     one octet C2 to DF followed by one   octet  in the range 80 to BF, or
     one octet E0 to E4 followed by two   octets in the range 80 to BF, or
     one octet F0 to F7 followed by three octets in the range 80 to BF.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Mon, 20 Aug 2007 10:20:14 +0100

> another encoding -- no matter how specified or flagged -- may break it.

On the other hand, the *syntax* allows any valid UTF-8 sequence, since it
doesn't forbid the octets %x80-9F. So it's unlikely that anything will
break unless someone is being very strict in their checking.

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Aug 2007 09:03:51 -0400

> Well, if "backwards compatible" means also supporting RFC 2047,
of course it does mean that, as you're not going to get rid of the need
to interoperate with the installed base of clients and servers anytime soon.
>  If the choice is between UTF-8 and RFC 2047,
if we had the luxury of starting from scratch, I'd agree with you.

though my (fuzzy) memory seems to say that at the time HTTP was
standardized lots of people insisted on using 8859-1 rather than either
ASCII or any form of Unicode - again for backward compatibility reasons.

but how much of a problem is this really?

---

From: Stefanos Harhalakis <[email-masked]>
Date: Mon, 20 Aug 2007 16:52:26 +0300

> but overall, will not cause any damage.

My 2c:

  UTF-8 introduces a requirement that ISO8859-X encodings don't have. UTF-8 
strings may be invalid, in which case a proper action may be needed (drop ?). 
Thus, all UTF-8 strings need to be validated.

  Apart from that, implementations may do various tricks like logging etc, 
where:
a) strlen() is used - not unicode aware
b) iconv() is used to convert ISO8859-1 to UTF-8 either for presentation or 
for internal storage (python or java perhaps?)

---

From: John C Klensin <[email-masked]>
Date: Mon, 20 Aug 2007 07:22:30 +0000

--On Monday, 20 August, 2007 13:40 +1000 Mark Nottingham
<[email-masked]> wrote:

>> - RFC 2616 prescribes that headers containing non-ASCII have
>> to use either iso-8859-1 or RFC 2047. This is unnecessarily
>>   complex and not necessarily followed. At the least, new
>>   extensions should be allowed to specify that UTF-8 is used.
> 
> My .02;
> 

And whatever is done about it should be consistent with the EAI
work.  Otherwise, we are likely to find ourselves in big trouble
going down the line.

> I'd like to see the text just require that the actual
> character set be 8859-1, but to allow individual extensions to
> nominate encodings *like* 2047,without being restricted to it.
> For example, the encoding specified in 3987 is appropriate for
> URIs. However, it *has* to be explicit; I've heard some people
> read this requirement and think that they need to check
> *every* header for 2047 encoding.

     john

---

From: der Mouse <[email-masked]>
Date: Mon, 20 Aug 2007 14:37:13 +0000

> iso-8859-1 essentially means storing (and resending) them as bytes.

Depends on how much checking is done.  The C0 and C1 ranges are not
valid 8859-x text (except for a few codes in C0, like HT), but, as
Clive points out, C1 does, in general, occur in UTF-8-encoded text.

I recognize there's a "who would bother to check" tendency.  While I
share it, I also believe the number of distinct implementations out
there is large enough that anything permitted by the spec has probably
been done (and, of course, a great many things not permitted by the
spec, but I see no reason to care about compatability with them).  In
particular, any implementation whose native text encoding is not 8859-1
may be recoding headers into its native encoding for storage and back
again on output, and that is almost certain to corrupt C1 octets.

The only fix I can see for that is to do something like UTF-8, but
tweaked to keep all octets in the ISO-8859-x printable space.  I've ben
unable to come up with a way of doing this by just changing the fixed
bits in UTF-8; it seems to me to require putting only five (rather than
six) bits of data in the second and later octets.  (I suspect this
wouldn't fly, simply because UTF-8 is too entrenched, but it's the only
way I can see to be strictly compatible.  It also has the disadvantage
that part of the BMP needs four octets rather than the three that UTF-8
needs.)

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Aug 2007 15:29:46 -0400

> again on output, and that is almost certain to corrupt C1 octets.
I suspect that the problem is not so much transparency, as
presentation.  The larger set of things broken by allowing utf-8 in
existing header fields (and to a lesser extent new fields) will not be
things that forbid C1 octet values, but rather things that try to
display those fields as if they were 8859/1.  Translation of the
presumed 8859/1 into other charsets is another version of the same problem.

---

From: Keith Moore <[email-masked]>
Date: Tue, 21 Aug 2007 12:51:01 -0400

> Thus, all UTF-8 strings need to be validated.
no.  the last thing we need in HTTP (or any protocol IMHO) is for
intermediaries to try to be smarter than their endpoints.
> a) strlen() is used - not unicode aware
strlen works the same for utf-8 as for ascii, as long as what you care
about is number of bytes in the string rather than, say, the amount of
space it will take up when displayed.
> for internal storage (python or java perhaps?)
valid point.

Keith