Subject: Character Set Encoding issues [Was Re: I-D ACTION:draft-ietf-cnrp-03.txt]

From: Leslie Daigle <[email-masked]>
Date: Wed, 5 Jul 2000 21:32:45 -0400

Howdy,

The subject of CNRP's requirement of UTF-8 and XML, raised by
James Seng and followed on by Martin Duerst is the one area that I'm
not sure we've found resolution/alternate text for.

[Martin J. Duerst wrote:]
> What you use above is called numeric character references.
> Correctly, it's &#xHHHH;, not &#0xHHHH;
>
> It is true that this is standard XML syntax, so every XML
> parser has to understand it, and I would consider any spec
> that ruled out numeric character references (which it looks
> like the CNRP spec might do) as clearly suboptimal.

Does anyone else have a comment as to whether we can/should include
support for these encodings?

[James Seng wrote:]
> But whatever the case, I find it seriously lacking information on I18N
> issues.
> Having it just as 'just-send-UTF-8' is definately not good enough...

James, it is true that the document punts on the issues of
I18N -- a lot of those issues are meant to be handled by specific
servers/services (cf. the thread from last year re. doing
locale-specific matching) and perhaps ultimately applications
of CNRP.  So, I think we need more specific pointers to areas that you
suggest need to be covered (even with that deliberate distancing),
and/or proposed text amendments.  Or maybe we need to be more explicit
that we are punting on the issues, and why?

Thanks,
Leslie.

--

-------------------------------------------------------------------
"My body obeys Aristotelian laws of physics."
   -- ThinkingCat

Leslie Daigle
[email-masked]
-------------------------------------------------------------------

---

From: "Martin J. Duerst" <[email-masked]>
Date: Thu, 6 Jul 2000 12:32:40 +0900

>support for these encodings?

It's not a question of whether you include them, but whether
you exclude (or don't exclude) them. If you use XML, then you
support them. They are part of XML in the same way that you
can use both single quotes and double quotes to delimit attribute
values. If you use an XML parser, you will have to work hard
to try an tweak it into not accepting these.

A few more comments on section 3.4:

 > 3.4 Character sets

Better call that 'character encoding'. There is no notion of
'set' involved anywhere. Change that throughout.

 >    The character set encoding of a common names is always UTF-8.

Better change to 'character encoding'. Also, saying the encoding
is always UTF-8 is weird. Can a common name be on a billboard?
What's the encoding then?

 >    Specifically:
 >    o  Within an XML query or response, the common name is represented
 >       in UTF-8,

Either your XML is in UTF-8, and then this statement is completely
unnecessary and confusing the way it's written, or your XML may
come across in other encodings, and then it would be totally
weird to do such a thing (XML has one encoding per external entity).

 > WITHOUT any hex-encodings

See above. Numeric character references (hex or dec) are on a different
level than the actual binary character encoding.

 >    o  Within a CNRP URI, the REPRESENTATION of the common name is
 >       hex-encoding of the UTF-8 common name -- because this is an issue
 >       for the CNRP URI, not the common name.
 >    o  Similarly, in other environments (e.g., particular display
 >       devices) the common name may be _presented_ in a different
 >       character-set, but that is a display/data entry issue. It is not
 >       a "common name" unless it's encoded in UTF-8.

Does that make it impossible to run a common name database
based on UTF-16?

Regards,   Martin.

---

From: James Seng <[email-masked]>
Date: Fri, 7 Jul 2000 11:23:06 +0800

> support for these encodings?

I strongly believe we should. But it is a tradeoff in complexity.

> that we are punting on the issues, and why?

Using UTF-8 for I18N is straight forward. Using UTF-8 for L10N is not,
as anyone who has tried to do L10N will tell you.

Using native encoding is bad idea from I18N but pretending it does not
exist is worst. I believe there must be explicit rules to state either

1. native encoding is accepted but how is it handled, e.g. having a new
   property specifying the charset; or

2. native encoding MUST NOT be used.

I shall not go into canonicalization and matching since I remember this
is one of the reasons for service differential.

It is possible that someone may wish to identify the language been used.
The I-D suggested using property "language" but why not use Unicode
language tag (yes, it is on Plane 14)?

And why should UTF-8 be hexencoded in URI but not allowed in the CNRP
scheme?

ps: Okay, some of these questions are dumb. So shoot me :-)

-James Seng

---

From: Leslie Daigle <[email-masked]>
Date: Fri, 7 Jul 2000 12:44:29 -0400

James,

> as anyone who has tried to do L10N will tell you.

Okay, is your point that UTF-8 isn't useful when you try to do
cross-locale (or "locale-independent") queries, which is where
most of canonicalization/matching comes in, or is your point that
UTF-8 is really suboptimal for certain specific locales (i.e.,
if you're doing a locale-specific service, why use such a rotten
encoding)?

> 2. native encoding MUST NOT be used.

As it stands, _everything_ is meant to be in UTF-8, which means
that CNRP isn't a universal database query/response language.  We
don't support raw data for anything.  If we did, I would go
with proposal 1' -- which is say that any property (except
common name -- for reasons of universality) may have an "encoding"
attribute, which could include other character encodings, data
types (e.g., jpeg), etc.  But, I think that is making this into
more than it was intended for, so perhaps we need to go with 2
and be more explicit that everything is meant to be text and
must be encoded in UTF-8.  Period.

Which doesn't, by the way, preclude the reply including URLs to
data that is in other formats.

> language tag (yes, it is on Plane 14)?

The language tag is meant to be more largely about the body from
which the data is drawn.  I don't know enough about Unicode - how
would using its language tag affect the content of what we're
proposing now?

> And why should UTF-8 be hexencoded in URI but not allowed in the CNRP

Don't entirely know what you're asking here -- though I'll point
out that the URI is conforming to URI syntax, which is outside
of our control.  The restrictions we've placed to date are on the
content of CNRP queries and results expression, which is what we're
trying to standardize.

And, it's still not clear to me, per what Martin Duerst has said,
that the current formulation of the document does exclude
them, by virtue of using XML.

Leslie.
--

Leslie Daigle
[email-masked]

---

From: James Seng <[email-masked]>
Date: Sat, 8 Jul 2000 16:04:25 +0800

> if you're doing a locale-specific service, why use such a rotten

I think you misunderstood me. I did not say UTF-8 is a "rotten"
encoding. UTF-8 is a good encoding. HOWEVER, UTF-8 is currently useful
only in I18N. Move into localization, you will see the resistant to
shift to UTF-8 as people are more comfortable with their own industry
set standard encoding.

Hence, forcing UTF-8 down the throat of L10N is okay but not likely
practical.

> must be encoded in UTF-8.  Period.

This is what I mean. CNRP should consider the between using UTF-8 *ONLY*
(and thus slowing down the adoption) or be more practical. There is no
right answer but make a choice.

> would using its language tag affect the content of what we're

Using the language tags would seem more nature since the property is
embedded within the UTF-8 string itself.

-James Seng

---

From: Leslie Daigle <[email-masked]>
Date: Wed, 12 Jul 2000 17:34:52 -0400

Howdy,

> shift to UTF-8 as people are more comfortable with their own industry

I think I did understand your characterization of UTF-8, but was
trying to get at why you are warning off it for localization.  What
I hear is:  for people who are interested in running CNRP servers
only to serve their local needs, they are going to be resistive to
using UTF-8, and hence won't adopt CNRP.

That sounds like a fairly standard argument to which teh IETF
response has traditionally been along the lines of "we focus on
solving wholescale Internet problems" -- i.e., in favour of asserting
that we only use UTF-8.

But, if you have more concrete examples of existing organizations
that would otherwise be interested in CNRP but for which this is
a showstopper, I'd like to hear it -- here or to me directly.

> This is what I mean. CNRP should consider the between using UTF-8 *ONLY*

Which is what I think the documents already say?

> embedded within the UTF-8 string itself.

Sure -- but what in what we've said so far prevents that?

Leslie.

--

Leslie Daigle
[email-masked]

---

From: "Martin J. Duerst" <[email-masked]>
Date: Thu, 13 Jul 2000 17:26:21 +0900

Hello everybody,

I read more of the CNRP spec, and found several problems of
differing magnitude. Maybe some of them are already know,
but some may not yet be known:

- The DTD contains errors. It looks like it has never seen
   a validating XML processor, otherwise it wouldn't.

- The content model
   ((service*, resourcedescriptor*, referral*, error*) | service)
   given in a comment is erroneous (because it's nondeterministic).
   It looks like what you want may be something like
   (service?, (resourcedescriptor | referral | error)*)
   or something similar (please explain). In any case, there
   is never a need to use ANY,
   (service | resourcedescriptor | referral | error)*
   accepts any combination and number of the above four elements,
   is much more precise than ANY, and still wider than you want,
   so you can go with this if you don't find anythin better.

- The examples also contain errors. At least for the complete
   examples, validating them shouldn't be considered a luxury
   (saying at the beginning of the spec that it's the DTD, and
   not the examples, which are normative still makes sense).

- Saying that all CNs are in UTF-8 is the wrong way of saying
   what you want (as I said earlier). You'r using XML, and
   XML solves the problem of how to encode characters, so
   if you want something specific, say it in terms of XML.

- For the language tag examples, most should only be one level
   (en, fr, sp, instead of en-US, fr-FR,...). In my opinion,
   the country subdivision will rarely be very relevant
   (if the CN is colour, it seems anyway clear that it's
   British English and not US, and if the language applies
   to the resources at the other end, people don't really
   care that much in most cases (I think the biggest exception
   is Chinese)). Also, of course to explain fr-FR as
   'European French' is a bad inaccuracy because there is
   fr-BE and fr-CH (Belgian and Swiss).

- RFC 1766 is being upgraded, to add 3-letter codes, which
   will make it possible to address a lot more languages.
   Any idea how CNRP plans to address this?

- In 4.1.3, no explanation of e.g. the syntax of Range.

- Messup in 4.1.3.1, messup in items 3, 4, and 5.

- On several instances, CN is described as a property,
   although it's not. (e.g. 4.1.4.1).

- 4.1.4.1.1 (lot's of levels for such a short doc :-):
   How does one know which types are enumerated, and which
   not?

- 4.1.4.2.1: Here the reader has no idea what ID is for.

- 4.1.4.3 'two clusters': In the example, I see two servers,
   but not two clusters. Can anybody explain. In the same example,
   there is a ? at the end of the URI, but there is none in other
   cases. Does that have any significance? Also, does CNRP
   define any rules for what to do with whitespace in content?
   There are quite a few examples that contain whitespaces/
   line breaks in content, and XML doesn't just remove them
   (it removes them between tags where no content can appear).
   I think CNRP needs to say something here.

- Reference [1]: I don't see a chance to locate this reference.
   Please give more information.

- 6.2 'This result set includes a referral and a non-fatal error':
   I don't see the error.

- Quite a lot of typos, e.g. 'server.There', 'new objects definition',
   'custom properties call the base properties', 'It is up to the
   serviced to', 'value 2 .OR property', 'MUST always includes',
   'can contain the Schema objects',

So much for the moment.    Regards,    Martin.

---

From: Michael Mealling <[email-masked]>
Date: Thu, 13 Jul 2000 09:31:56 -0400

>    a validating XML processor, otherwise it wouldn't.

Its been going through some revisions lately so I haven't
validated it yet. I'm putting in lots 'o edits before Friday...

>    so you can go with this if you don't find anythin better.

It may be a problem with the XML documentation I'm reading but
does this allow for any ordering? Due to how the child elements
can be generated, they can actually appear random...

>    not the examples, which are normative still makes sense).

Yep. I'm validating all of the examples today....

>    if you want something specific, say it in terms of XML.

I think this might have been for the cases where a CN is
not contained in XML. I think its needed because this
document also talks about CNs in other places (the URL
for example).

>    fr-BE and fr-CH (Belgian and Swiss).

I'll leave this one to the group. We could just create types
for these others...

>    Any idea how CNRP plans to address this?

The type that references 1766 should probably say something
about it....

> - In 4.1.3, no explanation of e.g. the syntax of Range.

I noticed that ommission too. Fixing it today...

> - 4.1.4.2.1: Here the reader has no idea what ID is for.

Fixing it...

>    but not two clusters. Can anybody explain.

Bad language. I'll fix it..

>    In the same example,
>    cases. Does that have any significance?

The CNRP URl has one. I'll check on this one and correct it..

>    Also, does CNRP

Nope. That's up to the service to be smart about...

>    I think CNRP needs to say something here.

Good point. My guess is that if whitespace were important (say as
part of the CN) then the spaces would be encoded as a character
reference...

>    'can contain the Schema objects',

Thanks for the proofread. With three authors and writing an RFC in
XML with /bin/vi probably introduces a goodly number of errors on its own.

-MM

--
--------------------------------------------------------------------------------
Michael Mealling        |      Vote Libertarian!       | www.rwhois.net/michael
Sr. Research Engineer   |   www.ga.lp.org/gwinnett     | ICQ#:         14198821
Network Solutions       |          www.lp.org          |  [email-masked]

---

From: "Marshall L. Moseley" <[email-masked]>
Date: Thu, 13 Jul 2000 08:22:38 -0700

>    the country subdivision will rarely be very relevant

I think the country subdivision is *highly* relevant.  It enables geographic
scoping so that a CNRP query can return meaningfull results.  For example,
'Today's business news' will have very different meanings in Paris then it
will in Quebec.

--Marshall

----- Original Message -----
From: Michael Mealling <[email-masked]>
To: <[email-masked]>
Sent: Thursday, July 13, 2000 6:31 AM
Subject: Re: More comments on CNRP spec (was: Re: Character Set Encoding
issues)

---

From: Michael Mealling <[email-masked]>
Date: Thu, 13 Jul 2000 11:49:14 -0400

> 'Today's business news' will have very different meanings in Paris then it

Martin is talking about the Language property, not the Location property.

I think the country subdivision is important for the language tag (ask
Leslie or Nico about what 'french' is one day). But if someone
doesn't like it they can create a new property type. Heck I can do
that in this document if someone can give me the appropriate citation.

-MM

---

From: Michael Mealling <[email-masked]>
Date: Fri, 14 Jul 2000 10:18:37 -0400

>    so you can go with this if you don't find anythin better.

Can someone verify this for me. My XML book isn't clear and
all of its examples don't agree...

>    if you want something specific, say it in terms of XML.

Unless I hear some suggestion otherwise I'm going to leave this
in for now since I think its speaking to common-names when
they are outside of an XML document. I'll read it again and
see if it needs clarifying...

-MM

---

From: Leslie Daigle <[email-masked]>
Date: Fri, 14 Jul 2000 10:56:13 -0400

Howdy,

Just to underscore the call to eX(ML)sperts, what Martin has
written:

> >    (service?, (resourcedescriptor | referral | error)*)

is indeed what we're looking for.  However, as the note in the DTD
attempts to explain, it was our (collective) understanding that
XML formalism doesn't support this.

If we were wrong in this understanding, please straighten this out!

> see if it needs clarifying...

Again -- agreed.  We may need to clarify things so that it's clear
that the CommonName in basic form is required to be in UTF-8, although
local context (URIs, XML) may "adjust" the on-the-wire bytes.

The goal is:  if you find a common name lying on the street,
its bytes will be interpreted as UTF-8 (e.g., any %'s will be
interpreted as part of the name, not a %encoding in a URL).

Leslie.

--

Leslie Daigle
[email-masked]

---

From: Michael Mealling <[email-masked]>
Date: Fri, 14 Jul 2000 11:13:49 -0400

> XML formalism doesn't support this.

I did find another book that did agree with this. Apparently the
secret is that if (a | b | c)* then any combination of a,b or c
satisfies some possible combination since a single one of
those by itself also satisfies the part in the parens...

> interpreted as part of the name, not a %encoding in a URL).

I think the text we have is pretty clear with one small clarifiation:

3.4 Character sets
      in UTF-8, WITHOUT any hex-encodings

If we change the first case (XML) to this:
   o Within an XML query or response, the common name is represented
     in whatever encoding is specified by the XML document.

That might help....Martin? Comments?

-MM

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sat, 15 Jul 2000 18:18:00 +0900

>interpreted as part of the name, not a %encoding in a URL).

Sorry, but while I have never found a common name lying on
the street, and therefore can't really be sure, I have
serious difficulties imagining where to find bits and bytes
(as opposed to characters) on the street.

Regards,    Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sat, 15 Jul 2000 18:15:36 +0900

At 00/07/14 11:13 -0400, Michael Mealling wrote:
> > is indeed what we're looking for.

You mean an optional service, followed by any number (starting
with zero) of resourcedescriptor, referral, and error in any order.
If that's what you want, then the above expression will give it.

>However, as the note in the DTD

'Collective understanding' isn't really relevant.
Check the specs, and run it through some software
(e.g. nsgmls with the appropriate XML settings,
see http://www.jclark.com/).

>those by itself also satisfies the part in the parens...

Not really a secret, unless you consider BNFs a secret :-).

>      in whatever encoding is specified by the XML document.

Yes. As I said before, you have two choices: Either say that
in CNRP, all XML exchanged is in UTF-8, in which case the
CNs embedded in XML also automatically will be in UTF-8,
or say that XML in any encoding is okay, as long as that
encoding is clear according to the rules of application/xml,
but that all CNRP processors must at least support UTF-8
and UTF-16, and that in this case, CNs embedded in XML
are in the encoding of the whole XML document.

Regards,   Martin.

---

From: Leslie Daigle <[email-masked]>
Date: Mon, 17 Jul 2000 10:14:05 -0400

s/lying on the street/in a database field/

Leslie.

--

Leslie Daigle
[email-masked]

---

From: "Martin J. Duerst" <[email-masked]>
Date: Wed, 19 Jul 2000 18:15:47 +0900

Leslie, many thanks for the clarification.

Sorry, but if this is the intention, then this is inappropriate.
Databases store characters if they are designed well, and they
may do that as UTF-8, or UTF-16 (I know both is being done) or
whatever else, e.g. use some kind of compression,...

There is really some work needed here, it seems.
I'm all for restricting things to UTF-8 only where it
makes sense, but doing it the wrong way can cause more
harm than benefit.

Regards,   Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Wed, 19 Jul 2000 18:29:48 +0900

>validated it yet. I'm putting in lots 'o edits before Friday...

Great. Sorry for a late response.

>document also talks about CNs in other places (the URL

For the URI, this is specified separately, and the case
is clear. For databases, I have to disagree with Leslie,
because it would unnecessarily restrict implementations.
There may be other things such as APIs,..., but there again
e.g. in a Java API, you would choose UTF-16, because that's
how strings are encoded.

>I'll leave this one to the group. We could just create types

Which 'others'? Do you mean fr-BE and fr-CH? I'm confused.

>The type that references 1766 should probably say something

Yes, please.

>Nope. That's up to the service to be smart about...

That works for CNs, but not necessarily for properties, and
not for other fields.

>part of the CN) then the spaces would be encoded as a character

That works only under some very special conditions, so it won't
fly.

Regards,   Martin.