Subject: Re: Troubles with UTF-8

From: Tim Bray <[email-masked]>
Date: Sun, 01 Jan 2006 23:32:11 -0800

On Dec 28, 2005, at 7:16 AM, Julian Reschke wrote:

>> The 'illegal syntax' is not yet an RFC but is in draft-ietf- 
>> netconf-ssh-05.txt
>> which says
>>    "As the previous example illustrates, a special character  
>> sequence,
>>     ]]>]]>, MUST be sent by both the client and the server after  
>> each XML
>
> Why don't you use an illegal *character* instead, such as Formfeed?  
> That's certainly easier to parse...

Or NULL, U+0000, which is illegal in XML. -Tim

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Tim Bray <[email-masked]>
Date: Sun, 01 Jan 2006 23:40:43 -0800

On Dec 28, 2005, at 5:05 AM, Masataka Ohta wrote:

> That problem is that Unicode is stateful with complex and
> indefinitely long term states

Has this ever caused a real problem to a real programmer in real life?

I have written a whole bunch of mission-critical code that reads and  
generates UTF-8, and any correct implementation will have to deal  
with the fact that there is no necessary connection between the  
number of glyphs on the screen and bytes in its encoding.  It would  
be perfectly reasonable for an implementation to declare a  
limitation, for example that it will not process than 32 trailing  
modifiers on any character, and this would not cause problems in  
production because sequences of such a length do not occur in the  
encoding of any known text.

Which is to say, Ohta's statement about statefulness is true, but the  
conclusion that this is a "problem" is erroneous. -Tim

---

From: Tim Bray <[email-masked]>
Date: Sun, 01 Jan 2006 23:43:39 -0800

On Dec 28, 2005, at 12:46 PM, Randy Presuhn wrote:

> Reserving NUL as a special terminator is a C library-ism.  I think  
> that
> history has shown that the use of this kind of mechanism, rather than
> explicitly tracking the string's length, was a mistake.

I used to think so too, but I don't any more; twenty years of doing  
text processing has convinced me that C's null-terminated strings  
simply cannot be improved on in a low-level programming language.   
For more on the subject see http://www.tbray.org/ongoing/When/200x/ 
2003/04/13/Strings  -Tim

---

From: Peter Dambier <[email-masked]>
Date: Mon, 02 Jan 2006 09:20:29 +0100

Tim Bray wrote:
>> Reserving NUL as a special terminator is a C library-ism.  I think  that

I guess variably lenght V-records of type

string {int type,
         int length,
         int data[] );

would be horror. That will lose you 4 bytes per word and 2 bytes for
every printable sign.

C-ASCII "Randy Presuhn" = 14 char + '\0'.

Compare it to

      99999, " R"," a"," n"," d"," y",
      99999, " P"," r"," e"," s"," h"," u"," n"

That is 28 characters now. No alternative.

> simply cannot be improved on in a low-level programming language.   For 
> more on the subject see http://www.tbray.org/ongoing/When/200x/ 
> 2003/04/13/Strings  -Tim
> 
> 

-- 
Peter and Karin Dambier
The Public-Root Consortium
Graeffstrasse 14
D-64646 Heppenheim
+49(6252)671-788 (Telekom)
+49(179)108-3978 (O2 Genion)
+49(6252)750-308 (VoIP: sipgate.de)
mail: [email-masked]
mail: [email-masked]
http://iason.site.voila.fr/
https://sourceforge.net/projects/iason/

---

From: Bob Braden <[email-masked]>
Date: Tue, 03 Jan 2006 19:31:16 -0800

The morality of string delimiters vs. character counts
  is a debate that has been going on for some 40 years!
Some things never change... ;-)

Bob Braden

>...
>
>I think the use of explicitly encoded length, rather than special terminator
>or deliminator sequences, is simpler to code and debug, as well as
>being more robust in avoiding buffer overflow problems, etc.

...

>Randy

---

From: James Seng <[email-masked]>
Date: Wed, 4 Jan 2006 13:50:27 +0800

On 12/23/05, Tom.Petch <[email-masked]> wrote:
>
> A) Character set.  UTF-8 implicitly specifies the use of Unicode/IS10646
> which
> contains 97,000 - and rising - characters.  Some (proposed) standards
> limit
> themselves to 0000..007F, which is not at all international, others to
> 0000-00FF, essentially Latin-1, which suits many Western languages but is
> not
> truly international.  Is 97,000 really appropriate or should there be a
> defined
> subset?

Why should there be a subset? You really really dont want to go into a
debate of which script is more important then the other.

B) Code point. Many standards are defined in ABNF [RFC4234] which allows
> code
> points to be specified as, eg,  %b00010011 %d13 or %x0D none of which are
> terribly Unicode-like (U+000D).  The result is standards that use one
> notation
> in the ABNF and a different one in the body of the document; should ABNF
> allow
> something closer to Unicode (as XML has done with &#000D;)?

Following RFC4234, Unicode code point U+ABCD will just be represented as
%xABCD.

I do not see the problem you mention or am I missing something?

C) Length. Text is often variable in length so the length must be
> determined.
> This may be implicit from the underlying protocol or explicit as in a
> TLV.  The
> latter is troublesome if the protocol passes through an application
> gateway
> which wants to normalise the encoding so as to improve security and wants
> to
> convert UTF to its shortest form with corresponding length changes
> (Unicode
> lacks a no-op, a meaningless octet, one that could be added or removed
> without
> causing any change to the meaning of the text).

While the simple byte counting obviously wont give you the accurate length
of the text (since one character in Unicode maybe represented by one or more
bytes), it is fairly trival to write a script to count the length of the
text accurately. Heck, perl 5.6 onwards even support Unicode natively.

> Other protocols use a terminating sequence.  NUL is widely used in *ix;
> some
> protocols specify that NUL must terminate the text, some specify that it
> must
> not, one at least specifies that embedded NUL means that text after a NUL
> must
> not be displayed (interesting for security).  Since UTF-8 encompasses so
> much,
> there is no natural terminating sequence.

NUL is defined in Unicode btw but I am disgressing; You already started with
a wrong foot if you think UTF-8 as some sort of programming encoding scheme
rather then what it is; an encoding scheme for a character reportairs.

> D) Transparency.  An issue linked to C), protocols may have reserved
> characters,
> used to parse the data, which must not then appear in text.  Some
> protocols
> prohibit these characters (or at least the single octet encoding of them),
> others have a transfer syntax, such as base64, quoted-printable, %xx or an
> escape character ( " \ %).  We could do with a standard syntax.

In those cases, Unicode U+ABCD or ANBF %xABCD do nicely. Why do we need
another one?

E) Accessibility.  The character encoding is specified in UTF-8 [RFC3629]
> which
> is readily accessible (of course:-) but to use it properly needs reference
> to
> IS10646, which is not.  I would like to check the correct name of eg
> hyphen-minus (Hyphen-minus, Hyphen-Minus, ???) and in the absence of
> IS10646 am
> unable to do so.

In absence of a dictionary, I couldn't understand most of the words you used
in an RFC. OMG, what should I do?

http://www.unicode.org/charts/

-James Seng

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 04 Jan 2006 16:12:18 +0900

> Has this ever caused a real problem to a real programmer in real life?

Yes, of course. State information preserved between lines is
really annoying.

But, you miss the point in my original mail:

: Unicode is not even finite state, which means some pattern
: matching and normalization problems are hard or insolvable.

that is, with Unicode, you can not search strings in reasonable
amount of time.

> generates UTF-8, and any correct implementation will have to deal  with 
> the fact that there is no necessary connection between the  number of 
> glyphs on the screen and bytes in its encoding.

You completely miss the point. It has nothing to do with the long
term state.

> It would  be perfectly 
> reasonable for an implementation to declare a  limitation, for example 
> that it will not process than 32 trailing  modifiers on any character, 
> and this would not cause problems in  production because sequences of 
> such a length do not occur in the  encoding of any known text.

I said "long term state", which, of course, is not confined in a
character with or without modifiers.

> conclusion that this is a "problem" is erroneous. -Tim

Instead, your statement: "I have written a whole bunch of mission-
critical code that reads and generates UTF-8" is untrustworthy.

Of course, it is perfectly reasonable for an implementation to
declare a limitation, for example, that it will not process
non-ASCII characters, which may also be the assumption of your
code.

						Masataka Ohta

---

From: "Tom.Petch" <[email-masked]>
Date: Thu, 5 Jan 2006 21:14:45 +0100

You say that a Unicode code point can be represented by %xABCD but that is not
spelt out in ABNF [RFC4234].  And when it refers to 'one printable character' as
'%x20-7E' I get the impressions that coded character sets like Unicode, with
more than 256 code points, do not fall within its remit.  I have yet to see any
use of this in an I-D or RFC. I did post a question about this to this list on
24th December and the lack of response reinforces my view that this is uncharted
territory.

Tom Petch
----- Original Message -----
From: "James Seng" <[email-masked]>
To: "Tom.Petch" <[email-masked]>
Cc: "ietf" <[email-masked]>
Sent: Wednesday, January 04, 2006 6:50 AM
Subject: Re: Troubles with UTF-8

> I do not see the problem you mention or am I missing something?
<snip>

---

From: Bill Fenner <[email-masked]>
Date: Fri, 6 Jan 2006 09:44:48 -0500

On 1/5/06, Tom.Petch <[email-masked]> wrote:
> spelt out in ABNF [RFC4234].

ABNF uses non-negative integers to represent characters - note that it
explicitly doesn't specify a range (2nd sentence of section 2.3).  RFC
4234 specifies the encoding of those integers into ASCII, and says
that other encodings may be specified.  There's an implicit (obvious)
encoding of ABNF characters into Unicode, and as Misha pointed out the
IRI spec uses it - technically maybe it needs to be spelled out
somewhere.

>  And when it refers to 'one printable character' as
> more than 256 code points, do not fall within its remit.

That's an example, which is probably missing the word "US-ASCII"
between "printable" and "character".

  Bill