Subject: Troubles with UTF-8

From: "Tom.Petch" <[email-masked]>
Date: Fri, 23 Dec 2005 11:36:30 +0100

The IETF mandates the use of UTF-8 for text [RFC2277] as part of
internationalisation.  When writing an RFC, this raises a number of issues.

A) Character set.  UTF-8 implicitly specifies the use of Unicode/IS10646 which
contains 97,000 - and rising - characters.  Some (proposed) standards limit
themselves to 0000..007F, which is not at all international, others to
0000-00FF, essentially Latin-1, which suits many Western languages but is not
truly international.  Is 97,000 really appropriate or should there be a defined
subset?

B) Code point. Many standards are defined in ABNF [RFC4234] which allows code
points to be specified as, eg,  %b00010011 %d13 or %x0D none of which are
terribly Unicode-like (U+000D).  The result is standards that use one notation
in the ABNF and a different one in the body of the document; should ABNF allow
something closer to Unicode (as XML has done with &#000D;)?

C) Length. Text is often variable in length so the length must be determined.
This may be implicit from the underlying protocol or explicit as in a TLV.  The
latter is troublesome if the protocol passes through an application gateway
which wants to normalise the encoding so as to improve security and wants to
convert UTF to its shortest form with corresponding length changes (Unicode
lacks a no-op, a meaningless octet, one that could be added or removed without
causing any change to the meaning of the text).

Other protocols use a terminating sequence.  NUL is widely used in *ix; some
protocols specify that NUL must terminate the text, some specify that it must
not, one at least specifies that embedded NUL means that text after a NUL must
not be displayed (interesting for security).  Since UTF-8 encompasses so much,
there is no natural terminating sequence.

D) Transparency.  An issue linked to C), protocols may have reserved characters,
used to parse the data, which must not then appear in text.  Some protocols
prohibit these characters (or at least the single octet encoding of them),
others have a transfer syntax, such as base64, quoted-printable, %xx or an
escape character ( " \ %).  We could do with a standard syntax.

E) Accessibility.  The character encoding is specified in UTF-8 [RFC3629] which
is readily accessible (of course:-) but to use it properly needs reference to
IS10646, which is not.  I would like to check the correct name of eg
hyphen-minus (Hyphen-minus, Hyphen-Minus, ???) and in the absence of IS10646 am
unable to do so.

Overall, my perception is that we have the political statement - UTF-8 will be
used - but have not yet worked out all the engineering ramifications.

Tom Petch

Tom Petch

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 23 Dec 2005 21:44:02 +0900

> used - but have not yet worked out all the engineering ramifications.

Correct. Like so many results of IETF, enforcing Unicode just does
not work.

But, never mind. Unicode has nothing to do with the internationalization.

> others to
> 0000-00FF, essentially Latin-1, which suits many Western languages but
> is not truly international.

The only appropriate subset of Unicode is 0000-007f, ASCII. Latin-1,
which introduced the confusions of the currency symbol and NBSP, is
already overkill.

> Unicode lacks a no-op, a meaningless octet,

The confusion of NBSP implies that spaces are not so meaningful
octets so that it may be replaced by line break characters.

So, the situation is worse than you would have considered and even
full Latin-1 is hopeless.

Just interpret UTF-8 ASCII.

							Masataka Ohta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Dec 2005 14:57:02 +0100

Tom.Petch wrote:
 
> should there be a defined subset?

There are already some defined subsets, e.g. SASLPREP, 
NAMEPREP, etc.
  
> Many standards are defined in ABNF [RFC4234] which allows
> code points to be specified as, eg,  %b00010011 %d13 or %x0D
> none of which are terribly Unicode-like (U+000D).

AFAIK you can use %x10FFFE, you're not limited to %x00-FF.
You can also use %x10.FF.FE if you're talking about octets.
   
> should ABNF allow something closer to Unicode (as XML has
> done with &#000D;)?

Dubious, %x10FFFF is valid ABNF, but U+10FFFF is no character.

> Unicode lacks a no-op, a meaningless octet, one that could
> be added or removed without causing any change to the meaning
> of the text.

Can't you add or remove U+FFFE to fully normalized text, and
the result is still fully normalized ?  I'm not sure, better
check this.

> Since UTF-8 encompasses so much, there is no natural
> terminating sequence.

If you want NUL as terminator you're free to use it, also in
UTF-8.  There's no UTF-8 character xx00 or xx00yy if that's
your problem.  If you have the octet %x00 somewhere in UTF-8
strings it is always U+00 (also known as ASCII NUL).

> D)

Sorry, I don't get your point D.  Unicode has a block of 32
non-characters, that could be good enough for all internal
and temporary purposes of an application.  It also has the
known 65 ISO 8859 control codes, use them as you see fit in
your protocol.

> others have a transfer syntax, such as base64,
> quoted-printable, %xx or an escape character ( " \ %).
> We could do with a standard syntax.

Pick what you need as transfer encoding, B64 or QP, MIME would
even allow to roll your own.

> I would like to check the correct name of eg hyphen-minus
> (Hyphen-minus, Hyphen-Minus, ???) and in the absence of
> IS10646 am unable to do so.

Maybe get the latest Unicode 4.1.0 data file (almost one MB):
<http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>

For SASLPREP / NAMEPREP you would take Unicode version 3.2,
and for a beta test of Unicode 5.0 take the beta data.

                       Bye, Frank

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Fri, 23 Dec 2005 15:46:46 +0100

>Correct. Like so many results of IETF, enforcing Unicode just does

Amen. This is an architectural feature decided for political reasons 
which does not scale.

>But, never mind. Unicode has nothing to do with the internationalization.

I beg to differ on wording. Internationalization is an IETF/Unicode 
word. It is part of the equation "globalization=global environment 
internationalization + local environment localization". Its IBM 
understanding is to reduce the lingual barrier between the core and 
the ends it relates with. I think it is appropriate to the IETF 
US-ASCII based Internet technology.

But the real world is "multinationalization" (if to keep the same 
image, or multilingualization): the same but for every end to end 
relation (and languages). Let consider the IETF RFC 2277 proposition: 
content must be in Unicode (client system) and the protocol is in 
US-ASCII (core system). A document may look being in a language, but 
when you read its source it is in English interspread with unicoded text.

The internationalization (RFC 3066bis) culture is unilateral. 
Networking calls for a multilateral culture architecture (RFC 4151 may help).

The only solution I see, which addresses the requirements of Tom 
Petch, is to go through a common universalisation layer (not charset 
dependent), accepting the existing US-ASCII environment of Masataka 
Ohta as a maximum. It should then down to Hexa. Getting rid of the 
Unicode based layer violations, and permitting a full charset support 
strategy where Unicode could fully play its role of common reference.

Obviously two-tier policies based on langtags could not develop as 
easily as planned.
jfc

---

From: "Tom.Petch" <[email-masked]>
Date: Fri, 23 Dec 2005 14:45:29 +0100

----- Original Message -----
From: "Frank Ellermann" <[email-masked]>
To: <[email-masked]>
Sent: Friday, December 23, 2005 2:57 PM
Subject: Re: Troubles with UTF-8

> and for a beta test of Unicode 5.0 take the beta data.
I do not think such a reference is stable enough.  The earlier references in RFC
to Unicode cited the books published by Addison-Wesley; now the later ones
mostly cite ISO 10646.  I am looking for a readily (cheaply) available list of
Unicode character names and code points, as in RFC1345.

---

From: Ned Freed <[email-masked]>
Date: Fri, 23 Dec 2005 10:13:17 -0800 (PST)

> truly international.  Is 97,000 really appropriate or should there be a defined

Short answer: No.

Longer answer: Subsetting makes sense in some places, such as when defining
canonicalization schemes like our various *prep profiles. You pretty much have
to limit yourself to what's currently defined for these and Unicode is, for
better or worse, open-ended.

However, limiting everything to a subset of Unicode simply because "97,000
characters are too many" is a terrible idea. This has been tried - take  a look
at the various string types for Unicode in ASN.1 or the text body part types in
X.400 - and the result has been nothing but a mess.

> something closer to Unicode (as XML has done with &#000D;)?

ABNF is charset-independent, mapping onto non-negative integers, not
characters. Nothing prevents a specification from saying that a given ABNF
grammar specifies a series of Unicode characters represented in UTF-8 and using
%xFEFF or whatever in the grammar itself.

> convert UTF to its shortest form with corresponding length changes

The various length issues and tradeoffs that exist in different Character
Encoding Schemes for Unicode are well known, have been extensivey debated, and
are well understood. There are inherent tensions between the various formats
(preserve ASCII vs. equal length for all characters vs. wasting space)
and this makes any choice a compromise.

> (Unicode

NBSP is used for this purpose.

> there is no natural terminating sequence.

This simply isn't true. NUL is present in Unicode and is commonly used as  a
terminator.

> escape character ( " \ %).  We could do with a standard syntax.

I disagree. Different environments have very different constraints, and in many
cases these constraints interact with the underlying character data in ways
that force the use of different escaping conventions. For example, the
differences between quoted-printable the content-transfer-encoding and the Q
encoding of encoded-words are NOT gratuitous.

> hyphen-minus (Hyphen-minus, Hyphen-Minus, ???) and in the absence of IS10646 am

The entire Unicode character database is readily available online:

   http://www.unicode.org/ucd/

A quick check shows that 0x002D is written as HYPHEN-MINUS in the database and
in the code charts.

I didn't need (and never have needed) a copy of ISO 10646 to find out stuff
like this. I do find a copy of the printed Unicode book to be useful but not
required (and I have versions 1 through 3), but it is readily available, albeit
not free.

The reason many of our standards documents refer to ISO 10646 is that at one
time there was concern that Unicode wasn't sufficiently stable, and it was felt
that reference to the ISO document would offer some protection against
capricious change. I think in retrospect this concern has been shown to be
unwarranted, and all things being equal I would prefer to see references to the
more readily available Unicode materials. (Given the wide deployment of Unicode
now there is effectively no chance of a major change along the lines of the
Hangul reshuffle between V1 and V2.)

> used - but have not yet worked out all the engineering ramifications.

Well, we have a lot more than a political statement - a huge amount of
engineering work has been done to make Unicode workable in IETF protocols and
elsewhere. Does more work  need to be done? Of course it does - these tasks are
by their very nature pretty much unending. But all of the points you have
raised here are either nonissues, settled issues, or engineering compromises
where there is no "right" answer.

				Ned

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 24 Dec 2005 13:26:13 +0900

> Amen. This is an architectural feature decided for political reasons 

Which means that, despite all the efforts wasted on it, we can't
have any satisfying solution. However, for some kind of shameless
people who, for example, say:

: A huge amount of engineering work has been done to make Unicode
: workable in IETF protocols and elsewhere. Does more work need
: to be done? Of course it does - these tasks are by their very
: nature pretty much unending.

it is a blessing rather than a curse, because they are assured not
to reach the final solution only to loss there work items. That's
the politics of so many useless IETF products.

> word.

It is not. IETF/Unicode abuses the word.
 
> The internationalization (RFC 3066bis) culture is unilateral.

Wrong. First, RFC 3066bis has nothing to do with the
internationalization. Second, it has nothing to do with
the culture.

It, of course, has nothing to do with engineering.

It is purely political.

							Masataka Ohta

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sat, 24 Dec 2005 13:06:07 +0100

Dear Ned,
I do not want to restart an issue on this. I thank you for your 
answer. I think I could support most of what you say .... but 
throughout different layers. The real issue appears to me a layer 
confusion. You describe it well when you oppose charsets and Unicode. 
Due to historical reasons the Internet has been developed and is 
maintained as a US-ASCII English based system. This certainly 
simplified many things at prototyping stage (and can be retained as a 
useful/test-bed default config). But the resulting mono-layer 
Internet meets architectural difficulties. It can address multi-layer 
demands only in blocking them (RFC 2277, 3066 bis) or through options 
(ex. IDNA tables). It does not scale to the need.

Also, I thank you for confirming that the IETF doctrine is Unicode 
and not ISO based. The work of Unicode is oriented towards 
globalization. The world does not want globalization but an equal 
opportunity multinatinalisation/multilingualisation (which is an 
equal internationalization for every language). This does not mean 
that Unicode is not a key element. But Unicode is not the _central_ 
element. The central element is the user, whatever his language.

jfc

---

From: "Tom.Petch" <[email-masked]>
Date: Sat, 24 Dec 2005 19:24:30 +0100

Dave

Is this an ok use of RFC4234?  Reading it, I am not clear whether U+FEFF should
be
specified as %xFE %xFF or whether %xFFEF is ok?  And what is the ABNF for any
possible ISO 10646 character, all 97000 of them?

Tom Petch

----- Original Message -----
From: "Ned Freed" <[email-masked]>
To: "TomPetch" <[email-masked]>
Cc: "ietf" <[email-masked]>
Sent: Friday, December 23, 2005 7:13 PM
Subject: Re: Troubles with UTF-8
<snip>

> %xFEFF or whatever in the grammar itself.
<snip>

---

From: "Tom.Petch" <[email-masked]>
Date: Sat, 24 Dec 2005 19:07:12 +0100

> NBSP is used for this purpose.
Thank you for that; it is not something I have seen documented before.

> This simply isn't true. NUL is present in Unicode and is commonly used as  a
Not sure which bit isn't true.  I agree NUL is present in Unicode and agree that
some protocols use it as a terminator and prohibit its use in the text.  But
some allow it in the text in which case another form of termination is needed or
else the NUL must be escaped/encoded.  Presented with a comparable problem where
XML is in use, one WG has chosen to use an illegal XML sequence as a terminator
so what I was fishing for is if there were any parallels with UTF-8, which has
many
illegal sequences of octets and so it would be easy to choose one as a
terminator.

Tom Petch

> Ned.

---

From: Tim Bray <[email-masked]>
Date: Sat, 24 Dec 2005 13:31:34 -0800

I agree with everything Ned said, this is a non-problem.

> NBSP is used for this purpose.

I think actually U+FEFF ZERO WIDTH NO-BREAK SPACE is appropriate in  
many situations.  Yes, I know it's also a BOM.

> Hangul reshuffle between V1 and V2.)

Yes.  It's intellectually satisfying that 10646 still exists, but for  
al practical purposes, Unicode is plenty stable enough, and widely  
accessible.  -Tim

---

From: Ned Freed <[email-masked]>
Date: Sat, 24 Dec 2005 15:35:30 -0800 (PST)

> else the NUL must be escaped/encoded.

None of this differs in any material way from the situation with plain
ASCII text. I fail to see why we have to do something with Unicode to deal
with a situation that's existed with ASCII for decades.

> Presented with a comparable problem where
> many illegal sequences of octets and so it would be easy to choose one as a
> terminator.

Using a construct that's syntactically illegal at a higher protocol level
is one thing - I still wouldn't do it, but it is arguagly OK. Using a sequence
of octets that's not allowed by the underlying charset, OTOH, is a really
bad idea. For one thing, various agents do perform syntax checks on charset
data, so this is bound to cause major problems. And for another, such sequences
are going to be specific to a particular character encoding scheme, which
will make agents that transcode from, say, UTF-8 to UTF-16 pretty unhappy.

If Unicode data needs to be self-terminated I strongly recommend using
NUL to do it.

				Ned

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 26 Dec 2005 03:41:25 +0100

--On 23. desember 2005 11:36 +0100 "Tom.Petch" <[email-masked]> 
wrote:

> A) Character set.  UTF-8 implicitly specifies the use of Unicode/IS10646
> which contains 97,000 - and rising - characters.  Some (proposed)
> standards limit themselves to 0000..007F, which is not at all
> international, others to 0000-00FF, essentially Latin-1, which suits many
> Western languages but is not truly international.  Is 97,000 really
> appropriate or should there be a defined subset?

I think Ned has answered most of your other points... I'll chime in on this 
one.....

My opinion: ALL attempts at defining an "useful" character set of any size 
between 128 and "all you can eat" for use internationally have been dismal 
failures. They have been used in some niche, sooner or later there's a need 
to work outside that box, and gateways or other forms of self-torture 
result. (Alvestrand's equality: gateways = pain).

At the moment, the only reasonable candidate for an "all you can eat" 
character set is the Unicode charset. All other alternatives, including the 
bizarrely byzantine character set switching schemes of ISO 2022, are 
basically dead in the marketplace.

So there are only two real choices for charset left: ASCII and Unicode.

ASCII is unsuitable for any language except the technologists' simplified 
version of English. So if you want text, and want it to work 
internationally, there's only one choice left.

Subsets are a mistake.

                            Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 26 Dec 2005 03:48:20 +0100

--On 23. desember 2005 14:45 +0100 "Tom.Petch" <[email-masked]> 
wrote:

> I do not think such a reference is stable enough.  The earlier references
> in RFC to Unicode cited the books published by Addison-Wesley; now the
> later ones mostly cite ISO 10646.  I am looking for a readily (cheaply)
> available list of Unicode character names and code points, as in RFC1345.

Cheap, stable, available. Pick two.

The Unicode lists are readily available from www.unicode.org. 
(Notwithstanding their problems with other stability guarantees, I think 
they try really hard to not break them more than they feel they have to. I 
haven't found a real problem myself with using them for reference yet).
They are also real easy to cut and paste from, being online, and easy to 
feed into programs for the same reason.

The ISO 10646 standard is stable, but expensive.

The Unicode books are stable (paper!), and somewhat less expensive. They 
also contain a CD-ROM.

Pick your pain.

(BTW, I've heard multiple people claim that RFC 1345 has errors in it. I 
haven't verified the claim myself, but I'm pretty sure the Unicode on-line 
files are checked by more people than the number of people that checked RFC 
1345. RFC 1345 is also quite a bit older than the recent Unicode and ISO 
10646 versions.)

                      Harald

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Mon, 26 Dec 2005 14:09:39 +0100

At 03:41 26/12/2005, Harald Tveit Alvestrand wrote: "So there are 
only two real choices for charset left: ASCII and Unicode."

This is a layer violation. Or this is internationalised-US-ASCII (to 
remove the barriers between ASCII and foreign customers). These 
barriers result from experienced needs. You may want to remove them 
at your own US-ASCII risk. But you cannot impose your solution to 
cross-foreign relations. Saying that your mission is to influence 
people to accept it (RFC 3935) will not help you.

The time of the car of any colour you want if it is English, is over. 
US-ASCII English protocols, documents and culture have helped a time 
to develop the technology in an US-ASCII English environment. They 
are now are a problem. Unless they are a default and an example to 
build equivalent opportunities for every other language. Or unless 
the technology is to stay US-ASCII English centric for ever.

Otherwise there is only one choice: the user keyboard. The master is 
the user not the developer of the machine he uses.

There is certainly a developer, manufacturer, salesman need for 
common simplification, to better support the user diversity. Unicode 
is one of them. But you have to be careful that the "simplification" 
does not increase the total complexity and hurt your market.
jfc

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 26 Dec 2005 18:26:19 +0100

Ned Freed wrote:

>> (Unicode lacks a no-op, a meaningless octet, one that could
>>  be added or removed without causing any change to the
>>  meaning of the text).

> NBSP is used for this purpose.

My proposal u+FFFE was nonsense, I wanted u+FEFF.  The latter
is "zero width nbsp" (aka BOM), did you also mean this beast ?

Not a no-op in some cases wrt hyphenation and composition, so
it's probably not what Tom wants.
                                  Frank

http://www.eki.ee/letter/chardata.cgi?ucode=FEFF
http://tools.ietf.org/html/3629#section-6

---

From: Ned Freed <[email-masked]>
Date: Mon, 26 Dec 2005 10:56:47 -0800 (PST)

> is "zero width nbsp" (aka BOM), did you also mean this beast ?

Yes.

> it's probably not what Tom wants.

It's close enough.

				Ned

---

From: "Tom.Petch" <[email-masked]>
Date: Wed, 28 Dec 2005 09:48:47 +0100

----- Original Message ----- 
From: "Ned Freed" <[email-masked]>
To: "Frank Ellermann" <[email-masked]>
Cc: <[email-masked]>
Sent: Monday, December 26, 2005 7:56 PM
Subject: Re: Troubles with UTF-8

> > it's probably not what Tom wants.
Thanks for that - as close as I am going to get:-)    

Tom Petch

---

From: "Tom.Petch" <[email-masked]>
Date: Wed, 28 Dec 2005 10:09:05 +0100

----- Original Message -----
From: "Ned Freed" <[email-masked]>
To: "TomPetch" <[email-masked]>
Cc: "Ned Freed" <[email-masked]>; "ietf" <[email-masked]>
Sent: Sunday, December 25, 2005 12:35 AM
Subject: Re: Troubles with UTF-8

> If Unicode data needs to be self-terminated I strongly recommend using

The Unicode data I am thinking of may have come from an upper layer protocol and
needs to be passed transparently (as with an error or hello message, identity
even); it may or may not already be NUL-terminated (ever had that security
foul-up where some userid/password are entered/stored NUL-terminated and some
are not?) - hence I see the need to terminate the string in some other way, or
to escape or in some other way transfer encode (parts of) the string.  I looked
at existing RFC, found many different approaches, all viable but none that
really said to me 'this is good engineering, this is best practice'.  Hence,
floating the issue to see if there were any better ones out there. I think not,
which is of itself worth knowing.

Tom Petch

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 28 Dec 2005 13:30:13 +0100

--On onsdag, desember 28, 2005 10:09:05 +0100 "Tom.Petch" 
<[email-masked]> wrote:

> The Unicode data I am thinking of may have come from an upper layer
> protocol and needs to be passed transparently (as with an error or hello
> message, identity even); it may or may not already be NUL-terminated
> (ever had that security foul-up where some userid/password are
> entered/stored NUL-terminated and some are not?) - hence I see the need
> to terminate the string in some other way, or to escape or in some other
> way transfer encode (parts of) the string.  I looked at existing RFC,
> found many different approaches, all viable but none that really said to
> me 'this is good engineering, this is best practice'.  Hence, floating
> the issue to see if there were any better ones out there. I think not,

could you give a little more detail, such as WG name, draft name and so on?

There are many strong opinions around "proper" treatment of XML and of 
text, and it would be a shame to ask for advice now, reach a seemingly 
reasonable conclusion, and then encounter violent objections at IETF Last 
Call.

(the WG that went for illegal syntax as "terminator" SHOULD have caused 
such a reaction, IMHO; I guess the people who care missed it....)

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 28 Dec 2005 22:05:57 +0900

> which is of itself worth knowing.

You can do nothing.

That problem is that Unicode is stateful with complex and
indefinitely long term states, which is a lot worse than
properly profiled ISO 2022 such as that of RFC1468, which
is the character encoding most widely used for Japanese.

Unicode is not even finite state, which means some pattern
matching and normalization problems are hard or insolvable.

OTOH, if you start from scratch, you can have encoding with
a lot shorter term and finite states.

						Masataka Ohta

---

From: "Tom.Petch" <[email-masked]>
Date: Wed, 28 Dec 2005 14:54:40 +0100

----- Original Message -----
From: "Harald Tveit Alvestrand" <[email-masked]>
To: "Tom.Petch" <[email-masked]>; "Ned Freed" <[email-masked]>
Cc: "ietf" <[email-masked]>
Sent: Wednesday, December 28, 2005 1:30 PM
Subject: Re: Troubles with UTF-8
> such a reaction, IMHO; I guess the people who care missed it....)

The 'illegal syntax' is not yet an RFC but is in draft-ietf-netconf-ssh-05.txt
which says
   "As the previous example illustrates, a special character sequence,
    ]]>]]>, MUST be sent by both the client and the server after each XML
    document in the NETCONF exchange.  This character sequence cannot
    legally appear in an XML document, so it can be unambigiously used to
    indentify the end of the current document, allowing resynchronization
    of the NETCONF exchange in the event of an XML syntax or parsing
    error."
For me, that is ok; the 'illegal syntax' is part of the transport syntax not
part of the XML syntax and so is not illegal, if you follow me:-)

The differing treatments of NUL I see in UTF-8 strings, out of many such RFC,
are
RFC3748 [EAP] only the portion of the field prior to the null is displayed, MUST
NOT be null terminated
RFC2444 [SASL] terminated by a NUL (0) octet
RFC2869 [RADIUS] MUST be able to deal with embedded nulls
RFC3315 [DHCP] MUST NOT be null-terminated.
and, just to be different,
RFC2595 [TLS+IMAP]  authorization identity followed by a US-ASCII NUL followed
by the authentication identity followed by a US-ASCII NUL character followed by
the clear-text password  [this also subsets UTF-8]
RFC3413 [SNMP tag]  Delimiter characters are defined to be one of the following
characters: - space character (0x20), TAB character (0x09), carriage return
character (0x0D), line feed character (0x0A)

All doubtless with good reason for being the way they are but it does cover the
spectrum.  As I said above, I quite like 'illegal syntax' and so would happily
terminate UTF-8 (and it is UTF-8 that IETF has mandated, not UTF-nnn) with
%xC1 - but I see that that will not find favour.

Tom Petch

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Dec 2005 16:16:25 +0100

Tom.Petch wrote:
> ...

Why don't you use an illegal *character* instead, such as Formfeed? 
That's certainly easier to parse...

Best regards, Julian

---

From: "Tom.Petch" <[email-masked]>
Date: Wed, 28 Dec 2005 16:19:55 +0100

----- Original Message -----
From: "Masataka Ohta" <[email-masked]>
To: "Tom.Petch" <[email-masked]>
Cc: "Ned Freed" <[email-masked]>; "ietf" <[email-masked]>
Sent: Wednesday, December 28, 2005 2:05 PM
Subject: Re: Troubles with UTF-8

> a lot shorter term and finite states.
You've lost me here.  I don't understand the use of state in the context of
Unicode (perhaps because I have just been reading about stateless serverside TLS
resumption:-).  I suspect that in the context of transporting
param = value
with value being supplied by an upper layer protocol in UTF-8 then it is not a
concern.  There is always base64 or quoted-printable:-(

Tom Petch

---

From: "Tom.Petch" <[email-masked]>
Date: Wed, 28 Dec 2005 17:06:05 +0100

----- Original Message -----
From: "Julian Reschke" <[email-masked]>
To: "Tom.Petch" <[email-masked]>
Cc: "ietf" <[email-masked]>
Sent: Wednesday, December 28, 2005 4:16 PM
Subject: Re: Troubles with UTF-8

> >> CalFrom [email-masked] Wed Dec 28 12:15:00 2005
From: "Tom.Petch" <[email-masked]>
To: "Julian Reschke" <[email-masked]>
Date: Wed, 28 Dec 2005 17:06:05 +0100
Cc: ietf <[email-masked]>
Subject: Re: Troubles with UTF-8
Reply-To: "Tom.Petch" <[email-masked]>
Sender: [email-masked]

----- Original Message -----
From: "Julian Reschke" <[email-masked]>
To: "Tom.Petch" <[email-masked]>
Cc: "ietf" <[email-masked]>
Sent: Wednesday, December 28, 2005 4:16 PM
Subject: Re: Troubles with UTF-8

> That's certainly easier to parse...

I agree, for XML, but my main concern is with UTF-8 encoded strings, where
FormFeed is a legal character, encoded as it would be in ASCII.  I was using the
'illegal syntax' to float an alternative approach, like using %xC1 - which is
illegal in
UTF-8 - to delimit a UTF-8 string, but as I say, that idea does not seem to have
caught on  within the IETF.

Tom Petch

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 28 Dec 2005 12:46:41 -0800

Hi -

> Sent: Wednesday, December 28, 2005 8:06 AM
...
...

I think the use of explicitly encoded length, rather than special terminator
or deliminator sequences, is simpler to code and debug, as well as
being more robust in avoiding buffer overflow problems, etc.  This
is especially true given the variable-length encoding inherent in UTF-8,
as well as the open-ended way that combining marks follow, rather than
precede the characters to which they apply.  (I think this was the "state"
that Masataka Ohta was referring to.)

Reserving NUL as a special terminator is a C library-ism.  I think that
history has shown that the use of this kind of mechanism, rather than
explicitly tracking the string's length, was a mistake.

Randy

---

From: James Cloos <[email-masked]>
Date: Thu, 29 Dec 2005 20:05:17 -0500

>>>>> "Tom" == Tom Petch <[email-masked]> writes:

Tom> You've lost me here.  I don't understand the use of state in the
Tom> context of Unicode

Masataka was refering to the fact that the universal character set
contains combining characters and some characters that otherwise alter
how subsequent and/or previous characters are treated.

As an example, the sequence of the two characters:

,----
| U+0061 LATIN SMALL LETTER A
| U+030B COMBINING DOUBLE ACUTE ACCENT
`----

which is enocoded in utf-8 as:

,----
| a̋
`----

has state between the base letter and the accent.  If the a is lost,
the accent will be added to whatever was before the a.

Similarly, U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK
affect how anything after them is displayed.  Their existance in the
standard, therefore, makes the standard statefull.

The combining accent characters can be added the base characters in
arbitrary number and sequence.  Not all combinations are currently in
use by any written language, of course, but they remain open ended.
You can even have multiple instances of a single combining character
in a sequence of combining characters.  (Consider, eg, a stack of
accents where there is a circumflex above a dieresis above a
circumflex above the base character.  Probably not used by anyone,
but it /could/ be.)

-JimC
-- 
James H. Cloos, Jr. <[email-masked]>

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 30 Dec 2005 11:09:44 +0100

Randy Presuhn wrote:

  [Tom Petch said:]
>> I was using the 'illegal syntax' to float an alternative
>> approach, like using %xC1 - which is illegal in UTF-8

Illegal today, it wasn't for some time.  My UTF-8 "decoder"
script would return one SUB for a %xC1 plus the next octet.
%xFF and %xFE were always illegal, %xFD was the worst case
for 5*6+1 bits u+7FFFFFF in UCS-4.
  
>> that idea does not seem to have caught on within the IETF.

u+FFFF (UTF-8 %xEFBFBF) is guaranteed to be no character, it
is AFAIK reserved for this purpose.  But not "on the wire".
 
> I think the use of explicitly encoded length, rather than
> special terminator or deliminator sequences, is simpler to
> code and debug, as well as being more robust in avoiding
> buffer overflow problems, etc.

Yes, abusing %xFF or similar tricks would be like an PDU with
an empty header and a constant trailer.  Your idea "length in
the header" (and maybe a checksum as trailer ?) is better.  

If that hits the limit for encoded lengths add a mechanism for
a "more" flag, or chunks with a "length = 0 is the end", etc.

> Reserving NUL as a special terminator is a C library-ism.

A leading length has its own drawbacks if you want a string
with more than 255 octets after one octet for the length. ;-)

> history has shown that the use of this kind of mechanism,
> rather than explicitly tracking the string's length, was a
> mistake.

<CRLF> or whatever isn't too bad with a decent maximal line
length (like 1000).  If you want arbitrary encoded lengths you
would need a delimiter to separate the length from the SDU, or
another trick for this effect.  Attackers could then try their
luck with huge encoded lengths.
                                Bye, Frank

---

From: "Tom.Petch" <[email-masked]>
Date: Fri, 30 Dec 2005 09:57:42 +0100

----- Original Message -----
From: "Randy Presuhn" <[email-masked]>
To: "ietf" <[email-masked]>
Sent: Wednesday, December 28, 2005 9:46 PM
Subject: Re: Troubles with UTF-8
> explicitly tracking the string's length, was a mistake.

I agree with you for 'binary' protocols, intended for machine consumption (OSPF,
SNMP), where the string is usually wrapped up in a binary-encoded TLV; but not
for character ones, intended for humans, in +-printable characters, with
positional or keyword parameters (LDAP[RFC2254], SDP[RFC2327], SASL OTP[RFC2444]
or MIME) where a numeric length, in ASCII characters, would look a little odd to
me.

I always saw NUL as an **IX-ism more than a C-ism, and so of wider use, could be
wrong on that.

Tom Petch