Subject: Re: [Geopriv] Domain identifier in common policy

From: Andrew Newton <[email-masked]>
Date: Wed, 4 Jan 2006 09:48:56 -0500

Henning,

Sorry to bring this up again, but Hannes made me reread the text and  
I caught a nit.

On Nov 14, 2005, at 7:32 PM, Henning Schulzrinne wrote:
> I looked at RFC 3490 (IDNA) in more detail just now. For the  
> 'domain' attribute, I think we can simplify the comparison process  
> as follows:
>
> --- begin text ---
>
> Common policy MUST use UTF-8 to store domain names in 'id' domain  
> attributes. For non-IDNs, lower-case ASCII SHOULD be used.

[ snip ]...

Is "MUST use UTF-8" intended to purposefully rule out UTF-16, which  
all XML parsers are required to understand?  That is fine if it is,  
but this would seem to then restrict the document to UTF-8.  If this  
is a protocol requirement, it needs to be stated.

-andy

_______________________________________________
Geopriv mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/geopriv

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 05 Jan 2006 11:44:06 -0500

I don't feel strongly about this; my only gut feeling is that it is a 
good thing to reduce options that don't add value. As far as I know, the 
set of strings representable by UTF-8 is exactly the same as the one 
representable in UTF-16. What is the advantage of allowing UTF-16?

(One disadvantage is that any comparison would have to convert to a 
common format if both are allowed.)

> Is "MUST use UTF-8" intended to purposefully rule out UTF-16, which all 
> XML parsers are required to understand?  That is fine if it is, but this 
> would seem to then restrict the document to UTF-8.  If this is a 
> protocol requirement, it needs to be stated.
> 
> -andy

---

From: Andrew Newton <[email-masked]>
Date: Thu, 5 Jan 2006 14:04:58 -0500

On Jan 5, 2006, at 11:44 AM, Henning Schulzrinne wrote:

> I don't feel strongly about this; my only gut feeling is that it is  
> a good thing to reduce options that don't add value. As far as I  
> know, the set of strings representable by UTF-8 is exactly the same  
> as the one representable in UTF-16. What is the advantage of  
> allowing UTF-16?
>

Unless you foresee the development of special purpose XML parsers for  
this application, I can see no advantage to ruling out UTF-16.  As  
for your stated disadvantage, which XML parsers pass raw bytes to the  
application instead of a common format?  Admittedly there are gobs of  
XML parsers out there, but I've never seen one that does this.

-andy

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 06 Jan 2006 04:39:39 -0500

How do XML parsers pass back string elements to the application?

> this application, I can see no advantage to ruling out UTF-16.  As for 
> your stated disadvantage, which XML parsers pass raw bytes to the

---

From: Andrew Newton <[email-masked]>
Date: Fri, 6 Jan 2006 08:43:45 -0500

Usually they pass back the element content as a unicode compatible  
string or an array of unicode compatible characters, and the  
application does not need to make a distinction between UTF-8 and  
UTF-16.  Some languages have native Unicode support, but in cases  
where that is not true special types are defined to represent xml  
character data.

-andy

On Jan 6, 2006, at 4:39 AM, Henning Schulzrinne wrote:

>> Unless you foresee the development of special purpose XML parsers  
>> for this application, I can see no advantage to ruling out  
>> UTF-16.  As for your stated disadvantage, which XML parsers pass  
>> raw bytes to the application instead of a common format?   
>> Admittedly there are gobs of XML parsers out there, but I've never  
>> seen one that does this.
>> -andy
>

---

From: Andrew Newton <[email-masked]>
Date: Fri, 6 Jan 2006 08:55:35 -0500

> common format if both are allowed.)

BTW, from BCP 70, last paragraph in Section 5.1:

    Restricting XML data to only be expressed in UTF-8 is an additional
    syntactic restriction (see Section 4.3) which, depending on
    circumstances, might add additional implementation complexity.

-andy

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 06 Jan 2006 09:23:02 -0500

FWIW, I just picked the first XML application that came to mind, in PHP. 
It doesn't support UTF-16. Source: http://de.php.net/xml and 
http://de.php.net/manual/en/ref.xml.php#xml.encoding

 From all I can tell for PHP, it represents strings in their 'native' 
(byte) representation, so that if you have two XML documents, one using 
8859-1 and one UTF-8, you need to convert between them to compare 
literal strings outside the ASCII range. 
(http://de.php.net/manual/en/language.types.string.php)

I don't see why restricting character sets to UTF-8 complicates processing.

Andrew Newton wrote:
> string or an array of unicode compatible characters, and the application 
> does not need to make a distinction between UTF-8 and UTF-16.  Some 
> languages have native Unicode support, but in cases where that is not 
> true special types are defined to represent xml character data.
> 
> -andy
>

---

From: Andrew Newton <[email-masked]>
Date: Fri, 6 Jan 2006 10:13:26 -0500

If it doesn't support UTF-16 as the source encoding, it is not  
compliant with the XML standard.

BTW, thanks for the reference point with PHP.  I started looking  
around to see what various parsers using languages without native  
Unicode do.  Both libxml2 and Xerces-C define datatypes for xml  
character data.  libxml2 always hands the application UTF-8 but it  
supports UTF-16 as a source encoding (making one draw the conclusion  
that it transcodes to UTF-8 when UTF-8 is not the source encoding).   
Expat does the same, but there is a compile-time option to have the  
parser hand the application UTF-16 instead of UTF-8.  From what I can  
tell, PHP is the exception and not the rule.

-andy

On Jan 6, 2006, at 9:23 AM, Henning Schulzrinne wrote:

> FWIW, I just picked the first XML application that came to mind, in  
> PHP. It doesn't support UTF-16. Source: http://de.php.net/xml and  
> From all I can tell for PHP, it represents strings in their  
> 'native' (byte) representation, so that if you have two XML  
> documents, one using 8859-1 and one UTF-8, you need to convert  
> between them to compare literal strings outside the ASCII range.  
>>>> Unless you foresee the development of special purpose XML  
>>>> parsers for this application, I can see no advantage to ruling  
>>>> out UTF-16.  As for your stated disadvantage, which XML parsers  
>>>> pass raw bytes to the application instead of a common format?   
>>>> Admittedly there are gobs of XML parsers out there, but I've  
>>>> never seen one that does this.
>>>> -andy
>>>

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 06 Jan 2006 11:03:09 -0500

At least, it's not a general problem...

I still fail to see the practical advantage of allowing UTF-16, given 
that we control the input and that this is not general text, but rather 
strings with a very specific purpose.

Andrew Newton wrote:
> If it doesn't support UTF-16 as the source encoding, it is not compliant 
> with the XML standard.
> 
> BTW, thanks for the reference point with PHP.  I started looking around 
> to see what various parsers using languages without native Unicode do.  
> Both libxml2 and Xerces-C define datatypes for xml character data.  
> libxml2 always hands the application UTF-8 but it supports UTF-16 as a 
> source encoding (making one draw the conclusion that it transcodes to 
> UTF-8 when UTF-8 is not the source encoding).  Expat does the same, but 
> there is a compile-time option to have the parser hand the application 
> UTF-16 instead of UTF-8.  From what I can tell, PHP is the exception and 
> not the rule.
> 
> -andy
> 
>> (byte) representation, so that if you have two XML documents, one 
>> using 8859-1 and one UTF-8, you need to convert between them to 
>> compare literal strings outside the ASCII range. 
>>>>> for this application, I can see no advantage to ruling out UTF-16.  
>>>>> As for your stated disadvantage, which XML parsers pass raw bytes 
>>>>> to the application instead of a common format?  Admittedly there 
>>>>> are gobs of XML parsers out there, but I've never seen one that 
>>>>> does this.
>>>>> -andy
>>>>

---

From: Andrew Newton <[email-masked]>
Date: Fri, 6 Jan 2006 11:27:06 -0500

On Jan 6, 2006, at 11:03 AM, Henning Schulzrinne wrote:

> I still fail to see the practical advantage of allowing UTF-16,  
> given that we control the input and that this is not general text,  
> but rather strings with a very specific purpose.

1) How does the application know that the source encoding was  
UTF-16?  Or asked another way, how does the application know the  
source encoding was not UTF-8?  Most can look at the XML declaration,  
but in the passing of the element content where this is actually  
relevant they may not be able to tell. This type of detection is  
extra logic and seems unnecessary if the application can handle  
both.  Also, on platforms that would output UTF-16, it is extra logic  
to force them to transcode to UTF-8, which is also unnecessary since  
the receiver of that output must understand UTF-16.

2) What practical advantage is there to disallowing UTF-16?

-andy

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 06 Jan 2006 19:10:10 +0100

>XML parsers out there, but I've never seen one that does this.

Note that e.g. RFC 3920 defines exactly this restriction.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Randall Gellens <[email-masked]>
Date: Fri, 13 Jan 2006 18:00:32 -0800

I think we need to resolve the issue of saying "MUST use UTF-8" (and 
hence disallowing UTF-16) or not.

Please reply by Friday January 20 if you feel strongly that we need 
to add the restriction: "Common policy MUST use UTF-8 to store domain 
names in 'id' domain attributes".  If there is no consensus to add 
this restriction, we won't.