Subject: RE: [Sipping] I-D ACTION:draft-fujimoto-sipping-header-lang-00.tx t

From: Adam Roach <[email-masked]>
Date: Fri, 20 Sep 2002 18:27:28 -0500

8 days without a peep on this thread. Anyone have any opinions
on this topic?

/a

> -----Original Message-----
> From: Adam Roach [mailto:[email-masked]]
> Sent: Thursday, September 12, 2002 10:58
> To: 'Shingo Fujimoto'; sipping-WG
> Subject: RE: [Sipping] I-D
> ACTION:draft-fujimoto-sipping-header-lang-00.tx t
> 
> 
> This is a really good start. The requirements section seems to be
> spot-on what we need.
> 
> It's a bit odd to have the requirements and the solution specified
> in the same document; you might want to split them out into separate
> documents. I suspect you will be able to get consensus on the
> requirements faster than the mechanism. Also, as a minor procedural
> point, if this work is put on the charters, I think the requirements
> will be specified in SIPPING, and the mechanism will be specified
> in SIP.
> 
> Speaking to the mechanism, the current draft has the following
> text, which acknowledges but rejects Unicode language tags:
> 
>     UTF-8 allows to represent any UCS-2/4 characters, 
> including recent 
>     proposed "language tag" which used to specifying the languages 
>     information for following character strings. However, the 
> deployed 
>     SIP implementations may not have enough knowledge to handle them 
>     correctly.  
> 
> Your assertion that most (if not all) deployed SIP implementations
> don't support Unicode language tagging is almost certainly correct;
> however, since *no* existing SIP implementations support the mechanism
> proposed in your draft, it seems to be a wash. In other words,
> this argument provides no motivation for discarding Unicode language
> tagging.
> 
> I invite members of the SIP community to chime in with their opinion
> on this topic. As I see it, the primary differences involved are:
> 
> - Selecting Unicode language tags would be more standard, inasmuch
>   as the representation of languages in this fashion is likely
>   to see far wider deployment than any SIP-specific solution. So,
>   for example, if the receiving UA uses Unicode libraries that
>   inherently understand language tags, it makes implementation much
>   simpler.
> 
> - Shingo Fujimoto's proposal is more space efficient. For example,
>   to encode a set of ISO639/ISO3166 (language, country) codes into
>   a header takes only 9 additional octets, instead of the 24 that
>   UTF-8 encoding of language tags would; encoding just the language
>   takes 6 additional octets instead of 12.
> 
> /a
> 
> > -----Original Message-----
> > From: Shingo Fujimoto [mailto:[email-masked]]
> > Sent: Wednesday, September 11, 2002 22:17
> > To: sipping-WG
> > Subject: Re: [Sipping] I-D 
> > ACTION:draft-fujimoto-sipping-header-lang-00.txt
> > 
> > 
> > Hi, all.
> > 
> > This I-D is about the issue which discussed on SIP-ML originally.
> > 
> > You can see the previous discussion from SIP-ML archives:
> > 
> > http://www1.ietf.org/mail-archive/working-groups/sip/current/m
> > sg06075.html
> > 
> > I would appreciate your comments on my I-D.
> > 
> > Best.
> > 
> > [email-masked] wrote:
> > > A New Internet-Draft is available from the on-line 
> > Internet-Drafts directories.
> > > 
> > > 
> > > 	Title		: SIP Header Language Information Extension
> > > 	Author(s)	: S. Fujimoto
> > > 	Filename	: draft-fujimoto-sipping-header-lang-00.txt
> > > 	Pages		: 6
> > > 	Date		: 2002-9-10
> > > 	
> > > This draft explains the problem when we use the UTF-8 
> character set 
> > > to represent the language text other than English. This 
> draft also 
> > > describes the requirements to solve the problems, and 
> proposes the 
> > > extended syntax to SIP protocol message syntax specification.
> > > 
> > > A URL for this Internet-Draft is:
> > > 
> http://www.ietf.org/internet-drafts/draft-fujimoto-sipping-hea
der-lang-00.tx
t

-- 
======================================
         SHINGO FUJIMOTO
    FUJITSU LABORATORIES LIMITED
E-mail: [email-masked]

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sun, 22 Sep 2002 23:04:50 -0400

I'm not entirely convinced that this is a real problem; the need is 
really driven by things like speech to text when the same characters are 
pronounced differently in different languages. Are people really 
encountering this problem in real life? I'm not opposed to this work; 
I'm just not convinced its really manifesting itself as an issue.

Assuming people do consider it a real problem, here are some comments on 
the requirements:

* on backwards compatibility - the requirement is vague. The question is 
- is it a requirement that headers containing strings with are to be 
languaged tagged will be rendered on older UA's as if the language 
tagging were not present? Your proposal does not have this property.

* multiple languages in one header - This is getting really obscure. 
Other proposals that do language tagging (such as CPIM msgfmt 
http://www.ietf.org/internet-drafts/draft-ietf-impp-cpim-msgfmt-06.txt) 
don't do this. Do we really need it?

* On this requirement:

  The i18n-ext MUST provide the mean to determine if the
     implementation can handle the specific part of i18n-ext header
     value.

Why? If you have the backwards compatibility requirement as I stated it 
above, this is not needed.

* On this requirement:

The i18n-ext MUST provide enough information for cultural dependent
     process.

I would propose that we not have requirements that go beyond existing 
standards for language tagging. If such standards support cultural 
dependent tags, great. If not, no way.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Shingo Fujimoto <[email-masked]>
Date: Tue, 24 Sep 2002 13:26:47 +0900

> I'm just not convinced its really manifesting itself as an issue.

I apologize the real issue is not clear in my I-D.

As Adam explained, most critical problem we need solve is world 
famous CJK code problem.

Unlike other languages, Chinese, Japanese, Korean uses a lot of 
characters in real life and from cultural background, we really care 
the letter difference on human name between the characters even they 
pronounced same way.
# Because each letter has its own mean in context.

Since Japan and Korea imported characters from ancient China, they 
look like same, but after more than thousand years past they changed 
their shapes and usages very differently. The difference are not 
small, and I can read nor write Chinese/Korean letters.
# I could guess their meaning sometime ...

The language tagging relieves this problem, with enabling to switch 
fonts for code mapping.

> tagging were not present? Your proposal does not have this property.

You are right. I should add the statement about this.

My proposal uses valid syntax even in current SIP specification. If 
UA (or gateway) which does not support 'new' extension, they can 
parse and process without errors.

The limitation on older UA is showed with ugly tag letters.

> don't do this. Do we really need it?

I think yes. From CJK code problem, we can not use Japanese title 
with Chinese word in UTF-8.

For example, the travel agency may use the chinese town name as the 
part of tour name in Japanese.

For another example, some of my Japanese friends uses compound From 
header in E-mail to express their name in English and Japanese in 
one header. That header can adapt both Japanese capable terminal and 
English only terminal, and eliminates the operation to switch 
between two versions of 'From' header.

> above, this is not needed.

Since CJK problem is exist, their are no universal font set for 
UTF-8 which can render all language. I think that is useful the 
terminal can determine which part of header can render.

> dependent tags, great. If not, no way.

I don't intend to add uncommon requirements. I used the word 
'cultural dependent' is not only language but country.

I think using RFC3066 "Tags for Indication of Languages" meets this 
requirement.

---

From: "Sean Olson" <[email-masked]>
Date: Mon, 23 Sep 2002 22:18:38 -0700

Based on some of your requirements (for example, multiple languages
in one header and ability to display on non-conforming devices),
I think the Unicode language tagging mechanism is preferrable to
creating
our own tagging format.

At the very least, I would like to understand why we would need to
escape
UTF-8 characters in this scheme. We are an 8-bit clean protocol with
inherent
UTF-8 support. Is there another issue I am overlooking?

Regards,
Sean Olson
Microsoft

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Shingo Fujimoto
Sent: Monday, September 23, 2002 9:27 PM
To: Jonathan Rosenberg
Cc: Adam Roach; sipping-WG
Subject: Re: [Sipping] I-D
ACTION:draft-fujimoto-sipping-header-lang-00.txt

Jonathan Rosenberg wrote:
> really driven by things like speech to text when the same characters 
> are pronounced differently in different languages. Are people really 

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip Use
[email-masked] for new developments of core SIP

---

From: Shingo Fujimoto <[email-masked]>
Date: Tue, 24 Sep 2002 17:09:53 +0900

> I think the Unicode language tagging mechanism is preferrable to

Even I could not find any written statement in IETF website, I heard 
Unicode language tagging mechanism is now discouraged to use.

I know the Unicode language tagging was published as RFC in 1999 
(RFC2482), but "Plane 14: characters for language tags" was 
superseded by the publication of Unicode 3.1.

See http://www.unicode.org/reports/tr7/

I also heard that most of Unicode guys dislike the approach since 
they thought Unicode should be the stateless code as much as possible.

Additional information can be found in Unicode specification 3.0 
chapter 5 "Implementation Guidelines", "5.11 Language Tagging" 
described this issue:

 From http://www.unicode.org/unicode/uni2book/ch05.pdf P.114
 > Avoiding Language Tags, Because of extra implementation burden,
 > language tags should be avoided in plain text unless language
 > information is required and it is known that the receivers of the
 > text will properly recognize and maintain the tags.

 From my investigation, RFC2231 meets most of our needs but 
sacrifice the efficiency of UTF-8 encoding. Except us-ascii 
characters letters, all of UTF-8 codes are 8 bit flagged. That means 
most (or all!) must be escaped with MIME B-encoding or Q-encoding. 
That against the requirement "3.2 Data Size" in my I-D.

My proposal is based on RFC2231 and modified with soften escaping 
target characters.

> UTF-8 support. Is there another issue I am overlooking?

Escaping is not for 8 bit characters, but for i18n-ext special 
characters such as '=' '?' '"'. Because those are misunderstood as 
the part of special sequence in i18n-ext syntax.

Best.

---

From: "Eric Burger" <[email-masked]>
Date: Tue, 24 Sep 2002 12:27:38 -0400

Comments in-line (and all the way to the bottom)

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Jonathan Rosenberg
[snip]
> I'm just not convinced its really manifesting itself as an issue.

Adam gave the text-only response.  I would also offer that this will become
an important issue with respect to support for the visually impaired.

> tagging were not present? Your proposal does not have this property.

Agreed.  If one really wanted the text to render "properly" on an "old"
device, one would need to introduce totally new, parallel headers (e.g.,
"UTF-To:", "UTF-From:", etc.).  Yuck.  Don't do that.

I think the idea is that the new header should not Break SIP.  If that is
the case, I would just drop the requirement.  NOTHING should Break SIP.
That is an implicit requirement.

That said, having mangled UTF-8 strings appear in ASCII is *exactly* what
one gets if you are reading e-mail on an old mail client or you don't have
the appropriate fonts loaded today.  We don't need to solve the unsolvable
problem.  People live with the font problem in the text/e-mail world.  We
don't need to do anything different in the text/SIP world.

To reiterate, I would just drop the requirement.

> don't do this. Do we really need it?

First of all, it is a SHOULD, not a MUST.  I can easily imagine From fields
with mixed-language tags (e.g., "La Keg"), so the requirement is a
reasonable one.

> above, this is not needed.

Agreed.

> dependent tags, great. If not, no way.

I think this is a cultural mis-interpretation :-)

How about the following text:

3.5 Country Information

   The i18n-ext MUST provide locale information.

   This means that even though the same characters and
   wording may be used, there may be different
   pronunciation or voicing.  For example, "Vacuumed
   Tomatoes" will have different renderings in en_UK
   and en_US.

This does bring up a sticky point.  If the purpose of these requirements is
to render text, then will there ever be a locale issue?  On the other hand,
if the purpose of these requirements is to render text as speech, then we
would be better served by specifying SSML.  That draws us back to having a
pair of parallel headers again (e.g., "To:" and "To-Speech:").  While that
made sense in the VPIM world, I don't think it will work well in the SIP
world.
[snip]
> >>From: Adam Roach [mailto:[email-masked]]
[snip]
> >>will be specified in SIPPING, and the mechanism will be specified

I wholeheartedly agree.  Section 4-6 do not belong in this document.

> >>  takes 6 additional octets instead of 12.

I'm ambivalent on this one.  Whatever is more "standard" would win for me.
What does e-mail use?  That's what we should use, if possible.
[snip]

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 30 Sep 2002 01:21:58 -0400

inline.

Shingo Fujimoto wrote:
> As Adam explained, most critical problem we need solve is world famous 
> CJK code problem
> 
> characters in real life and from cultural background, we really care the 
> letter difference on human name between the characters even they 
> pronounced same way.

Seems like these should just be different unicode characters, as a 
result. But, thats for someone else to decide. I'll accept that language 
tagging is generally something we want to add to SIP.

>> * on backwards compatibility - the requirement is vague. The question 
>> is - is it a requirement that headers containing strings with are to 
>> be languaged tagged will be rendered on older UA's as if the language 
> My proposal uses valid syntax even in current SIP specification. If UA 
> (or gateway) which does not support 'new' extension, they can parse and 
> process without errors.
> 

That would violate the definition of backwards compatibility as I have 
proposed. Indeed, my read of the recommendations in unicode 3.1 is that 
the reason that the language tagging is discouraged is EXACTLY this 
limitation. Your proposed solution relies in in-band tagging, just like 
unicode language tagging, and will therefore have exactly the same 
problems.

In XML, for example, the tagging is not in-band. Its done with the 
xml:lang attribute:

<subject xml:lang="de">Achtung!</subject>

An element which doesn't understand the de tag will still properly 
determine that the subject is "Achtung!", and will render it in a 
reasonable way, without any ugly escape characters.

We should probably do something similar in SIP. I would propose a header 
called Header-Lang which indicates the language of any free-form text in 
a header:

Header-Lang: from;lang="de", to;lang="en"

which says that the freeform text in the From header is in German, and 
in the To field, in English. This gets a bit messy for headers like 
Contact that have multiple values, each of which has freeform text that 
might have a different lanugage. In that case, an index that indicates 
which value can be included:

Header-Lang: contact;index=1;lang="de",contact;index=2;lang="es"

which says that the first contact header is in german, and the second, 
in spanish.

This approach has the same benefits as the out-of-band approach of XML; 
if the tag or even the extension is not understood, the headers are 
rendered as if there were no language tags present. They will be shown 
properly without any escaping.

> I think yes. From CJK code problem, we can not use Japanese title with 
> Chinese word in UTF-8.
> 
> For example, the travel agency may use the chinese town name as the part 
> of tour name in Japanese.
> 
> header in E-mail to express their name in English and Japanese in one 
> header. That header can adapt both Japanese capable terminal and English 
> only terminal, and eliminates the operation to switch between two 
> versions of 'From' header.

No other protocol I am aware of supports mixed language tagging like 
this. In fact, AFAICT, no XML-based document will be able to do this. 
The granularity of the language tags are per-element, and cannot be 
finer than that. I do not believe we need to do better than XML as far 
as language tagging is concerned.

>> Why? If you have the backwards compatibility requirement as I stated 
>> it above, this is not needed.
> 
> 
> Since CJK problem is exist, their are no universal font set for UTF-8 
> which can render all language. I think that is useful the terminal can 
> determine which part of header can render.

My point is, if you use the approach above I have proposed, it doesn't 
matter if the recipient understands it. If they do, they get the benefit 
of language tags. If they don't, the text is still rendered as if 
language tagging were completely absent. Thus, there is total backwards 
compatibility.

> I don't intend to add uncommon requirements. I used the word 'cultural 
> dependent' is not only language but country.
> 

OK.

Thanks,
Jonathan R.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 30 Sep 2002 08:44:36 -0400

> tagging is generally something we want to add to SIP.

I think this had to do with the 16-bit codespace of the earlier 
versions, but this is a topic which has apparently generated endless 
flame wars on the IDNS lists and many other venues, with rather 
different opinions between C, J and K, so this is probably best left alone.

> properly without any escaping.

I would prefer a solution that more closely fits the rest of the SIP 
model, where possible. This is just another parameter describing header 
information, so each header should be tagged, as in

From: "Gerhard Schroeder" <[email-masked]> ;lang="de"

and

Contact:  contact1;lang="de", contact2;lang="fr"

By definition, unknown parameters are ignored. This seems far less 
complicated and easier to parse, without having to do look-ahead. The 
only (relevant) header this does not work for is Subject, since it 
doesn't take parameters. There seem to be a number of kludges:

- Subject-Language (modeled after Content-Language)
- Header-Language (default for all headers unless overridden by ;lang 
parameters, as above)

> this. In fact, AFAICT, no XML-based document will be able to do this. 

Plus, it's not clear what a sequence of differently-tagged sub-strings 
would mean. It could mean "alternate", as in the English/Japanese 
example, or it could mean a true mixed-language string, as in

Subject: <lang=de>Achtung</>, <lang=en>said Colonel Klink</>

(never mind the bogus XML). Thus, you need a far more expressive tagging 
mechanism, with nesting and preferences, similar to MIME. This could get 
rather messy rather quickly. A reference to a vCard structure, assuming 
it can handle such expressive tagging, would be more appropriate in that 
case, via Call-Info.

---

From: Rohan Mahy <[email-masked]>
Date: Mon, 30 Sep 2002 08:50:21 -0700

On Sunday, September 29, 2002, at 10:21 PM, Jonathan Rosenberg wrote:

>>> encountering this problem in real life? I'm not opposed to this  
>>> work; I'm just not convinced its really manifesting itself as an  
>>> issue.
>> famous CJK code problem
> result. But, thats for someone else to decide. I'll accept that  
> language tagging is generally something we want to add to SIP.
>
>>> * on backwards compatibility - the requirement is vague. The  
>>> question is - is it a requirement that headers containing strings  
>>> with are to be languaged tagged will be rendered on older UA's as if  
>>> the language tagging were not present? Your proposal does not have  
>>> this property.
> proposed. Indeed, my read of the recommendations in unicode 3.1 is  
> that the reason that the language tagging is discouraged is EXACTLY  
> this limitation. Your proposed solution relies in in-band tagging,  
> just like unicode language tagging, and will therefore have exactly  
> the same problems.

The ugly tag letters are really not the huge problem that I expect you  
are imagining.  The UA would display whatever Unicode characters it  
could display whether using language tags or not.  Only the language  
tag on an older UA would be represented by a single ugly character.   
Because of the encoding of UTF-8, this ugly character would not  
interfere with additional characters. Note also that a UA can choose  
represent unknown language tags in a non-ugly way.

> We should probably do something similar in SIP. I would propose a  
> header called Header-Lang which indicates the language of any  
> free-form text in a header:
>
> Contact that have multiple values, each of which has freeform text  
> that might have a different lanugage. In that case, an index that  
> indicates which value can be included:
>

yuck!  this is hideous.  now you can't effectively perform end-to-end  
message integrity of the complete semantic content of the from header  
for example, because presumably, the Header-Lang header would need to  
be modified by intermediaries in case one added an Error-Info header  
and wanted to language tag it.

> This approach has the same benefits as the out-of-band approach of  
> XML; if the tag or even the extension is not understood, the headers  
> are rendered as if there were no language tags present. They will be  
> shown properly without any escaping.

I'd like to point out that for most languages where this matters (CJK),  
the UAs that have fonts for these characters are well incented to  
support the language tag, whereas implementations using just the ASCII  
part of UTF-8 or the Latin-1 extensions are likely to display the  
actual characters as ugly unknown character tags anyway.

>>> http://www.ietf.org/internet-drafts/draft-ietf-impp-cpim-msgfmt- 
>>> 06.txt) don't do this. Do we really need it?
>> header. That header can adapt both Japanese capable terminal and  
>> English only terminal, and eliminates the operation to switch between  
>> two versions of 'From' header.
>

I think mixed language tagging inside a header is useful functionality,  
especially in the Subject line.

>> which can render all language. I think that is useful the terminal  
>> can determine which part of header can render.
>
> matter if the recipient understands it. If they do, they get the  
> benefit of language tags. If they don't, the text is still rendered as  
> if language tagging were completely absent. Thus, there is total  
> backwards compatibility.
>
>>> * On this requirement:
>>>
>>> I would propose that we not have requirements that go beyond  
>>> existing standards for language tagging. If such standards support  
>>> cultural dependent tags, great. If not, no way.

FYI: an example of a culturally dependent process is selecting  
Simplified (PRC) vs. Traditional Chinese (Taiwan, Hong Kong SAR)  
characters.  These characters represent the same language in different  
countries/regions.

thanks,
-rohan