Subject: [Sip] language info in SIP headers

From: Shingo Fujimoto <[email-masked]>
Date: Tue, 23 Jul 2002 13:55:37 +0900

Hello. I would like to raise the new issue to solve.

SIP RFC specified UTF-8 friendly transport, but I think that did not
specified the way to use multi bytes characters in header part of
SIP messages.

If I use Japanese Kanji letters in From: header (as display name),
the receiver UA cannot determine which language of characters are
used in UTF-8 encoded text. Additionally, since UTF-8 CJK kanji
characters are overlapped between Chinese, Japanese, and Korean,
sometimes UA can not determine which font should be used to display
them.

MIME part 3 and updated RFC2231 introduced some solution for this
problem with combination of character set and language tag. However
RFC2231 always required escaping 8bit-flaged characters, there are
huge overhead to express multi byte characters.

As another option, recent Unicode 3.x specification includes
"language tag" characters to solve this problem, but it is not
common to support such new feature in the any UTF-8 handling
libraries. I guess that results corrupted display in UAs.

My suggestion about this, we should prepare supplemental document
for SIP which specify the way to express international characters
with language information in header part (modified RFC2231 or UTF-8
with "lang tag" support).

I would like to hear your opinion.

Thank you.
-- 
======================================
         SHINGO FUJIMOTO
    FUJITSU LABORATORIES LIMITED
E-mail: [email-masked]

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 23 Jul 2002 03:56:10 -0400

This is actually a fairly thorny problem. From a previous private
discussion, there are only a few possible solutions:

- RFC 2482 or http://www.unicode.org/unicode/reports/tr27/#tag provide
language tagging, as you noted. While I have not seen an official
statement from the IESG on this, I believe 10646 language tagging has
generally been deprecated. It's not quite clear whether the reasons (I
believe, nesting) for that apply to short text such as Subject or
Organization.

- For header fields that allow parameters, I believe the best option is
to add a language parameter, as in

From: "Somebody" <sip:[email-masked]> ;language="en"

This doesn't allow multiple languages within the same header field, but
is probably useful and sufficient as a hint for text-to-speech and other
UI rendering. It is also backward-compatible. This works for From, To
and a few others, such as the NAI header fields.

- For header fields that don't allow parameters (Subject, Organization,
etc.), there are no good options. In general, I believe we should avoid
such header fields in the future. One possible kludge suggested in
private conversation would be to add a descriptive header that tells you
what language each header field contains. Or, simpler but less
space-efficient, something like

Subject-Language: jp
Organization-Language: en-us

Any other suggestions?

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 23 Jul 2002 05:48:59 -0400

> The reason I prefer this approach is that SIP messages must be small
> if it runs over UDP. "language tag" will add only few bytes.
> 
> Problem is backward compatibility with widely deployed SIP implementations.
> Another option is use MIME part3 with small modification.
> Since current MIME part3 did not allow to use any 8 bit flagged
> characters in MIME headers, encoding always escapes UTF-8 characters.
> 
> This is an example of MIME part3 header.
> 
> From: =?UTF-8*JP?B?<base64 encoded utf-8 string>?= <sip:[email-masked]>
> 
> "base64" part is escaped and increased its size with 1.5 times or more.
> 
> If we define new encoding, say 'U', which only escape special
> charactors ('=','?',and '%' with '%xx' escaping).
> 
> From: =?UTF-8*JP?U?<raw utf-8 string>?= <sip:[email-masked]>
> 
> This extension saves overhead on 8 bit charactors, and will not
> suffer any older implementations.
> 
> Additionally, if we can assume utf-8 and 'U' encoding, then we can
> minimize the escaping syntax too.
> 
> From: =?JP?<raw utf-8 string>?= <sip:[email-masked]>
> 
> This still valid as display name value.
> 
> Both Keith and Ned suggested me that if we define 'U' encoding, it
> should be limited within SIP specification.
> 

I would suggest you write up a short I-D on the subject.

---

From: Shingo Fujimoto <[email-masked]>
Date: Tue, 23 Jul 2002 18:32:54 +0900

I am happy to hear you already had some discussions.

> believe, nesting) for that apply to short text such as Subject or

You are probably right.

Here is the comment from Ned Freed, one of the RFC2231 authors:

> Language indicators in a pure UTF-8 environment like SIP headers have to be
> stored as part of the UTF-8 string itself. The plane 14 language tags described
> in
> 
>    http://www.unicode.org/unicode/reports/tr27/#tag
> 
> are the only means of doing this that I'm aware of. However, the use
> of such tags is, as the text indicates, strongly discouraged.
> 
> It is the Unicode Technical Committee's belief that
> 
>   The requirement for language information embedded in plain text data is often
>   overstated.
> 
> As such, I see little chance for support for such tagging in the context of SIP
> headers being widely deployed.
> 
> 				Ned
> 

> and a few others, such as the NAI header fields.

I think this approach is fair. I guess it is quite rare to use
multiple languages are used in short text field.

> Organization-Language: en-us

I think these are also as good as previous one.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 24 Jul 2002 13:31:38 -0400

> I am happy to hear you already had some discussions.

Henning and I were trying to determine the "right" thing to
to regarding language tagging, and found, as you are
finding, that it is a far from simple answer....

specific comment inline:

>>Language indicators in a pure UTF-8 environment like SIP headers have
> 
> to be
> 
>>stored as part of the UTF-8 string itself. The plane 14 language tags
> 
> described
> 
>>in
>>
>>  The requirement for language information embedded in plain text data
> 
> is often
> 
>>  overstated.
>>
>>As such, I see little chance for support for such tagging in the
> 
> context of SIP
> 

I think the point Ned is making here, is that language
tagging is not all that useful in practice. With UTF-8,
you can render Kanji characters. You need the language
tagging IFF some automata needs to do something special
based on knowledge of the language, such as translation
or text to speech. However, it is far from clear whether
that information is truly needed. Translation of names
(like those in Subject, From, To) is not likely to work
in any case, nor is it clear that the language tag is
needed for it...

> if it runs over UDP. "language tag" will add only few bytes.

Well, as Ned indicated, and Henning repeated, the in-band
language tagging provided by UTF-8 is not considered
acceptable, and we would not be able to write a specification
which recommends it as the solution for sip.

-Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 24 Jul 2002 13:49:47 -0400

> in any case, nor is it clear that the language tag is

You're about to step into the Han unification minefield. From my limited
understanding, the same Unicode/10646 code point is rendered differently
depending on the language (Japanese, Chinese, etc.). I don't know if
this is also true for text-to-speech. Certainly, text-to-speech is
useful for all three headers, e.g., in a voicemail system. Jonathan
Rosenberg ;lang=de and Jonathan Rosenberg ;lang=en sound very different.

> which recommends it as the solution for sip.

I think he meant the ?=jp? marking, not UTF-8 language tagging. (That
said, it isn't clear to me why this would be inherently better, except
maybe to avoid tripping non-language-tag-capable 10646 renderers and
annoy the user with random characters instead... The arguments against
tagging seem to be about the same in either syntax - nesting, and all that.)

A pragmatic choice is to say that To/From don't need UTF or ?=? tagging,
since ;language=jp will work just fine, without annoying users with
random ASCII art. The Subject header is more difficult, but there, a
smart text-to-speech renderer can usually guess at the language by
checking its dictionary. From my vague recollection, very few words are
needed to do fairly accurate language recognition, excepting deliberate
attempts to create phrases that are meaningful in two languages. Google
seems to do an ok guessing job on most web pages, for example. (Guessing
isn't pretty, but many users aren't going to bother marking their
Subject line anyway and routinely use multiple languages, so that device
configuration isn't sufficient.)

---

From: Sean Olson <[email-masked]>
Date: Wed, 24 Jul 2002 13:46:08 -0700 (PDT)

Is there a problem to be solved here that cannot
be handled by Accept-Language and Content-Language?
I realize these only cover the payload and not the
headers. My assumption is that most of the information
that you would need to extract from the headers
can either be treated as a string of octets by an
automata 
-or-
is for human consumption and therefore it is
sufficient if these can be rendered appropriately
to the end user (ignoring the Han unification 
problem for now) which UTF-8 should cover. Proper
names as in the From/To are a good example of this.

What automata application cannot make use of the 
Accept-Language header to do its job? (I'm not
saying one does not exist, I would just like
clarification of the problem we are trying to solve 
here)

Regards,
Sean Olson

--- Henning Schulzrinne <[email-masked]> wrote:
> Sip mailing list 
> https://www1.ietf.org/mailman/listinfo/sip

__________________________________________________
Do You Yahoo!?
Yahoo! Health - Feel better, live better
http://health.yahoo.com

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 24 Jul 2002 17:13:30 -0400

I don't think this is a major issue, but there is a semantic difference. 
Accept-Language lists *all* the languages I'm willing to deal with, in 
terms of error messages, so it might be something like

Accept-Language: en, de, fr

This doesn't tell the receiver text-to-speech engine anything about how 
to pronounce my name. As noted, for personal names, even the same ASCII 
string (ignoring the much more complicated CJK sets) can be pronounced 
quite differently, particularly where the name has been "imported". I 
believe the problem is more pronounced with CJK since Chinese names are 
rendered in "classical" Chinese, rather than simplified. (And I'm 
probably getting the details wrong.)

Sean Olson wrote:
>>>I think the point Ned is making here, is that
>>
>>language
>>
>>>tagging is not all that useful in practice. With
>>
>>UTF-8,
>>
>>>you can render Kanji characters. You need the
>>
>>language
>>
>>>tagging IFF some automata needs to do something
>>
>>special
>>
>>>based on knowledge of the language, such as
>>
>>translation
>>
>>>or text to speech. However, it is far from clear
>>
>>whether
>>
>>>that information is truly needed. Translation of
>>
>>names
>>
>>>(like those in Subject, From, To) is not likely to
>>
>>work
>>
>>>in any case, nor is it clear that the language tag
>>
>>is
>>
>>>needed for it...
>>
>>>Well, as Ned indicated, and Henning repeated, the
>>
>>in-band
>>
>>>language tagging provided by UTF-8 is not
>>
>>considered
>>
>>>acceptable, and we would not be able to write a
>>
>>specification
>>

---

From: Shingo Fujimoto <[email-masked]>
Date: Thu, 25 Jul 2002 10:04:40 +0900

> I would suggest you write up a short I-D on the subject.

Yes, I will.

I just wonder if I should write it as a requirement doc, or just a specification proposal doc.

Because I think the requirements are some kind of simple, e.g. UTF-8 
friendly and multiple language in same field.

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 25 Jul 2002 11:41:53 -0400

The chairs will probably be helpful to provide guidance; I would imagine 
the formal approach would be a separate requirements document. Another 
pragmatic approach might be a document that presents and motivates the 
problem to be solved and then presents some solutions and why they may 
or may not work.