Subject: [Ltru] RE: For review: Tagging text with no language

From: "Richard Ishida" <[email-masked]>
Date: Wed, 16 May 2007 16:13:42 +0100

Thanks, Mark.  
 
I made some very slight adjustments to the article, and hopefully it is ok to publish now.
 
I18n core folks, please speak out if you don't feel we should agree to publish this during our telecon next Tuesday.
 
RI
 
============
Richard Ishida
Internationalization Lead
W3C (World Wide Web Consortium)
 
http://www.w3.org/People/Ishida/
http://www.w3.org/International/
http://people.w3.org/rishida/blog/
http://www.flickr.com/photos/ishida/

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Davis
Sent: 20 April 2007 21:00
To: Richard Ishida
Cc: [email-masked]; LTRU Working Group
Subject: Re: For review: Tagging text with no language

Richard, getting back to the topic of XML, it sounds like we have three cases, illustrated by the following example:

<p xml:lang="en">There are three cases:
  <span>foo</span>,
  <span xml:lang="">foo</span>,
  <span xml:lang="und">foo</span>
</p>

In the first foo construction, we are saying that it inherits the language "en". The second and third construction are equivalent: we are overriding the "en" setting, and saying that foo is undetermined with regard to language. The difference between them is that: 

*	the second construction can't be used in XHTML (or any other case where a DTD or schema requires a value). 

*	the third construction can always be used, but shouldn't be used if the second can be. 

Mark

On 4/11/07, Richard Ishida <[email-masked]> wrote: 

http://www.w3.org/International/questions/qa-no-language

Comments are being sought on this FAQ-based article prior to final release. Please send any comments to [email-masked]. We expect to publish a final version in one to two weeks.

============
Richard Ishida
Internationalization Lead
W3C (World Wide Web Consortium) 

http://www.w3.org/People/Ishida/
http://www.w3.org/International/
http://people.w3.org/rishida/blog/  <http://people.w3.org/rishida/blog/> 
http://www.flickr.com/photos/ishida/

-- 
Mark 

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Martin Duerst <[email-masked]>
Date: Fri, 18 May 2007 19:27:43 +0900

Hello Richard,

Overall, this looks good to publish. But a few points:

There is some hickup in the sentence "This is an attempt to capture some ideas in a thread on [email-masked] and a later reprise of that those ideas to which several people contributed."

Probably "that those" -> "those".

I'd also change "attempt to capture some
ideas" to something like "summary of discussion". The former doesn't
fit well to an FAQ item, and it doesn't reflect the fact that the
discussion was pretty conclusive.

At the very end, you write: "Martin Du"rst points out that you can redefine the XHTML/HTML format within the document to create an HTML/XHTML page that validates while using lang="" or xml:lang="". This is not recommended for widespread use, however."

I'd change the last sentence to use the point provided by Frank
(http://lists.w3.org/Archives/Public/www-international/2007JanMar/0141.html).
So I'd write something like "However, such a document is no longer
strictly conforming in the sense of XHTML 1.0."
This makes sure that people know what the issue is.

Another point, probably long-term: I found the text visually difficult
to read at some points because for me, there is not enough distinction
between the plain language of the text and the formal elements that are
interspersed. Things such as "zxx" and "und" appear in the middle of
the text as if they were English words. Looking at the issue in Amaya,
I discovered that most (but not all) of these instances are carefully
marked up with <code>, which is definitely appropriate. However, except
in Amaya, where the monospace font is serif (typical Currier style),
browsers (I checked Opera, Firefox, and IE6)
use a monospace font that is extremely close in x-height, width,
and everything else to the rest of the text. And the items themselves
are too short, and contain no letters like m/w or i/l which would
make it easy to distinguish monospace and proportional fonts.
So I think you should look at slightly reinforcing the style
difference.

Regards,   Martin.

At 00:13 07/05/17, Richard Ishida wrote:
><http://www.w3.org/People/Ishida/>http://www.w3.org/People/Ishida/
><http://people.w3.org/rishida/blog/>http://people.w3.org/rishida/blog/

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Richard Ishida" <[email-masked]>
Date: Fri, 18 May 2007 12:59:34 +0100

Hi Martin,

Thanks for these useful comments.  I made all the suggested changes, and I
changed the style so that for keywords the font family is set to "Courier
new", monospaced, rather than just monospaced, and coloured the text brown.
Hopefully that will help with the distinctiveness.

Cheers,
RI

============
Richard Ishida

> -----Original Message-----
> From: Martin Duerst [mailto:[email-masked]] 
> Sent: 18 May 2007 11:28
> To: Richard Ishida; 'Mark Davis'
> Cc: [email-masked]; 'LTRU Working Group'
> Subject: RE: For review: Tagging text with no language
> 
> Hello Richard,
> 
> There is some hickup in the sentence "This is an attempt to 
> capture some ideas in a thread on [email-masked] 
> and a later reprise of that those ideas to which several 
> people contributed."
> 
> I'd also change "attempt to capture some ideas" to something 
> like "summary of discussion". The former doesn't fit well to 
> an FAQ item, and it doesn't reflect the fact that the 
> At the very end, you write: "Martin Du"rst points out that 
> you can redefine the XHTML/HTML format within the document to 
> create an HTML/XHTML page that validates while using lang="" 
> or xml:lang="". This is not recommended for widespread use, however."
> 
> So I'd write something like "However, such a document is no 
> longer strictly conforming in the sense of XHTML 1.0."
> Another point, probably long-term: I found the text visually 
> difficult to read at some points because for me, there is not 
> enough distinction between the plain language of the text and 
> the formal elements that are interspersed. Things such as 
> "zxx" and "und" appear in the middle of the text as if they 
> were English words. Looking at the issue in Amaya, I 
> discovered that most (but not all) of these instances are 
> carefully marked up with <code>, which is definitely 
> appropriate. However, except in Amaya, where the monospace 
> font is serif (typical Currier style), browsers (I checked 
> Opera, Firefox, and IE6) use a monospace font that is 
> extremely close in x-height, width, and everything else to 
> the rest of the text. And the items themselves are too short, 
> and contain no letters like m/w or i/l which would make it 
> easy to distinguish monospace and proportional fonts.
> >I18n core folks, please speak out if you don't feel we 
> should agree to publish this during our telecon next Tuesday.
> > 
> >RI
> > 
> >============
> >Richard Ishida

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 18 May 2007 16:42:44 +0200

Richard Ishida wrote:
 
> I changed the style so that for keywords the font family is set to
> "Courier new", monospaced, rather than just monospaced, and coloured
> the text brown. Hopefully that will help with the distinctiveness.

With my CSS-agnostic browsers the strings marked as <code> are fine.

In the "zxx" section you say "not in any language", maybe add "human",
the "zxx" "no linguistic content" blurb affects only human languages.

Frank

---

From: Najib Tounsi <[email-masked]>
Date: Fri, 18 May 2007 22:25:47 +0000

Hi Richard,

My feedback is perhaps subjective. My feeling is that, in some places, 
the text is not sufficently clear for those who don't speak English 
fluently.

Anyway, here are some remarks (about 
http://www.w3.org/International/questions/qa-no-language#undetermined)

- You write
"For example, xml:lang="" might be used if text is included into a 
document from a database that doesn't provide language information..."
It is the text or the document which is from a database? The text of course.
Should I understand this:
For example, xml:lang="" might be used if text is to be included into a 
document and (the text) comes from a database that doesn't provide 
language information ...?

-You write
"The effect would be to cancel any language information declared higher 
up the hierarchy of elements in the document."
What do "cancel any language" means?
 - remove the language information declared higher up the hierarchy? Wrong
 - override this declaration by the new one "und"? Right

Finally the whole story (about the use of "und") is, if you can "leave 
out the markup", go ahead. Mark up only if "you have a particular need 
to indicate that the language is undefined". Right?

The 1st and 4th § of  
(http://www.w3.org/International/questions/qa-no-language#undetermined) 
say something similar, and could join into one §.

Regards,

Najib

Richard Ishida wrote:
> I18n core folks, please speak out if you don't feel we should agree to 
> publish this during our telecon next Tuesday.
>  
> RI
>  
> ============
> Richard Ishida
>     ------------------------------------------------------------------------
>     *From:* [email-masked]
>     [mailto:[email-masked]] *On Behalf Of *Mark Davis
>     *Sent:* 20 April 2007 21:00
>     *To:* Richard Ishida
>     *Cc:* [email-masked]; LTRU Working Group
>     *Subject:* Re: For review: Tagging text with no language
>
>     Richard, getting back to the topic of XML, it sounds like we have
>     three cases, illustrated by the following example:
>
>     In the first foo construction, we are saying that it inherits the
>     language "en". The second and third construction are equivalent:
>     we are overriding the "en" setting, and saying that foo is
>     undetermined with regard to language. The difference between them
>     is that:
>
>         * the second construction can't be used in XHTML (or any other
>           case where a DTD or schema requires a value).
>     On 4/11/07, *Richard Ishida* <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>
>
>
>         Comments are being sought on this FAQ-based article prior to
>         final release. Please send any comments to
>         [email-masked] <mailto:[email-masked]>. We
>         expect to publish a final version in one to two weeks.
>
>
>
>         ============
>         Richard Ishida
>         <http://people.w3.org/rishida/blog/>

-- 
Najib TOUNSI (mailto:tounsi @ w3.org)
Bureau W3C au Maroc (http://www.w3c.org.ma/)
Ecole Mohammadia d'Ingenieurs, BP 765 Agdal-RABAT Maroc (Morocco)
Phone : +212 (0) 37 68 71 50 (P1711)  Fax : +212 (0) 37 77 88 53
Mobile: +212 (0) 61 22 00 30

---

From: Martin Duerst <[email-masked]>
Date: Sat, 19 May 2007 10:30:03 +0900

Hello Richard, Najib,

At 07:25 07/05/19, Najib Tounsi wrote:
>
>Hi Richard,
>
>My feedback is perhaps subjective. My feeling is that, in some places, the text is not sufficently clear for those who don't speak English fluently.
>
>Anyway, here are some remarks (about http://www.w3.org/International/questions/qa-no-language#undetermined)
>
>- You write
>"For example, xml:lang="" might be used if text is included into a document from a database that doesn't provide language information..."
>For example, xml:lang="" might be used if text is to be included into a document and (the text) comes from a database that doesn't provide language information ...?

Very good point.

>-You write
>"The effect would be to cancel any language information declared higher up the hierarchy of elements in the document."
>Finally the whole story (about the use of "und") is, if you can "leave out the markup", go ahead. Mark up only if "you have a particular need to indicate that the language is undefined". Right?

I was also a bit surprised by this. It's easy to read this as
"language tagging, so who cares?". It looks like it's quite in
contrast to what we say on language tags otherwise.

Regards,    Martin.

---

From: Najib Tounsi <[email-masked]>
Date: Sat, 19 May 2007 10:07:02 +0000

> contrast to what we say on language tags otherwise.
In fact, to what I wanted to point is:

Suppose you have a text like "The speaker said 'Salam Alikoum' and began 
to talk".
You know that this English with something strange inside it. And you 
don't have a particular need to indicate that the strange language is 
undefined.

Which of the following two cases you recommend me to do? In which 
circumstances?

1. leave out the markup:
 <text xml:lang="en"> The speaker said
  <span>Salam Alikoum</span>
  and began to talk
 </text>

2. cancel any language information declared higher up the hierarchy 
using "und" (or xml:lang="",  depending on XML format):
 <text xml:lang="en"> The speaker said
  <span xml:lang="und">Salam Alikoum</span>
  and began to talk
 </text>
or
 <text xml:lang="en"> The speaker said
  <span xml:lang="">Salam Alikoum</span>
  and began to talk
 </text>

Now, if the English is not declared, is this the correct markup:
 <text> The speaker said
  <span>Salam Alikoum</span>
  and began to talk
 </text>

Regards, Najib

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 19 May 2007 15:00:28 +0200

Najib Tounsi wrote:

> Suppose you have a text like "The speaker said 'Salam Alikoum'
> and began to talk".
> You know that this English with something strange inside it.
> And you don't have a particular need to indicate that the
> strange language is undefined.

Does "no particular need" cover "no overzealous spell-checker" ?

Then you could leave it alone, apparently you want speech
browsers to read "Salam Alikoum" like English words.

Here's a similar idea: <div lang="de"> Aleikum Salem </div>

Frank

---

From: "Richard Ishida" <[email-masked]>
Date: Mon, 21 May 2007 19:42:50 +0100

> From: Najib Tounsi [mailto:[email-masked]] 
> Sent: 19 May 2007 11:07
> Suppose you have a text like "The speaker said 'Salam 
> Alikoum' and began to talk".
> Which of the following two cases you recommend me to do? In 
> which circumstances?
> 
> 1. leave out the markup:
> 2. cancel any language information declared higher up the 
> hierarchy using "und" (or xml:lang="",  depending on XML format):

Hi Najib,

Well, if you know that the span is Arabic, you should label it as such.  If
the span said something like "sowejc owkdjocwl eowekj", and the surrounding
text is English, then you may want to add xml:lang="" or xml:lang="und" to
the span if you want to prevent people treating this as English.  If you
leave out the markup on the span, spell checkers, text to speech
applications, etc will assume that the span text is English.

RI

---

From: "Richard Ishida" <[email-masked]>
Date: Mon, 21 May 2007 19:51:35 +0100

Najib and Martin,

Thanks for your comments.  I had another go at the document.
http://www.w3.org/International/questions/qa-no-language

RI

============
Richard Ishida

> Sent: 19 May 2007 02:30
> To: Najib Tounsi; Richard Ishida
> >My feedback is perhaps subjective. My feeling is that, in 
> some places, the text is not sufficently clear for those who 
> don't speak English fluently.
> >
> >"For example, xml:lang="" might be used if text is included 
> into a document from a database that doesn't provide language 
> information..."
> >For example, xml:lang="" might be used if text is to be 
> included into a document and (the text) comes from a database 
> that doesn't provide language information ...?
> 
> Very good point.
> 
> >-You write
> >"The effect would be to cancel any language information 
> declared higher up the hierarchy of elements in the document."
> >Finally the whole story (about the use of "und") is, if you 
> can "leave out the markup", go ahead. Mark up only if "you 
> have a particular need to indicate that the language is 
> undefined". Right?
> 
> "language tagging, so who cares?". It looks like it's quite 
> in contrast to what we say on language tags otherwise.
> 
> Regards,    Martin.
> 
> 
>

---

From: [email-masked]
Date: Mon, 21 May 2007 12:23:04 -0700

Sorry for piping up late on this issue.... 

I still question the practical applications of the "no linguistic content" 
semantic. I thought we had agreed that the most appropriate use of the 
"zxx" tag was to indicate that the association of a language with a piece 
of content is not applicable.  So if I'm classifying an instrumental 
musical work using a standard library cataloging system that is also used 
for lyrical works, I might indicate that the recording is "zxx"; a silent 
film might have a "zxx" audio track. This use of the zxx tag is not 
indicated in the text on the page. Should it be?

Second, I don't think the part number example on the page is useful if the 
intention is to code pages "so that applications such as voice browsers 
... can process that text." If we think about what behavior would be 
expected by a screen reader upon encountering a "zxx" tag, I would expect 
that it would ignore the text inside the tag -- just as it should with, 
say, binary junk. But clearly anyone trying to make sense of the content 
shown on this page would need to "read" those part numbers as well.  The 
same is true for programming code snippets that appear in technical 
tutorials. This is where I think there is a distinction between 
"non-applicable" and "non-linguistic" that is being ignored.

What purpose would your <span> tag in the example serve? While this may do 
the right thing for the spellchecker, this is not the right thing to do 
for a screen reader. 

I have always argued that "no linguistic content" is not appropriate for 
code or part numbers and I think recent examples show why I continue to 
think this is a problematic usage and that the "zxx" semantic should be 
"not applicable."

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 21 May 2007 14:23:26 -0700

I agree with Karen on those points. Some other comments on the text
(outdented)

Question

How do I mark up HTML or XML content for language when I don't know the
language, or the content is non-linguistic?

[Skip to the answer]<http://www.w3.org/International/questions/qa-no-language#answer>
 Background

You should always use attributes to identify the human language of the text,
when known, on the highest possible element of documents in HTML or a format
based on XML, so that applications such as voice browsers, style sheets, and
the like can process that text.

I disagree. There are a great many times when you don't need to tag the
language. An XML document that is used in AJAX may have no need whatsoever
for a language. Moreover, in our experience, the language tags are so often
set wrong on web pages that we have to ignore them completely. We only
really trust the language tags that are attached internally.

Probably better to just phrase this as "When you want to identify the
language of the text, here is how to do it."

In XML-based formats you would usually use the xml:lang attribute, and in
XHTML/HTML the lang and/or xml:lang attributes. (See Declaring Language in
XHTML and HTML <http://www.w3.org/International/tutorials/language-decl/>for
details about language tagging in HTML.)

You can override that initial language setting for a part of the document
that is in a different language, eg. some French quotation in an English
document, by using the same attribute(s) around the relevant bit of text.

Suppose you have some text that is not in any language, such as type
samples, part numbers, illustrations of binary data, etc. How would you say
that this was in no language in particular? Or how about a situation where
you extracted the text from a database and it came with no linguistic
information?
 Answer

There are two parts to the above question.
 When the text is
non-linguistic<http://www.w3.org/International/questions/qa-no-language#nonlinguistic>

Use the subtag zxx when the text is *known to be* not in any language.

This would apply for text such as type samples, part numbers, illustrations
of binary data, etc. The definition of zxx in the Language Subtag Registry
is 'no linguistic content'.

I wouldn't include part numbers. I would include binary data, if what that
binary data represented had no linguistic content.

For example:

<p>Here is a list of part numbers:
<span xml:lang="zxx" lang="zxx">9RUI34 8XOS12 3TYY85</span>.</p>

  When the language is
undetermined<http://www.w3.org/International/questions/qa-no-language#undetermined>

If the XML format you are using supports it, use
xml:lang=""<http://www.w3.org/TR/REC-xml/#sec-lang-tag>,
otherwise use the subtag und.

However, according to the people who define language subtags, you should
only tag text as undetermined if you can't just leave it as is. In practice,
this means that markup described in the previous paragraph should only be
used where the format you are using requires it, or where undetermined text
is embedded in some content that has already been labeled for language in
some way.

These values indicate that we cannot determine, for one reason or another,
what the appropriate language information is, or whether the text is
non-linguistic.

und does not mean "cannot determine". It means "has not determined". Maybe I
could have determined the language, maybe not. That isn't the meaning of the
tag.

For example, xml:lang="" might be used if text is to be included into a
document and the text comes from a database that doesn't provide language
information and you can't be reasonably sure what the language is.

Again "can't be reasonably" is phrasing that doesn't belong here.

The effect would be to prevent any language information declared higher up
the hierarchy of elements in the document from applying to the included
text.

Implications for XHTML/HTML Note that xml:lang="" only works if defined in
the XML schema that describes the format of your document. It is not
appropriate for XHTML because the XHTML DTDs define xml:lang in such a way
that an empty string value for the xml:lang attribute is disallowed. (The
xml:lang attribute takes NMTOKEN values in the schema, so they cannot be
empty.)

You cannot leave the lang attribute empty in HTML, either.

For XHTML and HTML, then, you should use und if you need to express the
undefined nature of some text embedded in a document. Note, again, that on
the very rare occasion when the whole document is in an undefined language
it is better to just not declare the default language of the document.
  By the way

This is a summary of a discussion in a
thread<http://lists.w3.org/Archives/Public/www-international/2005JulSep/0163.html>on
[email-masked], and a later
reprise<http://lists.w3.org/Archives/Public/www-international/2007JanMar/0123.html>of
those ideas to which several people contributed.

Martin Dürst points
out<http://lists.w3.org/Archives/Public/www-international/2007JanMar/0136.html>that
you can redefine the XHTML/HTML format within the document to create
an
HTML/XHTML page that validates while using lang="" or xml:lang="". This is
not recommended for widespread use, however, because such a document is no
longer strictly conforming in the sense of XHTML 1.0.
 Mark

On 5/21/07, [email-masked] <[email-masked]> wrote:
> semantic. I thought we had agreed that the most appropriate use of the "zxx"
> tag was to indicate that the association of a language with a piece of
> content is not applicable.  So if I'm classifying an instrumental musical
> work using a standard library cataloging system that is also used for
> lyrical works, I might indicate that the recording is "zxx"; a silent film
> might have a "zxx" audio track. This use of the zxx tag is not indicated in
> the text on the page. Should it be?
>
> intention is to code pages "so that applications such as voice browsers ...
> can process that text." If we think about what behavior would be expected by
> a screen reader upon encountering a "zxx" tag, I would expect that it would
> ignore the text inside the tag -- just as it should with, say, binary junk.
> But clearly anyone trying to make sense of the content shown on this page
> would need to "read" those part numbers as well.  The same is true for
> programming code snippets that appear in technical tutorials. This is where
> I think there is a distinction between "non-applicable" and "non-linguistic"
> that is being ignored.
>
> the right thing for the spellchecker, this is not the right thing to do for
> a screen reader.
>
> think this is a problematic usage and that the "zxx" semantic should be "not
> applicable."
>
> Regards,
>
> Karen Broome