Subject: [Ltru] Re: For review: Tagging text with no language

From: "Mark Davis" <[email-masked]>
Date: Wed, 11 Apr 2007 14:19:40 -0700

Your conclusion from 639-2 I find extremely odd, since "zxx" is ALSO in a
column that says "English name of language", so by your logic, "zxx" is also
a language, just one with no linguistic content. What we have here is not a
definitive situation, just a conflict between two standards, 639-3 and
639-2. And the important thing for xml:lang is BCP 47, which is based on
639-2. Reinterpreting "und" as being *only somewhat unknown* (that is, that
it can't correctly be applied to non-linguistic content, even if you don't
know that that content is non-linguistic) is definitely a breaking change
for many applications, and I would be strongly opposed to making that change
in BCP 47.

In the real world, it is often very difficult or impossible to tell whether
content is in an unknown language or whether the content is not in any
language at all. Saying that an implementation or protocol is not compliant
if it doesn't correctly make this determination is bizarre. What we have
done in BCP 47 is to say that if the protocol allows it, then one should
omit the language code entirely; otherwise one can use "und".

CCing the LTRU group, since this is relevant to the next version of BCP 47.

Mark

On 4/11/07, John Cowan <[email-masked]> wrote:
>
> Mark Davis scripsit:
>
> > I believe that that is adding an interpretation to "und" which is not
> > borne out by either the source standards, nor in common usage.
>
> ISO 639-2 says merely "Undetermined", but this is placed in a column
> labeled "English name of language", so I think it's fair to read it
> as "Undetermined language".  But ISO 639-3 is, I think, definitive.
> http://www.sil.org/iso639-3/scope.asp#S says (in part):
>
>         The identifier [und] (undetermined) is provided for those
>         situations in which a language or languages must be indicated
>         but the *language* cannot be identified [emphasis added].
>
> By contrast, "zxx" is explained in the next sentence thus:
>
>         The identifier [zxx] (no linguistic content) may be applied in a
>         situation in which a language identifier is required by system
>         definition, but the item being described does not actually
>         contain linguistic content.
>
> In any case, the document I'm commenting on says that "zxx" is
> non-linguistic content, and that "und" and "" are synonymous and
> represent linguistic content.  Whatever "und" may or may not mean,
> I think there's no doubt that "" can be applied to both linguistic
> and non-linguistic content.
>
> --
> You escaped them by the will-death              John Cowan
> and the Way of the Black Wheel.                 [email-masked]
> I could not.  --Great-Souled Sam                http://www.ccil.org/~cowan
>

-- 
Mark
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Wed, 11 Apr 2007 17:39:12 -0400

Mark Davis scripsit:

> Your conclusion from 639-2 I find extremely odd, since "zxx" is ALSO
> in a column that says "English name of language", so by your logic,
> "zxx" is also a language, just one with no linguistic content.

Bah.  I'll go no further with this.  My basic point remains:
whatever "und" means, "" can't mean "language is unknown", since it's
equally suitable for non-linguistic content.  XSLT documents don't
have xml:lang attributes, but if they did, "" would be a sensible value.

-- 
BALIN FUNDINUL          UZBAD KHAZADDUMU        [email-masked]
BALIN SON OF FUNDIN     LORD OF KHAZAD-DUM      http://www.ccil.org/~cowan

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 11 Apr 2007 15:52:43 -0700

I agree that "" should be treated as a sensible value, with the same meaning
as "und"; that is, an undetermined / unspecified value.

Mark

---

From: Jon Hanna <[email-masked]>
Date: Thu, 12 Apr 2007 11:46:01 +0100

To say that UND cannot apply to an item which *may* contain no 
linguistic data seems to be putting a bit of a limit on what 
"Undetermined" means.

"Determined to be some sort of language, but just which language remains 
undetermined" is not the same as "undetermined".

That cat is neither dead or alive.

---

From: Jon Hanna <[email-masked]>
Date: Thu, 12 Apr 2007 11:46:17 +0100

That cat is neither dead or alive until determined to be one or the other.

---

From: "Richard Ishida" <[email-masked]>
Date: Thu, 12 Apr 2007 15:35:15 +0100

Ok, I just found some additional emails between John and Mark that had been
scooped into my LTRU folder by my mail client because [LTRU] was prepended
to the subject. They shift the baseline for discussion and agreement.  

What I now see as the summary of where we are is:

[a] Determined, and a language (for which a subtag exists): <subtag(s)>

[b] Determined, and not a language: zxx

[c] Determined, but not a language for which a subtag exists: ???

[d] Undetermined, and not sure whether it is a language or not: xml:lang=""
if available, otherwise und

I have revised the article at [1] to make it clearer that whether text is
linguistic or not is unimportant wrt use of '' and und.

I'm still troubled however by the passage in RFC 4646, however, so I'll
repeat those comments here and copy the LTRU folks:

I'm also a little worried about the wording in section 4.1 of RFC 4646[2]
about und, which quite clearly says that you shouldn't use und unless the
*protocol* demands it, or sometimes when matching tags.  This doesn't make
any distinction between specifying the language of a resource and turning
off language declarations for a range of embedded text.  It seems that this
suggests a way in which xml:lang='' and xml:lang="und" are not equivalent,
since there are no such restrictions on xml:lang="". In my opinion, the text
of RFC 4646 needs some work, both to relax the use of und in scenarios where
'undefined text' occurs in a context with a defined language, and to clarify
the relationship of und to xml:lang=''.

RI 

[1] http://www.w3.org/International/questions/qa-no-language#answer

[2] http://www.rfc-editor.org/rfc/rfc4646.txt

============
Richard Ishida
Internationalization Lead
W3C (World Wide Web Consortium)
 
http://www.w3.org/People/Ishida/
http://www.w3.org/International/
http://people.w3.org/rishida/blog/
http://www.flickr.com/photos/ishida/

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Richard Ishida
> Sent: 12 April 2007 14:56
> To: [email-masked]
> Subject: RE: For review: Tagging text with no language
> 
> 
> So it seems the alternatives that John is suggesting are:
> 
> Determined, and a language (for which a subtag exists): <subtag(s)>
> Determined, and not a language: zxx
> Determined, but not a language for which a subtag exists: ???
> 
> Undetermined, and not sure whether it is a language or not: 
> xml:lang="" (if
> available)
> Undetermined, but sure that it's a language: und
> 
> The implications of this for X/HTML are that there is no way 
> to say that
> text is undetermined if you are not sure whether it's a 
> language or not.
> 
> This is very different from Jon Hanna's proposal at
> http://lists.w3.org/Archives/Public/www-international/2007JanM
ar/0178.html
> 
> Can we please discuss this.  I'm particularly hoping for 
> contributions from
> John, Jon, Mark, Martin and Addison (though he's on vacation 
> at the moment).
> 
> For my part, having experienced, even when trying to write 
> this email, how
> difficult it is to succinctly talk about the difference 
> between something
> that is unidentified and may or may not be a language, I'm a 
> little leery
> about accepting the evidence in the mail below, John.  Can we 
> be sure that
> the people who drafted that text were conciously making the 
> distinction you
> mention rather than just being a little imprecise in wording?
> 
> I'm also a little worried about the wording in section 4.1 of 
> RFC 4646[1]
> about und, which quite clearly says that you shouldn't use 
> und unless the
> *protocol* demands it, or sometimes when matching tags.  This 
> doesn't make
> any distinction between specifying the language of a resource 
> and turning
> off language declarations for a range of embedded text.  It 
> seems that this
> suggests another way in which xml:lang='' and xml:lang="und" are not
> equivalent. In my opinion, either the text of RFC 4646 needs 
> some work,
> either to relax the use of und in scenarios where undefined 
> text occurs in a
> context that is defined, or to clarify the relationship of und to
> xml:lang=''.
> 
> RI
> 
> 
> [1] http://www.rfc-editor.org/rfc/rfc4646.txt
> 
> ============
> Richard Ishida
> Internationalization Lead
> W3C (World Wide Web Consortium)
>  
> http://www.w3.org/People/Ishida/
> http://www.w3.org/International/
> http://people.w3.org/rishida/blog/
> http://www.flickr.com/photos/ishida/
>  
>  
> 
> > -----Original Message-----
> > From: [email-masked] 
> > [mailto:[email-masked]] On Behalf Of John Cowan
> > Sent: 11 April 2007 21:24
> > To: Mark Davis
> > Cc: John Cowan; CE Whitehead; [email-masked]
> > Subject: Re: For review: Tagging text with no language
> > 
> > 
> > Mark Davis scripsit:
> > 
> > ISO 639-2 says merely "Undetermined", but this is placed in a 
> > column labeled "English name of language", so I think it's 
> > fair to read it as "Undetermined language".  But ISO 639-3 
> > is, I think, definitive.
> > In any case, the document I'm commenting on says that "zxx" 
> > is non-linguistic content, and that "und" and "" are 
> > synonymous and represent linguistic content.  Whatever "und" 
> > may or may not mean, I think there's no doubt that "" can be 
> > applied to both linguistic and non-linguistic content.
> > 
> > --

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 09:13:09 -0700

The summary looks good. This discussion raises 2 items for the LTRU group.

Q1. What tag should be used where it is definitely a language, but there is
no code available yet? (This is an area where ISO 15924 is ahead of ISO 639
(and 3166), since it has Zzzz: Code for uncoded script.)

Q2. Clarify the wording around "und" vs "".

Mark

---

From: John Cowan <[email-masked]>
Date: Thu, 12 Apr 2007 13:00:13 -0400

Mark Davis scripsit:

> Q1. What tag should be used where it is definitely a language, but there
> is no code available yet? (This is an area where ISO 15924 is ahead
> of ISO 639 (and 3166), since it has Zzzz: Code for uncoded script.)

In principle, every natural-language item (text, audio, video) can be
coded with some 639-2 code; if the language does not have a code of its
own, it will belong to one of the 639-2 collections.

For example, the language Tarifit (639-3 code 'rif') does not have a 639-2
code, but it is a Berber language; consequently, an item in Tarifit may be
validly tagged 'ber', which represents the collection of Berber languages.
Similarly, the language Zumbun (639-3 code 'jmb') does not have an 639-2
code, nor does it belong to any of the smaller 639-2 collections, but it
does belong to the Afro-Asiatic language family; consequently, an item
in Zumbun may be validly tagged 'afa', which represents the collection
of Afro-Asiatic languages.

If all else fails, as for the language isolate Burushaski (639-3 code
'bsk'), the 639-2 collection code 'mis', representing the collection of
miscellaneous languages, may be applied.  This is the ultimate fallback
code, indicating that the language is known but nothing useful can be
said about it using 639-2 codes.

All of this lore, which represents the practice of the Library of Congress
(the ultimate source of 639-2), can of course go away when RFC 4646bis
goes into effect.  If it is necessary to be more specific before then,
and if strict compliance to 4646 is required, then rif-x-tarifit,
afa-x-jumbun, and mis-x-burushas may also be used.

> Q2. Clarify the wording around "und" vs "".

"" is not a well-formed language tag according to RFC 4646, so there is
nothing to say about it there.  It is defined by the XML Recommendation as
an extension to the set of language tags, and having the same significance
as no language declaration at all.

-- 
Dream projects long deferred             John Cowan <[email-masked]>
usually bite the wax tadpole.            http://www.ccil.org/~cowan
        --James Lileks

---

From: Martin Hosken <[email-masked]>
Date: Fri, 13 Apr 2007 00:06:17 +0700

Dear All,

>> of ISO 639 (and 3166), since it has Zzzz: Code for uncoded script.)

A related question is how to tag text that is definitely in a language
but I don't know what the language is. (But I might know the script).

Yours,
Martin

---

From: Marion Gunn <[email-masked]>
Date: Thu, 12 Apr 2007 18:19:49 +0000

Good question, albeit not a new one.:-)
mg

On 12 Apr 2007, at 17:06, scríobh Martin Hosken:
> ... A related question is how to tag text that is definitely in a  
> language

- -
Marion Gunn * EGTeo (Estab.1991)
27 Páirc an Fhéithlinn, Baile an
Bhóthair, Co. Átha Cliath, Éire.
* [email-masked] * [email-masked] *

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 10:23:40 -0700

For that, I'd suggest und-Latn (or whatever the script is). Since only
languages would have scripts, that is sufficiently determinate.

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Thu, 12 Apr 2007 20:29:18 +0300 (EEST)

On Thu, 12 Apr 2007, Mark Davis wrote:

[ about text that is in unknown language but known script ]

> languages would have scripts, that is sufficiently determinate.

I'm not so sure about it; it depends on what "script" really means. If the 
data is "JuUiYTlajUJO", which is not in any language as far as I know, 
can't we still say that it is in the Latin script?

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 10:29:20 -0700

Q1. I had missed the choice of "mis". I agree with that suggestion; we
should incorporate that into 4646bis. The problem is ameliorated
considerably once we add -3, but it doesn't disappear completely, so "mis"
remains a good choice for dealing with that situation.

Q2. The issue *does* remain, since we talk about "und" vs the absence of a
language tag, which "" represents.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 10:34:13 -0700

I was answering the question "A related question is how to tag text that is
definitely in a language but I don't know what the language is. (But I might
know the script).", assuming that one knows the script.

But after seeing John's suggestion, a better choice might be "mis-Latn" (if
one knows that it is some language but not sure which or not able to encode,
written in Latin) and "mis" if one knows that it is some language (but
doesn't know the script).

Mark

---

From: John Cowan <[email-masked]>
Date: Thu, 12 Apr 2007 14:56:29 -0400

Mark Davis scripsit:

> Q1. I had missed the choice of "mis". I agree with that suggestion;
> we should incorporate that into 4646bis. The problem is ameliorated
> considerably once we add -3, but it doesn't disappear completely, so
> "mis" remains a good choice for dealing with that situation.

I did not mean to suggest that "mis" is suitable in cases of ignorance
about the language in use: it is not a fallback *language* code.  Rather,
it is a fallback language *collection* code, suitable for languages that
don't appear in any other ISO 639-2 collection.  By the Ethnologue's
count, there are about 130 of these.

So it would be an error to tag a language you didn't recognize as 'mis',
because it is far more likely to be one of the non-'mis' languages, for
the same reason that it would be incorrect to use 'en' or 'nds' or 'afa'.
If you want a completely vague language tag, use 'und' (excluding for
the moment the question of whether non-linguistic content not recognized
as such can be tagged 'und').

> Q2. The issue *does* remain, since we talk about "und" vs the absence
> of a language tag, which "" represents.

I still don't see that there's anything more to say than we are
saying already, which is just a special case of "Tag wisely".

-- 
I now introduce Professor Smullyan,             John Cowan
who will prove to you that either               [email-masked]
he doesn't exist or you don't exist,            http://www.ccil.org/~cowan
but you won't know which.                               --Melvin Fitting

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 12 Apr 2007 23:25:53 +0200

FWIW, in CLDR 1.4 some of the translations for "und" has the word "language"
(translated of course) in them, in accordance with John Cowan's original suggestion:

da.xml:			<language type="und">Sproget kan ikke bestemmes</language>
de.xml:			<language type="und">Sprache nicht ermittelt</language>
it.xml:			<language type="und">lingua imprecisata</language>
sv.xml:			<language type="und">obestamt sprak</language>

(I would be to blame for the last one, but apparently I'm not the only one to (maybe)
be misguided). Perhaps those ones should be retranslated not to refer to language,
**if** "und" may apply also to "maybe not in any language".

		/kent k

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 15:06:47 -0700

Actually, we do have the ability to give fairly detailed messages based on
regular expression matches on the XML path in a "zoomed" view, and soon we
will have the ability to require people to go to the zoomed view before
editing, and thus see those messages. So we can add specific clarifications
on the use of "und" or other special cases.

Mark

On 4/12/07, Asmus Freytag <[email-masked]> wrote:
>
> On 4/12/2007 2:25 PM, Kent Karlsson wrote:
> >
> The sample translations show that there's general difficulty in agreeing
> on the concept. The German translation says "no language (has been)
> determined", while the Danish translation says that "no language could
> be determined". In my reading the Swedish allows both possibilities, but
> perhaps implies more strongly than the other two that assigning a
> language to the contents would be meaningful. (The Italian translation
> seems to most closely agree with the Swedish one to the extent of my
> command of Italian)
>
> The problem is that the scheme does not explicitly accounts for all the
> types of edge conditions that you can get into when analyzing text for
> language up front. Instead, labels are added here and there to handle
> some of these as they become urgent enough to require attention. As a
> result, all the translators have to go by is the shorthand English
> description for the label. And that's not written with enough precision
> to overcome the limitation of not having thought through all the
> possible cases.
>
> A./
>
>

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 00:09:59 +0200

> sv.xml: <language type="und">obestamt sprak</language>

For the German version I'd say that's "language not identified".
 
> Perhaps those ones should be retranslated not to refer to language,

I don't understand *_why_* anything about it should be changed.

If the process to identify / determine a language failed so far,
the final outcome can be still "no language at all", i.e. "zxx".

And that's precisely what Richard needs for those cases where an
empty string is (still) syntactically invalid, as in XHTML 1.0.

  <blockquote lang="en" xml:lang="en">
  Our sole example of pure Black Speech, then, is the inscription
  on the Ring: <q lang="und" xml:lang="und"> Ash nazg durbatulûk,
  ash nazg gimbatul, ash nazg thrakatulûk agh burzum-ishi krimpatul.</q>
  "One Ring to rule them all, One Ring to find them,  One Ring to
  bring them all and in the Darkness bind them." (LotR1/II ch. 2)
  </blockquote> 

John would likely have better ideas to tag this, but Richard's
article is for ordinary users.

Frank

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 13 Apr 2007 01:26:29 +0200

Stephen Deach wrote: 

> sv.xml:                       <language type="und">obestämt språk</language>

I thought "obestamt" was "unstated". 

"Obestämt" literally means "undetermined". "Unstated" would be "osagt", "outtalat", or "ej angett"
("not given", closer to the current German translation).
 
Though I would agree that xml:lang="" is closer to "unstated" than "undetermined". I'm not sure
that that nit-picking leads anywhere in this case. But "unstated" is not the same as "undetermined";
it may well be determined, but just not stated... So maybe there is a difference worth bothering about.
 
        /kent k

---

From: John Cowan <[email-masked]>
Date: Thu, 12 Apr 2007 19:26:07 -0400

> article is for ordinary users.

For automated tagging, "und" is probably the best you can do indeed,
unless you set up your automated tagger to detect precisely that
word sequence.  For manual tagging, I'd go with "art-x-bspeech" for
now, or something like that.

-- 
But that, he realized, was a foolish            John Cowan
thought; as no one knew better than he          [email-masked]
that the Wall had no other side.                http://www.ccil.org/~cowan
        --Arthur C. Clarke, "The Wall of Darkness"

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 12 Apr 2007 18:15:02 -0700

I think I agree with you in spirit, but not in precise details. The tag
"und" means "undetermined", so when I encounter it I don't know whether the
content contains one language, many languages, or no language. The tag "zxx"
would mean that there is no language content, "mis" would mean that there is
at least some language content, and "mul" would mean that there is language
content, with more than one language.

I think to try to consider what the motivations of the tagger are may lead
to misleading impressions. Assume for the moment that the tag is correct.
From the perspective of the tagger, using "und" could mean, as you say, that
the tagger doesn't know or care (or want to communicate, or what to spend
the time to determine) what the language is or whether there is any language
content there at all. There could be quite a variety of motivations for the
tagger's using "und"; the key is what the reader of "und" can assume about
the content, which is essentially nothing. With "mis", the situation is
similar, but slightly narrower. The tagger may still not know much, or care
much, but maybe cared enough to determine that there was something there, or
maybe there was language content there, but there is no language code that
correctly matches it (protogermanic, perhaps).

Similarly, using "may not" language is a bit too strong in your phrase
"Whereas zxx says I 'may not' apply any of those language-based services
because it is not a 'natural' language". Having content tagged with "zxx"
doesn't restrict me from doing anything I want to; it just means if it was
tagged correctly, it does not contain any language content. (I might decide
that the tagger was mistaken -- when we at Google look at the tagging people
actually do of web content, there is a fairly high percentage of both
invalid tags and valid-but-incorrect tags.)

Mark

On 4/12/07, Stephen Deach <[email-masked]> wrote:
>
>  I think much of this discussion is dealing with terminology differences
> that are so narrow that one is discussing "the number of angels who can
> dance on the head of a pin". (In other words we are debating theology, not
> practice.) In reality, specifications are worded as carefully as possible,
> but interpretation is open to the reader's most common
> definition/redefinition/translation of the exact terminology.  -- So rather
> than debate what the "exact meaning" of a word/phrase is in each of these
> languages, maybe we should take a looser interpretation of what is written
> and then clarify the intent.
>
> My reading of the ISO spec is that "und/undetermined" means "I don't know
> (or care, or am unwilling to state) what the language is (and have no closer
> alternative language identifier given the available options)". From a
> practical viewpoint, "und" indicates I can't assume any specific/preferred
> linguistic definitions for words in the content, nor can I assume any
> specific/preferred pronunciation-, spelling-, hyphenation-, and/or
> grammar-rules on the content; though I am allowed to attempt my own
> linguistic analysis to guess at the language. (Whereas zxx says I 'may not'
> apply any of those language-based services because it is not a 'natural'
> language and should not attempt any linguistic analysis to guess at the
> language.) I can't see any practical difference between "und" and "" (except
> that "" is disallowed in some processing environments) so why can't the
> documents simply say that 'a missing specification' or 'xml:lang=""' (should
> either occur), will be interpreted as "und".
>
>
> It has been a while since I considered myself fluent in Swedish (and I
> intentionally ignored the lack of the dieresis in the original text as an
> indication that the translations were "lossy"). I just thought that some
> comment would force the necessary clarification of the translations.
>
>
> At 2007.04.13-01:26(+0200), Kent Karlsson wrote:
>
> Stephen Deach wrote:
>
> ---Steve Deach
>    [email-masked]

---

From: John Cowan <[email-masked]>
Date: Thu, 12 Apr 2007 22:04:23 -0400

Mark Davis scripsit:

> I think I agree with you in spirit, but not in precise details. The
> tag "und" means "undetermined", so when I encounter it I don't know
> whether the content contains one language, many languages, or no
> language. The tag "zxx" would mean that there is no language content,
> "mis" would mean that there is at least some language content, and "mul"
> would mean that there is language content, with more than one language.

I'm okay with all of this except "mis".  "mis" is a collection code,
as I explained, and means "languages that don't belong to any other
collection."  It is not the universal collection.

-- 
Mark Twain on Cecil Rhodes:                    John Cowan
I admire him, I freely admit it,               http://www.ccil.org/~cowan
and when his time comes I shall                [email-masked]
buy a piece of the rope for a keepsake.

---

From: "CE Whitehead" <[email-masked]>
Date: Thu, 12 Apr 2007 23:10:57 -0400

Hi, my comments are below; I still sort of think that there may need to be some clarification about mis and und.

>empty string is (still) syntactically invalid, as in XHTML 1.0.

Frank,  the question for me is still: when is und recommended over "zxx"?

(I personally still want a clearer meaning for und; the meaning of zxx is clear to me)

Also I have questions about mis (miscellaneous)  (I agree with John's definition of miscellanous; that's the English definition; but there is nothing in the registry to indicate that mis refers to any collection of languages--it's just a three-letter subtag--so I have to say that what I get for mis is exactly what Mark gets,  maybe some language content; if so and if zxx means no known language content, then und may be redundant (caution:  this is an outside opinion from someone with limited programming skills).

Here's the registration for mis:

-------------------------------------------------------------------------

Type: language

Subtag: mis

Description: Miscellaneous languages

Added: 2005-10-16

------------------------------------------------------------------------------

Mark Davis [email-masked]

> I think I agree with you in spirit, but not in precise details. The tag "und" means

> "undetermined", so when I encounter it I don't know whether the content contains > one language, many languages, or no language.

+1 that is what I get from this, too, exactly.

> The tag "zxx" would mean that

> there is no language content, "mis" would mean that there is at least some language > content, and "mul" would mean that there is language content, with more than one > language.

John Cowan http://www.ccil.org/~cowan

> I'm okay with all of this except "mis". "mis" is a collection code, as I explained, and means "languages that

> don't belong to any other collection." It is not the universal collection.

--C. E. Whitehead

[email-masked]

Exercise your brain! Try Flexicon.

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 00:18:05 -0400

CE Whitehead scripsit:

> Also I have questions about mis (miscellaneous) 
> (I agree with John's definition of miscellanous; that's the English
> definition; but there is nothing in the registry to indicate
> that mis refers to any collection of languages--it's just a
> three-letter subtag--so I have to say that what I get for mis is
> exactly what Mark gets,  maybe some language content; 

It comes from ISO 639-2, and when English names from 639-2 contain
"languages" or "(Other)", then they are collections.  See the normative
text at http://www.loc.gov/standards/iso639-2/normtext.html section 4.1.1.
Even though the registry doesn't contain semantics, the subtags there
inherit their semantics from the corresponding ISO codes.

-- 
Normally I can handle panic attacks on my own;   John Cowan <[email-masked]>
but panic is, at the moment, a way of life.      http://www.ccil.org/~cowan

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 00:41:02 -0400

Stephen Deach scripsit:

> "mis" means it is "a language but I have no better identifier for it".

Well, "a known language for which no better identifier exists".  It would
be, as I keep saying, an error to tag German ("de") or Kiowa ("nai" or
"nai-x-kio" or "nai-x-kiowa") as "mis".

> One can have a separate debate over whether "zxx" or "art" should be
> used for computer-programming languages, or whether computer-programming
> (as a group or individually) deserve their own tag(s); but that is
> not an "Internationalization" issue.

BCP 47 explicitly excludes computer languages from its scope, as do the
ISO 639 family of standards.  So "zxx" is the only available tag.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
It's the old, old story.  Droid meets droid.  Droid becomes chameleon.
Droid loses chameleon, chameleon becomes blob, droid gets blob back
again.  It's a classic tale.  --Kryten, Red Dwarf

---

From: Peter Constable <[email-masked]>
Date: Thu, 12 Apr 2007 22:23:45 -0700

Agreed - seems pretty obvious to me.

Peter

________________________________
From: Mark Davis [mailto:[email-masked]]
Sent: Thursday, April 12, 2007 10:24 AM
To: Martin Hosken
Cc: LTRU Working Group; [email-masked]
Subject: Re: [Ltru] RE: For review: Tagging text with no language

For that, I'd suggest und-Latn (or whatever the script is). Since only languages would have scripts, that is sufficiently determinate.
On 4/12/07, Martin Hosken <[email-masked]<mailto:[email-masked]>> wrote:
Dear All,

Yours,
Martin

---

From: Peter Constable <[email-masked]>
Date: Thu, 12 Apr 2007 22:26:12 -0700

There is a difference between 'no information provide', and 'information is provided: this is unknown'.

I don't see what the issue is; I guess I must have missed the start of this thread.

Peter

________________________________
From: Mark Davis [mailto:[email-masked]]
Sent: Thursday, April 12, 2007 10:29 AM
To: John Cowan
Cc: [email-masked]; LTRU Working Group
Subject: Re: [Ltru] RE: For review: Tagging text with no language

Q1. I had missed the choice of "mis". I agree with that suggestion; we should incorporate that into 4646bis. The problem is ameliorated considerably once we add -3, but it doesn't disappear completely, so "mis" remains a good choice for dealing with that situation.

Q2. The issue *does* remain, since we talk about "und" vs the absence of a language tag, which "" represents.

Mark
On 4/12/07, John Cowan <[email-masked]<mailto:[email-masked]>> wrote:

--
Dream projects long deferred             John Cowan < [email-masked]<mailto:[email-masked]>>
usually bite the wax tadpole.            http://www.ccil.org/~cowan
        --James Lileks

---

From: Peter Constable <[email-masked]>
Date: Thu, 12 Apr 2007 22:27:37 -0700

+1

Peter

> From: John Cowan [mailto:[email-masked]]
> Sent: Thursday, April 12, 2007 11:56 AM
> To: Mark Davis

---

From: Peter Constable <[email-masked]>
Date: Thu, 12 Apr 2007 22:41:45 -0700

ISO 639 indicates that programming languages are out of scope. I interpret that to mean that no programming language or group of programming languages is positively represented - i.e. none of the entities represented (either individually or as part of a group) is a programming language. I interpret "zxx" to mean "the content so tagged is not any instance of the kind of entities encompassed by this coding standard". That would entail that "zxx" could appropriately be applied to content that is in a programming language with as much appropriateness as applying it to a part number or random text or an empty file or telemetry from a space probe: you may or may not be able to interpret the content, but you certainly cannot interpret it in terms of any human language.

Peter

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Stephen Deach
Sent: Thursday, April 12, 2007 8:23 PM
To: Mark Davis; Stephen Deach
Cc: Kent Karlsson; Asmus Freytag; John Cowan; Richard Ishida; LTRU Working Group; [email-masked]; CLDR list
Subject: Re: [Ltru] RE: For review: Tagging text with no language

My point was intended to be that this started out as a debate on the interpretation of "". I don't see that any existing tag other than "und" makes any sense as an alternate interpretation of "", so why the long debate and the discussion of the various re-translations of what "und" means in Danish, Swedish, German, Italian, et al. (At the nuance level of this discussion, such translations are ALWAYS imperfect.)
  I have no objection to your interpretations of what happens if someone explicitly says "mis", "mul", etc.
  I woudl note that "und" makes no statement of whether the content is a single or several languages, only that any/all language information is "not defined" for the specified scope.

If the user says nothing or says xml:lang="", it can't mean anything other than "undefined" (for which the tag is "und") or "unspecified" (for which there is no tag; regardless of whether the user intent was "I don't know" vs "I don't care" vs "I don't want to guess" vs "I don't want to say"; and regardless of if it is all a single language or a mixture of several languages). The tag "und" means that no input regarding the language is (languages are) provided. Why not say no xml:lang specifier and xml:lang="" are both interpreted as xml:lang="und" and be done with it.

It is quite clear that "zxx" means "I know this is not a linguistic"; that "art" means it is "an artificial language (invented or other non-natural language)"; that "mul" means "there are a mixture of languages which I may or may not choose to identify at a lower level in the document"; and that "mis" means it is "a language but I have no better identifier for it".

One can have a separate debate over whether "zxx" or "art" should be used for computer-programming languages, or whether computer-programming (as a group or individually) deserve their own tag(s); but that is not an "Internationalization" issue.

At 2007.04.12-18:15(-0700), Mark Davis wrote:

I think I agree with you in spirit, but not in precise details. The tag "und" means "undetermined", so when I encounter it I don't know whether the content contains one language, many languages, or no language. The tag "zxx" would mean that there is no language content, "mis" would mean that there is at least some language content, and "mul" would mean that there is language content, with more than one language.

I think to try to consider what the motivations of the tagger are may lead to misleading impressions. Assume for the moment that the tag is correct. From the perspective of the tagger, using "und" could mean, as you say, that the tagger doesn't know or care (or want to communicate, or what to spend the time to determine) what the language is or whether there is any language content there at all. There could be quite a variety of motivations for the tagger's using "und"; the key is what the reader of "und" can assume about the content, which is essentially nothing. With "mis", the situation is similar, but slightly narrower. The tagger may still not know much, or care much, but maybe cared enough to determine that there was something there, or maybe there was language content there, but there is no language code that correctly matches it (protogermanic, perhaps).

Similarly, using "may not" language is a bit too strong in your phrase "Whereas zxx says I 'may not' apply any of those language-based services because it is not a 'natural' language". Having content tagged with "zxx" doesn't restrict me from doing anything I want to; it just means if it was tagged correctly, it does not contain any language content. (I might decide that the tagger was mistaken -- when we at Google look at the tagging people actually do of web content, there is a fairly high percentage of both invalid tags and valid-but-incorrect tags.)

Mark

On 4/12/07, Stephen Deach <[email-masked]<mailto:[email-masked]>> wrote:
I think much of this discussion is dealing with terminology differences that are so narrow that one is discussing "the number of angels who can dance on the head of a pin". (In other words we are debating theology, not practice.) In reality, specifications are worded as carefully as possible, but interpretation is open to the reader's most common definition/redefinition/translation of the exact terminology.  -- So rather than debate what the "exact meaning" of a word/phrase is in each of these languages, maybe we should take a looser interpretation of what is written and then clarify the intent.
My reading of the ISO spec is that "und/undetermined" means "I don't know (or care, or am unwilling to state) what the language is (and have no closer alternative language identifier given the available options)". From a practical viewpoint, "und" indicates I can't assume any specific/preferred linguistic definitions for words in the content, nor can I assume any specific/preferred pronunciation-, spelling-, hyphenation-, and/or grammar-rules on the content; though I am allowed to attempt my own linguistic analysis to guess at the language. (Whereas zxx says I 'may not' apply any of those language-based services because it is not a 'natural' language and should not attempt any linguistic analysis to guess at the language.) I can't see any practical difference between "und" and "" (except that "" is disallowed in some processing environments) so why can't the documents simply say that 'a missing specification' or 'xml:lang=""' (should either occur), will be interpreted as "und".

It has been a while since I considered myself fluent in Swedish (and I intentionally ignored the lack of the dieresis in the original text as an indication that the translations were "lossy"). I just thought that some comment would force the necessary clarification of the translations.

Stephen Deach wrote:
> sv.xml:                       <language type="und">obestÃ¤mt sprÃ¥k</language>

I thought "obestamt" was "unstated".
"ObestÃ¤mt" literally means "undetermined". "Unstated" would be "osagt", "outtalat", or "ej angett"
("not given", closer to the current German translation).

        /kent k

---Steve Deach
   [email-masked]<mailto:[email-masked]>

--
Mar

---Steve Deach
   [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 12:48:23 +0200

John Cowan wrote:

> BCP 47 explicitly excludes computer languages from its scope, as do
> the ISO 639 family of standards.  So "zxx" is the only available tag.

Tagging source code snippets as "zxx" would be barbaric.  But it's a
case where "" is clearly better than "und".  Actually I think "" is
always better than "und" unless I intend to flag something for later
review.  In the context of Richard's article and XML documents, for
other purposes it might be different.  The use of "und" in XHTML 1.0
is IMO only a temporary kludge until the DTD is fixed.

Doug argued that "" is a placeholder, I think it's not in XML, it has
a clear effect of breaking any inherited xml:lang, resetting anything
interested in language tags (CSS, spell checkers, Web crawlers, etc.)
to their default "no language specified" behaviour.

It's IMO perfectly okay to have very different styles for "i-default",
"und", "zxx", "art", "mul", etc.   With "" meaning "none of those".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 12:56:46 +0200

John Cowan wrote:

> Even though the registry doesn't contain semantics, the subtags
> there inherit their semantics from the corresponding ISO codes.

Your "mis" explanation sounded interesting enough to deserve a
comment in the registry, something along the line of "whatever
you think 'mis' is, it's most likely wrong".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 13:52:41 +0200

LANGUAGE SUBTAG REGISTRATION FORM
1. Name of requester:                   Frank Ellermann
2. E-mail address of requester:         [email-masked]
3. Record Requested:

   Type:                                language
   Subtag:                              mis
   Description:                         Miscellaneous Languages
   Comments:                            A collection of languages which
                                        don't belong to any other collection

4. Intended meaning of the subtag:      n/a (no change)
5. Reference to published description
   of the language (book or article):   http://permalink.gmane.org/gmane.ietf.ltru/7478
6. Any other relevant information:      n/a (comment added)

---

From: Jon Hanna <[email-masked]>
Date: Fri, 13 Apr 2007 14:27:15 +0100

Frank Ellermann wrote:
> Tagging source code snippets as "zxx" would be barbaric.

Source code snippets often do contain no linguistic data.

However the following is in English:

alert("This is English");//This comment is English too.

> to their default "no language specified" behaviour.

Agreed.

> "und", "zxx", "art", "mul", etc.   With "" meaning "none of those".

I still don't buy that "" != "und".

If "" != "und" then the RFC is buggy, since it clearly requires that the 
latter not be used if a protocol permits the former - a decision which 
only makes sense if they are equivalent.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 16:07:33 +0200

> alert("This is English");//This comment is English too.

Maybe.  It also depends on what you're doing.  If your task
is "translation" then...

  alarm("Das is deutsch");//Dieser Kommentar ist auch deutsch

...is likely no working script anymore.  Or in the case of
ABNF in an Internet-Draft a translator better stays away from
it, including comments.  I'd use xml:lang="" or maybe -- if I
could take advantage of it with XSLT -- xml:lang="i-default".

Theoretical example, in practice I care only about the ASCII
output at the moment, meta-data details are a waste of time
for this output style.  But it could interest translators, if
they get hold of the XML source.

> I still don't buy that "" != "und".

I think it depends on the context.  In a context where either
"und" or "" are allowed they are semantically identical to
their syntactically invalid counterpart.

In a context where both are allowed you're free to assign a
meaning to "und" slightly different from "" (roughly RESET).

> If "" != "und" then the RFC is buggy, since it clearly
> requires that the latter not be used if a protocol permits
> the former - a decision which only makes sense if they are
> equivalent.

I think SHOULD NOT means that you need a very good excuse for
using it anyway.  One good excuse would be the NMTOKEN in the
XHTML 1.0 DTDs, that's explicitly listed as MAY in the RFC.

Another excuse could be "I need it for some CSS magic", but
the RFC doesn't say that this is a _good_ excuse.  Obviously
a good excuse are old documents written before the NMTOKEN
was replaced by a CDATA in some version of XML.  Or any old
tool designed to produce old XML.

It's "only" a SHOULD NOT, so consumers are not permitted to
crash and burn if somebody (ab)uses "und".  With a MUST NOT
I'd agree that "und" would be always wrong, and "abort with
error" would be a perfect implementation for a consumer.

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 07:57:25 -0700

"mis" is defined in 639-2 as "Miscellaneous languages". That does not mean
that it is limited to "languages that don't belong to any other collection".
You interpretation also breaks stability, since I could validly tag content
today with "mis", which would become invalid under your interpretation at
some point in the future.

On http://www.loc.gov/standards/iso639-2/php/code_list.php it is not listed
with "(other)", so it is not a collection.

> I interpret "zxx" to mean "the content so tagged is not any instance of
the kind of entities encompassed by this coding standard".

That is not born out by the name on
http://www.loc.gov/standards/iso639-2/php/code_list.php, which says " No
linguistic content". It does not say "no linguistic content that could
otherwise be represented by a code in this standard", a very different
thing.

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 09:24:25 -0700

I always like to think of these kinds of issues by looking at examples,
since it tends to focus the issues and make it clear when people are
misinterpreting others' terminology. I put out below some examples of what a
process should do if gets a stream of information and is to tag it, where we
assume that it is doing the best job it can. People can comment on these or
propose others.

   Content
 Tag
 Comment
  n/a
 und, or equivalently
"" , if that is available in the protocol  The tag where the process is not
equipped to analyze the text at all. und = "Undetermined"
143kl;ufa)iop(&uweiorqhjkl2341lkj#@!$Jkdfj;afe zxx
 Clearly some binary junk. zxx = "No linguistic content"   bok23
 und
 Maybe has linguistic content, maybe not. Can't really determine.
  chat  mul, if the protocol only permits a single tag
<en, fr> otherwise  mul = "Multiple languages"
maybe also others, since "chat" has entered the vocabulary of many languages
  Suzuki  ja-Latn
 maybe also others, since "I bought a Suzuki" is a perfectly reasonable
English sentence.   Igonda flatunicai vbinkli?  mis  some language the
process recognizes, but which is not in BCP 47   podstatné jméno  mis
 something the process recognizes as having linguistic content, and might be
in BCP 47, but it doesn't know which language it is.
  if (myInstance.getType() == Type.UNKNOWN) { throw new Exception(""); }
 art?
 unclear whether "art" can include, or is restricted to cases like Klingon
or Esperanto. art = "Artificial (Other)"
_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 19:16:08 +0200

Mark Davis wrote:

>                                   mul = "Multiple
>                mul, if the        languages"
>                protocol only      maybe also
>  chat          permits a          others, since
>                single tag         "chat" has
>                <en, fr>           entered the
>                otherwise          vocabulary of
>                                   many languages

Depends on the context.  If the context is something in the
direction of your comment it's fine.  But if the context is 
"I don't know if that's about a chat or a cat" I'd use "und".

>                                   some language
>                                   the process
>  Igonda flatunicai vbinkli?  mis  recognizes, but
>                                   which is not in
>                                   BCP 47

That would be wrong for almost all languages not yet in the
registry belonging to another collection like "ger".

>                                    something the
>                                    process
>                                    recognizes as
>                                    having
>                                    linguistic
>  podstatnÃ© jmÃ©no            mis  content, and
>                                    might be in BCP
>                                    47, but it
>                                    doesn't know
>                                    which language
>                                    it is.

If it doesn't know that it can use "und", abusing "mis" is
no option.  I think "art" is about artificial languages for
humans or in fiction, not for programming languages.

Frank

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 12 Apr 2007 14:53:27 -0700

A./

---

From: [email-masked]
Date: Fri, 13 Apr 2007 12:21:11 -0700

With respect to computer language snippets, isn't that what the <code> tag 
is for -- at least in XHTML?

Regards,

Karen Broome
Metadata Systems Designer
Sony Pictures Entertainment
310.244.4384

Frank Ellermann <[email-masked]> 
Sent by: [email-masked]
04/13/2007 03:48 AM

To
[email-masked]
cc
[email-masked]
Subject
Re: For review: Tagging text with no language

Frank

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 12:44:19 -0700

HTML is not the only place that language tags are used....

On 4/13/07, [email-masked] <[email-masked]> wrote:
>  *Frank Ellermann <[email-masked]>*
> [email-masked]  cc
> [email-masked]  Subject

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 13 Apr 2007 12:52:00 -0700

> "I don't know if that's about a chat or a cat" I'd use "und".

'und' is possible, but I think that mul conveys more information.

> registry belonging to another collection like "ger".

It's not wrong. I agree that where possible, one 'should' tag as precisely
as possible, but there is no requirement to in the RFC. Moreover, is unclear
to me that every human language falls under either one of the explicit codes
or into one of the collections. That may be true, I just don't know - Peter
might.

>                                    something the
> >                                    process
> >                                    recognizes as
> >                                    having
> >                                    linguistic
> >  podstatné jméno            mis  content, and
> >                                    might be in BCP
> >                                    47, but it
> >                                    doesn't know
> >                                    which language
> >                                    it is.
>
> no option.

I don't see how it is an abuse of 'mis'. If you could outline for me the
reasoning, based on the standards, behind saying that 'mis' is not correct,
I'd appreciate it.

I think "art" is about artificial languages for
> humans or in fiction, not for programming languages.

That may well be, it is just not clear to me from the specification.

Frank

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Fri, 13 Apr 2007 23:08:33 +0300 (EEST)

> is for -- at least in XHTML?

<code> has been in HTML since the beginning, and it indeed indicates 
computer code, or "computer language".

But to make things more difficult, computer code - such as program code, 
operating system commands, or markup - can be regarded as being in some 
human language(s) in the sense that the identifiers may have been formed 
from words in human languages and the comments are written in some human 
languages. Knowing the language of computer code can be relevant at least 
for two purposes:
a) the understandability of the code to human readers depends on whether
    they understand the human language(s) used
b) speech synthesis would benefit from the use of language information.
Moreover, a checking program could e.g. check that the code follows the 
syntactic rules of the programming, command, or markup "language" _and_ 
check the comments using a suitable human language spelling checker.

Thus, I would say that the human language some text and the role of that 
text as computer code are in a sense orthogonal.

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 16:21:41 -0400

> is for -- at least in XHTML?

Well, yes, but there is no standard way of saying *which* computer
language it is.  For one thing, there is no standard list of
computer languages.

-- 
Knowledge studies others / Wisdom is self-known;      John Cowan
Muscle masters brothers / Self-mastery is bone;       [email-masked]
Content need never borrow / Ambition wanders blind;   http://ccil.org/~cowan
Vitality cleaves to the marrow / Leaving death behind.    --Tao 33 (Bynner)

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Apr 2007 23:33:36 +0200

[email-masked] wrote:
 
> With respect to computer language snippets, isn't that what the <code>
> tag is for -- at least in XHTML?

Yes, typically interpreted as switch to a monospaced font.  But maybe
not good enough to convince spell-checkers that they should skip this
part, or to convince screenreaders that what follows might be not in
the inherited xml:lang.

While we're at it, IIRC you wanted a SHOULD NOT about script subtags
for "no linguistic content".  Mark's list of interesting examples also
contains a "zxx" use case, and it's clearly a zxx-Latn example.

Apparently the SHOULD NOT in RFC 4646 about "und" is already dubious,
so better we don't add another shaky SHOULD NOT about scripts for "zxx".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Apr 2007 00:02:03 +0200

Mark Davis wrote:

 [chat]
> 'und' is possible, but I think that mul conveys more information.

This "more info" could be wrong if you're sure that it's either
'chat' or 'cat'.  With "mul" you can justify 'en AND fr', arguably
you can also justify 'en OR fr'.  I'd draw the line at 'en XOR fr',
that's not "mul".

 [Igonda flatunicai vbinkli?]

> It's not wrong. I agree that where possible, one 'should' tag as
> precisely as possible, but there is no requirement to in the RFC.

Using "mis" instead of "ger" is like using "fr" instead of "en",
it's plain wrong based on John's explanations and the sources he
has cited.  The RFC doesn't forbid wrong or misleading tags, but
the registry also shouldn't encourage wrong or misleading tags.

Obviously I was like you also misled by "mis", it doesn't mean
that it belongs to "miscellaneous languages" from the POV of a
tagger.  It means that it belongs to a clearly defined set of
"miscellaneous languages" defined by ISO 639.

Above all a volatile set, all these collections (including "mis")
could be changed by ISO 639 whenever they find new evidence
justifying it.

That's about as horrible as your now long dead EU proposal, where
complete countries and territories on all continents could be
added (= minor problem) or removed (= major issue for stability).

This "mis" collection is IMO dangerous.  It's worse than generic
variants.  The least we can do is discourage its (ab)use.  All
registry users could have the same mis-conception about "mis" if
we do nothing.

> If you could outline for me the reasoning, based on the standards,
> behind saying that 'mis' is not correct, I'd appreciate it.

Your 2nd example sounds like Polish for me, and no language remotely
related to Polish belongs to the weird "mis" collection.  Trying
"ger" would be clearly wrong, but not as wrong as "mis".

>> I think "art" is about artificial languages for humans or in
>> fiction, not for programming languages.

> That may well be, it is just not clear to me from the specification.

 [A. Phillips, Ed.  Yahoo! Inc.  M. Davis, Ed.  Google  September 2006]
| Language tags are used to help identify languages, whether spoken,
| written, signed, or otherwise signaled, for the purpose of
| communication.  This includes constructed and artificial languages,
| but excludes languages not intended primarily for human
| communication, such as programming languages.

Frank

---

From: [email-masked]
Date: Fri, 13 Apr 2007 15:20:28 -0700

I agree with you. I would also say that indicating that computer languages 
have "no linguistic content" is not quite accurate for the same reasons 
you cite below and disagrees with the origin of the zxx tag, which was to 
indicate a total absence of language, not content which is not in a 
natural language. 

Karen Broome

> > With respect to computer language snippets, isn't that what the <code> 
tag

---

From: [email-masked]
Date: Fri, 13 Apr 2007 16:01:59 -0700

Well, if the code section is delimited with a consistent tag, you know to 
treat what's in there differently than the rest of the text. I used to 
edit technical tutorials for O'Reilly, so trust me, the <code> tag and I 
are old friends. On that site the code tag drives style sheets, but it is 
really a semantic delimiter -- not a design element -- and can potentially 
be used for a lot more. I'm a bit leery of spellcheckers for code.

But as long as the code section is delimited, you can write spellcheckers 
or screenreaders that  will handle that text differently.

I did not request a SHOULD NOT with "no linguistic content." My request 
was that script tags SHOULD NOT be used with linguistic *audio* content 
because spoken language, by definition, is not written down. I really 
don't have much use for the zxx tag.  I saw a use case that related to 
und-Latn that made sense, but zxx-Latn makes no sense to me. Did I miss 
that thread or is that a typo?

Regards,

Karen Broome

04/13/2007 02:33 PM

To
[email-masked]
cc
[email-masked]
Subject
[Ltru] Re: For review: Tagging text with no language

Frank

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 19:10:43 -0400

> that thread or is that a typo?

No, it makes sense:

<span xml:lang="zxx-Latn">kj;lqwe4ropiuasfgpoiu243poiuq3p4okjasl;jk</span>.

-- 
John Cowan   http://ccil.org/~cowan    [email-masked]
In might the Feanorians / that swore the unforgotten oath
brought war into Arvernien / with burning and with broken troth.
and Elwing from her fastness dim / then cast her in the waters wide,
but like a mew was swiftly borne, / uplifted o'er the roaring tide.
        --the Earendillinwe

---

From: John Cowan <[email-masked]>
Date: Fri, 13 Apr 2007 19:34:19 -0400

Stephen Deach scripsit:

> Is your intent to say:
>   "zxx: This string has no linguistic meaning, no-'language' (do not 
>   apply grammar, spelling, hyphenation, word recognition, or other 
> language-based services)"

Yes.

>   "Latn: format it as if it were a Latn script".

"However, a Latin-only font will suffice to render it."

-- 
"But I am the real Strider, fortunately,"       John Cowan
he said, looking down at them with his face     [email-masked]
softened by a sudden smile.  "I am Aragorn son  http://www.ccil.org/~cowan
of Arathorn, and if by life or death I can
save you, I will."  --LotR Book I Chapter 10

---

From: [email-masked]
Date: Fri, 13 Apr 2007 16:48:42 -0700

Hmm. How can you tell the "junk text" below isn't encrypted or encoded in 
some way? I think it's "und-Latn." 

Karen

[email-masked] wrote on 04/13/2007 04:23:12 PM:

> 
> At 2007.04.13-19:10(-0400), John Cowan wrote:
> 
> ><span 
xml:lang="zxx-Latn">kj;lqwe4ropiuasfgpoiu243poiuq3p4okjasl;jk</span>.
>    "zxx: This string has no linguistic meaning, no-'language' (do not 
apply 
> grammar, spelling, hyphenation, word recognition, or other 
language-based 
> services)"
>    -

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Apr 2007 02:58:30 +0200

[email-masked] wrote:

> I did not request a SHOULD NOT with "no linguistic content." My
> request was that script tags SHOULD NOT be used with linguistic
> *audio* content

Sorry, I confused it.  No problem with a SHOULD NOT for scripts
in audio content.

> I really don't have much use for the zxx tag.

Maybe for (printed) utterances like #?%zok##$%~boom!3<\# or ASCII
art.  IMO an example of zxx-Latn, whatever that means.

> zxx-Latn makes no sense to me. Did I miss that thread

Mark's table with interesting examples contained a zxx row:
43kl;ufa)iop(&uweiorqhjkl2341lkj#@!$Jkdfj;afe 

It could be a code, I guessed that it's zxx-Latn 

Frank

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 09:28:17 -0700

The meaning of zxx must be interpreted in terms of the coding standard of which it is a part. ISO 639 is explicitly about coding human languages. "No linguistic content" in the case of zxx means 'no content in any human language'. If a language tag must be applied to something like "ifdef DEBUG", then the appropriate language subtag would be zxx. ISO 639 cares not if an IETF tag zxx-Latn or zxx-progrmng is constructed; but consistency with ISO 639 would imply that the language subtag zxx be used.

Peter

From: [email-masked] [mailto:[email-masked]]
Sent: Friday, April 13, 2007 3:20 PM
To: Jukka K. Korpela
Cc: [email-masked]; [email-masked]
Subject: Re: [Ltru] Re: For review: Tagging text with no language

I agree with you. I would also say that indicating that computer languages have "no linguistic content" is not quite accurate for the same reasons you cite below and disagrees with the origin of the zxx tag, which was to indicate a total absence of language, not content which is not in a natural language.

Karen Broome

---

From: "CE Whitehead" <[email-masked]>
Date: Sat, 14 Apr 2007 13:09:01 -0400

Hi, my comment is below; I understand that mul is for where you know there 
is  content from more than one language.
Otherwise the code for content which you are sure is in some language where 
you do not know how many languages are involved is mis.

Is that right??

--C. E. Whitehead
[email-masked]

Mark Davis <[email-masked]>
On 4/13/07, Frank Ellermann <[email-masked]> wrote:

>'und' is possible, but I think that mul conveys more information.

'mul' only works I think if you are sure there is more than one language;
otherwise 'mis' or 'und' but und thus is being used rather freely I guess--
it seems to me.

That's what I gather.

>That may well be, it is just not clear to me from the specification.

_________________________________________________________________
Get a FREE Web site, company branded e-mail and more from Microsoft Office 
Live! http://clk.atdmt.com/MRT/go/mcrssaub0050001411mrt/direct/01/

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 10:40:26 -0700

From: CE Whitehead [mailto:[email-masked]]

> Otherwise the code for content which you are sure is
> in some language where you do not know how many
> languages are involved is mis.
>
> Is that right??

No! If you don't know how many languages are involved, then you have not made any determinations and the correct tag is und. This would be completely inappropriate use of mis.

Peter

---

From: "CE Whitehead" <[email-masked]>
Date: Sat, 14 Apr 2007 14:17:21 -0400

>No! If you don't know how many languages are involved, then you have not 
>made any determinations and the correct tag is und. This would be 
>completely inappropriate use of mis.
>
>
>
>Peter
>
>
Well, mul I think is when you are sure there is more than one;
is that wrong?

Thanks.

--C. E. Whitehead
[email-masked]

_________________________________________________________________
The average US Credit Score is 675. The cost to see yours: $0 by Experian. 
http://www.freecreditreport.com/pm/default.aspx?sc=660600&bcd=EMAILFOOTERAVERAGE

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 11:23:34 -0700

If you know there are multiple languages (as in 'this is deutsch', not Mark's ambiguous 'kind' cases), then that would be an acceptable place to use mul (if you find that useful for some purpose).

Peter

-----Original Message-----
From: CE Whitehead [mailto:[email-masked]]
Sent: Saturday, April 14, 2007 11:17 AM
To: Peter Constable; [email-masked]; [email-masked]
Subject: RE: [Ltru] Re: For review: Tagging text with no language

Thanks.

--C. E. Whitehead
[email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Apr 2007 21:14:26 +0200

Peter Constable wrote:

> The meaning of zxx must be interpreted in terms of the coding standard
> of which it is a part. ISO 639 is explicitly about coding human
> languages. ?No linguistic content? in the case of zxx means ?no
> content in any human language?. If a language tag must be applied to
> something like ?ifdef DEBUG?, then the appropriate language subtag
> would be zxx.

Okay, that's very important for Mark's table, Richard's aricle, and
IMO it deserves a comment for "zxx" in the language subtag registry:

art   = artificial human language, no programming language
zxx   = no linguistic content wrt. human languages, but it can be a
        programming language

Ignoring those real (non-fictional) alien languages for the moment.

So in a context where I can't use xml:lang="" I'd pick xml:lang="und"
if I don't know what it is, and it's likely a human language.

And I'd pick xml:lang="zxx" for a programming language if tools need
to know that it's certainly not in the otherwise inherited xml:lang.

Frank

---

From: "CE Whitehead" <[email-masked]>
Date: Sat, 14 Apr 2007 18:33:10 -0400

Hi,

More on tagging code:

From reading  "Internationalization Best Practices"

http://www.w3.org/International/geo/html-tech/tech-lang.html#ri20050128.175100333

if you have an element in HTML of XML code with an attribute value such as "Swedish" (for a title or something)

(here's the example from Internationalization Best Practices":

<p title="Swedish"><a xml:lang="sv" lang="sv" href="http://www.w3.org/International/geo/html-tech/index.sv.html">svenska</a></p>

then you do need to tag the language of that attribute value;

you do so by inserting in your document a second element if necessary (which serves to enclose the HTML or XML element, and set it off from the rest of the text).

But you only need to do this if the language of the attribute value is different from that of the text content elsewhere in the document.

But you never apparently tag the language of the attribute itself, the attribute "title" in this case, which is clearly English ( HTML and XML code are derived from English).  Just the language of the value!

Thus, if your title value were 'suédois' instead of 'Swedish,' you'd have to tag the language of this piece of code as French rather than as English,

in spite of the fact that title is an English word!

(if I understood the Best Practices document correctly)

>zxx   = no linguistic content wrt. human languages, but it can be a

(Still distraught as John C. has told me that there are now programming languages which make use of keywords in different languages.

So maybe just a note--programming languages are tagged as zxx even though of course code is derived from real languages; and maybe a reference to the existence of a few non-English programming language

but if no one else thinks this is proper, I'll withdraw my suggestion

)

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 14 Apr 2007 17:06:56 -0700

Hi -

As a technical contributor...

> From: "CE Whitehead" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>
> Cc: <[email-masked]>
> Sent: Saturday, April 14, 2007 3:33 PM
> Subject: [Ltru] Re: For review: Tagging text with no language
...
> But you never apparently tag the language of the attribute itself,
> the attribute "title" in this case, which is clearly English ( HTML
> and XML code are derived from English).  Just the language
> of the value!
...

That's because HTML and XML are no more English than C, ALGOL,
FORTRAN or Ada.  There are lots of borrowings, to be sure,but
they are no more English than English is French.

Randy