Subject: [Ltru] Re: STD (was: Last Call: 'Tags for Identifying Languages' to BCP)

From: "Doug Ewell" <[email-masked]>
Date: Sat, 27 Aug 2005 18:09:44 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

> JFTR, nothing new, I still prefer standards track:
>
> 3066bis   could obsolete 3066 (BCP 47) also as PS.
> 3066ter   oould integrate ISO 639-3 as DS (minor update)
> 3066tetra oould integrate ISO 639-6 as STD

I still have significant concerns about the assumption that ISO 639-6
will be, or should be, automatically integrated into a language tagging
scheme.  The Linguasphere database upon which 639-6 is based is still
not freely available to the public for evaluation and examination, and
seems unlikely ever to be, unlike the Ethnologue database upon which ISO
639-3 is based.

Moreover, with ISO 639-3 already claiming to provide coverage for "all
known human languages," numbering a bit over 7,600, and with
Linguasphere listing "over 20,000 languages and constituent dialects,"
one is left to wonder just what the 13,400 new identifiers will
contribute from the standpoint of identifying and requesting linguistic
content.  Will "en-US", "en-GB", "en-AU", and "en-IN" be given their own
four-letter codes in ISO 639-6, and what will be the rules for
correlating and equating these with the identifiers already in place?

Meanwhile, the claim that there are "over 20,000 languages" to be tagged
is being used as an argument against the current RFC 3066bis effort and
the plan to support 7,600 languages in RFC 3066ter.

I'm not saying anything against the Linguasphere effort per se, but with
the limited knowledge available to me, I don't think its eventual role
in language tagging should be considered a fait accompli.

--
Doug Ewell
Fullerton, California
http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 28 Aug 2005 07:14:01 +0200

Doug Ewell wrote:

> I still have significant concerns about the assumption that
> ISO 639-6 will be, or should be, automatically integrated
> into a language tagging scheme.

Yes, it wasn't my intention to say something else, I used it
only for my standards track example.  More about "BCP" on the
general list, a better solution could be a separate series
for "non-technical" (admin / meta) documents.

> the claim that there are "over 20,000 languages" to be tagged
> is being used as an argument against the current RFC 3066bis
> effort and the plan to support 7,600 languages in RFC 3066ter

Jefsey mumbled something in this direction, but IMHO it was not
convincing - we discussed potential size issues here.  His real
concern is "who controls IANA", that's interesting, but I'm not
worried, and it's no reason to block 3066bis.

BTW, I like number [7] in the rather long list. <g>  Bye, Frank

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 28 Aug 2005 12:22:55 +0100

Hi

See in line comments

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell
> Sent: 28 August 2005 02:10
> To: LTRU Working Group
> Subject: [Ltru] Re: STD (was: Last Call: 'Tags for Identifying Languages'
> toBCP)
> 

I don't think anyone should make assumptions where technology is concerned!
What I can say is that the database for the 639-6 standard will be made
freely available online, in the same way as the Ethnologue, as soon as it is
ready.  However, this is a WIP and there really is no point putting up half
information that has not yet been verified.  I am currently putting together
the verification methodology which will be implemented by various
collaboration partners internationally.  I cannot at this moment say when
the database will be ready but rest assured you will be notified.

> correlating and equating these with the identifiers already in place?

As it stands at the moment, 639-6 will provide each linguistic entity with
one language reference name.  This reference name is intended for use as the
unique data identifier (DI as per 11179). Unique Data Identifiers and
alpha-4 Representations for each linguistic entity written, spoken and
signed down to the level of dialect and component variety, where
appropriate, will be provided.  Because 639-6 is a hierarchical system (the
parent alpha-4 representation is provided), it lends itself very well to the
"mapping" of the other codes of the 639 family; something that should prove
invaluable for backward compatibility.  In other words, in the system as
proposed, you could use either the alpha-4 representation or the unique DI
to find the closest 639-1,-2,-3 or -5 tags should you so wish.  

> the plan to support 7,600 languages in RFC 3066ter.

Hmmmm... at this point there could be very lengthy off topic discussions as
to what is a language and I don't think that is a good idea.  I would not
say that there are 20,000 languages but rather nearer to 7,600 as stated in
639-3; just my humble opinion.  JFTR, I support the inclusion of 639-3 in
RFC3066ter.

> in language tagging should be considered a fait accompli.

That is your prerogative but IMHO very much a sensible approach (see first
line comment).  

ISO 639-6 is being created as an International Standard.  There is a long
way to go (earliest completion 2007) until it reaches full IS and no doubt
there will be twists and turns on the ISO journey :-)

As previously mentioned by Peter, there will be a draft CD for comment
available by the end of November.  At that time I will happily make it
available to this expert group for comment.  

Have a nice day!

Debbie Garside

---

From: r&d afrac <[email-masked]>
Date: Sun, 28 Aug 2005 15:54:14 +0200

Debbie,
you will see my comments in the mail of comments/support to Doug 
Ewell. All this is good and clear. Except your timing. When do you 
think we can have a test sample, to be able to develop? I suppose 
IANA would be interested too. Ditto, Doug: could you release your 
complete test ISO-639-3 included base?
jfc

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 28 Aug 2005 15:54:49 +0100

Hi JFC

I would not release any data until the November draft.

Cheers

Debbie

> From: r&d afrac [mailto:[email-masked]]
> Sent: 28 August 2005 14:54
> To: Debbie Garside; 'Doug Ewell'; 'LTRU Working Group'
> Subject: RE: [Ltru] Re: STD (was: Last Call: 'Tags for Identifying
> Languages' toBCP)
> 
> Debbie,

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 28 Aug 2005 11:36:09 -0700

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> I don't think anyone should make assumptions where technology is
> concerned!  What I can say is that the database for the 639-6 standard
> will be made freely available online, in the same way as the
> Ethnologue, as soon as it is ready.  However, this is a WIP and there
> really is no point putting up half information that has not yet been
> verified.  I am currently putting together the verification
> methodology which will be implemented by various collaboration
> partners internationally.  I cannot at this moment say when the
> database will be ready but rest assured you will be notified.

I'm a bit surprised that a work characterized as a work-in-progress and
not yet ready for public review is nevertheless deemed ready to be
considered as a draft international standard.  We're not talking about a
completely new invention, but a method of classifying languages, which
is a human endeavor dating back many centuries if not millenia.

But I'm certainly glad that my assumption that the information might not
be made freely available was incorrect.  Clearly, there can't be
anything proprietary or IP-restricted about the data available for
language tags (but of course Linguasphere can always provide proprietary
value-added data, such as the language hierarchy).

> As it stands at the moment, 639-6 will provide each linguistic entity
> with one language reference name.  This reference name is intended for
> use as the unique data identifier (DI as per 11179). Unique Data
> Identifiers and alpha-4 Representations for each linguistic entity
> written, spoken and signed down to the level of dialect and component
> variety, where appropriate, will be provided.  Because 639-6 is a
> hierarchical system (the parent alpha-4 representation is provided),
> it lends itself very well to the "mapping" of the other codes of the
> 639 family; something that should prove invaluable for backward
> compatibility.  In other words, in the system as proposed, you could
> use either the alpha-4 representation or the unique DI to find the
> closest 639-1,-2,-3 or -5 tags should you so wish.

But in language tags, either one value needs to be canonical -- sorry,
"preferred" -- over the others, or else the duplicative values should be
not be added at all.  Otherwise there is a significant burden on those
who would perform matching.

I would suggest that for compatibility, subtags directly mappable to ISO
639-1/2/3 should not be added.  This is similar to the situation with
ISO 639-2 in RFC 3066, where values like "eng" were not made available
for language tags because of the pre-existence of "en".

Will Linguasphere provide the mapping between the new alpha-4 codes and
ISO 639-1/2/3, or is that something a group like this would have to do?

>> Meanwhile, the claim that there are "over 20,000 languages" to be
>> tagged is being used as an argument against the current RFC 3066bis
>> effort and the plan to support 7,600 languages in RFC 3066ter.
>
> Hmmmm... at this point there could be very lengthy off topic
> discussions as to what is a language and I don't think that is a good
> idea.  I would not say that there are 20,000 languages but rather
> nearer to 7,600 as stated in 639-3; just my humble opinion.  JFTR, I
> support the inclusion of 639-3 in RFC3066ter.

I agree that the broad question of "what is a language" is out of our
scope. The more specific question "what is a taggable language
distinction" is perhaps more germane.  I just think the two figures
(7,600 and 20,000) could be seen as representing a fundamental
disagreement.  I'd like to cite a specfic example of a distinction that
is made in 639-6 but not 639-3, but I can't because the data's not yet
available.

I did not mean to imply that it was you (or Lee) who opposed either the
current draft or the proposed use of ISO 639-3 in future drafts.

>> I'm not saying anything against the Linguasphere effort per se, but
>> with the limited knowledge available to me, I don't think its
>> eventual role in language tagging should be considered a fait
>> accompli.
>
> That is your prerogative but IMHO very much a sensible approach (see
> first line comment).
>
> ISO 639-6 is being created as an International Standard.  There is a
> long way to go (earliest completion 2007) until it reaches full IS and
> no doubt there will be twists and turns on the ISO journey :-)
>

I will gladly wait until then before trying to make specific comments on
639-6.  Until then, I stand by my prerogative (in the U.S. we might say
that it's not a slam dunk).

> Have a nice day!

+1