Subject: [Ltru] Appropriate use of variants (was: Re: Language and script encoding standards)

From: "Doug Ewell" <[email-masked]>
Date: Wed, 26 Jul 2006 23:28:38 -0700

Frank Ellermann <nobody at xyzzy dot claranet dot de> wrote:

>> "The following types of language variations are suitable to
>> be registered as variant subtags:
> [...]
>
> Good idea, maybe also some known counter examples.  Saying
> "might be / might be not suitable" or similar, the review
> list should be free to make exceptions.

Excellent point.  RFC 3066bis leaves the scope of potential uses for 
variants fairly wide open, but there are a few inappropriate uses.  It 
might be a good idea to say, for instance, that a variant should not be 
proposed to denote a wholly independent language -- that's what 5- to 
8-letter primary language subtags are for -- or to denote a script or 
region unless the appropriate measures have been taken with ISO 3166/MA 
or ISO 15924/RA.

I also agree that the Reviewer, aided by the list, should have 
discretionary flexibility within the constraints of the document.  (I 
prefer that to "making exceptions," which sounds a bit too open-ended 
and above-the-law.)  The important point is that most of the time, the 
document should answer the meta-question "Is this a suitable use of a 
variant subtag?" in advance, so the Reviewer and list should not have to 
ask it.

--
Doug Ewell
Fullerton, California, USA
http://users.adelphia.net/~dewell/

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 26 Jul 2006 23:48:44 -0700

Mark Davis <mark dot davis at icu dash project dot org> wrote:

>> If this discussion is included in the Charter for RFC3066ter then I 
>> would recommend that ISO CD 639-6 be included also.
>
> As a CD, it's definitely out of scope. Even once it becomes a 
> standard, it is a matter of debate whether it should be used.

Timing will become a key criterion here -- the timing of ISO 639-6 as it 
progresses along the path from CD to DIS to FDIS to IS, and the timing 
of RFC 3066ter as it progresses along its own path.

Two points come to mind here.  First, the 2-year RFC 3066bis experience 
(so far -- we still don't have RFC numbers) does not give me great hope 
that 3066ter will just whiz through the WG and IETF in a matter of a few 
months, free from controversy and delay.  ISO 639-6 might be ready 
sooner than we think, relative to the 3066ter time frame.

Second, Debbie was only talking about mentioning ISO 639-6 in the 
3066ter charter.  Look at the 3066bis charter; it makes mention of ISO 
639-3, which is not yet an approved standard.  The effect of ISO 639-6 
on RFC 3066ter might simply be to firm up the projected purpose of the 
enigmatic "future use" 4-letter primary language subtags, just as 
3066bis broadly hints that extended-language subtags are "possibly for 
use with certain anticipated features of ISO 639-3."

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 27 Jul 2006 07:33:02 -0700

In my opinion, we should neither mention this in the charter nor expend any great amount of effort on it.

Section 2.2.5 (Variant) of the current document defines variants as "used to indicate additional, well-recognized variations that define a language or its dialects which are not covered by other available subtags." We then go on to say in Section 3.6 (Possibilities for Registration):

--
Dialect or other divisions or variations within a language, its orthography, writing system, regional or historical usage, transliteration or other transformation, or distinguishing variation MAY be registered as variant subtags. An example is the 'rozaj' subtag (the Resian dialect of Slovenian).
--

This is exactly the right level of distinction. We have some examples and we don't spend a lot of time restricting the possibilities, given that the range of variation in language is quite large.

If anything were to be added it would be, as Frank suggests, a few counter-examples. Variants are also not suitable (IMO) for charsets [en-us-latin1], obscure private distinctions [es-MX-auntsally], personal/individual identification [en-IE-joesmith], or transitory variations [de-DE-worldcup]. Still, each of these items is something that the list historically has been able to point out by itself. We don't need additional language to spell this out.

Addison

Addison Phillips
Internationalization Architect - Yahoo! Inc.

Internationalization is an architecture.
It is not a feature.  

> -----Original Message-----
> From: Doug Ewell [mailto:[email-masked]] 
> Sent: mercredi 26 juillet 2006 23:29
> To: LTRU Working Group
> Cc: Frank Ellermann
> Subject: [Ltru] Appropriate use of variants (was: Re: 
> Language and scriptencoding standards)
>

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 27 Jul 2006 08:40:00 -0700

It is more than that. The structure of 3066bis is explicitly designed with
639-3 in mind, with the extlang subtags. I see no need to mention 639-6 in
the charter unless we truely expect to incorporate it in some fashion. At
this point I see no value, only problems, in adding 639-6 to language tags.

Of course, Debbie and others can certainly present their case for why 639-6
adds value and won't cause complications, but that case has not yet been
made.

Mark

---

From: John Cowan <[email-masked]>
Date: Thu, 27 Jul 2006 12:04:30 -0400

Mark Davis scripsit:

> At this point I see no value, only problems, in adding 639-6 to
> language tags.
> 
> Of course, Debbie and others can certainly present their case for why
> 639-6 adds value and won't cause complications, but that case has not
> yet been made.

This is a confusion of levels.  If 639-6 isn't put in the LTRU charter,
it can't even be *considered*.  And as we have just painfully found out,
premature discussion doesn't help: it just has to be done all over again
(and indeed it may be harmful -- it gives certain elements an opportunity
to claim that everything is rigged in advance).

In short: we need to have 639-6 in the charter so that we can discuss it,
even if we end up rejecting it.  This is not like ISO 11179, where the
very relevance turned out to be doubtful; no one can doubt that 693-6
is relevant to language tags.

-- 
John Cowan        http://www.ccil.org/~cowan          [email-masked]
Please leave your values                Check your assumptions.  In fact,
   at the front desk.                      check your assumptions at the door.
     --sign in Paris hotel                   --Cordelia Vorkosigan

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 27 Jul 2006 09:09:08 -0700

FWIW, we did reserve alpha4 primary subtags for ISO 639-6. But I agree with Mark that it is far too early to commit to incorporating 639-6 into the registry.

What I would put into the charter would be text like this:

--
The primary focus of this update is to incorporate changes to the ISO 639 family of standards, specifically the adoption of ISO 639-3. ISO 639-3 makes an approximately 7000 additional language codes available, identifying many minority languages or dialects not previously distinguished. These language codes will be used as an additional source of primary language subtags, as well as the source for "extended language" (extlang) subtags. The working group must decide how ISO 639-3 codes are entered into the registry, what rules apply to associating a code with the language and extlang subtag types, and any additional rules or guidance on their adoption and use.

The working group should also consider ISO 639-6 and whether or how that evolving work might relate to the future development of language tags. Because this work is not mature, it might not appear in the document produced by this working group.
--

Best Regards,

Addison

Addison Phillips

> From: Mark Davis [mailto:[email-masked]] 
> Sent: jeudi 27 juillet 2006 08:40
> To: Doug Ewell
> Cc: LTRU Working Group
> Subject: Re: [Ltru] ISO CD 639-6 (was: Re: Language and 
> script encodingstandards)
> 
> It is more than that. The structure of 3066bis is explicitly 
> designed with 639-3 in mind, with the extlang subtags. I see 
> no need to mention 639-6 in the charter unless we truely 
> expect to incorporate it in some fashion. At this point I see 
> no value, only problems, in adding 639-6 to language tags. 
> 
> Of course, Debbie and others can certainly present their case 
> for why 639-6 adds value and won't cause complications, but 
> that case has not yet been made.
> 
> Mark
> 
>

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 27 Jul 2006 09:20:13 -0700

Oh, and...

My personal opinion, based on what I know about ISO 639-6, its approach, goals, structure, and so forth, is that it probably is not a suitable addition to the language subtag zoo. In many ways, 639-6 codes mirror what 3066-family language tags already do (identifying script, regional, and other variations within a language), with the complexity of compressing the identifiers into an alpha4 namespace. Because these codes duplicate, rather than complement, ISO 639-1/2/3 codes in meaning, incorporating them would vastly complicate language tag choice and matching algorithms, without apparent benefit to end-users.

That said, I don't know enough about ISO 639-6 to definitively say these things today: the foregoing is an impression and not a conclusion. Thus I support mentioning studying 639-6 in a 3066ter time frame, with no concrete deliverable promised, but I tend to expect that the sole support for it would be continuing this line in the ABNF for language tags:

              / 4ALPHA                 ; reserved for future use

Addison

Addison Phillips

> From: Addison Phillips [mailto:[email-masked]] 
> Sent: jeudi 27 juillet 2006 09:09
> To: 'Mark Davis'; 'Doug Ewell'
> Cc: 'LTRU Working Group'
> Subject: RE: [Ltru] ISO CD 639-6 (was: Re: Language and 
> scriptencodingstandards)
> 
> FWIW, we did reserve alpha4 primary subtags for ISO 639-6. 
> But I agree with Mark that it is far too early to commit to 
> incorporating 639-6 into the registry.
> 
> The primary focus of this update is to incorporate changes to 
> the ISO 639 family of standards, specifically the adoption of 
> ISO 639-3. ISO 639-3 makes an approximately 7000 additional 
> language codes available, identifying many minority languages 
> or dialects not previously distinguished. These language 
> codes will be used as an additional source of primary 
> language subtags, as well as the source for "extended 
> language" (extlang) subtags. The working group must decide 
> how ISO 639-3 codes are entered into the registry, what rules 
> apply to associating a code with the language and extlang 
> subtag types, and any additional rules or guidance on their 
> adoption and use.
> 
> The working group should also consider ISO 639-6 and whether 
> or how that evolving work might relate to the future 
> development of language tags. Because this work is not 
> mature, it might not appear in the document produced by this 
> working group.
> --
> 
> Best Regards,
> 
> Addison
> 
> Addison Phillips

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 27 Jul 2006 17:26:55 +0100

> Of course, Debbie and others can certainly present their case for why
639-6 adds value and won't cause complications, but that case has not yet
been made.
 
The essence of the case is the need to tag more accurately and I am sure
that this will cause a huge debate.  The question "what is a language" is
pertinent.  Many dialects have significantly different vocabularies and some
have different writing systems (especially historically).  Whether these
aspects of the standard are relevant to Internet Standards/RFCs etc is
something that needs to be discussed at some stage.
 
I agree with you that as 639-6 is technically at CD stage (moves to DIS
after Beijing in August - voting was positive on the CD ballot) perhaps it
would be a little premature to have the discussion now. However, I also
agree with Doug.  It depends on how long a time span you envisage for
RFC3066ter as to whether it would be prudent to at least mention 639-6
within the Charter.
 
As to complications, the change of direction in the research methods and
compilation of the supporting data since December 2005 means that there is
no duplication between the codes of ISO 639-3 and the hierarchical system of
ISO 639-6.  If there is a need for more granular tags it should prove
relatively easy to slot 639-6 alpha-4s into the Primary Language Subtag
position - a position that is currently reserved for them.
 
Best regards
 
Debbie

  _____  

From: Mark Davis [mailto:[email-masked]] 
Sent: 27 July 2006 16:40
To: Doug Ewell
Cc: LTRU Working Group
Subject: Re: [Ltru] ISO CD 639-6 (was: Re: Language and script
encodingstandards)

Mark

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 27 Jul 2006 09:34:36 -0700

I agree with that approach, but the last bit needs to be even more
tentative, something like:

The working group should also consider ISO 639-6 and whether or how that
evolving work might relate to the future development of language tags.
Because this work is not mature and may not be suitable for inclusion, it
might not appear in the document produced by this working group.

Mark

On 7/27/06, Addison Phillips <[email-masked]> wrote:
>
> FWIW, we did reserve alpha4 primary subtags for ISO 639-6. But I agree
> with Mark that it is far too early to commit to incorporating 639-6 into the
> registry.
>
> The primary focus of this update is to incorporate changes to the ISO 639
> family of standards, specifically the adoption of ISO 639-3. ISO 639-3 makes
> an approximately 7000 additional language codes available, identifying many
> minority languages or dialects not previously distinguished. These language
> codes will be used as an additional source of primary language subtags, as
> well as the source for "extended language" (extlang) subtags. The working
> group must decide how ISO 639-3 codes are entered into the registry, what
> rules apply to associating a code with the language and extlang subtag
> types, and any additional rules or guidance on their adoption and use.
>
> The working group should also consider ISO 639-6 and whether or how that
> evolving work might relate to the future development of language tags.
> Because this work is not mature, it might not appear in the document
> produced by this working group.
> --
>
> Best Regards,
>
> Addison
>
> Addison Phillips

---

From: John Cowan <[email-masked]>
Date: Thu, 27 Jul 2006 12:40:03 -0400

Debbie Garside scripsit:

> compilation of the supporting data since December 2005 means that there
> is no duplication between the codes of ISO 639-3 and the hierarchical
> system of ISO 639-6.

The concern is not (or no longer) about conflict between 639-6 codes
and 639-3 codes, but between 639-6 codes and RFC 3066 *tags* as a whole.
They cover the same semantic space (representations of arbitrarily
fine-grained distinctions between language varieties), but RFC 3066bis
tags are analyzable, whereas 639-6 codes are not, unless you have
the hierarchy information handy.

There's no simple way to see that a given 639-6 code is or is not
equivalent to some language-script-country-variant tag; furthermore,
it's a big question what, if anything, it means to specify a
639-6 code with script-country-variant following.

All of which is a fine example of the "premature discussion" I mentioned
in my previous posting.  So let's get 639-6 into the charter so that
Addison can properly argue against it.

-- 
On the Semantic Web, it's too hard to prove     John Cowan    [email-masked]
you're not a dog.  --Bill de hOra               http://www.ccil.org/~cowan

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 27 Jul 2006 17:45:43 +0100

Addison wrote:

> Because 
> these codes duplicate, rather than complement, ISO 639-1/2/3 
> codes in meaning, incorporating them would vastly complicate 
> language tag choice and matching algorithms, without apparent 
> benefit to end-users.

There is no duplication of codes between ISO 639-1/2B/2T/3/5 and ISO 639-6

Best regards

Debbie

> Sent: 27 July 2006 17:20
> To: 'Addison Phillips'; 'Mark Davis'; 'Doug Ewell'
> Cc: 'LTRU Working Group'
> Subject: RE: [Ltru] ISO CD 639-6
> 
> Oh, and...
> 
> My personal opinion, based on what I know about ISO 639-6, 
> its approach, goals, structure, and so forth, is that it 
> probably is not a suitable addition to the language subtag 
> zoo. In many ways, 639-6 codes mirror what 3066-family 
> language tags already do (identifying script, regional, and 
> other variations within a language), with the complexity of 
> compressing the identifiers into an alpha4 namespace. Because 
> these codes duplicate, rather than complement, ISO 639-1/2/3 
> codes in meaning, incorporating them would vastly complicate 
> language tag choice and matching algorithms, without apparent 
> benefit to end-users.
> 
> That said, I don't know enough about ISO 639-6 to 
> definitively say these things today: the foregoing is an 
> impression and not a conclusion. Thus I support mentioning 
> studying 639-6 in a 3066ter time frame, with no concrete 
> deliverable promised, but I tend to expect that the sole 
> support for it would be continuing this line in the ABNF for 
> language tags:
> 
> > The primary focus of this update is to incorporate changes 
> to the ISO 
> > 639 family of standards, specifically the adoption of ISO 
> 639-3. ISO 
> > 639-3 makes an approximately 7000 additional language codes 
> available, 
> > identifying many minority languages or dialects not previously 
> > distinguished. These language codes will be used as an additional 
> > source of primary language subtags, as well as the source for 
> > "extended language" (extlang) subtags. The working group 
> must decide 
> > how ISO 639-3 codes are entered into the registry, what 
> rules apply to 
> > associating a code with the language and extlang subtag 
> types, and any 
> > additional rules or guidance on their adoption and use.
> > 
> > The working group should also consider ISO 639-6 and whether or how 
> > that evolving work might relate to the future development 
> of language 
> > tags. Because this work is not mature, it might not appear in the 
> > document produced by this working group.
> > --
> > 
> > Best Regards,
> > 
> > Addison
> > 
> > Addison Phillips
> > > designed with 639-3 in mind, with the extlang subtags. I 
> see no need 
> > > to mention 639-6 in the charter unless we truely expect to 
> > > incorporate it in some fashion. At this point I see no 
> value, only 
> > > problems, in adding 639-6 to language tags.
> > > 
> > > Of course, Debbie and others can certainly present their case for 
> > > why 639-6 adds value and won't cause complications, but that case 
> > > has not yet been made.
> > > 
> > > Mark
> > > 
> > > 
> > > 	Timing will become a key criterion here -- the timing 
> of ISO 639-6 
> > > as it
> > > 	(so far -- we still don't have RFC numbers) does not 
> give me great 
> > > hope
> > > 	that 3066ter will just whiz through the WG and IETF in 
> a matter of 
> > > a few
> > > 	3066ter charter.  Look at the 3066bis charter; it makes 
> mention of 
> > > ISO
> > > 	639-3, which is not yet an approved standard.  The 
> effect of ISO 
> > > 639-6
> > > 	on RFC 3066ter might simply be to firm up the projected 
> purpose of 
> > > the
> > > 	3066bis broadly hints that extended-language subtags 
> are "possibly 
> > > for

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 27 Jul 2006 17:52:20 +0100

> There is no duplication of codes between ISO 639-1/2B/2T/3/5 

To make my meaning clearer:

There is no alpha4 allocated for linguistic entities that are already coded
within ISO 639-1/2B/2T/3/5.

Best regards

Debbie

> From: Debbie Garside [mailto:[email-masked]] 
> Sent: 27 July 2006 17:46
> > these codes duplicate, rather than complement, ISO 
> 639-1/2/3 codes in 
> > meaning, incorporating them would vastly complicate language tag 
> > choice and matching algorithms, without apparent benefit to 
> end-users.
> 
> > My personal opinion, based on what I know about ISO 639-6, its 
> > approach, goals, structure, and so forth, is that it 
> probably is not a 
> > suitable addition to the language subtag zoo. In many ways, 639-6 
> > codes mirror what 3066-family language tags already do (identifying 
> > script, regional, and other variations within a language), with the 
> > complexity of compressing the identifiers into an alpha4 namespace. 
> > Because these codes duplicate, rather than complement, ISO 
> 639-1/2/3 
> > codes in meaning, incorporating them would vastly 
> complicate language 
> > tag choice and matching algorithms, without apparent benefit to 
> > end-users.
> > 
> > That said, I don't know enough about ISO 639-6 to definitively say 
> > these things today: the foregoing is an impression and not a 
> > conclusion. Thus I support mentioning studying 639-6 in a 
> 3066ter time 
> > frame, with no concrete deliverable promised, but I tend to expect 
> > that the sole support for it would be continuing this line 
> in the ABNF 
> > for language tags:
> > 
> > > > 	>> If this discussion is included in the 
> Charter for RFC3066ter 
> > > > then I
> > > > 	progresses along the path from CD to DIS to 
> FDIS to IS, and the 
> > > > timing
> > > > 	Two points come to mind here.  First, the 
> 2-year RFC 3066bis 
> > > > experience

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 27 Jul 2006 10:12:23 -0700

> Because this work is not mature 
> and may not be suitable for inclusion, it might not appear in 
> the document produced by this working group.
>

That phrasing seems better to me.

Addison Phillips

---

From: "Addison Phillips" <[email-masked]>
Date: Thu, 27 Jul 2006 10:32:47 -0700

> the hierarchy information handy.

I agree that this is, perhaps, the main concern, given the "change in
direction". But there is also the immediate concern for language tag users
of subtag choice. If 'enxx' is narrower than 'en', which should I used to
form my language tag? Are these codes not in conflict on some level, since
they both represent some form of "English"? 

A hierarchy is a very different beast than a flat namespace. Comparing the
two leads to lots of interesting questions. A good look at the finished
product seems like a precondition to me.

> Addison can properly argue against it.
+1

Addison

Addison Phillips

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 27 Jul 2006 18:42:05 +0100

Addison wrote:

> A good look at the finished product seems like a precondition to me.

I will be releasing data tables displaying approximately 30-40% of the
product prior to the TC37 meeting in Beijing (August 22nd).  Will send a
link when it is online.  That said, the data released at that time will
still need to go through various verification procedures and thus will not
be stable. 

Best regards

Debbie

> Sent: 27 July 2006 18:33
> To: 'John Cowan'; 'Debbie Garside'
> Cc: 'Doug Ewell'; 'LTRU Working Group'
> > tags are analyzable, whereas 639-6 codes are not, unless 
> you have the 
> > hierarchy information handy.
> 
> I agree that this is, perhaps, the main concern, given the 
> "change in direction". But there is also the immediate 
> concern for language tag users of subtag choice. If 'enxx' is 
> narrower than 'en', which should I used to form my language 
> tag? Are these codes not in conflict on some level, since 
> A hierarchy is a very different beast than a flat namespace. 
> Comparing the two leads to lots of interesting questions. A 
> good look at the finished product seems like a precondition to me.
> 
> >

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 27 Jul 2006 22:44:33 -0700

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> product prior to the TC37 meeting in Beijing (August 22nd).  Will send 
> a link when it is online.  That said, the data released at that time 
> will still need to go through various verification procedures and thus 
> will not be stable.

This will be absolutely invaluable for making any sort of rational and 
informed decision about whether the ISO 639-6 "peg" will fit into the 
RFC 3066bis "hole."  I am eagerly awaiting a first look at these sample 
tables, and personally I'm not making any decision about whether the peg 
and hole are well matched until I've had a chance to sit down and play 
with them.

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 27 Jul 2006 23:17:45 -0700

Mark Davis wrote:

> that evolving work might relate to the future development of language 
> tags. Because this work is not mature and may not be suitable for 
> inclusion, it might not appear in the document produced by this 
> working group.

That's fine, but I don't think it would be productive to be any more 
tentative than that ("it probably will not appear in the document").  If 
the work really isn't suitable for inclusion, this will surely come out 
in the discussions.  We have to trust ourselves to make a good decision.