Subject: Re: [Ltru] I'm really confused by chinese in 3066bis

From: "Doug Ewell" <[email-masked]>
Date: Wed, 7 May 2008 20:26:38 -0600

Peter Constable <petercon at microsoft dot com> wrote:

> I will remind people that last November I sent an attachment with my 
> analysis of the macro-language entries in ISO 639-3, attached again 
> here for convenience, in which I take a stab at categorizing according 
> to degree of usage in ICTs and whether there is a single, dominant 
> language. Here is group #2:
>
> 2. Macrolanguage ID used in ICT, and there is a single, dominant 
> variety
> ara     (E15: arb dominant) = ar
> kok     (E14: knn dominant)
> msa     (E15: mly dominant) = ms
> swa     (E15: swh dominant) = sw (CLDR)
> uzb     (E15: uzn dominant) = uz
> zho     (E14: collection) = zh
>        cmn
> ...

Peter, I personally am comfortable with your research and judgment as to 
these cases.  I'm also comfortable with the implication that the other 
45-odd macrolanguages do not, by implication, have a dominant language 
such that the same rules would apply.

Where I become uncomfortable is with the idea that the rest of the world 
will accept this research and judgment.  You see how often people want 
to second-guess the work of SIL and the ISO MAs.  And you see the 
ongoing controversy over Suppress-Script, which as Stéphane points out 
is much more cut-and-dried than this.

I think if we (LTRU) head down this path of specifying "dominant" 
languages in any way more prescriptive than the wording we already have 
in draft-4646bis, we will be asking for endless controversy and 
accusations and haranguing.

--
Doug Ewell  *  Arvada, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Wed, 7 May 2008 19:41:31 -0700

We wouldn't necessarily have had to say these were dominant; we could simply have said "X applies to the following cases" and left it at that.

Peter

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Doug Ewell
> Sent: Wednesday, May 07, 2008 7:27 PM
> To: LTRU Working Group
> Subject: Re: [Ltru] I'm really confused by chinese in 3066bis
>

---

From: "Debbie Garside" <[email-masked]>
Date: Thu, 8 May 2008 10:21:37 +0100

Doug wrote:

> Where I become uncomfortable is with the idea that the rest
> of the world will accept this research and judgment.  You see
> how often people want to second-guess the work of SIL and the
> ISO MAs.

Firstly, linguistics is not an exact science and the old adage of you can
please some of the people some of the time etc. applies here.  I have not
got a problem with SIL, ISO or anyone else (and I know that you haven't
either).

Secondly, it is not a question of accepting research and judgement, it is a
question of standardization.  We are in the business of standardization and
sometimes tough decisions have to be taken for stability reasons.  Usually,
when we have problems with an ISO standard we go back to ISO and ask them to
sort it.  However. I think the decisions that Peter and Gary made with
regard to Macrolanguages were correct (and possibly the only decisions that
could have been made without deprecating all the macrolanguage codes - not a
good idea considering the extent of their historical usage).

So there is no scope for ISO to assist here.  But we cannot just sit on the
fence to the detriment of the work that we have spent the last few years
developing.  Let's put this into perspective.  We are talking about the
codes for some of the most spoken languages in the world here and we need to
get it right.

I think that where this especially goes awry is with the adjective
"dominant" (as pointed out by Shawn or Leif earlier) and would propose that
we change the description  to "most prolific" or some such.  I also think
that the RFC can be worded such that, however this information is
incorporated, it is deemed as guidance for backward compatibility purposes
and not a political statement about the languages themselves.

Just to be clear on my position, I do not, as stated previously, think,
under any circumstances, that 'ar' should be used in place of 'arb'.  All
that is required within the LSR is an indicator that 'ar' is the container
for several arabic languages and where it has been used historically the
related language code 'arb' is considered the most prolific of those
contained langauges for matching purposes.

I still think adding ;1 to the "dominant" language within the Macrolanguage
field would work.  Anyone able to tell me why it wouldn't?

best

Debbie

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Doug Ewell
> Sent: 08 May 2008 03:27
> To: LTRU Working Group
> Peter, I personally am comfortable with your research and
> judgment as to these cases.  I'm also comfortable with the
> implication that the other 45-odd macrolanguages do not, by
> implication, have a dominant language such that the same
> rules would apply.
>
> Where I become uncomfortable is with the idea that the rest
> of the world will accept this research and judgment.  You see
> how often people want to second-guess the work of SIL and the
> ISO MAs.  And you see the ongoing controversy over
> Suppress-Script, which as Stéphane points out is much more
> cut-and-dried than this.
>
> languages in any way more prescriptive than the wording we
> already have in draft-4646bis, we will be asking for endless
> controversy and accusations and haranguing.
>
> --

---

From: Peter Constable <[email-masked]>
Date: Thu, 8 May 2008 09:33:33 -0700

> Debbie Garside

> "dominant" (as pointed out by Shawn or Leif earlier) and would propose
> that we change the description  to "most prolific" or some such.

What people are seeing here is exactly the "major-language-variety" principle that Gary Simons and I came up with back when we began our analysis of ISO 639-2. You can find the complete details on that work here:

https://www.ethnologue.com/14/iso639/An_analysis_of_ISO_639.pdf

Here's the bit on MLV:

   4. The mapping for an individual language code should be to
      the unique language (if any) that resembles a major language.

Such assessment of languages was based on a prototype definition using various criteria,
including the following:

- The language is spoken by a substantially larger population than the other potential
candidates (we looked for a ratio on the order of 10:1 or greater).

- The language has literature, and preferably an established tradition of internal literature
development.*

- There is evidence that the language is used in mass media.

- The language is an official language in one or more of the countries in which it is spoken.

# That tradition may have been recently introduced, though a long-standing tradition provides better evidence.

(A prototype definition for a concept means that all of the criteria apply to an instance in the _ideal_ case, but it's not necessary that all apply in every case, and there's a fuzzy boundary. So, some entities are better fits than others, some are clearly in and others clearly out, and some are in between.)

Peter

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 8 May 2008 22:59:50 -0600

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> ... All that is required within the LSR is an indicator that 'ar' is
> the container for several arabic languages and where it has been used
> historically the related language code 'arb' is considered the most
> prolific of those contained langauges for matching purposes.
>
> I still think adding ;1 to the "dominant" language within the
> Macrolanguage field would work.  Anyone able to tell me why it
> wouldn't?

So are you suggesting something like the following?

%%
Type: language
Subtag: ar
Description: Arabic
Added: 2005-10-16
Suppress-Script: Arab
%%

(same as we already have in 4645bis)

%%
Type: language
Subtag: acq
Description: Ta'izzi-Adeni Arabic
Added: 2029-09-09
Macrolanguage: ar
%%

(same as we already have in 4645bis)

%%
Type: language
Subtag: arb
Description: Standard Arabic
Added: 2029-09-09
Macrolanguage: ar;1
%%

(note the ";1" added to the Macrolanguage field indicating that this
encompassed language is considered "dominant," however we define that)

While I'm still very nervous about making "dominant language"
pronouncements in the Registry for several reasons, I have to say that,
structurally speaking, this is the most palatable solution I've seen
yet.  I'm not sure about the ;2 ;3 ;4 options, but there was no
requirement to carry it to that level.  What do others think?

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 9 May 2008 09:46:15 +0100

Hi Doug

yes, this is exactly what I mean.

Thanks.

Debbie

> From: Doug Ewell [mailto:[email-masked]]
> Sent: 09 May 2008 06:00
> To: LTRU Working Group
> Cc: Debbie Garside
> (note the ";1" added to the Macrolanguage field indicating
> that this encompassed language is considered "dominant,"
> however we define that)
>
> pronouncements in the Registry for several reasons, I have to
> say that, structurally speaking, this is the most palatable
> solution I've seen yet.  I'm not sure about the ;2 ;3 ;4
> options, but there was no requirement to carry it to that
> level.  What do others think?
>
> --
> http://www.alvestrand.no/mailman/listinfo/ietf-languages  ^
>
>
>
>

---

From: "Phillips, Addison" <[email-masked]>
Date: Fri, 9 May 2008 08:46:13 -0700

> requirement to carry it to that level.  What do others think?

I am completely opposed to including pronouncements of "dominant language" in the registry. I'm not wild about making those pronouncements anywhere and tend to favor removing our statements about it in the drafts.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

---

From: "Debbie Garside" <[email-masked]>
Date: Fri, 9 May 2008 17:31:03 +0100

Hi Doug

As previously stated, this is what I mean but I think we need to also
identify the "container".

I am looking at future matching scenarios and wondering whether there is a
need to put something further within the Macrolanguage to facilitate
fallback the other way; e.g. 'ar' falls back to 'arb' then 'arq' etc.  It
has been clearly stated that there is an awful lot of digital data tagged
with macrolanguages.  Assuming that the RFC will in some way or another
recommend to tag wisely on this, e.g. 'arb' for standard Arabic, there will
be a future need to match 'ar' with the languages within it.  However, you
first need to identify that 'ar' itself is a Macrolanguage then you can do a
match on: Macrolanguage field = 'ar;1' or 'ar'.

Therefore, the record for 'ar' needs to be updated as follows:

%%
Type: language
Subtag: ar
Description: Arabic
Added: 2005-10-16
Suppress-Script: Arab
Macrolanguage: True
%%

Does that make sense?

Best

Debbie

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 04:53:04 +0200

Debbie Garside 2008-05-09 18.31:

> ... facilitate fallback the other way; e.g. 'ar' falls back to 'arb'
> then 'arq' etc.

For Norwegian and HTTP, the future is here now: My browser might ask for 
'no' without getting stuff tagged as 'nn' or 'nb'.

> ... a future need to match 'ar' with the languages within it. However, you
> first need to identify that 'ar' itself is a Macrolanguage 

Amen.

> then you can do a match on: Macrolanguage field = 'ar;1' or 'ar'.
>

Lot's of sence to me.
-- 
leif halvard silli

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 9 May 2008 23:17:36 -0600

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> ... However, you first need to identify that 'ar' itself is a 
> Macrolanguage then you can do a match on: Macrolanguage field = 'ar;1' 
> or 'ar'.
>

This would define the initial set of values for the Macrolanguage field 
as "ak", "ar", "ay", "az", "bal", etc., and also "True".  Is that right?

Why don't we just use the strategy we already have: if at least one 
language subtag "yyy" exists with a Macrolanguage field of "xxx", then 
"xxx" is a macrolanguage.  This is a bit more work for us programmer 
types, since it means we have to look through 7,768 language subtags, 
but that's our problem; everyone else can just use the 
Registry-searching engines we build.

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 14:21:03 +0200

Doug Ewell 2008-05-10 07.17:
> "xxx" is a macrolanguage.

Why would that be any better?

>   This is a bit more work for us programmer 

If BCP47 list doesn't list it, then one can always create alternative 
lists ... Programmer types can develope search tools anyhow. Programmer 
types outside IETF needs hooks and help as well, for developing their 
own tools. (Just look at Apache ...)

There allready is at least one search tool. But I thought the goal was 
to have only one place to look at. The search tools should also not 
appear to give more info than the list. If so, then the searh tool can 
be intrepreted as "interpretative".

Many will just look for Arab, and settle when they find "ar". If the 
goal is to get users to always use as spesific tags as needed/possible, 
then this is not good enough. If the field Macrolanguage answers "true", 
then at least there exist a insentive to try to understand what that as is.

Mentioning that it is a macrolanguage is neutral info. Developers/Users 
must then find out what to do with it - if it can be used for fallback, 
or whatever.
-- 
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 10 May 2008 09:20:45 +0100

Hi Doug

Yes, I could see that you could get a reverse match from the languages
containing the Macrolanguage field but I still feel that the actual
macrolanguage should be labelled.  If only to stop people from assuming that
they should tag all Arabic as 'ar'.  If I was tagging a document and looking
for the right code I might well do a search for "Arabic" or "Chinese".  The
result would not identify the subtag as a Macrolanguages and thus I may not
look any further.

For the sake of 50 or so fields, I think it is worth putting a Macrolangauge
field in the Registry - for humans to use :-)

Best

Debbie

> Sent: 10 May 2008 06:18
> This would define the initial set of values for the
> Macrolanguage field as "ak", "ar", "ay", "az", "bal", etc.,
> and also "True".  Is that right?
>
> Why don't we just use the strategy we already have: if at
> least one language subtag "yyy" exists with a Macrolanguage
> field of "xxx", then "xxx" is a macrolanguage.  This is a bit
> more work for us programmer types, since it means we have to
> look through 7,768 language subtags, but that's our problem;
> everyone else can just use the Registry-searching engines we build.
>
> --

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 10 May 2008 12:47:47 -0600

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

> macrolanguage should be labelled.  If only to stop people from 
> assuming that they should tag all Arabic as 'ar'.  If I was tagging a 
> document and looking for the right code I might well do a search for 
> "Arabic" or "Chinese".  The result would not identify the subtag as a 
> Macrolanguages and thus I may not look any further.
>
> For the sake of 50 or so fields, I think it is worth putting a 
> Macrolangauge field in the Registry - for humans to use :-)

I see three particular problems with using the Macrolanguage field to 
mean two opposite concepts, "this language HAS a macrolanguage" and 
"this language IS a macrolanguage":

1.  The software problem.  In beginning programming you learn to use 
values like -1 to mean "not a valid value" or "end of list" or similar. 
This works OK when the real values are non-negative, such as the 
population of a town, but not so well when the values could be negative, 
such as its elevation.  Also, other parts of the software have to know 
to treat -1 as a special case, not like an ordinary value.  Sometimes 
this gets confusing and you see -1 pop up in places it shouldn't.  In 
intermediate programming you learn to stop doing this, and represent 
special situations in other ways

Similarly, it is possible to imagine software looking fruitlessly for a 
language subtag 'True' that is the macrolanguage of 'ar' instead of 
remembering that 'True' is a special case.  Remember that 4-letter 
language subtags, though "reserved for future use" (for some standard, I 
forget which ;-), are valid in the ABNF, and that the casing of subtags 
doesn't matter, though we're supposed to get it right in the Registry.

2.  The human problem.  I can easily imagine readers of the Registry 
becoming confused over this dual usage of a single field.  They might 
wonder why other subtags don't have "Macrolanguage: False", or why 'ar' 
is considered the opposite of 'True'.  They might also experience 
confusion similar to problem 1, as 'tru' is a valid RFC 4646bis language 
subtag for Turoyo.  (At least the proposal wasn't to use "Macrolanguage: 
yes", thus causing instant confusion with Yeskwa.)

3.  The maintenance problem.  Technically the 'True' value is redundant 
information; it can be derived from the records for the encompassed 
languages.  Any time you have to maintain redundant information, 
especially in a different place, there is a much greater chance of 
making a human mistake.

Suppose your friendly team of Designated Experts, presented with a new 
batch of several dozen ISO 639-3 changes including a new classification 
of macrolanguage 'qma' with encompassed languages 'qea' and 'qeb', 
remembers to put "Macrolanguage: qma" on the records for 'qea' and 'qeb' 
but forgets to put "Macrolanguage: True" on the record for 'qma'. 
Suppose further that the ietf-languages list didn't catch this during 
the 1-week review.  We would end up with an internal inconsistency 
within the Registry.  Gosh, your friendly Experts would hate that.  They 
would also hate the inevitable e-mail flames about "process failure" and 
the possibility of removal or replacement at the IESG's discretion.

If it is really felt necessary to indicate that 'ar' is a macrolanguage 
in both ways, with a special value on the macrolanguage record as well 
as the encompassed languages (ignoring problem 3), then we should have 
two fields, something like Is-A-Macrolanguage and My-Macrolanguage-Is. 
(Suggestions for better names are solicited.)  As with the Comments 
field, I don't support overloading a single field for fundamentally 
different purposes.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 10 May 2008 12:57:33 -0600

Leif Halvard Silli <lhs at malform dot no> wrote:

>> language subtag "yyy" exists with a Macrolanguage field of "xxx", 
>> then "xxx" is a macrolanguage.
>

Because it's what we already do?  Also see a longer explanation in the 
reply to Debbie.

>> This is a bit more work for us programmer types, since it means we 
>> have to look through 7,768 language subtags, but that's our problem; 
> lists ...

BCP 47 would indeed list it, just under the encompassed languages and 
not under the macrolanguages themselves.  If you look for "Arabic" and 
find "Macrolanguage: True", you're going to have to search through the 
whole Registry anyway to find what it's a macrolanguage of.

> Programmer types can develope search tools anyhow. Programmer types 
> outside IETF needs hooks and help as well, for developing their own 
> tools. (Just look at Apache ...)

Are the descriptions of the Macrolanguage field in 4646bis really not 
adequate for this?

> goal is to get users to always use as spesific tags as 
> needed/possible, then this is not good enough. If the field 
> Macrolanguage answers "true", then at least there exist a insentive to 
> try to understand what that as is.

With 7,700 language subtags to choose from, simply stopping when you 
find the word "Arabic" is not going to be sufficient anyway.

> Mentioning that it is a macrolanguage is neutral info. 
> Developers/Users must then find out what to do with it - if it can be 
> used for fallback, or whatever.

Agreed.  What I'm saying is that we already provide this information in 
the draft-4645bis Registry, and document it in draft-4646bis, sections 
3.1.9 and 4.1.  I'm not convinced that this needs to be changed.

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 10 May 2008 12:01:42 -0700

If we need to indicate that a language is a macrolanguage in the registry, we should use a separate field. Presumably it would be something like:

Subtag: zh
Type: language
Encompasses: cmn, yue, wuu, ... etc.

Or perhaps:

Subtag: zh
Type: language
Encompasses: cmn
Encompasses: yue
Encompasses: wuu
... etc ...

And the encompasses languages would look like:

Subtag: cmn
Type: language // or extlang
Macrolanguage: zh

If we were to restore the use of 'extlang', perhaps the above wouldn't be necessary, since enclosed languages would not be of the same type and the 'cmn' record would look more like:

Subtag: cmn
Type: extlang
Macrolanguage: zh
Prefix: zh

... but we would have some enclosed languages grandfathered into the language slot:

Subtag: nn
Type: language
Macrolanguage: no

Addison

Addison Phillips

> Sent: Saturday, May 10, 2008 11:48 AM
> To: LTRU Working Group

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 10 May 2008 21:12:23 +0100

Why not:

Subtag: zh
Type1: Language
Type2: Macrolanguage

Indicating it can be used as a standalone language subtag or as part of an
extension subtag.

and...

Subtag: cmn
Type1: language
Type2: extlang
Macrolanguage: zh

Indicating that it may be presented as a language subtag but if it is
implementations should convert it to extlang using the Macrolanguage field
value.

This is where dual use comes in.  It doesn't matter how many pages you write
in the RFC saying how macrolanguage and extlang work people will still tag
with any of 'zh', 'cmn', or 'zh-cmn' and it shouldn't matter which they
choose as the registry facilitates fallback for all 3.

Best

Debbie

> Behalf Of Phillips, Addison
> Sent: 10 May 2008 20:02
> To: Doug Ewell; LTRU Working Group
> If we need to indicate that a language is a macrolanguage in
> the registry, we should use a separate field. Presumably it
> would be something like:
>
> Subtag: zh
> Type: language
> If we were to restore the use of 'extlang', perhaps the above
> wouldn't be necessary, since enclosed languages would not be
> of the same type and the 'cmn' record would look more like:
>
> Subtag: cmn
> Type: extlang
> Macrolanguage: zh
> Prefix: zh
>
> > From: [email-masked] [mailto:[email-masked]]
> On Behalf
> > Of Doug Ewell
> > > assuming that they should tag all Arabic as 'ar'.  If I
> was tagging
> > > a document and looking for the right code I might well do
> a search
> > > for "Arabic" or "Chinese".  The result would not identify
> the subtag
> > > as a Macrolanguages and thus I may not look any further.
> > >
> > population of a town, but not so well when the values could be
> > negative, such as its elevation.  Also, other parts of the software
> > have to know to treat -1 as a special case, not like an ordinary
> > value.  Sometimes this gets confusing and you see -1 pop up
> in places
> > it shouldn't.  In intermediate programming you learn to stop doing
> > this, and represent special situations in other ways
> >
> > Similarly, it is possible to imagine software looking
> fruitlessly for
> > a language subtag 'True' that is the macrolanguage of 'ar'
> instead of
> > language subtags, though "reserved for future use" (for
> some standard,
> > I forget which ;-), are valid in the ABNF, and that the casing of
> > subtags doesn't matter, though we're supposed to get it
> right in the
> > Registry.
> >
> > confusion similar to problem 1, as 'tru' is a valid RFC 4646bis
> > language subtag for Turoyo.  (At least the proposal wasn't to use
> > "Macrolanguage:
> > 3.  The maintenance problem.  Technically the 'True' value is
> > redundant information; it can be derived from the records for the
> > encompassed languages.  Any time you have to maintain redundant
> > information, especially in a different place, there is a
> much greater
> > chance of making a human mistake.
> >
> > batch of several dozen ISO 639-3 changes including a new
> > classification of macrolanguage 'qma' with encompassed
> languages 'qea'
> > and 'qeb', remembers to put "Macrolanguage: qma" on the records for
> > 'qea' and 'qeb'
> > If it is really felt necessary to indicate that 'ar' is a
> > macrolanguage in both ways, with a special value on the
> macrolanguage
> > record as well as the encompassed languages (ignoring
> problem 3), then
> > we should have two fields, something like
> Is-A-Macrolanguage and My-Macrolanguage-Is.

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 22:13:33 +0200

> Encompasses: cmn, yue, wuu, ... etc.

Sounds like a good way to do it.

[...]
> ... but we would have some enclosed languages grandfathered into the language slot:

For this reason, I prefer using

    "Encompasses: cmn, yue, wuu, ... etc."

instead of going the extlang route. Treating all macrolanguages the same 
way increases the  chance the issue is looked at and understood, for all 
the related languages.
-- 
leif halvard silli

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 10 May 2008 21:27:25 +0100

Doug wrote:

> I see three particular problems with using the Macrolanguage
> field to mean two opposite concepts, "this language HAS a
> macrolanguage" and "this language IS a macrolanguage":
>
> 1.  The software problem.  In beginning programming you learn
> to use values like -1 to mean "not a valid value" or "end of
> list" or similar.
> This works OK when the real values are non-negative, such as
> the population of a town, but not so well when the values
> could be negative, such as its elevation.  Also, other parts
> of the software have to know to treat -1 as a special case,
> not like an ordinary value.  Sometimes this gets confusing
> and you see -1 pop up in places it shouldn't.  In
> intermediate programming you learn to stop doing this, and
> represent special situations in other ways

OK

> fruitlessly for a language subtag 'True' that is the
> macrolanguage of 'ar' instead of remembering that 'True' is a
> special case.

Not if it is programmed correctly :-)

>Remember that 4-letter language subtags,
> though "reserved for future use" (for some standard, I forget
> which ;-), are valid in the ABNF, and that the casing of
> subtags doesn't matter, though we're supposed to get it right
> in the Registry.
>
> 2.  The human problem.  I can easily imagine readers of the
> Registry becoming confused over this dual usage of a single
> field.  They might wonder why other subtags don't have
> "Macrolanguage: False", or why 'ar'
> is considered the opposite of 'True'.  They might also
> experience confusion similar to problem 1, as 'tru' is a
> valid RFC 4646bis language subtag for Turoyo.  (At least the
> proposal wasn't to use "Macrolanguage:

OK

> redundant information; it can be derived from the records for
> the encompassed languages.  Any time you have to maintain
> redundant information, especially in a different place, there
> is a much greater chance of making a human mistake.

OK

>
> Suppose your friendly team of Designated Experts, presented
> with a new batch of several dozen ISO 639-3 changes including
> a new classification of macrolanguage 'qma' with encompassed
> languages 'qea' and 'qeb', remembers to put "Macrolanguage:
> qma" on the records for 'qea' and 'qeb'

I would sack them without hesitation ;-)

> Suppose further that the ietf-languages list didn't catch
> this during the 1-week review.  We would end up with an
> internal inconsistency within the Registry.  Gosh, your
> friendly Experts would hate that.  They would also hate the
> inevitable e-mail flames about "process failure" and the
> possibility of removal or replacement at the IESG's discretion.

Now come, come, this is not the GNSO GA forum you know!

> macrolanguage record as well as the encompassed languages
> (ignoring problem 3), then we should have two fields,
> something like Is-A-Macrolanguage and My-Macrolanguage-Is.
> (Suggestions for better names are solicited.)  As with the
> Comments field, I don't support overloading a single field
> for fundamentally different purposes.

OK  I could try to argue but your reasoning is quite sound and it is
Saturday night and I have a glass of red wine waiting :-)

Best

Debbie

---

From: Leif Halvard Silli <[email-masked]>
Date: Sat, 10 May 2008 22:28:28 +0200

> Why not:
....
> choose as the registry facilitates fallback for all 3.

So for Nynorsk, then we would simply just drop the Type2: field, I 
suppose? If so, then I would see this as the better solution, as it 
would be "compatible" with and optimal for all the macrolanguages.

[ I will probably take a break from this list 2-3 days. ]
-- 
best regards
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Debbie Garside" <[email-masked]>
Date: Sat, 10 May 2008 21:49:01 +0100

Actually, logically perhaps it should read:

Subtag: zh
Type: Macrolanguage

and...

Subtag: cmn
Type: extlang
Macrolanguage: zh;1

Subtag: yue
Type: extlang
Macrolanguage: zh

You can easily programme into implementations the scenarios for when a user
has used an extlang as a primary language subtag and you don't need a type 2
on the macrolanguage to say that it can be used as a standalone primary
language subtag - that's covered by the text.

Debbie

> Registry
> with a new
> this during

---

From: "Phillips, Addison" <[email-masked]>
Date: Sat, 10 May 2008 14:06:53 -0700

> language subtag - that's covered by the text.

No, actually we can't "program implementations" without breaking existing "well-formed" implementations (which I think represents the majority of implementations).

Reliance on metadata in the registry is, of course, something you are familiar and comfortable with---ISO 639-6 relies on it. But BCP 47--and the matching schemes in particular--have historically eschewed such reliance. If we introduce reliance on the registry, it breaks the implementations that rely solely on tag structure. Arguments between extlangistas and non-extlangistas have greatly revolved around this particular problem.

If this were just a one-time thing, such as the grandfathered list, it might not be so bad, since we could give the list to hardcode in the RFC. But there is no indication that the macrolanguage list is closed or particularly stable (even if entries are stable, new ones will probably be created).

BCP 47 is not the perfect language identifier system that suits all needs everywhere. It is a compromise that suits many applications.

Of course, I'm also pretty sure that a "perfect" language identifier system is not likely to appear, since any such system must make choices about how to split or lump languages and thus will ill-serve some application.

Addison Phillips

> From: Debbie Garside [mailto:[email-masked]]
> Sent: Saturday, May 10, 2008 1:49 PM
> To: Phillips, Addison; 'Doug Ewell'; 'LTRU Working Group'
> Subject: RE: [Ltru] I'm really confused by chinese in 3066bis
>
> > > would also hate the inevitable e-mail flames about "process
> failure"
> > > and

---

From: Peter Constable <[email-masked]>
Date: Sat, 10 May 2008 16:46:35 -0700

In terms of process and data integrity, I'm with Doug on this: don't conflate a data category; if you want pointers in both directions, have separate fields, or just treat your "true" data point as an informative annotation, adding "This is a macrolanguage entry" (or something like that) as a comment.

Peter

> language

---

From: Peter Constable <[email-masked]>
Date: Sat, 10 May 2008 16:51:19 -0700

This is getting worse by the moment: let's see just have many syntactic changes to our record-jar data files we can make, shall we?

Peter

> To: 'Phillips, Addison'; 'Doug Ewell'; 'LTRU Working Group'

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 10 May 2008 19:45:07 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

> If we need to indicate that a language is a macrolanguage in the 
> registry, we should use a separate field. Presumably it would be 
> something like:
>
> Subtag: zh
> Type: language

No other part of the Registry syntax requires record-jar parsers to 
subparse a comma-separated field.  I opposed this back during the 
discussions that ultimately led to the Macrolanguage field, and I oppose 
it now.

> ... etc ...

Much better syntax, though I maintain that the information is already in 
the Registry (in the records for 'cmn', 'yue', 'wuu', etc.) and that 
this "doubly-linked list" approach is unnecessary.

> Type: language // or extlang

As they do now.

> If we were to restore the use of 'extlang', perhaps the above wouldn't 
> be necessary, since enclosed languages would not be of the same type 
> and the 'cmn' record would look more like:
>
> Subtag: cmn
> Type: extlang
> Macrolanguage: zh
> Prefix: zh
>

This is what we were going to do anyway, before extlangs were removed. 
The Type and Prefix fields would tell how the subtag was to be used, 
while the Macrolanguage field would be used to offer suggestions to 
taggers and matchers about possibly related languages, as it is in the 
current draft.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 10 May 2008 19:52:12 -0600

Peter Constable <petercon at microsoft dot com> wrote:

> In terms of process and data integrity, I'm with Doug on this: don't 
> conflate a data category; if you want pointers in both directions, 
> have separate fields, or just treat your "true" data point as an 
> informative annotation, adding "This is a macrolanguage entry" (or 
> something like that) as a comment.

Choose one:

1.  The annotation is required for all macrolanguage records.
2.  The annotation is in the Comments field.

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 10 May 2008 20:03:41 -0600

Debbie Garside <debbie at ictmarketing dot co dot uk> wrote:

>> Similarly, it is possible to imagine software looking fruitlessly for 
>> a language subtag 'True' that is the macrolanguage of 'ar' instead of 
>> remembering that 'True' is a special case.
>

Check the ISO 639-6 data (I can't).  Is there any 4-letter code element 
with value 'true'?  Can you guarantee there won't be one in the future? 
Can you guarantee that Registry consumers who read this in 
draft-4646bis, Section 2.1:

"At all times, the tags and their subtags, including private use and 
extensions, are to be treated as case insensitive: there exist 
conventions for the capitalization of some of the subtags, but these 
MUST NOT be taken to carry meaning."

will still be careful to distinguish 'true' from 'True'?

>> Suppose your friendly team of Designated Experts, presented with a 
>> new batch of several dozen ISO 639-3 changes including a new 
>> classification of macrolanguage 'qma' with encompassed languages 
>> 'qea' and 'qeb', remembers to put "Macrolanguage: qma" on the records 
>> for 'qea' and 'qeb' but forgets to put "Macrolanguage: True" on the 
>> record for 'qma'.
>

I'm pretty sure you're kidding, but we've seen Michael get attacked on 
this list and ietf-languages for human errors such as this.

>> within the Registry.  Gosh, your friendly Experts would hate that. 
>> They would also hate the inevitable e-mail flames about "process 
>> failure" and the possibility of removal or replacement at the IESG's 
>> discretion.
>

One LTRU member, who has since left the WG, used the term "process 
failure" to describe the addition of new region subtags AC, CP, DG, and 
so forth.  While the IESG might not take this seriously, the objections 
can become pretty disconcerting.  The "removal or replacement" bit is 
from RFC 4646 and the current draft.

---

From: Leif Halvard Silli <[email-masked]>
Date: Sun, 11 May 2008 14:56:05 +0200

> Macrolanguage: zh

Instead of setting a number after the Macrolanguage field,

    Macrolanguage: zh;1

which it seems as if you have dropped anyhow, I would instead propose, 
building on Addison Phillips idea about a Encopassed field, to simply 
insert the Encompassed languages in a **particular order** - or at least 
place the dominant and second most dominant language first:

    Subtag: zh
    Type: language
    Encompasses: cmn, yue, wuu, etc.

Omitting the numbers has the advantage of not ranking the languages with 
a number. It also has the advantage of perhaps giving implementors the 
clue that 'cmn' can only be assumed to be number 1 as long as the 
application itself is localized into 'cmn'.

For a user of Cantonese, the default starting point shouldbe 'yue'. 
Thenafter it should go to the left and lookup 'cmn'. Thenafter either 
one of the other Encompassed languages - or a third langauge. (The issue 
of simplified versus traditional Chinese complicates even further, of 
course.)

To explain better what I mean, let's imagine that the encompassed 
languages had numbers instead of alphabetical codes - and imagine that 
the number '0' = macrolanguage. Thus for Chinese 0 = 'zh'. Then, for the 
dominant language within a macrolanguage, we would get this priority order:

    10, 2, (3, 4)

Note that '10' stands for 'cmn' (1) + zh' (0). It should check for 'cmn' first, but at the same time consider 'zh' as part of 'cmn' if 'cmn' is unavailable.

Then I would suggest that if the localisation of for instance the Web browser is the Encompassed language number 2, then browser should assume this *default* order instead:

    20, 1, (3, 4)

Again 'yum' (2) and 'zh' (0) are concatenated into 20.

Wheras an applications localized for Encompassed language 4 should 
rather assuem this default order:

    40, 1, (2, 3)

The paranthesis is meant to indicate that it is lesser certainity about 
whether the users actually would appreciate those languages in the 
fallback order.

Because, users of Encompassed language 2, 3 and 4 would probably all of 
them want to also include encompassed language number 1 as part of their 
fallback scheme. Thus a user of an Encompassed languages which isn't the 
dominant language, would normally want at least 3 codes as part of their 
lookup: the particular language itself, the macrolanguage and the 
dominant language. Their preference _after_ those 3 codes, is a bit more 
difficult to say anything about.

The users of dominant language, though, often requires perhaps would 
want dominant language, macrolanguage and often also the second most 
popular language. Thus, like this:

    10, 2, (3, 4)

Although users of a dominant language may sometimes prefer to set an 
paranthesis around all the Encompassed languages, except their own:

    10, (2, 3, 4)

I think it is important that the macrolanguage, 'zh' for instance, is 
not always assumed to be the dominant language. I.e. 'cmn' should not 
always be considered #1 within 'zh'. Doing so would only make the link 
between 'zh' and 'cmn' stronger. That link should only be made when the 
'zh' application actually is a 'cmn' application. If the localisation 
actually is a 'yum' applcation, then 'zh' should primarely be associated 
with 'yum'.

The twist is, that even if 'yum' gets associated with 'zh', then users 
would still - for example in the Web world - for the most part receive 
'cmn' content. But that is fine. That is what the users expect.

Questions to you: Do you agree that the above is a useful way of 
deciding fallback based on macrolangauge information? Or do you find 
anythign of this questionable? Localizers must of course always judge 
thing carefully, but to me the above seems sensisble, if I may say so.

If you agree with my thoughts here, then I hope it somehow can be 
incorporated into the RFC.

(I just  had to get this of my chest. I'll read mail tomorrow.)
leif halvard silli
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 11 May 2008 08:11:00 -0700

I agree.

   1. There is absolutely no need for having an "encompassed" field, since
   all the data is in the repository already; it would be completely redundant.
   2. If we wanted a field to indicate that a microlanguage was the
   predominant form for its macrolanguage, it would be possible to add a field
   like "PredominantForm: true" (I don't care about the exact name), and mark
   the cases that are in Table 8 of the draft. (I don't think such a new field
   would be particularly useful, but it wouldn't hurt either.)

Mark

On Sat, May 10, 2008 at 4:51 PM, Peter Constable <[email-masked]>
wrote:

> This is getting worse by the moment: let's see just have many syntactic
> changes to our record-jar data files we can make, shall we?
>
>
> Peter
>
> > > > They

-- 
Mark

---

From: "Debbie Garside" <[email-masked]>
Date: Sun, 11 May 2008 16:42:46 +0100

Doug wrote:

> Check the ISO 639-6 data (I can't).  Is there any 4-letter
> code element with value 'true'?  Can you guarantee there
> won't be one in the future?

I have to admit it took me a while to figure out the ref to 639-6 but you
are absolutely right, there is no saying that an alpha4 won't be used within
a Macrolangauge field in the future.  Good catch!

> I'm pretty sure you're kidding, but we've seen Michael get
> attacked on this list and ietf-languages for human errors
> such as this.

I am :-) But perhaps in bad taste... forgetting the history.

> One LTRU member, who has since left the WG, used the term
> "process failure" to describe the addition of new region
> subtags AC, CP, DG, and so forth.  While the IESG might not
> take this seriously, the objections can become pretty
> disconcerting.  The "removal or replacement" bit is from RFC
> 4646 and the current draft.

I see your point... being quite reasonable people ourselves we sometimes
forget that there are unreasonable people out there :-)

Best

Debbie

> Sent: 11 May 2008 03:04
> Check the ISO 639-6 data (I can't).  Is there any 4-letter
> code element with value 'true'?  Can you guarantee there
> won't be one in the future?
> "At all times, the tags and their subtags, including private
> use and extensions, are to be treated as case insensitive:
> there exist conventions for the capitalization of some of the
> subtags, but these MUST NOT be taken to carry meaning."
>
> I'm pretty sure you're kidding, but we've seen Michael get
> attacked on this list and ietf-languages for human errors
> such as this.
>
> One LTRU member, who has since left the WG, used the term
> "process failure" to describe the addition of new region
> subtags AC, CP, DG, and so forth.  While the IESG might not
> take this seriously, the objections can become pretty
> disconcerting.  The "removal or replacement" bit is from RFC
> 4646 and the current draft.
>
> --

---

From: Peter Constable <[email-masked]>
Date: Sun, 11 May 2008 15:21:33 -0700

> Leif Halvard Silli

> deciding fallback based on macrolangauge information?

Sorry, but no.

Peter

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 12 May 2008 08:45:11 +0100

I agree with Peter.  It seems a little more complex than is actually
necessary.

Best

debbie

> Behalf Of Peter Constable
> Sent: 11 May 2008 23:22
> To: 'LTRU Working Group'
> > Of Leif Halvard Silli

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 12 May 2008 09:01:11 +0100

Addison wrote:

> No, actually we can't "program implementations" without
> breaking existing "well-formed" implementations (which I
> think represents the majority of implementations).

I am not sure I understand this.

> Reliance on metadata in the registry is, of course, something
> you are familiar and comfortable with---ISO 639-6 relies on
> it. But BCP 47--and the matching schemes in particular--have
> historically eschewed such reliance. If we introduce reliance
> on the registry, it breaks the implementations that rely
> solely on tag structure. Arguments between extlangistas and
> non-extlangistas have greatly revolved around this particular problem.

Or this... I take it you mean that implementations are just looking at the
syntax to see if it is wellformed and not hardcoding the registry within.
Can you give an example?

> If this were just a one-time thing, such as the grandfathered
> list, it might not be so bad, since we could give the list to
> hardcode in the RFC. But there is no indication that the
> macrolanguage list is closed or particularly stable (even if
> entries are stable, new ones will probably be created).

But that is same across the Registry - new entities are created all the
time.

> BCP 47 is not the perfect language identifier system that
> suits all needs everywhere. It is a compromise that suits
> many applications.
>
> Of course, I'm also pretty sure that a "perfect" language
> identifier system is not likely to appear, since any such
> system must make choices about how to split or lump languages
> and thus will ill-serve some application.

Indeed.

Best

Debbie

> From: Phillips, Addison [mailto:[email-masked]]
> Sent: 10 May 2008 22:07
> To: [email-masked]; 'Doug Ewell'; 'LTRU Working Group'
> > You can easily programme into implementations the scenarios
> for when a
> > user has used an extlang as a primary language subtag and you don't
> > need a type 2 on the macrolanguage to say that it can be used as a
> > standalone primary language subtag - that's covered by the text.
>
> No, actually we can't "program implementations" without
> breaking existing "well-formed" implementations (which I
> think represents the majority of implementations).
>
> Reliance on metadata in the registry is, of course, something
> you are familiar and comfortable with---ISO 639-6 relies on
> it. But BCP 47--and the matching schemes in particular--have
> historically eschewed such reliance. If we introduce reliance
> on the registry, it breaks the implementations that rely
> solely on tag structure. Arguments between extlangistas and
> non-extlangistas have greatly revolved around this particular problem.
>
> If this were just a one-time thing, such as the grandfathered
> list, it might not be so bad, since we could give the list to
> hardcode in the RFC. But there is no indication that the
> macrolanguage list is closed or particularly stable (even if
> entries are stable, new ones will probably be created).
>
> BCP 47 is not the perfect language identifier system that
> suits all needs everywhere. It is a compromise that suits
> many applications.
>
> Of course, I'm also pretty sure that a "perfect" language
> identifier system is not likely to appear, since any such
> system must make choices about how to split or lump languages
> and thus will ill-serve some application.
>
> Addison Phillips
> > You can easily programme into implementations the scenarios
> for when a
> > user has used an extlang as a primary language subtag and you don't
> > need a type 2 on the macrolanguage to say that it can be used as a
> > standalone primary language subtag - that's covered by the text.
> >
> > Debbie
> >
> >
> > > From: [email-masked]
> [mailto:[email-masked]] On Behalf
> > > Of Phillips, Addison
> > > Sent: 10 May 2008 20:02
> > > wouldn't be necessary, since enclosed languages would not
> be of the
> > > same type and the 'cmn' record would look more like:
> > >
> > > Subtag: cmn
> > > Type: extlang
> > > Macrolanguage: zh
> > > Prefix: zh
> > >
> > > ... but we would have some enclosed languages
> grandfathered into the
> > > language slot:
> > >
> > > Subtag: nn
> > > Type: language
> > > Macrolanguage: no
> > >
> > > Addison
> > >
> > > Addison Phillips
> > > > mean two opposite concepts, "this language HAS a macrolanguage"
> > > > and "this language IS a macrolanguage":
> > > >
> > > > negative, such as its elevation.  Also, other parts of the
> > > > software have to know to treat -1 as a special case,
> not like an
> > > > ordinary value.  Sometimes this gets confusing and you
> see -1 pop
> > > > up
> > > in places
> experience
> > > > and 'qeb', remembers to put "Macrolanguage: qma" on the records
> > > > for 'qea' and 'qeb'
> > > > Comments field, I don't support overloading a single field for
> > > > fundamentally different purposes.
> > > >
> > > > --

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 12 May 2008 09:17:20 +0100

Just as an aside.  I checked... 'true' is not in the registry and I could
make it an unusable code for ISO 639-6 by adding it to the already
used/don't use Alpha4 check list that I have.

best

Debbie

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 12 May 2008 11:19:51 +0100

For me, record-jar is a dogs breakfast.

If we really want Joe Public to start tagging data correctly we need to
facilitate the provision of data in an XML format that can be imported into
proprietary software.  Frank and I did some work on this many moons ago and
we were about 99% of the way there - this needs to be completed and made
publicly available.  I'm not suggesting that this should form part of the
RFC but a tool should be created and (if possible) referenced within the
RFC.  (perhaps via Stephane's site)

And, whilst I am busy sticking spanners in the works, I would like to bring
up the idea (again) of being able to reference the RFC within the tag (if
desired).  This is extremely important for archiving purposes.

Best

Debbie

> From: Peter Constable [mailto:[email-masked]]
> Sent: 11 May 2008 00:51
> To: [email-masked]; 'Phillips, Addison'; 'Doug
> Ewell'; 'LTRU Working Group'
> This is getting worse by the moment: let's see just have many
> syntactic changes to our record-jar data files we can make, shall we?
>
>
> Peter
>
> > Of Debbie Garside

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 12 May 2008 08:13:41 -0700

Why bother?

I oppose having two completely different types of values in the same field. It making parsing a lot more difficult.

Addison

Addison Phillips

> Sent: Monday, May 12, 2008 1:17 AM
> To: 'Doug Ewell'; 'LTRU Working Group'

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 12 May 2008 08:21:26 -0700

> language subtag - that's covered by the text.

"Programming into implementations" information about which extlang subtags can be used as primary language subtags requires a table of items from the registry. Implementations that rely on remove-from-right solely (i.e. "well-formed" implementations, as opposed to "validating" implementations) will not have this information and will get it wrong.

> syntax to see if it is wellformed and not hardcoding the registry

Yes: well-formedness relies solely on the syntax rules. We don't require a copy of the registry for that. Validation requires a copy of the registry.

> But that is same across the Registry - new entities are created all the

New entries are created all the time. Validating implementations rely on the registry as of some particular registry date. But well-formedness is not affected by this. You don't have to know what the tags mean to do many operations on them.

What I'm suggesting is that: if we are going to *require* that tags of the form "yue-*" be treated as equivalent to tags of the form "zh-yue-*", that might be okay (I'm not convinced that it is, please note) if the list were short, closed, and stable. If, however, it changes over time, it is hard on implementations that otherwise have no use for the registry.

Addison

Addison Phillips

---

From: "Debbie Garside" <[email-masked]>
Date: Mon, 12 May 2008 16:58:02 +0100

So, basically, what I said is true as long as we don't create additional
Macrolanguages.

Are there likely to be additional Macrolanguages created in the future?  If
so, then you need to say when they can be registered - purely by following
ISO 639 or some other procedure.

If we take Macrolangauges as just a hack to get around what had happened
within previous ISO 639 standards there will be no problem.

best

Debbie

> Sent: 12 May 2008 16:21
> > > No, actually we can't "program implementations" without breaking
> > > existing "well-formed" implementations (which I think
> represents the
> > > majority of implementations).
> >
> "Programming into implementations" information about which
> extlang subtags can be used as primary language subtags
> requires a table of items from the registry. Implementations
> that rely on remove-from-right solely (i.e. "well-formed"
> implementations, as opposed to "validating" implementations)
> will not have this information and will get it wrong.
>
> >
> > > Reliance on metadata in the registry is, of course, something you
> > > are familiar and comfortable with---ISO 639-6 relies on
> it. But BCP
> > > 47--and the matching schemes in particular--have historically
> > > eschewed such reliance. If we introduce reliance on the
> registry, it
> > > breaks the implementations that rely solely on tag structure.
> > > Arguments between extlangistas and non-extlangistas have greatly
> > > revolved around this particular problem.
> >
> > Or this... I take it you mean that implementations are just
> looking at
> > the syntax to see if it is wellformed and not hardcoding
> the registry
> > within.
>
> Yes: well-formedness relies solely on the syntax rules. We
> don't require a copy of the registry for that. Validation
> requires a copy of the registry.
>
>
> > Can you give an example?
> >
> > > If this were just a one-time thing, such as the
> grandfathered list,
> > > it might not be so bad, since we could give the list to
> hardcode in
> > > the RFC. But there is no indication that the
> macrolanguage list is
> > > closed or particularly stable (even if entries are
> stable, new ones
> > > will probably be created).
> >
> > But that is same across the Registry - new entities are created all
> > the time.
>
> New entries are created all the time. Validating
> implementations rely on the registry as of some particular
> registry date. But well-formedness is not affected by this.
> You don't have to know what the tags mean to do many
> operations on them.
>
> What I'm suggesting is that: if we are going to *require*
> that tags of the form "yue-*" be treated as equivalent to
> tags of the form "zh-yue-*", that might be okay (I'm not
> convinced that it is, please note) if the list were short,
> closed, and stable. If, however, it changes over time, it is
> hard on implementations that otherwise have no use for the registry.
>
> Addison
>
> Addison Phillips

---

From: "Phillips, Addison" <[email-masked]>
Date: Mon, 12 May 2008 10:00:02 -0700

>
> So, basically, what I said is true as long as we don't create
> additional Macrolanguages.

Not necessarily. It still breaks existing implementations that don't know about the relationship between the subtag flavors.

>
> Are there likely to be additional Macrolanguages created in the future?

I don't really know, but suspect that the current ISO 639-3 is not the last word in language classification. I would be a little surprised if some of the languages that appear to the ISO 639-3 RA to be atomic today didn't turn out to be macrolanguages in the future.

> If

(boggle) The draft already says this. Macrolanguage relationships in the registry follow ISO 639's classifications.

> within previous ISO 639 standards there will be no problem.

Any decomposition of a language into two or more species of language will recreate the problem. Admittedly, I don't expect to see very many "major" languages (that is, ones of large commercial interest) exhibit this. But just because Quechua isn't important in commercial software doesn't make it unimportant to its speakers. So I tend to hesitate in treating "macrolanguage" as a hack.

Addison

---

From: "Doug Ewell" <[email-masked]>
Date: Mon, 12 May 2008 20:23:17 -0600

"Phillips, Addison" <addison at amazon dot com> wrote:

>> If we take Macrolangauges as just a hack to get around what had 
>> happened within previous ISO 639 standards there will be no problem.
>
> Any decomposition of a language into two or more species of language 
> will recreate the problem. Admittedly, I don't expect to see very many 
> "major" languages (that is, ones of large commercial interest) exhibit 
> this. But just because Quechua isn't important in commercial software 
> doesn't make it unimportant to its speakers. So I tend to hesitate in 
> treating "macrolanguage" as a hack.

ISO 639-3 introduces their description of macrolanguages by discussing 
the past practices of 639-1 and 639-2, but they go on to define 
macrolanguages as "clusters of closely-related language varieties 
that... can be considered distinct individual languages, yet in certain 
usage contexts a single language identity for all is needed."

I tend to agree with this practical definition, although I know there 
are some in this group who consider them just a hack.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 13 May 2008 14:25:59 +0900

At 00:21 08/05/13, Phillips, Addison wrote:

>"Programming into implementations" information about which extlang subtags 
>can be used as primary language subtags requires a table of items from the 
>registry. Implementations that rely on remove-from-right solely (i.e. 
>"well-formed" implementations, as opposed to "validating" implementations) 

[as a technical contributor, just thinking out loud]

I'm sure I forgot a lot of details, but it looks to me as if
using extlangs is a tiny bit better in this respect. What I
mean is that with something like
   zh-yue-Hant-hk
or so, the simple remove-from-right procedure would give fallbacks of:
   zh-yue-Hant
   zh-yue
   zh

It is my understanding that one of the arguments against exlangs
was that the following fallbacks would be more natular:
   zh-yue-Hant
   zh-Hant  ****
   zh

Having the relevant subtags in the tag might allow something lik
 "remove-from-right+extlang", where we would have fallbacks of:
   zh-yue-Hant
   zh-yue    **
   zh-Hant   **
   zh
or something similar. The relative order between zh-yue and zh-Hant
may need to be defined, or there may not be such a need because
we may be able to assume that for any single document (or other
matching target), we would not have zh-yue and zh-Hant at the
same time.

This wouldn't work with existing implementations, but it might
work with newer implementations. On the other hand, it would
NOT create a dependency on checking the registry to see that
zh is a fallback (or whatever we call it) for yue.

Again, for the moment, just thinking out loud.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 13 May 2008 12:58:34 +0200

On Sat, May 10, 2008 at 12:01:42PM -0700,
 Phillips, Addison <[email-masked]> wrote 

> Encompasses: cmn, yue, wuu, ... etc.

Note that it was already proposed (and rejected) one year ago.

http://www.ietf.org/mail-archive/web/ltru/current/msg08066.html
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 13 May 2008 14:57:05 +0200

On Mon, May 12, 2008 at 11:19:51AM +0100,
 Debbie Garside <[email-masked]> wrote 
 a message of 279 lines which said:

> Frank and I did some work on this many moons ago and we were about
> 99% of the way there - this needs to be completed and made publicly
> available.

As noted by Doug, this is already publicly available:

http://www.langtag.net/registries.html

Advices on the best process to debug, improve, extend, etc, this work
are welcome. The LTRU co-chairs will probably request that this is
discussed off-list.

---

From: "Debbie Garside" <[email-masked]>
Date: Tue, 13 May 2008 17:34:26 +0100

I have successfully imported the registry into MS Access using both XML
files and they work very well.  Couple of slight errors:

1.  Variant Table: Record 'baku1926': The comment field is more than 255
characters and thus is truncated.  This could be overcome by setting the
field type to memo but on import XML is automatically setting the filed type
to Text. Just a minor glitch.

2.   In LSR-2.xml Most (not all) of the fields seem to have a preceding
space for some reason which will make it difficult to create search
routines.

I won't bother the list with further discussion on this.  If you would like
me to test any further updates let me know.

best regards

Debbie

> From: Stephane Bortzmeyer [mailto:[email-masked]]
> Sent: 13 May 2008 13:57
> To: Debbie Garside
> Cc: 'LTRU Working Group'
> Subject: Re: Record-Jar Format (was RE: I'm really confused
> by chinese in 3066bis)
>
> On Mon, May 12, 2008 at 11:19:51AM +0100,  Debbie Garside
> <[email-masked]> wrote  a message of 279 lines which said:
>
> > Frank and I did some work on this many moons ago and we
> were about 99%
> > of the way there - this needs to be completed and made publicly
> > available.
>
> Advices on the best process to debug, improve, extend, etc,
> this work are welcome. The LTRU co-chairs will probably
> request that this is discussed off-list.
>
>
>
>
>

---

From: John Cowan <[email-masked]>
Date: Tue, 13 May 2008 13:09:36 -0400

> For me, record-jar is a dogs breakfast.

It's formally defined and has certain technical advantages.

> desired).  This is extremely important for archiving purposes.

The official denotation of a tag (as opposed to any unofficial
connotations it may have acquired over time) is independent of which
RFC was in force at the time it was defined.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
Linguistics is arguably the most hotly contested property in the academic
realm. It is soaked with the blood of poets, theologians, philosophers,
philologists, psychologists, biologists and neurologists, along with
whatever blood can be got out of grammarians. - Russ Rymer
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: 'Stephane Bortzmeyer' <[email-masked]>
Date: Tue, 13 May 2008 21:42:52 +0200

On Tue, May 13, 2008 at 05:34:26PM +0100,
 Debbie Garside <[email-masked]> wrote 
 a message of 55 lines which said:

> 1.  Variant Table: Record 'baku1926': The comment field is more than
> 255 characters and thus is truncated.

Here, MS-Access is too restrictive. RFC 4646 is not very talkative
about registry field sizes (unlike the issue of language tag size,
which is discussed at length). But the RFC says in 3.1:

   The content of this field [Comments] is not
   restricted, except by the need to register the information, the
   suitability of the request, and by reasonable practical size
   limitations.

255 seems quite low. Anyway, none of the two proposed XML schemas set
a limit on the size of fields (it would not be possible in DTD,
anyway).

> space for some reason which will make it difficult to create search

IMHO, a bug in the awk script. The way I read RFC 4646 3.1:

   record     = 1*( field-name *SP ":" *SP field-body CRLF )

Leading spaces at the beginning of a field body (*SP) should not be
significant. Frank?
_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 13 May 2008 13:50:52 -0700

> Leading spaces at the beginning of a field body (*SP) should not be

Leading whitespace is not significant (that is, it's not part of the value).

Addison

Addison Phillips

> 'Stephane Bortzmeyer'
> Sent: Tuesday, May 13, 2008 12:43 PM
> To: Debbie Garside
> Cc: 'LTRU Working Group'
> Subject: Re: [Ltru] Record-Jar Format (was RE: I'm really confused by
> chinese in 3066bis)
>

---

From: Leif Halvard Silli <[email-masked]>
Date: Thu, 15 May 2008 01:45:49 +0200

I can see that you think so, if you are only focusing on the majority 
language.

Can you offer something clearer and better for non-majority languages 
within a macrolangauge group?

Leif Halvard Silli