Subject: Re: [Ltru] requiring full prefixes

From: "Doug Ewell" <[email-masked]>
Date: Wed, 3 Dec 2008 22:24:15 -0700

I am simply amazed that at this stage of the game, we are still 
considering technical changes to what is and is not allowed in the 
Registry.

--
Doug Ewell  *  Thornton, Colorado, USA  *  RFC 4645  *  UTN #14
http://www.ewellic.org
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages  ˆ

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 3 Dec 2008 22:39:45 -0800

The problem is that we changed something two days ago that had been stable
for 1 1/2 years. So "no technical change" would be to revert to the #18
text.
So we have a bit of a problem. I don't think anyone is wedded to extension
syntax, but I think we do have to have at least some discussion about why we
should forbid a capability that we've had as part of the draft for 1 1/2
years.

But I'm happy for Martin to rule on this one.

Mark

On Wed, Dec 3, 2008 at 21:24, Doug Ewell <[email-masked]> wrote:

> I am simply amazed that at this stage of the game, we are still considering
> technical changes to what is and is not allowed in the Registry.
>
> --

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 3 Dec 2008 23:05:09 -0800

Hi -

> From: "Mark Davis" <[email-masked]>
> To: "Doug Ewell" <[email-masked]>
> Cc: "LTRU Working Group" <[email-masked]>
> Sent: Wednesday, December 03, 2008 10:39 PM
> Subject: Re: [Ltru] requiring full prefixes
>

As a technical contributor...

I agree that there doesn't seem to be anyone arguing specifically for
the extension syntax.  I think the question is whether we actually
*intended* there to be the "non-initial prefix" capability that
resulted as a side-effect (as I see it) of the use of that syntax
to accomplish subtag ordering. That this capability seems to have
only fairly recently been "discovered" calls into question whether
there was actually a WG consensus for it.  I don't think anything
terrible results whether we do it one way or the other.

I considered at some length whether it would be helpful with something
like -fonipa for indicating "schools" of transcription and so on.
My experience is that transcriptions always end up being "tuned" to
a particular language, so I'm skeptical of whether non-initial prefixes
would really be useful.  But I haven't thought of anything terrible
that would happen if we allowed them.

> But I'm happy for Martin to rule on this one.

As am I.

Randy

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 4 Dec 2008 07:52:01 -0800

Since I proposed the use of "extended language range" in the first place:

- It was NOT intended to allow non-initial prefix registrations. My intention was to simplify the description of both the field's value and the process of matching it to the recommended tag. Basically, I had an insight that what we were describing was a form of extended filtering.

- I don't think it was the working group's intention to allow non-initial ranges. This can been seen in the fact that we added text banning the wildcard when someone (Randy, I think) pointed out that the text didn't actually ban wildcards.

It is interesting to find that the text, aside from the ban on wildcards, doesn't ban the use of ranges to modify other subtags. If we only remove the ban on '*', we get variants that modify script, region, or generic variants. However, we *did* ban the wildcard in draft #18, so if we revert as Mark suggests, we don't get what he's asking for. The proposed text I sent along yesterday is clearly better, so I would support its inclusion in place of draft #18. If a consensus exists to allow non-initial prefix capabilities, we should also revisit text about generic variants and ensure that the whole is still sensible.

Addison

Addison Phillips
Globalization Architect -- Lab126

Internationalization is not a feature.
It is an architecture.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Randy Presuhn
> Sent: Wednesday, December 03, 2008 11:05 PM
> To: LTRU Working Group

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 4 Dec 2008 11:00:28 -0800

I have posted an "editor's copy" draft-20 on Inter-Locale. Diff links are available.

I corrected a problem with the generation of draft-19, vis: I didn't generate the TOC because the tool changed the format of the <?rfc ?> directive.

The changes in this document include:

1. Generation of the TOC
2. Addition of John Cowan's suggested text regarding private-use subtags vs. private use subtag sequences (with my modifications).
3. Correction of the typo on 'extension'.
4. A rewrite of the description of the 'Prefix' field on page 25 as previously discussed on this list.
5. A rewrite of Section 3.1.8 (Prefix Field). The rewrite itself has been discussed on this list, but this specific text has not been previously posted.
6. References now appear sorted (same cause as TOC generation)

Links follow:

  http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-20.html
  http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-20.txt
  http://www.inter-locale.com/ID/draft-ietf-ltru-4646bis-20.xml
  http://tinyurl.com/5t2nhb (side-by-side diff)
  http://tinyurl.com/5zmsmo (hwdiff, that is, a full diff)

Is there anything missing from this version? Any comments on the specific texts that were changed?

Addison

Addison Phillips

> Behalf Of Phillips, Addison
> Sent: Thursday, December 04, 2008 7:52 AM
> To: Randy Presuhn; LTRU Working Group
> - It was NOT intended to allow non-initial prefix registrations. My
> intention was to simplify the description of both the field's value
> and the process of matching it to the recommended tag. Basically, I
> had an insight that what we were describing was a form of extended
> filtering.
> 
> - I don't think it was the working group's intention to allow non-
> initial ranges. This can been seen in the fact that we added text
> banning the wildcard when someone (Randy, I think) pointed out that
> the text didn't actually ban wildcards.
> 
> It is interesting to find that the text, aside from the ban on
> wildcards, doesn't ban the use of ranges to modify other subtags.
> If we only remove the ban on '*', we get variants that modify
> script, region, or generic variants. However, we *did* ban the
> wildcard in draft #18, so if we revert as Mark suggests, we don't
> get what he's asking for. The proposed text I sent along yesterday
> is clearly better, so I would support its inclusion in place of
> draft #18. If a consensus exists to allow non-initial prefix
> capabilities, we should also revisit text about generic variants
> and ensure that the whole is still sensible.
> 
> Addison
> 
> Addison Phillips
> > > So we have a bit of a problem. I don't think anyone is wedded
> to
> > extension

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 4 Dec 2008 11:36:06 -0800

I don't find anything in #18 about removing wildcards.

http://tools.ietf.org/html/draft-ietf-ltru-4646bis-18

Unless I am missing something, the first time a change occurred was in
#19, on Dec 2.

> The Prefix MUST NOT include the wildcard '*'.

That was not in #18. So wildcards were allowed, and one could indicate
just a script up until Dec 2.

And note that even in #19, Section 2.2.5 says "The 'Prefix' indicates
a sequence of subtags", not a language tag.

So I think "we added text" is an inaccurate representation of a change
that first went in Dec 2, with text that didn't represent the last
message on the thread of Nov 2-4. And whether or not your intent was
to allow "partial" language tag prefixes, that's what the text had for
a long time.

   Note: I think part of the confusion is that we mention Prefix in
several places in the document, and we were probably talking about
different places, because we sort-of define Prefix in 3 (!) places.

I will suggest a change that allows us to leave Prefix as a language
tag (a complete one). It also cleans up the text so that there is a
single place where we define the Prefix field, and other instances are
clearly just glosses. I retained the text that you had about using the
extended filtering, with one small change.

That change maintains the ability to indicate that the primary subtag
is not material to the matching process for a Prefix, by using "und"
to stand for any language subtag in that case. That way a prefix of
"und-Latn" for tag X means that "en-Latn-X", "fr-Latn-X", etc are
valid. So it maintains the ability that was in up until Dec 2, but in
a much cleaner fashion, in that we can still now speak of a Prefix as
a valid language tag.

So here's what I'm proposing.

============================

http://tools.ietf.org/html/draft-ietf-ltru-4646bis-19#section-2.2.5

The 'Prefix' indicates a sequence of
subtags that would make a suitable prefix (with other subtags, as
appropriate) in forming a language tag with the variant. That is,
each of the subtags in the prefix SHOULD appear, in order, before the
variant.

=>

The 'Prefix' contains a valid language tag whose
subtags would make a suitable prefix (with other subtags, as
appropriate) in forming a language tag with the variant, according to
[Section 3.1.8].

=========

http://tools.ietf.org/html/draft-ietf-ltru-4646bis-19#section-3.1.2

Prefix's field-body contains an "extended language range"
(basically, a collection of subtags, see [RFC4647]) which
SHOULD be used with this subtag when forming the language tag.
The subtags in the prefix MUST appear before the subtag in
Prefix's record.  The Prefix MUST NOT include the wildcard '*'.
This field MUST appear only in records whose 'Type' field-body
is either 'extlang' or 'variant'.

=>

The 'Prefix' contains a valid language tag whose
subtags would make a suitable prefix (with other subtags, as
appropriate) in forming a language tag, according to [Section 3.1.8].
This field MUST appear only in records whose 'Type' field-body
is either 'extlang' or 'variant'.

=========

http://tools.ietf.org/html/draft-ietf-ltru-4646bis-19#section-3.1.8

The 'Prefix' field contains an "extended language range" (see:
[RFC4647]) whose subtags are appropriate to use with this subtag:
each of the subtags in one of the subtag's Prefix fields SHOULD
appear before the variant in a valid tag.

=>

Prefix's field-body contains a valid language tag. When a subtag's
record includes one or more Prefix fields, then any language tag
containing that subtag SHOULD match at least one of the Prefixes. That
matching is according to the "Extended Filtering" algorithm (see <xref
target="RFC4647"></xref>), whereby in the matching:
  * The Prefix behaves as if it had the subtag itself appended at the end.
  * "und" in the Prefix matches any primary language subtag.

This field MUST appear only in records whose 'Type' field-body is
either 'extlang' or 'variant'.

============================

[Discussion]

This puts the guts of the definition in exactly one place, with
glosses elsewhere.

You'd proposed:

<addison>
Prefix's field-body contains a valid language tag whose subtags SHOULD
be used when forming a language tag with this record's subtag. That
is, a language tag that includes a subtag whose record includes one or
more Prefix fields SHOULD match at least one of the Prefixes using the
"Extended Filtering" algorithm (see <xref target="RFC4647"></xref>).
Further, each of the subtags in the Prefix SHOULD appear before the
subtag itself. This field MUST appear only in records whose 'Type'
field-body is either 'extlang' or 'variant'
</addison>

I retained the Extended Filter language (I liked that). But there are
a number of referent problems still in this text. The first sentence
still requires that *each* prefix be a prefix of the language tag. The
second sentence is pretty convoluted. The third sentence also is a
requirement on *each* prefix. And this text was really in the wrong
place, since the formal, complete definition should be in 3.1.8.

So I tried the above formulation to clear up these problems. Note that
the first * is the one that has the Prefix fields (for the prefix that
matches!) come first.

Mark

Mark

---

From: "Phillips, Addison" <[email-masked]>
Date: Thu, 4 Dec 2008 13:06:52 -0800

> Unless I am missing something, the first time a change occurred was
> in #19, on Dec 2.

Hmm... you're right. We just didn't publish draft-19 for awhile and the memory does falter. We discussed this on the list some time ago. IIRC, Randy pointed out that draft-18 (must have been) didn't prohibit the use of wildcards, even though that was (as noted) the intention.

> 
> That was not in #18. So wildcards were allowed, and one could
> indicate just a script up until Dec 2.

It was added in draft-07, so wildcards were not allowed previously (the 4646-era text says "language tag"). The text doesn't indicate whether wildcards are banned or not elsewhere, so I guess they were permitted up until draft-19. The question is whether that was intentional or an oversight. And, if an oversight, whether there is anything to be gained from restoring the extended language range (including wildcards). I think there is some merit for doing so, but am a little leery of it (as a form of quondam generic variant).

> different places, because we sort-of define Prefix in 3 (!) places.

Yes. And this is fairly redundant stuff too. See proto-draft-20 for a stab at cutting back on the crossover.

> valid. So it maintains the ability that was in up until Dec 2, but

I really don't like the 'und' thing. It would be much better to just do extended language range and extended filtering and permit the wildcard in that case. We have instructions for implementation and everything. Then it would just be up to ietf-languages@ and the LSR as to whether this feature gets used and, if so, how. Using the wildcard calls attention to the fact that the Prefix matches any primary language subtag.

Perhaps a better question would be: if we go down that route, should we then REQUIRE a prefix for all variants? '*', after all, is a valid extended language range. The generic variant ickiness we have now looks problematic by comparison.

Good to include the link, but even better to remove repetitive examples, etc. I propose:

<t hangText="Note: ">Variant subtag records in the language subtag registry MAY include one or more 'Prefix' (<xref target="prefixfield"></xref>) fields. Each 'Prefix' indicates a suitable sequence of subtags (with other subtags, as appropriate) for forming a language tag when using the variant.</t>

> is either 'extlang' or 'variant'.

This is good. I modified my draft-20 to match thusly:

<t>Prefix<list>

<t>Prefix's field-body contains a valid language tag whose subtags are RECOMMENDED for use together as a prefix for this record's subtag. This field MUST appear only in records whose 'Type' field-body is either 'extlang' or 'variant'.</t>
</list>
</t>

The forward reference isn't necessary, I think, since we don't use it for the other field types.

> http://tools.ietf.org/html/draft-ietf-ltru-4646bis-19#section-3.1.8

Here's my proposed draft-20 text, revised to include your proposal:

<section anchor="prefixfield" title="Prefix Field">
					
<t>The field 'Prefix' contains a valid language tag whose subtags are RECOMMENDED for use together as a prefix for this record's subtag, perhaps with other subtags. That is, when including a subtag in a language tag, the tag SHOULD match at least one of its Prefix fields using the "Extended Filtering" algorithm (see <xref target="RFC4647"></xref>) and each of the subtags in the Prefix SHOULD appear before the subtag itself. </t>

<t>The 'Prefix' field MUST appear exactly once in a record of type 'extlang' and MAY appear multiple times (or not at all) in records of type 'variant'. Additional fields of this type MAY be added to a 'variant' record via the registration process, provided the 'variant' record already has at least one 'Prefix' field.
</t>

<t>Each 'Prefix' field indicates a particular sequence of subtags that form a meaningful tag with this subtag. For example, the extended language subtag 'cmn' (Mandarin Chinese) only makes sense with its prefix 'zh' (Chinese). Similarly, 'rozaj' (Resian, a dialect of Slovenian) would be appropriate when used with its prefix 'sl' (Slovenian), while tags such as "is-1994" are not appropriate (and probably not meaningful). Although the prefix for 'rozaj' is "sl", other subtags might appear between them. For example, the tag "sl-IT-rozaj" (Slovenian, Italy, Resian) matches the Prefix "sl".</t>

<t>The 'Prefix' also indicates when variant subtags make sense when used together (many that otherwise share a 'Prefix' are mutually exclusive) and what the relative ordering of variants is supposed to be. For example, the variant '1994' (Standardized Resian orthography) has several 'Prefix' fields in the registry ("sl-rozaj", "sl-rozaj-biske", "sl-rozaj-njiva", "sl-rozaj-osojs", and "sl-rozaj-solba"). This not only indicates that '1994' is appropriate to use with each of these five Resian variant subtags ('rozaj', 'biske', 'njiva', 'osojs', and 'solba') but also that it SHOULD appear following any of these variants in a tag. Thus, the language tag ought to take the form "sl-rozaj-biske-1994" rather than "sl-1994-rozaj-biske" or "sl-rozaj-1994-biske".</t>
					
<t>Once registered, fields of type 'Prefix' MUST NOT be removed from any record. The field-body for this type of field MAY be modified, but only if the modification broadens the meaning of the subtag. That is, the field-body can be replaced only by a prefix of itself. For example, the Prefix "be-Latn" (Belarusian, Latin script) could be replaced by the Prefix "be" (Belarusian) but not by the Prefix "ru-Latn" (Russian, Latin script).</t>
					
<t>The field-body of the 'Prefix' field MUST NOT conflict with any 'Prefix' already registered for a given record. Such a conflict would occur when no valid tag could be constructed that would contain the prefix, such as when two subtags each have a 'Prefix' that contains the other subtag. For example, suppose that the subtag 'avariant' has the prefix "es-bvariant". Then the subtag 'bvariant' cannot given the prefix 'avariant', for that would require a tag of the form "es-avariant-bvariant-avariant", which would not be valid.</t>
				
</section>

---

From: Kent Karlsson <[email-masked]>
Date: Thu, 04 Dec 2008 22:32:56 +0100

> So here's what I'm proposing.

For extlang this is a bit too general. The body of a Prefix field
for an extlang record must be exactly a primary language tag (and
there must be exactly one Prefix field for an extlang record) and
there can be no subtags between the prefix and the extlang.
The generality above is for "variant" records only.

>   * "und" in the Prefix matches any primary language subtag.

I'm not sure why "und" would be used in place for what elsewhere (in LTRU
documents) is denoted by "*". (I'm not having an opinion (yet) as to whether
this functionality should be included or not, just an opinion about using
"und" as a standin for "*" syntactically, but I do think this new syntax
(whether via "und" or "*") is a bit late in the making).
 
> either 'extlang' or 'variant'.

    /kent k

---

From: Kent Karlsson <[email-masked]>
Date: Thu, 04 Dec 2008 23:20:35 +0100

Den 2008-12-04 22.06, skrev "Phillips, Addison" <[email-masked]>:

> <t>The 'Prefix' field MUST appear exactly once in a record of type 'extlang'

...and is then a simpler case (as I mentioned in my previous email)...

...
> <t>Once registered, fields of type 'Prefix' MUST NOT be removed from any
> record. The field-body for this type of field MAY be modified, but only if the
> modification broadens the meaning of the subtag. That is, the field-body can
> be replaced only by a prefix of itself. For example, the Prefix "be-Latn"
> (Belarusian, Latin script) could be replaced by the Prefix "be" (Belarusian)
> but not by the Prefix "ru-Latn" (Russian, Latin script).</t>

>From "variant" records one could in principle remove all Prefix fields
(that's a widening), also if modifying Prefix fields so that they coincide
or overlap may be cause for removing Prefix fields. E.g.
Prefix: nnn-Latn
Prefix: nnn-Cyrl
modifying the first one to "Prefix: nnn" would be cause for removing the
"Prefix: nnn-Cyrl" field.

    /kent k

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 9 Dec 2008 10:43:24 -0800

Just to confirm, I'm not insisting on allowing "partial" language tags (or
using "und"), given the timing. My concern was that we might need this in
the future, but any subtag that would need it would just have to added with
prefixes, perhaps with a comment about intended usage.

I think the suggested language is not quite right yet, but will send a
separate note.

Mark

On Thu, Dec 4, 2008 at 13:06, Phillips, Addison <[email-masked]> wrote:

> Hmm... you're right. We just didn't publish draft-19 for awhile and the
> memory does falter. We discussed this on the list some time ago. IIRC, Randy
> pointed out that draft-18 (must have been) didn't prohibit the use of
> wildcards, even though that was (as noted) the intention.
>
> >
> It was added in draft-07, so wildcards were not allowed previously (the
> 4646-era text says "language tag"). The text doesn't indicate whether
> wildcards are banned or not elsewhere, so I guess they were permitted up
> until draft-19. The question is whether that was intentional or an
> oversight. And, if an oversight, whether there is anything to be gained from
> restoring the extended language range (including wildcards). I think there
> is some merit for doing so, but am a little leery of it (as a form of
> quondam generic variant).
>
> >
> I really don't like the 'und' thing. It would be much better to just do
> extended language range and extended filtering and permit the wildcard in
> that case. We have instructions for implementation and everything. Then it
> would just be up to ietf-languages@ and the LSR as to whether this feature
> gets used and, if so, how. Using the wildcard calls attention to the fact
> that the Prefix matches any primary language subtag.
>
> Perhaps a better question would be: if we go down that route, should we
> then REQUIRE a prefix for all variants? '*', after all, is a valid extended
> language range. The generic variant ickiness we have now looks problematic
> by comparison.
>
>
> >
> <t hangText="Note: ">Variant subtag records in the language subtag registry
> MAY include one or more 'Prefix' (<xref target="prefixfield"></xref>)
> fields. Each 'Prefix' indicates a suitable sequence of subtags (with other
> subtags, as appropriate) for forming a language tag when using the
> variant.</t>
>
> >
> > =========
> >
> >
> <t>Prefix's field-body contains a valid language tag whose subtags are
> RECOMMENDED for use together as a prefix for this record's subtag. This
> field MUST appear only in records whose 'Type' field-body is either
> 'extlang' or 'variant'.</t>
> </list>
> </t>
>
> <t>The field 'Prefix' contains a valid language tag whose subtags are
> RECOMMENDED for use together as a prefix for this record's subtag, perhaps
> with other subtags. That is, when including a subtag in a language tag, the
> tag SHOULD match at least one of its Prefix fields using the "Extended
> Filtering" algorithm (see <xref target="RFC4647"></xref>) and each of the
> subtags in the Prefix SHOULD appear before the subtag itself. </t>
>
> <t>The 'Prefix' field MUST appear exactly once in a record of type
> 'extlang' and MAY appear multiple times (or not at all) in records of type
> 'variant'. Additional fields of this type MAY be added to a 'variant' record
> via the registration process, provided the 'variant' record already has at
> least one 'Prefix' field.
> </t>
>
> <t>Each 'Prefix' field indicates a particular sequence of subtags that form
> a meaningful tag with this subtag. For example, the extended language subtag
> 'cmn' (Mandarin Chinese) only makes sense with its prefix 'zh' (Chinese).
> Similarly, 'rozaj' (Resian, a dialect of Slovenian) would be appropriate
> when used with its prefix 'sl' (Slovenian), while tags such as "is-1994" are
> not appropriate (and probably not meaningful). Although the prefix for
> 'rozaj' is "sl", other subtags might appear between them. For example, the
> tag "sl-IT-rozaj" (Slovenian, Italy, Resian) matches the Prefix "sl".</t>
>
> <t>The 'Prefix' also indicates when variant subtags make sense when used
> together (many that otherwise share a 'Prefix' are mutually exclusive) and
> what the relative ordering of variants is supposed to be. For example, the
> variant '1994' (Standardized Resian orthography) has several 'Prefix' fields
> in the registry ("sl-rozaj", "sl-rozaj-biske", "sl-rozaj-njiva",
> "sl-rozaj-osojs", and "sl-rozaj-solba"). This not only indicates that '1994'
> is appropriate to use with each of these five Resian variant subtags
> ('rozaj', 'biske', 'njiva', 'osojs', and 'solba') but also that it SHOULD
> appear following any of these variants in a tag. Thus, the language tag
> ought to take the form "sl-rozaj-biske-1994" rather than
> "sl-1994-rozaj-biske" or "sl-rozaj-1994-biske".</t>
>
> record. The field-body for this type of field MAY be modified, but only if
> the modification broadens the meaning of the subtag. That is, the field-body
> can be replaced only by a prefix of itself. For example, the Prefix
> "be-Latn" (Belarusian, Latin script) could be replaced by the Prefix "be"
> (Belarusian) but not by the Prefix "ru-Latn" (Russian, Latin script).</t>
>
> <t>The field-body of the 'Prefix' field MUST NOT conflict with any 'Prefix'
> already registered for a given record. Such a conflict would occur when no
> valid tag could be constructed that would contain the prefix, such as when
> two subtags each have a 'Prefix' that contains the other subtag. For
> example, suppose that the subtag 'avariant' has the prefix "es-bvariant".
> Then the subtag 'bvariant' cannot given the prefix 'avariant', for that
> would require a tag of the form "es-avariant-bvariant-avariant", which would
> not be valid.</t>
>
> </section>
>