Subject: RE: [Ltru] Re: For review: Tagging text with no language

From: "CE Whitehead" <[email-masked]>
Date: Sat, 14 Apr 2007 13:12:06 -0400

>
>The meaning of zxx must be interpreted in terms of the coding standard of 
>which it is a part. ISO 639 is explicitly about coding human languages. "No 
>linguistic content" in the case of zxx means 'no content in any human 
>language'. If a language tag must be applied to something like "ifdef 
>DEBUG", then the appropriate language subtag would be zxx. ISO 639 cares 
>not if an IETF tag zxx-Latn or zxx-progrmng is constructed; but consistency 
>with ISO 639 would imply that the language subtag zxx be used.
>
>
>Peter

Hi, Peter, I would only really disagree with what you say if there were 
programming languages where the words used to control the code were in other 
languages in use,
so that one language say made use of "while"
and another  of "lorsque"
but otherwise,

zxx I gather keeps the spelling checker from checking the language of the 
code,
but as authors embed alerts, which are in one language, in code where the 
keywords or whatever are in quite another language,
one might want a way to turn off the zxx code or to include a second code 
too and there is none that I know of.

(Example: you might have the programming language in English and the alert 
in German.
I guess it would be tough to pick a language identifier in that case.
I do not know what the spelling checkers do with mis or mul either
. . . though I know that zxx is the only subtag recommended so what the 
spelling checkers do is irrelevant if everyone uses zxx

You have to check the content language of alerts separately I guess)

--C. E. Whitehead
[email-masked]

>
>From: [email-masked]
>To: Jukka K. Korpela

>
>
>I agree with you. I would also say that indicating that computer languages 
>have "no linguistic content" is not quite accurate for the same reasons you 
>cite below and disagrees with the origin of the zxx tag, which was to 
>indicate a total absence of language, not content which is not in a natural 
>language.
>
>Karen Broome
>
>[email-masked] wrote on 04/13/2007 01:08:33 PM:
>
> >
> > On Fri, 13 Apr 2007, [email-masked] wrote:
> >
> > > With respect to computer language snippets, isn't that what the <code> 
>tag
> > > is for -- at least in XHTML?
> >
> > <code> has been in HTML since the beginning, and it indeed indicates
> > computer code, or "computer language".
> >
> > But to make things more difficult, computer code - such as program code,
> > operating system commands, or markup - can be regarded as being in some
> > human language(s) in the sense that the identifiers may have been formed
> > from words in human languages and the comments are written in some human
> > languages. Knowing the language of computer code can be relevant at 
>least
> > for two purposes:
> > a) the understandability of the code to human readers depends on whether
> >     they understand the human language(s) used
> > b) speech synthesis would benefit from the use of language information.
> > Moreover, a checking program could e.g. check that the code follows the
> > syntactic rules of the programming, command, or markup "language" _and_
> > check the comments using a suitable human language spelling checker.
> >
> > Thus, I would say that the human language some text and the role of that
> > text as computer code are in a sense orthogonal.
> >
> > --
> > Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/
> >
> >
> >

_________________________________________________________________
Mortgage rates near historic lows. Refinance $200,000 loan for as low as 
$771/month* 
https://www2.nextag.com/goto.jsp?product=100000035&url=%2fst.jsp&tm=y&search=mortgage_text_links_88_h27f8&disc=y&vers=689&s=4056&p=5117

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: Peter Constable <[email-masked]>
Date: Sat, 14 Apr 2007 11:39:48 -0700

From: CE Whitehead [mailto:[email-masked]]

>> If a language tag must be applied to something like "ifdef
>> DEBUG", then the appropriate language subtag would be zxx...

> I would only really disagree with what you say if there were
> programming languages where the words used to control the code
> were in other languages in use,

Excluding operators and such, which clearly are not words of English, the Microsoft C compiler recognizes this vocabulary:

#define #error #import #undef #elif #if #include
#else #ifdef #line #endif #ifndef #pragma
auto double int struct break else long switch
case enum register typedef char extern return union
const float short unsigned continue for signed void
default goto sizeof volatile do if static while
__asm dllimport2 __int8 naked2 __based1 __except
__int16 __stdcall __cdecl __fastcall __int32
thread2 __declspec __finally __int64 __try
dllexport2 __inline __leave

Some of those may be borrowings from English, but this is not English.

Not that I expect everyone to differentiate every different element in their document. If I discussing differences between the i607 and SMT5600 phones in an email, I'm probably not going to bother tagging the part numbers as zxx. But if I were editing a long document with lots of content of that nature, I might.

> zxx I gather keeps the spelling checker from checking
> the language of the code,

That is one possible reason for using such a tag among several.

Peter

---

From: [email-masked]
Date: Mon, 16 Apr 2007 11:28:20 -0700

Peter wrote:

> Excluding operators and such, which clearly are not words of 
> English, the Microsoft C compiler recognizes this vocabulary:
> 

Let's look at a real use case. Would you say this page is in "en" and 
"zxx" and that the sections of code have no linguistic value even though 
they are clearly intended to be read by humans and not machines? Or does 
context matter?

http://www.xml.com/pub/a/2000/11/29/schemas/part1.html?page=7

Interpreting "no linguistic content" as "not a human language, could be a 
programming language" could cause some problems. There may be a use case 
for programming languages to have their own tag if this is deemed 
appropriate for the 639 standards or IANA registry, and these languages 
are different than say, instrumental music in the Library of Congress or a 
sound effects track in a film (both zxx, I'd say). 

I think programming languages have specific identification and parsing 
needs and as such need to be treated differently. The code in the article 
above should be rendered in Braille, for example, so it must be parsed. 
This makes it different from non-linguistic content.

Are we sure we want to lump programming languages in with the "zxx" 
semantic? More and more people use programming language terms in their 
everyday speech.

How would you classify the page I cite?

Regards,

Karen Broome

---

From: Peter Constable <[email-masked]>
Date: Mon, 16 Apr 2007 17:38:45 -0700

From: [email-masked] [mailto:[email-masked]]

> Let's look at a real use case. Would you say this page
> is in "en" and "zxx" and that the sections of code
> have no linguistic value even though they are clearly
> intended to be read by humans and not machines? Or does
> context matter?

You're asking to have one tag that covers an entire document even though that document is mixed. What if I insert a quotation in Spanish in this mail? ¿Que vamos a hacer? It's no different. If you must use a single tag for the whole thing, then clearly this is predominantly in English. I don't know what you'd do if it were closer to 50-50.

> Interpreting "no linguistic content" as "not a human
> language, could be a programming language" could cause
> some problems. There may be a use case for programming
> languages to have their own tag if this is deemed
> appropriate for the 639 standards or IANA registry, and
> these languages are different than say, instrumental
> music in the Library of Congress or a sound effects
> track in a film (both zxx, I'd say).

Your argument is akin to someone saying that someone may want to code audio in Unicode. ISO 639 has defined a scope, human languages. Programming languages, electrical schematics, dance notation, bridge-hand notation, math formulas and engineering drawings are all graphic content that can be interpreted by humans. Some of these can be represented in text, but that does not change the fact that they are not a form of the kind of things coded by ISO 639, human languages.

> I think programming languages have specific
> identification and parsing needs and as such need to
> be treated differently.

As I suggested earlier, the scope defined by ISO 639 does not force RFC 4646bis to be limited to the same scope -- in fact, it cannot be. ("Language tags" already code things other than linguistic variety, written form in particular.) So, if you want to propose variant subtags to differentiate programming code from music notation, then I don't see why that couldn't be done.

But it would be out of scope for ISO 639 to code such a distinction, and it would be a non-conforming re-interpretation to say that zxx does not apply programming languages.

> The code in the article above should be rendered in
> Braille, for example, so it must be parsed. This makes
> it different from non-linguistic content.

You're confusing the language of content with the representation mode in some communicative technology. English content in Braille is still English, and so clearly different from zxx. That is not in any way comparable to discussing code in a programming language.

> How would you classify the page I cite?

As I mentioned above, this question is no different than asking how to come up with one tag for a page that contains content in both English and Spanish. On a *practical* level, I would tag that article as en and ignore the fact that it contains XML code snippets; but if someone was being careful to tag elements within the document correctly, then the code snippets should be tagged zxx. (That is, unless you want to register variant subtags to differentiate between different kinds of non-linguistic content.)

Peter

---

From: "CE Whitehead" <[email-masked]>
Date: Tue, 17 Apr 2007 14:59:39 -0400

> > intended to be read by humans and not machines? Or does

Such as comments in programming code.

I do personally happen to think that maybe there is a use for a variant of 
zxx that indicates it is a programming language (not a math formula or 
whatever)
and then there is a use to allow some additional identification of the 
language(s) involved
(clearly the basic code could be in one language, English, but user-defined 
functions might make use of French or German abbreviations and then the 
comments would actually be in French or German).

In the meantime,
one thing that might help for tagging code snippets in a page
is have an outer paragraph or whatever
lang="en"
(or "fr" or "de" or what have you, the language that is closest to the 
language in which the code is based)
with maybe a short header or something in the appropriate language

and just inside this a section pre which is where the code is
you could define the language of
pre
as zxx

Just my two cents' worth.

--C. E. Whitehead
[email-masked]

>
>You're asking to have one tag that covers an entire document even though 
>that document is mixed. What if I insert a quotation in Spanish in this 
>mail? ¿Que vamos a hacer? It's no different. If you must use a single tag 
>for the whole thing, then clearly this is predominantly in English. I don't 
>know what you'd do if it were closer to 50-50.
>
>
>Your argument is akin to someone saying that someone may want to code audio 
>in Unicode. ISO 639 has defined a scope, human languages. Programming 
>languages, electrical schematics, dance notation, bridge-hand notation, 
>math formulas and engineering drawings are all graphic content that can be 
>interpreted by humans. Some of these can be represented in text, but that 
>does not change the fact that they are not a form of the kind of things 
>coded by ISO 639, human languages.
>
>As I suggested earlier, the scope defined by ISO 639 does not force RFC 
>4646bis to be limited to the same scope -- in fact, it cannot be. 
>("Language tags" already code things other than linguistic variety, written 
>form in particular.) So, if you want to propose variant subtags to 
>differentiate programming code from music notation, then I don't see why 
>that couldn't be done.
>
>But it would be out of scope for ISO 639 to code such a distinction, and it 
>would be a non-conforming re-interpretation to say that zxx does not apply 
>programming languages.
>
>
>You're confusing the language of content with the representation mode in 
>some communicative technology. English content in Braille is still English, 
>and so clearly different from zxx. That is not in any way comparable to 
>discussing code in a programming language.
>
>
>As I mentioned above, this question is no different than asking how to come 
>up with one tag for a page that contains content in both English and 
>Spanish. On a *practical* level, I would tag that article as en and ignore 
>the fact that it contains XML code snippets; but if someone was being 
>careful to tag elements within the document correctly, then the code 
>snippets should be tagged zxx. (That is, unless you want to register 
>variant subtags to differentiate between different kinds of non-linguistic 
>content.)
>
>
>Peter

One might want to do so.