Subject: [Ltru] Google and marking the languages

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 22 Nov 2006 11:47:03 +0100

Very good talk but it seems noone uses language tagging properly (not
even Google internally, see the slide on "Stable identifiers").

http://www.macchiato.com/slides/unicode_at_google.ppt

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: "Jukka K. Korpela" <[email-masked]>
Date: Wed, 22 Nov 2006 14:33:18 +0200 (EET)

> even Google internally, see the slide on "Stable identifiers").

And why would people bother using even simple language markup on web pages 
where it's almost trivial (<html lang="en">), when even the most obvious 
candidates for utilizing such information, namely search engines, ignore 
it? This looks very much like a vicious circle.

It's natural that Google uses heuristic language guessing instead of using 
language information in markup (or HTTP headers), since most pages lack 
such information - or have it plain wrong. What Google could sensibly do 
is to use such information, when available, to decide between similar 
alternatives in borderline cases, like distinguishing Danish from 
Norwegian. But this would be of rather limited usefulness.

For any substantial amount of text, heuristic language guessing works 
sufficiently well. For small fragments of text, like quotations or foreign 
words, guessing is not very successful (as we know from experience with MS 
Word if we let it guess the language), but who wants to tag his quotations 
and foreign words, and do it properly? So we have the paradox of language 
tagging: it's easy when it's not needed, but not otherwise.

-- 
Jukka "Yucca" Korpela, http://www.cs.tut.fi/~jkorpela/

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 22 Nov 2006 12:50:26 -0000

Stephane wrote:

> Very good talk but it seems noone uses language tagging 
> properly (not even Google internally, see the slide on 
> "Stable identifiers").
> 

This is something that I have tried to bring up (privately) on a number of
occasions.  We can sit here and standardize until the cows come home but
unless we give the end user motivation to adopt the standards that we
create, there will be no real standardization.  Standardization occurs when
the end user adopts the standard and we have to devise ways to encourage our
target market to do that and then market those ways effectively.

If search engines (such as Google) make part of their search engine rating
whether a document has been "tagged wisely", this will spur on the end user
to follow the standard.  As a business owner with several web sites, I know
that I would definitely tag my sites wisely if I thought it would increase
my ranking on a search engine. (Please don't check out my web sites :-))

It is simple.  If search engines adopt this strategy and make it known that
this is the case wrt ranking, the end user will also adopt.  The pay
off...Better information for search engines and standardization in the very
meaning of the word, with the end user happy that he has done that little
bit extra to ensure his business is up there with the rest.

How many books/articles are published each year telling end users how to
increase their rankings on search engines?  Sounds like standardization
marketing made easy!

Best regards

Debbie

Debbie Garside

---

From: "Dave Pawson" <[email-masked]>
Date: Wed, 22 Nov 2006 13:50:33 +0000

View from the outside.

> target market to do that and then market those ways effectively.

I'd make it even plainer. How many users (commercial or individual) who
are unaware of I18N or have never had reason to consider it?
Does this group (or the others with similar interests) ever make a marketing
effort?
  Where is the list of ten good reasons to id the language?
  Where should those who see the benefit point others to, the 'sales pitch'?

> bit extra to ensure his business is up there with the rest.

Great piece of marketing; that would hit home with business users.

> increase their rankings on search engines?  Sounds like standardization

Yes!

regards

-- 
Dave Pawson
XSLT XSL-FO FAQ.
http://www.dpawson.co.uk

---

From: Peter Constable <[email-masked]>
Date: Wed, 22 Nov 2006 13:46:56 -0800

> From: Debbie Garside [mailto:[email-masked]]

> create, there will be no real standardization.

In relation to our recent discussion regarding phonetic transcriptions, it would be a serious mistake to be misled by statistics about Web documents. 

IMO, the primary need for tags for phonetic transcription is not in Web pages but rather in documentation of some 7000 languages recorded by linguists all over the world, held in various language data archives, and contained primarily in xml files using various schemas. 

Linguists around the world are very much engaged in such documentation projects, and they are depending on standards for various purposes to make the data that is compiled interoperable and enduring. Language tagging is an important aspect of standardization for language documentation, and tagging of content in various phonetic transcriptions is an important aspect of their language tagging needs.

Peter Constable

---

From: "Debbie Garside" <[email-masked]>
Date: Wed, 22 Nov 2006 23:08:57 -0000

Peter wrote:

> In relation to our recent discussion regarding phonetic 
> transcriptions, it would be a serious mistake to be misled by 
> statistics about Web documents. 

My comments were aimed at the statistics for web documents of which there
are one or two  :-)

However...

> 
> IMO, the primary need for tags for phonetic transcription is 
> not in Web pages but rather in documentation of some 7000 
> languages recorded by linguists all over the world, held in 
> various language data archives, and contained primarily in 
> xml files using various schemas. 

Yes, the archive industry is huge, without a doubt.  And, I can see their
needs quite clearly.  I assume they are using RFC4646 and thus would be
amenable to a variant subtag as proposed by Michael?
 
> Linguists around the world are very much engaged in such 
> documentation projects, and they are depending on standards 
> for various purposes to make the data that is compiled 
> interoperable and enduring. Language tagging is an important 
> aspect of standardization for language documentation, and 
> tagging of content in various phonetic transcriptions is an 
> important aspect of their language tagging needs.

Indeed.  But, given that IPA is considered by some to be out of scope for
ISO 15924, can we discuss any reasons why Michael's proposals will not meet
this need?

If any valid reasons are accepted by the majority of IETF-languages (or LTRU
for that matter) then we can go to proposal number two (Ken) and petition
the JAC of ISO 15924 to change the scope or just allow the tag.  If that
fails, you can always register a variant of any written language in IPA with
ISO 639-6 (as a last resort, of course).

Of course, this does not, in any way, prevent any one person from
petitioning the ISO 15924 JAC of their own accord.

Best regards

Debbie