Subject: Posted PaceSimpleLanguageTagging

From: David Powell <[email-masked]>
Date: Sat, 15 Jan 2005 17:30:47 +0000

PaceExtensionConstruct hopes to provide a basis for a mapping Atom to
models such as RDF [1], ER, and OO. I'm currently doing some work to see
whether there is anything in Atom that makes this unnecessarily
difficult.

I think that Atom's use of xml:lang is likely to be a significant
problem to many implementors. I've posted PaceSimpleLanguageTagging
explaining the problem.

http://www.intertwingly.net/wiki/pie/PaceSimpleLanguageTagging

[1] RDF supports language tagging of literals, so xml:lang doesn't
    actually cause a serious problem for an Atom implementation based
    on an RDF model.

-- 
Dave

---

From: Martin Duerst <[email-masked]>
Date: Mon, 17 Jan 2005 14:06:47 +0900

I think this proposal throws out the baby with the bathwater.
I don't see any reason to introduce an atom:language element;
xml:lang can serve exactly the same purpose. If you want to
reduce the effect on performance/DBs, there are the following
things that can be done:
- Allow xml:lang only on some elements, but not on others.
- Describe that inherited xml:lang does not apply to certain
   elements or attributes (e.g. things like dates, which are
   in a language-independent notation).
Disallowing xml:lang on text constructs would be quite a bad
idea in my view.

As for RDF's support of xml:lang, there is a very RDF-specific,
unfortunate issue that has to be taken into account when
converting from atom to RDF/XML: RDF/XML at certain points
cuts the inheritance of xml:lang.

Regards,    Martin.

---

From: Tim Bray <[email-masked]>
Date: Sun, 16 Jan 2005 22:39:49 -0800

> problem to many implementors. I've posted PaceSimpleLanguageTagging

I'm fairly unconvinced that this is will loom very large in the 
universe of implementor problems.  If it turns out that it's really an 
issue, couldn't you achieve the desired effect by restricting xml:lang 
to appear only on atom:entry and atom:feed?

I have to admit that I'm having trouble imagining a scenario where 
you'd want to scatter xml:lang's randomly around an atom instance, so 
maybe this is a handy simplification.

Obviously, all bets are off once you get inside atom:content, right?  
-Tim

---

From: David Powell <[email-masked]>
Date: Mon, 17 Jan 2005 08:47:31 +0000

> universe of implementor problems.

For an atom implementation where content is entered via HTML forms, or
from an existing CMS - then this isn't a problem.  The user won't be
able to scatter xml:lang's everywhere.  But for an implementation
based on Atom Protocol, the server needs to preserve all of this
stuff.

> If it turns out that it's really an

See my reply to Martin.  Restricting the position of xml:lang could
make things worse, unless we also restrict its inheritance - which I'm
not sure that we are allowed to do.

What do you think about only allowing it on atom:content and on Text
constructs?

> maybe this is a handy simplification.

I think that if we allow xml:lang then it should definitely be
restricted. The current "xml:lang everywhere" situation is only simple
to implement if you assume that your implementation stores all of it's
data in an XML DOM.

> Obviously, all bets are off once you get inside atom:content, right?

Well, HTML and XHTML can include lang, and xml:lang according to their
specs.  They inherit the default language from their envelope.  I
think that is pretty well defined.

-- 
Dave

---

From: David Powell <[email-masked]>
Date: Mon, 17 Jan 2005 08:47:30 +0000

>    in a language-independent notation).

Reading the XML spec, I'm not clear that we're allowed to restrict the
inheritance of xml:lang?

From the spec:

> The intent declared with xml:lang is considered to apply to all
> attributes and content of the element where it is specified, unless
> overridden with an instance of xml:lang on another element within
> that content.

> Disallowing xml:lang on text constructs would be quite a bad

I'm fine with allowing it on Text constructs. My worry is allowing to
inherit to elements where it is not appropriate (such as
atom:generator, and extension elements.)

If we allow it to inherit inappropriately, and we restrict it to
certain elements, then this makes the situation even worse, as we'll
have no way of re-declaring or un-declaring xml:lang on elements that
it shouldn't apply to.

I think it is very unlikely that a user would want to include a title,
summary, and content in a different language. I agree that content
might include more than one language, but that is already catered for
by XHTML/HTML's use of xml:lang/lang.

If we only allow xml:lang on Text constructs and atom:content, then
this means that the user would typically have to include it on titles,
summaries, and content separately. Would that be ok?

Would it be any better than a dc:language style atom:language?

> cuts the inheritance of xml:lang.

I didn't realize this - where is it?

-- 
Dave

---

From: Tim Bray <[email-masked]>
Date: Mon, 17 Jan 2005 10:11:22 -0800

> What do you think about only allowing it on atom:content and on Text

I'm very concerned.  Any time you're processing text and you want to 
display it or index it for search, or do pretty well anything, if you 
don't know what language it's in you're probably going to make damaging 
mistakes.  Consider a Turkish author whose name begins with 'I'... feed 
that to a search engine that doesn't know it's in Turkish and the 
results will be wrong.  Suppose Joi Ito wants to list his name in 
Japanese but still write in English; or the the reverse.  Suppose I'm 
publishing in Moscow and I want my copyright statement to be in Russian 
irrespective of what language I'm writing in?

> to implement if you assume that your implementation stores all of it's

This statement is not correct; it's simple to implement correctly if 
you're doing stream processing.

And I repeat, anything that is remotely general-purpose needs to 
associate each little chunk of text with its language, if it wants to 
operate correctly.  Sometimes being correct has convenience costs.

I'm veering towards -1 on this one, absent a few more people putting up 
their hands to say "we must have it". -Tim

---

From: Robert Sayre <[email-masked]>
Date: Mon, 17 Jan 2005 13:59:20 -0500

> you're doing stream processing.

I think David's issue with xml:lang concerns mapping Atom entries to, 
say, an RDBMS. I think he feels it will be challenging to store with 
that requirement. For instance, what does it mean to include an xml:lang 
declaration on atom:updated?

Robert Sayre

---

From: Tim Bray <[email-masked]>
Date: Mon, 17 Jan 2005 11:06:42 -0800

On Jan 17, 2005, at 10:59 AM, Robert Sayre wrote:

> that requirement. For instance, what does it mean to include an 
> xml:lang declaration on atom:updated?

Clearly, it means nothing, and such an occurrence can safely be ignored 
or discarded.  What am I missing? -Tim

---

From: Robert Sayre <[email-masked]>
Date: Mon, 17 Jan 2005 14:17:16 -0500

> or discarded.  What am I missing? -Tim

Let's say I use an Atom protocol client to PUT an entry that has an 
xml:lang attribute. Is it ok for the server to respond with a 204 if 
drops the xml:lang declaration? After all, that's exactly not what I PUT 
there.

I think David is asking for the spec to declare when xml:lang can be 
"safely ignored and discarded". Put another way, where is it part of the 
Atom model.

Robert Sayre

---

From: Tim Bray <[email-masked]>
Date: Mon, 17 Jan 2005 11:37:26 -0800

On Jan 17, 2005, at 11:17 AM, Robert Sayre wrote:

>>> For instance, what does it mean to include an xml:lang declaration 
>>> on atom:updated?
>> Clearly, it means nothing, and such an occurrence can safely be 
>> ignored or discarded.  What am I missing? -Tim
>
> drops the xml:lang declaration? After all, that's exactly not what I 
> PUT there.

The semantics of xml:lang (http://www.w3.org/TR/REC-xml/#sec-lang-tag) 
are very clear.  Given the definition of atom:updated, it is 
unambiguously true that such an attribute can have no effect.  I would 
drop it and return 204 without a moment's worry.

> "safely ignored and discarded". Put another way, where is it part of 
> the Atom model.

Draft format-04 says "Requirements regarding the content and  
interpretation of xml:lang are specified in XML 1.0 
[W3C.REC-xml-20040204] Section 2.12."  If the element or attribute so 
labeled contains free text, xml:lang is relevant.  If not, not.  If you 
don't know (i.e. on some other-namespace exception), you'd better 
preserve it.  -Tim

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 17 Jan 2005 14:32:48 -0500

David Powell wrote:
> problem to many implementors.
	Life is tough! So what?
	XML gives us a standard way to flag the language of things. Unless
there is a really compelling argument to do something different, it isn't
appropriate for us to diverge from the standard. If a compelling argument
*can* be found, then we probably should be talking about revising the base
XML specifications. We've spent decades working on providing the mechanisms
so that we can handle non-English data properly. Let's not go backwards with
Atom. RSS already made the mistake of including a "language" tag and it is a
mess that others are copying into other formats. Let's not contribute to
this pollution of the schema-space....
	There are some fields in Atom which are "language-independent" or
"neutral" and thus it might be useful to explicitly prevent the use of
xml:lang tags for these elements or simply state that they have no meaning
if used. For instance, atom:id, atom:modified, atom:updated, have semantics
and syntax which is independent of language. However, all other fields
should be candidates for xml:lang.

		bob wyman

---

From: Bill de hÓra <[email-masked]>
Date: Mon, 17 Jan 2005 19:45:46 +0000

> or discarded.  What am I missing? -Tim

I'm not too fussed about the outcome either way, but one problem seems 
to be this - when it means nothing isn't specified.

[I'm not sure we should spend a lot of time coping with fact that 
xml:lang isn't a universally applicable attribute for syndication markup.]

cheers
Bill

---

From: David Powell <[email-masked]>
Date: Mon, 17 Jan 2005 20:15:54 +0000

> results will be wrong.

Yes this does sound like a problem.

> Suppose Joi Ito wants to list his name in
> Japanese but still write in English; or the the reverse.

Let's hope he doesn't want to provide a name in more than one language.

> Suppose I'm

Copyright is a Text Construct, so at least this one would be ok.

> you're doing stream processing.

I agree, parsing is simple whether it is stream or tree parsing.  I'm
worried about what we do with this data afterwards.

It isn't clear what xml:lang should apply to? Does it apply to email
addresses? Aural browsers might use it as a pronunciation hint. So
does a database implementation need an "email_lang" column in the
table or not?  I think we need to decide on these issues rather than
leave it to implementors to guess.

> operate correctly.  Sometimes being correct has convenience costs.

I'm not suggesting that "xml:lang everywhere" is useless. I'm pointing
out that the costs are deceptively high. If the consensus is that the
current situation is acceptable then I'll be ok with that.

My Pace might not be an acceptable solution, but I think we perhaps
ought to be clear about what xml:lang applies to, including a strategy
for extensions.

-- 
Dave

---

From: David Powell <[email-masked]>
Date: Mon, 17 Jan 2005 20:16:05 +0000

Monday, January 17, 2005, 7:32:48 PM, you wrote:

> if used.

It is the inheritance which I have the biggest problem with.

> For instance, atom:id, atom:modified, atom:updated, have semantics

Which extension elements does it apply to?

-- 
Dave

---

From: Robert Sayre <[email-masked]>
Date: Mon, 17 Jan 2005 16:32:49 -0500

Tim Bray wrote:
> preserve it.

I'm not disputing your observations on the semantics of xml:lang, but I 
find that last sentence unacceptable and unrealistic in the context of 
the larger discussion on extensibility. I'll try to illustrate why.

Let's say a client generates the following Atom Entry Document to PUT at 
  the server, which returns 204:

<entry xml:lang="en" xmlns="..." xmlns:ext="...">
   ...
   <!-- a comment -->
   <updated xml:lang="de" ext:att="nein">time</updated>
</entry>

I think we can all agree that it would be acceptable for a subsequent 
GET to return this:

<entry xml:lang="en" xmlns="..." xmlns:ext="...">...<updated 
xml:lang="de" ext:att="nein">time</updated></entry>

where the whitespace and comments are missing. So, not all aspects of 
the infoset must be preserved, nor must it be identical the original 
bytestream in the PUT. I think it should be acceptable to return the 
following:

<entry xml:lang="en" xmlns="...">...<updated>time</updated></entry>

This is why David (and myself) are pushing to define locations 
(head,entry,person) where extension content has special significance. 
Extensions would still be allowed at any location. However, if not 
located in one of the predefined locations, it could be discarded, 
because it's outside the "Atom model."

Robert Sayre

---

From: James Aylett <[email-masked]>
Date: Mon, 17 Jan 2005 22:20:29 +0000

On Mon, Jan 17, 2005 at 04:32:49PM -0500, Robert Sayre wrote:

> I think it should be acceptable to return the following:
> 

I agree. However, given:

----------------------------------------------------------------------
<entry xml:lang="en" xmlns="...">
  ...
  <updated xml:lang="en">time</updated>
</entry>
----------------------------------------------------------------------

what should happen? If that can be accepted, with the xml:lang
silently dropped, then it makes it easy for implementors. (I don't
particularly care what happens - this is pretty boundary for me - but
it probably needs to be made clear, or at the very least unambiguously
decipherable from the spec.)

James

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  [email-masked]                               uncertaintydivision.org

---

From: Martin Duerst <[email-masked]>
Date: Tue, 18 Jan 2005 15:56:28 +0900

>> overridden with an instance of xml:lang on another element within

 From this text, it is indeed not clear. There is an erratum that
makes this clearer. Please see
http://www.w3.org/XML/xml-V10-3e-errata#E01.

 >have no way of re-declaring or un-declaring xml:lang on elements that

I don't understand this. If the spec says that it doesn't apply to
some element, there is no reason to un-declare it in the content,
and of course even less a reason to re-declare it.

Please also note that the current version of the XML spec allows
xml:lang="", i.e. the empty string as a value to say that you don't
have any information about the language. This is handy for cases
like 1) the element is natural language text, but the actual content
is e.g. only math or some other special notation that doesn't
belong to a natural language, and 2) the actual content is
natural language text, but the software doesn't know which
language it is.

As an aside: We should make sure that the Atom spec is very
clear for each element that inherits (e.g. copyright,...)
how to define the absence of such information. As experiece
with xml:lang and other things has shown, having something
like a 'reset' or 'neutral element' is extremely important
every time there is inheritance, in particular for information
that comes from different sources.

 >summary, and content in a different language.

Well, somewhat unlikely, but not impossible. What about e.g. a service
in Japan that takes some English feed and just changes the titles
to Japanese for quicker browsing by a Japanese audience?

 >summaries, and content separately. Would that be ok?

Why have to do that for feeds that are completely or mostly monolingual?

 >Would it be any better than a dc:language style atom:language?

Yes. It would still be better.

 >I didn't realize this - where is it?

Ok, here it is: The RDF Graph model doesn't deal with mixed content
(such as typical XHTML), but the data model has a special datatype
that in RDF/XML is indicated by the rdf:parseType="Literal" attribute.
The problem is that whenever you use this, RDF/XML requires you to
cut language inheritance.

From
http://www.w3.org/TR/2004/REC-rdf-concepts-20040210/#section-Literals:

For text that may contain markup, use typed literals with type 
rdf:XMLLiteral. If language annotation is required, it must be explicitly 
included as markup, usually by means of an xml:lang attribute. [XHTML] may 
be included within RDF in this way. Sometimes, in this latter case, an 
additional span or div element is needed to carry an xml:lang or lang attribute.

What this means in practice for Atom is that if it were not for this,
you could just use a DTD to declare a default attribute of
rdf:parseType="Literal" on things like <title>.

However, this won't work, because if you say

<entry xml:lang='en'>
     ...
     <title>My entry title.</title>
     ...
</entry>

which will look to a parser like

<entry xml:lang='en'>
     ...
     <title rdf:parseType="Literal">My entry title.</title>
     ...
</entry>

The XML Literal won't carry the language information.

Even if you put the language information right on title, as so:

<entry>
     ...
     <title  xml:lang='en' rdf:parseType="Literal">My entry title.</title>
     ...
</entry>

it still isn't picked up by RDF/XML. What RDF/XML requires you to do
is something like

  <entry xml:lang='en'>
     ...
     <title rdf:parseType="Literal"><span xml:lang='en'>My entry title.</span></title>
     ...
</entry>

i.e. in order to give the language information to the title in
RDF/XML, you have to introduce a 'dummy' element.

This means that the simple DTD approach won't work, but such
a transform of course can be done by XSLT. It also means that
Henry and friends have some more work to define how Atom
converts/relates to RDF, because they have to define when/how
to introduce dummy elements.

Regards,     Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 18 Jan 2005 15:06:14 +0900

>Let's hope he doesn't want to provide a name in more than one language.

Well, I can definitely imagine situations where somebody like him
may want to do that.

 >leave it to implementors to guess.

I think spelling out in the spec which elements take xml:lang
and which ones don't, and which attributes it applies to and
which not, would help. Anybody wants to start a list?

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 18 Jan 2005 15:08:33 +0900

>It is the inheritance which I have the biggest problem with.

Why? It seems silly to mark each and every entry (or even worse,
each and every title and content) in a feed as xml:lang="en" if
most or all of them are in English. We have other things that
get inherited from the feed to the entries, so I don't see why
this one would be such a problem.

Regards,    Martin.

---

From: David Powell <[email-masked]>
Date: Tue, 18 Jan 2005 09:05:07 +0000

> http://www.w3.org/XML/xml-V10-3e-errata#E01.

Good, that was one of the things that I was concerned about.

> that comes from different sources.

Yes, we should do this.  I think extension elements are the main
problem though.

In PaceExtensionConstruct, I proposed two classes of extension.
Simple Extension constructs were intended to be simple to interpret
and implement.

Should xml:lang apply to them?

If it does, the range of the property would be a (string, language)
pair, which would be useful for language sensitive text, but would be
unnecessary for a lot of cases. Looking at the RSS1.0 list of
extensions, most of them are not language dependant.

If it didn't apply, then Structured Extension constructs could still
be used to implement language sensitive text, but they are more opaque
which would be a disadvantage.

> cut language inheritance.

The solution for RDF that I was thinking of would be to add a BNode,
and attach the parseType="Literal" content to it together with the
xml:lang and xml:base context explicitly as properties; rather than
modify the parseType="Literal" content.

It would be like a virtual Content-Language / Content-Location header.

BTW - Does a HTTP Content-Language set the default language for Atom?

-- 
Dave