Subject: PaceAtomBidi

From: James M Snell <[email-masked]>
Date: Tue, 03 Oct 2006 20:33:19 -0700

This ought to give us something concrete to work from.

- James

#pragma section-numbers off

== Abstract ==

RFC4287 currently has no *efficient* means of indicating the base
directionality of text.  While the Unicode bidi control characters can
be used, they can be problematic to use correctly within XML and do not
allow publishers to set a single default direction for a complete Atom
document.

This proposal adds a new common attribute to all Atom elements called
"dir".  The attribute is optional and has three possible values, "ltr",
"rtl" and "" (representing 'unspecified').  If the attribute is missing,
the value is considered to be unspecified.  This should make the element
fully backwards compatible with existing compliant RFC4287
implementations that properly handle "unknown markup".

This proposal does add the requirement that bidi text in atom docs MUST
be rendered using the unicode bidi algorithm but makes it clear that the
use of dir on an element does not necessarily open a new level of
embedding... meaning that feed consumers have the responsibility of
determining how to apply the bidi algorithm based on how those consumers
choose to display the text (e.g. some readers may display entry content
in a <div> while others may display content in a <span>, both of which
handle bidi text differently depending on context).

I also specify that bidi text in (x)html should use the bidi markup as
defined by those specs.

== Status ==

Proposed.

== Rationale ==

(see abstract)

== Proposal ==

{{{

2.0 Add "dir" to atomCommonAttributes

   atomCommonAttributes =
      attribute xml:base { atomUri }?,
      attribute xml:lang { atomLanguageTag }?,
      attribute dir { "ltr" | "rtl" | "" }?,
      undefinedAttribute*

2.1 The "dir" Attribute

Any element defined by this specification MAY have a "dir" attribute
whose content specifies the base direction of directionally-neutral text
[UNICODE].  Possible values for the attribute are "ltr" and "rtl"
indicating "left-to-right" and "right-to-left" respectively or an empty
string "" indicating that no base-direction is unspecified.  If the"dir"
attribute is not specified, the value is assumed to be unspecified.

The direction specified by "dir" applies only to elements and attributes
whose values are specified as being "Language-Sensitive".  The attribute
is inherited by descendent elements and may be overridden.

Example atom:feed with right-to-left directionality

  <?xml version="1.0" ?>
  <feed xmlns="http://www.w3.org/2005/Atom" dir="rtl">
    <title>&#x0679&#x067A;&#x067B;</title>
    ...
  </feed>

If an Atom document contains any right-to-left characters, the Unicode
Bidirectional Algorithm [UAX#9] MUST be used to render that text.
Because consumers of Atom documents vary broadly in the way they display
text, the use of the "dir" attribute on an element does not necessarily
open an additional level of embedding with respect to the bidirectional
algorithm. Consuming applications that render bidirectional text are
responsible for determining the appropriate level of embedding.

When Atom Text Constructs or the atom:content elements contain
bidirectional text and the type attribute is set to either "html" or
"xhtml", the bidirectional markup mechanisms specific to each format
SHOULD be used.

Example atom:feed with bidirectional XHTML

  <?xml version="1.0" ?>
  <feed xmlns="http://www.w3.org/2005/Atom" dir="rtl">
    <title type="xhtml">
      <div xmlns="http://www.w3.org/1999/xhtml">
        <p dir="rtl">&#x0679&#x067A;&#x067B;</p>
      </div>
    </title>
    ...
  </feed>

The Unicode bidirectional control characters MAY also be used within
attributes and element values to indicate the directionality of text.

IRI's containing bidirectional characters MUST be stored, transmitted
and rendered as defined in [RFC3987] Section 4.
}}}

== Impacts ==

== Notes ==

----

CategoryProposals

---

From: David Powell <[email-masked]>
Date: Wed, 4 Oct 2006 08:00:45 +0100

It is hard to discuss the impacts without knowing what status we are
trying to achieve with this and any other proposed changes to Atom.

Are we planning on changing the Atom namespace?

Adding inheritable attributes seems like a breaking to change.
Existing compliant implementations will silently corrupt such feeds.

Example (incomplete):

<feed dir="rtl">
  <title>rtl feed title</title>
  <entry>
     <title>this should be right to left</title>
  </entry>
</feed>

If this feed is stored anywhere then, a) the undefined attribute may
get lost; and b) even if it doesn't, stateful implementations won't
know that they have to resolve each entry against it and store the
attribute with the entry data as well as the feed data.

Eg, if a later entry is posted to the same feed, and the feed polled
as:

<feed dir="ltr">
  <title dir="rtl">rtl feed title</title>
  <entry>
     <title>this should be left right</title>
  </entry>
</feed>

A stateful implementation needs to know that this implies a feed with
the latest feed metadata:

<feed dir="ltr">
  <title dir="rtl">rtl feed title</title>
</feed>

and two entries:

<entry dir="rtl">
   <title>this should be right to left</title>
</entry>
<entry dir="ltr">
   <title>this should be left right</title>
</entry>

-- 
Dave

---

From: "James Holderness" <[email-masked]>
Date: Wed, 4 Oct 2006 11:28:44 +0100

> The attribute is optional and has three possible values, "ltr",

How about also allowing the values "lro" and "rlo" as used in xhtml2. [1]

> whose content specifies the base direction of directionally-neutral text

I was looking through the various Atom elements that this might apply to and 
the one thing that struck me as problematic was the atom:category element. 
Would the dir attribute apply to the term, the label, both, neither? Is that 
one of the cases where you should just use unicode control characters?

Perhaps it would be better to limit the dir attribute to element content in 
general. Even for elements that only have one "text" attribute, you can 
never be sure that other attributes won't be added in the future. For 
example in atom:link you would think the direction would unambiguously apply 
to the title attribute, but say a future revision or extension added a 
description attribute - suddenly it's not so obvious anymore.

Regards
James

[1] http://www.w3.org/TR/xhtml2/mod-bidi.html

---

From: James M Snell <[email-masked]>
Date: Wed, 04 Oct 2006 07:29:21 -0700

> How about also allowing the values "lro" and "rlo" as used in xhtml2. [1]

Yeah, I'm just not yet sure whether these are strictly necessary.  We
need to explore this.

> I was looking through the various Atom elements that this might apply to
> and the one thing that struck me as problematic was the atom:category
> element. Would the dir attribute apply to the term, the label, both,
> neither? Is that one of the cases where you should just use unicode
> control characters?
> 

The dir should only ever impact Language-Sensitive elements and
attributes so term would not be affected but label would be.

> Perhaps it would be better to limit the dir attribute to element content
> in general. Even for elements that only have one "text" attribute, you
> can never be sure that other attributes won't be added in the future.
> For example in atom:link you would think the direction would
> unambiguously apply to the title attribute, but say a future revision or
> extension added a description attribute - suddenly it's not so obvious
> anymore.
> 

If the new attribute is Language-Sensitive then dir would apply.

- James

---

From: "James Holderness" <[email-masked]>
Date: Wed, 4 Oct 2006 16:14:24 +0100

> Yeah, I'm just not yet sure whether these are strictly necessary.  We

I don't think they are necessary - you can probably get by with Unicode 
control chars - but it might be a good idea for consistency. Otherwise 
you'll sometimes be using the dir attribute and sometimes Unicode control 
chars.

> attributes so term would not be affected but label would be.

Ok. I missed that.

> If the new attribute is Language-Sensitive then dir would apply.

But that means you can't have one attribute being RTL and the other LTR (say 
an Arabic title with an English description).

Regards
James

---

From: James M Snell <[email-masked]>
Date: Wed, 04 Oct 2006 11:46:06 -0700

James Holderness wrote:
> [snip]
> you'll sometimes be using the dir attribute and sometimes Unicode
> control chars.
> 

I had imagined that the feed consumer would decide whether to treat the
dir as an embed or an explicit override depending on context.  If the
feed publisher wants to set the direction without any ambiguity, then
they should use the unicode controls or the (x)html mechansisms (where
appropriate).

>>[snip]
> But that means you can't have one attribute being RTL and the other LTR
> (say an Arabic title with an English description).
> 

Right.  Folks who really want to do this can achieve it using the
unicode controls.

- James

---

From: James M Snell <[email-masked]>
Date: Wed, 04 Oct 2006 20:07:08 -0700

I've updated the Pace (apppended below) to include the "rlo" and "lro"
dir attr values as suggested by James Holderness.  I'm definitely not
happy with the wording, tho.

Example:

  <title dir="rlo">Testing</title>

  Would be generally equivalent to:

  <title type="xhtml">
    <div xmlns="...">
      <bdo dir="rtl">Testing</bdo>
    </div>
  </title>

  and

  <title>[RLO]Testing[PDF]</title>

  (where [RLO]  == the Unicode Right-to-Left Override and
         [PDF] == the Unicode Pop-directional format )

Even with this attribute, achieving proper mixed directionality in plain
text still requires the use of unicode control chars, e.g.

  <title dir="ltr">
    Testing "[RLM]A Hebrew Quotation![RLM]" Testing
  </title>

  (where [RLM] == the Unicode Right-to-Left Mark)

As an alternative, this whole thing can be defined in an extension
rather than as a part of the core spec.

Thoughts?

- James

---------------------------

http://www.intertwingly.net/wiki/pie/PaceAtomBidi

== Abstract ==

== Status ==

Proposed.

== Rationale ==

(see abstract)

== Proposal ==

{{{

   atomCommonAttributes =
      attribute xml:base { atomUri }?,
      attribute xml:lang { atomLanguageTag }?,
      attribute dir { "ltr" | "rtl" | "lro" | "rlo" | "" }?,
      undefinedAttribute*

2.1 The "dir" Attribute

Any element defined by this specification MAY have a "dir" attribute
whose content specifies the base direction of directionally-neutral text
[UNICODE].  Possible values for the attribute are "ltr" and "rtl"
indicating "left-to-right" and "right-to-left" respectively, "lro" and
"rlo" indicating explicit "left-to-right" and "right-to-left" overrides,
or an empty string "" indicating that no base-direction is unspecified.
 If the "dir" attribute is not  specified, the value is assumed to be an
empty string.

  <?xml version="1.0" ?>

If an Atom document contains any right-to-left characters, the Unicode
Bidirectional Algorithm [UAX#9] MUST be used to render that text.
Because consumers of Atom documents vary broadly in the way they display
text, the "ltr" and "rtl" values do not necessarily open an additional
level of embedding with respect to the bidirectional algorithm.
Consuming applications that render bidirectional text are responsible
for determining the appropriate level of embedding.  The "lro" and "rlo"
values are used to override the inherent directionality of characters as
defined by [UNICODE].

  <?xml version="1.0" ?>

== Impacts ==

== Notes ==

----

CategoryProposals

---

From: David Powell <[email-masked]>
Date: Fri, 6 Oct 2006 00:10:21 +0100

I don't have much experience with bidi. I've been having a quick read
up on it, and there seem to be the following features. Correct me if
I’m wrong.

a) Unicode implicitly supports bidi. Write a span containing Hebrew
characters, and it will be laid out right to left. We don’t need to do
anything to support this.

b) <span dir="xxx"> and Unicode controls such as RIGHT-TO-LEFT
EMBEDDING, give hints about how to layout neutral characters such as
punctuation marks that are used between flows of ltr and rtl text. We
don’t need to do anything to support these Unicode controls. <span
dir> can be used in Text constructs if required, but such a mechanism
isn’t necessary for the rest of Atom.

HTML benefits from <span dir> because it is easier to type it in HTML
source than Unicode escapes, but nobody really types Atom source by
hand, and operating systems do support embedding of Unicode escapes
via text widgets (eg, on Windows, install supplemental language
support via Regional Settings control panel - I think?). Allowing dir
to be used anywhere in Atom does not provide the same feature as <span
dir>, because the sole purpose of it is to disambiguate mixed
directions *within* a flow of text, and allowing @dir anywhere in Atom
would not achieve that granularity.

c) <bdo dir="xxx"> and Unicode controls such as RIGHT-TO-LEFT OVERRIDE
allow the Unicode algorithm to be disabled and direction to be
hard-coded.  Under what circumstances is this needed?  Is it
appropriate for Atom?

d) <div dir="xxx"> sets the base direction of the text. One of the
effects of this is that <div dir="rtl"> makes paragraphs right
aligned, and table columns to be laid out right-to-left.

I'm not convinced that we need support for this in Atom. I'd like to
hear more opinions. In HTML it makes sense to specify that a block of
text is right-aligned, but in Atom there are no blocks of text; just
fields. Isn't this largely a presentation feature?

Adding an attribute that is supposed to inherit from feed elements to
entry elements is a breaking change to Atom that is likely to lead to
silent data-loss with existing implementations. Implementations would
need to know to inherit the attribute down to entries and store it
with the entries rather than with the feed, so that retrospective
changes don't change the direction of previously polled entries.

I think that inheritance is useful for a hierarchically structured
document like HTML, but confusing for Atom, which is essentially a
bunch of XML-ified structs with a fairly arbitrary hierarchy
representing multiple entities with seperate life-cycles. (People deal
with feeds containing a subset of all entries ever, not with feed
documents)

e) <html dir="xxx"> in IE at least, this affects not only the page,
but the layout of browser chrome such as scrollbars and Javascript
alerts. I don't think that this feature is appropriate for syndication
format. The user preferences of the user-agent (probably via the OS)
would probably be better suited to control the application layout.

So, the question is: do we need the ability to markup the "base
direction" of text in Atom, at a block level. If so, what are the
applications of the "base direction" in Atom's layout-agnostic fields
such as category label?

Also at what granularity do we need to do this? On every element, or
would it be sufficient to use extension elements to describe the base
direction at the entry/feed level (cf: <html dir>) ?

-- 
Dave

---

From: James M Snell <[email-masked]>
Date: Thu, 05 Oct 2006 16:58:35 -0700

> anything to support this.

Only if rendered using the Unicode Bidirectional Algorithm.

> isn’t necessary for the rest of Atom.

Not quite.  Suppose, for instance, I have a category whose label
attribute consists of primarily (but not completely) right-to-left text.

Example taken from
http://www.w3.org/International/articles/inline-bidi-markup/#visual).
Uppercase letters represent RTL characters.

  <category term="..." label="ABCDEFG, W3C" />

When displayed, using the bidi algorithm, this should be rendered as
"W3C, GFEDCBA".  However, absent any directional information, it will be
incorrectly rendered as "GFEDCBA, W3C".

To get the direction correct in (x)html, we either need to wrap the
value using the unicode controls or use something like <span dir="rtl">.
 In Atom, the only current option is to wrap use the numeric entity
codes, e.g.,

  <category term="..." label="&#x202B;ABCDEFG, W3C&#x202C;" />

If you have multiple categories, text elements, etc that all contain
primarily RTL text, having to wrap all of the text values like this gets
to be quite annoying and problematic.

> would not achieve that granularity.

This is precisely why the Pace gives discretion to the consuming
application on how to apply the bidi alg.  The dir attribute in Atom
would serve only to specify the base directionality of neutral
characters contained in the feed (equivalent to explicitly wrapping
affected text in the equivalent unicode bidi embed codes).

> hard-coded.  Under what circumstances is this needed?  Is it

I imagine the "lro" and "rlo" values being used very rarely, if ever.

> fields. Isn't this largely a presentation feature?

Again, the Pace defers the responsibility of how to apply the bidi alg
to consuming applications.  For instance, some feed readers may choose
to display a RTL title in a block element while others may choose to
display it in an in-line element; both of which will yield different
results.

> changes don't change the direction of previously polled entries.

Yes, but this doesn't leave us any worse off than we currently are.
Implementations that are not direction-aware should treat the new
attribute as foreign markup and drop the data on the floor.  Those
implementations should continue to work as the do currently.

Further, implementors already need to deal with such issues with regards
to author, rights, xml:base and xml:lang so we're not introducing a new
problem.

> would probably be better suited to control the application layout.

Atom does not dictate the behavior of user agents.  Adding a dir
attribute would not change that fact.

> applications of the "base direction" in Atom's layout-agnostic fields

Strictly speaking, we can achieve the same result by using the unicode
bidi controls but doing so can be tricky at times.  The dir attribute
makes life easier.

> direction at the entry/feed level (cf: <html dir>) ?

The attribute should be capable of appearing anywhere xml:lang can
appear and like xml:lang, only affects certain elements and attributes
in the feed.  However, whether or not this works best as an attribute in
the Atom namespace or an attribute in some other extension namespace is
an open question.

- James