Subject: Proposal: Atomext WG

From: James M Snell <[email-masked]>
Date: Thu, 03 Jan 2008 15:27:31 -0800

Recent discussions around tombstones, the Atompub Features draft, and
other extensions have revealed an apparent need for some level of formal
review for I-D's intended for the Standards Track.  After discussing the
subject with Lisa Dusseault (the Area director), I believe that real
progress on these extensions can only be made within the context of a
formal IETF WG.  Therefore, I am proposing the creation of an "Atom
Extensions" (atomext) WG that will be chartered to review and publish
Standards Track extensions and revisions to the Atom format and
publishing protocol.  A draft charter is provided below.

- James

===========================================
Atom Extensions (atomext)

Description of Working Group:

The IETF Atom Extensions Working Group shall publish standards-track
extensions or compatible revisions to the Atom Syndication Format (RFC
4287) and Atom Publishing Protocol (RFC 5023).

This charter explicitly rules out the creation of new versions of the
syntax and the protocol, and any changes that are not compatible with
either RFC 4287 or RFC 5023.

The WG will initially focus on the review of existing proposals for
standards track extensions including:

 * Atom Bidi Extension, draft-snell-atompub-bidi-05
 * Atom Tombstones, draft-snell-atompub-tombstones-03
 * Atompub Features Extension, draft-snell-atompub-feature-12

 Ed Note: shoud fiql be included? draft-nottingham-atompub-fiql-00.txt

As proposals for new standards track extensions are published as
Internet-Drafts and brought to the attention of the WG, the WG will
review the drafts and decide whether or not to take the draft on as a
work product of the WG.  If a decision is made to pick up the draft, a
revision to this charter is required.  If the WG runs out of work items,
the WG will be closed.

The WG will use the same mailing lists used for the original Atom
Publishing Format and Protocol working group: [email-masked] for
discussing the Atom format and [email-masked] for discussing the
protocol.  Thw WG does not intend to hold regular face-to-face meetings,
though may do so if deemed necessary or helpful for resolution of
specific issues.

Goals and Milestones:

 May 2008  Submit Atom Bidi Extension for consideration as a
           Proposed Standard
 May 2008  Submit Atom Tombstones Extension for consideration as a
           Proposed Standard
 May 2008  Submit Atompub Features Extension for consideration as a
           Proposed Standard

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 6 Jan 2008 13:29:17 -0800

James M Snell wrote:
> Recent discussions around tombstones, the Atompub Features 
> draft, and other extensions have revealed an apparent need 
> for some level of formal review for I-D's intended for the 
> Standards Track.  After discussing the subject with Lisa 
> Dusseault (the Area director), I believe that real progress 
> on these extensions can only be made within the context of a 
> formal IETF WG.  Therefore, I am proposing the creation of an 
> "Atom Extensions" (atomext) WG that will be chartered to 
> review and publish Standards Track extensions and revisions 
> to the Atom format and publishing protocol.  A draft charter 
> is provided below.

I think it makes more sense to get people to get some working implementations that have informally agreed on some extensions, and then bring those psuedo-standards to the IETF for standardization. Honestly, though, I'm not really sure what benefit that IETF provides over informal agreement between mailing list members. 

> The WG will initially focus on the review of existing 
> proposals for standards track extensions including:
> 

How much of a problem is BIDI in Atom today? (This isn't a rhetorical question). Atom documents are almost never hand-entered, and there is already a specification in place for markup up BIDI and even ruby text in general XML. The odds that clients and servers are going to correctly implement this extension--except those targeted direclty towards BIDI users--seem pretty low to me. Personally, it seems much easier to implement the an existing BIDI markup mechanism (Unicode, XML, and/or XHTML) than a new standard. 

Most Atom constructs that hold human-readable text allow for XHTML, which has its own BIDI mechanism. The main problem seems to be with person constructs, and with extension elements. The Atom BIDI draft doesn't solve the internationalization problems with person constructs because it doesn't provide a way to do Ruby text markup, which is especially important for atom:name. It also doesn't solve the problem with atom:link/@title or other attributes that are language-sensitive. We should have a guideline that says that every extension element that contains human-readable text should be an atom text construct--i.e. that is should support XHTML markup, and the BIDI and Ruby subsets of XHTML in particular. And, the specification for atom:name and other existing language-sensitive elements should be updated so that they meet this guideline.

- Brian

---

From: "David Powell" <[email-masked]>
Date: Sun, 6 Jan 2008 22:46:56 -0000

Brian Smith wrote:

> How much of a problem is BIDI in Atom today? (This isn't a rhetorical
> question). Atom documents are almost never hand-entered, and there is
> already a specification in place for markup up BIDI and even ruby text
> in general XML. The odds that clients and servers are going to
> correctly implement this extension--except those targeted direclty
> towards BIDI users--seem pretty low to me. Personally, it seems much
> easier to implement the an existing BIDI markup mechanism (Unicode,
> XML, and/or XHTML) than a new standard.

I agree.  We already have HTML - which supports much more internationalization features than just bidi.  We also have Unicode bidi controls, for people who don't want to use HTML for whatever reason.  Do we really need 3 different interacting implementations of bidi?  This proposal will just cause interop problems when people try to use it with implementations that don't yet support it, when those people could probably have just used the HTML or Unicode techniques with more success.

And isn't the purpose of bidi markup usually to clarify ambiguities that would arise from applying the Unicode bidi algorithms?  To do that you need to be able to mark up the interior of these text-only elements, which this proposal wouldn't support anyway. 

As I mentioned in my previous email re: feed/entry metadata elements: Mixing feed and entry metadata is considered harmful (by me).  This applies here too, because applying inheritable bidi attributes at <feed> level and expecting it to apply to the contained <entry> elements is something that can only be done if the implementation of the component that decouples feed elements and entry elements from a stateful feed is aware of it.  And we shouldn't be designing extensions that require software to be updated everytime someone writes an Internet Draft in order that they don't corrupt people's data.

If we really need support for Ruby and bidi in <atom:name>, and we aren't immediately intending to supercede RFC4287, I think we would be better adding an additional entry/person metadata element that supports HTML names.  But to be honest, I'd like to see people knocking on our doors before we start writing drafts to solve potential problems.

-- 
Dave

---

From: James M Snell <[email-masked]>
Date: Sun, 06 Jan 2008 16:49:04 -0800

David Powell wrote:
> I agree.  We already have HTML - which supports much more 
> internationalization features than just bidi.  We also have 
> Unicode bidi controls, for people who don't want to use HTML 
> for whatever reason.  Do we really need 3 different interacting 
> implementations of bidi?  This proposal will just cause interop 
> problems when people try to use it with implementations that don't
> yet support it, when those people could probably have just used the 
> HTML or Unicode techniques with more success.

This has been covered many times in the past.  The xhtml mechanisms do 
not help when we're dealing with bidi text in attributes like 
atom:link/title and atom:category/label, the various plain text 
elements, and extensions.

Second, when dealing with (x)html text and content, the specification 
specifically recommends using the bidi mechanisms native to each.

Third, neither Unicode or the W3C recommend using the Unicode bidi 
formatting codes are not recommended in markup, for a variety of 
reasons.  The fact that Atom will typically be created by software 
doesn't solve the problem.

Fourth, even with the Unicode formatting codes, there are existing 
deployed feed readers that get it wrong.

Fifth, the bidi attribute defined in the spec does not introduce any 
additional interop issues.  Client implementations that do not support 
it will continue to act exactly as they do today.  If intermediaries 
drop the attribute on the floor, no critical data is lost and no 
existing applications will be broken.  I know this for a fact because I 
have about 100k internal Atom feeds that are currently using the bidi 
attribute and have been since May of last year.

> 
> And isn't the purpose of bidi markup usually to clarify ambiguities that 
> would arise from applying the Unicode bidi algorithms?  To do that you need 
> to be able to mark up the interior of these text-only elements, which this 
> proposal wouldn't support anyway. 
> 

Again, existing bidi markup does not help us outside of (x)html text and 
content elements.

The goal of this specification is to provide a means of indicating the 
text direction without changing the way existing implementations work -- 
meaning that existing implementations that do not support the extension 
will continue working as they always have while implementations that do 
support the extension can realize it's benefits.

> [snip]
> 
> If we really need support for Ruby and bidi in <atom:name>, and we aren't 
> immediately intending to supercede RFC4287, I think we would be better 
> adding an additional entry/person metadata element that supports HTML names.  

And how would that help us with atom:link/@title and atom:category/@label?

> But to be honest, I'd like to see people knocking on our doors before we start 
> writing drafts to solve potential problems.
> 

Ok, I'm knocking.  This is not a potential problem.  I implement and 
support existing deployed software that publishes feeds with bidi text. 
  I've seen the rendering problems first hand. There's nothing potential 
about it.

- James

---

From: James M Snell <[email-masked]>
Date: Sun, 06 Jan 2008 17:24:32 -0800

Brian Smith wrote:
> [snip]
> I think it makes more sense to get people to get some working 
> implementations that have informally agreed on some extensions,

I implemented extended bidi support in Apache Abdera months ago; and our 
internal blogging environment supports bidi entries.

> [snip]
> How much of a problem is BIDI in Atom today? (This isn't a rhetorical question). 

It's a problem. Quite a few readers I have tested do not even properly 
support bidi markup used in entry title's that use (x)html and there are 
language-sensitive attributes in the Atom schema that cannot use the 
existing markup oriented solutions at all; and since the values for 
those attributes are typically provided by humans, relying on the 
software to properly insert the unicode formatting codes (which aren't 
recommended for markup in the first place) is problematic at best.

Fortunately, there are ways of guessing the text direction with fairly 
decent accuracy given the value of xml:lang.  That is, the language or 
script identified could be RTL.  However, it's not always clear and 
guessing is not always accurate.  The bidi attribute gives us a good, 
explicit fallback.

> Atom documents are almost never hand-entered, and there is already a specification 
> in place for markup up BIDI and even ruby text in general XML. The odds that clients 
> and servers are going to correctly implement this extension--except those targeted 
> direclty towards BIDI users--seem pretty low to me. Personally, it seems much easier 
> to implement the an existing BIDI markup mechanism (Unicode, XML, and/or XHTML) than 
> a new standard. 
> 

What are you basing that on?  There's really not much guess work 
involved in the implementation and apps that choose not to implement 
support will be no worse or better off than they are currently.

> Most Atom constructs that hold human-readable text allow for XHTML, which 
> has its own BIDI mechanism. The main problem seems to be with person 
> constructs, and with extension elements. The Atom BIDI draft doesn't solve 
> the internationalization problems with person constructs because it doesn't 
> provide a way to do Ruby text markup, which is especially important for atom:name. 

Don't forget atom:link/@title and atom:category/@label. 
atom:category/@term can also cause problems when implementation use that 
value for display purposes.  Arbitrary extension elements can also have 
problems.

The bidi draft doesn't attempt to solve all the i18n issues with Atom. 
Ruby text is a problem for pretty much everything, especially given the 
fact that most browsers don't have a clue how to properly render ruby 
text yet.  The bidi draft rightfully focuses on one small part of the 
problem.

> It also doesn't solve the problem with atom:link/@title or other attributes that 
> are language-sensitive.

Yes, it does.

- James

---

From: David Powell <[email-masked]>
Date: Mon, 07 Jan 2008 02:24:47 +0000

James M Snell writes:

Well answered!  Thanks for the patience, you've convinced me.

Constructive criticism now - could you explain this:

  Implementers are reminded that unexpected results could occur when
  using both the "dir" attribute and the Unicode control characters
  within a single document.

Isn't using both the expected, and only, way to mark-up bidi text when
the Unicode heuristics won't do a good enough job?  It seems a bit odd
to warn against using the feature as intended?

> reasons.

True, but I'm not sure I'd describe a plain text element content or
attribute value as markup.  Isn't markup something that could
alternatively contain tags, and be "marked-up"?  Given that these
locations can't, the Unicode bidi codes will need to be used
regardless, so I'm not sure whether those recommendations apply here?

> deployed feed readers that get it wrong.

And will continue to.

-- 
Dave

---

From: James M Snell <[email-masked]>
Date: Sun, 06 Jan 2008 18:53:35 -0800

> Well answered!  Thanks for the patience, you've convinced me.

Good to hear.

> to warn against using the feature as intended?

No, both should not be used.

> regardless, so I'm not sure whether those recommendations apply here?

The markup in question here are the Atom elements.  In other words, the 
bidi formatting characters should not be used in XML, period, not just 
within specific kinds of elements.

>> deployed feed readers that get it wrong.

Yep.

- James

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 6 Jan 2008 23:27:05 -0800

James M Snell wrote:
> I implemented extended bidi support in Apache Abdera months 
> ago; and our internal blogging environment supports bidi entries.

That is a definitely a positive first step. What other implementations does Abdera interoperate with with regard to BIDI text?

> It's a problem. Quite a few readers I have tested do not even 
> properly support bidi markup used in entry title's that use 
> (x)html.

That is exactly my point. People are not even implementing the current standards for BIDI text. Adding another standard to implement is not going to make the situation better.

> Since the values for those attributes 
> are typically provided by humans, relying on the software to 
> properly insert the unicode formatting codes (which aren't 

If the software cannot correctly insert the formatting codes into the document, how would it be able to insert the correct directionality markup?

> > Atom documents are almost never hand-entered, and there is 
> > already a specification in place for markup up BIDI and even
> > ruby text in general XML. The odds that clients and servers
> > are going to correctly implement this extension--except
> > those targeted direclty towards BIDI users--seem pretty
> > low to me. Personally, it seems much easier to 
> > implement the an existing BIDI markup mechanism (Unicode, 
> What are you basing that on?

The Unicode/W3C guidelines (http://www.w3.org/TR/unicode-xml/#Bidi and http://www.w3.org/International/questions/qa-bidi-controls) say this:

* Use *XHTML* BIDI markup whenever possible.
* Otherwise *CSS* whenever possible.
* Otherwise, consider building BIDI markup into your markup schema.
* We have to support BIDI formatting codes anyway, since the above mechanisms don't solve all BIDI problems.

> There's really not much guess work 

That is not true. Consider a feed aggregator. If it doesn't support Atom BIDI, then it will not correctly rewrite entries to handle an inherited "dir" attribute from the atom:feed element. Since the directionality is also inherited by text constructs, whenever the implementation passes the text construct to a rendering engine, it needs to rewrite that content to handle the inherited directionality.

> value for display purposes.

I know that. But, the Atom BIDI draft does not eliminate all uses of BIDI formatting characters in these attributes, either. And, it doesn't specify BIDI support for language-sensitive content in Atom service documents, category documents, RSS feeds, or RSD documents.

> Arbitrary extension elements can also have problems.

The BIDI draft says it only applies to constructs that RFC4287 labeled "language sensitive." Accordingly, the BIDI draft does not apply to extension elements.

> The bidi draft doesn't attempt to solve all the i18n issues 
> with Atom. Ruby text is a problem for pretty much everything, 
> especially given the fact that most browsers don't have a
> clue how to properly render ruby text yet.  The bidi draft 
> rightfully focuses on one small part of the problem.

I agree that a narrow scope is good. But, a solution for Ruby text will also be applicable to BIDI, especially if that solution involves the reuse of XHTML markup and/or CSS. 

> > It also doesn't solve the problem with atom:link/@title or 

The Atom BIDI draft does not provide a way of specifying base different base directionalities for attributes on the same element, it doesn't eliminate all need for BIDI formatting characters in language-sensitive attribute values, it doesn't provide a mechanism for discovering which (nested) extension elements and attributes are affected by the proposed Atom BIDI markup. 

If I implement RFC4287, the Unicode BIDI algorithm, XHTML BIDI, HTML BIDI, and the "Unicode in XML" guidelines, I will have pretty good BIDI support. It will require me to adhere to four different BIDI standards in addition to RFC4287. That is a lot of work already. Now, your Atom BIDI and URI template BIDI proposals add two more specifications that I would have to support--for a total of SIX standards to adhere to and resolve conflicts between, JUST to support BIDI text. And, even if I create well-formed documents adhering to all six standards, whenever I open them up in any of my text editors, or any feed reader, they will look wrong since nobody else is implementing all of those standards. I think that is totally unreasonable. Abdera might have amazing support for BIDI, for which you should be commended, but unless all Atom software is going to be implemented on top of Abdera, Abdera will not be able to reliably interoperate with anything. If we want to provide interopera!
 ble support for BIDI, we need to make it as simple to implement as possible.

My counter-proposal is simple:

* Use XHTML/HTML BIDI/Ruby markup whenever possible.
* Otherwise, use Unicode BIDI/Ruby formatting codes, such that matching pairs of formatting codes are fully contained within a single text or attribute node.
* Editors of new documents must be meticulous about inserting the proper markup and formatting codes.
* Processors of existing documents must be meticulous about preserving BIDI/Ruby markup and/or formatting codes whenever any part of the contained text is preserved.

I recognize that this goes against the Unicode in XML guidelines. However, Atom already goes against the guidelines by having language-sensitive text in attribute values and other contexts where XHTML markup cannot be used.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 6 Jan 2008 23:27:05 -0800

David Powell wrote:
> Brian Smith wrote:
> If we really need support for Ruby and bidi in <atom:name>, 
> and we aren't immediately intending to supercede RFC4287, I 
> think we would be better adding an additional entry/person 
> metadata element that supports HTML names.  But to be honest, 
> I'd like to see people knocking on our doors before we start 

I have Japanese users and Japanese language learners that make extensive use of Ruby markup as an instructional device for teaching Kanji (the complex Japanese characters).

http://tools.ietf.org/html/rfc4287#section-6.4: Atom allows foreign markup anywhere in an Atom document, except where it is explicitly forbidden.

http://tools.ietf.org/html/rfc4287#section-3.2.1: The "atom:name" element's content conveys a human-readable name for the person.  The content of atom:name is Language-Sensitive.  Person constructs MUST contain exactly one "atom:name" element.

http://tools.ietf.org/html/rfc4287#section-6.3: Atom Processors that encounter foreign markup in a location that is legal according to this specification MUST NOT stop processing or signal an error. 

So, the following is legal already, according to RFC 4287:

  <author xmlns='http://www.w3.org/2005/Atom'
          xmlns:html='http://www.w3.org/1999/xhtml'>
    <name>
       <html:span dir='rtl'>foo</html:span>
    </name>
  </author>

And, I can't find anything that disallows the "type" attribute either:

  <author xmlns='http://www.w3.org/2005/Atom'
          xmlns:html='http://www.w3.org/1999/xhtml'>
    <name type='xhtml'>
       <html:span dir='rtl'>foo</html:span>
    </name>
  </author>

Keep in mind that the Relax NG schema in RFC 4287 is non-normative, and Appendix B already admits that the schema is too restrictive.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 6 Jan 2008 23:36:41 -0800

James M Snell wrote:
> > Isn't using both the expected, and only, way to mark-up 
> > bidi text when the Unicode heuristics won't do a good enough job?
> > It seems a bit odd to warn against using the feature as intended?
> 
> The markup in question here are the Atom elements.  In other 
> words, the bidi formatting characters should not be used in 
> XML, period, not just within specific kinds of elements.
> 

Even the W3C recommends Unicode control characters in some situations;
see [1]. I don't know why those W3C guidelines regarding markup vs.
formatting codes are different from [2], but the use of Unicode bidi
formatting characters in Atom is also reinforced by the language in [3]:
"Other situations where control characters may provide the only resort
are elements that only allow character content or that omit support for
directional attributes." IMO, that is exactly the case that Atom fits
into, since RFC4287 has no BIDI features built in.

[1] Internationalization Best Practices: Handling Right-to-left Scripts
in XHTML and HTML Content 
    http://www.w3.org/TR/i18n-html-tech-bidi/#ri20030218.135304584
[2] Unicode in XML and other Markup Languages
    http://www.w3.org/TR/unicode-xml/
[3] FAQ: Unicode controls vs. markup for bidi support
 
http://www.w3.org/International/questions/qa-bidi-controls.en.php#proble
ms

- Brian

---

From: James M Snell <[email-masked]>
Date: Sun, 06 Jan 2008 23:45:31 -0800

> * We have to support BIDI formatting codes anyway, since the above mechanisms don't solve all BIDI problems.

Sorry if I wasn't clear, I was referring specifically to your assertion 
that, "it seems much easier to implement the an existing BIDI markup 
mechanism"... Having already implemented it, I can see no additional 
complexity or difficulty.

> That is not true. Consider a feed aggregator. If it doesn't support 
> Atom BIDI, then it will not correctly rewrite entries to handle an 
> inherited "dir" attribute from the atom:feed element. Since the 
> directionality is also inherited by text constructs, whenever the 
> implementation passes the text construct to a rendering engine, it 
> needs to rewrite that content to handle the inherited directionality.

If an aggregator doesn't support the dir attribute, it will ignore it or 
drop it as if it wasn't there in the first place and will continue to 
operate as it always had, which is exactly what we want.  So again, 
those apps will be no worse or better off than they are currently.

> I know that. But, the Atom BIDI draft does not eliminate all uses of BIDI 
> formatting characters in these attributes, either. And, it doesn't specify 
> BIDI support for language-sensitive content in Atom service documents, 
> category documents, RSS feeds, or RSD documents.

I have no interest in eliminating all uses of bidi formatting characters.

And yes, the spec does address bidi support for language-sensitive 
content in Atompub service and category documents.  The spec language 
can be improved in this regard, but the spec alters the definition of 
the atomCommonAttributes production to add the dir attribute.  In both 
the Atompub service document and the Atom categories document, the 
langauge-sensitive text is provided by elements from the Atom namespace 
(e.g. atom:category and atom:title).  Also, the bidi spec predates the 
publication of rfc5023; I intend to have the next rev of the spec 
specifically discuss atompub service docs.

As for RSS and RSD doc's, I couldn't care less about solving the i18n 
issues of either.

> The BIDI draft says it only applies to constructs that RFC4287 labeled "language sensitive." Accordingly, the BIDI draft does not apply to extension elements.

Section 6.4.2: "Structured Extension elements are Language-Sensitive."

> I agree that a narrow scope is good. But, a solution for Ruby text will also be applicable to BIDI, especially if that solution involves the reuse of XHTML markup and/or CSS. 

if/when the need emerges to improve ruby text support in Atom, I will 
glady help work out a solution.

> The Atom BIDI draft does not provide a way of specifying base different 
> base directionalities for attributes on the same element, it doesn't eliminate 
> all need for BIDI formatting characters in language-sensitive attribute values, 
> it doesn't provide a mechanism for discovering which (nested) extension elements 
> and attributes are affected by the proposed Atom BIDI markup. 

atom:category and atom:link each have exactly one language-sensitive 
attribute.

there's no reason to try eliminating all need for bidi formatting 
characters in language sensitive attribute values.

The bidi attribute applies to language-sensitive elements and 
attributes. Defining which extension elements are language sensitive is 
up to the extension definition.  There's no reason to provide a 
mechanism for discovering which extensions are affected.

> 
> If I implement RFC4287, the Unicode BIDI algorithm, XHTML BIDI, HTML BIDI, and 
> the "Unicode in XML" guidelines, I will have pretty good BIDI support. It will 
> require me to adhere to four different BIDI standards in addition to RFC4287. 
> That is a lot of work already. Now, your Atom BIDI and URI template BIDI proposals 
> add two more specifications that I would have to support--for a total of SIX standards 
> to adhere to and resolve conflicts between, JUST to support BIDI text. And, even if 
> I create well-formed documents adhering to all six standards, whenever I open them up 
> in any of my text editors, or any feed reader, they will look wrong since nobody 
> else is implementing all of those standards. I think that is totally unreasonable. 
> Abdera might have amazing support for BIDI, for which you should be commended, but 
> unless all Atom software is going to be implemented on top of Abdera, Abdera will 
> not be able to reliably interoperate with anything. If we want to provide interopera!

None of this is simple. Please do not claim that it is.

> * Use XHTML/HTML BIDI/Ruby markup whenever possible.

Keep in mind the fact that the atom bidi spec very clearly indicates 
that the (x)html bidi mechanisms should be used in addition to the atom 
dir attribute.

> * Otherwise, use Unicode BIDI/Ruby formatting codes, such that matching pairs of formatting codes are fully contained within a single text or attribute node.

Whose responsibility is it to apply the formatting codes? The person 
typing the text or the software?  How does the software know when to 
apply the codes?  Also, what about when an Atompub client edits an 
entry?  Is the Atompub client responsible for preserving the unicode 
formatting characters?  What if they don't?  There are existing Atompub 
clients out there that, more than likely, will not, and since the 
formatting codes are non-visual, it's not likely the user will notice 
them either, causing unexpected rendering issues later on.  How are 
non-bidi enabled clients supposed to know what to do?  With the bidi 
attribute approach, per rfc5023, non-supporting clients are expected to 
at least preserve the bidi attribute but will otherwise continue working 
as they currently do, without risk of corrupting the text by 
inadvertently dropping or improperly nesting the bidi controls.

Also, imagine a case where we have a feed with 100 entries, each with 
about 5 atom:category elements.  Let's stay that the feed is generally 
all RTL.  Using your approach, that's at least 1000 extra characters in 
the feed, and 500 opportunities for the embedding to be screwed up.

Re: The ruby formatting codes: Even the Unicode spec warns against using 
the ruby formatting codes for anything other than internal storage.  We 
gain absolutely nothing by bringing ruby into this discussion.

> * Processors of existing documents must be meticulous about preserving BIDI/Ruby markup and/or formatting codes whenever any part of the contained text is preserved.

Again, what about older editors that know nothing about the proper 
markup or formatting codes?

> I recognize that this goes against the Unicode in XML guidelines. However, Atom 
> already goes against the guidelines by having language-sensitive text in attribute 
> values and other contexts where XHTML markup cannot be used.
> 

What is the benefit of going against the Unicode in XML guidelines?

- James

---

From: James M Snell <[email-masked]>
Date: Sun, 06 Jan 2008 23:56:52 -0800

With this, I think it is important to look at both the original intent 
of the spec and what clients are most likely to support.  While the 
schema is non-normative, the fact that atom:name is clearly and 
specifically marked as "text" in the schema makes the intent clear.  You 
*might* be able to stretch really far and claim that escaped HTML is 
permitted, but if that was the intent, the spec would more than likely 
say so.

If you want the Person construct to contain ruby text, the best approach 
would be to define an extension element designed for the purpose of 
providing a "display" version of the name, e.g.

<author>
   <name>name</name>
   <x:display-name>
     <ruby xmlns="http://www.w3.org/1999/xhtml">
       <rb>name</rb><rp>(</rp><rt>ruby</rt><rp>)</rp>
     </ruby>
   </x:display-name>
   ...
</author>

This would allow existing clients to continue to operate as they always 
have.

- James

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 10:27:43 -0800

James M Snell wrote:
> Brian Smith wrote:
> > The Unicode/W3C guidelines say:
> > * We have to support BIDI formatting codes anyway, since 
> >   the above mechanisms don't solve all BIDI problems.
> 
> Sorry if I wasn't clear, I was referring specifically to your 
> assertion that, "it seems much easier to implement the an 
> existing BIDI markup mechanism"... Having already implemented 
> it, I can see no additional complexity or difficulty.

 I wrote an Atom-to-HTML converter in XSLT, where, as long as the BIDI formatting characters do not span (text/attribute) nodes in the document, the BIDI directionality is maintained. This is trivial to do by simply preserving the values of the nodes. In fact, many applications should *accidently* get it right as long as they are not doing rewriting of the actual content of the nodes; even then, they will still probably get it right. This is the primary basis of my assertion: significantly more BIDI-oblivious applications will preserve Unicode formatting characters than BIDI markup.

> > The BIDI draft says it only applies to constructs that 
> > RFC4287 labeled "language sensitive." Accordingly, the BIDI 
> > draft does not apply to extension elements.
> 

Which attributes on elements and subelements of structured extension elements are language-sensitive? It seems it must be either all or none.

> characters in language sensitive attribute values.

I agree. If there are going to be formatting codes in the document anyway, then why do we need a mechanism that duplicates their functionality?

> > * Otherwise, use Unicode BIDI/Ruby formatting codes, such 
> that matching pairs of formatting codes are fully contained 
> within a single text or attribute node.
> 
> apply the codes?

> Also, what about when an Atompub client edits an entry?
> Is the Atompub client responsible for preserving the unicode 
> formatting characters? What if they don't?

My hypothesis is that an implementation ignorant of BIDI issues is more likely to preserve the formatting characters than Atom/XHTML/HTML BIDI markup, especially when the effects of those formatting characters never span multiple nodes in the document.

> There are existing Atompub clients out there that, more than likely,
> will not, and since the formatting codes are non-visual, it's not
> likely the user will notice them either, causing unexpected
> rendering issues later on.

I don't understand why a user that needs BIDI functionality would use a client that doesn't have BIDI support. Further, I don't understand how clients that are incapable of generating BIDI formatting codes can generate markup compliant with your proposal. Are you expecting the users to edit the markup directly? 

Markup is just as non-visual as formatting codes, except for people using "view source" and the like.

> With the bidi attribute approach, per rfc5023, non-supporting
> clients are expected to at least preserve the bidi attribute
> but will otherwise continue working as they currently do,
> without risk of corrupting the text by inadvertently dropping
> or improperly nesting the bidi controls.

RFC 4287 and RFC 5023 is pretty unclear about what is required to be preserved. Firstly, RFC 5023 says that implementations can do whatever they want as long as the results are well-formed. Otherwise, AtomPub implementations that use (X)HTML whitelists would be non-compliant. Also, the requirement to preserve unknown foreign markup seems to apply more to unknown extension elements than to unknown attributes on known elements. In particular, if I replace the atom:author element with a new one, then I am not going to preserve the old, unknown attributes on the previous atom:author element.

> about 5 atom:category elements.  Let's stay that the feed is 
> generally all RTL.  Using your approach, that's at least 1000 extra 
> characters in the feed, and 500 opportunities for the embedding to
> be screwed up.

The category labels are almost always going to be composed of strong RTL and strong LTR characters (only), so the BIDI algorithm will work correctly. And, when it doesn't work, it is unlikely that it will be due to the wrong base directionality--in these cases, formatting characters are going to be needed no matter what. Right? 

> Re: The ruby formatting codes: Even the Unicode spec warns 
> against using the ruby formatting codes for anything other than internal 
> storage.  We gain absolutely nothing by bringing ruby into this discussion.

I disagree. Bringing Ruby text into the discussion emphasizes the need for *new* language-sensitive constructs to be text constructs, and emphasizes the need to support (X)HTML in all text constructs instead of depending on Unicode formatting codes.

> > * Editors of new documents must be meticulous about 
> >   inserting the proper markup and formatting codes.
> > * Processors of existing documents must be meticulous about 
> >   preserving BIDI/Ruby markup and/or formatting codes whenever 
> >   any part of the contained text is preserved.
> 

All of these old editors will also fail to implement the Atom BIDI spec too. There are currently more editors that can insert formatting characters than can handle Atom BIDI markup, aren't there? I think that will always be the case.

> What is the benefit of going against the Unicode in XML guidelines?

1. It is extremely easy to implement. As long as an implementation copies text nodes and attribute node values verbatim, or using any transform that doesn't strip out formatting codes, the BIDI directionality will always be preserved, with no extra work. A processor can extract any entry, the content of any entry, any entry metadata, or any feed metadata, without having to rewrite the markup to preserve the BIDI information. Literally, it requires more work to screw it up than it is to implement it correctly.

2. It is general purpose. It works for RSS, Atom, and any other XML format.

3. It reuses existing markup without introducing any of its own.

Here is an expanded version of my proposal that provides BIDI and Ruby support for all language-sensitive constructs in Atom:.

In all cases, only use markup and/or formatting codes when necessary. It is recommended to avoid Ruby Unicode formatting characters whenever possible.

Text Constructs: Use (X)HTML, and use the (X)HTML BIDI and Ruby markup.

atom:content: Use (X)HTML whenever possible. For other XML content, use the existing BIDI and Ruby markup defined for that XML application; Otherwise, use Unicode formatting codes, and ensure that matching pairs of formatting codes do not span multiple text nodes. For cases where type="text" must be used, use Unicode formatting characters.

Language-sensitive attributes: Use Unicode formatting characters.

Extension elements: New language-sensitive extension elements should ensure that all language-sensitive text is enclosed in text constructs. In particular, do not use attributes for language-sensitive text, support at least type="xhtml", and support type="html" whenever possible. Be sure to support BIDI and Ruby markup.

atom:name: When consuming atom:name, treat it as though it was defined as a text construct, and allow for type="xhtml" and type="html". When producing atom:name elements, BIDI markup SHOULD be replaced with Unicode formatting characters when needed. Ruby markup may be replaced with Unicode formatting characters or stripped entirely. All markup should be stripped entirely, the type attribute should be removed.

- Brian

---

From: James M Snell <[email-masked]>
Date: Mon, 07 Jan 2008 09:33:51 -0800

Brian Smith wrote:
> [snip[

It depends on the definition of the extension.  If you're not familiar 
with the extension, you have no way of knowing.

> I agree. If there are going to be formatting codes in the document anyway, then why do we need a mechanism that duplicates their functionality?

Because their use in markup is problematic and is actively discouraged 
for a number of very good reasons.

> My hypothesis is that an implementation ignorant of BIDI issues is more 
> likely to preserve the formatting characters than Atom/XHTML/HTML BIDI 
> markup, especially when the effects of those formatting characters never 
> span multiple nodes in the document.

Have you tested this hypothesis using real editors?  Example, in our 
internal blogging environment, tags are entered in a single text box, 
each tag separated by a comma.  The system splits the tags into an array 
and saves each tag separately.  Each tag becomes a separate 
atom:category element.  Is the user responsible for adding the 
appropriate formatting codes around each individual tag?  When the user 
wishes to edit the entry later, perhaps to add a new entry, are they 
supposed to just know that there are non-visual bidi formatting codes 
interspersed into the comma separated list of tags?

> I don't understand why a user that needs BIDI functionality would use a 
> client that doesn't have BIDI support. Further, I don't understand how 
> clients that are incapable of generating BIDI formatting codes can generate 
> markup compliant with your proposal. Are you expecting the users to edit 
> the markup directly? 

You're assuming that all users have the same requirements.  In our 
environment, a single feed may include entries from many different 
users. We have group blogs where users from many different locales have 
edit rights on any entry in the blog.  Further, our users use many 
different editors to write and manage their blog entries.  Asking those 
users to be mindful of how they're using bidi formatting characters is a 
lot more difficult than what we currently do, which is provide a simple 
check box to indicate whether or not the entry is "right-to-left", which 
in turn, is translated into the appropriate dir="rtl" in the markup. 
Clients that do not understand the dir attribute simply ignore it, and 
since our software is written so that only explicit changes in value are 
recognized (e.g. a missing dir attribute does not mean the dir attribute 
value has changed) we're able to work seamlessly with editors that do 
not support the attribute.

> Markup is just as non-visual as formatting codes, except for people using "view source" and the like.

Yes, but with the marup we don't have to rely on users getting it right 
when they type in the values.

> RFC 4287 and RFC 5023 is pretty unclear about what is required to be preserved. 

RFC 5023, Section 9.3: To avoid unintentional loss of data when editing 
Member Entries or Media Link Entries, an Atom Protocol client SHOULD 
preserve all metadata that has not been intentionally modified, 
including unknown foreign markup as defined in Section 6 of [RFC4287].

Seems pretty darn clear to me.

> Firstly, RFC 5023 says that implementations can do whatever they want as long as 
> the results are well-formed. Otherwise, AtomPub implementations that use (X)HTML 
> whitelists would be non-compliant. Also, the requirement to preserve unknown foreign 
> markup seems to apply more to unknown extension elements than to unknown attributes 
> on known elements. In particular, if I replace the atom:author element with a new 
> one, then I am not going to preserve the old, unknown attributes on the previous 
> atom:author element.
> 

9.3 uses the term "unknown foreign markup", which, if RFC 4287 defines 
as unknown elements AND attributes.

> The category labels are almost always going to be composed of strong RTL and 
> strong LTR characters (only), so the BIDI algorithm will work correctly. And, 
> when it doesn't work, it is unlikely that it will be due to the wrong base 
> directionality--in these cases, formatting characters are going to be needed 
> no matter what. Right? 

"almost always" is not "always".  The Atom bidi draft covers the cases 
where "always" is more desirable than "almost always".

And no, the formatting characters are not always going to be needed.  If 
I am rendering the text in (x)html, I don't want the formatting 
characters in there at all; rather, I want to follow best practices and 
use the (x)html provided bidi markup mechanisms.

> All of these old editors will also fail to implement the Atom BIDI spec too. 
> There are currently more editors that can insert formatting characters than 
> can handle Atom BIDI markup, aren't there? I think that will always be the case.
> 

Failing to implement the atom bidi spec has significantly fewer 
consequences than improperly implementing the unicode formatting 
characters.  That is, existing applications will be no worse off than 
they currently are if the dir attribute is ignored or dropped; however, 
existing applications can be severely impacted by the improper use of 
the unicode bidi characters. Again, there are very good reasons behind 
the recommendation against using the formatting codes in markup.

> [snip]
> atom:name: When consuming atom:name, treat it as though it 
> was defined as a text construct, and allow for type="xhtml" 
> and type="html". When producing atom:name elements, BIDI markup 
> SHOULD be replaced with Unicode formatting characters when needed. 
> Ruby markup may be replaced with Unicode formatting characters or 
> stripped entirely. All markup should be stripped entirely, the 
> type attribute should be removed.
> 

As I pointed out in my previous note, this is not an interoperable 
suggestion and is not supported in any way by the specification.

- James

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 11:43:17 -0800

James M Snell wrote:
> With this, I think it is important to look at both the 
> original intent of the spec and what clients are most likely 
> to support.  While the schema is non-normative, the fact that 
> atom:name is clearly and specifically marked as "text" in the 
> schema makes the intent clear.  You
> *might* be able to stretch really far and claim that escaped 
> HTML is permitted, but if that was the intent, the spec would 
> more than likely say so.

I think "Atom allows foreign markup anywhere in an Atom document, except where it 
is explicitly forbidden" is explicit. I just tested Google Reader, IE7, RssBandit, FeedDeamon, and SharpReader. Of those, only SharpReader failed to strip out XHTML markup in an atom:name construct. I think changing atom:name to a text construct is something that could reasonably happen in a minor revision of RFC 4287, but I agree that currently it is not something that everybody should expect to work in every tool.

> If you want the Person construct to contain ruby text, the 
> best approach would be to define an extension element 
> designed for the purpose of providing a "display" version of 
> the name, e.g.
> 
> <author>
>    <name>name</name>
>    <x:display-name>
> This would allow existing clients to continue to operate as 
> they always have.

That seems reasonable. It also solves the problem of BIDI text in atom:name, and allows for using Microformats to distinguish between family names and given names.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 12:38:24 -0800

James M Snell wrote:
> Brian Smith wrote:
> > I agree. If there are going to be formatting codes in the document 
> > anyway, then why do we need a mechanism that duplicates their 
> > functionality?
> 

Even the latest W3C guidelines are now encouraging the use of some formatting characters (LRM and RLM) in addition to markup, where the previous guidelines recommended markup only:

So, Atom implementations have to be prepared to accept at least LRM and RLM in documents, anyway.

> > My hypothesis is that an implementation ignorant of BIDI issues is 
> > more likely to preserve the formatting characters than
> Atom/XHTML/HTML
> > BIDI markup, especially when the effects of those formatting 
> > characters never span multiple nodes in the document.
> 
> each tag separated by a comma.  The system splits the tags into an 
> array and saves each tag separately.
> Each tag becomes a separate atom:category element.  Is the user 
> responsible for adding the appropriate formatting codes around each 
> individual tag?  When the user wishes to edit the entry later, perhaps 
> to add a new entry, are they supposed to just know that there are 
> non-visual bidi formatting codes interspersed into the comma separated 
> list of tags?

When software breaks apart BIDI text and recombines it, it has to preserve the BIDI formatting. In this case, the system that splits apart the tags into an array and/or the system that recombines the tags into a comma-seperated list should transparently handle the formatting codes. 

> You're assuming that all users have the same requirements.  
> In our environment, a single feed may include entries from many 
> different users. We have group blogs where users from many different 
> locales have edit rights on any entry in the blog.  Further, our users 
> use many different editors to write and manage their blog entries.  
> Asking those users to be mindful of how they're using bidi formatting 
> characters is a lot more difficult than what we currently do, which is 
> provide a simple check box to indicate whether or not the entry is 
> "right-to-left", which in turn, is translated into the appropriate 
> dir="rtl" in the markup.
>
> since our software is written so that only explicit changes in value 
> are recognized (e.g. a missing dir attribute does not mean the dir 
> attribute value has changed) we're able to work seamlessly with 
> editors that do not support the attribute.

In my suggested mechanism, this concern is only relevent for language-sensitive attributes, and atom:name, since that is the only place where XHTML's BIDI markup cannot be used. Even then, it only applies to the few cases where there is a user that is editing part of an entry written in a RTL language, where the Unicode BIDI algorithm fails to work for that part of the entry, they are using software that doesn't support BIDI text entry (meaning they probably can't read or write any RTL languages), and their change somehow still requires the preservation of the RTL base directionality but doesn't require any other formatting codes. I agree that "extremely unlikely" isn't the same as "impossible," but in this case it seems pretty close.

> Yes, but with the marup we don't have to rely on users getting it 
> right when they type in the values.

I agree that we can't rely on users to manually enter formatting codes. But, the software can handle the formatting codes as transparently as it handles the "dir" attribute.

> > RFC 4287 and RFC 5023 [are] pretty unclear about what is required to 
> > be preserved.
> 
> RFC 5023, Section 9.3: To avoid unintentional loss of data when 
> editing Member Entries or Media Link Entries, an Atom Protocol client 
> SHOULD preserve all metadata that has not been intentionally modified, 

I will illustrate what I am saying with an example. An atom:link element is not unknown foreign markup. So, I can remove atom:link elements whenever I want. In particular, I can remove an atom:link element and then replace it with an another atom:link element that links to the same destination. That doesn't violate the specification but the old element might have a "dir" attribute and the new one might not.

> 9.3 uses the term "unknown foreign markup", which, if RFC
> 4287 defines as unknown elements AND attributes.

I don't see that anywhere in RFC 4287. Even if it was there, it would be very problematic to support that in all situations, and I would recommend against anybody depending on implementations to preserve unknown attributes on known elements, especially when the content of those known elements is being modified.

> > The category labels are almost always going to be composed of strong 
> > RTL and strong LTR characters (only), so the BIDI algorithm will 
> > work correctly. And, when it doesn't work, it is unlikely that it 
> > will be due to the wrong base directionality--in these cases, 
> > formatting characters are going to be needed no matter what. Right?
> 

Please give an example of a category label that requires RTL base directionality, fails to be rendered correctly using the Unicode BIDI algorithm, doesn't require any directionality modifiers other than the base directionality, which your software supports, and which you expect every Atom implementation to support. AFIACT, this would be a label that starts or ends with punctuation, or is a mixture of LTR and RTL text, with punctuation at the point where the directionality changes. 

> And no, the formatting characters are not always going to be needed.  
> If I am rendering the text in (x)html, I don't want the formatting 
> characters in there at all; rather, I want to follow best practices 
> and use the (x)html provided bidi markup mechanisms.

I agree 100%. A BIDI-enabled Atom-to-HTML converter should always convert Unicode formatting characters to markup, whenever possible.

> they currently are if the dir attribute is ignored or dropped; 
> however, existing applications can be severely impacted by the 
> improper use of the unicode bidi characters. Again, there are very 
> good reasons behind the recommendation against using the formatting 
> codes in markup.

Right, that is why my suggestion uses XHTML markup whenever possible, and fall back to formatting codes only as a last resort.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 12:54:23 -0800

James M Snell wrote:
> I wrote:
> > Atom/XHTML/HTML BIDI markup, especially when the effects of
> > those formatting characters never span multiple nodes in the
> > document.
> 
> Have you tested this hypothesis using real editors?

To clarify, I was thinking more of how aggregators and other non-interactive processors will treat markup vs. formatting codes. In my limited testing, I was able to get formatting codes to be preserved by AtomPub servers but I not always able to get the "dir" attribute preserved.

> >> Also, imagine a case where we have a feed with 100 
> >> entries, each with about 5 atom:category elements.
> >> Let's stay that the feed is generally all RTL. Using
> >> your approach, that's at least 1000 extra 
> >> characters in the feed, and 500 opportunities for the 
> >> embedding to be screwed up.
> > 
> > The category labels are almost always going to be composed 
> > of strong RTL and strong LTR characters (only), so the
> > BIDI algorithm will work correctly. And, when it doesn't
> > work, it is unlikely that it will be 
> > due to the wrong base directionality--in these cases, formatting 
> > characters are going to be needed no matter what. Right?
> 
> "almost always" is not "always".  The Atom bidi draft covers 
> the cases where "always" is more desirable than "almost always".

Here, I meant that using formatting codes would not result in 1000 extra characters, but almost no extra characters. Also, it is unlikely that errnoeously placed embedding characters will cause major problems since they are all localized to attribute values and atom:name in my suggested mechanism, and there is no cascading/inheritance of directionality that mechanism. In other words, any improper embedding will be limited and very localized.

- Brian

---

From: James M Snell <[email-masked]>
Date: Mon, 07 Jan 2008 11:28:15 -0800

> http://www.w3.org/TR/i18n-html-tech-bidi/#ri20030218.135304584

The guidelines state, very clearly, that the formatting characters 
should not be used when markup can be used instead.  The LRM and RLM 
characters do not have markup alternatives and do not carry the same 
risks as the other formatting characters.  This recommendation is very 
consistent, and is upheld by the Atom bidi draft, which provides a 
markup alternative to using the formatting characters.

Furthermore, the guidelines also state a best practice to "[u]se bidi 
markup only when necessary.", and gives the specific recommendation that 
  "[o]nce you have established the appropriate directionality for the 
html element you will only need to apply bidi markup to a block element 
if you want that element's directionality to be different."  By moving 
the dir as far up in the document tree as possible "simplif[ies] the 
document and reduce[s] bandwidth requirements."

> So, Atom implementations have to be prepared to accept at least LRM and RLM in documents, anyway.

Never said they didn't.  All the atom bidi draft does is provide a 
markup alternative in order to make life easier... which, reminds me, 
you still haven't explained why you think the dir attribute is more 
complicated to implement than the bidi characters.

> When software breaks apart BIDI text and recombines it, it has to preserve 
> the BIDI formatting. In this case, the system that splits apart the tags 
> into an array and/or the system that recombines the tags into a comma-seperated 
> list should transparently handle the formatting codes. 
> 

Let me ask again: is the user responsible for adding the appropriate 
formatting codes around each individual tag in the list?  When the user 
wishes to edit the entry later, are they supposed to just know that 
there are non-visual bidi formatting codes interspersed into the comma 
separated list of tags?

> In my suggested mechanism, this concern is only relevent for language-sensitive 
> attributes, and atom:name, since that is the only place where XHTML's BIDI 
> markup cannot be used. Even then, it only applies to the few cases where there 
> is a user that is editing part of an entry written in a RTL language, where 
> the Unicode BIDI algorithm fails to work for that part of the entry, they are 
> using software that doesn't support BIDI text entry (meaning they probably can't 
> read or write any RTL languages), and their change somehow still requires the 
> preservation of the RTL base directionality but doesn't require any other formatting 
> codes. I agree that "extremely unlikely" isn't the same as "impossible," but in 
> this case it seems pretty close.
> 

Your solution is based on a lot of assumptions and possibilities.  It 
*may* work in *many* cases.  It *likely* won't be a problem.  It's 
*possible* that clients will get it right.  It also goes against 
documented best practices and recommendations, defended only by an 
untested hypothesis that it's "simple" and "easy". That's not acceptable 
when we can do better.

The Atom bidi attribute is not the whole solution; it's just one part of 
the larger picture, intended to fill in the gaps and make certain things 
simpler.  Further, it is consistent with the approach taken by XHTML and 
the recommendations of both the W3C and the Unicode organization.

> I will illustrate what I am saying with an example. An atom:link element 
> is not unknown foreign markup. So, I can remove atom:link elements whenever 
> I want. In particular, I can remove an atom:link element and then replace 
> it with an another atom:link element that links to the same destination. 
> That doesn't violate the specification but the old element might have a 
> "dir" attribute and the new one might not.

Ok... and?  The meaning of the text in the spec is clear: clients should 
only change things they intentionally want changed.  If the atom:link 
element is replaced without the dir attribute, then obviously the only 
reasonable interpretation the server can make is that the client wanted 
that dir attribute to be changed.

- James

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 14:38:13 -0800

James M Snell wrote:
> Never said they didn't.  All the atom bidi draft does is 
> provide a markup alternative in order to make life easier... 
> which, reminds me, you still haven't explained why you think 
> the dir attribute is more complicated to implement than the 
> bidi characters.

* A processor still has to handle the formatting characters regardless of whether the document uses the Atom BIDI markup.

* The inheritance of directionality makes it more difficult to compose, decompose, and remix atom documents than if the directionality markup was localized near the text that needs it.

* In my testing, current Atom(Pub) implementations were more likely to strip the "dir" attribute than they were to strip Unicode formatting characters.

* My code already preserves BIDI directionality information encoded using the mechanism I suggested this thread, with no extra work on my part. Since the preservation of the formatting characters and (X)HTML BIDI markup required literally ZERO lines of code on my part, and support for the Atom BIDI draft requires more than zero lines of code, I find the Atom BIDI draft to be more difficult to implement. 

* The suggested mechanism doesn't all

> > When software breaks apart BIDI text and recombines it, it has to 
> > preserve the BIDI formatting. In this case, the system that splits 
> > apart the tags into an array and/or the system that recombines the 
> > tags into a comma-seperated list should transparently 
> > handle the formatting codes.
> > 
> 
> formatting codes around each individual tag in the list?  

Not if you don't already need to using your current implementation. Let's say the uesr has a list "abc, FED, ghi" with no control characters and checks the RTL checkbox. You break it into the three parts and store it in an array or database or wherever, along with the checkbox value. When you want to regenerate the list, you take all the entries and join then with ", ", you set the text box's directionality using the checkbox value, and then you put the regenerated list back in the text box--again, no formatting codes. When you generate your atom document, you don't need a "dir" attribute or unicode control characters for the @label value, because the BIDI algorithm will handle everything correctly.

That is assuming that the BIDI algorithm can already handle the list of labels without hints. If you send me some examples of labels that are not BIDI-algorithm-friendly (preferably ones that that your implementation supports), I will explain when the control characters are necessary. 

> When the user wishes to edit the entry later, are they supposed
> to just know that there are non-visual bidi formatting codes

No, I am not recommending that. The only case where that would be necessary is when you need to have nested embedding levels in your list of tags. That seems to be something that your implementation doesn't support anyway, based on your description.

> untested hypothesis that it's "simple" and "easy". That's not 
> acceptable when we can do better.

My main assumption is that very few implementations will implement the Atom BIDI mechanism properly or consistently. Consequently, I want to focus on a solution that can be "accidently" implemented correctly, as much as possible.

I guess I am just playing the "worse is better" card. I've asked you multiple times to supply some concrete examples where my idea breaks down (for BIDI IRI templates and for BIDI in Atom). Those requests were not rhetorical--having those examples really makes everything easier to understand for people like me, who aren't experts in BIDI text handling. Do you have links to real feeds that makes good use of the Atom BIDI draft features, or some other real-world examples, so we can study them?

> The Atom bidi attribute is not the whole solution; it's just 
> one part of the larger picture, intended to fill in the gaps
> and make certain things simpler. Further, it is consistent
> with the approach taken by XHTML and the recommendations of
> both the W3C and the Unicode organization.

I never doubted that. The only thing I doubt is that it is better or simpler to have a seperate mechanism for BIDI than the mechanism for Ruby text or other i18n problems with RFC 4287. 

I really liked your idea of a <x:display-name> element to solve the problem with Ruby text in <atom:name>. That same solution also be applied to atom:link/@title and atom:category/@label to allow all language-sensitive constructs in RFC 4287 to be compliant with the "Unicode in XML" guidelines. And, coincidently, it would also work for BIDI text and probably even Gaiji (which I didn't even know about until Murata-san mentioned it). I really think that the "use XHTML text constructs for i18n" idea is easy to understand and relatively simple to implement, particularly since the cascading/inheriting of directionality overrides becomes very localized.

> Ok... and?  The meaning of the text in the spec is clear: 
> clients should only change things they intentionally want
> changed. If the atom:link element is replaced without the
> dir attribute, then obviously the only reasonable
> interpretation the server can make is that the client
> wanted that dir attribute to be changed.

I am not talking about clients stripping the directionality information as much as servers stripping it. For example, in my AtomPub implementation, the server ignores any changes to "edit" and "edit-media" links, and any changes to atom:content for media link entries, including all attributes. I had difficulty with getting other AtomPub servers to preserve the "dir" attribute except on XHTML elements. It is my understanding that every server can strip off the "dir" attribute if it wants to. Do you know of any servers that preserve it?

Regards,
Brian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Jan 2008 15:22:21 -0800

James M Snell wrote:
> The guidelines state, very clearly, that the formatting 
> characters should not be used when markup can be used 
> instead.  The LRM and RLM characters do not have markup 
> alternatives and do not carry the same risks as the other 
> formatting characters.  This recommendation is very 
> consistent, and is upheld by the Atom bidi draft, which 
> provides a markup alternative to using the formatting characters.
> 
> Furthermore, the guidelines also state a best practice to 
> "[u]se bidi markup only when necessary.", and gives the 

How about a compromise? You don't like formatting codes, but they can't completely go away. I don't like the inheritance/cascading of directionality between atom elements because I think it is easy to get wrong. How about this:

* Don't allow @dir on atom:feed. This makes it easier to copy entries to/from a feed without requiring rewriting. It also makes digital signatures on entries easier to deal with.

* Don't allow @dir on atom:entry. Again, this reduces the likelihood that a processor will screw up the inheritance of directionality overrides. It also means that different users can more easily edit different pieces of metadata without clobbering each other's directionality overrides.

* Don't allow @dir on atom:content or text constructs. XHTML should be used instead, and the @dir should appear on the xhtml:div element of the text construct. This eliminates the other major case where a process will screw of the inheritance of directionality overrides, and increases the likelihood that processors will preserve the directionality information since it will be in the XHTML and not in the Atom markup.

* When any Unicode formatting characters are encountered, they must override the directionality in @dir. However, using formatting characters is "NOT RECOMMENDED." 

* Future language-sensitive extensions to Atom are strongly "RECOMMENDED" to use text constructs for all language-sensitive text.

* Punt on Ruby text for now. 

>   "[o]nce you have established the appropriate directionality 
> for the html element you will only need to apply bidi markup 
> to a block element if you want that element's directionality 
> to be different."  By moving the dir as far up in the 
> document tree as possible "simplif[ies] the document and 
> reduce[s] bandwidth requirements."

This recommendation is good for HTML because a HMTL document is usually not processed as a collection of individual parts, but instead as an indivisible document. However, it is counter-productive for Atom because Atom feeds ARE collections of individual documents.

- Brian

---

From: "David Powell" <[email-masked]>
Date: Mon, 7 Jan 2008 22:32:37 -0000

> should not be used when markup can be used instead.

But markup cannot be used inside XML attributes.

Read "Best Practice 11".  It describes our situation exactly:
<http://www.w3.org/International/geo/html-tech/tech-bidi.html#ri20030728.092841697>

  [...]
  On the other hand, attribute text and element text
  that allows no internal markup, ie. the title,
  textarea and option elements, cannot support use
  of dir on a span or other element to label part of
  its content.

  In these cases you can use Unicode characters to do
  the same job. The following table shows correspondences
  between markup and Unicode control codes:
  
  [...]
  RLE LRE RLO LRO PDF
  [...]
 
  The joint Unicode Technical Report #20 and W3C Note,
  Unicode in XML and other Markup Languages goes further.
  It explicitly recommends that only the markup be used
  [...]
  Of course, in attribute values or for the three elements
  listed above markup cannot be used, so the Unicode control
  characters are the only option available.

Do you disagree with these findings?

Do you have a better way to implement bidi embedding in attribute values or PCDATA elements?  The atom-bidi draft does not support bidi embedding.

If you are representing bidi text in a scope that supports internal markup you can either use Unicode or mixed content markup; and mixed content markup is the prefered choice.

If you are representing bidi text in a scope that does not support internal markup - which is what we are discussing here - then Unicode is the only choice.

-- 
Dave

---

From: James M Snell <[email-masked]>
Date: Mon, 07 Jan 2008 14:47:04 -0800

Let me say it again: the atom-bidi draft does not replace the use of the 
formatting characters.  There are cases where the formatting characters 
will still be needed.  I never said using the unicode characters was not 
an option; just that they weren't recommended. The *only* thing the 
atom-bidi draft does is provide a way of specifying the default 
direction of directionally neutral text.  It does not attempt to replace 
the functionality of the bidi formatting characters. The bidi draft does 
provide a markup alternative to using the bidi formatting codes when all 
that is needed is the ability to specify a default direction for all the 
text in the feed, entry, etc.  If you need anything more complicated 
than that, then you fall back to the unicode characters, which is not 
recommended but is still an option.

- James