Subject: Back on type=HTML, going a step farther...

From: Thomas Broyer <[email-masked]>
Date: Sun, 06 Mar 2005 22:01:46 +0100

Hi there,

First, let's introduce myself: I'm a 24 year-old French programmer 
living in Dijon, France. Some day, I'd have liked to create my blog but 
got stuck by all these RSS thingies and API stuff... I'm back now to 
syndication as a user and programmer, and found Atom as the new unified 
standard for syndication and web service API.

Reading the Atom spec (draft -05), one thing shocked me: what is this 
type=HTML for?

Why would I need to use HTML in my feed? Well, to add some structure to 
my entry content, such as headers, divisions, lists, tables, text 
emphasis, etc. Sometime, I'll also need to add formatting and visual 
effects, but this shouldn't happen often: the goal of a feed is to be 
read outside my website, either in a Gtk/Win32/etc. app or integrated in 
another website. Thus formatting and visual effects should be reduced to 
the strict minimum for my feed to comply to the feed reader appearance.

How would I insert HTML inside my feed? Err, an Atom feed (as well as an 
RSS feed) is an XML document, HTML has been rewritten in an XML syntax 
with XHTML, so I'll obviously use XHTML!

So why is Atom using that escaped-HTML thing? Where does it come from? 
It comes from RSS0.92 and RSS2.0, which have relaxed there syntax to 
reflect a "common use" by bloggers. This use comes from the fact feeds 
were first mainly integrated in websites, someone might have wonder how 
to add formatting and/or structure and found that trick that worked for 
the feed readers existing at that time.
Why does Atom use it? I think (correct me if I'm wrong) mainly to be 
adopted more easily by the existing CMS.

Why I think this is a bad practice?
1. If I want to program an Atom reader/aggregator, I'll need to deal 
with the HTML tag soup, which may not be wellformed (and wellformedness 
in SGML is something so light...) nor valid, which mean I'll also need 
to re-implement the error handling algorithms already used in existing 
browsers. This should already have discouraged me to start programming, 
but I'm really motivated and will try to find a solution... (using Tidy 
to "correct" the HTML and transform it into XHTML that I can parse ; not 
that this means using another parser than the one I use to process the 
Atom feed/entry). Depending on the language/environment I have chosen, 
this might not even be possible: how do you use such thing in an XSLT 
transformation or other XForms app?
If the Atom WG wants an interoperable format adopted by everyone, I'm 
not sure this is the good way...
2. If I want to integrate the HTML content in a website, my page 
structure and formatting could be totally broken if the HTML content of 
the feed is not wellformed or valid HTML: an unclosed element or too 
many closing tags, an un-wellformed tag that could "hide" my following 
tags, particularly if I don't use wellformed or valid HTML: tag soup 
inside tag soup can't always have a good taste... To prevent that sort 
of thing, the Atom spec would have to state that the escaped HTML 
content MUST be wellformed and valid HTML (wellformedness in SGML in not 
as strong as in XML and still might lead to weird stuff). However, even 
if the spec add such a rule, you can be sure most of the feeds won't 
follow it, simply because "it looks OK in the major readers/aggregators" 
(hey, remember the browser war which finally leave NS4 away?). It's much 
much easy to add welformedness constraints on XHTML which integrates 
well in XML, as this means the whole feed wouldn't be wellformed and 
couldn't be processed at all, than on "external content" like escaped 
HTML, even if they are encapsulated in the XML document. With schemas, 
it's even possible to add a validity constraint!

What disturbs me in the spec is that the content of a Text Construct is 
considered "external data" (call it foreign data or foreign object if 
you want).
In my opinion, the content of a Text Construct should be XML. The spec 
should add more constraints apart from *suggesting* not including 
metadata in the content and/or using XHTML to add structure and/or 
formatting, rather than DocBook, XSL-FO, SVG or else.
If there is a need to allow inlining of external content, such as 
images, there should be a special element to use as a container inside 
the Text Construct, similar to the foreignObject of SVG or the 
instream-foreign-object of XSL-FO.
This also means the @type and @mode attributes are discarded from the 
Text Construct element and added to the "Foreign Object Element" (let's 
rename them @type and @encoding, with @type being MIME media types (no 
more TEXT, HTML or XHTML special value).

Someone the list suggested stripping the tags from the escaped-HTML 
content if the reader doesn't support HTML processing ; I would suggest 
using HTML Tidy to turn it into XHTML (if the read supports XHTML). 
Can't this be done by the feed generator instead?
It's easy for blogging tools to produce XHTML from their WYSIWYG 
"richtext areas" to comply with an "XML-only Atom web service API"; it 
shouldn't be too hard for CMS tools to produce XHTML or plain text from 
legacy HTML content (either integrating HTML Tidy or a similar tool or 
stripping the HTML tags); it shouldn't be too hard either for CMS tools 
to integrate "migration tools" to transform legacy HTML content to XHTML 
(if the user wants too, otherwise the tool would produce plain text 
content in Atom feeds, for example); and it WILL be easy to program Atom 
readers/aggregators, because they won't have to deal with the HTML tag soup.

The Atom format aims to be:
  - an XML application, so it should, in my opinion, use XHTML to add 
structure or formatting to the content, instead of HTML
  - "implemented by everybody" [1], so it should be easy to parse (hence 
being XML only, and therefore preventing the need for a second 
processing) and produce (when using the Atom API for editing, input will 
be the same as output; when dealing with legacy HTML content or using an 
input tool not producing XHTML, migration algorithm or tools can be used)
  - computer-friendly [2]: HTML is not computer-friendly, or at least 
far less than XHTML...

If someone wants me to provide a clear "spec" for what I've told above, 
so that we can discuss it (rather than trolling), I'll create a Pace* on 
the wiki.

[1] http://www.intertwingly.net/wiki/pie/RoadMap
[2] http://www.intertwingly.net/wiki/pie/WhatIsAtomFor

-- 
Thomas Broyer

---

From: Julian Reschke <[email-masked]>
Date: Sun, 06 Mar 2005 22:52:49 +0100

Thomas Broyer wrote:
 > ...

I agree with what you say, but I fear this topic has been discussed and 
dicussed again in the past with no result. For some reason, people seem 
to prefer a format where "bad" data (tag soup) is allowed, and the task 
to fix it is put onto the recipients rather than the creators. For me 
it's very obvious that this is the wrong approach -- if you can require 
the recipients to do that, you can also require the same responsibility 
from the producers.

Anyway, I recently made a much more modest request that the spec should 
state that XHTML is preferred over HTML (because many recipients will 
not be willing to process tag soup, so in the best case formatting is 
lost). It seems that we can't even get a consensus for that, which is 
really disappointing considering the expectations I had 18 months ago :-(

Best regards, Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Graham <[email-masked]>
Date: Sun, 6 Mar 2005 21:49:41 +0000

I totally agree that publishers are all willing to convert their 
content to well-formed XHTML for us. They're very generous people. Do 
you not remember last week when we all got sent free lollipops?

Graham

On Mar 6, 2005, at 9:01 pm, Thomas Broyer wrote:

> living in Dijon, France. Some day, I'd have liked to create my blog 
> but got stuck by all these RSS thingies and API stuff... I'm back now 
> to syndication as a user and programmer, and found Atom as the new 
> unified standard for syndication and web service API.
>
> Why would I need to use HTML in my feed? Well, to add some structure 
> to my entry content, such as headers, divisions, lists, tables, text 
> read outside my website, either in a Gtk/Win32/etc. app or integrated 
> in another website. Thus formatting and visual effects should be 
> reduced to the strict minimum for my feed to comply to the feed reader 
> appearance.
>
> How would I insert HTML inside my feed? Err, an Atom feed (as well as 
> an RSS feed) is an XML document, HTML has been rewritten in an XML 
> syntax with XHTML, so I'll obviously use XHTML!
>
> were first mainly integrated in websites, someone might have wonder 
> how to add formatting and/or structure and found that trick that 
> worked for the feed readers existing at that time.
> with the HTML tag soup, which may not be wellformed (and 
> wellformedness in SGML is something so light...) nor valid, which mean 
> I'll also need to re-implement the error handling algorithms already 
> used in existing browsers. This should already have discouraged me to 
> start programming, but I'm really motivated and will try to find a 
> solution... (using Tidy to "correct" the HTML and transform it into 
> XHTML that I can parse ; not that this means using another parser than 
> the one I use to process the Atom feed/entry). Depending on the 
> language/environment I have chosen, this might not even be possible: 
> how do you use such thing in an XSLT transformation or other XForms 
> app?
> structure and formatting could be totally broken if the HTML content 
> of the feed is not wellformed or valid HTML: an unclosed element or 
> too many closing tags, an un-wellformed tag that could "hide" my 
> following tags, particularly if I don't use wellformed or valid HTML: 
> tag soup inside tag soup can't always have a good taste... To prevent 
> that sort of thing, the Atom spec would have to state that the escaped 
> HTML content MUST be wellformed and valid HTML (wellformedness in SGML 
> in not as strong as in XML and still might lead to weird stuff). 
> However, even if the spec add such a rule, you can be sure most of the 
> feeds won't follow it, simply because "it looks OK in the major 
> readers/aggregators" (hey, remember the browser war which finally 
> leave NS4 away?). It's much much easy to add welformedness constraints 
> on XHTML which integrates well in XML, as this means the whole feed 
> wouldn't be wellformed and couldn't be processed at all, than on 
> "external content" like escaped HTML, even if they are encapsulated in 
> the XML document. With schemas, it's even possible to add a validity 
> constraint!
>
> What disturbs me in the spec is that the content of a Text Construct 
> is considered "external data" (call it foreign data or foreign object 
> if you want).
> Text Construct element and added to the "Foreign Object Element" 
> (let's rename them @type and @encoding, with @type being MIME media 
> types (no more TEXT, HTML or XHTML special value).
>
> content if the reader doesn't support HTML processing ; I would 
> suggest using HTML Tidy to turn it into XHTML (if the read supports 
> XHTML). Can't this be done by the feed generator instead?
> shouldn't be too hard for CMS tools to produce XHTML or plain text 
> from legacy HTML content (either integrating HTML Tidy or a similar 
> tool or stripping the HTML tags); it shouldn't be too hard either for 
> CMS tools to integrate "migration tools" to transform legacy HTML 
> content to XHTML (if the user wants too, otherwise the tool would 
> produce plain text content in Atom feeds, for example); and it WILL be 
> easy to program Atom readers/aggregators, because they won't have to 
> deal with the HTML tag soup.
>
>  - "implemented by everybody" [1], so it should be easy to parse 
> (hence being XML only, and therefore preventing the need for a second 
> processing) and produce (when using the Atom API for editing, input 
> will be the same as output; when dealing with legacy HTML content or 
> using an input tool not producing XHTML, migration algorithm or tools 
> can be used)
> If someone wants me to provide a clear "spec" for what I've told 
> above, so that we can discuss it (rather than trolling), I'll create a 
> Pace* on the wiki.
>

---

From: Thomas Broyer <[email-masked]>
Date: Mon, 07 Mar 2005 00:20:27 +0100

Graham wrote :
> I totally agree that publishers are all willing to convert their content 
> to well-formed XHTML for us.

That's not the only solution I (and probably others before me) proposed.
Let's see below with some examples:

First, content can be "converted" to plain text without formatting (in 
the case the HTML markup serves only formatting needs, not structural ones):
<atom:entry>
<atom:id>tag:example.com,2005-03-06:PlainTextWithAlternateHTML</atom:id>
<atom:updated>2005-03-06T21:38Z</atom:updated>
<atom:title>Plain text in Atom linking to an HTML version</atom:title>
<atom:link rel="alternate" type="text/html"
       href="http://example.com/2005/03/06#PlainTextWithAlternateHTML" />
<atom:content>
The generator (not the person who publishes but the program which 
creates the feed) converts the entry content to plain text and links to 
the original HTML version.
</atom:content>
</atom:entry>

>> formatting, rather than DocBook, XSL-FO, SVG or else.

Sorry I would have written "the spec shouldN'T add more constraints 
apart from..."

>> types (no more TEXT, HTML or XHTML special value).

HTML can also be included as a "foreign object":
<atom:entry>
<atom:id>tag:example.com,2005-03-06:HTMLAsForeignObject</atom:id>
<atom:updated>2005-03-06T21:38Z</atom:updated>
<atom:title>HTML used as a foreign object</atom:title>
<atom:content>
<atom:foreignObject type="text/html">
&lt;p>The generator (not the person who publishes but the program which 
creates the feed) converts the entry content to plain text in an 
&lt;code>atom:fallback&lt;/code> element and inserts the original 
&lt;acronym title="HyperText Markup Language">HTML&lt;acronym> version 
in an &lt;code>atom:foreignObject&lt;/code> element.&lt;/p>
&lt;p>If no &lt;code>atom:fallback&lt;/code> is given, the entry 
&lt;em>must&lt;em> provide a &lt;code>link rel="alternate"&lt;/code>.
</atom:foreignObject>
<atom:fallback>
The generator (not the person who publishes but the program which 
creates the feed) converts the entry content to plain text in an 
atom:fallback element and inserts the original HTML version in an 
atom:foreignObject element.
If no atom:fallback is given, the entry MUST provide a link rel="alternate".
</atom:fallback>
</atom:content>
</atom:entry>

Using a foreignObject for HTML, the spec would not require (or strongly 
suggests (think about the SHOULD word in the spec)) readers/aggregators 
to support HTML.

As you can see, dealing with legacy HTML when producing an Atom feed is 
not an issue. Dealing with HTML when consuming a feed IS.

By allowing HTML inside a feed, you orient readers/aggregators 
developers to use an "HTML area" (integrated web browser) to render the 
entry content where they could use another "rich text area" (a "native 
Win32" rich text area, an SVG-based canvas, a "direct drawing" canvas) 
or convert the XHTML content into plain text (stripping "tags" is part 
of XML parsing: just parse the subtree without paying attention to 
elements, just keeping their text content; you could even skip SCRIPT 
and STYLE element content) or a "widget based" rendering.
Embedding a web browser widget is not always possible: how will you 
render HTML text in the title of a window (if, for example, a reader 
gives you the ability to open an entry in a new window, same as what you 
can do when reading Usenet news groups), or in a "bubble tooltip" like 
those seen on Windows 20 or WindowsXP?

>> tool or stripping the HTML tags);

CMS tools will need an update to add an Atom output, this update may 
integrate some processing to transform HTML into plain text or XHTML; or 
it can just produce an atom:foreignObject and an atom:link 
rel="alternate", with no effort at all!

>> and it WILL be easy to program Atom readers/aggregators, because they
 >> won't have to deal with the HTML tag soup.

...although I'm sure most readers/aggregators will use an embedded web 
browser to render then entry.
BUT, this will not be enforced by the spec, and that's the point!

It seems people here want to reinvent RSS with its tricks and drawbacks 
in the name of backwards compatibility instead of trying to produce a 
well-designed thing which also meets the backwards compatibility issue 
but in a clean approach!

-- 
Thomas Broyer

---

From: Tim Bray <[email-masked]>
Date: Sun, 06 Mar 2005 15:36:39 -0800

On Mar 6, 2005, at 1:52 PM, Julian Reschke wrote:

> I agree with what you say, but I fear this topic has been discussed 
> and dicussed again in the past with no result. For some reason, people 
> seem to prefer a format where "bad" data (tag soup) is allowed, and 
> the task to fix it is put onto the recipients rather than the 
> creators. For me it's very obvious that this is the wrong approach -- 
> if you can require the recipients to do that, you can also require the 
> same responsibility from the producers.

I agree also, but for some implementors, HTML is actually easier, if 
they are handing a chunk of bytes to an HTML rendering control, they'd 
rather not reconstruct the syntax from the infoset, they'd rather just 
take an opaque chunk of bytes.  So we really don't have community 
consensus in favor of XHTML, even though it's obviously cleaner. -Tim

---

From: Robert Sayre <[email-masked]>
Date: Sun, 06 Mar 2005 19:50:37 -0800

Tim Bray wrote:

>> and dicussed again in the past with no result. For some reason, 
>> people seem to prefer a format where "bad" data (tag soup) is 
>> allowed, and the task to fix it is put onto the recipients rather 
>> than the creators. For me it's very obvious that this is the wrong 
>> approach -- if you can require the recipients to do that, you can 
>> also require the same responsibility from the producers.
>
>
> consensus in favor of XHTML, even though it's obviously cleaner.

Agree w/ Tim. Also, HTML4 enjoys far better rendering support than XHTML 
does.

Robert Sayre

---

From: Julian Reschke <[email-masked]>
Date: Mon, 07 Mar 2005 08:23:01 +0100

> Agree w/ Tim. Also, HTML4 enjoys far better rendering support than XHTML 

That's true (in IE), but that's just a serialization problem. Nobody 
said that recipients should emit that XHTML directly to HTML controls.

Best regards, Julian

---

From: Martin Duerst <[email-masked]>
Date: Tue, 08 Mar 2005 03:31:25 +0900

At 08:36 05/03/07, Tim Bray wrote:

 >I agree also, but for some implementors, HTML is actually easier, if they 
are handing a chunk of bytes to an HTML rendering control, they'd rather 
not reconstruct the syntax from the infoset, they'd rather just take an 
opaque chunk of bytes.  So we really don't have community consensus in 
favor of XHTML, even though it's obviously cleaner. -Tim

I really hope they don't take an opaque chunk of bytes, because if
they do, they'll have no clue about the character encoding.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 08 Mar 2005 03:26:00 +0900

>Anyway, I recently made a much more modest request that the spec should 
state that XHTML is preferred over HTML (because many recipients will not 
be willing to process tag soup, so in the best case formatting is lost). It 
seems that we can't even get a consensus for that, which is really 
disappointing considering the expectations I had 18 months ago :-(

I made an even more modest request, namely to put XHTML ahead of HTML
in the spec. I still hope we can at least get consensus on that.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 07 Mar 2005 21:37:32 +0100

Tim Bray wrote:
> On Mar 7, 2005, at 10:31 AM, Martin Duerst wrote:
> 
>> >for some implementors, HTML is actually easier, if they are handing a 
>> chunk of bytes to an HTML rendering control, they'd rather not 
>> reconstruct the syntax from the infoset, they'd rather just take an 
>> opaque chunk of bytes.
>>
> Not true; if you can parse the XML to find the <atom:content> then you 
> know the character encoding and can tell your HTML renderer. -Tim

I think Martin was talking about the producer: if all he has is a chunk 
of *bytes*, there's no reliable way top put that into an HTML-typed text 
construct (you'll need characters, not bytes).

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Mon, 07 Mar 2005 12:31:33 -0800

---

From: Martin Duerst <[email-masked]>
Date: Tue, 08 Mar 2005 11:31:16 +0900

>>> >for some implementors, HTML is actually easier, if they are handing a 
chunk of bytes to an HTML rendering control, they'd rather not reconstruct 
the syntax from the infoset, they'd rather just take an opaque chunk of bytes.

It works both ways. If all you get is a chunk of bytes, things won't
work. Of course the Atom processor knows more, but Tim didn't say
that it would pass that info on to the renderer.

Regards,   Martin.

---

From: Thomas Broyer <[email-masked]>
Date: Thu, 10 Mar 2005 00:03:52 +0100

> the recipients to do that, you can also require the same responsibility 

I'm afraid it is a "business" problem more than a technical one...

Say you require publishing software to transform the content (either 
transform HTML to XHTML or strip tags, which not always acceptable (you 
should replace an IMG element by it's ALT attribute's value, tables and 
lists may produce non-understandable text when cells/items are 
concatenated, etc.)), they most likely won't do it. Either they'll 
produce not-wellformed Atom feeds, or they "just" won't implement Atom 
and keep their RSS feeds.
In the same time, readers/aggregators will most likely read Atom feeds 
(as it will be easier, particularly if they don't use an embedded web 
browser component) in _addition_ to RSS feeds (for which they already 
deal with multiple versions and extension modules).
There most likely won't exist Atom-only readers/aggregators, so if 
publishing software already produce RSS (in any widely supported 
flavor), they won't produce Atom.

That's why Atom feeds must be easy to produce, and particularly from 
legacy HTML4 content.

> lost).

Data may also be lost, for example if IMG are not replaced by their ALT 
attribute's value.
Content may become hard to read, for example if the content contains a 
table or a list. For lists, it could be leveraged putting commas at the 
end of each item: <ul><li>first item,</li><li>second item</li></ul>.
And extra non-content text may appear, for example if the ALT attribute 
of an IMG is not used as expected or if the content contains SCRIPT or 
STYLE elements. Moreover, HTML4 has defined a NOSCRIPT element (there is 
no NOSTYLE element, because STYLE must appear in the HEAD in a valid 
HTML4 document).

 > It seems that we can't even get a consensus for that, which is

However, Atom is not only a feed syntax, it's also a publishing/editing 
protocol. This means publishing software implementing the Atom protocol 
will have to deal with this escaped-HTML too.
And even if most publishing software treats HTML "just" as some "opaque 
text" that's eaten and regurgitated "as is", some weblogs aim to produce 
XHTML (b2evolution or WordPress), thus they will need to deal with 
escaped-HTML and transform it to XHTML.
That's why a consensus is hard to find...

I'll try to find an even-better solution than the one I proposed in my 
answer to Graham and post it here tomorrow (it's nearly midnight in 
France). Some solution that does not seem to bring back the 
multipart/alternative type...

In a word, I'm deeply convinced that an all-XML syntax (i.e. including 
XHTML) is THE good way, but I must admit we must provide solutions to 
integrate escaped-HTML if we want Atom to be widely adopted (even RSS 
1.0, and now RSS 1.1, have been extended to allow escaped-HTML...)

-- 
Thomas Broyer

---

From: "Thomas Broyer" <[email-masked]>
Date: Thu, 10 Mar 2005 08:58:07 +0100 (CET)

> escaped-HTML and transform it to XHTML.

I forgot to talk about publishing software that aim to produce not even
HTML/RSS/Atom but also a PDF or RTF version or something else... They'll
also need to parse escaped-HTML while it would have been so easier to just
use XHTML (as easy as applying an XSLT stylesheet and rendering through
Apache FOP for example).

There are many other cases where escaped-HTML will be hard to deal with,
at least harder than processing an XHTML document fragment, just because
the receiving software don't treat Text Constructs and atom:content
content as markup but only as "opaque text".
And this is NOT interoperable!

One thing we could do is requiring Atom to contain plain text or XHTML
only when using the Atom protocol, whatever the receiving software do with
it, whether it just takes the "innerXml" (Microsoft's way) of the Text
Constructs and atom:content elements and treat it as "opaque text", or
parse the XHTML to transform or validate it in any way.
Escaped-HTML would be preserved in the Atom syntax only for "backwards"
compatibility with RSS, thus only when "publishing" a feed.
Or we could state type=HTML is not required to be supported by a user
agent for it to be Atom-conformant. We would then need a way for user
agents to tell "servers" (would it be an Atom composing/publishing
software or a server publising a feed) whether or not they support
type=HTML, and that would add a bit complexity. Or we would have to state
that whenever escaped-HTML is provided, an alternate XHTML or plain text
version is required (except for atom:content, which can be replaced by the
atom:link when rendering an entry).
Or we could just nuke the type=HTML and do it the RSS0.90-way, allowing
escaped-HTML in a type=TEXT, but not requiring to be supported by the user
agent (not even stripping tags).
Or...

-- 
Thomas Broyer