Subject: Re: Atom Export

From: Alastair Rankine <[email-masked]>
Date: Mon, 2 Oct 2006 15:53:14 +1000

Thanks for the initial feedback.

It is a common dilemma for spec writers to decide when to stop. In  
this case: should the spec include non-atom resources or not?

I initially thought not because it didn't seem to fit easily but  
perhaps I was too hasty. There are some potential users of this whose  
important content is almost exclusively in binary format, such as  
podcasters. On reflection it seems an unwise exclusion.

So I haven't thought about this much either, but a couple of options  
present themselves. As you say below, zipfiles may be a good  
solution. There is also the XOP format (http://www.w3.org/TR/xop10/)  
which looks pretty suitable also. Of these two I lean towards zip,  
but I would have to think about it some more.

Any other suggestions or comments?

On 28/09/2006, at 6:38 AM, James M Snell wrote:

>
> I haven't done much deep thinking about this proposal yet, but the one
> thing that concerns me are the non-atom resources that go along  
> with the
> entries.  When I export from a blog, I want to be able to export
> everything in a single operation.  I've been stewing lately over the
> possibility of doing something similar to this but wrapping the atom
> documents and associated resources into a zip.  A single master atom
> feed would provide an index of the archive, with individual entries
> either contained in that index feed or as separate entry documents
> (modeled loosely after the distinction app makes between the  
> collection
> feed and individual editable entries.
>
> - James
>
> Bill de hOra wrote:
>>
>> Alastair Rankine wrote:
>>>
>>> Hello Atom folks,
>>>
>>> Here is a proposal for an Atom-derived format for exporting of  
>>> content.
>>>
>>> The problem I'm trying to solve is that of migration from one
>>> authoring system (eg blogging engine) to another. Atom is highly
>>> suited to this task.
>>>
>>> The full problem statement, and the reasons for basing the  
>>> solution on
>>> Atom, are all described in the proposal published here:
>>>
>>> http://girtby.net/offerings/atomexport
>>>
>>> There is no implementation yet.
>>>
>>> I look forward to your comments on this document.
>>
>>
>> Very quickly (I'm overloaded atm) - this is excellent.
>>
>> I've been looking at using Atom Entries for roundtripping/backup and
>> interchange, especially in CMSes, for some time. The ultimate dump
>> format in terms of expressiveness is RDF/XML, but it can be  
>> complicated.
>> Atom offers a lot of value with comparative simplicity largely  
>> because
>> atom:id can survive across systems*, but also because the flat  
>> format of
>> entries maps well onto relational data (it's particularly good for
>> dumping metadata). One hard problem is in usefully storing and  
>> indexing
>> received foreign markup.
>>
>> cheers
>> Bill
>>
>> * that said, for "consumer" data, the APP leans towards, or least
>> allows, rewriting of atom:ids due to trust issues.
>>
>>

---

From: James M Snell <[email-masked]>
Date: Sun, 01 Oct 2006 23:07:22 -0700

Using XOP is definitely an option, but it's one that is likely to be far
less efficient and likely quite a bit more complicated than a zip.
Using the zip gives you quite a few options for exporting multiple
collections, with associated resources, templates, configuration
settings, plugins, whatever.  Exporting the media resources is, at the
very least, a very important requirement for the types of cases we'd be
looking at to use this.

- James

Alastair Rankine wrote:
> It is a common dilemma for spec writers to decide when to stop. In this
> case: should the spec include non-atom resources or not?
> 
> I initially thought not because it didn't seem to fit easily but perhaps
> I was too hasty. There are some potential users of this whose important
> content is almost exclusively in binary format, such as podcasters. On
> reflection it seems an unwise exclusion.
> 
> present themselves. As you say below, zipfiles may be a good solution.
> There is also the XOP format (http://www.w3.org/TR/xop10/) which looks
> pretty suitable also. Of these two I lean towards zip, but I would have
> to think about it some more.
> 
>> thing that concerns me are the non-atom resources that go along with the
>> (modeled loosely after the distinction app makes between the collection
>>>> Here is a proposal for an Atom-derived format for exporting of content.
>>>>
>>>> The full problem statement, and the reasons for basing the solution on
>>> format in terms of expressiveness is RDF/XML, but it can be complicated.
>>> Atom offers a lot of value with comparative simplicity largely because
>>> atom:id can survive across systems*, but also because the flat format of
>>> dumping metadata). One hard problem is in usefully storing and indexing
>>> received foreign markup.
>>>
>>> cheers
>>> Bill
>>>

---

From: "A. Pagaltzis" <[email-masked]>
Date: Mon, 2 Oct 2006 10:12:41 +0200

* James M Snell <[email-masked]> [2006-09-29 14:17]:
> I've been stewing lately over the possibility of doing
> something similar to this but wrapping the atom documents and
> associated resources into a zip. A single master atom feed
> would provide an index of the archive, with individual entries

Why the zip? Atom is an envelope format that can transport binary
content. So is tar. Just gzip the Atom feed; instead of .tar.gz
(which usually compresses better than zips) you get .atom.gz.

Hmm. Maybe we should call these Atomballs?

Buckminsterfullerenes? :-)

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Elliotte Harold <[email-masked]>
Date: Mon, 02 Oct 2006 10:21:25 -0400

> very least, a very important requirement for the types of cases we'd be

XOP, if I understand it correctly, could in fact be packaged in a zip 
file, though no such packaging is described in the spec. If you went 
with a non-XOP zip as opposed to XOP, I suspect you'd probably end up 
reinventing a lot of XOP along the way. If you could get away with 20% 
of XOP, then it might be worthwhile to do something custom. Otherwise, 
I'd just go with XOP.

Another possibility: OpenOffice does something very much like this 
(zipped XML + media). Does anyone know if that format is sufficiently 
generic to handle this use case?

-- 
Elliotte Rusty Harold  [email-masked]
Java I/O 2nd Edition Just Published!
http://www.cafeaulait.org/books/javaio2/
http://www.amazon.com/exec/obidos/ISBN=0596527500/ref=nosim/cafeaulaitA/

---

From: Elliotte Harold <[email-masked]>
Date: Mon, 02 Oct 2006 10:27:36 -0400

> Why the zip? Atom is an envelope format that can transport binary

No, it isn't. An atom envelope can point to non-XML data binary data, 
but it cannot reasonably embed it. You could use an atomLink element to 
point to the binary data; then embed the whole file set in a zip file. 
Maybe that's all that really needed. However, you would need that extra 
level of packaging.

I suppose you could base64 encode everything in a big atom:entry; but 
that's a lot of extra work for no particular point. It would essentially 
rule out using DOM to process this stuff, for example. Better to let 
MP3s, MPEGs, and other non-XML media to reside elsewhere and just let 
the XML point to it.

---

From: James M Snell <[email-masked]>
Date: Mon, 02 Oct 2006 08:29:57 -0700

+1

---

From: Alastair Rankine <[email-masked]>
Date: Tue, 3 Oct 2006 20:24:35 +1000

On 03/10/2006, at 12:21 AM, Elliotte Harold wrote:

> XOP, if I understand it correctly, could in fact be packaged in a  
> zip file, though no such packaging is described in the spec. If you  
> went with a non-XOP zip as opposed to XOP, I suspect you'd probably  
> end up reinventing a lot of XOP along the way. If you could get  
> away with 20% of XOP, then it might be worthwhile to do something  
> custom. Otherwise, I'd just go with XOP.

Just to kill my own suggestion... Now that I have read the spec and  
thought about about it there is a good reason for *not* using XOP for  
the purpose of exporting content: it's simply that the resources that  
accompany an Atom feed -- such as images referenced within the XHTML,  
enclosures, etc -- do not form part of a larger XML infoset. If the  
resource were part of the XML of the Atom feed, you could use XOP as  
an alternative to base64-encoding it. But because the referenced  
binary objects are separate entites (in the generic non-XML sense of  
the word), I don't think XOP is applicable here.

> (zipped XML + media). Does anyone know if that format is  
> sufficiently generic to handle this use case?

After a quick scan (http://hacks.oreilly.com/pub/h/2097) I'd say  
almost certainly, but it adds a lot of other stuff which may or may  
not be useful for Atom export.

Instead, how about this: a zipfile with a single "index.xml" file  
containing the Atom export document (as currently proposed), with all  
other resources added to the zipfile according to their relative URLs  
from the site being exported.

Say we had a blog at http://example.com which in turn referenced  
images at http://example.com/images/* and enclosures at http:// 
example.com/enclosures/*, the resulting zipfile would contain:

index.xml
images/1.jpg
images/2.jpg
...
enclosures/1.mp3
enclosures/2.mp3
...

Seems simple enough, but there's almost certainly some complexity  
that I'm overlooking.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 3 Oct 2006 14:03:47 +0200

* Alastair Rankine <[email-masked]> [2006-10-03 12:40]:
> Instead, how about this: a zipfile with a single "index.xml"
> file containing the Atom export document (as currently
> proposed), with all other resources added to the zipfile
> according to their relative URLs from the site being exported.

I’d rather the normative URIs for the media resources be given by
way of an alternate link in entries associated with each of the
resources. You’ll want such entries anyway in order to store any
associated textual data (eg. the description of an image) and
preserve other metadata such as who uploaded a file. Within the
zip, the file would be identified by using the atom:content/@src
attribute. The zip structure could be completely free-form with
the exception of the name and placement of the master feed.

Regards,

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 3 Oct 2006 14:05:30 +0200

* Elliotte Harold <[email-masked]> [2006-10-02 16:40]:
> It would essentially rule out using DOM to process this stuff,
> for example.

OK, sold. I hadn’t thought of that.

Regards,

---

From: Elliotte Harold <[email-masked]>
Date: Tue, 03 Oct 2006 08:16:28 -0400

Alastair Rankine wrote:

> Seems simple enough, but there's almost certainly some complexity that 
> I'm overlooking.

Yep. What if the images aren't subdirectories of the blog. e.g.

Say we had a blog at http://example.com/foo/bar/blog which in turn 
referenced images at http://example.com/images/* and enclosures at 
http://example.com/enclosures/*, the resulting zipfile would contain:

foo/bar/blog/index.xml
images/1.jpg
images/2.jpg
...
enclosures/1.mp3
enclosures/2.mp3

Either you need to rewrite URLs are you need one new file at a know 
location, not present in the blog itself, to identify where the master 
index is.

Or maybe not. xml:base attributes in the index.xml might be sufficient 
to straighten this all out. Indeed that might be necessary because 
different blog entries live at different URLs themselves. e.g. 
images/dog.png means one thing at http://example.com/blog/2006/Hello and 
another at http://example.com/blog/2005/Hello

Perhaps it would be helpful to back up a step. Rather than starting with 
format design let's try to get a list of everything we need to include 
in this format. E.g.

1. Complete source text for every blog entry.
2. Metadata for each blog entry
      A. Title
      B. Catgeory
      C. Tags
3. All media referenced by relative URLs from blog entry
4. URL of blog entry

What else?

---

From: Alastair Rankine <[email-masked]>
Date: Wed, 4 Oct 2006 20:25:10 +1000

On 03/10/2006, at 10:16 PM, Elliotte Harold wrote:

> Perhaps it would be helpful to back up a step. Rather than starting  
> with format design let's try to get a list of everything we need to  
> include in this format. E.g.
>

Well here's my list:

1. Complete list of authors and categories defined
2. For each article:
	a. Source text
	b. All the relevant metadata from the Atom spec, namely:
		author, ID, published, rights, title, updated, summary, categories
	c. Some other metadata:
		draft status, syntax of source
	d. "Owned" media, whether linked to in the source text or enclosure
3. For each comment or trackback:
	a. Source text
	b. Atom spec metadata:
		author, ID, title, published, summary, avatar?
	c. Additional metadata:
		pointer to parent article or comment (ie "in-reply-to")

The tricky bit is defining what is meant by "owned" media.

If we assume that an input to this process is a URL, I would say that  
"owned" media is any referenced media which resolves to the same  
host. This would preclude separately-hosted media (eg  
"images.example.com") but I don't see that this can be handled  
easily: how would an importer handle media destined for more than one  
host?

I don't think it's worthwhile attempting to support arbitary  
differences between paths of the exported data and its desired import  
location. For example, I wouldn't expect to be able to migrate from  
http://example.com/blog to http://example.org/my/cool/blog AND have  
all the relative and absolute links to media magically work.

So in a nutshell that's the problem I'm trying to solve.

---

From: James M Snell <[email-masked]>
Date: Wed, 04 Oct 2006 08:08:22 -0700

Alastair Rankine wrote:
> [snip]

With permissions, metadata, etc.

>         pointer to parent article or comment (ie "in-reply-to")

I would add to this information about what plugins have been applied and
what templates have been used.  These, of course, are not going to be
portable to different blog environments but the information would be
necessary in order to faithfully recreate the entries later.

> "owned" media is any referenced media which resolves to the same host.
> This would preclude separately-hosted media (eg "images.example.com")
> but I don't see that this can be handled easily: how would an importer
> handle media destined for more than one host?
> 

I would say that it's any media linked to by an entry located on the
same host as the entry, with the exporter given some discretion as to
what to include and what not to include.

> I don't think it's worthwhile attempting to support arbitary differences
> between paths of the exported data and its desired import location. For
> example, I wouldn't expect to be able to migrate from
> http://example.com/blog to http://example.org/my/cool/blog AND have all
> the relative and absolute links to media magically work.
> 

+1. It is the responsibility of the importer/exporter to make sure the
references work.  The export should result in an internally consistent
package that can be processed without having to understand the URL
structure of the original blog.

- James

---

From: Alastair Rankine <[email-masked]>
Date: Tue, 17 Oct 2006 21:46:36 +1000

> With permissions, metadata, etc.

Metadata like URL and email, sure.

I'm not so sure about permissions. Do we do groups then? And do we  
try to represent permissions for actions that are not directly  
related to authoring content? (eg changing the theme) There is huge  
potential for scope creep here.

My intent with an Atom-based export file format is to facilitate  
migration from one blogging engine (or similar CMS) to another. But  
implementations will differ in the functions they offer, and the way  
they offer them, so it is unreasonable I think to expect that a  
single file format would allow seamless migration in all cases. Such  
a format would be a union of the collective data models of all  
current and future blogging engines, and hence doomed to  
unimplementable complexity.

So there is a compromise to be made here: on the one hand we want the  
exported file to contain as much important data as possible, but on  
the other hand the format needs to be simple enough to implement  
across multiple blogging engines.

I think it's a worthy goal to minimize, rather than eliminate, the  
amount of manual reworking required when migrating from one platform  
to another. In other words it is not a goal for this format to be  
suitable for backup/restore.

All this is a roundabout way of saying that each blogging engine is  
likely to have a unique permissions model, and that it is not a big  
win to attempt to represent this in the export file format.

> necessary in order to faithfully recreate the entries later.

Agree, but there is a minefield of complexity here.

I currently use Typo as a blogging engine and it supports macros and  
filters in addition to a set of basic markup langauges. Getting the  
right combination of macros and filters is important to producing  
correct HTML output. So it would seem pretty important to include  
this information in the export file.

But on the other hand the macros and filters are unlikely to be  
supported on other platform, so probably the safest course of action  
is to expand these when the content is exported? There is some loss  
of information here, and an export/import operation is no longer  
semantically neutral, but the alternative is potentially worse.

> what to include and what not to include.

Agree. In practice the decision is likely to be based on whether the  
exporter can find the linked media file on a local filesystem.

Each image included probably needs to be accompanied by the  
(relative?) URL at which it was originally published.

So the list is now:

1. Complete list of authors defined. For each author:
	a. Name
	b. URI
	c. email
2. Complete list of categories defined:
	a. Name
	b. URI
3. All articles. For each article:
	a. Source text
	b. All the relevant metadata from the Atom spec, namely:
		author, ID, published, rights, title, updated, summary, categories
	c. Some other metadata:
		draft status, syntax of source
4. All comments and trackbacks. For each comment or trackback:
	a. Source text
	b. Atom spec metadata:
		author, ID, title, published, summary, avatar?
	c. Additional metadata:
		pointer to parent article or comment (ie "in-reply-to")
5. All "Owned" media. For each media object:
	a. URI
	b. MIME type
	c. Binary data

Does this look about right? Obviously there would need to be a  
liberal sprinkling of extension points for proprietary information.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 17 Oct 2006 14:26:03 +0200

* James M Snell <[email-masked]> [2006-10-04 17:30]:
> I would add to this information about what plugins have been
> applied and what templates have been used. These, of course,
> are not going to be portable to different blog environments but
> the information would be necessary in order to faithfully
> recreate the entries later.

-1

These should be engine-specific extensions. The format should be
broad enough in scope that it can reasonably be used to transport
a weblog between engines, but not broader. What you’re looking
for is a full backup, but that’s a different use case than what
(I think) this spec aims for.

Regards,

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 17 Oct 2006 14:46:46 +0200

* Alastair Rankine <[email-masked]> [2006-10-17 14:05]:
> Does this look about right? Obviously there would need to be
> a liberal sprinkling of extension points for proprietary
> information.

I think it is a good idea to also include the translated text of
each article, comment, etc. Eg. if they’re written in Markdown,
they should be accompanied by an HTML version, so that when
migrating to an engine which does not have Markdown support, such
articles and comments are not lost.

The trouble is with the content model. Suppose a weblog
supports distinct summaries and main articles, and it supports
Markdown. Firstly, text constructs do not support arbitrary MIME
type; only atom:content does. Secondly, there are then two
instances of the textual content of every supported field. If
titles may contain inline Markdown markup, how do you preserve
that?

I’m not sure what to do about this. The first thing that comes
to mind is that to accompany each atom:foo text construct with
a corresponding src:foo element, but I don’t know whether this is
a good idea. There are two options here: either the elements are
placed directly inside the atom:entry as extension elements, or
they are collectively placed inside the atom:content as instances
of an independent XML vocabulary. The latter seems favourable,
but again I don’t know whether any of this is really a good idea.

Regards,

---

From: Alastair Rankine <[email-masked]>
Date: Tue, 17 Oct 2006 23:31:42 +1000

> articles and comments are not lost.

Aside from the implementation issues I am unconvinced that this is a  
good idea in the first place.

If we have two alternative representations for an entry in the export  
file, it forces the importer *code* to choose which
representation is the best match for the target blogging engine. It  
is not obvious to me that the code will always be able to make this  
decision in the best manner. Also, it's not obvious whether or not  
the redundant ("cooked") HTML representation will be "good enough".  
This content negotiation has the potential to significantly  
complicate the job of the importer.

I thought it might be simpler for the *exporter* to choose a content  
syntax - perhaps with the help of the user - which is deemed to be  
the most interoperable, and yet closest to the original.

For example, a Typo implementation would probably export in the  
source Textile/Markdown/whatever, and yet expand all the macros  
because these are by definition not interoperable. And yet if the  
user is exporting from one Typo installation to another, they  
probably don't want to expand these macros, so the decision probably  
needs to be under user control.

My assumption here is that the user prettymuch always knows what is  
the right level of translation to be done on the data when moving  
their data from one system to another. This may well be an incorrect  
assumption.

Even in the worst case, namely the user has authored their data in a  
format that is just not supported at all on the target platform,  
should it necessarily be the exporter that resolves this problem? It  
could be an *intermediate* tool that converts the content from one  
format to another, on behalf of the target platform. For example, a  
tool that renders all Markdown-formatted content into HTML would be  
very simple to write and would also achieve the required goal.

So despite all the above I'm not totally *against* the idea of  
supplying multiple representations in the export file, but I do think
there are alternatives which should be considered.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 17 Oct 2006 15:48:30 +0200

* Alastair Rankine <[email-masked]> [2006-10-17 15:35]:
> I thought it might be simpler for the *exporter* to choose
> a content  syntax - perhaps with the help of the user - which
> is deemed to be  the most interoperable, and yet closest to the
> original.

That is a good counter.

However, you still need to address the restriction of text
constructs, namely that in Atom, they can be only of type `text`
(with no semantics implied for whitespace), `html` or `xhtml`.
Without further provisions you can only export the cooked content
of elements other than atom:content.

Regards,

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Oct 2006 08:17:01 -0700

I agree, however, the format should be extensible enough to support
broader use cases.  With Atom extensibility, this is very easy to
achieve without allowing the scope of the spec to creep too far.

- James

---

From: Elliotte Harold <[email-masked]>
Date: Tue, 17 Oct 2006 11:39:48 -0400

> 4. All comments and trackbacks. For each comment or trackback:

This one needs to be expanded somewhat. In particular is it:

A. The text of the entry as published in an Atom feed?

or

B. The source of that text as it appears in the original?

In WordPress, for example, these two can be quite different. The real 
source is likely not HTML or XHTML, while the Atom output is.

I recommend choosing A. I think it's the only one that has a chance of 
being interoperable.