Subject: Why use Atom?

From: pkeane <[email-masked]>
Date: Thu, 4 Oct 2007 23:47:44 -0500 (CDT)

I wonder if I might get feedback on this question:

I am in the process of refactoring a web-based data repository (see 
daseproject.org), moving from PHP4 to PHP5 among other things, but also 
moving to a more fully realized service-based architecture.  The 
repository holds many collections of digital assets, primarily images,but 
also audio, video, pdfs, etc.

The repository is used by faculty at BUT Austin to 
store/organize/share/present digital 'stuff' for both instructional and 
research purposes.  When a faculty member creates a new collection they 
choose the set of attributes (i.e. "fields") they would like to use to 
describe their items.  Since these are often existing collections being 
ported from FileMaker, Access, etc. it is quite efficient to be able to 
use the same attributes.  Thus a collection is simply a set of "items" 
described by arbitrary key-value pairs with an associated media file (or 
files if there are different sizes, say, for an image, or an mp3 AND wav 
file for audio).

I understand the benefit of using Atom for outward facing services, 
apis, etc., but does Atom offer benefits for internal messaging, data 
transfer, AJAX, etc?  I suspect yes, but I cannot get past 
the fact that my ultra-generic xml schema is REALLY easy to deal with:

[note that there are only TWO possible child elements for an item (think 
atom 'entry'): metadata AND media_file]

<item serial_number="000435213">
<metadata ascii_id="admin_checksum">630230b057c511cbee87447960fff02e</metadata>
<metadata ascii_id="admin_filename">62-GT-06.jpg</metadata>
<metadata ascii_id="admin_file_size">318642</metadata>
<metadata ascii_id="admin_image_height">576</metadata>
<metadata ascii_id="admin_image_width">720</metadata>
<metadata ascii_id="admin_mime_type">image/jpeg</metadata>
<metadata ascii_id="admin_serial_number">000435213</metadata>
<metadata ascii_id="admin_upload_date_time">2007-07-18T15:59:28</metadata>
<metadata ascii_id="credit">Photographer: Unknown</metadata>
<metadata ascii_id="dase_rights">Restricted</metadata>
<metadata ascii_id="description">photo of Ben Barnes while Speaker, black 
and white</metadata>
<metadata ascii_id="keyword">Ben Barnes</metadata>
<metadata ascii_id="keyword">Capitol Building interior</metadata>
<metadata ascii_id="keyword">Lieutenant Governor</metadata>
<metadata ascii_id="keyword">Speaker of the House</metadata>
<metadata ascii_id="original_filename">62-GT-06.jpg</metadata>
<metadata ascii_id="rights_owner">Senate Media Services</metadata>
<metadata ascii_id="rights_status">Use in Texas Politics 
content</metadata>
<metadata ascii_id="scratch_pad">/62-GT-06.jpg</metadata>
<metadata ascii_id="title">Ben Barnes</metadata>
<metadata ascii_id="used_in_chapter">none</metadata>
<media_file filename="000435213_800.jpg" size="medium" height="576" 
width="720" mime_type="image/jpeg" />
<media_file filename="000435213_100.jpg" size="thumbnail" height="80" 
width="100" mime_type="image/jpeg" />
<media_file filename="000435213_640.jpg" size="small" height="480" 
width="600" mime_type="image/jpeg" />
</item>

Any thoughts on the benefits of using atom here?  While I realize the 
wisdom of NOT reinventing wheels 
(http://www.tbray.org/ongoing/When/200x/2006/01/08/No-New-XML-Languages) I 
also aim for maximum simplicity.

thanks!
Peter Keane

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 5 Oct 2007 12:10:04 +0700

Peter Keane:
> be able to use the same attributes.  Thus a collection is 
> simply a set of "items" 
> described by arbitrary key-value pairs with an associated 
> media file (or files if there are different sizes, say, for 
> an image, or an mp3 AND wav file for audio).

In one of the earlier messages, it was mentioned that you guys already
use WebDAV. Why don't you guys just keep on using WebDAV? If WebDAV
isn't working for you then AtomPub probably won't work either--they are
very similar to each other. In fact, it looks to me like it would be
easier to write a WebDAV client than an AtomPub client, because WebDAV
gives less flexibility to servers than AtomPub does.

If you just need a notification mechanism for your document repository,
then I suggest building a syndication feed for your WebDAV
implementation. Clients can subscribe to atom feeds to get notifications
of changes to the repository, and they can continue to use WebDAV to
retrieve documents and make changes.

I especially recommend sticking with WebDAV if you do document
versioning in your repository (don't you?). There is no (standardized)
versioning mechanism in AtomPub.

Regards,
Brian

---

From: pkeane <[email-masked]>
Date: Fri, 5 Oct 2007 00:43:52 -0500 (CDT)

Actually, my system (DASe - daseproject.org) does not use WebDAV at all, 
but I think I know where the confusion arose.   My earlier mention of 
WebDAV concerned NOT the data repository software, but the campus-wide 
shared storage system on which all UT Austin faculty/student/staff get 
150 Mb free storage. It is simply file space and that's it -- the vendor 
is Xythos and it does allow WebDAV access AND users can make items they 
upload visible over HTTP.  Thus, my aim was to use it simply as a way to 
get actual digital assets from the faculty member into my system (e.g., 
"Well Prof. Smith, if you can throw those 100 new images from your 
digital camera into the DASe_uploads folder in your WebSpace account, 
tomorrow when you go to DASe you will see them appearing in your 
private collection [since I will have run a batch process to grab 'em] 
ready to have metadata added").

My question regarding using Atom w/in the application was a totally 
different one -- this has to do with architecting the flow of metadata 
(not the assets themselves) throughout the application, mainly for 
AJAX-type interfaces. I would really like to have feeds that can pull 
double-duty both as outward-facing AND internally utilized data 
'services'.

Also, when I get to that point, APP seems like the perfect way to allow 
folks to update metadata (which is the most common task -- the uploading 
of the actual digital assets is, in most cases, a batch operation that 
occurs infrequently).

-peter keane

---

From: Bill de hOra <[email-masked]>
Date: Fri, 05 Oct 2007 10:59:47 +0100

pkeane wrote:
> repository holds many collections of digital assets, primarily 
> images,but also audio, video, pdfs, etc.
> 
> transfer, AJAX, etc?  I suspect yes, but I cannot get past the fact that 
> my ultra-generic xml schema is REALLY easy to deal with:
> 
> <metadata 
> ascii_id="admin_checksum">630230b057c511cbee87447960fff02e</metadata>
> <metadata ascii_id="description">photo of Ben Barnes while Speaker, 
> black and white</metadata>
> <metadata ascii_id="rights_status">Use in Texas Politics content</metadata>
> (http://www.tbray.org/ongoing/When/200x/2006/01/08/No-New-XML-Languages) 
> I also aim for maximum simplicity.

This looks like it will drop into Atom. Your main reason right now, 
would be integrate with a globally available toolchain so you can 
interchange information. Maximum simplicity is not defining yet another 
format you have to persuade others to support. For example rights_owner 
could be made go away in favour of atom:rights, dublin core or cc.

And I don't agree your ultra-generic schema will be be easy to deal 
with; ultra-genric things tend to be more complicated than expected.

<metadata ascii_id="title"><h3>Ben &amp; Jerry</h3></metadata>

what happens there?
cheers
Bill

---

From: "A. Pagaltzis" <[email-masked]>
Date: Sat, 6 Oct 2007 00:44:30 +0200

* pkeane <[email-masked]> [2007-10-05 07:00]:
> Any thoughts on the benefits of using atom here?

I don’t see the problem. Atom gives you an Entry where you can
put the metadata for a media resource. You have a bunch of
attributes that should be mapped to Atom elements; the rest you
stick into the content, possibly as RDF since your ad-hoc vocab
is more or less along those lines anyway.

> I cannot get past the fact that my ultra-generic xml schema is
> REALLY easy to deal with

It’s not actually very generic. It’s a very limited vocabulary
that expresses barely any more than a map of key-value pairs. Of
course such a simple data structure is easy to deal with. The
only genericity there is that the keys are arbitrary strings. It
looks easy now because you have to do almost no work up front:
the structure is rigid and the semantics are completely ad-hoc.

It won’t look very easy at all once you have a large dataset with
an inconsistent mess of key names.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: pkeane <[email-masked]>
Date: Fri, 5 Oct 2007 18:35:39 -0500 (CDT)

Yes, it is really nothing more than key-value pairs.  I am more wondering 
about the possible benefits of Atom than whether this system works -- I 
use it for data import/export of the collections and it is quite easy to 
create parsers and generators for this format that lets me move it in and 
out of the relational databse that the application uses.

The database itself is also quite generic: a "collections" table, and 
"items" table, a "values" table and an "attributes" table (each value has 
an item_id and and attribute_id).  It is important that the data model be 
able to grow organically -- as a user adds a new "field" (aka key or 
attribute) to describe the items they have, they'll have no knowledge at 
all of Atom or Dublin Core or any of that.  And it's fine -- every 
collection has a unique set of attributes (aka fields or keys).  The 
composite primary key for attribute is "ascii_id" plus "collection_id".

The system has been in production and heavily used for a couple years, 
and includes 88 collections comprising 300,000 items.  The are currently 
1358 rows in the attribute table (those are the keys in the key->value 
pairs) and 4.5 million rows in the value table.  We've had no problems at 
all with this current architecture.  And yet I wonder what Atom could do 
for me as a more standard XML format for data serialization...

thanks!
Peter Keane
daseproject.org

On Sat, 6 Oct 2007, A. Pagaltzis wrote:

> I don?t see the problem. Atom gives you an Entry where you can
> It?s not actually very generic. It?s a very limited vocabulary
> It won?t look very easy at all once you have a large dataset with

---

From: James M Snell <[email-masked]>
Date: Fri, 05 Oct 2007 17:06:59 -0700

Basically, if it's a closed system with specific clients, there likely
will not be any benefit to using Atom.  If you wish to enable
interchange and interop with other applications, there will be benefits
to using Atom, if only to leverage the existing tool support.

- James

pkeane wrote:
> 
> Yes, it is really nothing more than key-value pairs.  I am more
> wondering about the possible benefits of Atom than whether this system
> works -- I use it for data import/export of the collections and it is
> quite easy to create parsers and generators for this format that lets me
> move it in and out of the relational databse that the application uses.
> 
> "items" table, a "values" table and an "attributes" table (each value
> has an item_id and and attribute_id).  It is important that the data
> model be able to grow organically -- as a user adds a new "field" (aka
> key or attribute) to describe the items they have, they'll have no
> knowledge at all of Atom or Dublin Core or any of that.  And it's fine
> -- every collection has a unique set of attributes (aka fields or
> keys).  The composite primary key for attribute is "ascii_id" plus
> "collection_id".
> 
> pairs) and 4.5 million rows in the value table.  We've had no problems
> at all with this current architecture.  And yet I wonder what Atom could
> do for me as a more standard XML format for data serialization...
> 
> thanks!
> Peter Keane
> daseproject.org
> 
> 
> 
> 
>>> ascii_id="admin_upload_date_time">2007-07-18T15:59:28</metadata>

---

From: pkeane <[email-masked]>
Date: Fri, 5 Oct 2007 20:37:16 -0500 (CDT)

Yup, I am trying to decide if the tool support is enough to justify 
the effort.  And still I wonder if there is some other potential side 
benefit am not seeing. Here, by the way, is a collection as Atom Feed 
(with only one item shown).  Note that collection owners can declare their 
own custom attributes to "map" to Atom Elements if they wish, in which 
case they appear in the default Atom namespace, otherwise they are in the 
"dase" namespace (standard administrative metadata common to all 
collections) or in the collection's own namespace.

Note that there is no hand coding here, just a method on a collection 
object, e.g "print $collection->asAtom()".

-pk

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
   xmlns:dase="http://quickdraw.laits.utexas.edu/dase"
   xmlns:texpol="http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/1.0"
   xml:base="http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/">
  <title>Texas Politics Image Collection</title>
  <id>http://quickdraw.laits.utexas.edu/dase/texpol_image_collection</id>
  <author>
   <name/>
  </author>
  <updated>1969-12-31T18:00:00-06:00</updated>
  <entry xml:base="http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/">
   <id>http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/000435205</id>
   <updated>1969-12-31T18:00:00-06:00</updated>
   <title>Congress Avenue</title>
   <summary>photo of Congress Avenue</summary>
   <dase:admin_mime_type>image/jpeg</dase:admin_mime_type>
   <dase:admin_filename>PICA17902.JPG</dase:admin_filename>
   <dase:admin_checksum>c837f0abd05c8b7126b8dac15d510f30</dase:admin_checksum>
   <dase:admin_file_size>787705</dase:admin_file_size>
   <dase:admin_image_width>1408</dase:admin_image_width>
   <dase:admin_upload_date_time>2007-07-18T15:59:19</dase:admin_upload_date_time>
   <dase:admin_serial_number>000435205</dase:admin_serial_number>
   <dase:admin_image_height>1209</dase:admin_image_height>
   <texpol:keyword>Congress Avenue</texpol:keyword>
   <texpol:keyword>buildings</texpol:keyword>
   <texpol:scratch_pad>/PICA17902.JPG</texpol:scratch_pad>
   <texpol:rights_owner>Austin History Center</texpol:rights_owner>
   <texpol:rights_status>Use in Texas Politics content</texpol:rights_status>
   <texpol:credit>Photographer: Unknown</texpol:credit>
   <texpol:dase_rights>Restricted</texpol:dase_rights>
   <texpol:original_filename>PICA17902.JPG</texpol:original_filename>
   <texpol:used_in_chapter>executive</texpol:used_in_chapter>
   <link length="9701" type="image/jpeg" rel="http://quickdraw.laits.utexas.edu/dase/media/thumbnail" href="/media/thumbnail/000435205_100.jpg"/>
   <link length="78505" type="image/jpeg" rel="http://quickdraw.laits.utexas.edu/dase/media/viewitem" href="/media/viewitem/000435205_400.jpg"/>
   <link length="783340" type="image/jpeg" rel="http://quickdraw.laits.utexas.edu/dase/media/full" href="/media/full/000435205_3600.jpg"/>
   <content src="http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/media/thumbnail/000435205_100.jpg" type="image/jpeg"/>
  </entry>
  </feed>

---

From: James M Snell <[email-masked]>
Date: Fri, 05 Oct 2007 19:03:07 -0700

If you rely on a whole bunch of application specific extension elements,
you will not realize any significant benefit from using Atom.  If you go
with Atom, find a more general way of encoding your data (e.g. use RDF
for instance).

Regarding the question about tool support, that depends entirely on who
will be using the application.  If you're building a tool for a very
specific and limited audience, it's probably not worth the effort.  If
you're building a tool for an open audience, and the data from your
application might be used for purposes you hadn't originally intended,
use Atom and eliminate all the app-specific extensions.

- James

pkeane wrote:
> Yup, I am trying to decide if the tool support is enough to justify the
> effort.  And still I wonder if there is some other potential side
> (with only one item shown).  Note that collection owners can declare
> their own custom attributes to "map" to Atom Elements if they wish, in
> which case they appear in the default Atom namespace, otherwise they are
> in the "dase" namespace (standard administrative metadata common to all
>  <entry
>   <link length="9701" type="image/jpeg"
> rel="http://quickdraw.laits.utexas.edu/dase/media/thumbnail"
> href="/media/thumbnail/000435205_100.jpg"/>
>   <link length="78505" type="image/jpeg"
> rel="http://quickdraw.laits.utexas.edu/dase/media/viewitem"
> href="/media/viewitem/000435205_400.jpg"/>
>   <link length="783340" type="image/jpeg"
> rel="http://quickdraw.laits.utexas.edu/dase/media/full"
> href="/media/full/000435205_3600.jpg"/>
>   <content
> src="http://quickdraw.laits.utexas.edu/dase/texpol_image_collection/media/thumbnail/000435205_100.jpg"
> type="image/jpeg"/>
>  </entry>
>  </feed>
> 
>

---

From: pkeane <[email-masked]>
Date: Fri, 5 Oct 2007 23:50:36 -0500 (CDT)

> use Atom and eliminate all the app-specific extensions.

That strikes me as sound advice.  I saw a quote from Bill de hÓra cited on 
Aristotle Pagaltzis' blog (http://plasmasturm.org/log/463/) a while back:

"Any kind of data garden is fair game for AtomPub to rationalize."

In higher education we (faculty,librarians, etc.) are drowning in digital 
collections ("one off's") that eventually need to be shared, ported to the 
web (e.g. currently on some departmental Filemaker server), preserved, 
repurposed, etc. Current library tools (e.g. DSpace) generally hue to the 
Dublin Core way of looking at the world, which simply does not fit with 
the way faculty want to think of their stuff.  The way I see what we've 
done in DASe is a slightly more structured form of "tagging", but the tags 
here are allowed to have a type (i.e., not just all keywords).  I wouldn't 
call it a 'specific and limited audience' by any means, but for these 
purposes, perhaps so.  I am told that ArtStor (the largest vendor 
currently for web-based Art & Art History image collections) has now opted 
to go with key-value pairs (they too are "harvesting" collections 
that have originated in a wide variety of places) rather than "top-down" 
metadata schemas.  It'll be interesting to see how this all shakes out.

I don't really need/want the complexity of RDF and I certainly do 
not want to try to explain such a thing to a 
not-particularly-technology-saavy faculty member that I am trying to 
persuade NOT to simply build another Filemaker database!  The constraints 
provided by a flat key-value system has proven quite useful, actually.  I 
suspect that I will end up establishing some subset of Atom Elements 
(title, summary, updated, id) that every collection has as "common" 
attributes and simply throw the rest of the collection-specific key-value 
pairs into the <content> element as xml (or perhaps xhtml).

-peter keane
daseproject.org

---

From: Brendan Taylor <[email-masked]>
Date: Sat, 6 Oct 2007 09:23:16 -0600

On Fri, Oct 05, 2007 at 11:50:36PM -0500, pkeane wrote:
> I don't really need/want the complexity of RDF and I certainly do not want 
> to try to explain such a thing to a not-particularly-technology-saavy 
> faculty member that I am trying to persuade NOT to simply build another 
> Filemaker database!

The complexity of RDF is vastly overstated; as Aristotle said, your
homegrown XML is practically RDF already.

---

From: pkeane <[email-masked]>
Date: Sat, 6 Oct 2007 12:57:32 -0500 (CDT)

Well, I am revisiting RDF and I think I may have been influenced unduly by 
the fear of an "RDF complexity tax" which, as you say, is probably 
overstated.

Now I need to figure out how I can allow for the possibility of semantics 
changing in the lifecycle (or use case) of an entry in my system. For 
instance, I may want to export a collection 'encoded' as Dublin Core, 
MODS, VRA Core, Atom, etc.  In the library world thay call them 
"crosswalks" that establish equivalences between different metadata 
schemas.  My thought is that Atom provides the best and most widely used 
"lowest common denominator".  I think that some RDF in the <atom:content> 
may be just the trick.  And I am guessing that if I use RDF intelligently 
I might be able to 'dereference' a piece of metadata as its equivalence in 
whatever schema context pertains to that particular dereferencing 
operation.  In other words, I want to set up a (potentially dynamic) 
one-to-many relationship with a 'key' and it's equivalence out there in 
the wider information ecosystem.

While it may seem a bit crazy, the use case is quite simple: faculty 
members needing to organize their digital 'stuff' have resorted to 
FileMaker (mixes presentation w/ semantics and is remarkably difficult to 
repurpose) or Excel Spreadsheets (often a nice and easily re-purposeable 
format, but almost always includes unwise attempts at repeatable fields). 
I am simply looking to (and have succeeded at) rationalizing the whole 
mess, at least here at UT Austin, and now seek to rationalize things in a 
larger context.

thanks-
Peter Keane
daseproject.org

---

From: "Terris Linenbach" <[email-masked]>
Date: Sat, 6 Oct 2007 11:58:47 -0700

Since Atom has traction for describing who/what/when, I'm an Atom junkie as a matter of practicality. As for the "what", when you're talking about data instead of presentation, there seems to be two best choices: w3c xml schema or RDF (RDFXML until some binary representation of RDF becomes popular). xml schema of course is likely to be more common but interestingly I expect to see more reuse on the RDF side. RDFXML in my opinion makes RDF look scary initially. 
 
IMO it'll be a great day when ATOM toolsets integrate with RDF toolsets well enough for developers to be able to use both easily. The key word is "easily."
 
-----
From: [email-masked] on behalf of pkeane
Sent: Sat 10/6/2007 10:57 AM
To: Brendan Taylor
Cc: [email-masked]
Subject: Re: Why use Atom?

---

From: "A. Pagaltzis" <[email-masked]>
Date: Sun, 7 Oct 2007 03:02:04 +0200

* pkeane <[email-masked]> [2007-10-06 20:10]:
> And I am guessing that if I use RDF intelligently I might be
> able to 'dereference' a piece of metadata as its equivalence in
> whatever schema context pertains to that particular
> dereferencing operation. In other words, I want to set up a
> (potentially dynamic) one-to-many relationship with a 'key' and
> it's equivalence out there in the wider information ecosystem.

Take a look at OWL’s sameAs property. You can use that to
establish that two different properties in RDF mean the same
thing.

It’s only a partial answer though – it works only when the values
of the properties follow the exact same rules. But it’s at least
something.

(OWL defines more properties to deal with subtler mappings, but
I’m not well versed in it, and when I hear RDF-heads talk, they
only ever mention sameAs, so I have no opinion on how useful the
other stuff is.)

Regards,

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 13 Oct 2007 03:43:12 +0200

>    <texpol:used_in_chapter>executive</texpol:used_in_chapter>

If you could document these fields and their usage in the consuming  
application, it would be easier to devise a way of encoding them in a more  
Atom-friendly way.

As to your current implementation, why is 'dase:admin_mime_type' needed  
when you have 'atom:content/@type'? Can't 'texpol:rights_owner',  
'texpol:rights_status' and 'texpol:dase_rights' be implemented with  
'atom:rights' or 'link[@rel='copyright']' (or both)?

Can't 'texpol:original_filename' be an 'atom:link' with an appropriate  
'@rel' and a working (dereferencable) '@href'? The same goes for  
'texpol:used_in_chapter', 'dase:admin_filename' and 'texpol:scratch_pad'.  
I'm sure Dublin Core can help out with proper relationships here.

If you can use the Dublin Core elements directly, do that. I'm thinking  
that 'dase:admin_upload_date_time' has the same semantics as 'DC:  
Created'; <http://dublincore.org/documents/dcmi-terms/#created> and thus  
can safely be replaced with that element.

'texpol:keyword' - can't 'atom:category' be used here?

When excersising all of the changes I propose above (assuming all of them  
can be done successfully), you end up with a lot less extension elements  
and a much more interoperable format. The stuff left is really just  
metadata about the physical properties of the image (height, width, size)  
that can be stuffed into one extension element.

-- 
Asbjørn Ulsberg          -=|=-         [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: pkeane <[email-masked]>
Date: Fri, 12 Oct 2007 23:01:09 -0500 (CDT)

On Sat, 13 Oct 2007, Asbjørn Ulsberg wrote:

> As to your current implementation, why is 'dase:admin_mime_type' needed when 
> you have 'atom:content/@type'? Can't 'texpol:rights_owner', 
> Can't 'texpol:original_filename' be an 'atom:link' with an appropriate '@rel' 
> and a working (dereferencable) '@href'? The same goes for 
> 'texpol:used_in_chapter', 'dase:admin_filename' and 'texpol:scratch_pad'. I'm 
> sure Dublin Core can help out with proper relationships here.
>

I am not sure that I described my situation very effectively in my 
original post.  The DASe system, as we have it running at UT Austin, is 
what I'd call a "Data First" application (see Stefano Mazzocchi's
"Data First vs. Stucture First" 
http://www.betaversion.org/~stefano/linotype/news/93/).  What the RDF 
folks are thinking is more akin to this system: it is infinitely 
extensible, simply by the fact that any collection manager (we have 88 
collections, comprising 1000+ 'fields' -- we call them attributes) can 
create a new attribute.  All of the above prefixed 'tex_pol' are 
attributes that the folks managing the "Texas Politics Image Collection" 
created themselves -- they possess semantics *internal* to that collection 
and in almost every use case I have seen in 3+ years, that works fine. 
The application itself has a namespace as well (dase: in the example). 
This holds immutable administrative metadata captured/created when a new 
image (or other digital asset) is created.  This is handy for all kinds of 
application-specific housecleaning.

Applying externally recognized sematics here serves no useful purpose 
until/unless the data will be used outside of this system.  One example is 
in the case of RSS feeds.  I have given collection managers the ability to 
create "mappings" between their attributes and a set of Atom attributes 
(title, summary, rights, etc.), and the system iteself is 'smart' enough 
to map the obvious ones from the set of administrative attributes.  Thus 
we can easily offer RSS/Atom feeds for collections or subsets thereof. 
Other mappings are available as well -- I have built a proof-of-concept 
OAI-PMH publisher that uses an xml-described mapping to make the data 
availble using Dublin Core, for instance.

The metadata that we hold in what is essentially key-value pairs is the 
"raw" semantic stuff -- this is what the users, typically a faculty member 
with a bunch of images that they have "cataloged" in a FileMaker database 
or Excel spreadsheet that they designed to fit their own specific need. 
Our goals was to get this stuff up on the web and the only way to make 
that happen was to keep the original attributes they had created and use a 
very generic approach -- the system ONLY thinks in term of key->value 
pairs.  *Predictions of Doom* from all corners -- librarians and 
developers alike -- were, thankfully, not borne out.  It has been a 
resounding success and folks are adding new collections and assets daily.

Some subset of the collections are searchable/viewable and some remain 
private but they now ALL live in this system (rather than on a computer 
in a departmental office somewhere) and can be managed, preserved, 
repurposed, etc. quite easily.

All that said, I had (have) a hunch that deeper use of Atom and esp. 
AtomPub could make the system even more open and flexible.  Uploading 
happens in one of two ways: 1) the user uploads items one at a time 
through a web interface or 2) the user FTPs the assets to our servers and 
I do a batch upload that creates all of the proper admin metadata on the 
fly.  I am also experimenting with a system whereby the user puts the 
images on a web doc root somewhere and the system grabs them over HTTP 
(thus a previous questions abut how to create an Atom document describing 
the contents of a filesystem directory).

I have had an interesting few days reading the archives for this list and 
was particularly interested in the debates about RDF (whether Atom should 
be RDF like RSS 1.0) and whether Atom should be an abstract data model 
that Atom XML was simply one representation of.  For my part, I think the 
Atom folks hit the sweet spot on all counts.  But Atom cannot be 
everything to everyone and perhaps it is unrealistic of me to hope that it 
is the ONE TRUE data format that can contain my data and provide me with 
all of those tools, not to mention constraints that I need, to be able to 
make a flexible and extensible system.  I am looking with great interest 
at systems like Dynamo (Amazon) and CouchDB since I think on some level 
the problem domain is quite similar (well, not so much the massive 
scalability, but rather the flexible 'data first' approach to metadata).

-Peter Keane
daseproject.org

> If you can use the Dublin Core elements directly, do that. I'm thinking that 
> 'dase:admin_upload_date_time' has the same semantics as 'DC: Created'; 
> <http://dublincore.org/documents/dcmi-terms/#created> and thus can safely be 
> replaced with that element.
>
> When excersising all of the changes I propose above (assuming all of them can 
> be done successfully), you end up with a lot less extension elements and a 
> much more interoperable format. The stuff left is really just metadata about 
> the physical properties of the image (height, width, size) that can be 
> stuffed into one extension element.
>
> --

---

From: "A. Pagaltzis" <[email-masked]>
Date: Sat, 13 Oct 2007 18:41:13 +0200

* pkeane <[email-masked]> [2007-10-13 06:15]:
> All that said, I had (have) a hunch that deeper use of Atom and
> esp. AtomPub could make the system even more open and flexible.

Well, as long as you have sane mapping between your RDF(ish)
attributes and Atom elements, that should work just fine.

If you want to upload via Atompub, it’s probably best to make the
mapping two-way, so your collection-specific metadata is
automatically created from the Atom Entry metadata when an upload
comes in that way.

As long as the mapping is good, things should be peachy. (Ie.
make sure that the value domains of attributes mapped to each
other are exactly equal. An important reason for Atom to define
its own metadata elements rather than use Dublin Core was
precisely this – that DC generally places fewer constraints on
values than the WG wanted in Atom for interoperability reasons.)

Regards,

---

From: pkeane <[email-masked]>
Date: Sun, 14 Oct 2007 00:39:34 -0500 (CDT)

> attributes and Atom elements, that should work just fine.

And the "sanity" of those mapping will mostly depend on the end-users 
(i.e., collection managers) -- this is part of our philosopy (approach? 
survival-tactic?) to push as much control to the users as possible.  But 
what the heck -- if they create good mappings, the feeds look good out 
there and if not, they don't....
>
> If you want to upload via Atompub, it?s probably best to make the

I suspect that the only metadata I'll be able to grab upon upload is the 
admin stuff (and I'll be the one creating those mappings).  What we've 
done so far is to tell folks that they get the admin metadata to start 
with, and then they can add more once the items are in the system.  I have 
had SOME luck grabbing actual metadata from images manipulated in iPhoto 
that the users have entered temselves, but that all relies on the 
application using a standard (iPhoto uses IPTC embedded in the jpeg).  I 
believe Adobe is starting to use RDF, but it might be in a separate 
'sidecar' file, which is a whole other can o' worms.
> precisely this ? that DC generally places fewer constraints on

Yup.  I like what the DC folks have attempted, but I feel like the Atom 
effort really nailed things down in a much more useful way.

My ideal scenario, aside from making uploading simpler, is to create a 
system that holds all kinds of digital stuff, cataloged all sorts of 
different ways and if a faculty member wants to "repurpose" those assets 
as a web site, or learning module or whatever, they can hire a student web 
developer who simply has to set a few "data export settings" (don't want 
to frighten them into thinking they are creating metadata mappings -- I've 
had too many instances of faculty screaming in terror, running off into 
the night never to be seen again after I gave them my "friendly 
introduction to Dublin Core"), then creating a simple web-based "mash-up" 
to use that data feed.  Much simpler than hiring a programmer to build yet 
another PHP/MySQL application that'll need to be maintained after they 
leave.  This whole "expose the data with Atom" is a way to enforce some 
rational constraints.

I'd love to get to the point where Atom could fill in as a simpler 
solution in all sorts of domains:  simpler WebDAV, simpler Dublin Core, 
simpler OAI-PMH, etc.  Of course those other standards are trying to do 
other things as well, but maybe Atom will hit an 80/20 mark that will 
allow interesting work to be accomplished much more easily.

http://plasmasturm.org/log/463/
http://efoundations.typepad.com/efoundations/2007/07/app-moves-to-pr.html

-peter keane