Subject: Atom Link Extensions Use Case

From: Ed Summers <[email-masked]>
Date: Fri, 8 Jun 2012 09:04:03 -0400

Hi all,

I am using Atom to syndicate access to data dumps at the Library of
Congress. We have a web application that provides access to historic
newspapers [1], and we have received requests for access to the
underlying OCR data for research and commercial purposes. Despite the
fact that this is historic data, we are routinely adding new content
as it is digitized. Rather than require clients to issue millions of
requests to get at the OCR data (which is actually web addressable)
the plan is to periodically create a tarred and compressed dump file
of new OCR content, and publish the availability of the file in an
Atom feed, which interested parties can subscribe to. It's a similar
model to what Wikimedia does for various Wikipedia projects [2].

Here's a minimal example, to give you an idea of what I mean (warning
URLs don't currently resolve):

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>Chronicling America OCR Dumps</title>
    <link rel="self" type="application/atom+xml"
href="http://chroniclingamerica.loc.gov/dumps/ocr/feed/" />
    <id>info:lc/ndnp/dumps/ocr</id>
    <author>
        <name>Library of Congress</name>
        <uri>http://loc.gov</uri>
    </author>
    <updated>2012-06-08T08:35:27-04:00</updated>
    <entry>
        <title>part-00001.tar.bz2</title>
        <link rel="alternate" type="application/x-bzip2"
href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"
/>
        <id>info:lc/ndnp/dump/ocr/part-00001.tar.bz2</id>
        <updated>2012-06-07T13:57:23-04:00</updated>
        <summary type="xhtml"><div
xmlns="http://www.w3.org/1999/xhtml">OCR dump file <a
href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2">part-00001.tar.bz2</a>
with size 162.7 MB generated June 7, 2012, 1:57 p.m.</div></summary>
    </entry>
</feed>

So the reason why I am writing here is that I would like to add
checksum information to the feed to let clients verify that they have
downloaded the data dump file correctly. An argument could be made
that it's not necessary since a corrupted bz2 file would likely not
decompress. An argument could also be made that the Content-MD5 header
could be used. But I like the idea of making an explicit assertion
about the checksum in the Atom document.

After a bit of googling I ran across James Snell's Atom Link
Extensions draft, which provides a pattern for including an md5
checksum in the <link> element like so:

    <link rel="alternate" type="application/x-bzip2"
hash="md5:579758192095fde80896058af4ce0aee"
href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"
/>

Unfortunately it looks like the draft has expired. I was wondering:

a) are there other established patterns for adding checksum
information for resources in Atom
b) if it's worth it for James to update the draft and try to push it
forwards to an Informational status

As more and more data providers make dumps of their data available to
reduce crawling (like Wikipedia) it seems like a good use case for
Atom to support.

//Ed

[1] http://chroniclingamerica.loc.gov
[2] http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-abstract.xml-rss.xml
[3] http://tools.ietf.org/html/draft-snell-atompub-link-extensions-08

---

From: Richard Salz <[email-masked]>
Date: Fri, 8 Jun 2012 09:47:07 -0400

Way cool application!  Yes, it'd be good to refresh the I-D.  At a 
minimum, change the md5 examples to sha-256. :)
--
STSM, WebSphere Appliance Architect
https://www.ibm.com/developerworks/mydeveloperworks/blogs/soma/

---

From: Philippe Rathé <[email-masked]>
Date: Fri, 8 Jun 2012 10:26:19 -0400

Whatever the status of the I-D would change or not, I would use it
anyway as its current state. Unless that you find a standard that
provides exactly what you want.

In addition to that, it may be a good idea to add to your atom:entry a
link relation that advertise your potentially big file size with a
rel="alternate" link. So when crawler may handle it appropriately and
cache the result.

    <link rel="enclosure" type="application/x-bzip2" length="123456789"
            href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"/>

Well it is what I intend to do, I have similar needs.

-- Philippe

On Fri, Jun 8, 2012 at 9:47 AM, Richard Salz <[email-masked]> wrote:
> Way cool application!  Yes, it'd be good to refresh the I-D.  At a minimum,
> change the md5 examples to sha-256. :)
> --

---

From: James M Snell <[email-masked]>
Date: Fri, 8 Jun 2012 07:39:20 -0700

Good stuff. I can definitely resurrect the draft and move it forward
if there's enough interest.

- James

---

From: Tim Bray <[email-masked]>
Date: Fri, 8 Jun 2012 07:41:00 -0700

Why not just drop an element into the <entry> in your own namespace?  This
doesn’t feel like any kind of a link to me.

<feed xmlns:loc="http://whatever.loc.gov">
  ...
  <entry>
    ...
    <loc:checksum>3c89ea593c01483fd091</loc:checksum
    ...

On Fri, Jun 8, 2012 at 6:04 AM, Ed Summers <[email-masked]> wrote:

> [2]
> http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-abstract.xml-rss.xml

---

From: James M Snell <[email-masked]>
Date: Fri, 8 Jun 2012 07:46:53 -0700

> doesn’t feel like any kind of a link to me.

I don't get it. Per the example, the reference to the data dump is
specified as an alternate link... how is your suggested element
different than the suggested extension attribute on the link element?

- James

---

From: Ed Summers <[email-masked]>
Date: Fri, 8 Jun 2012 13:15:41 -0400

>     <loc:checksum>3c89ea593c01483fd091</loc:checksum

Since there can (potentially) be multiple <link> elements per <entry>
it would be difficult to know which resource the checksum applied to
no?

//Ed

---

From: Ed Summers <[email-masked]>
Date: Fri, 8 Jun 2012 13:17:06 -0400

>            href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"/>

I like it, thanks for the suggestion.

//Ed

---

From: Tim Bray <[email-masked]>
Date: Fri, 8 Jun 2012 10:17:27 -0700

D'oh, right. So why not

<link loc:checksum="3c89ea593c01483fd091" ...other link apparatus...
href="" />

---

From: Ed Summers <[email-masked]>
Date: Fri, 8 Jun 2012 13:20:34 -0400

On Fri, Jun 8, 2012 at 1:17 PM, Tim Bray <[email-masked]> wrote:
> D'oh, right. So why not
>
> <link loc:checksum="3c89ea593c01483fd091" ...other link apparatus... href=""
> />

Because I'm a helpless idealist? :-)

//Ed

---

From: James M Snell <[email-masked]>
Date: Fri, 8 Jun 2012 11:10:00 -0700

For the sake of discussion, I have revitalized the draft...

http://www.ietf.org/id/draft-snell-atompub-link-extensions-09.txt

If there is specific feedback regarding any portion, let me know and
I'll make the necessary changes.

- James

---

From: Philippe Rathé <[email-masked]>
Date: Fri, 8 Jun 2012 15:19:50 -0400

Thanks James,

certainly interesting.

---

From: James M Snell <[email-masked]>
Date: Fri, 8 Jun 2012 13:12:53 -0700

Tim, I'm curious... do have a particular objection to just using the
hash attribute in the draft?

- James

---

From: Ed Summers <[email-masked]>
Date: Fri, 8 Jun 2012 16:21:16 -0400

Hi Tim,

> Because I'm a helpless idealist? :-)

In hindsight I should clarify this silly quip, and answer your
question more directly.

I would prefer to re-use a pattern that even a small handful of other
people might be using rather than go-it-alone with my own local
extension. The thought being that maybe the use of the hash attribute
would spread a bit wider (that's the idealistic side). But I do
appreciate that Atom allows a solution like loc:checksum, and am
grateful for it.

//Ed

---

From: Tim Bray <[email-masked]>
Date: Fri, 8 Jun 2012 13:23:46 -0700

And actually, I confess to not having read James’ draft since a cursory
glance at an early version, and had simply forgotten that he’d proposed a
hash attribute. -T

On Fri, Jun 8, 2012 at 1:21 PM, Ed Summers <[email-masked]> wrote:

> href=""