Subject: Please Review: Dissemination of Earthquake / Tsunami data via Atom

From: "Bob Wyman" <[email-masked]>
Date: Sat, 8 Jan 2005 18:49:48 -0500

At PubSub, we're beginning to put together a live feed of earthquake and
tsunami information in the hope that by making this information more widely
and rapidly disseminated, we'll be able to prevent at least a tiny amount of
the troubles such as recently happened in the Indian Ocean. In collecting
this data, we're not simply polling the USGS RSS feeds - doing that would
introduce too much latency in the process and such high latency or delays
would be unacceptable for this kind of data. What we're doing is connecting
to live "push" feeds of data and then executing on our servers the complex
and messy "clean-up", validation and reformatting processes that are
required. The instant we have good data, we push it out to our subscribers
using "Atom over XMPP/PubSub" and, as always, insert the latest messages
into our subscriber's personal Atom and/or RSS files. This service will be
made available for at least experimental use in a few days after we've
completed some testing. We currently expect to provide both traditional
PubSub subscription service as well as a "telnet feed" of raw data that
other service providers will be able to use to build their own systems.

We think it is appropriate to feed this kind of data over PubSub and in Atom
files but before we get too far down this road, I'd like to get some input
from the Atom community about the format we're thinking about using. A
sample Atom 0.3 entry from the current system is provided below. Please take
a look at it, provide whatever comments you consider appropriate and also
consider my questions below:

1.    The first thing to note is that the content is xml, not text or XHTML.
The thinking here is that we really want to be able to provide "data" to end
users since it is likely that the best way to use this data is to have some
kind of application (like a mapper, etc.) that would process the data. It
is, I think, less interesting to provide text. However, some folk will want
a machine generated textual version of this message. Since Atom only allows
us one content element, there is no way to do both data and text - or, am I
missing something? (Note: I do not want to convert this XML to XHTML.)

2.    I've been thinking that I might include an atom:summary element that
contained a textual version of the XML data carried in the content field.
However, it appears that most aggregators today display either the summary
or the content but don't display both...

3.    I've provided a 'rel="alternate"' link that points to a USGS generated
HTML page describing the event. Is it legitimate to point to an "alternate"
that I haven't created myself? 

4.    The title field provides some reasonably compact user data summarizing
the key data for the event. Would that be sufficient as a "textual
representation"?

5.    What should a client do when presented with this data? Clearly, there
are all sorts of interesting things that one can do if you know the format
and build a custom app. However, it would be nice to allow "dumb" clients to
do something useful without having to force the format to
lowest-common-denominator (i.e. HTML or XHTML). I think what I'd like to do
is provide a pointer to an XSL-template in some standard manner. Then, a
client could fetch the style sheet and generate something for display. Is
this reasonable? If so, where would I identify the style sheet?

6.    This kind of data is really sensitive. I would very much like to be
able to use a Digital signature to ensure that we can repudiate any fake
messages that might get generated by spammers and other idiots as well as to
repudiate mangled versions of the data that might get produced down-stream
of us. The subject of signing Atom entries doesn't seem to be getting much
attention. Can we wake this one up again? I really do not think that signing
should be considered something "outside the core" and subject to individual
or proprietary extensions.

I look forward to your comments and assistance in getting this system
working - hopefully, long before the next disaster in which it might be
useful.

bob wyman

Note: The message below was published long after the original event. The
reason for the delay is that this message adds to information previously
published - much closer to the time of the event. (i.e. the "AddOn" element
with the links to the waveforms was not available when the event was first
published.) If a tsunami warning was issued, it would also appear as an
"AddOn".

<entry>

<modified>2005-01-08T16:49:18-05:00</modified>

<issued>2005-01-08T16:49:18-05:00</issued>

<id><id>tag:pubsub.com,2005:EQ:ci14117540</id></id>

<title>M 1.8 SOUTHERN CALIFORNIA 2005-01-08T21:00:21.000Z</title>

<link
rel="alternate">http://earthquake.usgs.gov/recenteqs/Quakes/ci14117540.htm</
link>

<content type='text/xml' xmlns='http://pubsub.com/xmlns'>

<Message>

<topic>Earthquakes</topic>

<publisher>PubSub Earthquake Publisher</publisher>

<messageID>14117540</messageID>

<revisionID>2</revisionID>

<publication-date>2005-01-08T16:49:18-0500</publication-date>

<contentType>EQMessage</contentType>

<content>

<Earthquake MsgNumber="1411754002" MsgVersion="1.0" >

<DataMessage Action="Add" TimeReceived="2005-01-08T16:49:18-0500">

<Identifier EventIDKey="ci14117540" DataSource="CI" Version="1"/>

<Event Type="Earthquake" Version="1" MsgTypeCode="E " Latitude="34.6391"
Longitude="-118.573" Depth="8.3" NumPhases="27" MinDistance="5"
AzimuthalGap="57.6" RMSTimeError="0.28" HorizontalError="0.5"
VerticalError="1.8" Time="2005-01-08T21:00:21.000Z" LocationMethod="H"
Verified="N">

<Magnitude Value="1.8" Type="L" NumStations="16" MagError="0.3"/>

</Event>

<AddOn Type="Waveform_CI" Version="1"
Link="http://pasadena.wr.usgs.gov/review_eventfiles/14117540.html"
Description="Waveforms"></AddOn>

</DataMessage>

</Earthquake>

</content>

</Message>

</content>

</entry>

---

From: Antone Roundy <[email-masked]>
Date: Sat, 8 Jan 2005 18:34:59 -0700

> <content type='text/xml' xmlns='http://pubsub.com/xmlns'>
One quick comment--would not the above XML place <content> in the 
namespace "http://pubsub.com/xmlns" rather than the Atom namespace?  
I'd think you need something like this:

<content type='text/xml'>
<Message xmlns='http://pubsub.com/xmlns'>

---

From: Robert Sayre <[email-masked]>
Date: Sat, 08 Jan 2005 20:38:29 -0500

>or the content but don't display both...

The same is true of <content:encoded> and <xhtml:body> in the various 
RSS versions. I would include the summary.
One might reasonably expect they would show the summary if they don't 
support the content.

>this reasonable? If so, where would I identify the style sheet?

This has been brought up before, people felt it was a security risk 
equal to object tags and javascript.

>or proprietary extensions.

DSig and XMLEnc are in core.
http://atompub.org/2004/10/20/draft-ietf-atompub-format-03.html#rfc.section.7

Also, the pubsub namespace declaration should be on <Message>.

Robert Sayre

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 9 Jan 2005 12:42:29 -0500

I've received a number of off-list comments pointing to CAP (Common Alerting
Protocol) defined by the OASIS Emergency Management WG. (see:
http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=emergency) It is
suggested that CAP would be a more appropriate format for disseminating
Earthquake reports. I am very familiar with CAP but am quite sure that it
isn't appropriate as a format for us to use in this case.

I'd like to point out that it is not our intention at PubSub to defining an
"Alerting" protocol - in the sense that CAP is an "alerting" protocol. What
we are doing with earthquake data, and what we've been doing for some time
with Airport Closing/Delay information, is distributing raw data. The
issuance of an "alert" of the type that CAP defines requires a subjective
and authoritative decision typically made by a governmental or other
authoritative organization - or by some machine process sponsored by such an
organization. It is one thing to report that an earthquake has occurred. It
is quite another to alert people that some action in response to the event
is appropriate. We do the former, not the later.

I paid the OASIS fees and joined the CAP WG last year, commented extensively
on the specification before it was adopted and we've had CAP support built
into PubSub since last spring. However, the fact that we couldn't find a
good feed of CAP messages combined with our obvious reluctance to generate
them ourselves has prevented us from making the topic public. If we could
get a stream of CAP messages, we would publish them. However, publishing CAP
messages would not replace our publishing of raw Airport, Earthquake,
Tsunami or other reports that we might publish in the future such as weather
reports, "terror level" reports, flood warnings, automobile traffic reports,
etc. All of these messages may be appropriate for inclusion in a stream of
CAP Alerts - however, a raw feed such as we're publishing would contain many
more messages then one would expect to see issued by a CAP system. Also, by
providing specialized formats and service for each of these types, we are
able to support them in ways that a generalized system cannot.

The CAP protocol is, of course, very general and is not really designed for
carrying the sort of detail that is appropriate for a domain specific
message such as one focused on earthquakes. Typically, one would expect that
a domain specific message would contain much data which is of concern to
those who have a specific interest in the domain (academics, hobbyists,
etc.) but much of that detail would be irrelevant to the local emergency
managers, governmental agencies, and even members of the general public who
might be the recipients of CAP messages. It should also be noted that CAP
has not, to date, managed to get much input from non-US domain experts or
agencies. Thus, it really can't be considered an "international" standard at
this point - however, the market for PubSub data is decidedly international.

For those who are interested in learning more about CAP, I highly recommend
that you check out the OASIS group pointed to above. Also, I've included a
sample CAP message (from the draft CAP 1.1 specification) below to show you
the CAP approach to alerting. I think even a quick review of the CAP message
will show why it might not be appropriate for our immediate purposes. For
instance, I would be very concerned about PubSub.com making statements
concerning things like "scope", "urgency", "severity", etc. Also, we provide
structure (AddOns) in our messages that can't be easily mapped to the
name/value "parameter" mechanism which is all that CAP provides for
communicating event-type-specific data.

It may be that in the future, we may be convinced that it would be
appropriate to publish some subset of our event notifications as CAP
messages. However, I'm quite sure that even if we did so, we would still
find it appropriate to publish the full detail as domain specific messages.
If we get demand to publish CAP messages, or a high-quality source of such
messages published by others, we will certainly consider publishing the
stuff.

            bob wyman

<?xml version = "1.0" encoding = "UTF-8"?>

<alert xmlns = "http://www.incident.com/cap/1.1">

   <identifier>TRI13970876.1</identifier>  

   <sender>[email-masked]</sender>  

   <sent>2003-06-11T20:56:00-07:00</sent>

   <status>Actual</status>  

   <msgType>Alert</msgType>

   <scope>Public</scope>

   <incidents>13970876</incidents>

   <info>

          <category>Geo</category>    

          <event>Earthquake</event>    

          <urgency>Past</urgency>    

          <severity>Minor</severity>    

          <certainty>Observed</certainty> 

          <senderName>Southern California Seismic Network (TriNet) operated
by Caltech and USGS</senderName>

          <headline>EQ 3.4 Imperial County CA - PRELIMINARY
REPORT</headline>

          <description>A minor earthquake measuring 3.4 on the Richter scale
occurred near Brawley, California at 8:53 PM Pacific Daylight Time on
Wednesday, June 11, 2003. (This is a computer-generated solution and has not
yet been reviewed by a human.)</description>

          <web>http://www.trinet.org/scsn/scsn.html</web>

          <parameter>EventID=13970876</parameter>

          <parameter>Version=1</parameter>

          <parameter>Magnitude=3.4 Ml</parameter>

          <parameter>Depth=11.8 mi.</parameter>

          <parameter>Quality=Excellent</parameter> 

          <area>        

                <areaDesc>1 mi. WSW of Brawley, CA; 11 mi. N of El Centro,
CA; 30 mi. E of OCOTILLO (quarry); 1 mi. N of the Imperial Fault</areaDesc>

                <circle>32.9525,-115.5527 0</circle>   

          </area> 

   </info>

</alert>

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 10 Jan 2005 16:24:34 -0500

We've made a good bit of progress on cleaning up our PubSub
Earthquake/Tsunami reporting service and would like to ask folk on the list
to take a look at the feed we're generating. (Note: This isn't "publicly"
announced yet, so please restrict comments to personal email or the list.)

You can see a sample Atom feed of all recent earthquakes (and tsunami
reports in "AddOn" fields - when we receive them) at:

http://atom.pubsub.com/90/69/6169a3b8e062b66f7afd592f.xml

Use "View Source" to see the actual Atom file if you are using a browser
that supports style-sheets.

I'm sure there is much to be said about this kind of service; however, my
interest in sending this message to this group is to get comments on our use
of Atom for this application. Please note that in the near future we'll be
allowing people to access this data via "Atom over XMPP PubSub" for
real-time push distribution. Thus, you'll be able to say things like: "Only
send me quakes over 5.0 magnitude that occur in Southern California and have
'waveforms' data in AddOn records." We're also going to be supporting a pure
TCP/IP "telnet" socket connection for folk who want the full stream.
Naturally, REST, SOAP, Email, and other distribution mechanisms will either
be enabled by default (as with all other PubSub services) or could be
provided if demand exists. 

I am particularly interested in "Atom-related" comments at this point. For
instance, you'll notice "Delete" messages in the feed. We've talked about
"Delete" or "Retract" capability often in the past and I've usually argued
against it. However, it turns out to be necessary with this kind of feed
since often earthquake reports are retracted as erroneous - often as the
result of telemetry glitches. How should we best handle the need for
"Delete" in this context? What will be easiest for clients to handle?

Also, please be aware that I am aware of no existing standards for this
data. If you know of any such standards, please let me know. (Note: I am
aware of CAP and have participated in that forum. It solves a different
problem.)

Thanks in advance for your assistance. My hope is that we'll be able to use
Atom to make a real contribution to making the world just a bit safer.

                        bob wyman

---

From: Ziv Caspi <[email-masked]>
Date: Sat, 15 Jan 2005 23:12:57 +0200

Bob Wyman:

[[[
I am particularly interested in “Atom-related” comments at this point. For
instance, you’ll notice “Delete” messages in the feed. We’ve talked about
“Delete” or “Retract” capability often in the past and I’ve usually argued
against it. However, it turns out to be necessary with this kind of feed
since often earthquake reports are retracted as erroneous – often as the
result of telemetry glitches. How should we best handle the need for
“Delete” in this context? What will be easiest for clients to handle?
]]]

If you consider the Atom entry stream as a set of idempotent synchronization
messages, "delete" can be represented as an empty entry with the same id as
a previous entry. (This is a very nice model to work with, in my experience:
you have a set of inputs representing state, a set of inference rules, and
an output state. Note that regardless of whether the entry stream represents
a sampled state or discrete events, you want the loop to be closed at the
consumer side, according to end-to-end principle.) 

Ziv Caspi
  cell: +972-53-668-751
    web: http://radio.weblogs.com/0106548/

---

From: Danny Ayers <[email-masked]>
Date: Sat, 15 Jan 2005 22:50:36 +0100

> consumer side, according to end-to-end principle.)

To save a bit of searching:

Empty content for deleted:
http://www.imc.org/atom-syntax/mail-archive/msg11456.html

Explicit <deleted/>
http://www.imc.org/atom-syntax/mail-archive/msg08585.html

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Sat, 15 Jan 2005 18:00:39 +0000

um... cancel messages?

look, errors or state change repudiation is latent in all reported state.
why "un-write" when you can "later-write"?

---

From: James Snell <[email-masked]>
Date: Mon, 17 Jan 2005 14:58:12 -0800

What I'm wondering here is whether or not a <retracted /> or <retracts
/> element could be used for this purpose.  The <retracted /> element
would be used within an entry to indicate that the entry has been
retracted.  The <retracts /> element would be used within a second
entry to identify it as a retraction for some preceding entry.  For
example:

<entry>
   ...
  <id>urn:my-entry-1</id>
  <retrated by="urn:my-entry-2">2005-01-07T12:00:00-8:00</retracted>
   ...
</entry>

<entry>
  ...
  <id>urn:my-entry-2</id>
  <retracts>urn:my-entry-1</retracts>
</entry>

The retraction elements can be defined externally to the Atom core.

On Sat, 15 Jan 2005 18:00:39 +0000, Eric Brunner-Williams in Portland

-- 
- James Snell
  http://www.snellspace.com
  [email-masked]

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 17 Jan 2005 19:36:49 -0500

James Snell wrote:
> The <retracted /> element would be used within an entry to indicate that
> the entry has been retracted.
	I must admit that I find myself going around in circles concerning
this issue of retractions. I may be overly analyzing this. In any case, the
following is my current thinking (subject to change...)

	An Atom feed is a record of a stream of entries that have been
published over time. Once a thing is published, that fact cannot be erased,
retracted, etc. While you can't erase a message previously published, you
can make it unavailable by removing it from the current feed and you can
supercede previously read copies of the entry by publishing a new entry that
shares the same atom:id as the original.
	I think <retract/> would only make sense if there was some semantic
associated with <retract/> that wasn't implied by republishing the entry. I
can't think of any useful semantics that replacement doesn't provide.

	There is, however, one problem that bothers me a bit... Atom
requires that an entry has a "rel='alternate'" link. However, when I'm
trying to remove something (such as an erroneous earthquake report), I want
to delete any HTML pages that might have been created for the deleted event.
Once something is deleted, I can see that I need to communicate that fact in
the feed, however, it doesn't seem right that I should have to clutter my
site with all sorts of dead pages... Or, should I just have the retracting
entry point to a standard "Entry Retracted" page that is shared by all
retracted entries? (i.e. http://example.com/no_such_page.html )

		bob wyman

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 18 Jan 2005 09:12:09 -0700

In article <[email-masked]>,

> retracted entries? (i.e. http://example.com/no_such_page.html )

If the page once existed (especially if the topic was something as 
potentially momentous as a natural disaster warning), to me it seems 
best that attempting to retrieve the page would result in a redirect to 
the correct information, not a 404. Further thoughts on linkrot here:

http://www.useit.com/alertbox/980614.html

Peter