Subject: If you want "Fat Pings" just use Atom!

From: "Bob Wyman" <[email-masked]>
Date: Sun, 21 Aug 2005 18:59:54 -0400

The subject of "Fat Pings" or full content streaming from blogs has come up
on the FeedMesh list and in a proposal by Brad
<http://www.livejournal.com/users/brad/2143713.html>  Fitzpatrick of
LiveJournal. I've responded to the FeedMesh list suggesting that the best
way to move forward is to simply use Atom feeds rather than invent new
formats. See my response at:
http://groups.yahoo.com/group/feedmesh/message/451

The problem being addressed here is that of increasing the efficiency with
which feed search and/or monitoring services (like PubSub, Feedster,
IceRocket, Technorati, BlogDigger, etc.) obtain posts from the major blog
hosting platforms. In the past, most services have limited what they do to
simply sending pings. However, while the pinging mechanism works fine in the
case of low volume publishers, it simply doesn't scale to the requirements
of high volume publishers like the major blog hosting platforms
(LiveJournal, TypePad, Blogspot, Bryght, etc.). The problem is that when a
service is pinged, it must reach back to the pinging site and retrieve an
RSS or Atom file that probably contains many duplicate entries. The service
must then filter out the dupes before indexing, publishing, or matching the
"new" or "changed" items discovered. 

LiveJournal has lead for some time in showing a more efficient and effective
way for search engines to obtain new and changed postings. What they do is
produce an aggregate feed that contains copies of all entries written on any
of their public blogs. This feed typically contains as many as 200-300 new
entries per minute. But, while that might sound like a great many entries to
process each minute, it is massively less than the number of entries that
would need to be processed if LiveJournal were to rely on a simple pinging
process. The reason is that search engines can focus their ingestion
processors only on the aggregate feed and thus never need to deal with the
wasted bandwidth and processing that comes from duplicate entries. Of
course, LiveJournal benefits as well since the bandwidth and processing cost
of serving external search and/or monitoring systems is drastically reduced.
At PubSub, because of the way that LiveJournal publishes updates, we find
that the cost of processing LiveJournal updates is very much lower than the
cost to process entries from other blog hosting services that use
traditional content-free ping formats.

But, as with most RSS based systems, the LiveJournal system has been based
on a polling model. Given the speed with which the feed updates, services
like PubSub have been forced to read the LiveJournal feed at least once a
minute if not more frequently. Given that the entire (massive) feed must be
downloaded very frequently and given that LiveJournal does not currently
support RFC3229+feed
<http://bobwyman.pubsub.com/main/2004/09/using_rfc3229_w.html> , there are
inevitably duplicates that appear in the feed. Also, the polling services
never have any idea what the publishing rate is in the feed and thus can't
slack off the frequency with which they poll during "slow" periods. The
result is that as the rate at which LiveJournal's users begins to slacken
during "slow" periods, the percentage of duplicate entries increases.
Clearly, the solution to the problem is to move to a push feed. In this
case, LiveJournal would push the data updates to services that were
interested in them rather than forcing  those services to poll LiveJournal.

Brad has proposed a somewhat bent and extended version of Atom which would
be streamed over a TCP/IP connection in much the same way that we currently
stream FeedMesh data. (I've included a snapshot of his proposed format
below.) He defines an "AtomStream" and suggests that individual posts from
the various LiveJournal hosted blogs would be included in the stream as a
sequence of single-entry feeds. This is a solution that would work. However,
I suggest that there is actually no need to do anything other than "vanilla
flavored" Atom in order to address the needs here. A stream which began with
an atom:feed element and continued with a series of atom:entry elements that
contained atom:source elements would be a much more natural solution than
the "stream of feeds" that Brad proposes. (A sample of what I think a
"proper Atom" format for Brad's sample appears below.)

The problem being addressed by "Fat Pings" is very much like the one
addressed by the
<http://www.xmpp.org/drafts/draft-saintandre-atompub-notify-04.html> "Atom
over XMPP" protocol and is very much like the service that we provide at
PubSub.com <http://pubsub.com/> .  I believe it will be an important test of
Atom to determine if it is adequate to handle this sort of problem. I would
greatly appreciate comments from others on this use of Atom.

It should be noted that "Fat Pings" are probably only properly generated by
large, trusted blog hosting platforms. One of the essential elements of
controlling spam in feeds is the ability to trace back to an actual network
resource which can be used to verify the data in a "ping" and can be used,
to some extent, to identify the publisher of the data. For a service like
PubSub to forgo actual verification that an entry exists as claimed by a
ping, we would have to be able to trust the pinger. Normally, creating such
trust relationships is very expensive. However, given that the vast majority
of posts are made on the large services, we can drastically increase the
efficiency of the overall system by having just a few of these
hosters/publishers who are permitted the privilege of publishing Fat Pings.
It is my hope that in the future we'll be able to rely on Atom's support for
Digital Signatures to expand drastically the number of publishers who could
be trusted to publish Fat Pings.

Brad Proposes:

<?xml version='1.0' encoding='utf-8' ?>
<atomStream>
<time>1124247941</time>
<feed xmlns='http://www.w3.org/2005/Atom'>
<title type='text'>some journal title</title>
<link href='http://www.livejournal.com/users/username/' />
<author><name>some name</name></author>
<entry>
<title>some entry title</title>
<link href='http://www.livejournal.com/users/username/12345.html' />
<content type='html'>
content
</content>
</entry>
</feed>

I believe that the sample feed above would be better represented as a
"simple" Atom feed which contains entries having source elements. Note: My
sample is a bit bigger than Brad's since I've included various bits that are
required in Atom but that Brad's proposal omits. He readily admits in his
postings that he has not yet gone to the effort of ensuring that he is
issuing compliant data.

I propose the following as an equivelant to Brad's sample:

<?xml version="1.0" encoding="utf-8"?>

<feed xmlns="http://www.w3.org/2005/Atom">

   <title>LiveJournal Aggregate Feed</title>

   <link href="http://livejournal.org/"/>

   <updated>2005-08-21T16:30:02Z </updated>

   <author><name>Brad</name></author>

   <id>tag:livejournal.org,2005:aggregatefeed-1</id>

<entry xmlns='http://www.w3.org/2005/Atom'>

   <source>

     <title type=text>'Example Feed'</title>

     <link href='http://www.livejournal.com/users/username/'/>

     <link rel='self' type='application/atom+xml'

         href='http://www.livejournal.com/users/username/atom'/>

     <id>tag:livejournal.org,2005:feed-username</id>

     <updated>2005-08-21T16:30:02Z</updated>

     <author><name>John Doe</name></author>

   </source>

   <title> some entry title </title>

   <link rel='alternate' type='text/html'

        href='http://www.livejournal.com/users/username/12345.html '/>

   <id>tag:livejournal.org,2003:entry-username-32397</id>

   <published>2005-08-21T16:30:02Z </published>

   <updated>2005-08-21T16:30:02Z </updated>

   <content type="html">

         This is some <b>content</b>.

   </content>

</entry>

. . .

</feed>

            What do you think? Is there any conceptual problem with
streaming basic Atom over TCP/IP, HTTP continuous sessions (probably using
chunked content) etc.? Is there any really good reason not just to use Atom
as defined?

            bob wyman

---

From: "A. Pagaltzis" <[email-masked]>
Date: Mon, 22 Aug 2005 03:01:07 +0200

* Bob Wyman <[email-masked]> [2005-08-22 01:05]:
> streaming basic Atom over TCP/IP, HTTP continuous sessions
> (probably using chunked content) etc.?

I wonder how you would make sure that the document is
well-formed. Since the stream never actually ends and there is no
way for a client to signal an intent to close the connection, the
<feed> at the top would never actually be accompanied by a
</feed> at the bottom.

If you accept that the stream can never be a complete well-formed
document, is there any reason not to simply send a stream of
concatenated Atom Entry Documents?

That would seem like the absolute simplest solution.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Joe Gregorio <[email-masked]>
Date: Sun, 21 Aug 2005 21:34:24 -0400

Why not POST the Atom Entry, ala the Atom Publishing Protocol?

   -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 21 Aug 2005 21:49:07 -0400

> Why not POST the Atom Entry, ala the Atom Publishing Protocol?
	This would be an excellent idea if what we were talking about was a
low volume site. However, a site like LiveJournal generates hundreds of
updates per minute. Right now, on a Sunday evening, they are updating at the
rate of 349 entries per minute. During peak periods, they generate much more
traffic. Generating 349 POST messages per minute to perhaps 10 or 15
different services means that they would be pumping out thousands of these
things per minute. It just isn't reasonable.
	Using an open TCP/IP socket to carry a stream of Atom Entries
results in much greater efficiencies with much reduced bandwidth and
processing requirements. 
	At PubSub, we've been experimentally providing "Fat Ping" versions
of our FeedMesh feeds to a small group of testers. We publish messages at a
rate much higher than LiveJournal does -- since we publish all of
LiveJournal's content plus everyone else's. We couldn't even consider Fat
Pings if we had to create and tear down a TCP/IP-HTTP session to post each
individual entry.
	There are many situations in which HTTP would work fine for Fat
Pings. However, for high-volume sites, it just isn't reasonable. The key, to
me, is that we establish the expectation that the Atom format is adequate to
the task (whatever the transport) and leave the transport selection as a
context dependent decision. Thus, some server/client pairs would exchange
streams of Atom entries using the POST based Atom Publishing Protocol while
others would exchange essentially the same streams using a more efficient
transport mechanism such as streaming raw sockets or even "Atom over XMPP".

	bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 21 Aug 2005 21:59:26 -0400

Aristotle Pagaltzis wrote:
> well-formed. Since the stream never actually ends and there
> is no way for a client to signal an intent to close the connection,
> the <feed> at the top would never actually be accompanied by a
> </feed> at the bottom.
	This is a problem which has become well understood in the use and
implementation of the XMPP/Jabber protocols which are based on streaming
XML. Basically, what you do is consider the open tag to have a virtual
closure and use it primarily as a carrier of stream metadata. In XMPP
terminology, your code works at picking "stanzas" out of the stream that can
be parsed successfully or unsuccessfully on their own. In an Atom stream,
the processor would consider each atom:entry to be a parseable atomic unit.

> If you accept that the stream can never be a complete
> well-formed document, is there any reason not to simply send a
> stream of concatenated Atom Entry Documents?
	You could certainly do that, however, you will inevitably want to
pass across some stream oriented metadata and you'll eventually realize that
much of it is stuff that you can map into an Atom Feed. (i.e. "created
date", unique stream id, stream title, etc.). Since we're all in the process
of learning how to deal with atom:feed elements anyway, why not just reuse
what we've got instead of inventing something new.
	A rather nice side effect of forming the stream as an atom feed is
the simple fact that a "log" of the stream can be written to disk as a
well-formed Atom file. Thus, the same tools that you usually use to parse
Atom files can be used to parse the log of the stream. It is nice to be able
to reuse tools in this way... (Note: At PubSub, the atom files that we serve
to people are, in essence, just slightly stripped logs of the proto-"Atom
over XMPP" streams that they would have received if they had been listening
with that protocol. In our clients we can use the same parser for the stream
as we do for atom files. It works out nicely and elegantly.)

		bob wyman

---

From: Joe Gregorio <[email-masked]>
Date: Sun, 21 Aug 2005 22:37:10 -0400

> results in much greater efficiencies with much reduced bandwidth and

Why can't you keep that socket open, that is the default 
behavior for HTTP 1.1.

   -joe

---

From: James M Snell <[email-masked]>
Date: Sun, 21 Aug 2005 19:40:59 -0700

>transport mechanism such as streaming raw sockets or even "Atom over XMPP".
First off, as a general FYI, take a look at PaceSimpleNotify... the 
current version uses basic HTTP POSTs to send one or more individual 
atom:entry's to a remote endpoint.  I'm hoping that the folks on the 
protocol list will pick this up in discussion in the near future as it 
is something that I definitely want to see incorporated. 

Secondly, I believe that the format is more than adequate to support 
this kind of mechanism.  I do not believe that Brad's atomStream 
container is necessary.  Either just stream a bunch of atom:feed or 
atom:entry elements directly over an open TCP/IP or a persistent 
(keep-alive) HTTP connection.  By no means would I ever suggest a new 
HTTP connection for each ping.

- James

---

From: "'A. Pagaltzis'" <[email-masked]>
Date: Mon, 22 Aug 2005 05:02:11 +0200

* Bob Wyman <[email-masked]> [2005-08-22 04:00]:
> Basically, what you do is consider the open tag to have a
> virtual closure and use it primarily as a carrier of stream
> metadata.

Shades of SGML…

> You could certainly do that, however, you will inevitably want
> to pass across some stream oriented metadata and you'll
> eventually realize that much of it is stuff that you can map
> into an Atom Feed

OT1H, you could put this data in the stream as an empty but
complete Atom Feed Document served as the first complete entity
in the feed –

> A rather nice side effect of forming the stream as an atom feed
> is the simple fact that a "log" of the stream can be written to
> disk as a well-formed Atom file.

– but OTOH this is a pretty good point.

Of course, the question is whether it is really any more work to
receive an empty Atom Feed Document + X * Atom Entry Documents
and to insert the Entry Documents into the Feed Document for
storage.

Note that in the case of prepending an empty Atom Feed Document,
all fully received Documents are well-formed entities of their
own, so you don’t need a recovering XML parser that can implement
the “virtual closing element” semantic – all entities can be
processed with any run-of-the-mill XML parser.

Regards,

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 21 Aug 2005 23:08:04 -0400

> Why can't you keep that socket open, that is the default
	In some applications, HTTP 1.1 will work just fine. However, HTTP
doesn't add much to the high volume case. It also costs a great deal. For
instance, every POST requires a response. This means that you're moving from
a pure streaming case to an endless sequence of application level ACK/NAKs
that are simply replicating what TCP/IP already does for you. Also, the HTTP
headers that would be required simply don't contribute anything useful. The
bandwidth overhead of the additional headers as well as the bandwidth,
processing and timing problems related to generating responses begins to
look pretty nasty when you're moving at hundreds of items per minute or
second...
	One really good reason for using HTTP would be to exploit the
existing HTTP infrastructure including proxies, caches, application-level
firewalls, etc. However, I'm aware of no such infrastructure components that
are designed to handle well permanently open high-bandwidth connections. The
HTTP infrastructure is optimized around the normal uses of HTTP. This isn't
"normal." 
	One of the really irritating things about the current HTTP
infrastructure is that it is very "fragile." This is a problem that has
caused unlimited headaches for the folk trying to do "notification over
HTTP" (mod-pubsub, KnowNow, various HTTP-based IM/chat systems, etc.). The
problem is that HTTP connections, given the current infrastructure and
standard components, are very hard to keep open "permanently" or for a very
long period of time. One is often considered lucky if you can keep an HTTP
connection open for 5 minutes without having to re-initialize... Of course,
during the period between when your connection breaks and when you get it
re-established, you're losing packets. That means that you have to have a
much more robust mechanism for recovering lost messages and that means
increased complexity, network traffic, etc. The added complexity and trouble
can be justified in some cases; however, not in all cases.
	HTTP is great in some cases but not all. That's why the IETF has
defined BEEP, XMPP, SIP, SIMPLE, etc. in addition to HTTP. One protocol
model simply can't suit all needs at all times and in all contexts.
	Whatever... The point here is that Atom already has defined all that
appears to be needed in order to address the "Fat Ping" requirement whether
you prefer individual HTTP POSTs, POSTs over HTTP 1.1 connections, XMPP, or
raw open TCP/IP sockets. That is a good thing.

		bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 21 Aug 2005 23:24:45 -0400

Aristotle Pagaltzis wrote:
> Shades of SGML.
	No! No! Not that! :-)

He continues with:
> ... many good points ....

	Basically, there are many really easy ways that one can handle
streams of Atom entries. You could prepend an empty feed to the head of the
stream, you could use "virtual" end-tags, you could just send entries and
rely on the receiver to wrap them up as required, etc... But, since all of
these are really easy and none of them really gets in the way of anything
rational that I can imagine someone wanting to do, why not just default to
doing it the way it is defined in the Atom spec? In that way, we don't have
to create one more context-dependent distinction between formats. Complexity
is reduced and we can avoid having to read yet-another-specification that
looks very, very much like hundreds we've read before. If Atom provides all
we need, lets not do something else unless there is a *very* good argument
to do so.

	bob wyman

---

From: "'A. Pagaltzis'" <[email-masked]>
Date: Mon, 22 Aug 2005 05:39:35 +0200

* Bob Wyman <[email-masked]> [2005-08-22 05:25]:
> If Atom provides all we need, lets not do something else unless
> there is a *very* good argument to do so.

I’m not inventing anything. Atom Entry Documents are part of the
spec and Atom Feed Documents may legally be empty.

And a consumer of a stream according to your proposition does not
get away without implementing some special ability to be able to
interpret it correctly anyway – you are using “plain Atom” at the
cost of requiring a specifically abled XML parser.

My proposition requires a trivial amount of extra semantics
implemented at the application logic level; yours requires extra
semantics at the protocol logic level (the protocol being XML).

I think doing this in application logic is so cheap that reaching
for the protocol logic is unwarranted.

Regards,

---

From: James M Snell <[email-masked]>
Date: Sun, 21 Aug 2005 20:44:13 -0700

>we need, lets not do something else unless there is a *very* good argument
+1. The basic format gives us everything we need to enable this.  Even 
looking back over my PaceSimpleNotify proposal.. in which I introduce a 
notification element used to identify the action that has occurred on 
the element (e.g. create, update or delete), I can see that there really 
is no need to have that element. 

- James

---

From: Sam Ruby <[email-masked]>
Date: Mon, 22 Aug 2005 00:36:17 -0400

> That would seem like the absolute simplest solution.

I think the keyword in the above is "complete".

SAX is a popular API for dealing with streaming XML (and there are a
number of "pull parsing" APIs too).  It makes individual elements
available to your application as they are read.  If at any point, the
SAX parser determines that your feed is not well formed, it throws an
error at that point.

With a HTTP client library and SAX, the "absolute simplest solution" is
what Bob is describing: a single document that never completes.

Note that if your application were to discard all the data it receives
before it encouters the first entry, the stream from there on out would
be identical.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Mon, 22 Aug 2005 00:36:03 -0400

> Why not POST the Atom Entry, ala the Atom Publishing Protocol?

Essentially, LiveJournal is making this data available to anybody who
wishes to access it, without any need to register or to invent a unique API.

I can, and have, accessed the LiveJournal stream from behind both a
firewall and a NAT device.  Doing so requires the client to initiate the
request.  Therefore, if you really wanted to turn this around, the
client would need to initiate a POST, and the server would need to
return the "Fat Pings" as the response.

I talked to Brad - in fact, I had independently made the same suggestion
that Bob did.  Brad indicated that if there were clients with different
requirements, he was amenable to accommodating each - endpoints are cheap.

- Sam Ruby

---

From: "A. Pagaltzis" <[email-masked]>
Date: Mon, 22 Aug 2005 07:34:26 +0200

* Sam Ruby <[email-masked]> [2005-08-22 06:45]:
> SAX is a popular API for dealing with streaming XML (and there
> are a number of "pull parsing" APIs too).

Of course – using a DOM parser is impossible with either approach
anyway.

> With a HTTP client library and SAX, the "absolute simplest
> solution" is what Bob is describing: a single document that
> never completes.

That can be argued both ways, I think. The important point is
that the connection can and almost certainly will be closed in
the middle of an entry document.

With a single, endless document, the application will have to
backtrack, discarding events until it has thrown away the last
seen start-element event for the incomplete entry, then close the
feed element.

With a series of concatenated complete documents, it can simply
discard everything that belongs to the current incomplete Entry
Document. There are implicit checkpoints in the stream.

Regards,

---

From: Bill de hÓra <[email-masked]>
Date: Mon, 22 Aug 2005 14:58:58 +0100

Bob Wyman wrote:
> XML. 

It is: every XMPP server has a hack to deal with it ;)

> Basically, what you do is consider the open tag to have a virtual

How XMPP does that opening stanza thing and then proceeds to not close
it is not a design to be emulated (imvho). The opening stanza should
arrived and be closed.

> what we've got instead of inventing something new.

In the Atom case, I don't see why you need to keep the atom:feed open.
Just send it once and then send entries in the raw.

> as we do for atom files. It works out nicely and elegantly.)

For high load scenarios using Atom/XMPP to decouple the atom processing
from xmpp stream handling you can use the log you're talking about as a
poor-man's message queue.

cheers
Bill

---

From: Joe Gregorio <[email-masked]>
Date: Mon, 22 Aug 2005 10:26:59 -0400

> wishes to access it, without any need to register or to invent a unique API.

Ahh, I had thought this was a more dedicated ping traffic stream. The 
never ending Atom document makes much more sense now.

  Thanks,
   -joe

---

From: Walter Underwood <[email-masked]>
Date: Mon, 22 Aug 2005 09:25:48 -0700

> what Bob is describing: a single document that never completes.

Except that an endless document can't be legal XML, because XML requires
the root element to balance. An endless document never closes it. So, the
endless document cannot be legal Atom. Worse, there is no chance for error
recovery. One error, and the rest of the stream might not be parsable.

So, it is simple, but busted.

The standard trick here is to use a sequence of small docs, separated
by ASCII form-feed characters. That character is not legal within an
XML document, so it allows the stream to resyncronize on that character.
Besides, form-feed actually has almost the right semantics -- start a
new page.

wunder
--
Walter Underwood
Principal Software Architect, Verity

---

From: "A. Pagaltzis" <[email-masked]>
Date: Mon, 22 Aug 2005 18:40:29 +0200

* Walter Underwood <[email-masked]> [2005-08-22 18:35]:
> The standard trick here is to use a sequence of small docs,
> separated by ASCII form-feed characters. That character is not
> legal within an XML document, so it allows the stream to
> resyncronize on that character.

Ooh! I like this – simple and very clever.

Regards,

---

From: Bill de hÓra <[email-masked]>
Date: Mon, 22 Aug 2005 17:51:02 +0100

> Besides, form-feed actually has almost the right semantics -- start a

In XMPP, you can reset on seeing </atom:entry>, assuming you don't have
atom:feed left hanging. You probably won't need atom:feed there anyway -
feeds are very much artefacts of going over HTTP.

cheers
Bill

---

From: "A. Pagaltzis" <[email-masked]>
Date: Mon, 22 Aug 2005 19:15:28 +0200

* Bill de hÓra <[email-masked]> [2005-08-22 19:00]:
> In XMPP, you can reset on seeing </atom:entry>,

Really?

    <![CDATA[
    </entry>
    ]]>

Or maybe

    <![CDATA[
    <![CDATA[
    </entry>
    ]]>

These are probably the only exceptions (I might be missing some,
though), but they’re enough to demonstrate that you will need to
write a parser, even if only a relatively simple one.

Using a character which is illegal in XML and can never be part
of a well-formed document as a separator is a clever way to avoid
having to do *any* parsing *whatsoever*. You just scan the stream
for the character and start over when you see it, end of story.
No need to keep state or look for patterns or anything else.

Regards,

---

From: James M Snell <[email-masked]>
Date: Mon, 22 Aug 2005 10:35:45 -0700

>Ooh! I like this – simple and very clever.
+1.. this seems a very elegant solution.

- James

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 22 Aug 2005 13:40:52 -0400

Walter Underwood wrote:
> Except that an endless document can't be legal XML, because XML
> requires the root element to balance. An endless document never closes
> it. So, the endless document cannot be legal Atom. Worse, there
> is no chance for error recovery. One error, and the rest of the
> stream might not be parsable.
	It's not quite as bad as you suggest. In fact, in an Atom streaming
application, clients should be expecting that that </feed> element could
arrive at any moment. This is how the server would announce that it is
terminating the stream. Servers might want to do this to shed load,
shut-down for maintenance, etc. So, it isn't right to say that the document
"never closes." In fact, I think the reason for the closure will typically
be that there is a network error or that the client (or some infrastructure
component serving the client) voluntarily stops reading the document. In
these cases, it is reasonable to suggest that the client should have
procedures to recover a partial document.
	Also, it should be noted that error recovery can be simplified
greatly if one is working with chunked content flowing over continuous HTTP
sessions. If one makes a simple rule that HTTP Chunk data will only be
inserted into the feed *in between* atom:entry elements, then you are
provided with the "framing" data needed to recover from any parse errors
with individual entries. This simple rule could be adopted by those who are
streaming over raw TCP/IP sockets as well. Perhaps what we need is "Atom
over Continuous HTTP" to be defined in addition to the "Atom over XMPP"
specification.

	I'm afraid that the point that is being argued is a fairly fine
technical detail that really shouldn't be allowed to get in the way of
implementing practical and useful systems. We have a great deal of industry
experience with this sort of XML streaming model if only because the Jabber
folk and the fact that the IETF has approved XMPP as an RFC. These "endless"
documents are not actually endless. We simply need to import some existing
knowledge about how to do partial document error recovery and we're where we
need to be.

	bob wyman

---

From: Joe Gregorio <[email-masked]>
Date: Mon, 22 Aug 2005 14:01:45 -0400

> +1.. this seems a very elegant solution.

+1.

Indeed both solutions, the never ending feed, and the FF separated
entries both have their advantages. The FF separated stream
has the advantage of being able to synchronize mid-stream. The 
never ending feed has the advantage that you are only initializing
a SAX parser instance just once. 

Interestingly enough the FF separated entries method would also work 
when storing a large quantity of entries in a single flat file where
appending an entry needs to be fast.

   -joe

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 22 Aug 2005 11:21:19 -0700

Yep; an existance proof is server push, which is very similar (but  
not XML-based);
   http://wp.netscape.com/assist/net_sites/pushpull.html

On 21/08/2005, at 9:36 PM, Sam Ruby wrote:

> solution" is

--
Mark Nottingham   Principal Technologist
Office of the CTO   BEA Systems

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 22 Aug 2005 11:23:37 -0700

Just as a data point, this should become less of a problem as event- 
loop based HTTP implementations become more popular; with them, the  
number of connections you can hold open is only practically limited  
by available memory (to keep fairly small amounts of connection- 
specific state). This technique can allow tens to hundreds of  
thousands of concurrent connections, leading to multi-hour HTTP  
connections (if both sides want them).

On 21/08/2005, at 8:08 PM, Bob Wyman wrote:

> The
> connection open for 5 minutes without having to re-initialize...

---

From: Walter Underwood <[email-masked]>
Date: Mon, 22 Aug 2005 11:33:32 -0700

> appending an entry needs to be fast.

The original application was logfiles in XML.

wunder

---

From: Bill de hÓra <[email-masked]>
Date: Mon, 22 Aug 2005 20:39:03 +0100

>>In XMPP, you can reset on seeing </atom:entry>,

Really. And I've even seen the two test cases below...

>     </entry>

... which are cute. I forget I didn't have an opening tag, or a buffer
for the entry. Oh wait, I do ;)

> write a parser, even if only a relatively simple one.

You already have an XML parser, that's not the problem; the problem is
managing the stream buffer off the wire for a protocol model that has no
underlying concept of an octet frame. I've written enough XMPP code to
understand why the BEEP/MIME crowd might frown at it - manipulating
infosets right on top on sockets make for funky code and isn't my notion
of what "bits on the wire" should be ;)

[Incidentally this is a non-problem for APP because we're piggy backing
on HTTP octets...]

> No need to keep state or look for patterns or anything else.

I see all the +1s, but don't understand why reinventing multi-part MIME
with formfeeds as a special case for Atom is more attractive that an
infinite list of entries whose closing atom:feed tag never arrives.
Still, I think this discussion is valuable: it speaks volumes on the use
of XML for wire protocols, especially for the single document school of
thought.

cheers
Bill

---

From: Graham <[email-masked]>
Date: Tue, 23 Aug 2005 00:49:07 +0100

Just out of interest, how well does any of this work with caches and  
proxies?

Graham

---

From: Tim Bray <[email-masked]>
Date: Mon, 22 Aug 2005 17:29:38 -0700

> never ending Atom document makes much more sense now.

It's got another advantage.  You connect and ask for the feed.  You get

<feed xmlns="http://www.w3.org/2005/Atom">
  ....... goes on forever ........

and none of the entry documents need to redeclare the Atom namespace,  
which saves quite a few bytes after the first hundred thousand or so  
entries. -Tim

---

From: Justin Fletcher <[email-masked]>
Date: Tue, 23 Aug 2005 01:55:34 +0100 (BST)

On Mon, 22 Aug 2005, Tim Bray wrote:

> and none of the entry documents need to redeclare the Atom namespace, which 
> saves quite a few bytes after the first hundred thousand or so entries. -Tim

I'm a little confused by all this discussion of never-ending XML 
documents, mainly because my understanding is that without the 
well-formedness checks the content might as well be free form, and the 
elements within the document may rely on parts that have 'yet to arrive'.

Taking as an example the atom:author element, with the above example of a 
never-ending document any atom:entry elements which exist would be quite 
valid in containing no atom:author element because they're not required to 
have one if the atom:feed element contains such an author. And because the 
feed has not yet finished the reading application cannot know that the 
document is invalid (or not - the atom:author element may arrive at some 
point in the future).

What I'm trying to say is that a never-ending 'atom' feed is ...

   * not parsable as XML because it is not well-formed
   * not parsable as Atom because the Atom constraints may not be
     satisfiable (the author example is what I'm thinking of)

For very specialist use it might be sensible, but as Atom and XML are 
quite well defined as to what they contain and such a stream isn't Atom or 
XML.

-- 
Gerph <http://gerph.org/>
... I need a friend, to make me happy; Not stand here on my own.

---

From: James M Snell <[email-masked]>
Date: Mon, 22 Aug 2005 18:08:07 -0700

Justin Fletcher wrote:

> Taking as an example the atom:author element, with the above example 
> of a never-ending document any atom:entry elements which exist would 
> be quite valid in containing no atom:author element because they're 
> not required to have one if the atom:feed element contains such an 
> author. And because the feed has not yet finished the reading 
> application cannot know that the document is invalid (or not - the 
> atom:author element may arrive at some point in the future).
>
If an atom:feed contains an author element, it is required by the spec 
to appear before any atom:entry elements in the atom:feed. 
(http://www.atompub.org/2005/08/17/draft-ietf-atompub-format-11.html#rfc.section.4.1.1) 
so this isn't a problem.  Further, the spec really does not define any 
metadata that would be dependent on parts yet-to-arrive.  There could be 
a challenge with same-document links that use fragment identifiers, but 
that's about it.

- James

---

From: Joe Gregorio <[email-masked]>
Date: Mon, 22 Aug 2005 21:11:32 -0400

> elements within the document may rely on parts that have 'yet to arrive'.

Not at all:

"""The "atom:feed" element is the document (i.e., top-level) element
of an Atom Feed Document, acting as a container for metadata and data
associated with the feed. Its element children consist of metadata
elements *followed by* zero or more atom:entry child elements."""

http://atompub.org/2005/07/11/draft-ietf-atompub-format-10.html#rfc.section.4.1.1

   -joe

---

From: Justin Fletcher <[email-masked]>
Date: Tue, 23 Aug 2005 02:15:39 +0100 (BST)

On Mon, 22 Aug 2005, James M Snell wrote:

> Justin Fletcher wrote:
>
>> I'm a little confused by all this discussion of never-ending XML documents, 
>> mainly because my understanding is that without the well-formedness checks 
>> the content might as well be free form, and the elements within the 
>> document may rely on parts that have 'yet to arrive'.
>> 
> If an atom:feed contains an author element, it is required by the spec to 
> appear before any atom:entry elements in the atom:feed. 
> metadata that would be dependent on parts yet-to-arrive.  There could be a 
> challenge with same-document links that use fragment identifiers, but that's 
> about it.

Ah; I misread that in the specification. Thanks. It's just the lack of 
well-formedness that is an issue in my head then.

-- 
Gerph <http://gerph.org/>
... Things get better second time around.

---

From: James M Snell <[email-masked]>
Date: Mon, 22 Aug 2005 18:23:20 -0700

> well-formedness that is an issue in my head then.
Hey, that's better than what I can claim.  For me, it's usually the lack 
of well-formedness in my head that's the issue ;-)

- James

---

From: Tim Bray <[email-masked]>
Date: Mon, 22 Aug 2005 19:38:15 -0700

On Aug 22, 2005, at 6:15 PM, Justin Fletcher wrote:

> Ah; I misread that in the specification. Thanks. It's just the lack  
> of well-formedness that is an issue in my head then.

If you encounter a busted tag on the Nth entry, per the XML spec  
that's a fatal error and you can't process any more.  But I'm pretty  
sure that there's nothing in the XML specifications that says you  
can't process the material before the breakage occurs.  Given that  
the overwhelming majority of these are going to be machine generated  
by production applications, I suspect the amount of breakage at the  
well-formedness level to be low. -Tim

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Aug 2005 12:01:11 +0900

>No need to keep state or look for patterns or anything else.

Well, modulo character encoding issues, that is. An FF will
look differently in UTF-16 than in ASCII-based encodings.

Regards,   Martin.

---

From: Walter Underwood <[email-masked]>
Date: Mon, 22 Aug 2005 20:35:58 -0700

> look differently in UTF-16 than in ASCII-based encodings.

Fine. Use two NULs. That is either one illegal UTF-16(BE or LE) character
or two illegal characters in ASCII or UTF-8.

Of course, a transport level multi-payload system would be preferred.

wunder

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 23 Aug 2005 05:54:19 +0200

* Bill de hÓra <[email-masked]> [2005-08-22 21:45]:
> ... which are cute. I forget I didn't have an opening tag, or a
> buffer for the entry. Oh wait, I do ;)

Sure. I didn’t say it was impossible. I was just saying that you
have to do more than scan the stream for the sequence “</entry>”.

> You already have an XML parser, that's not the problem;

The point is that if you get broken XML at some point, the parser
may be left in a state where it never closes an entry. Using an
illegal-in-XML character allows resynching without reconnecting
to the stream.

> [Incidentally this is a non-problem for APP because we're piggy
> backing on HTTP octets...]

Which Fat Pings are doing as well…

> I see all the +1s, but don't understand why reinventing
> multi-part MIME with formfeeds as a special case for Atom is
> more attractive that an infinite list of entries whose closing
> atom:feed tag never arrives.

Purely for the resynchronization aspect. If you believe that it’s
not an issue, then sure, there’s a lot less difference between
the two options.

> look differently in UTF-16 than in ASCII-based encodings.

Depends on whether you specify a single encoding for all entries
at the HTTP level or not. For this application, I would do just
that, in which case, as a bonus, non-UTF-8 streams would get to
avoid resending the XML preamble over and over and over.

Regards,

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 23 Aug 2005 06:27:29 +0200

* Tim Bray <[email-masked]> [2005-08-23 02:35]:
> and none of the entry documents need to redeclare the Atom
> namespace, which saves quite a few bytes after the first
> hundred thousand or so entries. -Tim

Hmm.

That’s the first really solid pro-single-doc argument I see…

And one I don’t think can be argued about, /realistically/.

• The ^L separation makes a lot of sense for logfiles, but less
  so for TCP-carried connections which ensure a basic level of
  data integrity. (With logfiles on disk all bets are off.)

• “Just discard the last incomplete document” logic can be
  implemented without require separate documents by processing
  and flushing the stack so far whenever an atom:entry
  start-element event is seen.

I still dislike the idea of a “virtual closing tag” – it ain’t
1995 anymore, after all. And I still think separate documents
would be cleaner and would require a bit less special case logic
to process.

But this *is* an application where saving bytes is a serious
enough concern, and I suppose that so long as the only thing
missing is exactly one atom:feed end-element event, the special
case is simple enough that I guess it’s acceptable.

Ugh. Reality.

Regards,

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 23 Aug 2005 06:56:19 +0200

* Tim Bray wrote:
>that's a fatal error and you can't process any more.

That's a bit misleading, a fatal error just means that the XML
processor must report the error to the application and that the
processor is not required by the XML specification to continue
processing; doing so is however an optional feature and further
processing would be implementation-defined. So this scenario is
unconstrained by the XML specifications.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Bob Wyman" <[email-masked]>
Date: Tue, 23 Aug 2005 01:59:59 -0400

Bill de hÓra wrote:
> the problem is managing the stream buffer off the wire for a
> protocol model that has no underlying concept of an octet frame.
> I've written enough XMPP code to understand why the BEEP/MIME crowd
> might frown at it
	Framing is, in fact, an exceptionally important issue. Fortunately,
HTTP offers us some framing capability in the form of chunked delivery. This
is much more light weight than what BEEP provides since HTTP assumes TCP/IP
as a transport layer while BEEP did not.
	The HTTP chunked delivery method would be vastly superior to the
suggestions for doing thing like including form-feeds or sequences of nulls
as entry boundary markers. If you accept a simple rule that says that you
will insert HTTP chunk length markers between each entry sent in a
"never-ending" Atom file, you get something like the feed I show below.
Simply strip out the chunk length data prior to stuffing data into your XML
parser. If an entry appears to continue beyond a chunk boundary, discard
that entry and continue by reading the next chunk.

See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
for more information on this method. Note that RFC2616 says: "All HTTP/1.1
applications MUST be able to receive and decode the "chunked"
transfer-coding,..."
Note: the chunk lengths are not "correct" in the following example.

GET /never-ending-feed.xml HTTP/1.1

HTTP/1.1 200 OK
Date: Fri Apr 8 17:41:11 2005
Server: FeedMesh/0.1
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml; charset=utf-8

ab
<feed ...>
...

a8
<entry>
...
</entry>

93
<entry>
...
</entry>

And so forth until finally you get a </feed>, the connection closes, or you
close the connection.

This is simple, requires no new specifications and provides for robust error
recovery in that broken entries can be easily detected and discarded.

bob wyman

---

From: Henri Sivonen <[email-masked]>
Date: Tue, 23 Aug 2005 09:40:44 +0300

On Aug 23, 2005, at 07:27, A. Pagaltzis wrote:

> 1995 anymore, after all.

You seem to be thinking that an XML parser needs to consider the whole 
document before reporting data to the app. This is not the case.

If you have a document that lacks the end tag, the fatal error is hit 
when the data stream ends. All the subtrees rooted at children of the 
document element will be just fine. Try it with any streaming SAX 
parser.

In this case, the data stream ends only if either party decides to 
close the connection. The parser will never know how long the document 
would have been. From the parsers point of view it looks like a broken 
stream while reading a finite doc. There's nothing in the XML spec 
requiring the app to throw away the data structures it has already 
built when the parser reports the error.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Bill de hÓra <[email-masked]>
Date: Tue, 23 Aug 2005 09:33:53 +0100

> which saves quite a few bytes after the first hundred thousand or so 

Any value in using this to exercise the PaceBatch? If it dealt with the
use-case in terms of bulk transfer that would make it more compelling imo.

cheers
Bill

---

From: Tim Bray <[email-masked]>
Date: Tue, 23 Aug 2005 06:23:40 -0700

> That’s the first really solid pro-single-doc argument I see…

There's another.  You don't have to create a new XML parser for each  
of the entries.  In most programming environments, that's a time- 
saver. -Tim

---

From: Tim Bray <[email-masked]>
Date: Tue, 23 Aug 2005 06:25:48 -0700

> unconstrained by the XML specifications.

No.  See, http://www.w3.org/TR/REC-xml/#sec-terminology, under "fatal  
error". -Tim

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 23 Aug 2005 15:57:47 +0200

>No.  See, http://www.w3.org/TR/REC-xml/#sec-terminology, under "fatal  

Yes, exactly what I wrote...
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Antone Roundy <[email-masked]>
Date: Tue, 23 Aug 2005 07:58:21 -0600

> avoid resending the XML preamble over and over and over.

Of course, if you do that, you won't be able to keep signatures for 
entries originally published in an encoding other than the one you've 
chosen.

If one were to want to signal an encoding change mid-stream, how might 
that work with what's been proposed thus far?

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 23 Aug 2005 16:22:09 +0200

* Antone Roundy <[email-masked]> [2005-08-23 16:05]:
> Of course, if you do that, you won't be able to keep signatures
> for entries originally published in an encoding other than the
> one you've chosen.

Why is that a concern for a site publishing a stream of its own
users’ posts?

Regards,

---

From: Walter Underwood <[email-masked]>
Date: Tue, 23 Aug 2005 07:27:01 -0700

--On August 23, 2005 9:40:44 AM +0300 Henri Sivonen <[email-masked]> wrote:

> There's nothing in the XML spec requiring the app to throw away the data
> structures it has already built when the parser reports the error.

There is also nothing requiring it. It is optional. The only 
reqired behavior is to report the error and stop creating parsed
information. Otherwise, "results are undefined" according to the spec.

The spec does require that normal processing stop at the error.
The parser can make data past the error available, but it "must not
continue to pass character data and information about the document's
logical structure to the application in the normal way".

This still feels like a hack to me. An unterminated document is 
not well-formed, and is not XML or Atom. Doing this should require
another RFC that says, "we didn't really mean that it had to be XML."

wunder

---

From: Graham <[email-masked]>
Date: Tue, 23 Aug 2005 15:48:33 +0100

> Yes, exactly what I wrote...

"the XML processor must report the error to the application and that the
processor is not required by the XML specification to continue
processing; doing so is however an optional feature and further
processing would be implementation-defined"

vs

"Once a fatal error is detected, however, the processor must not  
continue normal processing (i.e., it must not continue to pass  
character data and information about the document's logical structure  
to the application in the normal way)."

Graham

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 23 Aug 2005 17:08:58 +0200

>to the application in the normal way)."

Yes, the normal way could be for example to have the illformed flag on
each event not set, and an unnormal way would be to have the flag set.
In particular, "the processor MAY make unprocessed data from the
document (with intermingled character data and markup) available to
the application" and there is no constraint on what the application
may or must not do with such data.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Tue, 23 Aug 2005 08:29:15 -0700

On Aug 23, 2005, at 6:57 AM, Bjoern Hoehrmann wrote:

>>> processing; doing so is however an optional feature...
>>>
>>

No.  The specification clearly *forbids* proceeding with normal  
processing.  If you get a busted element in a feed, you'd better  
close that connection and open another one. -Tim

---

From: Martin Duerst <[email-masked]>
Date: Wed, 24 Aug 2005 16:21:16 +0900

At 22:58 05/08/23, Antone Roundy wrote:
 >> For this application, I would do just
entries originally published in an encoding other than the one you've chosen.

Wrong, unfortunately. XML Signature requires to transcode
to UTF-8 before signing, exactly to protect against such
problems.

You can't change an encoding midstream in an XML entity.
You can use different encodings for different external
entities, though.

Regrads,  Martin.