Subject: Re: PaceIntrospection: is another file format really needed?

From: Beecher Greenman <[email-masked]>
Date: Wed, 30 Jun 2004 16:44:13 -0400

Ken MacLeod wrote:

> Beecher Greenman <[email-masked]> writes:
>
>
>>Again, I ask the following question: what exactly is a "feed"? Is it
>>a concrete representation of a site or an entry, or is it an
>>abstract representation of a list? I don't think we're going to be
>>able to get much further in this until we nail down this definition.
>
> This is the critical point, so I moved it up here and reply to it
> first.
>
> I agree in principle that a general solution would be nicer than a
> specific format.  I would prefer to go to a wholly general solution,
> one that can be used to enable syndication but isn't inherently
> syndication.  I'm with you on that part.
>
> But, no.  To me, atom:feed is a syndication feed.

Then the spec needs to be altered to fit your opinion, because that is
not what it says right now.

> Extending it to an
> index/archive format has shown its weaknesses for general tasks (which
> feed headers are the "main" ones?  why do the rest need all the
> required elements when they could point to a site or category origin?)
> Entries are being forced to be both log entries and metadata wrappers,
> and they're pretty oddly defined metadata wrappers.

I'll admit that some of the metadata definitions are odd, and not things
I entirely understand (my biggest question, perhaps, being atom:issued
versus atom:created). These, however, can be refined in their own
proposals. It is entirely possible, in fact, that too many elements are
currently required in the feed format. This can be refined further, as
changing an element's status from required to optional (or the reverse)
is far less painful on the development side than creating completely new
elements or worse, completely new formats.

However, this -required versus optional elements- speaks only to minor
details of a format, mistakes made when people tried to make it too
specific. Better to refine a format which is 95% of the way, rather than
start from zero and changing definitions in the process. Changing the
definition of a feed, as a dedicated introspection format will require
in order to justify its existence, has far-reaching implications which
are best left untouched.

>>Again, XML models are cheap for standards developers, but not for
>>standards implementors. The most enduring standards have proven to
>>be the ones which are powerful enough to be useful in a wide array
>>of tasks but simple enough to be relatively easy to implement.
>
> My analogy to database records earlier was misunderstood as some
> specific kind of implementation, it was not; it was a comparison of
> flexibility.  XML content models are as flexible to work with as
> database tables.  I can't speak to writing tools that map flexible
> formats to hard-wired objects as I prefer to use tools that expose the
> flexibility.  Maybe that is why I see XML constructs as cheap.

Understood. At the same time, that is all theory; as with so many
things, the reality often proves to not quite live up to the theory
behind it. I agree with you that XML data models are as flexible to work
with as database tables on paper. But just as database models, once
they're set, they're set, and although changing them on paper remains
easy, changing them in the real world becomes extremely painful.

It's important, therefore, to get both of these things right the first
time. The whole point of working groups is to figure out exactly what
that means. The current proposals sacrifice the current spec genericity
-based largely on what seems to be a misunderstanding of what a site is
versus what a feed is- for a goal which needs no definitions to be changed.

>>Once again, it's all a matter of doing the simplest thing that could
>>possibly work.
>
> There are two parts to the DTSTTCPW rule: 1) implement a new
> capability in the simplest, then 2) refactor.  [1]
>
> You don't get a chance to refactor much after you've published a
> public format or protocol.  DTSTTCPW only applies during the iterative
> development phase before final release.  This is why we must pay
> attention to long term use and extension.

I agree completely, which is why I believe that we must keep things as
generic as we can.

- Beech

---

From: Beecher Greenman <[email-masked]>
Date: Wed, 30 Jun 2004 17:03:22 -0400

>>>> Perhaps the other question to ask is, should there really be any
 >>>> difference?
 >> No, the quesiton to ask is: "Should the Atom format requirements be
 >> extended to cater for Introspection?"

No, it should not. The entire point behind my argument is that the 
format does not need to be extended at all; it already supports 
everything needed in order to be used for Introspection. I am arguing 
that we do not need a new format for Introspection, because we already 
have one that is perfectly usable and ready to go.

It's odd; both sides here are arguing that "just because you can, 
doesn't mean you should"; we're just saying it about different things. I 
say that just because we can define a new XML format doesn't mean we 
should, because a perfectly usable one already exists and we must be 
wary of unnecessarily complicating our standard.
Meanwhile, my opposition states that just because we can use feeds for 
introspection doesn't mean we should, because they see introspection as 
being somehow fundamentally different from syndication even if the data 
models are identical.

The only real difference is that I believe the difference between 
introspection (as we define it) and syndication to be an artificial 
distinction, a limitation which does not currently exist and should not 
be added.

 >>>> "Subscribing" to a feed is something that one specific type of user
 >>>> will do. However, we cannot expect all users of Atom to be human.
 >> Those statements are not exclusive.

True, and please forgive my poor word choice there. My point is that 
while most human uses of feeds will be limited to subscription -a point 
with which my opposition appears to agree- machines cannot be counted on 
to always follow that model. Some machines -such as browser-based 
aggregators- certainly will. However, not all machines can be counted on 
to do that.

For an example of a machine use of feeds which does not deal with 
subscription, let us consider MemeGen, a popular tool on LiveJournal. 
This tool scrapes the RSS feeds of users' journals and "does stuff" (the 
"stuff" being largely determined by the user) depending on the contents 
it finds.

Is this a trivial example? It probably is. At the same time, it does 
illustrate that there are uses for feeds outside of syndication and 
subscription. Now, let's throw introspection into the mix, such that a 
tool could scrape _all_ of a user's journals. LiveJournal does not offer 
this, but other blogging services do, and it's inevitable that at some 
point, MemeGen-like tools will be offered for these.

If we were to take this to another level, let us consider a sort of 
"recursive MemeGen" which could spider the links from a feed, thus 
gathering not only the words a user has posted on his own site, but 
words from sites which the user deemed important enough to link. In a 
case like this, syndication and introspection become practically one and 
the same thing.

 >>>> Indeed, part of the reason to use XML for this format is that it
 >>>> can be relatively easily consumed by both humans and machines.

 >> This is not the same argument as requiring a unified format.

Indeed not, but again, I am not proposing any changes to any formats. I 
am arguing that an existing format is essentially "unified" already. In 
fact, I'll go a step further and try to show how this "unification" is 
inherent in the natures of syndication and introspection, and cannot be 
taken out of the file as long as either one of these uses remains important.

What does an Introspection file need? If the PaceIntrospection format is 
to be believed, then it basically needs the following:
   - It must be able to cover multiple sites.
   - It must be able to present links to each site. In Introspection's 
case, at least three particular types of links are needed if they exist: 
Feed, Post, and an alternate format which will usually (though not 
necessarily always) point to an HTML-based page.
   - It must provide some way of telling one site apart from another.

The feed format already provides all of these things. Everyone agrees 
that the primary use case of the feed format is syndication; where we 
disagree is on whether or not syndication is its only valid function.
However, if the feed format were ever to lose one of these three 
capabilities, it would not only be unsuitable for introspection, but it 
would also be unsuitable for syndication, which also requires these 
three capabilities. It would make no sense to render a format unsuitable 
for its primary use case, so we basically have a guarantee that the feed 
format will stay usable for introspection as well as syndication.

Onward to the four points on which you requested clarification:

 >>>> Most machine-based uses are unlikely to work on a model of
 >>>> subscriptions, and so they need no such distinction.
 >
 >> 1.

My inclusion of the word "most" was a poor one, and I apologize for the 
wording there. However, I maintain that many machine-centric uses of 
Atom will not work on a model of syndication and subscription, but will 
still require both introspection and "syndication" feeds. It seems, 
therefore, that developing the feed format strictly for syndication is 
unwise and unnecessarily limiting.

 >>>> Indeed, many of the scenarios under which a machine will use
 >>>> introspection are similar -if not identical- to the scenarios in
 >>>> which they would use syndication.
 >
 >> 2.

I have named one scenario which uses syndication without subscription, 
and could use introspection in the same ways as syndication. Would you 
like more examples?

 >>>> At the same time, it is not the least bit harmful for them to have
 >>>> this ability if they desire it.
 >
 >> 3.

The above quite, to restore context, references the ability of a user to 
"subscribe" to an Introspection file as though it were a feed; I 
maintain that there is no harm in this.

If you believe there is harm, then show it to me. How is a user harmed 
by the ability to subscribe to introspection? I will assume that you are 
not limiting the scope of the harm to the user who is doing the subscribing.

Do you feel that there might be some kind of privacy issue involved? 
 From where I sit, this is the only potential possibility for harm. 
However, it remains an issue no matter what form of Introspection we 
eventually choose; programs which "subscribe" to Introspection files can 
be written even if we make it unnecessarily complex to do so. We could 
"lock" Introspection files behind authentication, but this same thing 
-indeed, any security measures one might take- could also apply to 
syndication feeds, and in fact would be useful in many of the same ways. 
  A wise user, if he is worried about the privacy of his Introspection 
file, will also be worried about the privacy of his syndication feeds.

 >>>> I do, however, see harm in forcing an artificial distinction
 >>>> between different kinds of lists of data onto machines when the
 >>>> data models are essentially the same.
 >
 >> 4.

I have already stated why the the data models for syndication and 
introspection are basically the same, and my opposition appears to agree 
with me on this point. Therefore, the question becomes, is there harm in 
artificially separating the specifications of two lists which follow the 
same model? I argue that from an implementation standpoint there is 
harm, because it increases the complexity of the standard by adding yet 
another format, when adding that format yields no real gain, in 
usability or otherwise. Complicating a standard for no gain is meaningless.

A couple of messages ago, Kai mentioned the problem that this provides 
no ability to tell whether a feed is for syndication or introspection 
without looking at the URL. I maintain that this is not an issue, 
because clients already have to know the URL in order to access the 
file. This is a consequence of the REST design which we have undertaken 
to use, and if we intend to continue along that paradigm, then the URL 
is certainly a valid place from which to get data about a request, just 
as a function name would be.

If there is truly a need to make this kind of type-identification 
self-contained, then the  problem is by no means unique to syndication 
vs. introspection. There are already two "kinds" of feeds, to use the 
terms my opposition has used:
   - The feed represents a site, and the entries represent articles on 
that site.
   - The feed represents an article, and the entries represent comments 
on that article.

It stands to reason that a third "kind" of feed will become popular in 
short order, even though it's not explicitly defined:
   - The feed represents a category of articles within a site, and the 
entries represent articles within that category.

In the case of feed-based introspection, we add a fourth "kind":
   - The feed represents a user, and the entries represent sites for 
which that user is responsible.

If there is really a need to distinguish between these "kinds" of feeds 
inside the feeds themselves -I maintain that there is no need, but let 
us assume that one arises- this could be dealt with a new attribute, 
perhaps "/atom:feed@type,". This could take values of "site", "article", 
"category", "introspection", or whatever other sort of feed might be 
needed. According to my opposition's logic this element is already 
necessary, to distinguish between "site feeds" and "article feeds". I do 
not follow this logic, but there it is.

 >> I need an explanation why Introspection support ought to be a
 >> requirement for the format.

My point is that Introspection support need not be a "requirement" for 
the format. the requirements for the format already make it inherently 
usable for introspection, and that this is not going to change unless 
the whole idea of syndication is scrapped. I maintain that feed-based 
introspection is a perfectly logical use of the existing standard, being 
-at least as far as Atom understands the term- nothing more than a 
different sort of syndication: that of sites instead of articles.

- Beecher Greenman