Subject: Published extensibility Paces

From: Tim Bray <[email-masked]>
Date: Wed, 10 Nov 2004 15:04:47 -0800

I had a talk about Atom and extensibility with Dave Orchard this 
morning, and he convinced me that there is benefit in a must-understand 
facility, but then educated me as to how complex it can be to 
implement.

Based on that discussion, I have just published 
PaceMustUnderstandElement and PaceExtendingAtom.  Note that the WG 
could reject PaceMustUnderstandElement and I think that 
PaceExtendingAtom would still work.

For convenience:
  http://www.intertwingly.net/wiki/pie/PaceMustUnderstandElement
  http://www.intertwingly.net/wiki/pie/PaceExtendingAtom

   -Tim

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 10 Nov 2004 23:41:59 +0000

>  http://www.intertwingly.net/wiki/pie/PaceExtendingAtom

+1 to PaceExtendingAtom. These comments are mostly editorial.

"9.2 Extensions To the Atom Vocabulary

Future versions of this specification may add new elements and 
attributes to the Atom markup vocabulary."

The term 'vocabulary' should be defined in parens somewhere - I 
assume it implies all the names in a given namespace... ?

Please change,

"Software written to conform to this version of the specification 
will not be able to process such markup correctly and, in fact, will 
not be able to distinguish it from a markup error"

above, to,

"Software written to conform to this version of the specification 
will not be able to process such markup correctly and might not be 
able to distinguish it from a markup error"

I'm pretty sure I can write code to not treat 'foreign elements' as 
markup errors in the absence of ver/ext policies. The current text 
overstates the case. Anyway, the last sentence below,

"9.3 Software Processing of Foreign Markup

Software processing an Atom Document which encounters foreign markup 
in a location that is legal according to this specification MUST NOT 
stop processing or signal an error. It may be the case that the 
software is able to process the foreign markup correctly and does 
so. Otherwise, such markup is termed "unknown foreign markup"

contradicts the "will not be able to distinguish it from a markup 
error" assertion. Otherwise, I don't see a need to distinguish 
'foreign markup' and 'unknown foreign markup'. It's overcomplicated 
- either you spec to allow new stuff to pass at certain points or 
you spec to fail if new stuff attempts to pass. Speccing to allow 
stuff to pass where it couldn't pass before implies a new vocabulary.

cheers
Bill

---

From: Robert Sayre <[email-masked]>
Date: Wed, 10 Nov 2004 18:55:13 -0500

>  http://www.intertwingly.net/wiki/pie/PaceExtendingAtom

I'm having a hard time parsing the requirements on markup appearing 
somewhere not mentioned by the spec. For example, markup from future 
versions of Atom could appear in atom:feed, but would be 
indistinguishable from foreign markup to an Atom 1.0 processor. The spec 
doesn't say anything about processor behavior when foreign markup is 
encountered in an "illegal" location. Could we say something like the 
following?

"When foreign markup is encountered in another location, software 
behavior is undefined."

It's ok to halt and catch fire, ok to do something with it, ok to ignore 
it, and ok to preserve it.

Also, would a signed feed require a DSig element in atom:feed?

Robert Sayre

---

From: Tim Bray <[email-masked]>
Date: Wed, 10 Nov 2004 16:08:59 -0800

On Nov 10, 2004, at 3:55 PM, Robert Sayre wrote:

> indistinguishable from foreign markup to an Atom 1.0 processor. The 
> spec doesn't say anything about processor behavior when foreign markup 
> is encountered in an "illegal" location. Could we say something like 
> the following?
>

I'm OK with that, but do note that the proposed section 9.2 says that 
unrecognized atom stuff may appear anywhere.  But if I encounter a 
<link> element that's weirdly non-empty and contains markup from some 
other namespace, that's the kind of situation you're talking about.  I 
think it would be OK to leave behavior undefined as you say.

> It's ok to halt and catch fire, ok to do something with it, ok to 
> ignore it, and ok to preserve it.
>

Yep.  But I think we already covered that, right? -T

---

From: Robert Sayre <[email-masked]>
Date: Wed, 10 Nov 2004 19:27:36 -0500

> Yep.  But I think we already covered that, right? -T

Yes. I had wrongly conflated the Atom namespace with "Atom's XML markup 
vocabulary."

Robert Sayre

---

From: Tim Bray <[email-masked]>
Date: Wed, 10 Nov 2004 17:18:22 -0800

On Nov 10, 2004, at 3:41 PM, Bill de hÓra wrote:

> The term 'vocabulary' should be defined in parens somewhere - I assume 
> it implies all the names in a given namespace... ?

It's formally defined in the intro to the Namespaces rec... probably 
should have a reference to that. -Tim

---

From: Joe Gregorio <[email-masked]>
Date: Wed, 10 Nov 2004 22:48:18 -0500

> PaceExtendingAtom would still work.

-1 on PaceMustUnderstandElement, for the following reasons:

1. There is a mustUnderstand feature in SOAP and has proved of no value.
    You may attribute that to there being very few compliant implementations
     in SOAP, but that only leads me to my next point:
2. This is added complexity to the format that few, if any, developers
will implement.
    As you report Dave Orchard said, this is complex to implement.
3. The Pace, as written, doesn't specify any concrete benefit. There is
   only theoretical benefit. What situation, current or in the past, in
   syndication formats, would a mustUnderstand element have helped?

    -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: Henry Story <[email-masked]>
Date: Thu, 11 Nov 2004 11:09:37 +0100

-1 on PaceExtendingAtom

It looks like this would not allow the proposal I have just made that 
permits Atom to be extended into RDF. It is unnecessarily complex, over 
engineered, and even contradictory. Here is a much simpler proposal:

You can extend Atom any way you want (with foreign namespaced 
elements), as long as the following conditions are met:
	1- if all those elements are removed, you end up with a valid Atom 
document X
	2- the meaning of the document X is not incompatible with the 
unabbreviated document

(2) need hardly be mentioned, because it will be assumed that your 
document needs to be parsable by simple Atom parsers, and so it would 
be foolish to try to say something that most parsers would 
misunderstand.

That seems much more simple and much clearer. Anything more requires 
serious justification.

Henry Story

On 11 Nov 2004, at 00:04, Tim Bray wrote:

> morning, and he convinced me that there is benefit in a 
> must-understand facility, but then educated me as to how complex it 
> can be to implement.
>

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 11 Nov 2004 11:13:48 +0000

Henry Story wrote:

> document needs to be parsable by simple Atom parsers, and so it would be 
> foolish to try to say something that most parsers would misunderstand.
> 
> That seems much more simple and much clearer. 

No it doesn't. I have no idea what is meant by 2.

> Anything more requires serious justification.

I see nothing here to back that claim up.

cheers
Bill

---

From: Henry Story <[email-masked]>
Date: Thu, 11 Nov 2004 12:53:19 +0100

On 11 Nov 2004, at 12:13, Bill de hÓra wrote:
>
> [...] I have no idea what is meant by 2.

The wording of this can be improved clearly. But let me start with an 
example. Take the document with the following sentence:

Doc 1.
A. Jack has a house

You can add a sentences to Doc 1 to get Doc 2

A. Jack has a house
B. Jack a du travail.
C. Jack est marié.

B and C are in French. When you understand French you understand 
something more than when you understand english. But they have not said 
anything incompatible with Doc 1. They have just added information.

B says: Jack has work and
C says: Jack is married

You could have stripped Doc 2 of B and C and had still had something 
usable.

So the point is: whatever extensions you add to an Atom document, a 
parser that parses it as a pure Atom document cannot be 
misunderstanding the document. Ie: he is not liable to anything.

I am sure that with a little effort we can work out how to put this in 
one clear crisp sentence.

> I see nothing here to back that claim up.

We are working on a standard where reaching consensus is the goal. 
Adding any type of limitation requires serious justification. I don't 
see any at all at present for the PaceExtendingAtom proposal. I think 
this is just a simple matter of methodology: don't close options you 
don't have to.

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 11 Nov 2004 13:43:25 +0000

Henry Story wrote:

> parser that parses it as a pure Atom document cannot be misunderstanding 
> the document. Ie: he is not liable to anything.

I find the english examples 1) not pertinent, 2) misleading*. I'm 
also inclined to avoid terms like "meaning", "misunderstanding", 
"understanding " - they're highly subjective and don't strike me as 
operationally useful. I still have no idea what 2 means.

Anyway, this won't cater for the situation where the publisher wants 
to insist you do not process the document without be able to process 
a specified namespace. Tim's proposal does.

> one clear crisp sentence.

Then you should do that :) But even if you did, I doubt that you 
will be able to constrain things in the wild to be suitably  monotonic.

> see any at all at present for the PaceExtendingAtom proposal. 

I do. It's simpler than anything I've seen to date.

cheers
Bill

*  Atom documents are not collections of facts.

---

From: Henry Story <[email-masked]>
Date: Thu, 11 Nov 2004 15:28:27 +0100

On 11 Nov 2004, at 14:43, Bill de hÓra wrote:

> to insist you do not process the document without be able to process a 
> specified namespace. Tim's proposal does.

I was speaking particularly about PaceExtendingAtom [1], and was not 
commenting about
PaceMustUnderstandElement [2] where Tim Bray describes what you are 
mentioning. My proposal does not, on first glance, contradict 
PaceMustUnderstandElement.

The problem I have with Tim's Proposal on PaceExtendingAtom is that it 
limits where new
attributes can appear. I want them to be able to appear anywhere: in 
the feed, in the entries, etc... I don't see why the specification 
should try to limit this, other than
state that since this is an Atom document, any document containing 
externally namespaced elements (without a MustUnderstand element) 
should be a monotonic extension of the stripped document.

>> I am sure that with a little effort we can work out how to put this 
>> in one clear crisp sentence.
>
> Then you should do that :) But even if you did, I doubt that you will 
> be able to constrain things in the wild to be suitably  monotonic.

Thanks for "monotonic". I see you understand my proposal much better 
than you will admit. :-)

So perhaps I should rephrase:

You can extend Atom with foreign namespaced elements and attributes in 
any position, as long as the following conditions are met:
	1- if all those foreign name spaced elements are removed from DocOrig, 
you end up with a valid Atom document (call it DocMin)
	2- DocMin should be a well formed Atom Document as stated in this spec
     3- unless a mustunderstand element is present, a parser parsing 
DocMin is correctly
	parsing DocOrig

Henry

[1] http://www.intertwingly.net/wiki/pie/PaceExtendingAtom
[2] http://www.intertwingly.net/wiki/pie/PaceMustUnderstandElement

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 11 Nov 2004 18:17:39 +0000

Henry Story wrote:

> I was speaking particularly about PaceExtendingAtom [1]

So am I.

> limits where new attributes can appear. 

I think this will not be a concern by the time we're done. I don't 
think anyone is looking for text of the kind "you cannot put extra 
things here". In a roundabout way, when I said that 'unknown foreign 
elements' weren't needed, this is what I was getting at. Having as 
many places as possible open for extension should be a good thing.

>     1- if all those foreign name spaced elements are removed from 
> DocOrig, you end up with a valid Atom document (call it DocMin)

I might try and synthesize this with PaceExtendingAtom as it may be 
just another way of phrasing the essence of the Pace. Tim?

cheers
Bill

---

From: Tim Bray <[email-masked]>
Date: Thu, 11 Nov 2004 11:01:31 -0800

>    syndication formats, would a mustUnderstand element have helped?

I am not passionate about PaceMustUnderstandElement, if the WG doesn't 
want it, I'm OK with that.   Having said that, the benefit is real easy 
to understand: it allows extensions to declare themselves compulsory.  
I can easily imagine extensions that might want to do this, but I grant 
that none have appeared yet.

The only place I disagree with Joe is his #2; I think that it's *not* 
hard to implement as specified.  It would be if you could have 
MustUnderstand attributes scattered throughout the feed as you can in 
SOAP.  In this proposal, you just have to read a list of namespace 
names at the front of the feed and exit if there are any there you 
don't know.

Questions for Joe:
1. Why do you think this is hard to implement?
2. Assuming I can persuade you that it's cheap, would you be OK with it 
even given the lack of use in SOAP?

  -Tim

---

From: Tim Bray <[email-masked]>
Date: Thu, 11 Nov 2004 11:04:39 -0800

On Nov 11, 2004, at 6:28 AM, Henry Story wrote:

> attributes can appear.

I tend to agree.  I see little value in the current spec language 
saying foreign markup can appear only here, here, and here.  I don't 
see how it adds value since MustIgnore is easy to implement.  Also, the 
language is incomplete because it talks only about foreign elements, 
not foreign attributes.

How would people react to a proposal to take these details out of the 
spec and lightly rewrite PaceExtendingAtom to allow "foreign" markup 
anywhere?  I really like changes that make the spec shorter :)
  -Tim

---

From: Walter Underwood <[email-masked]>
Date: Thu, 11 Nov 2004 11:41:49 -0800

About "must understand":

> even given the lack of use in SOAP?

Speaking as "Joe for a day": We parse no formats with compulsory
extensions. It just doesn't exist for PDF, MS Office, OpenOffice,
WordPerfect, whatever. You want to change it, you rev the version.

Implementing this require a new result from parsing: it is a legal
document, but we are not allowed to tell you what is in it.

Feeds are in a funny place between API and document. But I think
they are more document than API. APIs have "must understand", but
documents don't.

What does a search engine do when faced with a "must understand"
element? Throw away the whole thing and log it? Ignore the spec
and salvage what we can? Which one results in fewer support calls?

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 11 Nov 2004 19:44:12 +0000

> anywhere?  I really like changes that make the spec shorter :)
+1

cheers
Bill

---

From: Graham <[email-masked]>
Date: Thu, 11 Nov 2004 19:52:31 +0000

On 11 Nov 2004, at 1:43 pm, Bill de hÓra wrote:

> specified namespace.

If they don't want you to process it, what are they doing putting it on 
the internet? I'm really lost as to what this situation might be.

On 11 Nov 2004, at 6:05 pm, David Orchard wrote:

> 1. Atom does not have mU.  Somebody comes up with a really really good
> mandatory extension.  The only way to get this feature in is to Rev 
> Atom
> to Atom next namespace (or V 2.0)

What the bloody hell is a "mandatory" "extension" ?

> 2. Atom has mU.  Nobody in the entire community comes up with a really
> really good mandatory extension.  Nobody uses mU and Atom
> namespace/version also never gets created.

Meanwhile every Atom processor is churning through every element in 
every document looking for elements it can artificially fail on.

More seriously, define "understand"? Is it OK for me to "understand" an 
element I do nothing with? Am I going to have to compile a list of 
extensions I "understand" but am consciously ignoring?

Graham

---

From: Joe Gregorio <[email-masked]>
Date: Thu, 11 Nov 2004 15:03:25 -0500

Answering some of your questions brought up some 
more questions on how this would work:

1. Does mustUnderstand apply no matter what I am doing to the feed? 
 For example, combining multiple feeds into a single synthesized feed?
 How about validating a feed?

2. How does this relate to SOAP's mustUnderstand role in the
   protocol part. From my understanding of SOAP the 
   mustUnderstand attribute indicates a header that must be 
   processed. From the 1.2 SOAP spec:
   "Mandatory SOAP header blocks are presumed to somehow 
    modify the semantics of other SOAP header blocks or SOAP 
    body elements."

   In the case of the protocol that 'body element' is an Atom feed 
   or entry. What if I were truly evil and spec'd a mandatory SOAP 
   header whose purpose was to force the atom:must-understand 
   element to be ignored? 

   And this is just the evil I could come up with after
   a few minutes of thinking about it. Give me some time and I could 
   probably do much worse :)

   -joe

---

From: Tim Bray <[email-masked]>
Date: Thu, 11 Nov 2004 12:12:06 -0800

>  How about validating a feed?

Yes.  It's a simple binary on/off switch.  Anything else gets very 
hairy very fast.

>    header whose purpose was to force the atom:must-understand

A conformant processor of Atom documents cannot (if this were adopted) 
ignore atom:must-understand.  I mean, I could write a mandatory SOAP 
header that forced all h1, h2, and h3 elements to be suppressed in 
XHTML content, but that would be silly and vacuous.  I think I'm not 
getting your point. -Tim

---

From: Robert Sayre <[email-masked]>
Date: Thu, 11 Nov 2004 15:23:28 -0500

> document, but we are not allowed to tell you what is in it.

Do we have any use cases?

Robert Sayre

---

From: Antone Roundy <[email-masked]>
Date: Thu, 11 Nov 2004 13:31:32 -0700

>  How about validating a feed?
I would say "yes", it must always apply.

This also brings up something that should be added to the proposal: 
when aggregating feeds, if one contains mU, that must be carried into 
the aggregate feed with it unless no entries coming from that feed are 
affected by anything from that namespace.  (I say "not affected by" 
rather than "contains" because I suppose it's possible that someone 
could write an extension which adds elements outside of an entry which 
affect that entry--it may or may not be a bad design, but that doesn't 
make it impossible).

Another point: if <mU> were to appears inside <head> rather before it, 
and we were to accept PaceHeadInEntry, we'd need to decide/clarify 
whether it could appear in the <head> in an <entry>, or whether it 
would have to appear in the <feed>'s <head>.  As now written, that's 
not an issue, because <mU> appears before <head>, not inside it.

---

From: Danny Ayers <[email-masked]>
Date: Thu, 11 Nov 2004 23:02:12 +0100

> anywhere?  I really like changes that make the spec shorter :)

I think that's moving in the opposite direction to what Atom needs for
future growth. Constraints make compatibility between extensions
easier by clearly demarcating them.  After all, why are we using
markup in the first place?

I think Robert's (unfinished I think) PacePropertyDesign is promising,
I'll try and add another compromise Pace in the next few days.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Robert Sayre <[email-masked]>
Date: Thu, 11 Nov 2004 18:20:33 -0500

> I think Robert's (unfinished I think) PacePropertyDesign is promising,

I don't think so. I pretty much hit the wall with that one.

Robert Sayre

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 12 Nov 2004 01:33:08 +0000

> I don't think so. I pretty much hit the wall with that one.

What you have there will work wrt RDF. The entry as data dictionary 
is a good fit:

<atom:entry rdf:about="http://www.dehora.net/journal/2004/05/a?id">
   <atom:feed rdf:resource="http://www.dehora.net/journal/" />
   <atom:title>Thus sprach metadata</atom:title>
   <atom:link>
     <rdf:Description
       rdf:about="http://www.dehora.net/journal/2004/05/a.html"
       atom:rel="alternate"
       atom:type="text/html"
       atom:href="http://www.dehora.net/journal/2004/05/a.html" />
   </atom:link>
   <atom:id
      rdf:resource="http://www.dehora.net/journal/2004/05/a?id" />
   <atom:created>2004-05-23T01:45:36Z</atom:created>
   <atom:summary>Seairth Jacobs</atom:summary>
   <atom:author>
     <rdf:Description rdf:about="mailto:[email-masked]">
       <atom:name>dehora</atom:name>
       <atom:url rdf:resource="http://www.dehora.net/journal" />
       <atom:email>[email-masked]</atom:email>
     </rdf:Description>
   </atom:author>
   <dc:subject 
xmlns="http://purl.org/atom/ns#">SemanticWeb</dc:subject>

   ... more stuff here ...

</atom:entry>

IOW it's mostly transform happy.

Things like Person constructs are trickier - those wind up in their 
own rdf:Description blocks as above, making the markup ugly. Otoh, 
if you make these XML literals you hide a lot of good data from the 
RDF - it's a bit like using CDATA in XML.

Last time I looke at this, I hadn't figured out a clean way to 
express atom:content

cheers
Bill

[LOL. I'm currently shipping a system that has a model for 'events' 
that does exactly what PacePropertyDesign does with the value of the 
event - literals, xml or resources. It works well enough, even 
though the types are disjoint.]

---

From: Danny Ayers <[email-masked]>
Date: Fri, 12 Nov 2004 11:43:17 +0100

On Fri, 12 Nov 2004 01:33:08 +0000, Bill de hÓra <[email-masked]> wrote:

> own rdf:Description blocks as above, making the markup ugly. 

Not got time to look properly right now, but why not:

   <atom:author rdf:parseType="Resource">
      <atom:name>dehora</atom:name>
      <atom:url rdf:resource="http://www.dehora.net/journal" />
      <atom:email>[email-masked]</atom:email>
  </atom:author>

-- 

http://dannyayers.com

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Tue, 16 Nov 2004 04:07:09 +0100

On Thu, 11 Nov 2004 12:12:06 -0800, Tim Bray <[email-masked]> wrote:

> ignore atom:must-understand.

Which is why I dislike any «must understand» mechanism very much. It is a  
huge burden to have to comply to this for implementors, and simple XSLT  
transforms of Atom to whatever is suddenly not so simple anymore.

Extensibility is nice, but I don't really see any great value in making it  
possible for third-parties to extend Atom in an incompatible way. The way  
I see it; if you want to make an incompatible version of Atom, then just  
fork it. Create your own format in your own namespace and don't bother  
anyone with it.

If you want it to be compatible, you can't use «must understand» anyway,  
and need to use the other extensibility options we give in the  
specification, such as @rel values and namespaced elements.

Iow, I'm -1 on any «must understand» extensibility mechanism.

-- 
Asbjørn Ulsberg     -=|=-    http://virtuelvis.com/quark/
«He's a loathsome offensive brute, yet I can't look away»

---

From: Joe Gregorio <[email-masked]>
Date: Mon, 22 Nov 2004 10:37:54 -0500

> even given the lack of use in SOAP?

I thought about this more after we talked about
mustUnderstand at XML 2004 and tried to clarify
my objection to mustUnderstand. 

For me it comes down to this: mustUnderstand is a form of validation. 

For the Atom Format we've already rejected validity constraints 
and have settled for well-formed documents, and I believe that 
should hold no matter the validation method, either schema, DTD
or mustUnderstand.

    -joe