Subject: Fwd: I-D ACTION:draft-nottingham-atompub-fiql-00.txt

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 09:31:38 +1100

FYI; I'd love any feedback folks have on this.

Begin forwarded message:

> From: [email-masked]
> Date: 19 December 2007 7:15:02 AM
> To: [email-masked]
> Subject: I-D ACTION:draft-nottingham-atompub-fiql-00.txt
> Reply-To: [email-masked]
>
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
>
>
> 	Title		: FIQL: The Feed Item Query Language
> 	Author(s)	: M. Nottingham
> 	Filename	: draft-nottingham-atompub-fiql-00.txt
> 	Pages		: 16
> 	Date		: 2007-12-18
> 	
>   The Feed Item Query Language is a simple but flexible, URI-friendly
>   syntax for expressing filters across the entries in a Web feed.  It
>   also specifies a mechanism to allow feeds to indicate what types of
>   queries are supported.
>
>
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-nottingham-atompub-fiql-00.txt
>
> To remove yourself from the I-D Announcement list, send a message to
> [email-masked] with the word unsubscribe in the body of
> the message.
> You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
> to change your subscription settings.
>
> Internet-Drafts are also available by anonymous FTP. Login with the
> username "anonymous" and a password of your e-mail address. After
> logging in, type "cd internet-drafts" and then
> "get draft-nottingham-atompub-fiql-00.txt".
>
> A list of Internet-Drafts directories can be found in
> http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
> Internet-Drafts can also be obtained by e-mail.
>
> Send a message to:
> 	[email-masked].
> In the body type:
> 	"FILE /internet-drafts/draft-nottingham-atompub-fiql-00.txt".
> 	
> NOTE:	The mail server at ietf.org can return the document in
> 	MIME-encoded form by using the "mpack" utility.  To use this
> 	feature, insert the command "ENCODING mime" before the "FILE"
> 	command.  To decode the response(s), you will need "munpack" or
> 	a MIME-compliant mail reader.  Different MIME-compliant mail readers
> 	exhibit different behavior, especially when dealing with
> 	"multipart" MIME messages (i.e. documents which have been split
> 	up into multiple messages), so check your local documentation on
> 	how to manipulate these messages.
>
> Below is the data which will enable a MIME compliant mail reader
> implementation to automatically retrieve the ASCII version of the
> Internet-Draft.
> Content-Type: text/plain
> Content-ID: <[email-masked]>
>
> _______________________________________________
> I-D-Announce mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/i-d-announce

--
Mark Nottingham     http://www.mnot.net/

---

From: "Hugh Winkler" <[email-masked]>
Date: Tue, 18 Dec 2007 17:14:14 -0600

> >       Filename        : draft-nottingham-atompub-fiql-00.txt

I'm uncomfortable with prescribing how to construct URIs (sec. 4),
aren't you? It takes control of the URI namespace away from the
server. You can get the effect you want by POSTing an entity
containing the FIQL string (give it mime type application/fiql say)
and allowing the server to respond directly with the result, or return
303 to the URI of a cacheable representation. Let the server construct
the URI.

Hugh

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 10:26:57 +1100

On 19/12/2007, at 10:14 AM, Hugh Winkler wrote:
> aren't you?

Why? HTML forms does it, and they're pretty successful.

> It takes control of the URI namespace away from the
> server.

Just as with forms, this only specifies a format that's meaningful if  
the server chooses to use it; it doesn't foist itself upon the URL.

> You can get the effect you want by POSTing an entity

POSTing a query works, but for whatever reason, people don't like the  
added step. The draft doesn't rule that out, of course, but most  
people will want to put it in a URI; it's up to the server to decide  
how it wants to do it. It would make sense to define a media type for  
a FIQL query if people want to do this, though; I'll add that to the  
next draft.

Cheers,

---

From: "Hugh Winkler" <[email-masked]>
Date: Tue, 18 Dec 2007 17:38:57 -0600

> Why? HTML forms does it, and they're pretty successful.

Sure. But the server implementer has control over the eventual
constructed URL. The server can even send javascript down and
construct an URL without a query part. Either way, using HTML form or
javascript, the server provides the code that constructs the URL.

> the server chooses to use it; it doesn't foist itself upon the URL.

Not sure I understand what you're saying here. Looked to me like you
contemplate clients sending GETs to URLs they construct without any
guidance from the server?

> added step.

What extra step? the client just sends the FIQL string as an entity in
a POST rather than tacking it to the URL of a GET.

>The draft doesn't rule that out, of course, but most

Heck, Mark, I even threw in a cacheability argument ;)

-- 
Hugh Winkler, CEO
Wellstorm Development

http://www.wellstorm.com/
+1 512 694 4795 mobile (preferred)
+1 512 264 3998 office

---

From: "James Holderness" <[email-masked]>
Date: Wed, 19 Dec 2007 00:25:10 -0000

> By default, a selector is treated as an XML QName
> [W3C.REC-xml-names-19990114] which selects any and all child elements
> of the entry element which share the same syntax (i.e., the same
> prefix and localname; the namespace URI is not considered)

I'm not sure I follow this. If the selector is "title", and your entry looks 
like this:

<a:entry xmlns:a="http://www.w3.org/2005/Atom">
  <a:title>Test</a:title>
</a:entry>

does that mean you won't match anything?

Regards
James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 11:56:17 +1100

Yes; if you wanted to select on a:title, you'd use a:title (or, more  
likely, use the default namespace for Atom).

On 19/12/2007, at 11:25 AM, James Holderness wrote:

> I'm not sure I follow this. If the selector is "title", and your  
> entry looks like this:
>

---

From: "Hugh Winkler" <[email-masked]>
Date: Tue, 18 Dec 2007 19:03:06 -0600

[ part of this exchange happened off list, replying to list]
On Dec 18, 2007 6:55 PM, Mark Nottingham <[email-masked]> wrote:
>
>
> On 19/12/2007, at 11:41 AM, Hugh Winkler wrote:
>
> > On Dec 18, 2007 5:48 PM, Mark Nottingham <[email-masked]> wrote:
> >>
> >> On 19/12/2007, at 10:38 AM, Hugh Winkler wrote:
> >> As does it here, using the hints in fq:interface. Admittedly, it's a
> >> pretty constrained set of hints, but so are HTML forms, really.
> >
> > Hrrm. I hadn't picked up the forms-like nature of fq:interface
> >
> >> Right, but then it has to remember the Location that it gets back,
> >> and
> >> remember to fetch it in the future. I get a certain amount of
> >> resistance to this when I suggest this to devs. *shrug*
> >
> > They don't have to do that if they don't want to. They can just POST
> > the entity each time and consume the response. That's no more trouble
> > than doing GET, and in most cases no less efficient since the '?'
> > suppresses many caches.
>
> But the point is to enable caching...
>

Right. Well the most cacheable way is POST + 303. Then the devs do
have to record the Location header. But GET url + '?' + query string
won't cache much of the time. Maybe that's getting better, and I'm
sure you're more up to date with the state of the art. But my pretty
recent Squid doesn't do it out of the box.

>
> >> In any case, I'm happy expanding section 4 to include the POST case,
> >> and defining a media type.
> >>
> > not sure I feel so strongly about this that I'd want to muddy the
> > waters with alternatives. Let's see if there's any traction.
> >
> > (btw you replied off-list)
>
> Oops, sorry, feel free to forward.
>
>
>
> >> You nearly had me :)

Hugh

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 19 Dec 2007 08:12:29 +0700

Mark Nottingham wrote:

> I-D ACTION:draft-nottingham-atompub-fiql-00.txt 

Why not XPath or XQuery or SPARQL (with an Atom/RDF mapping), or CSS
selectors or some subset of one of those? I guess one reason is to
abstract away Atom vs. RSS and the other reason is that namespace
proliferation in Atom/RSS documents is out of control. Those are valid
concerns but I think making namespace handling even more insane is not
the answer.

"By default, a selector is treated as an XML Qname which selects any and
all child elements of the entry element which share the same syntax
(i.e., the same prefix and localname; the namespace URI is not
considered), along with their descendant content if any." 

That is not acceptable. First, namespace handling in feed-consuming
software is bad enough as it is. And, secondly, this provides an
unnuecesssary burden on the server to preserve Qnames and prevent Qname
conflicts.

In my software, I preserve the Qnames of attributes and elements as long
as the underlying XML (SAX and XSLT) stack provides enough information
to preserve them. Unfortunately, a LOT of XML stacks do not provide
enough information to preserve QNames exactly; Python's default SAX
parser (pyexpat) and most XSLT processors (in any language binding)
don't preserve Qnames accurately enough. As a result, the Qname of any
element or attribute might change as it moves between storage and the
HTTP interface and cannot be relied upon.

- Brian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 12:12:20 +1100

Not out of the box, but there's been discussion of changing it IIRC.  
The RFC says that queries shouldn't be cached if there isn't explicit  
freshness information, so there's still the opportunity. Browsers will  
cache queries now, BTW.

Cheers,

On 19/12/2007, at 12:03 PM, Hugh Winkler wrote:

> Maybe that's getting better, and I'm

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 12:45:39 +1100

On 19/12/2007, at 12:12 PM, Brian Smith wrote:

> selectors or some subset of one of those?

In a nutshell, there are two reasons;

1) Those query languages are optimised for data models that aren't  
feeds; respectively, XML Infosets, Infosets again, RDF graphs and CSS  
cascades. While it's possible to contort them to fit feeds, they don't  
really lend themselves to it. XQuery and SPARQL also present a fairly  
high barrier to adoption (if you're not a big XML vendor or a SW-head,  
respectively ;) Contorting them so that they're easy to fit into a URL  
isn't too attractive, either.

2) When you expose a query interface, you're allowing people to  
consume compute power on your servers. An arbitrary query language  
allows arbitrary queries, which is unacceptable when you're working  
across administrative domains. FIQL gives you tools to constrain how  
queries are shaped.

I've been asked this many times, and should probably add it as a FAQ  
in an appendix. Certainly there are use cases for using XQuery, etc.  
against feeds, but it's also become apparent that there's a place for  
something simple, reasonably flexible, and Web-friendly.

> software is bad enough as it is.

While true, I don't see how that's relevant.

> And, secondly, this provides an

I'm open to other suggestions, but this seemed to be the simplest way  
to go about it. Note that the 'name' attribute on the fq:index element  
gives you a way to specify a stable name for a selector that's  
independent of the QName; if you're concerned about these issues, you  
can use that.

Cheers,

---

From: Peter Keane <[email-masked]>
Date: Tue, 18 Dec 2007 20:06:09 -0600

This is of great interest to me, having spent the last few weeks 
devising a search query syntax for a digital library which makes 
extensive use of Atom/AtomPub.  Have you see the Contextual Query 
Language which is part of SRU? 
(http://www.loc.gov/standards/sru/specs/cql.html).  Atom is making 
inroads into the library space (note that the recently released OAI-ORE 
spec has an Atom serialization http://www.openarchives.org/ore/0.1/atom) 
and I would LOVE to see a well-thought out query language.  Perhaps FIQL 
is to OpenSearch as CQL is to SRU??  Any thoughts?

-peter keane
daseproject.org

---

From: Peter Keane <[email-masked]>
Date: Tue, 18 Dec 2007 20:11:55 -0600

> something simple, reasonably flexible, and Web-friendly.
+1 this hits the same sort of complexity sweet-spot (i.e., as simple as 
possible w/o being too simple + easily extensible) as Atom/AtomPub and 
thus, it seems to me, has pretty good "conceptual integrity" w/ those.

-peter keane

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 19 Dec 2007 14:54:44 +0700

Mark Nottingham wrote:
> > Why not XPath or XQuery or SPARQL (with an Atom/RDF 
> > mapping), or CSS selectors or some subset of one of those?
> 
> 1) Those query languages are optimised for data models that 
> aren't feeds; respectively, XML Infosets, Infosets again, RDF 
> graphs and CSS cascades. While it's possible to contort them 
> to fit feeds, they don't really lend themselves to it. XQuery 
> and SPARQL also present a fairly high barrier to adoption (if 
> you're not a big XML vendor or a SW-head, respectively ;) 
> Contorting them so that they're easy to fit into a URL isn't 
> too attractive, either.
>
> 2) When you expose a query interface, you're allowing people 
> to consume compute power on your servers. An arbitrary query 
> language allows arbitrary queries, which is unacceptable when 
> you're working across administrative domains. FIQL gives you 
> tools to constrain how queries are shaped.

Developers are used to seeing subsets of XPath. Almost every XPath
implementation I have seen has been a subset of XPath; Sometimes the
explanation was avoidance of DoS (resource consumption) attacks; other
times, the explanation was "we didn't have the resources to implement
all of it." So, I think it is reasonable to attempt to use a subset of
Xpath as a query language.

> Certainly there are use cases for using XQuery, etc.  
> against feeds, but it's also become apparent that there's a 
> place for something simple, reasonably flexible, and Web-friendly.

I agree. But, I'd rather borrow small parts from a well-specified,
widely-deployed, well-understood language than deal with a brand new
one. 

> > That is not acceptable. First, namespace handling in
> > feed-consuming software is bad enough as it is.
> 

It creates confusion. We spend a lot of time explaining to people that
they should always prefer to match elements by their namespace URI and
never by their prefix. This is making an unnecessary exception to that
rule.

> I'm open to other suggestions, but this seemed to be the 
> simplest way to go about it. Note that the 'name' attribute 
> on the fq:index element gives you a way to specify a stable 
> name for a selector that's independent of the QName; if 
> you're concerned about these issues, you can use that.

The language should not require implementations to match elements and
attributes using the prefixes used in the actual documents. XPath
expressions have a mapping of "statically known namespaces" [1] and the
default element namespace [2]. The fq:index could be the means of
setting up this mapping. Additionally, some extra syntax could be added
to support binding prefixes that are not bound in the fq:index.

I will throw out this suggestion for using a subset of XPath 2.0 as the
constraint language. 

Expr         ::= AndExpr  ( "or"  AndExpr )*
AndExpr      ::= CompExpr ( "and" CompExpr )* 
CompExpr     ::= Value ValueComp Value
               | FunctionCall
               | "(" Expr ")"
Comp		 ::= "eq" | "ne" | "lt" | "le" | "gt" | "ge"
Value        ::= Literal 
               | Path
               | "(" Value ")"
Path		 ::= QName
Literal      ::= NumericLiteral (as defined in XPath)
               | StringLiteral  (as defined in XPath)
FunctionCall ::= "matches" "(" Qname "," RegEx ")"
	         | "matches" "(" Qname "," RegEx "," "'i'" ")"
	         | "not" "(" FunctionCall ")"
RegEx 	 ::= "'" ".*"? CHAR* ".*"? "'"
Char		 ::= <any character except ".", "'", or """> | "\" <any
character>

I believe this language to be very similar to the language that you
specify in the draft, while simultaneously being a pure subset of XPath.
As you can see below, it isn't beautiful. But, I don't think that it is
so ugly that it requires a whole new language to replace it, either. 

The expression would always be considered to be a XPath predicate. That
is, an expression X would be evaluated as:
    atom:entry[X] 
and the resulting node list would be wrapped in an atom:feed element.
The default element namespace binding is the Atom namespace. Analogous
considerations apply to RSS. The default function namespace is
http://www.w3.org/2005/xpath-functions.

FIQL:
  title==foo*;(updated=lt=-P1D,title==*bar)
XPath:
 
(matches(title,'foo.*'))and((updated)lt('-P1D')or(matches(title,'.*bar')
))

FIQL:
  title==*new*,author==bob*
XPath:
  (matches(title,'.*new.*'))or(matches(author,'bob.*'))
  
FIQL:
  title==*great*;ex:rating=gt=4
XPath:
  matches(title,'.*great.*', 'i')or(ex:rating)gt(4)

Further, lets extend the syntax the subset to include a little more
XPath:

Value        ::= Literal 
               | Qname 
               | "(" Value ")"
               | "xs:dateTime" "(" Value ")"
               | "xs:duration" "(" Value ")"
               | "xs:decimal"  "(" Value ")"

Now, the client can explicitly request certain values to be treated as a
specific type in variables, regardless of what the fq:index elements do
or do not say. 

Further, we can expand the syntax of paths to allow us to select nested
elements and/or attributes or anything else that XPath allows. For
example, atom:content/@type.

Regards,
Brian

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 19 Dec 2007 16:04:46 +0100

>> http://www.ietf.org/internet-drafts/draft-nottingham-atompub-fiql-00.txt

Also http://tools.ietf.org/html/draft-nottingham-atompub-fiql-00

(Opinion, if any, to follow later.)

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: James M Snell <[email-masked]>
Date: Wed, 19 Dec 2007 10:15:15 -0800

While the xpath examples are definitely more complicated, I think in
this case I've got to agree with Brian.  Using an existing, known query
mechanism is better that inventing yet another one.  Also, given that
Atom is, in fact, XML, it makes a lot of sense to use a query language
designed for use with the XML Infoset.  It may not look as pretty and
compact as Mark's new syntax but it works, it already exists, there is
existing tool support, and it already covers many of the difficult
issues like namespaces and extensions.

If we do use xpath, I would suggest that the default namespace *always*
be mapped to the Atom namespace so users will never have to worry about
the namespace prefix.  In fact, all known *standardized* atom extensions
should have prefixes defined in the spec.  E.g. the query spec could say
that the prefix thr: always maps to the threading extension and that
xhtml: always maps to XHTML so that we don't have to worry as much about
 namespaces.  For more complicated cases, XPath provides the means of
specifying namespaces, even if it's not that pretty.

We would, however, likely need to provide coverage of things like
relative uri's.  For instance, if I submit a query like
"matches(link/@href,'http://example.org/foo')", it should be capable of
matching against elements like <link href="/foo"
xml:base="http://example.org/bar" />.  With base XPath, AFAIK, this is
not currently possible.

Also, if it is at all possible, it would be ideal if we could
automatically map the atom date elements (updated, published and
app:edited) to the xs:dateTime in XPath so we can simplify those
operations a bit, e.g.

  updated eq fn:current-dateTime

  as opposed to

  xs:dateTime(updated) eq fn:current-dateTime

Both should be possible but the former is definitely easier.

Regarding fq:interface, I'm not convinced the fq:index bit is necessary;
and I'd much rather see a more generic way of inserting URI Templates
into a feed.  For instance, this is an example of a syntax I've been
experimenting with:

  <x:link-template
    rel="search"
    pattern="/feed-search?{-join|&|expr,count}">
    <x:var name="expr" ref="http://.../feed-expression" />
    <x:var name="count" pattern="\d+" />
  </x:link-template>

Given this, assuming I understand that "http://.../feed-expression"
refers to the query spec, I could construct a query like:

/feed-search?expr=(matches(title,'.*new.*'))or(matches(author,'bob.*'))&count=10

There are several use cases for embedding link-templates within feeds
that would make having a generic mechanism ideal.

As for RSS feeds, I would suggest that a canonical mapping of RSS to
Atom be developed such that expressions written for one can be
applicable to the other.  I know that such a mapping would be imperfect,
difficult to produce, and likely be controversial, but it would be
certainly be helpful.  Not having this, however, would certainly not be
a show stopper.

- James

---

From: "Danny Ayers" <[email-masked]>
Date: Wed, 19 Dec 2007 19:51:14 +0100

Coupla quick comments -

I accept most of your arguments for not making the language fully
generalised (despite being a fan of both SPARQL & XPath). But what I'd
caution though is making the language more specific than it needs to
be, the other points being equal.

To avoid reinvention, a selective reuse of XPath constructs would seem
to make sense, with added sugar for Atom. Can your proposal be
systematically mapped to XPath?

Having said that, although XML-orientation seems obvious given Atom
(and the possibility of using URI path parts..? ok, I only skimmed
your draft), I wouldn't entirely dismiss SPARQL-like features. Typed
links *are* important in Atom, so some kind of triple patterns could
be very useful.

As another ref, I'd suggest checking out Microsoft's Project Astoria -
it's essentially a RESTful front end on top of an entity-relation
backend model. Like RDF but without URIs as entity/relation
identifiers (baby...bathwater...hmm). But one thing I saw a demo of
which looked very promising was their query URI construction. I can't
find the spec proper, but there are some examples here:

http://alexbarnett.net/blog/archive/2007/04/30/microsoft-codename-quot-astoria-quot-data-services-for-the-web.aspx

Cheers,
Danny.
-- 

http://dannyayers.com

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 19 Dec 2007 20:05:20 +0100

* James M Snell <[email-masked]> [2007-12-19 19:30]:
> If we do use xpath, I would suggest that the default namespace
> *always* be mapped to the Atom namespace so users will never
> have to worry about the namespace prefix.

There is no notion of default namespace in XPath.

Regards,

---

From: James M Snell <[email-masked]>
Date: Wed, 19 Dec 2007 11:29:27 -0800

I meant only, for example, that regardless of whether the underlying
feed uses "a:feed", "atom:feed", "foobar:feed", etc the xpath query
would always just be "feed".

- James

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 20 Dec 2007 02:21:00 +0700

A. Pagaltzis wrote:
> Sent: Thursday, December 20, 2007 2:05 AM
> To: 'Atom-Syntax'
> Subject: Re: I-D ACTION:draft-nottingham-atompub-fiql-00.txt
> 
> 
> > *always* be mapped to the Atom namespace so users will 
> never have to 
> > worry about the namespace prefix.
> 

Yes, there is in XPath 2.0:
  http://www.w3.org/TR/xpath20/#dt-def-elemtype-ns

Sorry I edited out the links in my original message.

- Brian

---

From: Addison Phillips <[email-masked]>
Date: Wed, 19 Dec 2007 14:42:26 -0800

Hi Mark,

I glanced through this document, which looks pretty good... but I have a 
concern that the ABNF doesn't permit Unicode characters, although the 
text does.

Specifically, you cite the 'unreserved' and 'pct-encoded' productions of 
RFC 3986 (URI). Here they are:

       unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"
       pct-encoded   = "%" HEXDIG HEXDIG

Obviously, one *could* use the pct-encoded production to incorporate 
random byte values into queries. But I would tend to expect that the 
syntax would be better off if defined in terms of 'iunreserved' in RFC 
3987 (IRI) instead:

   iunreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar

... and then mention the use of transfer encodings (provided in IRI) as 
needed (which you already do).

Best Regards,

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "David Powell" <[email-masked]>
Date: Thu, 20 Dec 2007 00:39:11 -0000

Have you looked at "RFC 2254 - The String Representation of LDAP Search
Filters"?  I've used variants of that in URIs before.
Its use of &'s is probably asking for escaping bugs in URIs though.

Some questions and comments -

Attribute matching

  Matching atom:category would be useful, but atom:category depends on
attributes - is that
  supported at all?  Can we have a special element comparison type for
atom:category that supports
  attribute matching?

Existence checks

  RFC 2254 supports checks that a property (or element) exists or not.
  I'd think something like that would be pretty useful.  Eg: draft=*

Matching against QNames

  QNames are evil.  I'd rather have inexact *:name matches, than explicitly
  matching against a namespace prefix.

  Would it be possible to write qname match predicates, but have the prefix
in the
  predicate have no significance, and then optionally allow another type of
predicate,
  for the anal, to requre that the prefix to be mapped to a certain URI?

    ex:name=="Dave"                                  ; matches *:name, but
    ex:name=="Dave",xmlns:ex=="http://example.com/"  ; matches a specific
element.

Person constructs

  Do element match predicates only apply to direct children of atom:entry?
Does "email" match
  atom:author/atom:email?

Inheritance

  Are element match predicates supposed to apply to the logical properties
of the entry
  after things like author inheritance rules have been applied?

fq:index

  Is the default that no selectors can be used?  If you define fq:index
elements
  does that mean that only those selectors can be used?  Is there anyway to
say that
  any selectors can be used?

-- 
Dave

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Dec 2007 13:01:17 +1100

Wow, really? Are all of your use cases behind a firewall / within the  
same administrative domain?

Cheers,

On 20/12/2007, at 5:15 AM, James M Snell wrote:

> into a feed

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Dec 2007 12:59:28 +1100

> Xpath as a query language.

Hmm, I'm not sure the latter is a great advertisement for basing it  
upon XPath...

To my mind, XPath 2.0 (which you seem to be referring to) is too  
complex to base something like this on; printing the XQuery-related  
specs kills too many trees, and it takes a major investment to get up  
to speed with it as a user, much less an implementer. While it works  
if you're already invested in the XQuery world, my perception is that  
this group is still fairly small, and "enterprise" focused.

I don't keep up to date with the XQuery world, but last I checked,  
Perl, Python, PHP and Ruby all lacked implementations. What will they  
do?

In comparison, FIQL has been prototyped in a small amount of part-time  
effort by a single developer (yes, I'm trying to get that Open  
Sourced, but don't know when/if that'll happen).

> never by their prefix. This is making an unnecessary exception to that

Yes. I think the issue here is whether the convenience outweighs the  
confusion. Perhaps we could make the binding between selectors and  
QNames not on by default; instead, a feed would have to specifically  
say that it's in effect (e.g., in a fq:interface attribute)...

> so ugly that it requires a whole new language to replace it, either.

Now that's interesting! I've had the XQuery/XPath 2.0 discussion with  
others before, and the resolution was that as long as it was possible  
to map it to XPath for implementation purposes, that's good enough --  
something that it looks like you've taken a step towards.

I'm not married to the current syntax, but I do want it to be URI- 
friendly and easy for humans to understand. If the syntax meets those  
goals and happens to be a subset of XPath, I'm fine, but I don't want  
to define it as an explicit subset; doing so will force people to  
learn XPath whether they want to or not, and IMO will lead to interop  
problems as people start relying on XPath features, rather than what  
the spec says.

Anyway, this is good; I'd be interested in hearing what other Atom  
folks have to say about the alternative syntax, and whether it's good  
enough to have an easy way to turn it into XPath if necessary.

> specific type in variables, regardless of what the fq:index elements  

I wonder if that's a good thing or not...

> example, atom:content/@type.

Again, hmm. The ability of fq:interface to constrain the shape of  
queries is a good thing.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Dec 2007 13:07:54 +1100

> Its use of &'s is probably asking for escaping bugs in URIs though.

Yes, and that's the reason I moved on pretty early ;)

> atom:category that supports

See @path on fq:index.

>  I'd think something like that would be pretty useful.  Eg: draft=*

Yes; it's just a bare selector;

> A constraint with no comparison operator will yield True if the  
> selector matches any node.

>  QNames are evil.

I've heard that... :)

>  I'd rather have inexact *:name matches, than explicitly

Hmm, possibly; perhaps something you declare on fq:interface?

>  Do element match predicates only apply to direct children of  

Respectively, yes and no.

>  after things like author inheritance rules have been applied?

Ah, good question. This needs to be clarified (and I think it should  
inherit, but it may be tricky to cleanly specify).

>  any selectors can be used?

The default is that the server hasn't explicitly told you about any  
selectors if there aren't any fq:index's, so you're happy to try them  
(the server can only say 'no').

Thanks!

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Dec 2007 13:10:53 +1100

> systematically mapped to XPath?

I believe so, although I haven't done the work; as I said in a  
response to Brian a few minutes ago, other folks who want to implement  
with XPath 2.0 have looked at it and believed that it's possible (and  
relatively easy). Brian's message seems to support that.

> links *are* important in Atom, so some kind of triple patterns could

Yes, links are definitely not a strong point in the current draft (you  
can index them with @path, but it's fairly "dumb"). Do you have  
specific use cases in mind?

> http://alexbarnett.net/blog/archive/2007/04/30/microsoft-codename-quot-astoria-quot-data-services-for-the-web.aspx

Thanks, I've looked at it before, but I'll have another.

Cheers,

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 20 Dec 2007 12:35:31 +0700

Mark Nottingham wrote:
> To my mind, XPath 2.0 (which you seem to be referring to) is 
> too complex to base something like this on; printing the 
> XQuery-related specs kills too many trees, and it takes a 
> major investment to get up to speed with it as a user, much 
> less an implementer. While it works if you're already 
> invested in the XQuery world, my perception is that this 
> group is still fairly small, and "enterprise" focused.
> 
> I don't keep up to date with the XQuery world, but last I 
> checked, Perl, Python, PHP and Ruby all lacked 
> implementations. What will they do?

Parsing that subset of XPath 2.0 might be a little harder than parsing
FIQL, but building parsers is trivial nowadays. Actually implementing
the semantics shouldn't be any harder than implementing FIQL's
semantics, though, since they are equally expressive.

> I'm not married to the current syntax, but I do want it to be 
> URI- friendly and easy for humans to understand.

In the syntax you proposed, how can I search for all entries that have a
title starting with an asterisk ('*')? Or, in other words, it isn't
clear to me how you deal with quoting/escaping issues. Is the processor
supposed to treat an percent-encoded asterisk differently from an
unencoded asterisk character? And, if so, how does this interact with
URI templates?

> If the syntax meets those goals and happens to be a
> subset of XPath, I'm fine, but I don't want to define it as
> an explicit subset; doing so will force people to learn
> XPath whether they want to or not, and IMO will lead to
> interop problems as people start relying on Xpath
> features, rather than what the spec says.

There is no point in reusing the XPath syntax if it isn't 100%
upwards-compatible with XPath. If I were to write a specification for
this, the normative text would be written purely in terms of XPath; I
would have a separate non-normative tutorial describing the subset, and
a non-normative XQuery mapping should be included as well.

XPath is incompatible with URI templates; because of the
quoting/escaping requirements for strings, it isn't possible to write a
URI template (as they are defined today) that is guaranteed to result in
a syntactically-valid XPath expression. To see why, try to construct a
URI template that generates an the XPath containing a single quote and a
double quote at the same time, or even just one or the other. This is a
significant limitation of URI templates, and one that is difficult to
solve. Another problem with URI templates is that there is no "listjoin"
operation for dictionaries; this makes it difficult to pass an arbitrary
number of namespace prefix bindings along with the query. Finally, URI
templates cannot deal with nested structures like those used in the
AND/OR expressions in XPath, FIQL, and Gdata.

Besides syntax, I'm also not yet convinced that such a mechanism like
this is really needed. It seems like we can get really far with URI
templates and an explicit enumeration of searchable fields like GData,
if we remove the logical OR operator. And, once you *need* a logical OR
operator, or when you need to be able to search an open-ended set of
fields, you will probably be in a situation where it is worthwhile to
use (a significant of) XPath 1.0, XPath 2.0, XQuery, and/or some other
advanced structured document search language, plus a set of
Atom-specific extension functions. Do you have any real-world motivating
examples of something where a GData-ish query mechanism is insufficient,
but XPath is too much?

- Brian

---

From: Martin Duerst <[email-masked]>
Date: Thu, 20 Dec 2007 14:32:43 +0900

>this group is still fairly small, and "enterprise" focused.

I must admit that I haven't looked at the draft at all, but it seems
strange to me to propose XPath 2.0 for something simple when there is
XPath 1.0, and strange to reject any version of XPath based on the
fact/opition that XPath 2.0 is too complex.

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 20 Dec 2007 13:28:46 +0700

Martin Duerst wrote:
> I must admit that I haven't looked at the draft at all, but 
> it seems strange to me to propose XPath 2.0 for something 
> simple when there is XPath 1.0, and strange to reject any 
> version of XPath based on the fact/opition that XPath 2.0 is 
> too complex.

I agree that XPath 1.0 is generally preferable to XPath 2.0. The only
reason I chose XPath 2.0 was because I could map FIQL to it without
needing any extension functions. You could do the same for XPath 1.0,
but you would need at least one extension function to do date
comparisons, and another extension function to do convenient
case-insensitive string matching.

- Brian

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Thu, 20 Dec 2007 11:58:45 +0100

Hi Mark,

I was following the discussion around this draft you've proposed and 
rather than focusing on the choice of the most appropriate query 
language used I wanted to ask you about the purpose of this draft? Who 
is the target? Since you use URI-like query, are you thinking of public 
services to query Atom feeds? In such case, shouldn't the draft have a 
large security section (it looks like creating yet another query syntax 
means that its security is by default nil until proven otherwise)? Are 
you planning on making the draft informational only or more?

No matter what, FIQL will never be able to describe all the use cases of 
querying an Atom document and therefore will create a stress in an 
application which will have to make the decision about either dropping 
FIQL entirely in favor of home made XPath/XQuery/etc. queries or will 
potentially extend FIQL.

Overall, I'm not convinced there is a need for such specification. It 
might be great as best practices on the Atom wiki however. Also, even 
though some topics need an IETF draft, I'm worried that by defining 
specialized ones for Atom when standard alternatives already exist (e.g. 
XPath) the community will attract concerns about its capacity to re-use 
existing solutions.

That being said I'm not criticizing your work Mark, as I said just as a 
best practices document such specification would be very valuable to the 
community. But trying to go the IETF way sometimes feel too heavy.

Anyway just some thoughts :)
- Sylvain

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 20 Dec 2007 13:46:21 +0100

* James M Snell <[email-masked]> [2007-12-19 20:40]:
> I meant only, for example, that regardless of whether the
> underlying feed uses "a:feed", "atom:feed", "foobar:feed",
> etc the xpath query would always just be "feed".

Yes, and I said you can’t do that in XPath, at least in 1.0.

Regards,

---

From: Bill de hOra <[email-masked]>
Date: Fri, 21 Dec 2007 20:24:46 +0000

"updated=gt=2003-12-13T00:00:00Z"

use underscores instead for ranges, ie

  updated_gt==2003-12-13T00:00:00Z

I didn't understand why '==' is used as a token instead of '='. Perhaps 
the above suggestion helps with that.

"By default, a selector is treated as an XML QName 
[W3C.REC-xml-names-19990114] which selects any and all child elements 
of the entry element which share the same syntax (i.e., the same 
prefix and localname; the namespace URI is not considered), along   with 
their descendant content if any.
"

I'm no fan of Qnames/XMLNS, but that looks butchered to me. You at least 
need to figure out what to do with atom in the default ns and atom with 
atom:.

The spec doesn't say what should happen with acquired information or 
assumed values (ie, author). I have to assume since you're treating 
qnames literally, you're not catering for that either.

I'm not sold on the subset XPath arguments elsewhere btw. This language 
seems designed to match across branches not select subtrees.

cheers
Bill

---

From: James M Snell <[email-masked]>
Date: Fri, 21 Dec 2007 22:19:59 -0800

The issue I have with the fiql draft is not really the query syntax
itself, as much as it is with the poor definition of the qname-based
selectors and the definition of the fq:interface mechanism.  What I
think would be a significantly better approach, as I've mentioned, is a
more generic link-template mechanism that could be used for a broader
set of use cases.  Prior to the publication of fiql, I had already been
working on a syntax for the link-template, which would be simple to
extend to support the fiql requirements.  For instance,

<template xmlns="http://purl.org/atompub/templates/1.0"
  title="Search"
  rel="search" type="application/atom+xml" hreflang="en-US"
  template="http://example.org/{index=entries}?{-join|&amp;|filter,num}">
  <variable name="index" type="http://.../enum">
    <value>entries</value>
    <value>comments</value>
  </variable>
  <variable name="num" type="http://.../simple" pattern="\d+" />
  <variable name="filter" type="http://.../expression">
    <field name="title" />
    <field name="rating" type="http://.../numeric" />
    <field name="updated" type="http://.../date" />
  </variable>
</template>

With this, there is messing around with QNames.  There's just a URI
Template with a number of template variables that each map to some
distinct value definition.

The "index" variable is typed as an "enum" with two possible values,
"entries" and "comments"

The "num" variable is a simple string value matching the regex pattern
"\d+" (one or more alphanumeric digits).

The "filter" variable is an expression that would match the FIQL syntax.
 Within the filter variable definition are specific typed field
definitions that define the fields that can be used within the fiql
expression.

For example, given the values:

  index  = entries
  num    = 10
  filter = title==foo;rating=5

The URI Template expands to:

  http://example.org/entries?num=10&filter=title==foo;rating==5

Personally, I think this approach provides significantly more
flexibility than the fq:interface definition without adding too much
unnecessary complexity.  It's also something that can be reused easily
for other use cases beyond basic searching and filtering

For instance,

<template xmlns="http://purl.org/atompub/templates/1.0"
  title="Archive"
  rel="archive" type="application/atom+xml" hreflang="en-US"
  template="http://example.org/archives/{year}/{month}">
  <variable name="year"
            type="http://.../simple"
            pattern="\d{4}" />
  <variable name="month"
            type="http://.../simple"
            pattern="0[1-9]|1[0-2]" />
</template>

And another:

<template xmlns="http://purl.org/atompub/templates/1.0"
  title="Sync"
  rel="sync" type="application/atom+xml" hreflang="en-US"
  template="http://example.org/sync?{-join|&|start,end">
  <variable name="start" type="http://.../date" />
  <variable name="end" type="http://.../date" />
</template>

Getting back to the FIQL use case, however, let's break down the field
definition from earlier:

  <variable name="filter" type="http://.../expression">
    <field name="title" />
    <field name="rating" type="http://.../numeric" />
    <field name="updated" type="http://.../date" />
  </variable>

Given this, there is obviously no clear connection between each of the
fields and the elements in the Atom.  Personally, I think this is a good
thing, as it ought to be up to the server code to determine the
appropriate mapping between the fields and the elements.  However, if we
do need a means of explicitly connecting the two, that can be done via
the field element definition.  For instance, in order to avoid having to
mess with QNames, we could define a URI identifier for each of the Atom
elements, e.g.,

  <field name="title" ref="http://www.w3.org/2005/Atom/title" />

This would tell us that the title field maps to the atom:title element.

Perhaps I wanted to filter on title type (text, html, or xhtml). For
that I could do something like:

  <field name="title" ref="http://www.w3.org/2005/Atom/title" />
  <field name="type"
         type="http://.../enum"
         ref="http://www.w3.org/2005/Atom/title/type">
    <value>text</value>
    <value>html</value>
    <value>xhtml</value>
  </field>

This would allow me to create expressions like:

  title==foo;type==html
  title==bar;type==text

I'm not convinced it's necessary, but like the fq:index element, the
field element could have a path attribute as well:

  <field name="rating" path="snx:rating" type="http://.../numeric" />

The link-template mechanism has the benefit of being more generically
applicable to more scenarios that fq:interface; allows the use of any
URI Template, not just ones that contain {fiql-exp}; avoids the
inherently messy qname garbage; and is not prohibitively more
complicated than the fq:interface syntax.

- James

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 23 Dec 2007 09:04:50 +1100

On 22/12/2007, at 7:24 AM, Bill de hOra wrote:

> I didn't understand why '==' is used as a token instead of '='.  
> Perhaps the above suggestion helps with that.

Comparison operators are delimited with '=' so extension comparisons  
can be easily added down the road. "==" is necessary so that you don't  
confuse content with operator.

> "By default, a selector is treated as an XML QName [W3C.REC-xml- 
> names-19990114] which selects any and all child elements of the  
> entry element which share the same syntax (i.e., the same prefix and  
> localname; the namespace URI is not considered), along   with their  
> descendant content if any.
> "
>
> I'm no fan of Qnames/XMLNS, but that looks butchered to me. You at  
> least need to figure out what to do with atom in the default ns and  
> atom with atom:.

Yes, that seems to be where a lot of the pushback on the spec is  
centred. Let me ask explicitly: does anyone like the namespace binding  
between the URI and the document? I personally think it's not pretty,  
but better than some other attempts at referring to QNames in URIs.  
The alternatives, as I see them, are:

1) Specify this binding better (as you allude to)
2) Get rid of it completely, and require any non-Atom selectors to be  
defined in fq:interface.

> qnames literally, you're not catering for that either.

No, just haven't got to it yet.

> I'm not sold on the subset XPath arguments elsewhere btw. This  
> language seems designed to match across branches not select subtrees.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 23 Dec 2007 10:00:51 +1100

On 20/12/2007, at 9:58 PM, Sylvain Hellegouarch wrote:

> language used I wanted to ask you about the purpose of this draft?  
> Who is the target? Since you use URI-like query, are you thinking of  
> public services to query Atom feeds? In such case, shouldn't the  
> draft have a large security section (it looks like creating yet  
> another query syntax means that its security is by default nil until  
> proven otherwise)? Are you planning on making the draft  
> informational only or more?

Great questions.

I started this after looking at GData, which is really just APP +  
proprietary auth + proprietary feed-focused query.  Auth is a hard and  
luckily separable, but looking at the query portion of things, I  
thought we could do better.

So, yes, I'm thinking of public services, definitely. Yes, security  
considerations need to be filled out. Informational is the most I'd  
aspire to (unless there's *significant* interest in going further),  
and that may be too far.

> No matter what, FIQL will never be able to describe all the use  
> cases of querying an Atom document and therefore will create a  
> stress in an application which will have to make the decision about  
> either dropping FIQL entirely in favor of home made XPath/XQuery/ 
> etc. queries or will potentially extend FIQL.

Nor should it try to do everything; it's all about making the most  
common 80% as easy as possible to implement and use. That said, I  
tried to make sure that it is extensible in the right places.

> Overall, I'm not convinced there is a need for such specification.  
> It might be great as best practices on the Atom wiki however. Also,  
> even though some topics need an IETF draft, I'm worried that by  
> defining specialized ones for Atom when standard alternatives  
> already exist (e.g. XPath) the community will attract concerns about  
> its capacity to re-use existing solutions.

Perhaps, but if that's the case, why aren't people slapping generic  
XPath engines in front of their feeds? Why did Google choose to go for  
their own syntax instead of using XPath? Yes, FIQL is a new syntax,  
but it's one that people can understand after sitting down and playing  
with it for a few minutes. XPath 2.0 doesn't have the same property.

A lot of the discussion to date has been around the query language  
itself. To me, the most important part of FIQL is fq:interface; having  
a way to declare that there is a query interface -- whatever the query  
syntax -- is IMO necessary if people are going to be exposing these  
things.

> That being said I'm not criticizing your work Mark, as I said just  
> as a best practices document such specification would be very  
> valuable to the community. But trying to go the IETF way sometimes  
> feel too heavy.

To be clear, I submitted this not because I'm sure I want to  
standardise it; I did so because I want feedback from this community  
on this particular proposal, and to generally gauge interest in an  
openly defined feed query language.

Thanks!

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 23 Dec 2007 09:56:13 +1100

I discounted XPath 1.0 pretty quickly not only because of lack of  
datatyping, but also because it's inherently not URI-friendly (one of  
my bigger design goals).

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 23 Dec 2007 10:04:01 +1100

Sure, if we can get forward motion on a template-based analogy of the  
link element, I'm all for it; I just didn't want to try to boil the  
ocean in this one case. Seems like we have to settle on what a link  
template is first, however...

Cheers,

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 23 Dec 2007 07:01:04 +0700

Mark Nottingham wrote:
> A lot of the discussion to date has been around the query 
> language itself. To me, the most important part of FIQL is 
> fq:interface; having a way to declare that there is a query 
> interface -- whatever the query syntax -- is IMO necessary if 
> people are going to be exposing these things.

<snip>

> standardise it; I did so because I want feedback from this 
> community on this particular proposal, and to generally gauge 
> interest in an openly defined feed query language.

I suggest trying to define it as an extension to OpenSearch. OpenSearch
already has been implemented many times by many people, and it already
is an "80%" solution since it is about full text search. 

No matter what you do, URLs are always going to be hard to read. I
suggest giving up on the readability of the URL so that you can gain
readability in non-URL contexts. In particular, I recommend the
end-user-friendly syntax used by Google, Yahoo, Live Search, Lucene, and
many other existing systems: 

	keyword (keyword OR keyword) operator:"operator parameter"

Where operators are like:

	tag:HTTP tag:Standards tag:Web tag:"Web Services"
	updated:2007 
	updated:2007-12
	updated:2007-12-09
	author:Mark
	title:Foo

This ends up being pretty unreadable in URIs due to percent-encoding of
spaces, but some groups have been pretty successful with it.

- Brian

---

From: James M Snell <[email-masked]>
Date: Sat, 22 Dec 2007 21:10:27 -0800

I think the "what" is pretty simple: a link-template is essentially just
a parameterized atom:link built around a URI Template.  It shouldn't be
all that complicated.

A link-template would have the same basic attributes as an atom:link,
but would have a template in place of the the href.  Associated with
each are one or more variable definitions; each of which have a defined
type.  The type can be a simple value, a number, a date, an enum, or an
expression.  Simple values can optionally be defined to adhere to a
regex pattern.  Numbers and dates can have min/max values. Enums provide
a listing of static values. An expression could be a FIQL expression
which would have a set of field definitions that would basically echo
the variable definitions.

The link-template's rel attribute would identify the purpose of the
template.  There are a number of rel values we can very easily identify
now without much effort.  "search", "sync", "archives", etc.

Other than the definition of the expression language, which you've
already covered, there's really nothing else to it.  I'd be more than
happy to write up the necessary spec text.

- James