Subject: What good are stored VQUERYs?

From: [email-masked]
Date: Wed, 6 Nov 2002 18:14:55 -0500

It seems that the question never got resolved after the hall way 
discussions we had a couple meetings ago.  Throughout the draft there are 
lots of references to stored queries:

   NAME -  Several storeable components such as "VCAR" and "VQUERY" may
      have the "NAME" property contained in them to describe in various
      locals the purpose of the component.

and

   The conceptual model for a calendar store is shown below.  The
   calendar store (VCALSTORE - Section 9.2) contains "VCAR"s, "VQUERY"s,
   "VTIMEZONE"s, "VAGENDA"s and calendar store properties.

   Calendars (VAGENDAs) contain "VEVENT"s, "VTODO"s, "VJOURNAL"s,
   "VCAR"s, "VTIMEZONE"s, "VFREEBUSY", "VQUERY"s and calendar
   properties.

and

   VCALSTORE
   |
   +-- properties
   +-- VCARs
   +-- VQUERYs

but I have yet to see what use storing queries can be.  After all, I 
cannot 'wildcard' or macro expand anything in a query so I cannot create a 
stored query for "Find me all events on my calendar for today" or "What 
events have alarms that trigger in the next 15 minutes", etc.  Each VQUERY 
that a CUA sends to the CS needs to be custom built to have the right 
parameters (ie: dates/times).

So in the interest of KISS I ask why a CS should (MUST?) store VQUERYs if 
they are effectively 'single shot' queries?

Unless there is a common case that Im just not seeing I would like to 
propose that we remove the concept of stored queries entirely.

Bruce
===========================================================================
Bruce Kahn                                INet: 
[email-masked]
Messaging & Collaboration                 Phone: 978.399.6496
IBM Software Group                         FAX: and nothing but the FAX...
Standard disclaimers apply, even where prohibited by law...

---

From: Andrea Campi <[email-masked]>
Date: Thu, 7 Nov 2002 01:45:48 +0100

On Thursday, November 7, 2002, at 12:14 AM, [email-masked] 
wrote:
> cannot 'wildcard' or macro expand anything in a query so I cannot 
> create a stored query for "Find me all events on my calendar for 
> today" or "What events have alarms that trigger in the next 15 
> minutes", etc.  Each VQUERY that a CUA sends to the CS needs to be 
> custom built to have the right parameters (ie: dates/times).
>
> So in the interest of KISS I ask why a CS should (MUST?) store VQUERYs 
> if they are effectively 'single shot' queries?
>

IMHO the best reason to KEEP stored query is precisely what you 
mention, that is, the absence of parameters.
Nowhere in the draft I see anything which implies stored queries MUST 
translate to CAL-QL; the way I interpret this is, a CS can implement 
them anyway it wishes.
Looking at your example, a CS could expose a stored query named "All 
events on my calendar for today", with no corresponding CAP-QL. When a 
CUA executes the query, the CS simply "does the right thing", and 
retrieves events for today.
It's true that this doesn't improve interoperability at all, but it 
allows matching CUA and CS to do very powerful queries with minimum 
effort.

Now, if we extended CAP-QL to allow parameters in stored query 
invocation, I'd be even happier. However, allowing the syntax now and 
building on it later on is better than nothing...

Bye,
	Andrea

---

From: [email-masked]
Date: Mon, 11 Nov 2002 10:27:45 -0500

> translate to CAL-QL; the way I interpret this is, a CS can implement 

The actual storage of the query is not the concern here, its the utility 
of actually storing them in the first place.

Since the query has to be encoded in CAP-QL and there is no way to 
interpret what cannot be encoded in CAP-QL I dont see how a CS can 
transcode a 'fixed' query into one that means "All events on my calendar 
for today".   Should the CS somehow infer it from the explicit CAP-QL 
language?

> retrieves events for today.

Here is where Im a bit fuzzy.  If there is no way to encode the query in 
CAP-QL then how is the CUA to send it in the first place??

Oh, are you suggesting that there is some Out-of-band mechanism to place 
non-CAP-QL encodable queries into the CS and then the CUA just references 
them?  Seems like kind of a wishfull thing given CAP-QL and the robustness 
of it. 

Also, there is now way for the CUA to actually pull the VQUERY from the CS 
to check it or modify it.  After all my CUA should be able to check all 
VQUERYs in the CS so just how would the CS encode the unencodable VQUERY 
that is stored??  Would the CS just say "Sorry, you cant have that 
VQUERY!"?

I would prefer that we take a simpler approach instead of some assumption 
and hacks to deal w/your scenario.  CAP and iCalendar are complex enough 
as is.  There are 2 possible paths to take here:

1: Extend CAP-QL so that wildcarding is possible OR
2: Remove the concept of stored VQUERYs

I dont say remove named VQUERYs since I can have more than 1 VQUERY 'in 
use' at any given time. 

Im just for simplifying the model a bit.  If there is strong enough desire 
for #1 then some new ABNF and prose is in order and then we can discuss 
the impact of that change.  We may also need a way for the CS to respond 
"I dont support query storage" in GET-CAPABILITY for those more 
lightweight CS's (a comment I posted a year or so ago I think).

> allows matching CUA and CS to do very powerful queries with minimum 

But CAP-QL was supposed to be so rich that we could craft any query in it. 
(Ok a bit of sarcasm on a Monday AM...)

> building on it later on is better than nothing...

Care to propose the new ABNF and prose then?

Bruce

---

From: John Stracke <[email-masked]>
Date: Wed, 13 Nov 2002 15:07:37 -0500

> allows matching CUA and CS to do very powerful queries with minimum 

But such nonstandard implementations could just make up their own 
mechanisms to do those nonstandard queries.

-- 
/===============================================================\
|John Stracke      [email-masked]                      |
|Principal Engineer|http://www.centivinc.com                    |
|Centiv            |My opinions are my own.                     |
|===============================================================|
|If you're going to walk on thin ice, you might as well *dance*!|
\===============================================================/

---

From: Andrea Campi <[email-masked]>
Date: Fri, 15 Nov 2002 14:14:12 +0100

> mechanisms to do those nonstandard queries.

What part of the standard does it violate? Where is text to the effect
that a CUA MUST have stored the query before calling it? If the CUA
finds the query there, and is permitted to call it, and it knows what
to do with the results, I see nothing nonstandard.

Yes, a CUA would either need to fallback to using normal queries or
end up being tied to a CS. This could be a reasonable tradeoff for
special purpose CUAs where space or time are limiting factors.

Bye,
	Andrea

-- 
Andrea Campi                              mailto:[email-masked]
I.NET S.p.A. - BT Ignite                  http://www.inet.it
Technical Dept. - R&D			  phone: +39 02 32863 ext 1
v. Darwin, 85 - I-20019			  fax: +39 02 32863 ext 7705
Settimo Milanese (MI), Italy

---

From: Andrea Campi <[email-masked]>
Date: Fri, 15 Nov 2002 14:09:40 +0100

> for today".   Should the CS somehow infer it from the explicit CAP-QL 

Uhm no, I meant some kind of "synthetic" VQUERY - the CS put it there
itself.

> them?  Seems like kind of a wishfull thing given CAP-QL and the robustness 

I can understand if you questioned the usefulness or the interoperability
of such a thing, but I don't understand your objection. A CUA and a CS would
have no problem; heck, you could even write an RFC to standardize an "opaque"
VQUERY. As long as the CUA knows how the VQUERY is named and what it
returns, I think it would be both realistic and robust.

> that is stored??  Would the CS just say "Sorry, you cant have that 

Why not tell the truth? The CS would reply:

BEGIN:VQUERY
QUERYID:All VEVENTs for today
END:VQUERY

If the CUA and CS adhere to a common convention, the CUA would know what to
expect as a reply if it called this query.

> 1: Extend CAP-QL so that wildcarding is possible OR

Uhm... ok. But I don't see anything in the draft that forbids what I was
talking about. So I'd say, extending CAP-QL is good, but it won't remove
the usefullness of my example nor would outlaw it.

> I dont say remove named VQUERYs since I can have more than 1 VQUERY 'in 

I'm getting lost now. What use would named query be if they can't be stored?
I don't see how having more than one in use, whatever that means, affects
what we are discussing. If the CUA issues more than one query at once, it
will use cmd-id to find the answers, not the queryid.

> lightweight CS's (a comment I posted a year or so ago I think).

Sure.

> (Ok a bit of sarcasm on a Monday AM...)

It's also a question of workload split. For a very very simple example,
not meant to support the idea by itself, think about a query for "all events
in this week". The client needs to compute them by itself, then create a vquery
and send it (or possibly call a stored query with the right arguments). Depending
on the client architecture, the number of times the user quits and restarts its
CUA, etc this can be done a number of times. OTOH, since everybody is going to
use the same query, the server will probably have this information (when does
this week start and end?) cached.

This is a trivial example, but multiply it by a number of similar cases and
then by thousands of CUAs connected to the server at the same time... it saves
power on the CUA, on the CS (no need to parse the VQUERY again and again) and
network traffic.

Oh and by the way, this is one of the reasons why I like your idea of views -
the CUA would send the query and then ask the server to refresh it from time
to time...

> Care to propose the new ABNF and prose then?

I am not familiar enough with ABNF and I have no idea ATM if I have enough
time. I will try to come up with an informal proposal to see if we are
agree. OK?

But I, for one, will not stop using stored queries as I explained.

Bye,
	Andrea

---

From: [email-masked]
Date: Mon, 18 Nov 2002 17:19:55 -0500

> Uhm no, I meant some kind of "synthetic" VQUERY - the CS put it there

I got that part.  What I have a problem with is that if a CUA tried to 
actually retrieve the actual VQUERY from the CS then it would have to fail 
since the CS has no way (in CAP 1.0) to encode the non-CAP-QL-able query 
in CAP-QL.  So if a CUA cannot retrieve a VQUERY then should it try to use 
it in a query??  Its very unclear and vague to say the least and I think 
VQUERY storage as for CAP 1.0 is not needed.

Stored VQUERYs like you describe could be stored/retrieved in a subsequent 
draft if we make sure that CAP allows for storing of any components and 
the newer draft can encode up the non-CAP-QL query.  The System Model in 
Section 3.1 does not expressly show this (no IANA-Components in the tree) 
but I think the intent is there.

In the mean time I still think that building in VQUERY storage is a waste 
of cycles.  At a very minimum it should be a CS response to GET-CAPABILITY 
since I for one dont want to have to build it in when I cant see a strong 
usage for it given the current query model...

> returns, I think it would be both realistic and robust.

Actually part of my objection to your scenario is that unless all CUAs 
know that the VQUERY with "NAME=Magic01" is a special VQUERY that cannot 
be retrieved (since its not CAP-QL encodable) then when any CUA that tries 
to "SELECT VQUERY FROM VAGNEDA WHERE NAME = 'MAGIC01'" they will get some 
kind of failure response.  As such they will/should treat the named VQUERY 
as 'bad' or 'unusable'. 

IMHO there should be no data put into the CS by ANY means (Administrative 
or not) that cannot be retrieved and/or encoded using the CAP 1.0 or 
updated iCalendar specs.  Otherwise it just becomes a recipe for disaster.

> QUERYID:All VEVENTs for today

Thats not what I asked.  I didnt _use_ the VQUERY, I tried to SELECT it. 
What should be the response from the CS "Sorry, you cannot read the VQUERY 
but you can use it in a VQUERY command..."

> expect as a reply if it called this query.

Where in the CAP draft does it deal with this kind of response code?  I 
cannot find any text that describes a "You cannot read it but you can use 
it." kind of response.  The closest I can find would be a general failure 
response to the "SELECT VQUERY FROM VAGENDA ..." command.  I would not 
expect a CUA to use a VQUERY that it cannot read from the CS, would you?? 
(I like to practice Save Hex...)

> I'm getting lost now. What use would named query be if they can't be 
stored?

My bad.  I mistakenly mixed the concepts of cmd-id and NAME when I was 
thinking this over.  We no longer need the concept of a named VQUERY with 
this simplification.

> It's also a question of workload split. 

Actually its also an issue of CS design as well as performance and 
scalability.  Since I cant seem to craft any kind of meaningful VQUERY in 
CAP-QL that I would expect to store I question the need to require all 
CS's to implement it.  Currently in CAP-QL the CUA is going to have to 
recraft the VQUERY every time it wants to do that "What events are on the 
calendar for this week" since "this week" changes.  This inability to 
wildcard makes stored VQUERYs less than useful for CAP 1.0. As such I am 
strongly questioning why we do not remove it from CAP 1.0 and simplify the 
draft accordingly...

I have no qualms w/you or anyeone else coming up with a follow-on draft to 
use named queries (VQUERY2?) that can be stored, etc but I dislike all 
this baggage that has no current benefit.

>                                              OTOH, since everybody is 
going to
> this week start and end?) cached.

CS caching of results is not the same thing as caching of the query too.

> power on the CUA, on the CS (no need to parse the VQUERY again and 
again) and

The only benefit for this case is really at the network packet level, 
maybe some at the CS processing level. 

The CUA is still going to have to craft the VQUERY command to send to the 
CS on the already established session.  So there is some octet savings 
there but not that radical really. 

The CS will still have to locate the stored VQUERY, parse it, perform the 
propery internal query work and then send back the results.  Depending on 
how the CS implements VQUERY storage it may or may not have to reparse it; 
odds are it will have to reparse it so it can replace the 'wildcards' with 
the right stuff before it can do the actual query.  So again there is work 
to be done either in decoding the CUAs VQUERY or the stored VQUERY before 
the actual query can be done.  Depending on how the CS is designed this 
may be signficiant or it may be marginal (Im guessing marginal for the 
first release or so).

In any case there will still have to be CUA -> CS traffic followed by CS 
work of some kind and then CS -> CUA traffic so Im not sure we buy 
ourselves that much with VQUERY storage.  Especially since CAP-QL is the 
only codified search schema and it has no demonstratable benefits for this 
model.

> the CUA would send the query and then ask the server to refresh it from 
time

I dont think I sent that to the WG at large so I think you just confused 
some folks...

Seems like Im arguing against myself then does it?...  I could consider 
views a kind of stored, named VQUERY but I usually dont as I can typically 
search a view and the results may vary depending on the views content 
criteria (ala the SELECT of a VQUERY). 

> time. I will try to come up with an informal proposal to see if we are

Works for me.  Ill be happy to work on an ABNF after the dust settles.

> But I, for one, will not stop using stored queries as I explained.

I wont try to stop you but Im very curious how the CS is going to react to 
the CUA trying to read that VQUERY from the CS...   I think that no CUA 
should try to use something that it cannot read out of the CS myself; its 
just not safe. 

Bruce

---

From: Andrea Campi <[email-masked]>
Date: Tue, 19 Nov 2002 12:49:29 +0100

Hi,

> VQUERY storage as for CAP 1.0 is not needed.

Once again, I don't see the problem. If the CUA sends:

BEGIN:VQUERY
QUERY:SELECT VQUERY FROM VAGNEDA WHERE NAME = 'MAGIC01'
ENV:VQUERY

The CS replies:

BEGIN:VQUERY
QUERYID:MAGIC01
END:VQUERY

I don't see why you think it should be an error (although it would require
an ABNF change) - the CS has the query so it sends what it has. It just
happens to be unable to represent it in CAP-QL. Incidentally, that's the
reason why a few months ago I proposed we added a way to specify the VQUERY
is hardcoded in the CS and can't be represented in CAP-QL.

BEGIN:VQUERY
QUERYID:MAGIC01
NATIVE:YES	(NATIVE is just an example, you can choose a better name)
END:VQUERY

This would make clear to a CUA that it is looking at a valid VQUERY, but
nothing is know about its implementation - so the CUA had better know what
to do with it or leave it alone.

Also, do you really expect all CUAs to retrieve the VQUERY before calling
it? Then what, do a string compare of the QUERY? It would be even less
useful then.

> but I think the intent is there.

I agree that CAP should allow storing any component. Of course, an actual CUA
implementation should not expect to be able to store components the server
doesn't list as supported in GET-CAPABILITY.

> usage for it given the current query model...

I agree it should be in GET-CAPABILITY.

> updated iCalendar specs.  Otherwise it just becomes a recipe for disaster.

I agree. It just happens to have no QUERY property, so what? It's still
legal, so you can retrieve it.

> (I like to practice Save Hex...)

That's a CUA choice. In controlled environments (the CUA knows it can
trust the CS), it would be safe. Remember that I'm talking about hardcoded
VQUERY, so no malicious CUA could change them.

> strongly questioning why we do not remove it from CAP 1.0 and simplify the 

I agree that CAP-QL is currently not powerful enough to write generically
useful store queries. I will also admit that what I have in mind could be
not generally useful.

> this baggage that has no current benefit.

I guess I have to agree with you that there are some issues that need to
be addressed for stored queries to be really useful; among these are
parameters and security. I don't think I agree about the no benefit part,
since I can do my stuff without violating the draft, but this is a personal
opinion.
Since I seem to be the only one to defend it, I have no problem with taking
stored queries away and doing it properly in another draft. However, this
requires thinking NOW about how the new draft will extend the current
model.
Will it change or extend VQUERY? Then we might need to check VQUERY ABNF (I think
QUERY should be optional).
Will it introduce a new component? But all commands take VQUERY, so I
won't be able to send a SEARCH using VQUERY2...

Bye,
	Andrea