Subject: The intent/meaning of EXPAND and its usefullness

From: [email-masked]
Date: Thu, 11 Dec 2003 14:51:29 -0500

Ok, I know Im going to be out of touch soon but I wanted to ask in the 
interest of getting some WG discussion to resolve my confusion.

A while back I asked a question about the usefulness of EXPAND at all 
(Subject "EXPAND property: All instances?" back on 31-Oct-2003).  There 
was no technical discussion of it so Ill try again given the recent other 
thread on DTSTART/RECURRENCE-ID/EXPAND.

In CAP-12-e is defined:

8.16 EXPAND property

   Property Name: EXPAND

   Purpose: This property is to notify the CS if it should or should not
   expand any component with recurrence rules into multiple instances in
   a query reply.

[Snip, snip]

   Description: If a CUA wishes to see all of the instances of a
   recurring component the CUA sets EXPAND=TRUE in the "VQUERY"
   component. If not specified, the default is FALSE. Note that if the
   CS has its "RECUR-EXPAND" CS property value set to false then the
   "EXPAND" property will be ignored and the result will be as if the
   "EXPAND" value was set to false.

and in Section 6.1.1.15 Query by Date-Time range is also written:

          This works only for CSs that have the "RECUR-EXPAND"
   property value set to "TRUE" in the "GET-CAPABILITY" exchange.

So, my take on this is:

1: If a CS does not support expanding recurrence rules then it would send 
RECUR-EXPAND:FALSE when asked.
2: If RECUR-EXPAND is FALSE then:
        A) EXPAND on a QUERY is ignored and
        B) a CUA will be unable to query for entries using a date-time 
range (even if its not repeating )

This raises some questions which Id like to get some WG concensus on (and 
perhaps clarification in CAP too).  These include:

1: If a CS does not support recurrence rules then why would that preclude 
the use of RDATEs?  The CS does not have to evaluate any rules to 
correctly roll out the repeating set.

2: Why does the lack of recurrence rule support in the CS preclude dong 
date-time range searches for non-repeating entries ("Give me all the 
VEVENTs for tomorrow") or for those specified in RDATEs?  Date-time range 
searching is something nearly all CS need to have in order to function 
well at all.  Otherwise its usability would be somewhat questionable.

3: What real use is the EXPAND property in actual practice?  This question 
has 2 parts behind it:

        3A: The 1st line of the Description says to set EXPAND:TRUE if the 
CUA wants all instances of a recurring component.  Why not just use a 
simple QUERY like:

           BEGIN:VQUERY
           QUERY:SELECT * FROM VEVENT
           WHERE UID = 'uid123'
             AND STATE() = 'BOOKED'
           END:VQUERY

        which should cause the CS to return all instances of UID:uid123 in 
that calendar?

        3B: If there were multiple instances of a recurring component that 
match a date-time range query , wont the CS return both matching instances 
without EXPAND?  For example, I have a semi-weekly W & F Project Leaders 
meeting that repeats all year.  As such if my CUA were to use a date-time 
range query like:

           BEGIN:VQUERY
           EXPAND:TRUE
           QUERY:SELECT * FROM VEVENT
           WHERE DTSTART >= '20031208T050000Z'
           AND DTEND <= '20031213T045959Z'
           AND STATE() = 'BOOKED'
           END:VQUERY

to get all my booked entries for this week (Im in EST which is -5 from GMT 
hence the odd looking times) then I would expect the CS to return to me 
something like:

   BEGIN:VCALENDAR
   PRODID:-//someProdID
   VERSION:2.0
   BEGIN:VEVENT
   ORGANIZER:[email-masked]
   ATTENDEE;MEMBER="Dev Project Leads";PARTSTAT=ACCEPTED:[email-masked]
   DTSTART:20031210T140000Z
   DTEND:20031210T143000Z
   SUMMARY:Semiweekly PL Meeting
   RECURRENCE-ID:20031210T163000Z
   SEQUENCE:4
   UID:12345
   END:VEVENT
   PRODID:-//someProdID
   VERSION:2.0
   BEGIN:VEVENT
   ORGANIZER:[email-masked]
   ATTENDEE;MEMBER="Dev Project Leads";PARTSTAT=ACCEPTED:[email-masked]
   DTSTART:20031212T150000Z
   DTEND:20031212T153000Z
   SUMMARY:Semiweekly PL Meeting
   RECURRENCE-ID:20031212T163000Z
   SEQUENCE:6
   UID:12345
   END:VEVENT
   END:VCALENDAR

where the CS returns the data for the 2 repeat instances that occur this 
week.  There should be no need for an EXPAND:TRUE since the CS should 
return all entries (repeating instances or not) that match the search 
criteria without the CUA saying "Give me ALL instances of any repeating 
instances.". 

So I have to question either the usefullness of EXPAND or the way it is 
described in relation to the acutal intent of it.   As I mull over making 
sure the questions are accurate and clear I think I figured out what the 
intent may have been but just not well phrased. 
Could it be that the intent was to tell the CS "If Im getting all 
repeating instances of a particular component, I prefer to get back 
explicit, unwound individual VEVENTs instead of the base defintion 
followed by any changed instances."?  That is, for my question 3A, there 
are 2 possible ways to return the data:

1: The base set defintion and then all changed instances like:

   BEGIN:VCALENDAR
   PRODID:-//someProdID
   VERSION:2.0
   BEGIN:VEVENT
   ORGANIZER:[email-masked]
   ATTENDEE;MEMBER="Dev Project Leads
";PARTSTAT=NEEDS-ACTION:[email-masked]
   DTSTART:20030106T160000Z
   DTEND:20030106T163000Z
   SUMMARY:Semiweekly PL Meeting
   RRULE:FREQ=WEEKLY;UNTIL=20031231T235959Z;BYDAY=WE,FR
   SEQUENCE:0
   UID:12345
   END:VEVENT
... [Assorted instances that differ from their 'base' content including:]
   BEGIN:VEVENT
   ORGANIZER:[email-masked]
   ATTENDEE;MEMBER="Dev Project Leads";PARTSTAT=ACCEPTED:[email-masked]
   DTSTART:20031210T140000Z
   DTEND:20031210T143000Z
   SUMMARY:Semiweekly PL Meeting
   RECURRENCE-ID:20031210T163000Z
   SEQUENCE:4
   UID:12345
   END:VEVENT
...
   END:VCALENDAR

so that the CUA can unwind the RRULE and replaces those changed instances 
w/the ones it finds afterwards OR

2: All the instances rolled out as individual VEVENTs. 

As I try to compare the usefullness of both ideas one thought comes to 
mind as being a problem with #2 above.  If the repeating entry repeats 
infinitely, EXPAND:TRUE will cause the CS to unwind the instances 
indefinitely.  This is not goodness and something we should address in CAP 
(an implementation issue, security issue, or the like).  In any case, is 
anyone able to answer the questions I have regarding EXPAND:TRUE (or 
confirm my epiphany)?

Bruce
PS: Note to CAP editor(s): The Description text should read "EXPAND:TRUE" 
(in dquotes) not "EXPAND=TRUE" (no dquotes) and the multiple "false" 
should be "FALSE". 
===========================================================================
Bruce Kahn                                INet: 
[email-masked]
Messaging & Collaboration                 Phone: 978.399.6496
IBM Software Group                         FAX: and nothing but the FAX...
Standard disclaimers apply, even where prohibited by law...

---

From: Doug Royer <[email-masked]>
Date: Thu, 11 Dec 2003 13:28:33 -0700

See RECUR-LIMIT.

> indefinitely.  This is not goodness and something we should address in 
> CAP (an implementation issue, security issue, or the like).  In any 
> case, is anyone able to answer the questions I have regarding 
> EXPAND:TRUE (or confirm my epiphany)?

-- 

Doug Royer                     |   http://INET-Consulting.com
-------------------------------|-----------------------------
[email-masked]                 | Office: (208)520-4044
http://Royer.com/People/Doug   |    Fax: (866)594-8574
                               |   Cell: (208)520-4044

               We Do Standards - You Need Standards

---

From: Doug Royer <[email-masked]>
Date: Thu, 11 Dec 2003 13:45:49 -0700

There is (was?) a big vendor that always converted any recurring 
instances upon receipt to
an unspecified number of RDATEs. They removed the RRULE and EXRULE 
properties
then stored the object.

There is (was?) another big vendor that always expanded repeating 
instances to
single instances on the way out to an unspecified maximum number of 
instances.

There were some vendors that did NOT support RRULE or EXRULE, they would
only except RDATE.

We had discussions and no big vendor would budge at all, so we just set 
flags.
The RECUR-ACCEPTED, RECUR-LIMIT, and RECUR-EXPAND capability
replies were added so the CUA could be able to ask the CS (and CS ask 
the CUA).

As iTIP has fallback and errors to those same problems that pre-date 
CAP, we did
not feel that CAP should address those iTIP issues, just allow the 
endpoints to determine
that the problem existed prior to transferring data.

The only way to solve this problem is in iTIP-next, mandate some kind of 
minimum
or overlapping sets of rules.

I do not think CAP created this problem, I think it is an iTIP issue 
that CAP
allows you to find.

--

---

From: [email-masked]
Date: Fri, 12 Dec 2003 10:53:24 -0500

> See RECUR-LIMIT.

That and STORES-EXPANDED (which is interrelated) but I dont think all 
bases are covered yet.

RECUR-LIMIT is defined as:

   Purpose: This property specifies the maximum number of instances the
   endpoint will expand instances into at query or storage time.
[snip, snip]
   Description: For implementations that have the "STORES-EXPANDED"
   value set to TRUE, then this value specifies the maximum number of
   instances that will be stored and fetched. For all implementations
   this is the maximum number of instances that will be returned when
   the "EXPAND" parameter is specified as TRUE and the results contain a
   infinite or large number of recurring instances.

and STORES-EXPANDED is defined as:

   Purpose: This property specifies if the sending endpoint expands
   recurrence rules prior to storing them into the CS.
[Snip, snip]
   Description: If the value is TRUE then the endpoint expands
   recurrence rules and then stores the results into the CS. If this is
   TRUE then the "RECUR-LIMIT" property is significant because an
   infinitely recurring appointment will be stored no more than
   "RECUR-LIMIT" property values into the CS and all other instances
   will be lost.

The unclear/unspecified bits I find are:

1: RECUR-LIMIT says that its a limit for query results or for storage 
time.  However this latter bit is inaccurate if STORES-EXPANDED is FALSE.  
Both STORES-EXPANDED and RECUR-LIMIT need to have prose in them that 
clearly indicates that RECUR-LIMIT is not used as a storage constraint if 
STORES-EXPANDED is FALSE (there is only text now to the effect o 
STORES-EXPANDED:TRUE).

2: There is no indication of how to indicate "no limit", if we want to 
support that.  There is an example of RECUR-LIMIT:0 in the text however 
this violates the ABNF for RECUR-LIMIT which says its a posint1 which is 
defined as:

    posint1     = posintfirst 1*DIGIT

                  ; A number starting with 1 through 9.
                  ;
    posintfirst = %x31-39

so the example on p112 violates the ABNF or is intended to be "I allow 
infinite expansion".  If we do not want to support "No limit" in CAP, we 
need to add some explicit text to that effect.  If we do want to support 
it, we need to say exactly how we indicate that.

3: There is nothing to indicate how the endpoint can tell the requestor "I 
have expanded to my RECUR-LIMIT and there are other instances that  you 
did not get".   This is important for the CUA to know so it can properly 
inform the CU that what they see is NOT all that should be there.  In 
addition, if there is some way to 'resume' the expansion then that should 
be clearly defined.  For example, if the CS says RECUR-LIMIT:10 and 
STORES-EXPANDED:FALSE then if the CUA asks for EXPAND:TRUE on a 30 
instance repeat set there should be some clearly defined way for the CS to 
expand the 1st 10 (RECUR-LIMIT) instances and tell the requestor "Thats 
all ya get but thats not all I have."  The requestor could then refine the 
query so that they get the next 10 instances (11-20) and the response 
"Thats all ya get but not all I have." And so on until they are able to 
extract all of them from the CS w/o having to do the unwinding themselves.

This gets a little messy unless we clearly state that expansions are 
ALWAYS done in chronological order so that the requestor simply tweaks 
their QUERY to include a "AND DTSTART > "last returned instances"" kind of 
clause.  It may be implicit so far but I would expect the CS to return 
expanded results in chronological order but its not written CAP that it 
has to so its possible the results come back in any arbitrary order making 
this resuming of expansion hard to do.

If we do not provide some means for this we run into some usability 
issues.  A CU could store a RRULE;COUNT=364 repeating component into a 
STORES-EXPANDED:FALSE CS whose has RECUR-LIMIT:100 just fine but they 
would be unable to use EXPAND:TRUE to get the results back out and 
possibly even without EXPAND:TRUE set (see #6 below).

4: When a requestor tries to store something that would violate the 
RECUR-LIMIT of the CS, there is no clear describing what the failure 
results/state should be.  That is, is the entire request (for that 
component) rejected entirely?  Is it processed to RECUR-LIMIT and the rest 
ignored/lost? 

5: Does RECUR-LIMIT apply to manually stored instances or it is a limit 
per component no matter how the instances are stored?  That is, if a CS 
says they have RECUR-LIMIT:5 and I try to create a RRULE;COUNT=6 the CS 
would reject it (subject to #4 above).  However, what if the CUA manually 
rolls out the instances themselves (or they get created over time as the 
CU gets added to more and more instances bit by bit!).  Can my CUA create 
6 separate instances of a particular component without running astray of 
RECUR-LIMIT?  Since the CS is not doing the actual expansion Im not 
certain that STORES-EXPANDED or RECUR-LIMIT would be applicable.  After 
all, if I just ACCEPTed an invitation to a 6th instance of the "Super 
Secret Product Ship Party Planning Committee" then I would NOT expect to 
be rejected with some sort of "You tried to store too many instances of 
this repeating component" failure since form my POV I just accepted a 
single instance and not a bajillion repeats. 

6: It is unclear how RECUR-LIMIT is affected by NOT using EXPAND:TRUE. 
That is, is RECUR-LIMIT used to control the number of returned results 
even if the caller did not request EXPAND:TRUE?  For example, if the CS 
has RECUR-LIMIT:5 and a component repeats semi-weekly (W and F) then would 
a query for all of this months entries fail even if I did NOT use 
EXPAND:TRUE?

Bruce
PS: Note to CAP Editor(s): in RECUR-LIMITs Description, "EXPAND" is a 
property, not a parameter.  Also, may I suggest that the Descriptoin be 
split into 2 paragraphs at the point where it talks about the EXPAND 
property (so each related property is in its own paragraph.)
===========================================================================
Bruce Kahn                                INet: 
[email-masked]
Messaging & Collaboration                 Phone: 978.399.6496
IBM Software Group                         FAX: and nothing but the FAX...
Standard disclaimers apply, even where prohibited by law...

---

From: [email-masked]
Date: Fri, 12 Dec 2003 11:41:05 -0500

> We had discussions and no big vendor would budge at all, so we just set 

Given that all vendors (large, medium or small) have to deal with existing 
customer bases I dont find this astonishing.  In any case, the 
flags/settings we create need to be comprehensive and clear for all 
parties so that correct behaviour can be determined.

> that the problem existed prior to transferring data.

The iTIP fallbacks you refer to are protocol level behaviours like:

Method           Fallback
--------------   -----------------------------------------------------
PUBLISH          Required
REQUEST          PUBLISH
REPLY            Required
ADD              Required
CANCEL           Required
REFRESH          Required
COUNTER          Reply with Not Supported
DECLINECOUNTER   Required if EVENT-COUNTER is implemented; otherwise
                 reply with Not Supported

The issues with EXPAND are not germane to iTIP messaging so its not an 
iTIP issue/problem as you contend.  The questions related to EXPAND are 
related to CAP features/abilities and nothing directly related to iTIP.

> I do not think CAP created this problem, I think it is an iTIP issue 

iTIP defines the semantics for doing workflow between 2 parties and 
nothing more.  The issues with EXPAND are non-workflow related and thus 
CAP specific; iTIP does not cover accessing and manipulating calendar 
contents like CAP does.

Now can we try to address the questions so CAP is clear and as 
unambiguious as we can make it?

Bruce

---

From: Doug Royer <[email-masked]>
Date: Fri, 12 Dec 2003 13:37:30 -0700

> That and STORES-EXPANDED (which is interrelated) but I dont think all 

The LIMIT is covered - correct?

The ability to get-next (or what ever) was rejected previously by the WG 
because
when you get the last RECUR-LIMIT item, look at its date/time , start 
from there and
ask again. No explicit get-next needed.

--

---

From: Doug Royer <[email-masked]>
Date: Fri, 12 Dec 2003 13:42:25 -0700

[email-masked] wrote:

> support that.support it, we need to say exactly how we indicate that. 

Why would we want to explicitly consider breaking or hanging a CUA or CS?

--