Subject: RE: Can we fix BYSETPOS?

From: [email-masked]
Date: Fri, 8 Jun 2001 12:43:47 -0400

Shannon replied:
>Restrict BYSETPOS to events of value DATE - that is restrict it to events
>enumerated by DAY?

Sounds tempting at first blush but it would prevent the use of BYSETPOS in 
models like the SKiCAL where they do have stuff that can repeat sub day 
(ie: A city tour that repeats 3 times a day during the week).

>Just a thought - if it was intended for use when DAYS are enumerated, why
>not clearly define and specify that that is the case?

A scan of the archives will reveal a terse response from Frank to the 
original notice of the oversight saying it was a mistake.  He can also 
confirm this if he is still lurking around...

Bruce
===========================================================================
Bruce Kahn                                INet: [email-masked]
Iris Associates                          Phone: 978.392.5335
Westford, MA, USA 01886                    FAX: and nothing but the FAX...
Standard disclaimers apply, even where prohibited by law...

---

From: [email-masked]
Date: Fri, 8 Jun 2001 12:38:14 -0400

Eric wrote:
>BYSETPOS is problematic for a number of reasons, but there may be a way 
to
>fix it so that CPL can use it.

While this is a good idea I think the larger goal should be to 'fix' it so 
that it is useful for all implementors of the recurrence grammar, not just 
iptel/CPL.  Doing this has the side effect of making it work for everyone 
instead of just another WG that is reusing it.

>The way BYSETPOS is currently specified is that it can take values from 1
>to 366 and -1 to -366. Clearly is was intended to enumerate days, but
>there is no restriction to keep it from being used on a rule that can
>generate intra-day events, potentially millions of values in the
>occurrence set, such as by specifying every second of a year.
>It is silly for BYSETPOS to only let you access the first 366 and the 
last
>366 of a set of 31 million events.

We noted this a long time ago (earlier than last 1-Jun-00 "Max # of 
occurences in a recur rule?" thread on the topic and Frank concurred in a 
reply that indeed it was an oversight that needed to be fixed.

>                                There are a few possible fixes for
>this:
>
>                (1) Let BYSETPOS take a full range of vlaues from -31 
million to
>                +31 million.
>
>                (2) Declare that BYSETPOS cannot be used on a rule that 
gnenerates
>                more than 366 events before BYSETPOS is applied. ( Throw 
an error)
>
>                (3) Declare that when BYSETPOS appears, the 
implementation must
>                only consider the first 366 occurrences it generates to 
be part of
>                the set that BYSETPOS is applied to.
>
>(1) is the concept that makes BYSETPOS it unusuable for CPL.  However, 
(2)
>and (3) imply that an implementation only has to expand 366 events -- 
they
>are restrictions that bound the execution time for BYSETPOS.

I think we all agree that +/-366 is inadequate and needs to be corrected. 
Sure, +/-31 million is overly large for us to visualize now but Im not 
sure that its a reason to NOT go that route w/some caveats.

I think that the proper behavior of ANY CUA that does not like either 
BYSETPOS, its size or its values (ie: there are 68K values on it) should 
be to reject the rule as "too complex".  This said, I think that CPL or 
other implementors will find taking that approach a more robust one going 
forward.  As systems get faster or feature sets get more robust, they can 
extend the size of the grammar (ie: BYSETPOS values) that they support; 
all w/o requiring ANY RFC changes!

WRT the individual cases above, I think that (1) is probably the best 
course to logically take for reuse and growth reasons.  (2) is not good 
since it prevents something like SKiCAL from doing a "The City Tour begins 
at 9AM and again at 12PM every day of the year except New Years Day" kinds 
of stuff.  (Or actually any recurrence set that can generate > 366 
instances; not too good.) (3) is equally unlikable as (2) for many of the 
same reasons or worse yet, it could result in a recurrance instance that 
is actually desired but 'unreachable'.  (Consider the set that has 733 
instances in it, there is 1 instances that is 'unreachable'.)

Just as we have a way to indicate "I do not grok the recurrence 
subcomponent BYSETPOS (or BLECH, etc)" and rejecting the rule as 
"unsupportable" or "too complex" I think that the proper course of action 
is to adopt (1).  This coupled w/the fact that it is possible to actually 
specify a VERY big set using BYxxx modifiers (31,622,400 by my count w/o 
leap seconds) I think we really must have a way to filter out only the 
desired instances from the full pattern (the purpose of BYSETPOS) . 
Otherwise BYSETPOS really is only partly functioning compared to its 
intended use.

Now, do I actually expect someone to generate a recurrence rule that 
evenly remotely comes close to this; not in the forseeable future.  Does 
it go against my KISS mantra; to some degree, yes.  However from a 
technical perspective I think that having a 'complete' solution is better 
than one that has "holes" in it.  After all, just where should we draw the 
line at the limit of BYSETPOS between 366 and 31,622,400??  1K? 10K? 1M? 
10M?  For our (CalSched) uses I do not see any real cases of "An event 
that repeats every second, of every day, all year long".  However we are 
trying to make our design reusable to other areas where it may be 
necessary to have a value above our self-prescribed limit and below the 
theoretical max of 31,622,400.

Im not trying to sound like a broken record here but I do think we need to 
not restrict ourselves by fixing something for a smaller subset of 
potential users when we should be fixing it for any adopter of the 
grammar.

Bruce

---

From: [email-masked]
Date: Tue, 12 Jun 2001 17:06:09 -0400

Jonathan Lennox <[email-masked]> wrote on 06/11/2001 06:31:14 PM:
> > The FREQ is yearly so that alone would make the entry repeat on a 
basis of 
> > 1 year.  You do not factor in the INTERVAL value when trying to 
calculate 
> > the set; it is used when moving the starting point for the next 
'rollout'. 
> > As such, the only modifier you have to worry about is the BYSECOND. 
> >
> > So since your DTSTART date/time value is only being 'expanded' using 
> > BYSECOND, the only part of it that will be changed (and added to the 
> > 'set') is the seconds value of the time.  For example, had you 
provided a 
> > DTSTART;TZID=US-Eastern:20010611T090000 then the 'set' would evaluate 
out 
> > to:
> > 
> > 11-Jun-2001 09:00:[00, 01, 02, 03, 04,..., 58, 59] EST
> 
> Ah!  This is the thing I missed.  Again, a serious omission in the RFC 
2445
> examples.

May I suggest that when we rev the RFC we make an effort to include some 
examples like this for the future benefit of others??  I think thats 
something that Eric would have to take on (but at least he can copy my 
example for starters).  Id be willing to help out on some other examples 
(in a few weeks) too.

> The question is, what do you do for fields *larger* than your smallest
> BY* parameter, but smaller than FREQ?
> 
> This is only relevant if there's a "gap" in the information.
> 
> For instance:
>     DTSTART;TZID=US-Eastern:19970902T090000
>     RRULE:FREQ=YEARLY;BYMONTHDAY=1
> 
> I had interpreted this as meaning
>     ==> (1997 9:00 AM EDT) September 2, October 1
>         (1997 9:00 AM EST) November 1, December 1
>         (1998 9:00 AM EST) January 1, February 1, March 1
>   ...
> 
> (I.e. treat the "gap" information as "wild" -- as if we had
> BYMONTH=1,2,3,4,5,6,7,8,9,10,11,12.)
> 
> Instead, you're saying it's
>     ==> (1997 9:00 AM EDT) September 2
>         (1998 9:00 AM EDT) September 1
>         (1999 9:00 AM EDT) September 1
>   ...

Correct (it is akin to the "Every Friday the 13th, forever:" example).

> (I.e. treat it the same as if we had BYMONTH=9.)

Hmm, I had not viewed it that way so I don't know if that is accurate to do that.  I simply used the FREQ/INTERVAL values to pick my 
next effective DTSTART and then applied the BYxxx subcomponent in the 
prescribed order.

> Note that the only RFC 2445 examples that address anything like this
> question are the ones of the form
>      DTSTART;TZID=US-Eastern:19970902T090000
>      RRULE:FREQ=MONTHLY;INTERVAL=2;BYDAY=TU
>      ==> (1997 9:00 AM EDT)September 2,9,16,23,30
>          (1997 9:00 AM EST)November 4,11,18,25
>          (1998 9:00 AM EST)January 6,13,20,27;March 3,10,17,24,31
> 
> In this example, we must treat the week of the month as "wild" --
> i.e. BYDAY=TU is equivalent to BYDAY=1TU,2TU,3TU,4TU,5TU in a MONTHLY 
rule.
> (And to BYDAY=1TU,2TU,...,53TU in a YEARLY rule.)

Not sure Id say that I wildcarded anything again.  It is simply a matter 
of generating the proper 'filter' for expanding/contracting the set (in 
the prescribed order).  Since 2445 says that BYDAY=MO is equivalent to 
BYDAY=1MO,2MO,3MO,4MO,5MO (or the corresponding negatives for that matter) 
I can do an expansion internally in my grammar evaluator (which you 
consider making the weeks in the month "wild" but I just do not factor 
them in since that is the purpose of BYWEEKNO really).

I then seed my engine w/the DTSTART value and have it find "all Tuesdays 
in the month".  When it is done I move the DTSTART value (internally) by 
the prescribed FREQ/INTERVAL and begin again.  I find that it is possible 
to make the 'filter' up front since it never changes, just the seed that 
gets DTSTART that gets fed in changes.

> Is BYDAY "special" when it comes to filling in gaps, and in all other 
cases
> the gap should be filled in from the DTSTART?  Or are gaps different 
than
> times smaller than the smallest time unit mentioned in a BY*/FREQ?

BYDAY is special in that it can be 'shorthanded' for FREQ=MONTHLY or 
YEARLY.  It also allowed us to reuse the subcomponent w/o having to define 
different allowable values based on the FREQ (than say the expected '1 
off' of a FREQ=WEEKLY).  For a better explaination Id defer to Frank since 
he massaged all our original thoughts/comments into the prose that you 
have confounding you now...

> A very close reading of the RECUR definition seems to show that your
> interpretation -- the "BYDAY is special, gaps aren't" reading -- is
> *probably* right, and I guess it allows slightly more generality. 
However,
> it certainly wasn't clear, or illustrated.

I think that in our push to get iCalendar out we (the WG at the time) did 
not necessarily focus very strongly on the recurrence grammar (or the 
delegation related stuff) and it shows up now as we all try to implement 
it in a consistant manner.  The recurrence grammar addtions/changes were 
the last ones in and after such a long labor I think we just wanted it out 
and were not concerned about "a few warts".  Then again thats just my 
retrospective opinion...  I certainly hope we take the time now to correct 
our oversights/missteps and I think Eric has done a good job doing this so 
far.

Bruce

---

From: [email-masked]
Date: Wed, 20 Jun 2001 11:23:41 -0400

Jonathan pondered:
>Actually, the reason I was confused is that the definition of the set
>*isn't* clear from the examples, because the examples only show the use 
of
>BYSETPOS with expanding BY* rules.  The confusion comes from the use of
>BYSETPOS with contracting BY* rules, which no examples show.  (I suppose 
the
>use of INTERVAL with BYSETPOS isn't technically ambiguous, but *I* 
certainly
>mis-interpreted it.  No examples show this either.)

Sounds like we need to add a bigger variety of new examples to clarify 
things a bit.  Im eyeball deep in work and some outside projects but if 
need be I can try to come up with an example of BYSETPOS in a contracting 
set.  The only real requirement for BYSETPOS though is that the instance 
set the rule generates is not always constant.  Beyond that, its just a 
matter of finding a a contracting (yet variable) set of instances...

>Consider, e.g.:
>
>      DTSTART;TZID=US-Eastern:19970929T090000
>      RRULE:FREQ=WEEKLY;BYMONTH=4,6;BYDAY=TU,TH;BYSETPOS=-1
>
>Does this mean "The last Tuesday or Thursday of the week, every April and
>June"?  Or "The last Tuesday or Thursday of April, and the last Tuesday 
or
>Thursday of June, every year"?  Or "The last Tuesday or Thursday of 
(April
>or June), every year"?

Following the evaluation order from 2445 (p. 44, last paragraph) I would 
say that the rule decodes as follows:

1: The implied INTERVAL=1 means this rule is "Every week" (or "Every week 
in the entire year" if you prefer).
2: "BYMONTH=4,6" makes it "Every week in April & June" (or "In April & 
June, every week" if you strictly follow the RFCs phrasing style).
3: "BYDAY=TU,TH" makes it "Every Tuesday & Thursday [in the weeks] in April & June" if I clean the phrase up a bit (or "Every Tuesday & 
Thursday in April & June, every week" again).
4: "BYSETPOS=-1" means "The last instance in the set of ("Every Tuesday & 
Thursday in April & June")" which translates essentially down into "The 
last Tuesday or Thursday in June" since there are several June instances 
and we are only interested in the 'last one' of the combined set of April 
& June. 

As such, there are no actual used/referenced instances in April so the 
rule could have been simplified to:

RRULE:FREQ=WEEKLY;BYMONTH=6;BYDAY=TU,TH;BYSETPOS=-1

and the result would be the same. 

If the intent was to get "The last Tuesday or Thursday in April AND June" 
then you would need to use 2 RRULEs (1 per month) since the desired 
instance can vary w/in each month really (ie: TU 29-Apr-1997 then TH 
30-Apr-1998, TH 29-Apr-1999, TH 27-Apr-2000, TH 26-Apr-2001, TU 
30-Apr-2002, etc. and the same kind of flipping occurs in June but not 
necessarily in the same order). 

>I'd extend this further, by changing the current sentence in RFC 2445
>    It [BYSETPOS] MUST only be used in conjunction with another BYxxx 
rule
>    part.
>to
>    It MUST only be used in conjunction with other BYxxx rule parts.  At
>    least one of these BYxxx rule parts must be an "expanding" rule part,
>    that is, a BYxxx rule part representing an interval of time smaller 
than
>    FREQ.
>Do you object to this?

I dont have a thought either way on expanding vs contracting w/o trying to 
do a couple examples of contracting (yet variable) cases.  Let me mull it 
over a bit.

Is my assumption that BYSETPOS is only a concern when the instance sets 
are variable correct?  Im assuming it is but so far I haven't heard Yes or 
No from either you or Eric...

Bruce

---

From: Jonathan Lennox <[email-masked]>
Date: Wed, 20 Jun 2001 15:18:02 -0400 (EDT)

On Wednesday, June 20 2001, "Robert Ransdell/Iris" wrote to ""Jonathan Lennox, [email-masked]" saying:

> The BYSETPOS is evaluated over one instance of the set.
> In this case one instance of the set is defined by FREQ=WEEKLY;INTERVAL=1
> For every year, months January, February, March,  May, June, July, August, 
> September, October, November and December becomes empty sets because of the BYMONTH=4 rule.
> For April of every year the weekly expansion is evaluated and than the 
> BYSETPOS item is picked.

So are you proposing (or claiming) "the size of the set is the largest unit
of time mentioned in the rule"?

How would you resolve
    RRULE:DTSTART;TZID=US-Eastern:20010406T090000
    RECUR:FREQ=WEEKLY;BYMONTH=4,6;BYSETPOS=-1
?

-- 
Jonathan Lennox
[email-masked]

---

From: Jonathan Lennox <[email-masked]>
Date: Wed, 20 Jun 2001 16:11:13 -0400 (EDT)

On Wednesday, June 20 2001, "[email-masked]" wrote to "Jonathan Lennox, [email-masked]" saying:

> The FREQ = WEEKLY
> The INTERVAL =1
> The BYMONTH is a reduction restriction.
> 
> DTSTART is 4/6/2001 so starting there
> for WEEKLY in  April I have dates 4/6/2001 (set of one) BYSETPOS=-1  GET 
> 4/6
>                                                    4/13/2001 (set of one) 
> BYSETPOS=-1 get 4/13
>                                                    4/20/2001  (set of one) 
> BYSETPOS=-1 get 4/20
>                                        4/27/2001  (set of one) BYSETPOS=-1 
> get 4/27

... and so on ...

Okay.  So in other words, you agree with Eric and me, and disagree with
Bruce -- this rule means "the last Friday of the week, in April and June",
and the enclosing unit of a BYSETPOS set is the FREQ.  Thus, BYSETPOS is
mostly meaningless if only reduction restrictions (contracting rules) are
present, since only single-element sets are generated.  (This is the cause
of the rather bizarre qualifier "the last Friday of the week," above.)

Bruce was saying (I think) that this rule should instead be interpreted as
generating

6/29/2001
6/28/2002
6/27/2003
...

I.e. "the last Friday of (April or June) in the year", which is equivalent
to "the last Friday of June in the year" since DTSTART is in April.  I don't
understand his logic for this, though.

-- 
Jonathan Lennox
[email-masked]

---

From: [email-masked]
Date: Wed, 20 Jun 2001 16:58:46 -0400

> of the rather bizarre qualifier "the last Friday of the week," above.)

This is all starting to rehash an unresolved issue I raised wwaaaaayyyy 
back on 30-Jun-2000 (Subject: Seeking RRULE evaluation guidance) about handling disjoint BYxxx values in the same subcomponent.  Only 3 
folks even replied at first but we got no further traction on it.  This is 
a digression but perhaps it is relevant to resolve now...

I think we need to get some other vendors invovlved and I know that 
Netscape/iPlanet has done some work here and so has Steltor so Id love to 
hear their ideas on this unresolved issue.  Steve? Mario? Graham?

Bruce

---

From: [email-masked]
Date: Wed, 20 Jun 2001 17:20:45 -0400

> > The BYMONTH is a reduction restriction.
[Snip, snip]
> Bruce was saying (I think) that this rule should instead be interpreted 
as

At some point I was using FREQ=MONTHLY instead of FREQ=WEEKLY.  Im not 
sure where the disjoint happened so I went back to _my_ original example 
and it was:

    DTSTART;TZID=US-Eastern:19970929T090000
    RRULE:FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-2

so I think the thread for the past day or so has been a BIG 
miscommunication somehow.  I checked the archive and found that I did 
start w/MONTHLY 
(http://www.imc.org/ietf-calendar/mail-archive/msg04829.html) so lets try 
to reset here before we all blow some gray cells for nothing.

I need to leave now to run some family errands but Ill pick this up again 
tomorrow. 

Bruce
PS: Has anyone gotten the actual WG postings from 
[email-masked]??  I havent and they do not appear to be in the 
archives so Im curious if Im alone or not...
===========================================================================
Bruce Kahn                                INet: [email-masked]
Iris Associates                          Phone: 978.392.5335
Westford, MA, USA 01886                    FAX: and nothing but the FAX...
Standard disclaimers apply, even where prohibited by law...

---

From: [email-masked]
Date: Thu, 21 Jun 2001 01:04:53 -0400

Bruce, I have not seen posting from Robert Ransdell.
___________________
Patricia Egen Consulting
www.egenconsulting.com
423-875-2652

[email-masked]
Sent by: [email-masked]
06/20/01 05:20 PM

        To:     Jonathan Lennox <[email-masked]>
        cc:     [email-masked], [email-masked]
        Subject:        Re: Can we fix BYSETPOS?

[Snip, snip] 

so I think the thread for the past day or so has been a BIG 
miscommunication somehow.  I checked the archive and found that I did 
start w/MONTHLY (http://www.imc.org/ietf-calendar/mail-archive/msg04829.html) so lets try to reset here before we all blow some gray cells for 
nothing. 

Bruce