Subject: (#2) 4.1.1 Grammar for Search Mechanism

From: Doug Royer <[email-masked]>
Date: Wed, 16 Jan 2002 12:33:21 -0700

This posting is what I think is a summary (in detail) of the
"4.1.1 Grammar for Search Mechanism" issues.

If anything here is not what has been discussed, then it is
an error or my misunderstanding. I am NOT attempting to change
anything.   

   EXCEPT:
	(1) Note (3) in "4.1.2 CAP-QL notes" below, where
            I expanded how to sort for float, integer, date,
            and date-time values.

            PLEASE REVIEW.

        (2) I added note (6) at the end - literal values surrounded
            by single quotes.

            PLEASE REVIEW.

        (3) I PROPOSE the name change from SQL-MIN to CAP-QL.
            Where CAP-QL stands for CAP QUERY LANGUAGE.

            PLEASE REVIEW.

        (4) I added note (7) how to compare date to date-time.

            PLEASE REVIEW.

        (5) I added note (8) notes on usage of the LIKE clause.

Comments:

      (1) This proposal does not include the sections of CAP
          that describe the usage of this ABNF.

      (2) ABNF is tricky stuff - PLEASE REVIEW.
 
      (3) This proposes DOES NOT YET include the '(' ')' that
          we talked about for grouping logical operators. For
          two reasons:

              (a) Off of the top of my head, I did not know
                  where to put them in the ABNF.

              (b) They were just seriously re-proposed and there
                  has been no explicit debate yet.
    
----------------------------------------------------------- 

PROPOSAL text for 4.1.1 follows (and new 4.1.1.1)

-----------------------------------------------------------

4.1.1 Grammar for Search Mechanism

  (1) All components look like tables for the purpose of
      a VQUERY including VALARM. And their contained properties
      looked like columns in those tables.

  (2) All VAGENDAs and CS's look like tables for the purpose of a
      VQUERY. And all of their properties look like columns in
      those tables.

  (3) You CAN NOT do any cross component-type joins. And that means
      you can ONLY have one component, OR one VAGENDA OR one CALSTORE
      in the the FROM clause.

  (4) Everything in the SELECT and WHERE clauses MUST from the
      component type, or VAGINA OR CALSTORE in the FROM clause.

  (5) The '.' means <table>.<column> when needed for contained
properties
      or their associated contained components. As long as all were
      contained (as defined by iCalendar or CAP) in the SINGLE component
      type named in the FROM clause OR its contained component that
      MUST BE in the select clause. If you use a contained component
      in the WHERE clause, you MUST get ALL of that components data
      (and no double '.') in the SELECT clause for that contained
      component type.

      This prevents possible cross virtual table joins that could
      occur if the WHERE clause contained information that could
      be in another virtual table than the data in the SELECT clause.

  (6) A contained component without a '.' it is the same as
      <component>.* with the result being a properly formatted
      <component>(s) in the data stream, and correctly formatted
      in the contained component(s) in iCalendar (RFC2445) format.

        VALID:

                (a) SELECT VEVENT.<a-property-name> FROM VEVENT

                (b) SELECT VEVENT.VALARM FROM VEVENT

                (c) SELECT VALARM FROM VEVENT

                (d) SELECT VEVENT.* FROM VEVENT

                (e) SELECT * FROM VEVENT

                (f) SELECT * FROM VEVENT WHERE
                     TRIGGER < 20020201T000000Z
                     AND TRIGGER > 20020101T000000Z

                Note: (a) Selects all instances of <a-property-name>
                      from all VEVENTs.

                      That (b), (c), and (d) yield the same results.
                      And that is select all VALARMS from all VEVENTS.

                      (e) selects every property and every component
                          that is in any VEVENTs.
                  
        NOT VALID:

                (g) SELECT VEVENET.VALARM.TRIGGER FROM VEVENT

                (h) SELECT DTSTART,UID FROM VEVENT WHERE
                     VEVENT.TRIGGER < 20020201T000000Z
                     AND VEVENT.TRIGGER > 20020101T000000Z

                 Note: (g) Is NOT valid because it contains
                           two '.' characters in the SELECT clause.

                       (h) Is NOT valid because it violates rule (5).
                           The error is that VEVENT.TRIGGER is in the
                           WHERE clause and it is not directly or
                           indirectly (by '*' or <something>.'*') in
                           the SELECT clause.

4.1.1.1 "VQUERY ABNF"

    search     = "BEGIN:VQUERY" CRLF
                 [expand] querycomp
                 "END:VQUERY" CRLF

               # If not provided, EXPAND default to FALSE
    expand     = "EXPAND" ":" ( "TRUE" / "FALSE") CRLF

    comp-name  = "VEVENT"    / "VTODO"  / "VJOURNAL"
               / "VTIMEZONE" / "VALARM" / "VFREEBUSY"
               / "VAGENDA"   / "VCAR"   / "CALSTORE"
               / iana-name   / x-name

    querycomp  = ( query ) / ( queryname query ) / queryname

    queryname  = "QUERYNAME:" text CRLF

    query      = "QUERY:" capselect CRLF

                 # NOTE: There is exactly one space separating
                 # the various parts of capselect
                 #
    capselect  = ( "SELECT" " " cap-cols " "
                   "FROM"   " " cap-tbl " "
		   "WHERE"  " " cap-cmps

                 / "SELECT" " " cap-cols " "
                   "FROM"   " " cap-tbl

                 / "SELECT  " " cap-cols " "
                   "FROM"   " " cap-tbl  " "
                   "USING_PROPERTIES" " " cap-col cap-local
                   "WHERE"  " " cap-cmps )

    cap-col     = # Any property name found in the component
                  # named in the comp-tbl used in the FROM clause.
                  #
                  #   SELECT ORGANIZER FROM VEVENT ...
		  #
                  # OR
                  #
                  # A component name of an existing component contained
                  # inside of the cmp-tbl used in the FROM clause.
                  #
                  #   SELECT VALARM FROM VEVENT ...

    cap-tbl     = ( # Any known component type
                  / "CALSTORE"
                  / "VAGENDA" )

                  # NOTE: there is NO space around the "," on
                  # the next line
    cap-cols    = ( cap-col / cap-col "," cap-cols )
                  / "*"

    cap-param   = # Any parameter that may be contained in the cap-col
                  # in the supplied PARM() function.

    cap-local   = # Any string that is composed of the characters 
                  # that could be a cap-col name, but is not any
                  # cap-col name. It is suggested that the
                  # string start with "x-" to ensure it does not
                  # conflict with any existing or future cap-col name.
                  # This name MUST BE defined in the cap-using and
                  # can only be used in cap-cmps of the same query.
                  # And this name is only known and valid for the
                  # provided query and only for the lifetime of
                  # the query.

    col-literal = "'" literal-data "'"

    literal-data= # Any data that matches the value type of the
                  # column that is being compared. That is you can
                  # not compare PRIORITY to "some string" because
                  # PRIORITY has a value type of integer.
                  # 
                  # OR if the literal-data is proceeded by the LIKE
                  # element it may also contain the '%' and '_'
                  # wildcard characters.

    cap-ucol	=   cap-col / cap-local

    cap-cmp     = ( cap-ucol cap-cmd-rhs
                  / cap-ucol cap-oper cap-literal
                  / "PARAM(" cap-col "," cap-param ")" cap-cmp-rhs
		  / "CONTAINS(" cap-col "," col-literal ")"
                  / cap-logical

                  # NOTE: there is NO space around the "," on
                  # the next line
    cap-cmps    = cap-cmp
                / cap-cmp "," cap-cmps
                / cap-cmp cap-logical cap-cmps

    cap-cmp-rhs = ( cap-oper col-literal
                  / "IS NULL" )
                  / "IS NOT NULL"
                  / "LIKE" " " col-literal ) # Where the SQL '%' and '_'
                                             # Wildcard characters may
                                             # be used in col-literal

    cmp-oper    = ( " = " 
                  / " != "
                  / " < "
                  / " > "
                  / " <= "
                  / " >= " )

    cap-logical = ( " AND " / " OR " )

4.1.2 CAP-QL notes

      (1) No in-lined spaces are allowed if not in the grammar above.

      (2) Note that cmp-oper and cap-logical elements are
      surrounded by exactly one space.

      Use: "VEVENT,VTODO" 

      Not: "VEVENT, VTODO"  (Space before or after ',' not allowed)

      Use: "DTSTART <= '20000605T131313Z'"
 
      Not: "DTSTART<='20000605T131313Z'" (Exactly one space surrounds
                                          compare operators)

      Use: " AND " and " OR "            (Exactly one space surrounds
                                          compare operators)
 
      Not: "AND"   and not "OR"          (Exactly one space surrounds
                                          compare operators)

      (3) There is no ORDERBY.  Sorting will take place in the order the
      columns are supplied in the command.

      Float and integer values MUST BE sorted by their numeric value.

      This means the result of a sort on an integer value type will be:
          
                1, 2, 100, 1000

        and not

                1, 100, 1000, 2

      This means the result of a sort on an float value type will be:
          
                1.1, 2.23, 100.332, 1000.12

        and not

                1.1, 100.332, 1000.12, 2.23

      Date and date time values will be sorted by their equivalent
      value in UTC. No matter what the returned time zone in the result
      set returns. This is so that if multiple components are returned
      each in a unique time zone, the results will be sorted in UTC.
      This does not mean the values must be converted to UTC in the
      data returned to the CUA. It means the CS must do the sort in UTC.

      All other values are sorted according to the locale sorting order
      as specified in the calendar. Or the CS locale if the calendar
      does not have any locale set, or the host operating system
      locale if the CS does not specify a locale. And the locale to
      use for the sort is determined in that order.

      (4) The CS MUST sort at least the first column.
          The CS MAY sort additional columns.

      (5) If the cap-cols is only "*" and nothing else, then:

            If EXPAND=FALSE sorting will be by the DTSTART value
            ascending.

            If EXPAND=TRUE sorting will be by the RECURRENCE-ID value
            ascending.

          If one or more DTSTART or RECURRENCE-ID components have
          exactly the same value, the order for those matching
          components is unspecified.

      (6) All literal values are surrounded by single quotes ('), not
          double quotes ("), and not without any quotes. If the value
          contains quotes or any other ESCAPED-CHAR, they must be
          backslash escaped as described in section "4.3.11 Text" 
          of RFC2445. Any LIKE wildcard characters that are part
          of any literal data that is followed by a LIKE clause and
          is not intended to mean wildcard search, MUST BE escaped as
          described in [SQL92].

      (7) When comparing DATE-TIME to DATE value types and when
          comparing DATE to DATE-TIME value types, the result will
          be true if the DATE value is  on the same day as the DATE-TIME
          value. And they are compared in UTC no matter what time zone
the
          data may actual have been stored in.
          
            VALUE-1             VALUE-2            Compare Results

            20020304            20020304T123456    TRUE
            (in UTC-3)          (in UTC-3)

            20020304            20020304T123456    FALSE
            (in UTC-4)          (in UTC-4)

            20020304T123456Z    20020205T123456    FALSE
            (in UTC-0)          (in UTC-7)

         When comparing DATE and DATE-TIME values with the LIKE
         clause the comparison will be done as if the value is
         a RFC2445 DATE or DATE-TIME string value.

                LIKE '2002%' will match anything in the year 2002.
                LIKE '200201%' will match anything in January 2002.
                LIKE '%T000000' will match anything at midnight.

         Again all comparisons will be done in UTC.

     (8) LIKE notes:

         The pattern matching characters is the '%' that matches
         zero or more characters, and '_' that matches exactly one
         character (where character does not always mean octet).

         LIKE pattern matches always cover the entire string. To match
         a pattern anywhere within a string, the pattern must start and
         end with a percent sign. 

         To match a '%' or '_' in the data and not have it interpreted
         as a wildcard character, they must be backslash escaped.
         That is to search for a '%' or '_' in the string:

               LIKE '%\%%'    Matches any string with a '%' in it.
               LIKE '%\_%'    Matches any string with a '_' in it.

---

From: "Shannon J. Clark" <[email-masked]>
Date: Wed, 16 Jan 2002 14:18:54 -0600

Doug,

A couple of silly typos...

Correct text for 4.1 (4) follows: (added be and correct VAGINA to VAGENDA)

4.1 

	(4) Everything in the SELECT and WHERE clauses MUST be from the
      component type, or VAGENDA OR CALSTORE in the FROM clause.

Shannon
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Doug Royer
Sent: Wednesday, January 16, 2002 1:33 PM
To: [email-masked]
Subject: (#2) 4.1.1 Grammar for Search Mechanism

   EXCEPT:

            PLEASE REVIEW.

            PLEASE REVIEW.

            PLEASE REVIEW.

            PLEASE REVIEW.

Comments:

 

    

        VALID:

                (e) SELECT * FROM VEVENT

                  
        NOT VALID:

4.1.1.1 "VQUERY ABNF"

    cmp-oper    = ( " = " 
                  / " != "
                  / " < "
                  / " > "
                  / " <= "
                  / " >= " )

4.1.2 CAP-QL notes

      Use: "VEVENT,VTODO" 

 

          
                1, 2, 100, 1000

        and not

                1, 100, 1000, 2

        and not

          

     (8) LIKE notes:

---

From: Alan Davies <[email-masked]>
Date: Wed, 16 Jan 2002 16:23:37 -0500

Looks like a great step forward Doug; we're starting
to fill in the holes. I've not studied it in-depth,
but I've a couple of initial comments:

>                      AND TRIGGER > 20020101T000000Z

How does the CS know that TRIGGER refers to the VALARM
TRIGGER . What If I'd wanted to refer to the VALARM DURATION,
which would have been ambiguous with the VEVENT DURATION?

I suppose I could use VALARM.DURATION :

                  SELECT * FROM VEVENT WHERE
                      VALARM.TRIGGER < 20020201T000000Z
                      AND VALARM.TRIGGER > 20020101T000000Z
                      AND VALARM.DURATION < PT15M

but if there were multiple VALARMS, how would the CS
know which was being referred to?

Seems to me that we need to specify multi-components in a similar way
to multi-properties. These would therefore both be valid:

                  SELECT * FROM VEVENT WHERE
                      VALARM.TRIGGER < 20020201T000000Z
                      AND VALARM.TRIGGER > 20020101T000000Z
                      AND VALARM.DURATION < PT15M

                   (refers to the any one VALARM within the VEVENT)

                  SELECT * FROM VEVENT
                      USING_COMPONENTS VALARM alarm1, VALARM alarm2
                      WHERE
                      alarm1.ACTION = 'DISPLAY'
                      AND alarm1.TRIGGER < 20020116T000000Z
                      AND alarm1.TRIGGER > 20020117T000000Z
                      AND alarm2.ACTION = 'EMAIL'
                      AND alarm2.TRIGGER < 20020201T000000Z
                      AND alarm2.TRIGGER > 20020101T000000Z

(i.e. I'm looking for events with a 'DISPLAY' VALARM,
and an 'EMAIL' VALARM because... I couldn't think of a better
example right now)

>                            indirectly (by '*' or <something>.'*') in

I'm not sure what we're gaining from rule (5) here; would
it also disallow this, which seems perfectly sensible to me:

     SELECT UID FROM VEVEI Z

--Alan

---

From: Alan Davies <[email-masked]>
Date: Wed, 16 Jan 2002 16:40:33 -0500

Oops, that last line should have read:

>>     SELECT UID FROM VEVENT WHERE DTSTART < 20020201T000000Z

--Alan

---

From: Doug Royer <[email-masked]>
Date: Wed, 16 Jan 2002 16:09:24 -0700

> which would have been ambiguous with the VEVENT DURATION?

TYPO in my data :-) They should have been:

		VALARM.TRIGGER ... and not just TRIGGER

> know which was being referred to?

Any that matched. It was not trying to extract out a specific
VALARM, it was trying to extract out all VEVENTs where the TRIGGER
times were within a range. So any/all VALARMs that meets those
criteria would match.

>                    (refers to the any one VALARM within the VEVENT)

Yes, the example above would fetch all VEVENTs including ALL
of their VALARMS that met the criteria.

> and an 'EMAIL' VALARM because... I couldn't think of a better

After you correct for my typo (listed above), I think
we can do this? Correct?

>>     SELECT UID FROM VEVENT WHERE DTSTART < 20020201T000000Z

I have no clue :-)
It made sense when I wrote it :-)

---

From: Patrice Lapierre <[email-masked]>
Date: 17 Jan 2002 09:57:13 -0500

On Wed, 2002-01-16 at 14:33, Doug Royer wrote:
...
>             Where CAP-QL stands for CAP QUERY LANGUAGE.

I also prefer "CAP-QL".

> ...

  I included some corrections.

>                   where to put them in the ABNF.

I think they could be added be replacing cap-cmps by cap-expr 
as follows:

      cap-expr = "(" cap-expr ")"
               /  cap-term

      cap-term = cap-expr SP cap-logical SP cap-expr 
               / cap-factor 
               
      cap-factor = [ "NOT" SP ] "CONTAINS(" cap-lhs "," col-literal ")"
                 / cap-lhs SP cap-oper SP col-literal
                 / cal-lhs SP "IS" SP [ "NOT" SP ] "NULL"
               
      cap-logical = "AND" / "OR"      

      cap-lhs = "PARAM(" cap-ucol "," cap-param ")"
              / cap-ucol

      cap-oper    = ( "=" 
                    / "!="
                    / "<"
                    / ">"
                    / "<="
                    / ">=" 
                    / [ "NOT" SP ] "LIKE"
                    )

NOTE: For completeness, I also added an optional "NOT" in 
front of CONTAINS, and LIKE.

PLEASE REVIEW (I'm new to ABNF).

> ...

I think that EXPAND and QUERY should include the X-PARAM.

 expand     = "EXPAND" *(xparam) ":" ( "TRUE" / "FALSE") CRLF

(xparam is defined in rfc445). 

>                / "VAGENDA"   / "VCAR"   / "CALSTORE"

   "comp-name" is not used in the ABNF.

>     queryname  = "QUERYNAME:" text CRLF

 queryname  = "QUERYNAME" *(xparam) ":" text CRLF

>     capselect  = ( "SELECT" " " cap-cols " "

 This is a detail, but I prefer the use of the core rule SP 
instead of " ". Also see comments on (1) and (2) bellow. 

e.g., 
       pselect  = ( "SELECT" SP cap-cols SP
                    "FROM"   SP cap-tbl SP

>                    "USING_PROPERTIES" " " cap-col cap-local

 1. The USING_PROPERTIES property clause should accept a list.
 2. Missing space between in cal-col and cal-local

   ...
         USING_PROPERTIES" SP cap-local-decls
   ...
   
   cap-local-decls = cal-local-decl *( "," cap-local-decl )

>     cap-tbl     = ( # Any known component type

The avoid confusion supported element should be explicitly listed.

   "VQUERY" should also be included (not in comp-name above).

>                   / "PARAM(" cap-col "," cap-param ")" cap-cmp-rhs

    First argument of PARAM should be cap-ucol:
    
           / "PARAM(" cap-ucol "," cap-param ")" cap-cmp-rhs

> 		              / "CONTAINS(" cap-col "," col-literal ")"

 1. The first arg of CONTAINS should be cap-ucol + PARAM()
    (see suggested ABNF for sub-expressions).

 2. For completeness "NOT CONTAINS" should probably by added.   

>                   / cap-logical

                    )

Missing a closing ')'.

>                 / cap-cmp cap-logical cap-cmps

 cap-cmp is not defined (cmp-oper).

>                                              # Wildcard characters may

  For completeness, "NOT LIKE" should probably by added.

>     cmp-oper    = ( " = " 

 "cmp-oper" is referred to as cap-oper above.

>       Not: "AND"   and not "OR"          (Exactly one space surrounds

 I propose to remove (1) and (2) and to adjust the ABNF accordingly.
It only seems to add complexity (~10 lines of descriptions and 
comments in ABNF). Some spaces are required (e.g., around AND, OR),
but imposing exactly one is not needed. 

  I also propose to support tabs. This could be achieve by 
replacing most " " in the ABNF with *WSP.

WSP is defined in rfc2234 as:

   WSP = ( SP / HTAB )
   SP = %x20
   HTAB = %x09