Subject: Re: [Extra] Design of a new Sieve action to process iMIP messages

From: Ken Murchison <[email-masked]>
Date: Wed, 20 Oct 2021 09:48:44 -0400

Hi Alexey,

On 9/28/21 5:15 AM, Alexey Melnikov wrote:
>
> Hi Ken,
>
> On 22/09/2021 13:27, Ken Murchison wrote:
>>
>> All,
>>
>> We (Fastmail) are in the process of designing and implementing a new 
>> Sieve action which will process iMIP messages and add/update 
>> resources on a CalDAV calendar.  The main issue that I'm facing is 
>> that a user may want to have additional actions be executed depending 
>> on whether the 'processimip' action [un]successfully updates the 
>> calendar (E.g., add flags to and/or fileinto the email message into 
>> different mailbox).  Neither RFC 5528 nor any extension that I'm 
>> aware of defines a mechanism for determining whether an action was 
>> successful.  Failure of some actions, such as fileinto cause script 
>> execution to stop.  Failure of others such as vacation/notify are 
>> usually ignored.
>>
>> I have come up with 3 ways to solve this issue for the new 
>> 'processimip' action, in order of preference:
>>
>>  1. Have 'processimip' set a variable with a return code/status. 
>>     This variable can then be checked with a 'string' test.  There is
>>     precedent for an action setting a variable with the 'extracttext'
>>     action.
>>  2. Make 'processimip' a test rather than an action.  This would be
>>     the first test that I'm aware of with side-effects (other than
>>     setting match variables).
>>  3. Add optional :flags and :mailbox (and :mailboxid, :specialuse)
>>     args to 'processimip' which are only used if the action is
>>     successful and are used to execute an implicit 'fileinto'.  This
>>     is very ugly IMHO.  I'd prefer that an explicit 'fileinto' action
>>     be used.
>>
>> What do others think?  Any different/better design ideas?
>>
> Your list of preferences seems reasonable. #2 is not ideal, even 
> though it is otherwise compelling. #1 ties you to implementing another 
> extension, even if it is an important one.
>
> Can you tell me more about failure cases and maybe it would be 
> possible to extract some logic into a separate test?
>
Some reasons for failure, could be:

  * Invalid iCalendar data
  * Missing/invalid METHOD property
  * Owner of Sieve script isn't ORGANIZER/ATTENDEE
  * UID of a REPLY or CANCEL can't be found in script owner's calendars
  * Creation of new events from an invite aren't allowed (see
    :updatesonly below)

Here's what we have come up with so far and are beginning to test.  
These options support everything that we are currently doing in our 
perl-based iMIP processing.

The syntax of the action is:

processimip [ :calendarid <string> | :updatesonly ] [ :deletecanceled ] 
[ :status <var-name> ]

The options are as follows:

  * :calendarid - Specifies the ID of the calendar in which to create
    new events from invites.  If not specified, the value of the
    DAV:schedule-default-calendar property is used.
  * :updatesonly - Specifies that only updates to existing events will
    be processed.  New events from invites MUST NOT be created.
  * :deletecanceled - Specifies that METHOD:CANCEL on an entire event
    will cause the event to be removed from the calendar rather than
    simply setting the STATUS:CANCELLED property.
  * :status - Specifies a variable name into which a string representing
    the result status of the action will be placed.  A string of "ok"
    signals that the event was created/updated/removed from the
    calendar.  Any other string signals failure.

Note that :calendarid and :updatesonly are mutually exclusive.

The :status string values probably need a more formal syntax.  Perhaps 
steal something from IMAP or POP3, like "OK ..." and "NO ..." or "+OK 
..." and "-ERR ...".  I'm open to suggestions.

-- 
Kenneth Murchison
Senior Software Developer
Fastmail US LLC

---

From: Ned Freed <[email-masked]>
Date: Wed, 20 Oct 2021 07:20:07 -0700 (PDT)

> > >     setting match variables).

RFC 5463 (Sieve ihave extension) defines a test that has the side effect of
changing the interpretation of the rest of the script.

If I'm reading this correctly, the side effects here are all external to the
script. If so, in terms of language semantics they seem rather trivial in
comparison to ihave.

> > > What do others think?  Any different/better design ideas?

As the author RFC 5463, I'm clearly partial to the tests with side effects
approach. We've used in many private Sieve extensions we've defined, e.g., a
redis test/action.

The resulting syntax really is much cleaner, especially in simple scripts.

The place where using a test with side effects doesn't work well is when you
have more than two possible outcomes, e.g., success/tempfail/permfail. In this
case your only real option is to set a variable, although depending on the
semantics you may be able to combine the use of a success/fail test
result with an optional variable elaborating on why things failed.

>   * Creation of new events from an invite aren't allowed (see

The question here is whether or not it makes sense for the script to take
different actions depending on what kind of failure occurred. If it does then a
variable is about the only way and it needs to be carefully structured so it's
easy for scripts to handle.

There's also the question of logging. Regardless of what capabilities are or
are not provided to the script author, I cannot emphasize strongly enough how
important it is to log the reason why the script failed somewhere rather than
leaving it up to the script to do it. Failure to have some kind of logging
mechanism makes tracking down transient problems with external agents - CalDAV,
Redis, databases, whatever - next to impossible.

> Note that :calendarid and :updatesonly are mutually exclusive.

Does processimap default to operating on the first calendar object it finds in
the message?

I assume processimap can be used inside of foreverypart loops, in which
case it only processes the current part as a calendar object. (Yes, there are
messages containing multiple calendar objects.)

> ..." and "-ERR ...".  I'm open to suggestions.

The syntax doesn't really matter as long as it is well defined. What may
matter is being able to distinguish betweeen different error types, in which
case you're stuck with having some kind of token in there chosen from a
list you specify in the RFC.

				Ned

---

From: Ken Murchison <[email-masked]>
Date: Fri, 22 Oct 2021 08:33:58 -0400

Hi Ned,

Thanks for the feedback.

> script. If so, in terms of language semantics they seem rather trivial in

Yes.

> result with an optional variable elaborating on why things failed.

In our use case, we only care about pass/fail.  The reason for the 
failure is not pertinent as to how the script continues processing.

> easy for scripts to handle.

How about a hybrid approach?  Make 'processimip' a pass/fail test, but 
accepting an optional ':errstr' argument (if variables are supported).  
The :errstr could be used by the script to add a header to the email 
describing the cause of iMIP processing the failure.

> Redis, databases, whatever - next to impossible.

Agreed.

> Does processimap default to operating on the first calendar object it 

My implementation currently only uses the first text/calendar part.

> messages containing multiple calendar objects.)

Multiple as in different encodings of the same iCal object or as in 
different iCal objects (different UIDs)?

I have definitely seen the former in the wild, but not the latter.