Subject: Re: draft-ietf-sieve-variables-07

From: Ned Freed <[email-masked]>
Date: Sun, 08 Jan 2006 08:37:11 -0800 (PST)

> >> For the former: the lemonade extension adds a reserved variable which
> >> I'm currently calling "IMAPSieve.cause".  Its value is the action
> >> that caused the script to be run ("APPEND", "COPY", and so on).  It
> >> occurs to me that it'll be awkward for scripts to have to wonder
> >> what the absence of that variable means.  Wouldn't it be a good idea
> >> to have a reserved namespace called "Sieve.", and to have a variable
> >> called "Sieve.cause" that's ALWAYS available if variables are
> >> supported?  In the normal case, its value would be "DELIVERY", and
> >> extensions like mine can specify other values.  (This might also add
> >> an IANA item....)
> >
> > okay, so you think this namespace should be defined in the variables
> > draft?  I'm not too wild with the idea, since I really wanted -08
> > which I submitted yesterday to finally reach RFC status.

> Oh, well.  So it can be the -09 version that does; is that such a big
> deal?

Frankly, yes. We need to finish up this spec and get it out the door. The list
of potential additions of this general sort is for all intents and purposes
infinite.

> > I would expect the
> > extension to be required by anyone wanting to inspect that variable,
> > and so I don't see much benefit to making it a globally available
> > variable.

> The problem is that when a script is written to ask "Why did I get
> called?", it will have to assume that if it doesn't know, it's because
> it got called by "mail delivery".  Since we're at a point where we
> could get that modified in the variables draft now, I think we should.

I'm dubious about the notion that there will be lots of scripts intended to be
run in a variety of different contexts that need to test this sort of thing.
However, I have no objection to making such a test available, although I think
the right way to do it is as a separate extension and test, not as  a magic
variable. For one thing, this means that in order for such a test to work you
have to have the variables extension, and I worry that variables may not be
implemented or enabled in some performance-sensitive environments.

> So I'm suggesting that the variables spec define a base namespace (I'm
> suggesting "Sieve.").  And I'm suggesting that the variables spec
> define a variable that MUST be set (I'm suggesting "Sieve.cause",

"place" seems to me to be a more natural name for this, but I'm not wedded
to any particular term either.

> but
> I'm not particular about the name) that gives the reason the script was
> called.  I'm suggesting that the value for mail delivery be "DELIVERY".
> And I'm suggesting that the variables spec say that other extensions
> may add other values, if they create other causes for having Sieve
> scripts called.  It seems a simple change, and a useful one in the long
> run.

I don't necessarily see running sieves in different environments as
necessitating additional extensions. It seems to me a simple "places" registry
would be sufficient. And it probably should allow for vendor-supplied places
since I suspect there will be non-standard uses of sieve. (In fact I know there
will be because we already have two in our product and another two are
planned.)

I also think additional environmental information might be useful to  provide
to script. Indeed, I suspect that knowing, say, the name of the system you're
running on might be even more useful to scripts than being able to distinguish
between delivery and other places sieves are used.

I would therefore suggest an extension specifying an "environment" test as a
better approach. I suspect we can come up with half a dozen things such a test
could operate on.

				Ned

---

From: Aaron Stone <[email-masked]>
Date: Sun, 08 Jan 2006 15:04:48 -0800

On Sun, 2006-01-08 at 08:37 -0800, Ned Freed wrote:
[snip]
> > So I'm suggesting that the variables spec define a base namespace
> > (I'm suggesting "Sieve.").  And I'm suggesting that the variables
> > spec define a variable that MUST be set ...

>From a language implementation point of view, if you're going to reserve
a keyword or a namespace, do it sooner rather than later.

[snip]
> better approach. I suspect we can come up with half a dozen things such a test

Maybe? Sieve is a "little language" and I like to think of Sieve scripts
as being these little gnomes that push my mail around. Gnomes don't
think too hard about where they are or where they're going; gnomes just
focus on what they're doing right now.

Aaron

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 09 Jan 2006 02:14:42 +0100

> focus on what they're doing right now.

sorry, but I don't understand what this means.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Sun, 08 Jan 2006 18:01:24 -0800 (PST)

> a keyword or a namespace, do it sooner rather than later.

Since namespaces are bound to future extensions by the current variables
draft, all namespaces are effectively reserved already.

> focus on what they're doing right now.

I'm sorry, but I have to disagree. Lots of people have more than one email
account and one of the goals of sieve is for a user to be able to define
something that works in multiple places. Wanting to have somewhat different
behavior depending on the environment is a fairly obvious thing to want to do,
even when all the different environments correspond to the same place in email
processing. As a specific example, i might want to have a very different
spamtest threshhold for the filter I use on my ISP account than the one I use
on my corporate account.

This isn't to say that there aren't significant barriers to script portability.
The most obvious one is the very real possibility that the set of supported
extensions is quite different in different environments. Since requires cannot
be conditionalized this forces scripts intended for multiple environments to be
written us9ing the least common denominator set of extensions, which is often
pretty unappealing.

This all gets especially nasty when you change places. The set of available
actions at delivery time is unavoidably different from the set of actions
available within an IMAP server. This is reflected in the sieve-in-IMAP draft
in various ways, including the omission of reject. So what should

	require "reject";

do in such a script?

				Ned

---

From: "Nigel Swinson" <[email-masked]>
Date: Mon, 9 Jan 2006 13:37:42 -0000

> better approach. I suspect we can come up with half a dozen things such a test

I like this a lot.  I was struggling to articulate why I didn't like what was being proposed, but this sounds very extensible.  I already have a CSieveRunTimeEnvironment class in my solution and was already wanting to make more information available to the script.

I also agree that this should be an additional extension, primarily so it doesn't hold up variables, and secondarily cos I don't think it "needs" to be part of the base variables spec.

Nigel

---

From: Ned Freed <[email-masked]>
Date: Tue, 10 Jan 2006 09:07:25 -0800 (PST)

> I like this a lot.  I was struggling to articulate why I didn't like what was
> being proposed, but this sounds very extensible.  I already have a
> CSieveRunTimeEnvironment class in my solution and was already wanting to make
> more information available to the script.

I'm curious as to what sorts of things you have in this class. The things I
immediately see as useful are the server name, the place in the process where
the script is executing, and information relating to whether or not this script
is executing standalone or has been called in some way from another script.

> I also agree that this should be an additional extension, primarily so it
> doesn't hold up variables, and secondarily cos I don't think it "needs" to be
> part of the base variables spec.

Complete agreement here.

				Ned

---

From: "Nigel Swinson" <[email-masked]>
Date: Tue, 10 Jan 2006 18:35:57 -0000

> is executing standalone or has been called in some way from another script.

It holds state/manages things like:
- Mail message
- Owner info
- Server info
- Variables
- Method to obtain spam score/virus scoret
- A cache of the spam score, virus score
- Methods for reporting errors

But my reason for the comment was the relating to "Information about the owner".  ie I'm thinking it might be useful to access mailowner/domain/mailserver configuration from within a sieve script.  Perhaps things like the server name, but also possibly mailbox properties like return addresses.

In the case of scripts that are executed against lots of mailboxes then I was also thinking of stuff like:

 Do initial basic sieve filtering
 if (User has paid me for feature X) {
   Do expensive sieve filtering
 }
 if (User has paid me for feature Y) {
   Do slightly less expensive sieve filtering
 }
 Do basic sieve filtering

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 11 Jan 2006 00:43:16 +0100

On Tue, 2006-01-10 at 09:07 -0800, Ned Freed wrote:
> [...]

I'm also in favour of an "environment" extension rather than adding this
to variables.

in addition to server name, some identification for the site would be
useful, at least if the provider chooses to expose internal architecture
to end users (which we do).  other operational limits might be useful,
too, e.g. max size for messages, remaining quota (dynamic, so maybe not
appropriate), stuff like that.
-- 
Kjetil T.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 10 Jan 2006 19:06:53 -0500

>> called?", it will have to assume that if it doesn't know, it's
>> because it got called by "mail delivery".
> 
> I'm dubious about the notion that there will be lots of scripts
> intended to be run in a variety of different contexts that need to
> test this sort of thing.

We don't (or at least I don't) know what the actual use cases are
likely to be, so I don't know whether I agree with you or not.  If we
provide a good way to manage multiple scripts, perhaps with an
extension to ManageSieve, it might turn out that people always want to
do different things at the different processing points, and they'll
always use separate scripts for each.  In that case, no, this isn't
necessary.

Or it might be that there are common things that people will want done,
and they'll just occasionally want to test this to do something
different now and then.  In that case, we need this.

Anyway, as the discussion's gone since the note I'm responding to,
there seems to be agreement that this and other identifying stuff would
be useful in an "Environment" extension, or some such.  Really, I
didn't think it needed to be in Variables -- just that Variables seemed
a logical place to put it.  I'm happy to put it in its own extension
that defines some new test, like this (using one of the examples from
the imap-sieve draft):

     require ["copy", "environment"];

     if anyof (environment :is "cause" "APPEND",
               environment :is "cause" "COPY")  {
         if environment :is "mailbox" "ActionItems" {
             redirect :copy "[email-masked]";
         }
     }

Note, here, that I've used two "environment" items, called "cause" and
"mailbox".

I agree that I like this better than doing it with variables, and it
means that if the "environment" extension is available, it can specify
the "DELIVERY" cause that I'm asking for.  That satisfies everything.
I'll write up an "Environment" extension draft, if y'all think I
should.  Ned?  Alexey/Cyrus?

I still wonder whether there ought to be a way to test for the presence
of an extension, rather than only to have the choice of *requiring* it
or not using it at all.  I can see a script wanting, for example, to
send a notification IF the Notify extension is available, but not
wanting the script to FAIL if it's not.

Barry

--
Barry Leiba, Pervasive Computing Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba
http://www.research.ibm.com/spam

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 11 Jan 2006 02:14:59 +0100

> wanting the script to FAIL if it's not.

it certainly seems useful to me.  we could introduce an action, like:

  require "environment";
  # or perhaps a separate extension.
  want "notify";
  if environment "extensions" "notify" {
     ....
  }

since the grammar of Sieve is regular, an unknown extension can't
introduce elements which changes syntax parsing.  furthermore, the
validity of the elements in the script can easily be verified during
upload: if you find an unknown keyword or element, you look upwards in
the blocks leading down to the statement.  if a block is conditional on
the existence of an extension, and the extension is unknown, ignore the
statement, otherwise flag the error and refuse the upload.  a
bytecompiler can omit the complete block in its representation.

an implementation which does no verification during upload is no worse
off with this mechanism than without.  random syntax or semantic errors
can appear anywhere and must be handled anyway.

an anyof test does throw a spanner in the works, though.  I think it is
appropriate to disallow this explictly (although a _bit_ ugly), and have
the validation during upload assume that an anyof which includes an
extension test is _always_ true, even when the extension is unsupported.
(this is still the code for the script verifier I'm talking about.)

actually, the "want" action may be superfluous, but I think it should be
used to keep dependencies explicit up front.

just some random thoughts,
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Wed, 11 Jan 2006 08:19:10 -0800 (PST)

> likely to be, so I don't know whether I agree with you or not.

It is precisely because the use cases are unknown that I'm dubious
about the extent to which the feature will be useful. Designing software
in the absence of compelling use cases often results in features of
limited utility.

OTOH, environment tests serve other purposes, and the cost of another keyword
in such a test is incredibly low.

> If we

> different now and then.  In that case, we need this.

My problem isn't that the various different scripts won't have tests in
common. They will. The problem is rather that the actions you'll want to
take will be quite different (this is almost axiomatic) and the ways the
actions will be coupled to the tests will probably vary considerably.

> Note, here, that I've used two "environment" items, called "cause" and

Mailbox is a good one I hadn't thought of before. it clearly needs to be made
available. The notion of being able to distinguish between different sorts of
insertion operations is also interesting.

> should.  Ned?  Alexey/Cyrus?

Absolutely! Why isn't it already an I-D? ;-)
 
> wanting the script to FAIL if it's not.

I'm worried about this as well, but I'm not happy about the implications
of making such a change either. As Harald would say, "mumble".

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 11 Jan 2006 21:22:53 +0000

Barry Leiba wrote:

>be useful in an "Environment" extension, or some such.
>
I agree.

>  Really, I
Sure. Submit it as an individual draft first. The WG is sufficiently 
behind on existing milestones, so I think it is premature to start 
discussing adding this work to the charter.

---

From: Michael Haardt <[email-masked]>
Date: Thu, 12 Jan 2006 17:25:45 +0100

>   if environment "extensions" "notify" {

I must be missing something, but why would you want to write scripts
that way? Which application would profit from this extension?

I can imagine that a way to find out which extensions are available
is very useful for GUIs generating Sieve code, or even humans, but
for scripts?

If I used a GUI and it offers notifications, then I expect they are
available.  

I don't think the base specification unconditionally allows dead
code blocks that contain unknown extensions, like:

  if false {
     unknown_extension;
  }

Does the environment extension focus on the above usage and is it OK if
it enforces implementations to allow unknown extensions in dead code?

Michael

---

From: Barry Leiba <[email-masked]>
Date: Thu, 12 Jan 2006 15:22:59 -0500

> is very useful for GUIs generating Sieve code, or even humans, but

Well, the point of Sieve is for the scripts to be portable.  Ideally,
I should be able to move a script from one environment to another,
and it would still work.

But now we have to consider the meaning of "work" [or "is" (it's a US
politics joke; never mind)].  Suppose that a script was designed to
to a whole bunch of stuff, using fileinto and redirect and whatnot,
and then at the end it just wants to say, "OK, if, after all this,
the message is going into a mailbox called "important", and the
notify extension is available, then notify the recipient by pager.

If I should move that script to a system that doesn't support
notify, I might still want it to "work", in that I'd want it to do
everything except the pager bit.  But if the script be coded thus:

   require "environment", "fileinto", "notify"
   ...
   [do all the useful work]
   ...
   notify [...]

then it will fail completely when it's run somewhere that doesn't
support notify.  If I have the option of coding it this way:

   require "environment", "fileinto"
   ...
   [do all the useful work]
   ...
   if environment "extensions" "notify" {
     notify [...]
   }

then I get what I want, and I don't have to change the script.  And if,
some say, my system adds support for notify, I'll start getting paged
without having to find out that it's been upgraded, and without having
to put the notify code back into the script.

> If I used a GUI and it offers notifications, then I expect they are

(1) You're ignoring the script portability issue.
(2) I would not make that assumption anyway.  There's no way for a GUI
to query the Sieve system, and in a mix-and-match world there's no
reason to assume we'll have bought the Sieve engine from the same
source as we got the script builder from.

> it enforces implementations to allow unknown extensions in dead code?

I'm not sure I understand what you mean by any of that.
I *think* you're asking whether the error is generated by the parser or
by the execution, and I believe the answer is "execution".  So if
"unknown_extension" is never executed, there should be no error.

Barry

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 12 Jan 2006 21:44:59 +0100

> > it enforces implementations to allow unknown extensions in dead code?

this is not the focus of the "environment" extension.  it very well
might better be in a separate extension.

> "unknown_extension" is never executed, there should be no error.

either is allowed.

   Compile-time errors are ones in syntax that are detectable if a
   syntax check is done.
   [...]
   Implementations MAY choose to do a full parse, then evaluate the
   script, then do all actions.  Implementations might even go so far as
   to ensure that execution is atomic (either all actions are executed
   or none are executed).

-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Thu, 12 Jan 2006 14:24:27 -0800 (PST)

> then I get what I want, and I don't have to change the script.

I like this way of doing it. The "want" business used in an earlier example
didn't feel right, but this does.

There is precedent for this sort of thing, BTW. PostScript has the ability to
check for support for various extensions and only use them if they're
supported. (Or alternately, provide a different way to do something.) This
functionality used to be used a fair amount, although I haven't seen much of it
recently.

> And if,

Yes, although this does have a negative aspect - an upgrade changes behavior
and actives old code someone forgot was even there. This issue would certainly
need to be discussed in any specification we write.

> source as we got the script builder from.

Well, there has been talk about having some way in managesieve to get the list
of supported extensions, but AFAIK it has never made it into an actual
specification. Of course this may be more a result of no work having been done
on managesieve recently than anything else.

> by the execution, and I believe the answer is "execution".

Er, no. The base specification leaves this open. Implementations are free to
perform as much checking as they want before executing a script. It is
perfectly permissible to check and make sure that the script doesn't reference
any tests or actions that aren't listed in the require clause. Our
implementation does this, as a matter of fact.

The way we'd probably change this is to see if environment is listed  in the
require clause, and if it is disable this sort of checking. There's already a
way to disable this sort of strict checking, so this won't be hard for us to
implement.

I note in passing that only new tests, actions and comparators are amenable.
to this sort of thing. Something like

   if environment "extensions" "variables" ...

makes my head hurt.

				Ned

---

From: Aaron Stone <[email-masked]>
Date: Thu, 12 Jan 2006 17:26:37 -0800

> on managesieve recently than anything else.

I thought there was a CAPABILITY command... there certainly is in my
implementation ;-)  Capabilities are presented in the greeting, too!

Aaron

---

From: Ned Freed <[email-masked]>
Date: Thu, 12 Jan 2006 18:09:17 -0800 (PST)

> implementation ;-)  Capabilities are presented in the greeting, too!

Forgot about that, sorry.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Fri, 13 Jan 2006 10:18:48 +0100

On Thu, Jan 12, 2006 at 03:22:59PM -0500, Barry Leiba wrote:
> everything except the pager bit.

Ok, so the focus of this new test, whatever we call it, is to enhance
script portability.  I use Sieve in a number of places, but all scripts
are so different that portability is the least I would think of.
Your applications are obviously very different.

> to put the notify code back into the script.

Not something I would like.  Out of a sudden previously dead code
gets active.  My phone number might have changed and now someone else
gets my notifications (my fault).  There may be a semantic mistake in
the previously dead code, something managesieve could not check before,
and now that it is active, it causes an error and my script terminates,
leaving a mess to be discovered when coming back from vacation.

To me, we are talking about two different issues, and "environment"
as we talk about it, combines them in an unfortunate way.  I prefer to
put different things in different extensions:

o  Getting to know which extensions are supported.  For your applications,
   a Sieve extension is useful.  For mine, something like managesieve,
   although I would never implement it, is more suitable.  Ok, I see no
   problem there.  "environment" may be good solution for Sieve and
   any sieve implementation certainly had no problem to support that.

o  Having a way of not letting unknown extensions cause script failure.
   But why restrict that to unknown extensions? How about exception
   handling, something like "try & catch"? That does not only catch
   semantic errors at the action level, but also at the argument/option
   level.  Your "if" test above only catches the first.

I don't think I need either, but a new extension that looks like a
test and requires the fundamental change from strict to somewhat lazy
error checking sounds very wrong to me.  Exceptions may not be the best
solution, but at least you expect them to change error handling, and
they do so universally.

Unfortunately, the case spec forbids:

  try { require "notify"; } catch { }

The "require" statement is only allowed at the beginning of scripts.
Too bad.

Michael