Subject: Notes on draft-homme-sieve-variables-04

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 22 Oct 2004 16:11:12 -0400

Hi,

These are a few comments on draft-homme-sieve-variables-04 .  In another
message, I made some comments about variables in general:  specifically
that I'd like to see a time where variables are integrated into the
SIEVE language in a more fundamental way.  However I imagine that even
assuming anyone else agreed with me, that goal would take some time to
reach, and that something like this draft would be more immediately
useful.  In making allowance for that farther goal, I'd want to see:

 - This draft and its capability called something other than "variables"
   (maybe "stringvars")

 - The things that this draft calls "variables" be called something
   else (like, "stringvars").

Other than that, here's some specific stuff about the draft.

>    When a string is evaluated, substrings matching variable-ref SHALL be
>    replaced by the value of variable-name.  Only one pass through the
>    string SHALL be done.  Variable names are case insensitive, so "foo"
>    and "FOO" refer to the same variable.  Unknown variables are replaced
>    by the empty string.

I'd strongly prefer case sensitivity here.

> 
> 
>       variable-ref        =  "${" variable-name "}"
>       variable-name       =  num-variable / *namespace identifier
>       namespace           =  identifier "."

You might want to mention that "identifier" is as defined in rfc3028
(even though this is an extension- it doesn't hurt to be explicit).

There's not really a lot about namespaces in this draft, other than to
allow for future state variables associated with extensions (or so it
appears).  Maybe point out that this document specifies namespace syntax
only, without addressing anything else about namespaces.  (Also I'd
prefer something other than "." as a namespace suffix.)  The implication
seems to be that namespace-associated variables are read-only; if that's
true, might want to make that explicit.

>       num-variable        =  1*DIGIT

Why limit numbered variables to a single digit?  Some languages have had
this restriction to help them with parsing, but that's not the case
here; it's not that much of a trick to distinquish an all-digits
variable name from one that's not all-digits.

Not that I really see a lot of call for more than 9 match references,
but I hate that sort of limit- an implementation ought to be able to
provide more than 9 if it wants to, and limits should be about
resources rather than syntax.

>    The expanded string MUST use the variable values which are current
>    when control reaches the statement the string is part of.

At least one problem with this has already been brought up on the list:
the interaction between match results and sequential evaluation of
multiple tests.  It's much more expressive to be able to take advantage
of the side effects (the match results) within one test statement;
furthermore it's burdensome on an implementation (especially in terms of
efficiency) to have to freeze the current match results upon entry to a
test statement so that those frozen results will be available by each
step inside that test statement.  I suspect that the goal of this
prescription is to address deferred actions such as "fileinto" -- and
that's a good thing.  However, it does introduce these other real
problems.

>    Tests or actions in future extensions may need to access the unex-
>    panded version of the string argument and, e.g., do the expansion
>    after setting variables in its namespace.  The design of the imple-
>    mentation should allow this.

I suppose this is as good a place as any for this comment:  I'm not all
that comfortable with all strings being automagically eligible for
interpolation (once this extension's capability is enabled).  It seems
to me that it would be better to have a syntax that specifically
commands that the string be processed. e.g., with some character before
the opening quote:

    fileinto ?"${1}";

or via other different quoting style.  A bonus would be a syntax that
allows one or more rescannings of the resulting string.

>    For ":matches", the list will contain one string for each wildcard
>    ("?" and "*") in the match pattern.  Each string holds what the cor-
>    responding wildcard expands to, possibly the empty string.  The wild-
>    cards expand greedily.

I've lost track of the history: what's the reason for "*" expanding
greedily?  Is it to be compatible with what regex does?  My preference
(both in terms of what I'd expect and in terms of what is efficient to
implement) is the opposite.  For example, let's say you have:

    Subject: [filters] how to search for ']' 

I would want this to work:

    if header :matches "subject" "[*]*" {
        set "prefix" "${1}";
	set "actual_subject" "${2}";
    }

>    Numbered variables ${1} through ${9} MUST be supported.  References
>    to higher indices than the implementation supports should be treated
>    as a syntax error which MUST be discovered at compile-time.

I don't like the mandate that strings have to be inspected at
compile time-- a "MAY" would be preferable.

>    The introduction of variables makes advanced decision making easier
>    to write, but since no looping construct is provided, all Sieve
>    scripts will terminate orderly.

"orderly" is not an adverb :-)

Yours,
mm

---

From: Lyndon Nerenberg <[email-masked]>
Date: Fri, 22 Oct 2004 13:51:36 -0700

--On 2004-10-22 4:11 PM -0400 "Mark E. Mallett" <[email-masked]> wrote:

>>       num-variable        =  1*DIGIT
>
> Why limit numbered variables to a single digit?

They aren't. That construct means a <num-variable> consists of at least 
one digit, with no upper bound on the number of DIGITs it can contain. 
If it was limited to a single digit the ABNF construct would be 1DIGIT.
See RFC 2234 sections 3.6 and 3.7.

--lyndon

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 22 Oct 2004 17:00:15 -0400

> See RFC 2234 sections 3.6 and 3.7.

Right, of course- my mistake.  

mm

---

From: [email-masked]
Date: Fri, 22 Oct 2004 13:46:02 -0700 (PDT)

>  - This draft and its capability called something other than "variables"

I could not care less what the capability naame is.

>    else (like, "stringvars").

This strikes me as more confusing than useful. It something else comes along it
cann call whatever it defines builtin variables or something similar.

> I'd strongly prefer case sensitivity here.

I, OTOH, strongly prefer case insensitivity.

> (even though this is an extension- it doesn't hurt to be explicit).

Any time a document inherits ABNF from another document it needs to say so.
So yes, this issue definitely needs to be addressed.

> true, might want to make that explicit.

Pointing out that namespaces are basically a placeholder is fine and a good
idea. I like dot as a separator; there are other characters I could
live with but also a lot I would object to. I don't think stating that
namespace variables are read-only is a good idea at this point.

> Why limit numbered variables to a single digit?

This doesn't impose such a limit. "1*" means "one or more", not "one".
You'd write "1DIGIT" for "one".

> Some languages have had

Right, although perhaps banning leading zeroes in the ABNF would be a good idea.
Something like:

    num-variable = 1DIGIT / (%x31-39 1*DIGIT)

> resources rather than syntax.

Again, there's no such limit, although implementations are
allowed to impose a limit if they wish. I wonder if 9 is an
appropriate minimum maximum, however.

> that's a good thing.  However, it does introduce these other real

I agree, I can argue this one either way, but I actually prefer the
"numbered variable changes take effect immediately" model.

> allows one or more rescannings of the resulting string.

I think having this as a global setting is sufficient. You either 
require variables and get the effect or you don't and the effect is absent.

I also strongly object to making a syntax change to the language in order
to get this functionality. If we're gonna do that we might as well
make variables first class objects and add a concatenation operator.

> implement) is the opposite.  For example, let's say you have:

I'm afraid I don't buy your efficiency claims here. I can write efficient
greedy globbing code and I can write inefficient non-greedy globbing code.

I think the main reason for greedy globbing is exactly what you say: It
is what people have come to expect. I could live with non-greedy
globbing, although I'd have to change things a bit to support it.

> compile time-- a "MAY" would be preferable.

I missed this. I agree 100% with your assessment.

> "orderly" is not an adverb :-)

Indeed.

				Ned

---

From: "Mark E. Mallett" <[email-masked]>
Date: Fri, 22 Oct 2004 18:26:39 -0400

> require variables and get the effect or you don't and the effect is absent.

That was the thing that I was attempting to find a cure for.. the
necessity of interpolating every single string once the capability has
been enabled.  "require variables" isn't really a global setting,
in that you can't unset it.

I would imagine that in most scripts, very few strings will have
variable references.  It simply strikes me as wasteful to process all of
them for interpolation just for the few that need it; plus it's just my
inclination to want such processing to be explicit rather than implicit.
(with the emphasis on the latter, really- as the extra processing
probably isn't major.)  None of the alternatives that I could think of
were all that attractive to me though, including the one of
interpolating all strings all the time.  But I figured I'd throw it
out there.

> greedy globbing code and I can write inefficient non-greedy globbing code.

What I was thinking of here was that some implementations (including the
one I am most familiar with) have a local hack or extension to allow
tests on the message body.  And in time, when there are extensions to
support body and MIME part tests, existing match semantics will likely
be used.  Doing a search in a message body, some of which may be cached
on disk, would be better done non-greedily.  The efficiency of matches
on header strings is probably not all that big of a factor.

As for the utility of greedy ":matches" in scripts, I tend to think
non-greedy is more useful, but would welcome being educated by
examples of why that's not the case.

mm

---

From: [email-masked]
Date: Fri, 22 Oct 2004 16:22:00 -0700 (PDT)

> interpolating all strings all the time.  But I figured I'd throw it

Having implemented this, I can assure you that the waste involved is
insignificant compared to other costs in implementing sieve.

> on header strings is probably not all that big of a factor.

Interesting point. I hadn't considered body tests.

> examples of why that's not the case.

Well, greedy matches get some of the nested parens cases "more right"
than non-greedy. But again, I'm a bit ambivalent about whether or not
greedy is the right way to go.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sat, 23 Oct 2004 17:32:43 +0200

>  - This draft and its capability called something other than "variables"

if the capability is integrated, why does the name of an obsolete
extension matter?

>    else (like, "stringvars").

I can try to add a definition of "variable" to make it clear.  The
introduction now reads in my personal copy:

        Conventions for notations are as in [SIEVE] section 1.1,
        including use of [KEYWORDS] and [ABNF].  The grammar builds on
        the grammar of [SIEVE].  In this document, "character" means a
        [UNICODE] character, which may consist of multiple octets coded
        in [UTF-8], and "variable" is a named reference to data stored
        or read back using the mechanisms of this extension.

> I'd strongly prefer case sensitivity here.

almost all Internet standards are case insensitive, and I don't think it
is important enough to go against that flow.

> (even though this is an extension- it doesn't hurt to be explicit).

the excerpt above addresses this.

> only, without addressing anything else about namespaces.

- Future extensions may make internal state available through variables.
+ Namespaces are for future extensions which make internal state
available through variables.

> The implication

I don't think we want to restrict them to be read-only, but there needs
to be a new action if they are to change, since SET disallows setting
variables in namespaces.  one use could be a "scope" extension, so the
user says something like:

        LET "scope.local.myvar" "true"
 
> that's a good thing.  However, it does introduce these other real

the main issue is whether the results are defined or not.  there are no
other natural sequence points than the complete statement.  if this is a
to change, significant amounts of thought has to be put into defining
execution order.  (I'd be glad to be proved wrong, of course.)

> commands that the string be processed.

we have that, if the string contains "${", it should be processed.  a
bytecode compiler can do this trivially, and store the two types of
strings differently.

> e.g., with some character before the opening quote:
> 
>     fileinto ?"${1}";
> 

I'm not sure that is a bonus :-)

> greedily?  Is it to be compatible with what regex does?

yes, I think it would be strange if :matches was non-greedy,
while :regex was greedy.  you may be right that the improved usability
is worth it, though.

incidentally, if we had a "reverse" operator, users could implement
non-greedy themselves:

        set :reverse "pattern" "[*] *";
        if header :matches "Subject" "*" {
           set :reverse "subject" "${1}";
        }
        if string :matches "${subject}" "${pattern}" {
           set :reverse "prefix" "${2}";
           set :reverse "actual_subject" "${1}";
        }

> compile time-- a "MAY" would be preferable.

I don't like the possibility of an error spuriously occuring during
delivery.

> "orderly" is not an adverb :-)

strangely so.

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 25 Oct 2004 16:21:01 +0200

> I don't like the possibility of an error spuriously occuring during

re-reading 2.10.6 of RFC 3028, MUST may seem inappropriate, but on the
other hand, that section allows implementation to not have a
"compile-time" at all.

-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 26 Oct 2004 17:09:48 -0400

Hi,

> if the capability is integrated, why does the name of an obsolete

Because terminology and labels are sticky.  If at some point we have two
different things called "variables" there's ambiguity.  Native variables
would have a higher claim to the term; my thought was to reserve that
term for their eventual introduction.  As you can see, though, I don't
really have a better alternative; it's natural to think of these things
as "variables" and I don't have an idea for an equally natural term for
non-native variables.  But the fact that I don't have an answer doesn't
mean I don't think it's an issue :-)  Anyway it was something to point
out; apparently there's not much (any?) agreement with me.

> is important enough to go against that flow.

I guess I am on the short side of that opinion too :-)

Actually my preference for case sensitivity applies mainly to any future
implemention of native variables, since the trend in languages is for
case sensitivity in variables, and that is what many programmers will be
most comfortable with.  I'm mainly concerned with any precedent set
here.  I realize that RFC3028 specifies that statements (that is, all
tokens) are case-insensitive; I still think that script writers would
want case sensitive native variables.

> variables in namespaces.

Right, I was just trying to figure out what the namespaces were getting
at.  As you point out, the description of "SET" disallows specifying
a namespace, so my conclusion was that namespace-associated variables
are read-only.

> while :regex was greedy.  you may be right that the improved usability

Yes- it's a problem either way.

>            set :reverse "actual_subject" "${1}";

I'm all in favor of giving script writers options, such as a :non-greedy
option.  I don't really get what your ":reverse" does for "set" though.

mm

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 26 Oct 2004 23:23:07 +0200

On Tue, 2004-10-26 at 17:09 -0400, Mark E. Mallett wrote:
> >         set :reverse "pattern" "[*] *"; [...]
> 

it'd reverse the argument before storing it in the variable.  e.g.,
'set :reverse "var" "kram"' would set ${var} to "mark".

(the example was included for fun, to illustrate how arbitrary our
choice is for an implementation, it's a _lousy_ use case for :reverse,
in truth I don't see any utility of such a modifier :-)
-- 
Kjetil T.