Subject: [sieve] Notify (RFC 5435) questions

From: Hannah Schroeter <[email-masked]>
Date: Mon, 28 Sep 2009 16:36:51 +0200

Hello!

Section 3.4. Notify Tag ":importance" says that the :importance tag
specifies a numerical value from one through three as a string. However,
it doesn't specify what happens when a script fails to comply with that,
i.e. the string specifies a different number, or not a number at all
(the latter could have been avoided if one had specified the parameter
of :importance as a number to begin with).

This doesn't fit with the ductus of other parts of the spec (such as
:from, or the parameter method) where error handling or ignoring things
(e.g. from values that are syntactically correct but not allowed for the
script owner for policy reasons) are clearly specified.

And it's not "justified" by being delegated to method specific handling,
as RFC 5435 itself *does* specify it to be the string representation of
a number from one through three.

By the way, the ABNF suggests that the number has to be given literally,
but the specification doesn't spell it out whether that really implies
the :importance parameter has to be a *constant* string in the sense of
the variables extension.

So
  1. Is the parameter of :importance a *constant* string (as the ABNF
     suggests) or can it expand variables (as the variables extension
     says all strings do if one uses it unless *explicitly* told to be
     constant)?
  2. If the string of the parameter doesn't specify a number at all,
     is that an error or is the parameter silently ignored and the
     default importance ("2") used?
  3. If the string of the parameter *does* specify a number, but one
     out of range, is that an error or is it ignored? In the latter
     case, is the default importance used or is the number clipped
     to the range (from below 1 to 1, from above 3 to 3)?
  4. Are leading zeroes tolerated as the *text* suggests "a numeric
     value represented as a string" or not, as the ABNF suggests,
     [":importance" <"1" / "2" / "3">]?

Kind regards,

Hannah.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 28 Sep 2009 13:32:02 -0400

> i.e. the string specifies a different number, or not a number at all

What happens is the same as what happens with any other syntax error:
it causes a compile-time error (or, if variable substitution is used,
a run-time error after the variable substitution).

> script owner for policy reasons) are clearly specified.

I don't know what "ductus" means, but this isn't a case of policy.
There's no need to say specifically what happens if a script violates
the grammar in this case, because we know what to do when a script
violates the grammar in general.

> the :importance parameter has to be a *constant* string in the sense of

We thought it was clear that these were strings, which could be put in
as variables, but I see how it can be confusing.  The intent is
certainly that variables may be used here.  Perhaps we should enter an
erratum suggesting a comment in the ABNF that makes this clear.

A bit of history:
The parameter was originally called "priority", and had text-string
values.  From the -00 draft version:

    The priority parameter specifies the importance of the notification.
    The priority parameter has the following values: ":high" (very
    important), ":normal", and ":low" (not very important). If no
    priority is given, a default priority of ":normal" SHOULD be
    assumed.

After discussion about similar parameters in other protocols (XMPP and
SIP were mentioned), we decided to change it to "importance" in the
-04 version, but the value had already changed to the number-strings
in the -01 version.  One could search the mailing list archives to
find the discussion and see the reasoning, but searching the mailing
list archives is unnecessarily difficult.

I often wish we would leave some of the historical information in the
documents, so that people who come along later can see how we arrived
where we are, and perhaps understand it better (and so we, ourselves,
can go back and easily look).

>     says all strings do if one uses it unless *explicitly* told to be

The latter.

>     default importance ("2") used?

Error, as with any other incorrect parameter that doesn't give other
explicit guidance.

>     to the range (from below 1 to 1, from above 3 to 3)?

The same as 2.

>     [":importance" <"1" / "2" / "3">]?

We didn't explicitly consider this, but the grammar specifies a
string, and does not allow leading zeroes.  I would anticipate that
implementations might take that as specified.

Barry

---

From: [email-masked]
Date: Mon, 28 Sep 2009 10:15:19 -0700 (PDT)

> of :importance as a number to begin with).

Which generally should be taken to mean it is up to the implementation to
decide how to handle such a case. IMO throwing an error would be fine, but so
would silently ignoring the parameter.

> script owner for policy reasons) are clearly specified.

I think you mean "dictus", not "ductus", here.

In any case, while I generally beleive tightening up specifications is a Good
Thing, this reading goes MUCH too far. IETF specifications are in general
descriptive, not proscriptive, in nature. it is therefore unreasonable to
expect that every specification is going to cover exactly what to do
with every possible input.

Indeed, I can think of many other cases where the handling of bogus inputs
aren't specified. For example, the relational specification doesn't say what to
do when an unknown string is given to :value or :count. The vacation
specification doesn't say what to do if a :mime reason argument is not found to
be valid MIME. And so on.

> a number from one through three.

It isn't a question of justification, it's a question of reaonsableneess and
specification readability. Tying down every possible loose end in every
possible case sounds like a great idea until you  see the text that results,
which often ends up so laden with incredibly obscure exceptional cases
implementors end up missing the important stuff.

There are plenty of formal language specifications that have been written with
exhaustive or near-exhaustive proscriptive text. It has not proved to be recipe
for successful interoperation.

> the :importance parameter has to be a *constant* string in the sense of

I don't think this is unclear at all. Variables operates at a level between
syntactic processing and semantic interpretation. As such, unless variable
substutions are specifically prohibited by a given parameter, they are allowed.

Having to state whether or not variable substition can be done in every case
would again be a recipe for vast specifciation bloat with useless garbage text.

The latter.

>      default importance ("2") used?

That's currently up to the implementation to decide.

>      to the range (from below 1 to 1, from above 3 to 3)?

This is the same as 2.

>      [":importance" <"1" / "2" / "3">]?

I don't think the text suggests that, which reduces this to 2 again.

Again, if pepple think it is worth tightening up the text regarding the
:importance parameter, or other specification locations, I have no problem with
that. But this should not translate into searching around and adding such text
everytwhere. Our specifications will be the worse for it if twe were to do
that.

				Ned

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 28 Sep 2009 15:11:57 -0400

> I don't think the text suggests that, which reduces this to 2 again.

In other words, the behavior is undefined, and the implementation can do 
whatever it wants.  This sort of thing is fairly common in protocol and 
langauge specifications, including some which are considerably more formal 
and exhaustive than typical IETF specifications.  See, for example, ANSI C.

-- Jeff

---

From: [email-masked]
Date: Mon, 28 Sep 2009 14:21:56 -0700 (PDT)

> a run-time error after the variable substitution).

If you want to overload the term "syntax error" this way, fine, but it is then
important to distinguish between the core Sieve syntax as specified in the pase
specification and the layered syntax requirements imposed by specific tests and
actions.

It is quite clearly an error, detectable at compile time, for a sieve to have
invalid core syntax. So this case is easy.

But what about layered syntax requirements? THings get very tricky very
quickly. For one thing,  if there's a sentence in the specifications that say
"anything not conforming to any layered syntax requirements MUST, in the
absence of more direct guidance, be treated as an error" I for one don't know
where it is. The absense of such a statement is why I believe this to be
implementation-specific at the present time.

And adding such a requirement - assuming it isn't already there - is trickier
than you might think. One example where it is decidedly nontrivial is the
reason parameter in vacation when :mime is used. Perhaps you think that
requiring that sieve engines check the MIME syntax in this case is a good idea;
I most emphatically do not and I will state right now that I would have zero
problem violating any sort of requirement that says such a check should be done
on the grouns of total asininity.

> violates the grammar in general.

Again, there are two levels of grammar here, and violations of them don't
necessary have the same implications.

> erratum suggesting a comment in the ABNF that makes this clear.

I think that's a really bad idea because it puts us on the slippery slope of
specifying all the places where variables can be used. Unless we then go
through every specification and every parameter and add similar text will lead
people to the erroneous conclusion that variables can only be used in places
where the specifications say they can.

If this is in any way unclear - and I don't think it is - the place to explain
it better is in the variables specification. In fact as I recall the notion of
explicitly calling out all the placesa where variables can be used was
consideered during the development of that specifications and summarily
rejected.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Tue, 29 Sep 2009 00:55:41 -0400

> The absense of such a statement is why I believe this to be

Oh, I'm happy to call it implementation-specific.  I think what's
being described is a clear case of syntax violation, and I don't see
that it's a hard case or that syntax in an extension should be
different from syntax in the base spec in this regard.  And I don't
look at it as being the same as doing MIME validation.  But no matter;
it's not and important enough point to get stuck on.

In any case, I'm not suggesting changing the protocol definition.
Hannah asked for implementation advice, so I gave it.  You can say
that that advice is specific to my implementation.  :-)

> consideered during the development of that specifications and summarily

I agree that it'd be better to put it into the variables spec.  You
know I'm not suggesting going through everything to say where you can
and can't do variable substitution; you should know me better than
that.  But Hannah has a point: the variables spec says this:

   This extension changes the semantics of quoted-string, multi-line-
   literal and multi-line-dotstuff found in [SIEVE] to enable the
   inclusion of the value of variables.

...and the grammar for "importance" uses none of those three syntax
constructions.  Taken at its word, that would imply that variable
substitution doesn't apply here.  We might change RFC 5229 to add that
variables can substitute for literal strings as well.  Or we might
decide that it's clear enough, and doesn't need changing.

Barry

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 29 Sep 2009 15:17:37 +0400

> On Mon, Sep 28, 2009 at 10:36 AM, Hannah Schroeter <[email-masked]> wrote:
 [...]
>>     says all strings do if one uses it unless *explicitly* told to be

+1

> Error, as with any other incorrect parameter that doesn't give other

+1

>>     to the range (from below 1 to 1, from above 3 to 3)?

Actually I think the value should be ignored or rounded down (e.g. "5"
would become "3").
I intended for this range to be extensible.

> implementations might take that as specified.

Agreed.

---

From: Hannah Schroeter <[email-masked]>
Date: Tue, 29 Sep 2009 15:35:51 +0200

Hello!

>and exhaustive than typical IETF specifications.  See, for example, ANSI C.

Frankly, I consider C a bad example. C uses undefined and unspecified
behaviour in order to achieve performant execution for a relatively
low-level programming language. Sieve is a relatively high-level
language. Java would perhaps be a slightly more apt comparison, and I
think there's much less "undefined" or "unspecified" behaviour (even
though I haven't read the whole language specification yet).

Back to Sieve, I *do* see the need to leave room for implementors in
some areas. Sieve is used in different kinds of mail delivery (or even
other) scenarios, so interaction with the environment can vary. I *do*
see it as reasonable that e.g. the validity of mailbox names for
fileinto (if supported at all) may vary. I *do* see reason to leave room
for when to check the validity of strings that interpolate variables,
e.g. not forcing constant propagation onto implementors, i.e. allowing
to check every string value (e.g. of comparator names) at runtime.

But as an implementor who has compiler construction background, I see
the boundaries different. Specifications should, in my eyes, be
well-defined. If they intend "unspecified" or "undefined" semantics,
they should explicitly say so, so users *see* they may cause demons to
fly out of their nostrils. If they intend to allow only a limited
freedom to implementors, they should say so. A relatively good example
in Sieve specifications is the way numbers are defined, they *have*
to be unsigned, they *have* to have at least a specified range (so
users may rely on that), they *may* have a larger range. I say
*relatively* good, as users can't test for the exact range the
implementation permits, so allowing a bigger range isn't of so much use
anyway.

HTML is another bad example. Because implementations varied too widely,
*and because they were too tolerant*, one has created a mess and in fact
*hampered* interoperability. Stricter specifications (e.g. fewer or
even *no* optional tags or optional close tags, which would've been even
possible with SGML) and the early reference implementation(s) being
actually strict according to the specifications would perhaps have been
good for interoperability.

If Sieve leaves too much open for implementors to decide, and at the
same time, not even the implementations are registered and *announced*
(e.g. using a capability), it hinders interoperability. One might need
to code a script this way for site A and that way for site B even if
both support the set of extension the script needs.

Kind regards,

Hannah.

---

From: Hannah Schroeter <[email-masked]>
Date: Tue, 29 Sep 2009 16:09:49 +0200

Hello!

>I think you mean "dictus", not "ductus", here.

Was a false friend from my native language. I meant something like the
style of other parts of the specification which were clearer about
errors - which errors have to be handled strictly and which have to be
handled in a tolerant manner (e.g. :from syntax errors vs. valid syntax
but not viable values for policy reasons).

>with every possible input.

So you mean to imply IETF specs are meant with users, not implementors
as audience? At least I read "descriptive" as "describe to users what's
actually implemented out there" and "proscriptive" as "tell implementors
what they ought to do".

>do when an unknown string is given to :value or :count.

Other wrongs don't make this wrong better. (I'd have liked those match
types to be keywords in the whole, like :value_eq ... or two keywords in
order, like :value :eq, btw.)

>The vacation

Other specifications *do* tell, like the email address syntax for
redirect.

I'd like to look at the good examples more than the bad examples.

>It isn't a question of justification,

Therefore the quotes.

>it's a question of reaonsableneess and

It would be just one sentence. Either "if the string is not exactly
equal to one of the three possible value, it is an error" (perhaps insert
reference to section 2.10.6. of the base spec) or "if ..., the
:importance parameter is ignored and the default is used".

>for successful interoperation.

And there have been quite loose specifications that have not proved to
bee good for interoperation, like HTML.

>substutions are specifically prohibited by a given parameter, they are allowed.

That's what the variables extension says, yes.

>would again be a recipe for vast specifciation bloat with useless garbage text.

In principle you're right. And usually I read the specifications so. But
here, the ABNF-like notation suggests otherwise. Btw, your above example
was good, as in the relational extension, the ABNF-like notation for the
match types also suggests otherwise. There are other places where the
specifications have such not so explicit contradictions and could need
some clarification in my eyes, like the use of variable names in the
imap4flags extension in the hasflag test and in the setflag, addflag,
and removeflag actions. Especially the latter actions look like they are
parallels to "set", just with a bit different evaluation of the *value*
that is stored into the variable. In "set", the variable name (target of
the store) is *constant*. In *that* place, I would think a clarification
would be appropriate.

>everytwhere. Our specifications will be the worse for it if twe were to do

In many other places, the fact that e.g. variables are expanded by
default is not so obscured by suggestive ABNF-like notation.

Btw, I just see that the ABNF-like notation is
           [":importance" <"1" / "2" / "3">]

In ABNF, the quotes are *metasyntax*, so the spec is contradictory here,
anyway. From ABNF, that would imply a keyword and a *number*, not a
keyword and a string! Of course, the text clears it up well enough
`The ":importance" tag is followed by a numeric value represented as
a string'

So *if* the specification is amended, it *could* be changed into
          [":importance" <value:string>]
and value could be explained in the text below. The "string" would be
clear enough to imply variable expansion (just as in :from, etc.) and
the text could be clarified to say what to do if the value of the string
is not "1", "2", or "3". As said, one sentence could probably be enough.

Kind regards,

Hannah.

---

From: Hannah Schroeter <[email-masked]>
Date: Tue, 29 Sep 2009 16:17:16 +0200

Hello!

>a run-time error after the variable substitution).
   ^^^^^^^^^^^^^^           ^^^^^^^^^^^^^^^^^^^^^

>violates the grammar in general.

Grammar is usually not a run-time property in computer languages,
and that's why I think it *does* need specification.

>    priority is given, a default priority of ":normal" SHOULD be

Were those *keywords* in that version or *strings* with a value
containing a colon as first character?

>The latter.

Then, the ABNF-like spec is misleading, as it might suggest a deviation
from that "default" of expanding.

>Error, as with any other incorrect parameter that doesn't give other

This isn't a clear line in Sieve with respect to *run-time* values.

>implementations might take that as specified.

That means error, if I understand you right?

Kind regards,

Hannah.

---

From: Hannah Schroeter <[email-masked]>
Date: Tue, 29 Sep 2009 16:23:54 +0200

Hello!

>I intended for this range to be extensible.

I'm not sure whether that would be good. A script could come to rely
that 5 "equals" 3 and then later the range *does* get extended and the
script auther might be surprised. A stricter implementation would force
the script author to use 3 and the value would *keep* being 3 after an
upgrade of the Sieve implementation.

Kind regards,

Hannah.