Subject: [sieve] Issues with specifications

From: Hannah Schroeter <[email-masked]>
Date: Wed, 9 Sep 2009 18:01:20 +0200

Hello!

Is it okay to discuss issues (what seems like ambiguities or gaps to
me) in Sieve specifications (RFC texts, perhaps also drafts) here,
from an implementor's perspective?

And, what's the relationship between this list (sieve@) and
ietf-mx-filters?

Kind regards,

Hannah.

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 09 Sep 2009 12:04:01 -0400

Hi Hannah,

> from an implementor's perspective?

Yes, you are more than welcome to do that.

> And, what's the relationship between this list (sieve@) and

[email-masked] is the new home for the mailing list. The old @imc.org 
address wil forward to the new list.

-- 
Cyrus Daboo

---

From: Hannah Schroeter <[email-masked]>
Date: Wed, 9 Sep 2009 18:57:19 +0200

Hi, Cyrus!

>Yes, you are more than welcome to do that.

Okay, thanks for your prompt reply.

>address wil forward to the new list.

Ok, thanks.

I've currently got the task to add more extensions to our implementation.

I'm stalled with the Date/Index Extensions a bit because of a few things
that are unclear to me.

- Index extension: It seems it is not specified what happens if the
  index is out of range, that is it's greater than the number of headers
  actually found for the header names given to the test.

  if header :index 5 "Received" "..."

  if there are only 4 Received headers, for example.

  For this, I've gone on and made the test fail (not match) - even for
  :contains with an empty key -, but the RFC doesn't explicitly say
  this (in contrast to for example yielding a run-time error).

- The index extension doesn't specify its behaviour with respect to
  :count tests. (The date extension does.)
  For header, is it meant to yield 1 if the index is in range, 0
  otherwise? And for address, the number of addresses in the one
  selected header line (if the index is in range)?

- Date extension: 
  * Section 4.2 enumerates the possible values for the date-part argument.
    However, it does not specify any error handling for the case if the
    script uses an invalid value there; especially if that value is
    generated by use of the variables extension, thus possibly making
    static checking impossible.
      Should invalid date-part arguments be an error (static, if possible,
    run-time else)?
      A precedent would be the "envelope-part" argument for the envelope
    test where implementations SHOULD consider unknown envelope parts
    an error (but not "MUST"). For comparators (that are given as string),
    it's even stronger: unknown comparators are an error (because they
    are either declared with require and then require causes an error if
    it's not known, or they aren't, then it's an error to use an
    un-require'ed comparator; section 2.7.3 of the base spec, towards
    the end).

  * Section 4.3 tells about comparator vs. date-part interactions.
    Do I read it right and that section is just recommendations for
    users? I.e. the implementation shouldn't reject a script that
    uses i;ascii-numeric in a non-recommended way (e.g. for "date",
    where it will in fact just use the year), but at most warn the user
    that the comparator may lose information (because it strips
    the value at the dash separating year from month)?

  * Date extraction: The specification (in section 4, before 4.1) says
    the implementation must be able to extract a date from the entire
    field content or from the end of a field, following a semicolon.
    As the obsolete syntax is still not weeded out, even in RFC 5322,
    this extraction may prove difficult.

    Received: [... received-info ...]; Mon, 10 (This is a ridiculous
      comment containing a semicolon: see here: ; ) Aug 2009 ...

    Undoing the folding is probably easy, but finding the *relevant*
    semicolon is difficult if I want to implement/use only a *Date*
    parser and spare myself the work of parsing the structure of the
    Received header too. The Date parser will be able to remove/skip
    the obsolete comment, but not the leading received-info. So we'd
    have to do a first pass in *reverse* to skip over CFWS until we hit
    a semicolon (outside the CFWS we skipped).
      Does the spec (RFC 5260) require that or would a simple-minded
    implementation (strip everything up to the last semicolon,
    regardless of structure, which would fail in my contrived example,
    i.e. make the test return false and use a count of zero) fulfill the
    spec too?

Kind regards,

Hannah.

---

From: [email-masked]
Date: Wed, 09 Sep 2009 10:47:11 -0700 (PDT)

>   if there are only 4 Received headers, for example.

The test is supposed to fail in this case. I agree the RFC could be clearer
about this.

>   this (in contrast to for example yielding a run-time error).

That's the correct behavior.

>   selected header line (if the index is in range)?

The meaning of :count is tied to the test being performed. :index only
restricts the applicability of the test to a specific header field occurannce;
it does not change the interpretation of count in any way. So, when :index is
used with header, :count produces a 1 if the specific header with that index
exists, 0 otherwise. Address is more useful; it would return a count of all the
addresses in the header. Date would again be a 0 or 1 depending on the
existance of the header and whether or not it contains a valid date.

I don't see a need for further clarification here.

>       Should invalid date-part arguments be an error (static, if possible,

Yes, I think so. Our implementation certainly does.

>     un-require'ed comparator; section 2.7.3 of the base spec, towards

I'm not sure why it's a SHOULD as opposed to a MUST, but I have no problem with
it.

>     users?

Yes, that's all it is.

> I.e. the implementation shouldn't reject a script that

I'm not sure how you'd issue such a warning and the specification certainly
doesn't require it.

>     Received header too.

I have to say I'm not seeing the difficulty here. A common trick is to parse
from the end backwards. That way all you have to contend with is comments,
which requires nothing but a simple counter.

Alternately, you can preprocess the entire header and remove all the comments
first.

>     The Date parser will be able to remove/skip

>     i.e. make the test return false and use a count of zero) fulfill the

It's a border case, but I'd have to say I'd regard it as incompliant.

I should also note that we're planning on a revision to the date-index
speification to fix an issue with the example Julian date routines (they return
regular, not modified, Julian dates). We might as well clarify these issues as
well.

				Ned

---

From: Hannah Schroeter <[email-masked]>
Date: Thu, 10 Sep 2009 14:33:26 +0200

Hello!

>[...]

>The test is supposed to fail in this case. I agree the RFC could be clearer

Thanks for the heads on (I'm not fond of the wording "fail" for "return
false", as people not firm with the language used could more easily
confuse it with "error" or "temporary failure" than if one said "return
false" or "not match", but that's a general language issue with the
Sieve test specifications; sorry for the side note, if it's noise for
you). As you say at the end of your mail, there might be a clarification
or revision on the RFC, anyway, so one could include that.

>I don't see a need for further clarification here.

I do.

On one hand, yes, one can second guess the authors of the RFCs, as you
did, and as I did, too (but still asked). Our common conclusion is the
same, so it looks like a very reasonable second guess, that one can
deduce from the text of the specs,

RFC 5231

   The COUNT match type first determines the number of the specified
   entities in the message and does a relational comparison of the
   number of entities, as defined below, to the values specified in the
   test expression.

And :index modifies the specification of fields, voilà.

So yes, the deduction looks right.

Still, I'm in favor of specs where one doesn't have to use deduction
steps like those in the first place. Even if most would probably come to
the same conclusion, it takes some thought to reach it. People less apt
in English would have a more difficult time with it and one could ease
the issue with a spec that just doesn't leave room for any speculation.

Then, I've got some compiler construction background. And there, it's a
virtue of language specs to be just clear and well-defined and
unambiguous (even if existing specs often don't completely reach up to
that goal, even for "real" programming languages) and, IMO, also without
inducing the need to deduce too much about the intended syntax and
semantics. (Though one might have to deduce much about how to *achieve*,
how to *implement* it, for example if the syntax is difficult to parse
with commonplace techniques - C++ comes to mind - or if the semantic gap
between source and target is large - some functional or logic
programming languages come to mind.)

I'd be for clarification, even if our common result is the most, or even
the only, logical conclusion.

>Yes, I think so. Our implementation certainly does.

It's not completely clear. There's precedent for tolerant behaviour, e.g.
allowing syntactically invalid header names in tests (2.4.2.2. in the
basic Sieve spec, RFC 5228), making the tests fail (return false), but
not an error, which is opposite to the precedent for strict behaviour
in other places I already cited.

As the precedent is mixed, the conclusion is less clear than in the
above question, so this should in my eyes really be clarified in a
further revision of the RFC.

Does your implementation also flag an error if it can be detected only
during run-time? (Causing an implicit keep and some kind of notification
to the script owner about the error, as 5228 says...)

>I'm not sure why it's a SHOULD as opposed to a MUST, but I have no
>problem with it.

>I'm not sure how you'd issue such a warning and the specification certainly

*nods* We currently have no mechanisms for warnings or style warnings so
we probably wouldn't add one for this case, anyway.

>which requires nothing but a simple counter.

Yes, that was the thought in the next paragraph "first pass in
*reverse*".

>Alternately, you can preprocess the entire header and remove all the comments

Probably more work, as one would have to take possible other quoting
into account, where parens do *not* start comments - while when going
backwards, that isn't an issue, IIRC, as dates don't contain quoted
strings.

>It's a border case, but I'd have to say I'd regard it as incompliant.

Okay.

>regular, not modified, Julian dates). We might as well clarify these issues as

Yes, I saw the discussion about the (modified) Julian date code there in
the archives.

>				Ned

Thanks.

Kind regards,

Hannah.

---

From: [email-masked]
Date: Thu, 10 Sep 2009 08:54:36 -0700 (PDT)

> or revision on the RFC, anyway, so one could include that.

The base specification uses the phrase "returns false" for this sort
of thing so that's what I plan to use here as well.

> deduce from the text of the specs,

There is no second guessing involved. This is simply a matter of  reeading
what's there in the relevant specifications. RFC 5231 is quite clear on how
:count interacts with header and address tests, and RFC 5260 explains how it
interacts with date. All :index does is restrict the set of headers the test
can see. It does not change the semantics of :count in any way.

It would of course be possible to reiterate what RFC 5231 says, but
reiterating one specification in another is almost never a good idea.

>    number of entities, as defined below, to the values specified in the

You need to read a little further. This section fully explains how :count
interacts with address and header tests.

> And :index modifies the specification of fields, voilà.

No it doesn't.

> So yes, the deduction looks right.

There's no deduction involved as far as I can see.

> the only, logical conclusion.

And I'm opposed, for the reasons I've given.

> to the script owner about the error, as 5228 says...)

Our implementation only flags the error at runtime. In general this is
the only time it can be done.

				Ned

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 10 Sep 2009 16:22:39 -0400

--On Thursday, September 10, 2009 08:54:36 AM -0700 
[email-masked] wrote:

> There is no second guessing involved.

No, there's not.  I'm pretty sure you were there when RFC5260 was written.

---

From: Hannah Schroeter <[email-masked]>
Date: Tue, 15 Sep 2009 17:36:31 +0200

Hi!

>> >Yes, I think so. Our implementation certainly does.

Ok, I'll implement it that way, though I'd be happier if it were
explicitly specified that way. One could as easily make a case for
making the test just return false.

>the only time it can be done.

That's right *in general* - same as for comparators.

However, it's easy to know when a string is *constant*. It always is
when the script doesn't use the variables extension. Otherwise it is
when the string doesn't expand variables. The Sieve engine needs to know
that property of strings elsewhere (e.g. for checking whether the
variable name in the "set" command is constant).

So my Sieve engine checks comparator names statically if they're
constant and defers checking to runtime when they're not (in fact it's
not very well-optimized as the preference is clean, easy code over
speed, so it checks always at runtime, but additionally statically if
easily possible; no constant folding or other fancy tricks).

Doing the checks *only* at runtime is probably also correct.

Kind regards,

Hannah.