Subject: Invalid syntax to cause runtime error in encoded-character

From: Alexey Melnikov <[email-masked]>
Date: Wed, 08 Nov 2006 14:38:58 +0000

Kjetil,
We' ve discussed your suggested text for encoded-character Sieve 
extension during the San Diego meeting.
I think people are generally happy with your text (+ Philip's proposal 
for generic syntax for "magic things inside {}"), but both Ned and I 
don't think that invalid sequences should cause runtime errors. Can you 
elaborate on why you think causing runtime errors would be better than 
just leaving invalid sequences as is (like in variables)?

Thanks,
Alexey

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 08 Nov 2006 17:38:23 +0100

> just leaving invalid sequences as is (like in variables)?

hmm, runtime isn't too appealing, but compile time would be nice.  this
actually depends on the order we process encode-character and variables.
if we do variables first (so "${hex:${var}}" works), it must be runtime.
if we do encoded-characters first (so "${v${hex:61}r}" works), it can be
compile time.

I don't have any strong feelings about it.  it's good to catch as many
errors as possible at upload time, but it may be better to leave the
task of finding probable typos to a Sieve lint program.

-- 
Kjetil T.

---

From: Philip Guenther <[email-masked]>
Date: Wed, 8 Nov 2006 16:08:33 -0800

> if we do encoded-characters first (so "${v${hex:61}r}" works), it can be

Hmm, I thought it was settled that encoded-characters were to always be 
expanded before variables, such that encoded-characters could be 
implemented purely in an implementation's lexer, or even via 
pre-substitution.  This should be stated in the variables draft, to avoid 
creating a normative reference from 3028bis to variables.  Inserting that 
shouldn't be a huge process issue.

> task of finding probable typos to a Sieve lint program.

Okay.  Given that and the opinions expressed in the meeting, I propose we 
go with the following text, mostly based on a previous diff Kjetil posted:

1) the following addition to section 2.1 "Form of the Language"

       While this specification permits arbitrary octets to appear in
       sieve scripts inside strings and comments, this has made it
       difficult to robustly handle sieve scripts that are concerned with
       the enconding of data.  The "encoded-character" capability
       (section 2.4.2.4) provides an alternative means of representing
       such octets in strings using just US-ASCII characters.  As such,
       the use of non-UTF-8 text in scripts should be considered a
       deprecated feature that may be abandoned.

2) the addition of section 2.4.2.4, as follows:

2.4.2.4. Encoding characters using "encoded-character"

    When the "encoded-character" extension is in effect, certain
    character sequences in strings are replaced by the unencoded value.
    This happens after escape sequences are interpreted and dot-
    unstuffing has been done.  Implementations SHOULD support "encoded-
    character".

    Arbitrary octets can be embedded in strings by using the syntax
    encoded-arb-octets.  The sequence is replaced by the octets with the
    hexadecimal values given by each hex-pair.

    encoded-arb-octets   = "${hex:" hex-pair-seq "}"
    hex-pair-seq         = hex-pair *(WSP hex-pair)
    hex-pair             = 1*2HEXDIG

    It may be inconvenient or undesirable to enter Unicode characters
    verbatim, and in these cases the syntax encoded-unicode-char can be
    used.  The sequence is replaced by the UTF-8 encoding of the
    specified Unicode characters, which are identified by the hexadecimal
    value of unicode-hex.

    encoded-unicode-char = "${unicode:" unicode-hex-seq "}"
    unicode-hex-seq      = unicode-hex *(WSP unicode-hex)
    unicode-hex          = 1*6HEXDIG

    It is an error for a script to use a hexadecimal value that isn't in
    either the range 0 to D7FF or the range E000 to 10FFFF.  (The range
    D800 to DFFF is excluded as those character numbers are only used as
    part of the UTF-16 encoding form and are not applicable to the UTF-8
    encoding that the syntax here represents.)

    The capability string for use with the require command is "encoded-
    character".

    In the following script, message A is discarded, since the specified
    test string is equivalent to "$$$".

    Example:  require "encoded-character";
              if header :contains "Subject" "$${hex:24 24}" {
                 discard;
              }

3) the following addition to section 6.2.3 "Initial Capability Registrations"

    Capability name: encoded-character
    Description:     changes the interpretation of strings to allow
                     arbitrary octets and Unicode characters to be
                     represented using US-ASCII
    RFC number:      this RFC (Sieve base spec)
    Contact address: The Sieve discussion list <[email-masked]>

Philip Guenther

---

From: Philip Guenther <[email-masked]>
Date: Wed, 8 Nov 2006 16:52:52 -0800

On Wed, 8 Nov 2006, Philip Guenther wrote:
>      the enconding of data.

I've already corrected the obvious typos in the above (they were mine 
and not Kjetil's) and tweaked the wording to instead say:

       While this specification permits arbitrary octets to appear in
       sieve scripts inside strings and comments, this has made it
       difficult to robustly handle sieve scripts in programs that are
       sensitive to the encodings used.

Philip Guenther

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 09 Nov 2006 03:53:25 +0100

On Wed, 2006-11-08 at 16:08 -0800, Philip Guenther wrote:
> pre-substitution.

okay, I wasn't sure it was settled, but I agree this makes the most
sense.

> This should be stated in the variables draft, to avoid 

right.  I'll probably have to remove the [REGEX] reference, too, unless
a miracle happens :)
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Thu, 09 Nov 2006 11:13:16 +0100

> okay, I wasn't sure it was settled, but I agree this makes the most

Not to me, I'm afraid.  To me, the concept of ${} as boundaries for
"magic inside" asks for strict inside-out evaluation of all magic
enabled by extensions.  Should we ever get new ${} extensions, do we
really want to specify a total order for all of them?

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 09 Nov 2006 23:40:26 +0100

> really want to specify a total order for all of them?

yes, I think so.  being able to parse this without backtracking is a big
plus in my book.  I don't see a compelling use case for nesting these
constructs.  if we introduce functions in a similar syntax, I don't
think it will be a big problem that variables expand before the
functions are evaluated.  supporting non-constant variable names is just
asking for problems, IMHO.

note that with this ordering, the encoded-character extension can be
byte-compiled away entirely, a very nice property for a language which
is meant to be lightweight and simple.
-- 
Kjetil T.

---

From: Philip Guenther <[email-masked]>
Date: Thu, 9 Nov 2006 17:34:16 -0800

On Thu, 9 Nov 2006, Kjetil Torgrim Homme wrote:
...
> is meant to be lightweight and simple.

Exactly.

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Thu, 09 Nov 2006 18:06:26 -0800 (PST)

> Exactly.

Indeed. That's certainly how I intend to implement it.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Fri, 10 Nov 2006 11:02:40 +0100

> asking for problems, IMHO.

On evaluating variables before anything else: Of course that forbids
functions ever opening a new scope with new variables, or functions
having side effects on variables.  Exim uses the second to store the
results of lookups.  I don't particularly like that, but use it and must
admit I don't have a better idea.

To me, it does not feel like a good idea to forbid such extensions at
this point.

> is meant to be lightweight and simple.

It also means you can never ever add an extension to have ${hex}
work on variable expansions, i.e. one that adds true functions.

I think we are looking at ${hex} from two perspectives: I think of a
function of (in absence of other extensions constant) arguments, and
you think of it purely as quoting.  In case we don't find a unification
we are all happy with, I suggest to find a new name that does not look
like a function at all.  Like ${0xdeadbeef}, or ${\xdeadbeef} and
${\u0040}, or just anything that not being an identifier.

Michael

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 15 Nov 2006 18:39:04 -0500

> ${\u0040}, or just anything that not being an identifier.

I think that that difference in perspective is apparent.  Some want to
think of ${fn:} as invoking a run-time string function, If you have that
perspective, it seems to me that it's almost inevitable that one day you
will want nested evaluation of functions and variables (probably in an
order dictated by the nesting usage and not by arbitrary rules).  I
thought that the ${xx:} syntax was presented as a function reference,
allowing for further functions in the future.  So I guess I agree with
Michael Haardt here.

Imagine, for example, that :lower/:upper et al had never been thought of
for the variables draft (along with the pesky priority table).  One
might come up instead with, e.g., "${upperfirst: ${lower: ${var}}}"

That is perfectly byte-compilable..  you should be able to byte-compile
nested and mixed references anyway, as long as you don't allow for the
expansions to produce new references.  There's a difference between
nested/mixed evaluation, which I think has merit, and rescanning for new
substitutions, which if done implicitly (without an 'eval' sort of
function) could bring on a bunch of evils.

There was also this:

On Wed, Nov 08, 2006 at 04:08:33PM -0800, Philip Guenther wrote:
   ...
> implemented purely in an implementation's lexer, or even via 

If you implement this in the lexer, you are allowing a 'require'
statement to change the output of the lexer: not to mention having to
have the lexer work differently for "require" strings than for others
(assuming that the prohibition against expanding strings in "require"
still stands).  I suppose one could do that, but it doesn't seem
very, well, pure.

[I've been buried in other stuff for a while, but I did spend time
during the last couple of days to read every saved list message that
came in over that period.  I think.]

-mm-