Subject: Re: Proposal for escaping on non-UTF-8 sequences in Sieve

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sun, 01 Oct 2006 16:04:09 +0200

On Mon, 2006-09-18 at 15:12 -0600, Philip Guenther wrote:
> On Mon, 18 Sep 2006, Michael Haardt wrote:
> > On Mon, Sep 18, 2006 at 09:44:20AM -0700, Ned Freed wrote:
> >> There are all sorts of ways to do it. Here's an obvious one: If the
> >> octet-value extension is enabled, any occurances of ${x} where x takes
> >> the form of a space-separated list of decimal values is replaced with
> >> a sequence of octets corresponding to each value.
> >
> > What a brilliant idea! How about using the same syntax variables do,
> > and embedding arbitrary data written as a function?
> 
> You mean something like option (3) from my message of March 2005?
>     http://www.imc.org/ietf-mta-filters/mail-archive/msg02018.html

(sorry about responding so late to the discussion!)

the difference is that your suggestion added a dependency on variables.
the new idea is that we're using a syntax which is compatible with
variables, but can be used and implemented on its own.

some issues which would need ironing out:

variables don't allow a reference which acts as a function with
arbitrary input (e.g., "${hex:e6 f8 e5}"), the tail end has to be an
identifier or numbered variable.  unfortunately, this means ${hex:7e} is
disallowed, since "7e" is neither.  to solve this, you'd either need to
abandon hex and go for decimal values, or to introduce a prefix
character in addition to the namespace: ${hex:x7e}.  if we choose the
latter, we could consider using one namespace for several encodings,
e.g., ${data:x7e}, ${data:u007e} and, if we want to stretch it
maximally, ${data:q_53ort__of__q_2dp}

-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Mon, 2 Oct 2006 13:39:47 +0200

On Sun, Sep 24, 2006 at 10:06:44PM +0100, Alexey Melnikov wrote:
> After thinking more about this (and reviewing list of options provided 
> by Philip), I would like to propose a variables-like syntax close to 
> what you suggested above:
> $%<hex>
> controlled by a new Sieve capability "string-escape".
> This would give existing implementations an ability to provide the 
> escaping functionality in both quoted and multiline strings, without the 
> need to implement variables.
> Any objections?

Just curious: What speaks against using the syntax of variables?  It is
certainly a MUST that the new extension does not require variables,
but that does not forbid to use their syntax.

$%xx$%xx is OK for me, but ${hex:xxxx} is easier to read and we don't
get in trouble, should someone really miss ${frombase64:xxxx} and add it.

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 05 Oct 2006 11:16:31 +0200

> get in trouble, should someone really miss ${frombase64:xxxx} and add it.

as noted in my message on Sunday, you can't make ${frombase64:xxxx}.
even quoted-printable is a stretch, as I demonstrated (_jokingly_ I
hasten to add, although perhaps that wasn't clear).

I would still like to see a escape mechanism which also allows the entry
of Unicode (in UTF-8) in a pure US-ASCII script, I only mention this
since $%xx isn't really flexible enough for this -- the user shouldn't
have to do the UTF-8 transformation themselves.
-- 
Kjetil T.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 05 Oct 2006 16:58:47 +0100

>have to do the UTF-8 transformation themselves.
The current proposal ($%xx) is sufficient to address the issue in the 
base spec in order to be able to send it to IESG.
If you want to add Unicode, you need to suggest some text now.

Alexey,
Sieve co-chair

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 06 Oct 2006 03:24:37 +0200

On Sun, 2006-10-01 at 16:04 +0200, Kjetil Torgrim Homme wrote:
> disallowed, since "7e" is neither.

I'm sorry, I was very confused.  the "variables" syntax uses period, not
colon, to separate namespaces, so the suggested syntax would _resemble_
it, but be completely independent of it ("variables" only affects
substrings which match the specified syntax exactly, so there's
definitely no conflict).

in other words, we can do whatever we like with ${keyword:data}.  I
prefer an extensible syntax over a compact one (Alexey's $%xx
suggestion), so my vote is for ${hex:7e}.  please see suggested patch
below.

a couple of notes:

a) the extension name, "quoted-character", is off the top of my head.
feel free to use a different one if you prefer ("encode-char",
perhaps?).

b) the syntax requires spaces between the items.  it's possible to allow
${hex:ABCD} if we use this syntax instead:

   quoted-arb-octets   = "${hex:" hex-pair-seq "}"
   hex-pair-seq        = hex-pair *(*WSP hex-pair)
   hex-pair            = 2HEXDIG

on the other hand, some Unicode code points may need five hex digits
(such as U+1D10A, which is 𝄊, the musical symbol "da capo"), but these
are quite rare, so most people will probably want to write just four
digits (e.g. U+4e2d U+56fd, which is 中国, the name of China).  this
means a sequence of Unicode characters can't be written unambiguosly and
conveniently as one long string of hex digits.  therefore, to make them
consistent, both encodings require the values to be split by whitespace.
I think this improves readability, anyway.

c) it may be presumptious of me to add this extension to the "SHOULD be
implemented" list, I won't be offended if it's listed as "MAY" ;-)

d) I'm no expert on ABNF, so please review.

-- 
Kjetil T.

--- draft-ietf-sieve-3028bis-09.txt	2006-10-06 01:47:33.989869000 +0200
+++ draft-ietf-sieve-3028bis-kjetilho.txt	2006-10-06 03:22:35.025194000 +0200
@@ -385,13 +385,9 @@
    are permitted in quoted strings.  Quoted strings MAY span multiple
    lines.  NUL (US-ASCII 0) is not allowed in strings.
 
-   As message header data is converted to [UTF-8] for comparison (see
-   section 2.7.2), most strings will use the UTF-8 encoding.  However,
-   implementations MUST accept all strings that match the grammar in
-   section 8.  The ability to use non-UTF-8 encoded strings matches
-   existing practice and has proven to be useful both in tests for
-   invalid data and in arguments containing raw MIME parts for extension
-   actions that generate outgoing messages.
+   The extension "quoted-character" may be used to encode arbitrary
+   characters as a sequence of US-ASCII characters (see 2.4.2.4 for
+   details).
 
    For entering larger amounts of text, such as an email message, a
    multi-line form is allowed.  It starts with the keyword "text:",
@@ -470,6 +466,42 @@
    valid, but need not ensure that they actually identify an email
    recipient.
 
+2.4.2.4. Encoding characters using "quoted-character"
+
+   When the "quoted-character" extension is in effect, certain
+   character sequences in strings are replaced by the unencoded value.
+   This happens after escape sequences are interpreted and
+   dot-unstuffing has been done.
+
+   Arbitrary octets can be embedded in strings by using the syntax
+   quoted-arb-octets.  The sequence is replaced by the octets with the
+   hexadecimal values given by each hex-pair.
+
+   quoted-arb-octets   = "${hex:" hex-pair-seq "}"
+   hex-pair-seq        = hex-pair *(WSP hex-pair)
+   hex-pair            = 1*2HEXDIG
+
+   It may be inconvenient or undesirable to enter Unicode characters
+   verbatim, and in these cases the syntax quoted-unicode-char can be
+   used. The sequence is replaced by the UTF-8 encoding of the
+   specified Unicode characters, which are identified by the
+   hexadecimal value of unicode-hex.
+
+   quoted-unicode-char = "${unicode:" unicode-hex-seq "}"
+   unicode-hex-seq     = unicode-hex *(WSP unicode-hex)
+   unicode-hex         = 1*5HEXDIG
+
+   The capability string for use with the require command is
+   "quoted-character".
+
+   In the following script, message A is discarded, since the
+   specified test string is equivalent to "$$$".
+
+   Example:   require "quoted-character";
+              if header :contains "Subject" "$${hex:24 24}" {
+                    discard;
+              }
+
 2.5.     Tests
 
    Tests are given as arguments to commands in order to control their
@@ -1075,7 +1107,7 @@
    Implementations MUST support the "keep", "discard", and "redirect"
    actions.
 
-   Implementations SHOULD support "fileinto".
+   Implementations SHOULD support "fileinto" and "quoted-character".
 
    Implementations MAY limit the number of certain actions taken (see
    section 2.10.4).
@@ -1561,6 +1593,12 @@
    RFC number:      this RFC (Sieve base spec)
    Contact address: The Sieve discussion list <[email-masked]>
 
+   Capability name: quoted-character
+   Description:     changes the parsing of strings to allow arbitrary
+                    characters to be embedded
+   RFC number:      this RFC (Sieve base spec)
+   Contact address: The Sieve discussion list <[email-masked]>
+
    Capability name: comparator-* (anything starting with "comparator-")
    Description:     adds the indicated comparator for use with the
                     :comparator argument

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 06 Oct 2006 15:27:25 +0100

>suggestion), so my vote is for ${hex:7e}.  please see suggested patch
I would prefer if we pick a more unique prefix. Something starting with 
'$' but not followed by '{' would be great. However if others feel 
strongly in favor of your variant, that would be fine too. Apart from 
that your proposal is fine with me.

>feel free to use a different one if you prefer ("encode-char",
"encode-char" is slightly better, IMHO. quoted-* has strong association 
with quoted strings.

>implemented" list, I won't be offended if it's listed as "MAY" ;-)
Speaking personally, SHOULD is fine with me.

>d) I'm no expert on ABNF, so please review.
====

>+++ draft-ietf-sieve-3028bis-kjetilho.txt	2006-10-06 03:22:35.025194000 +0200
I am against this change, as it doesn't agree with the rough consensus 
in the group, which is to try keep existing implementations compliant.

However your change is fine, as long as the original text is not deleted.

Did you really want to allow for
${hex: 7 8 9}
?

The rest looks fine to me.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 06 Oct 2006 16:57:09 +0200

> that your proposal is fine with me.

glad to hear that.  yes, the resemblance to variables syntax is a mixed
blessing.

pro: it's easier to recognise magic sequences in the string.
con: it's easier to mix up the two syntaxes.

I want to note that a syntax mix-up can be flagged at upload time, so I
don't think it's a big problem in practice.

> in the group, which is to try keep existing implementations compliant.

this was a bit lazy editting on my part.  I made the argument for the
reinstating of status quo in a different thread, so please ignore the
removal here.

> Did you really want to allow for

not sure what you're pointing at here ...

a) yes, it may be a good idea to add leading and trailing WSP* in
quoted-arb-octets to allow arbitrary extra whitespace.

b) yes, it may be more readable to specify hex-pair as 2HEXDIG.  I made
it 1*2HEXDIG since the Unicode is 1*5HEXDIG, so I suggest we change the
latter to 2*5 if the former is 2HEXDIG.

so I'm game whatever you say :-)
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Fri, 6 Oct 2006 17:05:06 +0200

> that your proposal is fine with me.

Variables can be seen as functions without arguments.  I don't think
of the suggested extension it as quoting, but as function evaluation of
constant arguments.  Add variables, and you have functions of variable
arguments.

That's why I would like to use the syntax for argument-less functions,
aka variables, for functions with arguments as well. :)

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 06 Oct 2006 18:01:17 +0100

>con: it's easier to mix up the two syntaxes.
I think cons. outweigh pros. in this case. Somebody can forget to use 
':' after hex, etc.

>reinstating of status quo in a different thread, so please ignore the
So I will argue in another thread ;-).

>quoted-arb-octets to allow arbitrary extra whitespace.
This would be fine with me, but I was pointing out at single character 
hex-pair.

>latter to 2*5 if the former is 2HEXDIG.
I would rather use 2HEXDIG for octets and 1*5HEXDIG for Unicode, but 
2*5HEXDIG is fine as well.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 18 Oct 2006 19:49:55 +0200

> I think cons. outweigh pros. in this case. Somebody can forget to use 

we have three proposals here:

  $%<hex>
  ${hex:<hex>}

and a slightly more nebulous "a more unique prefix", perhaps this
qualifies:

  $[hex:<hex>]

that would be fine with me, in any case.

> >don't think it's a big problem in practice.

this is not entirely correct, a typo like ${hex.ff} can only be flagged
if "variables" is in use, otherwise it will be used verbatim.

> So I will argue in another thread ;-).

you haven't done so yet :-)  it may have been lost in the long thread,
I'll repost my suggestion in a new thread.

-- 
Kjetil T.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 18 Oct 2006 18:54:25 +0100

>that would be fine with me, in any case.
I like the last one, where <hex> can be a sequence of space separated 
hex pairs.

>you haven't done so yet :-)
>
No, I haven't. Sorry. I will post a message on this shortly.

>it may have been lost in the long thread,
No need.

---

From: Michael Haardt <[email-masked]>
Date: Thu, 19 Oct 2006 08:36:09 +0200

> that would be fine with me, in any case.

I like ${hex:<hex>}, because it looks like a function and we can use the
same syntax for other string expansions, too.

> if "variables" is in use, otherwise it will be used verbatim.

This particular typo will indeed be used verbatim, whereas ${hex:fg} can
be recognised.

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 19 Oct 2006 14:49:38 +0200

> same syntax for other string expansions, too.

I think it only looks like a function to you because you're used to
exim.conf :-)  if you know of other languages where functions looks like
this, I'm interested in hearing about it.

if we want a syntax well known from other languages, we could do

  $(hex <hex> <hex>)

this has a strong similarity to POSIX shell.

which of ${, $[ or $( we use is not important to me.
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Thu, 19 Oct 2006 15:16:07 +0200

> this, I'm interested in hearing about it.

Yes and no. :) Yes: It IS familar, because I use Exim, and I know of no
other language using that notation.  No: Seen from an abstract point
of view, a variable and a function call differ only in some syntactic
element that specifies arguments, because both deliver a rhs value
(assuming strict evaluation).

That's why I like to keep function calls very close to variables.

If you dislike the colon, I don't mind exchanging that against some other
token, as long as it is not in the follow-set of the variable identifier
in the variables extension.

I like ${...}, because it specifies the borders of "magic inside".
All the magic we currently have are variables, but that does not mean
things have to stay that way.

Michael

---

From: "Nigel Swinson" <[email-masked]>
Date: Thu, 19 Oct 2006 15:22:52 +0100

> things have to stay that way.

Me too.  :o)  I think $[ or $( just throw things more open to error, forgetting which braket to use :o/

Nigel

---

From: Philip Guenther <[email-masked]>
Date: Thu, 19 Oct 2006 18:28:42 -0600

On Wed, 18 Oct 2006, Kjetil Torgrim Homme wrote:
...
> we have three proposals here:

I dislike this as too specialized.

> that would be fine with me, in any case.

I think these are both acceptable; I have a slight preference for ${}

As for the name of the extension, I think it should be "encoded-character" 
instead of "quoted-character": this is not a quoting scheme but rather an 
encoding scheme.

Regarding the description of the language in section 2.1, how does the 
following addition (based on text from Alexey) look:

       While this specification permits arbitrary octets to appear in
       sieve scripts inside strings and comments, this has made it
       difficult to robustly handle sieve scripts in user interfaces.
       The "encoded-character" capability (section 2.4.2.4) provides an
       alternative means of representing such octets in strings using
       just US-ASCII characters.  As such, the use of non-UTF-8 text in
       scripts should be considered a deprecated feature that may be
       abandoned.

That would be inserted as the third paragraph in that section and given 
additional indentation (ala the behaviors described in section 2.7.2).

Philip Guenther

---

From: Michael Haardt <[email-masked]>
Date: Fri, 20 Oct 2006 09:05:50 +0200

> instead of "quoted-character": this is not a quoting scheme but rather an 

Or possibly "encoded-octet"?

> additional indentation (ala the behaviors described in section 2.7.2).

To me, it does not point out strong enough that new implementations
shold not allow non-UTF-8 octet sequences AT ALL.

How about:

       Sieve scripts SHOULD NOT use non-UTF-8 octet sequences.
       If non-UTF-8 octet sequences are required in strings, the
       "encoded-octet" capability (section 2.4.2.4) provides an
       alternative means of representing such octets in strings using
       just US-ASCII characters.

It's not a MUST and backwards compatibility of specific implementations
is certainly a very good reason to act against the SHOULD.

Michael

---

From: Philip Guenther <[email-masked]>
Date: Fri, 20 Oct 2006 02:32:37 -0600

There are a number of issues that need to be clarified before the proposed 
extension will be ready for inclusion:

Do implementations have to support encodings of NUL ala ${hex:0}?  If so, 
that will be a barrier to this extension being broadly supported.
(I think it should be a "MAY support".)

Which comes first, encoding replacement or variable expansion?  Or are 
they concurrent?  Whatever the answer, the variables I-D will need to make 
that clear.
(I think encoding replacement should come before variable expansion.)

unicode-hex should be 1*6HEXDIG: the Unicode range goes up to 10FFFF

The next rev of the base-spec includes this:
 	Extensions MUST NOT change the behavior of the "require"
 	control command.
I believe that means that this extension can't be used in the capability 
argument to 'require', so there's at least one place where Unicode 
characters can't be encoded using ${unicode:...}.

I'll note that while a script that uses this extension for all non-UTF8 
octets may be displayable without munging, the result may still be 
incomprehensible if, for example, an 8bit ISO-2022-JP MIME part is 
included in a string.  Indeed, such encoding will probably make it more 
difficult to display that MIME part readably: currently, if a user is 
viewing the script with raw ISO-2022-JP in it they can probably display 
that MIME part by overriding their browser's charset encoding for the 
page.

Philip Guenther

---

From: Michael Haardt <[email-masked]>
Date: Fri, 20 Oct 2006 11:43:20 +0200

> (I think it should be a "MAY support".)

Note that =00 in quoted printable has to be dealt with already.
An implementation that uses nul-terminated strings is already broken
and things will not get worse using an encoded NUL character in a string.

> (I think encoding replacement should come before variable expansion.)

I think they should be processed inside-out, but not in separate
passes.  Variables introduce of the concept of strings (aka test
and action arguments) not being literals, but string expressions that
are evaluated.  It makes sense that arguments of string functions
turn from literals to string expressions, too.

As a result, without variables, "${hex:${hex:4646}}" is a syntax error.

With variables, it is "FF".

> characters can't be encoded using ${unicode:...}.

That's right.  The require argument can not contain variables either,
and I consider that a good thing [tm].

> that MIME part by overriding their browser's charset encoding for the 

Indeed, a hack like that is not as useful any more.  But it was a hack
to begin with, because overriding the charset encoding renders contained
UTF-8 text useless.  Convert the ISO-2022-JP part to UTF-8.

Michael

---

From: Philip Guenther <[email-masked]>
Date: Fri, 20 Oct 2006 05:01:19 -0600

>> (I think it should be a "MAY support".)
nn> Note that =00 in quoted printable has to be dealt with already.

If you're referring to RFC 2047's Q encoding, 3028bis currently says:

       <...>  An encoded NUL octet
       (character zero) SHOULD NOT cause early termination of the header
       content being compared against.

So an implementation that refused to match beyond the =00 would still be 
"conditionally compliant".

> and things will not get worse using an encoded NUL character in a string.

So, in your opinion, being able to insert NUL into strings is as important 
as the ability to encode other, non-UTF-8, non-NUL octets, and that 
support for those capabilities should therefore be tied?  I think the 
result will be fewer implementations of this extension as well as 
implementation that don't actually comply.

The original MIME documents didn't make an exception for NUL...and that 
had to be changed when they were revised as part of moving to Draft 
Standard.  RFC 2049 says:
     (17)  The definitions of "7bit" and "8bit" have been
           tightened so that use of bare CR, LF can only be used
           as end-of-line sequences.  The document also no longer
           requires that NUL characters be preserved, which brings
           MIME into alignment with real-world implementations.

> As a result, without variables, "${hex:${hex:4646}}" is a syntax error.

No, it would be replaced with "${hex:FF}"

> With variables, it is "FF".

1) Given that variables are *explicitly* not handled that way, why should
    that be true of these?

2) Why would pulling in the variable capabilty change the behavior of the
    above?

2) The behavior you describe would be useful how?

> UTF-8 text useless.

No, US-ASCII is a subset of ISO-2022-JP, so at least the syntax portions 
of the script would still be readable.  Is it a hack?  Yes.  Does it work? 
Yes.  Will there be any equivalent when the raw ISO-2022-JP is encoded 
using ${hex}?  I strongly doubt it.

> Convert the ISO-2022-JP part to UTF-8.

My interactions with Sendmail's Japanese customers tell me that 
ISO-2022-JP is still heavily preferred over UTF-8 in email in Japan.  Are 
you claiming otherwise?  Or do you think that it doesn't matter?

Philip Guenther

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 20 Oct 2006 11:06:41 +0100

>and things will not get worse using an encoded NUL character in a string.
Right.

>With variables, it is "FF".
I disagree. I think encoded-octet and variables should be independent 
extensions.
Speaking as implementor I am more likely to implement encoded-octets first.

>and I consider that a good thing [tm].
Agreed.

---

From: Michael Haardt <[email-masked]>
Date: Fri, 20 Oct 2006 14:01:02 +0200

> "conditionally compliant".

So how about handling an encoded NUL in strings just the same? SHOULD NOT
is reasonable.

>            MIME into alignment with real-world implementations.

Without looking it up, it sounds as if that applies to unencoded NUL
characters.  I agree there is a real-world problem with them and the
above makes perfect sense.  I would not like if quoted-printable may
drop =00.

> 2) The behavior you describe would be useful how?

The variable capability changes how argument string are interpreted.
Arguments to string functions are strings, too.  That's why it makes
sense to me that variables cause recursive evaluation, but that may
just be me.

Implementations that do not implement variables can scan strings
with a linear effort and without the need for recursion or a parser.
Side-effect free functions of constant arguments can be evaluated until
you reach the top level, which can not be represented in UTF-8, hence
we need the top level to encode the result, but no more.

I don't insist on this behaviour, it was just a suggestion.  Using
${hex:...} opens the door to other string functions, and we are really
talking about introducing string functions here and the question is:
Which extension introduces recursive evaluation? I suggested variables,
but it may as well be some not-yet-specified extension.

> you claiming otherwise?  Or do you think that it doesn't matter?

Actually, my feedback from customers is that national character sets
are still preferred over UTF-8 - independent of their location.  I am
REAL GLAD Sieve went the UTF-8 route to begin with, because national
character sets are a royal pain and I hope that historic crap disappears
some day.  Each day Unicode is used a little more, we are a little closer
to that point.

Personally, I have no stuff capable of displaying UTF-8, and looking at
other (Linux) installations, the claim of being able to switch completely
to Unicode is not YET true.  iconv(1) and similar tools help a lot
with existing installations that are tied to national character sets,
e.g. when viewing a Sieve script stored in UTF-8 on my ISO-8859-15
display.  Japanese may be worse, but German has a non-US-ASCII letter
that only exists in lower case.  Translating it to upper case results
in two letters and you can not convert it back without a dictionary.
It's not like I would be ignorant of non-US-ASCII text.

I had hordes of customers yell and scream at me that they will cease to
exist without their open SMTP relay.  Some probably still hate me for
having to close it.  I don't always listen to their wishes.

Michael

---

From: Michael Haardt <[email-masked]>
Date: Fri, 20 Oct 2006 14:06:43 +0200

> I disagree. I think encoded-octet and variables should be independent 

That's 2:1 now saying we need a new extension to evaluate "${hex:${variable}}".
Anybody else?

> Speaking as implementor I am more likely to implement encoded-octets first.

So am I.

Michael

---

From: "Nigel Swinson" <[email-masked]>
Date: Fri, 20 Oct 2006 13:41:51 +0100

> I disagree. I think encoded-octet and variables should be independent 

I agree

> That's 2:1 now saying we need a new extension to evaluate "${hex:${variable}}".

I would want ${hex:${variable}} to be a syntax error if "hex" was requir()ed, and valid if variables was require()ed and "hex" was not.  I don't see a need for recursive evaluation.  I'm used to one pass, just like dequoting.  At the moment I can't think of any recursive string expansion out there...  Instead put your hex into ${variable} when constructing it.

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 20 Oct 2006 15:21:09 +0200

> instead of "quoted-character": this is not a quoting scheme but rather an 

agreed.

> additional indentation (ala the behaviors described in section 2.7.2).

this is good enough for me.  I think "deprecated" is stronger than
"SHOULD NOT" in practice :-)

-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 20 Oct 2006 15:17:48 +0200

> > I disagree. I think encoded-octet and variables should be independent 

ditto.

> I would want ${hex:${variable}} to be a syntax error if "hex" was
> requir()ed, and valid if variables was require()ed and "hex" was not.
> I don't see a need for recursive evaluation.  I'm used to one pass,
> just like dequoting.  At the moment I can't think of any recursive
> string expansion out there...  Instead put your hex into ${variable}
> when constructing it.

this implies variables should evaluated first.

I must admit I envisaged a single pass (concurrent evaluation) with no
backtracking, which means

  "${hex:${var}}" -> "${hex:VALUE}"
  "${v${hex:61}}" -> "${var}"

but I now think it makes more sense to order them explicitly.  it makes
most sense to have encoded-character evaluated before variables (since
it's defined in the base spec), but it's probably more useful to do
variables before encoded-characters, as Nigel demonstrates.

oh.  I just realised my suggestion has the same shortcoming as the
variables syntax:  it won't raise syntax errors, since it only matches
well-formed character sequences.  I may have a look at fixing that
during the weekend.
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Fri, 20 Oct 2006 14:50:00 +0200

> > I disagree. I think encoded-octet and variables should be independent 

Ok, so let both be top-level things, working independent of each other.

> I would want ${hex:${variable}} to be a syntax error if "hex" was requir()ed, and valid if variables was require()ed and "hex" was not.  I don't see a need for recursive evaluation.  I'm used to on…

Hmm.  So the result of the second would be "${hex:variable value}"?

I tend to say: If ${ is seen and there is no }, or stuff inside that
can't be processed for whatever reason, then cause an error.  But I am
not sure how variables defined that so far.

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sat, 21 Oct 2006 22:05:28 +0200

> well-formed character sequences.

here's my amended 2.4.2.4, which attempts to fix it.  the wording is a
bit weasely, but I think it will work in practice.  if anyone can fix it
more formally in ABNF, please help out.

I hope it addresses the issues brought up (6 digit Unicode, single pass,
NUL, extension name).  thanks for the feedback, everyone!

-- 
Kjetil T.

--- draft-ietf-sieve-3028bis-09.txt	2006-10-06 01:47:33.989869000 +0200
+++ draft-ietf-sieve-3028bis-kjetilho.txt	2006-10-21 21:52:27.461172000 +0200
@@ -393,6 +393,10 @@
    invalid data and in arguments containing raw MIME parts for extension
    actions that generate outgoing messages.
 
+   The extension "encoded-character" may be used to encode arbitrary
+   characters as a sequence of US-ASCII characters (see 2.4.2.4 for
+   details).
+
    For entering larger amounts of text, such as an email message, a
    multi-line form is allowed.  It starts with the keyword "text:",
    followed by a CRLF, and ends with the sequence of a CRLF, a single
@@ -470,6 +474,46 @@
    valid, but need not ensure that they actually identify an email
    recipient.
 
+2.4.2.4. Encoding characters using "encoded-character"
+
+   When the "encoded-character" extension is in effect, character
+   sequences in strings which match the encoded-seq syntax are
+   replaced by the decoded value.  This matching happens after escape
+   sequences are interpreted and dot-unstuffing has been done.  A
+   single pass is done.
+
+   encoded-seq         = "${" enc-method ":" enc-argument "}"
+   enc-method          = "hex" / "unicode"
+   enc-argument        = hex-list
+   hex-list            = hex-group *(WSP hex-group)
+   hex-group           = 1*6HEXDIG
+
+   Arbitrary octets can be embedded in strings by using the encoding
+   method "hex".  The sequence is replaced by the octets with the
+   hexadecimal values given by each hex-group.  Values greater than
+   255 ("ff") are a syntax error.
+
+   It may be inconvenient or undesirable to enter Unicode characters
+   verbatim, and in these cases the method "unicode" can be used. The
+   sequence is replaced by the UTF-8 encoding of the specified Unicode
+   characters, whose code points are identified by the hexadecimal
+   value of each hex-group.
+
+   Values for enc-method or enc-argument which don't match the above
+   syntax SHOULD cause a syntax error.  Implementations SHOULD support
+   encoded NUL octets.
+
+   The capability string for use with the require command is
+   "encoded-character".
+
+   In the following script, message A is discarded, since the
+   specified test string is equivalent to "$$$".
+
+   Example:   require "encoded-character";
+              if header :contains "Subject" "$${hex:24 24}" {
+                    discard;
+              }
+
 2.5.     Tests
 
@@ -1075,7 +1119,7 @@
 
-   Implementations SHOULD support "fileinto".
+   Implementations SHOULD support "fileinto" and "encoded-character".
 
@@ -1561,6 +1605,12 @@
 
+   Capability name: encoded-character

---

From: Philip Guenther <[email-masked]>
Date: Sun, 22 Oct 2006 23:30:38 -0600

...
+   encoded-seq         = "${" enc-method ":" enc-argument "}"
+   enc-method          = "hex" / "unicode"
...
+   Values for enc-method or enc-argument which don't match the above
+   syntax SHOULD cause a syntax error.

Hmm, that won't work, as there isn't a defined meaning for something to 
not match _part_ of a syntax.  The only sensible interpretation I can see 
would be to match *anything* for enc-method and enc-argument, and then 
compare them to their expected forms.  The problem, with that is that it 
would render this string a syntax error:
 	"${name}: ${value}"

because it's an attempt to use encoded-seq with an enc-method of "name}" 
and a enc-argument of " ${value".  That's obviously not the desired 
result.

To obtain the desired result, we have to give the syntax for all the 
sequences that should be covered by the encoded-character extension, both 
those that have a defined expansion and those that should be treated as a 
syntax error.  How broad do we want to make that?  The broadest would 
cover any sequence matching this:

encoded-seq     = "${" enc-method ":" enc-argument "}"
enc-method      = *(%x01-39 / %x3b-7c / %x7e-ff)
 		  ; zero or more characters other than ':' or '}'
enc-argument    = *(%x01-7c / %x7e-ff)
 		  ; zero or more characters other than '}'

I.e., any sequence that starts with '${', followed by zero or more 
octets other than ':' and '}', followed by a ':', then zero or more 
octets other than '}', then finally a '}', would be considered a use of 
the encoded-seq syntax.

To put it another way, if you find a '${', and there's at least one colon 
between that and the next '}', it's an encoded-seq.

That would leave
 	"${name}: ${value}"
with its expected value, but make this:
 	 "${name: sdlkfjs}"
or this:
 	"${a.44:}"
a syntax error.  On the other hand, this:
 	"${hex.ff}"
would _not_ be a syntax error, because it doesn't contain a colon between 
the braces.  That's correct, of course, because we _want_ it to be a 
variable reference instead.

The above is the broadest syntax as makes sense.  We could tighten it up 
and only cover sequences where the enc-method is, say, alphanumeric.  If 
we did that, then this:
 	"${a.44:}"
would not be a syntax error.  Opinions?

Philip Guenther