Subject: document status: 3028bis, body, editheader

From: Philip Guenther <[email-masked]>
Date: Mon, 20 Mar 2006 22:59:50 -0800

To either head-off or prime the conversation for tomorrow...

draft-ietf-sieve-3028bis-06.txt
-------------------------------

The primary issue for the base-spec has been whether it's description
of matching is inline with that description in the collation doc,
draft-newman-i18n-comparator.  I _believe_ the current drafts are
in agreement on the details.  Either confirmation or corrections
from others are needed.

I've made one tweak for the next rev to the wording in section 2.7.1
regarding ':match' and the definition of 'character'; I had overlooked
an off-list comment at the turn of the year.  The paragraph now
reads:
   The ":matches" match type specifies a wildcard match using the
   characters "*" and "?"; the entire value must be matched.  "*"
   matches zero or more characters in the value and "?" matches a single
   character in the value, where the comparator that is used (see 2.7.3)
   defines what a character is.  For example, the comparators "i;octet"
   and "en;ascii-casemap" define a character to be a single octet so "?"
   will always match exactly one octet when one of those comparators is
   in use.  In contrast, the comparator "i;basic;uca=3.1.1;uv=3.2"
   defines a character to be any UTF-8 octet sequence encoding one
   Unicode character and thus "?" may match more than one octet.  "?"
   and "*" may be escaped as "\\?" and "\\*" in strings to match against
   themselves.  The first backslash escapes the second backslash;
   together, they escape the "*".  This is awkward, but it is
   commonplace in several programming languages that use globs and
   regular expressions.

As listed in the doc, the open issues are:
 1) is 'reject' going back in here as a "do the the best you can" action?
 2) should 3028bis recommend/require that fileinto map UTF-8 to mUTF-7
    when working with an IMAP store?

The latter thought actually came about from pondering the description
of fileinto after a suggestion that all the fileinto examples in
spec should use a consistent style of folder naming (e.g., always
using a prefix of "INBOX.").  I felt it better to leave the spec
mixed, if just to avoid the implication that a particular style was
meant to be mandated.  Indeed, I went ahead and inserted the text
in section 4.1:
   Implementations MAY place restrictions on
   folder names; use of an invalid folder name MAY be treated as an
   error or result in delivery to an implementation-defined folder.

Should it say more than this in either direction?  I pondered adding
text about mapping UTF-8 to mUTF-7 and a supporting example (the
the open issues entry), but it couldn't find a wording that didn't
feel like one step too far beyond the scope of sieve.

draft-ietf-sieve-body-03.txt
----------------------------

This had been blocked pending the settling of the comparator/charset
wording in the base spec.  Again, I believe it to be in agreement
but confirmation is needed.  At that point, this can be put into
the submission queue behind the 3028bis.

draft-ietf-sieve-editheader-04.txt
----------------------------------

This has been reading for submission for a while; I think the
write-up is even done.  This last rev was basically editorial.

Philip Guenther

---

From: Michael Haardt <[email-masked]>
Date: Tue, 21 Mar 2006 15:26:52 +0100

> an off-list comment at the turn of the year.  The paragraph now

Sounds very good to me.

>     when working with an IMAP store?

Somehow the question does not sound right to me, see below.

>    error or result in delivery to an implementation-defined folder.

That is a very good path between all alternatives and I like it as it
is.

> feel like one step too far beyond the scope of sieve.

I would appreciate a requirement that says: If the implementation uses
a different unicode encoding scheme than UTF-8 for folder names, it MUST
translate the folder name from UTF-8 to its encoding scheme.

It would fit into the general picture of Sieve that uses UTF-8 everywhere
and relies on the implementation to convert.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 21 Mar 2006 07:48:48 -0800

>draft-ietf-sieve-3028bis-06.txt
I think you've missed a couple of issues I've reported:
1). I can't see any text about stripping leading/trailing whitespaces 
(text moved from the relational extension)
2). I think IANA template is not precisely defined, I am not clear on 
what is the difference between "capability name" and "capability 
keyword"? Is the former supposed to be a human readable phrase? Also, 
what should be put into "Capability arguments" if an extension supports 
multiple actions/tests.

>   and "en;ascii-casemap" define a character to be a single octet so "?"
I don't think en;ascii-casemap is working on octets. So either the 
comparators draft need to be changed, or this sentence need to be changed.

> 1) is 'reject' going back in here as a "do the the best you can" action?
I would rather keep it separate, as reject is still going through 
massive changes.

>    when working with an IMAP store?
Yes please.

>meant to be mandated.
I agree.

>Indeed, I went ahead and inserted the text
>in section 4.1:
Yes, the text should be carefully worded to make it clear that this 
restriction is on mailstores supporting IMAP.

>draft-ietf-sieve-body-03.txt
(Chair hat on): The draft needs to address the issue with 
leading/trailing spaces.
Also there might be some interaction with reject I would like to discuss 
(I will send a separate email on this).

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 21 Mar 2006 07:52:45 -0800

>translate the folder name from UTF-8 to its encoding scheme.
I like that.
UTF-8 to mUTF7 can be given as an informative example.

---

From: Philip Guenther <[email-masked]>
Date: Tue, 21 Mar 2006 13:37:28 -0800

Alexey Melnikov <[email-masked]> writes:
...
>(text moved from the relational extension)

Section 5.7:
   The "header" test evaluates to true if the value of any of the named
   headers, ignoring leading and trailing whitespace, matches any key.

(This was present in rev -04 too)

>what should be put into "Capability arguments" if an extension supports 

Hmm, the template is really aimed towards new tests and actions, while
just about every part of the sieve grammar has been extended in some
way.  Checking the previously published RFCs, we've tended to just
register the name and ignore the "Capability arguments" part.  For
example, the spamtest and virustest registrations in RFC 3685 don't give
the arguments.

My inclination would be to drop the "name" and "arguments" fields.
Opinions?

>Philip Guenther wrote:
...
>comparators draft need to be changed, or this sentence need to be changed.

Sure it is.  Section 9.2.1 of draft-newman-i18n-comparator-08 seems to
concur.

...
>(Chair hat on): The draft needs to address the issue with 

You just mean in deleteheader's matching?  Or are there other places I
don't see?

I'll change the text in deleteheader to refer to the base-spec's
'header' test for details of matching values against patterns.

Philip Guenther

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Mar 2006 07:13:31 -0800

>(This was present in rev -04 too)
Ah, my fault. I've only checked the diff between -03 and -04. I didn't 
realized that you've already addressed the issue.
[...]**

>Sure it is.  Section 9.2.1 of draft-newman-i18n-comparator-08 seems to
In another section it says that no comparator but i;octet should operate 
on octets.

I've already sent a comment to Arnt asking him to allow en;ascii-casemap 
to work on Unicode characters. The advantage of this is that Unicode 
characters outside of ASCII are legal and not going to cause matching 
failure anymore.
I though you agreed to this in one of your messages to the mailing list.

So, are there any objections to change en;ascii-casemap to work on 
Unicode characters?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 22 Mar 2006 16:38:18 +0100

Alexey Melnikov writes:
> I've already sent a comment to Arnt asking him to allow 
> en;ascii-casemap to work on Unicode characters. The advantage of this 
> is that Unicode characters outside of ASCII are legal and not going 
> to cause matching failure anymore.

I felt a little bad about that and sat on it. Call it intuition. And 
indeed I changed my mind later. Now I think the most appropriate policy 
is for ascii-casemap to treat all codepoints it doesn't know as 
non-letters, ie. there's no specific tie to unicode. A UTF-8 octet 
stream encoding a proper word or pure 8-bit junk: No difference to 
ascii-casemap.

Sound good?

Arnt

---

From: Michael Haardt <[email-masked]>
Date: Wed, 22 Mar 2006 16:51:33 +0100

> So, are there any objections to change en;ascii-casemap to work on 

I would appreciate it, because right now the base spec does not allow
to match one arbitrary character.  To begin with, I interpreted 3028 to
talk of unicode characters, not octets, because the text was confusing
and nothing else made sense to me.  I just changed my implementation to
match octets for RFC conformance, after we discussed the topic to death.
It sounded like all other implementations already worked that way before.

Changing the comparator breaks all existing implementations.

If it was just for me, I'd say: So what? It is a change for the better,
having en;ascii-casemap working on octets stinks, using variables
"?" matches a single octet of a multibyte UTF-8 sequence (which is bad
and unexpected to most users) and progress sometimes has to break things.

But I am afraid somebody is going to say something about "compatibility"
and "violating existing standards" and stuff.  I can almost hear it.
Sieve is now widespread and since we could always invent a new comparator,
that works like "en;ascii-casemap" but on characters, perhaps you really
shouldn't listen to me. ;-)

Michael

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 22 Mar 2006 17:06:42 +0100

I wrote:
> As it is, I'm suggesting that 0x80 0x82 is equal to precisely one 
> string, namely 0x80 0x20,

Ack! Typo. 0x80 0x82 both time, of course

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 22 Mar 2006 17:05:20 +0100

Could you please elaborate?

Do you mean that ascii-casemap should consider two octet strings to be 
equal if they are the same after decoding UTF-8? Should it do Unicode 
composition/decomposition?

As it is, I'm suggesting that 0x80 0x82 is equal to precisely one 
string, namely 0x80 0x20, and that independently of whether 0x80 0x82 
is meant to be a character encoded in UTF-8 or something else.

Arnt

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 22 Mar 2006 13:12:07 -0500

>    headers, ignoring leading and trailing whitespace, matches any key.

Now this brings up the question of what the "value of a header" is.  In
RFC2822-speak, it would probably really be the "field body of the named
header fields" .  "header" is used a lot in the draft to mean "header
field" so that's probably not worth going into (much as I would love to
suggest it), but the "value of a header" seems cloudy.  OTOH it's a lot
better than the current rfc3028.  To illustrage (not very good, but clearer):

    The "header" test compares the body of the named headers against the
    specified keys.  For the purpose of this test, The body of a
    header is what remains after removing the header's name and its
    following colon, and then ignoring any leading and trailing whitespace.

Then in:

   If a header listed in the header-names argument exists, it contains
   the empty key ("").  However, if the named header is not present, it
   does not match any key, including the empty key.  So if a message
   contained the header

would be "its body contains" and "its body does not contain"

BTW I notice that in 5.1:

   The address test matches Internet addresses ...

this is the only one of the descriptions in section 5 that doesn't put
the verb in quotes in its opening sentence.  i.e. for consistency's sake
it would be

   The "address" test ..

mm

---

From: Philip Guenther <[email-masked]>
Date: Wed, 22 Mar 2006 10:40:21 -0800

...
>better than the current rfc3028.

Fixing the draft to not use "header" when it means "header field"
wouldn't be hard.  Ditto for aligning with 2822 on "field body".
Does anyone actually think that change would be a _bad_ thing?

...
>the verb in quotes in its opening sentence.  <..>

Fixed.  I've also fixed the 'anyof' and 'allof' tests to match.

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Mar 2006 11:07:55 -0800 (PST)

> Does anyone actually think that change would be a _bad_ thing?

Sure sounds like a good thing to me. However, the material tested by header is
not, precisely speaking, the "body of the named header field". There's also
unfolding and decoding to consider. I suppose we could say "the unfolded and
decoded body of the named header field", although it's a bit of a mouthful.

				Ned

---

From: Michael Haardt <[email-masked]>
Date: Wed, 22 Mar 2006 20:57:10 +0100

On Wed, Mar 22, 2006 at 05:05:20PM +0100, Arnt Gulbrandsen wrote:
> equal if they are the same after decoding UTF-8?

That's what I would like indeed, but as I said, it would break all
implementations and for that reason and because RFC 3028 is interpreted
differently by the majority (read: anybody but me in the first place),
I doubt the change can happen.

> Should it do Unicode 

I don't know what that is, as I don't know much about Unicode at all.
Are you talking about encoding/decoding unicode characters in octets
or something different?

The way you ask implies that there are multiple ways how a unicode
character can be encoded in UTF-8.  Is that correct? I always assumed
UTF-8 was a 1:1 encoding of Unicode.

> is meant to be a character encoded in UTF-8 or something else.

You raise a very interesting point.  How do I match this?

Subject: =?utf-8?q?=80=82?=

That sequence is not valid UTF-8 encoding.  Sieve scripts must be valid
UTF-8, so you can not match the above.  That is fine for comparators
operating on characters, but a true octet-wise comparator will be
crippled.  It has been suggested to encode each of the code points
0x80 and 0x82 in UTF-8 and the comparator had to decode the literal.
Very ugly, and I don't think existing implementations work that way.

As far as I remember, the consensus was: The above is a problem not
worth discussing.  Let Sieve scripts continue to be valid UTF-8, and
let's be unable to match certain values.  If anybody has a problem,
he better invents some kind of "raw" header test.  I agree.

Thus, we can't talk about the octet sequence 0x80 0x82 in Sieve. :-)

But I got distracted.  As long as we consider ":is" or ":contains",
there is no problem really, because both behave the same, no matter
if they decode UTF-8 or just work on octets, as long as we have valid
UTF-8 encoding.  The "?" wildcard is the problem: Should it match an
octet or a character?

Since all (most?) implementations ignore unicode and just work on octets,
"?" matches an octet, too.  Now that is weird to users, because if their
unicode aware client displays a one-character subject, "?" will only match
it if it is an US-ASCII character, and if it is a German umlaut character,
"??" matches it, because those are encoded in two octets.

That's where it becomes obvious if a comparator works on octets or
characters.  RFC 3028 does not specify that as clear as required and
the installed base unfortunately decided for octets, not characters.
The latest 3028bis draft follows that decision, and clearly so.

Of course the comparator specification has the last word on this.  I guess
you see the problem now, and should you decide that "en;ascii-case"
works on characters and "?" matches a unicode character, you had my vote
and I would change my implementation within 5 minutes.  *ducks and hides*

Michael

---

From: Michael Haardt <[email-masked]>
Date: Wed, 22 Mar 2006 21:09:51 +0100

> decoded body of the named header field", although it's a bit of a mouthful.

Insert a reference to RFC 2822 for unfolding and say MIME-decoded with
a reference to RFC 2047 instead of decoded to make it worse, and I will
be happy, too.  Honestly, I am glad for each bit of precision when it
comes to specifications.

Michael

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Mar 2006 12:33:41 -0800 (PST)

> be happy, too.  Honestly, I am glad for each bit of precision when it

WFM

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 22 Mar 2006 23:24:33 +0100

>    commonplace in several programming languages that use globs and

I find the description of the awkwardness of "\\" highly amusing
juxtaposed with the requirement of all tests in Sieve[1] to have the
argument :comparator "i;basic;uca=3.1.1;uv=3.2" (and the matching
"require" statement).  to be honest, I find it absurd that such verbiage
is forced upon users.  we need to find a better way.

[1] outside of the US of A, anyway.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Mar 2006 16:12:07 -0800 (PST)

> "require" statement).

One man's meat is another man's poison... There are plenty of scripts that
depend on comparators continuing to work the way they always have.

  to be honest, I find it absurd that such verbiage

> [1] outside of the US of A, anyway.

This has nothing to do with geography and everything to do with backwards
compatibility. Some of the scripts I'm referring to were written and are used
outside the US.

And good luck using i;basic;uca=3.1.1;uv=3.2 to trap specific sequences of
illegal 8bit in headers. Such stuff is rarely if ever in UTF-8, in my
experience at least.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 23 Mar 2006 06:07:47 +0100

> depend on comparators continuing to work the way they always have.

that doesn't mean we can't find a better way than the above.

I thought Dave Cridland's suggestion to specify matching behaviour in
the comparator itself was intriguing:

http://permalink.gmane.org/gmane.ietf.mta-filters/2689

unfortunately, [draft-newman-i18n-comparator-08] says «the equality test
MUST be reflexive, symmetric and transitive», so "EQUAL" can't be used.
I must admit I don't quite understand how :matches and :regex work with
comparators, though.

another possibility is to have a capability which adds an action which
changes the default comparator to reduce the verbosity.

another possibility is to allow the wildcard comparator, so
that :comparator "*" «[selects] the collation with the broadest scope
(preferably international scope), the most recent table versions and the
greatest number of supported operations.»  (the comparators the server
chooses from would have to be "require"d in advance, I think, although
«require "comparator-*"» is a possibility)

> illegal 8bit in headers. Such stuff is rarely if ever in UTF-8, in my

this is impossible today, isn't it?  how do you specify the string to
compare with?  in any case, if you want to trap raw non-UTF8 in headers,
you should use i;octet.  but then again:

5.7.  Octet Collation

   The i;octet (Section 9.5) collation is only usable with protocols
   based on octet-strings.  Clients and servers MUST NOT use i;octet
   with other protocols.

which would disqualify the use of i;octet with Sieve, since 3028bis says:

   The language is represented in UTF-8, as specified in [UTF-8].

the collation-draft exempts RFC 3028 from this "MUST NOT", but it's not
clear to me that a 3028bis can get the same exemption.  notice that
"represented in UTF-8" only means constant strings can't contain raw
octets which are illegal UTF-8 sequences.  which brings us back to the
discussion on character escapes from a year ago.

http://comments.gmane.org/gmane.ietf.mta-filters/2030

I'd like to suggest we implement (2), but with the extension defined in
the base spec.  note that if the base spec adds the restriction on
extensions that they can't modify other "require" statements, the
analysis of string escapes can be performed statically by a byte
compiler.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Sat, 25 Mar 2006 09:26:25 -0800 (PST)

> that doesn't mean we can't find a better way than the above.

Even assuming we can "find a better way" (we haven't done so thus far) for
future scripts to use, the installed base issue remains.

> I must admit I don't quite understand how :matches and :regex work with

I think of it this way: A comparator has as one of it's components a
normalization operation. Pull that operation out, apply it, and then perform
the glob or regex operation on the result. Note that the output of the
normalization is best seen as a series of nonnegative integers or someting
similar, not octets or characters.

> changes the default comparator to reduce the verbosity.

It makes scripts a bit shorter, but at the expense of having something with
far-reaching impact specified at the top and not where the impact is felt. I'm
far from convinced this is a good tradeoff.

> «require "comparator-*"» is a possibility)

I really don't like this - now you have scripts working in subtlely different 
ways in different places. This is big steps backwards, I think.

> this is impossible today, isn't it?

It is done all the time and the base specification allows it, more or less.

> how do you specify the string to compare with?

You just do it. Nothing in the current sieve specification says that it is an
error to specify material that isn't valid utf-8. (This is now prohibited by
the ABNF in the revised base specification, and IMO this restriction needs to
be removed for string constants. It is at a minimum in direct conflict with the
vacation specification.)

> in any case, if you want to trap raw non-UTF8 in headers,
> you should use i;octet.

Agreed, but in practice scripts often don't specify this.

>  but then again:

>    The language is represented in UTF-8, as specified in [UTF-8].

Represent doesn't mean things are restricted to UTF-8. If it did it would be in
direct conflict with later language in the same document, which quite
specifically allows material in other charsets in strings.

The wording is a bit peculiar (and it needs to be clarified in the revision),
but the intent here was for UTF-8 to always be assumed when it was necessary to
assume a charset. So, in something like a string argument to fileinto, the
argument is to be interpreted in utf-8, never anything else. But in something
like, say, a whole MIME object, you have to be able to specify stuff in other
charsets.

Now, the base specification doesn't come out and say you can use this ability
in a header test. So I suppose it can be argued that it is more or less open as
to how an implementation should behave when this is done, which is why I said
"more or less" previously.

> clear to me that a 3028bis can get the same exemption.

It better or backwards compaatibility goes down the toilet, which is not
acceptable as far as I'm concerned.

> notice that
> octets which are illegal UTF-8 sequences.

I disagree 100% that that is what it means. And since the specification later
talks about putting non-UTF-8 material in constant strings in section 2.4.2.4,
it seems it agrees with me.

> which brings us back to the

> the base spec.

I have no problem with this and would like to see it happen, but it may or may
not be relevant to the matter at hand. Regardless of whether you write some
sequence that's illegal in UTF-8 directly as a series of octets in a string
constant or indirectly using some sort of encoding, you're still presenting the
sieve interpreter with text that isn't in utf-8 at some level. Any attempt to
enforce some sort of rule that nothing but utf-8 can be present is still going
to fail. And any +Unnnn sort of scheme is inherently incapable of representing
UTF-8 anyway.

The ability to write octets using some sort of encoding is certainly helpful
when the script is presented in an editor or otherwise displayed. And the
intent of the resulting script is much much much clearer. That's why I support
the extension, not because it adds a capability that isn't there now.

> note that if the base spec adds the restriction on

Agreed.

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 28 Mar 2006 04:07:20 +0200

On Sat, 2006-03-25 at 09:26 -0800, Ned Freed wrote:

> normalization operation. Pull that operation out, apply it, and then
> perform the glob or regex operation on the result. Note that the
> output of the normalization is best seen as a series of nonnegative
> integers or someting similar, not octets or characters.

thanks.

so you split your match pattern into constant strings without the
wildcards, send each to the comparator which returns a list of start and
end points for each matched constant string, and then see if you can
find a sequence of (start, end) for each constant string where the
intervals between the constant strings match the wildcards.  *phew*

that will work for :matches (although I doubt anyone will actually
implement it that way -- so the pluggable comparator idea goes out the
window), but not for :regex.  unless I'm missing something again :-)

> It makes scripts a bit shorter, but at the expense of having something
> with far-reaching impact specified at the top and not where the impact
> is felt. I'm far from convinced this is a good tradeoff.

what do you mean by impact?  server load?

> I really don't like this - now you have scripts working in subtlely
> different ways in different places. This is big steps backwards, I
> think.

you can reduce the problem by not doing require on more comparators than
you need.  e.g., if you do

  require "comparator-i;basic;uca=3.1.1;uv=3.2";
  require "comparator-i;ascii-numeric";
  require "comparator-*";

it's quite obvious which comparator has the «broadest international
scope» (just to make it clear, this is the wording about the behaviour
of "*" from [COLLATION]).

> It is done all the time and the base specification allows it, more or

well, that's in contention.

> You just do it. Nothing in the current sieve specification says that
> it is an error to specify material that isn't valid utf-8.

8.1.     Lexical Tokens

   Sieve scripts are encoded in UTF-8.  The following assumes a valid
   UTF-8 encoding; special characters in Sieve scripts are all ASCII.

"are" is considered equivalent with a MUST, isn't it?

> (This is now prohibited by the ABNF in the revised base specification,
> and IMO this restriction needs to be removed for string constants.

I strongly disagree.  let's not make a charset soup on purpose.

> It is at a minimum in direct conflict with the vacation
> specification.)

not relevant, since vacation hasn't left draft status.

> Agreed, but in practice scripts often don't specify this.

such scripts are broken and rely on undocumented behaviour IMO.

> Represent doesn't mean things are restricted to UTF-8.

yes, this is fine.  the message can contain octet strings even if the
script itself can not.  sorry about the red herring.

> If it did it would be in direct conflict with later language in the
> same document, which quite specifically allows material in other
> charsets in strings.

what language are you thinking of?

> It better or backwards compatibility goes down the toilet, which is
> not acceptable as far as I'm concerned.

how about implementations where "?" matches a Unicode character even for
the default comparator?  which implementation gets to define backwards
compatibility?

> I disagree 100% that that is what it means. And since the
> specification later talks about putting non-UTF-8 material in constant
> strings in section 2.4.2.4, it seems it agrees with me.

it's not clear that 2.4.2.4 expects the Sieve implementation to decode
the embedded MIME document into raw octets.  I would expect the
opposite, that the MIME headers and encoded data supplied in the script
are copied verbatim.

> I have no problem with this and would like to see it happen,

I'm glad to hear it.  I guess I should submit text for consideration :-)

> Regardless of whether you write some sequence that's illegal in UTF-8
> directly as a series of octets in a string constant or indirectly
> using some sort of encoding, you're still presenting the sieve
> interpreter with text that isn't in utf-8 at some level.

the Sieve interpreter needs to handle non-UTF-8 already. that is, header
bodies which are encoded in an unknown charset must be handled as
octet-strings.

looking at the current draft, I think this excerpt from 2.7.2 needs a
little tweak:

   If implementations fail to support the above behavior, they MUST
   conform to the following:

      No two strings can be considered equal if one contains octets
      greater than 127.

(this is unchanged since 3028).  I believe it should be allowed to
compare such strings using i;octet.  I don't have suitable replacement
text, though.

> Any attempt to enforce some sort of rule that nothing but utf-8 can be
> present is still going to fail. And any +Unnnn sort of scheme is
> inherently incapable of representing UTF-8 anyway.

well, yes, it represents a Unicode code point, not an octet sequence.
but as we have established, with the correct comparator, you can use "?"
to see that the *client visible* representation of Sieve strings is in
fact UTF-8.
-- 
Kjetil T.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 28 Mar 2006 08:20:23 +0100

> window), but not for :regex.  unless I'm missing something again :-)
I think we have to document reality here. The reality seems to be 
that not only do comparators have a normalization operation, but they 
also have a match operation. I *suspect* that the matching function 
has '?', '%', and '*' wildcards, although only two are used in Sieve.

I'd like to float the notion that somebody writes text suitable to 
document this at the comparator level.

We'd want to define this such that existing usage is captured, and 
sensible definitions for i;basic* apply, such that '?' wildcard 
behaviour works as expected.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw