Subject: base-spec issue #1: character escapes

From: Philip Guenther <[email-masked]>
Date: Tue, 08 Mar 2005 07:50:04 -0800

Per discussion during the meeting yesterday, I'm fleshing out the
options on how to provide character escapes here on the list:

0) do nothing; just repeat that \x means x, period.
   PRO: - no changes to the spec
	- breaks no implementations...unless they don't conform already
   CON: - scripts have no way to get strings that contain NUL or
	  invalid UTF-8

1) change the base spec to define \xFF, \uXXXX, etc
   PRO: - simplest to specify that adds the support
	- support is easy and consistent; no need to change string
	  interpreters in mid-stream
   CON: - breaks all implementations
	- breaks scripts that use superfluous backslashes
	- escapes only usable in quoted strings
	- scripts that need escapes can't guarantee they're getting them
	  (scripts would not be portable between versions)

2) change the base spec to say the \x maps to x unless overriden
   by an extension; extensions may redefine any \x except \\ and
   \".  Scripts SHOULD NOT contain extraneous escapes.  Then, create
   an extension which defines \xFF, \uXXXX, etc
   PRO: - neither implementations nor scripts broken by the change
	- script that needs escapes is guaranteed they're getting them
	  if they're supported
	- implementation similar to variable (or is that a CON?)
   CON: - more complicated to specify
	- another extension has to be defined and used when needed
	- escapes only usable in quoted strings
	- does there need to be a registry for the redefinitions
	  to prevent conflicts between such extensions?

3) define an extension to variables that implicitly creates variables
   (in a namespace) for each unicode codepoint and octet value whose
   values are the name codepoints/octets (e.g., ${unicode.00bf}
   would contain the UTF-8 representation of U+00BF (inverted
   question mark); ${octet.ff} would be the octet with value
   255, which is not valid UTF-8)
   PRO: - neither implementations nor scripts broken by the change
	- script that needs escapes is guaranteed they're getting them
	  if they're supported
	- usable in both quoted strings and multiline literal
	- avoids introducing another area of extension (c.f. last
	  CON of (2))
   CON: - more complicated to specify
	- more annoying/noisy to use
	- another extension has to be defined and used when needed
	- requires support for and use of variables

4) define an extension that covers all the changes in the base spec
   that are incompatible with RFC 3028.  Option (1) would be done
   under that extension.  If there are no other incompatible changes
   then this reduces to (2)
   PRO: - neither implementations nor scripts broken by the change
	- script that needs escapes is guaranteed they're getting them
	  if they're supported
   CON: - medium complexity?
	- another extension has to be defined and used when needed
	- escapes only usable in quoted strings
	- negative experience with version numbers in IMAP
	- no longer revising Sieve base spec but rather defining Sieve v2

Are there other options?  Did I miss (or misstate) any PROs or CONs?
Which of these PROs and CONs should be considered important and why?

Philip Guenther
editor

---

From: Philip Guenther <[email-masked]>
Date: Tue, 08 Mar 2005 08:43:09 -0800

Philip Guenther <[email-masked]> writes:
...
>   values are the name codepoints/octets

With my editor hat off now...

I dislike (4) the most.  Versioning the base spec like that is too
open-ended in my view.  Servers would have to continue to support
the old behavior as well as the updated/fixed behavior.  Are we
here to clarify the base-spec or create Sieve v2?

As much as the simplicity of (1) is tempting, breaking backward
compatibility just because the tweak was easy when there's a
straightforward means to do it via an extension is a bad policy.
This would be neither a clarification nor the removal of an ambiguity.
If it's still an acceptable change, then what's the line for just
rolling changes into the base spec?

Oh, and (1) shouldn't be that complex to implement for anyone who
has already done variables, as they already have the code to change
the interpretation of strings based on the use of an extension.

While (3) is a cleaner design, introducing a dependency on variable
create the possibility that some implmentation that doesn't want
to implement variables but needs access to NUL, etc will do (1) or
(2) instead, thus defeating the goal of interoperability.  If that
is a plausible risk, then (2) or (1) should be picked from the
start.

I therefore would prefer (2).

Philip Guenther

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 8 Mar 2005 12:02:27 -0500 (EST)

> here to clarify the base-spec or create Sieve v2?

I agree.  The changes aren't substantial enough to warrant this (even 
allowing tests to have side effects wouldn't require this).

> This would be neither a clarification nor the removal of an ambiguity.

Also agree.

> the interpretation of strings based on the use of an extension.

I think you mean (2) here.

> I therefore would prefer (2).

Same here.

-Rob

-------------------------------------------------------------------------
Rob Siemborski        |     PGP:0x5CE32FCC     |      [email-masked] 
-----BEGIN GEEK CODE BLOCK----
Version: 3.12
GCS/IT/CM/PA d- s+: a-- C++++$ ULS++++$ P+++$ L+++ E W+ N(-) o? K- w-- O-
M-- V-- PS+ PE+ Y+ PGP+ t+@ 5+++ X- R@ tv-- b+ DI+++ D++ G e++ h+ r- y?
------END GEEK CODE BLOCK-----

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 08 Mar 2005 12:28:23 -0500

Hi Rob,

>> I therefore would prefer (2).

With (2), do we even need to make the statement that an extension will 
override the base spec behaviour? After all, that is what extensions 
implicitly do. i.e. why can't we leave it exactly as it is now in the base 
spec, and an extension written independently of that.

-- 
Cyrus Daboo

---

From: Tony Hansen <[email-masked]>
Date: Tue, 08 Mar 2005 12:28:38 -0500

I came to the same conclusion you did, for almost exactly the same reasons.

This may be stating the obvious, but I'd go further and say that any 
such extension should ONLY define exactly what \x character sequences 
are overridden, and leave all other \x character sequences alone. This 
would potentially allow multiple such extensions to coexist.

I'm not sure I follow this CON that you list:
	- escapes only usable in quoted strings
Why can't the extension also be usable in multi-line strings?

As for this question:

I'd say "no". The published RFC spec should be a sufficient registration.

	Tony Hansen
	[email-masked]

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 8 Mar 2005 14:00:07 -0500 (EST)

> spec, and an extension written independently of that.

I think that we should definately add that scripts SHOULD NOT use 
extranious escapes.

But yes, I don't think we realy need to do anything beyond that in the 
base spec, though (mentioning that an extension can explicitly override 
this particular fact is probably unnecessary, unless we define the 
extension in the base document as well).

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 8 Mar 2005 15:34:11 -0500

> spec, and an extension written independently of that.

I think of the backslash character escapes in quoted strings as
something processed at the lexical stage.  That is, a backslash sequence
inside a quoted string is an instruction to the lexer that affects the
way the quoted string is internalized.  Some of the comments in this
thread have suggested that these escape sequences are handled in some
other way.  I suppose that's a valid implementation decision (though I
haven't though about it much: I hadn't really considered that this was
other than a lexical function).

If you're saying that an extension can change the way these escapes are
interpreted, then you are saying that "require" can change the lexical
input process (at least for implementations that process the quoted
strings in this way).  I don't think that's a good thing.  The way
extensions are specified allows "require" to be processed as a
runtime directive (unless I'm all wet on that).

By the time you get to where an extension can process other backslash
sequences, the sequences have already been processed.  So where the
lexer/tokenizer has already internalized "\a" as "a" -- what's an
extension to do?

With this interpretation, the only way to change the way that
single-backslash-style escapes in quoted strings are scanned is to do
the change at the base spec level.  This does indeed bring up
backward-compatibility issues.  One way of addressing that is to have
some kind of base-spec-version marker in the script.

Collaterally, being able to have an extension change the interpretation
of strings at run time would require some other way of embedding the
desired information, other than using a single backslash escape,
that is.  Double-backslash sequences would be ugly.

Another alternative would be to have a way to set a "variable" to
a particular code sequence.  Then you could include a "${var}"
reference to include that code.  I don't know if this is better or
worse than a namespace extension, though..  

Yours,
-mm-

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 08 Mar 2005 22:26:13 +0100

> runtime directive (unless I'm all wet on that).

depends on what you mean by "runtime directive".

   The require command, if present, MUST be used before anything other
   than a require can be used.  An error occurs if a require appears
   after a command other than require.

speaking of which, I think it would be a good idea to rule out that
require can affect other require statements in the base specification.
this is stated explicitly in the variables draft, but the same thing
will be a potential ambiguity with character escapes, e.g.,

  require "quotes";
  require "r\x65gex";

-- 
Kjetil T.

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 8 Mar 2005 16:33:28 -0500 (EST)

> will be a potential ambiguity with character escapes, e.g.,

This seems very sane, safe, and backwards compatible.

-Rob

-----BEGIN GEEK CODE BLOCK----
Version: 3.12
------END GEEK CODE BLOCK-----

---

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 10 Mar 2005 16:17:37 -0500

> depends on what you mean by "runtime directive".

Something that is processed at execution time as opposed to compile time
or, say, as a preprocessor directive outside of the compile/execution
process (which would be the place to affect lexical interpretation).

>    after a command other than require.

Yep.  Speaks to the ordering but not the processing (much like
the fact that "else" can only follow an "if" or "elsif").

(Personally I would be happier if the ordering rule for "require"
were relaxed.)

mm

---

From: [email-masked]
Date: Sun, 13 Mar 2005 10:39:08 -0800 (PST)

> I therefore would prefer (2).

I agree with this analysis and conclusion. Note that allowing (2) doesn't
preclude (3).

				Ned