Subject: Implementing encoded-character

From: Michael Haardt <[email-masked]>
Date: Wed, 04 Apr 2007 10:11:44 +0200

Hello,

Exim will implement encoded-character RSN, but I wonder if I understood
things correctly:

"$${hex:40}" -> "$@"

A dollar does not have to be quoted, does it?

"${hex:40" -> "${hex:40"
"${hex: 40 }" -> "${hex: 40 }"
"${hex:400}" -> "${hex:400}"
"${unicode:40}" -> "${unicode:40}"
"${unicode:1000000}" -> "${unicode:1000000}"

There is no word of the encoded-character grammar inside the string,
taking everything literal.

"${hex:40${hex:40}}" -> "${hex:40$}

It looks nested, but actually it's just some junk around a word of the
grammar.

"${unicode:020000}" -> error

Unicode range violation.

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 04 Apr 2007 10:51:52 +0200

On Wed, 2007-04-04 at 10:11 +0200, Michael Haardt wrote:
> Exim will implement encoded-character RSN,

excellent!

> but I wonder if I understood things correctly:
> 
> "$${hex:40}" -> "$@"

yes.

> A dollar does not have to be quoted, does it?

right, only for very unlikely sequences.

> "${hex: 40 }" -> "${hex: 40 }"

yes.  whitespace is only allowed between hex-pairs.  btw, how do you
feel about allowing CRLF as well as SPC and TAB between hex-pairs?

> "${hex:400}" -> "${hex:400}"

yes, hex-pair can't be three digits.

> "${unicode:40}" -> "${unicode:40}"

no, this is "@".

> "${unicode:1000000}" -> "${unicode:1000000}"

yes.

> taking everything literal.

I don't understand this statement.

> "${hex:40${hex:40}}" -> "${hex:40$}

no, "${hex:40@}"

> "${unicode:020000}" -> error

no, U+20000 is inside the Unicode range.  ${unicode:0020000} fails due
to not matching unicode-hex (too many digits), ${unicode:200000} fails
due to being outside the Unicode range.

-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Wed, 04 Apr 2007 11:23:22 +0200

> feel about allowing CRLF as well as SPC and TAB between hex-pairs?

Is CRLF allowed inside other ${} expressions (variables)?

> > "${unicode:40}" -> "${unicode:40}"

Good thing I asked.  I just reread RFC 2234 and found out that I
have to read 1*6HEXDIG as 1*6(HEXDIG), not as 1*(6HEXDIG).

> I don't understand this statement.

The grammar matches words inside the character sequence that makes up a
string.  No matter how much of a word is matched, if it is not complete,
it will be taken as the literal character sequence.  That means you
need infinite look-ahead.

Before looking at it, I expected that if ${hex: is found, it would be
an error if it were not followed by arguments and a closing brace.

> > "${hex:40${hex:40}}" -> "${hex:40$}

Oops, I meant to write @.  But you agree on my interpretation how things
are processed.

> due to being outside the Unicode range.

My mistake, again.  So we have:

"${unicode:200000}" -> error
"${unicode:2000000}" -> "${unicode:2000000}"

I don't particularly like that, because most likely the second was
never meant that way.  Is there any way to change that at this point?

Michael

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Wed, 04 Apr 2007 13:38:18 +0200

> Is CRLF allowed inside other ${} expressions (variables)?

variables doesn't allow any whitespace at all.

> an error if it were not followed by arguments and a closing brace.

well, you don't need to backtrack much:

  ${unicode:cafe ab ab ab ab ab ab
     ab ab add ${hex:40 41}}

you just go along, and as soon as you find a syntax "error", you bail
and copy what you've buffered so far verbatim (in this case,
"${unicode: ... add "), then restart the state machine.  worst case, the
buffering is the size of the script plus storage for the decoded Unicode
characters while parsing the script.

> So we have:

you want to change unicode-hex to 1*HEXDIG instead?  the wording should
already handle it, so it's just the ABNF which needs a tweak.  that's
fine with me.  I think it's Philip's call.

-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Wed, 04 Apr 2007 14:25:45 +0200

> variables doesn't allow any whitespace at all.

Hmm, right, variables contain no arguments and we don't have functions
yet.  Thinking about string expressions, I certainly would like to
have CRLF as white space, but I also would like embedded comments in
that case.  Just looking at encoded-character, I see no need for CRLF
and even have an odd feeling with, but considering it as syntactic
prototype for string expressions, both CRLF and comments sound useful.

> well, you don't need to backtrack much:

It's no problem really, just confusing.  If someone starts to write
${hex:, most likely he meant to encode data.  Only CS people think stuff
like "it's not a word of the grammar, thus of course being an literal
as specified". ;-)

> fine with me.  I think it's Philip's call.

Yes, that would be more logical.  I consider ${hex: and ${unicode: as
functions of constant arguments.  No matter which argument is passed
to them: Syntax errors (like a missing brace) should cause an error,
and semantic errors like range overflows should cause an error, too.
It's bizarre to see 0x200000 being an overflow, but 0x2000000 causing
everything to be taken literally.

Michael

---

From: Aaron Stone <[email-masked]>
Date: Thu, 05 Apr 2007 00:20:01 -0700

> prototype for string expressions, both CRLF and comments sound useful.

I kind of like the idea of things that look like variables but are
functions operating on the right side of the colon.

We had a bit of discussion in Prague about list expansions that access
external data sources. This would certainly be one way to handle it,
though we'd have to be careful about strict vs. lazy evaluation. Anyhow,
that should probably be the subject of a separate thread.

> like "it's not a word of the grammar, thus of course being an literal

I agree, it'd be confusing for that to happen.

[snip]

> > > "${unicode:2000000}" -> "${unicode:2000000}"

Ugh, if it looks like encoded-char and walks like encoded-char...

My test implementation left-shifts the current value of the encoded
character, then adds the next hex digit. When it hits whitespace, it
checks if the value is within appropriate bounds; if so, stores the
character then loops, if not, stores '?' then loops. Would we really
rather be very strict about this? I'm in favor of some flexibility.

Aaron

---

From: Michael Haardt <[email-masked]>
Date: Thu, 05 Apr 2007 09:49:29 +0200

> that should probably be the subject of a separate thread.

I suggest to look at Exim and the Exim filter, and their string
expressions, as a live and working example that's very similar.

> Ugh, if it looks like encoded-char and walks like encoded-char...

That's the point.

> rather be very strict about this? I'm in favor of some flexibility.

Given you use C and unsigned integers, or signed ones on a common
architecture where overflows are ignored, you already have a problem.
You could stop aggregating after the 6th nibble, simply parsing more,
and then generate an overflow error if there are really more.

I suggest to make the specification a bit more flexible and have
implementations obey it strictly.

Michael

---

From: Ned Freed <[email-masked]>
Date: Thu, 05 Apr 2007 10:40:11 -0700 (PDT)

> rather be very strict about this? I'm in favor of some flexibility.

You need to strictly implement the grammar in the specificaiton, whatever
that ends up being. Any flexibility will allow someone to write one of
these things that works in your implementation but silently fails and causes
wierd results elsewhere.

Past experience with RFC 2047 encoded-words has shown that allowing leeway in
this situations is a curse, not a blessing.

					ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 05 Apr 2007 10:25:13 -0700 (PDT)

> that case.

I would like to allow CRLF but comments IMO go way too far.

The argument for allowing CRLF is that it is really needed to allow reasonable
formatting of long runs of hex-encoded stuff. If, say, you want to write a few
hundred bytes worth of material, with the current proposal you either have to
put it all on one line, hope you can find a CRLF in there that you can leave
unencoded and thus create a line break (unlikely), or use a series of set
actions to build up the string piecemeal (ugly and requires variables support).

The same cannot be said of comments - you are free to put one in front of the
string or at the end and end up with something that's readable. Of course I
suppose you could argue that there are cases where it is clearer to have the
comment in the middle, but I have to say I find that to be a fairly contrived
case.

  Just looking at encoded-character, I see no need for CRLF

> everything to be taken literally.

I'm not wild about changing from 1*6 to 1* but I can live with it if need be.
That said, I completely disagree with your assessment of when it is appropriate
to generate an error. These "overlay" syntaxes always have a tension between
syntactic generality and potential collision with regular strings people might
want to use - the more general you make your syntax the more likely you are to
collide with some legitimate regular string. So, while I have no major issue
with allowing 1*HEXDIG, only to call overly long strings an error, I have a
major problem with allowing something like "${hex:" or even "${hex" or "${" to
match and then generate an error.

				Ned

---

From: "Aaron Stone" <[email-masked]>
Date: Thu, 5 Apr 2007 21:10:09 -0000

> this situations is a curse, not a blessing.

Indeed, point taken!

Aaron

---

From: Aaron Stone <[email-masked]>
Date: Mon, 09 Apr 2007 17:27:41 -0700

> > this situations is a curse, not a blessing.

It's not strict yet (I'll cross that bridge when we agree on where it is ;-),
it just translates the hex values to utf-8. And now, counting from 0-9 in
Western Arabic, Eastern Arabic and Amharic (thanks unicode.org/charts!)...

Converting [${unicode:30 31 32 33 34 35 36 37 38 39}]
        to [0123456789] length 11

Converting [${unicode:06f0 06f1 06f2 06f3 06f4 06f5 06f6 06f7 06f8 06f9}]
        to [۰۱۲۳۴۵۶۷۸۹] length 21

Converting [${unicode:1369 136a 136b 136c 136d 136e 136f 1370 1371 1372}]
        to [፩፪፫፬፭፮፯፰፱፲] length 31

(Are there any number systems up in the four bytes per symbol ranges?)

If anybody would like to use my code, I'd be happy to make it available
without restriction. It's all of 100 lines, and most of the fun was
generating utf-8 by hand.

Aaron

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 10 Apr 2007 18:08:45 +0100

>I would like to allow CRLF but comments IMO go way too far.
 From my reading of the mailing list it sounds like there is consensus 
to do the following change:

OLD:
   encoded-arb-octets   = "${hex:" hex-pair-seq "}"
   hex-pair-seq         = hex-pair *(WSP hex-pair)
                                     ^^^
NEW:
   encoded-arb-octets   = "${hex:" hex-pair-seq "}"
   hex-pair-seq         = hex-pair *(LWSP hex-pair)
                                     ^^^^

OLD:
   encoded-unicode-char = "${unicode:" unicode-hex-seq "}"
   unicode-hex-seq      = unicode-hex *(WSP unicode-hex)
                                        ^^^
NEW:
   encoded-unicode-char = "${unicode:" unicode-hex-seq "}"
   unicode-hex-seq      = unicode-hex *(LWSP unicode-hex)
                                        ^^^
   unicode-hex          = 1*6HEXDIG

And this needs to be updated if people want to allow for trailing LSWP 
(before the closing "}")

>comment in the middle, but I have to say I find that to be a fairly contrived
I don't think we have consensus to allow for comments.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 10 Apr 2007 18:12:06 +0100

>I'm not wild about changing from 1*6 to 1* but I can live with it if need be.  
Is there any consensus for the following change

OLD:
   unicode-hex          = 1*6HEXDIG
                          ^^^

NEW:
   unicode-hex          = 1*HEXDIG
                          ^^
?

(And any sequence of more than 6 HEXDIG will cause error)

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 10 Apr 2007 20:13:24 +0200

On Tue, 2007-04-10 at 18:08 +0100, Alexey Melnikov wrote:
> to do the following change: [...]

> And this needs to be updated if people want to allow for trailing LSWP 

if trailing LWSP is allowed, leading LWSP should be allowed, too.  I
don't mind making such a change.  I don't mind making the other changes,
either (including 1*HEXDIG).

> I don't think we have consensus to allow for comments.

I think we have consensus for not allowing them.
-- 
Kjetil T.

---

From: Michael Haardt <[email-masked]>
Date: Tue, 10 Apr 2007 20:55:16 +0200

> either (including 1*HEXDIG).

Trailing and leading white space would be great indeed, because together
with LWSP you could encode a regular hexdump.

> I think we have consensus for not allowing them.

Just looking at ${hex:...} and ${unicode:...}, I really don't need them,
but should that syntax (enriched by something to split arguments) be
used for functions, I disagree with Ned on not needing comments for that.
I did use comments to comment arguments given to functions in the past,
but I admit it's like once every few years.

But... I don't insist on comments.  Go ahead with LWSP and without comments.

Alexey: Could you please include an example that shows "${hex:" means
literally that and it is not an error? We can discuss that being right
or wrong, but at this time, everybody but me considers is being right
and that needs to be documented more clearly.

Btw: If you want to encode "${hex:00}" literally, you need to write
"${hex:24 7b 68 65 78 3a 30 30 7d}".  "\$\{\h\e\x\:\0\0\}" is a string
with one NUL byte, plus lacking the good reason to abuse \ asked for
by the SHOULD NOT in section 2.4.2.  I suppose everybody already knew
that, because the spec says, expansion happens after scanning a string.
If it surprises anybody, please speak up.

Hmm. 2.4.2 says further:

  NUL (US-ASCII 0) is not allowed in strings.

How about:

  An unencoded NUL (US-ASCII 0) is not allowed in strings, see section
  2.4.2.4 for encoded characters.

Michael

---

From: "Aaron Stone" <[email-masked]>
Date: Tue, 10 Apr 2007 19:12:24 -0000

> either (including 1*HEXDIG).

Something like this:

   encoded-character    = "${" encoded-char-scheme ":" encoded-char-seq
"}"
   encoded-char-scheme  = hex / unicode
   encoded-char-seq     = *(LWSP WSP 1*HEXDIG) LWSP

Note that LWSP is optional by definition, so we have to include SP or WSP
to force some kind of separator between 1*HEXDIG's. Note that this is not
valid according to the syntax above,

${unicode:
123
ABC
}

..because 123 and ABC do not have WSP between them. Use WSP / CR / LF? Is
there some variant of LWSP that mandates at least one character of
something be present?

I think there are three options for values that are out of range:

 1. Throw an error and reject the script.
 2. Ignore the offending value.
 3. Insert some placeholder like ' ' or '?'.

I concur that comments should not be allowed.

Aaron

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 10 Apr 2007 21:24:21 +0200

On Tue, 2007-04-10 at 20:55 +0200, Michael Haardt wrote:
> "${hex:24 7b 68 65 78 3a 30 30 7d}".

"${hex:24}{hex:00}" is shorter and IMHO clearer.

>   2.4.2.4 for encoded characters.

I think that clarification is useful.  "Literal NUL ..." works, too.
-- 
Kjetil T.

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 10 Apr 2007 21:51:41 +0200

>    encoded-char-seq     = *(LWSP WSP 1*HEXDIG) LWSP

if we allow ${hex:100} in the grammar, we need to say something in the
text about the valid range.  I would prefer to stick to separate
productions for encoded-arb-octets and encoded-unicode-char to keep the
text simple and to minimise the change to the text.

> Note that LWSP is optional by definition,

ouch, good catch!

> so we have to include SP or WSP

LWSP requires WSP after CRLF, too, so it's simply not what we want, we
need to add another basic terminal, perhaps

   blank = WSP / CRLF

I suggest we stick to the poll question from Alexey, but with "1*blank"
replacing LWSP in his suggested new text.

>  3. Insert some placeholder like ' ' or '?'.

I don't think we need to revisit this question.

-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Wed, 11 Apr 2007 15:44:26 -0400

> > Note that LWSP is optional by definition,

very.

> replacing LWSP in his suggested new text.

That looks right (name doesn't have to be "blank" but whatever).

> > I concur that comments should not be allowed.

ditto

mm

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 12 Apr 2007 20:20:28 +0100

> (And any sequence of more than 6 HEXDIG will cause error)

It looks to me that we've reached rough consensus in favor of this change.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 12 Apr 2007 20:30:28 +0100

> And this needs to be updated if people want to allow for trailing LSWP 

The update proposal is like this:

OLD:
  encoded-arb-octets   = "${hex:" hex-pair-seq "}"
  hex-pair-seq         = hex-pair *(WSP hex-pair)
                                    ^^^
NEW:
  blank                = WSP / CRLF
  encoded-arb-octets   = "${hex:" hex-pair-seq "}"
  hex-pair-seq         = *blank hex-pair *(1*blank hex-pair) *blank
                         ^^^^^^            ^^^^^^^           ^^^^^^

OLD:
  encoded-unicode-char = "${unicode:" unicode-hex-seq "}"
  unicode-hex-seq      = unicode-hex *(WSP unicode-hex)
                                       ^^^
NEW:
  encoded-unicode-char = "${unicode:" unicode-hex-seq "}"
  unicode-hex-seq      = *blank unicode-hex *(1*blank unicode-hex) *blank
                         ^^^^^^               ^^^^^^^              ^^^^^^
  unicode-hex          = 1*6HEXDIG

Does this look right?

Note that people have suggested some other editorial changes in the same 
section, e.g. extra examples. I will post a message on them separately. 
For now I want to get closure on the proposed syntax.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 12 Apr 2007 20:39:52 +0100

>text simple and to minimise the change to the text.
Speaking as a contributor: I agree.

>>Note that LWSP is optional by definition,
Indeed.

---

From: Michael Haardt <[email-masked]>
Date: Thu, 12 Apr 2007 22:08:58 +0200

>   hex-pair-seq         = *blank hex-pair *(1*blank hex-pair) *blank

The white space looks fine, but you missed defining hex-pair.

>   unicode-hex-seq      = *blank unicode-hex *(1*blank unicode-hex) *blank

You did define unicode-hex here, but unchanged, not as 1*HEXDIG.  How
about that?

  blank                 = WSP / CRLF
  encoded-arb-octets    = "${hex:" *blank hex-seq *blank "}"
  encoded-unicode-chars = "${unicode:" *blank hex-seq *blank "}"
  hex-seq               = hex *(1*blank hex)
  hex                   = 1*HEXDIG

IMHO, the leading and trailing white space is not part of the sequence.
If you think different, then put it back into hex-seq.  If we omit
the length restriction, we may as well merge the sequences into one
production.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 12 Apr 2007 21:36:06 +0100

>The white space looks fine, but you missed defining hex-pair.
No, it just stays as it is defined now. That is why I haven't listed it.

>You did define unicode-hex here, but unchanged, not as 1*HEXDIG.
>
Ah, sorry about that. Yes, I meant "1*HEXDIG".

>How
>If you think different, then put it back into hex-seq.
>
I think they are part of the hex-seq, but this is not something to argue 
about.

>If we omit the length restriction, we may as well merge the sequences into one
>production.
>  
>
I think we've only agreed to remove length restriction on "${unicode:".

BTW, are "${hex:"/"£{unicode:" prefixes case insensitive? I say yes.

---

From: Michael Haardt <[email-masked]>
Date: Fri, 13 Apr 2007 09:20:10 +0200

> I think they are part of the hex-seq, but this is not something to argue 

Indeed, it's not the same grammar, but still the same language. :)

> I think we've only agreed to remove length restriction on "${unicode:".

Oops.  Well, with hex I don't mind pairs, in case anybody here
likes them more.

> BTW, are "${hex:"/"£{unicode:" prefixes case insensitive? I say yes.

According to the grammar, they are.  Given that most everything in
sieve is case insensitive, it makes sense, too.

Michael

---

From: Philip Guenther <[email-masked]>
Date: Fri, 13 Apr 2007 10:54:11 -0600

> On Thu, Apr 12, 2007 at 09:36:06PM +0100, Alexey Melnikov wrote:
...
> sieve is case insensitive, it makes sense, too.

Yes, definitely.

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Sun, 15 Apr 2007 13:17:48 -0700 (PDT)

> Does this look right?

Modulo the change to make unicode-hex to 1*HEXDIG, yes.

> For now I want to get closure on the proposed syntax.

I'm find with this. I do note, however, that error handling here is kinda
tricky - since you cannot be sure until you reach the } that the syntax
rules are met, you have to flag any errors you see in unicode values and
only report them at the end. Perhaps a note about this would be in order...

				Ned

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 16 Apr 2007 03:11:11 +0200

On Sun, 2007-04-15 at 13:17 -0700, Ned Freed wrote:
> I'm fine with this. I do note, however, that error handling here is kinda

yes, I guess it's easy to trip over this, causing a potential
interoperability issue.  suggested text:

        It is an error for a script to use a hexadecimal value that
        isn't in either the range 0 to D7FF or the range E000 to
        10FFFF.[...]

        Note: Implementations MUST NOT raise an error for an out of
        range Unicode value unless the sequence containing it is
        well-formed according to the grammar.

-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Sun, 15 Apr 2007 19:44:54 -0700 (PDT)

>         well-formed according to the grammar.

Nice wording! I think thie definitely belongs in the specification.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 16 Apr 2007 20:12:33 +0100

Here is the collected list of changes to section 2.4.2.4:

3rd paragraph:

OLD:
 hex-pair             = 1*2HEXDIG

NEW:
 blank                = WSP / CRLF
 hex-pair             = 1*2HEXDIG

5th paragraph:

OLD:

Insert after the following paragraph:
   It is an error for a script to use a hexadecimal value that isn't in
   either the range 0 to D7FF or the range E000 to 10FFFF.  (The range
   D800 to DFFF is excluded as those character numbers are only used as
   part of the UTF-16 encoding form and are not applicable to the UTF-8
   encoding that the syntax here represents.)

a new paragraph:

8th paragraph:
OLD:
  In the following script, message A is discarded, since the specified
                                   ^
  test string is equivalent to "$$$".
NEW:
  In the following script, message B is discarded, since the specified
                                   ^
  test string is equivalent to "$$$".

And add the following paragraph to the end of the section:

  The following examples demonstrate valid and invalid encodings
  and how they are handled:
    "$${hex:40}" -> "$@"
    "${hex: 40 }" -> "@"
    "${HEX: 40}" -> "@"
    "${hex:40" -> "${hex:40"
    "${hex:400}" -> "${hex:400}"
    "${hex:40${hex:40}}" -> "${hex:40@}"
    "${unicode:40}" -> "@"
    "${ unicode:40}" -> "${ unicode:40}"
    "${UNICODE:40}" -> "@"
    "${UnICoDE:0000040}" -> "@"
    "${Unicode:40}" -> "@"
    "${Unicode:Cool}" -> "${Unicode:Cool}"
    "${unicode:1000000}" -> error
    "${unicode:200000}"  -> error
    "${Unicode:DF01}     -> error

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 16 Apr 2007 20:15:31 +0100

>    "${Unicode:DF01}     -> error

And I've missed Michael's favorite:

   "${hex:" -> "${hex:"

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 17 Apr 2007 01:32:27 +0200

> >  and how they are handled:

the list is quite long, I feel we can demonstrate more than one issue in
each example.

> >    "${HEX: 40}" -> "@"

the second is superfluous.

> >    "${hex:400}" -> "${hex:400}"

these are good.

> >    "${hex:40${hex:40}}" -> "${hex:40@}"

"${hex:40${hex:30}}" -> "${hex:400}"

would demonstrate that only one pass is done, too.

> >    "${ unicode:40}" -> "${ unicode:40}"

if we include the last example, the first in this set is superfluous
IMHO.  I'd like one of the examples to include backslash quoting, we
could extend that example, e.g., "\${UNICODE:40}" -> "@"

> >    "${UnICoDE:0000040}" -> "@"

fine.

> >    "${Unicode:40}" -> "@"

unnecessary, IMHO.

> >    "${Unicode:DF01}     -> error

ok.

> And I've missed Michael's favorite:

I think the fourth example from the top covers this already, that is
"${hex:40".
-- 
Kjetil T.

---

From: Aaron Stone <[email-masked]>
Date: Tue, 17 Apr 2007 00:54:13 -0700

> > >    "${Unicode:Cool}" -> "${Unicode:Cool}"

I continue to find this very unnatural, but I re-read variables, and it
does the same thing. There's certainly been lots of thought on the
issue, so I won't rehash.

Now my thinking is this: if we do end up going for a general
${function:argument} extension down the road, as was previously
suggested on the list, we'll have a precedent that the syntactic
validity of 'argument' depends upon the semantic value of 'function'.
That should be interesting ;-)

As for the other items, I thought Kjetil's suggestions about which were
superfluous and other edits were spot on.

Aaron

---

From: Michael Haardt <[email-masked]>
Date: Thu, 19 Apr 2007 10:04:31 +0200

> would demonstrate that only one pass is done, too.

Great!

> > >    "${unicode:200000}"  -> error

Now that we dropped the 6-digit-restriction, we no longer need two examples
of range violations.

> I think the fourth example from the top covers this already, that is

I agree.  Everything else is fine with me.  I'll adapt my code.

Michael

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 25 Apr 2007 11:15:48 +0100

>Now that we dropped the 6-digit-restriction, we no longer need two examples
Ok.
I've removed the first one.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 25 Apr 2007 11:15:27 +0100

>would demonstrate that only one pass is done, too.
I've dropped one 0 (i.e. "${hex:4${hex:30}}"), as 400 is invalid hex anyway.

>could extend that example, e.g., "\${UNICODE:40}" -> "@"
3028bis-12, section 2.4.2 says:

A quoted string starts and ends with a single double quote (the <"> 
character, US-ASCII 34). A backslash ("\", US-ASCII 92) inside of a 
quoted string is followed by either another backslash or a double quote. 
These two-character sequences represent a single backslash or double 
quote within the value, respectively.

Scripts SHOULD NOT escape other characters with a backslash.

So I would rather not demonstrate something which violates the SHOULD NOT.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 25 Apr 2007 15:29:45 +0100

>the second is superfluous.  
It shows a trailing space, so I let it be.

>could extend that example, e.g., "\${UNICODE:40}" -> "@"
I didn't delete the last one, but more examples is better than less.

>unnecessary, IMHO.
This one wasn't removed either.