Subject: [idn] Re: Privacy Policy

From: [email-masked] (Internet Draft Submission Manager)
Date: Thu, 07 Nov 2002 05:38:38 -0800

Greetings,

We are sorry, but the cut-off for Internet-Draft submissions was Monday
November 4 at 9AM ET. Your submission will not be retained (i.e. you
need to resubmit) after November 17, 2002.

If you receive this announcement, your submission will NOT be processed.

Any submissions received prior to November 18, 2002 will not be retained;
they must be resubmitted. Internet-Draft submissions received on or after
November 18 will be processed. However, Internet-Draft announcements will
not be sent until after the IETF Meeting in Atlanta.

AS you may surmise, this is an extremly busy week. Be advised
that it may take a number of days to process and announce your
document. Our goal is to process and announce all valid 
Internet-Draft submissions by November 12 at the latest.

Thank you for your understanding.

IETF Secretariat

---

From: [email-masked] (Internet Draft Submission Manager)
Date: Tue, 12 Nov 2002 08:21:13 -0800

Greetings,

IETF Secretariat

---

From: Martin Duerst <[email-masked]>
Date: Wed, 27 Nov 2002 02:00:01 +0900

The IDNA draft
(http://www.ietf.org/internet-drafts/draft-ietf-idn-idna-14.txt)
contains the word 'verify' in several critical locations, for
example:

  (a) Verify the absence of non-LDH ASCII code points; that is,
      the absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.

I haven't found any kind of 'definition' of 'verify'. My imagination
says that 'verify' means: Check, and throw an error if the check
isn't successful.
Can anybody confirm? Is it possible to make this clear in the draft?

Also, there is a typo in that the term 'UseSTD13ASCIIRules flag'
is used once, where otherwise, it is   'UseSTD3ASCIIRules flag'.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 27 Nov 2002 03:56:26 +0900

In http://www.ietf.org/internet-drafts/draft-ietf-idn-punycode-03.txt,
example (I) says:

  (I) Russian (Cyrillic):
         U+043F u+043E u+0447 u+0435 u+043C u+0443 u+0436 u+0435 u+043E
         u+043D u+0438 u+043D u+0435 u+0433 u+043E u+0432 u+043E u+0440
         u+044F u+0442 u+043F u+043E u+0440 u+0443 u+0441 u+0441 u+043A
         u+0438
         Punycode: b1abfaaepdrnnbgefbaDotcwatmq2g4l

The presence of the upper-case 'D' (not to say the string 'Dot' :-)
is confusing, because it seems completely arbitrary. There is no
upper-case letter in the Cyrillic string.

Is the result string actually correct? Has it been generated mechanically?
How did the upper-case D get in there?

Regards,    Martin.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 26 Nov 2002 22:14:43 +0000

Martin Duerst <[email-masked]> wrote:

> there is a typo in that the term 'UseSTD13ASCIIRules flag' is used
> once, where otherwise, it is 'UseSTD3ASCIIRules flag'.

Good catch.  I guess we'll fix this during the 48-hour author review (if
the RFC editor hasn't already caught it and fixed it by then).

> Can anybody confirm?

It's not a technical term, it's just the plain English word "verify".
"Verify the absence of foo" means look and make sure that there is no
foo.  If you look and see foo, then your attempt to verify the absence
of foo has failed.

> Is it possible to make this clear in the draft?

I thought it was clear from this:

    It is important to note that the ToASCII operation can fail.
    ToASCII fails if any step of it fails.

    ToUnicode never fails.  If any step fails, then the original input
    sequence is returned immediately in that step.

Do you think this needs clarification?

AMC

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 26 Nov 2002 22:50:13 +0000

Martin Duerst <[email-masked]> wrote:

> Is the result string actually correct?

Yes.  Uppercase and lowercase letters are equivalent in Punycode.
Section 5 says:

        41..5A (A-Z) =  0 to 25, respectively
        61..7A (a-z) =  0 to 25, respectively

    A decoder MUST recognize the letters in both uppercase and lowercase
    forms (including mixtures of both forms).

> Has it been generated mechanically?

Yes.  The sample encoder (that appears in the Punycode spec) was fed
the Russian text in u+ form (exactly as it appears in example I) and it
produced the Punycode encoding (exactly as it appears in example I).  If
the Punycode string (exactly as it appears in example I) is fed to the
sample decoder, it will produce the Russian text in u+ form (exactly as
it appears in example I, except for line breaks).  If you have an ANSI C
compiler, you can reproduce these results yourself.

If you have another Punycode implementation, it won't necessarily agree
with the sample implementation on the case of the letters in the encoded
string, but that's not significant.  Any correct encoder will produce
the same ASCII string (ignoring case), and any correct decoder will
produce the same numeric code points.

> How did the upper-case D get in there?

It corresponds to the uppercase U in one of the code points in the u+
notation.  The sample Punycode implementation uses the case of the u
as a 1-bit annotation.  The sample encoder uses these annotations to
determine the case of the letters in the Punycode string, and the sample
decoder recovers the annotations, using them to determine the case of
the u's in the u+ notation.

Section 5 says:

    An encoder SHOULD output only uppercase forms or only lowercase
    forms, unless it uses mixed-case annotation (see appendix B).

Appendix B says:

    ...mixed-case annotation is not used by the ToASCII and ToUnicode
    operations specified in [IDNA], and therefore implementors of IDNA
    can disregard this appendix.

    These annotations do not alter the code points returned by decoders;
    the annotations are returned separately, for the caller to use or
    ignore.  Encoders can accept annotations in addition to code points,
    but the annotations do not alter the output, except to influence the
    uppercase/lowercase form of ASCII letters.

    Punycode encoders and decoders need not support these annotations,
    and higher layers need not use them.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Wed, 27 Nov 2002 08:00:11 +0900

>Do you think this needs clarification?

Well, it would be very good if the words 'fail' and 'verify'
could be linked, for example something along the lines of
'If a condition cannot be verified, then ToASCII fails.'

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Nov 2002 05:41:28 +0900

Hello Adam,

Many thanks for your quick response.

>as a 1-bit annotation.

I see. I don't think this is a very good idea to use the U+ for
distinction, for the following reasons:

1) The u+ -> lower case, U+ -> upper case is not documented anywhere
    in the punycode draft (or at least I didn't find it). If used at
    all, it should be documented straight at the start of the examples.

2) The above convention is very easy to overlook, in particular because
    u+ and U+ look so very similar. It is close to a widely established
    convention, but differs slightly.

3) Punycode can be used in different ways, on mixed strings, on
    lc strings that still contain the original casing info, and
    on pure lc strings. Maybe there should be separate examples
    for all these three uses.

Regards,   Martin.

---

From: [email-masked] (Martin v. Loewis)
Date: 27 Nov 2002 23:48:51 +0100

>     all, it should be documented straight at the start of the examples.

This is no issue, since the case is not normative. To produce test
vectors, I find it reasonable to include mixed case, so that
implementors may find problems in their implementations when
confronted with mixed case.

> 2) The above convention is very easy to overlook, in particular
> because u+ and U+ look so very similar. It is close to a widely
> established convention, but differs slightly.

So what? Again, we are talking about test vectors here. The casing of
the output is not normative - let alone the syntax of the input;
usually, input won't be ASCII in the u+ notation anyway.

>     for all these three uses.

While this may be true for Punycode, that is not true for IDNA.  In
IDNA, nameprep will lower-case the input before applying Punycode.

Regards,
Martin

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 27 Nov 2002 23:23:01 +0000

Martin Duerst <[email-masked]> wrote:

> > The sample Punycode implementation uses the case of the u as a 1-bit
> > annotation.
>

It is not documented in the spec because it is not a feature of
Punycode.  The Punycode algorithm inputs and outputs code points, which
are numbers.  It does not input or output "u+".

The sample implementation inputs and outputs "u+".  Therefore the use
of the u as a 1-bit annotation is mentioned in the documentation of the
sample implementation, which is embedded in the source code (you can
either read the source code of the usage() function, or run the program
with no arguments).

I tried to downplay mixed-case annotation as much as possible in the
draft, because Paul and Patrik have never liked it.  I'll ask them if
they think the Examples section should call attention to the mixed-case
annotations.

>    established convention, but differs slightly.

I'm curious--can you explain or point me to that convention?

> 3) Punycode can be used in different ways, on mixed strings, on lc
>    strings that still contain the original casing info, and on pure
>    lc strings.  Maybe there should be separate examples for all these
>    three uses.

Long ago some of my drafts included explanations of various scenarios,
some of which are not applicable to domain names.  Since then, we
agreed that the Punycode draft should present itself as a piece of IDNA
which, by the way, could perhaps be useful outside IDNA; not as as a
general encoding that happens to be used in IDNA.  The goal was to avoid
confusing implementors of IDNA.  So I cut out all but the essentials of
mixed-case annotation, and confined its description to a single appendix
(and the sample implementation).

Maybe, if/when there is real interest in using mixed-case annotations
and/or using Punycode for things other than domain names, we could
update the spec, or augment it with a separate document.

AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 28 Nov 2002 15:13:16 +0100

...
> Punycode.  The Punycode algorithm inputs and outputs code 
> points, which are numbers.  It does not input or output "u+".
> 

I find this confusing.  And I think two things have gone wrong here.

In general, an example implementation (of any software specification)
should be as pure as possible, no extra bells and whistles.  Same
goes for example strings (if any).  So the "1-bit annotations" 
appear to not belong.  If they belong, they surely should not
manifest themselves as "U+"/"u+", which in addition is not explained.

Furthermore, in this case, there are example argument (to encode)
and result (from decoding) strings that are not readily presentable
in an ASCII document (as IETF documents apparently still have to be).
So there is an additional step of presenting characters that are not
ASCII printable ones. Using "U+" notation for such code points in
the example string presentation is fine, though.  This additional
step does NOT belong in a sample implementation, but should be kept
strictly outside of it.  It would still be nice to have the example
strings also in UTF-8 in the document, if possible, even if the
example implementation does not work directly on UTF-8 strings.

As for the "test wrapper" part of the sample implementation, if
at all presented, it should read/write test strings in at least one
of the "major" Unicode encodings (easiest would be to use UTF-32;
the conversion to/from UTF-32 and some other Unicode encoding
does not even belong in the *example test* wrapper, nor does
encode/decode of "U+"-notation).

Examples (strings, code) should be there to illustrate required
behaviour.  Not confuse the readers with irrelevant (and even
misleading and hard to see) extras.

		/kent k

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 28 Nov 2002 23:27:19 +0000

Kent Karlsson <[email-masked]> wrote:

> should be as pure as possible, no extra bells and whistles.

The sample implementation is divided into two parts, the implementation
itself (let's call it the core), and the test wrapper.  The core
is pure.  It implements the Punycode algorithm and the mixed-case
annotations described in appendix B, and provides a simple abstract
programming interface that makes it trivial to not use the annotation
support if you don't want it (you simply pass a null pointer instead of
a pointer to an array of flags).

> So the "1-bit annotations" appear to not belong.

I think it's good to have examples of the annotations, if for no other
reason than to serve as examples of mixed-case encoded strings, which
all implementations are required to handle, even if they don't interpret
them as annotations.

> If they belong, they surely should not manifest themselves as
> "U+"/"u+",

I don't see what's wrong with that representation.  The nice thing
about it is that it's very easy to ignore if you don't care about the
annotations.

> which in addition is not explained.

I can see how it might be considered unfortunate that the examples
section does not call attention to the annotations.  It might be too
late to change that, and I don't know if there would be consensus that
it should be changed.

> Furthermore, in this case, there are example argument (to encode) and
> result (from decoding) strings that are not readily presentable in an
> ASCII document (as IETF documents apparently still have to be).  So
> there is an additional step of presenting characters that are not
> strictly outside of it.

I agree that it does not belong in the core, and it's not.  I don't see
why it shouldn't be in the test wrapper.  The only purpose of the test
wrapper is to provide a convenient command-line interface on top of the
programming interface, to make it easy to write scripts to run tests.  I
put the conversion to/from U+ notation into the test wrapper because it
was getting to be a pain to generate and test the examples without it.
I figured that other readers of the document would find it convenient
(like I did) to be able to use the sample implementation directly on the
example strings.

If someone wants a different test wrapper that inputs and outputs
UTF-32, they can modify the test wrapper pretty easily.

> It would still be nice to have the example strings also in UTF-8 in
> the document, if possible, even if the example implementation does not
> work directly on UTF-8 strings.

It would, but I think RFCs are required to be ASCII.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Fri, 29 Nov 2002 06:48:21 +0900

Hello Adam,

I understand that the documents have been approved by the IESG,
so at this stage, changes are not appropriate. But maybe some
of the changes can be done in the next stage (draft standard,...).

> > I don't think this is a very good idea to use the U+ for

I agree that documenting it in the normative part of the spec
would be a bad idea. But what I was proposing was that it be
mentioned in the examples section.

>either read the source code of the usage() function, or run the program

I agree with what Kent Karlson said on this.

>they think the Examples section should call attention to the mixed-case

I agree with downplaying. But then the best thing would be to not
use it in the examples section. The alternative is to mention them
in the examples section.

>I'm curious--can you explain or point me to that convention?

The convention is the very widely established convention of writing
Unicode/UTC codepoints with an U+ followed by 4 to 6 digits. In that
convention, the U is always upper case. For many people used to this
convention, this convention is so well established that it's very
easy to overlook the lower-case u+. That happened to two of us on
Tuesday, and it took quite some time for us to figure out what was
actually going on.

>(and the sample implementation).

I think that basically, that was the right thing to do.

>update the spec, or augment it with a separate document.

There is definitely no interest from my side for using Punycode
for things other than domain names, and therefore no interest
for mixed-case annotations.

Regards,    Martin.