Subject: Re: SASLPrep vs Full Stop code points

From: Ienup Sung <[email-masked]>
Date: Fri, 07 Nov 2003 13:07:04 -0800 (PST)

I believe the question is whether to (incorrectly) generalize/map or not in my
opinion.

Actually the confusion will also and sure to occur if you map the all full
stops to the U+002E since Asian people distinguish the U+002E and the U+3002
even without any thinking if you wrongly use the full stops in the place of
decimal point.

The point that I can make further is that people have been using various
Asian codesets/codepages that have different code point values and different
input mechanisms for each of those Asian full stops and the ASCII decimal
points many years; they see them and use them differently.

The NFKC used in SASLprep will map the U+FF0E to the U+002E since they are
equivalent and should be normalized. On the other hands, the U+3002 and
the U+FF61 are not equivalent to the U+002E but only to themselves, i.e.,
the U+FF61 will map to the U+3002 after the NFKC.

Here is also another practical example: the ATOK is, arguably, the most popular
Japanese input system in my understanding. When you use ATOK to input
Japanese, what happens is that when you type with numbers, you will get '.'
automatically if you press dot key from your keyboard. When a sentense is
ending, used by itself alone, or surrounded with non-gidits, the dot key
will be interpreted as the ideographic/Asian full-stop of U+3002. In other
words, to input ideographic/Asian full stop (U+3002) with number isn't that
easy and when that happened, i.e., a user typed in the U+3002 among numbers,
that means it is most definitely not the full stop. Furthermore, you can ask
any Asian people around you if you can use full stop in a number/numeric
notation; most likely almost 100% of them will say, No Way.

(Also and at the same time, just as for info, Asian people will absolutely
accept and tolerate '.' being used in the place of U+3002 iff it is
used as the full stop. Usually, formal documents don't use '.' by the way as
you know.)

I also attached a sample screen shot in JPEG showing the ATOK12 based
Japanese input with numbers and would like to demonstrate how Asian input
system works in the SASL WG meeting at Tuesday.

And so once again my suggestion is that the mappings of all full stops should
be done as similary as defined in the IDNA for Internet style domain names
but outside and before the SASLprep if such mappings are needed for Internet
domain style names.

With regards,

Ienup

] Date: Fri, 07 Nov 2003 15:16:25 -0500
] From: Jeffrey Altman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: [email-masked], [email-masked]
] MIME-version: 1.0
] X-Accept-Language: en-us, en
] X-No-Spam-Score: Local
] X-Scanned-By: MIMEDefang 2.35
] 
] It is absolutely a trade off.  Avoidance of end user confusion because they
] could not tell the difference between the full stops and the ability to
] specify them as different strings.
] 
] By choosing to fold the full stops we are saying that we believe that there
] will be more confusion by users who try to use one form of full stop when
] they should be using another than the confusion which will be caused for
] system administrators who desire to create accounts which only differ by
] the type of full stop used.
] 
] Jeffrey Altman
] 
] 
] Ienup Sung wrote:
] 
] >Actually that's exactly the problem that I was also talking about since
] >people will treat the Asian full stops and U+002E differently at Asian
] >countries and when people tried to register a new user account with let say
] >3。14 (where it is three, Asian full-stop, and fourteen), the PLAIN based user
] >account system could say, sorry, we already have 3.14 (that is three point
] >one four) which is totally different user account name and would deny
] >the new user account name which will be quite confusing for Asian people.
] >
] >With regards,
] >
] >Ienup
] >
] >  
] >

---

From: Ienup Sung <[email-masked]>
Date: Mon, 10 Nov 2003 07:54:51 -0800 (PST)

In my opinion, yes,that's correct and especially for user names and
passwords.

With regards,

Ienup

] Date: Sun, 09 Nov 2003 20:36:37 -0500
] From: Sam Hartman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: Ienup Sung <[email-masked]>
] Cc: [email-masked], [email-masked]
] MIME-version: 1.0
] X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on 
frigga.ctd.anl.gov
] 
] >>>>> "Ienup" == Ienup Sung <[email-masked]> writes:
] 
]     Ienup> As U+002E is used as full stop as label separator (not as
]     Ienup> decimal point for number representations), the Asian full
]     Ienup> stops are used as full stop as label separator in the IDNA.
] 
]     Ienup> The IDNA's full stop mappings on the Internet domain style
]     Ienup> names are possible and rather desireable for convenience
]     Ienup> since we know that the full stops in that case are used as
]     Ienup> label separators and thus safely map the Asian full stops
]     Ienup> to the U+002E as the full stop.
] 
]     Ienup> With regards,
] 
] You believe there is not a similar convenience issue in usernames?
] 
] 
]

---

From: Ienup Sung <[email-masked]>
Date: Mon, 10 Nov 2003 08:12:42 -0800 (PST)

It is an explicit flag value that need to be given to the SASLprep and
any other Stringprep profile to be more specific. Otherwise, people will be
confused and wondering what to do with it on the Unicode version issues and
also whether and how to use query string vs. storage string (for simple
user names and passwords).

Especially, having some version related remarks or at least explicitly
giving the flag and also mark the flag with a reference would greatly
help people, e.g., "SASLprep(authcid, AllowUnassigned)" or
"SASLprep(passwd, AllowUnassigned)" since the flag processing would/better
need to be done at the Stringprep protocol processing.

With regards,

Ienup

] Date: Sun, 09 Nov 2003 14:13:08 -0500
] From: Jeffrey Altman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: [email-masked], [email-masked]
] MIME-version: 1.0
] X-Accept-Language: en-us, en
] X-No-Spam-Score: Local
] X-Scanned-By: MIMEDefang 2.35
] X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on 
frigga.ctd.anl.gov
] 
] Ienup Sung wrote:
] 
] >PS. By the way, on a bit different topic, reading the SALSprep and also the
] >PLAIN mechanism drafts, it appears there is no AllowUnassigned flag
] >setting or mentioning of it in the drafts. Has this issue been discussed
] >and/or if "Not Allowed" is implclitly meant in the SIMPLE mechanism draft?
] >If not, it seems either explicitly defining that the unassigned Unicode
] >characters are not allowed in the drafts, or, allow setting of the
] >AllowUnassigned flag as either True or False at the PLAIN draft appears
] >necesary so that the versioning issue is clearly stated... If this question
] >has been asked and answered, sorry, please let me know and I'll scan more of
] >the archived emails.
] >
] >  
] >
] Whether or not Unassigned Code Points are to be allowed or not is 
] dependent upon
] how the string being prepared is to be used.  As described in RFC 3454 
] section 7,
] if the use of the string is for storage in a database then Unassigned 
] Code Points
] are to be excluded.  However, if the string is to be used for the 
] purposed of
] querying a database, then the string may contain Unassigned Code Points.
] 
] I do not believe it is required for SASLprep to highlight this 
] distinction. 
] It should however be specified in the protocol documents which utilize 
] string
] preparation methods constructed within the framework of RFC 3454.
] 
] Jeffrey Altman
] 
]

---

From: Ienup Sung <[email-masked]>
Date: Mon, 03 Nov 2003 11:51:56 -0800 (PST)

I agree that the SASLprep is more suitable and applicable for simple
user names and passwords and if the additional mappings such as the full
stops are needed, they are more suitable and correct to be done at
the outside of any Stringprep/Stringprep profiles and most preferrably before
the application of the Stringprep profile(s). <-- These are also my points
all along.

The reasons are like the following that the U+002E (full stop and dot/decimal
point) isn't the same or equivalent to the ideographic/Asian full stops
all the time; if they were, the Unicode itself would define them as
equivalent forms/characters and show canonical/compatiblity decomposition(s)
at each characters so that the Unicode normalizations would normalize
those characters into a (normalized) character.

Also, if everyone would agree on the fact that the ideopgraphic/Asian
full stops are not equivalent to the U+002E and yet we still need to support
the mappings described in the IDNA for the compatibility/interoperability on
the Internet domain style names (only), then, I'm also thinking that everyone
would agree this conditional mappings should be done outside of the Stringprep
and any of the Stringprep profiles (since the Stringprep itself doesn't
really allow conditional mappings at the mapping step as of today even though
a profile *possibly and maybe* could define such conditional mappings).

Re user provided strings, I have the following example that shouldn't match
as the same thing:

	3.14
	３。１４
	3。14

The first one is the pi, three point one four, in ASCII and the second is
three, ideographic full-stop, and fourteen in Asian full-width digit characters
and with ideographic full stop in UTF-8. (The third one is a variation of the
second one, three, ideographic full-stop, and fourteen in ASCII digits and
with ideographic full stop in UTF-8; and the second and the third can be
seen as equivalent.)

With regards,

Ienup

] Date: Fri, 31 Oct 2003 20:52:15 -0800
] From: "Kurt D. Zeilenga" <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] X-Sender: [email-masked]
] To: Ienup Sung <[email-masked]>
] Cc: [email-masked], [email-masked]
] MIME-version: 1.0
] 
] At 05:42 PM 10/31/2003, Ienup Sung wrote:
] >Regarding the full stops, while I do understand the point of display vs.
] >storage/query string in my opinion, I think the issue boils down onto
] >the followign two things in my view:
] >1. Should/can we treat ideographic/Asian full stops as equivalent characters
] >   of the ASCII full stop (U+002E) in 100% of the possible cases/usages?
] 
] SASLprep is only applicable to two use cases: simple user names and password.
] 
] >   The answer is no since the U+002E is not only a full stop character but
] >   it is also a period and dot/decimal point character; CJK speakers do not 
use
] >   the ideographic/Asian full stops as the dot/decimal point character and
] >   hence some use of the ideographic/Asian full stops rather be preserved as
] >   they are to be distingushed. And in that cases, can we distinguish what to
] >   preserve and what not to preserve and map?
] 
] SASLprep scope is limited to preparation of user input strings for matching.
] Issues such as how to preserving the (unprepared) user input is simply beyond
] our scope as matching and hence SASLprep.
] 
] >2. Should/can we keep the compatiblity with the IDNA's additional and
] >   pre-processing on the label separators, i.e., mappings of all full stops 
to
] >   the U+002E, for the domain style (IDN) names before they are used?
] >   I think the answer should be yes.
] 
] SASLprep can define one preparation algorithm for simple user names and
] passwords.  Simple here implies the input has no structure.  That is,
] names have no particular style in respect to the algorithm.
] 
] It is possible that specification using SASLprep could apply additional
] transformations be done.  However, such transforms are beyond the scope
] of SASLprep.
] 
] >I know the above two sound conflicting and the example shown at the argument
] >#1 could be considered as a case that may not need to be really supported.
] >But I thought presenting them might be a valuable thing to do.
] 
] I would be interested in hearing whether you think the additional suggested
] mapping is appropriate or not for matching of simple user names or passwords,
] and why.
] 
] I think this issue boils down to whether the user provided strings:
]         <STRING1><U+002E><STRING2>
]         <STRING1><U+3003><STRING2>
]         <STRING1><U+FF0E><STRING2>
]         <STRING1><U+FF61><STRING2>
] 
] should match or not.  Because user names are often constructed from domain
] names, and IDNA says these stops are to be treated as equivalent, I believe
] they should match.
] 
] Kurt 
]

---

From: Ienup Sung <[email-masked]>
Date: Thu, 06 Nov 2003 16:28:38 -0800 (PST)

Yes, the POSIX/SUS/XPG specifies that the poratble user names be in
the portable file name character set and it includes [A-Za-z._-] with some
additional rules. I also have seen many people using the 0x2e in their user
names and passwords.

And, while I believe I do understand that the SASLprep is intended for
simple user names and passowrds at the moment, I believe I'm seeing
the full stops vs decimal point issue a bit differently... (and maybe in
that way, I'm wrong about it...)

I think there will be problems if we treat the Asian full stops as equivalents
to the ASCII decimal point/dot/full stop character simply because the Asian
full stops are not equivalent to the ASCII decimal points all the time.
In other words, Asian people are and will distinguish (almost implicitly) when
to use the Asian/ASCII full stops and when to use the decimal point character;
if coerced at the protocol level that the Asian full stops are all
mapped to the U+002E for all the cases, people will be, sometimes, confused.

I do see the pros of convenience but that will, in my opinion, come in with
the price of denying the fact that the Asian full stops are only and
conditionally equivalent to the U+002E (when the U+002E means the full stop).
In other words, by having the general full stop mappings, we are denying
the possibility of Asian people using the full stops and decimal points
differently in their user names and passwords.

I personally can live with such general mappings and so thus many other Asians
in my opinion but when we do know about the fact that the characters are
different and virtually and practically we are not capable of distinguishing
when the two groups of characters can be treated equal or not with the
Stringprep and SALSprep, it seems to me that it is not really a good idea to
have a generic, simple table-driven mappings at the SASLprep (even for
the simple user names and passwords)...

Should more opinions needed and if the WG folks agree and assign to me,
I can bring this exactly the same question of whether we can
treat the Asian full stops as the decimal point to the Unicode Technical
Committee and get the opinion of the UTC back to the WGs. 

With regards,

Ienup

PS. By the way, on a bit different topic, reading the SALSprep and also the
PLAIN mechanism drafts, it appears there is no AllowUnassigned flag
setting or mentioning of it in the drafts. Has this issue been discussed
and/or if "Not Allowed" is implclitly meant in the SIMPLE mechanism draft?
If not, it seems either explicitly defining that the unassigned Unicode
characters are not allowed in the drafts, or, allow setting of the
AllowUnassigned flag as either True or False at the PLAIN draft appears
necesary so that the versioning issue is clearly stated... If this question
has been asked and answered, sorry, please let me know and I'll scan more of
the archived emails.

] Date: Thu, 06 Nov 2003 16:35:13 -0500
] From: Sam Hartman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: Ienup Sung <[email-masked]>
] Cc: [email-masked], [email-masked]
] MIME-version: 1.0
] 
] >>>>> "Ienup" == Ienup Sung <[email-masked]> writes:
] 
]     Ienup> I agree that the SASLprep is more suitable and applicable
]     Ienup> for simple user names and passwords and if the additional
]     Ienup> mappings such as the full stops are needed, they are more
]     Ienup> suitable and correct to be done at the outside of any
]     Ienup> Stringprep/Stringprep profiles and most preferrably before
]     Ienup> the application of the Stringprep profile(s). <-- These are
]     Ienup> also my points all along.
] 
] I think Kurt's point was that since saslprep was for usernames and
] passwords, it is more appropriate to do full-stop mappings there than
] in some more general context.  Do we expect people to need decimal
] points in their usernames?  Do we think it a problem if someone uses a
] decimal point in a password, but can later use one of the other
] full-stop characters and have it work?
] 
] --Sam
]

---

From: Ienup Sung <[email-masked]>
Date: Fri, 07 Nov 2003 11:35:46 -0800 (PST)

Actually that's exactly the problem that I was also talking about since
people will treat the Asian full stops and U+002E differently at Asian
countries and when people tried to register a new user account with let say
3。14 (where it is three, Asian full-stop, and fourteen), the PLAIN based user
account system could say, sorry, we already have 3.14 (that is three point
one four) which is totally different user account name and would deny
the new user account name which will be quite confusing for Asian people.

With regards,

Ienup

] Date: Thu, 06 Nov 2003 21:21:08 -0500
] From: Jeffrey Altman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: [email-masked], [email-masked]
] MIME-version: 1.0
] X-Accept-Language: en-us, en
] X-No-Spam-Score: Local
] X-Scanned-By: MIMEDefang 2.35
] 
] The question is not whether or not the different full stops are used by
] different people in different ways at the user interface level.  The 
] transformations
] we are discussing for use with SASLPrep are specific to the use of the 
] strings
] as Storage and Query strings; not as Display strings.  When a string is 
] to be used
] as a username or a password is there a reason why we should be concerned 
] with
] distinguishing the differences?  If there is not a good reason to 
] preserve the
] differences then there is a reasonable argument to fold them similar to 
] the arguments
] for folding spacing characters.
] 
] Users will always be free to choose one representation over another.  It 
] is only
] on the wire that there should be a distinction.
] 
] Jeffrey Altman
] 
] 
] Ienup Sung wrote:
] 
] >I think there will be problems if we treat the Asian full stops as equivalents
] >to the ASCII decimal point/dot/full stop character simply because the Asian
] >full stops are not equivalent to the ASCII decimal points all the time.
] >In other words, Asian people are and will distinguish (almost implicitly) when
] >to use the Asian/ASCII full stops and when to use the decimal point character;
] >if coerced at the protocol level that the Asian full stops are all
] >mapped to the U+002E for all the cases, people will be, sometimes, confused.
] >
] >I do see the pros of convenience but that will, in my opinion, come in with
] >the price of denying the fact that the Asian full stops are only and
] >conditionally equivalent to the U+002E (when the U+002E means the full stop).
] >In other words, by having the general full stop mappings, we are denying
] >the possibility of Asian people using the full stops and decimal points
] >differently in their user names and passwords.
] >
] >  
] >

---

From: Ienup Sung <[email-masked]>
Date: Fri, 07 Nov 2003 13:07:04 -0800 (PST)

With regards,

Ienup

---

From: Ienup Sung <[email-masked]>
Date: Sat, 08 Nov 2003 17:48:21 -0800 (PST)

As U+002E is used as full stop as label separator (not as decimal point for
number representations), the Asian full stops are used as full stop as
label separator in the IDNA.

The IDNA's full stop mappings on the Internet domain style names are possible
and rather desireable for convenience since we know that the full stops in
that case are used as label separators and thus safely map the Asian full stops
to the U+002E as the full stop.

With regards,

Ienup

] Date: Sat, 08 Nov 2003 17:11:47 -0500
] From: Sam Hartman <[email-masked]>
] Subject: Re: SASLPrep vs Full Stop code points
] To: Ienup Sung <[email-masked]>
] Cc: [email-masked], [email-masked]
] MIME-version: 1.0
] Content-transfer-encoding: 8BIT
] 
] >>>>> "Ienup" == Ienup Sung <[email-masked]> writes:
] 
]     Ienup> Actually that's exactly the problem that I was also talking
]     Ienup> about since people will treat the Asian full stops and
]     Ienup> U+002E differently at Asian countries and when people tried
]     Ienup> to register a new user account with let say 3ã€‚14
]     Ienup> (where it is three, Asian full-stop, and fourteen), the
]     Ienup> PLAIN based user account system could say, sorry, we
]     Ienup> already have 3.14 (that is three point one four) which is
]     Ienup> totally different user account name and would deny the new
]     Ienup> user account name which will be quite confusing for Asian
]     Ienup> people.
] 
] OK, then why would people use the Asian full stop in domain names?
] What's different about the world of DNS and the world of user names?
] 
] Yours in confusion,
] 
] --Sam
] 
]

---

From: Ienup Sung <[email-masked]>
Date: Mon, 10 Nov 2003 07:54:51 -0800 (PST)

With regards,

Ienup

---

From: Ienup Sung <[email-masked]>
Date: Mon, 10 Nov 2003 08:12:42 -0800 (PST)

With regards,

Ienup