Subject: Re: [precis] WGLC: draft-ietf-precis-framework-09.txt

From: Andrew Sullivan <[email-masked]>
Date: Fri, 4 Oct 2013 23:07:51 -0400

Dear colleagues,

On Wed, Aug 28, 2013 at 03:46:03PM +0900, Yoshiro YONEYA wrote:
> The WGLC will end on Wednesday, Sep 11th.

On the principle of "better late than never", I at last got to this
document.  Many apologies for taking so long.

First, let me say that this is one of the best drafts I've read in
some time.  It's in very good shape, I think, and could go as it is.
Naturally, however, I cannot help suggesting some gilt to go on that
lily.  The editors should feel free to ignore any or all of this.

There was one largish issue that troubled me.  The discussion in 9.5
basically talks about the risks from enormous character repertoires,
and I wondered whether people won't start asking for a way to
negotiate locale or something similar as a mechanism for narrowing the
choices.  Certainly, something along these lines has been requested
(not to say "vehemently demanded") over and over for IDNA.  In IDNA
it's completely impractical, owing to caches, the need for
compatibility with existing DNS stuff, and so on.  It strikes me as
pretty impractical here, too, but I thought I'd raise it if only so we
can put it down.  (I'm also aware that it's pretty late in the game to
suggest this.  Why it only struck me today I don't know.  I have a dim
memory of having discussed this once before, but I didn't find
anything in the archive.)

The paragraph in section 3.1 starting, "Although members of the
community discussed the possibility of defining other PRECIS string
classes " read oddly to me.  As an alternative I can suggest,
"Although it might be possible to create an additional class that
falls somewhere between IdentifierClass and FreeformClass, it is not
clear how useful such a class would be.  In any case, because of the
ability to subclass FreeformClass, a protocol needing something more
particular is always able to create it."  I don't really care about
this; it was just something that struck me on the way by.

The order of operations is laid out in section 3.2, but is really sort
of explained in section 3.4.4.  It might be nice to put at least a
forward pointer in section 3.2 so that someone who knows what an NFKC
is won't start spluttering.

In section 6.7, I want to make sure we're ok with following IDNA2008's
lead on U+19DA, which moved from PVALID to DISALLOWED in Unicode 6.0.
In the precis case, it's FREE_PVAL.  I think that's fine, but I just
want to call attention.

I caught one typo in section 9.5: "stings".  (I think my giggling over
this might have alarmed the passenger next to me today.)

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 07 Oct 2013 19:21:46 +0900

> document.  Many apologies for taking so long.

I was just starting reading the document yesterday, so same applies here.

> anything in the archive.)

I'm not yet at 9.5, but I definitely agree with Andrew. Let's make sure 
we put a very big nail through that coffin (or whatever the correct 
idiom is).

> classes " read oddly to me.

Same here, but for different reasons. The WG is publishing a document 
with user name and password profiles at (roughly, at least?) the same 
time as this one. Why are we saying "two is enough" but then doing more? 
Maybe the user name/password ones aren't classes, or whatever, but a 
slight rewording would go a long way here to help avoid the impression 
that the WG is at odds with itself.

Something else occurred to me yesterday: Is PRECIS dealing with 
context-dependent characters (ZERO WIDTH JOINER/ZERO WIDTH NON 
JOINER,...)? IDNA2008 deals with them, but while a lot of other stuff is 
carefully noted in the introduction and in 3.1, this is totally missing. 
Either it should be added in these sections, or it should be added in 
PRECIS itself.

Regards,   Martin.

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 14:55:04 -0600

On 9/11/13 8:06 PM, Joseph Yee wrote:
> 
> On Wed, Sep 4, 2013 at 3:25 AM, Yoshiro YONEYA
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Dear all,
> 
>     This is reminder.  WG LC will end next week.
>     Please review the document and give your feedback.
> 
>     Regards,
> 
>     -- Marc & Yoneya, co-chairs
> 
>     On Wed, 28 Aug 2013 15:46:03 +0900 Yoshiro YONEYA
>     <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     > Dear all,
>     >
>     > This message starts two weeks Working Group Last Call (WGLC) on
>     > draft-ietf-precis-framework-09.txt (PRECIS Framework: Preparation and
>     > Comparison of Internationalized Strings in Application Protocols).
>     >
>     > Please review the document and send comments to the list
>     ([email-masked] <mailto:[email-masked]>),
>     > the co-chairs ([email-masked]
>     <mailto:[email-masked]>), or the authors
>     > ([email-masked]
>     <mailto:[email-masked]>) by the end of WGLC.
>     >
> Reviewed the draft, think the approach is good.  Just one minor comment.
> 
> Same as Florian, had the 'hmm' reaction when reading about
> directionality and application behaviour at Section 3.1.  It seems that
> the only application behaviour is permitted pattern.  It doesn't deal
> with visual appearance I believed.  Maybe replace 'application
> behaviour' with 'permitted patther of the string' (or 'allowed
> combination of the string')?

Hmm, I see why you and Florian don't like that text. :-)

How about this?

OLD
   Directionality:  defines application behavior in the presence of code
      points that have directionality, in particular right-to-left code
      points as defined in the Unicode database (see [UAX9]).

NEW
   Directionality:  defines which strings are to be considered
      left-to-right (LTR) and right-to-left (RTL), and the allowable
      sequences of characters in LTR and RTL strings.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 17:03:56 -0600

On 10/07/2013 04:21 AM, "Martin J. Dürst" wrote:
> I'm not yet at 9.5, but I definitely agree with Andrew. Let's make 
> sure we put a very big nail through that coffin (or whatever the 
> correct idiom is).
>
>
> time as this one. Why are we saying "two is enough" but then doing 
> more? Maybe the user name/password ones aren't classes, or whatever, 
> but a slight rewording would go a long way here to help avoid the 
> impression that the WG is at odds with itself.

That raises the question of whether our layering of classes and 
subclasses and profiles is really all that helpful. During the NEWPREP 
BoF, we had fairly strong agreement that we wanted to define a very 
limited number of classes (say, two or three) that could be re-used by 
application protocols. At the least, I think we need to more clearly 
describe the relationship between classes and profiles. I'll look at the 
existing text and propose better text on the list.
> JOINER,...)? IDNA2008 deals with them, but while a lot of other stuff 
> is carefully noted in the introduction and in 3.1, this is totally 
> missing. Either it should be added in these sections, or it should be 
> added in PRECIS itself.
PRECIS simply re-uses the CONTEXTJ and CONTEXTO rules from IDNA. This  
mentioned in various sections of the precis-framework spec (just search 
for "contextj" within the document), but is not specifically called out 
early in the spec. Would that be useful?

Peter

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 19:59:57 -0600

On 10/08/2013 05:03 PM, Peter Saint-Andre wrote:
> describe the relationship between classes and profiles. I'll look at 
> the existing text and propose better text on the list.
OK, I've thought about this further.

Currently we distinguish between classes, subclasses, and usages 
(although we often use the word "profiles" instead of "usages" because 
that seems to feel more natural). I think we can remove a layer here by 
defining classes and profiles, thus deleting subclasses. Right now, the 
only difference between a subclass and a usage is that a subclass can 
restrict the range of allowable codepoints beyond the range for a given 
class, whereas a usage doesn't restrict the codepoints but does define 
the rules for width mapping, additional mappings, case mapping, 
normalization, and directionality. This might be a distinction without a 
difference.

So far we have defined the following usages (where * indicates that the 
usage restricts the range of codepoints):

usernames
passwords
nicknames
localparts *
resourceparts

I suggest that we can simplify matters by using the current naming 
scheme for subclasses and applying it to profile names:

UsernameIdentifierClass (i.e., the Username profile of the IdentifierClass)
PasswordFreeformClass
NicknameFreeformClass
LocalpartIdentifierClass
ResourcepartFreeformClass

This way, each profile has a name, which might make it easier for future 
protocols to re-use what we've defined so far.

Peter

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 20:36:31 -0600

> lily.  The editors should feel free to ignore any or all of this.
Thanks, Andrew. As an editor, one gets so close to the document that 
it's hard to tell what's good and what's not.
> anything in the archive.)
I take it you're suggesting that we add a bit explaining that PRECIS 
does *not* include a way to specify the locale for purposes of 
restricting the range of codepoints that are allowed in a given profile?
> this; it was just something that struck me on the way by.
OK, I will try to find better wording, or just reuse what you've sent.
> is won't start spluttering.
Will do.
> In the precis case, it's FREE_PVAL.  I think that's fine, but I just
Given that we're defining PRECIS in terms of Unicode 6.2, it seems that 
it might be more appropriate to make it DISALLOWED. But I don't have a 
strong feeling about that.
> this might have alarmed the passenger next to me today.)
I'm happy that we were able to provide a bit of entertainment. :-)

Peter

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Oct 2013 21:20:38 -0600

Hi Florian, thanks for the review! Comments inline.

On 09/10/2013 08:06 PM, Florian Zeitz wrote:
> Am 28.08.2013 08:46, schrieb Yoshiro YONEYA:
>> Dear all,
>>
>> Please review the document and send comments to the list ([email-masked]),
>> the co-chairs ([email-masked]), or the authors
>> ([email-masked]) by the end of WGLC.
>>
> I've reviewed this draft, and generally think it takes a sensible approach.
> However, I do have one major grief about it, as well as some smaller
> comments.
>
> The major thing that bothers me about this draft is that string classes
> IMHO conflate to separate concepts. On the one hand they specify valid
> and disallowed codepoints. On the other hand they specify (or rather,
> let the application protocol specify) mappings and normalization.
> The problem I have with this is, that it makes it unclear which strings
> are valid in a certain class.

You are correct. Validity really applies at the level of a profile, not 
a class.
>
> E.g. consider an applications protocol that specifies FreeformClass
> mixed with NFKC. This means characters, which have a compatibility
> equivalent are valid in the sense that they are FREE_PVAL, but are
> invalid in the normalization form. It is unclear to me, whether a string
> containing characters with a compatibility equivalent would be contained
> in the FreeformClass, or more precisely, this specialization thereof.
>
> Similar considerations are true for e.g. mixing case mapping with
> IdentifierClass. Uppercase characters are PVALID/ID_PVAL, but shouldn't
> be present after mapping.
>
> I would prefer it if we specified classes solely in terms of valid and
> disallowed codepoints and directionality requirements.
When you suggest that we specify a class in terms of codepoints, are you 
suggesting that go back to something like the stringprep model, in which 
a class or profile defines a lookup table?
> We would then have separate text saying that an application protocol
> MUST also specify which mappings and normalization to apply, what entity
> needs to apply them (e.g. only the server), and when they need to be
> applied (e.g. when comparing strings, before storing them, before
> display to a user). Both StringPrep-bis and 6122bis already have text to
> this effect. It seems sensible to me to generally require application
> protocols to specify the "who", and "when" beyond the "what". E.g. it is
> often sensible to display identifiers with their case as entered, but
> compare them after case folding. The current text might suggest that
> mappings have to be applied to user input immediately.
I agree that all good application protocols that use PRECIS need to 
specify the enforcement rules, as we already do for SASL and XMPP. I am 
less sure that the PRECIS framework needs to legislate that.
>
> The following are smaller comments ordered by section:
>
> Section 3.1:
> This section talks about "safety" of strings, without ever defining what
> that means in this context. The term "very safe" used to describe the
> IdentifierClass also strangely reminds me of statements about "absolute
> security". Maybe there is a way to generally word this better?

I'll think about better wording and suggest something on the list.
>
> The sentence "Directionality:  defines application behavior in the
> presence of code points that have directionality" seems a bit off to me.
> It is very different from the explanation given later in Section 4.1.
>  From my understanding this is about the allowed combinations of
> characters with directionality, and not about "application beahvior" in
> their presence. It could be about both, but I have not seen a draft talk
> about anything but allowed combinations (i.e. the Bidi Rule) yet.

See other messages in this thread.
>
> Section 3.3.3 and 3.4.3:
> While "unassigned codepoints are unassigned" is a nice tautology, I'm
> not sure what this means in terms of their treatment. In general I feel
> like more explanation is needed about unassigned codepoints and their
> (possible) handling.
Good point. I'll propose text.
>
> Section 3.3.6:
> I think it would be sensible to suggest using the Unicode Default Case
> Folding algorithm, if case mapping is to be applied.
That seems reasonable.
>
> Section 5:
> I feel like this lacks a normative statement about contextual rules.
> E.g. "A character with the derived property value CONTEXTJ or CONTEXTO
>     (CONTEXTUAL RULE REQUIRED) MUST NOT be used unless an appropriate
>     rule has been established and the context of the character is
>
As mentioned, we just point to IDNA2008 here, but I think you and Martin 
are right that we need provide some more detals here. For example, RFC 
5891 says:

###

The Unicode string MUST NOT contain any characters whose validity is 
context-dependent, unless the validity is positively confirmed by a 
contextual rule. To check this, each code point identified as CONTEXTJ 
or CONTEXTO in the Tables document [RFC5892 
<http://tools.ietf.org/html/rfc5892>] MUST have a non-null rule. If such 
a code point is missing a rule, the label is invalid. If the rule exists 
but the result of applying the rule is negative or inconclusive, the 
proposed label is invalid.

###

IMHO your text is more to the point.

Peter

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 9 Oct 2013 00:23:58 -0400

> restricting the range of codepoints that are allowed in a given

Because I'm not as clever as you, it hadn't occurred to me to suggest
that exactly.  But that might well be another way to cope with the
topic: "Yeah, we know you want this.  If you really want it, you need
a special-purpose internationalization framework, and not a
general-purpose one."  The more I think about it, the more I think
that's true.  The user's linguistic environment has so many
tightly-bound implications for an application that if you really need
to know about it, your application needs to get dirty.  (Come to think
of it, this is another nice way of explaining the IDN problem around
this sort of request too.  Thanks!)

> OK, I will try to find better wording, or just reuse what you've sent.

It could be that, with your other proposal (in another thread) about
getting rid of subclassing and making it all use profiles, this point
will find a more natural expression.

> Given that we're defining PRECIS in terms of Unicode 6.2, it seems
> that it might be more appropriate to make it DISALLOWED. But I don't
> have a strong feeling about that.

Well, this was exactly my point.  If we do that, we really _are_
clearly treating at least one character differently than IDNA does.
In that case, we need to open the description in the text to point out
the difference.  Given the plain fact that U+19DA is so obscure as to
be practically irrelevant, I don't want to make a big deal.  But I
guess being picky about the details in this case allows us to see
where the seams are, and that's probably a good thing for the WG to
pay attention to.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 09 Oct 2013 18:02:43 +0900

>        sequences of characters in LTR and RTL strings.

That may be an improvement, but it's missing the fact that LTR and RTL 
strings are the only two alternatives allowed.

Also, it would be good to somewhere say that there is currently no 
widely accepted and implemented solution for the display of constructs 
with mixed pieces (e.g. domain names with LTR and RTL components 
(labels), because the problem is inherently extremely hard.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 09 Oct 2013 18:12:30 +0900

> (labels), because the problem is inherently extremely hard.

In addition, in the introduction, there is a paragraph:

    5.  Leave various mapping operations (e.g., case preservation or
        lowercasing, Unicode normalization, mapping of certain characters
        to other characters or to nothing, handling of full-width and
        half-width characters, handling of right-to-left characters) as
        the responsibility of application protocols, as was done for
        IDNA2008 through an IDNA-specific mapping document [RFC5895].

where "handling of right-to-left characters" is described as a mapping 
operation. That doesn't make sense to me, I think it should be moved out 
to a separate point.

Regards,   Martin.

> Regards, Martin.
> _______________________________________________
> precis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/precis
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 09 Oct 2013 18:24:29 +0900

> OK, I've thought about this further.

Me to. I haven't thought about how to organize the hierarchy much, but I 
have realized that at least it should be better documented.

That starts with the Abstract! It says:
                                                                  A
    specification that reuses this framework can either directly use the
    PRECIS string classes or subclass the PRECIS string classes as
    needed.

"the PRECIS string classes" implies that these have been mentioned 
before, but that didn't happen. I suggest something like:

This document defines several PRECIS string classes. Other 
specifications can either directly use such string classes or can 
subclass a string class defined here.

(please note the change from "reuse" to "use", this document doesn't use 
the classes, so there's no need for "re").

The abstract could then also talk about profiles (or whatever we end up 
if the changes below are adopted).

While we are at it, please shorten the abstract (one way to do this is 
to move historical/motivational stuff to the Intro or some other place 
like an appendix because it won't be very relevant anymore in a few 
years' time), or at least split it up into more than one paragraph.

Also, please explain the hierarchy (class-subclass-profile or whatever) 
in a bit more detail in the introduction, with pointers to examples that 
the WG is publishing in parallel.

Regards,   Martin.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 06:34:34 -0600

> operation. That doesn't make sense to me, I think it should be moved out

I think we can change "mapping operations" to "character-related
operations".

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 06:36:58 -0600

> strings are the only two alternatives allowed.

I think that's a good thing. We're not allowing mixed-direction strings.

> (labels), because the problem is inherently extremely hard.

Yes, which is why we don't allow those. Let's add a note about that.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 06:39:34 -0600

Martin, I agree with all of your points.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 06:45:23 -0600

> this sort of request too.  Thanks!)

Something along those lines sounds good.

> will find a more natural expression.

Perchance. :-)

> where the seams are, and that's probably a good thing for the WG to

It seems to me best not to special-case this character.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 15:05:00 -0600

> Something along those lines sounds good.

Here is proposed text (for the end of section 9.5):

   The challenges inherent in supporting the full range of Unicode code
   points has in the past led some to hope for a way to programmatically
   negotiate more restrictive ranges based on locale, script, or other
   relevant factors.  As a general-purpose internationalization
   technology, the PRECIS framework does not include such a negotiation
   mechanism.  Applications requiring a tighter binding to the user's
   linguistic environment might need to develop extensions to the PRECIS
   framework, or special-purpose internationalization technologies.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 9 Oct 2013 17:54:32 -0400

> Here is proposed text (for the end of section 9.5):

[elided] Looks good to me.  Thanks.

A
-- 
Andrew Sullivan
[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 10 Oct 2013 10:20:07 +0900

> I think we can change "mapping operations" to "character-related

Well, that wouldn't be wrong, but it would be extremely generic. There's 
nothing in the document that isn't about characters, or is there?

I think something like

5.  Leave various mapping operations (e.g., case preservation or
     lowercasing, Unicode normalization, mapping of certain characters
     to other characters or to nothing, handling of full-width and
     half-width characters) and the handling of right-to-left characters
     as the responsibility of application protocols, as was done for
     IDNA2008 through an IDNA-specific mapping document [RFC5895].

Would be at least slightly better.

Regards,   Martin.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 19:36:25 -0600

> Would be at least slightly better.

Sure, that works for me.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 10 Oct 2013 10:42:55 +0900

On 2013/10/10 6:05, Peter Saint-Andre wrote:

>     mechanism.

Mostly fine up to here, except that in this area, not only negotiation 
mechanisms, but also augmenting the identifiers themselves with 
language/locale information are regularly wished for. Using a specific 
example, that chat[English] and chat[French] would be two separate 
identifiers. We should also clearly warn against this.

>                 Applications requiring a tighter binding to the user's

I don't like this, because it essentially says "we don't do this, but 
you may well want to do it". It's something that makes a lot of sense 
e.g. in a content management system. But it does not make sense, and we 
shouldn't give the impression that it does, in the context of 
identifiers and similar stuff (even the PRECIS "FreeformClass" is about 
identifiers, not about content.

Regards,   Martin.

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 09 Oct 2013 20:16:18 -0600

On 10/09/2013 07:42 PM, "Martin J. Dürst" wrote:
> e.g. in a content management system. But it does not make sense, and 
> we shouldn't give the impression that it does, in the context of 
> identifiers and similar stuff (even the PRECIS "FreeformClass" is 
> about identifiers, not about content.
You make several good points. How about this?

    The challenges inherent in supporting the full range of Unicode code
    points have in the past led some to hope for a way to
    programmatically negotiate more restrictive ranges based on locale,
    script, or other relevant factors, to tag the locale associated with
    a particular string, etc.  As a general-purpose internationalization
    technology, the PRECIS framework does not include such mechanisms.

Peter

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 10 Oct 2013 17:36:03 -0600

> will find a more natural expression.

How's this?

   Future specifications might define additional PRECIS string classes
   (e.g., a class that falls somewhere between the IdentifierClass and
   the FreeformClass).  At this time, it is not clear how useful such a
   class would be.  In any case, because application developers are able
   to define profiles of PRECIS string classes, a protocol needing a
   construct between the IdentiferClass and the FreeformClass could of
   course define a restricted profile of the FreeformClass if needed.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 10 Oct 2013 19:40:53 -0400

> How's this?

[elided]  WFM.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Florian Zeitz <[email-masked]>
Date: Sat, 12 Oct 2013 03:39:56 +0200

> a class or profile defines a lookup table?
Well, yes and no. We certainly want the rule/category based algorithm in
order to have Unicode version agility, and I'm not suggesting we get rid
of it. I'm also not suggesting we drop the rules about having some
codepoints only valid in a certain context.
I do however think it may be more sensible to say a string is within a
PRECIS class iff all its characters are PVALID, CONTEXTO, or CONTEXTJ
for this class, and a contextual rule is fulfilled, if required.

This may even already be the intent, but as I said a profile can easily
be defined such, that a string matches this criteria, but can never be
produced after the specified normalization and all mappings were applied.
At any rate I think we need clearer text about the intention here,
answering the question: "When is a string allowed by a profile?". I
personally can not really tell from the draft right now.

> less sure that the PRECIS framework needs to legislate that.
I think not legislating this only gives people a great way to shoot
themselves in the foot. I could be convinced otherwise though.

>> [...]
I think we have agreement (or separate threads) about all other points.

Florian

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 11 Oct 2013 20:33:28 -0600

Heh, I was just now addressing your feedback in my working copy of the
spec. -)

> for this class, and a contextual rule is fulfilled, if required.

The way I see it, it doesn't make much sense to talk about a string
matching a class. In practice within an application protocol, a string
will be checked against the full set of rules as defined by a profile. A
string class provides a kind of "substrate", if you will, but it doesn't
define things in enough detail to perform string matching.

> personally can not really tell from the draft right now.

In part, I don't think it is the responsibility of this specification to
answer that question, other than to make it clear that you need to check
a string against the full set of rules defined by a profile. I do think
it would be helpful to provide some examples, although I think they
probably belong in the various specs that define the profiles (so far
that would be nickname, saslprepbis, and 6122bis).

> themselves in the foot. I could be convinced otherwise though.

Yes, we are trying to prevent such "foot guns". I don't think we can get
very specific (e.g., some technologies that use PRECIS might not have a
client-server architecture). I'll see about proposing some text here...

> I think we have agreement (or separate threads) about all other points.

Great, thanks.

Peter

---

From: Florian Zeitz <[email-masked]>
Date: Sat, 12 Oct 2013 13:25:52 +0200

> define things in enough detail to perform string matching.
What I'm trying to avoid here is a certain ambiguity I think we have
now. To give an example: Text we have in 6122bis now says «MUST consist
only of Unicode code points that conform to the "FreeformClass" base
string class». For arguments sake lets pretend it also specified NFKC.
Does "U+1D7D0 MATHEMATICAL BOLD DIGIT TWO" conform to "FreeformClass" in
this case?

It's either "Yes, that is clearly FREE_PVAL", or "No, that must be
normalized to U+0032 DIGIT TWO", depending on your reading.

> that would be nickname, saslprepbis, and 6122bis).
I think I agree. And I think that is why I suggested leaving
normalizations and mappings out of the classes. We want to tell people
that they have to normalize, and we want them to think about mappings.
But the exact order of those operations, who needs to perform them, what
is valid in protocol slots, etc. is their business.

And what that means in particular (to me) is that a profile would tell
you after which of the steps a string needs to be PVALID under a certain
class. E.g. for SASLPrepbis you would (as I understand it) split up the
simple username into its parts, perform normalization and all mappings
except case mapping, and then check whether the userparts are
PVALID/ID_PVAL. Case mappings would then be performed whenever the SASL
mechanism tells you to.

If everyone is required to specify such rules (usually less complex ones
I'd hope) I don't see the benefit of formally including normalization
and mappings in the definition of a class (in particular if that
definition pretty much is "that's up to you").
This would also allow making this text generic, and not repeating it for
IdentifierClass and FreeformClass.

> client-server architecture). I'll see about proposing some text here...
I don't think we have to be too specific here. Basically we want to get
people thinking. They must answer the questions "Which entity has to
perform which steps?" and "At what point in the protocol flow is this
required?" I don't think that restricts the text to client-server
architectures as such.

Florian

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 14 Oct 2013 09:36:18 -0600

On 10/12/13 5:25 AM, Florian Zeitz wrote:
> string class». 

Ah, I see your point. I think we'll need to adjust the text in all of
the documents that use the framework.

So for instance, currently we say:

   A resourcepart MUST consist only of Unicode code points that conform
   to the "FreeformClass" base string class defined in
   [I-D.ietf-precis-framework].  (Note that there is no XMPP-specific
   subclass for resourceparts.)

   The normalization and mapping rules for the resourcepart of a JID are
   as follows, where the operations specified MUST be completed in the
   order shown:

   1.  Fullwidth and halfwidth characters MAY be mapped to their
       decomposition equivalents.

   [etc.]

I think we'll need to change that to say something like this:

   A resourcepart MUST consist only of Unicode code points that conform
   to the "JIDresourceFreeformClass" profile, which is defined as
   follows:

   1. The base string class is the "FreeformClass" class specified in
   [I-D.ietf-precis-framework]

   2.  Fullwidth and halfwidth characters MAY be mapped to their
       decomposition equivalents.

That is, the base string class can immediately limit the characters that
you even consider. For the "JIDlocalIdentifierClass" profile (or
whatever we call it), if a character is disallowed by the
IdentifierClass then you don't need to consider it further, but if it's
allowed then you need to complete further processing (such as the
relevant mapping operations).

> For arguments sake lets pretend it also specified NFKC.

Here again I think conformance applies only at the level of the profile.
The base string classes just limit the universe of characters you need
to consider.

> class. 

s/class/profile/ (IMHO)

> E.g. for SASLPrepbis you would (as I understand it) split up the

I'm sorry, I've lost track of exactly what "this text" refers to here.

> required?" I don't think that restricts the text to client-server

True. Let me see if I can find a good place for that text, because I
agree with you that we really do want to get people thinking about those
topics.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Tue, 15 Oct 2013 12:16:40 +0200

> relevant mapping operations).
I'm not sure that is the right approach, see below.
It also occurs to me that the framework draft currently calls this
"Usage" instead of "Profile", or is that yet another concept?

>> you after which of the steps a string needs to be PVALID under a certain
I think that might be where we are talking cross purposes.
My understanding is, that we have an algorithm that will tell us whether
a codepoint is PVALID, according to the set of codepoints a class
allows. Profiles have no influence on this decision. I.e. nothing that
determines whether a codepoint is PVALID may be (re)defined by a
profile. This always requires a subclass.

> I'm sorry, I've lost track of exactly what "this text" refers to here.
I think I had this much clearer in my head then I managed to express it,
sorry. Let me try again:
What we have right now are classes. These do two things:
a) Limit the character set
b) Specify a set of properties usages need to define

Specifically a) encompasses Valid, Disallowed, and Unassigned, while
b) encompasses Width Mapping, Additional Mappings, Case Mapping,
Normalization, and Directionality

Note that the text describing b) is almost completely identical for
IdentifierClass and FreeformClass. This is what "this text" was refering to.

My suggestion is to restrict classes to include only the a) properties.
We would still say that a usage needs to specify everything in b) though.
The main benefit I see in this is that classes become self-contained.
Saying that a string has to conform to a class becomes a much more
self-explanatory statement. The usage would however need to further
define at which point in time this conformance is necessary.
E.g. if normalization is done server side, clients may already need to
produce strings conforming to the class. But when clients perform
normalization checking whether a string is in a class after
normalization might reduce user surprise.

Regards,
Florian

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 15 Oct 2013 06:45:16 -0600

> "Usage" instead of "Profile", or is that yet another concept?

Based on other last call feedback (see exchanges with Martin), I've
provisionally replaced the concepts of subclasses and usages with the
concept of a profile. This has (IMHO) simplified matters.

> profile. This always requires a subclass.

It all depends on what you mean by the "P" in PVALID. In practice, my
feeling is that you want to know if a given codepoint is allowed in,
say, the localpart of an XMPP address. The base class isn't always going
to answer that question for you.

> b) Specify a set of properties usages need to define

s/usages/profiles/ yes (if we accept the simplification that Martin and
I worked out).

> IdentifierClass and FreeformClass. This is what "this text" was refering to.

Thanks for the clarification.

> We would still say that a usage needs to specify everything in b) though.

I'm confused again, because that's what I *thought* we were doing all
along. However, if that wasn't clear to you then we need to improve the
text.

> normalization might reduce user surprise.

Indeed.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Florian Zeitz <[email-masked]>
Date: Tue, 15 Oct 2013 15:32:40 +0200

> to answer that question for you.
I'm actually talking about the calculated property here, be it named as
it will. My point is we have an algorithm to calculate PVALID, but we
don't have one for "is this an allowed string for a profile". Which, to
be clear, I think is fine for the framework document.

> along. However, if that wasn't clear to you then we need to improve the
Umm... I think I'm still not making myself clear :/. And I suspect that
is, because "include" is ambiguous here.
I do realize that we are already at the point where the classes
explicitly specify everything in a), while everything in b) is profile
dependent.
What I'm saying is that IMHO b) should not be part of a class at all.
I'd want to have classes, consisting of Valid, Disallowed and Unassigned
sets. For these we have an algorithm determining whether a codepoint is
within the class (i.e. PVALID or CONTEXT[JO] + rule), or not.
Each usage (I think the term is more appropriate for this scheme), would
then define everything from the b) set, and which class to restrict data to.

Florian

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 15 Oct 2013 11:56:23 -0600

> be clear, I think is fine for the framework document.

OK. I agree.

> What I'm saying is that IMHO b) should not be part of a class at all.

I understand. You're saying let's move everything about mapping and
normalization and directionality out of Section 3 (about string classes)
and move it to Section 4 (about profiles).

That makes a lot of sense. Sorry I was so dense. :-)

> then define everything from the b) set, and which class to restrict data to.

Got it.

BTW, I prefer the term profile because during WG meetings and list
discussions that's the term I've heard people naturally use. It seems
artificial to force people to call it a "usage" when their tendency is
to use the term "profile".

Peter

-- 
Peter Saint-Andre
https://stpeter.im/