Subject: [Ltru] NFC

From: "Doug Ewell" <[email-masked]>
Date: Sun, 19 Aug 2007 18:31:54 -0700

On Tue, 3 Jul 2007 08:07:18 -0700, Peter Constable <petercon at 
microsoft dot com> wrote:

> +1 to specifying NFC (whether we use UTF-8 or NCRs).

I don't see this yet in draft-4646bis-08, so I propose the following New 
Wording™ for Section 3.1.1:

OLD:
Each field can be considered a single, logical line of Unicode [Unicode] 
characters, comprising a field-name and a field-body separated by a 
COLON character (%x3A).

NEW:
Each field can be considered a single, logical line of Unicode [Unicode] 
characters, comprising a field-name and a field-body separated by a 
COLON character (%x3A).  Each Unicode character MUST be represented in 
Normalization Form C (NFC), as specified in Unicode Standard Annex #15 
[UAX15].

[Note: This will require a new normative reference to UAX #15.  I don't 
know whether it will also require the existing informative reference to 
the Unicode Standard to be made normative.]

OLD:
Folding is always done on Unicode code point boundaries (never in the 
middle of a multibyte UTF-8 sequence).

NEW:
Folding is always done on Unicode code point boundaries (never in the 
middle of a multibyte UTF-8 sequence), and SHOULD NOT be done 
immediately before a combining character as defined by Unicode 
[Unicode].

[Note: Again, the SHOULD NOT may require the Unicode reference to be 
normative.]

Are there any other places where NFC needs to be mentioned, in either 
4646bis or 4645bis?

--
Doug Ewell  *  Fullerton, California, USA  *  RFC 4645  *  UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

_______________________________________________
Ltru mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ltru

---

From: John Cowan <[email-masked]>
Date: Thu, 23 Aug 2007 08:22:29 -0400

> Normalization Form C (NFC), as specified in Unicode Standard Annex #15 

Being in NFC is a property of the field as a whole, not of each character.
For example, a + COMBINING ACUTE is not NFC, but q + COMBINING ACUTE is.
So:

s/Each Unicode character/The field

> the Unicode Standard to be made normative.]

A normative reference to Unicode is a normative reference to UAX #15,
because UAXes are logically part of Unicode, and are kept separate only
for maintenance reasons.  (They are physically included in TUS 5.0
and later.)

> immediately before a combining character as defined by Unicode 

Make that MUST NOT.  A combining character applies to the immediatelly
preceding base character, and if that happens to be whitespace, that
means something different.

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
'Tis the Linux rebellion / Let coders take their place,
The Linux-nationale / Shall Microsoft outpace,
We can write better programs / Our CPUs won't stall,
So raise the penguin banner of / The Linux-nationale.  --Greg Baker

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 23 Aug 2007 23:04:51 -0700

John Cowan <cowan at ccil dot org> wrote:

> Being in NFC is a property of the field as a whole, not of each 
> character.  For example, a + COMBINING ACUTE is not NFC, but q + 
> COMBINING ACUTE is.  So:
>

Of course it is.  I knew that.

> because UAXes are logically part of Unicode, and are kept separate 
> only for maintenance reasons.  (They are physically included in TUS 
> 5.0 and later.)

<covers face>

I'm starting to think I should claim this was really just a Unicode pop 
quiz in disguise, and John passed.

> means something different.

I almost had a defense here, that line folding would be collapsed to 
nothing, but of course that's not true either; lines are only folded 
where whitespace would occur anyway.

Any NFC-related edits to 4646bis should follow John's corrections.  And 
Doug needs to go back to Unicode school.

---

From: Addison Phillips <[email-masked]>
Date: Fri, 24 Aug 2007 08:47:28 -0700

> Doug needs to go back to Unicode school.

Actually, this is a flaw in our version of record-jar. What we should 
have is the line-continuation character at the end of the previous line. 
Otherwise languages that use spaces to separate words are at a 
disadvantage... or languages that don't use spaces will have spurious 
ones introduced (pick your poison).

That is, consider this text:

Field: This is a wrapped
    line of text.

The unwrapping would make that field-body into this string:

"This is a wrappedline of text."

If a space were introduced by the unwrapping, any text that does not use 
spaces (Japanese, Chinese, Korean, Thai, etc. etc.) would have spurious 
spaces added at each line wrap.

If we added line continuation, we would have:

Field: The one space is followed by \
    &#x301; which is a combining mark.

John's objection that the U+0301 combines with the leading whitespace on 
the second line wouldn't hold if unwrapping were considered a 
higher-order file formatting rule. That is, the whitespace at the start 
of the second line isn't considered to be part of the text. This, 
actually, is how I considered it to work in my own implementations and 
the reason I only specified code point separation previously.

However, since we don't supply a very full description of record-jar, I 
have made the edits, including the MUST NOT edit, to prevent others from 
falling into the trap of interpreting the file as text and then 
performing the unwrapping.

Addison

-- 
Addison Phillips
Globalization Architect -- Yahoo! Inc.
Chair -- W3C Internationalization Core WG

Internationalization is an architecture.
It is not a feature.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 24 Aug 2007 11:22:14 -0700

My real preference would be no broken lines at all. Then we don't have to
jimmy around with it. Parsing is simpler and straightforward, and If someone
wants to see wrapped lines, then they just turn that on in their viewer (or
copy the contents to one that does wrap).

Mark

-- 
Mark

---

From: Addison Phillips <[email-masked]>
Date: Fri, 24 Aug 2007 11:29:20 -0700

Yes, I agree completely.

However, we made 72-byte line length limits a part of RFC 4646. For 
compatibility's sake, we're probably stuck with wrapping. Note that this 
also means no line continuation character :-).

Addison

Mark Davis wrote:
> My real preference would be no broken lines at all. Then we don't have 
> to jimmy around with it. Parsing is simpler and straightforward, and If 
> someone wants to see wrapped lines, then they just turn that on in their 
> viewer (or copy the contents to one that does wrap).
> 
> Mark
> 
> On 8/24/07, *Addison Phillips* <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Doug Ewell wrote:
>     [email-masked] <mailto:[email-masked]>

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 24 Aug 2007 19:34:00 -0700

Addison Phillips <addison at yahoo dash inc dot com> wrote:

> have is the line-continuation character at the end of the previous 
> line. Otherwise languages that use spaces to separate words are at a 

Yeah, but I don't think we've got room right now for another change in 
the registry format, considering that 4646bis was widely portrayed as 
"ISO 639-3 plus a few minor tweaks."

I'd like to see both drafts go to WG Last Call by the end of September 
2007.  Can we manage that?

> John's objection that the U+0301 combines with the leading whitespace 
> on the second line wouldn't hold if unwrapping were considered a 
> higher-order file formatting rule. That is, the whitespace at the 
> start of the second line isn't considered to be part of the text. 
> This, actually, is how I considered it to work in my own 
> implementations and the reason I only specified code point separation 
> previously.

Actually, the way I view wrapping is that the whitespace at the start of 
the second line ISN'T part of the text.  Rather, the CRLF at the end of 
the first line and the two spaces at the start of the second line are 
removed, and then a single space is inserted.  I don't know if that's 
confusing to anyone else, but it makes sense to me.

Since we already reference Unicode, I suppose we could add a rule that a 
space should or should not be inserted at the line-wrap point depending 
on the Script category of the surrounding text.  That would be 
compatible with what we say now about the matter (viz. nothing).  But 
since this is only likely to affect Comments fields, you can imagine how 
low a priority I think it should be.

--
Doug Ewell · Fullerton, California, USA · RFC 4645 · UTN #14
http://users.adelphia.net/~dewell/
http://www1.ietf.org/html.charters/ltru-charter.html
http://www.alvestrand.no/mailman/listinfo/ietf-languages

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 24 Aug 2007 21:09:38 -0700

> 2007.  Can we manage that?

It really depends on whether we can resolve the extlang question -- I think
that is the only major outstanding issue.

Mark