Subject: [Ltru] Ticket #38, AD issue #5: ABNF vs UTF-8

From: "Phillips, Addison" <[email-masked]>
Date: Tue, 28 Apr 2009 20:58:16 -0700

In this issue, the AD suggests:

---
[ABNF error] While I understand what you mean here, I think the
definition of CHARS doesn't match the requirement to use UTF-8 encoding
stated earlier in section 3.1.1:

    The registry is a [Unicode] text file, using the UTF-8 [RFC3629]
    character encoding, and consists of a series of records stored in a
    format based on "record-jar" (described in [record-jar]).

IMHO, you can use ABNF productions from [RFC3629] to fix that.
---

I don't believe that this is necessary to change. The registry is a Unicode text file. The character encoding used to record it happens to be UTF-8 (by rule). Reproducing the RFC 3629 ABNF productions are not necessary, and, in fact, would make the ABNF unnecessarily complex. The ABNF production in the current document is:

   CHARS      = (%x21-10FFFF)      ; Unicode code points

This particular production was modeled on others related to code points, such as those in RFC 3987. It is not an error to talk about code points. Users are given the necessary pointers should they be unaware of what a character encoding is.

Propose resolution: no changes.

Addison Phillips
Globalization Architect -- Lab126
Member -- Unicode Editorial Committee

Internationalization is not a feature.
It is an architecture.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 29 Apr 2009 17:55:02 +0900

Addison, I suggest that where possible, you follow previous discussions 
on these issues. In particular, for this specific issue, I made a 
wording proposal, and Alex said that he would prefer the new wording.

My judgement as a co-chair and shepherd is that if our AD prefers 
something, and nobody from the WG opposed, then that's what we should do 
:-).

For more details, please see
http://www.ietf.org/mail-archive/web/ltru/current/msg12445.html and
http://www.ietf.org/mail-archive/web/ltru/current/msg12452.html.

Regards,    Martin.

On 2009/04/29 12:58, Phillips, Addison wrote:
> _______________________________________________
> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 29 Apr 2009 07:54:52 -0700

If we have resolutions, it would be really useful if they were, well, tracked in the tracker. I got upwards of 3000 emails while on vacation and, no, I didn't read them all :-).

I will extract the proposed text from these messages and incorporate it.

Addison Phillips

> -----Original Message-----
> From: "Martin J. Dürst" [mailto:[email-masked]]
> Sent: Wednesday, April 29, 2009 1:55 AM
> To: Phillips, Addison
> Cc: LTRU Working Group
> Subject: Re: [Ltru] Ticket #38, AD issue #5: ABNF vs UTF-8
> 
> > I don't believe that this is necessary to change. The registry is
> a Unicode text file. The character encoding used to record it
> happens to be UTF-8 (by rule). Reproducing the RFC 3629 ABNF
> productions are not necessary, and, in fact, would make the ABNF
> unnecessarily complex. The ABNF production in the current document
> is:
> >
> > This particular production was modeled on others related to code
> points, such as those in RFC 3987. It is not an error to talk about
> code points. Users are given the necessary pointers should they be
> unaware of what a character encoding is.
> >

---

From: "Phillips, Addison" <[email-masked]>
Date: Wed, 29 Apr 2009 08:10:27 -0700

The last email cited by Martin contains some additional discussion from the AD about the need to separate UTF-8 from the registry description, etc. So my proposed resolution isn't a simple cut-and-paste job. Here's what I propose:

Change the first paragraph from:

--
The registry is a [Unicode] text file, using the UTF-8 [RFC3629] character encoding, and consists of a series of records stored in a format based on "record-jar" (described in [record‑jar]. Each record, in turn, consists of a series of fields that describe the various subtags and tags.
--

To read:

--
The registry is a [Unicode] text file and consists of a series of records in a format based on "record-jar" (which described in [record-jar]). Each record, in turn, consists of a series of fields that describe the various subtags and tags. The actual registry file is encoded using the UTF-8 [RFC3629] character encoding.
--

Then, further down, change this para:

--
Although the file format uses the UTF-8 encoding, fields are restricted to the printable characters from the US-ASCII [ISO646] repertoire unless otherwise indicated in the description of a specific field-name (Record and Field Definitions).
--

To read:

--
<t>Although the file format uses the Unicode character set and the file itself encoded using the UTF-8 encoding, fields are restricted to the printable characters from the <xref target="ISO646">US-ASCII</xref> repertoire unless otherwise indicated in the description of a specific <xref target="recordformat">field-name</xref>.</t>
--

And finally, before the ABNF, change this text:

--
The format of the registry is described by the following ABNF (per [RFC5234] (Crocker, D. and P. Overell, “Augmented BNF for Syntax Specifications: ABNF,” January 2008.)):
--

To read:

--
The format of the registry is described by the following <xref target="RFC5234">ABNF</xref>. Character numbers (code points) are taken from Unicode and terminals in the ABNF productions are in terms of characters rather than bytes.
--

Addison Phillips

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Phillips, Addison
> Sent: Wednesday, April 29, 2009 7:55 AM
> To: "Martin J. Dürst"
> Cc: LTRU Working Group
> If we have resolutions, it would be really useful if they were,
> well, tracked in the tracker. I got upwards of 3000 emails while on
> vacation and, no, I didn't read them all :-).
>

---

From: Mark Davis <[email-masked]>
Date: Wed, 29 Apr 2009 08:19:20 -0700

+1

Mark

On Wed, Apr 29, 2009 at 08:10, Phillips, Addison <[email-masked]> wrote:

> The last email cited by Martin contains some additional discussion from the
> AD about the need to separate UTF-8 from the registry description, etc. So
> my proposed resolution isn't a simple cut-and-paste job. Here's what I
> propose:
>
> The registry is a [Unicode] text file, using the UTF-8 [RFC3629] character
> encoding, and consists of a series of records stored in a format based on
> "record-jar" (described in [record‑jar]. Each record, in turn, consists of a
> series of fields that describe the various subtags and tags.
> --
>
> To read:
>
> --
> The registry is a [Unicode] text file and consists of a series of records
> in a format based on "record-jar" (which described in [record-jar]). Each
> record, in turn, consists of a series of fields that describe the various
> subtags and tags. The actual registry file is encoded using the UTF-8
> [RFC3629] character encoding.
> --
>
> Although the file format uses the UTF-8 encoding, fields are restricted to
> the printable characters from the US-ASCII [ISO646] repertoire unless
> otherwise indicated in the description of a specific field-name (Record and
> Field Definitions).
> --
>
> To read:
>
> --
> <t>Although the file format uses the Unicode character set and the file
> itself encoded using the UTF-8 encoding, fields are restricted to the
> printable characters from the <xref target="ISO646">US-ASCII</xref>
> repertoire unless otherwise indicated in the description of a specific <xref
> target="recordformat">field-name</xref>.</t>
> --
>
> The format of the registry is described by the following ABNF (per
> [RFC5234] (Crocker, D. and P. Overell, “Augmented BNF for Syntax
> Specifications: ABNF,” January 2008.)):
> --
>
> To read:
>
> --
> The format of the registry is described by the following <xref
> target="RFC5234">ABNF</xref>. Character numbers (code points) are taken from
> Unicode and terminals in the ABNF productions are in terms of characters
> rather than bytes.
> --
>
>
>
> Addison Phillips

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 30 Apr 2009 12:44:23 -0700

Hi -

As co-chair...

I think we have a clear consensus that we want to clarify
that the ABNF is in terms of characters, rather than bytes.

Minor textual changes in several spots have been
identified as necessary to make this clarification, and we'll
trust the editor to figure out whether the text he proposed,
the text Martin proposed, or a merger of the two makes most sense.

So, I've closed http://trac.tools.ietf.org/wg/ltru/trac/ticket/38

Randy