Subject: [art] Modern Network Unicode

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Jul 2019 22:14:41 +0200

This week, I have been asked twice (for independent drafts) what these
drafts should say about their usage of Unicode.  It turns out, we have
an RFC that is almost, but not entirely unlike what is needed to
reference for a new protocol that uses text: RFC 5198.
I have tried to build on that in the very short draft:

https://tools.ietf.org/html/draft-bormann-dispatch-modern-network-unicode

I would expect that such a specification, once completed, would be a
useful point of reference for any new protocols that need to use text
without any specific requirements imposed by their applications.
(Protocols that already have been around probably benefit less,
because there always are legacy considerations.)

I'm not yet asking for dispatching this document; right now I would
just like to receive feedback from this esteemed community to further
improve the document.

Grüße, Carsten

---

From: "Manger, James" <[email-masked]>
Date: Mon, 8 Jul 2019 02:18:42 +0000

> https://tools.ietf.org/html/draft-bormann-dispatch-modern-network-unicode

Would be nicer if it wasn't written as a diff from RFC5198 (Network Unicode). That is, if you could get all the rules directly from this doc. For instance, I assume Clean Modern Network Unicode must/should be NFC. Keep the RFC5198 comparisons for an informative annex.

ABNF for uchar doesn't exclude U+2028 and U+2029.

--
James Manger

-----Original Message-----
From: art <[email-masked]> On Behalf Of Carsten Bormann
Sent: Monday, 8 July 2019 6:15 AM
To: [email-masked]
Subject: [art] Modern Network Unicode

This week, I have been asked twice (for independent drafts) what these drafts should say about their usage of Unicode.  It turns out, we have an RFC that is almost, but not entirely unlike what is needed to reference for a new protocol that uses text: RFC 5198.
I have tried to build on that in the very short draft:

I would expect that such a specification, once completed, would be a useful point of reference for any new protocols that need to use text without any specific requirements imposed by their applications.
(Protocols that already have been around probably benefit less, because there always are legacy considerations.)

I'm not yet asking for dispatching this document; right now I would just like to receive feedback from this esteemed community to further improve the document.

Grüße, Carsten

_______________________________________________
art mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/art

---

From: Tim Bray <[email-masked]>
Date: Sun, 7 Jul 2019 19:29:03 -0700

I agree with James. This is a good and worthwhile idea, but let's be
developer-friendly and just write the rules in the document. In particular,
starting out by saying “You can’t use U+000A” and then devoting much
following verbiage to saying “well, actually you can, with the following
considerations” feels particularly awkward.

On Sun, Jul 7, 2019 at 7:22 PM Manger, James <
[email-masked]> wrote:

> Would be nicer if it wasn't written as a diff from RFC5198 (Network
> Unicode). That is, if you could get all the rules directly from this doc.
> For instance, I assume Clean Modern Network Unicode must/should be NFC.
> Keep the RFC5198 comparisons for an informative annex.
>
> drafts should say about their usage of Unicode.  It turns out, we have an
> RFC that is almost, but not entirely unlike what is needed to reference for
> a new protocol that uses text: RFC 5198.
> I'm not yet asking for dispatching this document; right now I would just
> like to receive feedback from this esteemed community to further improve
> the document.
>
> Grüße, Carsten
>

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 08 Jul 2019 12:31:22 +1000

I like Section 5.  I was going to send a similar note to the authors of the Robots Exclusion Protocol, but now I can just say "What Carsten said".

I agree with the others about the standalone document thing.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 8 Jul 2019 22:21:10 +0200

Hi James, Tim, Martin,

thank you for the quick feedback!  As suggested, I have started turning this into a standalone document (of course, still requiring RFC 3629, the UTF-8 definition):

Status:         https://datatracker.ietf.org/doc/draft-bormann-dispatch-modern-network-unicode/
Htmlized:       https://tools.ietf.org/html/draft-bormann-dispatch-modern-network-unicode-02
Diff:           https://tools.ietf.org/rfcdiff?url2=draft-bormann-dispatch-modern-network-unicode-02

I hope I haven’t missed anything important from RFC 5198 that I wanted to keep.  
Maybe time to involve the authors of that RFC…

Grüße, Carsten

> On Jul 8, 2019, at 04:18, Manger, James <[email-masked]> wrote:
>

---

From: "Manger, James" <[email-masked]>
Date: Tue, 9 Jul 2019 00:43:41 +0000

A couple more quick comments:
* In section 2, please say "MUST be encoded in UTF-8 [RFC3629]", instead of "requires conformance to [RFC3629]" (without mentioning that it is UTF-8).
* U+FFFE and U+FFFF are forbidden, but not the other 64 permanently reserved noncharacters (eg U+1FFFF or U+FDD0). None are explicitly mentioned in RFC5198. It makes (almost) no sense to send these code points over the network (ie in Network Unicode), but I'm not sure it make sense for a receiver to explicitly look for these to, say, throw an exception.
* An abstract that doesn't mention RFC5198 would be great.

--
James Manger

-----Original Message-----
From: Carsten Bormann <[email-masked]> 
Sent: Tuesday, 9 July 2019 6:21 AM
To: Manger, James <[email-masked]>
Cc: [email-masked]
Subject: Re: [art] Modern Network Unicode — –02 submitted

Hi James, Tim, Martin,

Grüße, Carsten

---

From: Peter Occil <[email-masked]>
Date: Mon, 8 Jul 2019 21:14:44 -0400

Saying U+FFFE and U+FFFF (or any other noncharacters) “MUST NOT be used” merely “as per the Unicode specification” ought to be guided by looking at what the intent of Unicode actually is.

See corrigendum 9 of the Unicode Standard for that intent: http://www.unicode.org/versions/corrigendum9.html

“Noncharacters in the Unicode Standard are intended for internal use and have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange nor do they cause ill-formed Unicode text. This has always been the intent of the standard, as expressed by the Unicode Technical Committee.”

Since noncharacters have no standard meaning outside of internal use, however, they may be even more problematic than the Unicode paragraph and line separators (which do have standard meaning but are forbidden in the draft for Modern Network Unicode).  Many kinds of protocol strings, such as URIs, IRIs, and strings complying with the PRECIS framework, do not allow noncharacter code points, while other kinds of UTF-8 text, such as JSON, do allow noncharacters.  XML allows all noncharacters except for U+FFFE and U+FFFF.

--Peter

From: Carsten Bormann
Sent: Monday, July 8, 2019 4:24 PM
To: Manger, James
Cc: [email-masked]
Subject: Re: [art]Modern Network Unicode — –02 submitted

Hi James, Tim, Martin,

Grüße, Carsten

---

From: Larry Masinter <[email-masked]>
Date: Mon, 8 Jul 2019 21:17:39 -0600

I am confused how this could be to stand-alone without mentioning the https://developers.google.com/search/reference/robots_meta_tag robots html meta tag??? Which seems to me a better design
--
LarryMasinter.net

> On Jul 8, 2019, at 7:14 PM, Peter Occil <[email-masked]> wrote:
>

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jul 2019 17:09:10 -0400

Carsten and Peter,

Citing a corrigendum for Unicode 7 at this point is probably a
bad idea.  If you need a reference or more explanation for this,
see Section 23.7 of Unicode 11.0 (same section number in Unicode
10; I haven't taken the time to change Unicode 12).  That new
section is also more complete than the text Peter cites.

best,
    john

--On Monday, July 8, 2019 21:14 -0400 Peter Occil
<[email-masked]> wrote:

> Saying U+FFFE and U+FFFF (or any other noncharacters) "MUST
> NOT be used" merely "as per the Unicode specification"
> ought to be guided by looking at what the intent of Unicode
> actually is.
> 
> See corrigendum 9 of the Unicode Standard for that intent:
> http://www.unicode.org/versions/corrigendum9.html
> 
> "Noncharacters in the Unicode Standard are intended for
> internal use and have no standard interpretation when
> exchanged outside the context of internal use. However, they
> are not illegal in interchange nor do they cause ill-formed
> Unicode text. This has always been the intent of the standard,
> as expressed by the Unicode Technical Committee."
> 
> Since noncharacters have no standard meaning outside of
> internal use, however, they may be even more problematic than
> the Unicode paragraph and line separators (which do have
> standard meaning but are forbidden in the draft for Modern
> Network Unicode).  Many kinds of protocol strings, such as
> URIs, IRIs, and strings complying with the PRECIS framework,
> do not allow noncharacter code points, while other kinds of
> UTF-8 text, such as JSON, do allow noncharacters.  XML allows
> all noncharacters except for U+FFFE and U+FFFF.
> 
> --Peter
> 
> From: Carsten Bormann
> thank you for the quick feedback!  As suggested, I have
> started turning this into a standalone document (of course,
> still requiring RFC 3629, the UTF-8 definition):
> 
> Status:
> https://datatracker.ietf.org/doc/draft-bormann-dispatch-modern
> -network-unicode/ Htmlized:
> https://tools.ietf.org/html/draft-bormann-dispatch-modern-netw
> ork-unicode-02 Diff:
> https://tools.ietf.org/rfcdiff?url2=draft-bormann-dispatch-mod
> ern-network-unicode-02
> 
> I hope I haven't missed anything important from RFC 5198
> that I wanted to keep.   Maybe time to involve the authors of
> that RFC…
> 
> Grüße, Carsten
> 
> 
>> Would be nicer if it wasn't written as a diff from RFC5198
>> (Network Unicode). That is, if you could get all the rules
>> directly from this doc. For instance, I assume Clean Modern
>> Network Unicode must/should be NFC. Keep the RFC5198
>> comparisons for an informative annex.
>