Subject: [rfc-i] Unicode in ABNF (in RFC) draft-seantek-unicode-in-abnf-01.txt

From: Sean Leonard <[email-masked]>
Date: Sun, 2 Oct 2016 23:28:37 -0700

Dear ABNF-Discuss (and rfc-interest):

This draft by Chris Newman and I addresses an interesting topic: how to 
do Unicode in ABNF. Unicode has showed up in several different ways in 
protocols that are described in ABNF. These ways are not consistent 
across the RFC series, but now that Unicode is a pretty stable standard 
(for its basic parts) and now that UTF-8 RFCs are becoming a reality per 
draft-iab-rfc-nonascii-02, it is a good time to look at this issue. This 
is a fork from draft-seantek-abnf-more-core-rules.

This draft is currently proposed as Experimental. Special thanks to Paul 
Kyzivat for discussing the matters in this draft, although he is not 
formally a co-author.

The draft tries to be very conservative in its approach. Please read the 
draft for details. Some stuff was intentionally omitted as out-of-scope 
or too complicated for a general-purpose ABNF syntax parser, whether 
humans or machines.

Comments and feedback are appreciated.

Regards,

Sean

********

A new version of I-D, draft-seantek-unicode-in-abnf-01.txt
has been successfully submitted by Sean Leonard and posted to the
IETF repository.

Name:		draft-seantek-unicode-in-abnf
Revision:	01
Title:		Unicode in ABNF
Document date:	2016-10-01
Group:		Individual Submission
Pages:		11
URL:            https://www.ietf.org/internet-drafts/draft-seantek-unicode-in-abnf-01.txt
Status:         https://datatracker.ietf.org/doc/draft-seantek-unicode-in-abnf/
Htmlized:       https://tools.ietf.org/html/draft-seantek-unicode-in-abnf-01
Diff:           https://www.ietf.org/rfcdiff?url2=draft-seantek-unicode-in-abnf-01

Abstract:
    This experimental document adds support for Unicode strings in ABNF
    (Augmented Backus-Naur Form), and provides certain symbols related to
    Unicode code point ranges.

_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 3 Oct 2016 18:53:55 +0900

Hello Sean,

A few quick comments from a cursory reading:

First, I note that the choice you have made for representing Unicode 
codepoints seems to be the same that we made for RFC 3987, which is the 
one that I think RFC 5234 and its predecessors also implicitly suggest. 
If you have seen some discrepancies, I would appreciate a pointer. You 
may also want to reference some of the

In the Introduction, you mention security problems, but they are not 
detailed (no specifics, no examples) there and neither in the Security 
Considerations section.

In contrast to ASCII, Unicode (in any of its encoding forms) essentially 
introduces multiple levels at which protocols can be described: bytes, 
[code units (in the case of UTF-16xx),] code points, grapheme 
clusters,... I'm fine with limiting this document to the code point 
level, which is clearly what we need now, but it would be good to say 
somewhere at least that this document doesn't deal with other levels.

Starting sections/paragraphs with parentheticals (e.g. "(Consult Section 
2.3 of [RFC5234] in relation to this paragraph.)") is far away from good 
writing. At the minimum, put these parentheticals at the end of the 
paragraphs, but even better would be to convert them to actual text (in 
most cases still at the end of the paragraphs) and say explicitly what 
the "relation" is. (RFC 7405 looks much better in this respect.)

In the appendix, there are a lot of mostrosities such as 
"UVCHARBEYONDLATIN1". Why not change that to something a bit more 
readable, at the minimum something like UV_CHAR_BEYOND_LATIN_1 or so?

I don't see the point of defining aliases for C1 controls; it should be 
difficult to use these explicitly, not easy.

For some of the aliases, a property-based approach seems to be the right 
thing to do, although this may be difficult to align with the ABNF 
straightjacket.

The draft says:
    Formally, this document updates [RFC5234] but does not modify it in
    situ. Authors need to reference this document if they want to include
    these enhancements; bare references to [RFC5234] do not include this
    specification (or, for that matter, [RFC7405]).
There's no text whatsoever in RFC 7405 that would say that it doesn't 
update RFC 5234 directly. But I may be missing something. Please clarify.

I don't see the need to use %su for Unicode strings. The code points 
speak for themselves, just use %s. Leaving %i/%iu undefined for Unicode 
is indeed advisable, although it could be based on default case folding, 
but we know that this would be imperfect, in particular for Turkish.

Section 6 uses an example with actual Unicode characters. I'd definitely 
wait for the new way of publishing drafts/RFCs before the final 
publication of this document, so that this example (and hopefully a few 
more) can use actual Unicode characters.
(I'd also change 'notated' to 'annotated'. (several occurrences))

That's about it, hope it helps.

Regards,   Martin.

On 2016/10/03 15:28, Sean Leonard wrote:
> URL:
> https://www.ietf.org/internet-drafts/draft-seantek-unicode-in-abnf-01.txt
> Status:
> https://datatracker.ietf.org/doc/draft-seantek-unicode-in-abnf/
> Htmlized:
> https://tools.ietf.org/html/draft-seantek-unicode-in-abnf-01
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-seantek-unicode-in-abnf-01
>
> Abstract:
> .

-- 
Martin J. Dürst
Department of Intelligent Information Technology
Collegue of Science and Engineering
Aoyama Gakuin University
Fuchinobe 5-1-10, Chuo-ku, Sagamihara
252-5258 Japan
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Sean Leonard <[email-masked]>
Date: Mon, 3 Oct 2016 16:44:03 -0700

Thanks for checking it out!

> On Oct 3, 2016, at 2:53:55.000AM, Martin J. Dürst <[email-masked]> wrote:
> 
> Hello Sean,
> 
> First, I note that the choice you have made for representing Unicode codepoints seems to be the same that we made for RFC 3987, which is the one that I think RFC 5234 and its predecessors also implicitly suggest. If you have seen some discrepancies, I would appreciate a pointer. You may also want to reference some of the

? got cut off?

The main alternate syntax is direct UTF-8 encoding, where each terminal symbol represents an octet. I did a regular expression search through the RFC series for VCHAR and friends (%x21-7E, %d33-126, permutations thereof, etc.), and saw a lot come up.

See RFC 3629:

   UTF8-octets = *( UTF8-char )
   UTF8-char   = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4
   UTF8-1      = %x00-7F
   UTF8-2      = %xC2-DF UTF8-tail
   UTF8-3      = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
                 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
   UTF8-4      = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) /
                 %xF4 %x80-8F 2( UTF8-tail )
   UTF8-tail   = %x80-BF

Some RFCs cite RFC 3629’s ABNF directly. Others copy it. Yet others use some variation that boils down to UTF-8 in intent, but is not the modern formulation of UTF-8. For example, I have seen RFCs (mostly pre-2000) that allow up to 6 octets, formatted according to pre-modern UTF-8 (and also, notably, that permit “ill-formed” encodings). Some RFCs are really lazy and say *(%x80-FF) is “UTF-8”.

The approach to specify actual UTF-8 octets is not necessarily “wrong”. Some might argue it is “better” or has certain things to commend to it. I could go on with various arguments for and against, but suffice to say, it’s different.

That style of specifying Unicode is not addressed in draft-seantek-unicode-in-abnf-01. We may well address it in the next draft.

> 
> In the Introduction, you mention security problems, but they are not detailed (no specifics, no examples) there and neither in the Security Considerations section.

I will let Chris Newman chime in on that one. :)

> 
> In contrast to ASCII, Unicode (in any of its encoding forms) essentially introduces multiple levels at which protocols can be described: bytes, [code units (in the case of UTF-16xx),] code points, grapheme clusters,... I'm fine with limiting this document to the code point level, which is clearly what we need now, but it would be good to say somewhere at least that this document doesn't deal with other levels.

Good point; yes that should be mentioned explicitly.

> 
> Starting sections/paragraphs with parentheticals (e.g. "(Consult Section 2.3 of [RFC5234] in relation to this paragraph.)") is far away from good writing. At the minimum, put these parentheticals at the end of the paragraphs, but even better would be to convert them to actual text (in most cases still at the end of the paragraphs) and say explicitly what the "relation" is. (RFC 7405 looks much better in this respect.)

Okay.

Basically that part was trying to get at the relationship between “terminal values” (which are just non-negative integers in ABNF land) and “external encodings” (bits and bytes on the wire). An assumption crops up, when dealing with an “ASCII”-oriented protocol, that we are dealing in 8-bit bytes/code units and that the high-order byte is always 0 when it’s an ASCII encoding. This is because the 8-bit byte “won” in the format wars, but it was not always the case. When dealing with Unicode code points and characters, 8-bit byte != code unit. This is related to the UTF-8 issue above.

> 
> In the appendix, there are a lot of mostrosities such as "UVCHARBEYONDLATIN1". Why not change that to something a bit more readable, at the minimum something like UV_CHAR_BEYOND_LATIN_1 or so?

I don’t know about “monstrosities”; how about “gnarly” ? 🤘✌️🏄

It’s worth a discussion.

The low line _ is not a valid ABNF rule name character. Only ALPHA, DIGIT, and - are permitted. One could use - I suppose.

Appendix A is a set of Core Rules, just like RFC 5234 Appendix B.1. The aesthetic preference for ABNF’s Core Rules appears to be SHORTALLCAPSWITHNODELIMITERS. Examples: CHAR, CRLF, DQUOTE, HEXDIG, LWSP, VCHAR, WSP.

Basically, draft-01’s Appendix A tries to preserve that aesthetic. Hence: ASCII -> UNICODE, VCHAR -> UVCHAR, CHAR -> UCHAR, etc. The new thing is specifying ranges beyond the “Basic” ranges of %x00-7F (ASCII) %x00-9F (C1 controls), %x00-FF (Latin-1), and %x00-FFFF (BMP).

I also admit that I was in a Star Trek mood. 🖖 (But I did have a discussion about “Beyond” sometime last year on the Unicode mailing list.)

> 
> I don't see the point of defining aliases for C1 controls; it should be difficult to use these explicitly, not easy.

Basically I respectfully disagree about this.

Some people love to hate on the control characters, but people and protocols do use them. For some protocols (JSON-text-sequence), a control character (RS) turns out to be the natural choice. The same could be said about sentinel characters (aka non-characters). Judicious use of unused code points in a protocol, means you don’t have to escape or quote, which means less processing power and less risk of security bugs due to improper escaping or quoting.

The C1 controls are not used nearly as much; however, that is all the more reason to use them for out-of-band purposes (as RS is used in JSON text sequence). Another way to put it is: “why would anyone define an ABNF rule <SS2> that means anything other than SINGLE SHIFT 2”? Essentially this draft makes it a point to discourage that sort of thing, by having a common language.

We can discuss whether it makes sense to define a larger number of rules for the undisplayable characters. Beyond C1 (compare with C0 in draft-seantek-abnf-more-core-rules), the only ones I/we defined are NBSP (obvious), SHY (obvious), and LS and PS (obvious because SP and HT / HTAB are defined). I think that is a good number. For line-oriented protocols, PS should never appear, hence another good reason to call it out.

> 
> For some of the aliases, a property-based approach seems to be the right thing to do, although this may be difficult to align with the ABNF straightjacket.

Correct.

> There's no text whatsoever in RFC 7405 that would say that it doesn't update RFC 5234 directly. But I may be missing something. Please clarify.

I understand that there is some kind of crazy thread on [email-masked] about “what is the meaning of an RFC Update?” that I am not participating in because I am not presently subscribed to that list. Whatever comes out of that conversation is probably relevant to that text.

I think the point is that if an RFC references [RFC7405], then %s"foo" and %i"foo" are fair game in the ABNF. But if it only references [RFC5234], then that syntax is not supposed to appear. Ditto here (in intent).

> 
> Section 6 uses an example with actual Unicode characters. I'd definitely wait for the new way of publishing drafts/RFCs before the final publication of this document, so that this example (and hopefully a few more) can use actual Unicode characters.

Yes, that’s fine.

> (I'd also change 'notated' to 'annotated'. (several occurrences))

Okay; I will take a look.

Best regards,

Sean

(%su"foo" issue in next reply)

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 4 Oct 2016 13:05:36 +0900

Hello Sean,

>> First, I note that the choice you have made for representing Unicode codepoints seems to be the same that we made for RFC 3987, which is the one that I think RFC 5234 and its predecessors also imp…

Yes, I meant to say that you might want to cite some of these RFCs as 
examples.

> The approach to specify actual UTF-8 octets is not necessarily “wrong”. Some might argue it is “better” or has certain things to commend to it. I could go on with various arguments for and against,…

I agree with "not necessarily wrong". I think the main point is that it 
may be appropriate in a protocol that's otherwise just binary.

The main problem is not only that one has to repeat the definition of 
UTF-8 in terms of bytes, but that it becomes more and more difficult to 
specify various categories of characters.

> That style of specifying Unicode is not addressed in draft-seantek-unicode-in-abnf-01. We may well address it in the next draft.

If you mean that you point to the advantages and disadvantages of such a 
style, I'm all for it. But I don't think you need to go any further.

> Basically that part was trying to get at the relationship between “terminal values” (which are just non-negative integers in ABNF land) and “external encodings” (bits and bytes on the wire). An ass…

I didn't check what the relationship was in particular, but rather than 
just point to a section in RFC 5234, it might be better to say 
explicitly what the relationship is.

> The low line _ is not a valid ABNF rule name character. Only ALPHA, DIGIT, and - are permitted. One could use - I suppose.

If the '_' is prohibited, but '-' is okay, then please use '-'.

> Appendix A is a set of Core Rules, just like RFC 5234 Appendix B.1. The aesthetic preference for ABNF’s Core Rules appears to be SHORTALLCAPSWITHNODELIMITERS. Examples: CHAR, CRLF, DQUOTE, HEXDIG,…

That "aesthetic" preference makes sense for the actual examples because 
they are extremely short. But it makes less and less sense the longer 
these become.

> Basically, draft-01’s Appendix A tries to preserve that aesthetic. Hence: ASCII -> UNICODE, VCHAR -> UVCHAR, CHAR -> UCHAR, etc. The new thing is specifying ranges beyond the “Basic” ranges of %x00…

For UCHAR, and maybe even for UVCHAR, it should work. But for longer 
ones, the eyes and the brains of the readers would definitely appreciate 
some help with parsing.

> I also admit that I was in a Star Trek mood. 🖖 (But I did have a discussion about “Beyond” sometime last year on the Unicode mailing list.)

As long as the BEYOND is clearly separated and thus easily recognizable, 
I'm fine with it.

> Some people love to hate on the control characters, but people and protocols do use them.

Yes, but how often?

> For some protocols (JSON-text-sequence), a control character (RS) turns out to be the natural choice.

Yes, but how many similar protocols are there?
(I already think that JSON-text-sequence was overkill; it could have 
been done as a zip file or as a JSON array. But that's not the point here.)
We should have actual usage examples (and not just one or two) to 
support listing all these definitions. A format that decides it needs RS 
can always just define it locally at no additional cost to the whole 
community.

> The same could be said about sentinel characters (aka non-characters).

Non-characters may be used internally in an application. So NEVER use 
them in an actual protocol.

> Judicious use of unused code points in a protocol,

Unused code points may be assigned sooner or later, or may have been set 
aside for a specific purpose. If not, then they may still be used by 
somebody already. Using them as separators in the hope that nobody else 
uses them is just betting on unlucky circumstances.

> means you don’t have to escape or quote, which means less processing power and less risk of security bugs due to improper escaping or quoting.

Improper escaping is a problem that can be fixed. Missing escaping when 
it may still be needed is more difficult to fix, so we better don't 
start it.

> The C1 controls are not used nearly as much; however, that is all the more reason to use them for out-of-band purposes (as RS is used in JSON text sequence). Another way to put it is: “why would an…

The main point is not "why would anybody use SS2 for anything other than 
SINGLE SHIFT 2?". The main point is "if anybody ever gets the idea of 
using SS2, then let that be their problem, let's not pretend we suggest 
using SS2 by giving it an alias." Unicode already lists SS2 for U+008E. 
Everybody interested in knowing that can easily find it. The common 
language is already there. We don't need to repeat it for the odd case 
that somebody eventually may use it.

By the way, "gov't health warning: figment" is difficult to understand, 
better be more explicit.

> We can discuss whether it makes sense to define a larger number of rules for the undisplayable characters. Beyond C1 (compare with C0 in draft-seantek-abnf-more-core-rules), the only ones I/we defi…

Why don't you just mention a convention for converting unicode names 
into ABNF aliases, and leave it to specifications who need such aliases 
to define them?

> I understand that there is some kind of crazy thread on [email-masked] about “what is the meaning of an RFC Update?” that I am not participating in because I am not presently subscribed to that lis…

I see. Let's wait for that, then.

Regards,   Martin.

---

From: Sean Leonard <[email-masked]>
Date: Tue, 4 Oct 2016 11:08:33 -0700

On 10/3/2016 2:53 AM, Martin J. Dürst wrote:
> speak for themselves, just use %s. Leaving %i/%iu undefined for 
> Unicode is indeed advisable, although it could be based on default 
> case folding, but we know that this would be imperfect, in particular 
> for Turkish.

I like %su because it notifies the reader, and a parser, to expect UTF-8 
and "deal with it" in a way that %s alone doesn't. For example, accented 
e can be é (U+00E9) or é (U+0065 U+0301). When printed or in a medium 
that doesn't provide direct access to the encoded data (screenshot? 
mobile app? etc.), the quoted string is ambiguous. Saying %s"foo" means 
you know that foo is always in the ASCII range, and can't possibly be 
composed of anything else (including, for example, FULLWIDTH ASCII in 
U+FF00-U+FF5E). Are %s"foo" and %s"ｆｏｏ" the same? How about %s"·˙•․‥…‧"? 
%s"µ" and %s"μ"? And the bajillion different dashes? Then there is the 
issue that even if a code point is objectively, graphically distinct in 
this version of Unicode, some future version may assign a code point to 
a character that commonly looks exactly the same as an existing character.

Responding to your point, defining %su"" would mean %s"" is undefined 
for Unicode, which avoids the temptation of %i"" (or nothing "" aka the 
traditional approach). Perhaps if a need develops for a case-insensitive 
version in a protocol, %iu could take a parameter that indicates the 
language tailoring, such as %iu[tr]"çilek". (But, I suppose, one could 
make a converse argument that %i[tr]"çilek" would be a natural evolution.)

Those are a couple of arguments. I am happy to go with whatever (rough) 
consensus emerges, however.

Regards,

Sean