Subject: Re: [urn] UTF-8 ABNF Re: What Happened to UTF-8?

From: Sean Leonard <[email-masked]>
Date: Tue, 1 Dec 2015 11:18:10 -0800

On Nov 30, 2015, at 8:17 AM, Julian Reschke <[email-masked]> wrote:

> On 2015-11-30 16:52, Sean Leonard wrote:
>> ...
>>> -1. The ABNF is the wrong place to enforce constraints like these.
>> 
>> Explain? "Wrong" is not a technical argument.
>> 
>> The ABNF is not "enforcing constraints"; the standards document is (or
>> will). The ABNF accurately reflects the standard.
>> ...
> 
> But the *purpose* (IMHO) of the ABNF is not solely to "reflect the standard" but to also to assist people in writing conforming parsers.
> 
> Do you expect *parsers* to reject invalid UTF-8 sequences? I don't; I'd rather expect the consumer of the parser to inspect the octet sequences (that's why a UTF-8 decoder does, no?)
> 
> So, phrased differently, you're mixing multiple layers in the ABNF, which I believe wouldn't be helpful.

Actually after thinking about it, I respectfully but completely disagree. [The remainder of this e-mail message assumes that there is consensus about the assigned-name portion of a urn: URI being UTF-8, which is the status quo from RFC 2141. There seems to be a sub-thread with dissension in the ranks on this topic.]

The ABNF is supposed to define the behavior of conforming implementations: both parsers *and* generators. A generator that generates arbitrary "%" HEXDIG HEXDIG is non-conforming, plain and simple.

Let’s look at other IETF standards. Consider that UTF-8 itself (in the IETF) is standardized by RFC 3629. The ABNF of Section 4 of that RFC (quoted below) is normatively referenced by EAI [RFC6531] [RFC6532] (direct reference) and by LDAP [RFC4510] et. seq. including [RFC4512] (indirect reference). I am sure there are others. Therefore, a conforming LDAP system or EAI system MUST reject ill-formed UTF-8 sequences. Important features are that (well-formed) UTF-8 has no way to express different octet sequences for the same code point (overlong encoding hacks like C0 80 don’t work); furthermore, it can only express Unicode scalar values, *not* the surrogate Unicode code points U+D800-U+DFFF (in contrast to so-called Wobbly Transformation Format WTF-8, <https://en.wikipedia.org/wiki/UTF-8#WTF-8>). This means, among other things, that systems that do not check for unpaired surrogates (e.g., Windows filenames), cannot have their literal Unicode data shoved into those protocol slots. Whether this is a Good Thing or a Bad Thing, it’s a Thing and it needs to be acknowledged.

Do I expect parsers to reject invalid UTF-8 sequences? YES. It doesn’t matter if the parser is some hand-rolled thing or code automatically generated from ABNF or anything else. “Ill-formed” UTF-8 sequences are harmful for security and interoperability; there is plenty of written evidence on this point.

As a case in point: there is no reason why “%FF” or similar should ever appear in a (UTF-8) assigned-name. The octet FF never appears in UTF-8--it is out-of-band.

I have a great deal of skepticism that “the consumer of the parser [will] inspect the octet sequences”. As the Wikipedia article states: “Many systems that deal with UTF-8 work [the WTF-8] way without considering it a different encoding, as it is simpler.” Garbage-in, garbage-out. A consumer of the parsed output will likely only check for invalid UTF-8 subsequences if it is converting between UTF-8 and some other encoding, namely UTF-16 or UTF-32. Doing it “in the ABNF” is actually more efficient, because only one pass is needed to confirm validity. Requiring a system to parse the % values, and then re-parse, is not only double the work but something that most lazy/unaware/efficient implementers are unlikely to do.

Others have called for “pressure for 3986bis” [presumably, to promote UTF-8]. The proposed ABNF for percent-encoded UTF-8 may look complicated...but UTF-8 is just as “complicated” and you can abstract that out by referring to “utf8-pct-encoded” and ignoring the implementation details (as, presumably, the EAI standards did).

Consider how much easier the whole character encoding thing would have been, if URIs were always Unicode-oriented, rather than octet-oriented. We would have an unambiguous %-syntax akin to entity references like “B%FC%cher” for “Bücher”, and transitioning between URIs and IRIs would be much more seamless. Instead, we have a weird situation where that word could be maybe “B%FCcher”, maybe “B%C3%BCcher”, maybe “B%81cher”, or maybe even “xn--bcher-kva” depending on the position in the URI and the assumed character encoding in the environment.

Not only URN parts, but any URI part for other URI schemes *should* use utf8-pct-encoded when they want UTF-8 percent-encoded.

Best regards,

Sean

***
   UTF8-octets = *( UTF8-char )
   UTF8-char   = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4
   UTF8-1      = %x00-7F
   UTF8-2      = %xC2-DF UTF8-tail
   UTF8-3      = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
                 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
   UTF8-4      = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) /
                 %xF4 %x80-8F 2( UTF8-tail )
   UTF8-tail   = %x80-BF
***

> Best regards, Julian

---

From: Keith Moore <[email-masked]>
Date: Tue, 1 Dec 2015 19:34:12 -0500

On 12/01/2015 02:18 PM, Sean Leonard wrote:
> The ABNF is supposed to define the behavior of conforming implementations: both parsers*and*  generators. A generator that generates arbitrary "%" HEXDIG HEXDIG is non-conforming, plain and simple.
Nevertheless, you don't want the parser to invalidate existing URNs.

Also, it's not absolutely necessary that the grammar recognize exactly 
the set of syntactically valid URNs.   It is sufficient if the grammar 
recognizes all valid URNs but there are some strings that are valid 
under the grammar but invalid for other reasons. For instance, a URN 
isn't valid unless its namespace ID is registered, but we don't expect 
the grammar to recognize that. Similarly for most namespaces a URN isn't 
valid if its NSS hasn't been issued by the namespace, but we don't 
expect the grammar to recognize that either.

Actually I would suggest that rather than trying to impose a strict rule 
that expects all URNs to conform to UTF-8, and implicitly expects 
clients to do checking on URNs for UTF-8 validity, that instead it 
should be up to the namespace to enforce this rule at assignment time.   
Nobody else should care whether the NSS is UTF-8 or not.   The NSS 
should be considered opaque by everybody but the namespace anyway.

Finally, it's generally been my experience that overly strict parsers 
break interoperability.   If some namespace accidentally issues a NSS 
that doesn't conform to UTF-8, it does more harm than good for client 
implementations to refuse to resolve it.

Keith

---

From: Sean Leonard <[email-masked]>
Date: Wed, 2 Dec 2015 12:08:21 -0800

> Nevertheless, you don't want the parser to invalidate existing URNs.   

Correct. And such a UTF-8-enabled ABNF parser won’t invalidate existing URNs, because RFC 2141 says they are all UTF-8 percent-encoded. I don’t see any other way to interpret Section 2.2 of RFC 2141.

In the following conversation, I (going to) assume that by “URN” you mean the assigned-name production, which is NID and NSS. NID is ASCII-only. So really we’re just talking about NSS. As has been pointed out, r-components, q-components, and f-components have different encoding issues, respectively.

> 
> Also, it's not absolutely necessary that the grammar recognize exactly the set of syntactically valid URNs.   It is sufficient if the grammar recognizes all valid URNs but there are some strings that are valid under the grammar but invalid for other reasons.   For instance, a URN isn't valid unless its namespace ID is registered, but we don't expect the grammar to recognize that.   Similarly for most namespaces a URN isn't valid if its NSS hasn't been issued by the namespace, but we don't expect the grammar to recognize that either.

The other reasons you mentioned are not computable.

ABNF is a context-free grammar that can be interpreted (implemented by) a pushdown automaton. Specific NIDs and their NSS constraints are not specified in the URN grammar because they are open-ended and not known; no machine can compute such a thing. The argument is a red herring.

UTF-8 is known. Its constraints are computable by a pushdown automaton. (Actually it’s computable by a finite-state automaton, which is the “best” kind, because it can be implemented in the widest variety of resource-constrainted hardware.) Furthermore, it is a self-synchronizing code. Many character encodings—including the arcane Unicode ones that predate UTF-8—do not have these properties.

By this logic, I would say that not only is UTF-8 a “Good Idea”, UTF-8 is “Necessary” for URN interoperability. Otherwise people don’t know what characters map to what (percent-encoded) productions, making comparisons essentially impossible (i.e., uncomputable).

> 
> Actually I would suggest that rather than trying to impose a strict rule that expects all URNs to conform to UTF-8, and implicitly expects clients to do checking on URNs for UTF-8 validity, that instead it should be up to the namespace to enforce this rule at assignment time.   Nobody else should care whether the NSS is UTF-8 or not.   The NSS should be considered opaque by everybody but the namespace anyway.
> 
> Finally, it's generally been my experience that overly strict parsers break interoperability.   If some namespace accidentally issues a NSS that doesn't conform to UTF-8, it does more harm than good for client implementations to refuse to resolve it.

If an NSS is not UTF-8 encoded, it is not conforming with RFC 2141. Again, I call red herring.

Sean

---

From: Keith Moore <[email-masked]>
Date: Wed, 2 Dec 2015 17:14:42 -0500

Sent from my iPhone

> On Dec 2, 2015, at 3:08 PM, Sean Leonard <[email-masked]> wrote:
> 
> 

There were URNs in use before RFC 2141.  And it's a bad idea to assume that all URNs even after RFC 2141 was published adhered to the letter of the spec, particularly when the 2141 grammar wasn't that strict.

> In the following conversation, I (going to) assume that by “URN” you mean the assigned-name production, which is NID and NSS. NID is ASCII-only. So really we’re just talking about NSS. As has been…

Fair enough.  We should not presume anything about the character encoding of either q- or f- components because those are not within our control.

> The other reasons you mentioned are not computable.

True, but just because something can be recognized by an FSM doesn't mean we have to make the grammar strictly recognize it.

> 
> By this logic, I would say that not only is UTF-8 a “Good Idea”, UTF-8 is “Necessary” for URN interoperability.
> Otherwise people don’t know what characters map to what (percent-encoded) productions, making comparisons essentially impossible (i.e., uncomputable).

UTF-8 does not solve this problem because there is no single way of translating other character encoding schemes to Unicode.  To the extent existing namespaces are using UTF-8 to encode non-ascii characters in legacy names, they've presumably chosen their translation algorithms independently of one another.

And furthermore this is Not Our Problem, because it only comes into play when translating legacy named to URNs and vice versa.  That's inherently up to each namespace to define.  Nobody should be trying to decode or display the decoded versions of percent-encodings within an NSS unless they have namespace-specific knowledge and are trying to translate back to legacy format.  A URN is NOT some encoding of an arbitrary human-readable string; it is an opaque token.

> If an NSS is not UTF-8 encoded, it is not conforming with RFC 2141.

It doesn't matter, and that's not a good excuse for breaking non confirming URNs that might exist.

Keith

---

From: Sean Leonard <[email-masked]>
Date: Wed, 2 Dec 2015 14:41:47 -0800

> It doesn't matter, and that's not a good excuse for breaking non confirming URNs that might exist.

Remind me: what is the point of writing a standard that bends over backwards for non conforming URNs, when they never conformed in the first place?

It just seems that this WG or the chairs need to decide if the NSS is UTF-8 encoded or not. Or if it’s some convoluted well-things-not-in-pchar-are-something-else-but-we-cannot-agree-on-anything-so-we-will-make-the-world-more-convoluted-instead-of-actually-something-that-works-like-UTF-8.

The text of RFC 2141 Section 2.2 has a counterpart in urnbis Section 3.2, but the actual text is markedly different. The RFC 2141 text has already been quoted.

Here is proposed text for urnbis, in bold. Note that there are only two small changes.

***
3.2.  Namespace Specific String

   The namespace specific string (NSS) is a unique identifier that is
   assigned and managed in a consistent way and that conforms to the
   definition of the relevant namespace.  The combination of the NID
   (unique across the entire "urn" scheme) and the NSS (unique within
   the namespace) ensures that the resulting URN is a globally unique
   URI.

   This document modifies the syntax of the NSS to allow the following
   characters: "/", "~", and "&".  In particular, allowing the "/"
   character effectively makes it possible to encapsulate hierarchical
   identifiers from other naming systems.  For instance, consider the
   hypothetical example of a hierarchical naming system in which the
   identifiers take the form of a series of numbers separated by the "/"
   character, such as "1/406/47452/2".  If the naming authority for such
   identifiers were to use URNs, it would be natural to place the
   existing identifiers in the NSS, resulting in URNs such as
   "urn:example:1/406/47452/2".

   However, the foregoing changes to the syntax for the NSS do not
   modify the encoding rules for URN namespaces that were defined in
   accordance with [RFC2141].  Percent-encoded octets in the NSS,
   for example, are in UTF-8 [UNICODE] [RFC3629].
   If any such URN namespace that is used
   outside of the URN context (i.e., as a standalone identifier space)
   also allows the use of "/", "~", or "&" in the native form within
   that identifier space, then the encoding rules for that namespace are
   not changed by this specification.

*  Depending on the rules governing a namespace, names that are valid in
   a namespace might contain characters that are not allowed by the
   "pchar" production referenced above (e.g., characters outside the
   ASCII range or, consistent with the restrictions in RFC 3986, the
   characters "/", "?", "#", "[", and "]").  While such a string might
   be a valid name, it is not a valid URN until it has been translated
   into a conformant NSS.  In the case of URNs that are formed from
   names that exist separately in a standalone identifier space,
   translation of an identifier from its "native" format to URN format
   is accomplished by using the canonicalization and encoding methods
   defined for that URN namespace.  Software that is not aware of those
   namespace-specific canonicalization and encoding rules MUST NOT
   construct URNs from the names in the standalone identifier space.

   In order to make URNs as stable and persistent as possible when
   protocols evolve and the environment around them changes, namespaces
   SHOULD NOT allow characters beyond ASCII [RFC20]
   unless the nature of the particular namespace makes such
   characters necessary.

***

Go for vote.

Sean

* This paragraph, in my opinion, needs a lot of editorial work but if I contribute to that it will be in a separate message since the editorial changes are not germane to this thread.

---

From: Keith Moore <[email-masked]>
Date: Wed, 2 Dec 2015 19:21:25 -0500

On 12/02/2015 05:41 PM, Sean Leonard wrote:
>
> On Dec 2, 2015, at 2:14 PM, Keith Moore <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>>
>>
>> Sent from my iPhone
>>
>> On Dec 2, 2015, at 3:08 PM, Sean Leonard <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>>
>>>
> Remind me: what is the point of writing a standard that bends over 
> backwards for non conforming URNs, when they never conformed in the 
> first place?

Remind me: what's the point of making a change to an established 
standard for long-term stable names that might invalidate some of the 
names already assigned?   URNs are supposed to be persistent, and they 
can't be persistent if meddlers at IETF change the rules, even subtly, 
for no good reason.    (and "because we can" is not a good reason)

If we were seeing operational failures due to namespaces' use of 
encoding schemes other than UTF-8, maybe we'd need to change the rules.  
Even then we'd want to do it very carefully so as to minimize the harm 
done.   But I'm not aware of any operational failures of this nature.  
And if there were some, I expect that they would be the result of 
implementations doing things with URNs that they shouldn't do - like 
trying to pick apart an NSS without having a specification from the 
namespace for exactly how their NSSs are constructed.

>
> It just seems that this WG or the chairs need to decide if the NSS is 
> UTF-8 encoded or not. Or if it’s some convoluted 
> well-things-not-in-pchar-are-something-else-but-we-cannot-agree-on-anything-so-we-will-make-the-world-more-convoluted-instead-of-actually-something-that-works-like-UTF-8.
>
> The text of RFC 2141 Section 2.2 has a counterpart in urnbis Section 
> 3.2, but the actual text is markedly different. The RFC 2141 text has 
> already been quoted.
>
> Here is proposed text for urnbis, in *bold*. Note that there are only 
> two small changes.
>
> ***
>
>
>       3.2
>       <https://tools.ietf.org/html/draft-ietf-urnbis-rfc2141bis-urn-14#section-3.2>.
>       Namespace Specific String
>
>
>
>     accordance with [RFC2141 <https://tools.ietf.org/html/rfc2141>].*Percent-encoded octets in the NSS,*
> *for example, **are in UTF-8 [UNICODE] [RFC3629].*

No.   This is specifying things in the wrong way, and improper 
layering.   The burden of using UTF-8 is with the namespace, and the 
namespace alone.  The text implies that client implementations should be 
able to assume that percent-encoded strings in URNs are in UTF-8, and 
they really shouldn't assume that at all.

Actually I recommend that the preceding sentence be modified to read:

    However, the foregoing changes to the syntax for the NSS do not
    modify the encoding rules for existing URN namespaces.

Because it really doesn't matter to a client how a NSS was derived from 
a legacy name, and it doesn't matter whether that was done according to 
the rules in RFC2141.  If a namespace screwed up and derived its names 
in a different way, we still don't want to invalidate those names.
*
*
> SHOULD NOT allow characters *beyond ASCII [RFC20]*

I'm ambivalent about this proposal.

> Go for vote.

We don't vote in IETF.

Keith

---

From: Sean Leonard <[email-masked]>
Date: Thu, 3 Dec 2015 01:13:54 -0800

> On 12/02/2015 05:41 PM, Sean Leonard wrote:

The IETF operates on rough consensus (and ideally, running code). In 
in-person meetings, consensus can be ascertained by anonymous humming 
(for large crowds). Then the chairs figure out whether the hums. But on 
mailing lists, people's opinions are not anonymous. So that would be 
what I mean by voting: you have to write in and say what you want or 
don't want.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Thu, 3 Dec 2015 01:24:35 -0800

Corrected:

> (for large crowds). Then the chairs figure out whether the hums are 
> pointing to rough consensus in one direction or another. But on

---

From: Sean Leonard <[email-masked]>
Date: Thu, 3 Dec 2015 10:07:51 -0800

Good news!

I reviewed every URN registration, formal and informal, in the IANA URN 
Namespaces registry [1]. Every single one was registered under the rules 
of RFC 2141 / RFC 3406, after those were standardized. So that includes 
UTF-8 encoding.

RFC 2141 is the operating standard and has the IETF consensus; all 
namespace registrants are bound to follow it, including UTF-8 encoding. 
There is no evidence that any namespace registrants are bound to prior 
or different rules. If you want to change the UTF-8 part of RFC 2141 
[bis], it needs a different IETF consensus. Otherwise this whole UTF-8 
issue is just an editorial matter.

Should anyone have a different view, documented evidence is necessary to 
construct a credible argument.

Implementers can rely on the premise that the NSS percent-encoded 
characters are UTF-8 encoded. (Conveniently that makes IRIs directly 
transposable, at least for the NSS component. The other proposed 
components like f-, q-, and r-, not necessarily, but that's 
out-of-scope.) URNs are supposed to be persistent, and they can't be 
persistent if meddlers at IETF change the rules, even subtly, for no 
good reason.

Regards,

Sean

[1] http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml

---

From: Keith Moore <[email-masked]>
Date: Fri, 4 Dec 2015 08:32:46 -0500

On 12/03/2015 01:07 PM, Sean Leonard wrote:
> Good news!
>
> I reviewed every URN registration, formal and informal, in the IANA 
> URN Namespaces registry [1]. Every single one was registered under the 
> rules of RFC 2141 / RFC 3406, after those were standardized. So that 
> includes UTF-8 encoding.
>
> namespace registrants are bound to follow it, including UTF-8 
> encoding. There is no evidence that any namespace registrants are 
> bound to prior or different rules. If you want to change the UTF-8 
> part of RFC 2141 [bis], it needs a different IETF consensus. Otherwise 
> this whole UTF-8 issue is just an editorial matter.
>
> Should anyone have a different view, documented evidence is necessary 
> to construct a credible argument.
>

Disagree.   Not convinced, for reasons cited earlier.  Not going to keep 
repeating myself either.

Keith