Subject: [urn] What Happened to UTF-8?

From: Sean Leonard <[email-masked]>
Date: Thu, 26 Nov 2015 09:37:48 -0800

Hello...

According to:
https://tools.ietf.org/rfcdiff?url2=draft-ietf-urnbis-rfc2141bis-urn-04.txt

Sometime between Draft 4 (October 16, 2012) and Draft 5 (November 27, 
2012), UTF-8 encoding got removed. I guess that means that now it's up 
to the namespace to define what encoding is used, if any? (Unicode is 
still mentioned, but the relationship to percent-encoding is absent.)

What is up with that? Just want to know if that was a conscious choice 
by this WG, or what happened.

Regards,

Sean

---

From: Martin J. Dürst <[email-masked]>
Date: Sun, 29 Nov 2015 13:06:29 +0900

Hello Sean,

This is a very good catch.

On 2015/11/27 02:37, Sean Leonard wrote:
> 2012),

The dates are correct, but it's draft 3 and draft 4, as the diff URI shows.

The change happened with a change of author/editorship from Alfred Hönes 
to Peter Saint-Andre. Probably, Peter can explain best.

> UTF-8 encoding got removed. I guess that means that now it's up
> to the namespace to define what encoding is used, if any?

That would be a very bad idea. UTF-8  was explicitly put into RFC 2141 
to avoid useless/confusing variability for namespaces that originally 
contain non-ASCII characters and therefore need to be mapped to US-ASCII.

This definitely should be fixed asap.

Regards,   Martin.

> (Unicode is
> _______________________________________________
> urn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/urn
>

---

From: John C Klensin <[email-masked]>
Date: Sun, 29 Nov 2015 11:53:06 -0500

Sean, Martin, and others...

--On Sunday, November 29, 2015 1:06 PM +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>> Sometime between Draft 4 (October 16, 2012) and Draft 5
>> (November 27, 2012),
> 
> The change happened with a change of author/editorship from
> Alfred Hönes to Peter Saint-Andre. Probably, Peter can
> explain best.

Agreed.  However, my vague recollection is that there was an
explicit WG discussion on that matter, probably tied up with the
observation that the 2141 text was a little too close to the
now-discredited "just sand UTF-8 and all i18n issues will go
away" position.

>> UTF-8 encoding got removed. I guess that means that now it's
>> up to the namespace to define what encoding is used, if any?
> 
> That would be a very bad idea.

Yes, no question about that, at least IMO.  If any non-ASCII
encoding it to be used, it fairly clearly should be UTF-8.

> UTF-8  was explicitly put into
> RFC 2141 to avoid useless/confusing variability for namespaces
> that originally contain non-ASCII characters and therefore
> need to be mapped to US-ASCII.

I believe that is correct too.

> This definitely should be fixed asap.

Unfortunately, the fix does not appear to be as obvious as a
sentence that says "any non-ASCII octets MUST BE valid UTF-8" in
some appropriate place in the text.  As soon as we move away
form ASCII, we reintroduce issues with string comparison,
normalization, and case-folding.  Those issues imply a choice
between increased risk or false positives or false negatives,
potentially involving us with all of the issues that have
plagued PRECIS in recent months and a requirement to revisit our
comparison rules.

Or course. we could be draconian about things and say that two
URNs are equal only if the NID and NSS strings are exactly
identical at the octet level, but that implies that 

    urn:XXX:somestring-in-NFD-form 
will, in general, not be treated as equal to 
    urn:XXX:somestring-in-NFC-form

Maybe that is ok, but, if it what the WG wants, I'd like to be
sure before altering text.  And, if it isn't what the WG wants,
we need a plan.

So "asap" may not be quite as fast as one might think at first
glance.

Peter?  Andy?

     john

---

From: Sean Leonard <[email-masked]>
Date: Sun, 29 Nov 2015 09:17:08 -0800

> The dates are correct, but it's draft 3 and draft 4, as the diff URI 

Yes, thanks. I think I got confused with the ordinal version of the file 
(4th version / 5th version), as opposed to the cardinal number.

>
> The change happened with a change of author/editorship from Alfred 
> Hönes to Peter Saint-Andre. Probably, Peter can explain best.
>

Yes I agree 100%.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sun, 29 Nov 2015 09:25:22 -0800

On 11/29/2015 8:53 AM, John C Klensin wrote:
> identical at the octet level [...]

It's really not all that complicated. The original standard RFC 2141 
says it's UTF-8. Therefore anything that is RFC 2141-compliant will be 
RFC 2141bis-compliant, if we keep it the same way as it used to be.

Comparison of the identifier (aka, what hopefully now we can start 
calling "assigned-name" uniformly) was always specific to the namespace. 
Therefore things like Unicode case folding track existing 
namespace-specific issues, such as whether the NSS is case-sensitive.

Note that q-components, f-components, and (possibly) r-components need 
not be UTF-8. This introduces some subtleties in smushing problems 
(e.g., smushing q-component into a query component of another URI 
produced by resolution) but those are not more complicated than the 
ill-defined "smushing" operation itself. Regarding r-component: in 
-14-seantek I was very careful to define r-params as UTF-8, rop as 
ASCII-only (no percent encoding), and r-authority as whatever-reg-name-is.

Sean

---

From: Peter Saint-Andre <[email-masked]>
Date: Sun, 29 Nov 2015 14:58:20 -0700

> So "asap" may not be quite as fast as one might think at first

I agree that we need to discourage non-UTF-8 encodings, but I also agree 
that there can be devils lurking in the details. Thankfully we have 
enough i18n knowledge and experience here to get it right - well, as 
much as one can get things right when it comes to i18n...

Peter

---

From: John C Klensin <[email-masked]>
Date: Sun, 29 Nov 2015 21:59:58 -0500

--On Sunday, November 29, 2015 2:58 PM -0700 Peter Saint-Andre
<[email-masked]> wrote:

>...

>> sentence that says "any non-ASCII octets MUST BE valid UTF-8"
>> in some appropriate place in the text.  As soon as we move
>> away form ASCII, we reintroduce issues with string comparison,
>> plagued PRECIS in recent months and a requirement to revisit
>> our comparison rules.
>> 
>> sure before altering text.  And, if it isn't what the WG
>> wants, we need a plan.
>> 
 
> I agree that we need to discourage non-UTF-8 encodings, but I
> also agree that there can be devils lurking in the details.
> Thankfully we have enough i18n knowledge and experience here
> to get it right - well, as much as one can get things right
> when it comes to i18n...

Hi.  I am really reluctant to open new cans of worms if we can
avoid them.  To that end, because the relevant text points to
RFC 3986 for encoding discussions, I went back and looked at it.
It seems to strongly recommend UTF-8 (and nothing else) in
several places, but stops short of making that a requirement.
How would people feel about our simply going a half-step further
and saying something equivalent to "when characters outside the
ASCII repertoire are needed UTF-8 MUST be used as a basis for
coding and conversion, otherwise the rule are as specified in
RFC 3986".  I'd encourage everyone to recheck Section 4 of
2141bis to be sure it is comprehensive enough to avoid further
problems (and to suggest text if it is not), but, if it is, this
may be something we want to pass onto 3986 as much as possible.
If that creates more pressure for 3986bis, so much the better.

    john

---

From: Sean Leonard <[email-masked]>
Date: Sun, 29 Nov 2015 23:10:01 -0800

On 11/29/2015 6:59 PM, John C Klensin wrote:
>> [...]
> RFC 3986".

Ok, but how is that different from RFC 2141, appropriately massaged to 
reflect 2015:

RFC 2141:

    Depending on the rules governing a namespace, valid identifiers in a
    namespace might contain characters that are not members of the URN
    character set above (<URN chars>).  Such strings MUST be translated
    [...]. Translation is done
    by encoding each character outside the URN character set as a
    sequence of one to six octets using UTF-8 encoding [5 <http://tools.ietf.org/html/rfc2141#ref-5>], and the
    encoding of each of those octets as "%" followed by two characters
    from the <hex> character set above. The two characters give the
    hexadecimal representation of that octet.

-Sean

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 30 Nov 2015 16:28:19 +0900

On 2015/11/30 16:10, Sean Leonard wrote:

>     sequence of one to six octets using UTF-8 encoding [5
> <http://tools.ietf.org/html/rfc2141#ref-5>], and the

If we care about 2015, then please change "one to six octets" to "one to 
four octets". Also, RFC 3986 says that for hex characters (A-F), upper 
case should be used. That convention wasn't fixed yet in the RFC 2141 
time frame, but may lead to less false negatives when matching.

Regards,  Martin.

---

From: Sean Leonard <[email-masked]>
Date: Sun, 29 Nov 2015 23:46:46 -0800

On 11/29/2015 11:28 PM, Martin J. Dürst wrote:
> If we care about 2015, then please change "one to six octets" to "one 
> to four octets". Also, RFC 3986 says that for hex characters (A-F), 
> upper case should be used. That convention wasn't fixed yet in the RFC 
> 2141 time frame, but may lead to less false negatives when matching.

Yes. +1.

Except one thing: yes, URN producers and normalizers should use 
uppercase hex. However, it should not lead to false negatives when 
matching, because percent-encoded productions are case-insensitive. 
I.e., the case-insensitivity a URI thing (and an ABNF thing)--a URN 
comparison function that treats %7F as different from %7f is incorrect.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Mon, 30 Nov 2015 00:04:16 -0800

While we are at it, allow me to provide the ABNF of UTF-8 
percent-encoded octets:

utf8-pct-encoded = utf8-1-pct-encoded / utf8-2-pct-encoded / 
utf8-3-pct-encoded / utf8-4-pct-encoded

utf8-1-pct-encoded = "%" %x30-37 HEXDIG

utf8-2-pct-encoded = ("%C" (%x32-39 / %x41-46 / %x61-66) /  "%D" HEXDIG) 
utf8-tail-pct-encoded

utf8-3-pct-encoded = "%E0%" ("A" / "B") HEXDIG utf8-tail-pct-encoded / 
"%E" (%x31-39 / "A" / "B" / "C") 2( utf8-tail-pct-encoded ) / "%ED%" 
("8" / "9") HEXDIG utf8-tail-pct-encoded / "%E" ("E" / "F") 2( 
utf8-tail-pct-encoded )

utf8-4-pct-encoded = "%F0%" ("9" / "A" / "B") HEXDIG 2( 
utf8-tail-pct-encoded ) / "%F" %x31-33 3( utf8-tail-pct-encoded ) / 
"%F4%8" HEXDIG 2( utf8-tail-pct-encoded )

utf8-tail-pct-encoded = "%" ("8" / "9" / "A" / "B") HEXDIG

Replace "pct-encoded" with "utf8-pct-encoded" for appropriate parts of 
RFC2141bis.

Sean

PS This was straightforward to derive from RFC 3629.

---

From: Sean Leonard <[email-masked]>
Date: Mon, 30 Nov 2015 00:06:32 -0800

EDITORIAL fix:

Section 4.1:

    3.  any percent-encoded characters in the NSS (that is, all character
        triplets that match the <pct-encoding> production found in
        Section 2.1 
<https://tools.ietf.org/html/draft-ietf-urnbis-rfc2141bis-urn-14#section-2.1>  of the base URI specification [RFC3986 <https://tools.ietf.org/html/rfc3986>]), by

is wrong.

It should read "<pct-encoded>". pct-encoding is not a thing in RFC 3986.

But (somewhat less editorially), it should say <utf8-pct-encoded>, per 
my immediately preceding message. We are restricting NSS to UTF-8 so it 
is sloppy for the ABNF to permit all hex combinations in the <NSS> 
production.

Sean

---

From: Julian Reschke <[email-masked]>
Date: Mon, 30 Nov 2015 09:43:53 +0100

> PS This was straightforward to derive from RFC 3629.

-1. The ABNF is the wrong place to enforce constraints like these.

Best regards, Julian

---

From: Sean Leonard <[email-masked]>
Date: Mon, 30 Nov 2015 07:52:23 -0800

> -1. The ABNF is the wrong place to enforce constraints like these.

Explain? "Wrong" is not a technical argument.

The ABNF is not "enforcing constraints"; the standards document is (or 
will). The ABNF accurately reflects the standard.

Sean

---

From: Julian Reschke <[email-masked]>
Date: Mon, 30 Nov 2015 17:17:12 +0100

> will). The ABNF accurately reflects the standard.

But the *purpose* (IMHO) of the ABNF is not solely to "reflect the 
standard" but to also to assist people in writing conforming parsers.

Do you expect *parsers* to reject invalid UTF-8 sequences? I don't; I'd 
rather expect the consumer of the parser to inspect the octet sequences 
(that's why a UTF-8 decoder does, no?)

So, phrased differently, you're mixing multiple layers in the ABNF, 
which I believe wouldn't be helpful.

Best regards, Julian

---

From: Keith Moore <[email-masked]>
Date: Mon, 30 Nov 2015 21:41:04 -0500

I don't immediately see how to reconcile the desire for UTF-8 in URNs, 
with some fundamental URN requirements from RFC 1737 section 3:

    o Single encoding: The encoding for presentation for people in clear
      text, electronic mail and the like is the same as the encoding in
      other transmissions.

    o Simple comparison: A comparison algorithm for URNs is simple,
      local, and deterministic. That is, there is a single algorithm for
      comparing two URNs that does not require contacting any external
      server, is well specified and simple.

    o Human transcribability: For URNs to be easily transcribable by
      humans without error, they should be short, use a minimum of
      special characters, and be case insensitive. (There is no strong
      requirement that it be easy for a human to generate or interpret a
      URN; explicit human-accessible semantics of the names is not a
      requirement.)  For this reason, URN comparison is insensitive to
      case, and probably white space and some punctuation marks.

    o Transport friendliness: A URN can be transported unmodified in the
      common Internet protocols, such as TCP, SMTP, FTP, Telnet, etc., as
      well as printed paper.

Also, to the extent that any existing URN-eligible namespaces use 
characters outside the ASCII repertoire, the "legacy support" 
requirement might conflict with a desire to coerce those non-ASCII 
characters into UTF-8.   (I'm not saying that this couldn't be solved, 
it it sounds unnecessarily complex, with each namespace using non-ASCII 
characters in its NSS potentially having to describe how to map those 
characters unambiguously into UTF-8)

    o Legacy support: The scheme must permit the support of existing
      legacy naming systems, insofar as they satisfy the other
      requirements described here. For example, ISBN numbers, ISO
      public identifiers, and UPC product codes seem to satisfy the
      functional requirements, and allow an embedding that satisfies
      the syntactic requirements described here.

In addition, if you're trying to encode a significant amount of 
human-readable text in a URN, you're likely to be harming that URN's 
persistence - i.e. its ability to be bound to a single resource forever, 
because the meanings of human-readable words change over time.

So I seriously doubt that it's appropriate to try to impose UTF-8 on 
URNs.    URNs aren't intended to encode human-meaningful text. This is 
not a requirement of URNs, and trying to make URNs be good for that 
impairs their intended function.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 30 Nov 2015 21:42:16 -0500

On 11/29/2015 12:25 PM, Sean Leonard wrote:
> ill-defined "smushing" operation itself.

Since "smushing" is completely inappropriate, this isn't a problem.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 30 Nov 2015 21:43:17 -0500

> coding and conversion, otherwise the rule are as specified in
strongly opposed.    I have yet to see that this is a problem that needs 
solving, and which won't harm URNs if we try to solve it.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 30 Nov 2015 21:56:16 -0500

Ok, I'm looking at 2141 text regarding use of UTF-8 in NSS.   I think it 
was a mistake for 2141 to mention UTF-8 at all, but trying to fix that 
now would be an even greater mistake.   So it seems to me that the right 
thing to do going forward is to clarify 2141's language to be less 
ambiguous while still being consistent with 2141 and with all names 
assigned using 2141 rules.

    Depending on the rules governing a namespace, valid identifiers in a
    namespace might contain characters that are not members of the URN
    character set above (<URN chars>).  Such strings MUST be translated
    into canonical NSS format before using them as protocol elements or
    otherwise passing them on to other applications.

IMO this is fine up to this point and should not be changed.

    Translation is done
    sequence of one to six octets using UTF-8 encoding [5], and the

The problem I see with this is that it's ambiguous.  For many character 
encoding schemes there's not a single way to translate them into UTF-8, 
and different namespaces may have used different conventions.   So the 
best thing I know to do at this point is to require any legacy namespace 
that assigns NSSs containing non-ASCII characters to specify the 
algorithm by which those characters are translated into UTF-8.   Really 
this should be part of a general specification for translating legacy 
names into NSSs, because there are also cases in which a NSS might use 
an ASCII character that's not permitted by 2141 or 3986 syntax.

Keith