Subject: Re: [Ltru] [apps-discuss] Fwd: Defining a CBOR tag for RFC 5646 Language Tags

From: "Doug Ewell" <[email-masked]>
Date: Thu, 15 May 2014 08:39:55 -0700

Dave Cridland <dave at cridland dot net> wrote:

> Of course, an invalid-UTF-8 based proposal simply means that it's no
> longer UTF-8 per-se, and so needs itself to be tagged differently.
> Other than that, I don't see it's a bad idea from a technical
> standpoint. The use of the word "invalid" probably scares people, but
> I note that's really a shorthand for "not backwards compatible by
> existing UTF-8 processors".

The proposal from 1997 ("MLSF") did call it an extra layer on top of
UTF-8, and included lots of health warnings that it was not really
UTF-8. That didn't remove the danger, though, because it looked so much
like UTF-8. John's response about decoders was spot-on.

> Exactly the same caveats apply to Plane 14 tagging, mind, and
> moreover, we could invent our own - indeed, that's what we're doing by
> having these arrays of (tag, string) tuples.

Since CBOR processors are likely to be bespoke, as you said, but the
underlying UTF-8 processor is not -- as you also said -- it would be
much easier to strip out Plane 14 characters for display if necessary
than to implement a whole new UTF-8–like decoding-stream layer that
understands MLSF.

As Mark knows, I never bought into the deprecation argument about how
evil Plane 14 tag characters are. Handling them correctly just isn't
that difficult. For CBOR, you may be better off with the tag/string
tuples; the tags in that case are much easier to see and don't need to
be stripped from the string for display or comparison. But if this
"tagged text" model is too far out of step with the CBOR/JSON way of
thinking, Plane 14 is out there.

--
Doug Ewell | Thornton, CO, USA
http://ewellic.org | @DougEwell

---

From: Mark Davis ☕️ <[email-masked]>
Date: Thu, 15 May 2014 16:37:07 -0700

On Thu, May 15, 2014 at 8:39 AM, Doug Ewell <[email-masked]> wrote:

> But if this

​I was perhaps not forceful enough. Deprecated Unicode characters should
*never* be generated by any modern software, and that includes the language
tags. They certainly should not be used in any new protocol.

Mark <https://google.com/+MarkDavis>

*— Il meglio è l’inimico del bene —*

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 15 May 2014 20:32:49 -0600

Mark Davis 🍶 wrote:

>> But if this "tagged text" model is too far out of step with the
>> CBOR/JSON way of thinking, Plane 14 is out there.
>
> ​I was perhaps not forceful enough. Deprecated Unicode characters
> should *never* be generated by any modern software, and that includes
> the language tags. They certainly should not be used in any new
> protocol.

And if Peter and Dave had decided that an out-of-band tagging model for 
CBOR was simply unworkable (which they did not, thankfully), and they 
were left with a choice between Plane 14 tags, invalid UTF-8 sequences, 
and some other homegrown, MLSF-like hack, what would be your 
recommendation to them?

---

From: Mark Davis ☕️ <[email-masked]>
Date: Thu, 15 May 2014 20:00:05 -0700

On Thu, May 15, 2014 at 7:32 PM, Doug Ewell <[email-masked]> wrote:

> CBOR was simply unworkable (which they did not, thankfully), and they were
> left with a choice between Plane 14 tags, invalid UTF-8 sequences, and some
> other homegrown, MLSF-like hack, what would be your recommendation to them?
>

​I would say that they need to go back and look at out-of-band tagging.

CBOR already handles a variety of data formats and can handle sequences of
variable length text and other items, so there is no need ​to say what's
the best of a set of other choices, all of which are awful.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 16 May 2014 18:17:57 +0900

>> longer UTF-8 per-se, and so needs itself to be tagged differently.

But this is highly counter-productive. US computer and Internet 
technology was so successful among else because everything was ASCII. We 
are finally getting close to a place where (almost) everything is UTF-8. 
Some of us already in 1997 (or even earlier) knew that that was the 
direction to go. UTF-8 "variants" would have killed a lot of the 
advantages of moving towards UTF-8.

>> existing UTF-8 processors".

It was not such a bad idea on the level of "let's use a screw to hold 
these two pieces of metal together". I was a very bad idea because much 
of technological (prospect of) success is or has to be measured not by 
how many good pieces of technology you have (how many different sizes of 
screws), but how *few* of them you have.

> UTF-8.

Only after quite a bit of pressure on the authors (including from me). 
See http://tools.ietf.org/rfcdiff?url2=draft-ietf-acap-mlsf-01.txt.

> That didn't remove the danger, though, because it looked so much

It was an extremely ugly chameleon mixing character encoding with 
higher-level information, messing around with the structural cleanness 
and heuristic detectability of UTF-8, and inviting all kinds of other 
crazy cludges for other "UTF-8-but-not-quite" chimeras.

>> having these arrays of (tag, string) tuples.

Plane 14 language tags are strictly within UTF-8 and also of course work 
with UTF-16. They are therefore quite a bit less bad than MLSF, but 
still bad enough.

> that difficult.

There are hundreds of ideas that look "not that difficult" to implement. 
But usually, everything turns out to be more difficult than estimated, 
and what's more important, the combinations of the different ideas turn 
out to be the killer.

In some ways, plane 14 language tags were born dead. Putting them in 
plane 14 was an explicit decision that sent a clear message that there 
was no expectation that they would or should be used frequently.

> For CBOR, you may be better off with the tag/string

As far as I understand, the tagged text model should work well, about as 
well as lang/xml:lang attributes for HTML and XML.

Regards,   Martin.