Subject: Re: [Ltru] [apps-discuss] Fwd: Defining a CBOR tag for RFC 5646 Language Tags

From: "Doug Ewell" <[email-masked]>
Date: Wed, 14 May 2014 14:47:16 -0700

Mark Davis ☕️ <mark at macchiato dot com> wrote:

> I'm sure you're not implying that I think invalid UTF-8 would have
> been a good idea, but your statement might not be clear to others.

To clarify, I got that impression from Dave's remark, which is why I
originally quoted it:

] Many years ago, Mark Crispin and Chris Newman had a proposal for
] embedding language tags in invalid UTF-8; I seem to recall they
] publicly renounced their proposal rather dramatically in favour of a
] Unicode Consortium proposal for embedding the language tags somewhere
] in Plane 14 - published as RFC 2482.
] 
] The fact it was all initiated in order to support the pressing needs
] of ACAP might give you some hints as to why it never really took off,
] but as a counter-proposal to language tags in metadata, it might be
] worth re-examining.

I'm not sure, upon re-reading this, whether Dave meant to say that Plane
14 tags or invalid UTF-8 was worth re-examining.

--
Doug Ewell | Thornton, CO, USA
http://ewellic.org | @DougEwell

---

From: "Peter Occil" <[email-masked]>
Date: Wed, 14 May 2014 18:04:26 -0400

"I'm not sure, upon re-reading this, whether Dave meant to say that Plane
14 tags or invalid UTF-8 was worth re-examining."

I understood it all along to mean the Plane 14 characters.

That tag characters are deprecated is one of the reasons for the CBOR tag 
proposal.  How CBOR processors should handle invalid UTF-8 is definitely out 
of scope for my CBOR tag proposal.

-------------

I also have just one issue I want to resolve before I move my proposal 
forward; I repeat it here:

"I want to clarify that the language tag and the
tagged string can optionally be annotated with CBOR tags.  For example:

38([tagX("en"), tagY("Hello world")])

I think this will address some of the "scalability" issue that Martin Duerst
raised."

--Peter

-----Original Message----- 
From: Doug Ewell
Sent: Wednesday, May 14, 2014 5:47 PM
To: Mark Davis ☕️
Cc: LTRU Working Group ; Dave Cridland
Subject: Re: [Ltru] [apps-discuss] Fwd: Defining a CBOR tag for RFC 5646 
Language Tags

_______________________________________________
Ltru mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ltru

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 May 2014 08:44:00 +0100

> 14 tags or invalid UTF-8 was worth re-examining.

Either/neither/both.

Of course, an invalid-UTF-8 based proposal simply means that it's no longer
UTF-8 per-se, and so needs itself to be tagged differently. Other than
that, I don't see it's a bad idea from a technical standpoint. The use of
the word "invalid" probably scares people, but I note that's really a
shorthand for "not backwards compatible by existing UTF-8 processors".

Exactly the same caveats apply to Plane 14 tagging, mind, and moreover, we
could invent our own - indeed, that's what we're doing by having these
arrays of (tag, string) tuples.

Inline tagging is out of fashion, I agree - and as I said, it never really
caught on in part because the only protocol that cared - ACAP - never
caught on itself. What I was suggesting was that this technique exists, and
a new type for "language tagged string" is not impossible to arrange, and
actually aligns with the goal here.

Whether this is the right solution here is an entirely different question,
and the one I was hoping wouldn't be rejected out of hand - in no small
part because all we're doing here is inventing a new bespoke format for
CBOR to do the same thing.

The main consideration, I think, is what happens when a CBOR processor
encounters a language-tagged string when it doesn't understand the concept.
Extending UTF-8 seems generally undesirable since the UTF-8 processor is
likely to be an off-the-shelf black box, Plane 14 shows up as unknown
characters, and the array-of-tuple shows up as arrays of tuples. I'm not
sure any of those are desirable.

I'd further note that CBOR processors are, at this stage, likely to be
bespoke, so changes there seem acceptable. Either of the latter two can be
handled that way.

Finally, I don't have a horse in this race, I was just wondering if anyone
else had noticed there were other horses.

Dave.

---

From: John Cowan <[email-masked]>
Date: Thu, 15 May 2014 08:06:50 -0400

Dave Cridland scripsit:

> Of course, an invalid-UTF-8 based proposal simply means that it's no
> longer UTF-8 per-se, and so needs itself to be tagged differently.

The whole point of the invalid-UTF-8 was that the less fussy decoders of
the day would quietly drop the hidden information and display the string.
That is much less likely to happen now.

> Exactly the same caveats apply to Plane 14 tagging, mind, 

Not so much.  Unicode renderers that don't understand language tags will
not display them (because stuff on plane E is never displayed per its
Unicode properties), or if the renderer doesn't even understand that,
will at worst generate boxes or other "character unknown" glyphs,
often only three of them.

> encounters a language-tagged string when it doesn't understand the concept.

Agreed, and if that is a serious problem, Plane E starts to look better.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
[P]olice in many lands are now complaining that local arrestees are insisting
on having their Miranda rights read to them, just like perps in American TV
cop shows.  When it's explained to them that they are in a different country,
where those rights do not exist, they become outraged.  --Neal Stephenson

---

From: Dave Cridland <[email-masked]>
Date: Thu, 15 May 2014 13:31:22 +0100

Thanks for the corrections; I shall now slouch into silence because I
honestly have very little opinion.