Subject: Re: I-D file formats and internationalization

From: Douglas Otis <[email-masked]>
Date: Thu, 01 Dec 2005 00:15:11 -0800

On Wed, 2005-11-30 at 18:29 -0800, Paul Hoffman wrote:

> No escape mechanism is needed. Non-displayable characters are still 
> in the RFC, they simply can't be displayed by everyone (but they can 
> be displayed by many). This is infinitely simpler, and a much better 
> long-term solution, than "an escape mechanism". Further, there would 
> be no more "ASCII version" to be authoritative. The Internet Draft 
> clearly says that there is a single RFC, and it has a single encoding.

Why do you think there is a problem using all possible characters in an
ID, but not in an RFC?  Why would it be okay for the RFC not to be
readable for some, but then ensure the ID is limited to ASCII?

> >   I liked the idea that Frank suggested, use the HTML escape 
> >sequence to declare the unicode character.  This allows the ASCII 
> >version to remain authoritative.
> 
> ... as well as unreadable and unsearchable using normal search 
> mechanisms. The purpose of the proposal is to allow RFCs to be 
> readable and searchable using the encoding that is common on the 
> Internet, without resorting to sorta-kinda-HTML or an "escape 
> mechanism". Remaining with plain ASCII would be better than either of 
> the latter.

The suggestion of the HTML escape would ensure readability.  As only
ASCII would be used, there would be no issues related to searching.  It
would allow an alternative display that remains compatible with an ASCII
limitation as the authoritative version.  UTF-8 use would require
additional considerations regarding searching however.

-Doug   

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 1 Dec 2005 08:34:41 -0800

At 12:15 AM -0800 12/1/05, Douglas Otis wrote:
>ID, but not in an RFC?

I don't. I simply believe that, given the way the IETF deals with 
process changes, it is easier to change one process at a time.

>Why would it be okay for the RFC not to be

I don't. It makes good sense to have both publication formats be the 
same, but getting that to happen at the same time may be too much of 
a hurdle for the IETF.

>The suggestion of the HTML escape would ensure readability.

Fully disagree. Listing an author as Patrik F&auml;ltstr&ouml;m is 
not "readable".

>   As only
>ASCII would be used, there would be no issues related to searching.

Fully disagree. People would have to search using the escaped-HTML 
kludge, even though our documents are plain text.

>It
>limitation as the authoritative version.

It would mix escaped HTML into text/plain documents. It would also 
make RFCs that talk about HTML extremely difficult to read, because 
the reader would not know if entities in examples are supposed to be 
the escaped or unescaped versions.

>UTF-8 use would require

Please list those; they would be valuable for the Internet Draft.

--Paul Hoffman, Director
--VPN Consortium

---

From: Douglas Otis <[email-masked]>
Date: Thu, 1 Dec 2005 09:57:37 -0800

> Fully disagree. Listing an author as Patrik F&auml;ltstr&ouml;m is  

The suggestion was for an alternate field in the XML input file to  
contain non-ASCII versions of authors and titles of references used  
to augment the ASCII version when the output form allows.  The idea  
of a escape mechanism was to allow the definition of non-ASCII  
characters elsewhere as perhaps could be needed to clarify protocol  
related issues.

> the reader would not know if entities in examples are supposed to  
> be the escaped or unescaped versions.

This would require a convention with respect to what gets converted.   
How often are numeric HTML Unicode characters used?  If this appears  
to be problematic, then simply retain the escape sequence in all  
output forms.

> Please list those; they would be valuable for the Internet Draft.

You talk about some of the issues in your draft.  Even when just  
ASCII is used, there is difficulty discerning differences between  
characters, where one's ability is largely determined by familiarity  
with a particular font style.  Cyrillic could be an example of there  
being more than one character-repertoire that may be used.  Going  
from 94+ characters to thousands is obviously more of a challenge for  
those wishing to pose a search.

Perhaps within a few years, as Unicode becomes more ubiquitous and  
such issues have been resolved with better tools, resistance against  
full adoption may be less.  In my view, the place to start would be  
with the ID and alternative output forms, and not the ASCII RFC.  It  
seems regimenting where exceptions are allowed makes sense, where an  
authoritative ASCII document is retained for now.  At some point in  
time, you will be right.

-Doug

---

From: Tim Bray <[email-masked]>
Date: Thu, 01 Dec 2005 11:26:06 -0800

On Nov 30, 2005, at 2:54 PM, Frank Ellermann wrote:

> As Bob said raw UTF-8
> characters won't fly with `cat rfc4567 > /dev/lpt1` and other
> simple uses of RFCs.

1. I wonder what proportion of the population prints things this way?
2. If the file is correctly encoded in UTF-8 and the above doesn't  
work, then your operating system is buggy.  I print stuff with non- 
ASCII characters all the time on Linux, Windows, and Macintosh  
computers.  Granted, I don't use "cat > /dev/anything" to do it.

  -Tim

---

From: Keith Moore <[email-masked]>
Date: Thu, 1 Dec 2005 15:16:28 -0500

> work, then your operating system is buggy.  

I guess that depends on your definition of "buggy".  The most popular OS
in the world no longer natively supports printing of _any_ kind of flat
files - you have to have a special application to do that.  While most
people would agree that that OS is buggy, its inability to print flat
files was a deliberate design choice and is therefore more properly
termed a "crippling feature".

Also, the vast majority of printers in use don't natively support
printing of utf-8, thus forcing users to layer each of their computer
systems with more and more buggy cruft just to do simple tasks like
printing plain text.  Perhaps those are buggy also?

These days, your best bet for getting utf-8 files to print is to use a 
web browser's print command, which is doable but can be fairly
cumbersome as compared to typing a simple "lpr" command.  Unfortunately,
most web browsers fail to preserve page breaks (FF characters) when
printing flat text files, which makes the resulting documents hard to
read.

HTML with utf-8 actually displays and prints more portably than plain
text with utf-8, though it's not clear how many browsers support the
style sheet extensions enough to print page breaks in the right
places.  Also, HTML is still somewhat of a moving target and it is
somewhat unclear whether any particular subset of HTML that is used
today will still be effectively presented 10-20 years from now.  The
biggest problems with HTML are (a) no way to include images in the
document without external links (yes I know about MHTML but it's not as
widely supported); (b) difficulty in finding authoring tools that will
produce output in a subset of HTML that we define; (c) avoiding
the temptation to make the documents pretty rather than readable.

It's hard to escape the conclusion that we're trying very hard to make
our document processing much more complex for a very marginal gain.

Keith

---

From: Tim Bray <[email-masked]>
Date: Thu, 01 Dec 2005 12:48:37 -0800

> printing plain text.  Perhaps those are buggy also?

Uh, I print UTF-8 documents all the time.  Normally I do it from the  
app in which I'm viewing them (word processor, web browser, RSS  
reader, xml editor, whatever).

> cumbersome as compared to typing a simple "lpr" command.

Hmm; control-P, enter.

> Unfortunately,

Turn this around; when printing HTML, the browser inserts appropriate  
page breaks depending on the combination of font, styling, and paper  
size that's in effect.  This has the effect that when you're arguing  
about some text, you have to say "Look at 5.2.1.3, 2nd para" rather  
than "Look at page 13, 2nd para".  It's not clear that this is any  
better or worse.

> places.

Given the above, I agree with the first half of the sentence.  In  
fact, I am sitting behind a desk on which there's a macintosh and an  
Ubuntu linux box, and I wouldn't really know how to print plain-ASCII  
text on either of them, and when I've tried, the page breaks usually  
come out wrong.  On either of them, I can and do print HTML  
effortlessly and with excellent results.

>   Also, HTML is still somewhat of a moving target and it is
> today will still be effectively presented 10-20 years from now.

I think it is crystal clear that if you stick to HTML4 Strict or  
Transitional, that has an excellent chance of survival at least for  
decades.

>   The

I grant problem (a).  (b) and (c) can be solved using automated tools  
and compulsory stylesheets (or by using xml2rfc).

> our document processing much more complex for a very marginal gain.

There are two large populations for whom the gain is not "marginal".

1. Those, like me, who can't print ASCII files easily
2. Those whose names can't be spelled properly in ASCII.

I claim that both those groups either already do, or will soon,  
constitute large majorities of the interested population. -Tim

---

From: Keith Moore <[email-masked]>
Date: Thu, 1 Dec 2005 17:07:47 -0500

> reader, xml editor, whatever).

The point is that the apps have to support utf-8 or leverage OS support
for UTF-8, in order to print those documents.  Because the printer
doesn't support UTF-8, it's not as simple as simply sending the
characters to a printer anymore.   And while this might work fine for
you, and seem like a transparent change, it's inherently a much more
fragile setup.  (Which isn't really an argument against UTF-8, just a
digression about what is or is not "buggy".)

> > cumbersome as compared to typing a simple "lpr" command.

Well, sure, if you spend all of your time inside a particular web
browser.  

> than "Look at page 13, 2nd para".  It's not clear that this is any  

It's worse, because you really do want page numbers for when you print
the document and it's quite natural to reference them.  IMHO you really
want the HTML for an RFC to preserve pagination and page numbers and
make them visible (but not annoyingly so) even in a browser, while
causing page breaks when printed and still printing correctly on either
us-letter or a4 paper.  But I'm not sure that this can actually be done.

> come out wrong. 

Sometimes this is because there are no FFs in the source document
(especially for internet-drafts) and sometimes this is because the app
that you're using to print doesn't respect them (most web browsers
botch this).  OTOH, the command-line apps tend to do this right - not
because they are tied to a command-line but because they don't have
to deal with moby GUI libraries.

> and compulsory stylesheets (or by using xml2rfc).

Well, sure, if we can demand that everyone use the same tools we can
define the file format however we want.   What we want is to make the
RFCs editable and displayable with existing tools.  Compulsory
stylesheets don't make problem (c) go away - they just move part of the
problem.  Plain text has the nice attribute that it encourage you to
concentrate on substance rather than appearance.

> 1. Those, like me, who can't print ASCII files easily

I suspect it's not because you can't do so, but rather because you want
to be able to do so without changing the tools you use.  (after all,
lpr works fine to print text files on every UNIX/linux system I've seen
and also on my Mac)  Guess what, nobody else wants to change tools
either - and different kinds of people need different tools.
Changing the file format won't solve this problem, it will just move it.

> 2. Those whose names can't be spelled properly in ASCII.

It's a valid concern, but surely it's more important to communicate the
protocol specification clearly than the authors' names?  I certainly
don't object to making the native spellings of authors' names visible
if this can be done "for free", but any change to our document format
needs careful consideration, and the ability to print authors' names is
way down on the list of things to consider.

Keith

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Dec 2005 13:11:00 +0100

Paul Hoffman wrote:

> Listing an author as Patrik F&auml;ltstr&ouml;m is
> not "readable".

From my POV it's better than F=84lttr=94m (858 QP) or
F=E4ltstr=F6m (1252 QP) or F=C3=A4ltstr=C3=B6m (UTF-8 QP).

With your proposal I'd see FÃ¤ltstrÃ¶m (intentionally
raw UTF-8 encoded in the Latin-1 subset of windows-1252).

AFAIK xml2rfc would transform it to Faeltstroem for its
ASCII output formats - perfectly readable, but wrong.

With NCRs we'd get F&#xE4;ltstr&#xF6;m - not exactly the
best readable version, but at least it's correct, and
IMO better than F=C3=A4lttr=C3=B6m.  I'm not very good
at decoding UTF-8 (QP or raw) "manually".

> People would have to search using the escaped-HTML

Yes, we'd need a new "text/plane" or whatever, plain
ASCII plus NCRs isn't the same as text/plain ASCII.

Your idea text/plain UTF-8 also isn't the default for
text/plain.  Same problem, do you expect an update of
2046 introducing UTF-8 as new default for text/plain ?

I'd prefer to revisit your idea when the IEE / IMA
folks have something that works somewhere - in other
words 2007, not next year.
                          Bye, Frank

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Dec 2005 14:15:01 +0100

> raw UTF-8 encoded in the Latin-1 subset of windows-1252).

Frank,

I'll make the following assumption: those who still use operating 
systems with no builtin support for UTF-8 are knowledgeable enough to 
run a text file - in particular if it has a UTF BOM - through GNU recode.

> ...

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Dec 2005 15:35:06 +0100

Julian Reschke wrote:

> those who still use operating systems with no builtin support
> for UTF-8 are knowledgeable enough to run a text file - in
> particular if it has a UTF BOM - through GNU recode.

A mandatory BOM for those I-Ds / RFCs actually using UTF-8 is
an idea for Paul's next draft.  Just for fun I test it here.

I don't know "GNU recode" so far - my UTF-8 script would only
recode characters with a 1:1 correspondence in windows-1252.

So maybe Paul would also need "fully normalized" as defined
in http://www.w3.org/TR/charmod-norm/ (that's not yet ready).

                         Bye, Frank