Subject: RE: Source Format of RFCs

From: "Herman R. Silbiger" <[email-masked]>
Date: Mon, 30 Nov 1998 22:47:12 -0500

I agree that there is a certain universality about ASCII text.  The problems that you bring up that may be part of using a more modern format capable of figures and tables do exist.  I disagree somewhat with your position regarding the "archival" nature of RFCs that need to be read for decades.  The only true archival format is print on acid-free paper or microfilm, both of which can be produced without difficulty and could always be concerted back into digital format.

I do question why this is so all important to the IETF and not to any other standards organization.  The ITU, ISO, ANSI and most other  standards organizations do not seem to have any difficulty with using a word processor format.

What I miss most in the all text format is the ability to mark and track changes in successive versions of drafts. 

I also have found that many authors actually use word processors to produce drafts and RFCs, having given up their Teletypes some years ago.

Herman Silbiger

----------
From:  Keith Moore [SMTP:[email-masked]]
Sent:  Monday, November 30, 1998 8:28 PM
To:  Bruce Balden
Cc:  Dunlap, Randy; [email-masked]
Subject:  Re: Source Format of RFCs 

> A number of RFC's and drafts, such the ones regarding BGP, suffer negatively 
> from the text-only format. It would be nicer to have a richer format, 
> especially since such documents rarely originate in the text-only format, 
> but are rendered into that format by MS-Word, nroff, LaTeX or whatever.
> 
> Would it be asking too much for all RFC's too made available in some sort 
> of rich format, and in particular their source format since that doesn't 
> require additional work?

You may be asking for far more than you realize.

I agree with you that the ASCII-only RFC format is a bit restrictive.
However, ASCII is amazingly flexible.  It is one of the few formats 
that can be reasonably viewed (or searched) on any computer, and
printed on any printer.  Neither of these is true of the document 
formats associated with common word processors.  Also, since it is 
so simple, ASCII is a good choice for formatting documents that may 
need to be readable for many decades.  For comparison, word processor 
documents from the early 1980s are difficult to read today, and
it's not difficult to imagine that today's word processor documents
will be difficult to read ten or twenty years from now.

(The more complex the document format, the less interoperable it is.
There have been problems with PostScript RFCs because some tools
produce PostScript that cannot be printed on some printers.
Similarly, version X of FooWord may produce documents that cannot
be read by version Y of FooWord.)

There are also problems with making RFCs available in multiple formats.
One is that the RFC Editor and Internet-Drafts folks would then need
to deal with a wider variety of tools.  Another is that multiple formats
can make it more difficult to keep the various formats in sync.
For instance the RFC Editor generally needs to be able to make minor 
changes to a document before publication, yet may not have the 
tools neded to modify the "source" document.  If the author is asked
to make the changes, there is then a need to make sure that the
author does not make other modifications to the work after it has 
been reviewed and approved by a working group and IESG.  Again, 
this generally requires special tools.

Until a new format can supplant ascii text for all intents and purposes,
there will continue to be a need to produce ascii text versions of RFCs.
Many word processors do a very poor job of producing "RFC ASCII"
(with margins that allow printing on both A4 and US-Letter paper, 
no overprinting, and so forth)  Documents written for a wysiwyg 
word processor are often difficult to read as ASCII, e.g. because 
the word processor will try to put too many words on a line, or too
many characters in a column of a table. (I've seen various versions 
of this while reviewing RFCs for IESG.)

Then there is a need to make sure that new RFC formats are adequately
supported by all of the RFC and internet-drafts mirrors.

For some time I have been interested in exploring the use of HTML
for RFCs and Internet-Drafts.  This would provide several advantages,
including multifont and multi-faced text in a variety of sizes,
hyperlinks, images, and ease of generation with many word processors.
HTML is openly published and widely-used, and (provided a few restrictions
are observed) one of the few formats which can be read on every platform 
(and is easily converted to plain text).

However, as far as I can tell, there are several barriers to adoption 
of HTML as an RFC format:

0. Given that not all features of HTML are available on all HTML viewers,
   We need some (perhaps loose) standards for which features can be
   used, and which ones should be avoided.    (For example, the use of
   tables for formatting, or embedded images for bullets, should perhaps 
   not be allowed.  Even the use of CSS might be questionable since
   many viewers do not support it.) The standard would need to specify 
   things like which HTML tags could be used, the naming convention to
   be used for files, the format of links to other RFCs and to embedded 
   images (presumably relative rather than absolute), and "style" - 
   how a document should be formatted if it wants to look like an RFC
   (e.g. which tag should you use for a section header?).

1. Once an "RFCs in HTML" standard were adopted, common word processors 
   might be unable to produce HTML which conformed to the standard.
   It might turn out to be even more difficult to produce reasonable 
   HTML than it currently is to produce reasonable plain text.

2. The Internet-drafts and RFC Editor people would need tools to let
   them edit HTML, remove or alter tags that did not conform to the
   standard, and perhaps to convert HTML to "RFC format" ASCII text.
   (again, such tools would need to work with the garbage HTML that
   is generated by some word processors)

3. The mirror sites would have to adapt to accomodate HTML RFCs, 
   including any embedded images referenced by those RFCs.

..and there are probably a few more.  None of these is insurmountable,
but so far there has been a lack of critical mass behind the effort.

A year or two ago I set up a discussion list for rfcs-in-html.
To subscribe, send mail to [email-masked].

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 01 Dec 1998 01:24:18 -0500

> I agree that there is a certain universality about ASCII text.  The
> problems that you bring up that may be part of using a more modern
> format capable of figures and tables do exist.  I disagree somewhat with
> your position regarding the "archival" nature of RFCs that need to be
> read for decades.  The only true archival format is print on acid-free
> paper or microfilm, both of which can be produced without difficulty and
> could always be concerted back into digital format.

Archives are relative, I suppose.  I'd much rather have protocol 
specifications in ASCII on my disk, than paper on a shelf,
or microfilm which requires a special reader, or even scanned images
that are difficult to read, take up huge amounts of space, and are 
not searchable.

> I do question why this is so all important to the IETF and not to any
> other standards organization.  The ITU, ISO, ANSI and most other
> standards organizations do not seem to have any difficulty with using a
> word processor format.

The ITU, ISO, and ANSI documents I have are on paper, in binders on
a shelf.  Even when I get them in machine-readable form (and when the
format is one that my machine can read), I generally find that I have 
to print them out to make any use of them.  Since RFCs are easily
read without special tools, I rarely need to print them, they don't 
take up any precious shelf space, and I can carry them with me wherever 
I take my laptop.

> What I miss most in the all text format is the ability to mark and track
> changes in successive versions of drafts.

That's certainly a useful feature.
 
> I also have found that many authors actually use word processors to
> produce drafts and RFCs, having given up their Teletypes some years ago.

And I've personally found that RFC authors who write directly for pure-ascii 
format are only occasionally hampered by the inability to produce 
graphics.  By contrast, RFC authors who produce word processor documents 
and convert them to text later, often seem to be worrying far too much 
about the appearance of the document and too little about the quality 
of the prose.

Maybe some of them would do better with a Teletype, or at least a Selectric.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 01 Dec 1998 01:24:18 -0500

Keith

---

From: "Perry E. Metzger" <[email-masked]>
Date: Tue, 01 Dec 1998 10:54:06 -0500

"Herman R. Silbiger" writes:
> format capable of figures and tables do exist.

There are tables and figures in most ASCII RFCs and they seem quite readable.

> I do question why this is so all important to the IETF and not to
> any other standards organization.  The ITU, ISO, ANSI and most other
> standards organizati ons do not seem to have any difficulty with
> using a word processor format.

I was unaware I could download most of their source documents and
manage to read them twenty years later. Indeed, I was unaware that ANY 
of their documents were downloadable until very recently.

Perry

---

From: "Perry E. Metzger" <[email-masked]>
Date: Tue, 01 Dec 1998 10:54:06 -0500

Perry

---

From: "Matt Crawford" <[email-masked]>
Date: Tue, 01 Dec 1998 11:26:10 -0600

> any other standards organization.  The ITU, ISO, ANSI and most
> other standards organizations do not seem to have any difficulty
> with using a word processor format.

Yes, but for at least some of those organizations, the working group
mailing lists have appreciable instances of "I couldn't print your
file here."

---

From: "Matt Crawford" <[email-masked]>
Date: Tue, 01 Dec 1998 11:26:10 -0600

---

From: Fred Baker <[email-masked]>
Date: Tue, 01 Dec 1998 09:29:17 -0800

At 01:24 AM 12/1/98 -0500, Keith Moore wrote:
>a shelf.

ITU-T generally exchanges working documents in a variant of MS-word format.

---

From: Fred Baker <[email-masked]>
Date: Tue, 01 Dec 1998 09:29:17 -0800

---

From: Keith Moore <[email-masked]>
Date: Tue, 01 Dec 1998 13:27:43 -0500

> ITU-T generally exchanges working documents in a variant of MS-word format. 

Yep, and ms-word documents are damn near useless until they're printed out.

Keith

---

From: Ned Freed <[email-masked]>
Date: Tue, 01 Dec 1998 10:17:17 -0800 (PST)

> > using a word processor format.

Unfortunately this is factually incorrect, at least as far as the CCITT/ITU is
concerned. They have had all sorts of major problems over time with their use
of various word processor document formats. For some indication of how much
trouble they've had you might check out Carl Malamud's books (sorry Carl, I
don't remember which one).

And these problems continue to this day. If you buy individual standards from
the ITU you can get them in PostScript for A4 paper, which sometimes works and
sometimes doesn't. But if you buy the complete set on CD you get:

(1) A version in a proprietary hypertext format, which as far as I can tell
    doesn't work at all.
(2) ZIPped Word files, which again require A4 paper to print properly. (Yes,
    Word has some scaling options, but I find that they don't work very well
    in practice.)

In many cases you're actually better off with the ones from the the InfoSeek
disks (created during a brief period when the ITU made most of their standards
freely available), as the ones on the InfoSeek disks tend to have indices,
while the more recent official ones do not.

> of their documents were downloadable until very recently.

20 years ago the ITU used some proprietary Word processor system, which I doubt
anything could read these days, assuming the documents were available, which
they aren't.

And this is a real problem. There are times when people really need to see the
yellow books (1980) or the red books (1984) or the blue books (1988). The more
recent stuff won't do. (Indeed, there are cases in X.400 where you need  to see
material that's only in unpublished drafts, but that's a process problem, not a
format problem.)

When I need to look at, say, RFC 733, I don't have a problem. But the same
isn't the case for, say, X.25-1980. When it comes to yellow or red books or
FIPS PUBs or whatever I typically have to borrow one from someone I know has
the complete set, since the libraries don't have them and new printed copies
range from difficult to flat-out impossible to obtain.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 01 Dec 1998 10:17:17 -0800 (PST)

Ned

---

From: Keith Moore <[email-masked]>
Date: Tue, 01 Dec 1998 13:27:43 -0500

Keith

---

From: Robert Shaw <[email-masked]>
Date: Wed, 02 Dec 1998 14:17:17 +0100

> don't remember which one).

With all due respect to Carl, he really doesn't have any understanding of the 
history, requirements or constraints the ITU has. We have had for many years 
mandatory multilingual and complex doc architecture components (e.g., our Radiocom
Recommendations contain thousands of equations on stuff like radiowave propagation 
and satellite spot beam diagrams). This a priori ruled out ASCII. In fact, I remember 
one of my first jobs at ITU in 1982 was to patch a word processor to work in 
Cyrillic and burn new Cyrillic character set proms.

We had a customized publication text processing system to deal with this stuff in 
the late 70's but in the early 80's and the push for faster standards delivery, we 
realized we needed to really merge the standards processing and publication cycles 
instead of having grandiose ideas about elegant publication architectures (some 
will remember the daunting complexity of the ISO SGML Standards DTD).

> sometimes doesn't. 

First, we supply mostly in PDF and Word format now and the focus is really is
really away from Postscript as user complaints show it is really isn't too printer 
portable. As for A4, you're just pointing out the inherent problems of pre-paginated 
formats which is no different than RFC 2223 saying RFCs published as Postscript 
files be imaged at 8.5 x 11.
>                              But if you buy the complete set on CD you get:

The CD versions were speced by Dataware before HTML really took off. It will
be replaced shortly by an essentially off-line web version. It may not be ideal
but there are thousands of customers using it (don't know why yours doesn't
work but if you really care to follow up, let me know and I'll put you onto 
the people who can sort it).

>     Word has some scaling options, but I find that they don't work very well

In any case, you could say the same thing about RFCs since they also pre-paginated.
At least with Word, you can repaginate. 

> while the more recent official ones do not.

First, I assume you mean InfoMagic. Second, they're pirated as their web 
site will tell you. Third, these are for the most part dated and rubbish 
conversions. I can't count the number of complaints we have had from people 
thinking that this stuff originated from us. And like I said, we're doing new 
CD and DVD products. 

> anything could read these days, assuming the documents were available, which

See above. We basically had no choice, there wasn't anything at the time which 
could meet our multilingual and architecture requirements.

> range from difficult to flat-out impossible to obtain.

Next time you have a problem, contact me and I'll do my best to chase down a 
solution for you.

I'll note that most everyone preparing standards is using some commercial word 
processing software and discussions about which is the great and true final 
standards format are basically religious wars. After hearing these arguments for 
too many years, I've come to the conclusion that it doesn't really matter a whole 
lot. Use what works for you and what you're customers give you feedback that they
want today. Our customers aren't clamoring for ASCII. In fact, it's more the 
contrary, they want the original source formats. IETFers seem to like ASCII. 
So fine.

Bob
-- 
Robert Shaw <[email-masked]>
Head a.i., IED/Advisor, Global Information Infrastructure
International Telecommunication Union <http://www.itu.int>
Place des Nations, 1211 Geneva, Switzerland

---

From: Robert Shaw <[email-masked]>
Date: Wed, 02 Dec 1998 14:17:17 +0100

Bob

---

From: "Perry E. Metzger" <[email-masked]>
Date: Wed, 02 Dec 1998 11:19:14 -0500

Robert Shaw writes:
> history, requirements or constraints the ITU has. We have had for
> many years mandatory multilingual and complex doc architecture
> components

That certainly makes your life much harder. The IETF luckily isn't
under that constraint.

> Our customers aren't clamoring for ASCII. In fact, it's more the

Wackily, most of the original source format for most RFCs *was* ASCII
-- at least for the documents I've been involved in. I've worked on
several, and I've never seen them edited in anything but ASCII source.

Call me old fashioned, but I myself use Emacs and markup languages for
all the documents I prepare -- LaTeX, HTML and troff depending on on
situation (I'm now using MagicPoint for slides, but it is also ASCII
based input -- very cool program, btw, and it is open source).

I *do* prefer reading well typeset text, btw. I just find that I can
always manage to do more things more flexibly with ASCII as the input
format. I can write my own text manipulation tools, for instance. That 
probably puts me off the end of the curve as an "ordinary" text
processing user, but probably puts me in the mainstream as far as IETF 
document writers go.

The fact is, our needs are very different. We only produce documents
in English, the document editors are all highly technical people, and
we have very high standards for document viewing and editing
interoperability.

If people are REALLY concerned about producing IETF documents that
look good in both ASCII and in some printed format, I strongly suggest 
using troff/nroff. There are nroff macros that produce RFCs with very
little tweaking. The same macros could be used to generate postscript
with almost no effort. A bit of hacking with pic and tbl and, voila,
ASCII and PostScript versions of the same document (which is already
permitted by IETF rules.)

Perry

---

From: "Perry E. Metzger" <[email-masked]>
Date: Wed, 02 Dec 1998 11:19:14 -0500

Perry

---

From: Carl Malamud <[email-masked]>
Date: Wed, 2 Dec 1998 13:09:23 -0500 (EST)

> > don't remember which one).

It went out of print, so I posted it on-line at:

	http://not.invisible.net/eti/

> history, requirements or constraints the ITU has. 

After reverse-engineering your undocumented code, posting your doc
set on the net, maintaining a distribution network of 40 servers
in 20 countries, and then watching the international club of 
regulation-dependent telcos fumble their way through the next 5
years in a series of attempts to limit the spread of the Internet, 
I think I understand very well the "history, requirements, 
or constraints" of the ITU.  

Maintaining the ability of PTTs to syphon money from my pocket and
limit what I'm allowed to do is a wonderful goal and the ITU does a 
great job in that function.  I just don't happen to agree with the
goal.  That doesn't diminish my understanding for the constraints
the ITU faces, just a lack of respect for refusing to change those
constraints to become relevant to the modern world.

In any case, the ITU is one of those country clubs that is out of
my league.  Can't afford the membership fees and I never
really got much use out of the intellectual golf course you run.

Carl

---

From: Carl Malamud <[email-masked]>
Date: Wed, 2 Dec 1998 13:09:23 -0500 (EST)

Carl

---

From: Robert Shaw <[email-masked]>
Date: Wed, 02 Dec 1998 20:48:26 +0100

> or constraints" of the ITU.

Well now this has really gotten off topic and we're now in the fertile 
ground of netheads vs bellheads. Despite it being painful to hear, if you 
know the history of telecommunications, nothing here is really surprising 
or new. Dig up an essay from Wired called "Deja vu all over again" by Todd 
Lapin (sorry I couldn't find it, Wired seems to have moved it...) which talks 
about the Internet's cultural parallels with the early days of radiocommunications. 
Now ask yourself whether what's happening now is taking us inexorably down 
that path (hint, just go read the NTIA MoU with ICANN for enlightment where
things are heading).

> goal.  That doesn't diminish my understanding for the constraints

Huhhh...the ITU has nothing to do with PTO's being able to siphon money 
from your pocket. I really wonder where you get these ideas? Despite 
popular myth, the ITU is not a regulator. 

> constraints to become relevant to the modern world.

Oh please...

> really got much use out of the intellectual golf course you run.

Ouch!

See http://www.itu.int/aboutitu/smallm.html - look at the listings for
say, the "United States" - looks like your arch-enemy is now you, doesn't 
it?

Bob

---

From: Robert Shaw <[email-masked]>
Date: Wed, 02 Dec 1998 20:48:26 +0100

Oh please...

Ouch!

Bob

---

From: Keith Moore <[email-masked]>
Date: Wed, 02 Dec 1998 14:53:40 -0500

> just go read the NTIA MoU with ICANN for enlightment 

God help us all.

---

From: Keith Moore <[email-masked]>
Date: Wed, 02 Dec 1998 14:53:40 -0500

God help us all.

---

From: Ned Freed <[email-masked]>
Date: Wed, 02 Dec 1998 12:07:22 -0800 (PST)

> portable.

Uh huh. Exactly my point.

> As for A4, you're just pointing out the inherent problems of pre-paginated
> formats which is no different than RFC 2223 saying RFCs published as
> Postscript files be imaged at 8.5 x 11.

Actually, it is _very_ different. All RFCs are pre-paginated. Yet I have no
problems printing them out on either A or A4 paper, double-up, or anything else
I happen to want.

The same cannot be said of ITU docs.

Now, the solution to this problem is simple -- you use pagination when you
write the specifications that works on both A and A4. But that's not how these
documents were done.

> the people who can sort it).

I'm sorry, but I have neither the time nor the inclination to sort out bugs in
other people's hypertext systems. BTW, I've talked to at least half a dozen
"users" of this thing, and I've yet to find one who has had any success with
it.

> At least with Word, you can repaginate.

Been there, tried that. Doesn't work well in practice on ITU docs.

My solution to all this was to go out and buy 5000 sheets of A4 paper. Problem
solved, for me anyway.

> thinking that this stuff originated from us. And like I said, we're doing new

Well, all I can tell you is that I consistently have better results with theirs
than I do with yours. (The index for X.411 which theirs has and yours lacks,
for example.) Sorry, but I call them as I see them.

And yes, theirs are dated. But often what I need is the older version, so
this actually is an advantage, not a disadvantage.

> could meet our multilingual and architecture requirements.

Again, this is precisely my point. You decided on rich text, quite possibly for
very good or at least unavoidable reasons, and you picked what was current 20
years ago. And now nobody can read these documents. You then went through
various iterations and trials with other formats like PostScript, and had more
troubles. And how you've arrived at using Word files, and I'm telling you now
that while you may think this is working, it is a lot less than satisfactory
for at least some of your users (me, for example).

I never said that you didn't have good reasons for the path you chose. It is
quite possible you did. But this is not relevant to the matter at hand. The
matter at hand was the assertion that a rich text approach has been used by
various other standards organizations and it has been a huge and unqualified
success. But it hasn't, as you yourself have now stated. We may disagree on how
successful your present scheme is, but I don't think we disagree that there
have been major problems along the way.

So, when someone holds up the ITU as a shining example of the right path
to have followed when doing standards, I'm sorry, but I just don't buy it.

> Next time you have a problem, contact me and I'll do my best to chase down a

Bob, I have these sorts of needs all the time. This is the life of someone who
builds and supporting networking products in either the Internet or OSI world. 
I'm forever wanting to chase down some reference or other in some part of SMTP,
FTP, RFC822, MIME, X.400, X.500, ROSE, RTSE, FTAM, ASN.1, whatever. I usually
then want to see how the specification for this item has changed over time,
usually because I'm dealing with some interoperability problem where I've built
to a particular version of the specifications and I need to be really sure that
the specification I built to hasn't changed over time.

When this happens and it is an Internet standard at issue I always have all the
relevant documents to hand, and I always can read them. But when it is, say,
something in X.29 that may have changed since 1980, well, I've had to do a lot
of work, not to mention spending many thousands of dollars, to get to the 
point where I can do similar sorts of things.

However, I do still have lots of problems with ISO standards. If your offer to
help extends to those, then I'll be happy to contact you the next time I need
an old version of one of them.

> contrary, they want the original source formats. IETFers seem to like ASCII.

Well, you can count me as one customer who isn't at all satisfied with your
service and would clamor for something better if I knew how to do so. 

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 02 Dec 1998 12:07:22 -0800 (PST)

Ned

---

From: Christian Huitema <[email-masked]>
Date: Fri, 4 Dec 1998 09:42:53 -0500 (EST)

> Cyrillic and burn new Cyrillic character set proms.

Robert,

The ITU has a charter that mandates publication in several languages, a
problem that the IETF does not have.  Publishing standards in several
languages poses an impossible reference problem -- traductions are often
imperfect, and there are fairly publicized cases of documents where the
english and french versions don't have quite the same meaning (including
no less than a key UN resolution on the fate of Palestine.) Having exactly
one reference version is in fact a good thing.

However, even if we publish in just one language, you may have a point
there.  Just last month, I wanted to give an example of how we could
include a Unicode string in a protocol message -- to carry user frienly
names that should be displayed to the user.  I did format the appropriate
accents and cedillas in the NROFF source, but they were indeed wiped out
by the ASCII-only post-processing, resulting in some confusion.

Someone did mention that "monospace ASCII" was our only practical choice.
 But, in fact, "monospace Unicode" would have much of the same properties,
while allowing inclusion of accents, cyrillic letters or chinese
ideograms. And it would be relatively easy to produce an "unicode only"
postprocessor for NROFF.

-- 
Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Fri, 4 Dec 1998 09:42:53 -0500 (EST)

Robert,

-- 
Christian Huitema