Subject: Re: Automatically updated Table of Contents with Nroff

From: Stefan Santesson <[email-masked]>
Date: Thu, 17 Mar 2011 01:07:44 +0100

Julian,

Sorry for an awfully late response, but just spotted this and thought I
should clarify as author of the NroffEdit tool.

>>
>>
>>
>> NRoffEdit is an all-in-one wysiwyg tool in Java that maintains
>> the TOC for you (within the .nroff source itself).
>
>Which will only work properly as long the output isn't run through
>standard NROFF, and page breaks change. Just saying.

This is not correct.

The automatic ToC function (and now since version 1.40 also the automated
reference function) operates using commands hidden behind Nroff comments.
A standard NROFF compiler will ignore the comments and process the ToC as
if it was manually entered. The ToC will match the document even if
compiled by a standard NROFF compiler.

/Stefan

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Mar 2011 08:55:34 +0100

On 17.03.2011 01:07, Stefan Santesson wrote:
> ...
> ...

Yes, but only as long page breaks aren't changed, right? That is, the 
ToC contains fixed page numbers, no?

Best regards, Julian

---

From: Martin Rex <[email-masked]>
Date: Thu, 17 Mar 2011 14:59:32 +0100 (MET)

> ToC contains fixed page numbers, no?

That is a non-issue.  NRoffEdit produces the same output as the
nroff used by the RFC-Editor.

If you want to change the text, you would use NRroffEdit to do it
and update the TOC accordingly by a simple keypress or menu selection.

-Martin

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Mar 2011 15:12:33 +0100

> and update the TOC accordingly by a simple keypress or menu selection.

Martin,

the context of this was a discussion how to generate a ToC using NROFF.

My comment was regarding the claim that NRoffEdit somehow achieves this; 
it does not. It just does exactly what xml2rfc does: it paginates itself 
and adjusts the ToC accordingly. Once you feed the nroff output, be it 
from NRoffEdit or xml2rfc, into a standard nroff process, it'll get 
incorrect as soon as page breaks change.

This is relevant as changing vertical whitespace is one of the remaining 
reasons why the RFC Production Center uses nroff today (besides archival).

Best regards, Julian

---

From: Martin Rex <[email-masked]>
Date: Thu, 17 Mar 2011 16:36:30 +0100 (MET)

> incorrect as soon as page breaks change.

The page break will NOT change.  NRoffEdit produces the exact same
page breaks as "the standard nroff process". 

-Martin

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Mar 2011 16:42:36 +0100

> page breaks as "the standard nroff process".

Yes.

But it can only maintain the ToC as long as it is used.

The RFC Production Center, according to the xml2rfc SoW, does *not* use 
NRoffEdit, but a standard nroff installation.

Once you export from NRoffEdit, and start changing page breaks, the ToC 
will become incorrect.

If you say *inside* the tools (be it NRoffEdit or xml2rfc), there is no 
such problem.

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Thu, 17 Mar 2011 11:50:39 -0400

> page breaks as "the standard nroff process".

More context: At issue is the use of nroff by the RFC production center, 
which *does* manipulate the page breaks through hand tweaking.

     Tony Hansen
     [email-masked]

---

From: Stefan Santesson <[email-masked]>
Date: Thu, 17 Mar 2011 16:52:16 +0100

Julian,

I'm not sure what you have in mind that would change the page breaks.

NroffEdit accomplish this by iterating the task a number of times.

The following steps are executed:

1) Analyzing the Nroff document to determine which headings are present
and their data.
2) Compiles the text document.
3) Analyzing the text document to figure out on which pages the headings
appeared
4) Pasting the preliminary ToC in the Nroff document
5) compiling the text to allow pages breaks to adjust due to the added ToC
6) Analyzing the text to figure out where the headings finally ended up
7) Updating the ToC with final data.

These steps happens each time you press update <F3> or when you save the
document. The ToC is always correct in the compiled txt document.

/Stefan

On 11-03-17 3:12 PM, "Julian Reschke" <[email-masked]> wrote:

>_______________________________________________
>Ietf mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/ietf

---

From: Stefan Santesson <[email-masked]>
Date: Thu, 17 Mar 2011 16:55:33 +0100

OK, I understand what you say now.

All they have to do is to run it through NroffEdit once more after they
are done with their nroff editing.
They don't use NroffEdit as their main tool for nroff editing, but they do
have it and use it (at least last time I talked to them).

But agreed, NroffEdit can not maintain the ToC while the document is being
changed in another editor. That would be a tough one :)

/Stefan

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Mar 2011 17:00:11 +0100

> I'm not sure what you have in mind that would change the page breaks.

The RFC Production Center sometimes adds forced page breaks to manage 
vertical white space. When they do that in the nroff file, the ToC will 
get out of sync.

> These steps happens each time you press update<F3>  or when you save the

Yes. YES! (sorry :-).

But this thread is NOT about NRoffEdit, it's about how to automatically 
generate a ToC with NRoff. Because that's what the RFC Production Center 
uses and what is specified as target in the xml2rfc SoW.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Mar 2011 17:03:06 +0100

> changed in another editor. That would be a tough one :)

If this (running NroffEdit as a postprocessing step) could be 
established as standard procedure, this would simplify the output target 
for the xml2rfc SoW.

Best regards, Julian

---

From: John Levine <[email-masked]>
Date: 17 Mar 2011 16:26:51 -0000

>established as standard procedure, this would simplify the output target 

The current xml2rfc already does pagination and generates the TOC for
the text version, so the extra work to emit them surrounded by nroff
codes should be pretty minimal.

If we're going to put more work into xml2rfc, I would much rather
figure out what the production people are doing with nroff that
xml2rfc doesn't currenty do, and add twiddeles so they can do that in
xml2rfc and skip the nroff completely.

R's,
John

---

From: Tony Hansen <[email-masked]>
Date: Thu, 17 Mar 2011 12:36:05 -0400

> xml2rfc and skip the nroff completely.

Yup, this exactly matches conversations I and others have been having 
with the RFC production center.

Conversations along these lines have also been a part of why there's the 
xml2rfc SoW currently in progress: to generate a better code base from 
which modifications to xml2rfc can be more easily made.

     Tony Hansen
     [email-masked]

---

From: Stefan Santesson <[email-masked]>
Date: Thu, 17 Mar 2011 22:34:33 +0100

It's up to them, but it could easily be done if they want to.

It could even easily be done even if there is no nroff since NroffEdit can
generate nroff from text and then generate the ToC.

/Stefan

---

From: John C Klensin <[email-masked]>
Date: Mon, 21 Mar 2011 07:28:38 -0400

--On Thursday, March 17, 2011 12:36 -0400 Tony Hansen
<[email-masked]> wrote:

>> If we're going to put more work into xml2rfc, I would much
>> rather figure out what the production people are doing with
>> nroff that xml2rfc doesn't currenty do, and add twiddeles so
>> they can do that in xml2rfc and skip the nroff completely.
> 
> Yup, this exactly matches conversations I and others have been
> having with the RFC production center.
> 
> Conversations along these lines have also been a part of why
> there's the xml2rfc SoW currently in progress: to generate a
> better code base from which modifications to xml2rfc can be
> more easily made.

Tony,

While I believe this is a fine objective, I want to point out
one issue: the big advantage of generic markup (XML or
otherwise) over finely-controlled formatting markup (nroff or
otherwise) is that the former eliminates the need for authors
(and others early in the publication process) to worry about
formatting and, indeed, keeps them away from it.  The more one
goes down the path of letting (or, worse, encouraging or
requiring) authors fine-tune formatting and layout, the more we
reduce the advantages of generic markup.  In the extreme case,
xml2rfc could deteriorate into what might be described as nroff
plus a bunch of macros in an XML-like syntax.

I don't think we are there or that we are at immediate risk of
going there.  But I think we need to exercise caution.

In particular, if the idea is for the RFC Production Center to
be able to do detailed formatting (like page boundary tweaking)
using the general xml2rfc syntax and tools, I suggest that:

First, people think about whether there is a way to express the
requirements generically.   For example, a lot of the page
boundary tweaking that the Production Center has to do is
because the xml2rfc processing engine isn't good enough at
handling widow and orphan text.   If changes were made to the
engine to, e.g., bind section titles more closely to section
body text, and generally to permit the needed relationships to
be expressed better in generic markup, the requirement for
formatting-tweaking might be greatly reduced.

Second, if formatting control must be (further) introduced into
xml2rfc in order to make page layout control possible, can we do
it by inventing a processing directive family separate from
"<?rfc..."? If we had "<?rfc..." as something I-D authors were
expected to use a "<?rfcformat..." as something used only in
final formatting production, possibly even generating a comment
from nits checkers if present earlier, we would be, IMO, lots
better off --and lots closer to common publications industry
practice-- than mixing them together.

    john

---

From: Tony Hansen <[email-masked]>
Date: Mon, 21 Mar 2011 09:24:37 -0400

> formatting-tweaking might be greatly reduced.

John,  we're in total agreement here. And improved widow+orphan control 
is definitely top of the list in my mind for what would help the RFC 
Production Center the most -- doing it as a generic change within the 
formatting engine without requiring additional markup directives is 
certainly preferable to the other alternatives.

> practice-- than mixing them together

I think this is an excellent idea.

Further discussion of specific improvements and how to accomplish them 
should probably be done on the xml2rfc mailing list, as well as 
meta-discussions on how to vet ideas for improvement.

     Tony Hansen
     [email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Mar 2011 16:39:49 +0100

+1000

John, thanks for this good explanation!

Best regards, Julian

---

From: Stefan Santesson <[email-masked]>
Date: Thu, 24 Mar 2011 02:14:06 +0100

I can't escape the feeling that this discussion of using markup language
editing to produce RFCs, is a bit upside down.

I'm much more concerned with draft writers having to deal with markup
syntax than I am about drafters trying to put a page break in a sensible
location, or format their text in a readable fashion.

The latter is not a problem that consumes a lot of energy, neither do I
believe that drafters concern with readability is a matter that causes the
RFC production center a lot of headache. So why is this a matter of
concern?

I honestly think people waste a lot more time trying to figure out how to
properly form correct markup syntax, than they do with format tweaking.

In my ideal world, where XML would work at its best, drafters would
concentrate on writing text in a fashion that could be captured into XML
(or any functional markup language), making XML the output of the editing
process rather than the input.

That way it would not hurt the drafters if the XML syntax was extended to
capture both content and format, making it a complete input to the
rendering process.

Given the rather primitive structure of RFCs, writing such editor seem not
to be such a grim task. I'm even tempted to provide one in the next major
version of NroffEdit, where you could choose nroff and/or XML as markup,
but never bother with it when writing your draft.

If the XML syntax was expanded to capture formatting then that could
indeed render nroff obsolete . Right now I would have to keep relying on
nroff to ensure that the document is rendered as written.

/Stefan

---

From: [email-masked]
Date: Thu, 24 Mar 2011 13:48:16 -0700 (PDT)

> properly form correct markup syntax, than they do with format tweaking.

My experience has been the exact opposite. Markup syntax is a known quantity
that is easily accomodated, especially if you use a markup-aware editor. The
editor I use closes elements automatically, provides constant syntax checks,
and lets me toggle sections of the document in and out of view.

It's been a very long time since I've given any real thought to the supposed
difficulties of dealing with markup syntax.

But page breaks... I have on more occasions than I care to recall spent a
swacking big chunk of time adjusting them. Fix one widow, an orphan appears
somewhere else. And yes, I realize this is not really necessary for I-Ds, but
when the breaks are really bad I just can't help but try and fix them.

> process rather than the input.

Brian Reid once came up with a nice term for what results when this goal is
pursued to it's logical conclusion: What You Get is What You Deserve.

> but never bother with it when writing your draft.

The task may not be grim, but the end results of such exercises - and there
have been a lot of them - usually are.

				Ned

---

From: Stefan Santesson <[email-masked]>
Date: Thu, 24 Mar 2011 22:49:30 +0100

Ned,

>difficulties of dealing with markup syntax.

But you are probably pretty experienced user and you probably spent some
time setting up your environment to get where you are.

I believe having to deal with markup syntax poses a significant barrier to
those not as experienced as you.

>when the breaks are really bad I just can't help but try and fix them.

It's been a very long time since I experienced any problem with
formatting. :)
That was in the old days when I used a separate Nroff compiler. Using
NroffEdit's side by side view of source and text has completely removed
that issue for me.
And I think that is true also for an inexperienced user.

>pursued to it's logical conclusion: What You Get is What You Deserve.

Great one.... And so true.

>have been a lot of them - usually are.

I believe you are right, looking in the mirror. But time changes. I think
this is an area where open source development and open source libraries
really has provided a revolution. If we start with creating the
specifications that would allow such tool to be created, then you don't
need a huge software organization and kazillions of dollar any more to
piece together something that actually could be really useful..

I know I'm an idealist.. I still believe in simplicity.

/Stefan

---

From: [email-masked]
Date: Thu, 24 Mar 2011 14:53:58 -0700 (PDT)

> time setting up your environment to get where you are.

The answer is no to both. When I first started using xml2rfc I don't think I
had written a single line of XML. As for setting up the editing environment, I
installed a new version of BBEdit, done.

The hard part was actually getting xml2rfc up and running. This was back when I
was using Mac OS 9, and getting that combination working was a bit of a pain.
(I think my directions on how to do it eventually ended up in the readme.)

Since then I've done much more advanced work with XML involving schemata,
Relax, stylesheets, and so on and so forth. BBEdit is in no way adequate for
that stuff, and truth be told I'm not happy with any of the more powerful tools
I've tried. (I'm currently limping along with Exchanger, but it hasn't been
updated in a long time.)

> those not as experienced as you.

I realize that's your belief. But after having watched multiple technical
neophytes learn how to produce better results with TeX than I usually can - and
when it comes to markup, the rules for XML are the essence of simplicity
compared to TeX - it is not a belief I share.

> I know I'm an idealist.. I still believe in simplicity.

As do I, but what I'm not seeing is an approach that leverages all the stuff
you mention to actually produce a different outcome.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 25 Mar 2011 03:35:06 -0000

>I believe having to deal with markup syntax poses a significant
>barrier to those not as experienced as you.

>From long experience, I can assure you that whatever you are used to
seems obvious and natural, and whatever you aren't seems strange and
difficult.  I think nroff is swell, having been using it since about
1973, but on the rare occasions I show it to someone, their reaction
is about what I'd get if they looked into my kitchen and saw a bunch
of goat legs, a pile of coal, and a bellows.

Lots of people use XML editors like bbedit and xxe. They're not my
favorite, but they seem quite popular. Some of us weenies hand-edit
XML, which is not noticably more painful than hand-editing nroff or
any other text based markup language. (It's a lot easier than
hand-editing RTF, though.)

XML is clearly the future. The tools are way better, like saxon and
xsltproc which use templates to turn XML into web pages or whatever
else you want.  (And no, running nroff source through troff isn't the
same thing.)  If you want to use nroff, that's fine, but it's really a
dead end.

R's,
John

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Mar 2011 14:08:18 +0100

Hijacking this thread...:

There'll be an xml2rfc related session on Sunday:

   1500-1650  Tools for Creating Internet-Drafts Tutorial - Congress 
Hall I

...which I'll try to attend (and maybe demo stuff, and answer questions 
around the XSLT implementation, and the various tools that come with it).

Best regards, Julian