Subject: Re: [art] New RFCs text formatting

From: John C Klensin <[email-masked]>
Date: Thu, 28 Nov 2019 11:33:01 -0500

(copying the IETF list -- this is not an ART matter alone)

TL;DL warning: while not terribly long, this is a rant.

--On Thursday, November 28, 2019 01:50 +0100 Steffen Nurpmeso
<[email-masked]> wrote:

>...
> Yah, hm, hm.  It surely has merits, both HTML as well as PDF.
> I have a local RFC pool since, pooh, maybe 2003 or so.  The
> german computer magazine c't once put all (?) of them on one
> of their CDs by then, and i am only throwing onto that stock
> ever since. I personally am almost perfectly served with plain
> grep(1) on plain text.  Actually i do not even really care
> about missing form feeds, i was only surprised.

Hi.  I realize it is very late, probably too late, to be
complaining about this, but my clear recollection was that, when
the v3 project was started, the community was promised that,
while the authoritative/ canonical/ archival format would shift
to the XML and fully-functional HTML and PDF format (not just
HTML and PDF produced from plain text forms) would become
available, the plain text format would be preserved sufficiently
that tools [1] that were designed for, and used with, RFCs for
the last 40-odd years would continue to work and work equally
well [2].  If page breaks, running headers and footers, and TOCs
and indices with page references have disappeared, some of those
tools will cease working (even if grep still does) and that
violates that particular commitment.  If there were an xml2rfc
v3 option that was guaranteed to be supported and that would
generate the older format, even if that format were never stored
on IETF servers, I think that would be sufficient to let us (and
our tools) keep working, but that does not seem to be the case
at present (nor does it address any TOC or index issues).[3]

This is especially important because at least some of us who
prefer the text form when working with RFCs (even if some of us
might prefer something else for reading them), after losing the
argument about whether XML (or any other generic or formatting
markup arrangement) was appropriate as the canonical/archival
form essentially tuned out of the discussions of xml2rfc v3
details and decided to devote our available IETF time to
technical standards and other work that we felt had impact on
the Internet rather than just on the IETF and how it operated.
The tuning out was reinforced as it became clear that xml2rfc v3
was evolving into a odd hybrid between generic and formatting
markup, one that was as or more likely to share the disadvantage
of each rather than the advantages and that, in terms we use
when talking about networking, raises all of the issues of layer
violations.  So we didn't read the reports and documents
carefully; we just assumed that, as long as the plain text
format was stable (pagination and all), we would continue to be
able to work without having to spend time converting habits and
tools.

Perhaps what all of this means is that I (and a few others) are
just old and stuck in our ways and that the IETF would be better
off if we just retired or otherwise moved on.   The "old" part
is certainly true for some of us but I'm not convinced about the
"better off" part.  However, by requiring us to develop new
tools or learn new tricks, this change may have the effect of
raising the bar for effective participation in the IETF enough
to push us out.   That is especially likely when we need to
review and revise older documents.  Personally, I think that
pushing anyone out who is willing to do work and might have
useful contributions to make is a bad idea.

So, even if it is too late to reconsider these decisions, I hope
we can at least be aware of possible negative side effects.

best,
   john

[1] The concern is about user-developed tools, used by one
person or a cluster of people who have shared them, not ones
developed or maintained under IETF auspices.

[2] Reading RFC 7994 only after these changes were identified, I
find that Sections 4.1ff remove headers and footers and the page
numbers from the TOC.  But the introductory paragraph of Section
4 does say 

	"One plain-text output will be created during the
	publication process with basic pagination that includes
	a form feed instruction every 58 lines at most,
	including blank lines."

So perhaps that omission is a bug.

[3] The last argument for removing pagination in Section 2.1.3
of RFC 6949 is almost completely bogus.   The problems with
unbroken sections that span several pages, including
difficulties with references to material in those sections, was
noticed not long after RFC 1543 was published with its specific
comments about cross-references to sections and not pages, IIR
much earlier.  There was discussion even before then about
insisting on numbered paragraphs, an idea that was abandoned in
large measure because of difficulties with document preparation
and tooling.  For some years, the RFC Editor pushed back
aggressively or (or simply modified) such very long sections,
especially when there were cross-references to them (by either
section number or page number).  Later, for reasons unrelated to
this rant, the pushback became much less frequent.  The problem
is over-long sections; the solution is to manage RFC Style
decisions (during final editing if not earlier) so that they
don't happen, not to pretend that eliminating page numbers will
eliminate those sections by precluding the possibility of using
page number for more granular references.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2019 17:59:43 +0100

> we can at least be aware of possible negative side effects.

It is never too late to fix this bug.

I think the last few months have uncovered a lot of usages for the plain text form of RFCs, as well as the fact that “nobody will ever look at the plain text now that we have HTML” is maybe true for 80 %, but not for 100 %.

Grüße, Carsten

---

From: "John Levine" <[email-masked]>
Date: 28 Nov 2019 12:25:43 -0500

>It is never too late to fix this bug.

One of the biggest advantages of making the XML the canonical format
is that the other formats are *not* canonical so they can be recreated
in whatever form we want.

If people want a line printer page image version with form feeds, I
don't think that would be particularly hard to arrange in addition
to the other formats.

R's,
John

---

From: Claudio Allocchio <[email-masked]>
Date: Thu, 28 Nov 2019 18:55:06 +0100 (CET)

and just let me add to John, that "aestetic" is not always a synonym od 
"funcionality". More machine processable formats are ok if there is an 
advantage out of their use, but if not... then, why?

It's late... but do not foget this

PS: no I'm not a nostalgic of TXT formatting, expecially of the itme when 
there were no toold than "vi" or equivalent an formatting according to 
rules was a real pain (and very few wanted to be I-D editors also because 
of this)...

On Thu, 28 Nov 2019, John C Klensin wrote:

> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art
>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

      PGP Key: https://www.cert.garr.it/servizi/informazioni-su-pgp-keys

---

From: "Paul Hoffman" <[email-masked]>
Date: Thu, 28 Nov 2019 10:25:05 -0800

On 28 Nov 2019, at 8:33, John C Klensin wrote:

> well [2].

I strongly believe you are wrong: no such promise was made. If you can 
find any actual message that made such a promise, I'm happy to stand 
corrected.

--Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Thu, 28 Nov 2019 20:05:37 +0100

> developed or maintained under IETF auspices.

Do you have a concrete case of a tool that fails with the new format?

> So perhaps that omission is a bug.

AFAIR, the intent was to *paginate*, nothing more (yes, no headers,
footers, no page numbers).

Best regards, Julian

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 28 Nov 2019 11:57:56 -0800

> find any actual message that made such a promise, I'm happy to stand

https://www.rfc-editor.org/rse/format-faq/#10 points to
https://www.rfc-editor.org/rfc/rfc6949.html#section-3.3, which explicitly
retires pagination as a goal.

Jeffrey

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 29 Nov 2019 09:30:17 +1300

> footers, no page numbers).

What I care most about is to have a side-by-side diff format as useful as rfcdiff produces. From a practical point of view, that has been the greatest benefit of pagination and fixed width font for many years now. The ability to print on a line printer or a Diabolo hasn't been important for 25 years or so.

   Brian

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Thu, 28 Nov 2019 12:34:14 -0800

Step 1, apply the following patch to xml2rfc:

---
 cli/xml2rfc/run.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cli/xml2rfc/run.py b/cli/xml2rfc/run.py
index fc933f4..e75832f 100755
--- a/cli/xml2rfc/run.py
+++ b/cli/xml2rfc/run.py
@@ -407,7 +407,7 @@ def main():
     # Remember if we're building an RFC
     options.rfc = xmlrfc.tree.getroot().get('number')
     if options.rfc:
-        options.pagination = False
+        options.pagination = True

     # Check if we've received a version="3" document, and adjust accordingly
     if xmlrfc.tree.getroot().get('version') == '3':
-- 
2.24.0

Step 2: For all the RFCs published using xmlrfc 3, download the xml file.

Step 3: Regenerate the txt files from the xml files using the patched xml2rfc.

-- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: https://marc.petit-huguenin.org
Profile: https://www.linkedin.com/in/petithug

---

From: "Paul Hoffman" <[email-masked]>
Date: Thu, 28 Nov 2019 12:58:29 -0800

> retires pagination as a goal.

Yes, of course. That's a significant amount less that what John was 
(possibly mis-remembering) in the "promise" he spoke about.

--Paul Hoffman

---

From: Keith Moore <[email-masked]>
Date: Fri, 29 Nov 2019 00:22:55 -0500

> the last 40-odd years would continue to work and work equally

FWIW, I also recall this promise, at least being made verbally.

I hope it's not too late to fix this.

Keith

---

From: Bob Hinden <[email-masked]>
Date: Fri, 29 Nov 2019 08:35:22 -0800

Carsten,

> On Nov 28, 2019, at 8:59 AM, Carsten Bormann <[email-masked]> wrote:
> 

I think that is about right, I use both html and txt.  Html for reading on the screen, txt for doing comments and markups, excerpts in email, etc.

I have one active document that I converted to V3.   I note that the txt output of XML2RFC produces headers, footers, and page breaks.

The other thing I have noticed, since I now have the latest draft produced by V3 and the previous drafts produced by V2, is that several things changed that make this transition more difficult than needed.

The first I have found was when doing the conversion from V2 to V3, the default was to expand references, instead of the one line version.   This can be worked around when using the

--add-xinclude

when doing the conversion.

The other was default symbols in lists.   These can be worked around by using the --legacy-list-symbols when running XML2RFC.

My point is that in both cases I think the defaults should have been to make it closer to V2.  The current defaults make it harder to deal with converting an existing in progress draft from V2 to V3.  I suspect these are issues that many will run into when starting to use V3.

Thanks,
Bob

---

From: John C Klensin <[email-masked]>
Date: Fri, 29 Nov 2019 17:18:04 -0500

--On Friday, 29 November, 2019 09:30 +1300 Brian E Carpenter
<[email-masked]> wrote:

>> AFAIR, the intent was to *paginate*, nothing more (yes, no
>> headers, footers, no page numbers).
> 
> What I care most about is to have a side-by-side diff format
> as useful as rfcdiff produces. From a practical point of view,
> that has been the greatest benefit of pagination and fixed
> width font for many years now. The ability to print on a line
> printer or a Diabolo hasn't been important for 25 years or so.

I think others have commented on the assumption that pagination
is good only if the document is headed for a line printer, but
it may be worth pointing out that a recent errata submission may
have identified a weakness in rfcdiff.  An erratum submitted
last April complained about an extended example in RFC 5321
being internally inconsistent about leading spaces that turn out
to be important.  (see https://www.rfc-editor.org/errata/eid5711
for details, but they aren't important).  The problem is that
the XML source appears to be correct -- leading spaces in both
parts of the example -- but the XML output (under v2) isn't.
This wasn't caught caught during AUTH48 or earlier, or
apparently in the decade between 5321's publication and the
erratum, at least in because the author was relying on rfcdiff
and it is apparently insensitive to leading, as well as other
types of spaces.   I don't think that is a case for an emergency
fix to rfcdiff, especially one that might have side-effects I
haven't thought about, nor a fix to v2 of xml2rfc, but it should
be another reminder than we two be very careful about what
xml2rfc produces, especially in a world in which the XML is the
canonical form but that there is no requirement that the IETF's
favorite xml2rfc processor be used to process that source (and I
don't know enough to know whether the IETF-approved Style Sheet
covers this situation or not).

best,
   john

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 29 Nov 2019 23:34:57 +0100

On Nov 29, 2019, at 23:18, John C Klensin <[email-masked]> wrote:
> 
> identified a weakness in rfcdiff

The RFC 7386/7396 accident was also caused by this (root cause was apparently the use of HT characters, which need to be expunged from the universe, but rfcdiff helped in not detecting the problem before publication).

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sat, 30 Nov 2019 06:22:04 +0100

> The RFC 7386/7396 accident was also caused by this (root cause was apparently the use of HT characters, which need to be expunged from the universe, but rfcdiff helped in not detecting the problem…

rfc2629.xslt has been warning about HTAB since 2014-10-16 (likely
triggered by that accident). xml2rfc has a ticket open about that:
<https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/403> (dated
2019-04-14, still in status "new").

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 30 Nov 2019 06:38:18 +0100

> parts of the example -- but the XML output (under v2) isn't.

So what caused the problem in the XML output? (my copy of rfc5321.xml
from AUTH48 seems to be truncated, so I can't tell).

> types of spaces.   I don't think that is a case for an emergency

That's a good reminder. I had a similar issue in the past, which I
detected when doing an *actual* diff instead of rfcdiff. It's easy to
forget though.

> covers this situation or not).

Well, we'd have to know about the actual cause. If it was an HTAB (as
suggested by Carsten), then the answer is:

- rfc2629.xslt added a warning back in 2014

- RFC 7749 warns about HTAB in
<https://greenbytes.de/tech/webdav/rfc7749.html#element.artwork>: "Note
that processors differ in the handling of horizontal TAB characters
(some expand them, some treat them as single spaces), and thus these
ought to be avoided."

- RFC 7791 makes it an error:
<https://greenbytes.de/tech/webdav/rfc7991.html#element.artwork>: "Tab
characters (U+0009) inside of this element are prohibited."

- I'm not sure what xml2rfc does in the current version, but I reported
a bug back in April
(<https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/403>)

Finally, what do you mean by "IETF-approved Style Sheet"?

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Sat, 30 Nov 2019 21:26:54 +0100

Hi,

Sorry for becoming aware of this thread a bit late.  More below:

> I hope it's not too late to fix this.

The xml2rfc default setting for v3 text output for drafts is to paginate.

There's an option to turn off pagination for drafts.

Text output for RFCs is currently always unpaginated.

During development of the v3 text output it's been very clear, from the
specifications and also when I asked the RFC-Editor if the matter of
pagination had been reconsidered, that pagination of RFCs was not to be
done.

If being able to generate paginated text output for published v3 RFC XML
files is desirable, I'm happy to add a switch to do that.

Best regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 30 Nov 2019 23:22:27 +0100

> files is desirable, I'm happy to add a switch to do that.

Please do.

Grüße, Carsten

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Nov 2019 17:58:45 -0500

On 11/30/19 5:22 PM, Carsten Bormann wrote:

> On Nov 30, 2019, at 21:26, Henrik Levkowetz<[email-masked]>  wrote:

Yes, please do.   Though IMO this should be the default. (paginated, not 
necessarily including page numbers).

Keith

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 30 Nov 2019 18:46:08 -0500

It seems that it ought to take a consensus call (here, rfc-interest, 
somewhere else?) to change the default output prouced by the IETF tools. 
  The question of pagination of output was more contentious than I would 
have expected in previous rounds.

Yours,
Joel

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Nov 2019 18:55:41 -0500

On 11/30/19 6:46 PM, Joel M. Halpern wrote:

> somewhere else?) to change the default output prouced by the IETF 
> tools.  The question of pagination of output was more contentious than 
> I would have expected in previous rounds.

It appears that having the canonical format of RFCs be a format other 
than one that people actually use, has "interesting" consequences.   For 
example, anyone can generate RFCs in their own preferred format, using 
whatever typeface, pagination, TOC format, references format, etc. they 
wish, and it's difficult to tell (either by looking or comparison with 
the copy from the RFC Editor's site) whether such an RFC is genuine.

I think it would also be "interesting" if the preferred source for RFCs 
became a site other than rfc-editor.org.    (Which, for all I know, 
might already be the case.)

Keith

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 1 Dec 2019 01:16:02 +0100

Hi Keith, Carsten,

On 2019-11-30 23:58, Keith Moore wrote:
> Yes, please do.

Noted.

> Though IMO this should be the default. (paginated, not 

Making this the default for RFCs is a decision for the RFC Editor, I believe.

	Henrik

---

From: "John Levine" <[email-masked]>
Date: 30 Nov 2019 20:24:48 -0500

In article <[email-masked]> you write:
>wish, 

That's a feature.  If you don't like the format, you can pick another.
If you don't like any of the existing formats, you can write or adapt
tools to create any format you want.  If, say, you want line printer
pages for A4 paper, you can have them.

>and it's difficult to tell (either by looking or comparison with 

Aw, c'mon.  If that's a problem, take the canonical XML and run it through
your favorite formatter yourself to get a fresh authentic copy.

>might already be the case.)

If you google an RFC number you'll generally find links to
tools.ietf.org and datatracker.ietf.org ahead of rfc-editor.org, which
has been the case for a long time.

R's,
John

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Nov 2019 20:49:05 -0500

> pages for A4 paper, you can have them.
In some ways I agree that it is a feature.   But it may also bring some 
undesirable consequences with it.
> your favorite formatter yourself to get a fresh authentic copy.

On several occasions I've been asked to verify the authenticity of old 
documents (including some Usenet posts, some RFCs, and some academic 
papers) for patent litigation purposes.   Sometimes what I've been asked 
to do is verify a document that has already been used as an exhibit in a 
prior filing, so it's important that the version of the document that 
was used in the exhibit be verified.   If the exhibit was of a plain 
text RFC, it's pretty easy to do - download the same plain text RFC from 
rfc-editor.org, verify its date (say using FTP to the file server, and 
perhaps via other means, like an archived email announcement that was 
sent concurrent with RFC publication), and optically compare the two 
documents.

But on several occasions when I've been asked to do this, the exhibit 
was printed-out HTML and the document that could be readily 
authenticated (the one that the HTML was derived from) was in a 
different format.   It can be tricky to explain the differences in a way 
that is convincing to a jury.

I may be more sensitive to this issue than some because I've had to do 
this several times in 2019.    But having the old canonical format for 
RFCs also be the one that was most widely read, was indeed a feature.

Keith

---

From: John C Klensin <[email-masked]>
Date: Sat, 30 Nov 2019 21:32:57 -0500

Joel,

FWIW, I think that measuring consensus on rfc-interest and
assuming that represented consensus of RFC users, and others
dependent on them, or even that of the IETF, may be what got us
into this mess.
 
   john\

--On Saturday, November 30, 2019 18:46 -0500 "Joel M. Halpern"
<[email-masked]> wrote:

> It seems that it ought to take a consensus call (here,
> rfc-interest, somewhere else?) to change the default output
> prouced by the IETF tools.   The question of pagination of
> output was more contentious than I would have expected in
> previous rounds.
> 
> Yours,
> Joel
> 
>>>> If being able to generate paginated text output for
>>>> published v3 RFC XML files is desirable, I'm happy to add a
>>>> switch to do that.
>>> Please do.
>> 
>> Yes, please do.   Though IMO this should be the default.
>> (paginated, not  necessarily including page numbers).
>> 
>> Keith
>> 
>> 
>

---

From: Joel Halpern Direct <[email-masked]>
Date: Sat, 30 Nov 2019 22:12:17 -0500

That (the suggestions that RFC-interest is the wrong place to judge such 
things) may be, which is why I listed alternatives.  But just saying 
that two people liked the idea, and no one screamed, in one day, is 
clearly NOT enough to change the default.

Yours,
Joel

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 30 Nov 2019 22:59:20 -0500

> clearly NOT enough to change the default.

Quite. The botched pagination of the old plaintext format was one of the
main reasons I hated it. The parochial US-centric design ensures that it
only works reliably on US paper which is only used in North America.

So for years, I would have to make multiple attempts at tweaking page
lengths on printers to get RFCs to print out right. And after a while, I
gave up, and wrote myself a PERL script to convert them to HTML and those
are the docs I worked from all the time I was at the Web consortium and
then at VRSN until xml2rfc appeared with a HTML output.

I have never used the plaintext format. I have always regarded it as
garbage. This is a technology business. If we are going to get nostalgic
about stuff, perhaps we should start with all the newspaper people,
shopworkers, etc. etc. we put out of work. It seems rather incongruous and
not a little self-indulgent to find the vanguard of the digerati waxing
nostalgic over emulating 1960s line printer formats while they are merrily
shredding the fabric of the old economic order for everyone else.

What does concern me with the current state of affairs is that we have HTML
output of RFCs but not yet of IDs it would seem.