Subject: A modest proposal - allow the ID repository to hold xml

From: "Rosen, Brian" <[email-masked]>
Date: Wed, 27 Aug 2003 18:15:50 -0400

The list has been a little quiet this week, so starting a flame war
seems appropriate.  Harold noted that we had not had a format flame
in some time.

I therefore have a modest proposal:

Allow the submission of an xml file meeting the requirements of RFC2629
along with the text file (and optional ps file) for an Internet Draft.

This would change the "Guidelines to Authors of Internet-Drafts" 
document to add: "XML marked-up text is acceptable, but only when submitted 
with a matching ASCII version.  The xml file should be in conformance
with RFC2629."

I specifically propose a MAY strength, not a SHOULD

I specifically propose that we NOT change the RFC process at this time

I am not proposing any syntax checking, or use of any specific tool,
or any html output, or anything else other than allowing xml markup
on the text if the submitter so chooses to include it.

We have had enough experience with 2629 to know that it is of some value.  
I see that an interesting number of submissions appear to have been 
created with xml, and the wonderful xml2rfc tool (thank you Marshall 
and friends).  I merely seek to have a place where the xml files may be 
found, if they exist.

The additional disk space should not be a large burdon, I would hope.

Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 02 Sep 2003 12:09:52 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "Rosen" == Rosen, Brian <[email-masked]> writes:
    Rosen> I therefore have a modest proposal:

    Rosen> Allow the submission of an xml file meeting the requirements of
    Rosen> RFC2629 
    Rosen> along with the text file (and optional ps file) for an Internet
    Rosen> Draft. 

  This is a great idea.

    Rosen> This would change the "Guidelines to Authors of Internet-Drafts" 
    Rosen> document to add: "XML marked-up text is acceptable, but only when
    Rosen> submitted  
    Rosen> with a matching ASCII version.  The xml file should be in
    Rosen> conformance 
    Rosen> with RFC2629."

  I think that the DTD that xml2rfc uses has evolved a bit since 2629.
  The other tools have evolved too, I think. Point being that we may have
to do 2629bis first.
  It would be so nice to be able to refer to the draft itself in order to
cite it :-)

  I'd like it if the charters were also available by rsync.  
  Even better if we started using a directory per WG...

  I do the following with my drafts:
	  1) sort them by WG name.
	  2) copy the charter.
	  3) link in any RFCs that the charter references.

===

#!/usr/bin/perl

chdir('/corp/ietf');

system("rsync -avz ftp.rfc-editor.org::rfcs-text-only ftp.ietf.org/rfc");

system("rsync -avz optimus.ietf.org::internet-drafts ftp.ietf.org/internet-drafts");

system("cd html.charters; wget -r -l 1 -nv -np -nd -nc http://www.ietf.org/html.charters/");

opendir(DRAFTS,"ftp.ietf.org/internet-drafts");
@drafts=readdir(DRAFTS);
closedir(DRAFTS);

foreach $draft (@drafts) {
    next if ($draft =~ /^\./);
    $draft =~ m,draft-([^-]*)-([^-]*)-(.*),;
    
    if($1 eq "ietf") {
	$dir = "id/$1/$2";
	$base= $3;
    } else {
	$dir = "id/$1";
	$base ="$2-$3";
    }

    next if (-f "$dir/$base");

    print "$draft -> $dir/$base\n";
    system("mkdir -p $dir") unless (-d "$dir");
    link("ftp.ietf.org/internet-drafts/$draft","$dir/$base");
} 

opendir(CHARTERS,"html-charters");
@charters=readdir(CHARTERS);
closedir(CHARTERS);

foreach $charter (@charters) {
  next if ($charter =~ /^\./);

  if($charter =~ m,(.*)-charter.html,) {
    # got one, copy it, sanitizing the links.
    $wgname=$1;
    $infile="html.charters/$charter";
    
    $dir="id/ietf/$wgname";
    system("mkdir -p $dir") unless (-d "$dir");
    
    $outfile="$dir/$charter";
    
    if(-f $outfile ) {
      # if date of infile is older than outfile
      print STDERR "$infile: ".(-M $infile)." $outfile: ".(-M $outfile)."\n";
      
      if(-M $infile > -M "$outfile") {
	print STDERR "$infile not newer than $outfile\n";
      }
    }
    
    # okay, process it.
    open(CHARTER, $infile) || die "Can not open $infile: $!\n";
    open(OUTFILE, ">$outfile")|| die "Can not open $outfile: $!\n";
    
    while(<CHARTER>) {
      # localize references
      
      s,href="/internet-drafts/draft-ietf-(.*\.txt)",href="\1",g;
      
      if(m,href="/rfc/rfc(.*).txt",) {
	$rfcnum=$1;
	s,href="/rfc/rfc(.*).txt",href="rfc\1.txt",g;

	#print STDERR "looking for rfc$rfcnum.txt\n";
	if(! -f "$dir/rfc$rfcnum.txt" &&
	   ! -f "$dir/rfc$rfcnum.txt.Z") {

	  $rfc4 = sprintf("%d", $rfcnum);
	  
	  print STDERR "linking: ftp.ietf.org/rfc/$rfc4.txt\n";
	  link("ftp.ietf.org/rfc/rfc$rfc4.txt","$dir/rfc$rfcnum.txt") || die "Can not link ftp.ietf.org/rfc/rfc$rfc4.txt: $!\n";
	}
     }
      
      print OUTFILE;
    }
    close(CHARTER);
    close(OUTFILE);
  }
}

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 2 Sep 2003 10:05:58 -0700

>along with the text file (and optional ps file) for an Internet Draft.

You didn't say what the additional value would be. We know the 
additional value of a .ps file (drawings that don't translate to 
ASCII art). What is the value of XML? It certainly isn't 
searchability or readability.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Eliot Lear <[email-masked]>
Date: Tue, 02 Sep 2003 10:47:33 -0700

I don't know about about you, Paul, but I'm writing my drafts using 
EMACS and Marshall's tool.  That allows for generation of HTML, NROFF, 
and text.  The HTML allows for hyperlinks, which is REALLY nice.

Eliot

---

From: Zefram <[email-masked]>
Date: Tue, 2 Sep 2003 18:56:12 +0100

>along with the text file (and optional ps file) for an Internet Draft.

The value in this would be that it provides everyone with the document
source, suitable for generating patches for the author.  This is useful,
but if it's going to be allowed with XML then we should also allow it
with nroff, which historically we haven't.  I don't have particularly
strong feelings either way, but I do think these two cases should be
treated equivalently.

-zefram

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 2 Sep 2003 11:19:29 -0700

At 10:47 AM -0700 9/2/03, Eliot Lear wrote:
>EMACS and Marshall's tool.  That allows for generation of HTML, 
>NROFF, and text.  The HTML allows for hyperlinks, which is REALLY 
>nice.

Great! Why does that mean that the XML input should be published in 
the Internet Drafts directory along with the text output?

--Paul Hoffman, Director

---

From: [email-masked]
Date: Tue, 02 Sep 2003 11:10:02 -0700 (PDT)

> searchability or readability.

On the contrary, having the base XML is a huge benefit for some searches I, and
I suspect many others, often do: I want to find what's changed from one version
to the next. All too often minor changes lead to large differences in formatted
output, and even the various tricky diff utilities don't always overcome this. 

And if diffs of the base XML aren't to your taste, it would be easy to build a
tool that would perform the comparison and produce a modified XML document with
the changes highlighted. (Indeed, I'm sure utilities to do this already exist,
although how flexible they are in regards to different XML formats I couldn't
say.)

I'm sure there are also any number of other useful tricks that can be performed
if you have a number of Internet Drafts in a structured form. This is just the
one that interests me most initially.

As far as readability goes, having the base document gives me the ability to
generate whatever format I find most convenient to view documents. That's a big
readability boost in my book. In particular, generated HTML versions can
include useful hyperlinks.

				Ned

---

From: Bill Strahm <[email-masked]>
Date: Tue, 2 Sep 2003 15:33:13 -0400

I'll give you one good reason.  And that is updating the drafts once
the initial RFC is published.  If the origional XML/.doc/input language of the 
day is available, then I don't have to spend my time converting the text
into a usable form to get the formating done easily.

For this reason only, having origional input would be useful.
(Yes I know, I can always go ask the origional editor for the input source,
but that may take time locating that person and getting access to the
input doc)

Bill

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 2 Sep 2003 13:52:38 -0600 (MDT)

> searchability or readability.

While I normally run in horror from all things XML, this is one of the few
exceptions. XML would immediately solve a number of problems I face almost
daily:

- give me a list of all the documents "belonging" to a particular WG

- for any given RFC, show me the chain of document dependencies
(obsoletes, updated by, obsoleted by) that pass through this document

- for any given RFC, generate a dependency graph based on the normative
references in this document

You have to have a structured document format to programmatically solve
these sorts of problems, and XML provides that. (While I've become quite
adept at searching rfc-index.txt with less, I really want something
better.)

And I second Ned's comments about generating *useful* diffs between
document revisions. This is especially useful if we generate drafts in XML
format.

I'm not sure how to address the problem with legacy RFCs. I'll bet we
could find volunteers to generate XML equivalents from the existing plain
text documents. (We would need an XML tag to indicate which of the plain
text or XML documents is considered authoritative.)

And just to eliminate any lingering doubt, I'll note that I'm now using
xml2rfc for all my current and future drafts. (There, *that* just gave
several people heart attacks ;-)

--lyndon

---

From: Michael Richardson <[email-masked]>
Date: Tue, 02 Sep 2003 17:35:05 -0400

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "Lyndon" == Lyndon Nerenberg <[email-masked]> writes:
    Lyndon> - give me a list of all the documents "belonging" to a particular
    Lyndon> WG 

  Ironically, it is easier with IDs than RFCs :-)

    Lyndon> - for any given RFC, show me the chain of document dependencies
    Lyndon> (obsoletes, updated by, obsoleted by) that pass through this
    Lyndon> document 

    Lyndon> - for any given RFC, generate a dependency graph based on the
    Lyndon> normative  references in this document

...

    Lyndon> I'm not sure how to address the problem with legacy RFCs. I'll
    Lyndon> bet we could find volunteers to generate XML equivalents from the

  For this kind of thing, you don't need the entire document in XML, just
the references.... I believe that all of the RFCs have been converted such
that they can be referenced. Adding what they reference wouldn't be that
hard at this point.

    Lyndon> And just to eliminate any lingering doubt, I'll note that I'm now
    Lyndon> using xml2rfc for all my current and future drafts. (There,
    Lyndon> *that* just gave several people heart attacks ;-)

]      Out and about in Ottawa.    hmmm... beer.                |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another Debian/notebook using, kernel hacking, security guy");  [

---

From: Jean-Jacques Puig <[email-masked]>
Date: Wed, 3 Sep 2003 13:46:33 +0200

> adept at searching rfc-index.txt with less, I really want something

May I add in the same direction ? The structure of the document allows
for invoking relations between nodes, which helps for expressing
elaborate search both on meta informations (WG, Author, etc.) and on
content (the actual titles and text sections). You can surely look
within all 'xmlly' available drafts for draft belonging to a specific
WG AND referencing a list of specific docs in sections whose title contains
a specific word. This would be hard to express against the txt version.

> text or XML documents is considered authoritative.)

I wonder whether this has not already been done by zvon
(http://www.zvon.org/). Concerning referencing rfcs, citation libraries
from http://xml.resource.org look rather complete.

Several additions to the xml cause:

Edition in xml allows for good modularity, e.g. with the definition
of xml entities. This is an easy way to divide work between several
editors.  Availability of the xml source helps for editors to welcome
new contributors.

What I would suggest is the following:
	- Authors provide draft in xml XOR txt . This allows for a test
	  period of xml as an alternate default format; authors do not
	  provide both, this in order to avoid incoherences.
	- rfc editor generates txt, ps, etc. versions. html version may only
	  require reference to a stylesheet, though this is currently tricky
	  because few browsers actually support xml+xsl.

Newcomers to xml should be informed of the following:
	- xml is easy.
	- Grammar/spelling tools compatible with html generally work (I use
	  ispell and aspell indifferently).
	- Document structure and well-formedness can be validated.
	- Maintaining an xml source is easy, compared to maintaining a ASCII
	  formatted txt: this is a point authors may care about.
	- xml rendering generates classic formats and newers: this is what
	  reviewers, readers care about.
	- xml users are not hackers, extremists or ninjas. Common sense and
	  good pratice is the main source of xml advocacy. Also, xml is not
	  perfect; it is only better.

-- 
Jean-Jacques Puig

[homepage] http://www-lor.int-evry.fr/~puig/

---

From: Marshall Rose <[email-masked]>
Date: Tue, 2 Sep 2003 11:03:31 -0700

> and text.  The HTML allows for hyperlinks, which is REALLY nice.

and for folks who are of the xslt persusasion, julian's html output is
very sweet...
    
/mtr
    
ps: the point being, that there's not just one tool that works with this
    stuff...

pps: oh, and did i mention the every expanding bibliographic libraries
     already in 2629... the 3gpp reference set is coming up later this
     week (thanks miguel!)

---

From: Marshall Rose <[email-masked]>
Date: Tue, 2 Sep 2003 13:17:26 -0700

> text or XML documents is considered authoritative.)

actually, carl malamud and brad burdick wrote a script back in 99 that
had a 20% success rate on the legacy rfcs. the (unofficial) xml versions
of those rfcs is available online.
    
steven connor did some work for me after that to produce xml versions of
the remaining rfcs which excluded the middle (i.e., just the front
matter and references sections got translated).
    
so, it's not as much work as you'd think, but still far more work than
i'd like...
    
/mtr

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Sep 2003 14:26:58 -0400

Paul and Brian,

Since I've been using the quite attractive XML2RFC package for 
some, but not all, of the documents I'm working on, let me make 
a few observations about this...

	(1) The potentials to get quickly from I-D format to RFC
	format, and to edit either with a wide variety of
	editing tools (including some otherwise-broken ones)
	should be considered very attractive.
	
	(2) RFC 2629 (and 2629bis) impose some formatting and
	style requirements that are more restrictive than those
	supported/ required by the RFC Editor.  At the same
	time, the RFC Editor imposes --sometimes on a
	per-document basis-- some formatting and style
	requirements that are different from those provided by
	2629 and xml2rfc.  That is not a wonderful situation,
	and it would be well to get it sorted out before an XML
	format is promoted to any official status --rather than
	being a handy, if semi-hidden, tool for those who choose
	to use it.   If we are to sort it out, a good deal of
	the burden would need to fall on the RFC Editor... in
	particular, we would need to move significantly beyond
	the traditional "look at some recent RFCs and guess what
	we want" form of specification/documentation (as well as
	the "use simple nroff forms, but we don't really want to
	tell you what to you" tradition).
	
	(3) 2629bis and xml2rfc are, as might be predicted from
	the name, perhaps more suitable for generating RFCs than
	for generating ongoing, work-in-progress, forms of I-Ds.
	While one can fuss with comments and trick delimiters to
	simulate part of the functionality, there are no good
	generational change-tracking or "note in draft"
	facilities that are desirable for, e.g., tracking an
	evolving specification being developed by a WG.

	(4) While recent versions of 2629bis and xml2rfc have
	gotten _much_ better, there are things to be referenced,
	and forms of references, in RFCs that are not well and
	obviously supported in those tools.  Again, this is
	partially an RFC Editor issue -- if there were a real
	style manual for preferred reference formats (in 2223bis
	or elsewhere) that moved significantly beyond some
	simple cases and handwaving or "look at some examples
	and then guess", I assume Marshall and his colleagues
	would rapidly move to support them.

The above points argue more or less strongly for using 2629bis 
and xml2rfc when authors find them appropriate.  They don't do 
much to argue that posting the XML formats (even when available) 
for working-draft I-Ds would be significantly helpful (they 
argue more strongly for making the XML form of RFCs officially 
available, but there are separate and complex issues there, and 
it isn't what Brian proposed).   Availability of XML format I-Ds 
would almost certainly be helpful if a document were being 
developed by a WG or other committee and people were asked to 
submit text in pre-formatted form.  So would availability of 
MSWord or nroff formats if those were the "native" form being 
used by a document editor... at least to the extent that WG 
participants had easy access to the relevant formats, tools, and 
knowledge.

So I'm led to an even more modest suggestion than Brian's, at 
least as a first step:

	Where a WG is developing a document, and the WG Chair(s)
	and/or Editor(s) conclude that availability of the XML
	(or nroff, or MSWord, or WordPerfect, or...) editing
	form of a document would be    useful to the WG's work
	in addition to the ASCII-text-format I-D, they should
	make that happen.  In many cases, the best mechanism for
	making such a working draft available is probably a link
	from a WG home page to the document itself but, when
	appropriate, the Chair or Editor may request that the
	editing form, as well as the text, Postscript, or PDF
	one, be placed in the I-D directory (and the Secretariat
	should honor that request).  Whenever and however
	editing-format documents are made available outside
	editing teams, WG Chairs and Editors should be mindful
	of the importance of posted, sequentially-numbered, I-Ds
	as a means of marking WG progress and keeping WG work
	accessible to onlookers and new participants.

So...

(i) Submitters of individual documents, and documents outside 
the IETF standards-track and WG processes, are on their own. 
This approach (and Brian's) do mean more work for the 
Secretariat, and no case has been made that it would be worth 
the effort.

(ii) Availability of working drafts, in working/editing form, 
should not become a substitute for posting of I-Ds, no matter 
where those drafts are kept/ posted.

(iii) While I personally think that the other formats are less 
likely to be useful (or more controversial, or both), there is 
no strong reason to exclusively prefer XML as a posting format 
for working drafts.  If the posting of working draft formats is 
useful, then, within some limits, document editors should be 
permitted to post whatever working draft formats are actually 
being used... as long as the ASCII text I-Ds remain available 
and accurate.

(iv) In order to reduce the likelihood of chaos, the number of 
variant forms of editing formats should be minimal and 
well-defined.  If XML is to be permitted, it should conform to 
2629bis (and it might be in the interest of the IETF and the 
proponents of that format to become more systematic and public 
about versioning of definitions and DTDs).  If nroff is to be 
permitted, it should conform to a set of macros specified by the 
RFC Editor (or someone delegated by them).  And so on.   While 
we shouldn't expect the secretariat to police those rules, some 
procedure (ideally specified by the IESG without spending a lot 
of energy on it) would be appropriate by which a member of the 
community who notices a non-conformant format may request that 
it be removed.

       john

--On Tuesday, 02 September, 2003 10:05 -0700 "Paul Hoffman / 
IMC" <[email-masked]> wrote:

>> Allow the submission of an xml file meeting the requirements
>> of RFC2629 along with the text file (and optional ps file)
>> for an Internet Draft.
>
> additional value of a .ps file (drawings that don't translate
> to ASCII art). What is the value of XML? It certainly isn't

---

From: [email-masked] (Steve Conner)
Date: Wed, 3 Sep 2003 20:07:09 -0700

XML may not be the solution to the entire RFC generation process but it
could certainly help those of use who programmatically scrape the text based
documents to extract the basic info (references, obsoletes, updates, std,
bcp, fyi, yada yada yada...)

A subset of the DTD containing the summary info submitted along with the RFC
text could be very useful.

Cheers

---

From: Michael Richardson <[email-masked]>
Date: Thu, 04 Sep 2003 19:17:48 -0400

-----BEGIN PGP SIGNED MESSAGE-----

John, your reduced proposal would mean that the XML might not hang around
during the editing/publication processs. One key thing is that the XML
be available for later use in tracking references. 
  
It also makes it easier to do RFCXXXXbis.

---

From: John C Klensin <[email-masked]>
Date: Thu, 04 Sep 2003 19:48:22 -0400

--On Thursday, 04 September, 2003 19:17 -0400 Michael Richardson 
<[email-masked]> wrote:

> John, your reduced proposal would mean that the XML might not
> hang around during the editing/publication processs. One key
> thing is that the XML be available for later use in tracking
> references.
>

Michael,

Two responses/ observations...

(1) At best, one step at a time.  I was trying to stay with the 
spirit of the original proposal, which was, as I understand it, 
"if the XML for an I-D is available, then let's post it too".

(2) Today, the RFC publication process is one of

  author-supplied nroff -> RFC Editor nroff -> RFC
  text -> nroff -> RFC or
  XML -> nroff -> RFC

In each of those cases, the RFC Editor makes changes to the 
document, and those changes are made on the RFC Editor's nroff 
version of the document.  Now, 2629bis says (more or less) "if 
the RFC Editor makes changes, make those in the XML and submit 
them to the archive".  Good plan, except that

	* there is no way to guarantee that the RFC Editor
	changes will be reflected correctly in author
	modifications to the XML.  Remember that the RFC
	Editor's text files are authoritative, not original or
	the post-editing XML files.
	
	* because the RFC Editor's notions of flexibilities in
	formatting are broader than those permitted in the
	xml2rfc kit (that is both a good thing and a bad thing),
	there is no guarantee that the changes make by the RFC
	Editor even can be accurately reflected in the XML
	source.

Now, "tracking references" is a much easier problem than "copy 
of the document source in XML that exactly matches the published 
form of the RFC".  If nothing else, it is presumably much less 
sensitive to subtle changes made by the RFC Editor.   But both 
are rather different proposals, with their own sets of 
complications and issues, than the originally-proposed "if XML 
is available for an I-D, let's post it" or my modified version 
of "if document source is available for an I-D being processed 
by a WG, let's make it available if the WG Chair(s) and 
Editor(s) think that wise".

      john

---

From: Michael Richardson <[email-masked]>
Date: Thu, 04 Sep 2003 21:55:28 -0400

-----BEGIN PGP SIGNED MESSAGE-----

John, I concur with the issue about not changing the process.

I have contemplated asking the RFC editor to insert a copy of the
XML version of the references (and XML header) into the RFC output
itself. I.e. embedded some XML at the end of every text document.
I find this a bit dis-tasteful.

---

From: "wang liang" <[email-masked]>
Date: Fri, 5 Sep 2003 11:39:35 +0800

I think there should be a work group to discuss the  new possible  format of
ID.PDF or other format can be more  efficient than just text version, Why
should we just stick to the  ASCII text.

---

From: Jean-Jacques Puig <[email-masked]>
Date: Fri, 5 Sep 2003 10:05:09 +0200

Hi !

I think we should eventually keep the txt version of IDs and RFCs as the
reference format for `consumers` (reviewers, readers...).

The availability of sources (no matter if it is xml or nroff, etc.) has
different consequences for RFCs and I-Ds.

. For RFCs, I think sources should be available, if possible in a format
chosen by consensus; xml would be my choice here. Availability of RFCs
sources would be provided only for third parties to generate other
formats (like zvon or others). RFC repository would NOT host any of html,
ps, pdf etc formats except the ASCII .txt and the sources. IETF website
might provide such formats through cgi, servlets, etc. with an
appropriate cache if IETF decides such a functionality is relevant.

. For I-Ds, sources are important in order to relieve the editorial
process. Though I'm advocating xml, I can cope with any STANDARD format
for the representation of structured documents.

Last point is I mentioned earlier that either source or txt version
should be provided to the editor, but not both. In effect, providing two
versions of what should be `semantically` the same content may lead to
incoherences within the two versions. My trust goes more to the I-D
editor than to the actual authors here (sorry :).

The last pb is: what source formats are acceptable in the repository ?
Certainly we cannot have a huge choice here.

-- 
Jean-Jacques Puig