Subject: Processing of Expired Internet-Drafts

From: Fred Baker <[email-masked]>
Date: Wed, 14 Jan 2004 08:43:58 -0800

The secretariat sent a note to ietf-announce this morning, which I mostly 
consider a step forward - internet drafts will be aged out 185 days after 
their original posting barring certain procedural caveats such as being on 
a list waiting to be published.

I wonder what thoughts people have about this paragraph, though:

At 07:52 AM 1/14/2004, The IETF Secretariat wrote:
>When an Internet-Draft expires, a "tombstone" file will be created that 
>includes the filename and version number of the Internet-Draft that has 
>expired.  The filename of the tombstone file will be the same as that of 
>the expired Internet-Draft with the version number increased by one.  If a 
>revised version of an expired Internet-Draft is submitted for posting, 
>then the revised version will replace the tombstone file and will receive 
>the same version number as that previously assigned to the tombstone 
>file.  Tombstone files will never expire and will always be available for 
>reference unless they are replaced by updated versions of the subject 
>Internet-Drafts.

This affects me in two ways.

First, I maintain Cisco's internet draft and RFC mirrors. My procedure is 
pretty simple. Every night, I run a process which

  - obtains a list of the drafts/rfcs on the IETF server, via FTP
  - creates a list of the drafts/rfcs on my mirror
  - takes a diff between the 'ls' outputs
  - downloads the new files via FTP
  - deletes the ones that are no longer there

This has an interesting side-effect in our present (I suppose I should say 
past) system: since an expired draft gets a tombstone file that lasts for 
six months but does not change the name, the name remains on the IETF 
server. Hence, in *my* mirror, an expired draft is still available until 
the tombstone is removed six months later. For my purposes, that can be a 
nice side-effect, although it is one I'm not stuck on.

If I can have two separate files (a tombstone and a subsequent new file 
version) that have the same name, as described in the recent announcement, 
I am going to have to figure out a trigger that will tell me that I need to 
re-download the file.

Second, every 4 months we post ~700 IDs. Many of these are -00, meaning 
"new name". Every ID that is posted eventually expires, which means that 
according to the above procedure we are going to accumulate empty tombstone 
files at a rate of perhaps 1000-2000 per year ad infinitum.

It seems to me that there is a better approach to the above, at least in 
the context of the above. If the "tombstone" is literally as described, it 
would be far more space/search/etc efficient for us to have the tombstone 
consist of an added text line in a file indicating that the named draft 
expired on a certain date, and keep separate files for the active internet 
drafts. It seems to me that this makes it simpler to maintain a mirror and 
to find temporary documents.

Thoughts?

---

From: Zefram <[email-masked]>
Date: Wed, 14 Jan 2004 17:11:31 +0000

Definitions:

	"draft name" = filename stem, excluding version number (e.g.,
			"draft-iab-dos")
	"version number" = two-digit I-D version number
	"filename" = full filename, composed of draft name, version
		number, and a format suffix (e.g., "draft-iab-dos-00.ps")

(I've seen the term "filename" in this context used variously to refer
to (a) the full filename, (b) what I call the draft name, and (c) the
combination of draft name and version number.)

>I am going to have to figure out a trigger that will tell me that I need to 

I, too, dislike the idea of these filenames being ambiguous.  I'd prefer
that the tombstone file is the only file that ever gets that combination
of draft name and version number.  Any subsequent version of the draft
should get a new version number, higher than that used by the tombstone.

>files at a rate of perhaps 1000-2000 per year ad infinitum.

They're not empty.  I see no need for them to remain indefinitely --
six months seems fine to me -- though if we're going to maintain strong
uniqueness of draft names then at least the names need to be recorded
indefinitely.

-zefram

---

From: James M Galvin <[email-masked]>
Date: Wed, 14 Jan 2004 12:56:25 -0500 (EST)

On Wed, 14 Jan 2004, Fred Baker wrote:

    At 07:52 AM 1/14/2004, The IETF Secretariat wrote:

    >When an Internet-Draft expires, a "tombstone" file will be created
    >that includes the filename and version number of the Internet-Draft
    >that has expired.  The filename of the tombstone file will be the
    >same as that of the expired Internet-Draft with the version number
    >increased by one.  If a revised version of an expired
    >Internet-Draft is submitted for posting, then the revised version
    >will replace the tombstone file and will receive the same version
    >number as that previously assigned to the tombstone file.

This surprises me.  I would expect them to increment the version number
for the revised version.  At a minimum it is logically the right thing
to do because the information in the file is changing, i.e., it is not
the same file.

I am particularly surpised because it was more than 10 years that
incrementing the version number for the tombstone file started.  The
Secretariat didn't do that originally.

This was back before mirroring of the I-D directory was as popular as it
is today.  I was creating a mirror at my then employer and really wanted
to track the tombstone file, for consistency.  It is the same problem
you're having now although perhaps the content this time is more
valuable.

I had several conversations with Steve Coya about it back then and
pushed very hard to get that version number incremented for the
tombstone file.  It would be silly (if not shameful) to take a step
backwards now.

Jim

---

From: Fred Baker <[email-masked]>
Date: Wed, 14 Jan 2004 10:39:08 -0800

>tombstone file.  It would be silly (if not shameful) to take a step

I'm not sure it is backwards. If it is deemed to solve a problem (not 
obvious to me), it creates another.

I'd be happier bumping the number any time the file is changed, so that the 
tombstone supercedes the removed file and a subsequent posting supercedes 
the tombstone.

I am very concerned about the accumulation of tombstones forever, though. 
If we don't want to accumulate draft versions forever, what makes 
tombstones different? I would far rather age them out after some interval, 
such as six months.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 14 Jan 2004 19:35:31 +0000

Fred Baker wrote:

> I'd be happier bumping the number any time the file is changed, so 
> that the tombstone supercedes the removed file and a subsequent 
> posting supercedes the tombstone.
>
> I am very concerned about the accumulation of tombstones forever, 
> though. If we don't want to accumulate draft versions forever, what 
> makes tombstones different? I would far rather age them out after some 
> interval, such as six months. 

IMHO, if tombstones get removed after 6 months, the search interface on 
ietf.org should still be able to report that the document with a given 
name existed but expired.
I've been several times in a position when I had to search for a draft 
that has expired more than 6 months ago.

Alexey
__________________________________________
Isode Limited, http://www.isode.com

IETF standard related pages:
http://www.melnikov.ca/mel/devel/Links.html

Personal Home Page: http://www.melnikov.ca
__________________________________________

---

From: "Theodore Ts'o" <[email-masked]>
Date: Wed, 14 Jan 2004 14:48:45 -0500

> to find temporary documents.

I would prefer this as well.

						- Ted

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 14 Jan 2004 22:46:13 +0100

On 14-jan-04, at 17:43, Fred Baker wrote:

> It seems to me that there is a better approach to the above, at least 
> in the context of the above. If the "tombstone" is literally as 
> described, it would be far more space/search/etc efficient for us to 
> have the tombstone consist of an added text line in a file indicating 
> that the named draft expired on a certain date, and keep separate 
> files for the active internet drafts. It seems to me that this makes 
> it simpler to maintain a mirror and to find temporary documents.

> Thoughts?

This is probably orthogonal to mirroring issues, but it sometimes 
drives me mad when I have a draft filename but I can't find the draft 
itself and/or its status. Two things could help in this area:

- for all drafts, make it possible to determine the latest version
- for inactive drafts, supply some reference to the author(s)

A good and simple way to do this would be to create a file that matches 
the draft filename without the version number (would this be that 
tombstone thingy you guys keep talking about?) and say something like 
"version 34 was submitted 2003-04-05" or "version 00 was deleted 
1970-01-01" and copy the author's address section of the most recent 
version. Authors can then supply a link to the intended permanent 
resting place of the draft, if any, in this section.

If keeping all those files around is problematic, copying the author's 
address section into the I-D ACTION email message would also help a lot 
as the announcements mailinglist is well-archived.

---

From: Scott W Brim <[email-masked]>
Date: Wed, 14 Jan 2004 17:06:25 -0500

On Wed, Jan 14, 2004 10:46:13PM +0100, Iljitsch van Beijnum allegedly wrote:
> the draft filename without the version number 

I use a script that does: wget -k -nd -nH -r -np --glob=on -nc
--passive-ftp ftp://www.ietf.org/internet-drafts/$1\*

---

From: Tim Chown <[email-masked]>
Date: Thu, 15 Jan 2004 10:51:39 +0000

www.watersprings.org is helpful, if you know the draft name.

---

From: Geoff Huston <[email-masked]>
Date: Fri, 16 Jan 2004 08:01:06 +1100

>"version 34 was submitted 2003-04-05" or "version 00 was deleted 1970-01-01"

You can find an implementation of that form of indexing I-Ds at 
http://www.potaroo.net/ietf/xids-ietf.html

   Geoff