Subject: Is there a Git repository of RFCs? Or of Internet-Drafts?

From: [email-masked] (Dale R. Worley)
Date: Fri, 15 Mar 2013 11:01:34 -0400 (EDT)

Is there a publicly-available Git repository of RFCs or of
Internet-Drafts?

The reason I ask about a Git repository is that regular Git "pulls"
from such a repository seems like a straightforward and well-supported
way to maintain a local copy of the document collection.

Dale

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 11:09:37 -0400

> way to maintain a local copy of the document collection.

curious why rsync doesn't also seem 'straightforward' and 'well supported' ?

---

From: Jeffrey Schiller <[email-masked]>
Date: Fri, 15 Mar 2013 11:30:10 -0400

git show

-Jeff

-- 
_______________________________________________________________________
Jeffrey I. Schiller
Information Services and Technology
Massachusetts Institute of Technology
77 Massachusetts Avenue  Room E17-110A
Cambridge, MA 02139-4307
617.253.0161 - Voice
[email-masked]
http://jis.qyv.name
_______________________________________________________________________

---

From: Francis Galiegue <[email-masked]>
Date: Fri, 15 Mar 2013 17:45:49 +0100

On Fri, Mar 15, 2013 at 4:09 PM, Christopher Morrow
<[email-masked]> wrote:
[...]

> curious why rsync doesn't also seem 'straightforward' and 'well supported' ?

* rsync doesn't prevent corruption of data, git does;
* git show, git log, git bisect;
* git format-patch, git send-email etc.

I second this proposal.

-- 
Francis Galiegue, [email-masked]
JSON Schema in Java: http://json-schema-validator.herokuapp.com

---

From: Andrew Chi <[email-masked]>
Date: Fri, 15 Mar 2013 13:15:40 -0400

> * git format-patch, git send-email etc.

I like the proposal, though in my experience it's not a perfect match. 
I use git for version control on my own drafts, but renames are awkward 
when going from -02 to -03, etc.  And git's built-in diff can't compare 
to the IETF diff tool.  Of course, I'd be happy to be corrected, since 
it would improve my own workflow!

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 13:18:57 -0400

> * rsync doesn't prevent corruption of data, git does;

rsync does do checksums... and are there really that many failures
transfering 10k txt files?

> * git format-patch, git send-email etc.

I wasn't against it, just curious ... since rsync seems to do most of
this (hook it to mail you changes each run...)

---

From: Francis Galiegue <[email-masked]>
Date: Fri, 15 Mar 2013 19:01:17 +0100

> transfering 10k txt files?

What I mean is that if there is disk corruption on the server hosting
the drafts (which can happen post write), rsync will happily send the
checksum of the corrupted draft. Git's mechanism makes such a
probability infinitesimal.

> this (hook it to mail you changes each run...)

It depends on what features you really want out of a "draft revision
control system", I suppose ;)

---

From: Francis Galiegue <[email-masked]>
Date: Fri, 15 Mar 2013 19:03:39 +0100

On Fri, Mar 15, 2013 at 6:15 PM, Andrew Chi <[email-masked]> wrote:
[...]
>
> I like the proposal, though in my experience it's not a perfect match. I use
> git for version control on my own drafts, but renames are awkward when going
> from -02 to -03, etc.  And git's built-in diff can't compare to the IETF
> diff tool.  Of course, I'd be happy to be corrected, since it would improve
> my own workflow!

gitattributes(5) allows you to define a custom diff tool for path
globs... Disclaimer: I have never had the use for that, I only know
that it exists. Looking around will certainly yield useful examples,
though.

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 14:08:26 -0400

> probability infinitesimal.

wait, so.. if the disk fails things go bad... I'm confused.

---

From: Francis Galiegue <[email-masked]>
Date: Fri, 15 Mar 2013 19:15:51 +0100

On Fri, Mar 15, 2013 at 7:08 PM, Christopher Morrow
<[email-masked]> wrote:
[....]
> wait, so.. if the disk fails things go bad... I'm confused.

If the disk goes bad so as to provoke a misread of a sector, post
write, the file is effectively corrupted. If this happens with git,
the checksum calculated on write will fail to match, and the
corruption is detected.

Not only that, but:

* you may then recover the repository from another existing one;
* this mechanism detects corruption _even if you have a bare_ (since
what is checksummed is not the file in the working tree but its blob
representation in the object tree).

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 14:26:54 -0400

> the checksum calculated on write will fail to match, and the

you seem to be protecting against a very, very, very uncommon failure...
I think you'd be better off protecting against a host of much more
common failure modes, eh?

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 15 Mar 2013 14:40:09 -0400 (EDT)

> From: Christopher Morrow <[email-masked]>
> 

Is this an advocacy of a particular tool?  Or are you asserting that
rsync can be used to maintain a directory of RFCs?  If the latter,
could you supply some details (including, especially, how to get at
the public repository)?

Dale

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 15:25:47 -0400

> could you supply some details (including, especially, how to get at

/home/morrowc/scripts/ietf/internet-drafts-mirror.sh

has in it:
#!/bin/sh
#
# Location local: /prod/docs.as701.net/internet-drafts
# internet-drafts location: ftp.rfc-editor.org::
DST=<WHERETOPUTYOURSTUFF>
SRC="ftp.rfc-editor.org::"
RSYNC=/usr/bin/rsync
REPOS="rfcs-text-only ids-text-only"
for repo in ${REPOS} ; do
  ${RSYNC} --delete -rpva  ftp.rfc-editor.org::${repo} ${DST}/${repo}
done

I'm not sure why I was so lazy.. but there it is.

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Fri, 15 Mar 2013 15:30:31 -0400

you forgot the cron job instructions too.  and while you're at it, how to set up Dropbox. ;)

Tom

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 15 Mar 2013 16:20:45 -0400

> you forgot the cron job instructions too.  and while you're at it, how to set up Dropbox. ;)

jesus christmas...

30 7 * * * /home/morrowc/scripts/ietf/internet-drafts-mirror.sh >
/tmp/rfc-sync.log 2>&1

apt-get install dropbpx

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 15 Mar 2013 22:43:18 +0000

There  are consumer grade devices, where this is the most common failure mode. But usually a glitch in a video stream where these disks are typically used, isn't as critical as corrupting your bank account content (say, to the state prior to when you got your salary) :)

(And that consumer grade devices usually never get a bug fix to address that).

Richard Scheffenegger

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Christopher Morrow
> Sent: Freitag, 15. März 2013 19:27
> To: Francis Galiegue
> Cc: [email-masked]
> Subject: Re: Is there a Git repository of RFCs? Or of Internet-Drafts?
> 
> >>> What I mean is that if there is disk corruption on the server
> >>> hosting the drafts (which can happen post write), rsync will happily
> >>> send the checksum of the corrupted draft. Git's mechanism makes such
> >>> a probability infinitesimal.
> >>
> I think you'd be better off protecting against a host of much more common
> failure modes, eh?

---

From: "John Levine" <[email-masked]>
Date: 15 Mar 2013 23:14:07 -0000

>the checksum calculated on write will fail to match, and the

If you're worried about that (not totally unreasonable on modern
disks) wouldn't it be better to use a filesystem like zfs that checks
for corruption at the filesystem level?

In practice, rsync works great.  I pull the RFCs and I-Ds every night.

---

From: Michael Richardson <[email-masked]>
Date: Fri, 15 Mar 2013 17:41:33 -0400

>>>>> "Dale" == Dale R Worley <[email-masked]> writes:
    Dale> Is there a publicly-available Git repository of RFCs or of
    Dale> Internet-Drafts?

    Dale> The reason I ask about a Git repository is that regular Git
    Dale> "pulls" from such a repository seems like a straightforward
    Dale> and well-supported way to maintain a local copy of the
    Dale> document collection.

I could push to github from the nightly rsync on my desktop if you like.
rsync is also pretty good.  You'd get all the history in your git
repo. That's either a feature or a bug.

-- 
Michael Richardson
-on the road-

---

From: Michael Richardson <[email-masked]>
Date: Sat, 16 Mar 2013 11:04:20 -0400

>>>>> "Andrew" == Andrew Chi <[email-masked]> writes:
    Andrew> On 3/15/13 12:45 PM, Francis Galiegue wrote:
    >> * rsync doesn't prevent corruption of data, git does; * git show,
    >> git log, git bisect; * git format-patch, git send-email etc.

    Andrew> I like the proposal, though in my experience it's not a
    Andrew> perfect match. I use git for version control on my own
    Andrew> drafts, but renames are awkward when going from -02 to -03,
    Andrew> etc.  And git's built-in diff can't compare to the IETF diff
    Andrew> tool.  Of course, I'd be happy to be corrected, since it
    Andrew> would improve my own workflow!

I don't store my drafts as -00.txt, but just .txt.
I git add the -XX.txt as is (sometimes), or just ignore it.

=== % cat Makefile
VERSION=00

draft-ietf-roll-applicability-template-${VERSION}.txt: draft-ietf-roll-applicability-template.txt
	cp draft-ietf-roll-applicability-template.txt draft-ietf-roll-applicability-template-${VERSION}.txt

%.txt: %.xml
	XML_LIBRARY=$(XML_LIBRARY):./src xml2rfc $? $@

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 16 Mar 2013 08:56:27 -0700

> way to maintain a local copy of the document collection.

Another way is with the "ietf" command-line tool. The "mirror" function there uses rsync. The tool has lots of features that people have asked me for, such as being able to open an Internet Draft by entering just part of its filename, opening things in the IETF tracker or the IETF Tools site instead of locally, and being able to see the status of a draft from the command line.

See http://svn.tools.ietf.org/svn/tools/ietf-cli

--Paul Hoffman

---

From: James Cloos <[email-masked]>
Date: Sat, 16 Mar 2013 14:14:41 -0400

>>>>> "JL" == John Levine <[email-masked]> writes:

JL> In practice, rsync works great.  I pull the RFCs and I-Ds every night.

The benefits git would bring are history and reduced badwidth per pull
when updating.

The latter, of course, is only really relevant when the daily pull is
done over a low-bandwidth link.  (The impact of the cron job was very
noticeable over a DS0; it is not at all noticeable over doccis.)

-JimC
-- 
James Cloos <[email-masked]>         OpenPGP: 1024D/ED7DAEA6

---

From: Joe Abley <[email-masked]>
Date: Sat, 16 Mar 2013 14:47:02 -0400

What useful history can you possibly get when each file is only ever
substantively changed by publishing another file?

Aue Te Ariki! He toki ki roto taku mahuna!

---

From: Christopher Morrow <[email-masked]>
Date: Sat, 16 Mar 2013 18:52:35 -0400

>> The benefits git would bring are history and reduced badwidth per pull

'reduced bandwidth'? I suppose you lose on the initial 'what changed'
conversation with git vs rsync, but really beyond that you're going to
xfer the same bits...

this seems specious as well, to me at least.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 18 Mar 2013 10:40:58 -0400

https://github.com/credil/ietf-drafts-sorted	
  is the result of my "draftmirror" script which basically does s,-,/,
  in a selective way.  Each author and group gets its own directory.

  Previously I have just run draftmirror on each of my devices, but
  now, I think I'll just use git pull, so I saved the world some bandwidth.
  
https://github.com/credil/ietf-drafts
  is the unsorted rsync.

I probably have to sort out what ssh key I use to push.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: Robert Sparks <[email-masked]>
Date: Mon, 18 Mar 2013 17:28:43 -0500

> could you supply some details (including, especially, how to get at
<https://www.ietf.org/id-info/> -> 
<https://www.ietf.org/mirror-instructions.html>

alternatively, look at the third section on the right column at 
<https://tools.ietf.org/>
(starting with "Download the latest documents")

RjS

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 20 Mar 2013 16:10:12 -0400 (EDT)

Rob Evans mentions that there is documentation on using rsync to get
RFCs and internet drafts:

    http://www.rfc-editor.org/rsync-help.html

After some testing, rsync works well for me.  I use

rsync --stats -az --delete --exclude=ien \
    ftp.rfc-editor.org::rfcs \
    ~/in-notes

# Use --ignore-existing to prevent I-Ds from being deleted or replaced with
# tombstones.
rsync --stats -az --ignore-existing \
    ftp.rfc-editor.org::internet-drafts \
    ~/internet-drafts

Dale