Subject: Re: Next steps in IETF list email archiving

From: James Cloos <[email-masked]>
Date: Wed, 23 Aug 2017 03:49:42 -0400

>>>>> "RK" == Rich Kulawiec <[email-masked]> writes:

RK> One of the best approaches to ensure the long-term preservation of these
RK> archives is to (1) keep them in an open, standard, widely-supported
RK> format -- "mbox" is clearly the best choice and (2) make them trivially
RK> easy to mirror.

rsync://rsync.ietf.org/everything-ftp/ietf-mail-archive/

It is currently 23G.

-JimC
-- 
James Cloos <[email-masked]>         OpenPGP: 0x997A9F17ED7DAEA6

---

From: "John Levine" <[email-masked]>
Date: 23 Aug 2017 19:40:58 -0000

In article <[email-masked]> you write:
>One of the best approaches to ensure the long-term preservation of these
>archives is to (1) keep them in an open, standard, widely-supported
>format -- "mbox" is clearly the best choice

Er, not really.  See RFC 4155.  There are way too many not quite
compatible versions of mbox.  If I wanted a file format for a
multi-file archive, I'd probably use POSIX ustar which is an actual
standard.

On the other hand, I see that there are rsync-able archives in some
mbox variant, so that horse left the barn ages ago.  The archives are
kind of funky, e.g., the file dnsop/2001-01.mail contains messages
from March 1999 through January 2001, and there are empty files
1999-06.mail through 2000-12.mail.

R's,
John

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 23 Aug 2017 23:51:11 +0200

> multi-file archive, I'd probably use POSIX ustar which is an actual

Sure, but that would not be a complete spec because it does not cover
the format of individual messages nor any semantics of file names.

Whats the least restrictive sufficiently well specified generic mail encoding ?
Anything that would also pass via rfc822+mime ?

Might also be nice to have some well-known-filename with metadata
about the archive content, eg: self-URL, previous, next URL, contact
URL (if any), archive owner signed checksums of files. Template
spec of filenames (%YYYY-%MM.mail),...

> 1999-06.mail through 2000-12.mail.

See above: metadata that would help you figure out if you have a
complete set of archive files.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 23 Aug 2017 21:32:22 -0400

> Anything that would also pass via rfc822+mime ?

We already have a perfectly good mail encoding, the one defined in RFC 
5322 and its predecessors.  (Recall that MIME is entirely hidden inside 
the 5322 message structure.)  I suppose you could come up with a 
convention for filenames, but it hardly matters since anything interesting 
about the message is within the message itself.

As I noted, the current filenames are based on months, but the mail in the 
files only approximantely was sent during the months the names suggest. 
Doesn't matter.

> spec of filenames (%YYYY-%MM.mail),...

URL?  This is mail, not a web site.  In any event, if you want to make 
indexes and such on top of an archive, go ahead but it's not part of the 
archive.

> complete set of archive files.

A file listing the other files with hashes or checksums is all you need 
for that.

R's,
John

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 24 Aug 2017 05:46:57 +0200

On Wed, Aug 23, 2017 at 09:32:22PM -0400, John R Levine wrote:
> We already have a perfectly good mail encoding, the one defined in
> RFC 5322 and its predecessors.  (Recall that MIME is entirely hidden
> inside the 5322 message structure.)  I suppose you could come up
> with a convention for filenames, but it hardly matters since
> anything interesting about the message is within the message itself.

Great.

> URL?  This is mail, not a web site.  In any event, if you want to
> make indexes and such on top of an archive, go ahead but it's not
> part of the archive.
> 
> A file listing the other files with hashes or checksums is all you
> need for that.

Are there any RFCs specifying how mirrors can point to each others so
folks retrieving the information can load-balance/failover etc.. That ws
pretty much the level i was suggesting, and its certainly an orthogonal
problem.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 23 Aug 2017 21:10:20 -0700

>> with a convention for filenames, [...]

MH nailed down that standard decades ago.  One message per file, in a suitable directory hierarchy. E.g.

  ietf/<listname>/<year>/<nn>

This avoids the ambiguity of mbox message separator escapes (the '>From' problem), but creates a lot of overhead for things like rsync.

But for a public archive interface, why aren't/can't we use anonymous IMAP, with a folder structure as above?  This hides the underlying storage format.  It provides search capabilities to naïve clients, and it's easy to write mirroring tools.  It's trivial to make mirroring decisions based on UIDVALIDITY and UIDNEXT; the mirroring tool is essentially an offline IMAP client.

> pretty much the level i was suggesting, and its certainly an orthogonal

For the IMAP case, the fanout happens behind the scenes, with the list exploder feeding the IMAP archive servers by Submission is almost every case.  This doesn't guarantee complete synchronization between all the public IMAP servers, but they will be close enough in daily operation to be useful.  Certainly this is no different from load balancing across public anonymous FTP servers.

If people want local mirrors, there are already tools out there for remote-syncing IMAP accounts that could be easily adapted to the task.  But I'm sure lots of people would cook up dedicated tools in short order.

--lyndon

---

From: Michael Richardson <[email-masked]>
Date: Thu, 24 Aug 2017 11:22:48 -0400

Lyndon Nerenberg <[email-masked]> wrote:
    > But for a public archive interface, why aren't/can't we use anonymous
    > IMAP, with a folder structure as above?  This hides the underlying

We can, and we already do.

I think the problem is that many people would like to suck the archives back
in bulk, and they see IMAP as too chatty.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Lyndon Nerenberg <[email-masked]>
Date: Thu, 24 Aug 2017 13:55:24 -0700

> On Aug 24, 2017, at 8:22 AM, Michael Richardson <[email-masked]> wrote:
> 

I don't think that's a valid argument.  Once you authenticate and select your mailbox of interest, you're issuing a single FETCH command for each message you want, so it's no more overhead than FTP in that regard.  To mitigate the lock-step command RTTs, just run a few FETCH streams in parallel.

If I can find some spare time this weekend I'll see about putting together a simple downloader and run some timing tests.

--lyndon

---

From: "John Levine" <[email-masked]>
Date: 25 Aug 2017 02:10:50 -0000

In article <[email-masked]> you write:
>I don't think that's a valid argument.

We have mbox-ish archives available via wget and rsync, and have for
many years.  It is my impression that they are maintained
automatically and are not going away.  So if you want to do bulk
fetches via IMAP, that's fine, but anyone who wants to do it with
rsync will continue to do so.

I did, took a while but I got lots and lots of files of messages, and it
had the advantage of being a one-line command, no new programming needed.

R's,
John

---

From: Lyndon Nerenberg <[email-masked]>
Date: Thu, 24 Aug 2017 19:38:45 -0700

> On Aug 24, 2017, at 7:10 PM, John Levine <[email-masked]> wrote:
> 

I agree completely.  I'm just pointing out that there's no standard for the various "mailbox archive" formats.  E.g. mbox, MH, maildir.  IMAP provides a(n IETF) standard interface.  Maybe not the most efficient in this specific case, but beware of micro-optimization.

--lyndon

---

From: "John Levine" <[email-masked]>
Date: 25 Aug 2017 15:35:58 -0000

In article <[email-masked]> you write:
>I agree completely.  I'm just pointing out that there's no standard for the various "mailbox archive" formats.  E.g. mbox, MH, maildir.  IMAP
>provides a(n IETF) standard interface.  Maybe not the most efficient in this specific case, but beware of micro-optimization.

I agree about not micro-optimizing but it leads me to a different answer.

Having been schlepping mbox files for 40 years, I can report that even
though there are a lot of variations, they are minor enough that it is
easy to handle them all.  There have for decades been libraries and
applications that do so, such as mhonarc which turns mbox files into
pretty archive web pages.  Similarly, although the file naming conventions
in mh and maildor are different, the messages in the files are the same
and again it is easy to handle the naming differences.

IMAP is fine for what it's good for, but the question of how you
store archives of mail messages was answered a long time ago.

R's,
John