Subject: Next steps in IETF list email archiving

From: Robert Sparks <[email-masked]>
Date: Wed, 19 Jul 2017 09:43:24 +0200

Several years ago, we identified a set of issues with the way
we were archiving email messages and new functionality that a
searching and browsing solution should contain. The initial
requirements for improvements were captured in RFC 6778, and a
subsequent round of requirements were captured in RFC 7842.

One of the key issues was (and is) related to management of
the archives.  Currently if a message has to be removed
(rare, but not as rare as we would like due to spam hitting
some lists), the secretariat has to touch many places. One
of those is Mhonarc. Touching one message there involves
changing many files (any other message that contains a
navigation link to this one, and the index files that speak
about the message).

As we built the new systems, we've targeted having one source
of truth for the archives. For instance, as we added IMAP
access to the archives (RFC 7017), we ensured that it was
served from the same bits used by mailarchive.ietf.org.  It
has been our expectation from the beginning that
mailarchive.org would replace the Mhonarc archives (see
section 2.7 of RFC 6778).

We believe we are getting close to the point where we can
make that replacement. The rate of bug reports and feature
requests for mailarchive.ietf.org is slow. (As we do for
the datatracker, we plan to continuously improve the
mailarchive interface. We know already of some things that
need to be improved when viewing on phone-sized displays.)

If you use the mhonarc archives heavily, and have not yet
explored mailarchive.ietf.org, we encourage you to do so now,
and report any difficulties you find. We recognize that the
experience is different, but many of the RFC 7842 driven
improvements focused on minimizing the transition pain.

We have successfully tested the code that will redirect all
existing Mhonarc URLs into the mailarchive using the
testlist.

We are not going to make this transition immediately, but
we do plan to make it more in the near future than the far
future. Please help us identify any additional things we can
do to minimize the disruption to your current workflow.

RjS

---

From: Michael Richardson <[email-masked]>
Date: Wed, 19 Jul 2017 13:58:23 +0200

Robert Sparks <[email-masked]> wrote:
    > If you use the mhonarc archives heavily, and have not yet explored
    > mailarchive.ietf.org, we encourage you to do so now, and report any
    > difficulties you find. We recognize that the experience is different,
    > but many of the RFC 7842 driven improvements focused on minimizing the
    > transition pain.

I stopped using mailarchive and I almost exclusively use mhonarc
This is now easy since the dual links in the datatracker returned.
I was making up the correct links before, which was a pain.

I find the search-only interface to mailarchive annoying, and frankly slow.

While the thread support is better, it is still not anywhere as close
to MHonarc.  When I find an email that I care about, and I ask for the
thread view, I get the thread for the entire list --- yes, with that
email opened, but the entire thread is there.
If there is a URL for that thread, I don't know it, and it is not easily
found.

Frankly, I just feel stupid interacting with an active system when I
know a set of static files would satisfy my needs.   No matter how fast the
active system can be made...

    > We have successfully tested the code that will redirect all existing
    > Mhonarc URLs into the mailarchive using the testlist.

neat.  I understand the desire to get rid of mhonarc.  I want mailarchive
to succeed, but it still feels really klunky to me.

    > We are not going to make this transition immediately, but we do plan to
    > make it more in the near future than the far future. Please help us
    > identify any additional things we can do to minimize the disruption to
    > your current workflow.

It would be cool if I could get an IMAP URL from mailarchive, as that
would let me jump from searching the archives for a relevant thread,
and right into writing a reply to it.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 19 Jul 2017 13:03:51 +0100

without meaning to sound like a complete luddite (and top-posting for
extra effect), what Michael has written is exactly how I feel about both
systems, and I would be very unhappy to see mhonarc go.

Nick

---

From: "Andrew G. Malis" <[email-masked]>
Date: Wed, 19 Jul 2017 19:24:50 +0200

Robert,

At mailarchive.ietf.org, I see a “Sign in” button in the upper right.
What’s the advantage to signing in, and which credentials are accepted, the
same as datatracker?

Thanks,
Andy

---

From: Robert Sparks <[email-masked]>
Date: Wed, 19 Jul 2017 19:28:35 +0200

Datatracker credentials. If you are on any restricted archive lists, like the Iesg list, they'll show up. 

Sent from my iPhone

> On Jul 19, 2017, at 7:24 PM, Andrew G. Malis <[email-masked]> wrote:
> 
> Robert,
> 
> At mailarchive.ietf.org, I see a “Sign in” button in the upper right. What’s the advantage to signing in, and which credentials are accepted, the same as datatracker?
> 
> Thanks,
> Andy
> 
>

---

From: Samita Chakrabarti <[email-masked]>
Date: Wed, 19 Jul 2017 11:48:34 -0700

+1.

I find MHonarc is much easier interface to use and better viewing
experience.

Wish mailarchive.ietf had a similar user interface. Currently, I use
MHonarc for this reason.

 -Samita

On Jul 19, 2017 5:04 AM, "Nick Hilliard" <[email-masked]> wrote:

Nick

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 20 Jul 2017 08:54:04 +1200

I find it varies. For some types of search Mhonarch is definitely better.
If I have a vague idea about when a thread occurred and a vague idea about
the subject header, paging through Mhonarch to find the thread works much
better. And then following the thread itself is pretty natural in Mhonarch.
Also, if you send someone the link they can follow the thread too.

Here's a challenge. Follow the recent threads on 6man about RFC4291bis.
It's very easy at https://www.ietf.org/mail-archive/web/ipv6/current/threads.html
It's not at all easy with the new tool. In fact I found two issues while
trying to do so:

1. I simply can't see how I'm supposed to restrict the search to the subject header.
All searches appear to be on header+body. Maybe I'm missing something?

2. I noticed that a whole thread whose subject header is "RE: <draft-ietf-6man-rfc4291bis-00.txt>"
displays in the search results as just plain "RE: ", which seems to be bug.

So if I was doing that search in real life, I would definitely use Mhonarch.

Regards
   Brian Carpenter

---

From: Tim Chown <[email-masked]>
Date: Wed, 19 Jul 2017 23:19:44 +0100

> On 19 Jul 2017, at 21:54, Brian E Carpenter <[email-masked]> wrote:
> 

A good summary, and my strong preference also.

Tim

---

From: "Paul Hoffman" <[email-masked]>
Date: Thu, 20 Jul 2017 07:35:46 +0200

I apologize in advance for this engineering solution, but given how many 
people are saying "I prefer/need the MHonArc archives":

> navigation link to this one, and the index files that speak

You may be over-touching. If someone needs a message "removed", you can 
leave the message in-place and simply change to content to "Message 
content removed" in the one file. No links are broken, no indexes need 
to be regenerated. (I did this for a while at VPNC.)

--Paul Hoffman

---

From: Brandon Long <[email-masked]>
Date: Wed, 19 Jul 2017 22:51:54 -0700

If we're now storing messages in some datastore, wouldn't it be possible to
generate mhonarc like pages on the fly?

Brandon

On Jul 19, 2017 10:36 PM, "Paul Hoffman" <[email-masked]> wrote:

> leave the message in-place and simply change to content to "Message content
> removed" in the one file. No links are broken, no indexes need to be
> regenerated. (I did this for a while at VPNC.)
>
> --Paul Hoffman
>
>

---

From: Leonard Giuliano <[email-masked]>
Date: Thu, 27 Jul 2017 08:32:17 -0700

+1 for finding a way to keep both UIs.  I find these to be complementary 
tools- mailarchive.ietf.org is great for searching content and filtering, 
but my linear/non-Millenial brain finds the Mhonarc UI much easier to read 
and consume.

Do we really need to remove messages from the archive (sounds so 
Orwellian)?  I'd rather have spam in the archive than lose the Mhonarc UI.

-Lenny

| -------- Forwarded Message --------
| Subject: Next steps in IETF list email archiving
| Date: Wed, 19 Jul 2017 09:43:24 +0200
| From: Robert Sparks <[email-masked]>
| To: [email-masked] <[email-masked]>
| 
| Several years ago, we identified a set of issues with the way
| we were archiving email messages and new functionality that a
| searching and browsing solution should contain. The initial
| requirements for improvements were captured in RFC 6778, and a
| subsequent round of requirements were captured in RFC 7842.
| 
| One of the key issues was (and is) related to management of
| the archives.  Currently if a message has to be removed
| (rare, but not as rare as we would like due to spam hitting
| some lists), the secretariat has to touch many places. One
| of those is Mhonarc. Touching one message there involves
| changing many files (any other message that contains a
| navigation link to this one, and the index files that speak
| about the message).
| 
| As we built the new systems, we've targeted having one source
| of truth for the archives. For instance, as we added IMAP
| access to the archives (RFC 7017), we ensured that it was
| served from the same bits used by mailarchive.ietf.org.  It
| has been our expectation from the beginning that
| mailarchive.org would replace the Mhonarc archives (see
| section 2.7 of RFC 6778).
| 
| We believe we are getting close to the point where we can
| make that replacement. The rate of bug reports and feature
| requests for mailarchive.ietf.org is slow. (As we do for
| the datatracker, we plan to continuously improve the
| mailarchive interface. We know already of some things that
| need to be improved when viewing on phone-sized displays.)
| 
| If you use the mhonarc archives heavily, and have not yet
| explored mailarchive.ietf.org, we encourage you to do so now,
| and report any difficulties you find. We recognize that the
| experience is different, but many of the RFC 7842 driven
| improvements focused on minimizing the transition pain.
| 
| We have successfully tested the code that will redirect all
| existing Mhonarc URLs into the mailarchive using the
| testlist.
| 
| We are not going to make this transition immediately, but
| we do plan to make it more in the near future than the far
| future. Please help us identify any additional things we can
| do to minimize the disruption to your current workflow.
| 
| RjS
| 
| 
| 
|

---

From: Russ Housley <[email-masked]>
Date: Thu, 27 Jul 2017 13:02:37 -0400

In the most recent release of the mail archive search tool, many different ways of sorting the threads were added .What about the Mhonarc UI is useful to you?  Is that view not available in the recent release?  Just saying that your non-Millenial mind finds it easier does not help figure out the best next steps.

Russ

> On Jul 27, 2017, at 11:32 AM, Leonard Giuliano <[email-masked]> wrote:
> 
>

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 31 Jul 2017 09:25:08 -0400

This is partially tangential, but:

One of the best approaches to ensure the long-term preservation of these
archives is to (1) keep them in an open, standard, widely-supported
format -- "mbox" is clearly the best choice and (2) make them trivially
easy to mirror.  This has the desirable effect of allowing who wishes
to maintain their own copy and/or to use the search engine of their
choice on it.  (In my case, I find "grepmail" to be highly useful for
searching mail archives: it's better than anything else I've tried,
and I've tried a lot of things.)

It also allows folks to access the archives when disconnected from the
'net or when they're on a slow/expensive connection.  The former is
more than merely a matter of convenience: in some working environments:
it's a requirement, e.g., in environments that are always physically
disconnected from the 'net.

So whatever presentation format is chosen, the original/raw mbox files
should be available for download via -- at least -- https and rsync.
Gzip'd tarballs of them should be regenerated nightly and also made
available -- on a per-mailing-list basis and globally.  (To make that
clear: the latter means that it should be possible to download a single
gzip'd tarball that contains all archives of all IETF mailing lists.
Not only is that useful in its own right, but it'd be an excellent
way to initialize a mirror that can be subsequently maintained via rsync.)

A back-of-the-envelope calculation suggests that the complete archives of
all IETF mailing lists rendered as a gzip'd tarball would fit comfortably
on a 32G USB stick.

None of this is particularly hard to script and automate via cron: I've
set up similar things many times.  I'd be happy to do the work if nobody
else can spare the time.

---rsk