Subject: [testlist] TMDA / Outage

From: Glen <[email-masked]>
Date: Fri, 27 Jun 2008 10:16:28 -0700

Dear Working Group Chairs and Testers:

We had an outage last night, and I felt it appropriate to contact the 
WGChairs list directly about this, because we have some lingering 
cleanup to do.

So, first of all, we had an outage last night, once again because of TMDA.

It seems that TMDA knows it's on the way out.

Last night, an extreme spam run against many of the RT queues caused 
TMDA to overflow the server.  This caused the anomalous behavior 
observed at 1:45 AM, and was quickly leading to a crash and reboot.

To this I can only say that my objections against TMDA have been solid 
since day 1.

The bad news is that two lists were damaged by this outage - the IETF 
list, and the SIP list.

Because TMDA was eating everything alive, Mailman got a transient 
failure when it tried to read its databases for those two lists.

Rather than just try again, mailman produced 3-year-old data from an old 
copy of the databases for those two lists, and "rebuilt" the two lists 
with the old data.  By way of commentary, I find this behavior to be 
outrageous, but apparently that's what mailman does, and did.

As a result, messages to those two lists form people who were suddenly 
not on the list went to moderation instead.  Mail sent to those lists 
over the past couple of hours may not have been delivered to some 
individuals.  And servers attempting to deliver mail during the 
75-minute impact period may still be holding mail waiting on redelivery 
attempts.

*sigh*

I have restored the databases for those two lists from much more recent 
backups!  However, messages did go to moderation, and it would probably 
be prudent to send a message out to IETF-ANNOUNCE or something asking 
people to check and correct their subscriptions on those two lists.

The good news is that, yesterday, Henrik and I successfully installed 
his "postconfirm" TMDA replacement system, and implemented it on the 
testlist.  We still want to do some followup testing today, but I think 
we're finally getting out of the woods with this TMDA abomination.

So TMDA will be taken off, within the next day or so, and replaced with 
Henrik's system, which should put this ongoing stability issue, finally, 
to rest.

In addition, of course, we are proceeding with the split of web and 
email, using the servers we received from NSS, and setting up a 
multiple-server structure for the IETF.

We are finishing the data offload this week, and will begin the process 
of reloading servers next week.  Configuration and testing will occur 
after that, and we expect to switch over to the new setup right after 
Dublin.

I am now finish up final checks before I head into the office to work on 
Postconfirm with Henrik, who is visiting us this week.

As always, any questions, please let me know.

Glen
_______________________________________________
Testlist mailing list
[email-masked]
https://mail.ietf.org/mailman/listinfo/testlist