Subject: [Sip] Subscriber List Damage

From: Glen <[email-masked]>
Date: Sat, 28 Jun 2008 22:01:22 -0700

All -

I was asked by the IAOC to post a message to the IETF and SIP lists, to 
ensure that people were aware that the subscriber lists for the IETF and 
SIP lists were damaged as a result of an anomaly in TMDA and Mailman 
that occurred Thursday night.

Basically, TMDA misbehaved, and, in the process, caused Mailman to 
encounter a transient failure in the reading of its databases for these 
two lists.  As a result, rather than simply holding the mail and 
retrying it, Mailman decided to discard the current list databases and 
re-create them from 3-year-old data, for both the IETF and the SIP lists.

*sigh*

No email was lost to the system or the archives; however, some people 
may have missed some messages, or may still not be resubscribed to the list.

Of course we restored the files from backups; however, we want to make 
sure that everyone gets the mail they missed, and that everyone is 
subscribed to these lists who wishes to be subscribed.

So...

If you're reading this message in your email box, you're subscribed to 
the list identified in the subject line, and all should be okay.

If you're reading this message in the archives, wondering why you're not 
getting list mail, please take a moment to resubscribe yourself to the 
list, which should resolve your problem.

And regardless, if you feel you missed any mail, we do have the archives 
available for your reference.

IETF List Subscription Link:  https://www.ietf.org/mailman/listinfo/ietf
IETF List Archive Link:  http://www.ietf.org/mail-archive/web/ietf/

SIP List Subscription Link:  https://www.ietf.org/mailman/listinfo/sip
SIP List Archive Link:  http://www.ietf.org/mail-archive/web/sip/

We are in the home stretch of getting TMDA removed and replaced on the 
servers, and I apologize for any inconvenience caused by this issue. 
Because server problems apparently happen only in the dead of night, you 
can be sure that we feel any and all pain anyone may be experiencing.

If you need any assistance, please contact the IETF Secretariat, using 
the links at:  http://www.ietf.org/secretariat/

Thank you,
Glen Barney
IT Director
AMS (IETF Secretariat)
_______________________________________________
Sip mailing list  https://www.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: Michael Thomas <[email-masked]>
Date: Mon, 30 Jun 2008 15:48:10 -0700

1) Have you brought this up with the mailman folks? I've interacted with
    them and they seem like a responsive set of folks. I'm sure that 
this sort
    of thing would horrify them.
2) 3 years since the last backup? Oi.

       Mike

Glen wrote:
> All -
>
> I was asked by the IAOC to post a message to the IETF and SIP lists, 
> to ensure that people were aware that the subscriber lists for the 
> IETF and SIP lists were damaged as a result of an anomaly in TMDA and 
> Mailman that occurred Thursday night.
>
> encounter a transient failure in the reading of its databases for 
> these two lists.  As a result, rather than simply holding the mail and 
> If you're reading this message in the archives, wondering why you're 
> not getting list mail, please take a moment to resubscribe yourself to 
> the list, which should resolve your problem.
>
> And regardless, if you feel you missed any mail, we do have the 
> archives available for your reference.
>
> Because server problems apparently happen only in the dead of night, 
> you can be sure that we feel any and all pain anyone may be experiencing.
>
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Glen <[email-masked]>
Date: Mon, 30 Jun 2008 17:37:03 -0700

Michael Thomas wrote:
>    them and they seem like a responsive set of folks. I'm sure that this 
> sort

I haven't yet, but will try.  They'll probably want more information 
than I can give them.

> 2) 3 years since the last backup? Oi.

No - not three years since the last backup.  I have MUCH more recent 
backups than that, of course!

Three years is the age of the file that I suspect mailman used to pull 
its old configuration from.  An old copy of the database for those two 
lists that was just sitting around, not being used.

Glen

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 30 Jun 2008 18:42:18 -0700

At Mon, 30 Jun 2008 15:48:10 -0700,
>     of thing would horrify them.

I agree that this is horrifying.

More importantly, doesn't this mean that this is a problem we actually
need a solution for pronto? As I understand Glen's message, he's
saying that this is a bug in mailman triggered by some problem in
TMDA. I realize that TMDA is being replaced, but presumably Henrik's
code isn't perfect, so don't we have to worry about it triggering the
same behavior?

Glen, I'm sure there are some people on this list who understand
mailman well. I realize you may not have complete info, but if you can
provide us some more information--e.g., what file(s) got stomped and
which code you think stomped it--about what you think happened, maybe
they can help track it down?

-Ekr

---

From: Glen <[email-masked]>
Date: Mon, 30 Jun 2008 19:30:04 -0700

Unfortunately I really don't have anything but assumptions about what 
happened at this point.

An increase in TMDA activity caused the system to run out of resources. 
  We saw an extremely high load average and a sudden decrease in kernel 
buffer memory.  Processes started to fail to fork.  Our engineers 
connected in, and rebooted the server.

Upon reboot, we quickly discovered that the IETF list, which we 
moderate, was no longer letting us in with the normal list password.  A 
quick check showed that the database file config.pck was now only about 
50% of its original size, and that many subscribers had been removed - 
and others added.  Passwords and other settings had been "reverted" to 
pre-cutover values.  A comparison with our recent backup of the file 
showed massive differences - not just removals, but additions.  We 
hypothesize that mailman fell back to some type of cached copy of the 
database from 2005, which was also in the directory, and recreated the 
data from that.

Unfortunately, we have no way to verify what happened, and certainly 
don't want to try to cause it again.

I believe our solution is going to be to remove TMDA completely, which, 
I believe will eliminate the side-effects we've been seeing such as this.

I'm happy to discuss this further; however, I do not want to pollute the 
various lists with this.  I'm cc'ing the lists in this case so everyone 
understands we are responding to emails, but I'm going to take any 
further threads off-list to keep the lists focused on their primary 
purposes.

Glen