Subject: [testlist] No Outage (not at 0650Z anyway!)

From: Glen <[email-masked]>
Date: Tue, 13 May 2008 00:09:46 -0700

All -

Good morning.

At 06:55 GMT - five minutes past the time of our two outages this past 
week - the server is running quite smoothly and unencumbered, and has 
not (at least so far) gone down or shown any signs of doing so.

Progress was definitely made today.  Things we did include such elements 
as:  (Suppressing the desire to work "Fear" and "Surprise" into this 
message)...

* Did a bunch of digging through the kernel.  Found some strangeness in 
the way memory was being handled generally, found a possible way of 
adding some sanity to the oom-killer process, re-tuned the kernel to 
implement and fix that.

* In concert with Henrik, installed a modified version of the TMDA 
handler to fine-tune how TMDA is dispatched, thus reducing significantly 
the added load from TMDA (with great success, thank you Henrik!)

* Installed a logging system to capture snapshots of all kind of server 
data and log it; in the event that the server -does- go down this will 
hopefully give us something more to look at.

* Found a way to exempt certain processes from the oom-killer, applied 
those exemptions to the syslog and cron and sshd processes.  I -hope- I 
never get to test this, but I applied it nonetheless.

And of course,

* Sat up with the machine until now, to watch firsthand what is happening.

I did see a lot of python activity, and a lot of datatracker activity, 
and that activity increased dramatically after 06:30Z.  TMDA activity 
also increased dramatically; the new handler performed very well and our 
overall load remained low.  The usual other things all ran correctly and 
as they should, and the server performed perfectly, averaging less than 
2% cpu, peaking at 17% at one point, with memory staying well below the 
50% mark throughout the night.

So did these changes solve the problem?  The re-tuning and the TMDA 
stuff -may- have.  I won't stay up every night, but I *will* stay up 
Wednesday-into-Thursday again, since that's when the last crash 
occurred, just to see what happens.... and we'll go from there.

I also found a way to have the oom-killer reboot the server if it gets 
into that much trouble.  I haven't turned that on yet; however, when 
this little episode is all done, I will turn it on, so that any failures 
of this type in the future are hopefully limited to a few minutes of 
automatic recovery, rather than outages like this.

Have a good night/day as appropriate, and, if nothing happens between 
now and then, I'll post another email Thursday.

Glen Barney
_______________________________________________
Testlist mailing list
[email-masked]
https://mail.ietf.org/mailman/listinfo/testlist

---

From: Glen <[email-masked]>
Date: Thu, 15 May 2008 20:42:40 -0700

All -

I wanted to take a moment and post a final message regarding our recent 
outage, to let everyone know how things concluded and what we did about 
this.

As you recall, the problem began shortly after we "upgraded" the system 
to include TMDA processing for all of the IETF lists.  Two nights after 
applying that upgrade, we had a significant outage; then we had an 
additional shorter outage several days later.

Having spent quite a bit of time researching and discussing our 
findings, I've drawn the conclusion that the server crashed because it 
ran out of kernel page table memory.  The server itself is quite huge - 
with 8 processor cores running at 3GHz, 8GB of RAM, dual 300GB mirrored 
drive arrays, and so forth.  But a comparatively small portion of memory 
is used to manage all the rest of the memory in the system, and that 
small portion became exhausted as a result of the increased process load 
brought in by TMDA.

TMDA was not in itself the direct cause of the failure, TMDA was the 
"straw", so to speak.  The direct cause of the failure was that a key 
section of kernel memory became exhausted, and the server attempted to 
shut down processes to recover that memory.  This caused most public 
services (web, email, FTP) to be shut down, in essence causing the 
outage.  The server itself was still up and running... but no public 
processes were running... so the end result was the same.

Once we realized what had happened, the solution was quite easy - we 
simply increased (dramatically) the size of the memory area used by the 
kernel, to more than twice the maximum size needed to operate all of the 
existing services.  Observation of the server under regular load after 
this adjustment seems to have confirmed that this adjustment did in fact 
resolve the problem, and we have made these adjustments permanent on all 
IETF servers.

Another issue relating to the outage was the overall time of the outage. 
  In order to reduce the time of the outage, I've already implemented 
additional steps, such as the installation of remote power control, and 
remote server console access, on all IETF servers.  This, obviously, 
allows us to immediately interact with and restore a failed server, 
without the need to send on-call staff to the site in the middle of the 
night.  We had already planned these additions; however, because of the 
large volume of work pushed from "during cutover" to "post cutover", 
those items had not yet been implemented.  In consultation with AMS 
owners, I took those items to the top of the list and implemented them 
personally.

In addition, again thanks to commentary from the community, I've 
implemented additional kernel changes on the IETF servers.  Going 
forward, the server will no longer be permitted to selectively kill 
processes in an out-of-memory condition.  Instead, the server will 
execute an immediate reboot.  Moreover any type of software-based 
failure of this type (an "oops" or panic) will also now result in an 
immediate reboot.

Another project on our "to-do" list was the implementation of server and 
system monitoring software.  Again, this was planned for the transition 
period, but timing constraints forced us to delay this important 
component.  We have now implemented the popular NAGIOS system across our 
Fremont data center, and are configuring its quite complex test set and 
extending the system to our Orlando data center this coming week.

Obviously, all of these changes together will dramatically reduce the 
potential length (and possibility!) of any future outages.

Finally, the topic was raised about additional servers.

We currently operate four servers dedicated to the IETF.  Each is 
configured and built as I outlined above.  One server currently 
functions as the primary operations server, and handles all of the 
IETF's traffic without ever really breathing hard.  The recent failure, 
of course, is an exception, but that was a kernel utilization issue, 
which was a rather unique thing, and not an overload of the server's 
hardware resources.  Each one of the servers we've assigned to the IETF 
is the processing and power equivalent of three servers used by the 
former provider.  I was confident that a single server could handle all 
of the operations of the IETF, and I still am.

Indeed, I agree strongly with the comment someone made that "throwing 
another server at this won't solve the problem."  That turned out to be 
correct.  OTOH, I also agree strongly with the comment about not 
"putting all of our eggs in one basket."

So as I was saying, we have four servers, two in each of two data 
centers, and are currently running in such a way that an instant 
failover can be initiated to any of the other servers as desired, in the 
event of a longer-term or hardware failure of the production server.  So 
we have four identical eggs in two identical baskets, as it were.

In addition, we are planning to implement a fifth offsite backup server 
at ISOC, which will again be identical in function to the other four.

But I am more than open to the concept of putting in additional servers 
should the need arise.  One person commented today about adding 
additional mail exchange servers.  This is another comment I agree with; 
therefore, in the coming weeks I'll be setting up an additional server 
in Fremont to handle just incoming mail processing.  Obviously, that 
will not happen overnight, but it is something we will do going forward.

So, this outage, while unexpected, was not particularly catastrophic. 
No data was lost, the outage cause was identified and corrected, and 
we've taken appropriate steps to mitigate future outages and bring them 
under much more manageable levels.

Of course, my personal vote would be to never have an outage again.

I hope this information is useful and informative, and brings to closure 
any open questions regarding this outage.  TMDA remains in place, thanks 
primary to the efforts of Henrik Levkowetz, and the servers are running 
smoothly.  In addition, I wish to thank everyone who provided support 
and commentary during this highly stressful time.  Thank you all.

I'm now "closing this ticket," but please feel free to contact me at any 
time if you have further questions or concerns.  Please be reminded that 
I am not on the WGChairs list; therefore, you'll need to CC me if you 
want me to see something.

Best regards,
Glen Barney

---

From: Glen <[email-masked]>
Date: Fri, 16 May 2008 11:50:58 -0700

All -

Okay, having read all the email, and getting follow-up fom Jim and Russ, 
I think it's become clear that we are willing to face the burden of a 
potential increase in spam (lower case! ;-) ) to release these two other 
checks.

As such, I have now removed those checks, and we should be good to go.

A few closing points -

TOM PETCH - THANK YOU for taking the time to explain the consensus 
process to me, and your views on its operability.  I found that message 
to be extremely valuable, and I greatly appreciate it.

DANNY - Thanks for catching the IPV6 thing!  I'll get that blocked later 
today.  (I don't think that's been widely-used yet, and I hadn't even 
considered it, but you are quite right, and I very much appreciate you 
pointing that out.)

ERIC - Thanks for being the pro-forma guinea pig here, and taking the 
time to explain your position so clearly.

As for the list moderation messages, those occurred because we were all 
group-copying everyone on this thread, and mailman apparently doesn't 
like messages with "too many recipients."  I've released the last of the 
held messages (including my own, and this one, momentarily!) and that 
should do it.

Finally, I saw mention of interest in creating a web page that could 
provide bypass mechanisms to contact the IETF.  I'm referring that back 
to Jim Galvin to discuss with your ED, Alexa Morris.

As a point of interest, we do have the full ability to bypass individual 
senders, hosts, clients, recipients, etc., from being subject to these 
or any tests.  For example, if it was decided that you wanted one or 
more of these checks put in place, but you wanted all mail "TO" 
[email-masked] (or whatever) to be exempted from the checks, that is easily 
easily done.  Just say the word.

For now, the other two problematic checks have been removed, and, 
barring further direction, I'll "close this one out" on that note!

As always, feel free to contact me at any time for any reason.

Warm regards,
Glen Barney
Server Servant
AMS