Subject: [P2PSIP] Wired News and AP report Skype taken down by avalanche retsart problem

From: Dean Willis <[email-masked]>
Date: Mon, 20 Aug 2007 14:56:58 -0500

There's a design lesson we need to learn here. Even without a global
power failure, P2P networks are subject to avalanche restarts induced by
other sources. This requires designing robust restart characteristics
into the overlay.

From:

http://news.wired.com/dynamic/stories/G/GERMANY_SKYPE_OUTAGE?SITE=WIRE&SECTION=HOME&TEMPLATE=DEFAULT

Skype Outage Caused by Microsoft Update

By MATT MOORE
AP Business Writer

FRANKFURT, Germany (AP) -- A two-day outage that left millions of Skype
users unable to use the popular Internet phone service was caused by an
abnormally high number of restarts after people had downloaded a Windows
security update, the company said Monday.

The worldwide outage, which began on Thursday and ended on Saturday,
left millions of Skype users unable to log on to make phone calls or
send instant messages.

Luxembourg-based Skype Ltd., part of online auction giant eBay Inc., has
more than 220 million users in total but typically has 5 million to 6
million users online at any given time. In January, Skype reported that
it had counted 9 million users online at one time.

In an update to users on Skype's Heartbeat blog, employee Villu Arak
said the disruption was not because of hackers or any other malicious
activity.

Instead, he said that the disruption "was triggered by a massive restart
of our users' computers across the globe within a very short timeframe
as they re-booted after receiving a routine set of patches through
Windows Update," Arak wrote.

Microsoft Corp. released its monthly patches last Tuesday, and many
computers are set to automatically download and install them.
Installation requires a computer restart.

"The high number of restarts affected Skype's network resources. This
caused a flood of log-in requests, which, combined with the lack of
peer-to-peer network resources, prompted a chain reaction that had a
critical impact," Arak wrote.

Arak did not blame Microsoft for the troubles and said the outage
ultimately rested with Skype. Arak said Skype's network normally has an
ability to heal itself in such cases, but a previously unknown glitch in
Skype's software prevented that from occurring quickly enough.

In a statement, Microsoft described its patch as routine and reiterated
that the disruption resulted from a bug in Skype software.

Users from Vietnam to Brazil to Germany to the United States had
complained they could not log on and make phone calls or send instant
messages.

The outage was a critical moment for the company, founded in 2003 by
Niklas Zennstrom and Janus Friis, and was the first major outage since
October 2005 when its service was down only for a few hours.

"This disruption was unprecedented in terms of its impact and scope,"
Arak wrote. "We would like to point out that very few technologies or
communications networks today are guaranteed to operate without
interruptions."

_______________________________________________
P2PSIP mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/p2psip

---

From: "Francois Audet" <[email-masked]>
Date: Mon, 20 Aug 2007 15:02:14 -0500

There are quite many messages from this meltdown:

- Security by obscurity is bad

- Relying on single source proprietary implementations is bad

- Believing in a "magic box" to solve all problems is bad 

> -----Original Message-----
> From: Dean Willis [mailto:[email-masked]] 
> Sent: Monday, August 20, 2007 12:57
> To: P2PSIP Mailing List
> Subject: [P2PSIP] Wired News and AP report Skype taken down 
> by avalanche retsart problem
> 
> There's a design lesson we need to learn here. Even without a 
> global power failure, P2P networks are subject to avalanche 
> restarts induced by other sources. This requires designing 
> robust restart characteristics into the overlay.
> 
> 
> From:
> 
> FRANKFURT, Germany (AP) -- A two-day outage that left 
> millions of Skype users unable to use the popular Internet 
> phone service was caused by an abnormally high number of 
> restarts after people had downloaded a Windows security 
> update, the company said Monday.
> 
> The worldwide outage, which began on Thursday and ended on 
> Saturday, left millions of Skype users unable to log on to 
> make phone calls or send instant messages.
> 
> Luxembourg-based Skype Ltd., part of online auction giant 
> eBay Inc., has more than 220 million users in total but 
> typically has 5 million to 6 million users online at any 
> given time. In January, Skype reported that it had counted 9 
> million users online at one time.
> 
> In an update to users on Skype's Heartbeat blog, employee 
> Villu Arak said the disruption was not because of hackers or 
> any other malicious activity.
> 
> Instead, he said that the disruption "was triggered by a 
> massive restart of our users' computers across the globe 
> within a very short timeframe as they re-booted after 
> receiving a routine set of patches through Windows Update," 
> Arak wrote.
> 
> Microsoft Corp. released its monthly patches last Tuesday, 
> and many computers are set to automatically download and install them.
> "The high number of restarts affected Skype's network 
> resources. This caused a flood of log-in requests, which, 
> combined with the lack of peer-to-peer network resources, 
> prompted a chain reaction that had a critical impact," Arak wrote.
> 
> Arak did not blame Microsoft for the troubles and said the 
> outage ultimately rested with Skype. Arak said Skype's 
> network normally has an ability to heal itself in such cases, 
> but a previously unknown glitch in Skype's software prevented 
> that from occurring quickly enough.
> 
> In a statement, Microsoft described its patch as routine and 
> reiterated that the disruption resulted from a bug in Skype software.
> 
> Users from Vietnam to Brazil to Germany to the United States 
> had complained they could not log on and make phone calls or 
> send instant messages.
> 
> The outage was a critical moment for the company, founded in 
> 2003 by Niklas Zennstrom and Janus Friis, and was the first 
> major outage since October 2005 when its service was down 
> only for a few hours.
> 
> Arak wrote. "We would like to point out that very few 
> technologies or communications networks today are guaranteed 
> to operate without interruptions."
>

---

From: "David Barrett" <[email-masked]>
Date: Mon, 20 Aug 2007 13:13:48 -0700

What I don't understand is why it took *2 days* to get back up and running.

I mean, let's say all 9M Skype users restarted at the exact same time.  If
it takes 2 days for that many clients to log in, that's only 52 logins per
second: certainly there's something else going on here.  (As comparison,
consider how many logins per second Hotmail must handle?)

It seems to me that the overload problem cannot possibly be due to a lack of
central authentication capacity -- that's not the weak link in this chain.

It seems more probable that the problem is actually due to the P2P network
itself: that their DHT (or whatever they use) is able to handle minor blips
and growth, but given a massive churn event just died entirely.

(I'm visualizing how you can slowly pour vinegar into oil and it stays in a
big pool, but if you splash it in it explodes into a thousand tiny,
individual pools that can't reform.)

If that's true, I'm curious: what's the benefit that Skype is getting from
putting rendezvous functionality into a P2P datastructure rather than just
handling it centrally?

-david

> From: Francois Audet [mailto:[email-masked]]
> Sent: Monday, August 20, 2007 1:02 PM
> To: Dean Willis; P2PSIP Mailing List
> Subject: RE: [P2PSIP] Wired News and AP report Skype taken down by
> avalancheretsart problem
>

---

From: "Peter Pan" <[email-masked]>
Date: Mon, 20 Aug 2007 14:12:59 -0700

these msgs seem a "conversation" in chinese. ho~ho~pp

----- Original Message -----
From: "Francois Audet" <[email-masked]>
To: "Dean Willis" <[email-masked]>; "P2PSIP Mailing List"
<[email-masked]>

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 20 Aug 2007 18:40:06 -0400 (EDT)

Frankly, the only conclusion I will draw from this article is that churn 
mitigation is not a trivial problem. As Dean mentioned, the peer protocol 
should provide mechanisms to be robust against churn and must assume the 
worst churn scenario. The worst scenario is that the majority of nodes can 
(and will) die at the same time.