Subject: [p2pi] Real life torrent statistics

From: "Stas Khirman" <[email-masked]>
Date: Mon, 18 Aug 2008 01:07:50 -0700

To estimate a feasibility of ALTO/P4P for real life torrents , I collected
<ip,port> information for peers from one of the most popular "PirateBay"
torrents ( almost 20k peers) and maped their IPs to corresponded ASs. Please
find attached my working notes with some interesting statistics.

In short - for such kind of swarm (20k peers), almost half of peers belong
to ASs with 100 or more peers, almost 2/3 of peers belong to ASs with 50 or
more peers. Less then 15% of peers belong to ASs with 10 or less
"neighbors".

US broadband ISPs (Road Runner, Comcast, AT&T) have about 2-3% of total
peers each.

Also, I find it surprising geo distribution of the peers - majority were in
UK , not in US (probably because content is available in US theaters).
Places 3-5 taken by Sweden, Poland and Canada (in total - more peers then in
US).

Certainly, observed "heavy" neighboring of peers is a function of swarm
size. I intend to investigate a few medium/small size swarms to have a
multi-point picture for any future discussions.

I would appreciate comments, corrections and additional ideas on data
collection and analysis. 

_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: "Hao Chen" <[email-masked]>
Date: Mon, 18 Aug 2008 17:43:24 +0800

How many ASes does the overlay span totally? It seems the hotspot effect is
not very obvious as compared to previous study. What about cluster effect of
overlay by which I mean the ratio of neighbors of the peer in the same AS or
ISP?
2008/8/18 Stas Khirman <[email-masked]>

>  To estimate a feasibility of ALTO/P4P for real life torrents , I
> collected <ip,port> information for peers from one of the most popular
> "PirateBay" torrents ( almost 20k peers) and maped their IPs to corresponded
> ASs. Please find attached my working notes with some interesting statistics.
>
>
>
> In short – for such kind of swarm (20k peers), almost half of peers belong
> Also, I find it surprising geo distribution of the peers – majority were in
>  Places 3-5 taken by Sweden, Poland and Canada (in total – more peers then
> in US).
>
>
>

-- 
Cheers,Hao

Fudan University

---

From: "Stas Khirman" <[email-masked]>
Date: Mon, 18 Aug 2008 11:00:48 -0700

Sorry that I missed this number in my notes .

The dataset includes 19333 unique <ip,port> pairs ( some IPs are repeatable)
from 1459 ASs. Or 13.25 peers/AS in average.

While aggregated by "ISP name", I got 1368 unique ISP names ( not very
accurate estimation by second part of AS name) or 14.1 peers per "ISP"

  _____  

From: Hao Chen [mailto:[email-masked]] 
Sent: Monday, August 18, 2008 2:43 AM
To: Stas Khirman
Cc: [email-masked]; [email-masked]
Subject: Re: [p2pi] Real life torrent statistics

-- 
Cheers,Hao

Fudan University

---

From: Reinaldo Penno <[email-masked]>
Date: Mon, 18 Aug 2008 20:06:06 -0700

Hello Stas,

It is refreshing to get real life data like this instead of some Linux
distribution or small legal file that has little relevance. I was wondering
what kind of info we can get from your data that shed light on ALTO
discussions. One that caught my attention was that pure localization will
probably adversely affect certain swarms (even huge ones like this), since
you do not have enough peers in certain ASs.  If the connectivity of the
swarm is not good, getting certain pieces might be difficult even if the
ones ³around² you can be downloaded lightening fast.

I wonder what is the distribution of seeders vs. AS.  I¹m thinking certain
ASs might have enough peers but maybe no seeders. This is also a tricky
situation for optimization where the swarm are islands connected by relay
peers.  

It seems a lot of the bw savings ALTO can deliver is probably a cumulative
effect over many torrents and many peers instead of just one.

Thanks,

Reinaldo

On 8/18/08 1:07 AM, "Stas Khirman" <[email-masked]> wrote:

> <ip,port> information for peers from one of the most popular ³PirateBay²
> In short  for such kind of swarm (20k peers), almost half of peers belong to
> ASs with 100 or more peers, almost 2/3 of peers belong to ASs with 50 or more
> peers. Less then 15% of peers belong to ASs with 10 or less ³neighbors².
>  
> US broadband ISPs (Road Runner, Comcast, AT&T) have about 2-3% of total peers
> each.
>  
> Also, I find it surprising geo distribution of the peers  majority were in UK
> , not in US (probably because content is available in US theaters).  Places
> 3-5 taken by Sweden, Poland and Canada (in total  more peers then in US).
>  
> Certainly, observed ³heavy² neighboring of peers is a function of swarm size.
> I intend to investigate a few medium/small size swarms to have a multi-point
> picture for any future discussions.
>

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 19 Aug 2008 00:36:35 -0400

On Aug 18, 2008, at 4:07 AM, Stas Khirman wrote:

> collected <ip,port> information for peers from one of the most  
> popular “PirateBay” torrents ( almost 20k peers) and maped their IPs  
> to corresponded ASs. Please find attached my working notes with some  
> interesting statistics.
>
> In short – for such kind of swarm (20k peers), almost half of peers  
> belong to ASs with 100 or more peers, almost 2/3 of peers belong to  
> ASs with 50 or more peers. Less then 15% of peers belong to ASs with  
> 10 or less “neighbors”.
>

A pdf version of this is at

http://www.americafree.tv/papers/torrent_distribution.pdf

(Note, by the way, that some people block attachments sent through  
IETF mail lists.)

These data seem to follow a power law for part of their range and I  
would hypothesize that the
number of peers versus rank follows a Mandelbrot - Pareto distribution.

Here is a plot of the top 20 with a power law superimposed:

http://www.americafree.tv/papers/torrent.top_20.png

Regards
Marshall

>
> US broadband ISPs (Road Runner, Comcast, AT&T) have about 2-3% of  
> total peers each.
>
> Also, I find it surprising geo distribution of the peers – majority  
> were in UK , not in US (probably because content is available in US  
> theaters).  Places 3-5 taken by Sweden,Poland and Canada (in total –  
> more peers then in US).
>
> Certainly, observed “heavy” neighboring of peers is a function of  
> swarm size. I intend to investigate a few medium/small size swarms  
> to have a multi-point picture for any future discussions.
>
> I would appreciate comments, corrections and additional ideas on  
> data collection and analysis.
> <torrent_distribution.doc>

---

From: "Stas Khirman" <[email-masked]>
Date: Mon, 18 Aug 2008 23:48:38 -0700

[Stas Khirman] 

Marshall,

Thank for your investigation, but unfortunately, when considering tail of
the distribution I'm not sure it power distribution can be claimed - please
check

http://www.khirman.com/system/files/image/blog/torrent_distribution_blog_2.g
if 

By popular demand, I just published major points and raw data on my blog :

http://www.khirman.com/blog/p2p_localisation 

Working notes available at
http://www.khirman.com/system/files/image/blog/torrent_distribution.pdf 

Raw AS data http://www.khirman.com/system/files/image/blog/asn_groups.csv -
please feel free to "twist" it - will appreciate any input.

Regards

Stas Khirman

P.S. For those who in Silicon Valley - would be glad if you join our Club
event August 21 - http://www.khirman.com/ctc/20080821 . Hot controversial
discussion and cold beer - guaranteed! (please sign-up if you plan to
attend)

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 19 Aug 2008 10:35:52 -0400

On Aug 19, 2008, at 2:48 AM, Stas Khirman wrote:

> Thank for your investigation, but unfortunately, when considering  
> tail of the distribution I'm not sure it power distribution can be  
> claimed - please check
> http://www.khirman.com/system/files/image/blog/torrent_distribution_blog_2.gif

Well, you generally can't tell anything from a linear-linear plots in  
these cases.

However, it is common in these cases for there to be a flattening with  
small rank
(thus the Pareto-Mandelbrot or Zipf-Mandelbrot distribution), and a  
lack of power as the numbers per bin get to
be of order unity. Both of these can be seen with the full data set

http://www.americafree.tv/papers/asn_groups.png
http://www.americafree.tv/papers/asn_groups.2.png

With only 3 degrees of freedom, the modified power law fits well for  
about 2 decades of rank.

This is not too surprising to me - similar Zipf type modified power  
laws can be seen in the selection
of video content (see http://www.imconf.net/imc-2007/papers/ 
imc78.pdf ) and I would
expect node selection to trace that.

Regards
Marshall

> Working notes available at http://www.khirman.com/system/files/image/blog/torrent_distribution.pdf
>
> Raw AS data http://www.khirman.com/system/files/image/blog/asn_groups.csv 
>  - please feel free to “twist” it – will appreciate any input.
>
>
> Regards
> Stas Khirman
> P.S. For those who in Silicon Valley – would be glad if you join our  
> Club event August 21 - http://www.khirman.com/ctc/20080821. Hot  
> controversial discussion and cold beer – guaranteed! (please sign-up  
> if you plan to attend)
>
>

---

From: Salman Abdul Baset <[email-masked]>
Date: Tue, 19 Aug 2008 16:33:50 -0400 (EDT)

We have some related data from measuring Skype relays.
http://www1.cs.columbia.edu/~salman/publications/skyperelay08.pdf

-s

---

From: The 8472 <[email-masked]>
Date: Wed, 20 Aug 2008 01:40:39 +0200

Stas Khirman wrote:

To estimate a feasibility
of ALTO/P4P for real life torrents
, I collected <ip,port> information for peers from one of the
most
popular “PirateBay” torrents ( almost 20k peers) and maped their
IPs to corresponded ASs. Please find attached my working notes with
some
interesting statistics.

Ahh, there is a problem with this one. With torrents you have a
significant long tail when it comes to swarm sizes and content. I'm not
certain about distribution, but the long tail will probably outweight
the... let's say top 100 torrents. Torrents with only 10-20 peers
spread throughout several ASNs is a much harder to optimize than the
top 100.

This problem is aggreviated by swarm fragmentation due to private
trackers and since bittorrent does not aim to coalesce all torrents
with the same content, i.e. due to different piece sizes, file names
etc.

Also, I find it
surprising geo distribution of the peers –
majority were in UK
, not in US (probably because content is available in US theaters).
Places
3-5 taken by Sweden,
Poland and Canada
(in total – more peers
then in US).

This will probably be different if you sample torrents from regional
trackers or torrents aimed at other audiences. During some DHT tracing
on the weekend i saw a significant proportion of DHT traffic coming
from asian countries, though i suspect an inefficient implementation of
the DHT by a client that's popular in china to play some role in this
distribution.

Certainly, observed
“heavy” neighboring of peers
is a function of swarm size. I intend to investigate a few medium/small
size
swarms to have a multi-point picture for any future discussions.

as i mentioned above we should try to get the big picture, i.e. how
relevant the long tail is, measured in aggregate bandwidth. If the
small torrents actually make up the bulk of the traffic then any
solution will require a high degree of cooperation between ISPs, e.g.
caches that cooperate with each other.

--

The 8472

independent developer for the Azureus Vuze Bittorrent client