Subject: [Nmlrg] ENTRADA goes open source (.nl Hadoop platform)

From: "Giovane C. M. Moura" <[email-masked]>
Date: Fri, 29 Jan 2016 13:25:09 +0100

[apologies for multiple copies]

*************************************************************
ENTRADA GOES OPEN SOURCE
http://entrada.sidnlabs.nl/
*************************************************************

SIDN Labs [1] is happy to announce that we are releasing our ENTRADA
platform as an open source project [2]. ENTRADA is a Hadoop-based
platform designed to ingest and quickly analyze large amounts of network
data. More technically, it is a high-performance data streaming
warehouse (DSW).

Please refer to our research paper [3] for a performance evaluation and
more details.

WHO CAN USE IT:

* Internet measurement researchers who are in need of a
high-performance analytics platform
* Domain name registries and other network operators interested in
developing DNS big data applications (e.g:[4])

MAIN FEATURES:

* Performance: analyze the Parquet equivalent of 50 TB of pcap data
in under 3.5 minutes, with a small 6-node cluster (4 data processing
nodes).
* Interface: benefit from easy SQL statements to analyze your data
* Scalable: add more nodes for faster processing and more storage
* Built-in support for DNS (and TCP/UDP/IP fields too) and ICMP
network data; other protocols may be added too.

DEPLOYMENT:

We have been using ENTRADA at SIDN Labs for the past 1.5 years. It
currently runs on a relatively small 6 node cluster (with 4 data-nodes),
which store DNS traffic from the .nl authoritative servers. In total,
ENTRADA has more than 100 billion queries and responses stored, and 400
million are added on a daily basis. It can also be deployed on a cloud
environment.

We also make available open aggregated datasets at [5].

ABOUT SIDN AND SIDN LABS

SIDN [6] manages .nl, the country code TLD of the Netherlands.
SIDN Labs [1] is SIDN's R&D team, which develops
and evaluates new technologies and systems to improve both stability and
security of .nl zone.

References:

[1] https://www.sidnlabs.nl/
[2] http://entrada.sidnlabs.nl/
[3] https://www.sidnlabs.nl/sidn-noms2016.pdf
[4] http://iepg.org/2015-11-01-ietf94/iepg-moura.pdf
[5] http://stats.sidnlabs.nl
[6] https://sidn.nl