Subject: "Cottrell, Les": RE: [IMRG] Internet measurement: what next?

From: Mark Allman <[email-masked]>
Date: Wed, 09 Jul 2003 17:40:49 -0400

Folks-

I asked Les to distill the high points from the slides he posted
last week (pointer in his note).  He graciously agreed and distilled
the thoughts in the attached message (which he explicitly said I
could forward to the list).

My own feeling is that there is some good stuff in these notes to
mull over and discuss.  I'd be thrilled to hear opinions.

allman

------- Forwarded Message

Date: Sat, 05 Jul 2003 10:49:14 -0700
From: "Cottrell, Les" <[email-masked]>
Subject: RE: [IMRG] Internet measurement: what next?
To: "[email-masked]'" <[email-masked]>

[...]

=============================================================================
High performance networking measurement needs for Science:
Presentation to the US Interagancy Large Scale Networking (LSN) group
Les Cottrell, KC Claffy, Brian Tierney, Ronn Ritke, Nahs Wernetr Braun, June 10, 2003 
(for more details see http://www.slac.stanford.edu/grp/scs/net/talk03/lsn-jun03_files/frame.html including the notes appended to the slides)

Overview
* The end-to-end challenge, illustrations
* Solution
* End to end Monitoring Goals
* Current issues
* Problem analysis, measurement infrastructure, analysis tools, standards, collaborations
* Benefits to Science
* Consequences of not addressing issues
* Why not leave to industry
Appendix
* What is being done today
* Who is measuring?
* Who is using the measurements?
* What is being measured?
* What tools are being used?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Distributed systems are very hard
"A distributed system is one in which I can't get my work done because a computer I've never heard of has failed." Butler Lampson
When building distributed systems, we often observe unexpectedly low performance the reasons for which are usually not obvious
The bottlenecks can be in any of the following components:
* the applications
* the operating systems
* the disks, network adapters, bus, memory, etc. on either the sending or receiving host
* the network switches and routers, and so on
Problems may not be logical
* Most problems are operator errors, configurations, bugs 

Solution:
* A complete End-to-End monitoring framework that includes the following components: 
  - instrumentation tools (application, middleware, and OS monitoring) 
  - host and network sensors  (host and network monitoring)
  - sensor management  / activation tools
  - event publication service
  - event archive service
  - event analysis and visualization tools
  - a common set of protocols for describing, exchanging, and locating monitoring data
	o Need for applications (e.g. Grid middleware), diagnosis, perf. analysis
  - toolkit for streamlined problem diagnosis: detection, location, isolation & reporting 
	o glue to multiple sources of information, traceroute archives, router info, delay/loss archives, on-demand tests, baselines
	o analysis and heuristics
  - E2EPi working on solution, but only funded for coordination not for all the underlying work

End-to-end Monitoring Goals
* Have to solve the E2E performance, it is THE critical metric for user, not just a backbone bandwidth problem
* Improve end-to-end data throughput for data intensive applications in a high-speed WAN environments 
* Provide the ability to do performance analysis and fault detection in a Grid computing environment
* Provide accurate, detailed, and adaptive monitoring of all of distributed computing components, including the network
"Unfortunately, network management research has historically been very under-funded, because it is difficult to get funding bodies to recognize this as legitimate networking research," IAB Concerns & Recommendations Regarding Internet Research & Evolution

Current Issues:
1. Problem Analysis
  - Cultivate systematic studies of problems, causes, how to discover, how to report, how to by-pass
	o Analysis to help in deciding what are the most important problems, see how they are tackled manually today
	o Decide on which problems are most cost-effective to assist in developing tools to assist in diagnosis
2. Need to build infrastructures to support trouble-shooting
  - Requires repetitive and on-demand measurements with appropriate security model. 
  - Provide recommended/accepted set of tools for delay, RTT, loss, route tracking, "bandwidth" estimation. 
	o Include archiving and access to data, analysis and reporting of repetitive data. 
  - Allow for evaluation, validation and comparison of new measurement tools, TCP stacks, applications (e.g. file transfer).
  - Reverse traceroute, looking glass, remote tcpdump (e.g. SCNM), remote testing of connection (ANL NDT), 
  - Traceroute archives
  - Make tools easier to comprehend and use by scientists
  - Encourage efforts such as Internet2 E2Epi efforts to provide measurements inside the cloud
	o Extend to ESnet & other NRNs, and beyond
	o Fund collaboration across boundaries 
  - Ubiquitous coverage (require multiple toolkits): Inter agency, international, hi-speed, digital divide, long term and current
3. Analysis tools
  - Provide measurement tools to accurately & quickly identify performance problems, 
	o to automatically take action to investigate and provide information for:
		> Scientist
		> Grid support "NOC"
		> Network administrator or network person 
	o Promote well understood, accepted metrics for customers for realistic, enforceable SLAs, 
		>provide acceptable limits, 
		>provide tools to track  
4. Standards
  - All the above requires:
	o easy to use standard ways (e.g.web services) for applications to access data from existing and new monitoring projects. 
	o standard naming conventions and schemas.
  - This will provide the ability to share information from multiple measurement infrastructure projects
5. Collaboration
  - Need to build multi-disciplinary teams (incent orthogonal groups to work with one another): 
	o include people close to eventual customers (scientists, operational folks)
		> to ensure what is developed is useful, tested out in realistic environments
	o include vendors and providers in funded projects to bridge the gaps
  - E2Epi is funded to provide coordination
  - Multi agency funding!
	o This is not a problem a single agency can address
	o Science applications cross multi-agency networks, but barriers to interagency network monitoring collaborations

Benefits to Science
* Network reaches its potential
  - enable new ways of doing science: 
	o data intensive science (astrophysics, global weather, seismology, medicine), 
	o remote instrument control (SNS, fusion(ITER), surgery), 
	o remote visualization/insight (Terascale supernova, climate modeling), 
	o world-wide collaboration enabling (LHC, ITER)
  - enables scientists to do science
	o Wizard gap closure, not using all time fighting the network, rather network becomes a catalyst
  - Without good troubleshooting capabilities, the Grid vision will fail
  - Predictability, planning, expectations, raising the bar

What happnes if we do not address
* Data continues to ship inefficiently by truck/plane FedEx
  - Long delays (2 weeks), degraded collaboration, US scientists continue to lose leadership
  - Increased costs (manpower costs, lack of automation)
  - Inadequate reliability or performance for new applications, (e.g. Grid fails to reach its potential)
  -New capabilities do not emerge in US:
	o remote instrument control, real-time video, media distribution... 
	o US science loses leadership to Japan, Europe, Canada

Why no leave it to Industry
* Industry won't do it ("it's not my problem"):
* Has its interest and hands full elsewhere
* It's hard, does not sell products, little Return on Investment
* Historically poor record, competitive concerns
  - Management features are late in product development cycle
  - Early success with SNMP and Netflow
  - Commercial Network Management Platforms's (e.g. OpenView, Tivoli) limited success (network oriented, not user), not cost effective for small organizations
  - ISPs only measure own nets, not E2E, SLA guarantees are not cross-provider

------- End of Forwarded Message

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Mark Allman <[email-masked]>
Date: Wed, 09 Jul 2003 18:01:26 -0400

Les-

Les Cottrell:
> 	o Analysis to help in deciding what are the most important
> 	  problems, see how they are tackled manually today 
> 	o Decide on which problems are most cost-effective to assist
> 	  in developing tools to assist in diagnosis 

I think I have a little heartburn with this...  We **do** study
problems and report on the results.  We have papers on TCP
performance problems and BGP problems and DNS behavior and HTTP
problems and ....

So, we have all these problems sort of being worked on by various
arms of the community.  Is the real issue here that we don't have
these things pulled together in a way that folk can figure out what
has been done?  Or, are we fundementally doing things in a broken
way such that these studies are not Good Enough?

>     appropriate security model. 

Right on.

>   - Provide recommended/accepted set of tools for delay, RTT,
>     loss, route tracking, "bandwidth" estimation.  

Do we know what these tools are?  I.e., is really all that needs
done is to write something that says "thou shalt use these things to
facilitate comparisons"?  Or, are we still struggling with figuring
some or all of the above out?  (I.e., it seems to me that we have
not reached consensus on "bandwidth estimation".  Or, have we?)

> 	  reporting of repetitive data.  

Yes.

>   - Allow for evaluation, validation and comparison of new
>     measurement tools, TCP stacks, applications (e.g. file
>     transfer).

What do you envision?  This is really hard, right?  Validating a
tool that maps the topology (for instance) is quite hard, right?  We
have lots of techniques that we all admit have weaknesses.  So, what
do we validate against?  What is the approach here?

Or, how do I validate a new TCP stack?  There is such hetrogeneity
in TCP implemenations these days (even between linux versions!),
what can we compare to as "the truth"?  

(I realize this was a plea for an infusion of US government $$$,
but...) 

>   - Commercial Network Management Platforms's (e.g. OpenView,
>     Tivoli) limited success (network oriented, not user), not cost
>     effective for small organizations 

This doesn't seem right to me.  Some industry has a vested
interest and has demonstrated that with contributions to this field,
I believe.  Providers (no matter size), for instance, surely want to
be able to measure their networks to improve performance.  Companies
that serve web pages no doubt want to do it faster than their
competitors and would like to be able to get some good data on such
things.  Etc.  Etc.  Etc.

Plus, in your list you are noting that you need more visibility into
the middle of the network.  I don't doubt that.  However, that is
going to take a partnership with industry at a minimum, right?

I am certainly not claiming that industry is going to solve all the
problems.  On the other hand, I think that they have and will likely
continue to play a valuable and needed role.  Right?

Thanks!

allman

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/