Subject: RE: [IMRG] Internet measurement: what next?

From: "Cottrell, Les" <[email-masked]>
Date: Wed, 02 Jul 2003 13:18:21 -0700

A few of us (k claffy, Hans Werner-Braun, Brian Tierney, Ronn Ritke and myself) recently made a presentation to the government Inter-agency Large Scale Networks (LSN) group on this topic. There were two presentations, one on the needs/challenges of End-to-end monitoring especially for high performance applilcations, the second on a more Network centric needs for consumer grade applications.  The former talk may be found at:
http://www.slac.stanford.edu/grp/scs/net/talk03/lsn-jun03.html or http://www.slac.stanford.edu/grp/scs/net/talk03/lsn-jun03.ppt
The latter may be found at:
http://www.caida.org/outreach/presentations/2003/lsn20030610/

-----Original Message-----
From: Mark Allman [mailto:[email-masked]] 
Sent: Wednesday, July 02, 2003 9:30 AM
To: [email-masked]
Subject: [IMRG] Internet measurement: what next?

I am interested in seeing some discussion about the fairly generic topic of what sorts of outstanding problems exist in Internet
measurement:

    What are the core problems we need to solve?  
    What do we need to be able to measure that we cannot measure
      very well today?  
    Why?  
    What are the problems in taking measurements?  
    What can be done about those problems?
    What are practical problems that prevent wide-scale network
      measurement?
    What new tools does the community need to take, use, share and
      store measurements?

Please jump in with answers o any or all of the above.  Or, even with additional big-picture sorts of questions of your own.

Thanks!

allman
(imrg chair)

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: "Cottrell, Les" <[email-masked]>
Date: Mon, 14 Jul 2003 11:37:52 -0700

Mark: Thanks for the comments.

> Les Cottrell:
> > Current Issues:
> > 1. Problem Analysis
> >   - Cultivate systematic studies of problems, causes, how to
> >     discover, how to report, how to by-pass 
> > 	o Analysis to help in deciding what are the most important
> > 	  problems, see how they are tackled manually today 
> > 	o Decide on which problems are most cost-effective to assist
> > 	  in developing tools to assist in diagnosis
> 
> I think I have a little heartburn with this...  We **do** 
> study problems and report on the results.  We have papers on 
> TCP performance problems and BGP problems and DNS behavior 
> and HTTP problems and ....
> 
> So, we have all these problems sort of being worked on by 
> various arms of the community.  Is the real issue here that 
> we don't have these things pulled together in a way that folk 
> can figure out what has been done?  Or, are we fundementally 
> doing things in a broken way such that these studies are not 
> Good Enough?
This was in response to concerns raised by Vern Paxson and others, see for example: Vern's talk Aug2001 www.icir.org/vern/talks/vp-nrdm01.ps.gz and also see David Moore's talk http://www.caida.org/outreach/presentations/2002/ipam0203/ipam0203.pdf. Kc felt strongly that we needed to make a point of this. Your points are correct there is a bit of both. 
 
> >   - Provide recommended/accepted set of tools for delay, RTT,
> >     loss, route tracking, "bandwidth" estimation.
> 
> Do we know what these tools are?  I.e., is really all that 
> needs done is to write something that says "thou shalt use 
> these things to facilitate comparisons"?  Or, are we still 
> struggling with figuring some or all of the above out?  
> (I.e., it seems to me that we have not reached consensus on 
> "bandwidth estimation".  Or, have we?)
In some cases (e.g. loss & delay) the tools are reasonably well known (though there are problems about whether say ping measures the same losses or RTT as seen by TCP), in other cases (bandwidth estimation) the jury is still out and there are multiple tools that measure different things (available bandwidth, bottleneck capacity, achievable bandwidth etc.) and how they relate to one another is not fully understood, and in some cases (packet pair & train methods) may not work well at higher speeds with advanced ToE NICs. 

> >   - Allow for evaluation, validation and comparison of new
> >     measurement tools, TCP stacks, applications (e.g. file
> >     transfer).
> 
> What do you envision?  This is really hard, right?  
> Validating a tool that maps the topology (for instance) is 
> quite hard, right?  We have lots of techniques that we all 
> admit have weaknesses.  So, what do we validate against?  
> What is the approach here?
> 
> Or, how do I validate a new TCP stack?  There is such 
> hetrogeneity in TCP implemenations these days (even between 
> linux versions!), what can we compare to as "the truth"?  
> 
> (I realize this was a plea for an infusion of US government $$$,
> but...) 
You are right this IS really hard.  There are few if any absolute "truths" today.
For tools I was thinking to compare tools (e.g. iperf vs. pathload vs GridFTP vs say BW=a*MSS/(RTT*sqrt(loss)) to understand the regions of applicability, how well one can use one to deduce another (e.g. a lightweight measure (ping or pathload) to predict what a heavier weight measurement (say iperf or GridFTP) might see). In the case of capacity there may be something close to truth to validate against. Possibly if we had instantaneous utilizations of all network devices alonmg a path we might be able to validate some of the available bandwidth tools. I have not thought about topology.
For TCP stacks, applications and users want the best throughput, with the easiest way to use (e.g. do not have to optimize both number of parallel streams AND max window size), the best predictability, plus a feeling they are not being grossly unfair to others.  Networkers will put more emphasis on fairness, and maybe agility etc. There is currently no absolute "truth".
 
> > Why not leave it to Industry
> > * Industry won't do it ("it's not my problem"):
> > * Has its interest and hands full elsewhere
> > * It's hard, does not sell products, little Return on Investment
> > * Historically poor record, competitive concerns
> >   - Management features are late in product development cycle
> >   - Early success with SNMP and Netflow
> >   - Commercial Network Management Platforms's (e.g. OpenView,
> >     Tivoli) limited success (network oriented, not user), not cost
> >     effective for small organizations 
> >   - ISPs only measure own nets, not E2E, SLA guarantees are not
> >     cross-provider
> 
> This doesn't seem right to me.  Some industry has a vested 
> interest and has demonstrated that with contributions to this 
> field, I believe.  Providers (no matter size), for instance, 
> surely want to be able to measure their networks to improve 
> performance.  Companies that serve web pages no doubt want to 
> do it faster than their competitors and would like to be able 
> to get some good data on such things.  Etc.  Etc.  Etc.
You are right providers such as Sprint and AT&T want to measure their 
own networks ("you can't manage what you don't measure"), however for the end-users this does not provide end-to-end performance expectations.
As you say web services companies ought to be interested in this in order to provide better customer service. However there does not appear to be much out there are the moment, it's mainly traffic light red/yellow/green

> Plus, in your list you are noting that you need more 
> visibility into the middle of the network.  I don't doubt 
> that.  However, that is going to take a partnership with 
> industry at a minimum, right?
Right on. We need to engage industry to provide this visibility. Currently this is pretty limited. One way to make a start on this would be to engage the Academic & Research networks (e.g. Internet2, Esnet, GEANT, ...). From this an understanding of what is useful, and how to provide access (publicize and subscribe, standard naming etc.)could emerge. Internet2 seems to be particularly interested in this. However, it is still a long step to having the information available from industry at large.  

> I am certainly not claiming that industry is going to solve 
> all the problems.  On the other hand, I think that they have 
> and will likely continue to play a valuable and needed role. Right?
Right.