Subject: possible new item to add to the framework?

From: "Kastenholz, Frank" <[email-masked]>
Date: Tue, 24 Jul 2001 07:40:27 -0400

Hi,
Speaking as a technical contributor, NOT co-chair

802.17's performance subcommittee is putting together 
a document which includes TCP/UDP performance modelling 
and simulation as an optional item. 

A few weeks ago, the IEEE 802.17 group mentioned that
they would be doing simulations to determine performance,
etc, of 802.17. I responded that simulating Internet
traffic is not as useful as it seems since the traffic
patterns and mixes change and the TCP algorithms (the
key ones) are always under development. So it's not
clear to me that there is a lot of benefit to be
derived from this effort. Doing TCP modelling would,
of course, check that the current TCP and traffic
patterns do not break, but say nothing about future changes.

Also, Internet experience has shown that the best strategy
(with regard to TCP timeouts/retransmissions) for the lower-
layers is to keep things absolutely as simple as possible.
If they try to "help" TCP, things usually have ended up worse
rather than better.

So, should we add this to the framework?

Frank Kastenholz
Not-As-Co-Chair

---

From: Taylor Salman <[email-masked]>
Date: Tue, 24 Jul 2001 11:56:48 -0400

All,

Some clarifications:

The Performance Committee does not conduct simulations for IEEE 
802.17.  Simulations are conducted by individuals who would like to 
contribute to the standard development.  The committee is established to 
define simulation frameworks for different scenarios so that individuals 
running simulations can compare their results "apples to apples".  There 
are members of the group that adamantly would like to simulate the 
performance of TCP/IP over RPR, so we are trying to establish a framework 
for those simulations.  We have not agreed on a framework for TCP/IP 
simulations to date.

While personally I also have reservations about the utility of such 
simulations, the charter of the performance group does not allow us to 
determine what simulations are run by the individual members of 802.17.

Thanks,
Taylor
Reluctant Volunteer for the 802.17 Performance Committee Secretary Position...

>So, should we add this to the framework?

OPNETWORK 2001: Call for papers: Submit your paper for review
by the OPNETWORK Committee.
Visit http://www.opnet.com/opnetwork2001/home.html for submission guidelines.
We're pleased to announce that IEEE Communications Magazine is sponsoring 
the best-paper award at OPNETWORK 2001!

*****************************************************************************
P. Taylor Salman                               OPNET Technologies, Inc.
Senior Manager, Market Research       7255 Woodmont Avenue
Phone:  240.497.3000x2297                  Bethesda, MD 20814-7904
Fax:      240.497.3001                         www.opnet.com
Cell:      202.427.3319                         Pager:  888.912.5350
******************************************************************************

---

From: Siamack Ayandeh <[email-masked]>
Date: Tue, 24 Jul 2001 18:10:11 -0400

> So, should we add this to the framework?

Exactly what do you have in mind.  Do RED, WRED, and their new flavours
qualify for
keeping it simple.  RPR may also implement a MAC which is loss less (due
to congestion)
on the ring.

---

From: "Marcos R. Salvador" <[email-masked]>
Date: Wed, 25 Jul 2001 13:52:21 +0200

Dear all,

I think it is important to understand the effects of RPR access/fairness
mechanisms on higher layers. However, I believe that RPR is more likely to
connect routers rather than hosts. Therefore, I think simulations should
concentrate, at least at this point, on Internet traffic at some aggregation
level rather than on individual TCP/UDP flows.

Having said that, one issue I think deserves investigation is the effects of
RPR's access mode on queuing. Under non-heavy load condition, RPR provides a
random access behaviour. On the other hand, under heavy load condition, RPR
tends to a round robin access behaviour (enforced by the fairness
algorithm), meaning that access delays will increase and queues might
build-up fast. How much burstiness can a node tolerate (in one access
behaviour or another) assuming, for instance, that that node just
transmitted? What are the effects of packet dropping (in the event of, for
instance, queue overflow) on the incoming traffic (due to TCP
retransmissions), with and without RED?

BTW: As an active buffer management mechanism, RED should be considered in
simulations. However, RED is not a function of the MAC nor involves
signaling. Thus, I don't understand why RED or alike should imply a more
complex MAC layer. A congestion management mechanism, ECN could lead to a
more complex MAC layer though (?). But again, I personally believe that, at
this point at least, the MAC layer should not consider such functions.
Simplicity is the key!

Another interesting issue concerns QoS, if QoS is to be addressed in RPR at
all (personally I believe it should not, at least at this point). For
instance, it would be interesting to verify whether EF traffic's deadlines
could be met and in which conditions (the latest EF I-D elaborates on this
matter).

Regards,
Marcos
As a poor Ph.D. student :)

---

From: Siamack Ayandeh <[email-masked]>
Date: Wed, 25 Jul 2001 08:42:17 -0400

Marcos,

The RPR working group is soliciting input on this topic. You may want to forward
your email to their mailing list "[email-masked]".

Regards, Siamack

---

From: "Kastenholz, Frank" <[email-masked]>
Date: Wed, 25 Jul 2001 09:05:46 -0400

>keeping it simple.  RPR may also implement a MAC which is loss less (due

Speaking as technical contributor, not co-chair

All I'm saying is that perhaps two bits be added to the 
framework with regard to performance modelling and
simulations
1. Past experience has shown in the IETF & internet
   community that when there are multiple congestion,
   performance, retry, etc, etc, mechanisms, and one
   of them is TCP, they get into fights and the end
   result can be worse than if there was only one.
2. We believe that simulations and modelling are fine
   things, but internet traffic and technology are
   constantly changing, so the models and simulations
   may not have long term utility (because the environment
   changes and the base assumptions of the simulations are
   no longer valid, not because the simulation was itself
   systemically flawed).
and maybe item 3
3. We encourage the people doing this work to get in
   contact with the end-to-end research community (at
   <mailto:[email-masked]?subject=help>
   to learn the details with respect to ip&tcp

Frank Kastenholz
speaking-as-a-technical-contributor

---

From: Siamack Ayandeh <[email-masked]>
Date: Wed, 25 Jul 2001 09:17:48 -0400

>    result can be worse than if there was only one.

Valid point which has already been repeated a few times in RPR WG meetings.
I assume its inclusion in a frame work document also implies that some work
would be done by iporpr WG to supply summary/review of supporting
documentation.  Any statement should be limited to congestion control & how
best to be TCP friendly.

>    no longer valid, not because the simulation was itself

IMHO, better to leave this point out.

>    to learn the details with respect to ip&tcp

Already said on the .17 mailing list twice. No point repeating it in a FW
document.

---

From: Mike Takefman <[email-masked]>
Date: Wed, 25 Jul 2001 11:14:01 -0400

Speaking as a chair and not an individual,
and deleting as an editor and not a chair :)

> Already said on the .17 mailing list twice. No point repeating it in a FW

An email reflector, even one that is archived and available on
the web does not guarantee that people who missed the email
will go back and notice it at some later date.

I would encourage that you do put this somewhere in the framework
document as an informational item. I am not sure where to put
it in, but I like the idea of calling it out explicitly.

mike

-- 
Michael Takefman              [email-masked]
Manager of Engineering,       Cisco Systems
Chair IEEE 802.17 Stds WG
2000 Innovation Dr, Ottawa, Canada, K2K 3E8
voice: 613-271-3399       fax: 613-271-4867

---

From: Taylor Salman <[email-masked]>
Date: Wed, 25 Jul 2001 13:45:55 -0400

>speaking-as-a-technical-contributor

Hmmm...  that's an interesting line of thinking.  Perhaps we should 
continue along that line of logic.  Assumption:  Internet traffic and 
technology are constantly changing.  Hypothesis:  Base assumptions of 
internet traffic behavior do not have long term utility, so simulations are 
not valid in the long term since these assumptions do not hold in the long 
term.

It would seem to me that an obvious corollary to this hypothesis is that 
any new equipment designed and built on these same traffic assumptions also 
do not have long term utility for the same reasons.  Using this logic, 
perhaps we should cease to design and build new  equipment since there is 
no way they'll ever have long term utility and the time and monetary 
investment in designing and building new equipment is magnitudes larger 
than for a simulation...

The corollary above is obviously flawed and I think Frank would agree (or 
else start searching for a new employer...). Perhaps then we should also 
re-examine the original hypothesis.  It seems to me that the hypothesis has 
several hidden assumptions that are not stated in Frank's original 
e-mail.  First, it assumes that simulations are limited to modeling the 
behavior of existing traffic.  Second, it assumes that simulations need to 
have long term value.  Third, it assumes that simulations are static 
entities incapable of changing.  Fourth, it assumes that  there is no 
information provided by the simulation that is more accurate or useful than 
the experience and knowledge contained in an individual engineer's 
mind.  Let's address these assumptions one by one...

1)  Simulations are limited to modeling the behavior of existing traffic
Clearly false.  In fact, one of the inherent values of simulation is 
testing both measurable and unanticipated behaviors.  The ability to create 
unusual traffic behavior in a simulation to test extreme or boundary 
conditions is one of its greatest strengths.  These are the types of 
simulations that usually find latent flaws in a design.  I suppose it's 
better, however, to deploy the equipment and hope...

2)  Simulations need long term value
Also false.  The value of any one simulation only exists for the period of 
time to evaluate any single design issue.  Once that design decision is 
made, the simulation results only have value in explaining why that 
decision was made.  This doesn't mean that other simulations don't have 
value for evaluating other design decisions or predicting overall network 
behavior, but these are completely different simulations answering 
different questions at different times.  They also only have value in 
aiding the decision-making process...

3)  Simulations are static entities
False.  Again, one of the inherent values of simulation is that they can be 
re-configured quickly to test different scenarios, design implementations, 
traffic behavior, etc.  In fact, these changes can be made much more 
quickly than changes can be implemented to equipment or changes are seen in 
gross internet traffic behavior...

4)  There is no information provided by the simulation that is more 
accurate or useful than the experience and knowledge contained in an 
individual engineer's mind
Well, now we're into dogma.  So I'll say that 11 years experience in using 
simulations to design communications systems and feedback from over 2,500 
organizations with over 8,000 individuals involved in using simulation to 
design communications systems lead me to believe that this is also 
false.  However, my belief system doesn't necessarily preclude the 
existence of a communications "rain man" that has complete understanding of 
communications system behavior under all potential conditions, but without 
the ability to perform normal everyday tasks...

Taylor
High Priest of Simulations ;-)

Come to the networking industry's premier conference,
OPNETWORK 2001 August 27 - August 31
(http://www.opnet.com/opnetwork2001/home.html)
and to OPNET's cutting-edge seminars
(http://www.opnet.com/seminars.html)

---

From: "Khaled Amer" <[email-masked]>
Date: Wed, 25 Jul 2001 17:06:19 -0700

Frank,

I agree with Taylor and I'd like to consider myself an expert on the topic
(President of a company that does architecture analysis and performance
modeling for a living, as well as Chairman of the Performance committee for
the IEEE 802.17 WG).

Different traffic patterns can be applied to any models any time (if they
are designed properly). Changes in traffic patterns don't make models
invalid. The results can change a little based on new traffic patterns.
However, through my long years of experience doing performance modeling,
I've found that changes in traffic characteristics have some effect on the
results, but that most of the results are not changed that drastically,
especially if we're talking about changes in traffic on the same network
(e.g.. the Internet). The results can be more different if we're talking
about a completely different traffic distribution, especially if it's based
on new assumptions in the network setup, or traffic on a big network like
the Internet vs. traffic on my home network.

> individual engineer's mind

Needless to say, since my company does this for a living, we certainly
disagree with this statement and have a lot of input to provide that would
prove our point. Otherwise, we would not have established a company based on
this activity.

For a new design, there can be phenomena which are outside the realm of
experiences that an engineer has encountered. Besides, the interaction of
all the elements in modern complex systems make it impossible to get a
complete grasp of all the effects that need to be considered. When you have
a zillion flows going through a network element from so many different
applications with different characteristics and different QoS requirements
as well as complex scheduling algorithms, flow control mechanisms, and so
on, it becomes way beyond the capabilities of the human brain to accurately
(or even close to accurately) envision all the subtle effects (and sometimes
the major ones too) that can occur. Let alone, coming up with a statistical
distribution for various performance metrics under these complex conditions.
From my long years of personal experience in this, I can enumerate a whole
lot of examples where the architect came up with a great architecture, but
through performance modeling, we were able to optimize the design and show
where there are sub-optimal tradeoffs made, or point out subtle effects that
affect the overall performance.

My 2 cents on the topic anyhow ...
:-)

Khaled Amer
President, AmerNet Inc.
Architecture Analysis and Performance Modeling Specialists
Address:     13711 Solitaire Way, Irvine, CA 92620
Phone:        (949)552-1114                      Fax:     (949)552-1116
e-mail:         [email-masked]
Web:           www.performancemodeling.com

----- Original Message -----
From: "Taylor Salman" <[email-masked]>
To: "Kastenholz, Frank" <[email-masked]>; "Siamack
Ayandeh" <[email-masked]>
Cc: <[email-masked]>; "Albert Herrera" <[email-masked]>
Sent: Wednesday, July 25, 2001 10:45 AM
Subject: Re: possible new item to add to the framework?

> High Priest of Simulations ;-)
****************************************************************************
*
> Cell:      202.427.3319                         Pager:  888.912.5350
****************************************************************************
**