Subject: RE: [MPLS-OPS]: Jitter and MPLS

From: "Naidu, Venkata" <[email-masked]>
Date: Sat, 7 Dec 2002 20:43:01 -0500

Jing & Eric,

-> Although Eric's idea is taken by many people , our experiements shows
-> this is not the truth at least with the situation of our testbed.
-> 
-> In fact we set up a small testbed consisting a Linux based 
-> MPLS engine. In order to focus on difference between forwarding 
-> performance difference between MPLS and IP, we use CBR traffic 
-> with static routing table.
-> 
-> The testbed is consist of  3 hops of MPLS(IP) forwarder 
-> between source and destination. The experiments shows that:
  
  I think your discussion is little deviated. What I understood 
  from Jing's description is that, he didn't apply any form of TE 
  (neither IP-TE nor MPLS-TE) in his network. Jing's measurements 
  are based on straight line topology of 3 nodes, comparing IP and
  MPLS (forwarding) performance only.

  IMHO, in such a small topologies, the amortized performance
  improvements (overall gain of all operations in the worst case)
  may not be significant.

  In such straight line topologies (with out applying any form
  of queuing differentiation or some form of TE in the topology),
  I don't think MPLS out performs IP just because of MPLS 
  fast forwarding nature (may be small amount of improvement).
  Remember that, there is no much difference between IP QoS techniques
  and MPLS QoS techniques (same classes, same queuing methodologies).
  The major difference between IP and MPLS is in TE techniques.

  This discussion reminds me of old draft:
  ftp://ftp.netlab.ohio-state.edu/pub/jain/ietf/draft-bhani-mpls-te-anal-00.
txt
  Where, it is clearly shown that, there is a significant improvement 
  in the throughput of the UDP & TCP flows when MPLS-TE is applied.

  However, can the same performance improvements are achieved using 
  IP-TE methods (at least near to the performance of MPLS-TE) is 
  still an open ended question? 
  
  Some very interesting research has been done to prove the above 
  question. But, "at what cost?", "with how much ease of configuration
  & maintenance?", "with what added benefits?" are still unanswered.

  Finally, I strongly feel that, the performance matrices change a lot
  from small topologies to real-complex topologies. We can't directly
  map the simulation results to real-world topologies. In any case,
  the average/amortized performance measurements & analysis of 
  real-world complex topologies is not an easy task. If there is 
  any such work, I will be very glad to know. 

  Thank You.

--
Venkata.

---

From: Eric Osborne <[email-masked]>
Date: Sun, 8 Dec 2002 23:22:06 -0500

>   MPLS (forwarding) performance only.

Just to make it clear, I'm not talking about TE either.  TE is just a
more granular way of routing that can reduce the likelyhood of
congestion; once you have congestion, QoS kicks in and it doesn't
matter whether it's TE or LDP or whatever (ignoring any
implementations that do per-LSP scheduling for TE; since the once I
know best doesn't do that, I can't speak to it).  

>   fast forwarding nature (may be small amount of improvement).

May be small amount of performance hit.  May be large amount of
{improvement|performance hit}.

>   and MPLS QoS techniques (same classes, same queuing

Agreed - this is what I said to begin with.

>   The major difference between IP and MPLS is in TE techniques.

Well, for this discussion, yes.   

>   in the throughput of the UDP & TCP flows when MPLS-TE is applied.

I agree with this draft, although I don't think it's showing anything
revelationary.  If you put the same amount of traffic along a set of
links that has bandwidth X, and you do the test again with a set of
links of bandwidth Y, and Y >> X, then performance will be better.
This is irrespective of TE or not.  In the picture on p. 3 of the
draft, just make the diverse paths equal-cost and ECMP will probably
work fine.

>   still an open ended question? 

Yes, it is an open-ended question.

>   & maintenance?", "with what added benefits?" are still unanswered.

I agree completely.

eric

---

From: Jing Shen <[email-masked]>
Date: Mon, 09 Dec 2002 20:41:09 +0800

It's main conclusion of our experiments that policy based routing must 
be used with MPLS if e2e QoS is considered carefully. I agree with 
Naidu that experiments with large scale network consisting of commercial
router/switch is needed to find out more details, as theoretical 
analysis of e2e delay relates to same factor set in either MPLS or 
IP network, Jitter relates closely to queueing mechanism, but also 
accumulated processing overhead. 

there has been discussion on difference between MPLS-TE and IP-TE
on some other list, but I don't think IP-TE based only on traffic
metrix,
linear programing and policy database could be easily set up in
ISP networks, the obstacle exists with both traffic metix derivation, 
optimization period and the policy dealing with broken links. 

OMP gives a good start on traffic engineering but its traffic division
policy is too simple for self-similar internet traffic. 

I have another question, is OMP implemented in some commerical products?
if not, why? 

The overprovisioning of network resource makes current network works
smooth
util I come to know that DVE works at Gbps bandwidth requiement.

-- 
Jing Shen

State Key Lab of CAD&CG
ZheJiang University(YuQuan)
HangZhou, Z.J. 310027
P.R.China

Tel: +86-571-87932423
Mobile: (0)13516813753
Email: [email-masked]

**********************************************************************
* The SunShine of life is made up of very little beams which is      *
*  bright all the time                                               *
**********************************************************************

---

From: Eric Osborne <[email-masked]>
Date: Mon, 9 Dec 2002 08:46:38 -0500

On Mon, Dec 09, 2002 at 08:41:09PM +0800, Jing Shen wrote:
> be used with MPLS if e2e QoS is considered carefully.

Why?

eric

>  I agree with

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 09 Dec 2002 11:35:55 -0500

> policy is too simple for self-similar internet traffic. 

I don't see any basis for this assertion.

The IP src/dst hash is used almost universally in ECMP for OSPF, ISIS,
LDP (after payload snooping consistently indicates that payload is
IP), and MPLS/TE implementations (when L3PID indicates IP traffic), as
well as in proprietary link bundling (for example: Avici Composite
Links (tm) and others).  It works very well with Internet traffic.

> I have another question, is OMP implemented in some commerical products?

There have been a few research implementations that have come to my
attention, mostly grad student projects.

Implementing in a commercial router is a much larger undertaking due
to added complexity (forwarding cards are separate from router
processors, more is done in hardware or microcode, 100s or 1000s of
interfaces pose greater scaling challenges, etc) and the software
quality requirements.  Without a customer expressing strong interest
this hasn't happenned.

Curtis

---

From: Jing Shen <[email-masked]>
Date: Tue, 10 Dec 2002 21:58:02 +0800

We consider this with the following assumption:

1. it's nearly impossible for any user to negotiate a e2e
   bandwidth reservation for his/her accessing to some 
   other site. The most possible solution is to set up
   e2e LSPs which service a group of people( this is to 
   the situation of VPN or DiffServ-aware-TE ). Within 
   the LSP, sessions share bandwidth  reserved in a best-effort
   manner.
2. The network is increasing, and MPLS and IP will be the 
   most possible technology used to carry content.
3. The underlying technology is the same (DWDM  or the like),
   and the available physical bandwidth is the same. 
4. Real-time multimedia application will be the main stream
   application. This means most of applications require 
   high bandwidth.

So, our experiments are set up to find out what's the essential
part of MPLS networks if it's implemented on the same hardware 
platform. ( like that of upgrading IOS version). In order to focus
on switch routing itself, we don't invoke techniques like TE in 
the experiments. And, we do the experiments between IP and MPLS with
the same queueing mechanism, and the same packet classification 
mechanism. 

The experiment results shows, no matter how network is loaded,
if e2e bandwidth reservation is done packets transferring LSP show
 smaller IPDV and a little higher e2e delay. As the testbed is very 
small, we take the reason as: MPLS decap/encap smooth packet's entering 
procedure  into queue of outgoing interface.

On the other hand, if no bandwidth reservation is done: if network
is lightly loaded there is nearly no difference in e2e delay between
IP routing and MPLS switching, while MPLS shows bettter in IPDV; 
if network is heavy loaded MPLS behaves worse in both aspects than
IP does. As we know, the way to control jitter is smoothing packet
transfer 
by controling queueing behavior, and we find the overhead of IP routing
and MPLS switching is nearly the same. This means in our testbed MPLS's 
encap/decap introduces something good to jitter control.

We extend the result to a path consisting multi-hop because middle-LSP
could be considered a tunnel connecting LERs, and also it directly 
applies to LSPs consisting LERs. So, if e2e qos for individual
connection
is considered in MPLS network, it should take the LSP which pass a path
as highly loaded as possible.

From the discussion above, I come to know that implementation details
do have effect on e2e performance, and commerical implementation
of MPLS may introduce negelectable overhead in encap/decap overhead.
But it exists and accumulated result will come up when network is heavy
loaded.

And, also delay/IPDV is affected by fluctuation in network, we consider 
we extend the instant we observe. 

Some other people says that we could smooth any problem by
overprovisioning.
But, a phenomenon of current network is: sometimes when a part of
network
is heavy loaded the adjacent part idles. Why shoudn't we make use of 
those idled resource ? 

regards

-- 
Jing Shen

State Key Lab of CAD&CG

Tel: +86-571-87932423
Mobile: (0)13516813753

---

From: Jing Shen <[email-masked]>
Date: Tue, 10 Dec 2002 22:09:55 +0800

Curtis,

> Links (tm) and others).  It works very well with Internet traffic.

First of all, I take OMP is very good start in trying to make use of
idled network resource. And it shows good performance in simulation and
the academic testbed you mentioned. 

We take multipath routing is the way to make full use of network
resource
and traffic dispersion should be done according to the state of network
but
not the the src/dst hash function. The reason is, network load is not
symetricly
distributed and LRD is the built-in properties. So, even there exists 
multiple equal cost path between any src-dst pair hash based traffic
division
will not lead to a balanced network usage as well as guaranteeing e2e
qos.

> quality requirements.  Without a customer expressing strong interest

Thanks for the verification. I think this is for the same reason that
the network is usually overprovisioned, so people take it enough.

if there is something wrong correct me please.

-- 
Jing Shen

State Key Lab of CAD&CG

---

From: Eric Osborne <[email-masked]>
Date: Tue, 10 Dec 2002 10:41:56 -0500

> procedure  into queue of outgoing interface.

ok.  good.  great.  fine.  but you're testing a specific
implementation.  it may be yours, it may be mine, it may be someone
else's.  but any ipdv and other such performance characteristics are
not that of the mpls architecture, but of the implementation you're
testing.  I'm not sure if you agree with me or not, can you please
confirm that you do or don't?

> encap/decap introduces something good to jitter control.

MPLS encap/decap changing delay/jitter characteristics for better or
worse is a byproduct of the implementation.

> as highly loaded as possible.

I don't get this.  If you want qos, surely you want to strive for
low-load paths?

> But it exists and accumulated result will come up when network is heavy

Yes, it exists, but it _depends on the implementation_.  

> is heavy loaded the adjacent part idles. Why shoudn't we make use of 

I agree.

What does any of this have to do with policy routing, as you state
below?  Why do you give the impression that policy routing is only
necessary for MPLS networks, and not IP?  What, for that matter, is
policy routing?  I know what I think it is, but that's because we
happen to implement a feature called "policy-based routing".  I have
no idea whether this is what you refer to or not.  Also, if you
continue to insist on confusing architecture with implementation, you
may want to remember that a policy routing implementation will
probably have adverse effects on delay/jitter, as there's generally
more work involved in policy-routing a packet than in normal routing
or switching or whatever of the same packet.

eric

---

From: Jing Shen <[email-masked]>
Date: Wed, 11 Dec 2002 11:23:02 +0800

> confirm that you do or don't?

I agree with you that MPLS architecture does not have aparent difference
from IP when we are talking about forwarding performance. What I care
about
is , how could that affect e2e performance when we measure UDP/TCP. 
We consider such a protocol stack:
  [application protocol/TCP(UDP)/IP/MPLS], and we consider MPLS as a
protocol
adding values to IP networks. So, from the viewpoint of protocol
processing
there may be some different mechanism and implementation. I agree that 
different implementation has different effect on e2e performance. But,
the 
protocol processing  at LER/LSR should have the same logical procedure
which 
is a little different from pure IP networks.

What I care about is how may those changes affect the performance. From
your
word I come to see that our experimental result may only stick to our
implementation
and our experimental policy, but I'm still not clear that how could a
ASIC implemention
of MPLS engine compares to a ASIC implementation IP engine? Is there any
measurement data or paper on this?

> I don't get this.  If you want qos, surely you want to strive for

I mean, LSP should be set up along a path which is as low loaded as
possible.
And, also sessions for src-dst pair should chose the least loaded LSP if
there
are multiple LSPs  available. If the LSP is not set up, it should be
computed
to be along the least loaded path.

> >Why shoudn't we make use of those idled resource ?
> 
> I agree.
> 

I agree that both IP and MPLS need policy based routing ( constraint
based
routing), and to our implementation and experiments LER-LER switching
path is more
sensitive to load state. This may be only to our implementation, the
reason
I take it is I find that is changes with protocol stacks which is
definitely 
affect the implementation. I'm not so clear about how those commerical 
implementation different from academic implementation at the fundamental
theoretical level, so is there anyone could do me a favor to give some
explanation or
reference?

Thanks a lot! 

regards

-- 
Jing Shen

State Key Lab of CAD&CG

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 11 Dec 2002 14:23:40 -0500

> measurement data or paper on this?

Probably some work has been done but it doesn't make a whole lot of
difference.  The best modern ASICs forward at full line rate for
either MPLS or IP at OC192c rates of roughly 10Gb/s.  So figure a few
10s of nsec per table or route lookup.  This doesn't amount to a whole
lot of time compared to any normal speed of light delay for a real
network hop and even exceeds the packet serialization time.

For UDP or TCP, the difference is 4 bytes added to the packet for
MPLS.  For 1500 byte TCP MSS that's under 0.3%.  For 552 its about
0.7%.  For a 40 byte packet its 10%.  If you want traffic engineering
it certainly doen't have the overhead of ATM (9-62% overhead).

The network is usually not full so the <1% has negligible effect.  TCP
or UDP performance is typically dominated by window size and only
mildly impacted by transient queueing delays until the network fills,
then determined by drop characteristics after the network fills.

That's why this issue hasn't been a hot topic.

Curtis