Subject: RE: Some queries

From: "Bagasrawala, Abbas" <[email-masked]>
Date: Fri, 5 Jan 2001 09:36:28 -0500

Yi,
 I will attempt to answer this question and the experts can correct me if I
am wrong.

For IP over ATM (overlay networks) the ATM VC's are just a physical link (as
you noted) and thus each router at the end of the VC's need to be aware of
the adjacent router. In effect the ATM core is transparent and thus you have
1000 routers connected to each other. This requires that adjacencies at each
node is 999 (N-1). 
With densely meshed networks the number of adjacencies are less than of
fully meshed network.

Abbas Bagasra

-----Original Message-----
From: Yi Chu [mailto:[email-masked]]
Sent: Friday, January 05, 2001 9:22 AM
To: [email-masked]; Cheng-Yin Lee
Cc: Faisal S. Naik; mpls uunet; [email-masked];
[email-masked]
Subject: RE: Some queries 

Please forgive me if my question is so obvious and maybe stupid, since I am
new to this discussion.

Why 1000 nodes can have 10-15 adjacencies while IP over ATM needs 999
adjacencies per node?  I think IP over ATM uses ATM VC just as a physical
link, so there is no difference in adjacencies per node whether the routers
are connected through Sonet links or ATM VCs.

So what is the reason that IP over ATM requires full VC mesh of all nodes?

Yi Chu

-----Original Message-----
From:	[email-masked] [mailto:[email-masked]] On Behalf Of Curtis
Villamizar
Sent:	Thursday, January 04, 2001 10:27 PM
To:	Cheng-Yin Lee
Cc:	[email-masked]; Faisal S. Naik; mpls uunet; [email-masked];
[email-masked]
Subject:	Re: Some queries

In message <[email-masked]>, "Cheng-Yin Lee" writes:
> Curtis,
>
> Curtis Villamizar wrote:
>
> > There were a number of motivations for getting rid of ATM in IP
> > networks.  One was the cell tax which at a typical 20% would not alone
> > kill ATM.  Another was SAR.  You can't get fast ATM router interfaces
> > because SAR speed and SAR buffering becomes a problem.  This alone was
> > enough to kill ATM.
> >
> > Perhaps even larger was the problems of independent control planes and
> > the effect on IGP scaling.  Flooding is needed for reliability in the
> > IGP and flooding in a full mesh of N routers has some N^3 properties
> > with regard to messages sent when a router in the mesh fails and N^2
> > when a VC fails.  Attempts to constrain the flooding tended to slow
> > convergence and sometimes cause long lived IGP inconsistency when
> > change occurred.  This is not a problem is the flooding follows the
> > physical interconnections as it does with MPLS since the number of
> > adjacencies per node drops dramatically.
>
> Over time , would there be any inclinations to add direct connectivity
betwee
> n B
> and H, and then over F and D, and so on, leading to perhaps densely meshed
> networks in the long run ? What is your opinion on this?
>
>                           B-------C-------D
>                         /                                          \
>                       /                                               \
>                  A                                                    E
>                     \                                                /
>                       \                                            /
>                        F--------G--------H
>
> One could use a common control plane and may still run into IGP scaling
issue
> s
> because the network is densely meshed, right?

Densely meshed in a 1,000 node network might mean 10-15 adjacencies
per node.  That is still a lot less than full mesh of 999 adjacencies
per node if IP over ATM were used.

> > The IGP scaling problem was address by ISPs by partitioning their
> > network into core and regions but this reduced the effectivenes of TE.
>
> Could you kindly elaborate your thoughts on this?

Typically a provider would partition the topology into N regions where
N is 10s, then run VCs within an ATM core that interconnected the
regions and VCs within each region.  The routers had no visibility
into TE state so if there was mutliple routers bordering the core and
a given region and one of those core routers had a much less congested
path to a destination (recognizing that congestion was more likely
within specific regions than with the core) then the core routers had
no good way to pick the right edge router and get on the VC with a
cleaner shot at the destination.  [Its late.  I hope this is clear.]

> thanks,
> cyl

Curtis

---

From: "Yi Chu" <[email-masked]>
Date: Fri, 5 Jan 2001 09:54:16 -0500

Abbas Bagasra:
Thanks for explaining. But sorry I still do not get the point.  If I have
routers A,B and C, connected through an ATM core.  If I set up VCs to
connect A and B, B and C only.  Is it a legitimate configuration?  I mean it
is not necessary to have A adjacent to C.  Packet from A to C can still be
routed through B.  Admitted that those packets would traverse in and out
the ATM interface on B.  But is there any architectural reason why the above
configuration is prohibited?

yi

-----Original Message-----
From:	[email-masked] [mailto:[email-masked]] On Behalf Of Bagasrawala,
Abbas
Sent:	Friday, January 05, 2001 9:36 AM
To:	'Yi Chu'; [email-masked]; Cheng-Yin Lee
Cc:	Faisal S. Naik; mpls uunet; [email-masked]; [email-masked]
Subject:	RE: Some queries

Yi,

Abbas Bagasra

-----Original Message-----
From: Yi Chu [mailto:[email-masked]]
Sent: Friday, January 05, 2001 9:22 AM
To: [email-masked]; Cheng-Yin Lee
Cc: Faisal S. Naik; mpls uunet; [email-masked];

Yi Chu

-----Original Message-----

betwee
issue

Curtis

---

From: Juha Heinanen <[email-masked]>
Date: Fri, 5 Jan 2001 17:10:58 +0200 (EET)

> configuration is prohibited?

no, the configuration is not prohibited and in fact some routers allow
all three interface still to use an ip address from the same ip subnet.

so you could start with default connectivity between your routers and
then start adding more connections dynamically if there is enough
traffic between a pair. i documented this once in an internet draft that
has now expired.

-- juha

---

From: "Yi Chu" <[email-masked]>
Date: Fri, 5 Jan 2001 11:37:31 -0500

So based on the reply from Juha and Abbas, there is no need to have 999
adjacencies for IP over ATM either.  A densely meshed network would work for
ATM solution as well (10 to 15 adjacencies per router for a 1000 router
network).

I understand the need to improve the overlay model, and thus the need for
MPLS.  But I just can not stand false statements.

Yi

-----Original Message-----
From:	[email-masked] [mailto:[email-masked]] On Behalf Of Juha
Heinanen
Sent:	Friday, January 05, 2001 10:11 AM
To:	Yi Chu
Cc:	Bagasrawala, Abbas; [email-masked]; Cheng-Yin Lee; Faisal S. Naik;
mpls uunet; [email-masked]; [email-masked]
Subject:	RE: Some queries

* juha

Subject: RE: Some queries
From: "Bagasrawala, Abbas" <[email-masked]>
To: "'Yi Chu'" <[email-masked]>
Cc: "mpls uunet" <[email-masked]>
Date: Fri, 5 Jan 2001 10:09:12 -0500

Yi,
  There's no reason why you cannot configure as you suggested but then B
becomes the bottleneck. The use of mpls in atm core helps in scaling the
network by avoiding the full meshed vcc's to each of the nodes (routers).

Abbas

-----Original Message-----
From: Yi Chu [mailto:[email-masked]]
Sent: Friday, January 05, 2001 9:54 AM
To: Bagasrawala, Abbas; [email-masked]; Cheng-Yin Lee
Cc: Faisal S. Naik; mpls uunet; [email-masked];

Abbas Bagasra:

yi

-----Original Message-----
From:	[email-masked] [mailto:[email-masked]] On Behalf Of
Bagasrawala,
Abbas

Yi,

Abbas Bagasra

-----Original Message-----
From: Yi Chu [mailto:[email-masked]]
Sent: Friday, January 05, 2001 9:22 AM
To: [email-masked]; Cheng-Yin Lee
Cc: Faisal S. Naik; mpls uunet; [email-masked];

Yi Chu

-----Original Message-----

betwee
issue

Curtis

---

From: "Cheng-Yin Lee" <[email-masked]>
Date: Fri, 05 Jan 2001 12:05:33 -0500

Yi,
The related point that I was trying to get clarification from Curtis was that
it is not so much independent control planes that result in IGP scaling issues
(wrt the quote below from Curtis), but rather how fully-meshed one's network
is.  But I think this point is clearer now.

thanks,
cyl

> the effect on IGP scaling. ....
>
Yi Chu wrote:

>   ------------------------------------------------------------------------
>
>    Part 1.2    Type: message/rfc822
>            Encoding: 7bit

---

From: Vijay Gill <[email-masked]>
Date: Fri, 5 Jan 2001 15:43:18 -0500

> MPLS.  But I just can not stand false statements.

People, it doesn't hurt to trim headers.

There is nothing that precludes running ATM over a point to point links
like SONET, so that your adjacency count grows linearly as the number of
links in your network. That approach adds no major benefit over PoS and
increases your cost/bit of ip traffic transported over your network (cell
tax).  This is a reasonable approach if you are buying ATM pvc's across
some other providers cloud to get cheaper pipes.

This approach is perfectly analogous to just buying point to point
circuits and gives up a very important feature that some ISPs currently
use ATM for - the ability to get constrained based routing, so as to do
effective automatic traffic engineering.

To allow for automatic traffic engineering, some organizations terminate
their SONET handoffs into ATM switches and terminate routers into the ATM
switches as well. The ATM switches then talk amongst each other (various
levels of hierarchy go in here), and export a virtualized transport fabric
to the routers. The routers are then fully meshed with subinterfaces and
sPVCs to all the other routers in that particular area, such that no
router _transits_ another router. All routers appear to be directly
connected to each other, so that the IP traffic always takes what appears
to be the shortest path to any other router (a direct connection).

In general, hop by hop traffic forwarding can only super-aggregate
traffic, thus hiding edge-edge traffic flow information. Using a full mesh
of ATM pvcs such that there is no
edge-router----intermediate-router-----edge-router transit allows
operators to disambiguate the traffic flow on an edge-edge basis.

This, along with some hierarchy, allows for precise traffic measurement
between router-router flows and at the city-pair level, and results in
fairly tractable flow data that can be used to predict traffic levels,
allowing for proper sizing of sPVC contracts, etc.  These contracts are
then used by the ATM switch software to do constrained based routing of
sPVCs, allowing the network operators to do a good job of traffic
engineering and circuit layout/capacity planning.

This might explain why people have gone with the extra expense (both in
capex and SGnA) and complexity to implement a fully meshed ATM solution.

/vijay

---

From: "Manohar Ellanti" <[email-masked]>
Date: Sat, 6 Jan 2001 21:33:31 -0800

----- Original Message -----
From: "Vijay Gill" <[email-masked]>
> to be the shortest path to any other router (a direct connection).

Do you mean to say SPs such as large/national ISPs use ATMs on the backbone
and have complete mesh with DS3/OC-3/OC-12 links and then allow routers etc
connected to these ATMs switches  to have mesh connectivity as well between
all POPs using sub-interfaces or different physical interfaces. I would
think they use core routers to build the backbone rather than ATM switches.

Consider this;

POP1---------------------------DS3/OC-3/OC-12/OC-48-------------POP2

R[1..m]--Interface[1..n]--ATM----------------------------------ATM---Interfa
ce[1..j]--Router[1..k]
                   \
/
                    \
/
                      \
/
                         \
                         /
                            \
/
                               \
/
                                                  POP 3
                                                  ATM
                                          R[1..M]---Interface[1..N]

In this case R-to-R logical  mesh can be accomplished using PVCs (on per
router or per interface basis) with traffic engineering occurring in the
overlay IP network as well as in the ATM network.

-Manohar N Ellanti

---

From: "dave o'leary" <[email-masked]>
Date: Sat, 06 Jan 2001 10:34:06 -0800

>think they use core routers to build the backbone rather than ATM switches.

ATM in the some of the larger backbones was common throughout the mid-late 
90's,
and still remains in some of the networks - but due to some of the reasons 
discussed
on this thread, many are planning to migrate to routers/LSRs interconnected 
with
SONET/SDH links and no ATM switches (or encapsulation), using MPLS to
more or less replicate the traffic engineering capabilities of ATM between 
routers -
I believe this is where this thread started, right?

                                                 dave

---

From: Randy Bush <[email-masked]>
Date: Sat, 06 Jan 2001 12:06:35 -0800

> and have complete mesh with DS3/OC-3/OC-12 links 

no, complete mesh of vcs over semi-meshed physical circuits

randy

---

From: Alan Hannan <[email-masked]>
Date: Sat, 6 Jan 2001 13:02:10 -0800

It is useful to point out that within a domain
  such as the backbone, it is very common to have a partial
  mesh at L1 (circuit), and a full mesh at L2 (ATM PVC/sPVC or MPLS
  LSP) and L3 (IP Routing and/or forwarding adjacency)..

  Note however that all routers in the network/AS may not participate
  in a full mesh at L2/L3 (much les L1) due to hierarchy.  For example,
  Customer aggregations routers, even if they perform MPLS, are 
  unlikely to have a full mesh w/ all other routers in the AS.

  Additionally, it is not uncommon to find large/national/global 
  networks will have hierarchical domains, say, of Global Level, 
  National Level, Regional Level, and Metro Levels.  Within each 
  of these domains one can expect partial mesh at L1 and full mesh 
  at L2/L3.

  In such a network, you might find 40 routers in 20 pops on the 
  top-most backbone level, w/ each router having an average degree of 
  freedom of say 2.0 and each POP of 4.0.  Therefore you would expect 
  and observe about 80 physical backbone links (DOF*N/2) and about 
  39 L2/L3 adjacencies  (M-1).  Note that there should be (M*(M-1)) or 
  about 1560 total forwarding paths in the backbone domain alone.

  -alan

Thus spake Randy Bush ([email-masked])

---

From: "Manohar Ellanti" <[email-masked]>
Date: Sun, 7 Jan 2001 04:24:54 -0800

----- Original Message -----
From: "dave o'leary" <[email-masked]>
> I believe this is where this thread started, right?
Agreed. It is also possible to have much simpler and transparent solution to
provide point-to-point Service Provider (SP)  router connectivity while not
mandating that MPLS be used at the edge. We could use the
transport/emulation services such DS3/MPLS being proposed or some optimized
and data format aware payload point-to-point transport over MPLS. Network
Provider (NP)  traffic engineering and SP traffic engineering may use same
tools such as MPLS or some proprietary but  by keeping overlay model there
is a natural  separation which is simple and easy to operate , especially
for NP.

Considering that generally POP to POP physical topology is fairly static
with lead-planning times of anywhere from few months to an year or longer
before a topological change is effected, there is not so much of dynamic
nature to the underlying topology. It should therefore be possible to keep
provisioning and operational complexity as simple as possible , even that
means provisioning point-to-point circuits (LSP) using off-line TE tools and
installing the path in the NP network while SP could use dynamic TE
mechanisms. I am beginning to think that overlay may be in place despite all
the nice things being said of peer model.

It is not only avoidance of overhead with  ATM cell header ( with NP TE and
SP TE there may be stack of 32-bit headers as well) but also avoidance of
two control plane technologies:one at the ATM level and one at the IP level.
With MPLS, if used by NP we have single technology for control plane however
one should not forget that these are still two instance:  one instance by NP
another instance by SP.

As I see it , we do get benefits 1) substantial reduction in overhead (with
MPLS we do have overhead) 2) continued existence of two instance of control
plane.

-Manohar N Ellanti

---

From: "Cheng-Yin Lee" <[email-masked]>
Date: Tue, 09 Jan 2001 16:53:09 -0500

Curtis,

Do you agree that :
1) the scaling issues in IGPs, are a result of fully-meshing routers, not a
result of using independent control planes
2)  service providers who have requirements to fully-mesh routers  (for simple
edge to edge traffic measurement, bandwidth management, control, as described by
Vijay)  have to setup fully-meshed circuits,  VCs or LSPs. The needs of  service
providers to fully-mesh routers do not arise because of ATM (or MPLS).

thanks,
cyl

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 09 Jan 2001 17:40:49 -0500

> result of using independent control planes

Yes, but independent control planes was the reason dense meshing in
the IGP was needed.  The IGP had to run over the VCs.

You can have a full mesh of MPLS LSPs and a sparse mesh of IGP
adjacencies.  The same is not true for ATM.

> providers to fully-mesh routers do not arise because of ATM (or MPLS).

You don't seem to understand that with MPLS the mesh of MPLS LSPs is
decoupled from the IGP adjacencies.

With MPLS, the IGP does not run over the MPLS LSPs.  The IGP runs over
the physical links.  The IGP and TE mesh are decoupled.  You can have
BOTH a full mesh of LSPs AND a scalable IGP.

With ATM, the IGP runs over each VC.  The IGP and TE mesh are coupled.
You can have a full mesh of VCs OR a scalable IGP, but NOT BOTH.

> cyl

I'm repeating myself above in hopes that you'll finally see the point
that we've been trying to make.

Curtis

---

From: Juha Heinanen <[email-masked]>
Date: Wed, 10 Jan 2001 14:11:33 +0200 (EET)

> adjacencies.  The same is not true for ATM.

not true, see

ftp://lohi.eng.telia.fi/tmp/draft-ietf-ion-intra-area-unicast-01.txt

-- juha