Subject: [Idr] draft on virtual aggregation

From: Paul Francis <[email-masked]>
Date: Mon, 7 Jul 2008 12:52:53 -0400

Gang,

At the following URL is a draft on virtual aggregation that I'm posting to
IETF (it'll show up in a day or two), and which I'll present at IDR in
Dublin.  

 http://www.cs.cornell.edu/people/francis/draft-francis-idr-intra-va-00.txt

Title and abstract are below.  I hope to create a work item on this in IDR.
I would characterize this as falling under the general charter of scaling
BGP.

Any comments and discussion on this prior to Dublin is of course greatly
appreciated.

PF

Title:  Intra-Domain Virtual Aggregation

   Virtual Aggregation (VA) is a technique for shrinking the DFZ FIB
   size in routers (both IPv4 and IPv6).  This allows ISPs to extend the
   lifetime of existing routers, and allows router vendors to build FIBs
   with much less concern about the growth of the DFZ routing table.  VA
   does not shrink the size of the RIB.  VA may be deployed autonomously
   by an ISP (cooperation between ISPs is not required).  While VA can
   be deployed without changes to existing routers, doing so requires
   significant new management tasks.  This document describes changes to
   routers and BGP that greatly simplify the operation of VA.

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 09 Jul 2008 10:59:41 -0400

>    routers and BGP that greatly simplify the operation of VA.

Paul,

Is there a need?

  Are we still trying to do the equivalent of keeping an AGS+ with
  DFRT alive somewhere?  RAM is cheap and dense.  To get to on-chip
  RAM would require orders of magnitude reductions in DFRT size.

  Other techniques exist for dramatically reducing core router FIB
  size if that becomes a goal for a provider.

  For example, MPLS (or GRE) tunneling through a BGP free core reduces
  FIB size to about the size of the IGP (should easily fit in on-chip
  memory).  It requires no protocol change.  Only down side is no ICMP
  when tunnel faults in middle prior to the ingress knowing about it
  (usually the case anyway due to VPN and VRF) and no fallback to IP
  when ingress knows that the tunnel is down and hasn't yet rerouted.

Is the solution worse than the problem?

  This seems too operationally problematic.

Curtis

---

From: Paul Francis <[email-masked]>
Date: Thu, 10 Jul 2008 10:27:37 -0400

It is true that there are a number of routing problems that ultimately need
to be solved...RIB and FIB size, convergence time, security, multi-homing.
RRG is working on a single grand solution to all of this, but I'm not holding
my breath.

In the meantime, however, FIB size alone is an immediate problem for a lot of
ISPs, because it is specifically this that forces them to upgrade hardware
when they otherwise don't need to do it.  This is evidenced by the various
hacks that ISPs currently employ to extend the life of their routers.  One is
the "disconnected backbone" arrangement that Dan Ginsburg mentioned in his
email.  Another is to simply ignore /24's that you don't have enough space
for (see for instance
http://mailman.apnic.net/mailing-lists/pacnog/archive/2004/12/msg00000.html).
I'm even familiar with a Tier1 ISP (I'm not at liberty to say who) that is
actually using a hack that messes up the AS-path, and could create loops if
another ISP did the same thing).  

So this is already a problem that needs fixing.  Worse, as IPv4 addresses run
out, there is real concern that addresses will become less aggregatable, and
I'd like to have something in place should that happen.  VA represents about
the simplest architecturally sound solution I can imagine.

I guess you are suggesting that, since RAM is "cheap and dense", FIB really
isn't the problem?  I personally don't know router architecture very well,
but if what you say is true, why don't router vendors simply build routers
with more FIB?  I suppose you could argue that they try to, but that they
under-estimate DFRT growth?  This doesn't sound likely.  Tony Li has argued
(see RFC4984) that because router memories are built in volume, Moore's law
doesn't really apply.  This all suggests to me that there is a real cost to
huge FIBs. 

As for operational complexity, I don't think that you or I know if it is
"too" much or not.  Really it is a question of the value ISPs get out of this
versus the difficulty of doing it.  I'm not going to pretend that running VA
is trivial, but nor does it strike me as all that bad.  It strikes me for
instance as simpler than route reflectors (though that ain't saying much!).
And given that ISPs already deal with config complexity in the hacks I
mention above, there is a good chance in my mind that VA will be an
acceptable solution.

Regarding some specific points in your email:

Regarding core-router MPLS fix:  To be clear, VA isn't about reducing *core*
router FIB size, it is about reducing FIB size in *any* routers, all of them
if that's what you want to do.  Since most routers aren't core routers,
solutions that only address core routers are only so useful.  I've been told
by folks at ISPs that the greater concern are edge routers.

Regarding on-chip FIBs:  I've been told by Tony Li that he believes he can
fit 200K FIB entries on-chip.  VA can get you 5x reduction very easily, and
with some deployment creativity (which would take time and experience to
development) I'll be we could do much much better better.  But that is
somewhat besides the point...the goal here is not to get to single-chip
routers (though that might be where this leads us anyway), but to allow ISPs
to extend the lifetime of their routers, which is something that at least
some of them clearly want to be able to do (see below).

Thanks,

PF

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, July 09, 2008 11:00 AM
> To: Paul Francis
> Cc: [email-masked]
> Subject: Re: [Idr] draft on virtual aggregation
> 
> 
> In message
> <[email-masked]>
> Paul Francis writes:
> >
> >
> > Gang,
> >

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Jul 2008 00:01:15 -0700

Hi Paul,

 > In the meantime, however, FIB size alone is an immediate problem for a
 > lot of ISPs, because it is specifically this that forces them to
 > upgrade hardware when they otherwise don't need to do it.

As Curtis pointed out deployment of any form of tunneling in the core 
MPLS or IP ultimately addresses the FIB scaling of that part of the 
networks.

True that some networks do not have the core ... the network can be 
meshed edges or more specifically meshed POPs.

A very simple observation can be made that you can use a tunnel from the 
edge to one router (per POP for example) do IP lookup then encapsulate 
to the exit point. In that scenario your edge routers are free from 
carrying full table and due to shift in the place where single IP lookup 
is done and switching decision determined.

It is clearly not true that vendors today have any issues in delivering 
boxes which could keep today's Internet table and at least allow for 
5-10 time it's grow. I know at least two of them which have been 
shipping such routers for few years now.

And such architecture does not require dividing address space in any 
chunks and can be deployed today on any exiting hardware without waiting 
for any new protocol extensions.

Cheers,
R.

---

From: Paul Francis <[email-masked]>
Date: Fri, 11 Jul 2008 06:39:57 -0400

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Friday, July 11, 2008 3:01 AM
> To: Paul Francis
> Cc: [email-masked]; [email-masked]

Yes, but only that part of the network.  As I said, from the ISPs I've talked
to, the issue is more in the core than the edge.

> is done and switching decision determined.

This would overload that one router, as well as create a point of failure.
So you'd need to increase the capacity of that router, plus replicate it for
failure.  So in essence you are designing a core architecture.  The point
behind VA is that you don't force ISPs to change their architecture and
upgrade their routers to deal with table growth.  Plus anyway you can't do
this for your edge routers that peer with ISPs that require the full routing
table, so this fix is quite limited.

> shipping such routers for few years now.

Yes I understand that there are routers that can hold 5x or 10x the current
table.  Your company makes them.  How long will these routers last, given the
"end-game" of IPv4 address allocation?  5 years ago the new generation of
routers looked like they could handle a lot of table growth, but now they
have run out of space and yet ISPs want to keep them.  Basically you are
telling them that the solution is simple---buy your latest product.  And to
not worry about growth because you'll have yet another product for them a few
years down the road.  This is exactly what this draft is trying to avoid.

> for any new protocol extensions.

I'm not sure what you mean by "such architecture".  But many existing routers
in the field today cannot realistically use the kind of trick you mention
above to manage FIB size.  Rather, they resort to simply dropping some
fraction of routes, for instance.

To be clear, we are talking about one new attribute, zero changes to the data
plane, zero changes to the existing BGP decision process....just some rules
for automatically setting up tunnels and new address aggregates (virtual
prefixes).  Better to do this now well before the next generation of routers
runs out of FIB.

PF

> > http://mailman.apnic.net/mailing-
> lists/pacnog/archive/2004/12/msg00000.html).
> >>>  http://www.cs.cornell.edu/people/francis/draft-francis-idr-intra-
> va-
> >> 00.txt
> >>>    size in routers (both IPv4 and IPv6).  This allows ISPs to
> extend
> >> the
> >>>    with much less concern about the growth of the DFZ routing
> table.
> >> VA
> >>>    significant new management tasks.  This document describes
> changes
> >> to

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Jul 2008 04:29:21 -0700

Hi Paul,

 > So you'd need to increase the capacity of that router, plus replicate
 > it for failure.

It needs to be noticed that this one router can be any of today's edge 
router as if current edge handles full Internet table in FIB. The gain 
is that all other edge routers do not any more keep state in FIB.

Reg single point of failure ... of course not. Two such routers (per POP 
or in small AS per network) would anycast common next hop and we would 
relay on IGP for failure detection and even for data packets in flight 
redirection.

> So in essence you are designing a core architecture.  The point

Disagree 100%. VA is what I described + dividing the address space into 
chunks.

> table, so this fix is quite limited.

Why not ? Remember we are dealing with FIB only. (My idea works with RIB 
as well using multihop ebgp session, but let's focus on FIB). Why ASBRs 
peering with ISPs would require to store all the routes in FIB ? 
Core/POP full nodes would attract all traffic make switching decision 
then encapsulate and ship. It is very much the same as Lixia's APT model 
except no mapping plane is needed :).

> have run out of space and yet ISPs want to keep them.

If one have bought an ISP class router 5 years ago there is no reason to 
upgrade now capacity wise. 5 years ago internet table were about 130K 
today is 270K src: http://www.cidr-report.org/

Those routers 5 years ago were already design and ready to handle 1-2M 
routes and not due to expectation of Internet growth ... due to 
requirement for L3VPN deployments.

> Basically you are

If we compare Internet to Highways your draft still is trying to keep 
old Trabants going ... moreover bunch of them. It seems that they may be 
at some point became an obstacle on the road and it may be much better 
to find other uses for them then carrying passengers.

What I proposed is to keep all of those still running on the edge while 
based the

> fraction of routes, for instance.

What would be a single reason why at least pair of routers in any ISP 
could not carry full Internet table today in FIB ? I know quite a few 
ISP & SP networks worldwide and can't find any such example. Rest of the 
edge routers could function just fine without even slight need to drop 
anything.

My proposal is a special case of yours ... So saying that routers in the 
field can not support this means that your proposal suffers from the same.

The overall idea is good it is just I want to make sure that in whatever 
form this work goes forward ISP MUST be allowed to have single VA as well.

And that would not require any new protocol extension as BGP could 
inject such default route today with next hop being APR.

That means that the extended community attribute you are proposing 
should be optional as well.

> prefixes).  Better to do this now well before the next generation of routers

My concern is not about adding new attribute. The concern is about the 
additional deployment, operational and troubleshooting difficulties it 
will introduce.

Having forwarding boxes partitioned with different information (which 
will have to be statically managed by hand) is very error prone. It also 
have other consequences including the need redesign on behavior of fast 
reroute technics or HA features. It is not the steady state of operation 
where things go out of control.

Cheers,
R.

---

From: Paul Francis <[email-masked]>
Date: Fri, 11 Jul 2008 07:38:15 -0400

I mis-typed...I meant, the issue is more in the edge than the core...

PF

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Francis
> Sent: Friday, July 11, 2008 6:40 AM
> To: [email-masked]
> Cc: [email-masked]
> >  > In the meantime, however, FIB size alone is an immediate problem
> for
> > a
> it for
> > > RRG is working on a single grand solution to all of this, but I'm
> not
> > holding
> > > my breath.
> > >
> > > In the meantime, however, FIB size alone is an immediate problem
> for
> > a lot of
> > > hacks that ISPs currently employ to extend the life of their
> routers.
> > One is
> > > email.  Another is to simply ignore /24's that you don't have
> enough
> > space
> > > for (see for instance
> > > I guess you are suggesting that, since RAM is "cheap and dense",
> FIB
> > really
> > > isn't the problem?  I personally don't know router architecture
> very
> > well,
> > > with more FIB?  I suppose you could argue that they try to, but
> that
> > they
> > > under-estimate DFRT growth?  This doesn't sound likely.  Tony Li
> has
> > argued
> > > As for operational complexity, I don't think that you or I know if
> it
> > is
> > > "too" much or not.  Really it is a question of the value ISPs get
> out
> > of this
> > > is trivial, but nor does it strike me as all that bad.  It strikes
> me
> > for
> > > And given that ISPs already deal with config complexity in the
> hacks
> > I
> > > Regarding core-router MPLS fix:  To be clear, VA isn't about
> reducing
> > *core*
> > > router FIB size, it is about reducing FIB size in *any* routers,
> all
> > of them
> > > with some deployment creativity (which would take time and
> experience
> > to
> > >>> IETF (it'll show up in a day or two), and which I'll present at
> IDR
> > >> in
> > >>> Dublin.
> > >>>
> > >>> Title and abstract are below.  I hope to create a work item on
> this
> > >> in IDR.
> > >>>    lifetime of existing routers, and allows router vendors to
> build
> > >> FIBs
> > >>>    by an ISP (cooperation between ISPs is not required).  While
> VA
> > >> can

---

From: Paul Francis <[email-masked]>
Date: Fri, 11 Jul 2008 08:32:22 -0400

I see...so I think you are saying:

o   let's allow for a special case of VA where there is a single virtual
prefix which is a /0 (the default route),

o   that this works perfectly well for ISPs who design their POPs such that
the default routes feed into routers that contain the full FIB (using
existing anycast-like failover techniques), 

o   that if you run it this way, then the question of what goes in the FIB is
simplified (i.e. anything or nothing for edge routers, everything for "full"
routers), and therefore you no longer have to tag the default route with an
attribute calling it a VP.

o   edge routers can still keep the full RIB in order to peer with external
peers, or we use the multi-hop BGP trick to get the "full" router to peer
with external peers (FAIW, we mention this in the tech report cited in the VA
draft)

If I'm getting you right, this seems very reasonable to me.  Do you think it
makes sense to modify the current draft to include this case, or to make it a
separate thing?  

To extend your highway analogy, I agree that you don't want Trabants on the
autobahn, but it is reasonable to continue to use them on back roads...

PF

> Sent: Friday, July 11, 2008 7:29 AM
> To: Paul Francis

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Jul 2008 06:32:21 -0700

Paul,

That's exactly the point I was making. IMHO the current draft could be 
the right place to accommodate this as this is just a subset of the VA 
overall idea.

Cheers,
R.

PS. Also as said I agree to use Trabants on the back roads. Some people 
are really attached to them :)))

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 15 Jul 2008 10:26:02 -0400

IMHO:

If people are going to use products intended as enterprise switches or
otherwise not intended for full routing in provider networks they
better know what they are doing.

Some do.  Some don't.  Darwin ... and all that.

Idr should not be adding a hack for small gain in FIB size when better
techniques exist and products already exist with quite a few binary
orders of magnitude headroom.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 15 Jul 2008 10:57:04 -0400

In message <[email-masked]>
Paul Francis writes:
>  
> To be clear, we are talking about one new attribute, zero changes to
> the data plane, zero changes to the existing BGP decision
> process....just some rules for automatically setting up tunnels and
> new address aggregates (virtual prefixes).  Better to do this now well
> before the next generation of routers runs out of FIB.
>  
> PF

Paul,

Most providers amortize their routers in three years but keep them in
service for five or more.  Typical growth rates in healthy providers
are a doubling in about 1.5-2 years with some providers reporting 1
year (unconfirmed).  They keep routers in service by moving them
closer to the edge where the lower capacity of the router is less of
an issue, sometime redeploying from major cities to lesser cities.

A smart provider looks at their current default free routing size and
looks for at least 2 and better 4 or more times that in FIB and RIB
capacity, with RP memory size also dictated by the number of BGP peers
and peer groups that are expected to be supported.

Most of the providers with very large FIB and RIB are those top tiers
that do not do a good job of aggregating the routes for their own
infrastructure.  To aggregate well, they have to first allocate blocks
of addresses by POP and also subdivide their network into areas and
aggregate at area boundaries (possibly the only functionality where
confederations may be more straightforward and less error prone than
RR, but that is another topic).

If my memory serves me correctly, the target for major router vendors
(dictated by certain tier-1 providers) was over 1 million circe late
1990s, about 2 million early 2000 and some asked for as much as 4
million just to have headroom (and got it from some vendors).

RAM is cheap.  Once you go off chip (RAM off the forwarding ASIC)
memory bandwidth is much more an issue than memory size.

The problem is mainly "enterprise switch/routers" with on chip CAM and
TCAM and no provision for off chip RAM that have been a problem.  To a
lesser extent a few routers inteded as large enterprise routers or
default free provider routers will now require that you replace the
forwarding cards.

IMHO again: I think this is not a hack that IDR should pursue.  But I
have mostly worked with tier-1 providers and I am open to other
opinions.  Lets hear from some providers on this.

Curtis

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 15 Jul 2008 09:14:03 -0700

Hi Curtis,

> orders of magnitude headroom.

This is not a hack. In fact all what this is about can be accomplished 
today by sitting down and configuring three routers: edge, lookup router 
and egress (an option). If someone really want to divide the addressing 
space as Paul suggest marking can be reg community. This is provider 
contained solution - it does not cross any inter-as boundaries.

All this document attempts to is present such option for operators. Some 
may find them useful some may not.

Moreover there are other then edge FIB scaling real operational benefits 
for such solutions ... to name one I want to point at very fast 
reachability restoration in the event of failure.

To meet customer's and today's applications requirements one can forget 
about hop by hop switching, sending withdraws network wide, re-propagate 
new best paths etc ... That is gone model.

To meet such requirements requires to store backup paths for all 
destinations in FIB ahead of any failure + make sure that such router 
have very fast failure detection scheme. Every one could see that the 
less routers are configured with accelerated restoration features it 
scales much better ... or it allows for much faster OS upgrades on such 
boxes vs running around and upgrading all edge boxes.

Imagine also vast reduction in number of BFD sessions if someone would 
choose to use this technology for ASBR/PE failure detection.

Another one assuming that we would go one step fwd and remove RIB from 
the edge and just fully mesh those couple of IP lookup routers any 
problems with oscillations are history as well as vast reduction in 
intra-as triggered BGP instabilities.

For the FIB reduction indeed small/mid size ISPs or enterprise networks 
may benefit.

But for not directly mentioned in the draft benefits of moving to 
schemes which reduces the number of FIB lookup routers in the network 
many ISPs may be interested. And indeed I would like to hear their 
opinion on the list on that.

Cheers,
R.

---

From: Paul Francis <[email-masked]>
Date: Thu, 17 Jul 2008 13:11:45 -0400

Questions/comments inline:

> Sent: Tuesday, July 15, 2008 12:14 PM
> To: [email-masked]
> Cc: Paul Francis; [email-masked]
> and egress (an option). 

Are you saying here that no router modifications are needed to make this
work?

> If someone really want to divide the addressing

I'm not sure what you mean by "marking can be reg community".  Can you expand
a bit?

> may find them useful some may not.

I'm concerned that the folks that may find them useful (lower-tier ISPs)
aren't typically involved in IDR or IETF for that matter.  If there are folks
listening who can speak for lower-tier ISPs, please do so.  (My knowledge
here, such as it is, is all second- or third-hand.)

> new best paths etc ... That is gone model.

For all of this fast restoration stuff, I gather you are thinking in terms of
the core-edge structure that most (all?) tier-1 ISPs have?  Is your point
that simply shrinking the number of routers that have to participate in
re-routing will help scale ASBR/PE failure detection, and that we don't need
explicit new mechanisms along these lines (i.e. BGP Virtual Link Attribute)?

PF

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 17 Jul 2008 12:07:41 -0700

Paul,

>> today by sitting down and configuring three routers: edge, lookup
>> router and egress (an option). 
> 
> work ?

To make this work in the simplest form what is required is:

*A* Ability to attract traffic to lookup routers (Done)

*B* Ability to label switch to lookup router (Done)

*C* Ability to prevent sending what is in the RIB to FIB if not needed 
(This is vendor dependent anyway - does not require any standard work)

*D* Ability to forward traffic to external peer without lookup on the 
edge ASBR (This is also quite implementation dependent but with PHP and 
embedded service label this could be accomplished)

> I'm not sure what you mean by "marking can be reg community".  Can you expand

The community marking is needed only in the case of address space 
segmentation per lookup router in order for edge routers to recognize 
which out of the prefixes they receive should end up in FIB and which 
would only go as far as global RIB.

Such marking should be possible with standard community.

>> All this document attempts to is present such option for operators.
>> Some may find them useful some may not.
> 

I agree that IETF is not the best forum for that :). Much better would 
be NANOG, SANOG, APRICOT, JPNOG, RIPE etc ...

> explicit new mechanisms along these lines (i.e. BGP Virtual Link Attribute)?

Exactly. It helps scaling fast failure detection but what is more 
important it limits number of nodes where the redundant BGP paths must 
be present both in RIB and FIB.

Cheers,
R.

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 31 Jul 2008 09:06:59 -0400

The idea to keep FIB out-of-sync with RIB is a bit discomforting when
the troubleshooting has to be done for the traffic outage. Also, this
unnecessarily increases the complexity of the RIB and FIB interaction. 

For a lot of network operators, it may be cleaner to not even install
the qualified BGP paths into the RIB. FIB suppression would happen
automatically. 

Note that the decision to advertise the BGP paths to the downstream BGP
speaker can be independent of whether the paths are suppressed or not in
the RIB/FIB. This may be proposed by this specification.

Perhaps, you could clarify the rationale/advantages for keeping the
routes in the RIB. Thanks.

Cheers,
Rajiv

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Curtis Villamizar
> Sent: Tuesday, July 15, 2008 10:57 AM
> To: Paul Francis
> Cc: [email-masked]
> <[email-masked]>
> Paul Francis writes:
> >

---

From: Paul Francis <[email-masked]>
Date: Thu, 31 Jul 2008 18:11:28 -0400

Bottom line is that if the router doesn't have the full RIB, then it can't
participate fully in eBGP.  People already know how to shrink RIB and FIB in
edge routers that don't need to be full eBGP speakers.  Virtual Aggregation
makes it possible to shrink the FIB in any router.

You are certainly right that trouble shooting becomes more complex.  The
admin has to ask if the route is installed or not, and if not figure out what
aggregation point the packet is supposed to go to and so on.  This is one of
the trade-offs that the ISP has decide if it is worth making.

Could you be more clear about what you mean by "increases the complexity of
the RIB and FIB interaction"?  

PF

> From: Rajiv Asati (rajiva) [mailto:[email-masked]]
> Sent: Thursday, July 31, 2008 9:07 AM
> To: [email-masked]; Paul Francis
> Cc: [email-masked]
> Subject: RE: [Idr] draft on virtual aggregation
> 
>

---

From: Paul Francis <[email-masked]>
Date: Thu, 31 Jul 2008 18:41:49 -0400

> Sent: Thursday, July 31, 2008 6:11 PM
> To: Rajiv Asati (rajiva); [email-masked]
> Cc: [email-masked]

Well, alright, as somebody (Robert or Curtis?) pointed out already, you could
have a core router do multihop eBGP with the external peer and configure it
to dump only part of the RIB to the edge router...

But I must say that I find FIB suppression, as a general tool, to be quite
interesting.  The FIB is often the bottleneck in scaling, and if you have a
tool to control that, then I think long-term things will work out better.
(Which again is why I'm wondering what your concerns are.)

PF

> > > TCAM and no provision for off chip RAM that have been a problem.
> To
> > a

---

From: Paul Francis <[email-masked]>
Date: Thu, 31 Jul 2008 22:11:24 -0400

I've been thinking about this, so let me give a more thoughtful answer.

As I said, if you want to be able to do FIB reduction in edge routers whose
eBGP peers require the full DFZ, then either those edge routers must have the
full RIB, or some other router must do the peering (multi-hop).

So one limitation that you've bought yourself is that at least one router has
to have a full FIB.  This might be ok in many ISPs, but nevertheless there it
is.  With VA FIB reduction, all routers can have smaller FIBs.

When you shift peering to in internal router, then that router will have more
peers, and therefore bigger RIB and processing requirements.

You have to configure this internal router to give the appropriate routes to
the edge routers.

If the internal router doing the peering crashes, then a bunch of issues come
up.  Of course, connectivity with the peers is still good, so you need
another router to do the peering.  Without some kind of hot standby scheme,
this would require restarting all the BGP peering sessions.  Of course, it
requires more configuration, and that the internal routers monitor each
other's aliveness so as to know when to start peering.  In short, you step
away from the traditional model whereby the data path and the control path
(routing messages) are closely coupled, and there are a whole chain of
complexities that kick in that we don't know well how to deal with.

Now, if you keep the full RIB, then you get to run routing protocols as we've
run them for 25 years...no change at all.  You just add a new function that
decides what to install and what to suppress from the FIB.  Much of the VA
spec basically describes what these rules are (and they are quite simple).
You are absolutely correct in that torques how to debug...as I said now the
admin needs to know what was suppressed, and where packets should flow.  Plus
there are new failure modes (aggregation point crashing) so you need to
provision carefully, and there is some new config.  But overall it is much
cleaner, IMHO, than stepping away from the traditional packets-follow-control
routing model we've always had. 

I wasn't being entirely facetious when I suggested that all you need to do to
see how complex RIB reduction is is to look at the RRG proposals.  It is a
fundamentally hard problem, and we to date don't have good answers.  FIB
reduction allows you to avoid that path.

PF