Subject: [RRG] How to Incrementally Deploy APT

From: Dan Jen <[email-masked]>
Date: Thu, 21 Feb 2008 19:52:48 -0800

On Sat, 2008-02-16 at 14:18 +1100, Robin Whittle wrote:
>   APT        Hybrid push-pull.  Full database ITRs and query servers
>              are integrated into a Default Mapper.  All
>              participating ISPs have a few of these, with a slow
>              push scheme (separate BGP-like flooding system).  No
>              support for non-upgraded networks (Ivip's anycast ITRs
>              in the core == LISP's Proxy Tunnel Routers) - so it is
>              not incrementally deployable. 

Hey Guys,

So there have been some questions about how APT(and other proposals)
will be incrementally deployed.  We have devised a plan for incremental
deployment that we would like you to consider.  

The ideas in this plan may be applicable to other proposals that do not
require edge-network modifications.

Please hit us with any comments and questions.  We would love feedback
on how ridiculous or perfect you think our ideas are.

Dan Jen and Michael Meisel

---

From: "William Herrin" <[email-masked]>
Date: Thu, 21 Feb 2008 23:54:03 -0500

On Thu, Feb 21, 2008 at 10:52 PM, Dan Jen <[email-masked]> wrote:
>  We have devised a plan for incremental

Hi Dan,

What is the smallest unit? An "APT network" of which one or more are
contained in an "APT island"?

How do I go about deploying exactly one APT network into an existing
BGP AS? What new equipment do I install? What do I attach the
equipment to? What external information do I feed into it and how do I
get that information? What does it feed into my routing table? What do
I remove from my routing table? If you need to make assumptions about
the architecture of my existing BGP AS in order to give a concrete
answer, please do so.

What benefit do I get from being the first APT network? Are there new
services I can sell to my customers? Do I save money by having
deployed apt? Why should I be the first?

What benefit do I get when my neighbor deploys an APT network and we
form an island of two APT networks? Are there new services I can sell
to my customers? Do I save money by having deployed apt?

A-(B-C-D)-E-F

B, C and D are APT networks in an APT island. A and E connect to the
island with classic BGP. E connects to F with classic BGP. A host in A
wants to talk to a host in F. How does that work? Is BGP in the island
carrying all of E's BGP over to A and vice versa?

Now A wants to talk to C. You've stated that BGP in the island does
not contain prefixes in the island's mapping table. How does A get C's
prefixes?

Regards,
Bill Herrin

-- 
William D. Herrin [email-masked] [email-masked]
3005 Crane Dr. Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Robin Whittle <[email-masked]>
Date: Fri, 22 Feb 2008 23:45:45 +1100

Hi Dan and Micheal,

Here are some questions prompted by your explanation of how packets
from a host in a non-APT network will find their way to the
destination host which uses address space which is managed by APT.

I have redrawn your diagram to add ISP5 to the left island.

 ________________
|  APT Island 1  |
|     ______     |
|    / ISP5 \<=================\
|    \__,___/    |            ||
|   ___/         |            ||              ________________
|  /             | BGP      __\/__      BGP |  APT Island 2  |
| |   ______     | Routes  / ISP4 \  Routes |     ______     |
| |  / ISP1 \<===|========>\______/<========|===>/ ISP3 \    |
| |  \__,___/    |            /\            |    \__,___/    |
|_|_____|________| BGP Routes ||            |_______|________|
  |     |                   __\/__                  |
   \ ___|___               / ISP2 \              ___|___
    / Site1 \              \______/             / Site3 \
    \_______/                 /\                \_______/
                   BGP Routes ||
                           ___\/__
                          / Site2 \
                          \_______/

Please let me know if the following statements are true or how
they reflect my faulty understanding of APT.  Each statement
tends to assume that the previous ones were true.

 1 - Site1 can multihome with ETRs in ISP1 and ISP5.

 2 - Site1 can't multihome with ETRs in any other ISPs,
     including ISP3.

 3 - If Site2 uses less than 256 IPv5 addresses, for instance
     128, then your plan of advertising its prefix to BGP will
     fail because of the industry standard filtering of route
     advertisements limiting them to no longer than /24.

 4 - Assuming Site1 has only 128 IP addresses, its hosts can
     only be reached from hosts in Site2 if:

     a - There is no other user of the /24 in which Site1's
         prefix is half of - in which case the whole /24
         can be advertised to BGP and all will be well.

     or

     b - The other half of the /24 is used by Site4 (not shown)
         AND Site4 is currently using one or more ISPs in the
         same APT island that Site1 is using.  Then the whole
         /24 can be advertised and all is well.

 5 - Therefore, APT has very limited flexibility in serving the
     needs of end-users who have less than 256 IP addresses,
     unless all APT sites join up into a single "APT Island".

     This puts APT at a distinct disadvantage compared to LISP or
     Ivip, which can handle end-user networks (EID prefixes AKA
     micronets) as small as a single IP address, and enable the
     end users to use any ETR in any ISP in the world,
     irrespective of whether that ISP has ITRs and without any
     restriction on which ETRs are used by end-users with
     neighbouring micronets, or micronets in the same Mapped
     Address Block (BGP advertised prefix containing typically
     hundreds or thousands of micronets).

 6 - When Site1 uses APT's TE capabilities (I recall this is
     part of APT) to load share incoming traffic for its
     prefix over the two links from the ETRs of ISP1 and ISP5,
     this probably does not mean that packets from Site2 will
     be load shared between the two links ISP4-ISP5 and
     ISP4-ISP1.  More likely, all the traffic will flow from
     ISP4 to one of these, say ISP5.

 7 - This would make it difficult or inefficient for the APT
     system to somehow load share the incoming traffic over
     the two ETRs, since half the traffic would need to flow
     from ISP5 to ISP1.  (How would this actually be done?)

 8 - If:

     a - Site1 only uses an ETR of ISP5 which is located in
         Sydney, Australia.

     b - ISP5's network covers Australia, Japan and the UK.

     c - ISP5's BR linking to ISP4 is located in Sydney and
         has a link across the Pacific to a BR of ISP4's in
         Los Angeles.

     d - ISP4's network covers North America.

     e - ISP1's network covers North America and the UK.

     f - ISP1's BR with ISP4 is in Los Angeles, connected via
         a 1Gbps patch cord.

     g - Site1's prefix is 11.22.33.0/24.

     Then I understand that both ISP5's BR in Sydney and
     ISP1's BR in Los Angeles will be advertising
     11.22.33.0/24 to ISP4.

     h - Policy in ISP4 causes a preference to send traffic
         over the patch cord rather than over the expensive
         link between LA and Sydney.

     Then I understand packets from Site2 addressed to some
     host in Site will leave ISP4, bound for their
     destination in 11.22.33.0/24, via the patchcord-connected
     BR of ISP1.

     Assuming APT does provide backwards compatibility so that
     packets from Site2 will get to their intended destination
     in Site 1, then I understand they will have to travel
     through ISP1's network to a BR in the UK which connects
     with ISP5's network.  Therefore the packets will travel
     from LA, across the USA and the Atlantic, to the UK,
     then right round the world to Sydney before reaching
     the ETR.

     LISP, TRRP and Ivip have no such problem with islands or
     long path for packets, since the ordinary BGP system takes
     packets directly from the ITR to the ETR, irrespective
     of whether the networks those BGP routers are in have any
     upgrades for LISP, TRRP or Ivip.

 - Robin

---

From: Michael Meisel <[email-masked]>
Date: Fri, 22 Feb 2008 12:33:11 -0800

Hi Bill,

Thanks for your questions. Please see our answers inline. In case it 
wasn't clear, this scheme is specifically meant for APT, which is our 
design for a complete map & encap scheme for improved routing 
scalability. Our answers below may make more sense if you review our draft:

http://tools.ietf.org/html/draft-jen-apt

Though some things are out of date, it should be a decent overview.

> contained in an "APT island"?

We assume an APT network to be an entire BGP AS.

> BGP AS? What new equipment do I install?

You would need to make some modification to all border routers so that
they can do encap/decap and store a cache for mapping information. We
are hoping this can be a software-only upgrade for most hardware.

Additionally, you would need at least one new device called a default
mapper. You probably want at least two, just for redundancy purposes.

The details can be found in our draft.

> What do I attach the equipment to?

The border routers stay where they are, the default mappers can be
anywhere in your AS, preferably distributed in such a way as to minimize
latency from most border routers. You just need to assign an anycast
address for them, and configure the border routers to use that anycast
address.

>  What external information do I feed into it and how do I
> get that information?

There is a service running on the default mapper(s) that will exchange 
mapping information with other APT networks in your APT island. You 
merely need to manually configure the peering with your neighbors' 
default mappers, as you would for BGP routers. We are updating the 
details here, and will send another message to the list about that soon.

> What does it feed into my routing table? What do

Since APT is a map & encap system, you won't have any additions to your 
routing table, but the default mapper(s) store the full *mapping* table, 
and the border routers store a small cache of recently used mappings. 
You can remove from your internal routing tables any of your customers' 
PI prefixes, as well as any of the prefixes of other customer PI 
prefixes in the island. Any border routers on the border of the island 
will still need to keep these prefixes, however.

> deployed apt? Why should I be the first?

Primarily, you should be able to be much more competitive in supporting 
a large number of customers, regardless of whether they use PI prefixes. 
You will be able to remove their prefixes from the routing tables in 
most of your routers, and you will no longer need to have BGP peerings 
with your customers.

> to my customers? Do I save money by having deployed apt?

Now, you can both remove each others' customers' prefixes from your 
routing tables at all routers except at the border of the island.

> carrying all of E's BGP over to A and vice versa?

Our goal with APT is to separate end sites (or edge networks) and 
transit networks into different routing spaces. So, in your example, A 
must be an edge network, which does not participate in (B-C-D)'s routing 
space. So A's traffic enters B at an ITR, which encapsulates it with a 
destination address of a border router in D, which in turn decapsulates 
the traffic and forwards it to E based on its BGP tables.

> not contain prefixes in the island's mapping table. How does A get C's

Since we are intending to separate A from the routing space of the APT 
island, A CANNOT address any host in C. This is meant to be a feature. 
End users should not need to talk directly to nodes in the infrastructure.

-Michael and Dan

---

From: "William Herrin" <[email-masked]>
Date: Fri, 22 Feb 2008 17:47:23 -0500

>  are hoping this can be a software-only upgrade for most hardware.

Hi Michael,

Hmm. This is potentially a problem. In routers you have the "fast
path" and the "slow path."

The fast path is a hardware path through tailored ASICs. It can
generally only do trivial, well defined functions that the hardware
was specifically built for: decrement the ttl. Move to this interface
queue. That sort of thing. It does these tasks really really fast, all
the way from receiving the packet on one interface to transmitting it
on the next without the benefit of a CPU.

The slow path runs through the router's CPU. You can permute the
packet any way you want here but the capacity is much much less,
perhaps 1% of the router's normal capacity.

I know routers have been built that can handle things like GRE tunnels
and MPLS labeling in the fast path but I'm not sure how that stacks up
versus typical deployed hardware or to what extent those hardware
assists are software-reprogrammable to support some other encoding
scheme.

Is the encoder pulling data from the mapping cache? If yes and you
want it to live in the fast path then the map cache will have to go in
the TCAM. Depending on how large the map cache is (compared to the BGP
routes you've withdrawn) that could be a really bad thing. The FIB
size problem *is* a TCAM size problem.

If it's not possible to shoehorn APT into the fast path of existing
routers then because this change has to be made to every one of the
AS's border routers it could cost hundreds of thousands of dollars to
deploy for a modest size ISP.

Randy Bush keeps making this criticism wrt IPv6. If critical
functionality that was in the fast path the old way isn't in the fast
path the new way, deployment requires a forklift upgrade to devices
where it is in the fast path.

I think if I was in your shoes, I'd lavish some attention on exactly
which router models can be field-reprogrammed to do apt encoding not
just at a basic level but in the fast path.

>  and the border routers store a small cache of recently used mappings.

Okay, so my border router receives a packet from a non-APT neighbor.
It's a bare IP packet. I look up the destination in the TCAM FIB and I
get no routes. So, I send it to the APT encoder.

The APT encoder looks it up in the TCAM map cache and finds no map.
So, it bounces the packet to the slow path.

In the slow path, the router queries the default mapper for a prefix
and ETR map. The query comes back. The router encapsulates the packet
with an APT header for that ETR and sends it on. It also installs that
prefix and ETR in the TCAM map cache so that the next packet for that
destination stays in the fast path.

Do I understand correctly? If not, would you correct my understanding?

If yes then this system suffers from some of the performance issues
attributed to Cisco fast switching. The most serious of these is that
you overwhelm the router slow path whenever a worm on grandma's DSL
link starts spraying queries to random destinations that transit your
network without stopping there. Systems which place the encoder out of
line with the transit path don't suffer as much from this problem; she
still overwhelms the encoder but it doesn't take down the transit path
with it.

>  End users should not need to talk directly to nodes in the infrastructure.

Eh? But C is an entire APT network that was formerly a BGP AS. It
contains pron servers that I want to surf! How do I surf them from A?

Regards,
Bill Herrin

---

From: Michael Meisel <[email-masked]>
Date: Fri, 22 Feb 2008 15:47:12 -0800

> just at a basic level but in the fast path.

Yes, very true. As academics, we aren't exactly experts on the 
real-world hardware. =) We definitely see this as a necessary step at 
some point soon, but we don't believe that a narrow interpretation of 
current hardware capabilities should limit our exploration for the best 
design. It's much better to tweak a good design to work on the hardware 
than to miss a better design by focusing too much on hardware limitations.

I will say one thing that is encouraging for a fast mapping 
implementation in practice: our initial simulation results show that 
caches can realistically be quite small. So far, we've seen that a PE 
router at a reasonably sized POP would have a cache miss rate well under 
1% with a 4096-entry cache, and a single default mapper could support 
hundreds of such TRs. We'll be sharing more details about our 
simulations soon.

> Do I understand correctly? If not, would you correct my understanding?

There is one important thing you're missing here -- the router doesn't 
need to *wait* for a response from the default mapper, the default 
mapper will forward the packet. It does still get a reply, though, and 
adds the entry to its cache once it receives it. So, in the mean time, 
the router can continue to deal with other traffic. I'm not sure if this 
makes any difference to the issue you're describing, though.

> contains pron servers that I want to surf! How do I surf them from A?

Yes, but C is a transit network, by which we mean it's part of the 
infrastructure, think Level3 or Sprint. There aren't any pr0n servers 
for you in there (that we know of). If their customers have servers, 
then those customers need to have their prefixes in the mapping table.

-Michael and Dan

---

From: Dan Jen <[email-masked]>
Date: Fri, 22 Feb 2008 16:10:18 -0800

> I have redrawn your diagram to add ISP5 to the left island.

Point #2 is incorrect.  Site1 can multihome with ISP3.    

Perhaps this is your concern; the traffic engineering preferences that
Site1 expresses to ISP3(see draft for details) won't apply to ISP1 and
ISP5.  This is true, since the scope of TE preferences do not extend
between APT islands.  However, packet delivery works just fine.

>      hundreds or thousands of micronets).

APT considers edge networks to be on the granularity of ASes that do not
show up in the middle of an AS path.  Such ASes would have >256 IPs.
APT is not designed to allow home users to connect directly to an APT
island and have their packets tunneled.  

Thus you are correct in saying that during APT incremental deployment, a
site with a PI prefix longer than /24 will not be reachable outside the
island (unless the rest of the /24 is in the island, as you pointed
out).  Keep in mind that today's Internet does not support this
either.  

You claim that APT is at a disadvantage over LISP or Ivip.  Correct me
if I'm wrong, but don't these proposals have similar problems during
incremental deployment?  Let's say that I am a lisp/ivip customer site
with 111.111.111.128/25 and 111.111.0.0/16 is being used by a
non-upgraded ISP.  How would an anycast ITR or a PTR announce my prefix
in this case?  

>      from ISP5 to ISP1.  (How would this actually be done?)

Actually, we wouldn't even expect that ISP5 would forward any traffic 
for Site1 to ISP1, it would just deliver it directly.

Basically, the priorities and weights that APT uses for TE would only 
apply to senders within the island. There might be some tricks we could 
play with BGP to get data from ISP4 to go to across the right link
based 
on Site1's preferences, but that may not be advisable -- it will likely 
make a bit of a mess in the BGP announcements. Note that other
proposals 
have to pick a location in a similar trade-off space.

> <snip>

Tell us if we're mis-characterizing your example, but it seems that the 
larger point you're getting at is this: by removing the intermediate 
ASes in the APT island from the BGP path, we are providing an 
opportunity for ISP4 to unwittingly send traffic along an unnecessarily 
long path. That's absolutely right. However, this seems to us to be 
fundamentally the same problem faced by LISP PTRs or Ivip anycast ITRs 
in the core -- the ITR/PTR/BR/whatever becomes a necessary extra hop in 
the path which may be less than optimal. So we don't agree that other 
proposals avoid this problem.

In fact, APT islands are, first, more realistic since they require some 
business relationship between parties that encapsulate packets for each 
other, and, second, are likely to scale better. That is, during early 
deployment, APT islands will be small, only a few ISPs, meaning that an 
unnecessarily long path (at least one that could have been detected 
based on ASPATH length today) is unlikely. Only as APT islands grow 
larger are very long paths likely to be hidden by our scheme, at which 
point a smaller portion of the network will be non-APT networks,
meaning 
fewer parties will suffer.

Thanks for the comments.  They were very insightful.  

Dan and Michael

---

From: Robin Whittle <[email-masked]>
Date: Sat, 23 Feb 2008 14:28:42 +1100

Hi Dan and Michael,

> Perhaps this is your concern; the traffic engineering preferences
> that Site1 expresses to ISP3(see draft for details) won't apply
> to ISP1 and ISP5. This is true, since the scope of TE preferences
> do not extend between APT islands. However, packet delivery works
> just fine.

OK - ISP1, ISP5 and ISP3 all advertise a prefix in BGP for Site1's
EID space, so packets from non-upgraded networks will go to the
nearest such ISP.  This is an anycast approach, which is fine, since
the end-point is the host in Site1.

> APT considers edge networks to be on the granularity of ASes that
> do not show up in the middle of an AS path.  Such ASes would have
> >256 IPs. APT is not designed to allow home users to connect
> directly to an APT island and have their packets tunneled.

OK.  While LISP, TRRP (I think) and Ivip enable micronets (EID
prefixes) down to very small sizes, APT's EID sizes are constrained
by the current filtering limits of the BGP system.

This makes me think that APT would be a lot less effective at
enabling better utilization of IPv4 than the other proposals.

> Thus you are correct in saying that during APT incremental
> deployment, a site with a PI prefix longer than /24 will not be
> reachable outside the island (unless the rest of the /24 is in
> the island, as you pointed out).  Keep in mind that today's
> Internet does not support this either.

When, for instance, 80% of ISPs use APT and form a single island,
does this mean you could have edge networks with smaller address
assignments than the 256 lower limit imposed by your need to be
compatible with BGP's /24 limit?

I think that any such smaller edge networks would not be reachable
without the problems I raised from the other 20% of networks.

> You claim that APT is at a disadvantage over LISP or Ivip.
> Correct me if I'm wrong, but don't these proposals have similar
> problems during incremental deployment?  Let's say that I am a
> lisp/ivip customer site with 111.111.111.128/25 and
> 111.111.0.0/16 is being used by a non-upgraded ISP.  How would an
> anycast ITR or a PTR announce my prefix in this case?

Ivip "anycast ITRs in the core/DFZ" advertise Mapped Address Blocks.
 These are /24 or shorter.  A typical MAB might be a /16 or a /14.

It would contain the User Address Blocks (UABs) of hundreds or more
likely thousands of separate end-users.  In Ivip, these are of
arbitrary length and starting position - they are not constrained to
be binary boundary prefixes.

Within each UAB, the end-user defines as many micronets as they
like, down to single IP addresses.  Then they control the mapping of
each micronet to any ETR in the world.

LISP's "Proxy Tunnel Routers" are the same as Ivip's "anycast ITRs
in the core/DFZ".  They advertise a prefix, and the system only
makes sense in terms of reducing the number of advertised prefixes
due to the fact (typically) that each such prefix provides address
space for many end-users, who without LISP/Ivip would either not get
any multihomable space at all, or who would get it by conventional
means of getting an ASN, PI space and therefore by adding more
advertised prefixes to the DFZ.

LISP doesn't have a concept of User Address Block and its micronets
are called "EID prefixes" - which are constrained to be binary
boundary prefixes.

> Actually, we wouldn't even expect that ISP5 would forward any
> traffic for Site1 to ISP1, it would just deliver it directly.

OK, so load sharing wouldn't work for traffic coming from an
upgraded network?

> Basically, the priorities and weights that APT uses for TE would
> only apply to senders within the island. 

OK.

> There might be some tricks we could play with BGP to get data
> from ISP4 to go to across the right link based on Site1's
> preferences, but that may not be advisable -- it will likely
> proposals have to pick a location in a similar trade-off space.

Ivip doesn't attempt to do explicit load sharing.  If you want to
load share over multiple ETRs, it is necessary to have the incoming
traffic split over multiple IP addresses (or IPv6 /64s) and map each
one to a different ETR.  This may seem clunky, but (for a small fee
per update) you can fine-tune the starting points and lengths of
these micronets, and which ETR they are mapped to, with (ideally)
about a 5 second delay between giving the command and having all the
world's ITRs follow your instructions.

> Tell us if we're mis-characterizing your example, but it seems
> that the larger point you're getting at is this: by removing the
> intermediate ASes in the APT island from the BGP path, we are
> providing an opportunity for ISP4 to unwittingly send traffic
> along an unnecessarily long path. That's absolutely right.

> However, this seems to us to be fundamentally the same problem
> faced by LISP PTRs or Ivip anycast ITRs in the core -- the
> ITR/PTR/BR/whatever becomes a necessary extra hop in the path
> which may be less than optimal. So we don't agree that other

With Ivip, the destination host is in Sydney, and so it its ETR.

Sending hosts all over the world, in non-upgraded networks, have
their packets attracted to one or more "anycast ITRs in the core/DFZ".

If there is only one of these there will be longer paths, unless
that ITR is on the natural path for packets from the single sending
host and the destination.

The idea is that there are ITRs around the Net.

So if the sending host is in San Diego, the nearest "anycast ITR in
the core/DFZ" is in LA, then the packets go to LA in their raw
state, and are tunnelled to the ETR in Sydney.  Unless there was a
direct link from San Diego to Sydney, then this represents no extra
path length, because without Ivip, the path would have been through
LA anyway.

At the same time, a sending host in a non-upgraded network in
Edinburgh would probably find its outgoing packets being
encapsulated by an anycast ITR in London - which again would not be
a longer path than usual.

If there was a single anycast ITR in the core in Sydney, then all
would be well in our example - there would never be any longer
paths.  But that doesn't work when the end-user decamps to New York
and uses an ETR there.   So there needs to be anycast ITRs in the
core/DFZ pretty widely scattered around the Net for the system to
work with little or no extra path lengths.

The business model for this is discussed somewhat in:

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-db-fast-push-00.html

Basically, the companies how are renting out the micronet space pay
for these anycast ITRs, either by running them themselves or paying
another company who runs these ITRs for many other such Ivip address
companies.  Without a healthy set of such ITRs, the address space
would involve longer paths in many instances.   These companies
might need to charge on volume of traffic handled by the ITRs, not
just the number of IP addresses in each micronet they rent out.

I will reply to the rest of your message later.  I haven't yet read
your discussions with Bill Herrin.

Thanks again for your response.

  - Robin

---

From: Michael Meisel <[email-masked]>
Date: Sat, 23 Feb 2008 14:46:31 -0800

> compatible with BGP's /24 limit?

Not necessarily, see below...

> likely thousands of separate end-users.

Yes, but this is an economic issue, not a technical one. The reason you 
can do this is that you require that an entire /24 or shorter prefix is 
reserved for edge address space. There is no reason that APT cannot do 
this as well, assuming it is economically feasible.

> OK, so load sharing wouldn't work for traffic coming from an

That's right. The portion of your traffic that follows your TE 
preferences is based on the portion of traffic that comes from within 
your island. This may actually be a good thing, as it will provide a 
further incentive for islands to merge and customers to pick ISPs that 
are part of larger islands.

> With Ivip <snip>
> So there needs to be anycast ITRs in the

So it sounds like we agree.

> your discussions with Bill Herrin.

Thanks, Robin, we'll keep an eye out.

-Michael and Dan

---

From: Scott Brim <[email-masked]>
Date: Thu, 28 Feb 2008 16:34:23 -0800

On 2/21/08 10:52 PM, Dan Jen allegedly wrote:
 > The only new task for an edge network is to provide
 > traffic engineering information to its providers.

How is this done?

 > Second,
 > inside an APT island, the APT networks exchange their mapping
 > information with each other. This allows their default mappers to
 > maintain mapping information for the entire island.  Let's call this
 > table the Island Mapping Table.

I'm a little concerned about three things.

  - Scaling.  If I understand correctly, we will have inter-domain
    routing carrying all of the fine-grain intra-domain changes, e.g.
    ETR up/down.  In the past we have been concerned about fault
    isolation, which is one reason we still have strong domain
    boundaries: so that errors in one domain do not "poison" a
    neighboring domain.

  - Cooperation.  This seems to require providers to work together for
    the common good.  That's not something we can easily guarantee
    :-).  Do you have business motivators that would encourage them to
    share ops data?

  - Policy.  I believe it's possible to say that traffic to one
    sub-prefix of a site should come in to a site one way, and traffic
    to another sub-prefix should come in a different way.  Is is
    possible, using APT, to say that traffic *from* some remote site
    should come in to my site one way, while traffic from some other
    remote site should come in via some other set of CEs?

 > As APT islands grow larger and merge with each
 > other, this BGP table can be gradually reduced and eventually
 > contain only transit prefixes.  Smaller BGP table provides an
 > incentive for transit networks to deploy APT.

First, smaller BGP tables are an incentive only if the total difficulty 
of running both (small) BGP and the APT protocol(s) is less than the 
difficulty of running BGP right now.  I suspect that it will be years 
before everyone converts to a new routing architecture, therefore we 
can't talk about "transition" so much as "interworking" and 
"coexistence".  How much incremental difference do you believe there
is between an APT entry and a BGP entry?  What percent of a site's
routing needs to be converted to APT in order for BGP+APT to be less
complex to operate than BGP alone?

Getting a smaller BGP table is certainly an incentive for a domain to
get its *neighbors* to convert. :-)

swb

---

From: "William Herrin" <[email-masked]>
Date: Thu, 28 Feb 2008 20:47:57 -0500

On Fri, Feb 22, 2008 at 6:47 PM, Michael Meisel <[email-masked]> wrote:
>  the router can continue to deal with other traffic.

So the router doesn't ask the default mapper for a map; instead it
sends the packet whose destination was a cache-miss to the default
mapper. Then the default mapper encapsulates and forwards the packet.
The default mapper at that point can also choose to send the router a
new entry for its cache so that the next packet for that destination
can flow directly through the router.

Do I understand now?

>  I'm not sure if this

Yes, a big difference. This means that it should be possible to
implement the router's part entirely in hardware. It's the difference
between working and collapsing under the load at the engineering
level.

>  then those customers need to have their prefixes in the mapping table.

Okay, so C in my diagram is really two components: C1, the APT transit
network and C2 the endpoint network attached only to C1 which contains
my pr0n servers. I can surf the servers in C2 from A. Is that correct?

How do C2's prefixes get to A's BGP table so that A knows it should
send those destinations to B?

Stepping back, I think the single most important question you could
address at this point is: which of the currently deployed router
models can be software-upgraded to support APT in the fast (hardware)
path? This is make or break for your proposal. If you have to replace
every DFZ router to get APT in the fast path, deployment will cost
more than 4 Billion with a B dollars. If you only have to deploy
2*27000 default mappers, deployment could cost as little as 150
million dollars. I think you'll agree that's a -huge- difference.

Regards,
Bill Herrin