Subject: Re: [rrg] Fundamental objections to a host-based scalable routing solution

From: [email-masked] (Noel Chiappa)
Date: Mon, 24 Nov 2008 13:40:14 -0500 (EST)

> From: Robin Whittle <[email-masked]>

    > Here is a list of objections to any routing scaling solution which
    > pushes work relating to multihoming, TE and changing ISPs onto hosts.

    >                   Extra host traffic
    >                   Host operation more affected by packet loss
    >                   Increased cost and reliability problems
    >                   for mobile hosts operating over wireless
    >                   Extra complexity in the host
    > ...
    >                   General principle of solving a problem close
    >                   to its origin

A very similar list of objections could (and probably was, that discussion
would have been slightly before my time) have been raised to doing
reliability (retransmission, etc) in the hosts, back in the day (i.e. early
70's); as many will no doubt recall, in early networking (ARPANet, X.25),
reliability was the responsibility of the network, not the hosts.

Yet today we do reliability in the hosts, and to most people it's 'obvious'
that that's the right thing....

    > add architectural elements to the network to solve its scaling problem
    > when millions of end-user networks need portability, multihoming and TE.

It's not clear to me what you mean by "[in] the network", in saying the
above.

If you mean by this to put a function in the first-hop router, instead of the
host, that is not going to really change things like the capabilities (e.g.
response time) or cost (e.g. overhead in terms of number of packets needed to
run the mechanism). It may be more practical (less boxes to modify), but
that's all - and that more speaks to the deployment path, than the basic
architecture (e.g. deploy it in first-hop routers to start with, but
eventually it should migrate into the hosts).

(To me, 'extra complexity in the hosts' as a reason to not put something in
the hosts is a bit of a non-starter: most of the recurring costs, in terms of
code size, etc are trivial, given current OS sizes; and the non-recurring
costs, such as engineering time to write the code, are amortized over so many
hosts they are also not too significant.)

Do we agree that by "[in] the network", one cannot possibly mean 'have the
path selection do it', because it is precisely the case of "millions of
end-user [small entities]" which _cannot_ be supported in the path selection?

	Noel
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 25 Nov 2008 09:50:55 +1300

The good news is that the RRG doesn't get to decide
about this. A host based solution will or won't deploy,
and will or won't succeed, on its own merits and regardless
of both the routing community and the ISPs. So I don't
think we need to discuss it here at all.

   Brian

---

From: "Tony Li" <[email-masked]>
Date: Mon, 24 Nov 2008 13:04:13 -0800

|The good news is that the RRG doesn't get to decide
|about this. A host based solution will or won't deploy,
|and will or won't succeed, on its own merits and regardless
|of both the routing community and the ISPs. So I don't
|think we need to discuss it here at all.

The bad news is that we have to discuss this.  If a host based solution is
in fact undeployable, then there's little point in RRG coming to consensus
on one.

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 25 Nov 2008 10:30:08 +1300

> in fact undeployable, then there's little point in RRG coming to consensus

True, but while we're being pragmatic, it seems clear to me that the RRG
will never reach consensus on a host based solution in any case. And
since we have one in progress in the IETF already, aren't we done here?

   Brian

---

From: "Tony Li" <[email-masked]>
Date: Mon, 24 Nov 2008 14:09:03 -0800

|True, but while we're being pragmatic, it seems clear to me 
|that the RRG
|will never reach consensus on a host based solution in any case. 

Interesting.  Again, not everyone seems to agree with you.  There's clearly
a case (that you yourself have invoked) that it is possible to upgrade
hosts.  Thus, it seems like this is not wholly out of the question.

|And
|since we have one in progress in the IETF already, aren't we done here?

I assume that you're referring to shim6.  No, that doesn't at all imply that
we're done.  One host solution does not imply that it is the correct
solution.  In fact, as one of the original contributors to that effort, it's
pretty clear to me that it's very much a suboptimal 'solution' in that it
simply insulates the transport layer from the architectural changes that
need to be made. 

Regards,
Tony

---

From: "William Herrin" <[email-masked]>
Date: Mon, 24 Nov 2008 17:47:44 -0600

> in fact undeployable, then there's little point in RRG coming to consensus

At least one train of thought suggests that a host-based solution is
more easily deployed than a network-based one. The rationale goes
something like this:

1. IPv6 was supposed to be a phased deployment: a. Install the
software ubiquitously. b. Turn it on. c. Displace IPv4 as the primary
carrier. d. Turn IPv4 off.

2. We completed step a. Why suspect we can't do it again?

3. IPv6 deployment died at step b. There's no reason to suspect other
network-based strategies won't suffer the same fate.

4. If a host-based protocol allows us to use existing layer-3
protocols during step c, we can move step b after step c, improving
step b's chance of being doable.

Correct? Maybe, maybe not. I'm far from certain that its not.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "Fleischman, Eric" <[email-masked]>
Date: Mon, 24 Nov 2008 16:56:41 -0800

>| Brian Carpenter Wrote:
>|True, but while we're being pragmatic, it seems clear to me that the 
>|RRG will never reach consensus on a host based solution in any case.

>Tony Li wrote:
>Interesting.  Again, not everyone seems to agree with you.  There's
clearly a 
>case (that you yourself have invoked) that it is possible to upgrade
hosts.  
>Thus, it seems like this is not wholly out of the question.

I wasn't able to make the IETF so I don't have any insight as to whether
RRG community is converging or diverging. Regardless, it doesn't
surprise me that not everybody agrees with any given point -- RRG is too
large a community for that. However, I resonate with Brian's initial
observation because if we restrict RRG to routing only, then we also
restrict the diversity of solutions under consideration and are
therefore converging on a common solution. 

On the one hand, this is self-serving on my part because the solutions
that I resonate with are routing only. Some may argue that such a
limitation would eliminate the best alternatives. I have two
counter-arguments to that claim:
1) I don't discern any strong consensus building upon approaches
involving host modification so if they are a better approach, the
majority has yet to become aware of their superiority.
2) If modifying hosts are out-of-bounds for RRG, then perhaps
middleboxes can also be deemed out-of-scope as well? If this is the
case, then we are just that much closer to convergence.

My personal belief is that I'd like us to converge soon on a
routing-only solution. I think that it is time to begin to wrap the
theoretical discussions up and proceed to modeling and simulation and
limited deployment experimentation.

Failing that, I would like to better understand why the RRG community
hasn't yet converged on the desirability of map-and-encaps as a
desirable RRG vector.
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "Tony Li" <[email-masked]>
Date: Mon, 24 Nov 2008 18:27:00 -0800

Eric, 

|My personal belief is that I'd like us to converge soon on a
|routing-only solution. I think that it is time to begin to wrap the
|theoretical discussions up and proceed to modeling and simulation and
|limited deployment experimentation.
|
|Failing that, I would like to better understand why the RRG community
|hasn't yet converged on the desirability of map-and-encaps as a
|desirable RRG vector.

There is a portion of the community (significant, IMHO) that feels that
there are significant drawbacks to the map-&-encap strategy.  While I can't
claim to represent all of them or even a significant breadth of their
opinions, there are some issues that have been raised.

- Map-&-encap doesn't solve the problem where it truly occurs: at the host.
The convolution of locator and identifier happens precisely because the
transport layer has reused the address as part of the connection identifier.
Until we change this, we're simply band-aiding around the problem.  And
band-aids really aren't the way to create a lasting architecture.  Please
note that this also implies that routing-only solutions aren't the correct
direction.

- Map-&-encap comes with significant overhead.  It adds another network
layer header, and some additional per packet overhead.  In changing the MTU,
it requires additional mechanisms that some (tho I appear to be alone in
this ;-) find concerning.

- Map-&-encap creates some attack vectors.  If an attacker sends you a
packet from a forged (and possibly random) EID, and that causes you to
respond and perform a mapping lookup on the forged EID, then the attacker
can cause you to fill your mapping cache and thus create a significant
performance issue.  While it's already difficult in the Internet today to
trace forged source addresses, it would seem like tracking forged EIDs will
be even harder, as they are likely to come with addresses from forged ITR
addresses.  There are probably ways to address this, but they would seem to
come with Even More Mechanisms.

- Is virtualization really the right approach?  It's been noted that
map-&-encap is basically equivalent to running the entire Internet inside of
a VPN, where the mapping function conveys the edge of the 'primary'
infrastructure that terminates the VPN tunnels.  Some find it
architecturally disquieting to run our most essential function virtually
when we find that we cannot run it natively.

There are probably many other issues, but these are some of the ones that
have been raised.

Regards,
Tony

---

From: Robin Whittle <[email-masked]>
Date: Tue, 25 Nov 2008 15:01:35 +1100

Hi Noel,

> that that's the right thing....

OK - I think this is an interesting line of discussion.

One could argue that a host-based solution to the routing scaling
problem is simply upgrading the host's capacity to deal with lost
packets, by adding new methods to the current one of simply resending
to the same IP address.

The trouble with delegating reliable delivery to the "network" is
that the host has an unreliable link to the network - so it cannot be
a complete solution.

I think the core-edge separation systems can be complete solutions.
They give the host and the end-user network stable IP addresses, so
the host continues the current practice of ensuring reliable
communications by resending to the same address.  This covers
problems in the entire host-to-host path, including the link to the
nearest router, Ethernet switch, Wi-Fi access point, IP gateway etc.

I stand by my objections to hosts having to engage in processing,
probing, extra traffic etc. to cope with multihoming, TE and changing
ISPs.

> eventually it should migrate into the hosts).

Yes, I would object to pushing the functionality, extra traffic etc.
out almost to the hosts as well, such as to the router closest to the
hosts.

I think the best router-based core-edge separation solutions involve
the CE and PE routers - those routers at either side of the ISP /
end-user network division - with a mapping system and extra functions
such as query servers, ITRs, ETRs and OITRDs.  They do not require
any changes to the rest of the end-user network's routers, or to its
hosts.

> hosts they are also not too significant.)

I agree - a central reason for the Internet's success is hosts having
quite a complex stack - and arbitrarily complex, flexible and
updatable application programs.

For most PCs, adding another megabyte of code or RAM usage is not
going to cause significant costs or difficulties.

My objection is partly to the CPU and RAM requirements being
increased  in all hosts, including the smallest.  My objections are
more to do with:

  1 - Increased traffic to all hosts.

  2 - The extra costs and difficulties this causes for wireless
      hosts.

  3 - How much basic connectivity in the the new regime would be more
      dependent, fast, on reliable packet delivery to and from each
      host, including fast, reliable DNS.

> end-user [small entities]" which _cannot_ be supported in the path selection?

I think you are referring to souping up each DFZ's mechanism for
choosing the best path for packets matching any one end-user prefix.
 As far as I know, we all agree BGP can't be souped up to cope with
millions of end-user networks.  Nor do we have a direct alternative
to BGP which could do this - or a way of smoothly transitioning to
such an alternative.

What we do have is router-based core-edge separation schemes, which
could be introduced without disruption and without any changes to
hosts.  Ivip with OITRDs and LISP with PTRs could could provide
multihoming for 100% of incoming packets for adopters of the
technology, irrespective of how many sending hosts were in networks
with the technology (ITRs in the sending host's networks).

I think this is the best solution. Please see a later message:

  Tony's critique of map-encap - or of router-based core-edge
  separation

for my preferred solutions.

 - Robin

---

From: Christian Vogt <[email-masked]>
Date: Tue, 25 Nov 2008 19:10:36 +0200

> since we have one in progress in the IETF already, aren't we done  

Brian,

no doubt that discussions in RRG won't change how widely Shim6 will be
deployed.  But this does not mean that RRG should ignore the concerns
that people have raised against Shim6.  If RRG came up with a host-based
solution that mitigates these concerns, then this would be a big
achievement in my opinion.

And RRG can already claim some success in this regard:  One concern that
has been raised against Shim6 is the lack of network control on traffic
engineering.  In RRG, at least two new host-based solutions have been
proposed that mitigate this concern:  Six/One gives the network explicit
traffic engineering control through address prefix rewrites in routers.
And multi-path TCP gives the network an implicit means for traffic
engineering through bandwidth limits and packet loss.

Another disadvantage of Shim6 -- which applies just as well to other
host-based solutions that provide a "stable address" -- is the
complexity of the extra indirection layer:  This, plus the cryptographic
techniques by which the new indirection is secured, are not trivial.
They demand special expertise from network administrators, and they
limit network design freedom because they require certain address
configuration methods.  As I have tried to explain in Minneapolis, a
hostname-oriented stack architecture would mitigate these issues -- by
replacing the "stable address" with a human-readable hostname that maps
onto regular, non-cryptographic IP addresses.  The stack architecture
would thus become simpler, and more friendly for users, administrators,
operators, and application developers.

So overall, while Shim6 is for sure a great technology, there is room
for improvement.  And here, I believe, is where RRG can contribute.

- Christian

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 25 Nov 2008 10:57:26 -0800

Hi Tony,

>-----Original Message-----
>From: Tony Li [mailto:[email-masked]]
>Sent: Monday, November 24, 2008 6:27 PM
>To: Fleischman, Eric; 'Brian E Carpenter'
>Cc: [email-masked]; 'Noel Chiappa'
>Subject: Re: [rrg] Fundamental objections to a host-based
scalableroutingsolution
>note that this also implies that routing-only solutions aren't the
correct

I haven't been tracking the host-based discussions all that
closely, but AFAICT they should be compatible with map/encap.
For example, we have talked about end systems using HIP while
the routers (even those within the innermost recursive instance)
use map/encap. (I'm sure the scenario holds for any of the
other host-based schemes also.)

>this ;-) find concerning.

The Internet today is entering a transition phase in which
links with 1500 byte or smaller MTUs may soon be considered
legacy equipment in need of replacement - especially within
the core. Even at 1500 bytes, adding an extra 20 byte IPv4
header only adds ~1% per-packet overhead and moving out to
Gigabit Ethernet jumboframe sizes (9KB) the overhead is
further amortized.

The idea of the MTU handling mechanisms then is to detect and
provide short term interoperability for degenerate links. For
example, the mechanism can automatically activate itself when
an MTU-challenged link is on the path while at the same time
providing feedback to the operator that a specific link is in
need of replacement. As more and more degenerate links are
replaced, the mechanism is used less and less.

>come with Even More Mechanisms.

But, the mapping system gives the ETR a means for determining
the set of RLOCs from which packets that use specific EIDs may
originate. This could be termed EGRESS filtering and provides
a measure not available in some of the other approaches. Using
egress filtering, an ETR can always consult the mapping system
to determine whether an EID came from behind an acceptable
RLOC. When there is spoofing, the ETR then has a way to
determine which ITRs are not correctly implementing INGRESS
filtering and, if corrective actions are not taken, expose
the violators to the censorship of the greater community.

Notice the similarities with the MTU mechanism discussed above.
When the map/encaps solutions are deployed, there is incentive
for everyone to "play nice" together and an incremental migration
path toward the greater good. 

>when we find that we cannot run it natively.

I'm not sure I get this; one of the key features of map/encap
is to arrest (and preferably reduce) routing table sizes which
AFAICT it does very nicely. Is that what you mean?

Thanks - Fred
[email-masked]

---

From: "Fleischman, Eric" <[email-masked]>
Date: Tue, 25 Nov 2008 11:52:26 -0800

Tony,

Thank you for this wonderful summary. I am greatly relieved that the objections to map-and-encaps are based upon well-reasoned technical insights because we as a community can build a consensus together by examining contributions of this quality.

I resonate with these objections, particularly the first one (i.e., locator-identifier split), because the organization in which I work has found considerable power by implementing HIP to solve a variety of issues. Two examples are that (1) we cofounded the Secure Mobile Architecture (SMA) work within the Open Group which leverages HIP as a foundation for mobile security. We have applied SMA within our factories and have found that it is an outstanding technique to secure SCADA-based machine controllers -- by far the best alternative that we are aware of. (2) We've done applied research using HIP as a basis for maintaining streaming multimedia sessions within extremely mobile MANET networking environments. HIP is an effective technique for minimizing the impact of signal intermittence and mobility upon applications.

However, I believe that resisting map-and-encaps is like resisting NATs -- doing so is like "spitting into the wind" because it is already ubiquitous, so what's the point? More accurately, when I first read Fred Templin's VET-RANGER-SEAL trilogy, my reaction was that he had finally fingered what the Internet architecture has become. Many of you know that I have been concerned about the viability of the IETF's work ever since we abandoned our historical architectural foundations in the mid- to late-90s (e.g., the rise of middleboxes, the fattening of the middle of the hour glass, the confusion between core and edge). I don't object to modifying or changing architectures but I do object to having no architecture foundation at all, which is what we did in the IETF (in my personal opinion). Regardless, reading Fred's work was an epiphany for me because I realized that regardless of whether I liked it or not, the IETF indeed has finally created an architecture foundation again: map-and-encaps.

Everywhere I look I see coherent evidence that the IETF and industry has ubiquitously embraced map-and-encaps as our current Internet Architecture foundation. Witness:
1) IPv4-IPv6 coexistence: ISATAP, Teredo, IPvLX, etc.
2) Military Communications Security (COMSEC)
3) IPsec's ESP in tunnel mode
4) L3VPN (including how my employer relates to our ISPs (please note the plural))
5) ecommerce 
6) ARINC 664 enclave protections
7) Architectures that leverage IP to join together distributed legacy (i.e., non-IP) protocol systems. It is also used to merge legacy populations into IP networks (e.g., RFC 1006).

I've largely dropped out of IETF discussions since 2001 because I can't talk about what I do anymore. However, in all of the many topics that I have worked during that time I repeatedly see map-and-encaps based solutions. One example that I can share is that in 2006 I did a study on the safety and security of LAN-attached airborne avionics for the US Federal Aviation Authority (FAA). (Note: two of my books (DOT/FAA/AR-08/31 and DOT/FAA/AR-08-35) are now available on the FAA web site dealing with this topic.) I believe that the foundational issue of this problem devolves down to how to viably extend ARP 4754-based partitions into ARINC 664 enclaves. There are two basic technologies used for that approach: firewalls and VPNs. Local issues will determine which is preferable, though I argue that future airspace requirements (e.g., Eurocontrol's Single Sky, NASA's NextGen) imply using both and coupling VPNs with the Biba Security Model to create assurable distributed network deployments. My point is that wherever one discusses either "assurable networks" or "distributed networks" map-and-encaps is usually involved. 

Changing topics, in industry (e.g., ecommerce) firewalls have become increasingly passé because the majority of our internal network users, for example, are not our employees, but rather are our suppliers and customers. We call this de-parameterization. De-parameterization is a fact of life for most very large industrial companies. This is another type of problem which map-and-encaps is increasingly being used to solve. My point is that wherever one discusses distributed enclaves map-and-encaps is usually involved.

--Eric

>There is a portion of the community (significant, IMHO) that feels 
>that there are significant drawbacks to the map-&-encap strategy.  
>While I can't claim to represent all of them or even a significant 
>breadth of their opinions, there are some issues that have been raised.

>- Map-&-encap doesn't solve the problem where it truly occurs: at the 
>host. The convolution of locator and identifier happens precisely 
>because the transport layer has reused the address as part of the 
>connection identifier. Until we change this, we're simply band-aiding 
>around the problem.  And band-aids really aren't the way to create a 
>lasting architecture.  Please note that this also implies that 
>routing-only solutions aren't the correct direction.

>layer header, and some additional per packet overhead.  In changing the 
>MTU, it requires additional mechanisms that some (tho I appear to be 
>alone in this ;-) find concerning.

>- Map-&-encap creates some attack vectors.  If an attacker sends you 
>a packet from a forged (and possibly random) EID, and that causes 
>you to respond and perform a mapping lookup on the forged EID, then 
>the attacker can cause you to fill your mapping cache and thus create 
>a significant performance issue.  While it's already difficult in the 
>Internet today to trace forged source addresses, it would seem like 
>tracking forged EIDs will be even harder, as they are likely to come 
>with addresses from forged ITR addresses.  There are probably ways 
>to address this, but they would seem to come with Even More Mechanisms.

>map-&-encap is basically equivalent to running the entire Internet 
>inside of a VPN, where the mapping function conveys the edge of the 
>'primary' infrastructure that terminates the VPN tunnels.  Some find 
>it architecturally disquieting to run our most essential function 
>virtually when we find that we cannot run it natively.
>
>There are probably many other issues, but these are some of the ones 
>that have been raised.
>
>Regards,
>Tony

---

From: Christian Vogt <[email-masked]>
Date: Wed, 26 Nov 2008 10:49:57 +0200

[Forked from: "Fundamental objections to a host-based scalable routing  
solution"]

Christian Vogt wrote:

> [...]  As I have tried to explain in Minneapolis, a hostname-oriented
> stack architecture would mitigate these issues [...]

I should have been more elaborative for those who couldn't make it to
Minneapolis:

Assuming that the goals of RRG are to enable multi-homing and to
eliminate renumbering in a scalable manner:  The argument I brought
forth in Minneapolis was NOT that these goals could be fully solved with
a host-based solution.  The suggestion was instead for RRG to consider a
pair of host-based plus network-based solution.  Since the two goals are
independent of each other, they may well be best addressed with separate
solutions:  It is obvious that renumbering can be eliminated only with a
network-based solution.  And as previous email discussions indicate,
multi-homing may best be enabled with a host-based solution.  In fact, I
don't see a convincing technical reason to address both goals with a
single solution.  Trying to do that would simply make our job harder.

Regardless of which solution pair is picked, the solutions in the pair
would have to be independent of each other.  A mutual dependency between
host upgrades and network upgrades would impose deployment hurdles,
which in my opinion would be insurmountable.  But if each solution in
the pair provides benefits independently of the other, and if those
benefits are complementary, then the solution pair may deployment-wise
well be superior to a single one-size-fits-all solution.

- Christian

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 26 Nov 2008 09:09:11 -0800

>From: Christian Vogt [mailto:[email-masked]]
>Sent: Wednesday, November 26, 2008 12:50 AM
>To: Routing Research Group Mailing List
>Cc: Noel Chiappa
>Subject: [rrg] Maybe it's not "either-or": Considering a
host/network-basedsolution pair
>well be superior to a single one-size-fits-all solution.

That matches what I was saying yesterday:

http://www.ietf.org/mail-archive/web/rrg/current/msg03834.html

but it seems to me that the RRG's attitude has been one of
"let's take care of the network, and let the hosts take care
of themselves". Maybe I'm wrong...

Fred

---

From: "Christopher Morrow" <[email-masked]>
Date: Wed, 26 Nov 2008 16:45:38 -0500

(sorry, wrong source)

On Wed, Nov 26, 2008 at 4:43 PM, Christopher Morrow
<[email-masked]> wrote:
> On Wed, Nov 26, 2008 at 12:09 PM, Templin, Fred L
> <[email-masked]> wrote:
>>
>>
> I saw it as: The pain we see is in the network, host solutions don't
> (as of yet) provide the control required for large-scale TE issues,
> and often the network/host people are different groups with varied
> abilities to interact.
>
> The only host solution presented from the IAD workshop in AMS (in
> 2005?) was shim6, which may work fine for some discrete instances, but
> isn't going to help large enterprises multihome, nor is it going to
> relieve the pressure on the routing platforms.
>
> RRG ought to, in my mind, step back and decide what solution set (more
> than one solution) is going to best serve the Internet in the
> long-term (15-20-30 years), and leverage that over to the IETF to
> codify so vendors can implement. I don't have a lot of faith that the
> current routing paradigm is going to last 10-20 years at current costs
> with current growth curves.
>
> -Chris
>

---

From: "Tony Li" <[email-masked]>
Date: Thu, 27 Nov 2008 01:11:31 -0800

Hi Eric,
 
|My point is that wherever one discusses distributed 
|enclaves map-and-encaps is usually involved.

Thanks for your reply.  I agree that for solving the particular problems
that you're confronting, map-&-encap is appealing.  Because it is apparently
easy it would seem to have immediate attractiveness.  Real estate brokers
call this "curb appeal".

Unfortunately, for those of us working a bit deeper into the architecture,
our primary concern should be less about the exterior view of the edifice
and more about its fundamental underpinnings.  If the foundation of the
building is unstable, no amount of glitzy exterior is going to turn the
result into an acceptable outcome.

A relevant famous computer science quote is:  "Any problem in computer
science can be solved with another layer of indirection. But that usually
will create another problem."  (David Wheeler)

All of the Loc/ID split solutions are precisely adding another layer of
indirection and thus it behooves us to truly understand what problems we are
creating.

My concern is that if we turn towards map-&-encap solutions, where we can no
longer actually run hosts in the native Internet architecture (i.e., without
encapsulation), we have, in some sense, fundamentally given up on the
ability of the host to act as a first class element in that architecture.
Instead, it must now be shielded, protected from reality and encased in its
private bubble of non-reality.

Put another way: I see the strong appeal of a recursive architecture and I
embrace that portion of the goal.  However, all recursion starts with a base
case, and it troubles me that the host cannot be an element of that base
case.  It implies that the base is flawed and has limitations in its
functionality.  Noel likes to say that the measure of an architecture is its
ability to adapt to new requirements that are not yet forseen.  What new
requirements will arise in the base architecture where the fundamental lack
of host participation will cripple us?

Regards,
Tony

---

From: Christian Vogt <[email-masked]>
Date: Thu, 27 Nov 2008 15:34:53 +0200

> (as of yet) provide the control required for large-scale TE issues  
> [...]

Hi Chris -

On the other hand, RRG does have several host-based multi-homing
solutions that enable the network to exercise traffic engineering:
Six/One gives the network explicit traffic engineering control through
address prefix rewrites in routers.  Multi-path TCP gives the network an
implicit means for traffic engineering through bandwidth limits and
packet loss.  And both Six/One and multi-path TCP can be used within the
framework of a hostname-oriented network protocol stack.

The only argument that can be brought up against host-based multi-homing
is that it becomes effective only if a considerable portion of all hosts
supports it.  Even if networks can ensure that local hosts do support
multi-homing, there is still a dependency on remote hosts supporting it
as well.  However, the dependency on the remote side does exist also for
network-based multi-homing.  Like host-based multi-homing, network-based
multi-homing does require support on both sides of a connection.
Consequently, host-based multi-homing is IMO deployment-wise just as
feasible as network-based multi-homing.  And technically, host-based
multi-homing is IMO even preferable for two reasons:  (a) It is
inherently more robust because it does not add new single points of
failure.  (b) It is more reliable because only hosts can monitor the
availability of complete end-to-end paths.

FWIW:  The above considerations apply only to RRG's goal of enabling
multi-homing.  They do NOT apply to RRG's second goal of eliminating
renumbering.  Eliminating renumbering entirely is possible only with a
network-based solution.  And unlike enabling multi-homing, eliminating
renumbering is achievable with only unilateral support:  Six/One Router
Unilateral mode and NAT66 are example solutions.  Since the first goal
can IMO best be solved in hosts, whereas the second goal is achievable
only with a network-based solution, I was suggesting the dual approach
consisting of a host-based solution plus a network-based solution.

- Christian

---

From: "William Herrin" <[email-masked]>
Date: Thu, 27 Nov 2008 09:36:31 -0500

On Thu, Nov 27, 2008 at 4:11 AM, Tony Li <[email-masked]> wrote:
> Noel likes to say that the measure of an architecture is its

History suggests that's not a question we can answer until faced with
the unexpected requirements. Our crystal balls aren't that good.

History does, however, suggest that the more control you place
directly in the hands of the end-user's single PC, the greater
adaptability the system exhibits. This holds true whether the end user
exercises that control by writing code himself or merely selecting
which software he runs.

While this implies a host-based solution, it is not necessarily a
death knell for map-encap. Systems which follow strategy A1c/2c/3b
should be able to place the mapping brain (and hence the control) as
close to the host as desired for any given host.

In TRRP for example, individual hosts can, if desired, control their
maps directly. That dictates the behavior of remote ITRs trying to
reach them. TRRP also allows an individual host to upgrade its stack
in order to become an ITR, even if it has a mapped address, gaining
considerably more control over its network presence than it has now.
It even allows hosts to use base non-mapped addresses so that hosts
willing to accept renumbering can continue to perform as they do in
today's architecture.

My point is not to trumpet TRRP, but rather to point out that it's
possible to design a map-encap system in which the host is not
required to give up the ability to act as a first class element in the
architecture.

Regards,
Bill Herrin

---

From: "Flinck, Hannu (NSN - FI/Espoo)" <[email-masked]>
Date: Thu, 27 Nov 2008 18:01:55 +0200

Hello Christian

Could you please remind us again where the multi-path TCP draft is? I am
curious to know how the paths through the network are discovered and
made exclusive.

Best regards
Hannu

>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Christian Vogt
>Sent: Thursday, November 27, 2008 15:35
>To: Christopher Morrow
>Cc: Routing Research Group Mailing List; Noel Chiappa
>Subject: Re: [rrg] Maybe it's not "either-or": Considering 
>ahost/network-based solution pair
>
>
>On the other hand, RRG does have several host-based 
>multi-homing solutions that enable the network to exercise 
>traffic engineering:
>Six/One gives the network explicit traffic engineering control 
>through address prefix rewrites in routers.  Multi-path TCP 
>gives the network an implicit means for traffic engineering 
>through bandwidth limits and packet loss.  And both Six/One 
>and multi-path TCP can be used within the framework of a 
>hostname-oriented network protocol stack.
>
>The only argument that can be brought up against host-based 
>multi-homing is that it becomes effective only if a 
>considerable portion of all hosts supports it.  Even if 
>networks can ensure that local hosts do support multi-homing, 
>there is still a dependency on remote hosts supporting it as 
>well.  However, the dependency on the remote side does exist 
>also for network-based multi-homing.  Like host-based 
>multi-homing, network-based multi-homing does require support 
>on both sides of a connection.
>Consequently, host-based multi-homing is IMO deployment-wise 
>just as feasible as network-based multi-homing.  And 
>technically, host-based multi-homing is IMO even preferable 
>for two reasons:  (a) It is inherently more robust because it 
>does not add new single points of failure.  (b) It is more 
>reliable because only hosts can monitor the availability of 
>complete end-to-end paths.
>
>FWIW:  The above considerations apply only to RRG's goal of 
>enabling multi-homing.  They do NOT apply to RRG's second goal 
>of eliminating renumbering.  Eliminating renumbering entirely 
>is possible only with a network-based solution.  And unlike 
>enabling multi-homing, eliminating renumbering is achievable 
>with only unilateral support:  Six/One Router Unilateral mode 
>and NAT66 are example solutions.  Since the first goal can IMO 
>best be solved in hosts, whereas the second goal is achievable 
>only with a network-based solution, I was suggesting the dual 
>approach consisting of a host-based solution plus a 
>network-based solution.
>
>- Christian
>
>

---

From: "Tony Li" <[email-masked]>
Date: Thu, 27 Nov 2008 10:48:22 -0800

Hi Fred,

|I haven't been tracking the host-based discussions all that
|closely, but AFAICT they should be compatible with map/encap.

No doubt, but Occam's razor suggests that if we have one solution that
fulfills our requirements, that that would be preferable to the
juxtaposition of two solutions.

|The Internet today is entering a transition phase in which
|links with 1500 byte or smaller MTUs may soon be considered
|legacy equipment in need of replacement - especially within
|the core. 

The problem is less at the core than at the edge, where we are more
bandwidth constrained and the scale of upgrade is much harder and where we
are already talking about adding a significant overhead with the v6
transition.  And the transition (as with all transitions in networking)
never actually ends. 

|But, the mapping system gives the ETR a means for determining
|the set of RLOCs from which packets that use specific EIDs may
|originate.

Do you seriously think that an ETR is going to verify the source EID against
the source RLOC?

Even after significant efforts today, we can't get source address anti-spoof
filtering implemented to a significant extent.

|>- Is virtualization really the right approach?  It's been noted that
|>map-&-encap is basically equivalent to running the entire Internet
|inside of
|>a VPN, where the mapping function conveys the edge of the 'primary'
|>infrastructure that terminates the VPN tunnels.  Some find it
|>architecturally disquieting to run our most essential function
|virtually
|>when we find that we cannot run it natively.
|
|I'm not sure I get this; one of the key features of map/encap
|is to arrest (and preferably reduce) routing table sizes which
|AFAICT it does very nicely. Is that what you mean?

Not at all.  Let me see if I can try an aviation analogy.  Today we have
ample amounts of technology to construct UAVs that allow us to perform
numerous hazardous missions while the pilot is safe, secure and comfortable.
At the same time, we have most flights today where we actually spend
multiple pilots per flight physically at the controls.  One can easily
imagine that we will, over time, find it more efficient to transition more
air freight (e.g., FedEx) to UAV based flights.  However, are we really
going to move to passengers flown on UAV's?  In some ways (and I admit that
this is an imperfect analogy), map-&-encap asks our hosts to trust their
transport to a technology that they can no longer interact with.  How do we
manage the underlying network?  What happens when a host needs some
exceptional service from the network?  This seems problematic and
complicated, when the root architecture of the network must be truly simple
and elegant.

Tony

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Thu, 27 Nov 2008 10:57:35 -0800

> 
> |But, the mapping system gives the ETR a means for 
> determining the set 
> |of RLOCs from which packets that use specific EIDs may originate.
> 
> 
> Do you seriously think that an ETR is going to verify the 
> source EID against the source RLOC?
> 
> Even after significant efforts today, we can't get source 
> address anti-spoof filtering implemented to a significant extent.
> 
> 
>

Actually, as a point of fact, we have.  We've got 75-80% coverage, based
on studies presented at the *NOG forums.  That's significant.  It hass
dropped spoofed attacks from 50%+ of total attacks to less than 5% of
total attacks, IIRC.  Its MUCH easier today to just own a few tens of
thousands of hosts and not to bother spoofing.

I don't see why we can't enforce anti-spoofing on encap, since it
requires a mapping to be in place.  And we can on decap easily where the
traffic is symetrical (that is there is a mapping for the source
RLOC/EID pair.

-Darrel

---

From: Christian Vogt <[email-masked]>
Date: Thu, 27 Nov 2008 21:00:59 +0200

Hi Hannu,

I would take a look at the following paper:

The Resource Pooling Principle. Mark Handley, Damon Wischik, Marcelo  
Bagnulo Braun.
http://ccr.sigcomm.org/online/files/p47-handleyA4.pdf

Note that the work on multi-path TCP is still in its early stages, so
you won't find a full-fledged specification.

- Christian

---

From: "Tony Li" <[email-masked]>
Date: Thu, 27 Nov 2008 11:11:26 -0800

|I don't see why we can't enforce anti-spoofing on encap, since it
|requires a mapping to be in place.  And we can on decap easily 
|where the
|traffic is symetrical (that is there is a mapping for the source
|RLOC/EID pair.

Ok, but what about asymmetric decap?  That implies that the ETR does a
mapping lookup on the receipt of a packet, buffers the packet until the
lookup succeeds, and the does the compare.  This seems like another fine way
of DoSing the ETR.

Tony

---

From: "Christopher Morrow" <[email-masked]>
Date: Fri, 28 Nov 2008 02:43:06 -0500

On Thu, Nov 27, 2008 at 8:34 AM, Christian Vogt
<[email-masked]> wrote:
>> (as of yet) provide the control required for large-scale TE issues [...]
>
> Hi Chris -
>

sure

> supports it.  Even if networks can ensure that local hosts do support

and if a considerable number of sessions would benefit from it. If a
lot of the traffic is 30-50 (probably they are smaller than that
actually) packet flows I'm not sure host-based makes a lot of sense
with respect to the overhead associated with discovering paths,
performing TE decisions, etc... Also, keep in mind that there may be
some boundaries in administration to be aware of with respect to hosts
and network traffic. (network admin people manage traffic,
system-admin folks manage hosts, in someplaces this is a challenge)

of course this is also dependent upon the implementation on the hosts,
but so far the ones I've read will require some discovery and setup.

> multi-homing.  They do NOT apply to RRG's second goal of eliminating

fair enough.

> network-based solution.  And unlike enabling multi-homing, eliminating

ok.

> Unilateral mode and NAT66 are example solutions.  Since the first goal

nat66 or nat(anything) is a little scary in that it implies (to me)
the injection of lots of state into the 'network' (at least some
points in the network) and thus introducing large amounts of
complexity and troubleshooting complications. I agree something like
NAT would let you avoid renumbering (and that's one of the uses today
for NAT) but there is a cost for this. At the individual enterprise
level that may be cost effective, at the ISP level I'm not sure it's
cost effective.

> consisting of a host-based solution plus a network-based solution.

Sure... my point really was that there seem to be equal reasons for
each, and ratholing on just one seems counter productive in the long
term.

One strategy to permit the network to scale in the face of 'more
addresses' or 'more networks' or 'more multihomers' is great. Another
strategy to be used for smaller-scale multihoming like
home-network-based multihoming (cell, wifi, broadband-cable) also
seems like a good thing.

-Chris

---

From: "Mark Handley" <[email-masked]>
Date: Fri, 28 Nov 2008 10:47:08 +0000

We don't yet have a full draft spec, but we do have an implementation that
we're currently debugging and testing.  A spec will follow, once we're more
certain we understand all the more subtle interactions.

As for path discovery - there's the simple solution and the more complex
solution.

For now, we're assuming that at least one of the hosts is multi-homed, and
we simple use the path diversity that comes from this.  It certainly doesn't
guarantee the paths stay separated, but if you link the increase/backoff
algorithms in the right way, the hope is (and some of the theory dictates)
that the aggregate of the multiple subflows simply ends up behaving like a
regular TCP flow if the bottleneck is in the common part of the paths.  Thus
there's no requirement to enforce that the paths are exclusive, or even to
detect if they are.  If both ends are mult-homed, then you have more
possible paths to choose from, and more probability of finding one that is
uncongested or survives a network failure.

However, once you have multi-path capable transport, then this opens up the
option of doing smarter things for path discovery to try and ensure
diversity.  The goal then is to specify these mechanisms in such a way that
they can operate self-contained with whatever addresses and paths they're
given, or to give better results when you add extra path choice mechanisms
in the future.

Overall though, it remains a research question as to how much path diversity
is needed overall to realize the benefits of resource pooling, and this is
what we are currently starting to examine.

 - Mark

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 28 Nov 2008 10:29:43 -0800

Hi Tony,

>Sent: Thursday, November 27, 2008 1:12 AM
>Subject: Re: [rrg] Map and Encaps
>
>
>Hi Eric,
>

With RANGER/VET/SEAL, hosts can play in several ways:

  1) They can connect to a link behind an ITR/ETR
  2) They can perform encapsulation/decapsulation themselves
  3) They can connect directly to the native IPv4 Internet in
     Exactly the same way as they do today.
  4) They can connect to an IPv4 enterprise, and connect to
     the IPv4 Internet through a NAT in exactly the same way
     as they do today.

So, we don't see a full departure from the current architecture
at all. To the contrary, we see the extant IPv4 model carried
forward in next-generation enterprises and we see IPv6 coming
on line to satisfy the address scaling as well as map-encaps
coming on line to satisfy the routing scaling.

Fred

---

From: Christian Vogt <[email-masked]>
Date: Fri, 28 Nov 2008 20:36:11 +0200

Chris -

I agree with you that the overhead created by a host-based solution
needs to be carefully looked at.  Yet I don't think that high overhead
is inevitable for host-based solutions.  As you say, it depends on
protocol design.  Take, e.g., the overhead that is needed for
connectivity verification:  Whereas host-based solutions cannot
aggregate explicit probing messages as efficiently as network-based
solutions, host-based solutions can potentially use implicit probing
more efficiently than network-based solutions because hosts have session
state that may include useful connectivity information.  So the
overhead-efficiency of a host-based solution depends to a significant
extent on how effectively the solution exploits such session state.

> Also, keep in mind that there may be some boundaries in administration
> to be aware of with respect to hosts and network traffic. (network  
> admin
> people manage traffic, system-admin folks manage hosts, in someplaces
> this is a challenge)

Isn't that a challenge only if the host configurations needs to be
coordinated with the network configuration?  A good protocol design
would IMO avoid such a dependency between the hosts and the network.

> the
> injection of lots of state into the 'network' (at least some points in
> the network) and thus introducing large amounts of complexity and
> troubleshooting complications. [...]

Note that NAT66 (as well as Six/One Router Unilateral mode) is stateless
because it uses one-to-one address mappings.  It does need to be
configured with the pair of internal and external routing prefixes, but
there is no per-session state.  Hence I consider these solutions in fact
quite feasible.  Sure, their lack of transparency is a disadvantage. But
this is a phenomenon that is being dealt with already today.

> each, and ratholing on just one seems counter productive in the long

Yes.  We are definitely on the same page here.

> seems
> like a good thing.

Yes, good point.  And again this implies that RRG should not focus on
finding a one-size-fits-all solution.  RRG should rather develop a
"solution toolbox", including both host-based and network-based tools,
so that different causes for the routing scalability problem can be
addressed separately, and hence in general more effectively.

- Christian

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Fri, 28 Nov 2008 11:52:01 -0800

> |requires a mapping to be in place.  And we can on decap easily where 
> |the traffic is symmetrical (that is there is a mapping for the source

> |RLOC/EID pair.
> 
> 
> Ok, but what about asymmetric decap? 

As some one I know in the Industry likes to say 'there is no free lunch
here' ;-)

> That implies that the 
> ETR does a mapping lookup on the receipt of a packet, buffers 
> the packet until the lookup succeeds, and the does the 
> compare.  

Oh you mean like the IPv6 neighbor discovery process!?  

>This seems like another fine way of DoSing the ETR.
> 

Of course, so implementation has to be smart. 

If the outer header isn't spoofed, and the inner is, its trivial to find
the culprit.  Once you've determined the offending site, you have a
whole lot of interesting options - including using the control plane to
inform that site that he's dos'ing you and asking it to stop, or
building an ACL to block that particular source.  Post decap, this
spoofing is no different than what we have today.  If he outer header
and inner header source are both spoofed, and the flows are
asymmetrical, I'd suspect you need to have an implementation that can do
something like

1) turn on some sort of verification function and drop packets while it
waits to verify them
or
2) simply pass the decapsulated traffic to some ddos scrubbing box like
you'd do with any spoofed attack today.

The key, as you point out, is simply that the implementation doesn't
allow itself to be dos'd by traffic.  And this goes for encap/caching as
well as decap.

Regards,

-Darrel

---

From: "Tony Li" <[email-masked]>
Date: Fri, 28 Nov 2008 12:15:32 -0800

|> Ok, but what about asymmetric decap? 
|
|As some one I know in the Industry likes to say 'there is no free lunch
|here' ;-)

Or, in other words, it's just not going to happen.

|> That implies that the 
|> ETR does a mapping lookup on the receipt of a packet, buffers 
|> the packet until the lookup succeeds, and the does the 
|> compare.  
|
|Oh you mean like the IPv6 neighbor discovery process!?  

Two wrongs don't make a right.

|>This seems like another fine way of DoSing the ETR.

|Of course, so implementation has to be smart. 

Step 1 is going to be not doing this.

;-(

Tony

---

From: "Templin, Fred L" <[email-masked]>
Date: Sat, 29 Nov 2008 11:08:06 -0800

>Two wrongs don't make a right.

Why buffer the packet until the lookup succeeds? Why not
just accept the first few packets while a lookup is done
in parallel then, if subsequent packets appear to be
coming from an incorrect ITR, just start dropping? That
way we can defeat *sustained* DOS attacks, which are
the only ones we really care about anyway.

Fred

---

From: "Christopher Morrow" <[email-masked]>
Date: Sat, 29 Nov 2008 14:48:01 -0500

> just accept the first few packets while a lookup is done

a synflood is a bunch of 1 packet flows :( you lose, I win! yippee! :(
Seriously though, if you send through 'some' of the bad packets all
the attacker has to know is how many 'some' is... in the worst case
the answer is 'one'.

Buffering is bad, really, really bad.

-chris

---

From: "Templin, Fred L" <[email-masked]>
Date: Sat, 29 Nov 2008 14:57:50 -0800

Hi Chris,

>From: Christopher Morrow [mailto:[email-masked]]
>Sent: Saturday, November 29, 2008 11:48 AM
>To: Templin, Fred L
>Cc: [email-masked]; Darrel Lewis (darlewis); Routing Research Group
Mailing List
>Subject: Re: [rrg] Fundamental objections
toahost-basedscalableroutingsolution
>the attacker has to know is how many 'some' is... in the worst case

Still, AFAICT performing egress filtering of some sort during
decaps could be used to the ETR's advantage in establishing a
pattern of behavior from certain ITRs. In particular, it could
be used by the ETR to determine which ITRs are not correctly
implementing ingress filtering - right? 

>Buffering is bad, really, really bad.

Buffering on decaps does sound pretty bad, but buffering on
encaps may be a different story. I haven't looked at the
mapping schemes all that closely, but at least the APT
approach seems to have the ITR send to a "default mapper"
with side-effect of getting a mapping resolution in return.
That sounds great, but it essentially puts "default routers"
in the DFZ. Is that better or worse than buffering?

Fred

---

From: "Christopher Morrow" <[email-masked]>
Date: Sun, 30 Nov 2008 14:18:09 -0500

> implementing ingress filtering - right?

with some level of logging/stats-collection on the ETR you might be
able to determine that an ITR is improperly encapping, yes... though
it's possible that the ITR is performing encap but not decap for the
networks behind it so you may not be able to tell if this is a problem
or not.

> encaps may be a different story. I haven't looked at the

The encap buffering problem is 'solved' by encaping only when you have
a map for the destination, no? else you drop the 'first packet' (which
maybe multiple first packets certainly) while you await the mapping
reply.

> in the DFZ. Is that better or worse than buffering?

it sounds like default-mappers can get really busy, and cost some
stretch while maybe not forcing 'first packet drop'... depending on
the network that might not be a bad thing.

-Chris

---

From: "Templin, Fred L" <[email-masked]>
Date: Sun, 30 Nov 2008 13:18:26 -0800

Hi Chris,

>Sent: Sunday, November 30, 2008 11:18 AM
>To: Templin, Fred L
Mailing List
>the network that might not be a bad thing.

But, how bad that would be? In normal use, default mappers
would only have to forward the first packet out - not a
sustained stream of packets.

I'm wondering if there is a parallel to be drawn between
default mappers and the root domain name servers. Is there
some comparison to be drawn that could indicate anticipated
scaling properties? Maybe the APT people have thought about
this more and could comment?

Fred

---

From: "Christopher Morrow" <[email-masked]>
Date: Sun, 30 Nov 2008 17:41:09 -0500

> sustained stream of packets.

This really gets at 'where are the default mappers deployed' and 'what
is the traffic mix for the network deploying the default mapper(s)'.

you can minimize stretch, and lookup lag by pushing default mappers to
as many 'pops' as possible (thinking the network heirarchy is
something like: network -> region -> metro -> pop).

you can minimize mapping punts, at steady state, by keeping more map
entries 'local', but... you will still have a limited amount of the
map you can store on any one 'router' though, so if the traffic mix is
such that a router sees more destinations than their map space can
hold you'll always be sending some traffic through the default mapper.

For example say your map is only 100 items long, yet your network
sends to 150 locations per time period. In that time period you'd be
doing 50% default-mapper punts... If this default mapper is deployed
at the 'network' level you're going to get a lot of traffic punted
toward it...

There are places where default mappers are going to be the wrong
solution, there are places where they may fit the bil very nicely,
perhaps even in the same network. Say a network that has a sizable
dialup/dsl plant and wants to limit costs on the L3 devices there, but
has sizable routers/traffic through the network core and can spend on
mapping memory in L3 devices at that point, perhaps default mappers
inside the dial/dsl plant makes sense but not for the network as a
whole.

Anyway, my point wasn't that one solution is better than the other,
but that each tool has to be used properly, and we may want more than
one tool here.

> default mappers and the root domain name servers. Is there

I would bet only slightly since the lookup load is much lower on DNS
boxes (I think it is at least) than it is on routers. (per packet vs
per TTL)

> this more and could comment?

hopefully they can :)

-Chris

---

From: Michael Meisel <[email-masked]>
Date: Sun, 30 Nov 2008 15:33:53 -0800

>> this more and could comment?

Hi everyone, one of the APT people here. =) So we actually did an
analysis of potential default mapper load a while back, based on real
traffic traces at two POPs that serves mostly academic networks, taking
into account the RTT to the default mapper and the cache size and
expiration time. Even in the worst case, the results show that the load
on the default mappers is pretty small.

If you think about it, an academic network is actually a pretty good
test case here -- college campuses have residential customers (dorms),
administrative offices, and research labs, so they may have some of the
broadest spread of destinations out there.

The details and plots are in our technical report, which you can find
here (scroll to section 5):

http://fmdb.cs.ucla.edu/Treports/080004.pdf

Luigi Iannone and Olivier Bonaventure also did some work on caching
performance (for LISP, in this case) which showed similarly promising
results:

http://inl.info.ucl.ac.be/system/files/Conext-2007-CRV-UCL-v2-Clean.pdf

-Michael