Subject: [RRG] Map-encap space for "server" vs. "client" end-users?

From: Robin Whittle <[email-masked]>
Date: Sat, 16 Feb 2008 14:18:15 +1100

Hi Bill,

In "Re: [RRG] Why delaying initial packets matters - complex web
pages & real-time P2P" you wrote:

> It seems to me that one assumption which has run heavily through
> this thread is that man-n-encap would entirely replace the
> existing system where the EID and LOC are the same. It strikes me
> that not only is this unlikely, it would be undesirable as well.
>
> I won't spend a lot of time on why outright replacing BGP is
> unlikely. Suffice it to say that anything the size of the
> Internet has a great deal of inertia and no map-n-encap scheme
> discussed so far has enough punch behind it.

I intend Ivip's address space to be so attractive to end-users (who
want or need multihoming, TE and portability) that it will be very
widely adopted - ideally becoming the usual form of address space
for end-users.  I think we need the new map-encap space to be
attractive to end-users large and small, established and new:

  http://psg.com/lists/rrg/2008/msg00275.html

None of these map-encap schemes replace BGP.  They rely on the BGP
system for directly handling the "RLOC" address space which is used
by all ISPs and by end-user networks which for one reason or another
are not using the map-encap scheme.  All traffic between ISP
networks still relies on the BGP system.  The map-encap scheme
provides a way of dividing the address space down into smaller
chunks (micronets AKA EID prefixes), each individually handled by
any ETR (Ivip) or one or more ETRs (LISP and APT) in the world.

Each such division does not directly burden the BGP control plane,
other than to the extent to which Mapped Address Blocks (MABs) are
advertised in BGP, as they are in Ivip and LISP with "Proxy Tunnel
Routers".  The idea is that each MAB provides many micronets for
many end-users, so this burden on the BGP control system will be
small compared to the burden of serving that number of end end-users
with conventional "RLOC" space which is managed by BGP alone.

> I will spend a couple paragraphs on why a hybrid system is a
> desirable outcome.

I am not sure what you mean by "hybrid", but I think you mean some
end-users are better off with the current BGP-managed approach to PI
space while others are better off, or will be encouraged/forced to
use map-encap space.  This is very different from what I am
proposing with "hybrid push-pull" (APT and Ivip) and from the NERD
concept of "hybrid" in which ITRs are full database for some subset
of the EID space and rely on a global query server system to be a
caching ITR for the rest of the EID space.

> There are relatively few scenarios in which a client machine
> derives a benefit from a static IP address assignment. Mobility,
> at least with respect to long-lived connections. The occasional
> firewall admin where inbound access to a resource has been
> restricted by source IP address. Not a whole lot else.

OK - in part because widely used client applications are adapted to
operating behind NAT, where the public IP address may change from
time to time, as with the user's DSL/cable modem getting a new address.

But I think it would be best to have a map-encap scheme which didn't
require distinctions to be made between different methods of using
the address space, other than the dichotomy between space used by
ISPs for ETRs and other functions (including for PA space) and for
all other organisations and individuals ("end-users").

> Given a choice between a mild slowdown and the temporary
> assignment of an IP address from a highly aggregated pool, most
> clients would elect to use the temporary assignment.

There are more choices than between ALT (mild slowdown) and pushing
the "client" subset of end-users into dynamic addresses from RLOC space.

I am sure there are many end-users, large and small, who want
stable, portable, multihomable address space to run both servers and
client hosts on.  So trying to decide which end-users are "server"
end-users and which are "clients" would be difficult or impossible.

For instance, there is likely to be a huge demand for mobility with
persistent connections due to the device/network retaining the same
IP address no matter which network it uses for its care-of address -
with both IPv4 and IPv6, with generally optimal path lengths.
Cellphones and mobile PCs (probably the same thing in the future)
are the most obvious use for this.  Ivip is intended to provide this
form of mobility.

> Servers are a whole other can of worms. There are spam filters
> which block and permit by IP address. There's lots of crappy
> software which doesn't respect DNS TTLs. Renumbering servers is a
> difficult and fault-ridden process. Given a choice between an
> extra delay equivalent to a DNS lookup on every request and
> having to renumber, more than a few server administrators would
> choose the extra delay... Especially for email servers and other
> devices that move data in bulk where the delay simply doesn't
> matter.

Yes, I think for email servers the initial packet delay is not a
problem.  For web servers, game servers etc. having a bunch of
these, especially for a bunch of customers, would be extremely
disruptive and expensive for the end user - the hosting or
self-managed server company - and their customer end-users.

However, given the choice of keeping or attaining conventional space
(BGP-managed "RLOC" space) and changing over to a map-encap type of
address space which involved almost any perceivable degradation
(reliability, initial packet delay etc.) I think most existing and
new "server hosting" companies would avoid the new system like the
plague.  Competitors with the original, authentic, good-ol-days RLOC
space would be pointing out the superior responsiveness of their
servers compared to the slower nature of the new-fangled plastic
map-encap space.

> The conclusion, it seems to me, is that the desirable outcome is
> a hybrid system in which a fellow in Operations can consider both
> PI space and Fast space, and then choose the one best suited to
> his particular application.

The RRG needs to decide what sort of map-encap scheme to recommend
to the IESG for engineering development.  I assume we will be aiming
for consensus that one scheme alone should be developed.

I think the aim is to have a map-encap scheme which will be as
widely adopted by end-users as possible.  This includes end-users of
all sizes, existing and new.  Ideally, virtually all end-users would
use the space, leaving only the ISP's spaces (and the MABs) directly
managed by BGP.

I don't think any of the current schemes make conventional PI space
intrinsically less attractive.  However it is becoming increasingly
hard to obtain due to the IPv4 address shortage and concern about
the burden it places on the DFZ routing table.

I assume that by "Fast space" you mean the map-encap type of space.
 I don't think it will be "faster" than conventional RLOC space, in
terms of packet delivery - but it would probably be faster and
certainly cheaper and easier to alter its management for
multihoming, TE and portability.

The choice may be between:

  LISP-NERD  Full push - looks hard to sustain full database push to
             every ITR on the planet in the long term.

  LISP-ALT   Full pull - looks complex and unreliable, with long
             paths and delays for queries == initial traffic
             packets.  (The ALT network could be tweaked with more
             meshiness to reduce typical, but not worst-case, long
             paths due to the current focus on address aggregation
             over geographically short paths.)

  LISP-ALT with caching.  Typically improved mapping response times
             but is initial packet delivery any faster?

  Mix of NERD and ALT - but why bother with ALT if you can push the
             full database to even a handful of ITRs.  Put a query
             server at those sites and then the ALT network is not
             needed.

  TRRP       http://bill.herrin.us/network/trrp.html
             Pull via a DNS-like global distributed query server
             network.  Instead of ALT's circuitous path problem,
             TRRP has more direct paths to the query servers - but
             a similar delay problem arises due to the likely need
             for multiple recursive queries to find the
             authoritative query server.

  APT        Hybrid push-pull.  Full database ITRs and query servers
             are integrated into a Default Mapper.  All
             participating ISPs have a few of these, with a slow
             push scheme (separate BGP-like flooding system).  No
             support for non-upgraded networks (Ivip's anycast ITRs
             in the core == LISP's Proxy Tunnel Routers) - so it is
             not incrementally deployable.

  Ivip       Fast push (I will write an ID on this ASAP) to full
             database query servers and ITRs located anywhere - so
             it is highly flexible.

             Caching ITRs and optional caching ITR functions in
             sending hosts (not behind NAT) access the local query
             server(s) and get responses faster, less expensively
             and more reliably than with a global query server
             network (TRRP or ALT).   Optional caching query servers
             can be used as well, which query the full database
             query servers directly, or through other caching query
             servers.  Query servers respond with mapping
             information and a cache time, and send notifications to
             the querier (ITR or caching query server) if the
             mapping changes in that time.

             Incrementally deployable, fast mapping (also an ITR
             can let an initial packet go - it will be forwarded
             to a full database ITR) and capable of supporting
             a new form of mobility.

None of these are "faster" than current PI space for handling
packets, but those which rely on a global query server network will
be significantly slower with some initial packets.

> In other words, whether you favor LISP or IvIP or TRRP, your
> system should be designed as a supplement to BGP, not a
> replacement. A supplement which serves to enable PI space.

End-users with current PI prefixes could convert them into
individual MABs with Ivip (or use the space with any other scheme).
 Then, if they had been advertising the space in multiple longer
prefixes, the space could be a single MAB advertisement under BGP,
reducing the size of the DFZ routing table.  Then they could split
it up into as many micronets as they like, which they can't do at
present, due to the 256 IPv4 address granularity limits of the BGP
system.

I have been using "PI" to refer to conventionally BGP-managed PI space.

The new map-encap scheme provides Provider Independent space too, so
if you include map-encap-managed space as part of "PI", then I think
all the schemes accord with your last two sentences.

  - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "William Herrin" <[email-masked]>
Date: Fri, 15 Feb 2008 23:36:29 -0500

On Feb 15, 2008 10:18 PM, Robin Whittle <[email-masked]> wrote:
> use map-encap space.

Hi Robin,

By hybrid, I mean a system where the shorter, typically PA prefixes
continue to be routed via BGP as they are today while the longer,
frequently PI or TE prefixes are shifted to map-n-encap as driven by
the system's economics.

> end-users and which are "clients" would be difficult or impossible.

That's exactly my point: the choice should (and ultimately will) be
made by the folks purchasing Internet service. Our job shouldn't be
trying to figure out what they should or must do; it should be giving
them as many quality parallel options as we reasonably can so that
they can select the best fit.

> For web servers, game servers etc. having a bunch of

I disagree and I'll tell you why: given the option to reference
subservers by DNS name or by IP address, web site designers have
overwhelmingly chosen to reference them by name and accept the extra
lookup delay. This is the preference that the operations folk have
clearly demonstrated to us with their actions - that on initial access
to a server, a mild lookup delay is acceptable for the sake of
convenience.

> I assume that by "Fast space" you mean the map-encap type of space.

The opposite. "Fast" space is highly aggregated PA space which runs as
bare IP over the backbone without going through a man-n-encap scheme
and thus suffering no lookup delays or slow initial pathing.

> I think most existing and

What's wrong with that? If you're placing a /18 on the 'net, we can
easily accommodate you with the hardware that's already deployed, let
alone with hardware currently shipping.

I'm more interested in the guy who has 5 servers he wants to put on
the 'net and he never wants to have to deal with renumbering.
Map-n-encap would give him an option he doesn't have today, an option
he wants to have. That's where the initial market for map-n-encap is:
PI space for guys that are otherwise too small to have it.

Once map-n-encap exists, the definition of "too small" grows until it
correctly balances with the needs of the folks operating the DFZ.

>              authoritative query server.

I think ALT is actually pretty clever. Following that idea, its
possible to create a mechanism in TRRP to route initial packets while
waiting for the lookup to complete. That could strongly mitigate that
lookup delay. Unfinished work (read at your own risk) here:
http://bill.herrin.us/network/trrp-aapip.html

Another interesting point, which didn't make it to in the unfinished
document, is that this alternate initial route rather lends itself to
an accompanying classic BGP route which greatly eases the transition
from the current BGP-only system to a hybrid BGP+map-n-encap system.

Regards,
Bill Herrin

-- 
William D. Herrin                  [email-masked]  [email-masked]
3005 Crane Dr.                        Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Robin Whittle <[email-masked]>
Date: Thu, 21 Feb 2008 00:00:57 +1100

Short version:

   I still think the new map-encap space has to be free of stuff
   like significant initial packet delay, since we have to make it
   attractive to large and small end-users.  I would appreciate
   some debate of my message about this:

     Large companies with PI, small with map-encap space?
     http://psg.com/lists/rrg/2008/msg00275.html

Hi Bill,

You wrote, in part:

> By hybrid, I mean a system where the shorter, typically PA
> prefixes continue to be routed via BGP as they are today while
> the longer, frequently PI or TE prefixes are shifted to
> map-n-encap as driven by the system's economics.

OK - this means the map-encap address space does not completely
replace conventional BGP-managed prefixes.  However I am confused
with your use of "PA", "PI" and "TE" here.

I am using the term "hybrid" only as part of "hybrid push-pull",
which means something else.

>> stable, portable, multihomable address space to run both
>> servers and client hosts on.  So trying to decide which
>> end-users are "server" end-users and which are "clients" would
>> be difficult or impossible.
> 
> That's exactly my point: the choice should (and ultimately will)
> be made by the folks purchasing Internet service. Our job
> shouldn't be trying to figure out what they should or must do; it
> should be giving them as many quality parallel options as we
> reasonably can so that they can select the best fit.

Yes, however many options there are, if we want these options to be
widely used - which we do to solve the routing scaling problem -
then we have to ensure the end-users who adopt these options gain
direct, immediate, benefits.

>> For web servers, game servers etc. having a bunch of these,
>> especially for a bunch of customers, would be extremely 
> overwhelmingly chosen to reference them by name and accept the
> extra lookup delay. This is the preference that the operations
> folk have clearly demonstrated to us with their actions - that on
> initial access to a server, a mild lookup delay is acceptable for
> the sake of convenience.

OK.  Renumbering a bunch of web servers also means having to alter
the records in a bunch of nameservers - not all of which would be
controlled by the hosting company.

Quite a few web servers host dozens of sites, so renumbering one
such server means chasing around and individually changing zone
files in dozens of name servers, many of which are not going to be
accessible to the hosting company.  That means the company whose web
site it is needs to get involved, which makes the hosting company
look bad, leads to lots of support calls etc.

> The opposite. "Fast" space is highly aggregated PA space which
> runs as bare IP over the backbone without going through a
> man-n-encap scheme and thus suffering no lookup delays or slow
> initial pathing.

Not all map-encap schemes involve such delays!

>> I think most existing and new "server hosting" companies would
>> avoid the new system like the plague.  Competitors with the
>> original, authentic, good-ol-days RLOC space would be pointing
>> out the superior responsiveness of their servers compared to
>> the slower nature of the new-fangled plastic map-encap space.
> 
> What's wrong with that? 

As Marshall Eubanks wrote in response to Jari Arkko:

   http://psg.com/lists/rrg/2008/msg00381.html

    However, it is not enough to work; it also has to be salable.

    I don't think that any system that imposes significant user
    delays will be commercially viable if other options are
    available, and it is hard to see how they would not be.

    If I represent a service provider without these delays, and
    you represent one with these delays, I can guarantee you that
    you will not like my marketing materials pointing this out,
    and telling potential customers something like "ask how long
    it takes Jarinet to connect to the web." I have been on the
    loosing end of such campaigns; it is hard to fight simple
    facts with complicated explanations of why said facts are
    irrelevant most of the time.

    So, the delays cannot be "significant," which I would regard
    as meaning "being noticeably larger than typical DNS delays."
    Any such solution with significant user delays will be hammered
    in the marketplace.

    This is indeed likely to impose significant engineering
    constraints.

> If you're placing a /18 on the 'net, we
> can easily accommodate you with the hardware that's already
> deployed, let alone with hardware currently shipping.

I think you are saying that larger end-users might as well use the
conventional techniques we use today - a BGP advertised prefix.

> I'm more interested in the guy who has 5 servers he wants to put
> on the 'net and he never wants to have to deal with renumbering. 
> Map-n-encap would give him an option he doesn't have today, an
> option he wants to have. That's where the initial market for
> map-n-encap is: PI space for guys that are otherwise too small to
> have it.

I agree there is a huge market for this, but I believe we need to
make the map-encap address space equally attractive for all end
users, large and small, established, new and growing.

> Once map-n-encap exists, the definition of "too small" grows
> until it correctly balances with the needs of the folks operating
> the DFZ.

If map-encap space had deficiencies, such as frequent delays in
initial packets, then the big end-users (who have a choice whether
to adopt it or not) would avoid it like the plague.  Only the
smaller end-users, who can't afford or justify one or more /24s of
conventional BGP-managed PI space, would be choosing map-encap space
- because they have no other option.

I argue that the new address space must have no deficiencies which
affect end-users, and that it must be attractive to end-users of all
sizes:

  Large companies with PI, small with map-encap space?
  http://psg.com/lists/rrg/2008/msg00275.html

I would be glad if someone debated this.

  - Robin          http://www.firstpr.com.au

---

From: "William Herrin" <[email-masked]>
Date: Wed, 20 Feb 2008 11:21:05 -0500

Hi Robin,

Provider Aggregatable prefixes tend to be short enough that the
economics of the current BGP system make sense. Further, everybody who
wants to announce a PA prefix already can and does. If it ain't broke,
don't fix it.

Provider Independent and Traffic Engineering prefixes tend to be
longer and there are more of them putting pressure on the system.
Worse, engineering an anycast service on a single IP address or
providing provider independent space for a small number of servers
can't be done at all because of the filtering cutoffs that were
necessary to keep the system stable. This is where the system is
failing us, ergo this is a part we need to fix.

>  controlled by the hosting company.

Yes, this is among the many reasons why provider independent space is
such a desirable thing.

My point was in a different direction: content publishers already
tolerate delays on the same scale as the difference we're talking
about between push and pull. They tolerate these delays today solely
for the sake of their developers' convenience. There is simply no data
to support the conclusion that they wouldn't tolerate the same
character and scale of delay for the sake of their system
administrator's convenience.

The notion that a pull-based map system will necessarily be too slow
ain't nothin' but a theory and the theory is both unsupported by any
data and apparently contradicted by some of the data that is
available.

>  conventional techniques we use today - a BGP advertised prefix.

Yes. The worldwide cost of one advertised prefix works out to
something in the neighborhood of US $8000/year and falling. The system
is only broken for prefixes whose value is less than that.

>  - because they have no other option.

This is a bad thing? As an (er hem) larger gentleman, I can tell you
without reservation that one-size-fits-all never does.

>   Large companies with PI, small with map-encap space?

I would put it differently. Just 'cause I can't afford a Porsche
doesn't mean I care to ride the bus. Bare BGP prefixes for those who
cough up the cash. Map-encap for those who value it less but still
want something that isn't as ghetto as slices of PA space.

Regards,
Bill Herrin

---

From: Robin Whittle <[email-masked]>
Date: Thu, 21 Feb 2008 12:11:43 +1100

Hi Bill,

> Yes, this is among the many reasons why provider independent space is

We all agree on the need for more end-users to be able to keep their
addresses, meaning fully portable PI space, but by a different
mechanism than the current BGP approach.

> administrator's convenience.

If the entire Net changed to add some small delay, then this would
not be a barrier to adoption of the new map-encap scheme.  However,
as I and others have written, if the new address space performs
worse, in a way which really matters to end-users or which could be
construed by marketing people as such, then it will be very hard to
get widespread adoption of the map-encap scheme.

Yet I think we need this very widespread adoption for end-users
large and small if the system is to be effective at solving either
the routing scalability problem or the closely related problem of
improving IPv4 address utilization.

> data and apparently contradicted by some of the data that is

The ALT scheme is not just a global query server system (tending to
be slow, long-path and less reliable than using local query
servers), but is worse still due to the aggressive address
aggregation causing path lengths often far longer than the
geographic direct path.  See the thread "ALT's strong aggregation
often leads to *very* long paths".  Nobody disagreed, but there was
talk of more meshy cross-linking to give a better chance of a
geographically short path.  K. Sriram's diagram:

  http://www.antd.nist.gov/~ksriram/strong_aggregation.png

is also instructive.

TRRP has a similar delay problem, which would occur frequently
enough to be significant.  The ITR has to sequentially query and get
responses from several DNS-like servers in order to find the
authoritative server with the mapping information.  Unless you
anycast those intermediate servers (which does not scale well) then
there will be significant delays in quite a few circumstances.  This
seems to be far worse with IPv6, where each level in the DNS-like
hierarchy is a smaller number of bits, and there are more bits in
the EID address than with IPv4.

To some extent you could solve this by sending back a response,
giving authoritative nameservers for several levels of subdomain -
which enables the ITR to jump a few levels along its path of enquiry
without sending another query and response.  But the inherent
problems of global query server systems still remain in TRRP, even
if you could optimise it significantly.

Only anycasting most or all of the TRRP DNS-like nameservers so all
the queries were answered by  a somewhat local (same country or
continent) server would significantly remove the delays inherent in TRRP

Even if we think the delays are insignificant, they can still
prevent the map-encap scheme being adopted, as Marshall Eubanks wrote:

>From the point of view of large end-users with cash to burn, the
system isn't broken.  However, as the cost falls and more end-users
gear up to get this kind of space, the system becomes more broken
from the point of view of the companies who run DFZ routers, due to
the bloat in the number of DFZ routes - which is the primary problem
we are here to fix.

I argue that the new map-encap space must be equally attractive to
large and small end-users, including those with current BGP-managed
PI space:

No-one has yet debated this.

> without reservation that one-size-fits-all never does.

Ah yes, but many folks are establishing their nook in the Net with
the intention of growing enormously.  We need them to adopt
map-encap space - which means they need to be confident it will
still the best arrangement when they are bigger.

> want something that isn't as ghetto as slices of PA space.

This might work to some extent, but not if the price of BGP-managed
PI space comes down.

I think it strongly desirable that the new architecture provide a
form of address space which is really attractive to all end-users.
I think this is a feasible goal.

  - Robin

---

From: "William Herrin" <[email-masked]>
Date: Thu, 21 Feb 2008 02:23:23 -0500

>  get widespread adoption of the map-encap scheme.

Robin,

In the unlikely event that marketing folk convince users that a slice
of PA space is better than map-encap PI space, the RIB scalability
problem is solved and our job is done. The reason we're having this
discussion is that folks -aren't- satisfied with slices of PA space
and have been impinging on the RIB in any way they can connive.

>  there will be significant delays in quite a few circumstances.

Few if any of which will have a more meaningful impact than a DNS
query today. The users have spoken clearly on pull-style lookup
delays. Given a choice between reference by name and reference by IP
address, they have consistently favored convenience over the extra
edge in speed.

While none of us has collected direct data on this point, the
circumstantial data is not the slightest bit ambiguous. According to
users' behavior with the DNS, brief lookup delays solely at the front
of a series of transactions are A-OK.

>  This

There is no causative correlation whatsoever between IP address length
and DNS lookup depth. Lookup depth correlates with administrative
subdivision which looks no different for IPv6 than it does for IPv4.

This is a factor which is easily tunable at an operations level if
needed to meet the system's speed requirements. The lookup depth is
not bound to the number of elements in the name.
t.u.v.w.x.y.z.dirtside.com only has a lookup depth of 3.
a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.x.y.z.dirtside.com and
0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.x.y.z.dirtside.com
have the same lookup depth: 3. But that's not the end of the story:
you almost always have the first stop (.com) cached, so the actual
experienced lookup depth is rarely more than 2. Should it prove
operationally desirable, a map-pull system can easily accommodate that
degree of flatness in the hierarchy.

>  large and small end-users, including those with current BGP-managed

Nonsense. The map-encap scheme must be sufficiently attractive to a
large enough subset of the user base that they're willing to pay for
it to be deployed. No more and no less.

I argue that if you avoid schemes which disrupt "everybody else,"
you'll get much further with a map-encap scheme that well-accommodates
a small subset of the users than you will with one that poorly
accommodates a wide swath.

>  > I can tell you

No fundamental character of map-encap requires this. TRRP has no such
requirement inherent in its architecture. It doesn't require wide
adoption to begin delivering valuable results. Indeed, the theoretical
deployment timeline shows it producing valuable results with only a
handful of users.

>  This might work to some extent, but not if the price of BGP-managed

If the price of BGP-managed PI space comes down, the problem is solved
and we can all go home. The assumption we're working under, correct or
not, is that fundamental limitations in the technology will cause the
cost of BGP-managed prefixes to level off in a log-normal fashion at
some price which is still too high for the kind of granular
multihoming and mobility that you and I desire.

Regards,
Bill Herrin

---

From: Brian Dickson <[email-masked]>
Date: Thu, 21 Feb 2008 17:20:29 -0500

> and have been impinging on the RIB in any way they can connive.
IMHO, the comparison that needs to be made, is PI vs "foo", for any new 
thing "foo".
Whether "foo" in some way makes use of PA infrastructure is irrelevant, 
really.
It is only that we want to substantially limit the growth of PI space, 
to those who cannot live without it.

If "foo" involves trade-offs that are noticeable, such as delay or 
packet loss, *compared to PI*,
then that becomes a barrier to deployment.

If "foo" does not have such barriers, or fewer barriers, IMHO, that 
should be given significant weight, in comparing
various kinds of "foo".
> of a series of transactions are A-OK.
Not really.

DNS lookups will *block* most applications, until the DNS resolution 
succeeds or fails.

And that has perception in user-space, but no impact to the underlying 
protocols.

However, delays *within* the transport regime, are an entirely different 
can of worms.

Unless the transport itself is modified to handle this, I think it's 
best to see if we can avoid this.
I think it is a non-negligible issue.
> it to be deployed. No more and no less.
The problem is a spam-like asymmetry.

The problem with spam is, one party pays (the ISP delivering mail, and 
the end user ultimately) for the benefit of another party (the spammer).

For an ITR/ETR model, the benefit of the multihomed entity is only 
realized when the *clients* connecting to them have an ITR.

This is a prerequisite for EID address space that isn't routed - common 
to all map-and-encaps solutions.

The fact that the hardware used for the ITR function may already exist, 
and that the vendor may supply
software upgrades with ITR functionality for free (something that is 
IMHO an elephant in the room),
is less important than, the fact that until ITRs are universally 
available, the multihomed entity does not get the benefit.

Compare that with PI space, where the benefit is universal and immediate.

> deployment timeline shows it producing valuable results with only a
You've lost me. Without an ITR (or equivalent host stack modifications), 
how do I connect to an EID host?

And if I'm the server operator, why would I do this (switch to EIDs) 
unless *everyone* has the same quality of connectivity
reaching me?

The benefit for deployment needs to be universal, and absolute, not gradual.

Remember, the objective of someone deploying a site using EIDs, is 
extremely high availability, for inbound traffic.

That means no significant delays in client connections discovering 
alternate paths to the site if a link fails.

And that has to apply to every client on the planet, modulo client-side 
failures (where all sites become unreachable).
> multihoming and mobility that you and I desire.
I don't believe costs will go up, and certainly not with any 
proportionality, regarding PI space.

The DFZ will either be happy, or very very unhappy.

We need to find a solution that everyone will accept, and deploy, that 
prevents the latter.

This is a "commons" situation, with our goal being to avert tragedy.

It is one place where market economies will not do anything to help us, 
and in fact are likely
to be something we need to be keenly aware of - doing good often flies 
in the face of market
pressures.

In a financial model of a zero-sum game, money *is* the root of all evil.

Uh, I seem to have wandered a bit off-topic. Sorry. :-)

Brian Dickson

---

From: "William Herrin" <[email-masked]>
Date: Thu, 21 Feb 2008 18:38:52 -0500

On Thu, Feb 21, 2008 at 5:20 PM, Brian Dickson <[email-masked]> wrote:
>  thing "foo". Whether "foo" in some way makes use of PA infrastructure
>  is irrelevant, really.

Brian,

It will be hard for any solution we come up with to compare favorably
with $8k/yr worth of free service. That's what BGP PI amounts to.

On the other hand, there are lots of folk today who can't get PI and
want it. For them, our solution doesn't have to compare favorably with
BGP PI; it need only compare favorably with PA.

I make the modest claim that once a non-BGP solution to PI actually
exists, we'll find that the core operators' willingness to tolerate
giving away $8k/yr worth of free service quickly dries up... pushing
the bottom end of existing PI users into a situation where they too
must choose between map-encap PI or BGP PA.

>  I think it is a non-negligible issue.

If I correctly understand your criticism, it's that an ongoing
transport mustn't pause for an expired DNS TTL. I agree and this is
addressed in the TRRP document's section on ITRs.

>  the end user ultimately) for the benefit of another party (the spammer).

That is a remarkably insightful description of BGP PI as it exists today.

>  The benefit for deployment needs to be universal, and absolute, not gradual.

TRRP's theoretical deployment description starts with two to three
ITRs at one service provider. These ITRs get fed by something along
the lines of a /16 anycast-announced into the existing BGP system.
Folks who can't have PI space now, either they're too small or their
incumbent monopoly telco won't announce their prefix, folks who only
need it for a handful of servers, get /28's and /29's off this ITR.
Their end is simple: a couple DNS entries and a GRE interface on their
Cisco router or Linux server.

If there is no market for this, TRRP dies. But if there is a market,
it grows and a couple other companies sell a similar tunnel service.
Then you get a couple companies who decide that by feeding those
routes into their own ITRs rather than following BGP to someone else's
ITRs they can get a slight edge on their competition.

It grows bit by bit, running as a useful addition to the current BGP
system until, at some point, one of the large backbone providers
decides it's time to stop giving free FIB slots to the /24's. They put
out the word: if you're announcing a /24, implement a TRRP DNS entry
and a GRE ETR or you're off our section of the 'net. These being
ridiculously cheap things to implement (and /24 announcers having a
pretty good idea of the sufferance on which they survive), the
provider doesn't get a lot of pushback.

That provider realizes a cost savings in his router upgrade cycle, an
effect not lost on his competitors.

The long version is at
http://bill.herrin.us/network/trrp-implementation.html . If you have
comments along the lines of, "Where did X come from that you first
mentioned here," or "What advantage does this individual get from
doing the action you state there?" I'd like to hear them. If there is
any step which doesn't represent a realistic possibility given the
ones before then I need to do more work on it.

>  in the face of market pressures.

Counterpoint: The routing table is only a commons because we choose to
treat it so. It's well within our power to assess the cost of a BGP PI
prefix, create a payment clearinghouse and create software tools which
allow folks to automatically filter prefixes from those who don't pay.
Such an approach requires no changes to the routing architecture at
all and it handily solves the RIB/FIB size problem.

Regards,
Bill Herrin

---

From: Scott Brim <[email-masked]>
Date: Tue, 04 Mar 2008 14:54:11 -0800

> realized when the *clients* connecting to them have an ITR.

See draft-lewis-lisp-interworking for the concept of a proxy tunnel 
router.  PTRs can be close to sources or close to destinations, for 
different purposes.

On 2/21/08 5:57 PM, Brian Dickson allegedly wrote:
> Actually, I was referring to the ITR/ETR model. The ITR bears a large
> burden of the operation costs, scaling, performance, etc, while the
> benefit goes to the "other" guy at the ETR end - the multi-homed guy.

True, but compare to the costs today.  Today every BGP router must 
propagate and maintain every prefix announcement, big and small, whether 
they ever use it or not.  With an ITR/ETR model the only routers that 
get prefix mappings are the ones that actually want to use them.  So an 
ITR bears a burden that benefits an ETR, but only if the ITR wants to 
talk to the ETR.

> As soon as the packets are tunneled, you lose visibility on what is 
> happening.
> 
> And if you aren't the guy doing the tunneling, you *really* lose 
> visibility, and in turn, accountability, the ability to localize and
> diagnose problems, and generally you are left twisting in the wind if
> anything goes wrong. Which it will.
> 
> But, unlike, say, VPNs, this isn't a localized set of sites, in a
> closed community. This is the whole Internet, getting to your site.

We're talking about encapsulation across the DFZ.  Why would I want a 
core ISP to examine bodies of my packets?  What is the problem with 
"losing visibility"?  None of an intermediate ISP's operational 
responsibilities involve looking inside beyond the outer IP header.

swb

---

From: Brian Dickson <[email-masked]>
Date: Tue, 04 Mar 2008 19:08:22 -0500

> responsibilities involve looking inside beyond the outer IP header.

It's the other way around that matters - your packets being tunneled, 
means you don't get to see things like:
- ICMP unreachables concerning the outer header destination
- ICMP MTU exceeded concerning the outer header
- ICMP TTL exceeded concerning the outer header

If you operate the xTR, you have some ability to do diagnostics. If not, 
you are relying on the xTR operator.

And, when this is affecting inbound traffic, isolating common fault 
points becomes a big N^2 issue, where N is no longer aggregated...

Brian Dickson
(Not a big fan of operating networks using GRE, MPLS, or ATM, but have 
done so.)

---

From: "William Herrin" <[email-masked]>
Date: Tue, 4 Mar 2008 19:32:44 -0500

Brian,

How is this handled in today's MPLS networks? It seems like they
should have essentially the same problem: the packet faults at a node
which doesn't know how to interpret the contents of a packet with the
given label.

Regards,
Bill Herrin

---

From: Brian Dickson <[email-masked]>
Date: Tue, 04 Mar 2008 20:24:55 -0500

> which doesn't know how to interpret the contents of a packet with the
MPLS is, generally, a strictly internal mechansim. While there may be 
inter-provider MPLS, I'm not aware
of any significant deployments.

Which means, any problem can be identified as being entirely one ISP's 
problem.

Plus, there are knobs on MPLS configurations, that allow you to "expose" 
the hops, by decrementing IP TTL as it goes.

It gives a limited ability to traceroute, at least, when that is enabled.

(My experience was that customers are perturbed when the network shows 
one "hop" from LA to Frankfurt. :-))

Brian

---

From: "William Herrin" <[email-masked]>
Date: Tue, 4 Mar 2008 21:08:12 -0500

>  It gives a limited ability to traceroute, at least, when that is enabled.

Hi Brian,

Interesting. If you don't mind, I will restate that answer:

1. MPLS includes hacks which cause network analysis tools to behave as
expected most of the time.
2. Even when you hit a corner case where the tools don't work right
with MPLS, it's still possible for an end-user using only end-user
tools to determine which AS has dropped the ball.

This strikes me as a reasonable minimum standard for the support our
various approaches to the routing problem have to provide for network
analysis tools. What do you think?

Regards,
Bill Herrin

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 5 Mar 2008 04:13:16 -0500

On Mar 4, 2008, at 8:24 PM, Brian Dickson wrote:

> MPLS is, generally, a strictly internal mechansim. While there may  
> be inter-provider MPLS, I'm not aware

There are, however, people offering MPLS Tag translation to connect  
enterprise networks together.

Regards
Marshall

> Which means, any problem can be identified as being entirely one  
> ISP's problem.
>
> Plus, there are knobs on MPLS configurations, that allow you to  
> "expose" the hops, by decrementing IP TTL as it goes.
>
> (My experience was that customers are perturbed when the network  
> shows one "hop" from LA to Frankfurt. :-))
>
> Brian
>
> --

---

From: Scott Brim <[email-masked]>
Date: Wed, 5 Mar 2008 08:24:35 -0800

On 3/4/08 4:08 PM, Brian Dickson allegedly wrote:
> Scott Brim wrote:
>>> As soon as the packets are tunneled, you lose visibility on what
>>> is happening.
>>>
>>> visibility, and in turn, accountability, the ability to localize
>>> and diagnose problems, and generally you are left twisting in the
>>> wind if anything goes wrong. Which it will.
>>>
>> We're talking about encapsulation across the DFZ.  Why would I want
>> a core ISP to examine bodies of my packets?  What is the problem
>> with "losing visibility"?  None of an intermediate ISP's
>> operational responsibilities involve looking inside beyond the
>> outer IP header.
> 
> It's the other way around that matters - your packets being
> tunneled, means you don't get to see things like: - ICMP
> unreachables concerning the outer header destination - ICMP MTU
> exceeded concerning the outer header - ICMP TTL exceeded concerning
> the outer header
> 
> If you operate the xTR, you have some ability to do diagnostics. If
> not, you are relying on the xTR operator.
> 
> Brian Dickson (Not a big fan of operating networks using GRE, MPLS,
> or ATM, but have done so.)

and

On 3/4/08 5:24 PM, Brian Dickson allegedly wrote:
 > be inter-provider MPLS, I'm not aware of any significant
 > deployments.

It seems that the problem is that there's an encapsulation which
covers multiple DFZ SPs and which no one in particular has
responsibility for.

I think you might be looking at this like it's a tunnel, like ATM or
other "layer networks" (G.805).  Map-and-encap isn't a tunnel, it's
just an encapsulation.  It has no initial state synchronization
between ITR/ETR, no setup phase.  Anyone can throw packets at an ETR
without preamble.  So there is no separate "tunnel" entity for anyone
to have responsibility for.  Responsibility for packet delivery across
the DFZ from ITR to ETR is the same as it is today, since packets to
the ETR are just packets, with IP headers on them.

Propagating TTL expired for encapsulated packets is an interesting
thought.  I'll have to learn more about what other protocols do.

swb

---

From: "William Herrin" <[email-masked]>
Date: Thu, 6 Mar 2008 00:19:22 -0500

>  thought.  I'll have to learn more about what other protocols do.

Scott,

Implementations generating an ICMP error message like destination
unreachable are only required to return the IP header and the first 8
bytes of the layer 4 header. That won't include any knowledge about
the encapsulated packet.

Most modern implementations seem to return as much of the dead packet
as will fit in to 512 bytes. This includes enough of the encapsulated
packet for the ITR to reflect the unreachable back to the originating
host.

Regards,
Bill Herrin