Subject: [rrg] Summary of architectural solution space

From: "William Herrin" <[email-masked]>
Date: Tue, 11 Nov 2008 15:39:08 -0500

Hi Folks,

I'm trying to put together a more or less concise summary of the
general architectures we've discussed here these past couple years.
This is not a comparison of specific proposals (which Robin Whittle
has done an excellent job of) but rather a summary of the universe of
general strategies we've looked at and haven't resolutely rejected.
I'd appreciate your constructive criticism:

http://bill.herrin.us/network/rrgarchitectures.html

Particular answers I'm interested in:

1. Have I overlooked any viable approaches to the problem? If so, what are they?

2. Have I overlooked any architectural elements? I'm looking for
architectural elements here, not engineering issues. For example, I
left out path-MTU issues because that's a "how do we shoehorn this
into IPv4 of IPv6" engineering issue. It's only relevant in an
engineering compatibility context. Obviously engineering compatibility
issues will greatly inform the final architecture, but that's not what
I'm after in this document.

3. Do you see any areas where I could offer a more clear description?
How would you word it?

4. Have I listed anything for which we have a strong consensus that we
can discard the approach from consideration due to some uncorrectable
defect which is obvious even without an engineering viability study?
By strong consensus, I mean "nearly unanimous."

Thanks in advance,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: Scott Brim <[email-masked]>
Date: Tue, 11 Nov 2008 16:38:22 -0500

> By strong consensus, I mean "nearly unanimous."

Hi Bill.  Here are some notes.

Since you start by talking about the root cause of the routing scaling
problem, I'll start by saying that it isn't _just_ conflation of locator
and identifier.  Even if identifiers and locators were completely
separate, if routing stayed the way it is today you would still have
problems because sites will still want PI prefixes if only to avoid site
renumbering problems, and will still inject prefixes for traffic
"engineering" and to prevent hijacking.

Also there are three different identification issues that are themselves
conflated much of the time:

  (1) AAA identifiers used to check in with a home agent and/or source
      of funds and access a network at all.

  (2) Persistent identifiers that can be used for location discovery.
      They could even be DNS names, SIP URIs, HIP HIs, GSE ESIDs,
      whatever.

  (3) Potentially ephemeral identifiers for session control.

Some of the schemes we've been discussing separate #2 from IP addresses.
 Others separate #3.  Others don't separate them at all (leaving that up
to something else) but still solve routing scaling anyway.

Onward ...

It looks like Strategy A includes both what UCLA calls "separation" and
translation, and that Strategy B is "elimination" with the assumption of
rewriting at the edge.  I think there are four categories here, and they
are different enough that they should be separated.  In particular,
separation and elimination preserve packet headers end-to-end.  OK maybe
not in the face of carrier grade NAT but at least there is a chance that
headers will be preserved.  Translation rewrites headers at least once,
at the core boundary.  This gives you three cases: Separation with
headers maintained (e.g. map-n-encap), Elimination with headers
maintained (e.g. ILNP), Translation (e.g. Six/One Router).  Finally you
have a fourth, the GSE special hybrid which includes Elimination by
splitting ID and Locator at the endpoint, but also Translation at the
network edge.

Scott

---

From: Steven Blake <[email-masked]>
Date: Tue, 11 Nov 2008 16:59:08 -0500

> splitting ID and Locator at the endpoint, but also Translation at the

ILNP also fits into the last class, if network locator translation is
deployed at the network edge.
NLT is optional and does not need to be deployed universally (or even at
both ends of a connection).

If I recall correctly, Six/One only needs to translate the locator portion
of the address field, also.

Regards,

// Steve

---

From: "William Herrin" <[email-masked]>
Date: Tue, 11 Nov 2008 18:29:22 -0500

> "engineering" and to prevent hijacking.

Thanks Scott.

Isn't the renumbering hassle tied to the fact that the addresses
identify the servers? I don't hear of a lot of renumbering issues
arising from failed ethernet cards where the new card has a different
MAC address. If the ID always stayed the same and a locator change was
no more involved than every machine getting a new MAC address and
sending out a gratuitous ARP, would there still be a renumbering
issue?

> to something else) but still solve routing scaling anyway.

An excellent point.

When I talked about ID in the paper, I meant the second and third
elements. In the next draft I'll use GUID (for #2) and SID (for #3)
and drop the term "ID" altogether. Good?

Do we want to consider AAA in this context at all or is AAA adequately
served with protocols that sit at a higher level than routing? Does
the IPv4 or IPv6 address currently carry AAA information in a reliably
usable form?

> rewriting at the edge.

In truth, I didn't consider translation at all. In strategy A, I
expected that the encoders would either fill in header space reserved
for the RLOC or encapsulate the packet in a new one containing the
RLOC. The difference to me seems to be an engineering distinction.

Would you expand on translation? How does translation reduce the
number of entries in the routing table?

If translation is like Strategy A except that you replace the
destination address with an RLOC from the destination's aggregated set
of RLOCs (A1c) (and possibly replace the source address with one of
the source's RLOCs), what criteria identifies situations in which
translation would be preferable to retaining the destination address
and adding an RLOC?

Regards,
Bill Herrin

---

From: Scott Brim <[email-masked]>
Date: Wed, 12 Nov 2008 17:31:06 -0500

> sending out a gratuitous ARP, would there still be a renumbering

As far as renumbering goes, external access to servers can get sorted
out through DNS TTLs, but I'm told there are also problems because
changes are required in internal routing and forwarding configuration,
and in configuration of external sites you have special relationships
with.

> and drop the term "ID" altogether. Good?

Let's try it.

> the IPv4 or IPv6 address currently carry AAA information in a reliably

Right, I just mentioned it because I had a conversationa few days ago
where we had to separate it out.

> number of entries in the routing table?

Consider Six/One Router or NAT66 or LISP's "NAT".  If you translate a PI
prefix into one or more provider-aggregatable prefixes, you've
eliminated the PI prefix from global routing.  This does nothing about
prefixes injected for TE or anti-hijacking.

> translation would be preferable to retaining the destination address

Some people point out that translation is less complicated, but I don't
know what the side effects will be.  I'm uncomfortable breaking e2e
integrity at the IP layer, which encapsulation preserves.  Translation
is also (obviously) useful for interworking.

Scott

---

From: "William Herrin" <[email-masked]>
Date: Wed, 12 Nov 2008 19:16:04 -0500

On Wed, Nov 12, 2008 at 5:31 PM, Scott Brim <[email-masked]> wrote:
>> identify the servers?
> out through DNS TTLs,

Hi Scott,

Operational experience says no lifeline with DNS TTLs I'm afraid. DNS
TTLs only work right with applications which pay attention to DNS
TTLs. This works out to something south of 10% of deployed apps: most
interface with DNS via the gethostbyname call. Some undetermined and
indefinite time later, the other 90% are restarted or encounter some
other event that causes them to look up the DNS entry again.

> but I'm told there are also problems because

I'm not sure that makes sense in context. Given a protocol in which
the GUID, SID and LOC are separate, why would you ever statically
assign the LOC to anything? Borrowing the MAC address example again, I
can count on the fingers of one hand the situations in which they get
statically assigned, especially across multiple routing domains.

> eliminated the PI prefix from global routing.

If I understand Six/One router (and please correct me if I'm wrong),
it alters the the IPv6 source and destination addresses entering the
core by replacing the edge PI prefixes with provider-assigned
prefixes. The edge PI prefixes are fed (via the mapping system) to the
router where the packet exits the core. When exiting the core at the
other side, the edge PI prefixes are restored.

I'm not sure I'd call that translation. It seems like Strategy A
variant A1c where the "provider-assigned prefixes" are RLOCs. Instead
of adding space in the packet for the RLOC, a portion of the GUID is
mapped out to make room for the RLOC on entering the core and then
mapped back in on exiting the core. The extra map-in-on-exit is the
price paid for maintaining compatibility with the existing IPv6
protocol.

Maybe that's an engineering compatibility difference like the
gyrations for PMTU in map-encap rather than an architectural
difference? What do you think?

How do NAT66 and LISP's NAT differ from Six/One Router?

Regards,
Bill Herrin

---

From: Scott Brim <[email-masked]>
Date: Thu, 13 Nov 2008 13:16:53 -0500

> other event that causes them to look up the DNS entry again.

Thanks, I had forgotten that.  Site renumbering is just hard.

> statically assigned, especially across multiple routing domains.

I should probably leave this to others to answer but for example a site
may have restrictions on which prefixes (assigned by department) can
access which services.  I hope others will fill in here.

> prefixes. 

Right.  Maybe Christian will add more.

> The edge PI prefixes are fed (via the mapping system) to the

I was thinking about unilateral mode.  In Unilateral Mode, Six/One
Router does a 1:1 translation at a site boundary, without involvement of
mapping services or a requirement for reverse translation.  I get the
feeling unilateral mode may be OBEd by NAT66, but I haven't asked Christian.

> How do NAT66 and LISP's NAT differ from Six/One Router?

NAT66 and Six/One Router preserve the "host" part of the address, and
iirc rewrite to a different prefix at different upstream connection
points.

Scott

---

From: "William Herrin" <[email-masked]>
Date: Thu, 13 Nov 2008 14:19:50 -0500

> feeling unilateral mode may be OBEd by NAT66, but I haven't asked Christian.

Outbound with clients behind the Six/One Router, that looks like plain
old client NAT. Client NAT is a fine thing, but it's already widely
deployed so it isn't likely to improve the routing table versus what
we have today.

Inbound with servers behind the Six/One Router, I'm pretty sure it
doesn't work. The raw networking part works. The fault isn't obvious
until you step back and take a holistic view, but it would require an
impractically complex split-horizon DNS system. Hosts on legacy
networks must find the provider-assigned address and hosts on Six/One
Router-served networks must find the PI address.

I say impractically complex because all DNS requests are relayed by a
caching resolver, usually on a different machine than the client host.
The AAAA DNS query may even come from an IPv4 address! As a result,
you can't just build a smart authoritative DNS server that hands out
different IPs depending on the request source address. And you can't
just err on the side of handing out the provider-assigned address:
that address is unreachable to any user behind a Six/One Router
because the user's Six/One Router will alter the address in the
response!

IMO, this isn't deadly to Six/One Router. There is such a dearth of
IPv6 deployment that one could reasonably just say that "legacy" IPv6
users are required to add a Six/One Router at their border.

>> difference? What do you think?

I'm gonna back off on this assertion and the earlier one about
backwards compatibility not being an architectural issue. If I add the
following to strategy A, does it cover the strategies you've talked
about?

Strategy A.

Compatibility approaches include:

A4a. New IP protocol. Not compatible with IPv4 and IPv6.

A4b. Modified IP protocol. Not compatible with deployed IPv4 and IPv6.

A4c. Standard IPv4 and IPv6 packets are encapsulated in a tunnel
packet while they transit the core. Path-MTU issues are addressed by
setting an Internet-wide maximum packet size enforced by the encoders
and assuring that all core links support that size.

A4d. Standard IPv4 and IPv6 packets are encapsulated in a tunnel
packet while they transit the core. Path-MTU issues are addressed by
returning packets which breach the MTU while in the core to the
encoder who must act as a proxy by returning a sensible packet-too-big
message to the originating host.

A4e. The IPv6 address space is partitioned into end-user address space
and Internet core address space. The address to RLOC map is symmetric.
Part of the IPv6 end-user address is swapped for the RLOC when the
packet enters the Internet core and then restored when it leaves the
core. IPv4 is abandoned.

Regards,
Bill Herrin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 14 Nov 2008 15:34:20 +0100

> http://bill.herrin.us/network/rrgarchitectures.html

Constructiveness is in the eye of the beholder.  :-)

> The root cause of the routing table growth problem is that we're  
> using the same element (IP address) to express both the  
> communications identity (ID) needed by the layer 4 and higher  
> protocols to keep track of communication sessions with other hosts,  
> and the node's present locations (LOC) within the network topography.

Well, that was the rough consensus at the routing&addressing workshop  
two years ago. I was the only one who didn't agree. With a loc/id  
split multihoming and renumbering aversity (assuming we can contain it  
to the IDs and still renumber locs) would no longer have to increase  
routing tables. But the current routing table is 8 x the number of  
ASes and BGP not only fails to contain updates to a limited part of  
the network, it actually amplifies them as they circle the globe.  
Those issues are separate from an id/loc overload.

Also, we have no proof that the problems caused by a loc/id split are  
less than those fixed by it.

> changing a server's identity requires changes in many other hosts'  
> references which are unwieldy or impractical to make.

Disagree: if you can run both the old and new side by side for some  
time, there's nothing to it.

What I'm missing in your summary is the difference between the ID->loc  
mapping and the loc->reachability mapping. If we want to support  
multihoming, which we do or we still have a use case that can  
overinflate the routing table, we need to be able to determine which  
locators are reachable and which aren't. This gives us the following  
options:

- ID->loc and loc->reach are both flooded: this is what BGP does,  
doesn't scale
- ID->loc is flooded, loc->reach on demand: IMO, this is what we need
- ID->loc and loc->reach on demand: caching and initial packet issues
- ID->loc on demand, loc->reach flooded: doesn't make sense

---

From: "William Herrin" <[email-masked]>
Date: Fri, 14 Nov 2008 11:17:09 -0500

On Fri, Nov 14, 2008 at 9:34 AM, Iljitsch van Beijnum
<[email-masked]> wrote:
> Well, that was the rough consensus at the routing&addressing workshop two
> years ago. I was the only one who didn't agree. With a loc/id split
> multihoming and renumbering aversity (assuming we can contain it to the IDs
> and still renumber locs) would no longer have to increase routing tables.
> But the current routing table is 8 x the number of ASes and BGP not only
> fails to contain updates to a limited part of the network, it actually
> amplifies them as they circle the globe. Those issues are separate from an
> id/loc overload.

Hi Iljitsch,

That's an interesting point. Dissociating the GUID and SID from the
LOC would inherently suppress edge routing updates, even for basic
multihoming. However, updates due to link changes in the core still
have to propagate through the entire core, even if the change is so
distant from the instant router and the topology downstream from the
instant router is such that the change can't impact the downstream
FIBs.

I haven't heard a potentially successful solution strategy that
addresses this point. I think geoaggregation was the only one that
attempted it and we proved that geoaggregation has uncorrectable
theft-of-service anomalies even in small configurations.

The current solution strategies all imply the remaining impact from
this factor is small enough to be a non-problem. I think that's
probably right, but we should find a way to acknowledge the issue and
state the assumption explicitly.

> Disagree: if you can run both the old and new side by side for some time,
> there's nothing to it.

I mean with the current deployed protocols. I'm trying to explain why
the routing table size problem is an emergent property of the current
protocol design.

> multihoming, which we do or we still have a use case that can overinflate
> the routing table, we need to be able to determine which locators are
> reachable and which aren't.

Strategies A and B both depend on aggregated LOC reachability
information being flooded. It appears to be needed to compute a
network path.

I guess this ties in with your point about suppressing distant routes.
If you want to suppress distant routes then you can't flood LOC
reachability info. Is that correct?

Here's how I propose to include this knowledge in the document:

Strategy A.

Variants include:

A1a. Each core ISP has one RLOC. The RLOC's existence and reachability
is flood-propagated to the rest of the core.

A1b. Each core ISP has a small number of RLOCs for traffic engineering
(TE). The RLOCs' existance and reachability is flood-propagated to the
rest of the core.

A1c. Each core ISP has one aggregated set of RLOCs which it may
hierarchically assign to customers downstream and/or disaggregate for
TE. The aggregated RLOC's existance and reachability is
flood-propagated to the rest of the core.

Strategy B.

Assign heirarchically aggregatable LOCs to every host. Assign multiple
LOCs to each host such that in the network topography hosts appear as
stubs in multiple locations instead of forming distant connections in
the graph. Assign one aggregated set of LOCs to each core ISP where a
core ISP is one which has numerous major transit or peering links.
Flood-propagate the aggregated LOC's existance and reachability to the
rest of the core.

Having reduced the network topology to something relatively close to a
hierarchy, perform plain old hierarchical aggregation on the LOCs. Add
and remove LOCs to each host dynamically during operation as needed to
reflect changes in the nearby network hierarchies.

Strategy C.

Suppress distant routes by aggregating them into sets expected to be
available in a given direction. Because LOC reachability info is not
flooded, the routing tables each router must deal with are relatively
small.

Appears intractable.

Variants include:

C1. Geographic aggregation. All nodes within some geographic boundary
are assigned the same LOC. Routers move packets to any adjacent router
deemed to be "closer" to the LOC in question.

Geoag has been shown to have uncorrectable theft-of-service anomalies
in networks as small as 8 autonomous systems and two geographic areas.

Addendum:

Item 1. Additional causative factors worth mentioning:

Because even a distant route may present a better priority in one
coreward direction than another, it's not possible to suppress distant
routes. As a result, every default-free router in the core must carry
all routes.

Acceptable?

Regards,
Bill Herrin

---

From: "William Herrin" <[email-masked]>
Date: Fri, 14 Nov 2008 17:58:59 -0500

Hi Folks,

The second draft is now available at
http://bill.herrin.us/network/rrgarchitectures.html .  It adds a
strategy C to the solution universe, strategy A now addresses the
approaches to compatibility with deployed IP and I've made what I hope
are numerous small improvements.

The first draft is archived at
http://bill.herrin.us/network/rrgarchitectures1.html

For the folks who offered suggestions on the first draft: Thank you.
Have I addressed the issues you brought up? How can I further improve
the document?

For everyone: I'd appreciate your constructive criticism.

Regards,
Bill Herrin

P.S. Slightly off topic: anyone flying through or out of Washington
Dulles tomorrow on the way to the meeting?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 15 Nov 2008 21:49:53 +0100

On 14 nov 2008, at 17:17, William Herrin wrote:

> However, updates due to link changes in the core still

Right, the problem with suppressing updates that don't impact the FIB  
is that further upstream (downstream? confusing terminology, just  
credit and debit) the info could impact a FIB there because it's  
combined with other info. However, we could possibly solve this if we  
can let routing info skip routers or AS hops. I.e., if A and B are  
tier 1s and C and D are customers of those, respectively, and E a  
customer of both C and D, C and D could pretty much point a default to  
A cq B, but E would like to know if A loses a link and can now reach  
some destination only through a very long path or not at all, so E may  
want to subscribe to A and B their routing updates even though C and D  
don't care.

> theft-of-service anomalies even in small configurations.

The solution is to keep the geo stuff inside a single AS, then these  
issues don't occur.

> this factor is small enough to be a non-problem.

Famous last words.  :-)

But if we can keep the "core" limited to the actual core then FIB  
scalability shouldn't be an issue, so that only leaves processing  
issues such af flapping and amplification thereof.

> information being flooded. It appears to be needed to compute a

You can't aggregate reachability info. If a multihomer loses an ISP,  
everyone needs to start sending their traffic through another ISP.  
This is info that can't be aggregated. (Also note that this is only  
relevant for multihomers; for single homed destinations that are in  
the system for other reasons we don't need to know if they're  
reachable.)

> reachability info. Is that correct?

You can't but I don't think that's the reason. And you still need to  
know reachability for distant destinations if you actually talk to  
them. However, in the _current_ system we don't care so much about  
updates from distant networks because we can generally assume that any  
repairable failures will be repaired by the time we see the update. In  
other words: if a link in Tokyo dies I'm probably still sending my  
traffic to LA or SF, it's unlikey that I'm now going to circle the  
globe in the opposite direction.

> is flood-propagated to the rest of the core.

This is problematic because then each ISP still must have full  
customer prefixes in each location. For the largest ISPs just their  
customer prefixes is enough to be problematic, especially when success  
with our effort makes it easier to use EID prefixes.

> flood-propagated to the rest of the core.

I guess. Do we really care about the specifics? We just know that the  
number will be larger than 1.

> Assign heirarchically aggregatable LOCs to every host.

Ah, shim6.  :-)

> flooded, the routing tables each router must deal with are relatively

If we remove the burden of selecting a working LOC from the routing  
system this can work. We can do this in shim6 and it shouldn't be too  
hard to do with LISP, either. You just need a reachability detection  
protocol. As luck would have it I'm the co-author of one... (shim6- 
failure-detection)

> deemed to be "closer" to the LOC in question.

No, that would be geographic routing. If you think that geographic  
addressing is impossible, think about geographic routing...

The point of geo addressing is that you can point an aggregate in the  
right direction and find the more specific in the place the aggregate  
points to.

> routes. As a result, every default-free router in the core must carry

Depends on the topology restrictions you're willing to live with. I  
don't think we need to support the situation where a multihomer  
connects to ISPs in South Africa and Hawaii.

I think the correct constraint is that ISPs must carry their  
customer's prefixes in all their default free routers and other ASes  
must have sufficient routing info to get the packets to a working ISP  
of the destination. (Note that this is harder in the case when A talks  
to B where B has ISPs C and D and A points to C but then the link from  
C to B fails and C is not prepared to route traffic from A to B  
through D.)

Iljitsch

PS. Is fuel now so expensive that a flight leaving at 1530 and  
arriving at 1750, so flying during day time / evening has the lights  
turned off pretty much the whole flight? Good thing these new MacBooks  
have the illuminated keyboard.
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "William Herrin" <[email-masked]>
Date: Mon, 17 Nov 2008 11:08:08 -0500

On Sat, Nov 15, 2008 at 3:49 PM, Iljitsch van Beijnum
<[email-masked]> wrote:
> Depends on the topology restrictions you're willing to live with. I don't
> think we need to support the situation where a multihomer connects to ISPs
> in South Africa and Hawaii.

What about the one where he connects via a low earth orbit satellite
constellation which has redundant ground stations in Hawaii, New York,
Germany and India? Or any 4 points that actually fit on an orbit? The
cool thing about LEO is that it starts around 100 miles up, so you
don't have long speed of light delays or large transmit-power
requirements. But you do have a huge geographic spread.

In a developing country, it would be doubly-handy if a customer could
use a terrestrial radio network when the signal is good and fall back
on a more expensive LEO satellite when the terrestrial radio network
has problems, all without disturbing the ongoing end-to-end
connections.

That's perhaps an extreme case. The more obvious case is the company
with offices in London and San Francisco with a private line between
them. If the SF ISP link fails, the London link should take over and
vice versa. Under normal operating conditions, packets should head for
the SF link for the SF office and the London link for the London
office.

> I think the correct constraint is that ISPs must carry their customer's
> prefixes in all their default free routers and other ASes must have
> sufficient routing info to get the packets to a working ISP of the
> destination. (Note that this is harder in the case when A talks to B where B
> has ISPs C and D and A points to C but then the link from C to B fails and C
> is not prepared to route traffic from A to B through D.)

I figure: let's determine with width and breadth of the solution
universe before trying to decide which one is "correct." Otherwise we
risk assigning the label "correct" to a solution which is merely the
"first" we thought of. I suspect we're closer to having the solution
universe identified than we think we are, if we take the time to
consider it and write it down.

> PS. Is fuel now so expensive that a flight leaving at 1530 and arriving at
> 1750, so flying during day time / evening has the lights turned off pretty
> much the whole flight? Good thing these new MacBooks have the illuminated
> keyboard.

I thought they always turned the cabin lights off so folks can sleep
if they want to. Isn't that what the passenger-controlled directional
reading light above each seat is for?

-Bill

---

From: Robin Whittle <[email-masked]>
Date: Tue, 18 Nov 2008 13:24:56 +1100

Hi Bill,

Here are some thoughts on your second draft:

I haven't completely followed the recent discussions, and I am not
trying to check your description against every proposal.  My aim is
mainly to comment on how well it covers Ivip and to some extent APT.

The "Strategy A" section matches Ivip pretty well.  The "attaches a
second Remote LOC address" is neat, since it covers both
encapsulation and "Forwarding":

  ETR Address Forwarding (EAF) - for IPv4
  http://tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw

  Prefix Label Forwarding (PLF) - for IPv6
  http://www.firstpr.com.au/ip/ivip/ivip6/

as well as translation (Six/One Router).

I think there need to be some options here:

  1 - Encapsulation.  LISP, APT, TRRP and the original Ivip.

  2 - Translation.  Six/One Router.

  3 - Forwarding with the RLOC address, or part of it, in the
      existing header, as noted above.  This requires upgrades to
      most - essentially all - DFZ routers, and probably to many or
      most internal routers of ISPs.  There are currently two
      approaches:

      3a:  (EAF 30 bits in IPv4) Sufficient bits to uniquely identify
           the ETR.

      3b:  (PLF 19 or 20 bits in IPv6) Sufficient bits to identify
           the ISP, or one of multiple address ranges in the one ISP.
           This may be sufficient if there is a single ETR for that
           range. Otherwise, when the packet arrives at that
           ISP another lookup will be required with some other
           arrangement to get the packet to the correct ETR within
           this address block.

Question A1 concerns how ISPs organise their RLOC space.  None of the
existing options directly describes Ivip.  A1a and A1b doesn't fit,
because with Ivip, ISPs can and probably will have many separate RLOC
addresses (addresses on which an ETR could exist).

A1c doesn't quite fit either, since ISPs will probably have more than
one block of space they use for RLOCs.  Maybe:

   A1d.  Each core ISP has one or a small number of BGP-advertised
         prefixes which are suitable for use as RLOCs.  The ETRs
         at these RLOC spaces may be physically within the ISP's
         network, and/or physically within end-user networks, but
         they are always accessible via one of the ISP's one or a few
         BGP prefixes.

A2 concerns mapping.

A2a is full push - LISP-NERD.

A2b is probably closest to Ivip and APT.  However the central or
distributed registry pushes all changes in near real time to the full
database query servers - QSDs -(Ivip) or Default Mappers (APT).  A
full database ITR is really just a caching ITR integrated with, or
very close to, a QSD.

In Ivip, most ITRs are caching ITRs, and they request from a nearby
QSD (perhaps via intermediate caching Query Servers) whatever mapping
they need.  The QSDs remember the requests, and the caching time of
the responses they give.  If the QSD gets a changed mapping from the
central / distributed registry, it pushes it to the ITR or caching
query server which requested it.  I think APT has similar
arrangements.   So it "hybrid push-pull".  Ivip has a fast version of
this and APT a slow version.  In both cases, there is a full push of
the entire mapping database to all full database query servers, which
are physically close to the ITRs, but changes are only pushed in
real-time from the full database query servers to the ITRs based on
recent requests according to cache times.

A2c is full pull - LISP-APT or TRRP.

Perhaps there could be this addition to cover Ivip and APT:

   A2d.  GUIDs dynamically mapped to each RLOC are pushed towards a
         central or distributed registry as they change. The registry
         pushes all incremental changes in near-real time to all
         full database query servers in ISP and/or end-user networks.
         The encoders (ITRs) request mapping from these local
         query servers.  The response has a caching time and the
         local query server will push any changed mapping to an
         encoder if it receives such a change for mapping which
         matches a recent query which is still within its caching
         time.  There may be one or more full database query
         servers in each ISP and there may be one or more layers of
         caching query servers between these and the encoders.

Regarding how the system copes with RLOC addresses becoming
unreachable, neither A3a or A3b really describe Ivip.

A3a involves the ITRs determining reachability.  A3b doesn't specify
how the mapping is changed, and involves the concept of preferences
between multiple RLOC addresses for any EID address.  Ivip's mapping
does not involve preferences or ITRs detecting reachability or making
any other decisions.

For Ivip, perhaps you could add something like:

   A3c. End-user networks are responsible for controlling the
        mapping of their EID address space.  Each EID prefix -
        in the case of Ivip, not necessarily a prefix, since it
        can be one or more contiguous IPv4 addresses or IPv6
        /64s - is mapped to a single RLOC address.   For reasons
        such as a link failure, to implement inbound Traffic
        Engineering, or to implement address portability when
        moving to another ISP, the end-user network causes
        the mapping to change to the new RLOC address, and this
        is conveyed to all full database query servers in
        near real time.  These push the changes to any encoders
        (ITRs) which may need them, based on previous queries and
        the caching times of the responses.

The Ivip system - the ITRs etc. - are not involved in making any
mapping decisions.  The mapping has no preferences.  It is up to the
end-user to control their mapping for whatever purposes they desire -
and there is a small fee per change.  So the economics of mapping
changes pays for most of the global fast push system and ensures that
the costs and usage are matched.  This largely avoids BGP's tragedy
of the commons problem, which is at the heart of the routing scaling
problem.

In question A4, A4d is a good description of Ivip with encapsulation
and the tricks the ITR needs to do to solve PMTUD problems:

  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

There seem to be no PMTUD problems with the Forwarding approaches
mentioned above.

I think compatibility is more than coping with PMTUD problems.  I
think there is also the crucial question of how the system handles
packets sent by non-upgraded networks and/or hosts.  Without this,
the system only provides very limited benefits when few people have
adopted it - so it would never become widely enough adopted to solve
the routing scaling problem.

Ivip's "Open ITRs in the DFZ" (OITRDs) and LISP's "Proxy Tunnel
Routers) (PTRs) do much the same thing (although I think the PTR
description is much less informative about how address space will be
managed than with Ivip's description).

So I think there needs to be a new section on this.

For Ivip's OITRDs, and I guess LISP PTRs:

    Packets sent to EID addresses by hosts in non-upgraded networks
    are forwarded by those networks' routers to the DFZ.  There,
    OITRDs/PTRs advertise prefixes which encompass large blocks
    of RLOC space, typically encompassing the RLOC addresses used
    by many end-user networks.  Those OITRDs/PTRs encapsulate (or in
    the case of Ivip with Forwarding, convert the packet to the
    new format with with the RLOC address inside the existing header)
    the packet so it is forwarded by the DFZ routers and any internal
    routers to the correct ETR.

Strategy B seems to cover SHIM6, ILNP and perhaps others.  I believe
this is 100% impractical, since it only provides benefits to adopters
in proportion to how many other networks adopt it.  Also, it involves
host stack and probably application changes, if sessions are to
survive multihoming transitions.  Please see my previous message:

  http://www.irtf.org/pipermail/rrg/2008-November/000169.html

for why I believe these approaches are of theoretical interest only.
 This includes some "clean slate" approach where the introduction
date is so far away that people think we can ignore the difficulties
of changing everyone to the new system.  Whenever it happens, it is
still going to be impossible.

I think Strategy C is completely impractical.  It involves a complete
change to the routing and addressing structure which would be
difficult or impossible to introduce, even if everyone agreed it was
a good thing.  Then there are various problems with it, not least
that geography is not necessarily the most important determinant of
how the Internet's routing system can be broken into local zones, for
the purpose of enabling routers to work with only a local view of the
Net.

  - Robin

---

From: "William Herrin" <[email-masked]>
Date: Thu, 20 Nov 2008 17:05:26 -0600

>      existing header, as noted above.  This requires upgrades to

Hi Robin,

For the moment I'm arguing that these are compatibility differences in
A4. Functionally (architecturally) adding an RLOC is adding an RLOC
whether its filling in a field in the packet, adding a label to the
front of the packet, encapsulating the packet into another which
contains the RLOC or temporarily and symmetrically swapping out part
of the address.

> one block of space they use for RLOCs.  Maybe:

Is there an important architectural distinction between one
disaggregatable block and many blocks? If not, my inclination is to
leave this alone and say that IVIP "approximately" fits A1c. Maybe
I'll weaken the language a bit and say it "has an aggregated set of
RLOCs" instead of "has one aggregated set of RLOCs."

>         full database query servers in ISP and/or end-user networks. [...]

My inclination is that if it's a full push to any of the map servers
then its a full push system. After all, leaf nodes in the existing BGP
system can discard distant routes in favor of a default saving quite a
bit of space in the router as things are now. Yet we still reasonably
consider BGP to be a full push system.

I'm open to arguments why that viewpoint is wrong.

> does not involve preferences or ITRs detecting reachability or making

The minimum preference you can give an RLOC is "remove". ;)

I'll change A3b to "destination host or set of hosts" to reflect
single-eid maps, prefix-based maps and arbitrary ranges of eids as
IVIP uses.

> and there is a small fee per change.

Then in IVIP the preference ranking for an ETR is "yes" or "not present."

> the routing scaling problem.

"Come to think of it, you can't get there from here." - from "Which
way to Millinocket"

I don't know how to intelligently address the deployment issue from
anarchitectural standpoint rather than a specific design standpoint.
It may not fit as an architectural issue beyond noting that any
non-optional changes to the architecture present a deployment problem.
I'm open to suggestions.

> in proportion to how many other networks adopt it.
> I think Strategy C is completely impractical.

You may be right. Nevertheless, I want to enumerate it as a strategy a
least until we have a strong consensus that we can't possibly build a
successful protocol that way. If nothing else, doing so will save us
from, "Why didn't you consider?" questions later on. Or at least arm
us with good answers. :)

Regards,
Bill

---

From: Robin Whittle <[email-masked]>
Date: Fri, 21 Nov 2008 16:57:24 +1100

Hi Bill,

Regarding my comments on your second draft:

> contains the RLOC or temporarily and symmetrically swapping out part

OK.

> RLOCs" instead of "has one aggregated set of RLOCs."

Yes, I think it is best to describe it in a way which indicates one
or a low number of blocks, rather than exactly one.

> I'm open to arguments why that viewpoint is wrong.

I have described APT and Ivip as "hybrid push-pull" systems, and I
think this is an important distinction from the "full push" of
LISP-NERD.  Here is my understanding of the 3 options so far:

  A2a. GUIDs statically mapped to each RLOC are periodically pushed
       towards a central or distributed registry. The full list is
       periodically downloaded to the encoders which add RLOCs to the
       packets.

I think this matches only LISP-NERD.  Although the ITRs requests the
download of mapping data, it is still a "full push" system, since the
mapping distribution system ensures that the full database is sent to
every ITR.

  A2b. GUIDs dynamically mapped to each RLOC are pushed towards a
       central or distributed registry as they change. The registry
       pushes all incremental changes in near-real time to all
       encoders which add RLOCs to the packets.

I think this doesn't match any of the proposals so far.

  A2c. GUIDs dynamically mapped to each RLOC are pushed towards a
       central or distributed registry as they change. Encoders
       request and briefly cache individual mappings from the
       registry as needed.

I think this matches LISP-ALT and TRRP.  Six-One Router has no
specific mapping distribution system yet.

I think APT and Ivip have an architecturally similar arrangement and
 that it would be good to add something like the A2d paragraph I
suggested.  The last paragraph could be chopped.

> single-eid maps, prefix-based maps and arbitrary ranges of eids as

OK, but A3b still doesn't describe Ivip at all.

A3a and A3b both refer to changes to how the ITR chooses which RLOC
to apply according to reachability failures which have in fact been
detected by some means.  Both involve an EID having multiple RLOCs
and each RLOC having a particular preference.  Ivip doesn't have
multiple RLOCs - it only has one - so there are no preferences.

A3a refers to the ITR detecting a reachability problem between itself
and the ETR, and then deciding that this this high preference RLOC
address appears to be useless at present, that it will instead use
whichever other RLOC has the next highest preference.

A3b seems to be a description of how an ETR tells the ITR that it
can't reach some or all of the end-user network, so that the ITR
should do as described above.  However, I think that the current
description is confusing, since:

   cause a dynamic change to the address-RLOC map, depreferencing the
   affected RLOC.

is too terse.  I think it needs to be clear that the ETR is not
altering the mapping, but is sending a message back to the ITR that
this ETR's RLOC should no longer be chosen - allowing the ITR to
hopefully choose another one from the two or more RLOCs which are
typically specified in the mapping.  Below is an attempt at rewriting
it.  Part of the trouble, I think, is that in your attempt to be
terse you haven't got a term for the ETR.

The terms "ITR" and "ETR" are fine for LISP, APT (if the Default
Mapper is counted as a type of ITR), Ivip and TRRP.  Six-One Router
calls them both "Translation Routers" I think.

Does "tunnel" imply encapsulation?  If so, then Ivip's forwarding
approaches are not really tunnels.  Still, I will use "ITR" and "ETR"
because it is compatible with the other schemes and nutty to invent a
new name for no really good purpose.  Maybe if Christian Vogt could
rename his routers "Ingress Translation Routers" and "Egress
Translation Routers" we could all use the same acronyms!

Here is an attempt at rewriting A3b.  I need to use "ETR".

  A3b. Link failures which prevent parts of the RLOC's network from
       reaching a destination host it serves are detected by the
       ETR, which sends a message to the one or more encoders
       (ITRs) which are sending it packets, to cause those
       encoders to dynamically change how they choose which RLOC
       to use for this EID.  The mapping information has a list of
       RLOCs with preferences, and the encoder would typically
       choose the next highest preference RLOC instead.

I think something like my suggested A3c is required to describe Ivip,
which operates on very different principles to the other schemes in
terms of how failures are detected, and how the ITR behaviour is
changed.  Here is is again:

> Then in IVIP the preference ranking for an ETR is "yes" or "not present."

I think it is confusing to try to shoehorn Ivip into a set of
descriptions which are suitable for the other schemes, which involve
preferences.  Ivip has no preferences because it specifies only a
single RLOC for each EID.

> non-optional changes to the architecture present a deployment problem.

I do not regard OITRDs or PTRs as merely a question of "deployment".
 That would be true only if we assume that the "deployment" phase is
a transitory one which will come to an end when every last end-user
network gets Religion and deploys ITRs in its own network.

I see OITRDs and PTRs as a lasting part of the system - a vital part
of the architecture - since I think it is unrealistic to assume all
end-user networks will ever install ITRs.

A4 at present covers PMTUD questions and whether there is to be a
fundamentally new protocol or not.  A4e also seems to describe
Six-One Router's approach, which only works for IPv6.

So perhaps the A4 material needs to be split into multiple questions:

  Is there a new protocol to replace IPv4 or IPv6?

  Is there a modified form of IPv4 or IPv6?

  How to deal with PMTUD problems, which result from encapsulation by
  a device other than the sending host.

  How to ensure adoptors of the new kind of address space have all
  their incoming packets handled fully for multihoming, TE and
  portability, despite them being sent from non-upgraded networks.

For the last question, I think that Ivip's OITRDs and LISP's PTRs do
much the same thing.  Here is a description:

     APT, as currently defined, can advertise prefixes into BGP to
     achieve much the same result, but due to the current (and likely
     to remain) /24 administrative limit on IPv4 BGP propagation of
     routes, it can only work if the EID prefix is /24 or shorter.
     This is assuming there are multiple APT islands.  If APT is
     changed so all ISPs link, via tunnels rather than direct
     router-to-router links, to share mapping data and so make a
     single global APT island, then APT perform the same task as
     OITRDs and PTRs, with some or all border routers of
     participating ISPs acting as OITRDs and tunneling packets
     directly to the ETRs, including across the DFZ - irrespective
     of the size of the EID prefix.

Your TRRP approach:

  http://bill.herrin.us/network/trrp-implementation.html

seems to be a Heath Robinson affair with 8 Carrots and 6 Sticks.
That could be tricky to condense into a paragraph, but it needs to be
done because it is architecturally distinct from the other approaches.

I think Six-One Router doesn't have a way of coping with packets sent
from non-upgraded networks.

> from, "Why didn't you consider?" questions later on. Or at least arm

Yes, I agree.

No-one is suggesting that SHIM6 is a solution.  I guess this is
covered by your initial description of being "resolutely rejected":

  http://www.irtf.org/pipermail/rrg/2008-November/000127.html

Ran Atkinson and Steven Blake attest that ILNP is practical - but I
don't know of anyone else who does.

We have little consensus.

I would have thought that the idea of changing all hosts and
applications to solve the routing scaling problem would be resolutely
rejected, but apparently not.  See my previous message and some
earlier ones for why I think a host-based solution is a can of worms.

LISP, APT, Ivip, TRRP and Six-One Router seem to be based on the
assumption that it is easier and/or better to add something to the
routing system than to change all hosts so they need to be concerned
with coping with changes in addresses of other hosts and/or with
changes in connectivity in the routing system.

  - Robin

---

From: "William Herrin" <[email-masked]>
Date: Fri, 21 Nov 2008 14:42:54 -0600

Hi Folks,

The third draft is now available at
http://bill.herrin.us/network/rrgarchitectures.html .  It adds a
strategies D, E and F to the solution universe and moves the major
criticisms of each strategy to a separate subheading so that the
criticisms don't clutter the description of the approach.

The second draft is archived at
http://bill.herrin.us/network/rrgarchitectures2.html

For the folks who offered suggestions on the earlier drafts: Thank
you. Have I addressed the issues you brought up? How can I further
improve the document?

For everyone: I'd like to close this document after this draft. If you
have any additional comments or criticism, now's the time.

Regards,
Bill Herrin

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 24 Nov 2008 14:25:35 +1300

> http://bill.herrin.us/network/rrgarchitectures.html .  

Looking at Strategy B, I have two comments.

One is that
"Assign multiple LOCs to each host such that in the network
topography hosts appear as stubs in multiple locations..."
appears to me to be quite clearly the *standard* model for
IPv6, a.k.a. Plan A, so I would suggest inverting the names
of Strategy A and Strategy B.

The second comment is that
"LOCs dynamically mapped to each host are pushed towards a
distributed registry as they change."
is only one variant. The other variant is that they aren't
considered to be dynamically mapped but rather administratively
mapped (in solutionism, that's called DNS). That doesn't change
most of what you write, but it does affect the two major
criticisms:

1. There's no problem with transport protocols as long as the
IP stack conceals the address dynamics from the upper layer.
It would be solutionism to point out that there's already running
code for that.

2. If the LOCs are assigned administratively, the firewalls
can deal with multiple LOCs.

   Brian

---

From: "Tony Li" <[email-masked]>
Date: Sun, 23 Nov 2008 18:21:24 -0800

|"Assign multiple LOCs to each host such that in the network
|topography hosts appear as stubs in multiple locations..."
|appears to me to be quite clearly the *standard* model for
|IPv6, a.k.a. Plan A, so I would suggest inverting the names
|of Strategy A and Strategy B.

I thought the standard model for IPv6 was to assign sites a PI /48?

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 24 Nov 2008 16:23:03 +1300

> I thought the standard model for IPv6 was to assign sites a PI /48?

No, that's the standard heresy. The standard model has always
been multiple PA's.

    Brian

---

From: Robin Whittle <[email-masked]>
Date: Mon, 24 Nov 2008 19:07:29 +1100

Hi Bill,

Regarding your 3rd draft:

and my comments:

 http://www.irtf.org/pipermail/rrg/2008-November/000172.html
 http://www.irtf.org/pipermail/rrg/2008-November/000216.html

I accept that you don't want to go into the differences between
encapsulation and forwarding - however, it does affect some other
things, such as PMTUD and source address filtering.

You wrote in:

  http://www.irtf.org/pipermail/rrg/2008-November/000212.html

     My inclination is that if it's a full push to any of the map
     servers then its a full push system.

     ...

and I replied (msg 216) that I think this description and its A2b
description doesn't adequately describe Ivip or APT.

I don't see any change in any of the A2 options.  I think APT and
Ivip need a separate description.  I describe them as "hybrid
push-pull" systems.  If you leave your page as it is, perhaps you
could add a footnote pointing to this message in the archives saying
that I think some further descriptions are needed to cover Ivip, at
least:

Likewise, I see no change in the A3 section.  To encompass Ivip, I
think something like this needs to be added:

I understand you don't consider OITRDs and PTRs as part of the
architecture, but I think they are an essential element of Ivip and
LISP which is not covered in your page at present.

I don't see any changes in the A4 descriptions.

I wrote that A4d is a good description of Ivip's approach to PMTUD
problems:

On second thoughts, A4d is not a good description at all.

Here is an attempt at describing Ivip's "IPTM: ITR Probes Tunnel MTU"
approach.

   A4f. The encoder encapsulates all packets which are short enough
        that once encapsulated, they will be no longer than some
        constant N, where it is assured that the MTU between all
        encoders (ITRs) and ETRs is at least N.  Longer packets
        are either encapsulated and sent, or rejected with a
        packet too big message, depending on how their length
        if encapsulated, compares to two variables the encoder
        maintains for each particular RLOC (ETR) address.  The
        encapsulated packets function as probes, and depending
        on whether the encoder receives PTBs for them, it
        adjusts its two variables.  These represent the
        encoders upper and lower limits to its uncertainty about
        the true MTU to this RLOC.

That is about as good a description as I can think of without making
it a lot longer.  It is a complex thing to do, but I think it will work.

Your description doesn't cover Ivip's two forwarding modes:

Here is an attempt to do so:

  A4g  For IPv4, all DFZ and some internal routers are upgraded
       so that an alternative format of IPv4 header can be used.
       The "Evil bit" is set to 1 and 30 bits of the RLOC address
       is encoded into the header, with routers forwarding
       the packet according to that address, rather than the
       destination address.  The ETR restores the header and
       forwards the packet to the end-user network.  This involves
       no encapsulation overhead or PMTUD problems, but
       fragmentable packets longer than a certain length are
       dropped by the encoder.

BTW, I think dropping overly long fragmentable packets is just fine -
applications have had more than a decade to get over burdening the
network with fragmenting their too long packets.

  A4h  For IPv6, all DFZ routers are upgraded to recognise an
       alternative format of the IPv6 header in which 19 or
       20 bits are used to determine the packet's forwarding.
       These bits map to a predefined set of advertised prefixes
       of ISP networks, and so the packet is delivered across the
       DFZ, to a border router of the correct ISP network, without
       PMTUD problems.  If there is more than one ETR at that
       network, a second lookup will be required and a similar, or
       different, mechanism internal to the ISP network will be
       required to forward the packet to the correct ETR.

Here are some comments on your summary of the major criticisms of the
various approaches to Strategy A:

*  There don't appear to be any genuinely clean ways of implementing
*  strategy A. Handling path-MTU is a problem since the packets in
*  the core are different than the origin host would recognize.

This doesn't apply to the two forwarding approaches.  I know it is
radical to suggest altering DFZ routers, but maybe it is not so hard
to do.  The IPv4 approach only concerns the FIB, and the IPv6
approach also involves the RIB to a certain extent.  AFAIK, these
could be done with firmware upgrades.  There is no change at all to
the BGP implementation.   Maybe this would be easier than trying to
develop and deploy the more complex ITRs required to handle the PMTUD
problems.

*  Extra bandwidth is consumed by the ITR figuring out whether the
*  ETR is still available and functioning.

This doesn't apply to Ivip, since the ITRs don't do any reachability
testing.  The end-user network typically would hire a specialised
company to continually probe reachability of its network via all the
ETRs, and to update the mapping accordingly, but that is an entirely
user-selectable burden of probing, and it does not affect ITRs.

*  Border filtering of source addresses becomes problematic.  It's a
*  mess.

I think this is a fair description of LISP, APT and TRRP.

This is not so with Ivip's forwarding approaches - the filtering
happens normally and naturally with those.

Nor is it a problem with Ivip's encapsulation approach of using the
sending host's address as the source address of the encapsulation
header (LISP, APT and I guess TRRP use the ITR's address as the outer
source address).  Ivip's approach of:

  1 - Outer source address = sending host's address.

  2 - ETRs drop any packet where the inner source address does not
      match the outer source address.

solves the problem neatly.  The ISP's border routers carry on as
usual, rejecting packets with a source address matching a list of
internal addresses.  There's nothing more to do.

When, as with LISP, APT and TRRP, the outer source address is that of
the ITR, then the only way of implementing this source address
filtering is to implement it at every ETR.  This would be extremely
expensive and unwieldy, since there might be a large number of ETRs
this list has to be sent to - and because the only fast way to do the
filtering is with hardware (TCAM?), or a rather large lookup table in
RAM.

*  Deployment may require heavy weight "for the public good" relays
*  in the non-upgraded part of the Internet to facilitate migration.

This doesn't match Ivip's OITRD business model at all.  They would be
paid for by the organisation who has the MAB (Mapped Address Block)
which they split up into smaller spaces and rent to end-user networks.

  Business incentives for LISP PTRs and Ivip OITRDs
  http://psg.com/lists/rrg/2008/msg02021.html

I think it would be helpful to include pointers to actual proposals.
 I understand you want to use a more generalised language to describe
the various proposals, but for the benefit of anyone who is not
totally up with RRG discussions (or who is, but is tired), I think it
would help to have such pointers.

I haven't yet read the Strategies B to F.

   - Robin

---

From: "Teco Boot" <[email-masked]>
Date: Mon, 24 Nov 2008 09:08:21 +0100

Agreed with Brian.

Moreover, I think the strategy is orthogonal to the others
and it provides additional functionality, like multipath transport.
I would call this strategy "map and forward" as encap is not needed; 
assuming edge routers are upgraded for Source Address to 
Border Router mapping.

Teco.

> -----Oorspronkelijk bericht-----
> Van: [email-masked] [mailto:[email-masked]] Namens Brian E
> Carpenter
> Verzonden: maandag 24 november 2008 2:26
> Aan: William Herrin
> CC: RRG
> Onderwerp: Re: [rrg] Summary of architectural solution space
> 
>

---

From: "William Herrin" <[email-masked]>
Date: Mon, 24 Nov 2008 08:03:36 -0500

> of Strategy A and Strategy B.

Hi Brian,

Neither strategy A nor strategy B are reflected within the standard
model for IPv6.

Some specific differences between standard and strategy B:

1. In strategy B, communication sessions survive the loss of any of
the locators. This is a new requirement on layer 4. Neither TCP nor
any of the common UDP protocols handle this. The dynamic map updates
serve two purposes: first to let the remote endpoints know that a new
set of locators is available to reach you and second as a lightweight
authentication mechanism so that the remote endpoint knows which
locators are valid as sources in the packet.

2. In strategy B, locator assignment is dynamic. Not just automatic
and not just on the LAN. It starts at the first service provider in
the core and dynamically propagates hierarchically down to you with
each router receiving the block of addresses from an "upstream"
router, assigning subblocks "downstream" and swapping peer routes
laterally.

As a result, ** the network often adjusts to a edge-network link
failure by propagating a new set of locators (layer-3 addresses)
downstream and expiring the old ones, ** an approach that retains the
hierarchic aggregation of the locators regardless of the current
network topology. Note that this requires the impacted hosts to
dynamically update their respective map entries.

Also as a result, hosts require layer-4 protocols capable of dealing
with multiple layer-3 addresses (locators) in each communication
stream.

3. In strategy B, a route has to match both the source and
destination. This enforces the hierarchy. A route source and
destination can't both be 0/0. Announced coreward, 0/0->Dest/long.
Announced hostward, Dest/long->0/0. Announced laterally,
Source/long->Dest/Long. Thus your packet automatically goes out the
correct service provider for the selected source locator.

While this is possible with a number of IPv6 implementations it is not
a standard part of the architecture and not a requirement.

Would you suggest changes to the strategy B description that make this
more clear?

> IP stack conceals the address dynamics from the upper layer.

Yeah, there is. Sessions don't survive the loss of a layer-3 address
and start slow when one of the addresses isn't available. As a result,
simple multihoming still has to be carried in the core. SHIM6 and SCTP
try to tackle the problem and one or the other may be a viable
connection-oriented layer-4 protocol in a strategy B network. I
personally think that with the right selection of layer-3 semantics we
can do better. Regardless, we'll also need to address the problem for
connectionless (UDP-based) protocols.

Regards,
Bill Herrin

---

From: "Tony Li" <[email-masked]>
Date: Mon, 24 Nov 2008 09:53:05 -0800

|> I thought the standard model for IPv6 was to assign sites a PI /48?
|
|No, that's the standard heresy. The standard model has always
|been multiple PA's.

It seems that the IAB then needs to have a chat with the RIR's then.  When
can we expect this?

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 25 Nov 2008 10:21:30 +1300

Bill,

On 2008-11-25 02:03, William Herrin wrote:
> the locators. This is a new requirement on layer 4. 

It was introduced (as a goal) by the MULTI6 WG in RFC 3582.
I don't see it as new.

> Neither TCP nor

That's a very good brief description of SHIM6, which does not need
a global dynamic map as it works purely end to end. It also hides the
dynamics from the transport layer, so TCP and UDP are unaffected.
Soa global dynamic map, and changes to the transport layer, are not
necessary properties of strategy B.

> 
> 2. In strategy B, locator assignment is dynamic. 

Why is that a necessary property? If that *is* a necessary property
of strategy B, then you have a missing strategy which is essentially
B with administratively assigned locators (which is, I believe, the
current IPv6 standard strategy, and always has been). My suggested
changes were to include this as a variant withing your strategy B.
In any case, I believe it needs to be covered in the solution space.

> Not just automatic

That can work either administratively or dynamically, too.
> dynamically update their respective map entries.

Not to mention DNS. But once again, it will work whether the locators
are assigned dynamically or administratively. (Incidentally, if they're
assigned dynamically, there will still need to be an underlying invariant
administrative handle that is used for the assignment process to work.)
> with multiple layer-3 addresses (locators) in each communication

As noted, a shim can solve that problem.

> a standard part of the architecture and not a requirement.

Whether that works or not gets quite complicated.
draft-ietf-6man-addr-select-sol-01.txt
draft-baker-6man-multiprefix-default-route-00

> Would you suggest changes to the strategy B description that make this

Well, either include the multiple administratively assigned LOC case
or add a new strategy which is actually the original IPv6
strategy.
> connectionless (UDP-based) protocols.

I don't understand why you think there's an unsolved problem, given that
SCTP solves it explicitly and SHIM6 solves it for UDP and TCP (and, I
think, DCCP). That applies to administratively assigned LOCs (which can
be found in the DNS). I agree that there may be a harder problem for
dynamically assigned LOCs - as noted above, that will require an
administratively assigned handle to exist, and that will have to be
used as the key to find the current LOCs. But once you have done that,
the SCTP or shim solution works.

   Brian

---

From: "William Herrin" <[email-masked]>
Date: Mon, 24 Nov 2008 17:47:15 -0600

Hi Robin,

When I read this, I see A2b in the large with some local
optimizations. Let me ask others on the group to offer an opinion on
this: is Robin's hybrid method described above distinctive enough from
the other three mapping methods that it should be listed separately in
the document?

>         the caching times of the responses.

This is really what A3b is all about.

I'll adjust it to:

A3b. Link failures which prevent parts of the RLOC's network from
reaching a destination host or set of hosts it serves cause an
external analysis element to make a dynamic change to the address-RLOC
map, depreferencing or removing the affected RLOC.

Better?

>        the true MTU to this RLOC.

I think you were right the first time: this is a fancy version of A4d
but architecturally its still A4d.

>       required to forward the packet to the correct ETR.

Whoops, I missed that one. How about:

A4f. The IPv6 flow label or some other component(s) of the IPv6 header
are used to contain the RLOC. The flow label is set before the packet
enters the core. Non-local packets are routed based on the flow label.
IPv4 is abandoned.

>  http://tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw

That's a bit broken. You can't discard the fragment offset and MF bit.
I'm pretty sure hosts are allowed to pre-fragment the packets and then
set the DF bit on each fragment. Hosts routinely fragment UDP and ICMP
packets that are too large for the wire, even if they don't set the DF
bit. These packets need to successfully cross the core.

There's also no point carrying the DF bit if you're not going to carry
the fragment offset. By definition those packets are always DF=1.

You could rely on the L2 checksum to maintain packet integrity. Many
if not most core links are a form of ethernet today anyway, and the
rest could theoretically be upgraded. But you're going to need to find
some more bits somewhere; 16 isn't enough. I suppose you could steal 4
bits from the header length since there's not a lot of point in
passing anything in the core with IP options attached anyway. Just
send an "administratively prohibited" message if someone tries to send
a packet with options. But that still only buys you 20 bits.

ICMP errors are gonna be a bit hairy.

Meh. I've included less likely approaches in the document. How about this:

A4g. Steal bits from other functions in the IPv4 header (e.g.
checksum) to make space for an RLOC. Discard those components and set
the RLOC when the packet enters the core. Restore the original bits
when the packet leaves the core. Use another strategy for IPv6.

> would help to have such pointers.

I'm trying to keep this document higher level than that. I think a
second document which maps from the strategies to the proposals and
vice versa would be very helpful.

Regards,
Bill Herrin

---

From: "William Herrin" <[email-masked]>
Date: Mon, 24 Nov 2008 17:53:50 -0600

Hi Brian,

Some of the best new ideas are old ones whose time has finally come.

> necessary properties of strategy B.

You say Shim6 doesn't need a global map. I say it can't make effective
use of one.

A shim6-enabled host has trouble initiating connections to hosts where
the first chosen address is inaccessible. It also runs into a number
of corner cases where its behavior is either broken or ambiguous. For
example, what happens if:

1.2.3.4 starts talking to 5.6.7.8 which is also 8.7.6.5.
5.6.7.8 dies. 1.2.3.4 starts talking to 8.7.6.5 instead. The app still
thinks its talking to 5.6.7.8.
5.6.7.8 is assigned to a new machine unrelated to 8.7.6.5.
The app at 1.2.3.4 asks for a new connection to 5.6.7.8.

Does 1.2.3.4 connect to the new 5.6.7.8 or does it connect to 8.7.6.5?

With a map, the answer is straightforward:

1.2.3.4 starts talking to example.com (currently 5.6.7.8 and 8.7.6.5)
5.6.7.8 dies. 1.2.3.4 keeps talking to example.com, now at 8.7.6.5.
5.6.7.8 is assigned to a new machine unrelated to 8.7.6.5.
The app at 1.2.3.4 asks for a new connection to example.com (currently 8.7.6.5).

That having been said, ubiquitous deployment of Shim6+DNS combined
with the kind of dynamic address assignment I described could
reasonably form a strategy-B protocol. I don't think it forms a
particularly good one, but I doubt I could do better under Shim6's
mapless design constraint. With a map I can do better. Much better.

> Why is that a necessary property?

Because we want to optimize the network for hierarchical aggregation
in the instant topography with whichever link sets are currently
working. We've already learned that administratively optimizing a
static "best case" view of the network for hierarchic aggregation is
burdensome and dysfunctional because the network state isn't static.

And we want the addresses to change regularly enough that no one makes
the mistake of writing software which assumes otherwise. We won't
achieve either goal with administrative assignment.

> draft-baker-6man-multiprefix-default-route-00

Thanks for the refs!

> or add a new strategy which is actually the original IPv6

The original IPv6 strategy was:

Make everybody attach to only one ISP using IPv6 and a subset of the
ISP's single set of PA addresses.

That may not have been the original hope, but that's what popped out
of the process.

Okay, maybe I see your point. Let me repeat it in my words to make
sure before I commit it to the document.

I think what you're saying is that if we make sure that the layer-4
protocols properly continue the communication session as layer-3
addresses are added and removed and we deprecate and remove variants
of the layer-4 protocols which don't achieve this, we should be able
to use multiple statically (aka administratively) assigned subnets
with IPv6's layer 3 to achieve an acceptable reduction of the routing
table size. Theoretically, IPv6's deployment is meager enough that
deprecating the non-compliant variants of the layer-4 protocols now
wouldn't be too painful.

How close did I get?

Regards,
Bill Herrin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 25 Nov 2008 11:25:03 +0100

> It seems that the IAB then needs to have a chat with the RIR's  

Note that the RIRs are not in charge of their own policies; these are  
set by members of the community who care enough to show up. For some  
strange reason these are mostly people with some financial stake in  
these policies.
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg