Subject: Re: [RRG] Taxonomy: 25 questions - updates & web page

From: Robin Whittle <[email-masked]>
Date: Mon, 21 Apr 2008 13:23:22 +1000

Hi Bill,

Thanks very much for your detailed responses to my "Taxonomy: 25
questions" message.

I have prepared a web-version of this, with various updates, at:

  http://www.firstpr.com.au/ip/ivip/comp/

and have updated it according to what you wrote.  In one instance I
link to your message:

  http://psg.com/lists/rrg/2008/msg01094.html

to direct the reader to a fuller explanation than I think would fit
well on this page.

Here I respond to your comments, and mention any other significant
changes I made to my original text when putting it on the web page.

  Cheers

    - Robin

Q02
---

>>  For the proposals for which the Q01 answer was No, do they aim to
>>  provide a mechanism by which there is no loss of connectivity for
>>  any hosts, as the system is deployed, including for communications
>>  to and from networks with no upgrades?
>>
>>   TRRP            Yes, but I can't recall exactly how at present.
> 
> The answer is indeed yes, but the how is a little complex. BGP never
> actually goes away with TRRP. TRRP starts by extending PI from BGP's
> /24 boundary down into micronets (longer than /24) and announcing
> covering routes into BGP which lead to an ITR for folks who haven't
> yet deployed their own ITR. Once deployment begins to get wide, the
> boundary between routes allowed into BGP and maps delivered by TRRP
> starts to shift so that BGP stops carrying most /24's, /23's, etc.
> This shift continues until it reaches the optimal equilibrium between
> full-push BGP and cached-pull TRRP as determined by the network
> operators and their customers.
> 
> This is accomplished in lots of little steps. At each step it never
> takes more than a cheap, simple action to stay connected. The most
> disruptive part of the transition comes when the the first few folks
> in the core want to drop the /24's from BGP. At that point, some folks
> will have to deploy a machine that can host a GRE endpoint and DNS
> entries that point to it.

I gave a short description and referred to your message for a fuller
account.

Q03
---

I added to both the LISP-ALT and TRRP sections saying that each
proposal was probably not "pure pull", because both had planned
extensions by which the ITRs handling traffic packets for some
ETR could be given, or induced to seek, freshly updated mapping
information.

For TRRP, I linked to your description of the 2 methods and to my
critique of these:

  http://bill.herrin.us/network/trrp-preempt.html
  http://psg.com/lists/rrg/2008/msg00532.html

I think there is more food for discussion in this, and I also want
to find out more about what Dino said about ETRs prompting ITRs
to request fresh mapping. (Friday 14th March RRG meeting - no audio
archive, and my live recording ended in the middle of this part of
Dino's presentation.)

Q04
---

>>  How fast can an end-user's mapping change be sent to all ITRs?
>>
>>   TRRP       As for LISP-ALT, since both use a global query
>>              server network.  TRRP doesn't have a workable
>>              Notify system.  It has a limited cache invalidation
>>              system where an unreachable ETR leads to an ICMP
>>              message to an ITR, which can then issue a new
>>              mapping request.
> 
> TRRP has a notification system: http://bill.herrin.us/network/trrp-preempt.html
> 
> Notification is an optional component. The system is designed to work
> acceptably without it but work faster with it.
> 
> Without notifications, the maximum effective change rate per map is
> about once every 10 seconds. That interval has enough overhead that
> folks who are just doing multihoming service restoration will normally
> back it off further. For every communication where notification is
> supported at both the ITR and map server, TRRP can in the average case
> switch to the new map within a fraction of a second after the map is
> published.

I included your text, and noted that I don't yet fully understand it.

Q06
---

>>  Does the end-user have to pay for traffic directed to their micronet
>>  / EID prefix?
> 
> I expect the answer is yes, but it's not formal part of the
> architecture. The most accurate answer is that the end-user can do
> whatever he wants to do. If he gets acceptable connectivity without
> paying someone to reroute traffic his direction from a BGP supernet
> route then the answer is no. If he doesn't, then something like
> http://bill.herrin.us/network/trrp-aapip.html 

Waypoint routers.

> or http://lists.arin.net/pipermail/ppml/2008-March/010475.html
> comes in to play and he'll  pay the aggregator.

I included your text, but have not read the PPML message yet.

Q08
---

>>  Therefore, what functionality is required in the ITR and ETR?
>>
>>   Ivip       No reachability functionality, except as a by-product
>>              of Ivip's inbuilt PMTUD and fragmentation management
>>              system (yet to be finalised, but see my PMTUD-frag
>>              page).
>>
>>   APT        } ITRs must individually determine reachability
>>   LISP-NERD  } of each ETR and whether the ETR can reach the
>>   LISP-ALT   } destination.  Also, I think in LISP, the ETR
>>   TRRP       } may, or must, somehow tell ITRs about reachability
>>                of other ETRs which are in the mapping.  (But
>>                how does the ETR know exactly what mapping the
>>                ITR has??)
> 
> For TRRP, the ETR need only be a dumb GRE endpoint with an acl
> specifying the destination addresses which are allowed to use it. It
> may do more than that and you can get better results if it does, but
> dumb GRE endpoint is the base requirement. The ITR is responsible for
> finding currently valid ETRs in a timely manner (primarily by looking
> them up from the map server) and engaging in much of the complex
> activity.

I included your text.

Q09
---

>>  Therefore, is the system a monolithic approach, permanently
>>  integrating reachability and multihoming service restoration
>>  decision-making functions in the map-encap scheme, or enabling and
>>  requiring end-users to supply their own reachability and
>>  decision-making systems?
>>
>>   APT        } Monolithic system.  Since all these do not
>>   LISP-NERD  } get end-user mapping changes to all ITRs
>>   LISP-ALT   } quickly, the map-encap system must do all
>>   TRRP       } reachability testing and make all decisions.
>>                ITRs and ETRs must do this, using relatively simple
>>                mechanisms and mapping data which are hardwired into
>>                the entire map-encap system.  This will not
>>                satisfy all needs of end-users, since it is
>>                impossible to anticipate all the ways end-users might
>>                want to test reachability and since it is impossible
>>                to define exceedingly complex or extensible
>>                arrangements into the map-encap scheme itself.
>>                Also, ITRs acting alone are a poorly scalable and
>>                inefficient mechanism for testing reachability and
>>                for making decisions.
> 
> This is not an accurate description for TRRP. TRRP requires end-users
> to supply their own reachability and decision-making systems. The ITRs
> have some flexibility based on local knowledge but they're not
> expected to be able to be able to make complete routing decisions on
> their own.
> 
> The workings of the decision-making systems are intentionally not a
> part of TRRP's specification. I foresee them starting as simple
> pingers and advancing to something akamai-like.

I added your text, but with the note:

  OK - it seems I have misunderstood TRRP in this important
  respect.  I will try to understand it this way in the future,
  but will have to discuss this more with you.

Doesn't the mapping information give the ITR options for ETRs to use
when one fails?  I thought it was like LISP or APT in this respect.

Since I don't think either of your Notification systems is reliable,
and since you say TRRP is supposed to work OK without them, I don't
see how an ITR can decide which alternative ETR to use if that ETR
goes down, since the ITR doesn't have a reliable way of knowing
the ETR is down, or that the ETR is not able to reach the
destination network, or any other reason why the end-user wants the
ITRs to use a different ETR.

Perhaps we could discuss this in a separate thread, such as "TRRP is
not monolithic" or "TRRP's end-user supplied reachability and
detection systems".

Q10
---

>>  Does the proposal tackle the PMTU and Fragmentation problem?
>>
>>   TRRP       No.
> 
> Yes, it does. See http://bill.herrin.us/network/trrp.html subheading
> "Fragmentation."

I added:

   Bill refers to the Fragmentation section of:

      bill.herrin.us/network/trrp.html

   but this looks inadequate to me.

   For instance, there is no suggestion of how the ITR
   can discover the Path MTU to each ETR.

   Also, the suggestion that the ITR alter traffic
   packets to adjust TCP MSS in any SYN packets sounds
   really undesirable and impractical to me.

Q13
---

>>  Does the system aspire to support mobility, and if so, how?
>>   TRRP       } have not yet been explored in detail.
> 
> Yes as described in http://www.ops.ietf.org/lists/rrg/2008/msg00766.html

I added:

    All these map-encap systems can support the TTR
    approach to mobility:

      http://www.firstpr.com.au/ip/ivip/#mobile

    but I think only Ivip promises to provide fast changes
    to the mapping for all ITRs which need it, which would
    make the mobility system much more attractive.

    Bill discussed TRRP's support for mobility:

       psg.com/lists/rrg/2008/msg00766.html

    but I don't yet know how fast it could get fresh
    mapping to all ITRs, since I don't believe either
    of the TRRP approaches to Notification are really
    useable.

How fast could a mobile TRRP end-user could have their mapping
changed when they want to use a new TTR?  My understanding is that
without a reliable Notify system, you would depend on short caching
times for your pull mapping responses, which would be extremely
onerous.  With mobility, the old TTR is still reachable, and
probably has a tunnel from the Mobile Node (MN), but the end-user
wants all ITRs to send packets to the new TTR because it is closer
to the MN's currently preferred access network.

Q14
---

>>  What is the granularity of address management?
>>
>>   APT        } Not yet clear, but I think they can all support
>>   LISP-NERD  } EID prefixes starting at any IPv4 address or
>>   LISP-ALT   } IPv6 /64, in CIDR prefix format and therefore
>>   TRRP       } powers of two lengths on binary boundaries.
> 
> TRRP: primary granularity is one IP address. Administrative grouping
> can be arbitrary; its an implementation detail in the map server.
> Efficiencies are gained at power-of-two boundaries, 4-bit boundaries
> and 8-bit boundaries.

I added your text and the note:

   I think this relates to how many DNS servers the
   ITR needs to query before one of them responds with
   authoritative mapping.

Q17
---

>>  Where are ITRs placed?
>>   TRRP       I am not sure - I guess in provider and perhaps
>>              end-user networks.
> 
> "Anywhere." The ITR can be on any originating host (or nat gateway)
> with a public IP address, way deep in the core, around the other side
> being fed by a supernet route or anywhere in between. It's more cost
> effective to place them close enough to the edge that they only deal
> with sub-gigabit traffic flows, but not mandatory.

I added your text with the note:

    TRRP, like Ivip, enables the sending host to be its
    own ITR, as long as it is on a public IP address.

    I think this includes addresses which are mapped by
    TRRP or Ivip.

Q18
---

>>  Where are ETRs placed?
>>
>>   TRRP       I am not sure - I guess in provider and perhaps
>>              end-user networks.
> 
> The ETR can be anywhere that can natively reach the tunneled
> destinations via a direct connection or an IGP.

I added your text.

Do you expect all packets from Sending Hosts outside a given
TRRP-mapped edge network to flow through an ITR and the ETR,
including those sent from Sending Hosts in the same ISP network the
ETR is located in?

This is what I intend for Ivip.

It is not good enough to rely on the internal routing systems to
respond as fast or as reliably as ITRs, so they can't be used to
send packets direct to the destination network, bypassing the ETR.

This really relates to the next question:

Q19
---

>>  Does the internal routing system of ISP networks need to know about
>>  each end-user network's micronet (EID prefix)?  In other words, can
>>  an ETR in an ISP network put out raw, decapsulated, traffic packets
>>  it receives from an ITR and expect the internal routing system to
>>  take these packets to the proper destination?
>>
>>   TRRP       ? I guess so.
> 
> In part, it depends on the ETR placement. The ISP only needs to know a
> route if the ETR is at the ISP. If the ETR is deployed at the customer
> premise then the ISP doesn't need an IGP route to it.
> 
> The ISP will need to know not to drop packets with the micronet's
> source addresses.

I added your text.

Q20
---

>>  Does every packet addressed to a micronet (EID prefix) have to go
>>  through an ITR and an ETR?  If the answer is No to this question,
>>  then it means that the proposal expects packets to be delivered to
>>  the correct ETR not just by the map-encap scheme, but also by the
>>  local routing system of any ISP network which is configured to have
>>  the end-user's prefix in its local routing system.  Where two ISP
>>  networks have ETRs for the one end-user network, and they rely on
>>  the internal routing system to deliver raw decapsulated packets from
>>  the ETR to the proper point which links to the end-user network,
>>  then each such ISP must have the micronet / EID of the end-user's
>>  network in its internal routing system.  For consistent operation,
>>  this means the internal routing system needs to perform the same
>>  functions as an ITR, including deciding whether this ETR is
>>  reachable and has a connection to the end-user network (and if not
>>  so, then tunneling the packet to another ISP's network and its ETR,
>>  which requires an ITR function).
>>
>>   TRRP       ?
> 
> As with Q19.

OK, I hope to discuss this further.

Q22
---

>>  Following the above answers, can the sending host traceroute through
>>  the tunnel?
>>
>>   Ivip       Yes, with a somewhat modified traceroute program.  This
>>              would also be able to identify the ITR and ETR.
>>
>>   APT        } No, unless the ITR did heroics to convert the ICMP
>>   LISP-NERD  } messages it receives into ICMP messages with the
>>   LISP-ALT   } correct data for a standard or modified traceroute
>>   TRRP       } program on the sending host to recognise.
> 
> Note #1: as the originating host may function as a TRRP ITR,
> a modified traceroute would also work for TRRP. TRRP's traceroute
> issues are noted here:
> http://bill.herrin.us/network/trrp-breaks.html
> 
> Note #2: ICMP unreachables as commonly implemented include enough of
> the original packet that the ITR can translate it into valid
> unreachable for the source with no particular heroics. Not every case
> to be sure and the standards don't require it, but when I looked at
> the packets they usually had enough data.

I added your text, with a note:

   If this is true to a large extent, then maybe the same
   is true for Packet Too Big messages from these same
   routers.  If so, then this would be contrary to my
   arguments about it being impractical to have ITRs
   respond to PTBs from the tunnel in a way which
   reliably created a PTB the Sending Host would
   recognise:

http://www.firstpr.com.au/ip/ivip/pmtud-frag/
http://www.ietf.org/mail-archive/web/ram/current/msg01766.html
http://www.ietf.org/mail-archive/web/ram/current/msg01769.html

Q23
---

>>  [Q23]
>>
>>  Following the answer about outer source address, how can an ETR
>>  perform the same filtering on packets it decapsulates as an ISP
>>  border router might perform on packets coming into the ISP network -
>>  specifically, rejecting any packets from outside the ISP network
>>  which have source addresses matching any internal prefix of that
>>  network?  See note below my signature for more on this.
>>
>>   APT        } The ETR would need to do a complete, expensive,
>>   LISP-NERD  } filter operation on the inner source address
>>   LISP-ALT   } against whatever list of prefixes the ISP's
>>   TRRP       } border routers use - which could be huge in a
>>                large ISP network.
> 
> With TRRP, the ETR need only filter for the addresses which it serves.
> This could be an ISP's entire border list but it could also be a
> customer's single micronet or anything in between depending on where
> you place the ETR.

I added your text.

Note on Filtering
-----------------

>>  Even if the end-user runs their own ETR, the ISP - or ISPs - who
>>  provide connectivity are likely to want to ensure there is some
>>  source address filtering on the decapsulated packets.
>>
>>  An attacker could send a packet from anywhere to the ETR and the
>>  decapsulated packet could have the source address of any host in the
>>  ISP's network.  This is likely to be a security concern, if only
>>  because an attacker could prompt the destination host to send
>>  packets to the host at that spoofed address.
> 
> I think this concern fits in a different location than you think.
> 
> Addresses used within an end-user's network are out-of-scope for the
> ISP and always have been. A user can generate bad packets just as
> easily now as they can after deploying an ETR.
> 
> The ISP needs to assure that the source addresses on packets reaching
> them from the user may legitimately do so. That's not so problematic
> at the ETR but it becomes a problem in two other locations:
> 
> 1. The ITR. The tunnel packet's source may be valid but the
> encapsulated packet's source may not be. I gather this doesn't apply
> to Ivip but it certainly applies to all the rest.
> 
> 2. The ISP border with his upstream. Filtering for the ISP's 10
> prefixes that change once a year is not so problematic. Filtering for
> the hundreds of downstream micronets that change weekly is. This
> problem will appear with any solution that attempts to restore
> provider-independent addresses to the masses, not just map-encap.

I included your text and will think about this more in the future

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "William Herrin" <[email-masked]>
Date: Thu, 24 Apr 2008 18:43:36 -0400

>  archive, and my live recording ended in the middle of this part of

Hi Robin,

I think there's an advantage to the ETR being able to tell the ITR
that it is no longer a valid selection for a particular destination.
That having been said, there are a number of cautions:

1. The ETR can't be relied upon to correctly tell the ITR anything. As
near as I can tell, most if not all protocols that rely on an
intermediate node telling the originator or some other intermediate
node about its failure tend to run in to trouble.

Perhaps an example where it almost works would be more instructive
than the many discarded protocols where it fails outright: Path MTU
discovery. Need I say more?

2. In a pull-based protocol, I think there is some value to making
notification elements like this optional. Doing so both allows
compliant implementations that aren't excessively complex and serves
to remind us that the preemptive notification is frequently-useful
speed-up, not a reliable core part of the protocol.

>  I included your text, and noted that I don't yet fully understand it.

The speed with which a TRRP map can be updated depends most directly
on the originator of the map and the completely arbitrary timeout
value he gives it. My comments above were primarily an observation
that once you get below about 10 seconds, other factors besides that
TTL start to provide a noticeable percentage of the update delay.

When the preemptive notifications are implemented and the notification
packet isn't lost, cutover to the updated map will be faster, often
less than a second, regardless of the timeout. The preemptive
notifications should usually work but they're not guaranteed to.

>  I included your text, but have not read the PPML message yet.

This was a brainstorm for how implement long prefixes without the
"everybody else pays" problem. It's not a part of TRRP or any other
proposal.

One idea has the long PI prefixes paying into a fund and all backbones
who agree to carry those prefixes receiving distributions from the
fund.

The other implements a partially-meshed Internet in which the
operators of long prefixes make private arrangements with the dozen or
so providers necessary to get back to the tier-1's who advertise a
supernet prefix. They still get full connectivity and most of the
multihomed reliability but the routes themselves only propagate nearby
and only carry the cost of propagating nearby.

Both were intended to be approaches that could be implemented without
new routing protocols.

>  when one fails?  I thought it was like LISP or APT in this respect.

Yes it does; it allows multiple ETRs to be specified, multiple
encapsulation protocols and a ranked priority for each.

>  Since I don't think either of your Notification systems is reliable,

Correct; only the timeout-rerequest is deemed reliable. The ITR
regularly re-requests map entries from the authoritative map server
for any communications which are ongoing. At that time it picks up any
map changes designated by the originator of the map, which will
include any changes because of link failures or whatever.

The connectivity or routing tests that the originator uses to
determine what maps to include in the map reply are beyond the scope
of the TRRP protocol itself. It could range from "I'll change it by
hand if I need to" to an Akamai-like system that takes packet loss and
other complex factors into account.

>  ITRs to use a different ETR.

IF the ITR receives a preemptive (early) notification from the ETR (or
a router in the path to the ETR) that a problem has occurred, the ITR
can decide prior to the map update to fall back on a lower priority
ETR. Regardless of that decision, it will still seek the originator's
updated map and follow the originator's instructions once it gets
them.

This is almost the exact opposite of, "the map-encap system must do
all reachability testing and make all decisions."

>    can discover the Path MTU to each ETR.

Bog standard path MTU detection: an overlarge packet with the DF flag
induces a too-big message back to the ITR.

>    really undesirable and impractical to me.

It's one of the solutions Cisco came up with in their
currently-deployed GRE implementation in order to deal with broken
PMTUD. If you know that packets beyond a certain size are likely to be
a problem, just tweak the MSS option that occurs near the start of a
TCP connection so that you never get packets that large.

Obviously there are pluses and minuses to this, which is why I offer
it as a recommendation rather than a requirement.

>     of the TRRP approaches to Notification are really

Assuming "fast"=fraction of a second and "not fast"=10 seconds, the
point of my linked article is that TRRP doesn't need to distribute the
update "fast" in order to support mobility. All it needs to do is
communicate with both the approaching and receding towers during the
overlap period.

Since only the ITRs actively communicating with me care what my map
is, and they will re-lookup my map at whatever interval I, the
originator of the map, tell them to, I can guarantee that those ITRs
will get the new map between the time I decide that a tower is
receding and the time that it's no longer reachable to me.

>  changed when they want to use a new TTR?

As previously discussed, a user who knows he's mobile should be able
to set a timeout that guarantees update propagation within 10 seconds.

>  My understanding is that
>  onerous.

My back-of-the-napkin calculations suggest that an active mobile
station would have as much as a 5% traffic overhead in order to keep
the mapping updated and propagate the changes to the interested ITRs
when the delay is dropped to 10 seconds.

Is that onerous? That depends on your perspective. A CD-Rom has about
a 15% overhead for error correction bits. Satellite Internet has
similar error correction overheads. That's simply what makes it work.

If it takes a 5% traffic overhead, mostly concentrated in the
land-bound fiber optic lines, in order to make mobility work smoothly
and eliminate the handoff problem, that's not necessarily a
dealbreaker.

>    ITR needs to query before one of them responds with

Generally no. Its more a question of how many queries the ITR has to
make to get the full span of IP addresses covered by the map and how
many map slots the result will take in the ITR's lookup table. If it's
not on a power-of-two boundary then the full span of addresses will
take multiple slots in the lookup table and if it's not on a DNS
boundary (4 or 8 bit) then the full span will take multiple lookups.

>     I think this includes addresses which are mapped by

Correct.

>  including those sent from Sending Hosts in the same ISP network the

No.

I also expect it'll be asymmetric: if a TRRP-mapped host communicates
with a non-mapped host, the packets from the TRRP host will be bare on
the Internet while the packets returning from the non-mapped host will
have to find an ITR in order to get to the TRRP-mapped host.

>  http://www.ietf.org/mail-archive/web/ram/current/msg01769.html

I haven't rigorously explored this. I spent a few minutes poking at it
with tcpdump and noticed that instead of seeing just the N bytes of
payload that my TCP/IP Illustrated book told me to expect for an ICMP
unreachable, I was usually seeing payload out to the 500+ byte minimum
MTU cutoff.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked] [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004