Subject: [rrg] Why a host-based solution does not necessarily add signalling load

From: Pekka Nikander <[email-masked]>
Date: Mon, 19 Jan 2009 08:23:19 +0100

One of the repeated arguments against host-based solution is that it  
increases signalling load.  Of course, that is true for a pure host- 
based solution where each host individually maintains the mapping from  
EIDs to RLOCs.  So, from a signalling load point of view, such a  
solution is not very good if the mapping granularity is at the site  
level.

There is a relatively easy solution to this, which brings the inter- 
site signalling load to the level of tunnel-router-like maps-and- 
encaps solution.  The basic idea is that the hosts *delegate* the  
signalling to a tunnel-router-like device at the site border.  (At the  
same time, the hosts can still signal individually through another  
interface; consider e.g. a laptop which is attached with a wire to a  
corporate site but at the same time has a wireless interface using an  
open WLAN.)

So, when both a host and a site support a (multi-homing/mobility)  
solution, the following can be done:

1. The host detects that the site supports the solution, e.g. through  
DHCP/ND.
2. The host delegates any mobility/multihoming/tunnelling signalling  
to the TR at the site.
3. The site border TR/whatever takes care of signalling on the host's  
behalf.

When considering HIP, the theory is discussed in length in Jukka  
Ylitalo's Ph.D.
http://lib.tkk.fi/Diss/2008/isbn9789512295319/

For a practical example, see Section 4 of draft-melen-hip-mr-01.txt

For measurement results, see Jukka Ylitalo, Jan Melén, Patrik Salmela,  
and Henrik Petander, "An Experimental Evaluation of a HIP based  
Network Mobility Scheme", in Proc. of the 6th International Conference  
on Wired/Wireless Internet Communications (WWIC 2008), pp. 139-151,

--Pekka Nikander

_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 Jan 2009 09:49:08 +1300

Pekka,

On 2009-01-19 20:23, Pekka Nikander wrote:
> increases signalling load.  Of course, that is true for a pure
> host-based solution where each host individually maintains the mapping
> from EIDs to RLOCs.  So, from a signalling load point of view, such a
> solution is not very good if the mapping granularity is at the site level.
> 
> There is a relatively easy solution to this, which brings the inter-site
> signalling load to the level of tunnel-router-like maps-and-encaps
> solution.  The basic idea is that the hosts *delegate* the signalling to
> a tunnel-router-like device at the site border.  

Ran Atkinson has observed that for IPv6, RA messages could be used as a
way to indicate prefix "liveness" in real time (see section 2.3 of
draft-carpenter-renum-needs-work-01). No doubt a similar mechanism could
be retrofitted to IPv4. That would make it equally possible to delegate
the liveness function, if needed, to the border device.

    Brian

> (At the same time, the
> hosts can still signal individually through another interface; consider
> e.g. a laptop which is attached with a wire to a corporate site but at
> the same time has a wireless interface using an open WLAN.)
> 
> 2. The host delegates any mobility/multihoming/tunnelling signalling to
> the TR at the site.
> and Henrik Petander, "An Experimental Evaluation of a HIP based Network
> Mobility Scheme", in Proc. of the 6th International Conference on
> Wired/Wireless Internet Communications (WWIC 2008), pp. 139-151,
> 
> --Pekka Nikander
>

---

From: Scott Brim <[email-masked]>
Date: Mon, 19 Jan 2009 16:37:01 -0500

Excerpts from Pekka Nikander on Mon, Jan 19, 2009 08:23:19AM +0100:
> based solution where each host individually maintains the mapping
> from  EIDs to RLOCs.  So, from a signalling load point of view, such
> a  solution is not very good if the mapping granularity is at the
> site  level.
>
> There is a relatively easy solution to this, which brings the
> inter-site signalling load to the level of tunnel-router-like
> maps-and-encaps solution.  The basic idea is that the hosts
> *delegate* the signalling to a tunnel-router-like device at the site
> border.  (At the same time, the hosts can still signal individually
> through another interface; consider e.g. a laptop which is attached
> with a wire to a corporate site but at the same time has a wireless
> interface using an open WLAN.)
>
> 1. The host detects that the site supports the solution, e.g.
>    through  DHCP/ND.
> 3. The site border TR/whatever takes care of signalling on the
>    host's  behalf.

Pekka, I don't think this is a win.  

As an endpoint, you are interested in reachability of specific
endpoints.  You don't care if a whole prefix is reachable, if the
particular endpoint isn't reachable within that prefix.  So if you
delegate liveness monitoring to an agent, you are asking it to monitor
specific locators.  

First of all the number of locators that need to be monitored does not
decrease.  The problem is a lack of parallelism between the different
sides of the transaction.  Intermediaries like ITRs and ETRs can
monitor the health of interfaces to whole prefixes, since that is what
they are responsible for.  Endpoints are responsible for communication
with other endpoints, not just prefixes, so that is the information
they need.

Second, as Fred Baker pointed out in many cases exchanges are short
and you don't need probing or monitoring, and in some cases there may
be different "grades" of continuity required which may allow "relaxed"
monitoring ... but a liveness agent won't be able to keep track of
that nearly as well as endpoints, so it will over-probe unnecessarily.

Does that make sense?  Do I understand correctly?

> Ylitalo's Ph.D.  http://lib.tkk.fi/Diss/2008/isbn9789512295319/
>
> For measurement results, see Jukka Ylitalo, Jan Melén, Patrik
> Salmela,  and Henrik Petander, "An Experimental Evaluation of a HIP
> based Network Mobility Scheme", in Proc. of the 6th International
> Conference on Wired/Wireless Internet Communications (WWIC 2008),
> pp. 139-151,

I'll take a look.

Scott

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 20 Jan 2009 13:01:02 +1300

Scott,

> delegate liveness monitoring to an agent, you are asking it to monitor

The e2e principle seems to make it clear that, ultimately, liveness is
the endpoint's responsibility, but that is usually interpreted as a
statement about what the transport layer or above should do. And they
already do it, 100% independently of what we do in layer 3 and below.
I think that limiting layer 3 liveness detection to the prefix level
(except presumably in the case of Mobile IP) is a perfectly good
engineering compromise, unless we're planning to repudiate [Saltzer].

    Brian

[Saltzer] End-To-End Arguments in System Design, J.H. Saltzer,
   D.P.Reed, D.D.Clark, ACM TOCS, Vol 2, Number 4, November 1984, pp
   277-288.
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 19 Jan 2009 20:57:48 -0500

There are a couple of different issues, that are all too easy to 
confuse.  (No, I don't think Brian, Scott, Dave, etc. have individually 
confused them.  But I think that the discussion may have switched from 
one to another unintentionally.

There are prefixes on RLocs.  Such prefixes appear in routing.  Clearly, 
the presence of such a prefix in routing does not tell you enough to 
judge the reachability or liveness of an individual xTR within that 
prefix.  (The absence of a prefix, that's useful.  But that's the easy 
part of the problem.)

As far as I know, in terms of liveness testing of ETRs, there is no 
granularity finer than the ETR to test.  So if we are using an ID / RLoc 
split, the liveness testing we are interested in is the reachability / 
efficacy of a given ETR.  [There is the additional complication of the 
ETR being live, but not having reachability to the endpoint.  That is a 
different problem than the liveness problem described in the draft. 
There are many additional problems...)

Conversely, although the liveness testing must be on the basis of 
individual ETRs, it does seem likely that many hosts in a site will be 
trying to reach endpoints behind the same set of ETRs.  As such, having 
a border guy (or someone else, if you really want to complicate life) 
testing / monitoring that liveness / reachability on behalf of the 
various sources within the site would seem likely to
1) reduce the probing traffic
2) increase the odds of having accurate data when packets need to be sent.

Yours,
Joel

---

From: Pekka Nikander <[email-masked]>
Date: Tue, 20 Jan 2009 12:28:44 +0200

> individual ETRs, it does seem likely that many hosts in a site will  
> be trying to reach endpoints behind the same set of ETRs.  As such,  
> having a border guy (or someone else, if you really want to  
> complicate life) testing / monitoring that liveness / reachability  
> on behalf of the various sources within the site would seem likely to

Exactly.

To recap what I tried to say:

So far we've had two models on the table:  tunnel-router-based and  
host-based.  The tunnel-router based obviously checks reachability at  
the TR granularity, and host-based on host granularity.  This has  
received criticism that a host-based model causes more signalling/ 
reachability testing traffic than the tunnel-router-based one.

My point was a simple one: not necessarily so.  If TR-level  
granularity is acceptable, then it is doable with a hybrid host+TR- 
based solution (like the one I've been hand waving during the last  
month or so), and in that case the amount of reachability testing /  
signalling will be at the *same* level as for a TR-only solution.

Of course, in a hybrid solution the hosts can do reachability testing  
themselves.  However, if the only interface they have is behind a TR,  
then it doesn't give any extra pay off to the host.  That is, the host  
has a slight pay off if it delegates the task to the TR, since the TR  
may do more efficiently, as it has better information about the  
multiple local links, and (at least sometimes) on the behalf of  
multiple hosts at the same time.

Still, in a hybrid solution, a multi-homed the host has the option of  
doing reachability testing on some interfaces itself, and on others  
delegate the task to the TR.

--Pekka

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 20 Jan 2009 07:04:47 -0800

Hi Pekka,

> -----Original Message-----
> From: Pekka Nikander [mailto:[email-masked]] 
> Sent: Tuesday, January 20, 2009 2:29 AM
> To: RRG
> Subject: Re: [rrg] Why a host-based solution does not 
> necessarily addsignalling load
> 
> > 2) increase the odds of having accurate data when packets 
> need to be  
> > sent.
> 
> Exactly.
> 

I'm not sure why you're still thinking of a hybrid HIP/xTR
as the preferred mode of operation. IMHO, in many cases the
HIP end system should have direct access to the underlying
physical links (ethernet, 802.11, etc. etc.) and not the
tunnel interfaces of a co-resident xTR. This to allow
choosing the correct physical link for the desired mode
of operation and possibly based on link metrics. It also
saves encapsulation overhead over underprivileged links.

The other thing is that the HIP end system may need to
do active testing of the path, where with RANGER the
xTR gets to do both active and passive testing of the
path (i.e., it listens for ICMP unreachables). So, I
think the two reachability tests are complementary and
not conflicting/interfering. So my answers are:

HIP? yes
xTR? yes
HIP/xTR hybrid? no

Fred
[email-masked]

---

From: Paul Jakma <[email-masked]>
Date: Tue, 20 Jan 2009 16:05:31 +0000 (GMT)

On Tue, 20 Jan 2009, Pekka Nikander wrote:

> This has received criticism that a host-based model causes more 
> signalling/reachability testing traffic than the 
> tunnel-router-based one.

What's the basis for thinking this, and are there any good 
studies/data to back it up? Intuitively, it doesn't seem like it 
should be so, to any significant degree at least.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Tact is the art of making a point without making an enemy.
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: David Conrad <[email-masked]>
Date: Tue, 20 Jan 2009 09:38:50 -0800

On Jan 20, 2009, at 8:05 AM, Paul Jakma wrote:
>> signalling/reachability testing traffic than the tunnel-router- 
>> based one.
>
> What's the basis for thinking this, and are there any good studies/ 
> data to back it up? Intuitively, it doesn't seem like it should be  
> so, to any significant degree at least.

Well, for one thing, a tunnel-router-based solution would typically  
cover some number of hosts.

Regards,
-drc

---

From: Paul Jakma <[email-masked]>
Date: Tue, 20 Jan 2009 17:56:24 +0000 (GMT)

On Tue, 20 Jan 2009, David Conrad wrote:

>>> signalling/reachability testing traffic than the tunnel-router-based one.
>> 
>> What's the basis for thinking this, and are there any good studies/data to 
>> back it up? Intuitively, it doesn't seem like it should be so, to any 
>> significant degree at least.
>
> Well, for one thing, a tunnel-router-based solution would typically cover 
> some number of hosts.

Ok, sure.

I assumed there was an implied ", to a significant enough extent for 
it to matter" at the end of the sentence I quoted. It seems 
reasonable to assume that, for without it the criticism can be 
dismissed straight away.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
That must be wonderful: I don't understand it at all.
 		-- Moliere
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: Scott Brim <[email-masked]>
Date: Tue, 20 Jan 2009 13:24:20 -0500

Excerpts from Brian E Carpenter on Tue, Jan 20, 2009 01:01:02PM +1300:
> The e2e principle seems to make it clear that, ultimately, liveness
> is the endpoint's responsibility, but that is usually interpreted as
> a statement about what the transport layer or above should do. And
> they already do it, 100% independently of what we do in layer 3 and
> below.  I think that limiting layer 3 liveness detection to the
> prefix level (except presumably in the case of Mobile IP) is a
> perfectly good engineering compromise, unless we're planning to
> repudiate [Saltzer].

First let's step back and reiterate that we're talking about how to
scale e2e liveness monitoring and failure recovery.  xTRs and
map-and-encap are not (necessarily) involved, although _monitoring_
intermediaries may be.

Disregarding layers and what their responsibilities might be, if I
understand correctly you're saying you would do prefix-level liveness
monitoring for possible paths but only do specific address-level
monitoring for address pairs that are currently in use?  If so, are you
saying that if that's the best you can get, you'll take it?  Or do you
really think that's good enough for end users?

> prefix.  (The absence of a prefix, that's useful.  But that's the easy  

xTRs are only involved in a map-and-encap situation.  We _may_ have a
map-and-encap situation here but it's not required.

> There are many additional problems...)

Right ... the requirements on xTRs for liveness is completely different
from those for end-to-end.  xTRs take responsibility for getting traffic
between _prefixes_, as opposed to endpoints.

Excerpts from Pekka Nikander on Tue, Jan 20, 2009 12:28:44PM +0200:
>> trying to reach endpoints behind the same set of ETRs.  As such,  
>> having a border guy (or someone else, if you really want to complicate 
>> life) testing / monitoring that liveness / reachability on behalf of 
>> the various sources within the site would seem likely to
> My point was a simple one: not necessarily so.  If TR-level granularity 
> is acceptable, 

That's the big question.  I'm surprised people would find it acceptable.
Think about how individual machines are multihomed between different
providers today.  If one of your interfaces goes down, the prefix you
are part of in that provider is still available, so your peers would
continue to try to talk to you there.  I don't see how prefix liveness
tests could take the place of e2e liveness tests.  If I understand Brian
correctly he thinks higher layer tests will take care of this problem,
but we still need a way for those higher layer tests to be performed on
src/dst pairs other than the ones in use by transport already.

Happy Obama Day ... Scott

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 21 Jan 2009 10:16:12 +1300

> really think that's good enough for end users?

I think I'm saying that's good enough for transport and session layers
that apply the e2e principle, and that the nature of RFC791 is such
that they all apply that principle today. It is not the network layer's
job to be perfect.

One thing that may be useful is an API that allows the upper layer to
tell the local IP stack "Hey, I'm seeing a liveness problem, please
check it out." That would make constant liveness checks by the network
layer unnecessary.

    Brian

---

From: Robin Whittle <[email-masked]>
Date: Thu, 22 Jan 2009 14:21:28 +1100

However, neither of them scale acceptably, so
                      there is no point seriously considering a
                      hybrid.

                      The only reachability testing system which
                      scales acceptably is Ivip's approach of
                      using a separate set of servers to probe full
                      reachability, via multiple ETRs, to the
                      destination network.  The probing load does
                      not increase in proportion to either the number
                      of ITRs or sending hosts currently sending
                      packets to the destination network.

                      If the ITR was in the ISP, then Ivip's approach
                      does not cope with certain patterns of outages
                      which ITR-based or host-based reachability
                      testing approaches could probably cope with.
                      However, as long as the ITR is in the sending
                      host or the multihomed sending host's network
                      (or is an OITRD) then the outcome is just as
                      good as today, since BGP copes with those
                      outages pretty quickly.

                      While host-based probing could cope well with
                      an outage to the upstream ISP compared with
                      ITR-based probing from an ISP-located ITR, this
                      is already coped with perfectly well by LISP
                      etc. or Ivip where the multihomed end-user
                      network has its own ITRs.  (Non-multihomed
                      end-user networks can have their own ITRs or
                      use ITRs in the single upstream ISP.)

>>   2) increase the odds of having accurate data when packets need to be

A further reduction in probing, and greater flexibility in deciding
which ETR the traffic packets will be tunneled to can be achieved by
the Ivip approach of using a completely separate reachability probing
system - separate from ITRs and from sending hosts.  The end-user
would either do this, or have some company do it for them.  The
probing could be done from multiple points all over the Net, to the
ETRs or more likely through the ETRs to routers and/or hosts in the
end-user network.

Most likely, that company's specialised probing system would be
configured to make the decision on how best to map the destination
network's micronets - and would change the mapping within seconds,
according to whatever criteria were specified by the administrators
of the destination network.  Within a few seconds, all ITRs in the
world handling packets addressed to these micronets would be
tunneling these packets to the ETR chosen in the new mapping decision.

This is a generalised DFZ->destination-network probing approach,
since the probing servers would, broadly speaking, be in the DFZ.
This is the only scalable approach, since the same level of probing
would still occur if 100,000 ITRs were sending packets to the
destination network as if one, none or a few were sending packets.

The key to using a separate, dedicated, reachability probing system
(quite outside the Ivip system itself, and so which can be made to
work on any principles, any protocols etc. which suit the destination
network being probed) is Ivip's real-time mapping distribution
system.  This tells all the ITRs which need to know which ETR to
tunnel the packets to.  This greatly simplifies the ITR and ETR
design and separates out reachability testing and the resulting
decision-making from the core-edge-separation system itself. (LISP,
APT, TRRP and Six/One Router monolithically integrate them.)

So having something other than the ITRs doing the probing and making
the decisions does involve some extra complexity - a real-time
mapping system.  I believe this is a small price to pay for the
greater flexibility, more robust probing (all the way to the
destination network, not just to the ETRs), greater simplicity in
ITRs and ETRs, reduction in probing traffic etc.  Also, it enables
real-time control of ETR address for incoming TE.

> the TR granularity, and host-based on host granularity.  

The last sentence applies to LISP, APT, TRRP etc. but not to Ivip.

With Ivip, end-user networks can and generally will probe
reachability as they choose, including most likely through each ETR
to their network, including to individual hosts if they like.

So this is finer grained at the receiving end than just to ETR
granularity.  One ETR in an ISP may be perfectly reachable and may
reach multiple destination end-user networks, but not the destination
end-user network in question.  Ivip copes with this perfectly well -
that network's micronets are mapped to an ETR (typically in another
ISP) which can reach the network.

The probing would typically be done from multiple sites, including
from actual ITRs or through actual ITRs, all over the Net.

Host-to-host reachability testing is arguably the most adaptable,
since it tests the full path used by traffic packets.  So this
approach can, in principle, cause the choices made by the sending
host (which address of the destination use to send the packets to) to
adapt to every conceivable network outage.

ITR-to-ETR probing on its own doesn't tell each ITR about the
destination network being unreachable - so it needs to be done in a
way that the ETR confirms the network is reachable.  This could be
done via:

1 - ITR->ETR probing, with a special protocol telling the ETR which
    end-user network (or perhaps an EID address within that network)
    the probe concerns.  In this way, the ETR can return a message
    that it is reachable, but that the destination network is not.

2 - ITR->(transparently through the ETR)->destination-network probing.
    This would require some new protocol or some identified router
    in the destination network which would always respond to
    existing probe protocols.  This would require extra information
    in the mapping data, to specify the one or more addresses in the
    destination network each ITR should probe.

I figure LISP etc. will need to do one of the above.  AFAIK, at
present, LISP only involves ITRs probing reachability of whole ETRs -
not the ability of the ETR to deliver the packets to any particular
destination EID.

In this diagram (from my message Re: No liveness requirement in the
ID/Loc Split concept 15 Jan) the sending host is HA, the ITRs are
shown in the ISP networks.  (However, the ITRs could be in the NA
sending host's network, or with Ivip, the ITR function could be built
into the sending host.  I depict this in Fig 2.)

                        ~~~~~~~~~~~
   NA           ISP1   ~           ~   ISP3         NB
            PE1      BR1---     ---BR3     PE3
 ........  /    ITR1   ~\         /~   ITR3   \  ........
 .      . /     ETR1   ~ \       / ~   ETR3    \ .      .
 . HA---CEA            ~    DFZ    ~           CEB---HB .
 .      . \            ~ /       \ ~           / .      .
 ........  \   ISP2    ~/         \~   ISP4   /  ........
            PE2      BR2---     ---BR4      PE4
               ITR2    ~           ~   ITR4
               ETR2     ~~~~~~~~~~~    ETR4

(Fig 1. LISP etc. or Ivip with ITRs in ISPs.)

Either form of probing from the ITR is able to cope fine with any DFZ
outages between the ITR (ITR1 or ITR2, depending on how CEA sends out
the packets) and the two ETRs in question: ETR3 and ETR4.

In addition, either form of probing from the ITR can cope separately
with (for instance) an outage between ITR1 and ETR4, while from some
other ITR, for another sending host, there is an outage to ETR3 but
not to ETR4.

Ivip's approach is not so flexible.  With Ivip, I am assuming that
each ETR is either:

1 - Reachable from essentially every ITR, or

2 - Unreachable from some (or all) ITRs to the degree that the
    destination-network's probing system will detect the problem and
    change the mapping to the ETR which is reachable.

So ITR-based probing appears to have this advantage over this
approach h (ITRs in the ISPs) to Ivip.  The advantage is limited,
however, since BGP already does a pretty good job of resolving loss
of links and routers between ISPs.   The proper way to deploy Ivip is
to ensure that every multihomed end-user network has its own ITRs.

Host-based probing has the additional advantage that it can detect
outages from the HA to the upstream ISPs ISP1 and ISP2.  The same
would apply for an ITR-based probing system, if the ITR was located
in the sending host's network NA.

However, the problem of outages from the sending host to the upstream
ISPs is not one which requires any special solutions, since the
sending-host's network's Customer Edge (CEA) router does this task
just fine.  This will continue to work fine for Ivip and for LISP,
APT, TRRP etc. no matter whether the ITR is located in the sending
host's network or in the upstream ISP.

Also, maybe in the case of non-upgraded networks in the Ivip-era or
LISP-era which are either single-homed or multihomed with current BGP
techniques,  here are no ITRs in the ISP or the sending host's
network.  In that case, there are Ivip OITRDs (Open ITRs in the DFZ)
or LISP PTRs (Proxy Tunnel Routers) somewhere in the DFZ.  Again, the
existing behavior of the CE router in the sending host's network
automatically sends outgoing packets to whichever ISP is currently
preferred according to outgoing TE preferences and which of the ISPs
are currently reachable and advertising the micronet (Ivip) or EID
(LISP) prefixes which are advertised by the OITRDs or PTRs.

The next section concerns how ITR-based probing or Ivip's approach
works when an upstream ISP is reachable, but that ISP has either no
connectivity to the rest of the Net, or patchy connectivity.

  LISP etc. with ITRs in the ISPs:

     Packets would be lost if the CE router sent packets to ISP1
     where ITR1 accepted them, but was then unable to send them to
     the ETR.

     (Would ITR1 recognise it couldn't reach beyond ISP1 and then
     somehow signal this so CEA would no longer send packets to
     ISP1?)

  Ivip with ITRs in the ISPs:

     As for LISP, but this is not how Ivip should be deployed -
     multihomed networks should either have their own ITRs, or
     rely entirely on OITRDs outside their network and outside
     their ISPs.

The problem in both cases is that CEA is sending packets to ISP1
which can't get them to their destination.  That is beyond the scope
of LISP etc. or Ivip to solve.

A better outcome is if the ITR is in the sending network, assuming
this is a multihomed end-user network.  For singlehomed end-user
networks, there is no difference in this probing respect between
having the ITR in the end-user network or in the single upstream ISP.
 (Also, OITRDs or PTRs are beyond these ISPs and are somewhere in the
DFZ.)

                            ~~~~~~~~~~~
   NA               ISP1   ~           ~   ISP3         NB
                PE1      BR1---     ---BR3     PE3
 ............  /           ~\         /~   ITR3   \  ........
 .          . /     ETR1   ~ \       / ~   ETR3    \ .      .
 . HA-ITRA-CEA            ~    DFZ    ~           CEB---HB .
 .          . \            ~ /       \ ~           / .      .
 ............  \   ISP2    ~/         \~   ISP4   /  ........
                PE2      BR2---     ---BR4      PE4
                           ~           ~   ITR4
                   ETR2     ~~~~~~~~~~~    ETR4

(Fig 2 - Sending host network has its own ITRs, and/or ITR function
in the sending host.)

  LISP etc. with ITRs in the sending host's network (NA):

    Now there is no problem, since CEA is handling packets which
    have already been encapsulated.  If ISP1 is disconnected from the
    Net, its PE router will not be advertising any prefixes, so CEA
    will send all outgoing packets to ISP2.

    Note, this is not applicable to Six/One Router - the equivalents
    of ITRs are always in the ISPs.

  Ivip with ITRs in the sending host's network (NA):

    As above and likewise with Ivip's forwarding approaches, based on
    the ETR address (or some of the address, for IPv6) being encoded
    into a modified IP header.  There is no need for the actual ITRs
    to do probing, since the disconnection of ISP1 from the Net is
    detected by CEA due to ISP1's PE1 router no longer advertising
    routes to any addresses outside ISP1.

So while host-based probing also detects problems as close as the
upstream ISP, the core-edge separation systems (LISP, APT, TRRP and
Ivip) have no problem in this regard, as long as each upgraded
multihomed end-user network has its own ITRs.  (Or non-upgraded
BGP-multihomed networks rely on OITRDs/PTRs rather than ITRs in their
upstream ISPs.)

> This has
> received criticism that a host-based model causes more

My critique along these lines is:

  Fundamental objections to a host-based scalable routing solution
  http://www.irtf.org/pipermail/rrg/2008-November/000233.html

> is acceptable, then it is doable with a hybrid host+TR-based solution
> (like the one I've been hand waving during the last month or so), and in
> that case the amount of reachability testing / signalling will be at the
> *same* level as for a TR-only solution.

I don't think ITR-based reachability testing is acceptable.

I think the only scalable approach is to have neither sending hosts
nor ITRs doing the reachability probing and decision-making.  The
only proposal so far which achieves this is Ivip.

 - Robin                           http://www.firstpr.com.au/ip/ivip/

---

From: Shane Amante <[email-masked]>
Date: Wed, 21 Jan 2009 22:28:22 -0700

I find this whole discussion of a separate reachability probing system  
to be fraught with a number of economic, security, more importantly,  
operational problems.

> ETRs or more likely through the ETRs to routers and/or hosts in the

Really?  There is no way companies are going to allow one of these 3rd- 
party "probing/reachability companies", even if they have a paid  
relationship with them, to probe through their ETR's to their internal  
network.  Who's to say that the probing company (or companies) doesn't  
get hacked and they're used as a launching point to attack the inside  
network of a company using them for services?

> APT, TRRP and Six/One Router monolithically integrate them.)

This won't fly operationally.  First, I can't envision an economic  
model that would cause someone or, better yet, multiple probing/ 
reachability companies to be launched as you envision.  Ultimately,  
you're talking about not only a lot of fixed costs for servers and  
such, but more importantly the OpEx for colocation costs and bandwidth  
that company would be burning to send out millions, billions or  
trillions of "probes" to everyone's ETR on the planet.

Second, these probes from a 3rd-party probing/reachability company  
will not reliably tell end-user networks that there is, in fact, known  
"good" connectivity to an ETR, because of ECMP and/or LAG being used  
along certain paths within SP's networks and not others.  More  
specifically, ECMP and/or LAG are load-balancing mechanisms that SP's  
widely use to scale the physical BW between nodes in their network,  
(e.g.: to scale BW between a city-pair to 100G and [much] larger by  
logically bonding together multiple OC-192's, etc. into a single  
"bundle").  The key part of those technologies is core routers use IP  
header information, (L3 addresses and/or L4 port information), as  
input keys to their load-hashing algorithms to determine the  
particular output link in a LAG or ECMP "bundle" that a particular  
flow goes on.  The problem observed in operational networks today is  
that "soft-failures" cause a particular link in a bundle to stop  
forwarding traffic, which goes unnoticed to IGP's (OSPF or IS-IS) or  
BGP and, unfortunately, results in blackholing of customer traffic  
until the problem is isolated and "bad" link in the bundle is taken  
out of service.  Unfortunately, there aren't tools to diagnose this  
problem today so it's very difficult and time-consuming to  
troubleshoot & resolve these problems.  More to the point of why your  
proposal of 3rd-party probing companies will not work is:
- When these 3rd-party probing companies are transmitting probes  
toward an ETR they're going to be unable to construct *identical* EID  
within RLOC packets that would cause them to be hashed and push on the  
same links that are bound for that customer's ETR.  IOW, the 3rd-party  
probe company *will* get false-positives that will either: a)  leave  
the ETR's in service when there is a bad link blackholing traffic on a  
parallel path unseen by the 3rd-party company; or, b) falsely take an  
ETR out-of-service because the 3rd-party probing company saw traffic  
being blackholed on a link that may, in fact, carry very little or  
know end-user traffic toward that ETR.
- Economically, it's infeasible for this imaginary probing company, or  
set of probing companies, to be able to deploy servers to every POP in  
every ISP to ensure full coverage of these LAG & ECMP paths ... or,  
for that matter, a decent percentage of each operator's network that  
it can reasonably approximate reachability across all paths in that  
every ISP's networks.
- As an operator, I don't have much (any?) faith in trusting others to  
determine connectivity (or lack thereof) to/from my ETR's.  I trust my  
network and the configuration of my ITR's/ETR's to determine that.   
Furthermore, when someone goes wrong (e.g.: probes aren't returned) I  
can easily login to those devices (since I own them) and quickly  
troubleshoot the problem and restore service how *I* deem appropriate  
for my network.

So, to summarize, *if* we have to do probing, the only operationally  
and economically feasible method is either to/from the ITR's/ETR's or  
the end-systems themselves.  (More on this just below).

> real-time control of ETR address for incoming TE.

This discussion does raise an interesting architectural point with  
respect to all map-and-encap solutions, which depend on reachability  
probing between ITR's and ETR's.  Specifically, do these reachability  
probes faithfully represent end-user (host-machine originated and  
terminated) flows so they are travelling along the same path as host- 
to-host data flows?  I would argue that they do not, (without  
substantially increasing the probing traffic load/bandwidth to  
"search" through all possible paths used by end-user traffic).

OTOH, solutions that either:
a) do proactive probing originating from hosts at something at or just  
below the TCP/UDP layer; or, better yet,
b) piggyback ***reachability*** detection along with active TCP/UDP  
traffic flow (cf: TCP ACK's) between hosts
... would faithfully represent that a given path really is working.

FWIW, I don't think there's a straightforward solution for map-and- 
encap solutions to the aforementioned probing over LAG/ECMP paths  
problem.  If there is, I'm all ears.  OTOH, if there's not a solution,  
I do not think we should gloss over it and, instead, I would strongly  
recommend we should point this out out very clearly within an  
"Operational Considerations" section of any/all map-and-encaps  
protocol specs or an overall/all-encompassing companion "Operational  
Considerations" document noting this and other 'challenges' that map- 
and-encaps can't feasibly address.  Said document(s) may prove helpful  
in evaluating the final set of candidate solutions that the RRG will  
offer its recommendation on.

-shane

---

From: David Conrad <[email-masked]>
Date: Wed, 21 Jan 2009 21:37:54 -0800

On Jan 20, 2009, at 9:56 AM, Paul Jakma wrote:
>>>> a host-based model causes more signalling/reachability testing  
>>>> traffic than the tunnel-router-based one.
>>> data to back it up?
> it to matter" at the end of the sentence I quoted.

What meets your requirement of enough "to matter"?  Depending on  
deployment scenario, a tunnel-router-based solution could cover  
thousands of hosts or more.  Is that enough to matter?

Regards,
-drc

---

From: Paul Jakma <[email-masked]>
Date: Thu, 22 Jan 2009 13:12:40 +0000 (GMT)

> Paul,
>
>> I assumed there was an implied ", to a significant enough extent for it to 
>> matter" at the end of the sentence I quoted.
>

It's going to be subjective to a point, but determining "to matter" 
would involve comparing the control-overhead to the actual 
data-bearing traffic, rather than focusing solely on the difference 
in control traffic between end-host and intermediary based solutions.

Having a tunnel-router do the liveliness tests for 1000 hosts might 
save 999 instances of control traffic, but if the 999 instances of 
control traffic comprise 1-2% of the data traffic - then should we 
care? (Is there a data-communications version of Amdahl's Law I can 
quote here?).

By my back-of-the-envelope calculations, a Shim6/REAP solution has 
Order(1.2%) overhead relative to data-traffic (on a low amount of 
traffic, 24kB - e.g. fetching the google front-page) for the normal 
case. Failure cases presumably might take slightly more - but failure 
is not the common case. Solid quantitative studies would be really 
useful.

So basically, where's the awful overhead of host-based signalling, 
relative to what matters: the actual data?

It seems it would be good to have data to back up such an assertion, 
rather than accepting it blindly and introducing a lot of extra 
complexity (and losing much fate-sharing) to optimise an apparently 
relatively-trivial overhead.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
   "I always avoid prophesying beforehand because it is much better
   to prophesy after the event has already taken place. " - Winston
   Churchill
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: Scott Brim <[email-masked]>
Date: Thu, 22 Jan 2009 14:19:19 -0500

> On Wed, 21 Jan 2009, David Conrad wrote:
>
>> Paul,
>>
> would involve comparing the control-overhead to the actual data-bearing 
> traffic, rather than focusing solely on the difference in control traffic 
> between end-host and intermediary based solutions.
>
> Having a tunnel-router do the liveliness tests for 1000 hosts might save 
> 999 instances of control traffic, but if the 999 instances of control 
> traffic comprise 1-2% of the data traffic - then should we care? (Is 
> there a data-communications version of Amdahl's Law I can quote here?).
>
> case. Failure cases presumably might take slightly more - but failure is 
> not the common case. Solid quantitative studies would be really useful.

Could you show us your BOTEC?  I've been meaning to go gather some
real numbers.

Scott

---

From: Christian Vogt <[email-masked]>
Date: Wed, 28 Jan 2009 17:10:10 -0800

> check it out." That would make constant liveness checks by the network

Right.  This would also account for the fact that the result of probing
does not necessarily correspond to the success/failure of data delivery
-- independent of whether the probing packets are sent by a host or a
router.  One potential cause for incorrect probing results is that data
packets get filtered due to the protocol they carry, whereas probing
packets pass through filters.  Another potential cause is that data
packets and probing packets are routed via different paths (cf. ECMP) of
which only one is functional.

Only the use of data packets themselves as probes can avoid this issue.
But then, if we use data packets as probes, we eliminate the overhead of
probing in the first place...

- Christian