Subject: Re: Last Call on LDP Fault Tollerance

From: Curtis Villamizar <[email-masked]>
Date: Tue, 03 Apr 2001 15:06:44 -0400

In message <[email-masked]>, George Swallow writes:
> This message commences a workgroup last call on LDP Fault Tollerance,      
> <draft-ietf-mpls-ldp-ft-01.txt>.  The last call closes April 11, 12 pm
> GMT.
> 
> ...George
> 
> ======================================================================
> George Swallow       Cisco Systems                   (978) 244-8143
>                      250 Apollo Drive
>                      Chelmsford, Ma 01824

Adrian, et al,

I would like to make a comment and leave it up to the authors
discression as to whether to incorporate any change into the document.

There appears to be a school of thought where it is deemed beneficial
to keep forwarding state intact in only very limited cases such as
midnight controlled reload of routers for purposes such as software
upgrade.  This case can be even further limited to maintaining
forwarding state only if network topology has not changed while
adjacency is unavailable.

The justification for retaining forwarding state is to eliminate
unnecessary forwarding state churn that may cause transient forwarding
disruptions.  The justification for not retaining forwarding state if
topology changes is to prevent persistent forwarding problems that
could occur if loops or black holes formed as a result of inconsistent
forwarding state on a router in which the means to synchronize state
was unavailable.

Stated more simply -- Its nice to avoid dumping a few packets on the
floor but if topology changes it may be better to endure some
transient route churn than to endure inconsistent routing state that
lasts for the duration of a router reloading new software and could
dump a whole lot more packets on the floor than what the transient
would cause.

If changes are made, one possible addition would be to add a section
4.4.2:

  4.4.2  Optional Reconnect Expire on Topology Change

  <add some subset of the wording above or some explanation of why
  this is needed from the authors>

   If topology change is detected through some means outside LDP such
   as change in an Interior Gateway Protocol Link State Database
   (IGP-LSDB) it may be desireable to prematurely expire the FT
   Reconnection Timer.  An option SHOULD be provided to enable such a
   feature on a per peer basis.

I leave it entirely up to the authors as to whether they would like to
add this text.  I would prefer if something to this effect were added
unless there is objection to this addition on the list.  A vendor
could always include such a feature and make it optional and disabled
by default regardless of whether it is stated in the document so there
is little sense of urgency.

Curtis

---

From: Eric Gray <[email-masked]>
Date: Tue, 03 Apr 2001 16:49:45 -0400

Curtis,

    In general, I found this proposal not unreasonable.  On
reflection however, I have two relatively minor issues with
it - each having an additive effect making the resulting
concern a little less minor.

    The first issue is actually a two parter.  Once you start
adding knobs for the user to configure when the protocol bumps
this way and when it bumps that way you do two things:

    1) you make the implementing and testing more complex and
    2) you open the door to suggestions for similar knobs.

    The second is that we currently have no experiential evidence
to indicate that implementers will get this wrong and deployers
will get it right.  I believe we should wait for such evidence to
start coming in before adding an optional knob for optional
behaviors based on various permutations of the presence, or
absence of other optional knobs and behaviors.

    I believe that the knob you are asking for will actually only
complicate implementation and testing for those people who
are trying to get the implementation right.  In other words, a
correct implementation should internally determine on its own
whether or not the topology information on which LDP bases
LSP forwarding is suspect.  Adding a knob to have such an
implementation either not perform the topology sanity check
- or ignore the results if it does (by assuming insanity) will
introduce additional test cases.

    I also suspect that you may be understating the advantages
of maintaining session context.  Consider the impact to the
network if LSPs are dropped lightly and these LSP are in turn
being used to carry (stacked) LSPs.

--
Eric Gray

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 03 Apr 2001 17:16:07 -0400

> being used to carry (stacked) LSPs.

You wouldn't be using LDP in that case.  ;-)

Curtis

---

From: Eric Gray <[email-masked]>
Date: Tue, 03 Apr 2001 17:33:31 -0400

Curtis,

    As I understand it, LDP is the preferred protocol for use
with MPLS BGP when BGP peers are not adjacent with
respect to an MPLS domain.  Suppose that MPLS BGP is
used to establish many VPN services which are possibly
being used to provide a label switching service to enterprise
customers.

    For that matter, suppose that LDP is being used to signal
and maintain <your favorite> VPN technology's LSPs.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 03 Apr 2001 20:11:17 -0400

> and maintain <your favorite> VPN technology's LSPs.

Eric,

You wouldn't be using LDP on the outer tunnel, just the inner VPN
tunnel.  The outer tunnel is a RSVP-TE tunnel.  We are already lab
testing this at a customer and it interoperates nicely with other
people's BGP/VPN LDP implementations.

Because the TE tunnel is rerouted around a failure the LDP adjacency
that ride inside the TE tunnels never go down.  The LDP TCP connection
sees some temporary loss at worst and then continues as if no topology
change occurred because the logical adjacency never went down.  This
works as long as there is a brief holddown on the TE tunnel and it
reroutes before the holddown expires.  In practice works great.

Where you want to maintain LDP (and IGP) adjacency is where a router
is reloaded for something like software upgrade and nothing else in
the IGP changes during the reload time.  If you have two route
processors on the router that was reloaded, that should be a switch in
well under a second.  Rumor has it that some routers with one route
processor take a few minutes to reload so for those, if the topology
changes you'd rather route around them.

So please remind me, exactly what was the point of your objection.
:-)

Curtis

---

From: Eric Gray <[email-masked]>
Date: Wed, 04 Apr 2001 12:13:53 -0400

Curtis,

    The discussion of whether or not a network operator would
use LDP - with or without fault tolerance - as a mechanism
for establishing LSPs for the purpose of transporting other
LSPs is a side-track, down which I do not wish to proceed
any farther.  I raised the point only to establish that there is
another perspective and I believe I have accomplished that.

    Back on the issue you raised.  As anyone who might have
come close to correctly implementing LDP will know, routing
drives LSP set-up and tear down.  If an LSR looses a peer in
a routing protocol (pick one, any one),  routes are re-computed.
For routes that have changed upon re-computation, LDP gives
a rigorous set of procedures for determination of appropriate
response with respect to existing LSPs (and - for the liberal
retention case - validity of labels not even in use).

    Note that LDP procedures are defined indepedent from the
specific _internal_ procedures that a router uses to determine
that a route has changed.  If you do not feel that this is the
case, then I suggest you raise this issue with the working
group as an issue with LDP rather than with LDP FT.

    In reading the current LDP FT specification, you probably
discovered that it describes exactly how a fault tolerant LDP
implementation is expected to deal with labels that are made
invalid for the entire period during which it is unable to obtain
confirmation that a peer has been informed of the invalidation.
If there are details of this process that are not clear to you,
please address those details.

    If, as I assert, LDP procedures are independent of specific
routing protocols and LDP FT defines procedures for ensuring
that LDP procedures are adhered to across multiple sessions,
then LDP FT is equally independent of specific routing protocols.

    Hopefully, this will clear up any fear, uncertainty or other
doubts that this discussion may have raised. :-)

--
Eric Gray

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 04 Apr 2001 20:20:33 -0400

> another perspective and I believe I have accomplished that.

I agree.  Whether LDP or CR-LDP would ever be used on the outer tunnel
is a separate issue.

> group as an issue with LDP rather than with LDP FT.

If you have A-B-D-E and A-C-D-E and A-W-X-Y-Z-E and B just went down,
a topology change at D-E (D-E went down) may make the path A-B-C a
poor choice.  C may have decided that the best path is by way of A
since D-E is unavailable and is dutifully trying to inform B of the
situation with a Label Withdraw, who will take note of it and inform A
in due time after its finished rebooting.

In your example on page 24, C would play the role of P2 and queue the
label withdraw.  B would be P1.  A is on the other side of P1.  Maybe
I'm missing something.

> please address those details.

Ok.  I just went through it again and I don't see how the above
example is handled.  Maybe I'm just dense but I missed it and I'd like
you to explain how the LDP-FT procedures handle the above example in
the event that it take B some 20-30 seconds to come back up.

> doubts that this discussion may have raised. :-)

Your explanation will certainly clear things up.

Thanks in advance,

Curtis

---

From: Eric Gray <[email-masked]>
Date: Wed, 04 Apr 2001 22:19:57 -0400

Curtis,

    First of all, I thought you were concerned that the fact
that the LDP peer relationship was on hold would cause
special problems when re-establishing the relationship.
It is possible that this is not clear.  However - based on
your example, it now seems to me that your concern is
about forwarding during the TCP outage.  This should not
be a problem because the specification is intended to
support LDP sessions across a TCP connection failure.
LDP itself should continue to function - in terms of the
procedures defined with respect to the use of LSPs and
the impact of route changes.  If routing says that a path
is not valid, then the corresponding LSP is not either.

    Please see below.

You wrote:

> >     ...

Since you didn't draw a picture, I'll try to unscramble this
alphabet soup.

My guess is the picture is supposed to look like this:

        B
       /|\
      / | \
 --- A  |  D --- E ---
     |\ | /      |
     | \|/       |
     |  C        Z
     |           |
     W --- X --- Y

Is that the case?

Also, guessing what you mean by B going down and D-E
going down you mean that B is undergoing some kind of
software maintenance which results in a temporary TCP
failure and, subsequent to this, the link between D and E
becomes inoperative.

First of all, this is a two-failure scenario and - as such -
is notoriously hard to protect against in any case.  I can
imagine a double failure of a single routing protocol that
would produce worse results.

But I look at this specific problem and I see the following:

B is being upgraded (not rebooted - that would affect more
than TCP and TCP-based LDP sessions).

The link D-E goes down.

Since (A,B,C,D,E,W,X,Y,Z) are all in the same routing
domain and all are still active routers (or are you suggesting
a third simultaneous failure?), all of them will eventually
detect the missing link D-E.

C determines that the best route for what used to be the
path A-C-D-E is now A-W-X-Y-Z.  C drops packets
it already has received from A because split horizon dis-
allows forwarding packets received from A back to A.
In any case, A also will (eventually) know that these
packets should go along the path A-W-X-Y-Z so on
their merry way they go.

Meanwhile - assuming that B is not yet aware of the topology
change - B continues to forward packets on the path A-B-D-E
but D is unable to forward them via E and sends them on a new
path through C-A-W-X-Y-Z - making the path that packets
traverse A-B-D-C-A-W-X-Y-Z-E for the smallish period of
time it may take A to realize that it should not be forwarding the
packets via B (it will eventually know that the path A-B-D-E
is no longer valid and that A-B-D-C-A-W-X-Y-Z-E is not
the shortest path) and B to realize that it should not be forwarding
the packets via D.

(Note - I am assuming that there is some reason why the
packets went A-B-D-E in the first place - as opposed to
A-C-D-E and that reason is related to the fact that they
were received on the 'left' interface of A.  If this is not
the case, then packet will loop along A-B-D-C-A...
until A figures out that it shouldn't be sending packets
to B that are intended to go through E.)

As soon as B becomes aware that the shortest path for packets
it is receiving from A for the path A-B-D-E is now through
A-W-X-Y-Z-E, it will start dropping packets (split horizon
again).  As soon as A realizes that the ONLY path to E is
A-W-X-Y-Z-E, it will stop forwarding packets to B and C.

So the fact that B is not in a position to pass on label release
and withdraw messages should be irrelevant to the problem.
The labels may be there, but nobody should be using them.
B should not continue to forward packet on a bad path and -
even if it does, A, C and D will do the right thing.  Yes, you
will have temporary looping of data, but you would have had
the exact same looping data without MPLS, LDP or LDP FT.
And, unless B is the ingress for CR-LDP based LSPs, the
ingress (at least to the routing domain) _will_ know about
the topology change and _will_ be able to re-route LSPs
accordingly.

Obviously, there are special problems with having the ingress
to an explictly routed LSP go off-line.  It will not be able to
signal new explicit routed LSPs until after it re-establishes
LDP peer relationships.  But it will not erroneously forward
packets on a bad path either.

This is just a specific analysis showing how what I said is
generally true is true in this specific case.  What is it you
think the specification needs to add to this?

> the event that it take B some 20-30 seconds to come back up.

As explained above, the example I believe you are citing is
handled by LDP and routing based finesse.

> Your explanation will certainly clear things up.

I hope so.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 05 Apr 2001 12:51:49 -0400

> My guess is the picture is supposed to look like this:

My description was off.  Should be:

  If you have A-B-D-E and A-C-D-E and A-W-X-Y-Z-E and B just went
  down, a topology change at D-E (D-E went down) may make the path
  A-B-D a poor choice.  D may have decided that the best path is by
      ^                 ^
  way of A since D-E is unavailable and is dutifully trying to inform
  B of the situation with a Label Withdraw, who will take note of it
  and inform A in due time after its finished rebooting.

Note the two places where C is changed to D above.  This is the same
picture you have without the B-C link.

I answered my own question.

In RFC3036 "A.1.7. Detect Change in FEC Next Hop" the "Received Label
Mapping" event generated in NH.12 and Note 2 cover this case as long
as the next hop changed.

If we look at a similar example where C were not there and there was a
B-E link (or B-...-E path) that was higher cost than B-D-E, then A
would not see a next hop change.

If we want to prevent a persistent loop, the very minimum would be for
A to reject consideration of any FEC through B for which the next hop
from B has changed because B has not reflected the change.  There is
no such procedure in draft-ietf-mpls-ldp-ft-01.txt.

A simpler approach was taken in the OSPF WG.  Adjacencies to the
router that was down were declared down (the IGP Grace timer was
prematurely expired) when there was a change in the topology.

By virtue of the IGP Grace timer expiring LDP would see a next hop
change so this case too is covered as long as the IGP prematurely
expires its Grace timer.

This would mean that LDP does not also have to prematurely expires its
timer, though it would seem that there would be no difference in
behaviour since all traffic is already diverted away by virtue of all
IGP next hops being changed to go elsewhere.

It would only be if the IGP did not expire its Grace timer that there
would be a problem in LDP (and the IGP and therefore things like IBGP
would be even worse off, so the IGP would clearly need to do
something).

Whether the LDP timer is prematurely expired does not effect whether
LDP routes are diverted away from the downed router.  If the timer is
prematurely expired, it just changes the behavior during recovery,
only impacting how soon FEC are sent through that router again.

Either keeping the text as is or recommending prematurely expiring the
timer both seem to be fine options.  Since it is already last call and
there is no urgency I can see good reason not to make this change.
Thanks for the time taken for this discussion.

Curtis

---

From: Eric Gray <[email-masked]>
Date: Thu, 05 Apr 2001 13:47:08 -0400

Curtis,

    Thanks for the clarifications.

> would not see a next hop change.

Nor would it need to.  Now you have proposed the following
picture:

         B -------.
        / \       |
       /   \      |
  --- A     D --- E ---
      |\   /      |
      | \ /       |
      |  C        Z
      |           |
      W --- X --- Y

If path A-B-E is more expensive than path A-B-D-E,
but less expensive than A-W-X-Y-Z-E, then A will
not see a route change and will continue to forward
to B.  So what?  B has the perfectly legitimate
path via E (useful if B - for example - was using
liberal retention).  If B cannot, in fact, forward
packets on the link B-E, then it should not include
this link in its advertisements (a fact that I had
pointed out to the authors of the MPLS Framework
document 2 years ago).

In any case, this is not persistent in any serious
sense in the redundant control plane case.  In that
case, the switch over is in hundreds of milliseconds
at worst - making the recommended 5 second timeout
roughly equivalent to setting your watch by the
progression of lunar phases.

>
>
> A simpler approach was taken in the OSPF WG.

I'm happy for them.  However, they are solving a different
problem.

> Adjacencies to the

This is OSPF routers dealing with the fact that an OSPF
router that is known to be unable to process OSPF transactions
is now known to have an invalid topological picture.  The behavior
you describe makes sense in that context.

You had suggested that LDP should adopt a similar (or at least
related) approach.  I disagree.  In the LDP case, the co-resident
routing function is _not_ unable to process <whatever> routing
transactions.  Therfore, there is little justification in hamming
up the already reasonably well-defined interactions between
LDP and the still-functioning routing control plane by adding
the behavior proposed for OSPF.

> only impacting how soon FEC are sent through that router again.

Agreed.

> Thanks for the time taken for this discussion.

You're welcome.

--
Eric Gray