Subject: RE: Reconvergence micro-loops: problem and solutions overview

From: "Don Fedyk" <[email-masked]>
Date: Thu, 23 Sep 2004 15:51:42 -0400

Mike

Tunnels require mechanisms to set them up, and remove them. The details are
out of scope in your current tunnels draft. One can assume that the
procedures must be at least analogous to setting CSPF tunnels for MPLS. A
path, complete or partial, must be built and state maintained in some form.
But regardless they are not pure IP forwarding based. 

Regards,
Don 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of mike shand
> Sent: Thursday, September 23, 2004 3:05 PM
> 
> 
> At 14:39 23/09/2004 -0400, Alia Atlas wrote:
> >At 01:17 PM 9/23/2004, mike shand wrote:
> >>At 12:20 23/09/2004 -0400, Alia Atlas wrote:
> >>The advantage of the other two mechanisms is they actually 
> work when a
> >>link is added or the cost of a link is changed.  If you are 
> arguing that 
> >>micro-loop prevention is vital to support SLAs and avoid 
> traffic loss, 
> >>those same arguments seem to me to apply for when a link is added.
> >>>
> >>>Both the incremental cost advertisement and the ordered SPF 
> >>>approaches
> >>>solve the problem for the new link case as well as the 
> failed link case.
> >>
> >>And so do tunnels!
> >
> >How so?  I can infer what one might do, but it is not 
> described at all 
> >in
> >your draft, and I'm not certain what this would do to the 
> traffic load on 
> >the links.
> 
> In draft-bryant 6.2 (for example), globally replace "failure" by "new 
> adjacency" and that is pretty much it.
> 
> 
> 
> 
> >How and when does the use of the tunnels get terminated?
> 
> When everyone has converged (as for link down). e.g. on a timer
> 
> >What happens if there is a second change?  How is a new node handled,
> >where multiple new links may be learned of simultaneously, 
> or similarly a 
> >failed node?
> 
> Unrelated second changes are generally a problem (even with 
> ordered SPFs), 
> but the related set of changes caused by node down/up turn 
> out to be OK.
> 
>          Mike
> 
> 
> 
> >Alia
> >
> 
> 
> _______________________________________________
> Rtgwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rtgwg
> 
> 

_______________________________________________
Rtgwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rtgwg

---

From: mike shand <[email-masked]>
Date: Mon, 27 Sep 2004 14:50:13 +0100

>path, complete or partial, must be built and state maintained in some form.

I think one would assume wrongly :-) All that is required for the tunnel is 
the endpoint of the tunnel. There is no other state to be maintained, and 
the path is the natural forwarding path. i.e. to get a packet from X to 
Y,  X encaps the packet in a header with Y as the destination address, and 
forwards it. That's all. There is no path to be maintained. Nothing is 
required to "remove" the tunnel. You just stop using that FIB entry which 
tell you to encap.

         Mike

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 27 Sep 2004 15:32:52 +0100

Don Fedyk wrote:

>Mike
>
>Tunnels require mechanisms to set them up, and remove them. 
>
For IPFRR we would use multi-point IP tunnels. We propose that when you
configure a router to allow it to be used as an IPFRR repair point it 
multicasts
its tunnel endpoint identity in the IGP. The other routers, when they 
chose their
P nodes, just encapsulate their packets to that tunnel endpoint. Unlike MPLS
IP requires no further setup, maintainance or removal.

>The details are
>out of scope in your current tunnels draft.
>
>One can assume that the
>procedures must be at least analogous to setting CSPF tunnels for MPLS. 
>
Not at all.

>A
Not for IP.

>But regardless they are not pure IP forwarding based. 
In what way impure? The header on the front of a tunnelled packet is an 
IP header
so it's pure IP forwarding from the encap to the decap. The repairing router
has to encap, but logically this is just using an alternative next-hop. The
decapsulating router (at the chosen tunnel endpoint)  is doing IP tunnel
removal which is a well known IP packet operation.

- Stewart

---

From: Alia Atlas <[email-masked]>
Date: Mon, 27 Sep 2004 10:43:39 -0400

Mike,

At a minimum, there will need to be some agreement/signaling that the 
tunnel egress can handle that role, since Y will need to decapsulate the 
packet and agree that it could do so at the necessary rate.

Depending on the tunneling method, that could be a rather non-trivial 
change to the forwarding mechanism.

Of course, to obtain the coverage that draft-bryant is claiming, you also 
require the directed forwarding, which also requires more signaling 
communication and potential extensions to the forwarding mechanism.

As Pekka pointed out, there may also be additional complications about 
fragmentation caused by tunnels.  Does X  know the MTU to Y and fragment as 
necessary before encapsulation or does Y have to reassemble at line rate 
before forwarding the packets?  Of course, you could say that the path MTU 
is pre-configured, so it's mostly the operator's problem.

What happens though when there is a second failure?  Say there is traffic 
already encapsulated in a tunnel from X to Y which goes through T on its 
way to Y.  If T experiences a failure and decides to use its alternate, 
then T will try and encapsulate that traffic again.  Now you will run into 
the same MTU problem.  Obviously, IP FRR isn't trying to solve the multiple 
uncorrelated failures problem, but something reasonable should happen to 
the traffic.  For instance, that traffic shouldn't end up going to Y's 
control plane because the second and so on packet fragments don't have the 
original destination in them.

If a tunnel encapsulation were to be used, this would need to be described 
clearly.

Perhaps, rather than claiming that the tunneling mechanism is out of scope, 
when the type very clearly affects the scalability and operator complexity, 
you might care to tell the working group what the suggested tunneling type 
is?

Alia

At 09:50 AM 9/27/2004, mike shand wrote:
>I think one would assume wrongly :-) All that is required for the tunnel 
>is the endpoint of the tunnel. There is no other state to be maintained, 
>and the path is the natural forwarding path. i.e. to get a packet from X 
>to Y,  X encaps the packet in a header with Y as the destination address, 
>and forwards it. That's all. There is no path to be maintained. Nothing is

---

From: Alia Atlas <[email-masked]>
Date: Mon, 27 Sep 2004 11:01:08 -0400

The other complexity which any type of tunnel introduces is for additional 
protocols, such as LDP, which requires a label from the tunnel end-point.

With tunnels, X would need to maintain an LDP adjacency with every 
potential tunnel-endpoint Y, so that the labels are available.  If you go 
to a mechanism like the directed forwarding, then an adjacency would need 
to be maintained with every router to whom the traffic might be directed.

Since VPNs and pseudo-wires are a source of the traffic which needs better 
resiliency, this is a serious concern.

Alia

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Sep 2004 13:42:12 -0400

> required to "remove" the tunnel. You just stop using that FIB entry which 

That's interesting.

    Z -- N -- X
    |         |
    A -- B -- C

Cost of Z-A is 10 (or anything >5).  The path from A to Z is
A-B-C-X-N-Z.  If the link X-Y goes down, there is no "Y" that X can
forward to such that traffic will go to Z without coming back to X.
Are you saying that in this case tunneling just doesn't work.  If so,
the spec internet-draft needs to be changed to remove any mention of
complete coverage.

Since the i-d is rather vague about the tunneling this was not
previously clear.  You'd need an explicit path setup machanism, or in
something like IP-in-IP you'd need to put each hop on a forwarding
stack and suffer multiple layers of encapsulation (and possibly exceed
the MTU).

In Alia's U turn stuff, a break in B-C would be covered, but that's
it since A would forward traffic to its next best hop if the traffic
came from the normal best hop.  A break in C-X would not be covered
because it becomes too computationally expensive to determine cases
where cascading U-turn forwarding would help.  Since cycles like this
tend to be small in size (mostly 3 or 4 hop cycles) coverage for the
networks that Alia had studied was quite good and in the case where it
wasn't 100% could be made 100% with only a few circuit additions.

If the cost from Z-A was 4, then X still can use A as the "Y".  This
is a case where tunneling would work, but not U-turn.

This is the simplest case where there would be no traffic normally
flowing on the backup.  Arguably this situation would not exist on a
provider core and such a link would not make a very good backup if
there was reason to avoid it for all primary traffic.

There are other cases where for a node failure between X and Z would
result in failure with such a tunneling but where U-turn would work
and where the scenario is more likely to occur in a real network.  For
example, if the cost Z-A was 2 and there was a N-B link with cost 1.
For failure of node N, A would still be trying to send traffic to B.
The U-turn is fine with this since A would send traffic from its
normal next hop B, to Z if N failed, and X C would send traffic to B
if it came from its normal next hop.  The rule that you take a link
disjoint alternate next hop even if you aren't sure about node
disjoint, would have X sending to C in U-turn, even if X hadn't
figured out that B would not be sending to N either if N failed.

This means that neither U-turn or the proposed tunneling have 100%
coverage and the two methods have different topologies which result in
less than 100% coverage.  The only thing that works in this case, and
many similar cases, is MPLS FRR or a tunneling mechanism that supports
an explicit-path either in the setup or in the forwarding encaps.  At
the very least this should be reflected in the internet-draft.

Curtis

---

From: Albert Tian <[email-masked]>
Date: Mon, 27 Sep 2004 11:31:04 -0700

Hi,

It seems we've been overly harsh on tunnels lately :-)

First of all I feel tunnels are unavoidable for IPFRR for a number of reasons:

a) to protect IP multicast traffic, one need to encapsulate the protected
traffic and tunnel it all the way to a node that have existing group membership.
You can not just send multicast traffic to an alternative nexthop and hopes that
one knows how to handle the specific (*,G) or (S,G).

b) for dual homed IP prefixes, there would be cases where you need to
encapsulate and tunnel traffic to the alternative egress when the primary egress
fails. Without tunneling, traffic could be drawn back to the primary egress and
defeat the FRR. This would happen when the alternative egress has a less favored
BGP path.

Also IMHO we should maintain some level of seperation between the algorithm and
its specific implementations. This may not be the best forum to discuss non IP
issues such as MPLS, but we all know that MPLS provides the most inexpensive
implementation of tunnels. My point is that we should not discard a solution
just because the known IP based implementation has certain drawbacks. For many
customers, all they need is a good solution, be it IP or MPLS.

Now lets look at some of the issues. Several issues have been raised against
tunnels, more specifically IP based tunnels.

a) Setup overhead

Tunnels can be treated just as an encapsulation and there need not be any per
tunnel signaling overhead.
The mpls-in-ip-or-gre draft has pointed out such a case already. Also for LDP
based LSP tunnels, the overhead is very low. For RSVP-TE LSPs, there is per
tunnel overhead, but if we are only talking about a hundred or so LSPs, any
implementation can handle it.

b) decapsulation cost

For MPLS the decapsulation is as simple as a pop operation. If PHP is enabled,
there is NO additional cost at the decapsulating node at all.

b) MTU issue

This is an issue specifically for IP based tunnels. For MPLS LSPs this is not an
issue at all. In most implementations MPLS is considered below IP and do not eat
into IP mtu. In other words, encapsulate IP packets in MPLS do not increase the
IP payload size.

My point is that arguments against IP tunnels should not be used agains tunnels
in general.

My 2c,

Albert

---

From: mike shand <[email-masked]>
Date: Mon, 27 Sep 2004 19:38:49 +0100

Curtis,

This is where "directed forwarding" comes in (see section 3.2.3 and 4.2). 
The tunnel to C includes the information that C should forward the packet 
to X. Depending on what tunnel technology was used, this would be encoded 
in different ways, but it could for example be a specific GRE key. The 
basic idea is that each router (which is able to be a tunnel end point), 
distributes in the IGP, the information to be used to cause the data to be 
forwarded to each of its neighbors. i.e. that is just one extra piece of 
information per neighbor. When the router at the failure computes the 
repair path it includes this piece of information (if necessary; most 
repairs don't need directed forwarding) in the tunnel header. There is no 
explicit path setup mechanism. If you like there is an implicit path always 
set up to each neighbor. Note that it is only ever one hop to a neighbor.

         Mike

---

From: mike shand <[email-masked]>
Date: Mon, 27 Sep 2004 19:47:10 +0100

>one knows how to handle the specific (*,G) or (S,G).

Absolutely!

>defeat the FRR. This would happen when the alternative egress has a less 

and again, I have to agree.

>customers, all they need is a good solution, be it IP or MPLS.

Yes, that is why we have avoided being specific about the actual tunnelling 
mechanisms.

>tunnel signaling overhead.

Yes.

>there is NO additional cost at the decapsulating node at all.

PHP may be an issue with multicast, but in general I agree.

---

From: Albert Tian <[email-masked]>
Date: Mon, 27 Sep 2004 12:03:25 -0700

Hi, Alia,

Yes that is an issue needs to be solved, but it may not be as serious:

a) there are solutions that do not require LDP full mesh. For example,
draft-shen-mpls-ldp-nnhop-label-01.txt that would let you learn LDP next-nexthop
labels.

b) there are people who would not mind LDP full mesh.

Thanks,

Albert

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Sep 2004 15:24:42 -0400

Mike,

You might want to reread the example.  X is the source of the
traffic and Z is the destination.  The link X-N breaks and C is left
forwarding all traffic to all nodes toward X.  Whatever tunnel
technology used must override the IP decision at each hop all the way
to the destination.

After you reread the first example, then look at the second.

Curtis

---

From: Alia Atlas <[email-masked]>
Date: Mon, 27 Sep 2004 16:31:23 -0400

Hi Albert,

An LDP full mesh doesn't scale very well.
As for learning the LDP next-nexthop labels via your proposed extension, I 
am concerned about the potential gaps where the next-nexthop 
changes.  Also, just next-nexthop labels isn't sufficient for the proposed 
mechanism with directed forwarding or non-TE tunnels.

It seems to me that you are prioritizing a solution which allows easier 
support of multicast at the cost of more complex support of LDP.

As for the idea of TE tunnels for the backups, I don't see how that 
simplifies a solution beyond what TE fast-reroute can provide today.  It 
seems to me to be equally operationally complex.

Alia

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 27 Sep 2004 22:14:47 +0100

>forward to such that traffic will go to Z without coming back to X.
You need to correctly and fully state your problem not the least
because you have no Y in the diagram.

Then you need to remember two elements of the tunnel solution

- A packet may backtrack, but it will only do this in a tunnel and
  therefore will not loop. This is in our internal spec, but I think
  we deleted it when we compressed the text down to draft format
  because it was an obvious property of tunnels.

- A node instructed to directed forward must do as it is told, even
  if it "knows better" - See para 4.2

I think that you are saying how does A get a packet to Z when XN has
failed?

A forwards to B
B forwards to C
C forwards to X
X knows that XN has failed, so must repair.
X encaps to A with directed forward operation to Z
X forwards to A via C
C forwards to A via B
B forwards to A via direct path - last three ops are just normal
  forwarding
A decapsulates the packet, but notes that it contains a directed
  forwarding vector.
Now read what I said about directed forwarding above.
A forwards directly to Z (NOT to Z via B) - by definition.
So packet arrives correctly at Z via A

>Are you saying that in this case tunneling just doesn't work. 
>
No.

> If so,
Since we get complete coverage there is no need to make this change.

>previously clear. 
We abstracted the problem of path computation and packet delivery
from the many viable  methods of implementation.

>You'd need an explicit path setup mechanism, or in
In the above we need just one encap that contains a DF vector, or one
that does not contain a DF vector + a single MPLS LSE. We clearly
need to operate the network with sufficient margin between permitted
entry MTU and repair MTU. That is in the nature of IP tunnels.

>where cascading U-turn forwarding would help.  
>
We did it in two SPFs, one of which is needed for normal forwarding.

>Since cycles like this
Unfortunately we are both dealing with networks that we are unable to
disclose. What we need is reference network of realistic size and
topology, say 500 nodes, that we can use to illustrate the properties
of the various solutions.

>example, if the cost Z-A was 2 and there was a N-B link with cost 1.
It's easier with a diagram:

    Z -- N -- X
  2 |    |    |
    A -- B -- C

N has failed and we are sending traffic from Z to X ?

We need a secondary repair, but we can repair it.

It's easier to see backwards. If the packet ever got to B, then
B (which is aware of the failure) would repair via its
downstream path to X.

In this case Z just has to push to A with a directed
forwarding vector to B (may or may not need to tunnel to A 
depending on how the DF vector is carried).

There are many more interesting secondary repair cases
than this, but we (or strictly the algorithm executed 
in the simulator) has always found a solution using the 
elements described in our draft (i.e. at worst encap to
another neighbour of the failure, repair to that neighbour,
decap at that neighbour and let that neighbour deal with 
the native packet we are trying to deliver)

| For failure of node N, A would still be trying to send traffic to B.

>coverage 
No. You have demonstrated to us all that U-turns do not have 100%
coverage, but you have not demonstrated a fault with the tunnel
solution.

>and the two methods have different topologies which result in
>an explicit-path either in the setup or in the forwarding encaps. 
>
Wrong

> At
If we had found a case that did not work, we would have stated it.
We have not yet found such a case in symmetric cost networks.

- Stewart

---

From: Albert Tian <[email-masked]>
Date: Mon, 27 Sep 2004 15:42:49 -0700

Hi, Alia,

First I need to clarify that if we are doing link protection only (for LDP
traffic or multicast traffic), we need NO change to LDP, and need very little
extension to RSVP/PIM to fix the RPF interface. In this case the repair path
terminates at the nexthop, which is exactly the node where you learned the LDP
labels from. So no need for extra LDP sessions at all for link protection. Still
the repair path has to be fully encapsulated (tunneled) to make sure multicast
traffic arrive at the nexthop, which is guaranteed to have proper (*,G) or (S,G)
state.

The complexity comes in when we want to do node protection, in which case the
repair paths terminate at the next-nexthops. This is where we need to learn
next-nexthop LDP labels, and next-nexthop group memberships.

> An LDP full mesh doesn't scale very well.

I was thinking the same way as you do before San Diego. But from the feed back I
got from San Diego, LDP full mesh is fine, at least according to those who
spoke. BTW I do have solutions even if LDP full mesh don't scale :-)

> mechanism with directed forwarding or non-TE tunnels.

I feel you maybe mixing how directed forwarding is implemented and how directed
forwarding is used. Any tunnel as long as it can deliver rerouted traffic to
nexthop or next-nexthop, it is sufficient. To use such tunnels to implement node
protection for LDP traffic, next-nexthop LDP labels are the only thing extra we
need. The LDP next-nexthop label scheme works with any kind of tunneling
technology.

> support of multicast at the cost of more complex support of LDP.

The problem is that this seems to be the only way to support multicast.

> seems to me to be equally operationally complex.

To use TE tunnels as backups does not mean the operator also has to use TE for
his/her normal traffic.

The existing TE fast-reroute scheme can only protect TE traffic. So before IP
traffic can be protected, it has to be placed on some TE tunnels first, which is
VERY complex. You need a lot of (mesh of some kind) TE tunnels in your network
for normal day to day operation even without any protection being activated.

We are proposing to use TE tunnels as backups, and only as backups. This does
not require the operator to place IP traffic onto TE tunnels in the normal
operation. The TE tunnels are needed only when some failure happens. Therefore
there will be far less of these TE tunnels to maintain.

Here are some numbers posted by Mike earlier on this mailing list:

"the highest neighbor's neighbor count we have observed in a real network is 129
(with a max neighbor count of 22)"

This basically means to support link protection we need maximum 22 TE tunnels
per PLR, and to support node protection we need max 129 TE tunnels per PLR.
These numbers are far below the thousands normally required to form a full mesh
of TE tunnels.

So IMHO there is huge advantage for using TE tunnels as backups comparing to the
existing TE fast-reroute.

Thanks,

Albert

---

From: Pekka Savola <[email-masked]>
Date: Tue, 28 Sep 2004 08:08:50 +0300 (EEST)

> First of all I feel tunnels are unavoidable for IPFRR for a number of reasons:
[...]

Are you forgetting that we are addressing *IP* FRR, not MPLS FRR?

Anything with 'LSP' in it seems like a non-starter.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Albert Tian <[email-masked]>
Date: Mon, 27 Sep 2004 23:42:18 -0700

> Are you forgetting that we are addressing *IP* FRR, not MPLS FRR?

Thanks for the reminder, but no I did not forget that. I'm talking about the
need for some form of encapsulation/tunneling to protect pure IP traffic
(unicast and multicast). The tunneling technology could either be IP based, or
MPLS based.

> Anything with 'LSP' in it seems like a non-starter.

Please replace 'LSP' with 'tunnel', if that concerns you.

Thanks,

Albert

---

From: Pekka Savola <[email-masked]>
Date: Tue, 28 Sep 2004 10:35:46 +0300 (EEST)

> (unicast and multicast). The tunneling technology could either be IP based, or

Good.  I think it's pretty clear that we must solve the generic case,
i.e., IP based and anything else (e.g., MPLS optimizations) is just
extra.

> Please replace 'LSP' with 'tunnel', if that concerns you.

Sure.  But keep in mind that that just means that one has to care
about MTU issues, encapsulation, decapsulation, set-up issues, etc. as
well.

---

From: Albert Tian <[email-masked]>
Date: Tue, 28 Sep 2004 01:21:36 -0700

> i.e., IP based and anything else (e.g., MPLS optimizations) is just

sure.

> about MTU issues, encapsulation, decapsulation, set-up issues, etc. as

agreed.

Thanks,

Albert

---

From: Albert Tian <[email-masked]>
Date: Tue, 28 Sep 2004 01:26:19 -0700

Albert Tian wrote:

[snip]

> (with a max neighbor count of 22)"

ps. thanks to Mike for posting these interesting numbers ...

---

From: mike shand <[email-masked]>
Date: Tue, 28 Sep 2004 10:54:20 +0100

>After you reread the first example, then look at the second.

Curtis,
         Sorry I misinterpreted your example (which was confusing since it 
referred to an X-Y failure, when it didn't contain a Y). However, the 
answer is still the same. Directed forwarding allows this to be repaired, 
and Stewart has described the scenario.

         It can be mathematically proven that directed forwarding will 
always work provided the link costs are symmetric. In fact it is rarely 
needed in real networks, but almost every real network has a few cases 
which DO need it.

e.g. for link repairs in a variety of networks

Downstream repairs      Tunnels Directed forwarding
(No tunnels)
1347                    263             10
1273                    163             8
1062                    212             0
139                     49              38
869                     127             13
1006                    248             7

Mike

---

From: Albert Tian <[email-masked]>
Date: Tue, 28 Sep 2004 09:00:20 -0700

Hi, Mike,

Thanks again for the numbers. One clarification question below.

mike shand wrote:

[snip]

> Downstream repairs      Tunnels Directed forwarding

I'm assuming these are the total repair path counts for 

- link protection
- repair path terminate at nexthop
- symmetric link cost

right?

Thanks,

Albert

---

From: mike shand <[email-masked]>
Date: Tue, 28 Sep 2004 18:28:52 +0100

>- repair path terminate at nexthop

Sorry, my mistake, these are for node repairs (repair path terminating at 
next next hop).

Some figures for link repair only are

Downstream      Tunnels DF      Asymmetric Links
60              52              38      0
693             227             12      51
486             213             111     11
859             311             0       4
1037            317             14      122 (4 failures due to asymmetry)
111             27              0       0
971             266             5       7
1723            169             14      117
483             141             4       14

Note that most of these have asymmetric costs, but only one has any 
failures, and in these cases the costs are seriously asymmetric
(e.g. 10/1000000)  and the failure only prevents reaching the node at the 
other end of the link (all others can be repaired).

         Mike

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 28 Sep 2004 13:49:42 -0400

In message <[email-masked]>
> because you have no Y in the diagram.

The prior statement was that all the tunneling needed to do was for
some X to forward to Y and it needed no state.  I retrained the
terminology of the message I replied to which was included in my

> > required to "remove" the tunnel. You just stop using that FIB entry which

Those were the lines in my message above the words "That's
interesting" which is where you apparently started reading.

I did state that if X-Y goes does and it should have been if X-N goes
down.  Then the problem is find a "Y" which would be the endpoint of a
tunnel and if you are going to be using IP forwarding rather than an
explicit path, find a "Y" which is reachable without going through the
failure and forwards to Z.

Based on your response below you had no trouble determining what the
point was of the example.

>   because it was an obvious property of tunnels.

With a tunnel destination of A this works.  The part I missed was the
directed forwarding.

>   if it "knows better" - See para 4.2

So what type of tunnel has a "pop and IP directed forward" already
defined for it?  Even putting an IP source route option in there
doesn't override the forwarding at A.  

Did we just invent a new IP option field or its equivalent?  I also
notice that the spec says:

   3.2.3. Directed forwarding.

   Directed forwarding must be supported such that the router at the
   tunnel endpoint (P) can be directed by the router at the tunnel
   source (A) to forward the packet directly to a specific neighbor.
   Specification of the directed forwarding mechanism is outside the
   scope of this document.

A lot of things to make this work are out of scope and this may be
part of the problem that people are having, myself included.

A lot of us have been looking for an existance proof to the many
things that have been declared out of scope.  Note that there were
prior complaints on this list about you proposing a new forwarding
mechanism in this.

[... lots of stuff deleted ...]

> We have not yet found such a case in symmetric cost networks.

Thanks for the clarification.  I can see how you can find some node,
in the above cases a node immediately adjacent to the destination,
which can be reached by the IGP and needs only one hop reversed by the
directed forwarding.  It was the directed forwarding that I missed.
I stand corrected on the examples I gave being a coverage issue.

I can also see why you've done things the way you have.  The tunnel
can always follow the IP path in reverse for assymetric paths at worst
case up to one hop from the destination, then reverse the forwarding
on that one hop.  The advantage over an explicit-path all the way to
the destination to that is plain IP directed tunneling can be used to
reach the point where the one directed forwarding has to be used.

The only cases that it appears cannot be covered are those related to
assymetric path and possibly SRLG.

You've acknowledged that when assymetric paths are present there may
not be a viable backup using this method.  The question of SRLG has
come up but has not been addressed.

Consider the following:

     Z -- N -- X -- E
  6  |         |    |
     A -- B -- C -- D

Link cost on Z-A is 6.  X-N and X-C share an SRLG.  Note that U-turn
does not cover this either, however if router D was not there (E and C
directly connected) it would.

If you want to consider SRLG then you need explicit paths.

> - Stewart

It is more clear to me now why you are pursueing this.  You give up TE
and SRLG support but you don't have any path setup.  OTOH unless I've
missed something you have just invented a new forwarding type in the
directed forwarding which raises compatibility issues with current
hardware and might raise some intellectual property snags if the
directed forwarding on which this strongly depends has issues.

I personally think MPLS FRR is a better solution, but at least I see
that there is greater simplicity to this, giving up a few things for
that simplicity.  Prior to this discussion I didn't see any advantage
over MPLS FRR tunneling so thanks for the response.

Curtis

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 28 Sep 2004 23:51:40 +0100

>doesn't override the forwarding at A.  
Exactly the vector need to be a vector.

A new tunnel type is not strictly necessary, one obvious mechanism
is a GRE tunnel with an MPLS Ethertype (0x8847) followed by an MPLS LSE.
Alternatively one could use L2TPv3 and vector off the session IP.
This would then we the same as an IP pseudo-wire carried over L2TPv3.

The tunnel can therefore be built with existing IETF defined
encapsulations.

>part of the problem that people are having, myself included.
We put it out of scope because there is no novelty to it - see above
- and we did not want the choice of the specific implementation
technology to cloud our description of the repair mechanism.

>[... lots of stuff deleted ...]
Neither MPLS nor GRE are new, nor is the use of them in combination.
The MPLS world does it all the time to hop between disjoint MPLS islands.

>reach the point where the one directed forwarding has to be used.
Correct

>assymetric path and possibly SRLG.
We have always stated that there was a problem with asymetric costs,
however you will see from the analysis of the networks that we have
presented that it is a very rare problem easily fixed by making the
costs symetric. Now before you say "so you need to modify the topology"
I should point out that when we were given the topologies we asked
how common asymetry was and we were told in most cases that there
was none. When we looked at the asymetric costs that we could not
repair we found that there was usually some typo at the root of it
(ie 6666 vs 66666). Again before anyone asks we did NOT fix these
typos before analysing the networks, so our results are illustrative
of the repair coverage degradation that you get in a real network
that includes misconfiguration.

SRLG is something that we have given little thought to, mainly
because we have little confidence in the validity of SRLG marking.
(i.e. we are concerned at how often failure correlation is
predicted correctly). The approach that we have considered is to
start by triming both P and Q space by excluding nodes with a
reachable path that includes members of the SRLG. If that does
not produce a solution then we would need to analyse the repair
strategy of each of the SRLG guardians and try to find a set that
were not mutually repairing. But note that the special SRLG case of node
failure is covered.

>come up but has not been addressed.
Actually I am not sure that any of the proposed (non TE) methods
have a good SRLG strategy. However given that we have more control
over the packet for the first two hops than U-turn (we push the
first hop and DF the second), I would have thought that our
longer reach approach was at worst identical in coverage to
U-turn, and at best more likely to find a solution.

>directly connected) it would.
N-X and X-C as a SRLG is equivelent to a node failure of X, which
we cover in the draft.

I guess that you are trying to go from Z to E?

Z forwards normally to N.
N, knows that there is a failure, and assumes that it is node X
N encaps to C (nearest reachable neighbour of X) and then
repairs to X
To repair to X, N tunnels to Z with a DF vector to A.
Packet goes from N to Z
Z decaps.
Z looks at the DF vector and pushes to A.
A now has a packet addressed to C, and the packet
follows normal forwarding to C.
C decaps and finds a packet addressed to E.
If CX *is* still up the packet goes to E via normal path
If CX is down, C notices and uses the pathsplit to E via D.
If the cost CD or DE was greater such that there was no
path split, then our usual combination of first hop push
of a tunneled packet with a DF vector would get the packet
safely to E.

>If you want to consider SRLG then you need explicit paths.
In some cases, but not the one above.

>and SRLG support but you don't have any path setup.  
>
Strictly we have not properly considered any SRLG case except the case of
node failure which we have given considerable thought to. However
if we were prepared to undertake the extra SPFs I am sure we could
address additional SRLG cases.

>OTOH unless I've
>hardware 
Not really. See above. I am sure I can find a better reference, but
draft-ietf-l3vpn-gre-ip-2547-02.txt illustrates the point.
On the other hand U-turn is a completely novel forwarding mechanism
that I do not recall seeing in any RFC.

>t raise some intellectual property snags if the
I cannot comment on IPR, other than to note that all of the drafts
under consideration include IPR statements.

>over MPLS FRR tunneling so thanks for the response.
You are welcome.

- Stewart

---

From: Albert Tian <[email-masked]>
Date: Tue, 28 Sep 2004 16:55:44 -0700

Hi, Stewart,

> is a GRE tunnel with an MPLS Ethertype (0x8847) followed by an MPLS LSE.

If we use MPLS, then why not use a stack of LDP labels as proposed in
draft-tian-mpls-lsp-source-route-01.txt? I went extra length in the draft trying
to avoid the LDP full mesh problem, which now seems to be a non-issue (according
to the feedback I got). Remove the domain wide label part from the draft, just
use a stack of LDP labels to build a source routed repair path seems better than
the mixed GRE/MPLS solution (less overhead, more efficient, no MTU issue,
support multiple hops, support all strict/loose combinations).

Thanks,

Albert

---

From: Alex Zinin <[email-masked]>
Date: Tue, 28 Sep 2004 17:17:13 -0700

Albert,

> support multiple hops, support all strict/loose combinations).

How would this solve the case of a pure-IP network with no MPLS switching
in the core?

Alex

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 28 Sep 2004 20:37:45 -0400

In message <[email-masked]>
> The tunnel can therefore be built with existing IETF defined

Both of these forwarding mechanism require prior setup.  The third
party setup of the "vector" as you call it, is not defined.

> technology to cloud our description of the repair mechanism.

Infinite choice leads to total lack of interoperability.  Unless of
course that is an explicit goal, it would be considered to be bad.

> The MPLS world does it all the time to hop between disjoint MPLS islands.

Yes but it did take a while to get the tunnel technologies that are
being considered out of this.  A few other choices were mentioned for
tunneling in other threads, IP-in-IP, L2TPv3.

[... stuff deleted ...]
> were not mutually repairing. But note that the special SRLG case of node

There are many cases where SRLG are very helpful.  If two core links
over DWM traverse a common length of fiber, an SRLG is certainly
helpful.  Putting the backup for one of these links on the other would
certainly be undesirable if an alternate existed, even if the
alternate had a higher cost.

There are two types of SRLG that can be supported (terminology
varies).  One where if the SRLG cannot be avoided the backup is not
considered viable due to very high probability of correlated failure.
A second where backup paths are in effect ranked or biased (often by
artificially increasing the cost of the link) so that backup paths
with a relatively low but non-zero probability of correlated failure
can be avoided.

Some providers are more interested in making use of SRLGs than others.

> I guess that you are trying to go from Z to E?

Same as all of the other examples, from X to Z.  I mistyped again.
Node failure of N.

[... stuff deleted ...]

> In some cases, but not the one above.

Sorry.  I mistyped in giving the example but we do agree on this.

> address additional SRLG cases.

Fine.  Maybe you should say that in the draft.  Perhaps you can
revisit the handling of SRLG later.

> draft-ietf-l3vpn-gre-ip-2547-02.txt illustrates the point.

The setup of the forwarding is undefined.

> that I do not recall seeing in any RFC.

By your definition we have not invented a new forwarding.  It is IP
forwarding based on the IP destination.

The difference is that the IP forwarding in the outbound interface is
normally set up to generate ICMP if the packet comes in on that
interface.  Instead it is set up to go a different way since it will
be used for backup in case of a failure at the next node and will
forward to the next best path which does not use the downstream node
on the best path.

So it is IP forwarding and it is a matter of setup.

Furthermore, no signailing extension is needed except an indication of
capability which is passed among peers.  Not only that but Alia has
specified what that signaling is.

So unlike your draft, all of the details are on the table rather than
remaining a mystery, and interoperability is possible, unlike your
draft where interoperability is impossible given what you have
specified.

> under consideration include IPR statements.

And the critical details that are out of scope are of course not
covered by the IPR statements.  So we have no chance for
interoperability based on the draft and probably have IPR noose
waiting if we try to interoperate by some other means.

This doesn't seem like something that would be in the best interest
of the overall community for the IETF to move forward with.  It seems
like a proprietary solution.

> >over MPLS FRR tunneling so thanks for the response.

We might have to add impossible to interoperate with to the list of
things that are given up with this method of IPFRR.

Curtis

---

From: Albert Tian <[email-masked]>
Date: Tue, 28 Sep 2004 23:42:29 -0700

Hi, Alex,

> How would this solve the case of a pure-IP network with no MPLS switching

No this would not. I was just saying that MPLS in GRE as Bryant has suggested
would require MPLS, so we might as well replace the outer GRE tunnel with LDP
LSP tunnel -- basically using a stack of LDP labels instead.

Thanks,

Albert

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 29 Sep 2004 08:40:48 +0100

>LSP tunnel -- basically using a stack of LDP labels instead.
The only MPLS required is a 1 hop PHP tunnel. BUT there is no need for 
LDP or RSVP signalling.
The LSE can be distributed via the IGP as an opaque 32 bit number. The 
interest in using MPLS
here is ONLY to use the MPLS forwarding support available in the 
routers. Under those circumstance
it is not clear whether MPLS is realy there of not.

- Stewart

---

From: Alex Zinin <[email-masked]>
Date: Wed, 29 Sep 2004 00:42:09 -0700

Albert, guys-

 I have nothing against MPLS in general, but FRR in MPLS networks is not the
 problem we're solving here. The mandatory to implement part of the spec that
 will ensure interoperability will need to work in pure IP networks. If support
 of e.g. MPLS/LDP networks comes as an added benefit--sure.
 
-- 
Alex
http://www.psg.com/~zinin

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 29 Sep 2004 10:37:52 +0100

>support multiple hops, support all strict/loose combinations).
That does not work in a pure IP network.

Remember the only node that needs to actually know that this is MPLS is 
the tunnel endpoint.
The tunnel endpoint floods in the IGP: one bit that says that it is 
willing to act as a repair endpoint
and for each of its neighbors a 32 bit opaque data type. That this 
happens to be an MPLS label
is a local matter at that endpoint.

The use of MPLS here is just a convenient re-use of the MPLS forwarder 
just that one
router at the end of the repair.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 29 Sep 2004 12:49:35 +0100

>course that is an explicit goal, it would be considered to be bad.
Seems better to start by agreeing the requirements, and then the repair 
mechanisms
and then the implementation spec. I am not sure that we have agreed the 
first two
items yet.

>tunneling in other threads, IP-in-IP, L2TPv3.
Indeed they are still possibilities.

>alternate had a higher cost.
Agreed.

>with a relatively low but non-zero probability of correlated failure
The method I suggested is consistent with both of these.

>Some providers are more interested in making use of SRLGs than others.
Fine.

>revisit the handling of SRLG later.
None of the methods on the table say anything about SRLG. If what you 
are saying
is that ALL the methods should define their SRLG capability as part of 
the design
selection process I would be happy with that.

As I said before we can direct a packet through any two first hops which 
is better
than U-turn which uses an implicit second hop, IN ADDITION we can separate
the two points of direction by an arbitrary length natural forwarding 
path. We can
therefore run any path selection method that you suggest for the first 
two hops
and then look for paths that repair those SRLGs that were not caught by your
method..

>things that are given up with this method of IPFRR.
Not at all. When we agree the design we can then agree the necessary 
protocol changes.

- Stewart

---

From: Albert Tian <[email-masked]>
Date: Wed, 29 Sep 2004 15:56:57 -0700

Hi, Stewart,

> it is not clear whether MPLS is realy there of not.

IMHO MPLS clearly is there. It is MPLS over GRE.

- You just created a new MPLS signaling mechanism that uses IGP to achive the
same effect as LDP full mesh. BGP is used to distribute VPN labels but that is
still MPLS.

- These IGP distributed labels still need to be installed into LFIB and
associated with pop operation.

- The payload inside the GRE header is a labeled packet. 

- The label has to be looked up in the LFIB to find the outbound interface and
identify the pop operation. 

All these are 100% MPLS.

Thanks,

Albert

---

From: Alia Atlas <[email-masked]>
Date: Wed, 29 Sep 2004 21:21:51 -0400

Hi Albert,

I agree with you that MPLS is clearly there.  That mechanism does has MPLS 
in it with a new undefined signaling mechanism.   Fundamentally, the router 
is making a switching decision based solely on the received label; that is 
MPLS.

At least using an LDP tunnel with a directed label underneath removes the 
tunnel fragmentation concerns.  It also seems more likely to involve 
substantially fewer changes to the forwarding path.  Of course, it is still 
an MPLS only solution which requires a full mesh of adjacencies for LDP.

The benefit of U-Turn alternates is that it doesn't require MPLS or any 
tunneling mechanism; without the tunneling, the full mesh for LDP (if that 
is of interest) is not necessary.

The claim that only the tunnel egress needs to understand this mechanism 
appears to conflict with the argument being made that 100% coverage without 
any network change is utterly critical and worth any complexity.

Alia