Subject: Re: FW: MPLS OAM (was Re: Last Call on RSVP Label Allocation for Bac kup Tunnels)

From: Peter Willis <[email-masked]>
Date: Wed, 04 Apr 2001 16:25:45 +0100

From experience I think that the "rare" faults are not as "rare" as us network 
operators would like. I have colleagues who manage a national network with 
9000 routers in it and a global network with hundreds of routers in it. This 
network
has a "rare" fault about every 3 months. A "rare" fault is something like a 
router sending packets in a different direction to that indicated by the 
routers' routing table. In other words the user plane isn't doing what the 
control plane thinks.

A much more common fault is bugs in the implementation of the control plane 
protocols. I've seen routers "forget" their routes many times. This is so 
common a fault I considered writing a script to check all the routing tables 
in all the routers to look for these "forgetting" events.

Why wouldn't LSRs manifest similar bugs which would result in the LSPs
going down in a manner the control plane protocols would not detect?
According to Dave Cooper (gblx.net) MPLS implementations do manifest
these types of faults.

So "rare" events do happen and without some monitoring functions built into
the routers and the protocols then Network Operators have to resort to 
customers detecting faults or create their own OAM systems to monitor the 
network.

So you can debate how many milliseconds it takes to restore a LSP but 
milliseconds down-time is going to pale into insignificance against the
hours of downtime it takes to detect and fix problems caused by software
bugs.

I'm not complaining about bugs in router software, all software has bugs,
I'm complaining about the lack of capability to detect faults caused by
the bugs.

Giles Heron wrote:
> Seriously I think it is the case that for MPLS the control and user
> plane will almost always go through the same path
.....
> I think a fair assumption is to say that if the LSP fails from the
> control plane point of view then it is considered to be down.  I know
> this doesn't catch the "rare" cases, but I think it is workable for all
> intents and purposes (especially since the OAM code to catch the "rare"
> cases would have failure modes itself!)

Peter.

---

From: [email-masked]
Date: Wed, 4 Apr 2001 17:58:15 +0100

Giles Heron [mailto:[email-masked]] wrote 04 April 2001 12:58
<snipped>
> cases would have failure modes itself!)
NH=> I note Dave Cooper does not agree with you nor do the people who run
our IP networks.....and I see that Peter Willis has already responded with
our experiences of these (not so) 'rare' defects due to mismatches between
control and user planes.
> 
> > -       where are the 'consequent actions' on failure specified?
> 
> Presumably in the case of George's draft the "consequent 
> action" should
> be to reroute over a backup tunnel?
NH=> But there are other consequent actions that need to be considered, and
invoking a control-plane restoration action is only one of these (assuming
this option is available).  Others to consider are:
-	is there a potential security breach (eg leaking customer traffic)
such that traffic squelching is required?
-	is there a need to raise an alarm?
-	what information needs to be sent to the NMS?
-	what needs to be sent to the client layers supported by the failed
layer.....not a big issue when one's only focus is intra-domain (since one
can, perhaps, appeal to NMS correlation), but it is an issue that should be
considered for the inter-domain case (could result in
geographically/organisationally disjoint alarms and uncessary Ops actions)?
-	what local action is needed in the NE, eg has the defect caused a
(i) QoS perf hit only, (ii) a short-break or (iii) an unavailability
event.....and the in latter 2 cases I would argue that QoS count aggregation
(eg in the MIB say) needs suspending?
-	is there any impact (from the above) into the SLAs that affect
customer billing and hence infomation needs to be transferred to the OSS? 
> 
> > -       where is the definition of availability specified?
> 
> I'd specify it as the % of time for which the LSP is up from 
> the control
> plane point of view?
NH=> It would need a more precise definition than that since we would want
consistency across all vendors and for all customer SLAs.
> 
> > -       and how can we move on to define QoS 
> metrics/objectives if we can't
> > define availablity?
> 
> Agreed - so let's take a simple definition as above.
NH=> Glad we have at least 1 point of common ground.....but it needs a
proper definition, and this isn't one.
> 
> > As an operator we are coming under increasing pressure to 
> be able to define
> > these (esp for VPN customers), and in any case these are 
> issues that should
> > be thought about up-front and not later (when it might be 
> too difficult to
> > make changes).  It seems clear that you don't have any 
> answers to these
> > questions, but you also don't have any alternative proposals either.
> 
> I'd say I just made a perfectly workable proposal - from the point of
> view of a pragmatist who has to build and run a production network.  I
> suspect your viewpoint is somewhat different.
NH=> Not really, but I do look ahead further than what is deployed today.
Believe it or not, I honestly thought we had something simple that operators
would like......clearly not.
> 
> > BTW - I am surprised you thought the proposals complex.  
> This is about as
> > simple as we could make them, and Shahram (who works for a 
> chip supplier)
> > did not think these complex.
> 
> I'd say 29 pages for a first revision of a draft is pretty meaty. 
> Especially when the P flow and support for merging LSPs are 
> FFS, and the
> figures are all missing?
NH=> The figures are missing since they don't port to ascii easily.  On the
other points:
-	the basic premise is very simple.....a 1s CV keepalive with unique
source ID (it has to be a fixed period to ensure consistent/simple defect
entry/exit criteria)
-	if the MIBs can do detailed ingress/egress pkt/octet counting then
we don't need the P flow I guess
-	if you don't care about alarms storms and the inter-domain defect
location info  then you can dispense with the FDI (consequent action)
too.....I expected this one to cause the most trouble (though its funny how
some people take it for granted that an FDI will be provided from other
server layer technologies, like SDH, when they want/appeal defect detection
(in a client layer), and seem to ignore that its only there because someone
specified the defects properly in a *different* layer network)
-	merging LSPs cannot be easily dealt with in the user-plane as I am
sure you appreciate (and there are many other areas that make this
FFS)....but I don't think one would use them for important traffic (such as
VPNs perhaps?)
So......the OAM could indeed be very, very simple and still give strong
customer protection and reasonable decent Ops help.