Subject: Re: [mpls] draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

From: Alia Atlas <[email-masked]>
Date: Tue, 9 Jul 2013 18:12:49 +0000

Can you clarify why you think constraining the links explored based upon a latency isn't a practical approach?  It is still O(n log n) - granted, it's a heuristic and may not find a path.

Alia

-----Original Message-----
From: Xuxiaohu [mailto:[email-masked]] 
Sent: Tuesday, April 23, 2013 9:44 PM
To: [email-masked]; MPLS WG Mailing List; draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS Co-Chairs
Subject: re: [mpls] draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

>     s/While it has been possible to compute a CSPF/It is possible to
>     compute a CSPF/
>     s/Instead of this approach to minimize path latency, an/An
>     alternative to this approach to minimize path latency is an
>     approach to place a upper bound on path latency.  An/
>     Note: both approaches are valid.

I think the alternative approach (i.e., to compute a CSPF path based on the TE metric while placing a upper bound on the path latency) is not a practical approach due to computationally complexity.

Best regards,
Xiaohu

---

From: Alia Atlas <[email-masked]>
Date: Tue, 9 Jul 2013 19:59:44 +0000

Hi Curtis,

Thanks for your detailed comments and suggestions.  My responses are in-line, as always.

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, April 23, 2013 12:59 PM
To: MPLS WG Mailing List; draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS Co-Chairs
Cc: [email-masked]
Subject: draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

Loa, authors, et al,

So far I have seen two of the four MPLS-RT reviews (maybe I missed the others).  I've commented before on the mailing list about this draft but at this point I would like to provide a detailed review.

I think there are very major issues with this document as it now stands.  IMO these issues should be addressed before the draft is even accepted as a WG document.

You can choose to consider this during MPLS-RT review or after.

Curtis

Major issues:

  Jitter and loss are very close to meaningless if queueing delays and

  loss are not considered.  Links that are losing packets at the link

  layer are generally taken down.  Oscillations can occur if queueing

  delay and loss is considered, such as measurements at a low priority

  and therefore possible to congest or measurements which are

  otherwise affected by traffic load.  Unless there is adequate

  discussion of stability and mechanisms to insure stability, then

  jitter and loss should be removed.

[Alia] I have changed the second paragraph of the introduction to be:

" Queuing latency is specifically excluded to insure freedom from

   oscillations and stability issues that have plagued prior attempts to

   use delay as a routing metric.  If application traffic which follows

   path based upon latency constraints, the same traffic might be in an

   Expedited Forwarding Per-Hop-Behavior [RFC3246] with minimal queuing

   delay or another PHB with potentially very substantial per-hop

   queuing delay.  Only traffic which experiences relatively low

   congestion, such as Expedited Forwarding traffic, will experience

   delays very close to the sum of the reported link delays"

Removing queuing delay is the agreed result for draft-ietf-ospf-te-metric-extensions.

As far as jitter and loss goes, these can still be somewhat meaningful.  For instance, jitter does allow capturing the differences in serialization delays - which is relevant in some parts of the network.  The acceptable loss to take a link down may vary.   I don't actually see the reasoning or need to remove jitter and loss from the draft - given that we have constrained the measurements in the draft-ietf-ospf-te-metric-extensions to not include queuing delay.

General:

  It needs to be much more clearly stated that the NPO (s/SLA/NPO/g,

  see nits below) is specific to a link, and not an NPO per LSP.  A

  "non-conformance to NPO" flag in the IGP link advertisement is

  intractable if there are multiple NPO being applied to the link.

[Alia] As you may recall from draft-ietf-ospf-te-metric-extensions, the Anomalous bit is specified per characteristic advertised per link.   This draft (draft-atlas-mpls-te-express-path-02) describes the Anomalous bit as clearly inside a links' sub-TLV as in Sec 2.3.1:

"If the answer to (a) is no for latency SLAs, then any link which has

the Anomalous bit set in the Unidirectional Link Delay sub-

TLV[I-D.ietf-ospf-te-metric-extensions]

[I-D.previdi-isis-te-metric-extensions] should be removed from the

topology before a CSPF calculation is used to compute a new path."

  The use of a "non-conformance to NPO" flag as the only means to

  alert the set of LSP ingress of a significant change is also at best

  a poor solution.  For example, a change from 2 msec to 15 msec may

  be a problem for some LSP.  That same change may not be a problem

  for others, such as LSPs where only this one hop is needed.  So

  advertising out of NPO in the IGP doesn't help the second case.  LSP

  ingress must evaluate the path delay constraint, each time a change

  in IGP link advertised delay is received.

[Alia] Sure - different LSPs may have different requirements as to the Anomalous flag.  Using it doesn't replace paying attention to the value that is advertised.  The anomalous flag is reporting that the link is not complying to the expected performance - this can indicate that there is something strange going on and some LSPs should avoid using that link due to administrative policy.

  The use of a "non-conformance to NPO" flag solely as a constraint

  makes sense, where some LSP are configured to exclude links with

  this flag set.

[Alia] Right - case (a) and (c) for Sec 2.3

  For path delay computation it would also be useful if the NPO

  threshhold were advertised.  In some cases, it may make sense to

  minimize or place bounds on the sum of NPO delays.

[Alia] I think that should be a comment on draft-ietf-ospf-te-metric-extensions.  This draft doesn't define what is flooded.   A question though is whether in the same network it would make sense to consider both the NPO delays and the actual measured delays?  If only one or the other - then that could be a local router's decision as to which is flooded.

Specific:

  Abstract:

    Remove mention of jitter and loss.

[Alia] I am interested in the opinion of the WG on this.  In my view, this draft is describing how the information flooded via the new sub-TLVs in draft-ietf-ospf-te-metric-extensions-04 should be used.  Loss and jitter are included in that draft; of course, that draft could change as well - but I'd like to hear stronger arguments for why it is harmful to include them than that they're only relevant when queuing behavior is included and that can lead to oscillations.  I am, perhaps stubbornly, not convinced that they are irrelevant.

  1.  Introduction:

    This sentence is awkward but I think I know what you are trying to

    say.

      The method suggested is not optimal for both minimizing path

      cost and additional constraints, such as latency; optimal

      solutions are computationally complex.

    Please consider this replacement.

      Methods of optimizing path selection for multiple parameters are

      generally computationally complex.  The method proposed here are

      to make use of either a single metric in path selection, such as

      minimal path delay, or to make use of another single metric,

      such as the existing TE metric with additional constraints, such

      as target link delay bounds and target path delay bounds.

[Alia] Thanks for the better text - put in...

    Please note:

      The path selection mechanisms described in this document apply

      to paths that are fully computed by the head-end of the LSP and

      then signaled in an ERO where every sub-object is strict.  This

      allows the head-end to consider IGP-distributed performance data

      without requiring the ability to signal the performance

      constraints in an object of the RSVP Path message.

    There is work in MPLS or CCAMP by George Swallow and others on

    cummulative metrics in RSVP-TE with delay as a major motivation.

    Perhaps that work should be considered.

[Alia]  I believe you are referring to http://tools.ietf.org/html/draft-ietf-ccamp-te-metric-recording-01?  That draft appears to me to be about how to get measurements for latency, jitter, and cost for a given Forwarding Adjacency or Routing Adjacency so that those values can be advertised into the IGP.  I think this draft is providing a means to collect the data to flood in draft-ietf-ospf-te-metrics-04.

I did add the last sentence in the following:

"This document does not specify how a router determines what values

to advertise by the IGP; it does assume that the constraints specified

in [I-D.ietf-ospf-te-metric-extensions] and [I-D.previdi-isis-te-metric-extensions] are followed.  Mechanisms for determining latency and delay variation for Forwarding

Adjacencies and Routing Adjacencies are defined in [I-D.ietf-ccamp-te-metric-recording]."

    This paragraph could use rewording:

      When considering performance-based data, it is obvious that

      there are additional contributors beyond just the links.

      Clearly end-to-end latency is a combination of router latency,

      queuing latency, physical link latency and other factors.

      However, if application traffic requires paths to be selected

      based upon latency constraints, the same traffic might be in an

      Expedited Forwarding Per-Hop- Behavior[RFC3246] with minimal

      queuing delay or another PHB with known maximal per-hop queuing

      delay.  While traversing a router can cause delay, that can be

      included in the advertised link delay.

    What you are getting at is that where traffic levels can't

    possibly have a significant impact the measurements, such as low

    levels of EF traffic in a WAN with high geographic delays, and

    delay meansured with EF priority, stability is not impacted if

    delay measurements are considered.  In this case, loss MUST be

    zero or the EF service is broken (replace routers and try again).

    OTOH jitter will increase as traffic levels increase, even in such

    a case where the traffic is only a few percent, potentially

    causing oscillations and impacting stability.  For this reason,

    jitter and loss should not be considered.

    I will leave the rewording up to you.  I suggest that you create a

    subsection of "Introduction" which discusses potential

    oscillations and stability in greater detail.  If you like, I will

    provide some text.

[Alia] I do hear your concern that jitter could cause oscillations and impact stability.  I am not fully persuaded that this is a practical issue - given the delays in ingress re-optimization, not trying to minimize the jitter, and the ability for an LSP to reserve bandwidth.  I would be interested in what text you might provide and a focused discussion on whether this is something that needs to have restrictions clearly described to avoid oscillations.

For now, I have changed this paragraph to:

"When considering performance-based data, it is obvious that there

are additional contributors beyond just the links. Clearly end-to-end

latency is a combination of router latency, queuing latency, physical

link latency and other factors.  While traversing a router can cause

delay, that can be included in the advertised link delay.  As

described in [I-D.ietf-ospf-te-metric-extensions] and

[I-D.previdi-isis-te-metric-extensions], queuing delay

should not be included in the measurements advertised by OSPF or ISIS.

Queuing latency is specifically excluded to insure freedom from

oscillations and stability issues that have plagued prior attempts to

use delay as a routing metric.  If application traffic which follows

path based upon latency constraints, the same traffic might be in an

Expedited Forwarding Per-Hop-Behavior [RFC3246] with

minimal queuing delay or another PHB with potentially very substantial

per-hop queuing delay.  Only traffic which experiences relatively low

congestion, such as Expedited Forwarding traffic, will experience

delays very close to the sum of the reported link delays."

  1.1.  Basic Requirements:

    Drop "packet loss, jitter" from the first numeric item.  Otherwise

    OK except use of the word SLA.  See nits below.

[Alia] Changed SLAs to NPO

    After the list please state that "For existing MPLS constraints,

    corresponding RSVP-TE signaling allows midpoint LSR to use Path

    Tear and/or notification and would use this mechanism to

    accomplish items 3-6.  In the absense of RSVP-TE signaling

    corresponding to these new constraints, new mechanisms at the LSP

    ingress are needed."  Alternately, consider citing the work by

    George Swallow et al and explain how that solves it in the same

    way that a change in admin attr of a link would (or should, poor

    implementations not withstanding).

[Alia] Frankly, I'm confused by this.  I don't agree that RSVP-TE signaling extensions would solve, for instance, (3):

"3. Ability to periodically verify that a TE tunnel's current LSP

complies with its configured end-to-end performance requirements."

Even if the ingress signaled the end-to-end performance requirements, I don't see how that stops the ingress from verifying compliance?

Similarly, only the ingress could do:

"   4.  Ability to move tunnels, using make-before-break, based upon

       computed end-to-end performance complying with configuration

   5.  Ability to move tunnels away from any link that is violating an

       underlying SLA

   6.  Ability to optionally avoid setting up tunnels using any link

       that is violating an SLA, regardless of whether end-to-end

       performance would still meet requirements."

I have looked for the additional work that you are talking about by George Swallow unsuccessfully.  Can you find a pointer to explain what you're talking about?  The anomalous bits provide a trigger mechanism that a router can use to tell the ingress to do (5); different admin attributes could do a similar behavior - and similarly for (6) - but again that is the flooding for notification aspect.  I think the lack of RSVP-TE extensions doesn't cause an issue - these are handled by IGP instead and thus don't have to be signaled per LSP.

I am not irrevocably opposed to RSVP-TE extensions - if we really need them for practical use-cases.  This draft was trying to do the minimum that is sufficient and then, if and when there is a need for more, what extra is needed could be better defined.

  2.1.  End-to-End Constraints:

    Note: I've requested that jitter and loss be dropped from

    draft-ietf-ospf-te-metric-extensions and

    draft-previdi-isis-te-metric-extensions for the same potential

    oscillations and stability reasons cited above.

[Alia] Yes - I think we clearly need to have a good email discussion with those interested about what oscillations might actually occur and what could be done to prevent that.

    compute a CSPF/

    Delete next paragraph starting with "This is illustrated as

    follows."  This seems to be taken from email and is good email

    discussion but not needed in the draft.

[Alia] I've put in some pseudo-code so I'm ok with taking the example out - but there does seem to have been confusion even with it in.

    Delete next paragraph starting with "An end-to-end bound on delay

    variation".  Lets get rid of jitter altogether.  (Let the old SNA

    networks be damned. :)

    Delete next paragraph starting with "For link loss".  Get rid of

    link loss altogether.

[Alia] Not done - discussion is needed.  I understand that you really really really don't want to see loss or delay variation in any of these drafts.

  2.2.  Link Constraints:

    Drop delay variation and link loss.

    If we are dealing with EF traffic then using Unidirectional

    Available Bandwidth and Residual Bandwidth makes no sense.  If we

    are dealing with low priority traffic and we are using

    Unidirectional Available Bandwidth and Residual Bandwidth in path

    selection will be prone to oscillations and network instability.

    Therefore delete the entire second paragraph (the one starting

    with "When doing path selection for TE tunnels,".

[Alia]  If we are trying to avoid congesting the link, then Residual Bandwidth is important and useful regardless of the LSP traffic class.  What is your specific concern with oscillation for bandwidth?  Why is it different than for the bandwidths already advertised and used for TE path computation?   Come on - the Residual Bandwidth is just the Link Capacity minus that reserved by RSVP-TE - so pretty darn similar characteristics to the Unreserved Bandwidth per priority...   The Unidirectional Available bandwidth does include an actual traffic measurement - that is averaged over a reasonable interval, that can only change with limited frequency - and then the LSPs need to be reoptimized to use them.  Please explain precisely the oscillation concern with a clear example.

    Also delete the entire third paragraph (starting with "Similarly,

    only links whose loss is").  As stated earlier, use of loss is

    either a NOOP (low volume of EF traffic) or can lead to

    oscillations.  Get rid of loss entirely.

  2.3.  Links out of SLA:

    s/SLA/NPO/g (see nits below).

    General: A better mechanism than an "Anomalous State" flag is

    needed to provide notifications of change.  One mechanism would be

    to put the commulative constraint and the cummulative total in the

    ERO and RRO.  A link adding delay can then notify the ingress of

    any LSP for which the commulative constraint is violated.  See

    work by Swallow et al and perhaps align with that work.

[Alia] That could be done as well - but the signaling extensions seem like overkill for the basic problem of letting the ingress compute a complete ERO.  Carrying both the cumulative constraint and the cumulative total just means that the midpoint detects a changed link and then has to verify the performance on each LSP and individually signal it.  Having an anomalous flag provides a succinct notification to the ingress which can then do the verification and be known to have the updated information.  This solution also requires that the midpoints all support it before it can be used.  An advantage of the ingress computation is that only the ingress needs to have updated code.

    The "Anomalous State" flag is helpful for c in the list, but not

    for b.  The case where a link is out of NPO by 1 msec then changes

    to out of NPO by 10s of msec is an example.  The flag is already

    set and therefore the trigger is unavailable.

[Alia] Right - but LSPs that are particular sensitive (i.e. a or c) can have been moved already.   Having the Anomalous flag set is expected to be unusual.  I agree that it is more a hint for (b) than a full solution.

  2.3.1.  Use of Anomalous Links for New Paths

    Delete second paragraph regarding jitter and loss.

  2.3.2.  Links entering the Anomalous State

    This section ignores two cases in which the Anomalous State does

    not change but a change makes the path violate a delay

    constraint.  The first is where all of the links are within NPO

    but a change to a link has make the path delay sum exceed the path

    delay constraint.  The second is where a link which is already in

    the Anomalous State by a small margin but the path delay is still

    within the constraint.  A large change in delay at that point will

    not affect the Anomalous State since it is already set.

[Alia]  The Anomalous bit is NOT meant to replace reading the actual value associated with the link and checking each potentially affected LSP.  In the case of (b), it is a hint to focus on those LSPs first.

    A better means of handling case (b) in "2.3.  Links out of SLA" is

    needed.

[Alia]  I've added the following paragraph at the end of 2.3.2:

"It is not sufficient to just look at the Anomalous bit in order to

determine when TE tunnels must have their compliance verified.  When

changing to set, the Anomalous bit merely provides a hint that

interested TE tunnels for case (b) should have their continued

compliance verified."

  2.3.3.  Links leaving the Anomalous State

    Same issue as in "2.3.2.  Links entering the Anomalous State".  A

    better means of handling case (b) in "2.3.  Links out of SLA" is

    needed.

[Alia]  I've added the following sentence:

"The hint provided by the Anomalous state change may help optimize when to recompute for a better path."

XML version nits:

  CV: you really should remove the template comments.

[Alia]  I find them helpful for when I want to do additional things... and very very few people read the XML.

  You should also enable strict mode.  For example, you have one

  author too many and strict would catch that.

[Alia]  So does basic arithmetic :)   It will be resolved before the draft is passed to the RFC editor.

Other nits:

  The "A" in SLA is "Agreement" as in contract.  The acronym NPO for

  network performance objective seems to be in vogue for that reason.

  IETF since diffserv has wanted to steer clear of making

  recommendations regarding provider contracts (agreements) with

  customers, peer, or anyone else.

[Alia]  True - changed

  I agree with Sri on the suggestions to change the title, short name

  and document filename but I'm not fond of his suggested new names.

  Authors please suggest new title, short name, and filename.

[Alia]  I did do a new short name and title.  As for filename, we'll deal with that if/when the draft is adopted as a WG draft.

Thanks again,

Alia

------- Forwarded Message

On 4/5/13 9:50 PM, "Loa Andersson" <[email-masked]<mailto:[email-masked]>> wrote:

>Xiaohu, Sri, Rajiv and Pranjal,

>You have been selected as an MPLS Review team reviewers for

>draft-atlas-mpls-te-express-path-02.

>Note to authors: You have been CC'd on this email so that you can know

>that this review is going on. However, please do not review your own

>document.

>Reviews should comment on whether the document is coherent, is it

>useful (ie, is it likely to be actually useful in operational

>networks), and is the document technically sound?  We are interested in

>knowing whether the document is ready to be considered for WG adoption

>(ie, it doesn't have to be perfect at this point, but should be a good

>start).

>Reviews should be sent to the document authors, WG co-chairs and WG

>secretary, and CC'd to the MPLS WG email list. If necessary, comments

>may be sent privately to only the WG chairs.

>Are you able to review this draft by April 20, 2013?

>Thanks, Loa

>(as MPLS WG chair)

>/Loa

>--

>Loa Andersson                        email: [email-masked]<mailto:[email-masked]>

>Senior MPLS Expert                          [email-masked]<mailto:[email-masked]>

>Huawei Technologies (consultant)     phone: +46 739 81 21 64

_______________________________________________

mpls mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/mpls

------- End of Forwarded Message

---

From: <[email-masked]>
Date: Wed, 10 Jul 2013 12:42:11 +0000

I'd like to get more feedback on removing jitter and loss from OSPF TE ME before doing so. So far, Curtis, it's only you asking for it (to be removed). Also, no one else seems uncomfortable with the A bit.. I am not personally in favor of removing them, and I feel the A bit is fine for my use case.

From: Alia Atlas [mailto:[email-masked]]
Sent: Tuesday, July 09, 2013 4:00 PM
To: [email-masked]; MPLS WG Mailing List; draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS Co-Chairs
Subject: RE: draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

Hi Curtis,

Alia

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]]
Sent: Tuesday, April 23, 2013 12:59 PM
To: MPLS WG Mailing List; draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS Co-Chairs
Cc: [email-masked]
Subject: draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

Loa, authors, et al,

Curtis

Major issues:

General:

  this flag set.

Specific:

  Abstract:

  1.  Introduction:

    say.

    Please note:

    provide some text.

  1.1.  Basic Requirements:

       underlying SLA

    compute a CSPF/

    networks be damned. :)

    link loss altogether.

  2.2.  Link Constraints:

  2.3.  Links out of SLA:

    needed.

compliance verified."

    needed.

XML version nits:

Other nits:

[Alia]  True - changed

Thanks again,

Alia

On 4/5/13 9:50 PM, "Loa Andersson" <[email-masked]<mailto:[email-masked]>> wrote:

mpls mailing list

---

From: Edward Crabbe <[email-masked]>
Date: Wed, 10 Jul 2013 17:27:44 +0100

I'm not uncomfortable with the A bit.  I just think it's redundant, as
discussed in Orlando.

On Wed, Jul 10, 2013 at 1:42 PM, <[email-masked]>wrote:

>  I’d like to get more feedback on removing jitter and loss from OSPF TE
> ME before doing so. So far, Curtis, it’s only you asking for it (to be
> removed). Also, no one else seems uncomfortable with the A bit.. I am not
> personally in favor of removing them, and I feel the A bit is fine for my
> use case. ****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Alia Atlas [mailto:[email-masked]]
> *Sent:* Tuesday, July 09, 2013 4:00 PM
> *To:* [email-masked]; MPLS WG Mailing List;
> draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS
> Co-Chairs
> *Subject:* RE: draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of
> ...)****
>
> ** **
>
> Hi Curtis,****
>
> ** **
>
> Thanks for your detailed comments and suggestions.  My responses are
> in-line, as always.****
>
> ** **
>
> Alia****
>
> ** **
>
> Subject: draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)**
> **
>
> ** **
>
> ** **
>
> Loa, authors, et al,****
>
> ** **
>
> So far I have seen two of the four MPLS-RT reviews (maybe I missed the
> others).  I've commented before on the mailing list about this draft but at
> this point I would like to provide a detailed review.****
>
> ** **
>
> I think there are very major issues with this document as it now stands.
> IMO these issues should be addressed before the draft is even accepted as a
> WG document.****
>
> ** **
>
> You can choose to consider this during MPLS-RT review or after.****
>
> ** **
>
> Curtis****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> Major issues:****
>
> ** **
>
>   Jitter and loss are very close to meaningless if queueing delays and****
>
>   loss are not considered.  Links that are losing packets at the link****
>
>   layer are generally taken down.  Oscillations can occur if queueing****
>
>   delay and loss is considered, such as measurements at a low priority****
>
>   and therefore possible to congest or measurements which are****
>
>   otherwise affected by traffic load.  Unless there is adequate****
>
>   discussion of stability and mechanisms to insure stability, then****
>
>   jitter and loss should be removed.****
>
> ** **
>
> [Alia] I have changed the second paragraph of the introduction to be:****
>
> ** **
>
> " Queuing latency is specifically excluded to insure freedom from****
>
>    oscillations and stability issues that have plagued prior attempts to**
> **
>
>    use delay as a routing metric.  If application traffic which follows***
> *
>
>    path based upon latency constraints, the same traffic might be in an***
> *
>
>    Expedited Forwarding Per-Hop-Behavior [RFC3246] with minimal queuing***
> *
>
>    delay or another PHB with potentially very substantial per-hop****
>
>    queuing delay.  Only traffic which experiences relatively low****
>
>    congestion, such as Expedited Forwarding traffic, will experience****
>
>    delays very close to the sum of the reported link delays"****
>
> ** **
>
> Removing queuing delay is the agreed result for
> draft-ietf-ospf-te-metric-extensions.****
>
> As far as jitter and loss goes, these can still be somewhat meaningful.
> For instance, jitter does allow capturing the differences in serialization
> delays - which is relevant in some parts of the network.  The acceptable
> loss to take a link down may vary.   I don't actually see the reasoning or
> need to remove jitter and loss from the draft - given that we have
> constrained the measurements in the draft-ietf-ospf-te-metric-extensions to
> not include queuing delay.****
>
> ** **
>
> General:****
>
> ** **
>
>   It needs to be much more clearly stated that the NPO (s/SLA/NPO/g,****
>
>   see nits below) is specific to a link, and not an NPO per LSP.  A****
>
>   "non-conformance to NPO" flag in the IGP link advertisement is****
>
>   intractable if there are multiple NPO being applied to the link.****
>
> ** **
>
> [Alia] As you may recall from draft-ietf-ospf-te-metric-extensions, the
> Anomalous bit is specified per characteristic advertised per link.   This
> draft (draft-atlas-mpls-te-express-path-02) describes the Anomalous bit as
> clearly inside a links’ sub-TLV as in Sec 2.3.1:****
>
> ** **
>
> “If the answer to (a) is no for latency SLAs, then any link which has****
>
> the Anomalous bit set in the Unidirectional Link Delay sub-****
>
> TLV[I-D.ietf-ospf-te-metric-extensions]****
>
> [I-D.previdi-isis-te-metric-extensions] should be removed from the****
>
> topology before a CSPF calculation is used to compute a new path.”****
>
> ** **
>
>   The use of a "non-conformance to NPO" flag as the only means to****
>
>   alert the set of LSP ingress of a significant change is also at best****
>
>   a poor solution.  For example, a change from 2 msec to 15 msec may****
>
>   be a problem for some LSP.  That same change may not be a problem****
>
>   for others, such as LSPs where only this one hop is needed.  So****
>
>   advertising out of NPO in the IGP doesn't help the second case.  LSP****
>
>   ingress must evaluate the path delay constraint, each time a change****
>
>   in IGP link advertised delay is received.****
>
> ** **
>
> [Alia] Sure – different LSPs may have different requirements as to the
> Anomalous flag.  Using it doesn’t replace paying attention to the value
> that is advertised.  The anomalous flag is reporting that the link is not
> complying to the expected performance – this can indicate that there is
> something strange going on and some LSPs should avoid using that link due
> to administrative policy.  ****
>
> ** **
>
>   The use of a "non-conformance to NPO" flag solely as a constraint****
>
>   makes sense, where some LSP are configured to exclude links with****
>
>   this flag set.****
>
> ** **
>
> [Alia] Right – case (a) and (c) for Sec 2.3****
>
> ** **
>
>   For path delay computation it would also be useful if the NPO****
>
>   threshhold were advertised.  In some cases, it may make sense to****
>
>   minimize or place bounds on the sum of NPO delays.****
>
> ** **
>
> [Alia] I think that should be a comment on
> draft-ietf-ospf-te-metric-extensions.  This draft doesn’t define what is
> flooded.   A question though is whether in the same network it would make
> sense to consider both the NPO delays and the actual measured delays?  If
> only one or the other – then that could be a local router’s decision as to
> which is flooded.****
>
> ** **
>
> Specific:****
>
> ** **
>
>   Abstract:****
>
> ** **
>
>     Remove mention of jitter and loss.****
>
> ** **
>
> ** **
>
> [Alia] I am interested in the opinion of the WG on this.  In my view, this
> draft is describing how the information flooded via the new sub-TLVs in
> draft-ietf-ospf-te-metric-extensions-04 should be used.  Loss and jitter
> are included in that draft; of course, that draft could change as well –
> but I’d like to hear stronger arguments for why it is harmful to include
> them than that they’re only relevant when queuing behavior is included and
> that can lead to oscillations.  I am, perhaps stubbornly, not convinced
> that they are irrelevant.****
>
> ** **
>
>   1.  Introduction:****
>
> ** **
>
>     This sentence is awkward but I think I know what you are trying to****
>
>     say.****
>
> ** **
>
>       The method suggested is not optimal for both minimizing path****
>
>       cost and additional constraints, such as latency; optimal****
>
>       solutions are computationally complex.****
>
> ** **
>
>     Please consider this replacement.****
>
> ** **
>
>       Methods of optimizing path selection for multiple parameters are****
>
>       generally computationally complex.  The method proposed here are****
>
>       to make use of either a single metric in path selection, such as****
>
>       minimal path delay, or to make use of another single metric,****
>
>       such as the existing TE metric with additional constraints, such****
>
>       as target link delay bounds and target path delay bounds.****
>
> ** **
>
> [Alia] Thanks for the better text – put in…****
>
> ** **
>
>     Please note:****
>
> ** **
>
>       The path selection mechanisms described in this document apply****
>
>       to paths that are fully computed by the head-end of the LSP and****
>
>       then signaled in an ERO where every sub-object is strict.  This****
>
>       allows the head-end to consider IGP-distributed performance data****
>
>       without requiring the ability to signal the performance****
>
>       constraints in an object of the RSVP Path message.****
>
> ** **
>
>     There is work in MPLS or CCAMP by George Swallow and others on****
>
>     cummulative metrics in RSVP-TE with delay as a major motivation.****
>
>     Perhaps that work should be considered.****
>
> ** **
>
> [Alia]  I believe you are referring to
> http://tools.ietf.org/html/draft-ietf-ccamp-te-metric-recording-01?  That
> draft appears to me to be about how to get measurements for latency,
> jitter, and cost for a given Forwarding Adjacency or Routing Adjacency so
> that those values can be advertised into the IGP.  I think this draft is
> providing a means to collect the data to flood in
> draft-ietf-ospf-te-metrics-04.  ****
>
> ** **
>
> I did add the last sentence in the following:****
>
> ** **
>
> “This document does not specify how a router determines what values****
>
> to advertise by the IGP; it does assume that the constraints specified****
>
> in [I-D.ietf-ospf-te-metric-extensions] and
> [I-D.previdi-isis-te-metric-extensions] are followed.  Mechanisms for
> determining latency and delay variation for Forwarding****
>
> Adjacencies and Routing Adjacencies are defined in
> [I-D.ietf-ccamp-te-metric-recording].”****
>
> ** **
>
> ** **
>
>     This paragraph could use rewording:****
>
> ** **
>
>       When considering performance-based data, it is obvious that****
>
>       there are additional contributors beyond just the links.****
>
>       Clearly end-to-end latency is a combination of router latency,****
>
>       queuing latency, physical link latency and other factors.****
>
>       However, if application traffic requires paths to be selected****
>
>       based upon latency constraints, the same traffic might be in an****
>
>       Expedited Forwarding Per-Hop- Behavior[RFC3246] with minimal****
>
>       queuing delay or another PHB with known maximal per-hop queuing****
>
>       delay.  While traversing a router can cause delay, that can be****
>
>       included in the advertised link delay.****
>
> ** **
>
>     What you are getting at is that where traffic levels can't****
>
>     possibly have a significant impact the measurements, such as low****
>
>     levels of EF traffic in a WAN with high geographic delays, and****
>
>     delay meansured with EF priority, stability is not impacted if****
>
>     delay measurements are considered.  In this case, loss MUST be****
>
>     zero or the EF service is broken (replace routers and try again).****
>
>     OTOH jitter will increase as traffic levels increase, even in such****
>
>     a case where the traffic is only a few percent, potentially****
>
>     causing oscillations and impacting stability.  For this reason,****
>
>     jitter and loss should not be considered.****
>
> ** **
>
>     I will leave the rewording up to you.  I suggest that you create a****
>
>     subsection of "Introduction" which discusses potential****
>
>     oscillations and stability in greater detail.  If you like, I will****
>
>     provide some text.****
>
> ** **
>
> [Alia] I do hear your concern that jitter could cause oscillations and
> impact stability.  I am not fully persuaded that this is a practical issue
> – given the delays in ingress re-optimization, not trying to minimize the
> jitter, and the ability for an LSP to reserve bandwidth.  I would be
> interested in what text you might provide and a focused discussion on
> whether this is something that needs to have restrictions clearly described
> to avoid oscillations.****
>
> ** **
>
> For now, I have changed this paragraph to:****
>
> ** **
>
> “When considering performance-based data, it is obvious that there****
>
> are additional contributors beyond just the links. Clearly end-to-end****
>
> latency is a combination of router latency, queuing latency, physical****
>
> link latency and other factors.  While traversing a router can cause****
>
> delay, that can be included in the advertised link delay.  As****
>
> described in [I-D.ietf-ospf-te-metric-extensions] and****
>
> [I-D.previdi-isis-te-metric-extensions], queuing delay****
>
> should not be included in the measurements advertised by OSPF or ISIS.****
>
> ** **
>
> Queuing latency is specifically excluded to insure freedom from****
>
> oscillations and stability issues that have plagued prior attempts to****
>
> use delay as a routing metric.  If application traffic which follows****
>
> path based upon latency constraints, the same traffic might be in an****
>
> Expedited Forwarding Per-Hop-Behavior [RFC3246] with****
>
> minimal queuing delay or another PHB with potentially very substantial****
>
> per-hop queuing delay.  Only traffic which experiences relatively low****
>
> congestion, such as Expedited Forwarding traffic, will experience****
>
> delays very close to the sum of the reported link delays.”****
>
> ** **
>
>   1.1.  Basic Requirements:****
>
> ** **
>
>     Drop "packet loss, jitter" from the first numeric item.  Otherwise****
>
>     OK except use of the word SLA.  See nits below.****
>
> ** **
>
> [Alia] Changed SLAs to NPO****
>
> ** **
>
>     After the list please state that "For existing MPLS constraints,****
>
>     corresponding RSVP-TE signaling allows midpoint LSR to use Path****
>
>     Tear and/or notification and would use this mechanism to****
>
>     accomplish items 3-6.  In the absense of RSVP-TE signaling****
>
>     corresponding to these new constraints, new mechanisms at the LSP****
>
>     ingress are needed."  Alternately, consider citing the work by****
>
>     George Swallow et al and explain how that solves it in the same****
>
>     way that a change in admin attr of a link would (or should, poor****
>
>     implementations not withstanding).****
>
> ** **
>
> [Alia] Frankly, I’m confused by this.  I don’t agree that RSVP-TE
> signaling extensions would solve, for instance, (3):****
>
> ** **
>
> “3. Ability to periodically verify that a TE tunnel's current LSP****
>
> complies with its configured end-to-end performance requirements.”****
>
> ** **
>
> Even if the ingress signaled the end-to-end performance requirements, I
> don’t see how that stops the ingress from verifying compliance?****
>
> ** **
>
> Similarly, only the ingress could do:****
>
> “   4.  Ability to move tunnels, using make-before-break, based upon****
>
>        computed end-to-end performance complying with configuration****
>
> ** **
>
>    5.  Ability to move tunnels away from any link that is violating an****
>
>        underlying SLA****
>
> ** **
>
>    6.  Ability to optionally avoid setting up tunnels using any link****
>
>        that is violating an SLA, regardless of whether end-to-end****
>
>        performance would still meet requirements.”****
>
> ** **
>
> I have looked for the additional work that you are talking about by George
> Swallow unsuccessfully.  Can you find a pointer to explain what you’re
> talking about?  The anomalous bits provide a trigger mechanism that a
> router can use to tell the ingress to do (5); different admin attributes
> could do a similar behavior – and similarly for (6) – but again that is the
> flooding for notification aspect.  I think the lack of RSVP-TE extensions
> doesn’t cause an issue – these are handled by IGP instead and thus don’t
> have to be signaled per LSP.****
>
> ** **
>
> I am not irrevocably opposed to RSVP-TE extensions – if we really need
> them for practical use-cases.  This draft was trying to do the minimum that
> is sufficient and then, if and when there is a need for more, what extra is
> needed could be better defined.****
>
> ** **
>
>   2.1.  End-to-End Constraints:****
>
> ** **
>
>     Note: I've requested that jitter and loss be dropped from****
>
>     draft-ietf-ospf-te-metric-extensions and****
>
>     draft-previdi-isis-te-metric-extensions for the same potential****
>
>     oscillations and stability reasons cited above.****
>
> ** **
>
> [Alia] Yes – I think we clearly need to have a good email discussion with
> those interested about what oscillations might actually occur and what
> could be done to prevent that.****
>
> ** **
>
>     s/While it has been possible to compute a CSPF/It is possible to****
>
>     compute a CSPF/****
>
>     s/Instead of this approach to minimize path latency, an/An****
>
>     alternative to this approach to minimize path latency is an****
>
>     approach to place a upper bound on path latency.  An/****
>
>     Note: both approaches are valid.****
>
> ** **
>
>     Delete next paragraph starting with "This is illustrated as****
>
>     follows."  This seems to be taken from email and is good email****
>
>     discussion but not needed in the draft.****
>
> ** **
>
> [Alia] I’ve put in some pseudo-code so I’m ok with taking the example out
> – but there does seem to have been confusion even with it in.****
>
> ** **
>
>     Delete next paragraph starting with "An end-to-end bound on delay****
>
>     variation".  Lets get rid of jitter altogether.  (Let the old SNA****
>
>     networks be damned. :)****
>
> ** **
>
>     Delete next paragraph starting with "For link loss".  Get rid of****
>
>     link loss altogether.****
>
> ** **
>
> [Alia] Not done – discussion is needed.  I understand that you really
> really really don’t want to see loss or delay variation in any of these
> drafts.****
>
> ** **
>
>   2.2.  Link Constraints:****
>
> ** **
>
>     Drop delay variation and link loss.****
>
> ** **
>
>     If we are dealing with EF traffic then using Unidirectional****
>
>     Available Bandwidth and Residual Bandwidth makes no sense.  If we****
>
>     are dealing with low priority traffic and we are using****
>
>     Unidirectional Available Bandwidth and Residual Bandwidth in path****
>
>     selection will be prone to oscillations and network instability.****
>
>     Therefore delete the entire second paragraph (the one starting****
>
>     with "When doing path selection for TE tunnels,".****
>
> ** **
>
> [Alia]  If we are trying to avoid congesting the link, then Residual
> Bandwidth is important and useful regardless of the LSP traffic class.
> What is your specific concern with oscillation for bandwidth?  Why is it
> different than for the bandwidths already advertised and used for TE path
> computation?   Come on – the Residual Bandwidth is just the Link Capacity
> minus that reserved by RSVP-TE – so pretty darn similar characteristics to
> the Unreserved Bandwidth per priority…   The Unidirectional Available
> bandwidth does include an actual traffic measurement – that is averaged
> over a reasonable interval, that can only change with limited frequency –
> and then the LSPs need to be reoptimized to use them.  Please explain
> precisely the oscillation concern with a clear example.****
>
> ** **
>
> ** **
>
>     Also delete the entire third paragraph (starting with "Similarly,****
>
>     only links whose loss is").  As stated earlier, use of loss is****
>
>     either a NOOP (low volume of EF traffic) or can lead to****
>
>     oscillations.  Get rid of loss entirely.****
>
> ** **
>
>   2.3.  Links out of SLA:****
>
> ** **
>
>     s/SLA/NPO/g (see nits below).****
>
> ** **
>
>     General: A better mechanism than an "Anomalous State" flag is****
>
>     needed to provide notifications of change.  One mechanism would be****
>
>     to put the commulative constraint and the cummulative total in the****
>
>     ERO and RRO.  A link adding delay can then notify the ingress of****
>
>     any LSP for which the commulative constraint is violated.  See****
>
>     work by Swallow et al and perhaps align with that work.****
>
> ** **
>
> [Alia] That could be done as well – but the signaling extensions seem like
> overkill for the basic problem of letting the ingress compute a complete
> ERO.  Carrying both the cumulative constraint and the cumulative total just
> means that the midpoint detects a changed link and then has to verify the
> performance on each LSP and individually signal it.  Having an anomalous
> flag provides a succinct notification to the ingress which can then do the
> verification and be known to have the updated information.  This solution
> also requires that the midpoints all support it before it can be used.  An
> advantage of the ingress computation is that only the ingress needs to have
> updated code.****
>
> ** **
>
>     The "Anomalous State" flag is helpful for c in the list, but not****
>
>     for b.  The case where a link is out of NPO by 1 msec then changes****
>
>     to out of NPO by 10s of msec is an example.  The flag is already****
>
>     set and therefore the trigger is unavailable.****
>
> ** **
>
> [Alia] Right – but LSPs that are particular sensitive (i.e. a or c) can
> have been moved already.   Having the Anomalous flag set is expected to be
> unusual.  I agree that it is more a hint for (b) than a full solution.****
>
> ** **
>
>   2.3.1.  Use of Anomalous Links for New Paths****
>
> ** **
>
>     Delete second paragraph regarding jitter and loss.****
>
> ** **
>
>   2.3.2.  Links entering the Anomalous State****
>
> ** **
>
>     This section ignores two cases in which the Anomalous State does****
>
>     not change but a change makes the path violate a delay****
>
>     constraint.  The first is where all of the links are within NPO****
>
>     but a change to a link has make the path delay sum exceed the path****
>
>     delay constraint.  The second is where a link which is already in****
>
>     the Anomalous State by a small margin but the path delay is still****
>
>     within the constraint.  A large change in delay at that point will****
>
>     not affect the Anomalous State since it is already set.****
>
> ** **
>
> [Alia]  The Anomalous bit is NOT meant to replace reading the actual value
> associated with the link and checking each potentially affected LSP.  In
> the case of (b), it is a hint to focus on those LSPs first.****
>
> ** **
>
>     A better means of handling case (b) in "2.3.  Links out of SLA" is****
>
>     needed.****
>
> ** **
>
> [Alia]  I’ve added the following paragraph at the end of 2.3.2:****
>
> “It is not sufficient to just look at the Anomalous bit in order to****
>
> determine when TE tunnels must have their compliance verified.  When****
>
> changing to set, the Anomalous bit merely provides a hint that****
>
> interested TE tunnels for case (b) should have their continued****
>
> compliance verified.”****
>
> ** **
>
>   2.3.3.  Links leaving the Anomalous State****
>
> ** **
>
>     Same issue as in "2.3.2.  Links entering the Anomalous State".  A****
>
>     better means of handling case (b) in "2.3.  Links out of SLA" is****
>
>     needed.****
>
> ** **
>
> ** **
>
> [Alia]  I’ve added the following sentence:****
>
> “The hint provided by the Anomalous state change may help optimize when to
> recompute for a better path.”****
>
> ** **
>
> XML version nits:****
>
> ** **
>
>   CV: you really should remove the template comments.****
>
> ** **
>
> [Alia]  I find them helpful for when I want to do additional things… and
> very very few people read the XML.****
>
> ** **
>
>   You should also enable strict mode.  For example, you have one****
>
>   author too many and strict would catch that.****
>
> ** **
>
> [Alia]  So does basic arithmetic J   It will be resolved before the draft
> is passed to the RFC editor.****
>
> ** **
>
> Other nits:****
>
> ** **
>
>   The "A" in SLA is "Agreement" as in contract.  The acronym NPO for****
>
>   network performance objective seems to be in vogue for that reason.****
>
>   IETF since diffserv has wanted to steer clear of making****
>
>   recommendations regarding provider contracts (agreements) with****
>
>   customers, peer, or anyone else.****
>
> ** **
>
> [Alia]  True - changed****
>
> ** **
>
>   I agree with Sri on the suggestions to change the title, short name****
>
>   and document filename but I'm not fond of his suggested new names.****
>
>   Authors please suggest new title, short name, and filename.****
>
> ** **
>
> [Alia]  I did do a new short name and title.  As for filename, we’ll deal
> with that if/when the draft is adopted as a WG draft.****
>
> ** **
>
> Thanks again,****
>
> Alia****
>
> ** **
>
> ** **
>
> ------- Forwarded Message****
>
> ** **
>
> On 4/5/13 9:50 PM, "Loa Andersson" <[email-masked]> wrote:****
>
> >** **
>
> >Xiaohu, Sri, Rajiv and Pranjal,****
>
> >** **
>
> >You have been selected as an MPLS Review team reviewers for ****
>
> >draft-atlas-mpls-te-express-path-02.****
>
> >** **
>
> >Note to authors: You have been CC'd on this email so that you can know **
> **
>
> >that this review is going on. However, please do not review your own ****
>
> >document.****
>
> >** **
>
> >Reviews should comment on whether the document is coherent, is it ****
>
> >useful (ie, is it likely to be actually useful in operational ****
>
> >networks), and is the document technically sound?  We are interested in *
> ***
>
> >knowing whether the document is ready to be considered for WG adoption **
> **
>
> >(ie, it doesn't have to be perfect at this point, but should be a good **
> **
>
> >start).****
>
> >** **
>
> >Reviews should be sent to the document authors, WG co-chairs and WG ****
>
> >secretary, and CC'd to the MPLS WG email list. If necessary, comments ***
> *
>
> >may be sent privately to only the WG chairs.****
>
> >** **
>
> >Are you able to review this draft by April 20, 2013?****
>
> >** **
>
> >Thanks, Loa****
>
> >(as MPLS WG chair)****
>
> >** **
>
> >/Loa****
>
> >--****
>
> >** **
>
> >** **
>
> >Loa Andersson                        email: [email-masked]****
>
> >Senior MPLS Expert                          [email-masked]****
>
> >Huawei Technologies (consultant)     phone: +46 739 81 21 64****
>
> ** **
>
> _______________________________________________****
>
> mpls mailing list****
>
> [email-masked]****
>
> https://www.ietf.org/mailman/listinfo/mpls****
>
> ** **
>
> ------- End of Forwarded Message****
>
> ** **
>
> [email-masked]

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 10 Jul 2013 12:52:37 -0400

Hi Alia,

I think the confusion here is that there are approaches to minimize
multiple metrics which are too computationally complex to be practical
but that is not what is being proposed.

There are two things being proposed.  One is minimizing the delay by
using the delay as a single metric.  The other is bounding the delay.
This is done using the existing CSPF but truncating the CSPF search
when the cumulative delay exceeds the target.

Perhaps it is not sufficiently clear that the former approach *is not*
requiring or even recommending the use of algorithms that some vendors
have toyed with in the past to optimize for multiple metrics.  Those
algorithms are not prohibited except any practical prohibition which
will arise from an inability (so far) in meeting convergence time and
restoration time requirements of providers.

Curtis

In message <[email-masked]>
Alia Atlas writes:
 
> Can you clarify why you think constraining the links explored based
> upon a latency isn't a practical approach?  It is still O(n log n) -
> granted, it's a heuristic and may not find a path.
>  
> Alia
>  
> I think the alternative approach (i.e., to compute a CSPF path based
> on the TE metric while placing a upper bound on the path latency) is
> not a practical approach due to computationally complexity.
>  
> Best regards,
> Xiaohu

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 11 Jul 2013 15:38:20 -0400

In message <[email-masked]>
Alia Atlas writes:
> in-line, as always.

Hi Alia,

I have to say that it was *very* hard to determine the context in the
mail below from the formating alone, in other words what I wrote vs
your reply.  The HTML had MsoPlainText tags and the mail headers had
X-MS- headers so the culprit can be identified but there was no header
identifying the sending MUA (mail user agent).  This is the first
email I got from you that suffered from this.  It would be helpful if
you could use an alternate mailer for at least IETF work.

Anyway, comments are inline.  I tried to reformat to regain context.

btw- NPO is no longer in vogue as you are now well aware (due to the
perception that the ITU may feel they own the term and we are misusing
it).  Therefore s/NPO/performance objective/.  Or perhaps for this set
of documents you could define link performance objective (LPO) and
avoid any issues with NPO (btw ITU defines NPO as UNI to UNI and given
that definition NPO makes no sense at all in this context).

> > others).  I've commented before on the mailing list about this draft
> > but at this point I would like to provide a detailed review.
> >  
> > I think there are very major issues with this document as it now
> > stands.  IMO these issues should be addressed before the draft is even
> > accepted as a WG document.
> >  
>    oscillations and stability issues that have plagued prior attempts
>    to use delay as a routing metric.  If application traffic which
>    follows path based upon latency constraints, the same traffic might
>    be in an Expedited Forwarding Per-Hop-Behavior [RFC3246] with
>    minimal queuing delay or another PHB with potentially very
>    substantial per-hop queuing delay.  Only traffic which experiences
>    relatively low congestion, such as Expedited Forwarding traffic,
>    will experience delays very close to the sum of the reported link
>    delays
>  
> draft-ietf-ospf-te-metric-extensions.

OK

> As far as jitter and loss goes, these can still be somewhat
> meaningful.  For instance, jitter does allow capturing the differences
> in serialization delays - which is relevant in some parts of the
> network.  The acceptable loss to take a link down may vary.  I don't
> actually see the reasoning or need to remove jitter and loss from the
> draft - given that we have constrained the measurements in the
> draft-ietf-ospf-te-metric-extensions to not include queuing delay.

The argument above regarding jitter and loss doesn't pass the
"reasonable" threshhold.

Lets do some quick math on the jitter number.  On a 10 Gb/s link the
time it takes to pass 1532 bytes is (1532*8)/(10*10^9) or 1225*10^-9
or 1.225*10^-6 sec or a bit over 1.2 usec.  That is the delay in about
240 meters of fiber.  That amount of fiber won't even get you down the
elevator shaft in a lot of Manhattan buildings let alone down the
street.  The argument for jitter as a measure of serialization delay
doesn't seem to me to be very credible.

Besides, serialization is queuing jitter when only occupancies of 0
and 1 are considered.  In the real world when you add more traffic
even for EF the probabilities of occupancies of 2, 3, or more rises
even if there is no congestion.  If you include this jitter
measurement which is inherently a queuing effect and extremely
sensitive to loading, then you risk oscillations and instability.

Your statement "The acceptable loss to take a link down may vary" also
doesn't justify including loss.  If you want to advertise a protection
or restoration time, that might be more reasonable.  These things are
not measured in "loss" units because how much gets lost depends on the
utilization during the protection or restoration time.  This is also
something that today is generally handled by rfc3209 administrative
attributes (colors) to indicate protected vs unprotected links if it
matters to some LSP.  Note that the difference between end-to-end
protection and link by link protection only makes sense for fairly
long delay links, where the delay (time in flight before detection) is
long compared to the typical protection times of about 10 msec for
small number of LSP or under 45 msec for lots of LSP (except some
implementation that don't acheive this for lots of links but certainly
won't be advertising that).

I have made the same comment regarding
draft-ietf-ospf-te-metric-extensions.  There is no reason to have
jitter and delay if queuing impacts are not considered.

> [Alia] As you may recall from draft-ietf-ospf-te-metric-extensions,
> the Anomalous bit is specified per characteristic advertised per link.
> This draft (draft-atlas-mpls-te-express-path-02) describes the
> Anomalous bit as clearly inside a links' sub-TLV as in Sec 2.3.1:
>  

I think s/SLA/link performance objective/ (or LPO as mentioned above)
in a lot of places would help.  Where you mean end-to-end, just say
end-to-end performance objective.  The document would then read easier
for those who haven't already read the document a dozen times (or read
the email conversations that make it already very clear).

> [Alia] Sure - different LSPs may have different requirements as to the
> Anomalous flag.  Using it doesn't replace paying attention to the
> value that is advertised.  The anomalous flag is reporting that the
> link is not complying to the expected performance - this can indicate
> that there is something strange going on and some LSPs should avoid
> using that link due to administrative policy.
>  
> draft-ietf-ospf-te-metric-extensions.  This draft doesn't define what
> is flooded.  A question though is whether in the same network it would
> make sense to consider both the NPO delays and the actual measured
> delays?  If only one or the other - then that could be a local
> router's decision as to which is flooded.

NPO (or link performance objective) delay and measured delay would be
two different numbers.  If this document is about the use of
draft-ietf-ospf-te-metric-extensions, then whether it needs to have a
link performance objective advertised in order for "anomolous" to make
sense is an issue for this document.

> [Alia] I am interested in the opinion of the WG on this.  In my view,
> this draft is describing how the information flooded via the new
> sub-TLVs in draft-ietf-ospf-te-metric-extensions-04 should be used.
> Loss and jitter are included in that draft; of course, that draft
> could change as well - but I'd like to hear stronger arguments for why
> it is harmful to include them than that they're only relevant when
> queuing behavior is included and that can lead to oscillations.  I am,
> perhaps stubbornly, not convinced that they are irrelevant.

See argument above.

btw- If we add queuing delay and loss, then we can do OSPF-OMP,
ISIS-OMP, and MPLS-OMP as defined back in the mid to late 1990s
without requiring any protocol changes.  :-)
Scary thought of the day.

> http://tools.ietf.org/html/draft-ietf-ccamp-te-metric-recording-01?
> That draft appears to me to be about how to get measurements for
> latency, jitter, and cost for a given Forwarding Adjacency or Routing
> Adjacency so that those values can be advertised into the IGP.  I
> think this draft is providing a means to collect the data to flood in
> draft-ietf-ospf-te-metrics-04.

Yes, that is the draft.  But no I don't think that the intent is
strictly to readvertise the delay for an FA given that the ingress
could just sum the values advertised for the links.  I think by
recording the sum, the ingress can more easily check the latest RESV
for a change that could require a route computation.

The motivation could also be for MPLS with loose hops where a midpoint
LSR could change the hops (but I'm not sure how often if ever that is
used, not being a firm beleiver that multidomain RSVP-TE ever got much
deployment or persisted where experimented with).

Perhaps we should ask George et al about the intended usage.

The stated motivation in the abstract is:

   This draft provides extensions for the Resource ReserVation
   Protocol Traffic Engineering (RSVP-TE) for the support of the
   discovery of cost, latency and latency variation of an LSP.

The introduction gives examples, most of which are multidomain
related.  Another example is the egress of a bidirectional LSP.

>    This document does not specify how a router determines what values
>    to advertise by the IGP; it does assume that the constraints
>    specified in [I-D.ietf-ospf-te-metric-extensions] and
>    [I-D.previdi-isis-te-metric-extensions] are followed.  Mechanisms
>    for determining latency and delay variation for Forwarding
>    [I-D.ietf-ccamp-te-metric-recording]."

You wouldn't use a direct measurement of delay using IP-Prec 6 or 7 on
an FA?

If so, you might want to cite the MPLS-TP DM RFCs and note that an
Entropy Label may be needed in some cases to insure that the
measurement is accurate.

Better to just not cite anything as the means of measurement.

> impact stability.  I am not fully persuaded that this is a practical
> issue - given the delays in ingress re-optimization, not trying to
> minimize the jitter, and the ability for an LSP to reserve bandwidth.
> I would be interested in what text you might provide and a focused
> discussion on whether this is something that needs to have
> restrictions clearly described to avoid oscillations.

Here is a start:

   1.2  Oscillation and Stability Considerations

   Past attempts to use delay or loss as metric sufferred from severe
   oscillations [].  The use of performance based data MUST be such
   that ocillations are not possible and stability cannot be impacted.

   The use of timers is often cited as a cure.  Oscillation that is
   damped by timers is known as "slosh".  If advertisement timers are
   very short relative to the jitter applied to RSVP-TE CSPF timers,
   then a partial oscillation occurs.  If RSVP-TE CSPF timers are
   short relative to advertisement timers, full oscillation (all
   traffic moving back and forth) can occur.  Even a partial
   oscillation causes unnecessary reordering which is considered at
   least minimally disruptive.

   Delay variation or jitter is affected by even small traffic levels.
   At even tiny traffic levels, the probability of a queue occupancy
   of one can produce a measured jitter proportional to or equal to
   the packet serialization delay.  Very low levels of traffic can
   increase the probability of queue occupancies of two or three
   packets enough to further increase the measured jitter.  Because
   jitter measurement is extremely sensitive to even very low traffic
   levels, any use of jitter is likely to oscillate.  There may be
   legitimate use of a jitter measurement in path computation that can
   be considered free of oscillation.

   Delay measurements that are not sensitive to traffic loads may be
   safely used in path computation.  Delay measurements made at the
   link layer or measurements made at a queuing priority higher than
   any significant traffic (such as DSCP CS7 or CS6 [RFC4594], but not
   CS2 if traffic levels at CS3 and higher or EF and AF can affect the
   measurement).  Making delay measurements at the same priority as
   the traffic on affected paths is likely to cause oscillations.

   Delay measurements that include queuing delay or loss measurement
   that includes queuing loss would be very difficult to use in path
   computation in a way that can be assured to be stable.  No
   technique to date has successfully accomplished this.  Timers
   values must reflect the number of contributors to traffic on each
   given link or a very conservative estimate of the potential
   contributors.  Moving large LSP can itself be problematic and
   techniques which allow movement of partial LSP traffic, such as
   multipath techniques, may help.
   
   If queuing delay or loss measurement is considered, a proof of
   stability should be undertaken.  These proofs insure that no
   positive feedback exists such that small or moderate changes in
   traffic patterns could cause path decisions to fluctuate wildly.
   Proof of stability for an arbitrary topology with arbitrary traffic
   patterns and using a set of rules for determining timer values and
   traffic adjustment amounts can be exceedingly difficult.

   Until a path selection technique is available which is provably
   stable, a condition that today has not been met, queuing delay and
   queuing loss MUST NOT be included in delay and loss measurements.

Note that most of RFC4594 provides nothing more than recommendations
is routinely ignored except that CS6 is normally used for control and
management (IGP, BGP, RSVP-TE control, LDP control, SNMP, etc), though
often SNMP is often given lesser priority.

Proof of stability is very difficult (I tried to get help with this
with OMP back in the 1990s).  Note that OMP could move traffic in one
direction and then move half that amount in the other direction if
overshoot occurred, yet we still couldn't come up with a stability
proof for an arbitrary topology, even with timers and backoffs that
considered the number of "other" contributors to traffic (number of
other LSP ingress ingress in the topology).

>    When considering performance-based data, it is obvious that there
>    are additional contributors beyond just the links. Clearly
>    end-to-end latency is a combination of router latency, queuing
>    latency, physical link latency and other factors.  While traversing
>    a router can cause delay, that can be included in the advertised
>    link delay.  As described in [I-D.ietf-ospf-te-metric-extensions]
>    and [I-D.previdi-isis-te-metric-extensions], queuing delay should
>    not be included in the measurements advertised by OSPF or ISIS.

... queuing delay MUST NOT be included ...

>    oscillations and stability issues that have plagued prior attempts
>    to use delay as a routing metric.  If application traffic which
>    follows path based upon latency constraints, the same traffic might
>    be in an Expedited Forwarding Per-Hop-Behavior [RFC3246] with
>    minimal queuing delay or another PHB with potentially very
>    substantial per-hop queuing delay.  Only traffic which experiences
>    relatively low congestion, such as Expedited Forwarding traffic,
>    will experience delays very close to the sum of the reported link
>    delays.

The traffic levels can't impact the measured delay.  For that to
occur, the delay measurements have to be queued ahead of the traffic.
Changing the path of EF traffic and making the delay measurements at
EF is unsafe.  If CS7 and CS6 is queued ahead of EF, then delay must
be measured at CS6 or better yet CS7 (or still better at link layer).

> [Alia] Changed SLAs to NPO

Sorry, but now change it to "performance objective" or "link
performance objective" or LPO.  See discussion way above.

> [Alia] Frankly, I'm confused by this.  I don't agree that RSVP-TE
> signaling extensions would solve, for instance, (3):
>  
>    3. Ability to periodically verify that a TE tunnel's current LSP
>       complies with its configured end-to-end performance
>       requirements.

Sure.  The RESV changes if the delay of a link along the way changes.
If the link delay or end-to-end delay is a constraint, the LSP can be
rejected which would best be handled using soft preemption.

> Even if the ingress signaled the end-to-end performance requirements,
> I don't see how that stops the ingress from verifying compliance?

If the RESV has to be updated with the latest value, the ingress at
the very least gets a verification without requiring the use of probe
packets by the ingress.  Link to link DM packets are much more
efficient that each ingress sending probe packets to verify delay.

>    4.  Ability to move tunnels, using make-before-break, based upon

Any change in the RESV or a soft preempt give the ingress a higher
priority heads up that something changed.  A soft preempt is a good
heads up on an SLA violation for any LSP that cares about that.

> I have looked for the additional work that you are talking about by
> George Swallow unsuccessfully.  Can you find a pointer to explain what
> you're talking about?  The anomalous bits provide a trigger mechanism
> that a router can use to tell the ingress to do (5); different admin
> attributes could do a similar behavior - and similarly for (6) - but
> again that is the flooding for notification aspect.  I think the lack
> of RSVP-TE extensions doesn't cause an issue - these are handled by
> IGP instead and thus don't have to be signaled per LSP.

The work by Swallow does not solve your problems.  It is a good
starting point and like the prior situation with multiple delay metric
related drafts consolidating very similar efforts is a good thing.  So
if you would consider RSVP-TE extensions, then try to align your work
with George Swallow's work.

> I am not irrevocably opposed to RSVP-TE extensions - if we really need
> them for practical use-cases.  This draft was trying to do the minimum
> that is sufficient and then, if and when there is a need for more,
> what extra is needed could be better defined.

The point is to be realistic about what the limited mechanisms do well
and don't do well.

Note: At one point at least one major RSVP-TE implementation didn't
check to see if constraints for LSP were met after an IGP change.  For
example if a set of LSP include the admin attribute constraint
"exclude color blue" and a link gets changed to color blue, then those
LSP could sit there for hours before being rerouted.  That (or those)
implementation(s) only prioritized CSPF for LSP where a notification
or PATH/RESV TEAR had been received.  You might want to check to see
how your employers RSVP-TE handles this.  It used to do this but I
don't know if it still does.

> [Alia] Yes - I think we clearly need to have a good email discussion
> with those interested about what oscillations might actually occur and
> what could be done to prevent that.
>  
> [Alia] I've put in some pseudo-code so I'm ok with taking the example
> out - but there does seem to have been confusion even with it in.
>  
> [Alia] Not done - discussion is needed.  I understand that you really
> really really don't want to see loss or delay variation in any of
> these drafts.

Yes, for reasons very clearly stated above.

> What is your specific concern with oscillation for bandwidth?  Why is
> it different than for the bandwidths already advertised and used for
> TE path computation?  Come on - the Residual Bandwidth is just the
> Link Capacity minus that reserved by RSVP-TE - so pretty darn similar
> characteristics to the Unreserved Bandwidth per priority...  The
> Unidirectional Available bandwidth does include an actual traffic
> measurement - that is averaged over a reasonable interval, that can
> only change with limited frequency - and then the LSPs need to be
> reoptimized to use them.  Please explain precisely the oscillation
> concern with a clear example.

On rereading draft-ietf-ospf-te-metric-extensions this is OK as
defined.

Residual Bandwidth doesn't use any measured traffic and is therefore
OK.

Available Bandwidth also subtracts out just the non-RSVP-TE traffic
and so that too should be OK.  The non-RSVP-TE traffic (IP and LDP
traffic) will not move as long as the IGP metrics don't change.

> [Alia] That could be done as well - but the signaling extensions seem
> like overkill for the basic problem of letting the ingress compute a
> complete ERO.  Carrying both the cumulative constraint and the
> cumulative total just means that the midpoint detects a changed link
> and then has to verify the performance on each LSP and individually
> signal it.  Having an anomalous flag provides a succinct notification
> to the ingress which can then do the verification and be known to have
> the updated information.  This solution also requires that the
> midpoints all support it before it can be used.  An advantage of the
> ingress computation is that only the ingress needs to have updated
> code.

Coding this at the midpoint is very easy and efficient.  First when an
LSP is signaled, compute the difference between the LSP delay target
and the experienced delay (the PATH can give the delay total of prior
links, the RESV can give delay total for downstream links.  Take that
headroom figure and install a pointer to the LSP in a sorted
collection with good insert/delete times (a balanced tree for
example).  When another link delay changes, the PATH or RESV changes
and the LSP has to be scheduled for removal and reinsertion into the
sorted collection.  Initially put it on a DLL to make this
computationally trivial.  Clean up the DLL entries periodically in
spare time.  When a link changes, start at the end of the sorted
collection with the most sensitive LSP.  Go through the list doing
soft preempt on all those in violation.  Simply update the delay value
in the PATH and RESV on the rest and that can be done in spare time.

Those LSP that don't care about delay don't get on any of these lists.

> [Alia] Right - but LSPs that are particular sensitive (i.e. a or c)
> can have been moved already.  Having the Anomalous flag set is
> expected to be unusual.  I agree that it is more a hint for (b) than a
> full solution.
> 
> [Alia] The Anomalous bit is NOT meant to replace reading the actual
> value associated with the link and checking each potentially affected
> LSP.  In the case of (b), it is a hint to focus on those LSPs first.
> 
>    It is not sufficient to just look at the Anomalous bit in order to
>    determine when TE tunnels must have their compliance verified.
>    When changing to set, the Anomalous bit merely provides a hint that
>    compliance verified.

The point that I am making is that a change to the "Anomalous State"
flag is an unreliable hint for (b) where (b) is:

   b.  Should LSPs using this link be immediately verified for continued
       compliance to their end-to-end constraints?

Perhaps you should take (b) out of the list and state that for LSP
which have end-to-end constraints, but for which the "Anomalous State"
flag does not automatically disqualify a link, the advertised
parameters will have to be checked on every received advertisement
with a change.  Or take out (b) and leave the paragraph that you have
suggested adding.

>    The hint provided by the Anomalous state change may help optimize
>    when to recompute for a better path.

It is an unreliable hint so using that hint is a bad practice.
Unreliable hints don't "help".

> [Alia] I find them helpful for when I want to do additional
> things... and very very few people read the XML.

OK

> draft is passed to the RFC editor.

Strict mode does a lot of other checks that are supposed to get run
when a doc becomes a WG doc, long before it goes to the RFC Editor.

I just used datatracker to do a check nits and it had a few things to
complain about.

  ** The document seems to lack a both a reference to RFC 2119 and the
     recommended RFC 2119 boilerplate, even if it appears to use RFC
     2119 keywords.
  
  == Unused Reference: 'RFC5420' is defined on line 291, but no
     explicit reference was found in the text

  == Outdated reference: A later version (-04) exists of
     draft-ietf-ospf-te-metric-extensions-02

  == Outdated reference: A later version (-03) exists of
     draft-previdi-isis-te-metric-extensions-02

The first two need to be fixed.  The second two are just a matter of
updating the references.

> >   customers, peer, or anyone else.

Sorry about this but s/NPO/?/
Where "?" is "performance objective", "link performance objective",
LPO, or descriptive term of your choice.  See above.

> [Alia]  I did do a new short name and title.  As for filename, we'll
> deal with that if/when the draft is adopted as a WG draft.

I haven't seen the new name.

> Alia

Thanks,

Curtis

> [ ... trimmed forwarded MPLS-RT message from Loa ... ]

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 12 Jul 2013 02:52:14 +0000

Hi Alia,

Let me give an example as illustrated below:

A--------(Metric:1; Delay:1)-------B---(Metric:1; Delay:10)---D-----------F
|                        |                           / |
|                        +---(Metric:2; Delay:10)---E--------/  |
|                                                     |
|                                                     |
+-------(Metric:10; Delay:1)------C--------(Metric:1; Delay 1)--------------+

Assume A wants to find an optimal path to F with delay upper bound of 10. In step 1, A selects B among all of its adjacent routers as the next_router since its distance to other adjacent routers (i.e., C) is larger than that to B. in step 2, B finds that neither of its adjacent routers (i.e., D and E) could be selected as next_router since the latency of the either path (i.e., A->B->D and A->B->E) is already beyond the e2e delay bound. At a result, in step 3, A has to give up B and then explore its other adjacent routers. Is that the heuristic algorithm you meant? If so, it would be better to describe the step 3 (i.e., a retreat step) as well in your pseudo-code while mentioning that retreat step may be performed multiple times back and forth.

Best regards,
Xiaohu
   
> -----邮件原件-----
> 发件人: Alia Atlas [mailto:[email-masked]]
> 发送时间: 2013年7月10日 2:13
> 收件人: Xuxiaohu; [email-masked]; MPLS WG Mailing List;
> 主题: RE: [mpls] draft-atlas-mpls-te-express-path-02 (was MPLS-RT review
> of ...)
> 
> Can you clarify why you think constraining the links explored based upon a
> latency isn't a practical approach?  It is still O(n log n) - granted, it's a heuristic
> and may not find a path.
> 
> Alia
> 
> To: [email-masked]; MPLS WG Mailing List;
> I think the alternative approach (i.e., to compute a CSPF path based on the TE
> metric while placing a upper bound on the path latency) is not a practical
> approach due to computationally complexity.
> 
> Best regards,
> Xiaohu
> 
>

---

From: Alia Atlas <[email-masked]>
Date: Fri, 12 Jul 2013 05:05:18 +0000

Hi Xiaohu,

No, there is no retreating.  I think you're describing something based on a depth-first-search that's cost aware??? I don't know but it's not based on a normal SPF.  The pseudo-code I gave is correct.  

Here's the brief example -  I mean an SPF where links aren't explored if they result in a path that is too long.

Step 1 CSPF:   A has cost 0, delay 0   - explore A's links
          B.cost = 1  B.delay= 1  insert B into the fibheap ordered by cost
          C.cost = 10  C.delay = 1   insert C into the fibheap ordered by cost

Step 2 CSPF:  remove first node from fibheap - it is B  - explore B's links
          Via B->D would give D.delay to be 11 - over bound so don't save 
          Via B->E would give E.delay to be 11 - over bound so don't save

Step 3 CSPF: remove first node from fibheap  - it is C - explore C's links
            Via C->F would give F.delay=2 -under bound so save
                 F.cost = 11   F.delay = 2  insert F into fibheap

Step 4 CSPF:  remove first node from fibheap - it is F
         CSPF was from A to F and F is now minimized so terminate.

Alia                   

-----Original Message-----
From: Xuxiaohu [mailto:[email-masked]] 
Sent: Thursday, July 11, 2013 10:52 PM
To: Alia Atlas; [email-masked]; MPLS WG Mailing List; draft-atlas-mpls-te-express-path authors; The Great and Mighty MPLS Co-Chairs
Subject: 答复: [mpls] draft-atlas-mpls-te-express-path-02 (was MPLS-RT review of ...)

Hi Alia,

A--------(Metric:1; Delay:1)-------B---(Metric:1; Delay:10)---D-----------F
|                        |                           / |
|                        +---(Metric:2; Delay:10)---E--------/  |
|                                                     |
|                                                     |
+-------(Metric:10; Delay:1)------C--------(Metric:1; Delay 
+1)--------------+

Best regards,
Xiaohu

---

From: Alia Atlas <[email-masked]>
Date: Fri, 12 Jul 2013 02:27:17 -0400

Hi Curtis,

Comments in-line

On Thu, Jul 11, 2013 at 3:38 PM, Curtis Villamizar <[email-masked]>wrote:

>
> In message <
> [email-masked]
> >
> Alia Atlas writes:
> >
> > Hi Curtis,
> >

[Alia] Sorry about that - I'm using the same mail programs as always.

> that definition NPO makes no sense at all in this context).

[Alia] Right - I knew I didn't like NPO - but couldn't recall the reason
readily.
I've changed to using link performance objective - but didn't turn it into
an acronym.
I don't have a terminology section yet.

> doesn't seem to me to be very credible.
[Alia] Not all links are 10 Gb/s - there are still slow links in the world.
 There are still leased paths.

> sensitive to loading, then you risk oscillations and instability.

[Alia] Curtis, I hear your claims on this.  I know about what happened in
ARPAnet (from many
sources :-) and I am not convinced that pruning links that have too much
jitter to use is going to
cause horrific instability.  Why doesn't pruning links based on the
reservable bandwidth cause the
same problems??

Can you please describe a clear example where the potential is clear? I
don't claim to have
thought through, much less simulated, every possible case.

> jitter and delay if queuing impacts are not considered.

[Alia] I have people who run networks who want to have jitter and loss in
there.  I understand
that you don't see a reason - and I am certainly willing to encourage
discussion from those who
are interested in using this technology (and diligent enough to read this
far).

> the email conversations that make it already very clear).

[Alia] done - as above

> sense is an issue for this document.

[Alia]  Please explain the use-case where it is important to know the
actual link
performance objective.  There can be administrative policy for violating
it; that just
cares about the violation.  There can be LSP performance objectives that
depend on
the actual measured delays.   I don't see a use-case where the specific
value for the
link performance objective is important.

> Scary thought of the day.

[Alia] blast from the past :-)  but what might be better done with ECMP is
a different
story.

> related.  Another example is the egress of a bidirectional LSP.

[Alia] My skim of the draft was that it was for the measurements.  In a
multidomain case,
one couldn't just sum up the link measurements - particularly if the
details of the path in a
different domain were not shared due to policy.   We could certainly verify
with George if there's
an additional motivation.  (There's a lot of work in ccamp for years on
multidomain - hard to
believe it's to little purpose.)

> Better to just not cite anything as the means of measurement.

[Alia] Ok -  I added that statement in to try and place the draft you
mentioned into context.
It doesn't say that one needs to use them - but one could do direct
measurements or have
the control plane compute and measure them.

I do agree that there's no reason to talk about the means of measurement in
this draft.

>    that ocillations are not possible and stability cannot be impacted.

[Alia] Were they all uses of unbounded delay like ARPAnet?  It's certainly
the standard problem of everyone rushing to the shortest line.  I'm also
caveating
"oscillations are not possible" to "undampened oscillations are not
possible".  Let's
go for the merely very difficult :-)

>    the traffic on affected paths is likely to cause oscillations.

[Alia] I think I will cut it here.  The
draft-ietf-ospf-te-metric-extensions clearly specifies
that queuing delay and queuing loss are not included.  I don't think we
need to hammer
it in again here.

> other LSP ingress ingress in the topology).

[Alia] Proofs can be hard.  How did you approach it?  What tool did you use?
It may be interesting to revisit for other uses.

I have a hard time believing that changing demand in response to loading
hasn't
been studied a great deal for other applications (load-balancing servers,
etc). I
haven't gone poking yet though to understand the similarities and
differences.

> ... queuing delay MUST NOT be included ...

[Alia] yes

> be measured at CS6 or better yet CS7 (or still better at link layer).

[Alia] Yes - but THIS DRAFT is not defining how the measurements are made.

> performance objective" or LPO.  See discussion way above.

[Alia] yup - agreed and done

> rejected which would best be handled using soft preemption.

[Alia] So you are suggesting hauling the link delay values back in every
RESV instead
of just flooding it in the IGP.  Then we also run into race conditions
where the IGP hasn't
updated the link delay but RSVP knows that it's violated and CSPF keeps
landing on the
same path.   I don't see this as a superior solution to just flooding the
link constraints in
the IGP.

> efficient that each ingress sending probe packets to verify delay.

[Alia] ??  I don't think the ingress is sending probe packets.  I think it
is getting
the results of the link delay measurement reported via the IGP - and then
the
ingress is summing up the delays of the links in the path to determine
conformance.

> heads up on an SLA violation for any LSP that cares about that.

[Alia] If the midpoint knows what LSPs care about which link performance
violations,
it could, I guess, do a soft-preempt which might or might not cause the
ingress to do
the right thing.  I think we've moved into "how else could it be designed
with more
complexity".

> with George Swallow's work.

[Alia] Considering the effort to get this draft acceptable - as a purely
informational
here's how you could compute paths that use the extra flooded info - I'd
really like
to hear substantial and significant use-cases for why we would need RSVP-TE
extensions.

> don't know if it still does.

[Alia] Obviously an implementation that considers the anomalous bit would
need
to respond to it for relevant LSPs.  Thanks for the heads-up.  Markus has
his hands
in the RSVP-TE code; I  haven't needed to yet.

> traffic) will not move as long as the IGP metrics don't change.

[Alia] Exactly so

> Those LSP that don't care about delay don't get on any of these lists.

[Alia] LOL - I'm not arguing that the coding is hard or even the RSVP-TE
extensions
to communicate the requirements and accumulation.  It's a deployment
consideration
and a question of "is it necessary".

> with a change.  Or take out (b) and leave the paragraph that you have

[Alia] Agreed - I removed (b) but left in the section and description.

> Unreliable hints don't "help".

[Alia] It is not unreliable if the LSP was reoptimized due to
administrative policy instead
of, for example, delay bound.  I clarified "whose LSPs were changed due to
administrative
policy when the link entered the Anomalous state"...

>      recommended RFC 2119 boilerplate, even if it appears to use RFC

[Alia] Yes - I'm not convinced it's meaningful to have RFC 2119 keywords in
an
informational draft.  I've changed them to lower case.

>      explicit reference was found in the text

[Alia] Yup

> The first two need to be fixed.  The second two are just a matter of
[Alia] Done automatically with the new draft.

> [Alia] yup

> I haven't seen the new name.

[Alia]  The shortname is "Path Selection with TE Metric Extensions".  I'll
probably
name the draft "Fred" unless the WG chairs have a better solution ;-)

I'm taking the draft with all these changes and publishing it.

Thanks,
Alia

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 12 Jul 2013 06:33:43 +0000

Oh, yes, you are right. Sorry for my confusion.

Best regards,
Xiaohu

> 发送时间: 2013年7月12日 13:05
> No, there is no retreating.  I think you're describing something based on a
> depth-first-search that's cost aware??? I don't know but it's not based on a
> normal SPF.  The pseudo-code I gave is correct.
> 
> To: Alia Atlas; [email-masked]; MPLS WG Mailing List;
> Assume A wants to find an optimal path to F with delay upper bound of 10. In
> step 1, A selects B among all of its adjacent routers as the next_router since its
> distance to other adjacent routers (i.e., C) is larger than that to B. in step 2, B
> finds that neither of its adjacent routers (i.e., D and E) could be selected as
> next_router since the latency of the either path (i.e., A->B->D and A->B->E) is
> already beyond the e2e delay bound. At a result, in step 3, A has to give up B
> and then explore its other adjacent routers. Is that the heuristic algorithm you
> meant? If so, it would be better to describe the step 3 (i.e., a retreat step) as
> well in your pseudo-code while mentioning that retreat step may be performed
> multiple times back and forth.
> 
> Best regards,
> Xiaohu
> 
> > -----邮件原件-----

---

From: Alia Atlas <[email-masked]>
Date: Fri, 12 Jul 2013 02:35:18 -0400

Happens to all of us.  Thanks for doing the review.

Alia

On Fri, Jul 12, 2013 at 2:33 AM, Xuxiaohu <[email-masked]> wrote:

> > Here's the brief example -  I mean an SPF where links aren't explored if
> they
> > result in a path that is too long.
> >

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 12 Jul 2013 17:36:17 -0400

In message <[email-masked]>
[email-masked] writes:
 
> I'd like to get more feedback on removing jitter and loss from OSPF TE
> ME before doing so. So far, Curtis, it's only you asking for it (to be
> removed). Also, no one else seems uncomfortable with the A bit.. I am
> not personally in favor of removing them, and I feel the A bit is fine
> for my use case.

Spence,

I made comments about the use of the A-bit.  I did not ask that it be
removed (someone else did, not me) but asked that its usefulness (or
lack of) as a hint when the delay value also has to be checked be more
accurately described.  Briefly, in that case (when the delay value
also has to be checked) I feel the A-bit is useless.  In the case
where an LSP is configured such that it would never use a link with
the A-bit set on a given performance measure, then it is useful.

My recomendation is that for the case where the A-bit is not useful at
all, just say so.  Don't dance around and say it may be useful because
an unreliable hint is generally more harmful than useful.

As for the inclusion of jitter and loss, in the past I suggested a few
options.  Briefly:

  1.  Remove jitter and delay.

      --- OR ---

  2.  Keep jitter and delay and effectively say that jitter and loss
      MUST NOT be used unless a provably stable means of using them is
      discovered, which has not occurred to date.  And be very clear
      about this - no watered down wording.

Adding Section 1.2 "Oscillation and Stability Considerations" is a
step towards 2, but not adequate.

See below for further discussion on that.

Tony Li used to say that (sic) "If a customer asks us for rope, we
give them plenty or rope to hang themselves with".  Having been at a
vendor before I'll add "If a customer asks for rope and tells us they
plan to use it to hang themselves, we explain in detail that hanging
themselves with it will be unpleasant at best and likely fatal.  If
they continue to insist we give them the rope and reiterate the advice
and hope the advice sinks in before its too late."

With that in mind I would like to offer a compromise.

Leave jitter and loss in the OSPF, ISIS, and TE-express-path drafts
but reference an additional document, to be written by me, Alia, and
you, entitled "Oscillation and Stability Considerations for
Performance based MPLS Path Selection" and including the brief
information I suggested for draft-atlas-mpls-te-express-path plus
further details on stability considerations, including stability
discussion regarding advanced multipath (was CL work in RTGWG).

I'm fine with you keeping jitter and loss in the documents, if you are
willing to cite this document [Oscillation and Stability
Considerations for Performance based MPLS Path Selection] from the
OSPF, ISIS, and draft-atlas-mpls-te-express-path and state that "Any
use of jitter or loss measurements or any use of queuing delay in
delay measurements is strongly discouraged.  Before considering
whether to use jitter or loss measurements or any use of queuing delay
in delay measurements network operators MUST consider the warnings in
[Oscillation and Stability Considerations for Performance based MPLS
Path Selection]."

With this compromise you get the rope you wanted and you get a clear
warning about specific ways to use it that are strongly inadvisable.

Please let me know if you think that is a good compromise.

Curtis

Regarding other prior discussion on this topic:

Here was my recommendation for Section 1.2 from prior email.

   

BTW- I meant to say "There may be *no* legitimate use of a jitter
measurement in path computation that can be considered free of
oscillation."  Amazing what omiting one word can do.

Here is what Alia added which includes most of the above.

   1.2.  Oscillation and Stability Considerations

   Past attempts to use unbounded delay or loss as metric sufferred
   from severe oscillations.  The use of performance based data must
   be such that undampened oscillations are not possible and stability
   cannot be impacted.

And finally my latest comments on this on the MPLS-RT review thread
which runs to the end of this email (74 lines, not that long).

 OLD

The last sentence is unjustified and in conflict with the remaining
paragraph.  Please change that one sentence.

 OLD

   There may be legitimate use of a jitter measurement in path
   computation that can be considered free of oscillation.

 NEW

   While there may in the future be legitimate use of a jitter
   measurement in path computation that can be considered free of
   oscillation, except for uses with extremely low utilization within
   a given very high priority traffic class, and therefore almost no
   jitter to measure, no legitimate use of a jitter measurement in
   path computation that can be considered free of oscillation is
   known today.

Nit:

s/undampened oscillations/undamped oscillations/
I make this mistake all the time too.  The former is "dry".

There is no warning about using queuing loss measurements and the
warnings about using delay measurements in certain circumstances was
omitted.  The text I had previously suggested is:

Provably stable is not an insurmountable goal.  TCP's feedback
mechanism is provably stable and with a relatively simple proof.  For
the case of LAG or other multipath where local adjustment affects only
traffic into adjacent links, a provable stable load balance can occur.
This is more difficult for longer distance load balancing.  It is even
more difficult with the granularity of moving whole LSP rather that a
portion of a hash space over a diverse key (ie: IP src/dst or Entropy
Label) and it may turn out that given the use of path selection,
and therefore moving whole LSP, stability cannot be assured.
Therefore there is a strong argument that neither jitter or loss have
any legitimate use for which stability cannot be assured.