Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

From: William Britto A J <[email-masked]>
Date: Mon, 1 Mar 2021 10:16:09 +0000

Hi Robert,

Thanks for your comments.

Currently there are customers who deploy separate networks, of which one could be assigned metrics relative to the interface bandwidths, while other could be based on other parameters like latency, etc. Flex-Algo which facilitates different metric-types for SPF, is a very useful feature for such network consolidation use-cases. We are trying introduce the protocol constructs to simplify the use of metric based on bandwidth via Flex-Algo.

Any existing tools and techniques used by operators who configure metric relative to link bandwidth to optimize the network can be used with these Flex-Algo constructs too. The Flex-Algo bandwidth constraints defined here can be used to automatically derive a metric based on reference bw vs link bw; and if required, this can be overridden using the individual link bandwidth-metric sub-TLV. We will make this clear in the next version.

RFC 8570 support advertising link delay parameters in ISIS. Protocols like TWAMP support dynamically measuring the delay. Whether egress queueing delay is included in the link delay depends on the measuring mechanism. The Exclude Max Delay sub-TLV introduced in this draft is only meant for pruning out high latency links from a Flex-Algo, and it is upto the operator to define the maximum bounds based on the measuring mechanisms deployed in his network.

Thanks,
William

From: Robert Raszuk <[email-masked]>
Date: Saturday, 27 February 2021 at 5:54 PM
To: William Britto A J <[email-masked]>
Cc: [email-masked] <[email-masked]>, Rajesh M <[email-masked]>, Shraddha Hegde <[email-masked]>, DECRAENE Bruno IMT/OLN <[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
[External Email. Be cautious of content]

Hi William & co-authors,

I read the draft and have two basic questions.

1.
Both bw & delay can be used as defined in the draft to construct new forwarding topologies. But how practical such topologies would be in the real life when 40GB links may be heavily occupied with bursty traffic and 10G links can sit idle ? I suppose you are trying to address the case where say 12 gbps holographic stream needs to be sent across a network.. But then I don't think if sending it in a single flow instead of spreading into many sub-flows and use as much as possible ecmp would not be a better option.

2.
Likewise how good is my accumulated link delay value if in between there are deep buffer network elements and say egress queuing to each link (which max is unaccounted for in your draft) can significantly alter the end to end delay ? Have you consider to add MAX_EGRESS_QUEUE_DELAY on a per link basis (still as static value).  So if some traffic is delay sensitive we will have a much better accuracy not to get into a trap of queuing related delays.

Thx a lot,
Robert.

On Fri, Feb 26, 2021 at 8:37 AM William Britto A J <bwilliam=[email-masked]<mailto:[email-masked]>> wrote:
All,

We would like to draw your attention to a new ID: https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKV2Tba5Qg$>

The draft talks about introducing link bandwidth related constraints in Flex-Algorithm which can be used to define a Flex-Algorithm based on bandwidth based metric.

Please review. Any questions and comments are welcome.

Thanks,
William

From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Date: Monday, 22 February 2021 at 10:56 PM
To: Bruno Decraene <[email-masked]<mailto:[email-masked]>>, Rajesh M <[email-masked]<mailto:[email-masked]>>, Rajesh M <[email-masked]<mailto:[email-masked]>>, Shraddha Hegde <[email-masked]<mailto:[email-masked]>>, William Britto A J <[email-masked]<mailto:[email-masked]>>, William Britto A J <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-hegde-lsr-flex-algo-bw-con-00.txt
[External Email. Be cautious of content]

A new version of I-D, draft-hegde-lsr-flex-algo-bw-con-00.txt
has been successfully submitted by Shraddha Hegde and posted to the
IETF repository.

Name:           draft-hegde-lsr-flex-algo-bw-con
Revision:       00
Title:          Flexible Algorithms Bandwidth Constraints
Document date:  2021-02-22
Group:          Individual Submission
Pages:          21
URL:            https://urldefense.com/v3/__https://www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$>
Status:         https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$>
Htmlized:       https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$>
Htmlized:       https://urldefense.com/v3/__https://tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$<https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$>

Abstract:
   Many networks configure the link metric relative to the link
   capacity.  High bandwidth traffic gets routed as per the link
   capacity.  Flexible algorithms provides mechanisms to create
   constraint based paths in IGP.  This draft documents a set of
   bandwidth related constraints to be used in Flexible Algorithms.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<https://urldefense.com/v3/__http:/tools.ietf.org__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKXnuJRcew$>.

The IETF Secretariat

Juniper Business Use Only
_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKV0XHjSNQ$>

Juniper Business Use Only

---

From: William Britto A J <[email-masked]>
Date: Mon, 1 Mar 2021 10:24:00 +0000

Hi Gyan,

This draft aims to provide the protocol constructs to define a flex-algorithm which is suitable for sending high bandwidth traffic. Flex-Algo is a very useful feature for network consolidation use-cases which requires different metric-types for SPF. We are trying introduce the protocol constructs to simplify the use of metric based on bandwidth via Flex-Algo.

This draft does not attempt to do bandwidth management nor reservation like what RSVP does. For LDP based networks that use igp metric relative to bandwidth, Flex-Algo provides an easy alternate.

Thanks,
William

From: Gyan Mishra <[email-masked]>
Date: Saturday, 27 February 2021 at 9:40 PM
To: Robert Raszuk <[email-masked]>
Cc: DECRAENE Bruno IMT/OLN <[email-masked]>, Rajesh M <[email-masked]>, Shraddha Hegde <[email-masked]>, William Britto A J <[email-masked]>, [email-masked] <[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
[External Email. Be cautious of content]

Hi William & Co-authors

>From first read of the draft it does appear your are trying to apply RSVP TE PCALC path and reserve message link attributes constraints such as concept of affinity bits to exclude low bandwidth or delay of individual links without taking into account all of what RSVP TE is reserving of bandwidth in the end to end path with the Path and Reserve message.  As mentioned Looking at individual links will not provide the end to end path view or bandwidth requirements for the entire path to be reserved as accomplished by RSVP TE.

As Tony and Robert have mentioned I agree this is a good first step but does need more refinement to make useful.

Kind Regards

Gyan

On Sat, Feb 27, 2021 at 7:24 AM Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
Hi William & co-authors,

1.

2.

Thx a lot,
Robert.

We would like to draw your attention to a new ID: https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8sz2YPxkw$>

Thanks,
William

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<https://urldefense.com/v3/__http:/tools.ietf.org__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vQC8uTvg$>.

The IETF Secretariat

Juniper Business Use Only
_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vByLZnBg$>

Gyan Mishra

Network Solutions Architect

M 301 502-1347
13101 Columbia Pike
Silver Spring, MD

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 1 Mar 2021 11:48:02 +0100

Hi Wiliam,

I get yr point on bw. I was not saying to make any changes in the draft on
this. I was more hinting that perhaps deployment scenarios could better
articulate pros and cons of use of such static bw parameter.

Regarding the delay - Oh so the delay is a dynamic variable here ? I was
under the impression that this would be a static cfg based value.

Can you clarify ?

Thx,
R.

On Mon, Mar 1, 2021 at 11:16 AM William Britto A J <[email-masked]>
wrote:

> Currently there are customers who deploy separate networks, of which one
> could be assigned metrics relative to the interface bandwidths, while other
> could be based on other parameters like latency, etc. Flex-Algo which
> facilitates different metric-types for SPF, is a very useful feature for
> such network consolidation use-cases. We are trying introduce the protocol
> constructs to simplify the use of metric based on bandwidth via Flex-Algo.
>
>
>
> Any existing tools and techniques used by operators who configure metric
> relative to link bandwidth to optimize the network can be used with these
> Flex-Algo constructs too. The Flex-Algo bandwidth constraints defined here
> can be used to automatically derive a metric based on reference bw vs link
> bw; and if required, this can be overridden using the individual link
> bandwidth-metric sub-TLV. We will make this clear in the next version.
>
>
>
> RFC 8570 support advertising link delay parameters in ISIS. Protocols like
> TWAMP support dynamically measuring the delay. Whether egress queueing
> delay is included in the link delay depends on the measuring mechanism. The
> Exclude Max Delay sub-TLV introduced in this draft is only meant for
> pruning out high latency links from a Flex-Algo, and it is upto the
> operator to define the maximum bounds based on the measuring mechanisms
> deployed in his network.
>
>
>
> Thanks,
>
> William
>
>
>
> *From: *Robert Raszuk <[email-masked]>
> *Date: *Saturday, 27 February 2021 at 5:54 PM
> *To: *William Britto A J <[email-masked]>
> *Cc: [email-masked] <[email-masked]>, Rajesh M <[email-masked]>,
> Shraddha Hegde <[email-masked]>, DECRAENE Bruno IMT/OLN <
> [email-masked]>
> *Subject: *Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
>
> *[External Email. Be cautious of content]*
>
>
>
> Hi William & co-authors,
>
>
>
> Both bw & delay can be used as defined in the draft to construct new
> forwarding topologies. But how practical such topologies would be in the
> real life when 40GB links may be heavily occupied with bursty traffic and
> 10G links can sit idle ? I suppose you are trying to address the case where
> say 12 gbps holographic stream needs to be sent across a network.. But then
> I don't think if sending it in a single flow instead of spreading into many
> sub-flows and use as much as possible ecmp would not be a better option.
>
>
>
> 2.
>
> Likewise how good is my accumulated link delay value if in between there
> are deep buffer network elements and say egress queuing to each link (which
> max is unaccounted for in your draft) can significantly alter the end to
> end delay ? Have you consider to add MAX_EGRESS_QUEUE_DELAY on a per link
> basis (still as static value).  So if some traffic is delay sensitive we
> will have a much better accuracy not to get into a trap of queuing related
> delays.
>
>
>
> Thx a lot,
>
> Robert.
>
>
>
>
>
> On Fri, Feb 26, 2021 at 8:37 AM William Britto A J <bwilliam=
> [email-masked]> wrote:
>
> All,
>
>
>
> We would like to draw your attention to a new ID:
> https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKV2Tba5Qg$>
>
>
>
> The draft talks about introducing link bandwidth related constraints in
> Flex-Algorithm which can be used to define a Flex-Algorithm based on
> bandwidth based metric.
>
>
>
> *From: [email-masked] <[email-masked]>
> *Date: *Monday, 22 February 2021 at 10:56 PM
> *To: *Bruno Decraene <[email-masked]>, Rajesh M <
> [email-masked]>, Rajesh M <[email-masked]>, Shraddha Hegde <
> [email-masked]>, William Britto A J <[email-masked]>, William
> Britto A J <[email-masked]>
> *Subject: *New Version Notification for
> draft-hegde-lsr-flex-algo-bw-con-00.txt
>
> URL:
> https://urldefense.com/v3/__https://www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$
> <https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$>
> Status:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$>
> Htmlized:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$>
> Htmlized:
> https://urldefense.com/v3/__https://tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$
> <https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$>
>
>
> Abstract:
> until the htmlized version and diff are available at tools.ietf.org
> <https://urldefense.com/v3/__http:/tools.ietf.org__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKXnuJRcew$>
> .
>
> The IETF Secretariat
>
>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lsr
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!WgwtY1NpSXZkAtkBfmMRfgZABaIBWv534VYF7vvI4eGGuMbmdKMK_VpWFKV0XHjSNQ$>
>
>

---

From: Tony Przygienda <[email-masked]>
Date: Mon, 1 Mar 2021 17:58:27 +0100

On Sat, Feb 27, 2021 at 2:56 AM Tony Li <[email-masked]> wrote:

>
> Hi William & co-authors,
>
> Thank you for your contribution. It’s definitely interesting. As bandwidth
> management is one area where FlexAlgo lags legacy traffic engineering, this
> is definitely one small step in the right direction.
>
> But I have many comments…
>
> 1) I’ll start with the title: there is MUCH more here than bandwidth
> constraints. Perhaps this should be generalized somehow? Sorry, I don’t
> have a constructive suggestion.
>

there is good general, theoretical body of work here that shows what is
possible to compute (NP completeness, algorighms, heuristics). short search
yields e.g.

http://www.csun.edu/~ctoth/Handbook/chap31.pdf
https://engineering.purdue.edu/~ece624/Week-9.pdf

JSAC '96 Crowcroft is a starting point on additive/multiplicative, other
papers even eqarlier called that stuff max/additive, convex/concave (that
has to do with linear programming space properties) etc. I'm aware of work
going as far back as '92-'93 here.

Generally, one max/min & one additive is doable (you need additive to make
sure you don't start hamiltonian walks based on a constraint ;-)  Mixing up
addvitive & mix/min is a fairly poor idea unless one is happy with
heurestics that kind of "almost always work" and customer is not looking
for hard guarantees but "wishes that maybe granted".

>
>
> 2) Section 2: I concur that bandwidth is an important consideration in
> path computation. At a first reading, it is not at all clear to me that it
> is optimal to somehow transmogrify things into a metric. Why not simply
> deal with the bandwidth directly?  I did (eventually) realize that you did
> this because you want SPF to somehow select the path with the minimum
> product of (# links) * bandwidth.  I think you need to be explicit about
> this motivation and also mention that traditional SPF is not set up to deal
> with a floating point metric. Note that a bandwidth metric is STILL
> somewhat counter-intuitive to me, as I would expect that you’d mostly want
> to route things along the highest bandwidth paths, not the lowest ones.
> More motivation behind the bandwidth metric would be most welcome.
>

yeah, BW can be made a metric and work, IEEE encoding is simply a funky
number and different max/min/additive/multiplicative/any increasing
function (whether it has to be stricly monotonic is a more complex
discussion).  old but very solid stuff for that space

https://tools.ietf.org/html/rfc2676

based on even earlier

   [BP95]   J.-Y. Le Boudec and T. Przygienda.  A Route Pre-Computation
            Algorithm for Integrated Services Networks.  Journal of
            Network and Systems Management, 3(4), 1995.
   [GOW97]  R. Guerin, A. Orda, and D. Williams.  QoS Routing Mechanisms
            and OSPF Extensions.  In Proceedings of the 2nd IEEE Global
            Internet Mini-Conference, pages 1903-1908, Phoenix, AZ,
            November 1997.

expired draft-ietf-qosr-framework-04.txt

 but given graph theory doesn't change things roughly still the same ;-)

There is also good body of work on something that quickly interacts with
that work, namely k-disjointness. e'thing looking for "fattest pipe" or
"shortest-widest" pretty quickly crams all traffic on the same links ;-)
unless somehting like RSVP-TE reserves & changes metric.

All those things have been implemented and worked in products so we know
they work, maybe flex stuff is just a framework now to unify it all a bit.

my only small nit on the draft is that the stepping is in practical terms
almost always exponewntial so you don't have to encode full value
necessarily. the rounding has to adjust to link size as well or is better
served as % value.

-- tony

---

From: Tony Li <[email-masked]>
Date: Mon, 1 Mar 2021 10:42:30 -0800

Hi William, Gyan, Robert, Tony, et. al.,

Please permit me to wax a bit philosophic here.

William is exactly correct that the goal is not to make FlexAlgo deal with reservations like RSVP does.  Without some kind of setup protocol or global computation, that’s simply not possible. Moreover that’s not the real problem that we’re out to solve.

Reservations are just a first order approximation to a traffic flow in any case. We characterize them as having a fixed constant bandwidth, but we all know that that is far from the truth. Each flow is diurnal and fractally bursty. Every user who ever clicks on a link creates bandwidth demand and while the Law of Large Numbers helps us out with some averages we all know that we have no good way of characterizing the traffic that we’re trying to carry. Claiming that it is a single 12Gbps LSP is truly a huge over simplification and a good step towards solving the real problem.

So what is the real problem that we’re trying to solve?

We are trying to not drop packets.

Dropping packets is bad because it forces retransmissions and impacts someone’s Internet experience. Dropping packets is our response to congestion events. If we could manage to have a network that never congested and always delivered packets without giant latency, all would be good.

To date, we have created traffic engineering mechanisms to help us steer traffic so that we could delivery traffic without congesting. It has been a means to an end. The mechanisms that we’ve created have a non-trivial overhead and approximate our goals, but they do NOT preclude, anticipate, or avoid congestion. They do not react when we have unexpected inputs. We do extensive pre-computation to deal with even single failures and have no serious mechanisms that handle multiple failures.

Right now, FlexAlgo does nothing to help with bandwidth management. Wiilliam et. al., are proposing some first steps, which are to be encouraged. Much more will be needed, not to recreate legacy mechanisms but because we should be striving for a more sophisticated, real-time approach to bandwidth management and congestion avoidance.

Regards,
Tony

> On Mar 1, 2021, at 2:24 AM, William Britto A J <bwilliam=[email-masked]> wrote:
> 
> Hi Gyan, 
>  
> From: Gyan Mishra <[email-masked] <mailto:[email-masked]>>
> To: Robert Raszuk <[email-masked] <mailto:[email-masked]>>
> Cc: DECRAENE Bruno IMT/OLN <[email-masked] <mailto:[email-masked]>>, Rajesh M <[email-masked] <mailto:[email-masked]>>, Shraddha Hegde <[email-masked] <mailto:[email-masked]>>, William Britto A J <[email-masked] <mailto:[email-masked]>>, [email-masked] <mailto:[email-masked]> <[email-masked] <mailto:[email-masked]>>
> On Sat, Feb 27, 2021 at 7:24 AM Robert Raszuk <[email-masked] <mailto:[email-masked]>> wrote:
> Hi William & co-authors, 
>  
> On Fri, Feb 26, 2021 at 8:37 AM William Britto A J <bwilliam=[email-masked] <mailto:[email-masked]>> wrote:
> All,
>  
> We would like to draw your attention to a new ID: https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00 <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8sz2YPxkw$>
>  
> From: [email-masked] <mailto:[email-masked]> <[email-masked] <mailto:[email-masked]>>
> To: Bruno Decraene <[email-masked] <mailto:[email-masked]>>, Rajesh M <[email-masked] <mailto:[email-masked]>>, Rajesh M <[email-masked] <mailto:[email-masked]>>, Shraddha Hegde <[email-masked] <mailto:[email-masked]>>, William Britto A J <[email-masked] <mailto:[email-masked]>>, William Britto A J <[email-masked] <mailto:[email-masked]>>
> URL:            https://urldefense.com/v3/__https://www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$ <https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-hegde-lsr-flex-algo-bw-con-00.txt__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3v6TruoA$>
> Status:         https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$ <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-hegde-lsr-flex-algo-bw-con/__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD1VexjHPQ$>
> Htmlized:       https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$ <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD3X5nPQbA$>
> Htmlized:       https://urldefense.com/v3/__https://tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$ <https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!QGg36p91zPfVMznYY91xs-zx70Qp5BE1nJx-Thnl14sTCkvwgOjEzjGBtD2aqSYcuQ$>
> 
> 
> Abstract:
> until the htmlized version and diff are available at tools.ietf.org <https://urldefense.com/v3/__http:/tools.ietf.org__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vQC8uTvg$>.
> 
> The IETF Secretariat
> 
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/lsr <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/lsr <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
> -- 
>  <https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vByLZnBg$>
> Gyan Mishra
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/lsr <https://www.ietf.org/mailman/listinfo/lsr>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 1 Mar 2021 21:22:28 +0100

Hi Tony,

Constructing arbitrary topologies with bw constrain is useful work. For
example I want to create a topology without links of the capacity less then
1 Gbps. All cool. Of course if I have a case where two nodes have 10 L3
1Gbps links nicely doing ECMP I will not include those which may be a
problem.

However my observation is precisely related to your last sentence.

Is this extension to be used with static or dynamic data ? If static all
fine. But as William replied to me earlier link delay may be dynamically
computed and may include queue wait time. That to me means something much
different if Flex-Algo topologies will become dynamically adjustable. And I
am not saying this is not great idea .. My interest here is just to
understand the current scope.

Thx,
R.

On Mon, Mar 1, 2021 at 7:42 PM Tony Li <[email-masked]> wrote:

> William is exactly correct that the goal is not to make FlexAlgo deal with
> reservations like RSVP does.  Without some kind of setup protocol or global
> computation, that’s simply not possible. Moreover that’s not the real
> problem that we’re out to solve.
>
> Reservations are just a first order approximation to a traffic flow in any
> case. We characterize them as having a fixed constant bandwidth, but we all
> know that that is far from the truth. Each flow is diurnal and fractally
> bursty. Every user who ever clicks on a link creates bandwidth demand and
> while the Law of Large Numbers helps us out with some averages we all know
> that we have no good way of characterizing the traffic that we’re trying to
> carry. Claiming that it is a single 12Gbps LSP is truly a huge over
> simplification and a good step towards solving the real problem.
>
> Dropping packets is bad because it forces retransmissions and impacts
> someone’s Internet experience. Dropping packets is our response to
> congestion events. If we could manage to have a network that never
> congested and always delivered packets without giant latency, all would be
> good.
>
> To date, we have created traffic engineering mechanisms to help us steer
> traffic so that we could delivery traffic without congesting. It has been a
> means to an end. The mechanisms that we’ve created have a non-trivial
> overhead and approximate our goals, but they do NOT preclude, anticipate,
> or avoid congestion. They do not react when we have unexpected inputs. We
> do extensive pre-computation to deal with even single failures and have no
> serious mechanisms that handle multiple failures.
>
> Right now, FlexAlgo does nothing to help with bandwidth management.
> Wiilliam et. al., are proposing some first steps, which are to be
> encouraged. Much more will be needed, not to recreate legacy mechanisms but
> because we should be striving for a more sophisticated, real-time approach
> to bandwidth management and congestion avoidance.
>
> Regards,
> Tony
>
>
> On Mar 1, 2021, at 2:24 AM, William Britto A J <
> bwilliam=[email-masked]> wrote:
>
> Hi Gyan,
>
> This draft aims to provide the protocol constructs to define a
> flex-algorithm which is suitable for sending high bandwidth traffic. Flex-Algo
> is a very useful feature for network consolidation use-cases which requires
> different metric-types for SPF. We are trying introduce the protocol
> This draft does not attempt to do bandwidth management nor reservation
> like what RSVP does. For LDP based networks that use igp metric relative to
> bandwidth, Flex-Algo provides an easy alternate.
>
> Thanks,
> William
>
>
> *From: *Gyan Mishra <[email-masked]>
> *Date: *Saturday, 27 February 2021 at 9:40 PM
> *To: *Robert Raszuk <[email-masked]>
> *Cc: *DECRAENE Bruno IMT/OLN <[email-masked]>, Rajesh M <
> [email-masked]>, Shraddha Hegde <[email-masked]>, William
> Britto A J <[email-masked]>, [email-masked] <[email-masked]>
> From first read of the draft it does appear your are trying to apply RSVP
> TE PCALC path and reserve message link attributes constraints such as
> concept of affinity bits to exclude low bandwidth or delay of individual
> links without taking into account all of what RSVP TE is reserving of
> bandwidth in the end to end path with the Path and Reserve message.  As
> mentioned Looking at individual links will not provide the end to end path
> view or bandwidth requirements for the entire path to be reserved as
> accomplished by RSVP TE.
>
> As Tony and Robert have mentioned I agree this is a good first step but
> does need more refinement to make useful.
>
> Kind Regards
>
> Gyan
>
> On Sat, Feb 27, 2021 at 7:24 AM Robert Raszuk <[email-masked]> wrote:
>
> Hi William & co-authors,
>
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-00__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8sz2YPxkw$>
>
>
> <https://urldefense.com/v3/__http:/tools.ietf.org__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vQC8uTvg$>
> .
>
> The IETF Secretariat
>
>
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
>
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8umPEf2Zg$>
>
> --
>
>
> *Gyan Mishra*
> *Network Solutions Architect *
>
>
> *M 301 502-134713101 Columbia Pike *Silver Spring, MD
>
>

---

From: Tony Przygienda <[email-masked]>
Date: Mon, 1 Mar 2021 21:44:54 +0100

dyanmic queue lengths are still poor indicators of delay (in routing
network wide sense @ realistic routing flood/processing resolution),
nothing changed much since 1980 AFAIK

https://www.cs.yale.edu/homes/lans/readings/routing/mcquillan-darpa_routing-1980.pdf

delay/jitter can be talked about meaningfully if the context of the
topologies involved is controlled, pretty much stuff Lou's DETNET is
chewing

my 2c

-- tony

On Mon, Mar 1, 2021 at 9:22 PM Robert Raszuk <[email-masked]> wrote:

>> William is exactly correct that the goal is not to make FlexAlgo deal
>> with reservations like RSVP does.  Without some kind of setup protocol or
>> global computation, that’s simply not possible. Moreover that’s not the
>> real problem that we’re out to solve.
>>
>> Reservations are just a first order approximation to a traffic flow in
>> any case. We characterize them as having a fixed constant bandwidth, but we
>> all know that that is far from the truth. Each flow is diurnal and
>> fractally bursty. Every user who ever clicks on a link creates bandwidth
>> demand and while the Law of Large Numbers helps us out with some averages
>> we all know that we have no good way of characterizing the traffic that
>> we’re trying to carry. Claiming that it is a single 12Gbps LSP is truly a
>> huge over simplification and a good step towards solving the real problem.
>>
>> [email-masked]>, William Britto A J <[email-masked]>,
>> William Britto A J <[email-masked]>

---

From: Tony Li <[email-masked]>
Date: Mon, 1 Mar 2021 12:47:52 -0800

Robert,

> Constructing arbitrary topologies with bw constrain is useful work. For example I want to create a topology without links of the capacity less then 1 Gbps. All cool. Of course if I have a case where two nodes have 10 L3 1Gbps links nicely doing ECMP I will not include those which may be a problem. 

I agree that it may be a problem. Maybe it’s not the right tool for the job at hand. That doesn’t make it a bad tool, just the wrong one. I try not to turn screws with a hammer. And I try not to drive nails with a screwdriver.

I will happily stipulate that we need more tools and that these are not enough.  We should not reject a tool simply because it doesn’t solve all problems. Let’s work towards the right set of tools. Linear algebra tells us that we want an orthogonal set of basis vectors. What are they? Adding them one at a time is not horrible progress.

> Is this extension to be used with static or dynamic data ? If static all fine. But as William replied to me earlier link delay may be dynamically computed and may include queue wait time. That to me means something much different if Flex-Algo topologies will become dynamically adjustable. And I am not saying this is not great idea .. My interest here is just to understand the current scope. 

Link delay was dynamic before this draft.  As William mentioned, TWAMP can already be used to provide a dynamic measurement of link delay.  That, coupled with the link delay metric already gave us dynamic path computation requirements and the possibilities of oscillation and instability. We have chosen to charge ahead, without addressing those concerns already.

Regards,
Tony

---

From: Jeff Tantsura <[email-masked]>
Date: Mon, 1 Mar 2021 13:45:53 -0800

In ol’ good RSVP-TE days we already used “severity/relevance indicator” to decide whether changes in link  attributes (BW/etc) are significant enough and should be propagated in into TED and trigger re-optimization/rerouting, this is no different,  define your threshold for a trigger.
Note - flex-also requires contiguous topology to work, self isolation as the result of (dynamic) topology re-computation would not be a great thing.

Cheers,
Jeff

---

From: "Dongjie (Jimmy)" <[email-masked]>
Date: Tue, 2 Mar 2021 01:48:35 +0000

Hi Robert,

I have similar question as yours: whether the proposed mechanism is based on static or dynamic bandwidth/latency metric?

If static, it is easy for Flex-Algo based distributed computation, while the result may not be that helpful, as Tony said, all traffic may be steered to the same link.

If dynamic, the changes in the metric will result in dynamic computation with Flex-Algo constraints, thus more need to be considered: measurement and advertisement of dynamic metrics, overhead in dynamic computation, loop prevention during dynamic computation and convergence, the possibility of an unacceptable result… And with all this overhead in mind, it is better to understand what would be the gains, can it help to reduce the congestion or packet drop? Or what possible problem it is targeted to solve?

Best regards,
Jie

From: Lsr [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Tuesday, March 2, 2021 4:22 AM
To: Tony Li <[email-masked]>
Cc: Gyan Mishra <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>; Shraddha Hegde <[email-masked]>; Rajesh M <[email-masked]>; [email-masked]; William Britto A J <bwilliam=[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Hi Tony,

Thx,
R.

On Mon, Mar 1, 2021 at 7:42 PM Tony Li <[email-masked]<mailto:[email-masked]>> wrote:

Regards,
Tony

On Mar 1, 2021, at 2:24 AM, William Britto A J <bwilliam=[email-masked]<mailto:bwilliam=[email-masked]>> wrote:

Hi Gyan,

Thanks,
William

From: Gyan Mishra <[email-masked]<mailto:[email-masked]>>
Date: Saturday, 27 February 2021 at 9:40 PM
To: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: DECRAENE Bruno IMT/OLN <[email-masked]<mailto:[email-masked]>>, Rajesh M <[email-masked]<mailto:[email-masked]>>, Shraddha Hegde <[email-masked]<mailto:[email-masked]>>, William Britto A J <[email-masked]<mailto:[email-masked]>>, [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
[External Email. Be cautious of content]

Hi William & Co-authors

Kind Regards

Gyan

1.

2.

Thx a lot,
Robert.

Thanks,
William

Abstract:

The IETF Secretariat

[图像已被发件人删除。]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!U3HBQwaZL7T4mjz-MUHE7VjFtnOyNghGDh5vwJXUXkHdMxlravmRvJwdZ8vByLZnBg$>
Gyan Mishra
Network Solutions Architect
M 301 502-1347
13101 Columbia Pike
Silver Spring, MD

---

From: Tony Przygienda <[email-masked]>
Date: Tue, 2 Mar 2021 07:46:50 +0100

the pesky operational issue of those computations to suddenly partition the
graph (if used with dynamic resource updates) or otherwise pile everything
on the same link that led to RFC5120 being built the way it is. It is one
thing to get from RSVP-TE a "no resources to get there" indication and
another for IGP to magically not be getting there anymore while in terms of
best-effort functionality the destination looks perky and happy. One could
argue that you just need a routing table to see whether it's there but the
nature of those algorithms is that people show up with "I want 100MB" and
the answer there is very different from another request for "I want 1GB".
In Fore systems we basically had bunch of pre-defined/configurable profiles
we were pre-computing the answers (obviously also for performance purposes)
and comibned with the Q.2931 doing roughly what RSVP-TE is doing  +
crankback to do low catches stuff was working pretty well. There is a
patent for that still somewhere AFAIR.

-- tony

On Mon, Mar 1, 2021 at 10:48 PM Jeff Tantsura <[email-masked]>
wrote:

> In ol’ good RSVP-TE days we already used “severity/relevance indicator” to
> decide whether changes in link  attributes (BW/etc) are significant enough
> and should be propagated in into TED and trigger re-optimization/rerouting,
> this is no different,  define your threshold for a trigger.
> Note - flex-also requires contiguous topology to work, self isolation as
> the result of (dynamic) topology re-computation would not be a great thing.
>
> Cheers,
> Jeff
> I agree that it may be a problem. Maybe it’s not the right tool for the
> job at hand. That doesn’t make it a bad tool, just the wrong one. I try not
> to turn screws with a hammer. And I try not to drive nails with a
> screwdriver.
>
> I will happily stipulate that we need more tools and that these are not
> enough. We should not reject a tool simply because it doesn’t solve all
> problems. Let’s work towards the right set of tools. Linear algebra tells
> us that we want an orthogonal set of basis vectors. What are they? Adding
> them one at a time is not horrible progress.
>
>
> Link delay was dynamic before this draft. As William mentioned, TWAMP can
> already be used to provide a dynamic measurement of link delay. That,
> coupled with the link delay metric already gave us dynamic path computation
> requirements and the possibilities of oscillation and instability. We have
> chosen to charge ahead, without addressing those concerns already.
>
> Regards,
> Tony
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 2 Mar 2021 08:13:45 +0100

Hey Tony,

I think there are few things to observe here.

1.
It very much depends how one is to use Flex-Algo topologies. If you attempt
to use it as solid fixed topology for some applications I would be very
cautious not to create such topology with dynamic constraints. Not only
worrying about sudden disconnected or partitioned areas, but also that
micro loops during topology changes may not be so micro anymore. It would
be rather rear that all network elements compute topology and activate it
in the same time.

And that is quite different from local protocol convergence when typical
microloop can and do happen all the time, but rest of the topology is
unchanged.

2.
If you however are using such tools as optimizations and you can always
fall back to base topology the risks are much lower. I think there is room
for at least testing it in real scale.

3.
Of course to get #2 going we would need not only control plane topology
creation but also build in data plane ingress to egress live topology
validation - before you shift some traffic on it or remove from it
to fallback to base.

So back to the subject - we still do not know if the current proposal
assumes static bw & static delay or dynamic delay to take into
consideration.

Many thx,
R.

On Tue, Mar 2, 2021 at 7:47 AM Tony Przygienda <[email-masked]> wrote:

> the pesky operational issue of those computations to suddenly partition
> the graph (if used with dynamic resource updates) or otherwise pile
> everything on the same link that led to RFC5120 being built the way it is.
> It is one thing to get from RSVP-TE a "no resources to get there"
> indication and another for IGP to magically not be getting there anymore
> while in terms of best-effort functionality the destination looks perky and
> happy. One could argue that you just need a routing table to see whether
> it's there but the nature of those algorithms is that people show up with
> "I want 100MB" and the answer there is very different from another request
> for "I want 1GB". In Fore systems we basically had bunch of
> pre-defined/configurable profiles we were pre-computing the answers
> (obviously also for performance purposes) and comibned with the Q.2931
> doing roughly what RSVP-TE is doing  + crankback to do low catches stuff
> was working pretty well. There is a patent for that still somewhere AFAIR.
>
> -- tony
>
>> In ol’ good RSVP-TE days we already used “severity/relevance indicator”
>> to decide whether changes in link  attributes (BW/etc) are significant
>> enough and should be propagated in into TED and trigger
>> re-optimization/rerouting, this is no different,  define your threshold for
>> a trigger.

---

From: Tony Li <[email-masked]>
Date: Tue, 2 Mar 2021 15:59:40 -0800

Hi William,

> 1) I’ll start with the title: there is MUCH more here than bandwidth constraints. Perhaps this should be generalized somehow? Sorry, I don’t have a constructive suggestion.
> [William] Agree. We are introducing few ‘definitions’ for a bandwidth based Flex-Algo, along with couple of link constraints which can be used for any type of flex-algo. Any suggestions are welcome.

In the hopes of inspiring others with better thoughts:

“Flexible Algorithms: Bandwidth Management Part 1”
“Flexible Algorithms: Some Stuff We Forgot”
“Flexible Algorithms: Bandwidth and Delay, Metrics and Constraints”
“Flexible Algorithms: A Few More Bits”

> 4) Sectiion 2.1: Why are there two octets reserved in this sub-TLV? If you’re hoping for alignment within an IS-IS LSP, you’ve already lost. Nothing in IS-IS is aligned.
> [William] Apologies. This was an error in the figure. Intention was one octet for Reserved/future flags, hence the Length of 5 octets. Will correct this in the next version.

And if you agree with a more general metric, maybe a few of those reserved bits could be used as the index for the user defined metric.

> 5) Section 3.1.1: Here you have chosen to refer directly to subTLV 9.  To be consistent with the rest of FlexAlgo, should we use subTLV 9 inside of the Application Specific Link Attribute subTLV.
> [William] Shouldn’t there be only one Max link BW per link applicable for all applications? However, agree that it could still be in a common ASLA. Will add relevant text to clarify this.

That argument might well apply to all of our other link attributes as well.  Why do we need a separate administrative group for FlexAlgo?  Why do we need a separate set of SRLGs?  I have to confess that we have added a bit of complexity that I don’t think is very valuable. However, what’s done is done and there’s no point in arguing about it now.  We should simply strive to be consistent with what’s already in place so that we don’t create even MORE complexity.

> 6) Section 3.1.2: I’m unclear on the utility of this. I can understand optimizing for path delay against the path metric. I can even understand putting an upper bound on the total path delay. I don’t understand why a bound on an individual link delay is important.  If my path delay budget is 100ms, then does it matter if it is exceeded in one hop or ten? Could you please provide more motivation here? Also, wouldn’t a FlexAlgo system be advertising the link delay in the Application Specific Link Attributes subTLV?
> [William] This constraint could be useful in a Flex-Algo whose Metric-type is anything but Link-delay. Here, Link-delay doesn’t influence the ‘cost’ of a link in that Flex-Algo, but can be used to prune out certain links with very high delay from that Flex-Algo. 

Thank you, that’s helpful motivation.  Please add something like this to the draft.

Continuing on from there…

7) Section 4. You’ve hidden a big part of what you want to do down here, 9 pages into the document.  You’ve got one whole sentence in the introduction to motivate it.  It deserves a good bit more.

8) Section 4.2. You write that an implementation "considers cumulative bandwidth of the parallel links while arriving at the metric for the link”. This seems a bit vague.  I think you’re trying to say that in interface group mode the bandwidth of an adjacency is the sum of the bandwidths of the individual links.  Typically today, if we have L3 parallel links they are encoded as separate adjacencies, complete with unique interface addresses.  How does interface group mode work with that? Does each adjacency advertise the same metric based on the total bandwidth of all of the links?  The discussion about reference bandwidths and the staircase mapping seems a bit confusing and redundant.  It appears to be orthogonal to the mode, correct? Why not describe it independently? More proofreading is necessary here.

9) Section 4.3.1. The reserved byte is wasteful. Your explanation of the round-off bandwidth is unclear.  Are we expecting dynamic bandwidth changes?  If so, that is an entire discussion unto itself.

10) Section 4.3.1. The reserved byte here is also wasteful.  Your explanation of how to use this is a bit unclear.  Are bandwidth ranges allowed to overlap?  I hope not. Is max[1] == min[2]?  I hope so, otherwise there is a gap in the ranges.  If there is a gap, and a link bandwidth falls in the gap, then what do we do?  Assuming that max[i] == min[i+1], then is there a point in carrying both values all of the time? What happens when a bandwidth exceeeds the highest max? Is below the smallest min?

Regards,
Tony

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 10:34:06 +0100

Hi Tony,

> Link delay was dynamic before this draft.  As William mentioned, TWAMP can already be used to provide a dynamic measurement of link delay.  That, coupled with the link delay metric already gave us…

TWAMP provided Min Unidirectional Link Delay is a dynamic one. On the 
other side this value is calculated based on multiple measurements over 
period of time and an average is used. Also, smart implementations can 
normalize the value so that a small fluctuation of the delay is not 
causing the traffic to shift or cause ECMP loss.

What is important here is that the Min Unidirectional Link Delay is a 
link characteristic, not something that is affected by the amount of 
traffic on the link or subject to queuing delay. Same applies to Maximum 
link bandwidth.

thanks,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 10:58:28 +0100

Hi Peter,

To your last point ...

Authors stated: "Whether egress queueing delay is included in the link
delay depends on the measuring mechanism."

So sure there will be thresholds etc ... but this may very well depend on
the traffic.

Thx,
R.

On Wed, Mar 3, 2021 at 10:34 AM Peter Psenak <[email-masked]> wrote:

> >> Is this extension to be used with static or dynamic data ? If static
> all fine. But as William replied to me earlier link delay may be
> dynamically computed and may include queue wait time. That to me means
> something much different if Flex-Algo topologies will become dynamically
> adjustable. And I am not saying this is not great idea .. My interest here
> is just to understand the current scope.
> >
> >
> > Link delay was dynamic before this draft.  As William mentioned, TWAMP
> can already be used to provide a dynamic measurement of link delay.  That,

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 11:06:03 +0100

Robert,

> delay depends on the measuring mechanism."

I disagree with that statement - the Min Unidirectional Link Delay is 
the value that does not include the queueing delay - that's why it is 
called Min.

> 
> So sure there will be thresholds etc ... but this may very well depend 
> on the traffic.

no.

thanks,
Peter

> 
> Thx,
> R.
> 
> 
> 
> On Wed, Mar 3, 2021 at 10:34 AM Peter Psenak <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Tony,
> 
>      >> Constructing arbitrary topologies with bw constrain is useful
>     work. For example I want to create a topology without links of the
>     capacity less then 1 Gbps. All cool. Of course if I have a case
>     where two nodes have 10 L3 1Gbps links nicely doing ECMP I will not
>     include those which may be a problem.
>      >
>      >
>      > I agree that it may be a problem. Maybe it’s not the right tool
>     for the job at hand. That doesn’t make it a bad tool, just the wrong
>     one. I try not to turn screws with a hammer. And I try not to drive
>     nails with a screwdriver.
>      >
>      > I will happily stipulate that we need more tools and that these
>     are not enough.  We should not reject a tool simply because it
>     doesn’t solve all problems. Let’s work towards the right set of
>     tools. Linear algebra tells us that we want an orthogonal set of
>     basis vectors. What are they? Adding them one at a time is not
>     horrible progress.
>      >
>      >
>      >> Is this extension to be used with static or dynamic data ? If
>     static all fine. But as William replied to me earlier link delay may
>     be dynamically computed and may include queue wait time. That to me
>     means something much different if Flex-Algo topologies will become
>     dynamically adjustable. And I am not saying this is not great idea
>     .. My interest here is just to understand the current scope.
>      >
>      >
>      > Link delay was dynamic before this draft.  As William mentioned,
>     TWAMP can already be used to provide a dynamic measurement of link
>     delay.  That, coupled with the link delay metric already gave us
>     dynamic path computation requirements and the possibilities of
>     oscillation and instability. We have chosen to charge ahead, without
>     addressing those concerns already.
>

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 11:10:11 +0100

Hey Peter,

> the value that does not include the queueing delay - that's why it is

But draft we are discussing here does not talk about "Min" delay. Contrary
it talks about "Max"

*Maximum* Delay sub-TLV

That is also I asked that very question up front.

Thx,
R.

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 11:22:01 +0100

Robert,

On 03/03/2021 11:10, Robert Raszuk wrote:
> But draft we are discussing here does not talk about "Min" delay. 
> Contrary it talks about "Max"
> 
> *Maximum*  Delay sub-TLV
> 

I'm afraid you misunderstood it. FA uses "Min Unidirectional Link Delay" 
as one of its metrics. The "Maximum Delay sub-TLV"  is used to advertise 
the maximum value of the "Min Unidirectional Link Delay" that is allowed 
for the particular FA.

The text should be improved in that regard though, it's not obvious, but 
I believe that's what it is.

thanks,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 11:27:46 +0100

I am not sure I follow your logic here ...

If we are already advertising "Min Unidirectional link delay" as described
in https://tools.ietf.org/html/draft-ietf-lsr-flex-algo-13 why would we
need to define it again here in this draft ?

Also does it really make sense to advertise maximum value of minimum value
?

Thx,
R.

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 11:31:42 +0100

On 03/03/2021 11:27, Robert Raszuk wrote:
> If we are already advertising "Min Unidirectional link delay" as 
> described in https://tools.ietf.org/html/draft-ietf-lsr-flex-algo-13 why 
> would we need to define it again here in this draft ?

we are not defining the metric here, we are defining the constraint that 
says what is the maximum value of that metric that can be used.

thanks,
Peter
> 
> Also does it really make sense to advertise maximum value of 
> minimum value ?
> 
> Thx,
> R.
> 
> On Wed, Mar 3, 2021 at 11:22 AM Peter Psenak <[email-masked] 
>      >      > Authors stated: "Whether egress queueing delay is included
>     in the
>      >     link
>      >      > delay depends on the measuring mechanism."
>      >

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 11:41:47 +0100

Sorry but to me the draft is very clear that it does not care about min
delay, but possible maximum delay of a link  ...

After all for time sensitive applications we do care how long it will take
to actually traverse a path in practice not what would be the theoretical
min amount of time needed for this path to be traversed.

And it does define it here as brand new metric.

Just read this paragraph as well as sections 3.1.2 and 3.2.2.:

   Similarly, exclude maximum link delay constraint is also defined in
   this document.  Links may have the link delay measured dynamically
   and advertised in delay metric in IGP.  For usecases that deploy low
   latency flex-algo, may want to exclude links that have delay more
   than a defined threshold.

Thx,
R.

On Wed, Mar 3, 2021 at 11:31 AM Peter Psenak <[email-masked]> wrote:

> >      >      > Authors stated: "Whether egress queueing delay is included
> >     in the
> >      >     link

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 11:47:18 +0100

Robert,

> delay, but possible maximum delay of a link  ...

"maximum link delay constraint" !=  "max link delay"

You are not listening.

Peter

> 
> After all for time sensitive applications we do care how long it will 
> take to actually traverse a path in practice not what would be the 
> theoretical min amount of time needed for this path to be traversed.
> 
> Just read this paragraph as well as sections 3.1.2 and 3.2.2. 
> <http://3.2.2.>:
> 
> On Wed, Mar 3, 2021 at 11:31 AM Peter Psenak <[email-masked] 
>      > described in
>     https://tools.ietf.org/html/draft-ietf-lsr-flex-algo-13 why
>     <mailto:[email-masked]>
>      > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>      >
>      >     Robert,
>      >
>      >      > Hey Peter,
>      >      >
>      >      >      > Authors stated: "Whether egress queueing delay is
>     included
>      >     in the
>      >      >     link
>      >      >      > delay depends on the measuring mechanism."
>      >      >
>      >      >     I disagree with that statement - the Min
>     Unidirectional Link
>      >     Delay is
>      >      >     the value that does not include the queueing delay -
>     that's
>      >     why it is
>      >      >     called Min.
>      >      >
>      >      >
>      >      >
>      >      > But draft we are discussing here does not talk about "Min"
>     delay.
>      >      > Contrary it talks about "Max"
>      >      >
>      >      > *Maximum*  Delay sub-TLV
>      >      >
>      >      > That is also I asked that very question up front.
>      >
>      >     The text should be improved in that regard though, it's not
>     obvious,
>      >     but
>      >      >
>      >      > Thx,
>      >      > R.
>      >      >
>      >      >
>      >      >
>      >      >

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 11:54:52 +0100

Not sure what's the difference between the two.

But I guess let't wait for authors to clarify their intentions here.

Cheers,
R.

On Wed, Mar 3, 2021, 11:47 Peter Psenak <[email-masked]> wrote:

> >      >      >     I disagree with that statement - the Min
> >     Unidirectional Link
> >      >     Delay is
> >      >      >     the value that does not include the queueing delay -
> >     that's
> >      >     why it is
> >      >      >     called Min.
> >      >      >
> >      >      >
> >      >      >
> >      >     I'm afraid you misunderstood it. FA uses "Min Unidirectional
> Link
> >      >     Delay"
> >      >     the maximum value of the "Min Unidirectional Link Delay" that
> is
> >      >     allowed
> >      >     for the particular FA.
> >      >

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 12:11:30 +0100

Robert,

looking at the text from the draft:

    "The link delay [RFC8570].as advertised by the sub-TLV 33
    of the TLV 22/222/23/223/141 is compared against the Max link delay
    advertised in FAEMD sub-TLV."

sub-TLV 33 is "Unidirectional Link Delay", which defined as "average" 
link delay.

I would argue this should be Min Delay.

The draft proposes to use "Maximum link bandwidth" (TLV-9) as a metric 
(indirectly) and is using the same TLV-9 to compare against the "Min 
Bandwidth" constraint.

The same should be used for metric - it uses Min delay (TLV-34) as 
metric and it should use the same TLV-34 to compare against the "Maximum 
Delay" constraint.

Usage of the "average" delay that is subject to change depending on the 
traffic crossing the link would make the problem of traffic osculation 
much more real and I'm not sure we want to go that path - history has 
proved that to be quite difficult.

thanks,
Peter

On 03/03/2021 11:54, Robert Raszuk wrote:
> On Wed, Mar 3, 2021, 11:47 Peter Psenak <[email-masked] 
>      >      > I am not sure I follow your logic here ...
>      >      >
>      >      > If we are already advertising "Min Unidirectional link
>     delay" as
>      >      > described in
>      >      > would we need to define it again here in this draft ?
>      >
>      >     we are not defining the metric here, we are defining the
>     constraint
>      >     that
>      >      > Also does it really make sense to advertise maximum value of
>      >      > minimum value ?
>      >      >
>      >      > Thx,
>      >      > R.
>      >      >
>      >      > On Wed, Mar 3, 2021 at 11:22 AM Peter Psenak
>     <[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>      >      > <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>      >      >
>      >      >     Robert,
>      >      >
>      >      >     On 03/03/2021 11:10, Robert Raszuk wrote:
>      >      >      > Hey Peter,
>      >      >      >
>      >      >      >      > Authors stated: "Whether egress queueing
>     delay is
>      >     included
>      >      >     in the
>      >      >      >     link
>      >      >      >      > delay depends on the measuring mechanism."
>      >      >      >
>      >      >      >     I disagree with that statement - the Min
>      >     Unidirectional Link
>      >      >     Delay is
>      >      >      >     the value that does not include the queueing
>     delay -
>      >     that's
>      >      >     why it is
>      >      >      >     called Min.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      > But draft we are discussing here does not talk
>     about "Min"
>      >     delay.
>      >      >      > Contrary it talks about "Max"
>      >      >      >
>      >      >      > *Maximum*  Delay sub-TLV
>      >      >      >
>      >      >      > That is also I asked that very question up front.
>      >      >
>      >      >     I'm afraid you misunderstood it. FA uses "Min
>     Unidirectional Link
>      >      >     Delay"
>      >      >     as one of its metrics. The "Maximum Delay sub-TLV"  is
>     used to
>      >      >     advertise
>      >      >     the maximum value of the "Min Unidirectional Link
>     Delay" that is
>      >      >     allowed
>      >      >     for the particular FA.
>      >      >
>      >      >     The text should be improved in that regard though,
>     it's not
>      >     obvious,
>      >      >     but
>      >      >     I believe that's what it is.
>      >      >
>      >      >     thanks,
>      >      >     Peter
>      >      >
>      >      >      >
>      >      >      > Thx,
>      >      >      > R.
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >
>      >
>

---

From: Shraddha Hegde <[email-masked]>
Date: Wed, 3 Mar 2021 13:03:05 +0000

Robert,

The draft is not trying to define new delay metric.
A new constraint called " Exclude Maximum link delay " is being defined in the draft.
This constraint when included in the FAD should be used prune links that have RFC 8570 advertised
Unidirectional link delay larger than the value defined in this FAD constraint.
We will post the -01 version when the window opens. We have clearer text and also
fixed some confusions in IANA section.

Rgds
Shraddha

Juniper Business Use Only
From: Robert Raszuk <[email-masked]>
Sent: Wednesday, March 3, 2021 4:12 PM
To: Peter Psenak <[email-masked]>
Cc: Tony Li <[email-masked]>; Gyan Mishra <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>; Shraddha Hegde <[email-masked]>; Rajesh M <[email-masked]>; [email-masked]; William Britto A J <bwilliam=[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Sorry but to me the draft is very clear that it does not care about min delay, but possible maximum delay of a link  ...

After all for time sensitive applications we do care how long it will take to actually traverse a path in practice not what would be the theoretical min amount of time needed for this path to be traversed.

Just read this paragraph as well as sections 3.1.2 and 3.2.2.<https://urldefense.com/v3/__http:/3.2.2.__;!!NEt6yMaO-gk!Qi6nctWtUC5MsX--CcSHucSj6ja5VJIBkRYNQtm3EOTpOgWBEzDcIQDmqwM1R9Mc$>:

Thx,

R.

On Wed, Mar 3, 2021 at 11:31 AM Peter Psenak <[email-masked]<mailto:[email-masked]>> wrote:
On 03/03/2021 11:27, Robert Raszuk wrote:
> described in https://tools.ietf.org/html/draft-ietf-lsr-flex-algo-13<https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-ietf-lsr-flex-algo-13__;!!NEt6yMaO-gk!Qi6nctWtUC5MsX--CcSHucSj6ja5VJIBkRYNQtm3EOTpOgWBEzDcIQDmq-eiJLT-$> why

thanks,
Peter
> On Wed, Mar 3, 2021 at 11:22 AM Peter Psenak <[email-masked]<mailto:[email-masked]>
> <mailto:[email-masked]<mailto:[email-masked]>>> wrote:
>
>     Robert,
>

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 14:19:39 +0100

Shraddha,

Yes your proposal defines constrains for FAD. But ny point is that if you
are defining such constrain called Max Link Delay you better make sure that
parameter used to measure such Maximum is well generated and flooded.
Otherwise this constrain becomes questionable.

What if implementation will choose to advertise Minimum Link Delay of
the period of 1 week or have this as constant value configured wheen you
test your link before deploying it in production ? What if it never will
include egress queueing delay. How practical will be your constrain in such
cases ?

Many thx,
R.

On Wed, Mar 3, 2021 at 2:03 PM Shraddha Hegde <shraddha=
[email-masked]> wrote:

> A new constraint called “ Exclude Maximum link delay “ is being defined in
> the draft.
>
> *From:* Robert Raszuk <[email-masked]>
> *Sent:* Wednesday, March 3, 2021 4:12 PM
> *To:* Peter Psenak <[email-masked]>
> *Cc:* Tony Li <[email-masked]>; Gyan Mishra <[email-masked]>;
> DECRAENE Bruno IMT/OLN <[email-masked]>; Shraddha Hegde <
> [email-masked]>; Rajesh M <[email-masked]>; [email-masked];
> William Britto A J <bwilliam=[email-masked]>
> *Subject:* Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
>
>
>
> <https://urldefense.com/v3/__http:/3.2.2.__;!!NEt6yMaO-gk!Qi6nctWtUC5MsX--CcSHucSj6ja5VJIBkRYNQtm3EOTpOgWBEzDcIQDmqwM1R9Mc$>:
>
>
>
>
> > described in https://tools.ietf.org/html/draft-ietf-lsr-flex-algo-13
> <https://urldefense.com/v3/__https:/tools.ietf.org/html/draft-ietf-lsr-flex-algo-13__;!!NEt6yMaO-gk!Qi6nctWtUC5MsX--CcSHucSj6ja5VJIBkRYNQtm3EOTpOgWBEzDcIQDmq-eiJLT-$>
> why

---

From: Shraddha Hegde <[email-masked]>
Date: Wed, 3 Mar 2021 13:52:04 +0000

Robert,

Pls see inline...

Juniper Business Use Only
From: Robert Raszuk <[email-masked]>
Sent: Wednesday, March 3, 2021 6:50 PM
To: Shraddha Hegde <[email-masked]>
Cc: Peter Psenak <[email-masked]>; Gyan Mishra <[email-masked]>; Rajesh M <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>; Tony Li <[email-masked]>; [email-masked]; William Britto A J <[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Shraddha,

Yes your proposal defines constrains for FAD. But ny point is that if you are defining such constrain called Max Link Delay you better make sure that parameter used to measure such Maximum is well generated and flooded.
<Shraddha> The Maximum delay in FAD is compared against min delay advertised per link as  in RFC 8570.
                       so maximum delay is not required to be generated and flooded  for every link.

What if implementation will choose to advertise Minimum Link Delay of the period of 1 week or have this as constant value configured wheen you test your link before deploying it in production ? What if it never will include egress queueing delay. How practical will be your constrain in such cases ?
<Shraddha> If statically configured link delay values are used in production, the existing exclude link affinity FAD constraints will be good enough to achieve any required  topology pruning in the Flex-algo. The "Exclude maximum link delay" defined in this document is useful when dynamic link-delay measurement is used.

Many thx,
R.

On Wed, Mar 3, 2021 at 2:03 PM Shraddha Hegde <shraddha=[email-masked]<mailto:[email-masked]>> wrote:
Robert,

Rgds
Shraddha

Juniper Business Use Only
From: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, March 3, 2021 4:12 PM
To: Peter Psenak <[email-masked]<mailto:[email-masked]>>
Cc: Tony Li <[email-masked]<mailto:[email-masked]>>; Gyan Mishra <[email-masked]<mailto:[email-masked]>>; DECRAENE Bruno IMT/OLN <[email-masked]<mailto:[email-masked]>>; Shraddha Hegde <[email-masked]<mailto:[email-masked]>>; Rajesh M <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; William Britto A J <bwilliam=[email-masked]<mailto:[email-masked]>>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Thx,

R.

thanks,
Peter
_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/lsr__;!!NEt6yMaO-gk!RV3wuZC53zWIexlY8lVqmUuzLvZyHt4Z5gUf91AdddmCAXzdENrH84mBLZU0lVDO$>

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 14:55:50 +0100

> and flooded  for every link.

I was talking about requirements of generation and flooding of  min delay
for the needs of this new constrain.

Thx,
R.

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 3 Mar 2021 07:39:49 -0500

William,

Understood.  From following the thread and draft basically like the RSVP TE
ERO link attributes "include" or "exclude" or applying infinity bits to
prune off low bandwidth links from the topology graph based on either new
"exclude maximum link delays" dynamic values or existing exclude link
affinity FAD static values.  Do I have it right?

Kind Regards

Gyan

On Mon, Mar 1, 2021 at 5:24 AM William Britto A J <[email-masked]>
wrote:

> *M 301 502-1347 13101 Columbia Pike  *Silver Spring, MD
>
>
>

-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 15:42:56 +0100

Hi Robert,

On 03/03/2021 14:55, Robert Raszuk wrote:
> 
>     Yes your proposal defines constrains for FAD. But ny point is that
>     if you are defining such constrain called Max Link Delay you better
>     make sure that parameter used to measure such Maximum is well
>     generated and flooded. ____
> 
>     <Shraddha> The Maximum delay in FAD is compared against min delay
>     advertised per link as  in RFC 8570.____
> 
> I was talking about requirements of generation and flooding of  min 
> delay for the needs of this new constrain.

yes, but the min delay is already being used by flex-algo as one of the 
possible metrics, so noting new is required.

thank,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 15:46:55 +0100

Hi Peter,

> possible metrics, so noting new is required.

I think it depends on one's use case.

The fact that we are advertising this min delay today to me does not mean
that we are done with it for ever and every use case :).

All,

Is there any document or implementation report summarizing how different
implementations generate and how (by default) trigger flooding of the
changed min delay value per link ?

Thx,
R.

---

From: Tony Li <[email-masked]>
Date: Wed, 3 Mar 2021 09:21:34 -0800

Peter,

> TWAMP provided Min Unidirectional Link Delay is a dynamic one. On the other side this value is calculated based on multiple measurements over period of time and an average is used. Also, smart implementations can normalize the value so that a small fluctuation of the delay is not causing the traffic to shift or cause ECMP loss.
> 
> What is important here is that the Min Unidirectional Link Delay is a link characteristic, not something that is affected by the amount of traffic on the link or subject to queuing delay. Same applies to Maximum link bandwidth.

I do understand that the minimum link delay is not meant to include queuing delay. That, however, does NOT make it a constant.  

There are several link types in use that exhibit variable delay: satellite links (e.g., Starlink), microwave links, and ancient link layers that deliver reliability through retransmission.

Any of these (and probably a lot more) can create a noticeable and measurable difference in TWAMP. That would be reflected in an FA metric change. If you imagine a situation with multiiple parallel paths with nearly identical delays, you can easily imagine an oscillatory scenario.  IMHO, this is an outstanding concern with FA. 

Please note that I’m NOT recommending that we back away. Rather, we should seek to solve the long-standing issue of oscillatory routing.

Regards,
Tony

---

From: Shraddha Hegde <[email-masked]>
Date: Wed, 3 Mar 2021 17:36:18 +0000

Tony,

> Please note that I'm NOT recommending that we back away. Rather, we should seek to solve the long-standing issue of oscillatory routing.

It's a fair point and I see Robert is also making a comment on Implementation report of how the link-delay value is measured and flooded. Seems like either a clarification draft on RFC 8570 or an 8570bis would make sense.

RFC 8570 has been around for sometime. I would be interested to hear if there are deployments of dynamically measured link-delay.

Rgds
Shraddha

Juniper Business Use Only
From: Tony Li <[email-masked]> On Behalf Of Tony Li
Sent: Wednesday, March 3, 2021 10:52 PM
To: Peter Psenak <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; Gyan Mishra <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>; Shraddha Hegde <[email-masked]>; Rajesh M <[email-masked]>; [email-masked]; William Britto A J <bwilliam=[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Peter,

Please note that I'm NOT recommending that we back away. Rather, we should seek to solve the long-standing issue of oscillatory routing.

Regards,
Tony

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 19:04:21 +0100

Tony,

On 03/03/2021 18:21, Tony Li wrote:
>> other side this value is calculated based on multiple measurements 
>> over period of time and an average is used. Also, smart 
>> implementations can normalize the value so that a small fluctuation of 
>> the delay is not causing the traffic to shift or cause ECMP loss.
>>
>> traffic on the link or subject to queuing delay. Same applies to 
>> Maximum link bandwidth.
> 
> 
> I do understand that the minimum link delay is not meant to include 
> queuing delay. That, however, does NOT make it a constant.

I never claimed it is constant. It should not be affected though by the 
amount of traffic on the link itself, but rather by the characteristics 
of the underlying physical path.

> 
> There are several link types in use that exhibit variable delay: 
> satellite links (e.g., Starlink), microwave links, and ancient link 
> layers that deliver reliability through retransmission.
> 
> Any of these (and probably a lot more) can create a noticeable and 
> measurable difference in TWAMP. That would be reflected in an FA metric 
> change. If you imagine a situation with multiiple parallel paths with 
> nearly identical delays, you can easily imagine an oscillatory scenario. 
>   IMHO, this is an outstanding concern with FA.
yes, and that is what I referred to as "delay normalization", which can 
avoid that oscillation. There are implementations that do that today. 
You normalize the measured value so that values that are close enough 
are advertised as same value. That works well for min delay case, which 
does not change significantly very often.

> 
> Please note that I’m NOT recommending that we back away. Rather, we 
> should seek to solve the long-standing issue of oscillatory routing.

not that I disagree. History tells us that the generic case of 
oscillation which is caused by the traffic itself is a hard problem to 
solve.

thanks,
Peter

---

From: Tony Li <[email-masked]>
Date: Wed, 3 Mar 2021 10:14:03 -0800

Peter,

> yes, and that is what I referred to as "delay normalization", which can avoid that oscillation.

It can also negate the benefits of the feature. One might well imagine that Starlink would want to follow a min-delay path for optimality.  If the delay variations are “normalized” out of existence, then the benefits are lost.  The whole point is to track the dynamics.

> not that I disagree. History tells us that the generic case of oscillation which is caused by the traffic itself is a hard problem to solve.

Any oscillation is difficult to solve.  Positive feedback certainly can exacerbate the problem. But solving hard problems is why we are here.

Yours in control theory,
Tony

---

From: Peter Psenak <[email-masked]>
Date: Wed, 3 Mar 2021 19:40:57 +0100

Tony,

> It can also negate the benefits of the feature. One might well imagine that Starlink would want to follow a min-delay path for optimality.  If the delay variations are “normalized” out of existence…

for all practical purposes that we use it for, the two values of min 
delay that differ by few microsecond can be treated as same without any 
loss of functionality. So it's about the required normalization interval 
- something that can be controlled by the user.

thanks,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 20:57:10 +0100

Peter,

>  that differ by few microsecond

Really you normalize only single digit microseconds ???

What if link delay changes in milliseconds scale ? Do you want to compute
new topology every few milliseconds ?

Out of curiosity as this is not a secret -  What are your default min delay
normalization timers (if user does not overwrite with their own). Likewise
how Junos or Arista normalizes it today ?

Thx,
R.

On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak <[email-masked]> wrote:

> satellite links (e.g., Starlink), microwave links, and ancient link layers
> that deliver reliability through retransmission.
> avoid that oscillation.
> > It can also negate the benefits of the feature. One might well imagine
> that Starlink would want to follow a min-delay path for optimality.  If the
> delay variations are “normalized” out of existence, then the benefits are
> lost.  The whole point is to track the dynamics.
>
> > Any oscillation is difficult to solve.  Positive feedback certainly can
> exacerbate the problem. But solving hard problems is why we are here.
> >
> > Yours in control theory,
> > Tony
> >
> >
> >
>
>

---

From: Tarek Saad <[email-masked]>
Date: Wed, 3 Mar 2021 22:34:18 +0000

Hi Robert,

The RSVP-TE world has had to deal with such churn resulting from frequent link attribute changes (e.g. specific to available BW). In that case, such frequent changes made their way to the network at periodic intervals and in the event they crossed a threshold. In my mind, the link delay attribute is no different and IGPs can react to it just like RSVP-TE did.

Obviously, any path that was computed and placed on a set of links based on a certain view of the network may quickly become stale. However, IMO, any per-path e2e SLA need to be validated (independent of the network topology) e.g., by active measurement using probes or other means.

My 2cents.

Regards,
Tarek

On 3/3/21, 2:57 PM, "Lsr on behalf of Robert Raszuk" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Peter,

What if link delay changes in milliseconds scale ? Do you want to compute new topology every few milliseconds ?

Out of curiosity as this is not a secret -  What are your default min delay normalization timers (if user does not overwrite with their own). Likewise how Junos or Arista normalizes it today ?

Thx,
R.

On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak <[email-masked]<mailto:[email-masked]>> wrote:
Tony,

thanks,
Peter

---

From: Tony Li <[email-masked]>
Date: Wed, 3 Mar 2021 14:41:12 -0800

Hi Tarek,

Please recall that in FA there is no path setup. If the delay changes and it propagates to other nodes, then the network will SPF and paths may change immediately.

Tony

> On Mar 3, 2021, at 2:34 PM, Tarek Saad <[email-masked]> wrote:
> 
> Hi Robert,
>  
> On 3/3/21, 2:57 PM, "Lsr on behalf of Robert Raszuk" <[email-masked] <mailto:[email-masked]> on behalf of [email-masked] <mailto:[email-masked]>> wrote:
>  
> Peter,
>  
> On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak <[email-masked] <mailto:[email-masked]>> wrote:
>> Tony,
>>

---

From: Tarek Saad <[email-masked]>
Date: Wed, 3 Mar 2021 22:47:28 +0000

Hi Tony,

Yes, I’m aware FA is hop-by-hop..
My point is per-link delay changes can be suppressed and advertised at periodic intervals (usually order of mins) or immediately based on crossing a threshold.
The per-path delay cannot be accurately extracted from a “snapshot” of the view of the topology (e.g. adding per link delays for traversed links - as those values may quickly become stale). Such validation IMO (if any are needed) will have to come based on active/passive measuring mechanisms..

Regards,
Tarek

On 3/3/21, 5:41 PM, "Lsr on behalf of Tony Li" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Hi Tarek,

Tony

On Mar 3, 2021, at 2:34 PM, Tarek Saad <[email-masked]<mailto:[email-masked]>> wrote:

Hi Robert,

My 2cents.

Regards,
Tarek

Peter,

Thx,
R.

thanks,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 3 Mar 2021 23:54:22 +0100

Hi Tarek,

Yes as Tony also just indicated it is completely different game here.
Headend can do whatever it likes.

But I think your point and also what Peter said earlier is to actually
throw the baby with the bath water by suppressing  advertisements/flooding.

It is all subject to proper suppression tuning/timers and I think we have
little experience with it (yet).

Most importantly we are talking about topology wide changes so IMHO doing
it on the receiving node is not an option. It must be done on sender.
Receiver side suppression would be only valid if timers are hard in stone
in the RFC.

Thx,
R.

On Wed, Mar 3, 2021 at 11:34 PM Tarek Saad <[email-masked]> wrote:

> Hi Robert,
>
>
>
> The RSVP-TE world has had to deal with such churn resulting from frequent
> link attribute changes (e.g. specific to available BW). In that case, such
> frequent changes made their way to the network at periodic intervals and in
> the event they crossed a threshold. In my mind, the link delay attribute is
> no different and IGPs can react to it just like RSVP-TE did.
>
>
>
> Obviously, any path that was computed and placed on a set of links based
> on a certain view of the network may quickly become stale. However, IMO,
> any per-path e2e SLA need to be validated (independent of the network
> topology) e.g., by active measurement using probes or other means.
>
>
>
> My 2cents.
>
>
>
> Regards,
>
> Tarek
>
>
>
> On 3/3/21, 2:57 PM, "Lsr on behalf of Robert Raszuk" <[email-masked]
> on behalf of [email-masked]> wrote:
>
>
>
> Peter,
>
>
>
> Out of curiosity as this is not a secret -  What are your default min
> delay normalization timers (if user does not overwrite with their own).
> Likewise how Junos or Arista normalizes it today ?
>
>
>
> Thx,
>
> R.
>
>
>

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 10:07:23 +0100

Robert,

On 03/03/2021 20:57, Robert Raszuk wrote:
> What if link delay changes in milliseconds scale ? Do you want to 
> compute new topology every few milliseconds ?

let me repeat again.

Min delay is not something that changes every few milliseconds 
significantly. It's a semi static variable that reflects the property of 
the underlying physical path. It only changes when the physical path 
properties changes - e.g. the optical path reroutes, etc. We 
deliberately picked Min delay for flex-algo purposes for this semi 
static property.

The small, non significant changes can be filtered by the normalization.

If the min delay changes significantly every few milliseconds there's 
something wrong with the link itself - we have standard dampening 
mechanisms in LS protocols to deal with unstable links that would kick 
in. Similar to what we do if the link flaps every few milliseconds.

> delay normalization timers (if user does not overwrite with their own).

there is no timer needed for the normalization itself.

You are likely referring TWAMP computation interval which is 30 sec, 
with probes being sent every 3 seconds in our implementation by default, 
if I'm not mistaken.

Normalization is applied to the value that come from the above.

thanks,
Peter

>      >>> There are several link types in use that exhibit variable
>     delay: satellite links (e.g., Starlink), microwave links, and
>     ancient link layers that deliver reliability through retransmission.
>      >>> Any of these (and probably a lot more) can create a noticeable
>     and measurable difference in TWAMP. That would be reflected in an FA
>     metric change. If you imagine a situation with multiiple parallel
>     paths with nearly identical delays, you can easily imagine an
>     oscillatory scenario.   IMHO, this is an outstanding concern with FA.
>      > It can also negate the benefits of the feature. One might well
>     imagine that Starlink would want to follow a min-delay path for
>     optimality.  If the delay variations are “normalized” out of
>     existence, then the benefits are lost.  The whole point is to track
>     the dynamics.
> 
>      >>> Please note that I’m NOT recommending that we back away.
>     Rather, we should seek to solve the long-standing issue of
>     oscillatory routing.
>      >>
>     oscillation which is caused by the traffic itself is a hard problem
>     to solve.
>      >
>      >
>      > Any oscillation is difficult to solve.  Positive feedback
>     certainly can exacerbate the problem. But solving hard problems is
>     why we are here.
>      >
>      > Yours in control theory,
>      > Tony
>      >
>      >
>      >
>

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 10:12:22 +0100

Robert,

> throw the baby with the bath water by suppressing  advertisements/flooding.

no, we are not throwing the baby. You seem to by trying to find the 
problem where it does not exist.

> 
> It is all subject to proper suppression tuning/timers and I think we 
> have little experience with it (yet).

I would not say so.

> 
> Most importantly we are talking about topology wide changes so IMHO 
> doing it on the receiving node is not an option. It must be done on 
> sender. 

at least we agree on the above.

thanks,
Peter

>Receiver side suppression would be only valid if timers are hard 
> in stone in the RFC.
> 
> Thx,
> R.
> 
> On Wed, Mar 3, 2021 at 11:34 PM Tarek Saad <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Robert,____
> 
>     __ __
> 
>     The RSVP-TE world has had to deal with such churn resulting from
>     frequent link attribute changes (e.g. specific to available BW). In
>     that case, such frequent changes made their way to the network at
>     periodic intervals and in the event they crossed a threshold. In my
>     mind, the link delay attribute is no different and IGPs can react to
>     it just like RSVP-TE did.____
> 
>     __ __
> 
>     Obviously, any path that was computed and placed on a set of links
>     based on a certain view of the network may quickly become stale.
>     However, IMO, any per-path e2e SLA need to be validated (independent
>     of the network topology) e.g., by active measurement using probes or
>     other means.____
> 
>     __ __
> 
>     My 2cents.____
> 
>     __ __
> 
>     Regards,____
> 
>     Tarek____
> 
>     __ __
> 
>     On 3/3/21, 2:57 PM, "Lsr on behalf of Robert Raszuk"
>     <[email-masked] <mailto:[email-masked]> on behalf of
>     [email-masked] <mailto:[email-masked]>> wrote:____
> 
>     __ __
> 
>     Peter,____
> 
>     __ __
> 
>      >  that differ by few microsecond ____
> 
>     __ __
> 
>     Really you normalize only single digit microseconds ???____
> 
>     __ __
> 
>     compute new topology every few milliseconds ? ____
> 
>     __ __
> 
>     Out of curiosity as this is not a secret -  What are your default
>     min delay normalization timers (if user does not overwrite with
>     their own). Likewise how Junos or Arista normalizes it today ? ____
> 
>     __ __
> 
>     Thx,____
> 
>     R.____
> 
>     __ __
> 
>     On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak <[email-masked]
>     <mailto:[email-masked]>> wrote:____
> 
>         Tony,
> 
>          >>> Any of these (and probably a lot more) can create a
>         noticeable and measurable difference in TWAMP. That would be
>         reflected in an FA metric change. If you imagine a situation
>         with multiiple parallel paths with nearly identical delays, you
>         can easily imagine an oscillatory scenario.   IMHO, this is an
>         outstanding concern with FA.
>          >> yes, and that is what I referred to as "delay
>         normalization", which can avoid that oscillation.
>          >
>          >
>          > It can also negate the benefits of the feature. One might
>         well imagine that Starlink would want to follow a min-delay path
>         for optimality.  If the delay variations are “normalized” out of
>         existence, then the benefits are lost.  The whole point is to
>         track the dynamics.
> 
>          >> not that I disagree. History tells us that the generic case
>         of oscillation which is caused by the traffic itself is a hard
>         problem to solve.
>          >
>          >
>         certainly can exacerbate the problem. But solving hard problems
>         is why we are here.
>          >
>          > Yours in control theory,
>          > Tony
>          >
>          >
>          > ____
> 
>

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 4 Mar 2021 17:16:20 +0800

Hi, Peter:

If we use such unpredicted parameters for the dynamic IGP calculation, will the network be operated automatically in non-consistent manner and let the operator stuck in a mess, and busy to find which semi static value was changed and what's it the cause?
Is this the right direction for network automation?

Best Regards

Aijun Wang
China Telecom

-----Original Message-----
From: [email-masked] <[email-masked]> On Behalf Of Peter Psenak
Sent: Thursday, March 4, 2021 5:07 PM
To: Robert Raszuk <[email-masked]>
Cc: Gyan Mishra <[email-masked]>; Rajesh M <[email-masked]>; Shraddha Hegde <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>; Tony Li <[email-masked]>; [email-masked]; William Britto A J <bwilliam=[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Robert,

let me repeat again.

Min delay is not something that changes every few milliseconds significantly. It's a semi static variable that reflects the property of the underlying physical path. It only changes when the physical path properties changes - e.g. the optical path reroutes, etc. We deliberately picked Min delay for flex-algo purposes for this semi static property.

If the min delay changes significantly every few milliseconds there's something wrong with the link itself - we have standard dampening mechanisms in LS protocols to deal with unstable links that would kick in. Similar to what we do if the link flaps every few milliseconds.

thanks,
Peter

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 10:23:54 +0100

Peter,

Completely disagree.

Real say enterprise networks are being build with emulated circuits
(example VPWS). In one company I was working at it was about 80% of
emulated links all over the world in their WAN. Yes for me it was a shock
as I did not realize how much this emulated links took over the world. In
most geographies you even can not get any link of less then 10Gbps to be
real any more. Only emulated option is on the table.

Emulated circuits run over someone's IP backbones. You can understand the
consequences of this. Not only link delay changes a lot but you run into
very interesting set of issues.

Maybe you think of the backbones of the 90s or 2000s where dark fiber or
SDH or SONET were in use for interconnects.

Well no more. IETF came with such brilliant ideas to emulate L2 over L3 and
here we go.

Reality is not what we wish it to be.

Cheers,
R.

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 10:25:01 +0100

Aijun,

On 04/03/2021 10:16, Aijun Wang wrote:
> Hi, Peter:
> 
> If we use such unpredicted parameters for the dynamic IGP calculation, 

I'm not sure what unpredicted parameters you are talking about.

> will the network be operated automatically in non-consistent manner and let the operator stuck in a mess, and busy to find which semi static value was changed and what's it the cause?

no, if it is done properly. We have examples of deployments in the field 
that prove it to work.

> Is this the right direction for network automation?

delay based traffic optimization has been requested from the field for 
years.

thanks,
Peter

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 10:41:28 +0100

Robert,

> Completely disagree.

that's what you seem to be doing from the beginning of this conversation ;)

> emulated links all over the world in their WAN. Yes for me it was a 
> shock as I did not realize how much this emulated links took over the 
> world. In most geographies you even can not get any link of less then 
> 10Gbps to be real any more. Only emulated option is on the table.
> 
> Emulated circuits run over someone's IP backbones. You can understand 
> the consequences of this. Not only link delay changes a lot but you run 
> into very interesting set of issues.

look at it from the opposite direction. The provider of the VPWS is the 
one who can use this technology to guarantee the delay bound of the WPWs 
service. And if it does, the user of the WPWs would not experience the 
wild variation in the min delay.

So you have to apply right set of tools at right location. Obviously, if 
the min delay fluctuates wildly, one can not achieve delay optimized 
forwarding no matter what. That does not mean that the network will get 
unstable.

Peter

> Well no more. IETF came with such brilliant ideas to emulate L2 over L3 
> and here we go.
> 
> On Thu, Mar 4, 2021 at 10:07 AM Peter Psenak <[email-masked] 
>      >  >  that differ by few microsecond
>      >
>     min
>      >      > Peter,
>      >      >>> There are several link types in use that exhibit variable
>      >      >>> Any of these (and probably a lot more) can create a
>     noticeable
>      >      >> yes, and that is what I referred to as "delay normalization",
>      >     which can avoid that oscillation.
>      >      >
>      >      >
>      >      > It can also negate the benefits of the feature. One might well
>     track
>      >      >>> Please note that I’m NOT recommending that we back away.
>      >      >>
>      >      >> not that I disagree. History tells us that the generic
>     case of
>      >      > Any oscillation is difficult to solve.  Positive feedback
>      >      > Yours in control theory,
>      >      > Tony
>      >      >
>      >      >
>      >      >
>      >
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 10:50:54 +0100

Peter,

Sorry but the draft does not have a disclaimer section which states:

"Extensions defined below are only applicable to networks with not even a
single emulated circuit in the IGP."

> Obviously, if the min delay fluctuates wildly, one can not achieve delay
optimized
> forwarding no matter what.

If VPWS provider's IP network reconverges once per day or per hour I see
nothing wrong with flooding new link delays and recomputing the topology of
the network running on top of such constructs. My point is that changes are
real, pretty unpredictable and in ms or 10s of ms. And IMO the proposal on
the table is specifically designed to catch and address those cases - not
just dismiss it as you can not use it in your network - sorry mr customer.

Best,
R.

On Thu, Mar 4, 2021 at 10:41 AM Peter Psenak <[email-masked]> wrote:

> well
> >      >     loss of functionality. So it's about the required
> normalization
> >      >     interval

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 11:06:31 +0100

Robert,

On 04/03/2021 10:50, Robert Raszuk wrote:
> "Extensions defined below are only applicable to networks with not even 
> a single emulated circuit in the IGP."
> 
>  > Obviously, if the min delay fluctuates wildly, one can not achieve 
> delay optimized
> nothing wrong with flooding new link delays and recomputing the topology 
> of the network running on top of such constructs. 

the question is how much sense would it make to try to optimize based on 
delay if it fluctuates every ms and we optimize once per day. But feel 
free to give it a shot if you believe it's a good idea.

Peter

> My point is that 
> changes are real, pretty unpredictable and in ms or 10s of ms. And IMO 
> the proposal on the table is specifically designed to catch and address 
> those cases - not just dismiss it as you can not use it in your network 
> - sorry mr customer. >
> Best,
> R.
> 
> 
> 
> 
> 
> 
> On Thu, Mar 4, 2021 at 10:41 AM Peter Psenak <[email-masked] 
>      >      >  >  that differ by few microsecond
>      >      >
>      >      > Really you normalize only single digit microseconds ???
>      >      >
>      >      > What if link delay changes in milliseconds scale ? Do you
>     want to
>      >      > compute new topology every few milliseconds ?
>      >
>      >     let me repeat again.
>      >
>      >      > Out of curiosity as this is not a secret -  What are your
>     default
>      >     min
>      >      > delay normalization timers (if user does not overwrite
>     with their
>      >     own).
>      >
>      >      > Likewise how Junos or Arista normalizes it today ?
>      >      >
>      >      > Thx,
>      >      > R.
>      >      >
>      >      > On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak
>      >      >     Tony,
>      >      >     On 03/03/2021 19:14, Tony Li wrote:
>      >      >      >
>      >      >      > Peter,
>      >      >      >
>      >      >      >>> There are several link types in use that exhibit
>     variable
>      >      >     delay: satellite links (e.g., Starlink), microwave
>     links, and
>      >      >     ancient link layers that deliver reliability through
>      >     retransmission.
>      >      >      >>> Any of these (and probably a lot more) can create a
>      >     noticeable
>      >      >     and measurable difference in TWAMP. That would be
>     reflected
>      >     in an FA
>      >      >     metric change. If you imagine a situation with
>     multiiple parallel
>      >      >     paths with nearly identical delays, you can easily
>     imagine an
>      >      >     oscillatory scenario.   IMHO, this is an outstanding
>     concern
>      >     with FA.
>      >      >      >> yes, and that is what I referred to as "delay
>     normalization",
>      >      >     which can avoid that oscillation.
>      >      >      >
>      >      >      >
>      >      >      > It can also negate the benefits of the feature. One
>     might well
>      >      >     imagine that Starlink would want to follow a min-delay
>     path for
>      >      >     optimality.  If the delay variations are “normalized”
>     out of
>      >      >     existence, then the benefits are lost.  The whole
>     point is to
>      >     track
>      >      >     the dynamics.
>      >      >
>      >      >     for all practical purposes that we use it for, the two
>     values
>      >     of min
>      >      >     delay that differ by few microsecond can be treated as
>     same
>      >     without any
>      >      >     loss of functionality. So it's about the required
>     normalization
>      >      >     interval
>      >      >     - something that can be controlled by the user.
>      >      >
>      >      >     thanks,
>      >      >     Peter
>      >      >
>      >      >
>      >      >
>      >      >      >
>      >      >      >
>      >      >      >>> Please note that I’m NOT recommending that we
>     back away.
>      >      >     Rather, we should seek to solve the long-standing issue of
>      >      >     oscillatory routing.
>      >      >      >>
>      >      >      >> not that I disagree. History tells us that the generic
>      >     case of
>      >      >     oscillation which is caused by the traffic itself is a
>     hard
>      >     problem
>      >      >     to solve.
>      >      >      >
>      >      >      >
>      >      >      > Any oscillation is difficult to solve.  Positive
>     feedback
>      >      >     certainly can exacerbate the problem. But solving hard
>      >     problems is
>      >      >     why we are here.
>      >      >      >
>      >      >      > Yours in control theory,
>      >      >      > Tony
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >
>      >
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 11:52:38 +0100

I guess now you are not listening ;)

I am saying it is about finding the right normalization timers and values
which can satisfy the need. And those should reside on the senders.

That is why I was asking what is there today and so far no one gave precise
answer. You said links don't change delay characteristics which by itself
only applies to some link categories. Not satellite not even 5G.Leave alone
VPWS.

So I guess your conclusion is that this is hard problem and we should not
go there. But if so let's not pretend we are sending even min link delay as
dynamic value and redefine it as static approximated link delay.

Thx
R.

On Thu, Mar 4, 2021, 11:06 Peter Psenak <[email-masked]> wrote:

> >      >      > delay normalization timers (if user does not overwrite
> >     with their
> >      >     own).
> >      >
> >      >     with probes being sent every 3 seconds in our implementation
> by
> >      >     default,
> >      >     if I'm not mistaken.
> >      >
> >      >      > On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak
> >      >      > <mailto:[email-masked] <mailto:[email-masked]>
> >      >      >     and measurable difference in TWAMP. That would be
> >     reflected
> >      >     in an FA
> >      >      >     oscillatory scenario.   IMHO, this is an outstanding
> >     concern
> >      >     with FA.
> >      >      >     for all practical purposes that we use it for, the two
> >     values
> >      >     of min
> >      >      >     delay that differ by few microsecond can be treated as
> >     same
> >      >     without any
> >      >      >      >> not that I disagree. History tells us that the
> generic
> >      >     case of
> >      >      >     oscillation which is caused by the traffic itself is a
> >     hard
> >      >     problem
> >      >      >     to solve.
> >      >      >      >
> >      >      >      >

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 12:05:58 +0100

On 04/03/2021 11:52, Robert Raszuk wrote:
> I am saying it is about finding the right normalization timers and 
> values which can satisfy the need. And those should reside on the senders.

that's about what tend to agreed on so far.

But that will not address the entire problem. No timers would make it 
work in environment where the delay fluctuates constantly and 
significantly. There timers would only make sure the stability is not 
compromised. But the delay optimized forwarding will not be achieved.

> 
> That is why I was asking what is there today and so far no one gave 
> precise answer. 

I did provide a pretty detailed answer about what is there in one 
implementation.

> You said links don't change delay characteristics which 
> by itself only applies to some link categories. Not satellite not even 
> 5G.Leave alone VPWS.
> 
> So I guess your conclusion is that this is hard problem and we should 
> not go there. But if so let's not pretend we are sending even min link 
> delay as dynamic value and redefine it as static approximated link delay.

no, I did not conclude anything. I was providing a reasoning of why min 
delay was chosen versus another, even more dynamic values.

Peter
> 
> Thx
> R.
> 
> 
> 
> 
> 
> 
> On Thu, Mar 4, 2021, 11:06 Peter Psenak <[email-masked] 
>      >  > Obviously, if the min delay fluctuates wildly, one can not
>     achieve
>      > delay optimized
>      >  > forwarding no matter what.
>      >
>      >      > Completely disagree.
>      >      > Real say enterprise networks are being build with emulated
>     circuits
>      >      > (example VPWS). In one company I was working at it was
>     about 80% of
>      >      > emulated links all over the world in their WAN. Yes for me
>     it was a
>      >      > shock as I did not realize how much this emulated links
>     took over
>      >     the
>      >      > world. In most geographies you even can not get any link
>     of less
>      >     then
>      >      > 10Gbps to be real any more. Only emulated option is on the
>     table.
>      >      >
>      >      > Emulated circuits run over someone's IP backbones. You can
>      >     understand
>      >      > the consequences of this. Not only link delay changes a
>     lot but
>      >     you run
>      >      > into very interesting set of issues.
>      >
>      >     one who can use this technology to guarantee the delay bound
>     of the
>      >     WPWs
>      >      > Maybe you think of the backbones of the 90s or 2000s where
>     dark
>      >     fiber or
>      >      > SDH or SONET were in use for interconnects.
>      >      >
>      >      > Well no more. IETF came with such brilliant ideas to
>     emulate L2
>      >     over L3
>      >      > and here we go.
>      >      >
>      >      > Reality is not what we wish it to be.
>      >      >
>      >      > Cheers,
>      >      > R.
>      >      >
>      >      >
>      >      > On Thu, Mar 4, 2021 at 10:07 AM Peter Psenak
>      >      >     On 03/03/2021 20:57, Robert Raszuk wrote:
>      >      >      > Peter,
>      >      >      >
>      >      >      >  >  that differ by few microsecond
>      >      >      >
>      >      >      > Really you normalize only single digit microseconds ???
>      >      >      >
>      >      >      > What if link delay changes in milliseconds scale ?
>     Do you
>      >     want to
>      >      >      > compute new topology every few milliseconds ?
>      >      >
>      >      >     let me repeat again.
>      >      >
>      >      >     Min delay is not something that changes every few
>     milliseconds
>      >      >     significantly. It's a semi static variable that
>     reflects the
>      >      >     property of
>      >      >     the underlying physical path. It only changes when the
>      >     physical path
>      >      >     properties changes - e.g. the optical path reroutes,
>     etc. We
>      >      >     deliberately picked Min delay for flex-algo purposes
>     for this
>      >     semi
>      >      >     static property.
>      >      >
>      >      >     The small, non significant changes can be filtered by the
>      >     normalization.
>      >      >
>      >      >     If the min delay changes significantly every few
>     milliseconds
>      >     there's
>      >      >     something wrong with the link itself - we have
>     standard dampening
>      >      >     mechanisms in LS protocols to deal with unstable links
>     that
>      >     would kick
>      >      >     in. Similar to what we do if the link flaps every few
>      >     milliseconds.
>      >      >
>      >      >      >
>      >      >      > Out of curiosity as this is not a secret -  What
>     are your
>      >     default
>      >      >     min
>      >      >      > delay normalization timers (if user does not overwrite
>      >     with their
>      >      >     own).
>      >      >
>      >      >     there is no timer needed for the normalization itself.
>      >      >
>      >      >     You are likely referring TWAMP computation interval
>     which is
>      >     30 sec,
>      >      >     with probes being sent every 3 seconds in our
>     implementation by
>      >      >     default,
>      >      >     if I'm not mistaken.
>      >      >
>      >      >     Normalization is applied to the value that come from
>     the above.
>      >      >
>      >      >     thanks,
>      >      >     Peter
>      >      >
>      >      >
>      >      >
>      >      >      > Likewise how Junos or Arista normalizes it today ?
>      >      >      >
>      >      >      > Thx,
>      >      >      > R.
>      >      >      >
>      >      >      > On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>      >      >      > <mailto:[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>      >      >      >
>      >      >      >     Tony,
>      >      >      >
>      >      >      >     On 03/03/2021 19:14, Tony Li wrote:
>      >      >      >      >
>      >      >      >      > Peter,
>      >      >      >      >
>      >      >      >      >>> There are several link types in use that
>     exhibit
>      >     variable
>      >      >      >     delay: satellite links (e.g., Starlink), microwave
>      >     links, and
>      >      >      >     ancient link layers that deliver reliability
>     through
>      >      >     retransmission.
>      >      >      >      >>> Any of these (and probably a lot more) can
>     create a
>      >      >     noticeable
>      >      >      >     and measurable difference in TWAMP. That would be
>      >     reflected
>      >      >     in an FA
>      >      >      >     metric change. If you imagine a situation with
>      >     multiiple parallel
>      >      >      >     paths with nearly identical delays, you can easily
>      >     imagine an
>      >      >      >     oscillatory scenario.   IMHO, this is an
>     outstanding
>      >     concern
>      >      >     with FA.
>      >      >      >      >> yes, and that is what I referred to as "delay
>      >     normalization",
>      >      >      >     which can avoid that oscillation.
>      >      >      >      >
>      >      >      >      >
>      >      >      >      > It can also negate the benefits of the
>     feature. One
>      >     might well
>      >      >      >     imagine that Starlink would want to follow a
>     min-delay
>      >     path for
>      >      >      >     optimality.  If the delay variations are
>     “normalized”
>      >     out of
>      >      >      >     existence, then the benefits are lost.  The whole
>      >     point is to
>      >      >     track
>      >      >      >     the dynamics.
>      >      >      >
>      >      >      >     for all practical purposes that we use it for,
>     the two
>      >     values
>      >      >     of min
>      >      >      >     delay that differ by few microsecond can be
>     treated as
>      >     same
>      >      >     without any
>      >      >      >     loss of functionality. So it's about the required
>      >     normalization
>      >      >      >     interval
>      >      >      >     - something that can be controlled by the user.
>      >      >      >
>      >      >      >     thanks,
>      >      >      >     Peter
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >      >
>      >      >      >      >
>      >      >      >      >>> Please note that I’m NOT recommending that we
>      >     back away.
>      >      >      >     Rather, we should seek to solve the
>     long-standing issue of
>      >      >      >     oscillatory routing.
>      >      >      >      >>
>      >      >      >      >> not that I disagree. History tells us that
>     the generic
>      >      >     case of
>      >      >      >     oscillation which is caused by the traffic
>     itself is a
>      >     hard
>      >      >     problem
>      >      >      >     to solve.
>      >      >      >      >
>      >      >      >      >
>      >      >      >      > Any oscillation is difficult to solve.  Positive
>      >     feedback
>      >      >      >     certainly can exacerbate the problem. But
>     solving hard
>      >      >     problems is
>      >      >      >     why we are here.
>      >      >      >      >
>      >      >      >      > Yours in control theory,
>      >      >      >      > Tony
>      >      >      >      >
>      >      >      >      >
>      >      >      >      >
>      >      >      >
>      >      >
>      >
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 12:25:22 +0100

Ok so there are some agreements. Good.

As to the *minimum* you keep highlighting IMO minimum without the
definition of what time window this minimum is related to is meaningless.

Is it minimum ever, is it min of the year, month, week, day ... etc

Best
R.

On Thu, Mar 4, 2021, 12:06 Peter Psenak <[email-masked]> wrote:

> >      >      > shock as I did not realize how much this emulated links
> >     took over
> >      >     the
> >      >      > world. In most geographies you even can not get any link
> >     of less
> >      >     then
> >      >      > the consequences of this. Not only link delay changes a
> >     lot but
> >      >     you run
> >      >      > Maybe you think of the backbones of the 90s or 2000s where
> >     dark
> >      >     fiber or
> >      >      > Well no more. IETF came with such brilliant ideas to
> >     emulate L2
> >      >     over L3
> >      >      > and here we go.
> >      >      >
> >      >      > On Thu, Mar 4, 2021 at 10:07 AM Peter Psenak
> >      >      >      > What if link delay changes in milliseconds scale ?
> >     Do you
> >      >     want to
> >      >      >     significantly. It's a semi static variable that
> >     reflects the
> >      >      >     property of
> >      >      >     deliberately picked Min delay for flex-algo purposes
> >     for this
> >      >     semi
> >      >      >     static property.
> >      >      >
> >      >      >     The small, non significant changes can be filtered by
> the
> >      >     normalization.
> >      >      >
> >      >      >     If the min delay changes significantly every few
> >     milliseconds
> >      >     there's
> >      >      >     mechanisms in LS protocols to deal with unstable links
> >     that
> >      >     would kick
> >      >      >      > Out of curiosity as this is not a secret -  What
> >     are your
> >      >     default
> >      >      >     min
> >      >      >     You are likely referring TWAMP computation interval
> >     which is
> >      >     30 sec,
> >      >      >     with probes being sent every 3 seconds in our
> >     implementation by
> >      >      >     default,
> >      >      >     if I'm not mistaken.
> >      >      >
> >      >      >      > <mailto:[email-masked]
> >      >      >      >      >>> There are several link types in use that
> >     exhibit
> >      >     variable
> >      >      >      >     delay: satellite links (e.g., Starlink),
> microwave
> >      >     links, and
> >      >      >      >     ancient link layers that deliver reliability
> >     through
> >      >      >     retransmission.
> >      >      >      >     paths with nearly identical delays, you can
> easily
> >      >     imagine an
> >      >      >      >      > It can also negate the benefits of the
> >     feature. One
> >      >     might well
> >      >      >      >     imagine that Starlink would want to follow a
> >     min-delay
> >      >     path for
> >      >      >      >     optimality.  If the delay variations are
> >     “normalized”
> >      >     out of
> >      >      >      >     existence, then the benefits are lost.  The
> whole
> >      >     point is to
> >      >      >     track
> >      >      >      >     the dynamics.
> >      >      >      >
> >      >      >      >      >>> Please note that I’m NOT recommending that
> we
> >      >     back away.
> >      >      >      >      >> not that I disagree. History tells us that
> >     the generic
> >      >      >     case of
> >      >      >      >      > Any oscillation is difficult to solve.
> Positive
> >      >     feedback
> >      >      >      >     certainly can exacerbate the problem. But
> >     solving hard
> >      >      >     problems is
> >      >      >      >     why we are here.
> >      >      >      >      >
> >      >      >      >      > Yours in control theory,
> >      >      >      >      > Tony
> >      >      >      >      >
> >      >      >      >      >
> >      >      >      >      >
> >      >      >      >
> >      >      >
> >      >
> >
>
>

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 12:28:46 +0100

> Is it minimum ever, is it min of the year, month, week, day ... etc

https://tools.ietf.org/html/rfc8570#section-4.2

Peter

> 
> Best
> R.
> 
> On Thu, Mar 4, 2021, 12:06 Peter Psenak <[email-masked] 
>      >      > Sorry but the draft does not have a disclaimer section
>     which states:
>      >      >
>      >      > "Extensions defined below are only applicable to networks with
>      >     not even
>      >      > a single emulated circuit in the IGP."
>      >      >
>      >      >  > Obviously, if the min delay fluctuates wildly, one can not
>      >     achieve
>      >      > delay optimized
>      >      >  > forwarding no matter what.
>      >      >
>      >      > If VPWS provider's IP network reconverges once per day or per
>      >     hour I see
>      >      > nothing wrong with flooding new link delays and
>     recomputing the
>      >     topology
>      >      > of the network running on top of such constructs.
>      >
>      >      > My point is that
>      >      > changes are real, pretty unpredictable and in ms or 10s of ms.
>      >     And IMO
>      >      > the proposal on the table is specifically designed to
>     catch and
>      >     address
>      >      > those cases - not just dismiss it as you can not use it in
>     your
>      >     network
>      >      > - sorry mr customer. >
>      >      > Best,
>      >      > R.
>      >      >
>      >      >
>      >      >
>      >      >
>      >      >
>      >      >
>      >      > On Thu, Mar 4, 2021 at 10:41 AM Peter Psenak
>      >      >     On 04/03/2021 10:23, Robert Raszuk wrote:
>      >      >      > Peter,
>      >      >      >
>      >      >      > Completely disagree.
>      >      >
>      >      >     that's what you seem to be doing from the beginning of
>     this
>      >      >     conversation ;)
>      >      >
>      >      >      >
>      >      >      > Real say enterprise networks are being build with
>     emulated
>      >     circuits
>      >      >      > (example VPWS). In one company I was working at it was
>      >     about 80% of
>      >      >      > emulated links all over the world in their WAN. Yes
>     for me
>      >     it was a
>      >      >      > shock as I did not realize how much this emulated links
>      >     took over
>      >      >     the
>      >      >      > world. In most geographies you even can not get any
>     link
>      >     of less
>      >      >     then
>      >      >      > 10Gbps to be real any more. Only emulated option is
>     on the
>      >     table.
>      >      >      >
>      >      >      > Emulated circuits run over someone's IP backbones.
>     You can
>      >      >     understand
>      >      >      > the consequences of this. Not only link delay changes a
>      >     lot but
>      >      >     you run
>      >      >      > into very interesting set of issues.
>      >      >
>      >      >     look at it from the opposite direction. The provider
>     of the
>      >     VPWS is the
>      >      >     one who can use this technology to guarantee the delay
>     bound
>      >     of the
>      >      >     WPWs
>      >      >     service. And if it does, the user of the WPWs would not
>      >     experience the
>      >      >     wild variation in the min delay.
>      >      >
>      >      >     So you have to apply right set of tools at right location.
>      >      >     Obviously, if
>      >      >     the min delay fluctuates wildly, one can not achieve delay
>      >     optimized
>      >      >     forwarding no matter what. That does not mean that the
>      >     network will get
>      >      >     unstable.
>      >      >
>      >      >     Peter
>      >      >
>      >      >
>      >      >      >
>      >      >      > Maybe you think of the backbones of the 90s or
>     2000s where
>      >     dark
>      >      >     fiber or
>      >      >      > SDH or SONET were in use for interconnects.
>      >      >      >
>      >      >      > Well no more. IETF came with such brilliant ideas to
>      >     emulate L2
>      >      >     over L3
>      >      >      > and here we go.
>      >      >      >
>      >      >      > Reality is not what we wish it to be.
>      >      >      >
>      >      >      > Cheers,
>      >      >      > R.
>      >      >      >
>      >      >      >
>      >      >      > On Thu, Mar 4, 2021 at 10:07 AM Peter Psenak
>      >      >      >     Robert,
>      >      >      >     On 03/03/2021 20:57, Robert Raszuk wrote:
>      >      >      >      > Peter,
>      >      >      >      >
>      >      >      >      >  >  that differ by few microsecond
>      >      >      >      >
>      >      >      >      > Really you normalize only single digit
>     microseconds ???
>      >      >      >      >
>      >      >      >      > What if link delay changes in
>     milliseconds scale ?
>      >     Do you
>      >      >     want to
>      >      >      >      > compute new topology every few milliseconds ?
>      >      >      >
>      >      >      >     let me repeat again.
>      >      >      >
>      >      >      >     Min delay is not something that changes every few
>      >     milliseconds
>      >      >      >     significantly. It's a semi static variable that
>      >     reflects the
>      >      >      >     property of
>      >      >      >     the underlying physical path. It only changes
>     when the
>      >      >     physical path
>      >      >      >     properties changes - e.g. the optical path
>     reroutes,
>      >     etc. We
>      >      >      >     deliberately picked Min delay for flex-algo
>     purposes
>      >     for this
>      >      >     semi
>      >      >      >     static property.
>      >      >      >
>      >      >      >     The small, non significant changes can be
>     filtered by the
>      >      >     normalization.
>      >      >      >
>      >      >      >     If the min delay changes significantly every few
>      >     milliseconds
>      >      >     there's
>      >      >      >     something wrong with the link itself - we have
>      >     standard dampening
>      >      >      >     mechanisms in LS protocols to deal with
>     unstable links
>      >     that
>      >      >     would kick
>      >      >      >     in. Similar to what we do if the link flaps
>     every few
>      >      >     milliseconds.
>      >      >      >
>      >      >      >      >
>      >      >      >      > Out of curiosity as this is not a secret -  What
>      >     are your
>      >      >     default
>      >      >      >     min
>      >      >      >      > delay normalization timers (if user does not
>     overwrite
>      >      >     with their
>      >      >      >     own).
>      >      >      >
>      >      >      >     there is no timer needed for the normalization
>     itself.
>      >      >      >
>      >      >      >     You are likely referring TWAMP computation interval
>      >     which is
>      >      >     30 sec,
>      >      >      >     with probes being sent every 3 seconds in our
>      >     implementation by
>      >      >      >     default,
>      >      >      >     if I'm not mistaken.
>      >      >      >
>      >      >      >     Normalization is applied to the value that come
>     from
>      >     the above.
>      >      >      >
>      >      >      >     thanks,
>      >      >      >     Peter
>      >      >      >
>      >      >      >
>      >      >      >
>      >      >      >      > Likewise how Junos or Arista normalizes it
>     today ?
>      >      >      >      >
>      >      >      >      > Thx,
>      >      >      >      > R.
>      >      >      >      >
>      >      >      >      > On Wed, Mar 3, 2021 at 7:41 PM Peter Psenak
>      >      >     <[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>
>      >      >      >      > <mailto:[email-masked]
>     <mailto:[email-masked] <mailto:[email-masked]>>>>>> wrote:
>      >      >      >      >
>      >      >      >      >     Tony,
>      >      >      >      >
>      >      >      >      >     On 03/03/2021 19:14, Tony Li wrote:
>      >      >      >      >      >
>      >      >      >      >      > Peter,
>      >      >      >      >      >
>      >      >      >      >      >>> There are several link types in use
>     that
>      >     exhibit
>      >      >     variable
>      >      >      >      >     delay: satellite links (e.g., Starlink),
>     microwave
>      >      >     links, and
>      >      >      >      >     ancient link layers that deliver reliability
>      >     through
>      >      >      >     retransmission.
>      >      >      >      >      >>> Any of these (and probably a lot
>     more) can
>      >     create a
>      >      >      >     noticeable
>      >      >      >      >     and measurable difference in TWAMP. That
>     would be
>      >      >     reflected
>      >      >      >     in an FA
>      >      >      >      >     metric change. If you imagine a
>     situation with
>      >      >     multiiple parallel
>      >      >      >      >     paths with nearly identical delays, you
>     can easily
>      >      >     imagine an
>      >      >      >      >     oscillatory scenario.   IMHO, this is an
>      >     outstanding
>      >      >     concern
>      >      >      >     with FA.
>      >      >      >      >      >> yes, and that is what I referred to
>     as "delay
>      >      >     normalization",
>      >      >      >      >     which can avoid that oscillation.
>      >      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >      > It can also negate the benefits of the
>      >     feature. One
>      >      >     might well
>      >      >      >      >     imagine that Starlink would want to follow a
>      >     min-delay
>      >      >     path for
>      >      >      >      >     optimality.  If the delay variations are
>      >     “normalized”
>      >      >     out of
>      >      >      >      >     existence, then the benefits are lost. 
>     The whole
>      >      >     point is to
>      >      >      >     track
>      >      >      >      >     the dynamics.
>      >      >      >      >
>      >      >      >      >     for all practical purposes that we use
>     it for,
>      >     the two
>      >      >     values
>      >      >      >     of min
>      >      >      >      >     delay that differ by few microsecond can be
>      >     treated as
>      >      >     same
>      >      >      >     without any
>      >      >      >      >     loss of functionality. So it's about the
>     required
>      >      >     normalization
>      >      >      >      >     interval
>      >      >      >      >     - something that can be controlled by
>     the user.
>      >      >      >      >
>      >      >      >      >     thanks,
>      >      >      >      >     Peter
>      >      >      >      >
>      >      >      >      >
>      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >      >>> Please note that I’m NOT
>     recommending that we
>      >      >     back away.
>      >      >      >      >     Rather, we should seek to solve the
>      >     long-standing issue of
>      >      >      >      >     oscillatory routing.
>      >      >      >      >      >>
>      >      >      >      >      >> not that I disagree. History tells
>     us that
>      >     the generic
>      >      >      >     case of
>      >      >      >      >     oscillation which is caused by the traffic
>      >     itself is a
>      >      >     hard
>      >      >      >     problem
>      >      >      >      >     to solve.
>      >      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >      > Any oscillation is difficult to
>     solve.  Positive
>      >      >     feedback
>      >      >      >      >     certainly can exacerbate the problem. But
>      >     solving hard
>      >      >      >     problems is
>      >      >      >      >     why we are here.
>      >      >      >      >      >
>      >      >      >      >      > Yours in control theory,
>      >      >      >      >      > Tony
>      >      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >      >
>      >      >      >      >
>      >      >      >
>      >      >
>      >
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 14:28:32 +0100

> https://tools.ietf.org/html/rfc8570#section-4.2

Maybe it is just me but in section 4.2 I see NOTHING about how often that
delay should be measured which was the point above.

Thx,
R.

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 14:30:31 +0100

On 04/03/2021 14:28, Robert Raszuk wrote:
> Maybe it is just me but in section 4.2 I see NOTHING about how often 
> that delay should be measured which was the point above.

it says it clearly - "over a configurable interval"

Peter

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Mar 2021 17:54:34 +0100

So it is undefined in the spec.

All along I was just asking what is your default probing interval.

It can be: 0 ms - means never probe.

It can be probe every N ms|s|min\hours etc ..means prove every N ms|s|min\h
etc ...

It can also be probe every time after link down -> up event but not more
often then M ms

Care to share that with the group ?

Thx,
R.

---

From: Peter Psenak <[email-masked]>
Date: Thu, 4 Mar 2021 18:06:12 +0100

> All along I was just asking what is your default probing interval.

I thought I have responded to that earlier today, please look at my 
first response today.

> It can be: 0 ms - means never probe.

one can even statically provision the min delay value, if that is what 
is preferred.

> 
> It can be probe every N ms|s|min\hours etc ..means prove every N 
> ms|s|min\h etc ...
> 

yes, all that control exist and most of it is under the control of the 
operator.

thanks,
Peter
> On Thu, Mar 4, 2021 at 2:30 PM Peter Psenak <[email-masked] 
>      >      > Is it minimum ever, is it min of the year, month, week,
>     day ... etc
>      >

---

From: Shraddha Hegde <[email-masked]>
Date: Fri, 5 Mar 2021 07:28:05 +0000

Tony,

Thanks for the comments.

Pls see inline for replies...

-----Original Message-----
From: Tony Li <[email-masked]> On Behalf Of Tony Li
Sent: Wednesday, March 3, 2021 5:30 AM
To: William Britto A J <[email-masked]>
Cc: [email-masked]; Rajesh M <[email-masked]>; Shraddha Hegde <[email-masked]>; DECRAENE Bruno IMT/OLN <[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints

Hi William,

<SH>  “Flexible Algorithms: Bandwidth, Delay, Metrics and Constraints”  looks like a good title

<SH> This is a good idea. Its useful to have flexibility in assigning various metrics to the link and be able to find flex-algo SPF paths for those metric types.
We will update the draft.

> [William] Shouldn’t there be only one Max link BW per link applicable for all applications? However, agree that it could still be in a common ASLA. Will add relevant text to clarify this.

That argument might well apply to all of our other link attributes as well.  Why do we need a separate administrative group for FlexAlgo?  Why do we need a separate set of SRLGs?  I have to confess that we have added a bit of complexity that I don’t think is very valuable.
<SH> Totally agree with you on this. 

 However, what’s done is done and there’s no point in arguing about it now.  
<SH> Arguing about this during working group adoption of such work didn’t help either! 

We should simply strive to be consistent with what’s already in place so that we don’t create even MORE complexity.
<SH> Agree.

<SH> Sure.

<SH> ok

8) Section 4.2. You write that an implementation "considers cumulative bandwidth of the parallel links while arriving at the metric for the link”. This seems a bit vague.  I think you’re trying to say that in interface group mode the bandwidth of an adjacency is the sum of the bandwidths of the individual links.  Typically today, if we have L3 parallel links they are encoded as separate adjacencies, complete with unique interface addresses.  How does interface group mode work with that? 
<SH> It continues to work the same way. I mean the topological representation does not change. It is going to be represented as multiple parallel 
Adjacencies. How the metric is assigned to these links differs in simple mode and interface group mode. In simple mode, single l3 link bandwidth is taken and metric is derived by using either of two modes of metric derivation (reference bw or staircase bandwidth thresholds). In  interface group mode, cumulative bandwidth of parallel links is used derive the metric (again either ref bw or staircase method can be used) and same metric is assigned to all parallel links.

Does each adjacency advertise the same metric based on the total bandwidth of all of the links?  
<SH> In automatic metric calculation method, each node derives the metric  based on advertised maximum link bandwidth .
In interface group mode,  metric is derived based on cumulative bandwidth of parallel  links and same metric is assigned for all parallel links.
In automatic metric calculation method, metric is not advertised.

In cases where operators do not want to use this automatic metric derivation, they can advertise bandwidth metric.
How this bandwidth  metric is assigned, whether same metric on all parallel links or different metric and actual metric value is all upto the operator.
When bandwidth metric sub-TLV is advertised on a link, simple mode or interface group mode does not come into play for that link. Bandwidth metric advertisement overrides, automatic metric derivation.

The discussion about reference bandwidths and the staircase mapping seems a bit confusing and redundant.  It appears to be orthogonal to the mode, correct? 
<SH> yes. It is orthogonal to the mode.

Why not describe it independently? More proofreading is necessary here.
<SH> agree. Will update in next revision.

9) Section 4.3.1. The reserved byte is wasteful. Your explanation of the round-off bandwidth is unclear.  Are we expecting dynamic bandwidth changes?  If so, that is an entire discussion unto itself.
<SH> yes we are expecting dynamic bandwidth changes. These dynamic bw changes are applicable to L2 bundled links only. When constituent links go up/down link bw width changes and that’s why you have the dynamic changes. Just to reiterate, these dynamic bw changes are not reflecting any path placement or bw utilization etc its is still total link bandwidth. We will add more text for round-off bw and also about bw changes.

10) Section 4.3.1. The reserved byte here is also wasteful.  Your explanation of how to use this is a bit unclear. 
<SH> ok will fix it.

 Are bandwidth ranges allowed to overlap?  I hope not. Is max[1] == min[2]?  I hope so, otherwise there is a gap in the ranges.  If there is a gap, and a link bandwidth falls in the gap, then what do we do?  Assuming that max[i] == min[i+1], then is there a point in carrying both values all of the time? What happens when a bandwidth exceeeds the highest max? Is below the smallest min?
<SH> yes we have scope to optimize the encodings.  We will post an updated encoding with clarifications.

Regards,
Tony

---

From: Tony Li <[email-masked]>
Date: Fri, 5 Mar 2021 09:28:10 -0800

Hi Shraddha,

> Adjacencies. How the metric is assigned to these links differs in simple mode and interface group mode. In simple mode, single l3 link bandwidth is taken and metric is derived by using either of tw…

So just to be very clear: we continue to see multiple, parallel adjacencies advertised. Each of them that has a metric that was computed using the sum of the bandwidths on the parallel links.

When one of those links fails, the remainder are then advertised with a revised metric that was computed using the sum of the bandwidths of the remaining links.

If this is correct, it would be good to state this very clearly. If you intend something else… please be equally clear.

> In automatic metric calculation method, metric is not advertised.

I don’t understand your last sentence. Section 4 is all automatic metric calculation. Why would you calculate the metric and then not advertise it using the Bandwidth Metric subTLV?

> When bandwidth metric sub-TLV is advertised on a link, simple mode or interface group mode does not come into play for that link. Bandwidth metric advertisement overrides, automatic metric derivati…

I see that I have misunderstood a big chunk of your intent here. Please add more clarifying text.  How does a node know whether we are using automatic mode or are being explicit in advertising our metrics? Is it on a per-link basis? Do we default to automatic mode if no bandwidth metric is advertised?

Is the metric automatically calculated if no bandwidth metric is advertised?

What does a remote node use as a bandwidth the compute the bandwidth of a link? I would assUme that it’s a Maximum Link Bandidth subTLV of the ASLA subTLV. If so, please be explicit.

If we are to use the bandwidth metric in a FAD, we need to specify that in the FAD.   This implies that we would need a new value (or set of values if we want multiple bandwidth metrics) for the Metric-Type field in the FAD subTLV.  I see that you’ve added a specific request for this in section 7.1, but it deserves mentioning somewhere in the text as well.

Regards,
Tony

---

From: William Britto A J <[email-masked]>
Date: Mon, 8 Mar 2021 13:14:03 +0000

Hi All,

Thanks for your valuable comments and feedback. We have tried to address most of them in the new version which has been uploaded now.

https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-01

Thanks,
William

From: Tony Li <[email-masked]> on behalf of Tony Li <[email-masked]>
Date: Friday, 5 March 2021 at 10:58 PM
To: Shraddha Hegde <[email-masked]>
Cc: William Britto A J <[email-masked]>, [email-masked] <[email-masked]>, Rajesh M <[email-masked]>, DECRAENE Bruno IMT/OLN <[email-masked]>
Subject: Re: [Lsr] New draft on Flex-Algorithm Bandwidth Constraints
[External Email. Be cautious of content]

Hi Shraddha,

Regards,
Tony