Subject: Alvaro Retana's Discuss on draft-ietf-rtgwg-backoff-algo-07: (with DISCUSS and COMMENT)

From: Alvaro Retana <[email-masked]>
Date: Mon, 19 Feb 2018 22:02:48 -0800

Alvaro Retana has entered the following ballot position for
draft-ietf-rtgwg-backoff-algo-07: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-rtgwg-backoff-algo/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I am balloting DISCUSS because I believe that this document presents an
incomplete and vague description of a specification, which (as is) won't result
in consistent implementations.  Consistency, through the specification of a
standard algorithm is used as the basis to justify this work:  "To allow
multi-vendor networks to have all routers delay their SPF computations for the
same duration, this document specifies a standard algorithm."

I am specifically and specially concerned about the fact that there are no
defaults or even suggested values:

   This document does not propose default values for the parameters because
   these values are expected to be context dependent. Implementations are free
   to propose their own default values.

If the whole purpose of standardizing an algorithm is for different
implementation to behave the same way and (specifically) "to have all routers
delay their SPF computations for the same duration", then not defining defaults
(and not being clear in the recommendations -- more on this below) makes the
specification incomplete and vague!

Section 6 tries to provide guidelines about defaults, but it falls short!

   In order to satisfy the goals stated in Section 2, operators are
   RECOMMENDED to configure delay intervals such that SPF_INITIAL_DELAY
   <= SPF_SHORT_DELAY and SPF_SHORT_DELAY <= SPF_LONG_DELAY.

Why are the operators not REQUIRED to meet that relationship?  Are there cases
when it is ok not to follow those guidelines?  Would (for example) the
SPF_LONG_DELAY ever be less than SPF_INITIAL_DELAY?

The other Normative Language in this section can't really be enforced, and
provide (at best) very weak guidance.

   When setting (default) values, one SHOULD consider the customers and
   their application requirements, the computational power of the
   routers, the size of the network, and, in particular, the number of
   IP prefixes advertised in the IGP, the frequency and number of IGP
   events, the number of protocols reactions/computations triggered by
   IGP SPF (e.g., BGP, PCEP, Traffic Engineering CSPF, Fast ReRoute
   computations).

"SHOULD consider..."  How can this statement be Normatively enforced?  Using
"SHOULD" implies that it is ok to only partially consider the list you
provided, or even a different set of criteria.

Based on the suggestions above, I can't imagine how a vendor can set default
values (even if "free to propose their own")...or how the average network
operator could configure anything beyond the numbers that you mentioned as
examples in the text.  For example, the average network operator might ask:
under the same circumstances, should my bigger routers (ones with presumably
more computational power) have lower or higher delays with respect to my
smaller routers?  ...

   Note that some or all of these factors may change over the life of
   the network.  In case of doubt, it's RECOMMENDED to play it safe and
   start with safe, i.e., longer timers.

How can "playing it safe" be Normatively enforced?

   For the standard algorithm to be effective in mitigating micro-loops,
   it is RECOMMENDED that all routers in the IGP domain, or at least all
   the routers in the same area/level, have exactly the same configured
   values.

[A similar statement is made in Section 7.]

If it is so important, why is consistency not mandatory?  IOW, why is it only
"RECOMMENDED" and not "REQUIRED"?  When is it ok to not do it?

Back to the point of this DISCUSS, the importance of consistent values is
clear!  Based on the experience of existing implementations, please specify
"safe" default values.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

[I know that some of these comments have been brought up in the SecDir and
GenArt reviews, but I have not seen an update yet.]

(1) Besides the lack of guidance (see above), there are several other
inconsistencies throughout the document:

(1.1) Section 3: "The HOLDDOWN_INTERVAL MUST be defaulted or configured to be
longer than the TIME_TO_LEARN_INTERVAL."  Which one, defaulted OR configured? 
Is it ok for the implementation to provide a default value that doesn't comply
with the expectation that the operator will configure the correct value?  It
seems to me that the definition of MUST doesn't fit with an option.

(1.2) Section 4: "If subsequent IGP events are received in a short period of
time (TIME_TO_LEARN_INTERVAL)...In this situation, we delay the routing
computation by SHORT_SPF_DELAY."  Note that Section 3 provided example values
for TIME_TO_LEARN_INTERVAL and SHORT_SPF_DELAY as 1 sec and 50-100 ms,
respectively.  If IGP events are received within the TIME_TO_LEARN_INTERVAL
window, then the SPF_DELAY ("delay between the first IGP event...and the start
of that routing table computation") set to SHORT_SPF_DELAY will be triggered
before TIME_TO_LEARN_INTERVAL...which means that the SPF run after
SHORT_SPF_DELAY won't cover all the changes.  Is that what you meant, or are
you assuming that the SPF_DELAY will start *after* the TIME_TO_LEARN_INTERVAL?

(1.3) Section 5.1: "LONG_WAIT: State reached after TIME_TO_LEARN_INTERVAL.  In
other words, state reached after TIME_TO_LEARN_INTERVAL in state SHORT_WAIT." 
But Section 3 defines TIME_TO_LEARN_INTERVAL as "the maximum duration typically
needed to learn all the IGP events related to a single component failure" --
why don't the events from that single failure start while in QUIET state?  OR
are you saying (in 5.1) that the TIME_TO_LEARN_INTERVAL is not measured from
the initial IGP Event?

(1.4) What is the relationship between HOLDDOWN_INTERVAL and the *_SPF_DELAY? 
I would assume that *_SPF_DELAY is always less than HOLDDOWN_INTERVAL, but the
document doesn't specify that relationship anywhere.

(1.5) Section 6: "All the parameters MUST be configurable
[I-D.ietf-isis-yang-isis-cfg] [I-D.ietf-ospf-yang] at the protocol instance
granularity."  Given that the references to the YANG models are listed as
Informative, what does that statement mean?  Is it a directive to what must be
included in the models?  What about implementations that don't use YANG (yet)?

(2) Section 5.4 (FSM Events)

(2.1) When will "Transition 7: SPF_TIMER expiration, while in QUIET" happen? 
Because when an IGP Event occurs in QUIET state, the FSM moves to SHORT_WAIT,
the SPF_TIMER should never expire in QUIET state.

(2.2) "Transition 3: LEARN_TIMER expiration." is defined between SHORT_WAIT and
LONG_WAIT, which (at first glance) seems to match how 5.1 defines
"LONG_WAIT:...state reached after TIME_TO_LEARN_INTERVAL in state SHORT_WAIT." 
However, the LEARN_TIMER in only started when an IGP event happens in
QUIET_STATE (transition 1).

(2.3) For completeness, the HOLDDOWN_TIMER expiration events (5 and 6) should
include resetting all the timers, just in case...and to be consistent with the
initialization description.

(3) From Section 3: "Routing table computation: Computation of the routing
table..."  This is a circular definition [1].  I'm sure the authors can figure
out a clear way to explain the meaning without using the terms being defined...

(4) "Note that previously implemented SPF delay algorithms counted the number
of SPF computations."  References?  Knowing that the references may not be
stable (pointing to a vendor's website), you might want to simply remove this
sentence and simply make the point in the paragraph as to why a time interval
is used.  Note that the point of this document is not to compare the
specification to "previously implemented algorithms".

(5) I am surprised that no other documents "must be read to understand or
implement the technology" [2] resulting in no Normative References (beyond
rfc2119).  I would think that at least the OSPF and ISIS specs should be
Normative.

(6) For the rtgwg-chairs/Shepherd: A quick scan of the mail archive shows that
this document wasn't reviewed by the ospf/isis WGs.  Given that what is
specified here affects the protocols directly, I would think that formal review
is needed.  [I note that a couple of the Chairs of the ospf/isis WGs are
co-authors of this document, and that a note was indeed sent when the -00
version of the individual draft was published -- still, it would have been nice
to at least explicitly inform of the progress.]

(7) Nit: "(e.g.  Loop Free Alternates..."   The closing parenthesis is missing.

(8) Nit: Please put a forward reference to 5.1 when the QUIET state is
mentioned in Section 4.

(9) Nit: s/QUIET_STATE/QUIET state.

[1] https://en.wikipedia.org/wiki/Circular_definition
[2] https://www.ietf.org/iesg/statement/normative-informative.html

---

From: <[email-masked]>
Date: Tue, 27 Feb 2018 10:43:41 +0000

Alvaro,

Thanks for your review and comments.

Please see inline [Bruno]

Also -08 has just been posted
https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-backoff-algo-08

Diff: https://www.ietf.org/rfcdiff?url2=draft-ietf-rtgwg-backoff-algo-08

 > -----Original Message-----
 > From: Alvaro Retana [mailto:[email-masked]]
 > Sent: Tuesday, February 20, 2018 7:03 AM
> 
 > in consistent implementations.
 
[Bruno] I believe that the specification of the algorithm itself is complete and clearly defined.

I agree with you that in a given deployment (IGP domain), timers needs to be consistent across nodes. This is indicated in the document in both sections 6 "Parameters" and 7 "Partial deployments"
In addition, section 6 specifically covers the requirements for implementations to provide overlapping parameters value ranges and granularity. (in order to avoid that implementation A allows values [2-10] while implementations B allows values [20-50].

Hence I believe that the specification is clear and detailed enough to provide interoperable implementations. And we do have 3 independent implementations, two of which have been tested in the same IGP area and were interoperable, i.e. provided consistent FSM states and SPF delay.

More below regarding default values, including a proposed text at the very end.

 > Consistency, through the specification of a

[Bruno] ok. Here I think that you are asking for different implementations to provide the same result "out of the box" i.e. in the absence of specific configuration from the network operator. (which IMHO is different from "interoperable implementations". Note that I'm not saying that my interpretation is the good one. I'm just trying to explain where I'm coming from).
I think that this would indeed be useful if this backoff-algo was enabled by default.
On the other hand, if it is enabled via explicit configuration from the network operator, I think that we can rely on the network operator to provide the complete configuration, i.e. with the value to use. Especially since this is discussed in both section 6 and 7.

 > SPF_LONG_DELAY ever be less than SPF_INITIAL_DELAY?
 
[Bruno] It is not required because the FSM would still work and provide consistent SPF delay across implementations.
If this is the choice to the network operator, SPF_LONG_DELAY could be less than SPF_INITIAL_DELAY, although I could not see the reason for this. But I also don't see a reason to forbid this configuration to a network operator.

 > provided, or even a different set of criteria.
 
[Bruno] Regarding this specific example, I'm fine with changing SHOULD to should.
But I don't think that this applies to all "other Normative Language" in this section. e.g., in the below sentence, I'd rather use a normative language.
"For the standard algorithm to be effective in mitigating micro-loops, it is RECOMMENDED that all routers in the IGP domain, or at least all the routers in the same area/level, have exactly the same configured values"

 > examples in the text.
 
[Bruno] As of today, and for the last 15+ years, all vendors have implemented a proprietary SPF algo, chosen and implemented default values, and that those values have been tuned by network operator. So I don't think that the argument "can't be done" really applies.
 
 >  For example, the average network operator might ask:

[Bruno] Within an IGP domain, section 6 clearly states that all nodes should use the same value.
Between IGP domains, CPU(s) capabilities may indeed be a reason to set different values. For example between a backbone and a backall area.

 > How can "playing it safe" be Normatively enforced?

[Bruno] If the question is whether normative language may be used to provide requirements to network operators, I defer to the IESG.
Otherwise, RFC 2119's definition of RECOMMENDED seems a good fit to me:

"3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course."

 > [A similar statement is made in Section 7.]

[Bruno] Indeed.
Section 7 is about the partial deployment of this specification. However, even with a full deployment, we do need consistent timers (discussed in section 6)

 > "RECOMMENDED" and not "REQUIRED"?  When is it ok to not do it?
 
[Bruno] IMHO, it is not required because the network would not melt down.
I would personally use the same parameters for all nodes on the IGP. However, for PE routers which are leaf in the IGP (vs providing transit within the IGP), I would personally find acceptable to have different values. Especially since PE tend to have lower CPU capabilities and may be older (some people would even say weaker for some boxes). Plus for those leaf PE, any topology change typically does not translate into any FIB update (as they just send all their traffic to the core network). So in short less motivation for fast IGP reaction, more motivation to reuse the churn, e.g. on the BGP side.
Also, at the (10) millisecond scale, I guess one could try to "trick" the algo in some cases. e.g., by setting smaller delays for nodes needing more times to perform the SPF computation/FIB installation.
 
 > clear!  Based on the experience of existing implementations, please specify
 
[Bruno] Ok.
First of all, I do think that the "best" default are likely to change over time (as both CPU power and customer requirements increase). Over the last 15+ years, this has already happened on some implementations https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/211432-Change-of-Default-OSPF-and-IS-IS-SPF-and.html  Also for the BGP protocol, this also happened for BGP Route Flap dampening parameters (cf RFC 2439 & 7196). They are also likely to be dependent of the segment market (e.g. backbone vs backhaul vs "pre-aggregation").

I would propose the following addition:
NEW:
If this SPF backoff algorithm is enabled by default, then in order to have consistent SPF delays between implementations with default configuration, the following default values SHOULD be implemented: 
INITIAL_SPF_DELAY 50 ms, SHORT_SPF_DELAY 200ms, LONG_SPF_DELAY: 5 000ms, TIME_TO_LEARN_INTERVAL 500ms, HOLDDOWN_INTERVAL 10 000ms.

[Bruno] both.
NEW: The HOLDDOWN_INTERVAL MUST be defaulted and configured to be longer than the TIME_TO_LEARN_INTERVAL."  Which one, defaulted OR configured?

(:s/or/and)

 > you assuming that the SPF_DELAY will start *after* the TIME_TO_LEARN_INTERVAL?

[Bruno]  Correct. This is what we mean.
Ideally, we'd like the time for the network to learn all IGP events for a given failure to be small. However, it's quite easy to miss this target for the whole network. e.g., in case of a node failure, one adjacency failure is detected by loss of IGP hellos, while others by loss of signal/BFD. In this case, indeed, the SPF run after SHORT_SPF_DELAY won't cover all changes. But waiting for the max time would be too conservative/slow in the average. (however, that's up for the network operator to configure its desired values)

 > why don't the events from that single failure start while in QUIET state? 

[Bruno] Indeed, the first event is handled by the QUIET state. However, FSM is immediately (0ms) transitioned to the SHORT_WAIT state. Hence all the TIME_TO_LEARN_INTERVAL   _time_ is spent in the SHORT_WAIT state. Or let's say ]0;TIME_TO_LEARN_INTERVAL] is spent in the SHORT_WAIT state.

I'm open to reformulation, but I'm not sure what to propose as to me all sentences seems valid. Can you propose some text that you would find cleared? 

 > OR

[Bruno] no we are not saying this. Quite the contrary. cf § "FSM events"

   Transition 1: IGP event, while in QUIET_STATE.

   Actions on event 1:

[...]

   o  Start LEARN_TIMER with TIME_TO_LEARN_INTERVAL.

[...]
 
 > document doesn't specify that relationship anywhere.
 
[Bruno] There is not required relationship.
This point has already been discussed in Ben's review.
(I agree that it seems a priori logical to have *_SPF_DELAY less than HOLDDOWN_INTERVAL. However, I don't see a reason to provide additional restrictions, which would then need to be enforced and handled as error conditions)

 > included in the models?  What about implementations that don't use YANG (yet)?
 
[Bruno] The statement was intended to means that "  All the parameters MUST be configurable at the protocol instance granularity." This applies independently of the way the parameters are configured. In this regards, I agree that the reference to yang models are not well placed or may be not even needed. I propose to remove them. IMHO, this is probably the other way around: the yang model should normatively reference this document.
 
 > the SPF_TIMER should never expire in QUIET state.
 
[Bruno] Good question.
2 answers:
- to make the FSM more robust, previous comments were on the side of adding all transitions
- if LONG_SPF_DELAY > HOLDDOWN_INTERVAL, then the SPF_TIMER initiated in the LONG_WAIT state will expire in the QUIET state.

 > QUIET_STATE (transition 1).
 
[Bruno] Same point as your question "(1.3) Section 5.1:" cf above.
 
 > initialization description.
 
[Bruno] Good point a priori. However, I don't think anything needs to be changed because:
- SPF_TIMER must not be reset as it can still run (if LONG_SPF_DELAY > HOLDDOWN_INTERVAL)
- HOLDDOWN_TIMER has already expired by hypothesis
- For transition 5 (from LONG_WAIT state), LEARN_TIMER has already expired by hypothesis.
- For transition 6 (from SHORT_WAIT state) LEARN_TIMER has already expired because of the following constraint: " The
   HOLDDOWN_INTERVAL MUST be defaulted or configured to be longer than the TIME_TO_LEARN_INTERVAL."

 > out a clear way to explain the meaning without using the terms being defined...
 
[Bruno] agreed based on the extract that you have picked. However here the goal is to say that:
- routing table is limited to the one handled by the IGP
- no distinction is made between the type of computation.

Hence we define the scope of the "routing table computation" rather than define what is a "routing table computation".

Actually that term and definition has already been quite debated in order to fit both IS-IS and OSPF, and the different king of computations that different implementations can make (e.g. full SPF, incremental SPF, PRC). Hence I'm reluctant to change the consensus reached.
I could propose the following change:

OLD:
   Routing table computation: Computation of the routing table, by the
   IGP, using the IGP LSDB.  No distinction is made between the type of
   computation performed. e.g., full SPF, incremental SPF, Partial Route
   Computation (PRC).

NEW
    Routing table computation, in this document, is scoped to the IGP. So this is the computation of the IG RIB, performed by the
   IGP, using the IGP LSDB.  No distinction is made between the type of
   computation performed. e.g., full SPF, incremental SPF, Partial Route
   Computation (PRC).

 > specification to "previously implemented algorithms".

I agree that the point is not to make comparison. However, I think that this is useful to explain the motivations for the design choices. (especially since this is a change compared to existing deployments)

OLD:
Note that previously implemented SPF delay algorithms counted the number of SPF computations. However, as all routers may receive the IGP events at different times, we cannot assume that all routers will perform the same number of SPF computations or that they will schedule them at the same time. For example, assuming that the SPF delay is 50 ms, router R1 may receive 3 IGP events (E1, E2, E3) in those 50 ms and hence will perform a single routing computation. While another router R2 may only receive 2 events (E1, E2) in those 50 ms and hence will schedule another routing computation when receiving E3.
That's why this document uses a time (TIME_TO_LEARN_INTERVAL) from the initial event detection/reception as opposed to counting the number of SPF computations to determine when the IGP is unstable.

NEW:
Note that in order to increase the consistency network wide, the algorithm uses a delay (TIME_TO_LEARN_INTERVAL) from the initial IGP event, rather than the number of SPF computation performed. Indeed, as all routers may receive the IGP events at different times, we cannot assume that all routers will perform the same number of SPF computations. For example, assuming that the SPF delay is 50 ms, router R1 may receive 3 IGP events (E1, E2, E3) in those 50 ms and hence will perform a single routing computation. While another router R2 may only receive 2 events (E1, E2) in those 50 ms and hence will schedule another routing computation when receiving E3.

 > rfc2119).  I would think that at least the OSPF and ISIS specs should be
 
[Bruno] I think that this document generally applies to any link state protocol. I'm not seen adherence to a specific link state protocol, be it OSPFv2, OSPFv3, IS-IS, RIFT possibly, LSVR possibly.
Definitely, someone implementing this document for the IS-IS protocol does not need to read the OSPF specification nor need OSPF implemented. Hence I don't feel that this fall under the definition of normative reference  "Normative references specify documents that must be read to understand or implement the technology in the new RFC, or whose technology must be present for the technology in the new RFC to work."

 > (7) Nit: "(e.g.  Loop Free Alternates..."   The closing parenthesis is missing.
 
[Bruno] thanks. done from previous review.
 
 > (8) Nit: Please put a forward reference to 5.1 when the QUIET state is
 
[Bruno] Done.
 
 > (9) Nit: s/QUIET_STATE/QUIET state.
 
[Bruno] thanks.

Thanks again for the careful review and comments.

--Bruno 

 > [2] https://www.ietf.org/iesg/statement/normative-informative.html

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Wed, 28 Feb 2018 18:36:25 +0000

Alvaro,

Please find below a follow up on one point

> From: DECRAENE Bruno IMT/OLN
 > Sent: Tuesday, February 27, 2018 11:44 AM

 [...]

[...]

[...]

 >    HOLDDOWN_INTERVAL MUST be defaulted or configured to be longer than the
 
Thinking more about this,
Given that transition 6 should not be used if the constraint is enforced on the timers, but transition 6 is still documented for completeness and robustness, I think it makes sense to talk about the LEARN_TIMER during this transition 6.
But actually, this is already in the draft:

   Transition 6: HOLDDOWN_TIMER expiration, while in SHORT_WAIT.

   Actions on event 6:
   o  Deactivate LEARN_TIMER.
   o  Transition to QUIET state.

https://tools.ietf.org/html/draft-ietf-rtgwg-backoff-algo-08#section-5.1
 
Thanks
--Bruno