Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Fri, 17 Apr 2020 20:40:49 +0000

Bruno -

Returning to this old thread...

The following is a generic description of how receive path for IS-IS PDUs functions today - based on examination of implementations on a variety of platforms from my employer. It is, for obvious reasons,  generic and intentionally omits any discussion of implementation details.
But it is hopefully detailed enough to illustrate some of the challenges in regards to providing real-time  accurate feedback to the control plane that could be used by a receiver based flow control mechanism.

Stage 1: Input Policing

As a first step, a policer operates at ingress to rate limit the number of packets which will be punted for local processing. This typically operates in an interface independent manner - limiting the total number of packets per second across all interfaces.
The policer may rate limit different classes of packets or simply limit all types of packets. A given platform may apply a limit specific to IS-IS PDUs or apply a limit to a class of packets (e.g., OSPF+BGP+IS-IS combined).

Stage 2: Punt Queue Shaping

Received packets are then placed in a queue for eventual transfer to the control plane. The number of queues varies. In some cases a single queue for all packet types is used. In other cases packets are placed on different queues based on packet classes.
Each queue is typically bounded to a maximum number of packets. As the queue usage approaches a limit, shaping policies are applied to prioritize certain packet types over others.
An upper limit specific to IS-IS packets may be employed - or a limit may be applied to a larger class of packet types of which IS-IS is only one of the packet types in the class.

Stage 3: Transfer to control Plane

Packets are then transferred to the control plane. Control plane input queues may map 1-1 with the data plane queues or map many to 1.
If the incoming packets are encapsulated (for example GRE) they may be transferred to a media specific control plane queue to process the encapsulation header. In some cases encapsulation may be processed in the data plane prior to transfer to the control plane.

Stage 4: Transfer to IS-IS

Packets are then transferred to a queue which is read directly by IS-IS. In the event there are multiple IS-IS instances, implementations may choose to have a shared queue which drives the execution of all instances or have instance specific queues filtered based on incoming interface.

A single queue is typically used for all interfaces and all IS-IS packet types. Subsequent processing may requeue packets based on packet type e.g., separating processing of hellos from processing of LSPs/SNPs.

*********

A receiver based flow control mechanism which attempts to make dynamic adjustments needs to obtain real-time feedback from one or more of the above stages. Monitoring the state of the input queue to IS-IS is easy to do, but would not account for drops at previous stages.
Obtaining feedback from earlier stages requires real-time updates from data plane to IS-IS in the control plane. This is much more challenging. Idiosyncrasies of platforms will have a significant impact on how to meaningfully interpret and use the data. How to integrate data from the various stages - especially when the numbers are not specific to IS-IS packets - is not intuitive.

https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03 provides no guidance on how information from the dataplane could be used.
As an alternative, it suggests that static parameters derived from offline tests could be advertised. But static parameters would necessarily have to be conservative as they would need to account for the worst case scenarios.

draft-ginsberg-lsr-isis-flooding-scale proposes dynamic flow control based on the state of the transmitter. In this model, there is no dependency on platform implementation. The number of unacknowledged LSPs sent on an interface is used as input to the flow control algorithm. This accounts for all reasons why a receiver may be slow to acknowledge without requiring knowledge of which stage(s) described above are affecting the receiver's ability to provide timely acknowledgements.

   Les

From: [email-masked] <[email-masked]>
Sent: Wednesday, February 26, 2020 11:03 AM
To: Les Ginsberg (ginsberg) <[email-masked]>
Cc: [email-masked]
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Les,

Please see inline[Bruno]

From: Lsr [mailto:[email-masked]] On Behalf Of Les Ginsberg (ginsberg)
Sent: Wednesday, February 19, 2020 3:32 AM
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Base protocol operation of the Update process tracks the flooding of
LSPs/interface and guarantees timer-based retransmission on P2P interfaces
until an acknowledgment is received.

Using this base protocol mechanism in combination with exponential backoff of the
retransmission timer provides flow control in the event of temporary overload
of the receiver.

This mechanism works without protocol extensions, is dynamic, operates
independent of the reason for delayed acknowledgment (dropped packets, CPU
overload), and does not require additional signaling during the overloaded
period.

This is consistent with the recommendations in RFC 4222 (OSPF).

Receiver-based flow control (as proposed in https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ )
requires protocol extensions and introduces additional signaling during
periods of high load. The asserted reason for this is to optimize throughput -
but there is no evidence that it will achieve this goal.

Mention has been made to TCP-like flow control mechanisms as a model - which
are indeed receiver based. However, there are significant differences between
TCP sessions and IGP flooding.

TCP consists of a single session between two endpoints. Resources
(primarily buffer space) for this session are typically allocated in the
control plane and current usage is easily measurable..

IGP flooding is point-to-multi-point, resources to support IGP flooding
involve both control plane queues and dataplane queues, both of which are
typically not per interface - nor even dedicated to a particular protocol
instance. What input is required to optimize receiver-based flow control is not fully specified.

https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ suggests (Section 5) that the values
to be advertised:

"use a formula based on an off line tests of
   the overall LSPDU processing speed for a particular set of hardware
   and the number of interfaces configured for IS-IS"

implying that the advertised value is intentionally not dynamic. As such,
it could just as easily be configured on the transmit side and not require
additional signaling. As a static value, it would necessarily be somewhat
conservative as it has to account for the worst case under the current
configuration - which means it needs to consider concurrent use of the CPU
and dataplane by all protocols/features which are enabled on a router - not all of whose
use is likely to be synchronized with peak IS-IS flooding load.
[Bruno] _Assuming_ that the parameters are static, those parameters

     *   are the same as the one implemented (configured) on multiple implementations, including the one from your employer. Now do you believe that those parameters can be determined?

        *   If yes, how do you do _today_ on your implementation? (this seems to contradict your statement that you have no way to figure out how to find the right value)
        *   If no, why did you implement those parameters, and ask network operator to configure them?
        *   There is also the option to reply: I don't know but don't care as I leave the issue to the network operator.

     *   can still provide some form of dynamicity, by using the PSNP as dynamic acknowledgement.
     *   are really dependent on the receiver, not the sender.

        *   the sender will never overload itself.
        *   The receiver has more information,  knowing its processing power (low end, high end, 80s, 20s (currently we are stuck with 20 years old value assuming the worst possible receiver (and worst there were, including with packet processing partly done in the control plane processor)), its expected IS-IS load (#neighbors), its preference for bursty LSP reception (high delay between IS-IS CPU allocation cycles, memory not an issue up to x kilo-octet...), its expected control plane load if IS-IS traffic has not higher priority over other control plane traffic...), it's expected level of QoS prioritization [1]

           *   [1] looks for "Extended SPD Headroom". E.g. "Since IGP and link stability are more tenuous and more crucial than BGP stability, such packets are now given the highest priority and are given extended SPD headroom with a default of 10 packets. This means that these packets are not dropped if the size of the input hold queue is lower than 185 (input queue default size + spd headroom size + spd extended headroom)."

              *   And this is for distributed architecture, 15 years ago. So what about using the above number (in the router configuration), applies Tony's proposal (*oversubscription/number of IS neighbhors), and advertise this value to your LSP sender?

[1] https://www.cisco.com/c/en/us/support/docs/routers/12000-series-routers/29920-spd.html

  *
--Bruno

Unless a good case can be made as to why transmit-based flow control is not a good
fit and why receiver-based flow control is demonstrably better, it seems
unnecessary to extend the protocol.

    Les

From: Lsr <[email-masked]<mailto:[email-masked]>> On Behalf Of Les Ginsberg (ginsberg)
Sent: Tuesday, February 18, 2020 6:25 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Two recent drafts advocate for the use of faster LSP flooding speeds in IS-IS:

https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/
https://datatracker.ietf.org/doc/draft-ginsberg-lsr-isis-flooding-scale/

There is strong agreement on two key points:

1)Modern networks require much faster flooding speeds than are commonly in use today

2)To deploy faster flooding speeds safely some form of flow control is needed

The key point of contention between the two drafts is how flow control should be implemented.

https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ advocates for a receiver based flow control where the receiver advertises in hellos the parameters which indicate the rate/burst size which the receiver is capable of supporting on the interface. Senders are required to limit the rate of LSP transmission on that interface in accordance with the values advertised by the receiver.

https://datatracker.ietf.org/doc/draft-ginsberg-lsr-isis-flooding-scale/  advocates for a transmit based flow control where the transmitter monitors the number of unacknowledged LSPs sent on each interface and implements a backoff algorithm to slow the rate of sending LSPs based on the length of the per interface unacknowledged queue.

While other differences between the two drafts exist, it is fair to say that if agreement could be reached on the form of flow control  then it is likely other issues could be resolved easily.

This email starts the discussion regarding the flow control issue.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

---

From: Tony Przygienda <[email-masked]>
Date: Sat, 18 Apr 2020 10:27:47 -0700

roughly same experience over couple commercial, heavy-lifting vendors on my
side. modulo that implementation in user space internally sometimes manages
queues per interface & different queues in/out per ISIS packet types +
plays with size of socket buffers across interfaces (don't forget, that's
not infinite resource either). As Les said, each stage can loose
independently and there are no feedbacks in any implementaiton/platform I
know of between the different buffering stages (albeit they could be built
on some platforms but that would make the protocol code mostly
nightmarishly platform dependent). Observe that each stage has different
timing characterics in terms of processing & that e.g. line card load & CPU
load can change the characterics very drastically (that has direct
implications on what the max. capacity to "soak packets in" is and with
that what the "acceptable flooding rate" is). It's not unheard of on low
end, highly loaded platforms to introduce 10s of millisecs delay between
packet hitting the link and being read out the socket. On high end or very
simple boxes you may be in tens of microsecs, that's a cool 3-4 orders of
magnitude. With the advent of VMs and containers things got _much_ worse,
both in variation and range. I  personally spent a day puzzling over an
impossible failing test case in a virtual env until I measured the delay
between one node sending a packet and reading it out on the other side in
my protocol implementation in this setup. It was _5_ seconds ....

-- tony

On Fri, Apr 17, 2020 at 1:42 PM Les Ginsberg (ginsberg) <ginsberg=
[email-masked]> wrote:

> Bruno –
>
>
>
> Returning to this old thread…
>
>
>
> The following is a generic description of how receive path for IS-IS PDUs
> functions today – based on examination of implementations on a variety of
> platforms from my employer. It is, for obvious reasons,  generic and
> intentionally omits any discussion of implementation details.
>
> But it is hopefully detailed enough to illustrate some of the challenges
> in regards to providing real-time  accurate feedback to the control plane
> that could be used by a receiver based flow control mechanism.
>
>
>
> Stage 1: Input Policing
>
>
>
> As a first step, a policer operates at ingress to rate limit the number of
> packets which will be punted for local processing. This typically operates
> in an interface independent manner - limiting the total number of packets
> per second across all interfaces.
>
> The policer may rate limit different classes of packets or simply limit
> all types of packets. A given platform may apply a limit specific to IS-IS
> PDUs or apply a limit to a class of packets (e.g., OSPF+BGP+IS-IS combined).
>
>
>
> Received packets are then placed in a queue for eventual transfer to the
> control plane. The number of queues varies. In some cases a single queue
> for all packet types is used. In other cases packets are placed on
> different queues based on packet classes.
>
> Each queue is typically bounded to a maximum number of packets. As the
> queue usage approaches a limit, shaping policies are applied to prioritize
> certain packet types over others.
>
> An upper limit specific to IS-IS packets may be employed - or a limit may
> be applied to a larger class of packet types of which IS-IS is only one of
> the packet types in the class.
>
>
>
> Packets are then transferred to the control plane. Control plane input
> queues may map 1-1 with the data plane queues or map many to 1.
>
> If the incoming packets are encapsulated (for example GRE) they may be
> transferred to a media specific control plane queue to process the
> encapsulation header. In some cases encapsulation may be processed in the
> data plane prior to transfer to the control plane.
>
>
>
> Packets are then transferred to a queue which is read directly by IS-IS.
> In the event there are multiple IS-IS instances, implementations may choose
> to have a shared queue which drives the execution of all instances or have
> instance specific queues filtered based on incoming interface.
>
>
>
> A single queue is typically used for all interfaces and all IS-IS packet
> types. Subsequent processing may requeue packets based on packet type e.g.,
> separating processing of hellos from processing of LSPs/SNPs.
>
>
>
> *********
>
>
>
> A receiver based flow control mechanism which attempts to make dynamic
> adjustments needs to obtain real-time feedback from one or more of the
> above stages. Monitoring the state of the input queue to IS-IS is easy to
> do, but would not account for drops at previous stages.
>
> Obtaining feedback from earlier stages requires real-time updates from
> data plane to IS-IS in the control plane. This is much more challenging.
> Idiosyncrasies of platforms will have a significant impact on how to
> meaningfully interpret and use the data. How to integrate data from the
> various stages – especially when the numbers are not specific to IS-IS
> packets – is not intuitive.
>
>
>
> https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03
> provides no guidance on how information from the dataplane could be used.
>
> As an alternative, it suggests that static parameters derived from offline
> tests could be advertised. But static parameters would necessarily have to
> be conservative as they would need to account for the worst case scenarios.
>
>
>
> draft-ginsberg-lsr-isis-flooding-scale proposes dynamic flow control based
> on the state of the transmitter. In this model, there is no dependency on
> platform implementation. The number of unacknowledged LSPs sent on an
> interface is used as input to the flow control algorithm. This accounts for
> all reasons why a receiver may be slow to acknowledge without requiring
> knowledge of which stage(s) described above are affecting the receiver’s
> ability to provide timely acknowledgements.
>
>
>
>    Les
>
>
>
>
>
> *From:* [email-masked] <[email-masked]>
> *Sent:* Wednesday, February 26, 2020 11:03 AM
> *To:* Les Ginsberg (ginsberg) <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* RE: Flow Control Discussion for IS-IS Flooding Speed
>
>
>
> Les,
>
>
>
> Please see inline[Bruno]
>
>
>
> *From:* Lsr [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Les Ginsberg (ginsberg)
> *Sent:* Wednesday, February 19, 2020 3:32 AM
> *To:* [email-masked]
> *Subject:* Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed
>
>
>
> Receiver-based flow control (as proposed in
> https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ )
>
> suggests (Section 5) that the values
>
> to be advertised:
>
>
>
> [Bruno] _*Assuming*_ that the parameters are static, those parameters
>
>    - are the same as the one implemented (configured) on multiple
>       implementations, including the one from your employer. Now do you believe
>       that those parameters can be determined?
>
>
>    - If yes, how do you do _*today*_ on your implementation? (this seems
>          to contradict your statement that you have no way to figure out how to find
>          the right value)
>          - If no, why did you implement those parameters, and ask network
>          operator to configure them?
>          - There is also the option to reply: I don’t know but don’t care
>          as I leave the issue to the network operator.
>
>
>    - can still provide some form of dynamicity, by using the PSNP as
>       dynamic acknowledgement.
>       - are really dependent on the receiver, not the sender.
>
>
>    - the sender will never overload itself.
>          - The receiver has more information,  knowing its processing
>          power (low end, high end, 80s, 20s (currently we are stuck with 20 years
>          old value assuming the worst possible receiver (and worst there were,
>          including with packet processing partly done in the control plane
>          processor)), its expected IS-IS load (#neighbors), its preference for
>          bursty LSP reception (high delay between IS-IS CPU allocation cycles,
>          memory not an issue up to x kilo-octet…), its expected control plane load
>          if IS-IS traffic has not higher priority over other control plane
>          traffic…), it’s expected level of QoS prioritization [1]
>
>
>    - [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and link
>             stability are more tenuous and more crucial than BGP stability, such
>             packets are now given the highest priority and are given extended SPD
>             headroom with a default of 10 packets. This means that these packets are
>             not dropped if the size of the input hold queue is lower than 185 (input
>             queue default size + spd headroom size + spd extended headroom).”
>
>
>    - And this is for distributed architecture, 15 years ago. So what
>                about using the above number (in the router configuration), applies Tony’s
>                proposal (*oversubscription/number of IS neighbhors), and advertise this
>                value to your LSP sender?
>
>
>
> [1]
> https://www.cisco.com/c/en/us/support/docs/routers/12000-series-routers/29920-spd.html
>
>
>
>    -
>
> --Bruno
>
>
>
>
>
> *From:* Lsr <[email-masked]> *On Behalf Of *Les Ginsberg (ginsberg)
> *Sent:* Tuesday, February 18, 2020 6:25 PM
> *To:* [email-masked]
> *Subject:* [Lsr] Flow Control Discussion for IS-IS Flooding Speed
>
>
>
> advocates for a receiver based flow control where the receiver advertises
> in hellos the parameters which indicate the rate/burst size which the
> receiver is capable of supporting on the interface. Senders are required to
> limit the rate of LSP transmission on that interface in accordance with the
> values advertised by the receiver.
>
>
>
>  advocates for a transmit based flow control where the transmitter monitors
> the number of unacknowledged LSPs sent on each interface and implements a
> backoff algorithm to slow the rate of sending LSPs based on the length of
> the per interface unacknowledged queue.
>
>
>
> While other differences between the two drafts exist, it is fair to say
> that if agreement could be reached on the form of flow control  then it is
> likely other issues could be resolved easily.
>
>
>
> _______________________________________________
> Lsr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lsr
>

---

From: <[email-masked]>
Date: Mon, 20 Apr 2020 11:47:00 +0000

Les,

After nearly 2 months, can we expect an answer from your side?

More specifically, the below question

[Bruno] _Assuming_ that the parameters are static, the parameters proposed in draft-decraene-lsr-isis-flooding-speed are the same as the one implemented (configured) on multiple implementations, including the one from your employer.
Now do you believe that those parameters can be determined?

§  If yes, how do you do _today_ on your implementation? (this seems to contradict your statement that you have no way to figure out how to find the right value)

§  If no, why did you implement those parameters, and ask network operator to configure them?

Thank you,
--Bruno

From: DECRAENE Bruno TGI/OLN
Sent: Wednesday, February 26, 2020 8:03 PM
To: 'Les Ginsberg (ginsberg)'
Cc: [email-masked]
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Les,

Please see inline[Bruno]

To: [email-masked]

o   are the same as the one implemented (configured) on multiple implementations, including the one from your employer. Now do you believe that those parameters can be determined?

§  If yes, how do you do _today_ on your implementation? (this seems to contradict your statement that you have no way to figure out how to find the right value)

§  If no, why did you implement those parameters, and ask network operator to configure them?

§  There is also the option to reply: I don't know but don't care as I leave the issue to the network operator.

o   can still provide some form of dynamicity, by using the PSNP as dynamic acknowledgement.

o   are really dependent on the receiver, not the sender.

§  the sender will never overload itself.

§  The receiver has more information,  knowing its processing power (low end, high end, 80s, 20s (currently we are stuck with 20 years old value assuming the worst possible receiver (and worst there were, including with packet processing partly done in the control plane processor)), its expected IS-IS load (#neighbors), its preference for bursty LSP reception (high delay between IS-IS CPU allocation cycles, memory not an issue up to x kilo-octet...), its expected control plane load if IS-IS traffic has not higher priority over other control plane traffic...), it's expected level of QoS prioritization [1]

·          [1] looks for "Extended SPD Headroom". E.g. "Since IGP and link stability are more tenuous and more crucial than BGP stability, such packets are now given the highest priority and are given extended SPD headroom with a default of 10 packets. This means that these packets are not dropped if the size of the input hold queue is lower than 185 (input queue default size + spd headroom size + spd extended headroom)."

o   And this is for distributed architecture, 15 years ago. So what about using the above number (in the router configuration), applies Tony's proposal (*oversubscription/number of IS neighbhors), and advertise this value to your LSP sender?

-
--Bruno

    Les

From: Lsr <[email-masked]> On Behalf Of Les Ginsberg (ginsberg)
Sent: Tuesday, February 18, 2020 6:25 PM
To: [email-masked]
Subject: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

---

From: <[email-masked]>
Date: Mon, 20 Apr 2020 12:20:40 +0000

Les,

Thank you for this first answer. This is inline with my understanding of the behavior.
I wish some of the behaviors and values (e.g. queue size, rate limiting) could be shared. At minimum a range of typical values for platform which are not end of sale. I don't think this is secret. Actually, this is something that network provider would need in order to configure the right value for the parameters which are implemented on the platforms of your employer, so this should be on the public documentation of your employer. One could also think that you may be proud of your implementations and eager to show that it performs well.

Waiting some more details, as per you below email, the IS-IS receiver does have a queue, sometimes dedicated to IS-IS, but in general relatively dedicated to very important and time sensitive traffic to the control plane. Details are indeed implementation specific. But in general, this queue is designed to protect the IS-IS traffic from lower priority traffic, e.g. bursty BGP. So can we assume that the receiver have (or at least may have) such a queue, and work with this?

--Bruno

From: Les Ginsberg (ginsberg) [mailto:[email-masked]]
Sent: Friday, April 17, 2020 10:41 PM
To: DECRAENE Bruno TGI/OLN
Cc: [email-masked]
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Bruno -

Stage 1: Input Policing

*********

   Les

Les,

Please see inline[Bruno]

·
--Bruno

    Les

Thank you.

---

From: <[email-masked]>
Date: Mon, 20 Apr 2020 12:38:26 +0000

Thank you Tony for the additional information.
Originally, draft-decraene-spring-srv6-vlsid focused on static parameters, for e.g. input (receiver) queue size, so let’s discuss base on this.
I don’t think anyone spoke about infinite resources. Quite the contrary, our goal is to not exceed the resources of the receiver. Could you please share an approximation/range of the buffer size associated to these IS-IS queues? (either per interface or per system)

Thanks for the extra information regarding the timing within the device. The feedback loop between the LSP sender and the receiver need to also account for the link RTT, IS-IS scheduling and processing, possibly the LSP acknowledgement time if we want to more quickly retransmit lost LSPs…so I don’t think that in the general case we are thinking about micro seconds or even 10s of milli-second. At least I’m not working with this assumption. That’s why advertising a receive window seem interesting as it allows for sending multiple LSPs without waiting for the feedback loop.

--Bruno

From: Tony Przygienda [mailto:[email-masked]]
Sent: Saturday, April 18, 2020 7:28 PM
To: Les Ginsberg (ginsberg)
Cc: DECRAENE Bruno TGI/OLN; [email-masked]
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

roughly same experience over couple commercial, heavy-lifting vendors on my side. modulo that implementation in user space internally sometimes manages queues per interface & different queues in/out per ISIS packet types + plays with size of socket buffers across interfaces (don't forget, that's not infinite resource either). As Les said, each stage can loose independently and there are no feedbacks in any implementaiton/platform I know of between the different buffering stages (albeit they could be built on some platforms but that would make the protocol code mostly nightmarishly platform dependent). Observe that each stage has different timing characterics in terms of processing & that e.g. line card load & CPU load can change the characterics very drastically (that has direct implications on what the max. capacity to "soak packets in" is and with that what the "acceptable flooding rate" is). It's not unheard of on low end, highly loaded platforms to introduce 10s of millisecs delay between packet hitting the link and being read out the socket. On high end or very simple boxes you may be in tens of microsecs, that's a cool 3-4 orders of magnitude. With the advent of VMs and containers things got _much_ worse, both in variation and range. I  personally spent a day puzzling over an impossible failing test case in a virtual env until I measured the delay between one node sending a packet and reading it out on the other side in my protocol implementation in this setup. It was _5_ seconds ....

-- tony

On Fri, Apr 17, 2020 at 1:42 PM Les Ginsberg (ginsberg) <ginsberg=[email-masked]<mailto:[email-masked]>> wrote:
Bruno –

The following is a generic description of how receive path for IS-IS PDUs functions today – based on examination of implementations on a variety of platforms from my employer. It is, for obvious reasons,  generic and intentionally omits any discussion of implementation details.
But it is hopefully detailed enough to illustrate some of the challenges in regards to providing real-time  accurate feedback to the control plane that could be used by a receiver based flow control mechanism.

Stage 1: Input Policing

*********

A receiver based flow control mechanism which attempts to make dynamic adjustments needs to obtain real-time feedback from one or more of the above stages. Monitoring the state of the input queue to IS-IS is easy to do, but would not account for drops at previous stages.
Obtaining feedback from earlier stages requires real-time updates from data plane to IS-IS in the control plane. This is much more challenging. Idiosyncrasies of platforms will have a significant impact on how to meaningfully interpret and use the data. How to integrate data from the various stages – especially when the numbers are not specific to IS-IS packets – is not intuitive.

draft-ginsberg-lsr-isis-flooding-scale proposes dynamic flow control based on the state of the transmitter. In this model, there is no dependency on platform implementation. The number of unacknowledged LSPs sent on an interface is used as input to the flow control algorithm. This accounts for all reasons why a receiver may be slow to acknowledge without requiring knowledge of which stage(s) described above are affecting the receiver’s ability to provide timely acknowledgements.

   Les

From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, February 26, 2020 11:03 AM
To: Les Ginsberg (ginsberg) <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Les,

Please see inline[Bruno]

implying that the advertised value is intentionally not dynamic. As such,
it could just as easily be configured on the transmit side and not require
additional signaling. As a static value, it would necessarily be somewhat
conservative as it has to account for the worst case under the current
configuration - which means it needs to consider concurrent use of the CPU
and dataplane by all protocols/features which are enabled on a router - not all of whose
use is likely to be synchronized with peak IS-IS flooding load.
[Bruno] _Assuming_ that the parameters are static, those parameters
o    are the same as the one implemented (configured) on multiple implementations, including the one from your employer. Now do you believe that those parameters can be determined?
§  If yes, how do you do _today_ on your implementation? (this seems to contradict your statement that you have no way to figure out how to find the right value)
§  If no, why did you implement those parameters, and ask network operator to configure them?
§  There is also the option to reply: I don’t know but don’t care as I leave the issue to the network operator.
o    can still provide some form of dynamicity, by using the PSNP as dynamic acknowledgement.
o    are really dependent on the receiver, not the sender.
§  the sender will never overload itself.
§  The receiver has more information,  knowing its processing power (low end, high end, 80s, 20s (currently we are stuck with 20 years old value assuming the worst possible receiver (and worst there were, including with packet processing partly done in the control plane processor)), its expected IS-IS load (#neighbors), its preference for bursty LSP reception (high delay between IS-IS CPU allocation cycles, memory not an issue up to x kilo-octet…), its expected control plane load if IS-IS traffic has not higher priority over other control plane traffic…), it’s expected level of QoS prioritization [1]
·         [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and link stability are more tenuous and more crucial than BGP stability, such packets are now given the highest priority and are given extended SPD headroom with a default of 10 packets. This means that these packets are not dropped if the size of the input hold queue is lower than 185 (input queue default size + spd headroom size + spd extended headroom).”
o    And this is for distributed architecture, 15 years ago. So what about using the above number (in the router configuration), applies Tony’s proposal (*oversubscription/number of IS neighbhors), and advertise this value to your LSP sender?

·
--Bruno

    Les

Thank you.
_______________________________________________
Lsr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lsr

---

From: [email-masked]
Date: Mon, 20 Apr 2020 09:28:37 -0700

Bruno,

> Waiting some more details, as per you below email, the IS-IS receiver does have a queue, sometimes dedicated to IS-IS, but in general relatively dedicated to very important and time sensitive traffic to the control plane. Details are indeed implementation specific. But in general, this queue is designed to protect the IS-IS traffic from lower priority traffic, e.g. burstyBGP.. So can we assume that the receiver have (or at least may have) such a queue, and work with this?

That would be ideal, but probably not realistic. Not all implementations are going to have separate queues for BGP and IS-IS.

IMHO, that’s a very good thing, of course, but not all merchant silicon is quite that sophisticated. Yet. And it takes years to change, so it’s best that we proceed without.

Tony

---

From: [email-masked]
Date: Mon, 20 Apr 2020 09:49:54 -0700

Gentlebeings,

This discussion is producing far more heat than light.  Can we please refocus our attentions?

I think that we all agree that the legacy parameters are no longer serving us well and that we need to reconsider our flooding parameters and mechanisms.

If we fail to reach a constructive consensus, we will end up with a protocol that severely underperforms, hurting our end user experience and allowing other
protocols to supplant what we’ve worked so hard for. It behooves us to all work together to find a common ground that allows all implementations to 
converge rapidly.

At this point, arguing back and forth does not seem to be helping.

Our goal is to enable rapid flooding of a large LSP database. We need everyone to be able to do this. While it may help an implementation to be 
rapid when operating with just itself, in today’s industrial environment, not being able to do this across implementations is not helpful. We need
all implementations to be able to be performant.

We have iterated on the points of the discussion repeatedly. Further repetition is not helpful. What is lacking is experimentation and facts. 
We need the results of running code. Packet traces with analysis of flow control mechanisms. Demonstrations of how particular parameters and
mechanisms perform across implementations and across hardware platform classes. Comparisons of our flooding mechanisms with the throughput 
of other transport protocols.

What is going to help all platforms be their best?

Don’t just tell us what you think: prove it.

The competition is not the other implementation. It is other protocols who will supplant everything.

Regards,
Tony

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Tue, 21 Apr 2020 21:47:58 +0000

Bruno -

You have made an assumption that historically vendors have tuned LSP transmission rates to a platform specific value.
That certainly is not true in the case of my employer (happy to hear what other vendors have been doing for the past 20 years).

The default value is based on minimumBroadcastLSPTransmissionInterval specified in ISO10589.
A knob is available to allow tuning (faster or slower) in a given deployment - though in my experience this knob is rarely used.

We already discuss in https://tools.ietf.org/html/draft-ginsberg-lsr-isis-flooding-scale-02#section-2 that this common interpretation isn't the most appropriate, but historically the concern has been to avoid flooding too fast - not to optimize flooding speed.
Obviously, we are revisiting that approach and saying it needs to change - which is something I think we have consensus on.

I have provided a description in my recent response as to why it is difficult to derive an optimal value on a per platform basis. You may disagree - but please do not claim that we actually have been doing this for years. We haven't been.

  Les

From: [email-masked] <[email-masked]>
Sent: Monday, April 20, 2020 4:47 AM
To: Les Ginsberg (ginsberg) <[email-masked]>
Cc: [email-masked]
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Les,

Thank you,
--Bruno

Les,

Please see inline[Bruno]

-
--Bruno

    Les

Thank you.

---

From: Tony Przygienda <[email-masked]>
Date: Tue, 21 Apr 2020 16:19:13 -0700

neither am I aware of anything like this (i.e. per platform/product
flooding rate constants) in any major vendor stack for whatever that's
worth. It's simply unmaintanable, point. All major vendors have extensive
product lines over so many changing hardware configuration/setups it is
simply not viable to attempt precise measurements (and even then, user
changing config can throw the stuff off in a millisecond). There may have
been here and there per deployment scenario some "recommended config" (not
something I immediately recall either) but that means very fixed
configuration of things & how they go into networks and even then I'm not
aware of anyone having had a "precise model of the chain in the box". yes,
probes to measure lots and lots of stuff in the boxes exist but again,
those are chip/linecard/backplane/chassis/routing engine specific and
mostly used in complex test/peformance scenarios and not to derive some
kind of equations that can predict anything much ...

Bruno, if you're so deeply interested in that stuff we can talk 1:1
off-line about implementation work on rift towards adapatable flooding rate
(algorithm you see in the -02 draft Les put out is a _very rough_
approximation of that BTW. I joined as co-author after we had some very
fruitful discussions and I consider the draft close to what can be
_realistically_ done  today in ISIS. I don't consider further details
generic enough to merit wide forum discussions). And RIFT put a couple of
things into packet formats we can't put into ISIS (I talked with Les about
it) to improve the adaptability of the flooding rate BTW and some
interesting, coarse indication from receiver. Again, this is not a constant
that is calculated, it's all adaptive driven almost completely from the
transmitter side and the feedback it gathers.

all my very own 2c

-- tony

On Tue, Apr 21, 2020 at 2:48 PM Les Ginsberg (ginsberg) <ginsberg=
[email-masked]> wrote:

> Bruno –
>
>
>
> You have made an assumption that historically vendors have tuned LSP
> transmission rates to a platform specific value.
>
> That certainly is not true in the case of my employer (happy to hear what
> other vendors have been doing for the past 20 years).
>
>
>
> A knob is available to allow tuning (faster or slower) in a given
> deployment – though in my experience this knob is rarely used.
>
>
>
> We already discuss in
> https://tools.ietf.org/html/draft-ginsberg-lsr-isis-flooding-scale-02#section-2
> that this common interpretation isn’t the most appropriate, but
> historically the concern has been to avoid flooding too fast – not to
> optimize flooding speed.
>
> Obviously, we are revisiting that approach and saying it needs to change –
> which is something I think we have consensus on.
>
>
>
> I have provided a description in my recent response as to why it is
> difficult to derive an optimal value on a per platform basis. You may
> disagree – but please do not claim that we actually have been doing this
> for years. We haven’t been.
>
>
>
>   Les
>
>
>
> *Sent:* Monday, April 20, 2020 4:47 AM
> [Bruno] _Assuming_ that the parameters are static, the parameters proposed
> in draft-decraene-lsr-isis-flooding-speed are the same as the one
> implemented (configured) on multiple implementations, including the one
> from your employer.
>
> §  If yes, how do you do _today_ on your implementation? (this seems to
> contradict your statement that you have no way to figure out how to find
> the right value)
>
>
>
> *From:* DECRAENE Bruno TGI/OLN
> *Sent:* Wednesday, February 26, 2020 8:03 PM
> *To:* 'Les Ginsberg (ginsberg)'
> *Cc:* [email-masked]
> o   are the same as the one implemented (configured) on multiple
> §  If yes, how do you do _*today*_ on your implementation? (this seems to
> contradict your statement that you have no way to figure out how to find
> the right value)
>
> §  The receiver has more information,  knowing its processing power (low
> end, high end, 80s, 20s (currently we are stuck with 20 years old value
> assuming the worst possible receiver (and worst there were, including with
> packet processing partly done in the control plane processor)), its
> expected IS-IS load (#neighbors), its preference for bursty LSP reception
> (high delay between IS-IS CPU allocation cycles, memory not an issue up to
> x kilo-octet…), its expected control plane load if IS-IS traffic has not
> higher priority over other control plane traffic…), it’s expected level of
> QoS prioritization [1]
>
> ·          [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and
> link stability are more tenuous and more crucial than BGP stability, such
> o   And this is for distributed architecture, 15 years ago. So what about
> using the above number (in the router configuration), applies Tony’s

---

From: <[email-masked]>
Date: Wed, 22 Apr 2020 18:03:23 +0000

Tony, all,

Thanks Tony for the technical and constructive feedback.
Please inline [Bruno]

Sent: Wednesday, April 22, 2020 1:19 AM

neither am I aware of anything like this (i.e. per platform/product flooding rate constants) in any major vendor stack for whatever that's worth. It's simply unmaintanable, point. All major vendors have extensive product lines over so many changing hardware configuration/setups it is simply not viable to attempt precise measurements (and even then, user changing config can throw the stuff off in a millisecond). There may have been here and there per deployment scenario some "recommended config" (not something I immediately recall either) but that means very fixed configuration of things & how they go into networks and even then I'm not aware of anyone having had a "precise model of the chain in the box". yes, probes to measure lots and lots of stuff in the boxes exist but again, those are chip/linecard/backplane/chassis/routing engine specific and mostly used in complex test/peformance scenarios and not to derive some kind of equations that can predict anything much ...
[Bruno] Good points.
Yet, one of the information that we propose to advertise by the LSP receiver to the LSP sender is the Receive Window.

-          This is a very common parameter and algorithm. Nothing fancy nor reinvented. In particular it’s a parameter used by TCP.

-          I would argue that TCP implementations also run on a variety of hardware and systems, must wider range than IS-IS platform. And those TCP implementations on all those platform manage to advertise this parameter (TCP window)

-          I fail to understand that when some WG contributors proposed the use of TCP, nobody said that determining and advertising a Receive Window would be an issue, difficult or even impossible. But when we propose to advertise a Receive Window in an IS-IS TLV, this becomes difficult or even impossible for some platforms. Can anyone help me understand the technical difference?

Bruno, if you're so deeply interested in that stuff we can talk 1:1 off-line about implementation work on rift towards adapatable flooding rate
[Bruno] Sure. My pleasure. Please propose me some timeslot offline. Please note that I’m based in Europe (CEST), so a priori during your morning and my evening.
If you can also extend the offer to discuss the implementation work on the IS-IS implementation of your employer with regards to adaptable flooding rate, and/or how network operator can configure the CLI parameters of the LSP senders so as to improve flooding rate without overloading the Juniper receiver (possibly depending on the capability of the receiver, its number of IS-IS neighbors… and/or whatever parameter that you feel are relevant) that would be most appreciated. And if you believe that the Juniper LSP receiver can handle any rate from any reasonable (e.g. 50)  number of IGP neighbors, without (significantly) dropping the received LSPs, that would be even simpler, please advise.

--Bruno
(algorithm you see in the -02 draft Les put out is a _very rough_ approximation of that BTW. I joined as co-author after we had some very fruitful discussions and I consider the draft close to what can be _realistically_ done  today in ISIS. I don't consider further details generic enough to merit wide forum discussions). And RIFT put a couple of things into packet formats we can't put into ISIS (I talked with Les about it) to improve the adaptability of the flooding rate BTW and some interesting, coarse indication from receiver. Again, this is not a constant that is calculated, it's all adaptive driven almost completely from the transmitter side and the feedback it gathers.

all my very own 2c

-- tony

On Tue, Apr 21, 2020 at 2:48 PM Les Ginsberg (ginsberg) <ginsberg=[email-masked]<mailto:[email-masked]>> wrote:
Bruno –

The default value is based on minimumBroadcastLSPTransmissionInterval specified in ISO10589.
A knob is available to allow tuning (faster or slower) in a given deployment – though in my experience this knob is rarely used.

We already discuss in https://tools.ietf.org/html/draft-ginsberg-lsr-isis-flooding-scale-02#section-2 that this common interpretation isn’t the most appropriate, but historically the concern has been to avoid flooding too fast – not to optimize flooding speed.
Obviously, we are revisiting that approach and saying it needs to change – which is something I think we have consensus on.

I have provided a description in my recent response as to why it is difficult to derive an optimal value on a per platform basis. You may disagree – but please do not claim that we actually have been doing this for years. We haven’t been.

  Les

Les,

Thank you,
--Bruno

Les,

Please see inline[Bruno]

•  If yes, how do you do _today_ on your implementation? (this seems to contradict your statement that you have no way to figure out how to find the right value)

•  If no, why did you implement those parameters, and ask network operator to configure them?

•  There is also the option to reply: I don’t know but don’t care as I leave the issue to the network operator.

•  the sender will never overload itself.

•  The receiver has more information,  knowing its processing power (low end, high end, 80s, 20s (currently we are stuck with 20 years old value assuming the worst possible receiver (and worst there were, including with packet processing partly done in the control plane processor)), its expected IS-IS load (#neighbors), its preference for bursty LSP reception (high delay between IS-IS CPU allocation cycles, memory not an issue up to x kilo-octet…), its expected control plane load if IS-IS traffic has not higher priority over other control plane traffic…), it’s expected level of QoS prioritization [1]

•          [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and link stability are more tenuous and more crucial than BGP stability, such packets are now given the highest priority and are given extended SPD headroom with a default of 10 packets. This means that these packets are not dropped if the size of the input hold queue is lower than 185 (input queue default size + spd headroom size + spd extended headroom).”

-
--Bruno

    Les

Thank you.

---

From: <[email-masked]>
Date: Wed, 22 Apr 2020 18:04:13 +0000

Les,

Please see inline

From: Les Ginsberg (ginsberg) [mailto:[email-masked]]
Sent: Tuesday, April 21, 2020 11:48 PM
To: DECRAENE Bruno TGI/OLN
Cc: [email-masked]
Subject: RE: Flow Control Discussion for IS-IS Flooding Speed

Bruno -

You have made an assumption that historically vendors have tuned LSP transmission rates to a platform specific value.
[Bruno] I don't think so. What makes you think so?
In all cases, that is not my assumption, and for multiple reasons.

The default value is based on minimumBroadcastLSPTransmissionInterval specified in ISO10589.
A knob is available to allow tuning (faster or slower) in a given deployment - though in my experience this knob is rarely used.
[Bruno] I would agree on both. More interestingly is the why: why aren't those existing sending parameters tuned, while we  agree that default configuration are suboptimal?
My take is that:

-          We don't want to overload the receiver and create loss of LSP (as this will delay the LSDB synchronization and decrease the goodput). Hence this (the parameters) is receiver dependent (e.g., platform type,  number of IGP adjacencies, ...).

-          We don't know which value to configure : difficult for the network operator to evaluate without a significant knowledge of the receiver implementation (in particular QoS parameters for incoming LSP), vendors are not really proposing value or guidance, especially since the sending behavior is not standardized and slightly different between vendors. So everyone stay safe with default 20 years old values.

I have provided a description in my recent response as to why it is difficult to derive an optimal value on a per platform basis. You may disagree - but please do not claim that we actually have been doing this for years. We haven't been.
[Bruno] I'm not sure how to rephrase my below email so that we could understand each other, have an answer from your side (I mean employer side), and progress.
More succinctly: How do network operator correctly configure the flooding parameters value on the implementation of your employer? In particular given the variety of conditions on the LSP receiver side.

--Bruno

  Les

Les,

Thank you,
--Bruno

Les,

Please see inline[Bruno]

-
--Bruno

    Les

Thank you.

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 22 Apr 2020 12:25:14 -0700

On Wed, Apr 22, 2020 at 11:03 AM <[email-masked]> wrote:

> *From:* Tony Przygienda [mailto:[email-masked]]
> *Sent:* Wednesday, April 22, 2020 1:19 AM
> *To:* Les Ginsberg (ginsberg)
> *Cc:* DECRAENE Bruno TGI/OLN; [email-masked]
> been here and there per deployment scenario some "recommended config"
> (not something I immediately recall either) but that means very fixed
> Yet, one of the information that we propose to advertise by the LSP
> receiver to the LSP sender is the Receive Window.
>
> -          This is a very common parameter and algorithm. Nothing fancy
> nor reinvented. In particular it’s a parameter used by TCP.
>
> -          I would argue that TCP implementations also run on a variety
> of hardware and systems, must wider range than IS-IS platform. And those
> TCP implementations on all those platform manage to advertise this
> parameter (TCP window)
>
> -          I fail to understand that when some WG contributors proposed
> the use of TCP, nobody said that determining and advertising a Receive
> Window would be an issue, difficult or even impossible. But when we propose
> to advertise a Receive Window in an IS-IS TLV, this becomes difficult or
> even impossible for some platforms. Can anyone help me understand the
> technical difference?
>
>
>

Bruno, I was waiting for that ;-) Discounted for the fact that I'm not a
major TCP expert: TCP is a very different beast. it has a 100-200msec fast
timer & 500msec slow (which have to be quite accurate, it's really one
timer for all connections + mbuf and other magic) and it sends a _lot_ of
packets back and forth with window size indications so the negotiation can
happen very quickly.  Also, TCP can detect losses based on sequence number
received contrary to routing protocols (that's one of the things we added
in RIFT BTW) and it can retransmit quickly when it sees a "hole". Contrary
to that in ISIS ACKs may or may not come, they may be bundled, hellos may
or may not come and we can't retransmit stuff on 100msec timers either.
It's an utterly different transport channel.

In more abstract terms, TCP is a sliding N-window protocol (where N is
adjusted all the time & losses can be efficiently detected) whereas LSR
flooding is not a windowing protocol (or rather LSDB-size window protocol
with selective retransmission but no detection of loss [or only very slow
based on lack of ACK & CSNPs]). Disadvantage of something like TCP (I think
I sent out the RFC with UDP control protocol work to make it more TCP like)
is that you are stuck when you put something into the pipe, no
prioritization possible and if receiver is slow you may have multiple
obsolete copies in the pipe waiting & lots retransmission BW when holes are
punched into the data through loss. And plain TCP  is actually quite bad
for control protocol traffic @ scale, almost all vendor run special version
of it for BGP for that reason. Why that is is out of scope of this list I
think ... Flooding is really good to send lots of data prioritized/in
parallel but on losses re-TX is slow.

> [Bruno] Sure. My pleasure. Please propose me some timeslot offline. Please
> note that I’m based in Europe (CEST), so a priori during your morning and
> my evening.
>
> If you can also extend the offer to discuss the implementation work on the
> IS-IS implementation of your employer with regards to adaptable flooding
> rate, and/or how network operator can configure the CLI parameters of the
> LSP senders so as to improve flooding rate without overloading the Juniper
> receiver (possibly depending on the capability of the receiver, its number
> of IS-IS neighbors… and/or whatever parameter that you feel are relevant)
> that would be most appreciated. And if you believe that the Juniper LSP
> receiver can handle any rate from any reasonable (e.g. 50)  number of IGP
> neighbors, without (significantly) dropping the received LSPs, that would
> be even simpler, please advise.
>
>
>
>
>

ping me for that 1:1 on company email pls

-- tony

---

From: Peter Psenak <[email-masked]>
Date: Thu, 23 Apr 2020 10:30:14 +0200

Bruno,

On 22/04/2020 20:04, [email-masked] wrote:
> Les,
> 
> Pleasesee inline
> 
> *From:*Les Ginsberg (ginsberg) [mailto:[email-masked]]
> *Sent:* Tuesday, April 21, 2020 11:48 PM
> *To:* DECRAENE Bruno TGI/OLN
> *Cc:* [email-masked]
> [Bruno] I don’t think so. What makes you think so?
> 
> That certainly is not true in the case of my employer (happy to hear 
> what other vendors have been doing for the past 20 years).
> 
> *//*[Bruno] I would agree on both. More interestingly is the why: why 
> aren’t those existing sending parameters tuned, while we agree that 
> default configuration are suboptimal?
> 
> My take is that:
> 
> -We don’t want to overload the receiver and create loss of LSP (as this 
> will delay the LSDB synchronization and decrease the goodput). Hence 
> this (the parameters) is receiver dependent (e.g., platform type, number 
> of IGP adjacencies, …).
> 
> -We don’t know which value to configure : difficult for the network 
> operator to evaluate without a significant knowledge of the receiver 
> implementation (in particular QoS parameters for incoming LSP), vendors 
> are not really proposing value or guidance, especially since the sending 
> behavior is not standardized and slightly different between vendors. So 
> everyone stay safe with default 20 years old values.
> 
> We already discuss in 
> Obviously, we are revisiting that approach and saying it needs to change 
> – which is something I think we have consensus on.
> 
> *//*[Bruno] I’m not sure how to rephrase my below email so that we could 
> understand each other, have an answer from your side (I mean employer 
> side), and progress.
> 
> More succinctly: How do network operator correctly configure the 
> flooding parameters value on the implementation of your employer? In 
> particular given the variety of conditions on the LSP receiver side.

the answer is test and see which value fits best in your specific 
environment.

One reason to have some sort of feedback mechanism (being it tx or rx 
based) is to avoid the need to tune today's static parameters and flood 
as fast as the receiver is able to consume and slow down if the receiver 
is not able to cope with the rate we flood.

thanks,
Peter

> 
> --Bruno
> 
>    Les
> 
> *From:[email-masked] <[email-masked]>
> [Bruno] _Assuming_ that the parameters are static, the parameters 
> proposed in draft-decraene-lsr-isis-flooding-speed are the same as the 
> one implemented (configured) on multiple implementations, including the 
> one from your employer.
> 
> *From:*DECRAENE Bruno TGI/OLN
> *Cc:* [email-masked] <mailto:[email-masked]>
> *From:*Lsr [mailto:[email-masked]] *On Behalf Of *Les Ginsberg 
> (ginsberg)
> *To:* [email-masked] <mailto:[email-masked]>
> [Bruno] _/Assuming/_ that the parameters are static, those parameters
> 
> oare the same as the one implemented (configured) on multiple 
> implementations, including the one from your employer. Now do you 
> believe that those parameters can be determined?
> 
> §If yes, how do you do _/today/_ on your implementation? (this seems to 
> §If no, why did you implement those parameters, and ask network operator 
> to configure them?
> 
> §There is also the option to reply: I don’t know but don’t care as I 
> leave the issue to the network operator.
> 
> ocan still provide some form of dynamicity, by using the PSNP as dynamic 
> acknowledgement.
> 
> oare really dependent on the receiver, not the sender.
> 
> §the sender will never overload itself.
> 
> §The receiver has more information,  knowing its processing power (low 
> assuming the worst possible receiver (and worst there were, including 
> with packet processing partly done in the control plane processor)), its 
> expected IS-IS load (#neighbors), its preference for bursty LSP 
> reception (high delay between IS-IS CPU allocation cycles, memory not an 
> issue up to x kilo-octet…), its expected control plane load if IS-IS 
> traffic has not higher priority over other control plane traffic…), it’s 
> expected level of QoS prioritization [1]
> 
> · [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and link 
> oAnd this is for distributed architecture, 15 years ago. So what about 
> *From:*Lsr <[email-masked] <mailto:lsr-bounces@ietf..org>> *On 
> *To:* [email-masked] <mailto:[email-masked]>
> advocates for a receiver based flow control where the receiver 
> advertises in hellos the parameters which indicate the rate/burst size 
> which the receiver is capable of supporting on the interface. Senders 
> are required to limit the rate of LSP transmission on that interface in 
> accordance with the values advertised by the receiver.
> 
>   advocates for a transmit based flow control where the transmitter 
> monitors the number of unacknowledged LSPs sent on each interface and 
> implements a backoff algorithm to slow the rate of sending LSPs based on 
> the length of the per interface unacknowledged queue.
> 
> that if agreement could be reached on the form of flow control  then it 
> is likely other issues could be resolved easily.
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> 
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez 
> recu ce message par erreur, veuillez le signaler
> 
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> 
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
>

---

From: <[email-masked]>
Date: Thu, 23 Apr 2020 08:57:33 +0000

Peter,
 
> From: Peter Psenak [mailto:[email-masked]] 
> 
> Bruno,
> 

I don't think that's good enough, or even feasible given the very diverse set of platforms and environments in large networks, and/or the availability of human resources in smaller networks.
Since Les is reporting that virtually nobody is changing the default value -although we now all agree that they are suboptimal- I think that this is an indication that this strategy is not working.
That's why I brought this to the LSR WG.

We need the IS-IS protocol to work adequately without requiring constant or personalized tuning from the network operator. Coming back to TCP, general user/terminals are not been asked to run test per terminal and per destination/server. It just work relatively adequately.
 
> is not able to cope with the rate we flood.

+1
 
Thanks,
Bruno

---

From: Peter Psenak <[email-masked]>
Date: Thu, 23 Apr 2020 11:14:25 +0200

Bruno,

> That's why I brought this to the LSR WG.

we agree on the problem and the solution, for most part anyway.

I responded with the only option that is available today with existing 
protocol behavior and parameters available today.

thanks,
Peter

---

From: <[email-masked]>
Date: Thu, 23 Apr 2020 17:15:11 +0000

Tony,
Thanks for engaging.
Please inline [Bruno2]

From: Tony Przygienda [mailto:[email-masked]]
Sent: Wednesday, April 22, 2020 9:25 PM
To: DECRAENE Bruno TGI/OLN
Cc: [email-masked]; Les Ginsberg (ginsberg)
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

On Wed, Apr 22, 2020 at 11:03 AM <[email-masked]<mailto:[email-masked]>> wrote:
Tony, all,

From: Tony Przygienda [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, April 22, 2020 1:19 AM
To: Les Ginsberg (ginsberg)
Cc: DECRAENE Bruno TGI/OLN; [email-masked]<mailto:[email-masked]>
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Bruno, I was waiting for that ;-)
[Bruno2] Good ;-)

Discounted for the fact that I'm not a major TCP expert: TCP is a very different beast. it has a 100-200msec fast timer & 500msec slow (which have to be quite accurate, it's really one timer for all connections + mbuf and other magic) and it sends a _lot_ of packets back and forth with window size indications so the negotiation can happen very quickly.  Also, TCP can detect losses based on sequence number received contrary to routing protocols (that's one of the things we added in RIFT BTW) and it can retransmit quickly when it sees a "hole". Contrary to that in ISIS ACKs may or may not come, they may be bundled, hellos may or may not come and we can't retransmit stuff on 100msec timers either. It's an utterly different transport channel.
[Bruno2] I would distinguish two things, which I think we have done in https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03

-          How fast you can adapt the sending rate. This seems mostly dependent on the speed of the feedback loop, rather than the format of message. E.g. In IS-IS the receiver can give a feedback more or less quickly (e.g. depending on how fast/bundled it sends the PSNP). In theory, I don’t see a major different. From an in implementation standpoint, I’m guessing that the difference is probably bigger (e.g. TCP is probably lower level/closer to the system/hardware, than IS-IS which is more user space and possibly Platform Independent in some organizations))

-          How fast you can detect packet loss. I agree that TCP & IS-IS are very different on this. We have proposed to improve this by allowing the receiver to advertise to the sender how fast it will ack the LSP. Currently the timer/behavior is known to receiver but no to the sender. Hence the sender needs to assume the wort case (ISO default).

In more abstract terms, TCP is a sliding N-window protocol (where N is adjusted all the time & losses can be efficiently detected) whereas LSR flooding is not a windowing protocol (or rather LSDB-size window protocol with selective retransmission but no detection of loss [or only very slow based on lack of ACK & CSNPs]). Disadvantage of something like TCP (I think I sent out the RFC with UDP control protocol work to make it more TCP like)
[Bruno2]  If you are referring to DCCP (Datagram Congestion Control Protocol) (RFC 4340), yes you did and thank you for this. Constructive feedback.

-          Regarding flow control, I’ve quickly looked at DCCP and it does not provides flow control.

-          Regarding congestion control, possibly the algorithm part may be reused. There are two algo and DCCP is open to others. May be one question is how much we want IS-IS to be fair to TCP (control plane TCP, not dataplane/user plane TCP). To me, IS-IS is more important than BGP traffic, given their relative importance to the network, their delay requirements, their typical volume of traffic. But that is probably a “detail” down the road. This is also depends on whether TCP & IS-IS compete for the same resources (e.g. same queue) or not (ideally TCP and IS-IS have different queues).

is that you are stuck when you put something into the pipe, no prioritization possible and if receiver is slow you may have multiple obsolete copies in the pipe waiting & lots retransmission BW when holes are punched into the data through loss. And plain TCP  is actually quite bad for control protocol traffic @ scale, almost all vendor run special version of it for BGP for that reason. Why that is is out of scope of this list I think ... Flooding is really good to send lots of data prioritized/in parallel but on losses re-TX is slow.
[Bruno2] Good that we seem to make the same distinction between the control loops for the sending rate vs the retransmission.
Regarding clarifying distinctions, draft may need to better introduce the distinction between flow control and congestion control, at least to structure the work and the discussion.

Thanks
--Bruno

-- tony

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 23 Apr 2020 10:28:51 -0700

I was refering to RFC4960. Bruno, for all practical purposes I think that
seems to go down the path of trying to reinvent RFC4960 (or ultimately use
it). Or, changing the packet formats heavily to incorporate all the control
loop data you need to the point you have a different control channel along
those lines since you'll find most of the problems RFC4960 is describing
(minus stuff like multiple paths). Nothing wrong with that but it's
ambitious on a 30 years old anitque artefact we're nursing forward here ;-)
As entertaining footnote, I saw in last 20 years at least 3 attempts to
allow multiple TCP sessions in BGP between peers to speed/prioritize things
up. All failed, after the first one I helped to push I smarted up ;-)

As another footnote: I looked @ all the stuff in RIFT (tcp, quic, 4960,
more ephemeral stuff). I ended up adding to rift bunch very rudimentary
things and do roughly what Les/Peter/Acee started to write (modulo algorith
I contributed and bunch things that would be helpful but we can't fit into
existing packet format). This was as much decision as to "what's available
& well debugged" as well as "does it meet requirements" as "how complex is
that vs. rtx in flooding architecture  we have today + some feedback". Not
on powerpoint, in real production code ;-) rift draft shows you the outcome
of that as IMO best trade-off to achieve high flooding speeds ;-)

my 2c

-- tony

On Thu, Apr 23, 2020 at 10:15 AM <[email-masked]> wrote:

> *Sent:* Wednesday, April 22, 2020 9:25 PM
> *Cc:* [email-masked]; Les Ginsberg (ginsberg)
> Discounted for the fact that I'm not a major TCP expert: TCP is a very
> different beast. it has a 100-200msec fast timer & 500msec slow (which have
> to be quite accurate, it's really one timer for all connections + mbuf
> and other magic) and it sends a _lot_ of packets back and forth with window
> size indications so the negotiation can happen very quickly.  Also, TCP
> can detect losses based on sequence number received contrary to routing
> protocols (that's one of the things we added in RIFT BTW) and it can
> retransmit quickly when it sees a "hole". Contrary to that in ISIS ACKs may
> or may not come, they may be bundled, hellos may or may not come and we
> can't retransmit stuff on 100msec timers either. It's an utterly different
> transport channel.
>
> [Bruno2] I would distinguish two things, which I think we have done in
> -          How fast you can adapt the sending rate. This seems mostly
> dependent on the speed of the feedback loop, rather than the format of
> message. E.g. In IS-IS the receiver can give a feedback more or less
> quickly (e.g. depending on how fast/bundled it sends the PSNP). In theory,
> I don’t see a major different. From an in implementation standpoint, I’m
> guessing that the difference is probably bigger (e.g. TCP is probably lower
> level/closer to the system/hardware, than IS-IS which is more user space
> and possibly Platform Independent in some organizations))
>
> -          How fast you can detect packet loss. I agree that TCP & IS-IS
> are very different on this. We have proposed to improve this by allowing
> the receiver to advertise to the sender how fast it will ack the LSP.
> Currently the timer/behavior is known to receiver but no to the sender.
> Hence the sender needs to assume the wort case (ISO default).
>
>
>
> based on lack of ACK & CSNPs]). Disadvantage of something like TCP (I
> think I sent out the RFC with UDP control protocol work to make it more TCP
> like)
>
> [Bruno2]  If you are referring to DCCP (Datagram Congestion Control
> Protocol) (RFC 4340), yes you did and thank you for this. Constructive
> feedback.
>
> -          Regarding congestion control, possibly the algorithm part may
> be reused. There are two algo and DCCP is open to others. May be one
> question is how much we want IS-IS to be fair to TCP (control plane TCP,
> not dataplane/user plane TCP). To me, IS-IS is more important than BGP
> traffic, given their relative importance to the network, their delay
> requirements, their typical volume of traffic. But that is probably a
> “detail” down the road. This is also depends on whether TCP & IS-IS compete
> for the same resources (e.g. same queue) or not (ideally TCP and IS-IS have
> different queues).
>
>
>
> [Bruno2] Good that we seem to make the same distinction between the
> control loops for the sending rate vs the retransmission.
>
> Regarding clarifying distinctions, draft may need to better introduce the
> distinction between flow control and congestion control, at least to
> structure the work and the discussion.
>
>
>
> Thanks
>
> --Bruno
>

---

From: <[email-masked]>
Date: Fri, 24 Apr 2020 09:25:22 +0000

Tony

Sent: Thursday, April 23, 2020 7:29 PM

I was refering to RFC4960. Bruno, for all practical purposes I think that seems to go down the path of trying to reinvent RFC4960 (or ultimately use it).
[Bruno] I don’t think that SCTP (RC4960) is a better fit than TCP. Many more features and options than TCP, way more than needed given existing IS-IS flooding mechanism. Much less implementations experience and improvement than TCP.
Or, changing the packet formats heavily to incorporate all the control loop data you need to the point you have a different control channel along those lines since you'll find most of the problems RFC4960 is describing (minus stuff like multiple paths).
[Bruno] Really, adding one sub-TLV in IS-IS is not “changing the packet formats heavily”.
Nothing wrong with that but it's ambitious on a 30 years old anitque artefact we're nursing forward here ;-)
[Bruno] I’m perfectly fine with revolution approaches. I think that we can also provide incremental improvement to IS-IS.
As entertaining footnote, I saw in last 20 years at least 3 attempts to allow multiple TCP sessions in BGP between peers to speed/prioritize things up. All failed, after the first one I helped to push I smarted up ;-)
 [Bruno] On my side, I’ll try once and I think the LSR WG should also try to improve IS-IS performance. May be if we want to move, we should first release the brakes. I’m seen some progress, e.g., from “there is no need to improve flooding” to “we all agree to improve flooding”, or from “Network operator just need to configure existing CLI” to “We agree that we need something more automated/dynamic”. But this has been very slow progress over a year.

--Bruno

As another footnote: I looked @ all the stuff in RIFT (tcp, quic, 4960, more ephemeral stuff). I ended up adding to rift bunch very rudimentary things and do roughly what Les/Peter/Acee started to write (modulo algorith I contributed and bunch things that would be helpful but we can't fit into existing packet format). This was as much decision as to "what's available & well debugged" as well as "does it meet requirements" as "how complex is that vs. rtx in flooding architecture  we have today + some feedback". Not on powerpoint, in real production code ;-) rift draft shows you the outcome of that as IMO best trade-off to achieve high flooding speeds ;-)

my 2c

-- tony

On Thu, Apr 23, 2020 at 10:15 AM <[email-masked]<mailto:[email-masked]>> wrote:
Tony,
Thanks for engaging.
Please inline [Bruno2]

Cc: [email-masked]<mailto:[email-masked]>; Les Ginsberg (ginsberg)

Thanks
--Bruno

-- tony

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 24 Apr 2020 12:41:39 +0200

Hi Bruno  & all,

[Bruno] On my side, I’ll try once and I think the LSR WG should also try to
improve IS-IS performance. May be if we want to move, we should first
release the brakes.

Well from my observations releasing the breaks means increasing the risks.

Take BGP - breaks are off and see what happens :)

My personal observation is that ISIS implementations across vendors are
just fine for vast majority of deployments today. That actually also
includes vast majority of compute clusters as they consists of max 10s of
racks.

Of course there are larger clusters with 1000+ or 10K and above network
elements itself and x 20 L3 computes, but is there really a need to stretch
protocol to accommodate those ? Those usually run BGP anyway. And also
there is DV+LS hybrid too now.

ISIS flooding churn (and room for optimization) becomes a problem when node
boots up (IMHO this is not a problem) and when node fails while having many
neighbors attached. Yes maybe second case could be improved but well
designed and operated network should have pre-programmed bypass paths
against such cases so IMO stressing IGP to "converge" faster while great in
principle may not be really needed in practice.

Last I am worried that when IETF defines changes to core protocol behaviour
the quality of the implementations of those changes may really differ
across vendors overall resulting in much worse performance and stability as
compared to where we are today.

I am just not sure if possible gains for few deployments are greater
then risk for 1000s of today's deployments. Maybe one size does not fit all
and for massive scale ISIS we should define a notion of "ISIS-DC-PLUGIN"
which can be optionally in run time added when/if needed. If that requires
protocol changes to accommodate such dynamic plugins - that work should
take place.

Many thx,
R.

PS. Does anyone have a pointer to any real data showing that performance of
real life WAN ISIS deployments is bad ?

On Fri, Apr 24, 2020 at 11:26 AM <[email-masked]> wrote:

> *Sent:* Thursday, April 23, 2020 7:29 PM
> it).
> [Bruno] I don’t think that SCTP (RC4960) is a better fit than TCP. Many
> more features and options than TCP, way more than needed given existing
> IS-IS flooding mechanism. Much less implementations experience and
> improvement than TCP.
>
> Or, changing the packet formats heavily to incorporate all the control
> (minus stuff like multiple paths).
>
> [Bruno] I’m perfectly fine with revolution approaches. I think that we can
> also provide incremental improvement to IS-IS.
>
>  [Bruno] On my side, I’ll try once and I think the LSR WG should also try
> to improve IS-IS performance. May be if we want to move, we should first
> release the brakes. I’m seen some progress, e.g., from “there is no need to
> improve flooding” to “we all agree to improve flooding”, or from “Network
> operator just need to configure existing CLI” to “We agree that we need
> something more automated/dynamic”. But this has been very slow progress
> over a year.
>
>
>
> --Bruno
>
>
>

---

From: <[email-masked]>
Date: Mon, 27 Apr 2020 08:34:04 +0000

Ø  ISIS flooding churn (and room for optimization) becomes a problem when node boots up (IMHO this is not a problem) and when node fails while having many neighbors attached. Yes maybe second case could be improved but well designed and operated network should have pre-programmed bypass paths against such cases so IMO stressing IGP to "converge" faster while great in principle may not be really needed in practice.

I don’t think that FRR is a replacement for “fast” (I’d rather say adequate) IGP convergence & flooding.

For multiple reasons such as:

-          Multiple ‘things’ depends on the IGP, such as BGP best path selection, CSPF/TE/PCE computations, FRR computations

-          Slow flooding increase the likelihood of multiple IGP SPF computations which is harmful for other computations which are typically heavier and manifolds (cf above)

-          Multiple IGP SPF computations also create multiple transient forwarding loops. There are some techniques to remove forwarding loops but this is still an advanced topic and some implementations do not handle consecutives IGP SPF (with ‘overlapping’ convergences and combined distributed forwarding loops)

-          For FRR, you mostly need to pre-decide/configure whether you want to protect link or node failures. Tradeoff are involved and given probability of events, link protection is usually enabled (hence not node protection)

-          …

Also, given the current “state of the art”, there is no stressing involved. Really. Using TCP, my 200€ mobile running on battery and over wifi+ADSL+Internet can achieve better communication throughput than a n*100k€ high end IS-IS router.
I think many persons agree that IS-IS could do better in term of flooding. (possibly not as good as a brand new approach, but incremental improvement also have some benefits). Eventually, we don’t need everyone to agree on this.

Ø  PS. Does anyone have a pointer to any real data showing that performance of real life WAN ISIS deployments is bad ?

In some of our ASes, we do monitor IS-IS by listening to and recording flooded LSPs. I can’t share any data.
Next question could be what is “good enough”. I guess this may depend on the size of your network, its topology, and your requirements.

We also ran tests in labs. I may share some results during my presentation. (no names, possibly no KPI, but some high level outcomes).

Regards,
Bruno

From: Robert Raszuk [mailto:[email-masked]]
Sent: Friday, April 24, 2020 12:42 PM
To: DECRAENE Bruno TGI/OLN
Cc: Tony Przygienda; Les Ginsberg (ginsberg); [email-masked]
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Hi Bruno  & all,

[Bruno] On my side, I’ll try once and I think the LSR WG should also try to improve IS-IS performance. May be if we want to move, we should first release the brakes.

My personal observation is that ISIS implementations across vendors are just fine for vast majority of deployments today. That actually also includes vast majority of compute clusters as they consists of max 10s of racks.

Of course there are larger clusters with 1000+ or 10K and above network elements itself and x 20 L3 computes, but is there really a need to stretch protocol to accommodate those ? Those usually run BGP anyway. And also there is DV+LS hybrid too now.

ISIS flooding churn (and room for optimization) becomes a problem when node boots up (IMHO this is not a problem) and when node fails while having many neighbors attached. Yes maybe second case could be improved but well designed and operated network should have pre-programmed bypass paths against such cases so IMO stressing IGP to "converge" faster while great in principle may not be really needed in practice.

Last I am worried that when IETF defines changes to core protocol behaviour the quality of the implementations of those changes may really differ across vendors overall resulting in much worse performance and stability as compared to where we are today.

I am just not sure if possible gains for few deployments are greater then risk for 1000s of today's deployments. Maybe one size does not fit all and for massive scale ISIS we should define a notion of "ISIS-DC-PLUGIN" which can be optionally in run time added when/if needed. If that requires protocol changes to accommodate such dynamic plugins - that work should take place.

Many thx,
R.

PS. Does anyone have a pointer to any real data showing that performance of real life WAN ISIS deployments is bad ?

On Fri, Apr 24, 2020 at 11:26 AM <[email-masked]<mailto:[email-masked]>> wrote:
Tony

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 12:08:53 +0200

> Slow flooding increase the likelihood of multiple IGP SPF computations

True.

But if you keep your IGP nicely organized in area and levels, get rid of
flooding anything incl. /32s domain wide to address bugs in MPLS
architecture then your flooding radius is usually very small.

That in turn allows for both fast flooding and fast topology computation
while only dealing with few external summaries. I am yet to see a
practical case where a well designed network with today's ISIS requires
flooding speedup.

Best,
R.

On Mon, Apr 27, 2020 at 10:34 AM <[email-masked]> wrote:

> Ø  ISIS flooding churn (and room for optimization) becomes a problem when
> node boots up (IMHO this is not a problem) and when node fails while having
> many neighbors attached. Yes maybe second case could be improved but well
> -          Multiple ‘things’ depends on the IGP, such as BGP best path
> selection, CSPF/TE/PCE computations, FRR computations
>
> -          Slow flooding increase the likelihood of multiple IGP SPF
> computations which is harmful for other computations which are typically
> heavier and manifolds (cf above)
>
> -          Multiple IGP SPF computations also create multiple transient
> forwarding loops. There are some techniques to remove forwarding loops but
> this is still an advanced topic and some implementations do not handle
> consecutives IGP SPF (with ‘overlapping’ convergences and combined
> distributed forwarding loops)
>
> -          For FRR, you mostly need to pre-decide/configure whether you
> want to protect link or node failures. Tradeoff are involved and given
> probability of events, link protection is usually enabled (hence not node
> protection)
>
> -          …
>
>
>
> Also, given the current “state of the art”, there is no stressing
> involved. Really. Using TCP, my 200€ mobile running on battery and over
> wifi+ADSL+Internet can achieve better communication throughput than a
> n*100k€ high end IS-IS router.
>
> I think many persons agree that IS-IS could do better in term of flooding.
> (possibly not as good as a brand new approach, but incremental improvement
> also have some benefits). Eventually, we don’t need everyone to agree on
> this.
>
>
>
> Ø  PS. Does anyone have a pointer to any real data showing that
> performance of real life WAN ISIS deployments is bad ?
>
>
>
> Next question could be what is “good enough”. I guess this may depend on
> the size of your network, its topology, and your requirements.
>
>
>
> We also ran tests in labs. I may share some results during my
> presentation. (no names, possibly no KPI, but some high level outcomes).
>
>
>
> Regards,
>
> Bruno
>
>
>
>
>
> *From**:* Robert Raszuk [mailto:[email-masked]]
> *Sent:* Friday, April 24, 2020 12:42 PM
> *Cc:* Tony Przygienda; Les Ginsberg (ginsberg); [email-masked]
> ISIS flooding churn (and room for optimization) becomes a problem when
> node boots up (IMHO this is not a problem) and when node fails while having
> many neighbors attached. Yes maybe second case could be improved but well
> Last I am worried that when IETF defines changes to core protocol
> behaviour the quality of the implementations of those changes may really
> differ across vendors overall resulting in much worse performance and
> stability as compared to where we are today.
>
>
>
> PS. Does anyone have a pointer to any real data showing that performance
> of real life WAN ISIS deployments is bad ?
>
>
>
>
>

---

From: <[email-masked]>
Date: Mon, 27 Apr 2020 12:14:49 +0000

Robert,

Sent: Monday, April 27, 2020 12:09 PM

True.

But if you keep your IGP nicely organized in area and levels, get rid of flooding anything incl. /32s domain wide to address bugs in MPLS architecture then your flooding radius is usually very small.
[Bruno] First of all, the use of areas/levels brings tradeoffs. Then, after their initial design, networks grow and change.

Coming back to flooding, if you have a core router with 50 IGP neighbors, the failure of this neighbor requires flooding 50 LSPs. At 33ms pacing between LSPs that’s a 1.6s delay/tax, before any computation & FIB update. As you see, it’s not related to the number of /32 nor the network diameter.
Some may be fine with this additional 1.6s. Some may not.

Best
--Bruno

That in turn allows for both fast flooding and fast topology computation while only dealing with few external summaries. I am yet to see a practical case where a well designed network with today's ISIS requires flooding speedup.

Best,
R.

-          …

Regards,
Bruno

From: Robert Raszuk [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Friday, April 24, 2020 12:42 PM
To: DECRAENE Bruno TGI/OLN
Cc: Tony Przygienda; Les Ginsberg (ginsberg); [email-masked]<mailto:[email-masked]>
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Mon, 27 Apr 2020 12:38:38 +0000

Hi Bruno,

From: Lsr <[email-masked]> on behalf of Bruno Decraene <[email-masked]>
Date: Monday, April 27, 2020 at 8:15 AM
To: Robert Raszuk <[email-masked]>
Cc: "Les Ginsberg (ginsberg)" <ginsberg=[email-masked]>, "[email-masked]" <[email-masked]>, Tony Przygienda <[email-masked]>
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Robert,

True.

I’m not nearly as familiar with IS-IS deployments as OSPF. Are there any implementations that don’t offer configuration to override the 33ms inter-LSP interval? At Redback (circa 2000), our OSPF implementation defaulted to fast flooding and for the MinLSInterval and MinLSArrival OSPF values, you had to explicitly remove the fast flooding default if  you wanted to follow RFC 2328. Thanks,
Acee

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

---

From: <[email-masked]>
Date: Mon, 27 Apr 2020 13:02:45 +0000

Hi Acee,

Please see inline [Bruno2]

From: Acee Lindem (acee) [mailto:[email-masked]]
Sent: Monday, April 27, 2020 2:39 PM
To: DECRAENE Bruno TGI/OLN; Robert Raszuk
Cc: Les Ginsberg (ginsberg); [email-masked]; Tony Przygienda
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Hi Bruno,

Robert,

True.

I’m not nearly as familiar with IS-IS deployments as OSPF. Are there any implementations that don’t offer configuration to override the 33ms inter-LSP interval?
[Bruno2] AFAIK, all implementations allow the configuration of the inter-LSP interval.
The question is which value do you set while not risking to overload your IS-IS neighbor. Which brings two issues:

-          This depends on the receiver. E.g. High end router vs low end; PE with only 2 high adjacencies vs P with 50 adjacencies, router generation… While this is currently configured on the sender on a per receiver basis

-          Even though the vendor know the design and how it is implemented, some vendors may not commit on scaling values supported by a given receiver.

So there are two options:

-          Have the receiver advertise static values (default but according to its platform capability=

-          Use dynamic values. Flow control is likely to also require some signaling from the receiver to the sender.

In both cases, this call, IMO a signaling capability from the receiver to the sender. The same signaling may then be used to advertise either static/default or dynamic values, depending on what the receiver prefer (some tend to prefer static values, some tend to prefer dynamic values)

Thanks,
BR
--Bruno
At Redback (circa 2000), our OSPF implementation defaulted to fast flooding and for the MinLSInterval and MinLSArrival OSPF values, you had to explicitly remove the fast flooding default if  you wanted to follow RFC 2328. Thanks,
Acee

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Mon, 27 Apr 2020 14:12:23 +0000

Hi Bruno,

From: Bruno Decraene <[email-masked]>
Date: Monday, April 27, 2020 at 9:03 AM
To: Acee Lindem <[email-masked]>
Cc: "Les Ginsberg (ginsberg)" <ginsberg=[email-masked]>, "[email-masked]" <[email-masked]>, Robert Raszuk <[email-masked]>, Tony Przygienda <[email-masked]>
Subject: RE: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Hi Acee,

Hi Bruno,

Robert,

True.

I think we’ve come full circle with this discussion and don’t agree that we need explicit signaling and believe that this would undoubtedly add unneeded complexity and could, in the wrong implementer’s hands, do more harm than good. I guess this is why we have separate drafts 😉
Thanks,
Acee

Thanks,
BR
--Bruno

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 16:38:55 +0200

> In both cases, this call, IMO a signaling capability from the receiver to
the sender.

So essentially you are asking for per peer flooding queue.

Now this get's a little bit of tricky (especially if you are dealing
with relatively small timers) if one peer sends you 1 ms, second 50 ms and
10th  250 ms.

Imagine that the LSP to be flooded to the 10th peer is already overwritten
due to new LSP but still sitting in the out queue ... do you drain that
queue and start over with new LSP or you in place replace the old one
keeping the running timer ?

I am just curious what will happen under the hood :)

Cheers,
R.

On Mon, Apr 27, 2020 at 3:02 PM <[email-masked]> wrote:

> *From:* Acee Lindem (acee) [mailto:[email-masked]]
> *Sent:* Monday, April 27, 2020 2:39 PM
> *To:* DECRAENE Bruno TGI/OLN; Robert Raszuk
> *Cc:* Les Ginsberg (ginsberg); [email-masked]; Tony Przygienda
> *From: *Lsr <[email-masked]> on behalf of Bruno Decraene <
> [email-masked]>
> *Date: *Monday, April 27, 2020 at 8:15 AM
> *To: *Robert Raszuk <[email-masked]>
> *Cc: *"Les Ginsberg (ginsberg)" <ginsberg=[email-masked]>, "
> [email-masked]" <[email-masked]>, Tony Przygienda <[email-masked]>
> *Subject: *Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed
>
>
>
> Robert,
>
>
>
> *From:* Robert Raszuk [mailto:[email-masked]]
> *Sent:* Monday, April 27, 2020 12:09 PM
> [Bruno] First of all, the use of areas/levels brings tradeoffs. Then,
> after their initial design, networks grow and change.
>
>
>
> Coming back to flooding, if you have a core router with 50 IGP neighbors,
> the failure of this neighbor requires flooding 50 LSPs. At 33ms pacing
> between LSPs that’s a 1.6s delay/tax, before any computation & FIB update.
> As you see, it’s not related to the number of /32 nor the network diameter.
>
> I’m not nearly as familiar with IS-IS deployments as OSPF. Are there any
> implementations that don’t offer configuration to override the 33ms
> inter-LSP interval?
>
> -          This depends on the receiver. E.g. High end router vs low end;
> PE with only 2 high adjacencies vs P with 50 adjacencies, router
> generation… While this is currently configured on the sender on a per
> receiver basis
>
> -          Even though the vendor know the design and how it is
> implemented, some vendors may not commit on scaling values supported by a
> given receiver.
>
>
>
> -          Use dynamic values. Flow control is likely to also require
> some signaling from the receiver to the sender.
>
>
>
> In both cases, this call, IMO a signaling capability from the receiver to
> the sender. The same signaling may then be used to advertise either
> static/default or dynamic values, depending on what the receiver prefer
> (some tend to prefer static values, some tend to prefer dynamic values)
>
>
>
> Thanks,
>
> BR
>
> --Bruno
>
> At Redback (circa 2000), our OSPF implementation defaulted to fast
> flooding and for the MinLSInterval and MinLSArrival OSPF values, you had to
> explicitly remove the fast flooding default if  you wanted to follow RFC
> 2328. Thanks,
> Acee
>
>
>
> Best
>
> --Bruno
>
>
>
>
>

---

From: <[email-masked]>
Date: Mon, 27 Apr 2020 15:25:23 +0000

Robert,

From: Robert Raszuk [mailto:[email-masked]]
Sent: Monday, April 27, 2020 4:39 PM
To: DECRAENE Bruno TGI/OLN
Cc: Acee Lindem (acee); Les Ginsberg (ginsberg); [email-masked]; Tony Przygienda
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

> In both cases, this call, IMO a signaling capability from the receiver to the sender.

So essentially you are asking for per peer flooding queue.
[Bruno] Not exactly, at least not as your below text. But this is not the point as this part is already implemented. lsp-interval can be configured per interface. E.g.
interface GigabitEthernet0/0/0/1
  circuit-type level-2-only
  point-to-point
  lsp-interval 1

(and different delays between LAN and P2P interfaces are also in the original ISO spec.)

--Bruno

Now this get's a little bit of tricky (especially if you are dealing with relatively small timers) if one peer sends you 1 ms, second 50 ms and 10th  250 ms.

Imagine that the LSP to be flooded to the 10th peer is already overwritten due to new LSP but still sitting in the out queue ... do you drain that queue and start over with new LSP or you in place replace the old one keeping the running timer ?

Cheers,
R.

On Mon, Apr 27, 2020 at 3:02 PM <[email-masked]<mailto:[email-masked]>> wrote:
Hi Acee,

From: Acee Lindem (acee) [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, April 27, 2020 2:39 PM
To: DECRAENE Bruno TGI/OLN; Robert Raszuk
Cc: Les Ginsberg (ginsberg); [email-masked]<mailto:[email-masked]>; Tony Przygienda
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Hi Bruno,

From: Lsr <[email-masked]<mailto:[email-masked]>> on behalf of Bruno Decraene <[email-masked]<mailto:[email-masked]>>
Date: Monday, April 27, 2020 at 8:15 AM
To: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: "Les Ginsberg (ginsberg)" <ginsberg=[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Tony Przygienda <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Robert,

True.

Thanks,
BR
--Bruno

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

---

From: [email-masked]
Date: Mon, 27 Apr 2020 09:21:50 -0700

Robert,

> So essentially you are asking for per peer flooding queue. 

You already have a per-interface flooding ‘queue' through the implementation of the SRM bit in the LSDB, which must be managed on a per-interface basis.

> Now this get's a little bit of tricky (especially if you are dealing with relatively small timers) if one peer sends you 1 ms, second 50 ms and 10th  250 ms. 

If the peers are on the same interface, then you clearly have to not overrun the slower peer.  Note that both are slower than the currently mandated 33ms.

If the peers are on different interfaces, then this tells you how to configure the per-interface timers.

> I am just curious what will happen under the hood :) 

You’re assuming a strict queue, when in fact, what happens under the hood is simpler: each LSP has a per-interface bit saying that it must be sent on that interface. When it’s time to transmit you walk the LSDB looking for an LSP with the bit set, and then transmit it. If an LSP is overwritten, that’s fine, the bit is still set and you would send the newer version.

Walking the LSDB every time is obviously inefficient, so an implementation is free to maintain other data structures to optimize this if it chooses, but the semantics are quite clear: send LSPs that haven’t been sent yet.

Regards,
Tony

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Mon, 27 Apr 2020 16:26:18 +0000

Bruno –

What would be helpful – in the spirit of Tony Li’s call for “more light and less heat” – is some detail on the inputs required to derive the values you propose to advertise in https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03#section-2
I – and others – have thought about this – and I believe it will be very difficult to obtain the input required to derive accurate values – but maybe you have ideas that did not occur to me.
If you do not need per interface input, what do you require? Could you provide a candidate algorithm? This would allow us to better understand what inputs the dataplane needs to provide  to IS-IS in the control plane and therefore better evaluate the implementation costs.

Less important, but just to set the historical record straight, ISO 10589 never specified a delay for P2P interfaces. You might want to read ISO 10589 Section 12.1.2.4.3e – which says in part:

“On point-to-point links the peak rate of arrival is limited only by the speed of the data link and the other traffic flowing on that link.”

Modern implementations chose to apply the pacing config – originally specified only for Broadcast interfaces – to all types of interfaces – but this was not based on ISO 10589.

   Les

From: [email-masked] <[email-masked]>
Sent: Monday, April 27, 2020 8:25 AM
To: Robert Raszuk <[email-masked]>
Cc: Acee Lindem (acee) <[email-masked]>; Les Ginsberg (ginsberg) <[email-masked]>; [email-masked]; Tony Przygienda <[email-masked]>
Subject: RE: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Robert,

Cc: Acee Lindem (acee); Les Ginsberg (ginsberg); [email-masked]<mailto:[email-masked]>; Tony Przygienda

--Bruno

Cheers,
R.

Hi Bruno,

Robert,

True.

Thanks,
BR
--Bruno

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

---

From: <[email-masked]>
Date: Mon, 27 Apr 2020 17:48:03 +0000

Les,

From: Les Ginsberg (ginsberg) [mailto:[email-masked]]
Sent: Monday, April 27, 2020 6:26 PM
To: DECRAENE Bruno TGI/OLN; Robert Raszuk
Cc: Acee Lindem (acee); [email-masked]; Tony Przygienda
Subject: RE: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Bruno –

What would be helpful – in the spirit of Tony Li’s call for “more light and less heat” – is some detail on the inputs required to derive the values you propose to advertise in https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03#section-2
I – and others – have thought about this – and I believe it will be very difficult to obtain the input required to derive accurate values – but maybe you have ideas that did not occur to me.
If you do not need per interface input, what do you require? Could you provide a candidate algorithm? This would allow us to better understand what inputs the dataplane needs to provide  to IS-IS in the control plane and therefore better evaluate the implementation costs.
[Bruno] ok. I can update the draft with more details and explanations. Not likely by next meeting/Wednesday.
However, in the meantime please note that:
1) the two parameters advertised are the same as the ones on the implementation of your employer:
  lsp-interval 10

  lsp fast-flood threshold 10

2) A receive window is advertised by all TCP implementations. Somehow, this seem possible for TCP.

It seems strange to assume that the sender can know those values which are related to a receiver capability/scaling, but the receiver can’t.

With the same spirit, can you explain how the parameters used in your implementation are determined? (Was already asked by Tony during latest LSR meeting)

“On point-to-point links the peak rate of arrival is limited only by the speed of the data link and the other traffic flowing on that link.”
[Bruno] I’m aware of this.
This essentially means that the lsp-pacing time is equal to 0 ms on point to point links.   That’s one value.
With 33ms on broadcast interfaces, that’s a second value.
So what’s wrong with my text “different delays between LAN and P2P interfaces are also in the original ISO spec”? 0ms and 33ms seem different delays to me.

--Bruno

   Les

Robert,

--Bruno

Cheers,
R.

Hi Bruno,

Robert,

True.

Thanks,
BR
--Bruno

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 19:55:19 +0200

Hi Tony,

> You already have a per-interface flooding ‘queue' through the
> implementation of the SRM bit in the LSDB, which must be managed on a
> per-interface basis.
>

Today from what I see operators (if they even change the default) normally
apply same timer value on all interfaces. If the timer is static what would
be the incentive for any implementation not to group interfaces with
identical transmit delay ?

- - -

While this thread is very interesting I must observe that from my
experience the issue is usually on the receiver. If LSR would publish a one
page draft/rfc mandating that links state packets MUST or SHOULD be
recognized and separated from any other control plane traffic at the
ingress interface level (on their way to local RE/RP) we likely wouldn't be
having such debate.

Slowing senders just due to bad implementation of the receiving router is
IMHO a little suboptimal (not to say wrong) thing to do.

Kind regards,
R.

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Mon, 27 Apr 2020 18:09:35 +0000

Bruno =

Thanx for the quick response.
Inline.

From: [email-masked] <[email-masked]>
Sent: Monday, April 27, 2020 10:48 AM
To: Les Ginsberg (ginsberg) <[email-masked]>; Robert Raszuk <[email-masked]>
Cc: Acee Lindem (acee) <[email-masked]>; [email-masked]; Tony Przygienda <[email-masked]>
Subject: RE: [Lsr] Flow Control Discussion for IS-IS Flooding Speed

Les,

Cc: Acee Lindem (acee); [email-masked]<mailto:[email-masked]>; Tony Przygienda

Bruno –

[Les:] This is not what I am asking for. Yes – your draft is very clear about the meaning of the values you intend to advertise.
I am asking how you calculate values to advertise.
I assume you intend to use some sort of input from the dataplane about IS-IS PDU input status – drops, queue state, etc. – but as you do not discuss this at all I have no idea what the requirements are on the dataplane to provide input to IS-IS in the control plane.

[Les:] Saying that it “should be possible” does not help. An implementation would have to have some logic based on a set of inputs – but your draft currently does not discuss the inputs nor the algorithm that might be used to derive the advertisements from the inputs.
Again, I have thought about this and I have been unable to come up with a set of inputs which would both allow calculation of accurate values and be readily available from the dataplane.
Please specify.

With the same spirit, can you explain how the parameters used in your implementation are determined? (Was already asked by Tony during latest LSR meeting)
[Les:] https://tools.ietf.org/html/draft-ginsberg-lsr-isis-flooding-scale-02#section-3.1 fully specifies a candidate algorithm.
It specifies what is configured and what is calculated.
The input is:

CurrentUackLSP:  Current number of unacknowledged LSPs already
                     transmitted

This is already known today because implementation of the IS-IS Update process requires tracking of what LSPs have been sent and are still waiting for acknowledgment – and this is tracked on a per interface basis.  This is the combination of the “SRM’ flag and the retransmit timer.

[Les:] This is a NIT – no need to discuss this further.

   Les

--Bruno

   Les

Cc: Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>>; Les Ginsberg (ginsberg) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Tony Przygienda <[email-masked]<mailto:[email-masked]>>

Robert,

--Bruno

Cheers,
R.

Hi Bruno,

Robert,

True.

Thanks,
BR
--Bruno

Best
--Bruno

Best,
R.

-          …

Regards,
Bruno

Hi Bruno  & all,

Many thx,
R.

--Bruno

my 2c

-- tony

Thanks
--Bruno

-- tony

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

Thank you.

---

From: [email-masked]
Date: Mon, 27 Apr 2020 11:18:12 -0700

Hi Robert,

> Today from what I see operators (if they even change the default) normally apply same timer value on all interfaces. If the timer is static what would be the incentive for any implementation not to group interfaces with identical transmit delay ? 

Why should the timer be static in an optimal system?  We want to avoid the need for the timer and have systems be adaptive.

> While this thread is very interesting I must observe that from my experience the issue is usually on the receiver. If LSR would publish a one page draft/rfc mandating that links state packets MUST or SHOULD be recognized and separated from any other control plane traffic at the ingress interface level (on their way to local RE/RP) we likely wouldn't be having such debate. 
> 
> Slowing senders just due to bad implementation of the receiving router is IMHO a little suboptimal (not to say wrong) thing to do. 

Heck, we can do better than that: we can outlaw bad implementations of anything.  Do you think that will help? 

The fact of the matter is that even good implementations can congest.  As silicon continues to scale, the ratio of interface bandwidth to control plane processing power continues to shift. Silicon has completely taken over our forwarding planes and scales upwards, where a single chip is now forwardinging for hundreds of interfaces. Meanwhile, buffering is finite and control planes really can’t keep up. Forwarding is a parallel activity. The control plane is not.  This presents us with a situation where congestion is pretty much inevitable. We need to deal with it.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 20:38:45 +0200

> Meanwhile, buffering is finite and control planes really can’t keep up.
> Forwarding is a parallel activity. The control plane is not.  This presents
> us with a situation where congestion is pretty much inevitable. We need to
> deal with it.

At least we are lucky that ISIS LSPs are produced by some other control
plane not by a silicon chip :)

If we have 1000 of interfaces and all peers *all in the same time* will
send us an LSP of max size of 1492 octets that our control plane buffer RAM
size required to store them would be as huge as 1.5 MB. And that assumes we
did not process any from arrival of the first to the arrival of the last
one.

Thx,
R.

---

From: [email-masked]
Date: Mon, 27 Apr 2020 11:47:32 -0700

> If we have 1000 of interfaces and all peers *all in the same time* will send us an LSP of max size of 1492 octets that our control plane buffer RAM size required to store them would be as huge as 1.5 MB. And that assumes we did not process any from arrival of the first to the arrival of the last one. 

And that’s only one LSP.  If they don’t stop there and each sends 1000 LSPs, then you can have 10^6 incoming packets, requiring 1.6GB.

Further, since the bottleneck is likely the queue of packet on the forwarding chip(s) to the CPU, this 1.6GB needs to exist on the forwarding silicon.  Needless to say, it doesn’t.

Yes, the CPU can probably keep up with one of the peers. This implies that the forwarding plane queue grows at the rate that 999 peers are sending at. Thus, congestion.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 21:11:32 +0200

Ahh ok.

I was under the assumption that flooding reduction is something we will
have sooner then LSP flow control. But maybe this was too optimistic.

- - -

For per peer flow control I do not get how receiver's ISIS process is to
come up with per peer timer if it may never see under congestion given
peer's LSPs (being dropped on the single RE cp queue or at the interface).

Perhaps such flag to "slow down guys" could be send by receiver uniformly
to all peers when under LSP flooding congestion ? And instead of
specifying an absolute number just mean: slow N times (ex: slow 2 times or
slow 4 times). Such timer could be either temporary with fixed duration or
in effect until relaxed explicitly.

LSP senders would still require to handle it on a per peer basis but as
indicated this is not an issue.

Thx.
R.

On Mon, Apr 27, 2020 at 8:57 PM <[email-masked]> wrote:

> And that’s only one LSP.  If they don’t stop there and each sends 1000
> LSPs, then you can have 10^6 incoming packets, requiring 1.6GB.
>
> Further, since the bottleneck is likely the queue of packet on the
> forwarding chip(s) to the CPU, this 1.6GB needs to exist on the forwarding
> silicon.  Needless to say, it doesn’t.
>
> Yes, the CPU can probably keep up with one of the peers. This implies that
> the forwarding plane queue grows at the rate that 999 peers are sending at.
> Thus, congestion.
>
> Tony
>
>

---

From: [email-masked]
Date: Mon, 27 Apr 2020 12:19:59 -0700

Robert,

> For per peer flow control I do not get how receiver's ISIS process is to come up with per peer timer if it may never see under congestion given peer's LSPs (being dropped on the single RE cp queue or at the interface). 

I’m sorry, but I can’t parse this comment.  The intent is not for the receiver to specify a timer value.  The point is for the receiver to provider the LSP sender with feedback about available resources on the receiver. This can inform the sender’s computation of a reasonable transmit bandwidth.

> Perhaps such flag to "slow down guys" could be send by receiver uniformly to all peers when under LSP flooding congestion ?

That’s effectively what we’re proposing, tho it need not be a binary flag.  It allows us to do simpler things saying “we’re running out of buffer space, please slow down a bit”. Again the goal is to find the optimal goodput.  Granularity in the feedback will be helpful.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 27 Apr 2020 21:40:19 +0200

>
> > Perhaps such flag to "slow down guys" could be send by receiver
> uniformly to all peers when under LSP flooding congestion ?
>
> That’s effectively what we’re proposing, tho it need not be a binary
> flag.  It allows us to do simpler things saying “we’re running out of
> buffer space, please slow down a bit”. Again the goal is to find the
> optimal goodput.  Granularity in the feedback will be helpful.
>

Apologies if I missed it but so far I understood that the new signalling
from the receiver could be different per each LSP sender (per each Hello).

Above I am suggesting that such signalling to be "global" per receiver
(ie.sent identical to all LSP senders) under moments of stress/congestion.

Robert.

---

From: [email-masked]
Date: Mon, 27 Apr 2020 12:52:44 -0700

Robert,

> Apologies if I missed it but so far I understood that the new signalling from the receiver could be different per each LSP sender (per each Hello). 

Correct.  We propose to piggyback feedback information in IIH’s, and *SNPs.

> Above I am suggesting that such signalling to be "global" per receiver (ie.sent identical to all LSP senders) under moments of stress/congestion. 

How does that help?

Also, in systems where the area multiple forwarding plane silicon instances, the resources for two different interfaces may have very different control plane resources available.  A single response would not allow the receiver to accurately describe his situation.

Tony