Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and meth-05

From: Al Morton <[email-masked]>
Date: Sat, 01 Aug 2009 07:13:25 -0400

At 12:27 PM 7/31/2009, Al Morton wrote:
>...
>I'm hoping to finish this task on the flight home...
>
>At 02:24 PM 7/2/2009, Al Morton wrote:
>>This message begins a Last call on the Sub-IP Protection terms and methods.
>>
>>http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-protection-term/
>>http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-protection-meth/

Comments on term-06:

Section 1.2

While the General Model applies well to some protection
mechanisms (e.g. SONET), it omits key aspects of others.
Since the first technology bmwg where bmwg is applying these
terms is MPLS-FRR, and MPLS-FRR is not well-described by the
General Model, it seems appropriate to give the reader
a heads-up. So at the end of Sec 1.2, I suggest:
NEW
Some protection switching technologies may use a series of
steps that differ from the general model. The specific differences
SHOULD be highlighted in each technology-specific methodology.
Note that some protection switching technologies are endowed
with the ability to re-optimize the working path after a
node or link failure.

Section 3.5 (and 3.6) - suggested top-level text:
The following Benchmarks MAY be assessed on a per-flow basis using
at least 16 flows spread over the routing table (more flows is 
better). Otherwise, the impact of a prefix-dependency in the implementation
of a particular protection technology could be missed.
However, the test designer must be aware of the number of
packets per second sent to each prefix, as this establishes
sampling of the path and the time resolution for measurement
of Failover time on a per-flow basis.

in 3.6:
The following Methods MAY ... <same as above>

Section 3.6.1, 3.6.2, 3.6.3 units are in seconds, but
Section 3.5.3. and 3.5.4 units are in milliseconds.
These should match.

----------------------
Comments on meth-05:
----------------------

Following the changes in the term draft Introduction,
I suggest to add the following at the end of the meth Introduction:

...
MPLS Fast Re-Route (MPLS-FRR)
allows for the possibility that the Label Switched Paths can
be re-optimized in the minutes following Failover. IP Traffic
would be re-routed according to the preferred path for the post-failure
topology. Thus, MPLS-FRR includes an additional step to the
General model:

    1. Failover Event - Primary Path (Working Path) fails
    2. Failure Detection-  Failover Event is detected
    3. a. Failover - Working Path switched to Backup path
    3. b. Re-Optimization of Working Path (possible change from Backup Path)
    4. Restoration - Primary Path recovers from a Failover Event
    5. Reversion (optional) - Working Path returns to Primary Path

----------------------------------------------------------------------
Is there any limitation that should be added to the Scope
(with this more complete view of MPLS-FRR in mind)?
In other words, are the methodology and benchmarks robust-enough to
measure the impact of *both* Failover and Re-optimization?

Attempting to answer these questions myself, it appears that all the
3 methods are currently worded to emphasize measurement of the initial
Failover event.

So, this should be made clear in the Scope section (2):
NEW
As described above, MPLS-FRR may include a Re-optimization of the
Working Path, with possible packet transfer impairments.
Characterization of Re-optimization is beyond the scope of this memo.

-----------------------------------------------------------------------

Section 5.7
Suggest replacing some text with the following:
NEW
At least 16 flows should be used, and more if possible (3 is not sufficient).
Prefix-dependency behaviors are key in IP and tests with route-specific
flows spread across the routing table will reveal this dependency.

Section 7.
At present, all of the three Failover Time methods are equally
preferred, but it appears that at least a recommendation would be
appropriate.
Both the Time-based Loss Method and the Packet Loss
based method will have difficulty assessing any
protection switch that is successfully conducted in a
make-before-break fashion. The Time-stamp based method
would be able to detect Reversion impairments beyond loss, thus it
should be RECOMMENDED method in section 7.

Also in section 7:
There should be a reference to Section 4 of RFC 2544 to capture
important considerations for the results, such are repeatability:
NEW
The considerations of Section 4 of [RFC2544] are applicable
when evaluating the results obtained using these methodologies as well.

Section 7.1:
This seems like an appropriate place to refer to
draft-ietf-bmwg-mpls-forwarding-meth- ...
and this passage in particular, from section 4:
    ... Throughput may vary depending on the number of ports used,
    and other factors. The number of ports (p) used SHOULD be reported.
    Please see Test Setup in Section 6, and recommended to follow Fig 1
    from S6 of RFC2544.

Section 7.5: references to "6.5 to 6.8" are no longer valid
(there are no such sections in the meth draft).

Section 8:
There is a note at the end which is difficult to interpret:
OLD
    Note: If the primary is configured to be dynamic, and if the primary
    is to reroute, make before break should occur from the backup that
    is in use to a new alternate primary. If there is any packet loss
    seen, it should be added to failover time.
Question:
Is this Note referring to "Re-optimization" as described above?
The intent needs to be made more clear in this text.
Also, if we limit the Scope to Failover Time, then this "reroute"
referred to above would not be in scope.
BTW, I don't think it is appropriate to simply add the loss
observed during Failover with any loss during reroute/re-optimization.

---

From: "Kris Michielsen" <[email-masked]>
Date: Mon, 3 Aug 2009 15:56:11 +0200

Hi,

Please find some comments/questions below.

comments on term-06:

* The definitions don't seem to be consistent. A "Path" is defined in the draft as a sequence of nodes/links from R1 (ingress of IP
packets) and Rn (egress of IP packets). Other definitions in the draft point out that a "Backup Path" is not necessarily end-to-end,
but can start and end at any node along the "Primary Path". If the Backup Path is not end-to-end, one can also not claim that the
Backup Path becomes the Working Path upon a Failover Event.

* It needs to be made clear that the definitions of the different paths are not applicable to MPLS TE LSPs. The definition of a path
in the draft is a physical sequence of nodes/links, while a TE LSP is a logically signalled path. A headend may resignal a new TE
LSP following a failure. This new TE LSP may in general follow another sequence of nodes/links and may possibly no longer be routed
over the "backup TE LSP" while the failed link/node is still not restored.

* I think it is difficult to distinguish Failover Events (for sub-IP) from Convergence Events (for IGP). The distinction is possible
for e.g. SONET APS. But how do we distinguish the two for e.g. MPLS TE FRR? A link or node failure will trigger IGP convergence as
well in that case and could potentially result in additional packet loss.

* The Benchmarks (3.5) give no detail about which packet loss needs to be observed. Is it for traffic going over all Primary Paths,
or a single Primary Path? Is it to a single destination reachable over a Primary Path or all destinations reachable over a Primary
Path?

* The definition of Failover Packet Loss (3.5.1) doesn't need to point out start and end of traffic loss, how the start end end are
defined doesn't seem right anyway (not for multiple streams anyway). I think it is important to take the time between Failover Event
and first observed packet loss as well in case the failure does not cause instantaneous packet loss. Shouldn't there be a "Sustained
Failover Validation Time", i.e. a period in which there is no aditional packet loss before declaring Failover as completed?

* What are the definitions of the "Time(xxx)" terms used in 3.6.1?

* It needs to be pointed out that the PLBM gives the _average_ loss over the measured destinations.

comments on meth-05:

* 5.1 I don't think shutdown is a good method to simulate a failure since the actions taken by a shutdown are very much
implementation dependent. Shutdown should only be considered as an administrative action.

* 5.2 Can't the remote fault indication in the ethernet auto-negotiation scheme be considered as a type of link failure indicator
for directly connected devices? In that configuration it doesn't need to rely on layer3 failure detection.

* 5.6 Shouldn't reoptimization (signal a new LSP while the failure still persists) be added to this section?

* 5.7 If the failover is prefix/LSP dependent, chosing only 3 routes to measure is not a good idea. In that case as much prefixes as
possible (limited by required accuracy and Throughput) are needed. Maybe a scheme to randomly select as much prefixes as possible
from the total number could be considered. One can also first test if the failover is prefix/LSP independent and make use of that
characteristic to reduce the number of traffic destinations that need to be measured. If there is a dependency on the number of
prefixes, the number of LSPs, ... then how is this reflected in the measurements? Is only the fastest failover time important, or
the slowest, or the average?

* 5.8 When tunnels are established from the Tester, it should be capable to resignal/reoptimize LSPs following the receipt of a path
error from the PLR, just as real devices do

* 6 It is confusing to mark R1 as HE and R2 as MID while their actual roles may vary depending on the testcase

* 8 "Packet Size" is it layer2, layer3 ? "Forwarding rate" should be "Offered Load"?
* 8 BGP routes, is this only applicable for VPN PEs?
* 8 What is a "FRR tunnel", the primary or the backup? I think it's important to indicate the scale of what is protected, besides
the global network scale. The Benchmarks need to indicate how they are measured, which method.
* 8, p 24, bullet 1. s/Rate/Loss/
* 8 The note at the end of the section should be more elaborated on (and may have implications throughout the document).

Thanks,
Peter and Kris

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Al Morton
> Sent: 02 July 2009 20:25
> To: [email-masked]
> Subject: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and meth-05
> 
> BMWG,
> 
> The Last Call with end on July 31, 2009.
> 
> We've been discussing this topic in BMWG almost as long as I 
> been chairman.  Two original authors of the terms had to 
> leave us, but others stepped-up and have brought this work to 
> a good state of completeness.  In fact, I've recently shared 
> our terms draft with some colleagues in the MEF who were 
> quite impressed.
> 
> Please weigh-in on whether or not these Internet-Drafts 
> should be given to the Area Directors and IESG for 
> consideration and publication as an Informational RFCs.  Send 
> your comments to this list or [email-masked].
> 
> Al
> bmwg chair
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
>

---

From: "Jay Karthik (jakarthi)" <[email-masked]>
Date: Tue, 4 Aug 2009 13:33:28 -0400

Hi Kris,

Thanks for your detailed review. I have responded to your comments on
the methodology draft here. We will get back with our response to your
comments on the terminology draft.

Please see inline.

Cheers,
Jay

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kris Michielsen (kmichiel)
Sent: Monday, August 03, 2009 9:56 AM
To: 'Al Morton'; [email-masked]
Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and meth-05

<Snip>

comments on meth-05:

* 5.1 I don't think shutdown is a good method to simulate a failure
since the actions taken by a shutdown are very much implementation
dependent. Shutdown should only be considered as an administrative
action.

Jay: Agreed that the shutdown is very much implementation dependent and
hence the reason for including this as failover trigger event. As you
can see this is 1 of the failure events and we have listed a few other
events as well.

* 5.2 Can't the remote fault indication in the ethernet auto-negotiation
scheme be considered as a type of link failure indicator for directly
connected devices? In that configuration it doesn't need to rely on
layer3 failure detection.

Jay: Sure - we shall modify the text with your suggestion explicitly for
directly connected device.

* 5.6 Shouldn't reoptimization (signal a new LSP while the failure still
persists) be added to this section?

Jay - The authors discussed this and based on the comments we received
earlier, we have mentioned the make-before-break scenario towards the
end of section 8. Would you prefer that we include this in section 5.6
as well ?

* 5.7 If the failover is prefix/LSP dependent, chosing only 3 routes to
measure is not a good idea. In that case as much prefixes as possible
(limited by required accuracy and Throughput) are needed. Maybe a scheme
to randomly select as much prefixes as possible from the total number
could be considered. One can also first test if the failover is
prefix/LSP independent and make use of that characteristic to reduce the
number of traffic destinations that need to be measured. If there is a
dependency on the number of prefixes, the number of LSPs, ... then how
is this reflected in the measurements? Is only the fastest failover time
important, or the slowest, or the average?

Jay - Kris, we have not mandated 3 routes/streams. You probably picked
the 3 routes from the example we had furnished. We also mention at the
beginning of this section that more streams could be used as long as the
flow is steady across all the traffic streams/routes.

* 5.8 When tunnels are established from the Tester, it should be capable
to resignal/reoptimize LSPs following the receipt of a path error from
the PLR, just as real devices do

Jay - Sure but we did not see much value in including benchmarking
scenarios that would validate the Tester as opposed to the DUT. Btw,
most if not all of the testers we know do not support reoptimization as
of now.

* 6 It is confusing to mark R1 as HE and R2 as MID while their actual
roles may vary depending on the testcase

Jay - Not sure where the confusion arises from. We believe all the
different scenarios could be tested by positioning R1 and HE and R2 as
MID nodes

* 8 "Packet Size" is it layer2, layer3 ? "Forwarding rate" should be
"Offered Load"?

Jay - Packet Size is w.r.t layer3. Forwarding Rate is used in "IGP
dataplane convergence term" documents and we are consistent with that.

Jay - We meant BGP routes from global table, just like we mentioned IGP
routes. Please note, we have explicitly listed VPN routes as well 

* 8 What is a "FRR tunnel", the primary or the backup? I think it's
important to indicate the scale of what is protected, besides the global
network scale. The Benchmarks need to indicate how they are measured,
which method.

Jay - "FRR tunnel" as in "Protected tunnel". We have left the decision
making on which method of the PBLM, TBLM and TBM to the individual
measuring the benchmarks.

Jay - Not sure what you are referring to. I looked for this pattern and
don't see any.

* 8 The note at the end of the section should be more elaborated on (and
may have implications throughout the document).

Jay - We can elaborate on that.

<Snip>

Thanks,
Jay

---

From: "Kris Michielsen" <[email-masked]>
Date: Wed, 5 Aug 2009 18:03:31 +0200

Jay,

see below.

> From: Jay Karthik (jakarthi) [mailto:[email-masked]] 
> Sent: 04 August 2009 19:33
> To: Kris Michielsen (kmichiel); Al Morton; [email-masked]
> Subject: RE: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 
> and meth-05
> 
> Hi Kris,
> 
> Thanks for your detailed review. I have responded to your 
> comments on the methodology draft here. We will get back with 
> our response to your comments on the terminology draft.
> 
> Please see inline.
> 
> Cheers,
> Jay
> 
> Behalf Of Kris Michielsen (kmichiel)
> * 5.1 I don't think shutdown is a good method to simulate a 
> failure since the actions taken by a shutdown are very much 
> Jay: Agreed that the shutdown is very much implementation 
> dependent and hence the reason for including this as failover 
> trigger event. As you can see this is 1 of the failure events 
> and we have listed a few other events as well.
> 

If the shutdown is implementation dependent, how can we compare benchmarks of different vendors' implementations?
An example of an "Interface shutdown on remote side with POS Alarm": One implementation "A" immediately stops receiving traffic
after an interface shutdown and sends AIS. Another implementation "B" just sends AIS after interface shutdown but keeps forwarding
traffic coming in from the shutdown interface. Implementation "B" is also very slow in triggering FRR after receiving AIS (e.g.
~200ms), while implementation "A" reacts swiftly to AIS (e.g. ~10ms). The benchmark test for implementation "A" will see some packet
loss (e.g. 10ms), while implementation "B" will see zero packet loss, but for a _real_ failure implementation "B" may see more
packet loss (e.g. ~200ms).

To test the reaction to different kinds of layer1 failure indications some additional equipment (same as used in real networks) is
required to inject these failure indications in the DUTs. This can go with or without traffic loss, just as in reality. Also higher
layer failures may cause immediate traffic loss or not. To be able to compare benchmarks the characteristics of a Failover event
need to be the same or at least documented/reported.

An interface shutdown is not to be considered as a real "failure", but merely as an administrative action. Measuring packet loss
following such an administrative action is useful, but a user/tester is free to decide how to take an interface administratively out
of service (which can be more complicated than just doing a "shut" on an interface).

> * 5.2 Can't the remote fault indication in the ethernet 
> auto-negotiation scheme be considered as a type of link 
> failure indicator for directly connected devices? In that 
> configuration it doesn't need to rely on
> Jay: Sure - we shall modify the text with your suggestion 
> explicitly for directly connected device.
> 
> Jay - The authors discussed this and based on the comments we 
> received earlier, we have mentioned the make-before-break 
> scenario towards the end of section 8. Would you prefer that 
> we include this in section 5.6 as well ?

It needs to be explained that a reoptimization can take place while the failure is still not recovered, but it doesn't need to take
place, also depending on the DUT configuration. Since it doesn't fit the title/subject of 5.6 it may be better to put it in a new
section before 5.6.
In the Reporting format it needs to be indicated if this reoptimization takes place or not.

> 
> * 5.7 If the failover is prefix/LSP dependent, chosing only 3 
> routes to measure is not a good idea. In that case as much 
> prefixes as possible (limited by required accuracy and 
> Throughput) are needed. Maybe a scheme to randomly select as 
> much prefixes as possible from the total number could be 
> considered. One can also first test if the failover is 
> prefix/LSP independent and make use of that characteristic to 
> reduce the number of traffic destinations that need to be 
> measured. If there is a dependency on the number of prefixes, 
> the number of LSPs, ... then how is this reflected in the 
> measurements? Is only the fastest failover time important, or 
> Jay - Kris, we have not mandated 3 routes/streams. You 
> probably picked the 3 routes from the example we had 
> furnished. We also mention at the beginning of this section 
> that more streams could be used as long as the flow is steady 
> across all the traffic streams/routes.

Guidelines are needed on how much and which streams need to be used when benchmarking. This is required to avoid misinterpretations
of the benchmark metrics. If one picks only a handfull stream destinations out of a total of e.g. 5000, then in a prefix or tunnel
dependent implementation one cannot know what is being measured (unless maybe when using white-box information). Are those traffic
streams going to the prefixes and/or tunnels that failover first or last or somewhere inbetween?

> 
> * 5.8 When tunnels are established from the Tester, it should 
> be capable to resignal/reoptimize LSPs following the receipt 
> of a path error from the PLR, just as real devices do
> 
> Jay - Sure but we did not see much value in including 
> benchmarking scenarios that would validate the Tester as 
> opposed to the DUT. Btw, most if not all of the testers we 
> know do not support reoptimization as of now.

Then why is there a need for Tester capability 1. in section 5.8?
While we're at it, are Tester capabilities 2. and 5. really needed? If so, when?

> 
> * 6 It is confusing to mark R1 as HE and R2 as MID while 
> their actual roles may vary depending on the testcase
> 
> Jay - Not sure where the confusion arises from. We believe 
> all the different scenarios could be tested by positioning R1 
> and HE and R2 as MID nodes

I meant the indications in the topology figures, such as Figure 2. below (indicated with =><=).

#             -------    -------- PRI  --------  
#            |  R1   |  |   R2   |    |   R3   | 
#         TG-|=>HE<= |--|=>MID<= |----|    TE  |-TA 
#            |       |  |  PLR   |----|        | 
#             -------    -------- BKP  -------- 
#      
#                          Figure 2. 

And e.g. the testcase section 7.2 where R2 is Headend. This does not match with "HE" and "MID" in Figure2, and leads to confusion.

# 7.2. Headend PLR with Link Failure  
...
#      1. Establish the primary LSP on R2 required by the topology 
#         selected. 
#      2. Establish the backup LSP on R2 required by the selected 
#         topology. 

> Jay - Packet Size is w.r.t layer3. Forwarding Rate is used in 
> "IGP dataplane convergence term" documents and we are 
> consistent with that.

Then it needs to be indicated in the draft it's layer3 packet size.
Forwarding Rate is not the correct term to use here. It should be Offered Load. Forwarding Rate varies during the test (e.g. drops
to zero when pulling the cable), Offered Load is constant/predefined.

> Jay - We meant BGP routes from global table, just like we 
> mentioned IGP routes. Please note, we have explicitly listed 
> VPN routes as well 

In the IGP dataplane convergence draft, all other protocols are banned from the DUT. If you permit to have BGP on the DUT, many more
things need to be specified. How many BGP nexthops? How many BGP prefixes are routed on tunnel(s)? ...

> 
> * 8 What is a "FRR tunnel", the primary or the backup? I 
> think it's important to indicate the scale of what is 
> protected, besides the global network scale. The Benchmarks 
> need to indicate how they are measured, which method.
> 
> Jay - "FRR tunnel" as in "Protected tunnel". We have left the 
> decision making on which method of the PBLM, TBLM and TBM to 
> the individual measuring the benchmarks.

Then it's better to indicate it as "Protected tunnel" in the draft.

It needs to be reported which method is being used to derive the benchmark metrics.

> Jay - Not sure what you are referring to. I looked for this 
> pattern and don't see any.

This is the bullet I refer to.

#      1. Packet-Based Loss method (PBLM): (Number of packets 
#        dropped/packets per second * 1000) milliseconds. This method 
#        could also be referred as Rate Derived method. 
                                   ^^^^^^^^^^^^--> Loss-Derived

I just noticed that the PBLM is called Packet-Loss Based Method (PLBM) in term-06

> 
> * 8 The note at the end of the section should be more 
> elaborated on (and may have implications throughout the document).
> 

Thanks,
Kris

---

From: "Scott Poretsky" <[email-masked]>
Date: Wed, 5 Aug 2009 19:52:02 +0300

Kris,

The purpose of the BMWG is to develop informational standards to
benchmark performance of different vendor implementations.  If every
vendor has the same implementation then there would be no need for BMWG
because every vendor would have the same performance.  How a Cisco admin
shut impacts layer 3 performance versus a Juniper admin shut is a very
real world performance issue that is within scope of the BMWG.

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kris Michielsen
Sent: Wednesday, August 05, 2009 12:04 PM
To: 'Jay Karthik (jakarthi)'; 'Al Morton'; [email-masked]
Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and meth-05

Jay,

see below.

> and we have listed a few other events as well.

If the shutdown is implementation dependent, how can we compare
benchmarks of different vendors' implementations?
An example of an "Interface shutdown on remote side with POS Alarm": One
implementation "A" immediately stops receiving traffic
after an interface shutdown and sends AIS. Another implementation "B"
just sends AIS after interface shutdown but keeps forwarding
traffic coming in from the shutdown interface. Implementation "B" is
also very slow in triggering FRR after receiving AIS (e.g.
~200ms), while implementation "A" reacts swiftly to AIS (e.g. ~10ms).
The benchmark test for implementation "A" will see some packet
loss (e.g. 10ms), while implementation "B" will see zero packet loss,
but for a _real_ failure implementation "B" may see more
packet loss (e.g. ~200ms).

To test the reaction to different kinds of layer1 failure indications
some additional equipment (same as used in real networks) is
required to inject these failure indications in the DUTs. This can go
with or without traffic loss, just as in reality. Also higher
layer failures may cause immediate traffic loss or not. To be able to
compare benchmarks the characteristics of a Failover event
need to be the same or at least documented/reported.

An interface shutdown is not to be considered as a real "failure", but
merely as an administrative action. Measuring packet loss
following such an administrative action is useful, but a user/tester is
free to decide how to take an interface administratively out
of service (which can be more complicated than just doing a "shut" on an
interface).

> we include this in section 5.6 as well ?

It needs to be explained that a reoptimization can take place while the
failure is still not recovered, but it doesn't need to take
place, also depending on the DUT configuration. Since it doesn't fit the
title/subject of 5.6 it may be better to put it in a new
section before 5.6.
In the Reporting format it needs to be indicated if this reoptimization
takes place or not.

> across all the traffic streams/routes.

Guidelines are needed on how much and which streams need to be used when
benchmarking. This is required to avoid misinterpretations
of the benchmark metrics. If one picks only a handfull stream
destinations out of a total of e.g. 5000, then in a prefix or tunnel
dependent implementation one cannot know what is being measured (unless
maybe when using white-box information). Are those traffic
streams going to the prefixes and/or tunnels that failover first or last
or somewhere inbetween?

Then why is there a need for Tester capability 1. in section 5.8?
While we're at it, are Tester capabilities 2. and 5. really needed? If
so, when?

> and HE and R2 as MID nodes

I meant the indications in the topology figures, such as Figure 2. below
(indicated with =><=).

And e.g. the testcase section 7.2 where R2 is Headend. This does not
match with "HE" and "MID" in Figure2, and leads to confusion.

Then it needs to be indicated in the draft it's layer3 packet size.
Forwarding Rate is not the correct term to use here. It should be
Offered Load. Forwarding Rate varies during the test (e.g. drops
to zero when pulling the cable), Offered Load is constant/predefined.

> mentioned IGP routes. Please note, we have explicitly listed 

In the IGP dataplane convergence draft, all other protocols are banned
from the DUT. If you permit to have BGP on the DUT, many more
things need to be specified. How many BGP nexthops? How many BGP
prefixes are routed on tunnel(s)? ...

It needs to be reported which method is being used to derive the
benchmark metrics.

I just noticed that the PBLM is called Packet-Loss Based Method (PLBM)
in term-06

Thanks,
Kris

---

From: "Kris Michielsen" <[email-masked]>
Date: Thu, 6 Aug 2009 10:22:30 +0200

Scott,

I'm not arguing against testing performance when administratively taking a link out of service. It's indeed a valid real world
performance issue. But a network adminstrator that cares about customer network uptime usually doesn't do that by a plain interface
shutdown instead of following procedures such as set overload bit, cost-out link, shut adjacency, ...

I have a problem with claiming that an interface shutdown is a _failure_ scenario. Please reread my comment below with the example
of implementations "A" and "B". Which of the two devices is the best performing (and hence expected to show the best benchmark
metric) in your opinion?

Performance of taking an interface adminstratively out of service (using a real procedure/sequence of actions chosen by the
user/tester -- even plain interface shutdown if that is what the user/tester cares about) should be measured, but under the title of
Adminstrative Actions, not claiming it to be a failure.

Thanks,
Kris

> From: Scott Poretsky [mailto:[email-masked]] 
> Sent: 05 August 2009 18:52
> To: Kris Michielsen; Jay Karthik (jakarthi); Al Morton; [email-masked]
> The purpose of the BMWG is to develop informational standards 
> to benchmark performance of different vendor implementations. 
>  If every vendor has the same implementation then there would 
> be no need for BMWG because every vendor would have the same 
> performance.  How a Cisco admin shut impacts layer 3 
> performance versus a Juniper admin shut is a very real world 
> performance issue that is within scope of the BMWG.
> 
> Scott
> 
> Behalf Of Kris Michielsen
> > Subject: RE: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and 
> > meth-05
> > 
> > Hi Kris,
> > 
> comments on 
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Kris Michielsen (kmichiel)
> > Jay: Agreed that the shutdown is very much implementation dependent 
> > and hence the reason for including this as failover trigger 
> event. As 
> > you can see this is 1 of the failure events and we have 
> listed a few 
> > other events as well.
> > 
> 
> If the shutdown is implementation dependent, how can we 
> compare benchmarks of different vendors' implementations?
> An example of an "Interface shutdown on remote side with POS 
> Alarm": One implementation "A" immediately stops receiving 
> traffic after an interface shutdown and sends AIS. Another 
> implementation "B"
> traffic coming in from the shutdown interface. Implementation 
> "B" is also very slow in triggering FRR after receiving AIS (e.g.
> The benchmark test for implementation "A" will see some 
> packet loss (e.g. 10ms), while implementation "B" will see 
> zero packet loss, but for a _real_ failure implementation "B" 
> may see more packet loss (e.g. ~200ms).
> 
> To test the reaction to different kinds of layer1 failure 
> indications some additional equipment (same as used in real 
> networks) is required to inject these failure indications in 
> the DUTs. This can go with or without traffic loss, just as 
> in reality. Also higher layer failures may cause immediate 
> traffic loss or not. To be able to compare benchmarks the 
> characteristics of a Failover event need to be the same or at 
> least documented/reported.
> 
> An interface shutdown is not to be considered as a real 
> "failure", but merely as an administrative action. Measuring 
> packet loss following such an administrative action is 
> useful, but a user/tester is free to decide how to take an 
> interface administratively out of service (which can be more 
> complicated than just doing a "shut" on an interface).
> 
> > auto-negotiation scheme be considered as a type of link failure 
> > indicator for directly connected devices? In that configuration it 
> > doesn't need to rely on
> It needs to be explained that a reoptimization can take place 
> while the failure is still not recovered, but it doesn't need 
> to take place, also depending on the DUT configuration. Since 
> it doesn't fit the title/subject of 5.6 it may be better to 
> put it in a new section before 5.6.
> > * 5.7 If the failover is prefix/LSP dependent, chosing only 
> 3 routes 
> > to measure is not a good idea. In that case as much prefixes as 
> > possible (limited by required accuracy and
> > Throughput) are needed. Maybe a scheme to randomly select as much 
> > prefixes as possible from the total number could be considered. One 
> > can also first test if the failover is prefix/LSP 
> independent and make 
> > use of that characteristic to reduce the number of traffic 
> > destinations that need to be measured. If there is a 
> dependency on the 
> > number of prefixes, the number of LSPs, ... then how is 
> this reflected 
> > in the measurements? Is only the fastest failover time 
> important, or 
> probably picked 
> > beginning of this section that more streams could be used 
> as long as 
> > the flow is steady across all the traffic streams/routes.
> 
> Guidelines are needed on how much and which streams need to 
> be used when benchmarking. This is required to avoid 
> misinterpretations of the benchmark metrics. If one picks 
> only a handfull stream destinations out of a total of e.g. 
> 5000, then in a prefix or tunnel dependent implementation one 
> cannot know what is being measured (unless maybe when using 
> white-box information). Are those traffic streams going to 
> the prefixes and/or tunnels that failover first or last or 
> somewhere inbetween?
> 
> > 
> > * 5.8 When tunnels are established from the Tester, it should be 
> > capable to resignal/reoptimize LSPs following the receipt of a path 
> > most if not all of the testers we know do not support 
> reoptimization 
> > as of now.
> 
> And e.g. the testcase section 7.2 where R2 is Headend. This 
> does not match with "HE" and "MID" in Figure2, and leads to confusion.
> 
> Forwarding Rate is not the correct term to use here. It 
> should be Offered Load. Forwarding Rate varies during the 
> test (e.g. drops to zero when pulling the cable), Offered 
> Load is constant/predefined.
> 
> > 
> > Jay - We meant BGP routes from global table, just like we mentioned 
> > IGP routes. Please note, we have explicitly listed VPN 
> routes as well
> 
> In the IGP dataplane convergence draft, all other protocols 
> are banned from the DUT. If you permit to have BGP on the 
> DUT, many more things need to be specified. How many BGP 
> nexthops? How many BGP prefixes are routed on tunnel(s)? ...
> 
> > 
> > important to indicate the scale of what is protected, besides the 
> > global network scale. The Benchmarks need to indicate how they are 
> > measured, which method.
> > 
> > Jay - Not sure what you are referring to. I looked for this pattern 
> > and don't see any.
> 
> > * 8 The note at the end of the section should be more elaborated on 
> > (and may have implications throughout the document).
> >

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Mon, 10 Aug 2009 17:30:19 -0400

I agree with Kris. In fact, one would reason that the interface failure
due to 'shutdown' (executed by the admin) may provide different result
from that of the interface failure that happened on its own (fiber cut,
for ex).

It is a mere title change, IMO.

Cheers,
Rajiv

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Kris
> Michielsen (kmichiel)
> Sent: Thursday, August 06, 2009 4:23 AM
> To: 'Scott Poretsky'; Jay Karthik (jakarthi); 'Al Morton';
[email-masked]
> I'm not arguing against testing performance when administratively
taking a
> link out of service. It's indeed a valid real world
> performance issue. But a network adminstrator that cares about
customer
> network uptime usually doesn't do that by a plain interface
> shutdown instead of following procedures such as set overload bit,
cost-out
> link, shut adjacency, ...
> 
> I have a problem with claiming that an interface shutdown is a
_failure_
> scenario. Please reread my comment below with the example
> of implementations "A" and "B". Which of the two devices is the best
> performing (and hence expected to show the best benchmark
> metric) in your opinion?
> 
> Performance of taking an interface adminstratively out of service
(using a
> real procedure/sequence of actions chosen by the
> user/tester -- even plain interface shutdown if that is what the
user/tester
> cares about) should be measured, but under the title of
failure
> > > Jay: Agreed that the shutdown is very much implementation
dependent
> > > Jay - We meant BGP routes from global table, just like we
mentioned
> > > Jay - Not sure what you are referring to. I looked for this
pattern

---

From: "Scott Poretsky" <[email-masked]>
Date: Tue, 11 Aug 2009 00:48:31 +0300

Great.  We will maintain the 'shutdown' test case in the document.

Scott

-----Original Message-----
From: Rajiv Asati (rajiva) [mailto:[email-masked]] 
Sent: Monday, August 10, 2009 5:30 PM
To: Kris Michielsen (kmichiel); Scott Poretsky; Jay Karthik (jakarthi);
Al Morton; [email-masked]
Subject: RE: [bmwg] WGLC: draft-ietf-bmwg-protection term-06 and meth-05

Cheers,
Rajiv

Of Kris
[email-masked]
taking a
customer
cost-out
_failure_
(using a
user/tester
failure
dependent
mentioned
pattern