Subject: Re: [bmwg] Proposal on Protection Benchmarking

From: <[email-masked]>
Date: Wed, 31 May 2006 22:54:44 -0600

BMWG:

I support the continued development of this work item and will endeavour to review the relevant document(s) by 6/23.

Regards,
Martin Lai
Agilent Technologies 

-----Original Message-----

Date: Fri, 26 May 2006 16:45:34 -0400
From: Al Morton <[email-masked]>
Subject: Re: [bmwg] Proposal on Protection Benchmarking
To: [email-masked]
Message-ID:
	<[email-masked]>
Content-Type: text/plain; charset="us-ascii"; format=flowed

REMINDER:

The comment period on this proposal closes next week (and messages on this thread are few).

At 03:30 PM 5/3/2006, Al Morton wrote:
>BMWG,
>
>The proponents of the Protection Benchmarking Work Proposal have 
>prepared the description of this work effort, below.
>
>BMWG discussed this work at the Dallas IETF-65 session, where there was 
>strong support and involved membership (see meeting minutes).
>
>Please weigh-in on whether this topic should become part of BMWG's 
>chartered work, by
>
>                    June 2, 2006
>
>And, if you support the work, please say:
>
>+  HOW you intend to support the development in BMWG,
>    (by reviewing draft X by MM/DD/YY, for example),  or
>
>+  WHY this work will be beneficial to BMWG's user community, or
>
>+  Modifications that would make the proposal more useful
>    (which we will discuss on the list), and
>
>+  (anything else that's constructive)
>
>And remember, we'd like to hear your opinion on the list, even if you 
>spoke in favor of this proposal at the meeting.
>
>Thanks in advance for your efforts and commitment to BMWG!
>
>Al
>bmwg co-chair
>
>=======================================================================
>==
>
>Proposal for Protection Benchmarking
>
>
>Summary:
>
>This is a proposal for a BMWG work item for black-box benchmarking of 
>sub-IP protection mechanisms at the IP Layer. These mechanisms function 
>at sub-IP layers with the purpose of protecting IP traffic, making it 
>possible to develop benchmarks for these mechanisms by measuring the 
>data plane at the IP layer.  This work item will develop a common 
>terminology to be used for protection benchmarking at the data plane of 
>the IP Layer and multiple methodology documents for each sub-IP 
>protection mechanism.  Such mechanisms include but are not limited to 
>MPLS Fast Reroute, SONET APS, and High Availability.  The methodologies 
>and benchmarks can be used to compare performance of a protection 
>mechanism on one device to another or to compare performance of two or 
>more protection mechanisms.
>
>
>Goal:
>
>The objective is to produce a terminology document for protection 
>benchmarking at the IP Layer and methodology documents for benchmarking 
>protection mechanisms at the IP layer.  Path failures and rerouting are 
>benchmarked through packet loss measurement on the data plane.  There 
>will be a common terminology to be used for protection benchmarking at 
>the IP Layer and multiple methodology documents for each sub-IP protection mechanism.
>
>
>Scope:
>
>The focus of this effort is to benchmark Sub-IP protection mechanisms at 
>the IP Layer for a DUT or SUT.   Such mechanisms include but are not 
>limited to MPLS Fast Reroute, SONET APS, and High Availability.  The 
>benchmark parameters that are measured are independent of the layer 3 
>routing protocol being run.  The data plane is measured to obtain the 
>IP layer benchmarks. Control Plane scaling will be considered in the 
>test methodologies to measure its impact to the IP Layer benchmarks. In 
>addition, this effort will characterize the duplicate and reordered 
>packets within the definition of recovery time. The presence of 
>duplicate and errored packets may be tied to the failover event.
>
>
>Motivation:
>
>Service Providers are currently deploying sub-IP protection mechanisms 
>to minimize or eliminate IP layer packet loss and control plane 
>failure.  The networking device failover performance times have become 
>critical metrics for deployment selection.  Faster networking device 
>failover times have directly led to faster network failure recovery times.
>
>  Network users consider packet loss a key service metric. Intervals of 
> high packet loss are externally observable events having direct impact 
> on user application performance. Numerous sub-IP protection mechanisms 
> have been developed and are now being deployed to minimize IP packet 
> loss upon a failure.  For this reason it is very important to develop 
> uniform protection mechanism benchmarks for comparing networking 
> device performance.  Service providers can use these benchmarks to 
> compare performance of a protection mechanism on one device to another 
> or to compare performance of two or more protection mechanisms.
>
>
>Schedule:
>
>"Terminology For Protection Benchmarking,"
>
>draft-poretsky-protection-term-01.txt, Ready for WGLC 07/06 and Ready 
>for IESG 10/06.
>
>
>"Methodology For MPLS Protection Benchmarking,"
>
>draft-poretsky-mpls-protection-meth-05.txt
>
>
>MERGE WITH
>
>"Methodology for benchmarking fast fail-overtime with local protection,"
>
>draft-vapiwala-bmwg-frr-failover-meth-00.txt
>
>
>TO CREATE
>
>"Methodology for MPLS Protection Benchmarking,"
>
>draft-ietf-bmwg-mpls-protection-meth-00.txt
>
>
>FIRST SUBMITTAL 06/06 and Ready for WGLC 12/06 and Ready for IESG 04/07.
>
>
>These documents (Terminology and Methodology) have already undergone 
>extensive working group review.

------------------------------

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

End of bmwg Digest, Vol 23, Issue 11
************************************

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Thu, 15 Jun 2006 15:47:34 -0400

Curtis,

Thanks for suggesting this text. We are adding this (mostly as is) to
the merged draft that will be posted shortly.

Jay 

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Wednesday, June 14, 2006 4:41 PM
To: Jay Karthik (jakarthi)
Cc: [email-masked]; [email-masked]
Subject: Re: FW: [bmwg] Proposal on Protection Benchmarking 

Jay,

How about this text.

    Two or more failures are considered correlated if those failures
    occur more or less simultaneously.  Correlated failures are often
    expected where two or more logical resources, such as layer-2
    links, rely on a common physical resource, such as common
    transport.  TDM and WDM provide multiplexing at layer-2 and
    layer-1 that are often the cause of correlated failures.

    Where such correlations are known, such as knowing that two
    logical links share a common fiber segment, the expectation of a
    common failure can be compensated for by specifying Shared Risk
    Link Groupgs [cite something like FRR RFC].

    Not all correlated failures are anticipated in advance of their
    occurance.  Failures due to natural disasters or due to certain
    man-made disasters or mistakes are the most notable causes.
    Failures of this type occur many times a year and generally a
    quite spectacular failure occurs every few years.

    There are two factors impacting service availablity.  One is the
    frequency of failure.  The other is the duration of failure.  FRR
    improves availability by minimizing the duration of the most
    common failures.  Unexpected correlated failures are less common.
    Some routers recover much more quickly than others and therefore
    benchmarking this type of failure may also be useful.

    Benchmarking of unexpected correlated failures should include
    measurement of restoration with and without the availability of IP
    fallback.  The use BGP free core may be growing, making the latter
    case an important test case.

    This document focuses on FRR failover benchmarking with MPLS TE.
    Benchmarking of unexpected correlated failures is out of scope but
    may be covered by a later document.

Trim, snip, cut or dispose of as you pleae.  Consider it a suggestion.

Curtis

---

From: "Arun Gandhi" <[email-masked]>
Date: Mon, 19 Jun 2006 14:09:40 -0400

Please see my response ("in favor") below ...

> From: Jay Karthik (jakarthi) [mailto:[email-masked]] 
> Sent: Wednesday, June 14, 2006 11:25 AM
> To: [email-masked]; Samir Vapiwala (svapiwal)
> Cc: Rajiv Papneja
> Subject: RE: [bmwg] Proposal on Protection Benchmarking
> 
> 
> Hi Arun,
> 
> Thanks for getting back on this.
> 
> Could you please send your "in favor" mail to the list as 
> well (cc'ing Al).
> 
> Cheers and TIA,
> Jay 
> 
> From: Arun Gandhi [mailto:[email-masked]] 
> Sent: Wednesday, June 14, 2006 10:42 AM
> To: Samir Vapiwala (svapiwal)
> Cc: Jay Karthik (jakarthi); 'Rajiv Papneja'
> Subject: RE: [bmwg] Proposal on Protection Benchmarking
> 
> Hi Samir:
> 
> Sorry for the delay as was out travelling with limited
> email access.
> 
> I do support this work proposal and look forward to 
> contributing to it. I believe it shall cater well to
> the network operators/service providers by providing
> equable way for them to compare performance between
> different vendor devices and methodologies.
> 
> Thanks,
> Arun
> 
> From: Samir Vapiwala (svapiwal) [mailto:[email-masked]]
> Sent: Tuesday, May 30, 2006 10:55 AM
> To: [email-masked]
> Cc: Jay Karthik (jakarthi); Rajiv Papneja
> Subject: FW: [bmwg] Proposal on Protection Benchmarking
> 
> 
> Hi Arun,
>  
> Thanks for your review comments to the draft
> "draft-vapiwala-bmwg-frr-failover-meth-00.txt ". We have
> addressed all your comments in the merged draft which will
> be posted shortly.
>  
> As you might have seen Al's email on the bmwg list, we
> are currently proposing a "benchmarking protection"
> work item and like to seek your support on this. We
> would greatly appreciate your support to this work item. 
>  
> Please respond to Al's email saying why (or why not) you 
> support (or not support) this work proposal.
> 
> Al's mail is attached below
> 
> Thanks 
> Samir
>  
> > From: Al Morton [mailto:[email-masked]]
> > Sent: Friday, May 26, 2006 4:46 PM
> > To: [email-masked]
> > >BMWG discussed this work at the Dallas IETF-65 session, where
> > >there was strong support and involved membership (see meeting 
> > >minutes).
> > >
> > >sub-IP protection mechanisms at the IP Layer. These mechanisms 
> > >function at sub-IP layers with the purpose of protecting 
> IP traffic, 
> > >making it possible to develop benchmarks for these mechanisms by 
> > >measuring the data plane at the IP layer.  This work item will 
> > >develop a common terminology to be used for protection 
> benchmarking 
> > >at the data plane of the IP Layer and multiple methodology 
> documents 
> > >for each sub-IP protection mechanism.  Such mechanisms include but 
> > >are not limited to MPLS Fast Reroute, SONET APS, and High
> Availability.
> > >The methodologies and benchmarks can be used to compare performance
> > >of a protection mechanism on one device to another or to compare 
> > >performance of two or more protection mechanisms.
> > >
> > >
> > >Goal:
> > >
> > >benchmarking at the IP Layer and methodology documents for 
> > >benchmarking protection mechanisms at the IP layer. Path 
> failures and 
> > >rerouting are benchmarked through packet loss measurement 
> on the data 
> > >plane. There will be a common terminology to be used for 
> protection 
> > >benchmarking at the IP Layer and multiple methodology 
> documents for 
> > >each sub-IP protection mechanism.
> > >
> > >
> > >Scope:
> > >
> > >The focus of this effort is to benchmark Sub-IP protection 
> mechanisms 
> > >at the IP Layer for a DUT or SUT. Such mechanisms include 
> but are not 
> > >test methodologies to measure its impact to the IP Layer 
> benchmarks. 
> > >In addition, this effort will characterize the duplicate and
> reordered
> > >Service Providers are currently deploying sub-IP protection 
> > >mechanisms to minimize or eliminate IP layer packet loss 
> and control 
> > >plane failure.  The networking device failover performance 
> times have 
> > >become critical metrics for deployment selection.  Faster 
> networking 
> > >device failover times have directly led to faster network failure 
> > >recovery times.
> > >
> > >high packet loss are externally observable events having direct 
> > >impact on user application performance. Numerous sub-IP protection 
> > >mechanisms have been developed and are now being deployed 
> to minimize 
> > >IP packet loss upon a failure. For this reason it is very 
> important 
> > >to develop uniform protection mechanism benchmarks for comparing 
> > >networking device performance. Service providers can use these 
> > >benchmarks to compare performance of a protection mechanism on one 
> > >device to another or to compare performance of two or more 
> protection 
> > >mechanisms.
> > >
> > >
> > >Schedule:
> > >

---

From: "Rajiv Papneja" <[email-masked]>
Date: Sun, 25 Jun 2006 11:29:06 -0400

Hello Aamer,

Further to our previous email exchange. The merged draft has been posted,
and here is the link to the merged draft 

www.ietf.org/internet-drafts/draft-papneja-mpls-protection-meth-merge-00.txt

Your comments are greatly appreciated. 

Regards,
Rajiv

> >  From: Rajiv Papneja [mailto:[email-masked]]
> >  Sent: Wednesday, May 31, 2006 11:17 AM
> >  To: Aamer Akhter (aakhter); 'Al Morton'; [email-masked]
> >  Thanks Aamer,
> >  We will be posting the merged draft very soon on the mailing
> >  list. Thanks
> >  for considering reviewing the document. Once we post the
> >  draft, we will post
> >  the link to the mailing list.
> >
> >  Regards,
> >  Rajiv
> >
> >
> >  > From: Aamer Akhter (aakhter) [mailto:[email-masked]]
> >  > Sent: Wednesday, May 31, 2006 9:33 AM
> >  > To: Al Morton; [email-masked]
> >  > BMWG folks,
> >  > Apologies for the delay in responding.
> >  >
> >  > I do support the work on measurement of protection
> >  technologies. In the
> >  > coming drive for stricter SLAs we need to have a discussion and
> >  > methodology on what aspects a tester should consider and
> >  how to go about
> >  > this testing.
> >  >
> >  > I will review (by 6/25):
> >  >
> >  > or if
> >  > is out by next week, I'll review that.
> >  >
> >  >
> >  > --
> >  > Aamer Akhter / [email-masked]
> >  > NSITE, cisco Systems
> >  >
> >  >
> >  > >  Sent: Wednesday, May 03, 2006 3:30 PM
> >  > >  To: [email-masked]
> >  > >  Subject: [bmwg] Proposal on Protection Benchmarking
> >  > >
> >  > >  BMWG,
> >  > >
> >  > >  The proponents of the Protection Benchmarking Work Proposal
> >  > >  have prepared the description of this work effort, below.
> >  > >
> >  > >  BMWG discussed this work at the Dallas IETF-65 session,
> >  > >  where there was strong support and involved membership
> >  > >  (see meeting minutes).
> >  > >
> >  > >  Please weigh-in on whether this topic should become part
> >  > >  of BMWG's chartered work, by
> >  > >
> >  > >                      June 2, 2006
> >  > >
> >  > >  And remember, we'd like to hear your opinion on the list,
> >  > >  even if you spoke in favor of this proposal at the meeting.
> >  > >
> >  > >  =============================================================
> >  > >  ============
> >  > >
> >  > >  sub-IP protection mechanisms at the IP Layer. These
> >  > >  mechanisms function at
> >  > >  sub-IP layers with the purpose of protecting IP traffic,
> >  > >  making it possible
> >  > >  to develop benchmarks for these mechanisms by measuring the
> >  > >  data plane at
> >  > >  the IP layer.  This work item will develop a common
> >  > >  terminology to be used
> >  > >  for protection benchmarking at the data plane of the IP
> >  > >  Layer and multiple
> >  > >  methodology documents for each sub-IP protection
> >  mechanism.  Such
> >  > >  mechanisms include but are not limited to MPLS Fast Reroute,
> >  > >  SONET APS, and
> >  > >  High Availability.  The methodologies and benchmarks can be
> >  > >  used to compare
> >  > >  performance of a protection mechanism on one device to
> >  another or to
> >  > >  compare performance of two or more protection mechanisms.
> >  > >
> >  > >
> >  > >  Goal:
> >  > >
> >  > >  benchmarked through packet loss measurement on the data
> >  > >  plane.  There will
> >  > >  be a common terminology to be used for protection
> >  > >  benchmarking at the IP
> >  > >  Layer and multiple methodology documents for each
> >  sub-IP protection
> >  > >  mechanism.
> >  > >
> >  > >
> >  > >  Scope:
> >  > >
> >  > >  mechanisms at
> >  > >  routing protocol being run.  The data plane is measured to
> >  > >  obtain the IP
> >  > >  layer benchmarks. Control Plane scaling will be considered
> >  > >  in the test
> >  > >  methodologies to measure its impact to the IP Layer
> >  benchmarks. In
> >  > >  addition, this effort will characterize the duplicate and
> >  > >  reordered packets
> >  > >  within the definition of recovery time. The presence of
> >  > >  duplicate and
> >  > >  errored packets may be tied to the failover event.
> >  > >
> >  > >
> >  > >  Motivation:
> >  > >
> >  > >  mechanisms to
> >  > >  minimize or eliminate IP layer packet loss and control plane
> >  > >  failure.  The
> >  > >  networking device failover performance times have become
> >  > >  critical metrics
> >  > >  for deployment selection.  Faster networking device failover
> >  > >  times have
> >  > >  directly led to faster network failure recovery times.
> >  > >
> >  > >  high packet loss are externally observable events having
> >  > >  direct impact on
> >  > >  user application performance. Numerous sub-IP protection
> >  > >  mechanisms have
> >  > >  been developed and are now being deployed to minimize IP
> >  > >  packet loss upon a
> >  > >  failure.  For this reason it is very important to
> >  develop uniform
> >  > >  protection mechanism benchmarks for comparing networking device
> >  > >  performance.  Service providers can use these
> >  benchmarks to compare
> >  > >  performance of a protection mechanism on one device to
> >  another or to
> >  > >  compare performance of two or more protection mechanisms.
> >  > >
> >  > >
> >  > >  Schedule:
> >  > >
> >  > >  draft-poretsky-protection-term-01.txt, Ready for WGLC 07/06
> >  > >  and Ready for
> >  > >  IESG 10/06.
> >  > >
> >  > >

---

From: "Aamer Akhter \(aakhter\)" <[email-masked]>
Date: Mon, 26 Jun 2006 00:35:08 -0400

BMWG folks,

I'd like to thank and commend the authors of the document for the through job they have done in capturing the failure cases. Well done.

A number of concerns have already been addressed (eg direction of fiber pull etc). Please find my comments for draft-papneja-mpls-protection-meth-merge-00.txt below:

============

++ general

* bidir traffic. I realize that TE tunnels are unidirectional, and I don't think there is any strong benefit in bi-directional traffic for these cases. However, I would still like to pose the question to the wider group: Are there any cases were having bidirectional tunnels might make a difference? One area I can think of is in the area of scalability (it is not as trivial as just dividing the unidirectional scale limit).

++ 1. Introduction

* are "Failover Time, Additive Latency, and 
     Reversion Time" defined?

++ 3.1

* "            - Interface bearing multiple sub-interfaces " isn't a failure

++ 3.2

* For Ethernet failures, Ethernet oam (specifically ethAis) can provide indication of failure

++ 3.3

*  "In extreme cases, where measurement of errored and duplicate 
        packets is difficult, these packets could be attributed to lost 
        packets." Should this be 'these anomalies' ?

* It might be nice to make a note of being able to detect multiple failures. For example the case where the protection initially works, then fails, then really starts working again.

++ 3.4

* "        Failover time performance may vary with the number of established 
        primary and backup LSPs and routes learned." Should be 

"        Failover time performance may vary with the number of established 
        Primary LSPs, backup LSPs and routes learned."

Also, is 'routes learned' the correct term. 'routes protected directly by PLR' perhaps?

++ 3.7

* I personally would like to see the multi-traffic stream changed from suggestion to a requirement. We already know that there is a delta between the time the protection mechanism starts and when it is complete. This is something that should be caught if it is too high. 

* Perhaps some discussion on why round-robin traffic generation is not recommended. This would be helpful in that the concerns might be mitigated via some changes to how the traffic is injected into the topology (via multiple ports and traffic generators). I'm not comfortable with an outright unrecommendation of traffic to all prefixes as that is the 'true' coverage case.

++ 4.x

* It would be nice to have a table for each node mapping to [P,PE].

* I think the P and B in the diagrams stand for (P)rimary and (B)ackup. But can we get a explicit legend somewhere.

* kind of a stupid point, but some of the diagrams (fig 12) are split across page boundaries

* Indentation for ip traffic (P-P) traffic matrix seems to be off

++ 5. 

* NEED TO CHECK. Why does the backup tunnel have to match bandwidth of the primary?

++ 5.1 

* "(If the node downstream of the PLR is not 
               A simulated node, then the Ingress of the tunnel should have  
               one link connected to the traffic generator and the node  
               downstream to the PLR or the egress of the tunnel should have 
               a link connected to the traffic analyzer)."

	Firstly, the above passage is confusing. And I'm not sure where the simulated node downstream from the PLR came from. It's not in a topology, so is it an option ?

* step 4: 'Verify FRR protection'. I think this means to verify that FRR is enabled/ready for whatever we want it to be enabled for. If not, can we make this statement clearer?

* after the failover is done, should there be a period of time that the traffic is kept on the backup path-- for example to make sure that RSVP for the primary path does not time out?

* I've already made a point about being able to detect the presence of multiple traffic failures.

* should there be guidance on (or capturing of) having 'background' LSPs and prefixes that are not protected?

++6 

* In reporting, should there be something about
	- % of prefixes that are being protected vs. not protected. 
	- % of LSPs that are being protected vs not protected
	- which topology was used
	- what kind of failure was used

Regards,

>  Sent: Sunday, June 25, 2006 11:29 AM
>  To: Aamer Akhter (aakhter)
>  Cc: [email-masked]; 'Al Morton'

---

From: "Rajiv Papneja" <[email-masked]>
Date: Fri, 30 Jun 2006 12:10:11 -0400

Hello Aamer,

Thanks for reviewing the merged draft so quickly and providing your
feedback. Please see inline.

/R

-----Original Message-----
From: Aamer Akhter (aakhter) [mailto:[email-masked]] 
Sent: Monday, June 26, 2006 12:35 AM
To: Rajiv Papneja; Samir Vapiwala (svapiwal)
Cc: Al Morton; [email-masked]
Subject: RE: [bmwg] Proposal on Protection Benchmarking

BMWG folks,

I'd like to thank and commend the authors of the document for the through
job they have done in capturing the failure cases. Well done.

A number of concerns have already been addressed (eg direction of fiber pull
etc). Please find my comments for
draft-papneja-mpls-protection-meth-merge-00.txt below:

============

++ general

* bidir traffic. I realize that TE tunnels are unidirectional, and I don't
think there is any strong benefit in bi-directional traffic for these cases.
However, I would still like to pose the question to the wider group: Are
there any cases were having bidirectional tunnels might make a difference?
One area I can think of is in the area of scalability (it is not as trivial
as just dividing the unidirectional scale limit).

[rp] We are interested in learning the response as well, and we will be very
happy to update the draft accordingly.

++ 1. Introduction

* are "Failover Time, Additive Latency, and 
     Reversion Time" defined?
[rp] Yes, they are defined in the terminology document,
draft-poretsky-protection-02.txt

++ 3.1

* "            - Interface bearing multiple sub-interfaces " isn't a failure
[rp] Just trying to understand your view point here, if VLANs are carrying
different LSPs, then the interface failure could trigger multiple LSP
rerouting.

++ 3.2

* For Ethernet failures, Ethernet oam (specifically ethAis) can provide
indication of failure

[rp] Certainly.

++ 3.3

* It might be nice to make a note of being able to detect multiple failures.
For example the case where the protection initially works, then fails, then
really starts working again.

[rp] Good scenario, we can consider this adding to the next revision of the
draft. 

++ 3.4

Also, is 'routes learned' the correct term. 'routes protected directly by
PLR' perhaps?
[rp] May be this is a better term. We can also define this in the term
document. 
[rp] We will try to add this in the next revision of the term document

++ 3.7

* I personally would like to see the multi-traffic stream changed from
suggestion to a requirement. We already know that there is a delta between
the time the protection mechanism starts and when it is complete. This is
something that should be caught if it is too high. 
[rp] We have had lot of discussion about this already. May be we should
consider adding multiple streams as a requirement. 

* Perhaps some discussion on why round-robin traffic generation is not
recommended. This would be helpful in that the concerns might be mitigated
via some changes to how the traffic is injected into the topology (via
multiple ports and traffic generators). I'm not comfortable with an outright
unrecommendation of traffic to all prefixes as that is the 'true' coverage
case.

[rp] Agree, we should address this and providing the reasoning in the next
revision about the why round-robin traffic generation is not recommended.

++ 4.x

* I think the P and B in the diagrams stand for (P)rimary and (B)ackup. But
can we get a explicit legend somewhere.
[rp] We should perhaps defined in the test setup.

* kind of a stupid point, but some of the diagrams (fig 12) are split across
page boundaries
[rp] We tried out best to post the draft accurately and making sure that
this does not happen, however, the posting of the draft and Internet
explorer could you some different formatting. I will send you a text file
offline, to show the draft that we submitted had accurate page breaks.

[rp] Thanks. We will fix it.

++ 5. 

* NEED TO CHECK. Why does the backup tunnel have to match bandwidth of the
primary?

++ 5.1 

	Firstly, the above passage is confusing. And I'm not sure where the
simulated node downstream from the PLR came from. It's not in a topology, so
is it an option ?
[rp] This is the scenario, when the test equipment is simulating all the
positions accept the DUT. 

* step 4: 'Verify FRR protection'. I think this means to verify that FRR is
enabled/ready for whatever we want it to be enabled for. If not, can we make
this statement clearer?
[rp] Yes, we can.

* after the failover is done, should there be a period of time that the
traffic is kept on the backup path-- for example to make sure that RSVP for
the primary path does not time out?

* I've already made a point about being able to detect the presence of
multiple traffic failures.

* should there be guidance on (or capturing of) having 'background' LSPs and
prefixes that are not protected?

[rp] Probably yes, could you please recommend something here.

++6 

[rp] Yes, these should be part of reporting but not reporting matrix.

Regards,