Subject: [bmwg] RE: comments on draft-sdry-bmwg-mvpnscale-00

From: "Silvija Andrijic Dry \(sdry\)" <[email-masked]>
Date: Thu, 1 Mar 2007 06:36:45 -0500

Hi Thomas and all,

Thomas, thank you for thorough review of the "00" version of this draft
and valuable feedback !
Btw, we just posted "01" version that addresses some of your feedback:
http://www.ietf.org/internet-drafts/draft-sdry-bmwg-mvpnscale-01.txt

Let me start this email by addressing couple of key points from the
summary of your email, and other details I will put inline (labeled with
[SD] and [SD-end]) In this email I'll refer to draft-l3vpn-2547bis-mcast
as [l3vpn-mcast], draft-rosen-vpn-mcast-08  as [rosen-8] and
draft-sdry-bmwg-mvpnscale as [mvpn-bm] (or this draft).
Also to avoid confusion I will use term "MVPN architectures" to refer to
various different sets of technologies/protocols that can be used to
implement MVPN service as described in [l3vpn-mcast] and term "MVPN
implementations" to refer to practical implementations of any given
specific "MVPN architecture".
I will refer to portion of [l3vpn-mcast] that addresses architecture
from [rosen-8] as "rosen-8 architecture".

1) Changing scope of the [mvpn-bm] to be agnostic of "architecture"
used:
------------------------------------------------------------------------
---
- We have considered your suggestion of making [mvpn-bm] agnostic of
MVPN "architecture" used, specifically agnostic of PE-PE C-Multicast
routing transmission protocol and core transport/signaling mechanism
used.
- We decided NOT to change the scope of [mvpn-bm], i.e. scope remains to
be on "rosen-8 architecture". However we do agree that large portion of
this draft is applicable to other architectures from [l3vpn-mcast] so we
did on your suggestion reformat the text of the draft so that it is easy
to identify parts that are applicable to all architectures from
[l3pvn-mcast] and we explicitly noted such parts in 01 doc. That will
allow this draft to be leveraged by other MVPN architectures from
[l3vpn-mcast] but at the same time we feel that this draft is not
complete for all architectures, thus opted not to change the scope of
[mvpn-bm]. Note that I feel that no architecture agnostic benchmarking
draft would be complete either and would not provide enough detail
necessary for fair comparison of different implementations of the same
architecture. Basically, I am of opinion that [mvpn-bm] includes all
relevant architecture agnostic components already but that architecture
agnostic benchmarking draft is not sufficient for any (or at least most)
of architectures described in [l3vpn-mcast]. 

2) Value of having benchmarking draft addressing only portion of
[l3pvn-mcast] ("rosen-8 architecture" in this case):
------------------------------------------------------------------------
--------
- I disagree that there is no value in draft that addresses only portion
of [l3vpn-mcast] (specifically "rosen-8 architecture"):
	a) significant number of operators that have "rosen-8
architecture" deployed today need now benchmarking document for this
architecture. Note that some of such operators are co-authors of this
([mvpn-bm]) draft.
	 b) Today "rosen-8 architecture" is only architecture from
[l3vpn-mcast] that is widely deployed and has multiple publicly
available implementations. Some of architectures and protocols proposed
in [l3vpn-mcast] are still being discussed and defined in IETF, thus it
will realistically take some time until all of proposed architectures
from [l3vpn-mcast] gain enough practical implementations, test and
deployment experience to write good and comprehensive benchmarking draft
for all of them. And also it is not yet clear that all of architectures
from [l3vpn-mcast] will have wide acceptance.
	 c) Given a), b) and the opinion that agnostic draft is not
sufficient we feel it is unwise to wait with the document needed by many
SPs for architecture that has number of implementations and deployments
today.
       D) even if b) wasn't true  benchmarking document that would
include specifics of all possible architectures from [l3vpn-mcast] would
be too long and complex. 

3) Suggestion to move to different document PE metric and tests that are
also applicable to P routers:
------------------------------------------------------------------------
-----------------------
- I agree it would be nice to be able to leverage single set of test
results for both P and PE router.
- However, I feel that completely moving to different document some of
test cases and metric from [mvpn-bm] that apply to PIM P-instance would
leave this draft incomplete and its test results of significantly lesser
value to operators deploying "rosen-8 architecture". Reason being is:
	a) On PE router both P and C PIM instances (and few other MVPN
related components) share same resources thus increased scale of
P-instance components will leave less resources for C instance
components and vice versa. 
	b) BMWG precludes use of device resource usage (such as CPU and
memory) as their measurement methods are not standardized across
implementations. Thus if you get benchmark on P instance components from
one draft and C instance components from other draft I don't see a way
for one 	to conclude what would happen when both P and C instance
components co-exist on the same device (which is always the case in
"rosen-8" MVPN on PE device). If you have any suggestions on how would
you address this problem I would be happy to hear.
	c) Btw, I'm not as concerned about removing test cases (as there
is really only one) as much as I'm concerned about removing metric
aspects of P instance as that's applicable to all test cases.

4) Terminology:
----------------
- Intent was to use both terminology from [l3vpn-mcast] and [rosen-8]
for reader's convenience as [rosen-8] terminology is more widely known
than [l3vpn-mcast] terminology. Note that [l3vpn-mcast] also makes
references and draws parallels where applicable to [rosen-8]
terminology.
- I agree "00" version was not consistent in this matter and this is
already rectified in "01" version. If there are any inconsistencies in
"01" I will be happy to fix those.

5) References to [rosen-8]:
-------------------------------
- [l3vpn-mcast] is written in such a manner that it is difficult to
describe any individual architecture (including [rosen-8] one) in short
way so that it can be easily referred to from other documents such as
[mvpb-bm]. For example it takes a very long sentence or paragraph to
describe "[rosen-8] architecture" using [l3vpn-mcast] terminology. Thus
for sake of readability you will notice in 01 we continue to refer to
[rosen-8] but only as a abbreviation for longer sentence describing this
architecture in [l3vpn-mcast] terms. I would be interested in any better
suggestions ? 

Rest of the comments are inline labeled as [SD]. I would be interested
on yours and the thoughts of rest of the BMWG on the topics discussed in
this email.

Btw, yes I absolutely agree document needs to be reviewed by L3VPN WG. I
sent email to mailing list before with 00 version and will now with 01
version. I also asked chair to ask WG during the Prague WG meeting to
review this draft.

Thx,

Silvija

-----Original Message-----
From: Thomas Morin [mailto:[email-masked]]
Sent: Tuesday, February 20, 2007 11:38 AM
To: Silvija Andrijic Dry (sdry)
Cc: Al Morton; [email-masked]
Subject: Re: comments on draft-sdry-bmwg-mvpnscale-00

Hi Silvija, Al,

I had a bunch of comments, which I made in San Diego, but which I didn't
find the time to write down before now. 

Here is a summary:
- having benchmarking guidelines on multicast vpn is definitely
something useful and a welcome initiative
- there is an issue regarding the terminology: the terms used comes most
of the time from draft-rosen-vpn-mcast-08 (has expired 1.5 year ago)
though the document is said to apply to both draft-rosen-vpn-mcast-08
and a subset of draft-l3vpn-2547bis-mcast (said differently: normative
references only should be required for understanding the document)
- I generally feel that it is a unwise use of resources to only target a
subset of a standard, especially when many of the benchmarks (if not
most) are relevant independently of the implementation choice (like it
is the case here).
- there is an issue regarding the mvpn implementation option to which
this draft applies : to exactly which sections of
draft-l3vpn-2547bis-mcast does it apply ? the document says it targets
implementations which "[use] PIM protocol to create tunnels that
instantiate MI-PMSIs and S-PMSIs", but reading the doc it seem to
actually implicitly make the additional assumption that option "Full
Per-MVPN PIM Peering Across a MI-PMSI" (described in section 5.2.1 of
draft-l3vpn-2547bis-mcast) also has to be used...  It seems to me that
the draft actually targets a _very_ restricted subset of the procedures
described in draft-l3vpn-2547bis-mcast. 
- some of the tests proposed are relevant to multicast in general,and
not specifically to multicast when used to carry VPN traffic. I believe
such tests would deserve being placed in a more generic document (when
benchmarks defined in RFC3918 are not enough).  Numerous tests in this
document are related to control plane stress due to how the mvpn
application uses multicast as provided by the operator infrastructure :
this is relevant not only for PE-routers, but also for P-routers. To
evaluate a platform or engineering options, multicast scalability should
separately be benchmarked, whether or not mvpn will be used. I think
such tests belong to a separate doc.

My suggestion would be to restructure the document so that everything
that is generic enough is described without making too much assumption
on how mVPN is implemented. When and if required, specific tests
applicable to a specific way of implementing mvpn could be also
described. Also, when applicable pointers to benchmarking documents
related to multicast should be referred (e.g. : RFC3918, RFC2432).
This would have the benefit of making the document more readable (eg.
the number of parameters would be greatly reduced).

Then, I would expect a review of such a benchmarking document by the
l3vpn working group, at some point.

[ Inlined below are a few more detailed comments on different sections,
including suggestions about which tests/section I feel should be
"factored", which could be kept separate, and which could be maybe
replaced by pointers to more generic documents.  ]

Cheers,

-Thomas Morin

It seems to me that benchmarking "steady state" and "failure state" is
fine for unicast, but that multicast-related benchmarking would require
testing also the "membership changes stress" case (i.e. how a device
performs when a lot of multicast membership changes happen).

[SD] Can you elaborate further what exactly you mean by "membership
changes"? If you refer to tests of how device performs when subjected to
large number of PIM Join/Prunes at the same time then failure recovery
tests should be able to assess this aspect as each of failure recovery
events will trigger such a stress condition. Or do you imply constant
series of Joins and prunes over longer period of time while rest of the
protocols are in steady state, i.e. something like channel flipping
scenarios? Note in section 7 all failure stimulus and execution
procedure and let me know what additional tests would you like to see
and we'll consider them.
[SD-end]

>    The MVPN Metric is defined as a tuple of the following 13
variables: 
> 
>    Global Domain Related Variables
> 
>   1. Num_mVPN: Number of multicast VPN routing instances configured on

>      DUT that have MI-PMSI active and forwarding. 
>   2. Num_*G_P: Total number of (*,G) multicast routes on DUT capable
of 
>      forwarding and created by PIM P-instance on DUT. 
>   3. Num_SG_P: Total number of (S,G) multicast routes on DUT capable
of 
>      forwarding and created by PIM P-instance. 
>   4. Num_OIF_P: Total number of OIFs (outgoing interfaces) on DUT
across 
>      all multicast routes created by PIM P-instance. 

Here is an example of how things may not be abstracted enough in this
draft: "Variables" 2 and 3 above are not in fact variables independent
from any other variables, they are dependent on the number of VPNs,
likely the average number of PEs in each multicast VPN, the option
chosen to carry multicast in the provider backbone. But they are not
intrinsic metrics of the mvpn deployment/simulation.

[SD]
It is true not all metric variables are independent of each other but if
you remove any of them you can not compare fairly test results. However
note that for your example "the average number of PEs in each multicast
VPN, the option chosen to carry multicast in the provider backbone" are
NOT defined as a METRIC in this draft, thus variables Num_SG_P and
Num_*G_P you mention are actually NOT dependant on any other METRIC
variables besides Num_mVPN. Of course I agree in general Num_SG_P and
Num_*G_P do depend on choice of P instance PIM variant chosen but( as
you can't assign numeric value to it) P instance PIM variant choice is
specified in the test setup of each individual test case and not in
metric. So if person A and person B execute same test case from this
draft and provide Num_SG_P and Num_*G_P (plus other metric) you can
compare apples to apples without having to complicate metric with non
numeric variables. 
In general principle in this draft is that test environment and
execution of each test case specifies any non numeric aspects and also
you will notice that numeric variables in test environment of each test
case are independent of each other. However most of them are not the
most suitable for metric as additional calculations are necessary to get
a picture of overall DUT limits. For ease of reading "01" version
provides relationships between metric variables for each test case as
well as relationship of metric variables to test environment variables. 

Btw, most of metric variables do depend on Num_mVPN variable but if you
exclude Num_mVPN you can end up making unfair comparisons. For example
in practical implementations maximum number of Num_*G_C,Num_SG_C and
Num_OIF_C might not be the same if all of them are in 1 vs 100 mVPNs for
example. Same goes for PIM neighborships for example (see discussion in
test case 9.3). 

This is precisely example for my argument that practical implementations
and test experience is required in order to come up with comprehensive
benchmarking document. Have we not had this for "rosen-8 architecture"
one might of excluded Num_mVPN from the metric if rest of variables are
dependant on it, which would of lead to comparing two unequivalent
environments.

Btw, criteria for choice of METRIC variables was driven by desire to
have set of variables that combined uniquely describes DUT scale profile
for any given test and which DIRECTLY take resources from PE routers
(for example Num_SG_P directly takes memory and CPU resources, while
choice of PIM protocol  in the core doesn't directly utilize any
resources but indirectly does by impacting Num_SG_P....similarly number
of PEs indirectly impact couple of metric variables but don't take
resources on the router themselves). Also metric variables are chosen as
aggregate values over all of mVPNs on the given PE router (which makes
them dependant on Num_mVPN). This is because most of people like to see
in the scale benchmark limits that reflect totals per DUT rather then
per for example mVPN per DUT (which requires additional calculations to
get totals per DUT)

[SD-end]

>   VPN Domains Related Variables: 
> 
>   1. Num_MC_C_ints: Number of PIM C-interfaces on DUT 
>   2. Num_PIM_C_neigh: Total number of PIM neighbors in PIM C-instances

>      across all mVPNs on DUT. 

Here we have I think another example: "Total number of PIM neighbors in
PIM C-instances across all mVPNs on DUT" should I think be reformulated
into "Total number of remote PEs across all mVPNs defined on DUT". It is
equivalent, more generic, and I think a little easier to read.

[SD] "Total number of remote PEs across all mVPNs defined on DUT" is not
really equivalent but "Average number of remote Pes per mVPN" (together
with Num_mVPN which is already defined) would provide equivalent
information. I considered the later, but at the end decided to go with
neighborships in the metric and PE's in test setup as PIM neighborships
are one of the most asked scale metric and most of people prefer to
easily identify it in the test result rather than having to calculate it
from other variables. In addition in spirit of trying to chose metric
variables that directly take router resources #PIM neighborships were
better suitable than #Pes. However, if community prefers to use "Average
number of remote Pes per mVPN" instead of neighborships as defined now
I'm open to it.
[SD-end]

> If there are multiple sources per group in a C-instance then they 
>    MUST be located behind the same PE router. 

(Why so ?)

[SD] Only bcs of simplification of setup and consistency between
different people implementing this draft. I removed this statement from
"01".
[SD-end]

> Test apparatus ports that are physically directly connected to DUT 
>    (port A1 in Figures 1 and 2) MUST not use IGMP protocol to emulate 
>    multicast receivers. Instead PIM protocol must be used, i.e. the
DUT 
>    MUST not be the last hop router.  

Why such a limitation ?
It should surely be precised in a report which of IGMP and PIM was used,
and the use of PIM should be recommended but I fail to see why
benchmarking the PE device as a last hop device would be precluded.

[SD] This is changed in 01, where IGMP is allowed to be used and
documented as an exception to the setup in the report.
[SD-end]

> 5.5 Test Apparatus Considerations
> 
>    Different test tools must generate PIM protocol control messages in
a 
>    consistent way since they are directly connected to the DUT. 
> 
>    The following MUST be implemented on all PIM sessions on the test 
>    apparatus: 
> 
>      1) PIM Join/Prune aggregation MUST be supported and set such that

>         80 PIM J/P messages are aggregated in each PDU      
> 
>      2) PIM Join/Prune aggregated PDUs MUST be sent at 10 PDUs/sec
rate 
>         per PIM session, i.e. this translates to maximum of 
>         80*10*60=48,000 state per minute. 

(Why these values ?)

[SD] To ensure consistancy when different test tools are used we need to
define some values, as note in the topology test tool directly attaches
to the DUT, thus stress on DUT will depend on the rate at which messages
are sent and we need to ensure consistency.
I'm open to other values as well. These are chosen as follows:
1) Picked to be such that close to maximum aggregation is achieved for
Ethernet frame. We considered just to put maximum possible aggregation
instead of value but then you would have to worry about L2 encap choice
of interfaces used on DUT, and didn't feel that's necessary for
scalability draft.
2) picked to be default rate of one test tools [SD-end]

> 6.2 Failure Recovery Testing
> 
> [...]
>      7) If any one of following conditions are reached the test case 
>        instance is considered failed and next test case instance with 
>        reduced value of scaled variable/s needs to be performed: 
> 
>           a. Value of MVPN metric in steady state reached after
failure 
>              stimuli (step 6 above) is not the same as in original 
>              steady state. 
> 
>           b. Multicast latency [RFC2432] averaged over all C-instance 
>              multicast flows in steady state after failure recovery 
>              stimuli is more than 10% larger than in original steady 
>              state

I'm not sure of the reason for this "10%" : e.g. having latency go from
50ms to 60ms is no sign of failure, and no issue either.  Maybe
introducing a lower value under which the test is considered a success,
would be a solution (eg. 80ms) ?

[SD]
For the "suggested" load detailed in the document, we consider that a
10% increases in latency  needs to be noticed  by the tester / reader
and based on the specific implementation, further investigation may need
to take place in order to determine whether scaled system tends to
experience any degradations in latency after recovery from stressed
conditions created by network failure events. 
[SD-end]

> 
>       8.4   Default MDT's Multicast State Scale.....................20

>       8.5   VRF Multicast State Scale...............................22

>       8.6   VRF Multicast OIF Scale.................................24

>       8.7   Joined Data MDT Scale...................................26

[ This one is an example of one of my introductory comment on including
tests related to multicast in general in this mvpn specific draft. 8.4
and 8.7 actually test the same thing : how PE devices scale in terms of
multicast states. This is duplicated work. Notwithstanding the fact I
would expect such a test to be described in a more generic document
(note that RFC3918 section 7.1 describes something similar).  ]

[SD] Actually 8.4 and 8.7 are NOT nearly the same. It is true that
Joined Data MDTs contribute to P instance PIM state creation and
maintenance just as Default MDTs do. However, there is more to default
and data MDTs than just mcast state maintenance, plus there are
significant differences between data and def MDT themselves. Here are
just some examples of the components that can contribute to scale for
def and data MDT but not for general mcast (RFC3918) or that differ for
def and data MDT's:
- mechanics of creation of default MDT and Data MDTs are different, thus
could incur different resources usage
- maintenance of mapping of C-instance state to P-instance state can be
different for def and data MDT's
- there are implementations that have additional type of resources (such
as tunnel interfaces) that are associated with both def and data MDT's
and impact overall scale capabilities of the system. While also there
are implementations that have such additional resources utilized only
for def MDT  but not for data MDTs for example.

Basically, point is that there is enough differences between default and
data MDTs and that these potential impacts needed to be considered.
After consideration test experience on practical implementations had
shown that these differences indeed do impact system scale significantly
enough that test cases we have in [mvpn-bm] (9.4,9.7, 9.8 and 9.9) need
to be included in benchmarking draft if one wishes to get complete
picture of scale capabilities of the implementation.
This is another example of point I was trying to make earlier in email
when discussing scope: that it is required to have couple of practical
implementations and some deployment/test experience prior to defining
benchmarking draft in order to fully understand impact and relevance of
all scale impacting variables, and this draft reflects such experience
for "rosen-8 architecture".
 
[SD-end]

>       8.9   Data MDT Reuse..........................................30

This specific test is I think complete nonsense : it implicitly suggests
that a PE device allowing more data-MDT groups to be reused is better.
This is nonsense because, depending on the deployment context, the
operator may very well have reason to actually prefer that no reuse
would happen, or to have controlled reuse (reuse data mdt groups up
to...) of groups allocated for data-MDTs.

[ This is IMHO a very illustrative example of a benchmarking suite
written by one vendor for his own platform. ]

[SD] Intent wasn't to suggest that having reuse is better and I don't
see such implications made in the text. Sorry if it came across as such.
Definitely benchmarking draft is not a place for such a claims. In "01"
I added some explicit statements to clarify this.
Yes, in ideal world no one would need to use reuse and agree depending
on the deployment context its benefits/drawbacks vary, however due to
various valid reasons and circumstances many operators do use it today.
All we want to do in this draft is to assess it's impact when it is used
and provide data for both operators using it and operators not using it.
That's why 8.7 and 8.8 requires that NO reuse is utilized (providing
data for operators such as yourself) and 8.9 is providing data for
operators that do use reuse. In "01" we actually go step further and
state not only that reuse test case is optional but also that all data
MDT related test cases are optional as they are not mandatory by
[l3vpn-mcast].
[SD-end]

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Silvija Andrijic Dry \(sdry\)" <[email-masked]>
Date: Thu, 1 Mar 2007 06:51:20 -0500

Hi Thomas and all,

4) Terminology:
----------------

Thx,

Silvija

-----Original Message-----
From: Thomas Morin [mailto:[email-masked]]
Sent: Tuesday, February 20, 2007 11:38 AM
To: Silvija Andrijic Dry (sdry)
Cc: Al Morton; [email-masked]
Subject: Re: comments on draft-sdry-bmwg-mvpnscale-00

Hi Silvija, Al,

Here is a summary:

Cheers,

-Thomas Morin

of 
of 
across 

[SD]

[SD-end]

(Why so ?)

DUT 

Why such a limitation ?

a 

rate 

(Why these values ?)

failure 

[SD]

 
[SD-end]