Subject: Re: draft-so-yong-rtgwg-cl-framework

From: Curtis Villamizar <[email-masked]>
Date: Thu, 14 Jun 2012 20:36:15 -0400

Iftekhar,

I'm replying to this email because this email thread accounts for most
of the changes that I just made to draft-so-yong-rtgwg-cl-framework.

Since the call for support as a WG item is ongoing I will not submit
the updates until that call is completed.

Briefly the changes to the document are:

  1.  Author affiliation change for Ning, address change for Lucy.
      @@ -61,23 +61,17 @@
      @@ -107,12 +101,12 @@

  2.  Some clarification in "Flow Identification" section resulting
      from your comments.
      @@ -474,29 +470,35 @@

  3.  A brief summary added to the "Flow Identification" section.
      @@ -474,29 +470,35 @@  (last addition in that set of diffs)

  4.  Wording changes suggested in this thread (see inline below).
      @@ -648,17 +680,18 @@

  5.  Significant rewording in "Composite Link in Data Plane" section
      to reduce redundancy and put the general description of traffic
      distribution here and issues later, but referencing here.
      @@ -770,24 +803,26 @@
      @@ -795,34 +830,38 @@

  6.  Clarified wording about large networks and scalability.
      @@ -833,12 +872,20 @@

  7.  Clarified wording about large routing changes.
      @@ -851,15 +898,32 @@

  8.  Removed redundant description of issues in "Data Plane
      Challenges" that were already covered in "Flow Identification"
      section or "Composite Link in Data Plane" section, and moved
      some text that was not covered in either place.
      @@ -1391,59 +1455,14 @@

  9.  Added clarifications and cross reference to "Very Large LSP"
      section.
      @@ -1453,8 +1472,18 @@

 10.  Better defined what a "very large microflow" is.
      @@ -1464,8 +1493,8 @@

 11.  Added some text to the acknowledgements.
      @@ -2506,9 +2535,22 @@

Diffs will be sent in a separate email - too long to attach.  The @@
notation above refers to specific diff chunks.  Followup to the most
recent email is inline.

It was pointed out in another thread that some text on carrying IP and
LDP is in more than one place and similar enough to be redundant.
I'll get that in a next set of changes.

Best Regards,

Curtis

In message <[email-masked]>
Curtis Villamizar writes:
>  
> Iftekhar,
>  
> Thank you for the detailed comments.  Responses to individual
> comments/suggestions are inline.
>  
> This is a long email message, so if I missed anything, please point
> out what I missed.
>  
> Please let me know if you are in general agreement with the responses
> and if so I'll make specific proposals for replacement text or propose
> more detailed action items.
>  
> Best Regards,
>  
> Curtis
>  
>  
> In message <[email-masked]>
> Iftekhar Hussain writes:
>  
> > Dear  Authors,
> >
> > Please find below some comments on the
> > http://tools.ietf.org/html/draft-so-yong-rtgwg-cl-framework-05.
> >
> >
> > ----------------------------
> >
> > 2.1. Flow Identification
> >
> > "Operator may have other objectives such as ...composite link energy
> > saving, and etc. These new requirements are described in
> > [I-D.ietf-rtgwg-cl-requirement]"
> >
> > Comment:
> >
> > I don't recall any energy saving related requirement (or discussion)
> > in the [I-D.ietf-rtgwg-cl-requirement.  Suggest removing the text
> > "energy saving etc."
>  
> You are entirely correct that there are no energy savings mentioned in
> the requirements document.  This wording came from one of the other
> authors but I think I understand the intent.  There is a daily cycle
> (and less so a weekly cycle) to traffic levels.  During the low
> traffic hours (generally late at night for local hours of night)
> traffic can be rebalanced to reduce the number of active component
> links.  Where it is possible to depower interfaces, this could result
> in energy savings.
>  
> I'm OK with removing this example, though it is a very good example
> and an goal that we should be pursuing.  Another option is to add a
> "MAY" in the requirements docuemt somewhere that indicates that "Load
> balancing MAY be used during sustained low traffic periods to reduce
> the number of active component links for the purpose of power
> reduction".
>  
> I'll wait for further comments on this before making a suggestion on
> how we will proceed.

No further comments on the mailing list on this.  The requirements was
added to CL requirements.

> > 2.2. Composite Link in Control Plane
> >
> > "LDP follows the IGP, therefore failure to forward on  the IGP path
> > will often result in loss of connectivity if the IGP adjacency is not
> > withdrawn when an LDP FEC is refused.  This is a pathologic case that
> > can occur if LDP is carried natively and there is a high volume of LDP
> > traffic.  This situation can be avoided by carrying LDP within RSVP-TE
> > LSP."
> >
> >
> > Comment:
> >
> > Is it the loss of connectivity referring to LDP control plane
> > connectivity only or LDP signaled LSP data plane or both?
>  
> The bottom line is that admission control cannot be applied to LDP
> without loss of connectivity.  Complete loss of connectivity for a
> subset of traffic, no matter how low its priority, is unacceptable.
>  
> LDP does not support TE.  There are two options.  Either don't carry
> enough LDP traffic such that this situation can occur (this is the
> case when a small subset of traffic is high priority traffic carried
> within LDP, such as VOIP or enterprise VPN traffic mixed with a much
> larger volume of Internet traffic).  The alternative is to carry the
> LDP traffic within RSVP-TE LSP when enough LDP traffic is aggregated
> such that TE is needed.
>  
> The goal was to refer to this problem without going into an
> excessively long explanation.  If you think it is too brief and
> therefore is unclear, then I'll reword it.

No further comments so I'm assuming that the wording was adequate as
is.

> > Comment:
> >
> > "Composite link capacity is aggregated capacity and MAY be larger than
> > individual component link capacity."
>  
> Before the MAY insert "LSP capacity" and the sentence makes more
> sense.  I'll reread this and see if that is what was intended.

Fixed in @@ -648,17 +680,18 @@ (#4 above)

> > Composite link aggregate capacity should always be larger than
> > individual component link capacity. Did I misunderstand?
>  
> The statement as written is not useful.  The sum of positive
> quantities is always greater than the quantities being summed.  Thanks
> for pointing out this statement.  I'll look at the context and figure
> out the intent.

Looked fine after the two word insertion mentioned above.

> > "...1. If no other information is available the largest microflow is
> > bound by one of the following:"
> >
> > Suggested text:
> >
> > If no other information is available the largest microflow is bound
> > (i.e., signaling extensions don't indicate a bound) by one of the
> > following:
>  
> I'll reword this.  The other information could be signaled or
> configured.

Added this in @@ -648,17 +680,18 @@ last change in the chunk.

> > Comment:
> >
> > Section 2.2 provides architectural guidelines e.g., "  ...Available
> > capacity in other component links MUST be used to carry impacted
> > traffic.  The available bandwidth after failure MUST be advertised
> > immediately to avoid looped crankback" and provides illustrative
> > examples e.g.,"... no microflow larger than 10 Gb/s will be present on
> > the RSVP-TE LSP that aggregate traffic across the core, even if the
> > core interfaces are 100 Gb/s interfaces."
>  
> Be careful not to take the last sentence out of the context of the
> example (where no interface feeding the core is greater than 10 Gb/s).

No issue here.

> > In contrast, section 2.1 appears to be little bit too vague e.g., "
> > ... technique of grouping flows, such as hashing on the flow
> > identification criteria, becomes essential to reduce the stored state,
> > and is an essential scaling technique.  Other means of grouping flows
> > may be possible"
> >
> > Suggestion:
> >
> > Add some examples which flow identification scheme to use for
> > composite links or add a reference to section 4.2 which discusses flow
> > identification trade-offs.
>  
> The intent in Section 2.1 is to make three points.  First, tracking
> every flow is not scalable.  Second, IP src/dst address hashing has
> proven (in over two decades of experience) to be an excellent way of
> identifying groups of flows (for reasons briefly listed).  Third, if
> you find a better way to identify groups of flows, then use it.
>  
> We don't want to require the use of IP address hashing, but wanted to
> strongly encourage its use, given its long history of successful
> deployment.
>  
> I'll look at rewording the section.

Above comments insired the changes in diff chunk @@ -474,29 +470,35 @@

> > 4.1.4. Requirements for Contained LSP
> >
> > Comment:
> >
> > Add reference to specific relevant to requirements in
> > I-D.ietf-rtgwg-cl-requirement] similar to section 4.1.2 and 4.1.3.
>  
> OK.  The sentence "[I-D.ietf-rtgwg-cl-requirement] calls for new LSP
> constraints." needs followup indicating which requirements are being
> referred to.

The existing text identifies the requirement topics.  I don't want to
embed the requirement numbers here since the numbers can change.

> > 4.2. Data Plane Challenges
> >
> > Comment:
> >
> > Minor typo: "very course..." should be "very coarse..."
>  
> Of coarse.  How could I miss that?  :-)
>  
> I searched for other uses of the word "course" and found quite a few
> that should be "coarse".  Thanks for the catch.
>  
> > Comment:
> >
> > "In practice  using the MPLS label stack alone has proven too course
> > to acheive a reasonably good load balance, due to bin-packing issues
> > and discrpencies between signaled bandwidth and actual traffic loads
> > on LSP."
> >
> > Suggest adding a reference to an IETF standard or best practices
> > document that discusses these issues.
>  
> OK, if I can find something.  This has been discussed on and off on
> the mailing lists for about a decade.  There may be something in the
> original link bundling RFC or elsewhere.

That text is gone.  The topic is covered in Flow Identification.
Another reference to draft-symmvo-rtgwg-cl-use-cases was added.
RFC4928 indicates that more than MPLS label stack is needed but
doesn't say why.  There is a little more detail in the Entropy Label
draft (now in last call).

> > Comment:
> >
> > Sections 4.2, 2.1, 2.3, appear to be somewhat redundant. Suggest
> > trimming section 2.1 and absorbing that information in section 4.2.
>  
> Thanks for pointing this out.  I'll reread and try to reduce
> redundancy, using cross references where appropriate rather than
> repeating a point.

Redundant information in 4.2 was removed.  Some information in 4.2 was
put into 2.3.  This left 2.1 with the background on current flow
identification and 2.3 with background on current traffic
distribution.

> > 3. Architecture Tradeoffs
> >
> > Comment:
> >
> > "Composite Link is applicable to large networks, and therefore
> > scalability must be a major consideration."
> >
> > What is a typical definition of a large network?  Suggestion: Add an
> > example (or a forward reference to section 3.3.1 which has an example)
>  
> Any network that one of your customers wants to build would be a good
> example of large.  :-)  [But not a good definition of large.]
>  
> By "large" we mean the type of networks that service providers or
> large content providers are building today.
>  
> In the future we may find that enterprises are building large enough
> private networks to have certain scaling problems that had previously
> only been experienced by service providers and content providers in
> the past.  A good example of that in the past is overuse of bridging.
> Some of the NSFNET regional networks in the late 1980s used bridging
> but experienced severe scaling issues very quickly.  It wasn't until
> the early to mid 1990s that large enterprises began having similar
> problems.
>  
> The point of the phrase "Composite Link is applicable to large
> networks" is that if you need parallel links because the single link
> capacity je jour is too small, then your network is large relative to
> most other networks.  Considering the magnitude of "large" where
> 10Gb/s links are too small, the phrase "and therefore scalability must
> be a major consideration" may seem almost rhetorical to some with
> deployment experience with IP networks.
>  
> The reason that the statement here is brief is that there are a number
> of IETF base documents dating back to the mid-1990s on the importance
> of scalability.  These realization came from deployment experience.
> Despite this someone occasionally makes a naive comment about
> processors getting faster and memory getting larger, ignoring the fact
> that comparable grouth with order NlogN or N-squared growth in
> computation or memory requirements far outpaces Moore's Law
> improvements.
>  
> I think at least one somewhat ancient RFC indicatea that scalability
> must be considered in every document in the IETF routing area.  If I
> find it I'll cite it.  It will probably have Brian Carpenter or
> Christian Huitman on the author list, making it easier to find.  Maybe
> Fred Baker.  I think it was an IAB or IESG RFC.

Best general IAB statement I found was "All designs must scale readily
to very many nodes per site and to many millions of sites." in RFC
1958 "Architectural Principles of the Internet" (1996).  RFC 2791
"Scalable Routing Design Principles" by Jessica Yu (2000) is very good
but focused on BGP.  RFC 3439 "Internet Architectural Guidelines" by
Randy Bush and David Meyer (2002) was an IAB document about scaling
but touched on a lot of topics without driving home any one point very
well (IMHO).

I didn't add any reference here.  I did make changes saying a bit
about what a "large network" means.

> > 3.1. Scalability Motivations
> >
> > Comment:
> >
> > "....a large routing change to be accomplished more quickly,"
> >
> > What is a typical definition of a large routing change?  Suggestion:
> > Add an example.
>  
> Not entirely serious: Hurricane Katrina resulted in a number of "large
> routing changes".  There was the collaspse of the Raratan River bridge
> in NJ taking out fibers on both sides of the bridge (thought to be
> redundant), an earthquake east of San Diego taking out three of four
> fiber paths, etc.
>  
> This is again a relative term that has been used for quite some time.
> A customer circuit going down or a new customer coming online results
> in a tiny routing change.  A metro fault may be handled entirely in
> the metro and have no effect at all on routing elsewhere in the
> provider network or global network.  Major fiber outages generally
> result in large routing changes.
>  
> In an MPLS context, a large routing change is one that affects a large
> number of LSP.  A classic example is where one hop along one of two or
> three east-west paths across the continental US goes down (or comes
> up, though this can be made far less disruptive).  Many LSP traverse
> the small number of US east-west paths and terminate in a large number
> of medium to large sized cities along either coast.
>  
> Again, I'll look at clarifying, but in less words than this response.

I added changes giving some limited information about what a large
routing change entails in an MPLS network.

> > 7.2.5. Dynamic Multipath Balance
> >
> > Comment:
> >
> > "...uses a course granularity, the adjustments would have to be
> > equally  course, in the worst case moving entire LSP"
> >
> > Minor typo "course" should be "coarse".
>  
> s/course/coarse/ with selective replacement.
>  
> > 7. Required Protocol Extensions and Mechanisms
> >
> > Comment/suggestion:
> >
> > Organizing section 7 as  a matrix containing something like:
> > Requirement#, Existing Mechanisms, Gaps, Ongoing/new extensions...,
> > might be more clearer. AT a minimum, add a traceability to each
> > requirement in [I-D.ietf-rtgwg-cl-requirement] and the section number
> > in framework document that addresses each requirement (or set of
> > requirement).
>  
> Section 7.2 groups the requirements but there is no where that lists
> every requirement in numeric order and points to where in the grouping
> it falls.  It should be easy enough to add that.
>  
> At the moment, potential issues with the requirements or in this
> document are listed in Section 7.3.  This is a different approach,
> listing the omissions rather than cross referencing.  Ideally we
> address all of the issues and drop this section.
>  
> Section 7.4 lists the requirements by protocol or functionality
> affected.  This as stated in the first paragraph is for the benefit of
> implementors.  Subsections of 7.4 refers back to the groupings in
> subsections of 7.2.  It would not be difficult to put in the reverse
> cross references (functional group refers to protocol change or
> functionality supporting it).

I didn't add this cross reference because the requirement numbers
change.  There is no way to add a tag in the requirements document
that would allow a tag to number mapping if the numbers change.

Keeping it this way I only have to fix the requirement numbering in
Section 7.2 if the CL requirements document changes.

> > Is there a minimal set of requirements which must be met to form a
> > deployable (useful) composite link based solution?
>  
> That's a great question.
>  
> "Enough to be useful" is interpreted differently by different network
> operators deploying a network.  For an equipment vendor the best
> approach is to plan to implement everything over time under the
> assumption that every feature will appear in a check box in someone's
> RFI/RFP/RFQ eventually, but ask current key customers which features
> to prioritize.  That process may yield a different answer for
> different equipment vendors, depending on who their current key
> customers are.
>  
> BTW- Our friends at Verizon made sure that almost every requirement is
> a MUST, so ask them to identify the requirements that they are not
> really serious about.  Wear protective fireproof undergarments.  :-)

Did you ever ask?

> > -----------------------
> >
> > Regards,
> > Iftekhar
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 14 Jun 2012 20:38:50 -0400

In a prior email I wrote:

  Diffs will be sent in a separate email - too long to attach.

Here they are:

Index: draft-so-yong-rtgwg-cl-framework.xml
===================================================================
RCS file: /home/cvs/CVS-occnc/customers/ietf/rtg-cl/draft-so-yong-rtgwg-cl-framework.xml,v
retrieving revision 1.10
diff -u -w -r1.10 draft-so-yong-rtgwg-cl-framework.xml
--- draft-so-yong-rtgwg-cl-framework.xml	7 Mar 2012 21:35:14 -0000	1.10
+++ draft-so-yong-rtgwg-cl-framework.xml	14 Jun 2012 20:46:17 -0000
@@ -61,23 +61,17 @@
 <?rfc inline="yes" ?>
 
 <rfc category="info" ipr="trust200902"
-     docName="draft-so-yong-rtgwg-cl-framework-05">
+     docName="draft-so-yong-rtgwg-cl-framework-06-preview1">
 
   <front>
     <title abbrev="Composite Link Framework">
       Composite Link Framework in Multi Protocol Label Switching (MPLS)</title>
 
     <author
-	    fullname="So Ning" initials="N." surname="So">
-      <organization>Verizon</organization>
+	    fullname="So Ning" initials="S." surname="Ning">
+      <organization>Tata Communications</organization>
       <address>
-        <postal>
-          <street>2400 N. Glenville Ave.</street>
-          <city>Richardson, TX</city>
-	  <code>75082</code>
-        </postal>
-        <phone>+1 972-729-7905</phone>
-        <email>[email-masked]</email>
+        <email>[email-masked]</email>
       </address>
     </author>
 
@@ -107,12 +101,12 @@
       <organization>Huawei USA</organization>
       <address>
         <postal>
-          <street>1700 Alma Dr. Suite 500</street>
+          <street>5340 Legacy Dr.</street>
           <city>Plano, TX</city>
-	  <code>75075</code>
+	  <code>75025</code>
         </postal>
-        <phone>+1 469-229-5387</phone>
-        <email>[email-masked]</email>
+        <phone>+1 469-277-5837</phone>
+        <email>[email-masked]</email>
       </address>
     </author>
 
@@ -380,7 +374,8 @@
 	described in greater detail than given requirements alone.
       </t>
 
-      <section title="Flow Identification">
+      <section anchor="sect.flow-id"
+	       title="Flow Identification">
 
 	<t>
 	  Traffic mapping to component links is a data plane
@@ -399,17 +394,18 @@
 	</t>
 
 	<t>
-	  Operator may have other objectives such as placing a
-	  bidirectional flow or LSP on the same component link in
-	  both direction, load balance over component links, composite
-	  link energy saving, and etc.  These new requirements are
-	  described in <xref target="I-D.ietf-rtgwg-cl-requirement"
-	  />.
+	  The network operator may have other objectives such as
+	  placing a bidirectional flow or LSP on the same component
+	  link in both direction, load balance over component links,
+	  composite link energy saving, and etc.  These new
+	  requirements are described in <xref
+	  target="I-D.ietf-rtgwg-cl-requirement" />.
 	</t>
 
 	<!--
-	  The feasibility of symetric paths for all flows is questionable!
-	  Perhaps the emphasis on this (mis)feature should be reduced.
+	  The feasibility of symetric paths for all flows is
+	  questionable!  Perhaps the emphasis on this (mis)feature in
+	  the above paragraph should be reduced.
 	-->
 
 	<t>
@@ -474,29 +470,35 @@
 	  flow identification.  If some LSP may require strict packet
 	  ordering but those LSP cannot be distinguished from others,
 	  then only the top label can be used as a flow identifier.
-	  If only the top label is used, then there may not be
-	  adequate flow granularity to accomplish well balanced
-	  traffic distribution and it will not be possible to carry
-	  LSP that are larger than any individual component link.
+	  If only the top label is used (for example, as specified by
+	  <xref target="RFC4201" /> when the "all-ones" component
+	  described in <xref target="RFC4201" /> is not used), then
+	  there may not be adequate flow granularity to accomplish
+	  well balanced traffic distribution and it will not be
+	  possible to carry LSP that are larger than any individual
+	  component link.
 	</t>
 
 	<t>
 	  The number of flows can be extremely large.  This may be the
 	  case when the entire label stack is used and is always the
 	  case when IP addresses are used in provider networks
-	  carrying Internet traffic.  Current practice at the time of
-	  writing were documented in <xref target="RFC2991" /> and
-	  <xref target="RFC2992" />.  These practices as described,
-	  make use of IP addresses.  The common practices were
-	  extended to include the MPLS label stack and the common
-	  practice of looking at IP addresses within the MPLS payload.
-	  These extended practices are described in <xref
-	  target="RFC4385" /> and <xref target="RFC4928" /> due to
-	  their impact on pseudowires without a PWE3 Control Word.
+	  carrying Internet traffic.  Current practice for native IP
+	  load balancing at the time of writing were documented in
+	  <xref target="RFC2991" />, <xref target="RFC2992" />.  These
+	  practices as described, make use of IP addresses.  The
+	  common practices were extended to include the MPLS label
+	  stack and the common practice of looking at IP addresses
+	  within the MPLS payload.  These extended practices are
+	  described in <xref target="RFC4385" /> and <xref
+	  target="RFC4928" /> due to their impact on pseudowires
+	  without a PWE3 Control Word.  Additional detail on current
+	  multipath practices can be found in the appendices of <xref
+	  target="I-D.symmvo-rtgwg-cl-use-cases" />.
 	</t>
 
 	<t>
-	  Using only the top label supports too course a traffic
+	  Using only the top label supports too coarse a traffic
 	  balance.  Using the full label stack or IP addresses as flow
 	  identification provides a sufficiently fine traffic balance,
 	  but is capable of identifying such a high number of distinct
@@ -506,9 +508,39 @@
 	  technique.  Other means of grouping flows may be possible.
 	</t>
 
+	<t>
+	  In summary:
+	  <list style="numbers">
+	    <t>
+	      Load balancing using only the MPLS label stack provides
+	      too coarse a granularity of load balance.
+	    </t>
+	    <t>
+	      Tracking every flow is not scalable due to the extremely
+	      large number of flows in provider networks.
+	    </t>
+	    <t>
+	      Existing techniques, IP source and destination hash in
+	      particular, have proven in over two decades of
+	      experience to be an excellent way of identifying groups
+	      of flows.
+	    </t>
+	    <t>
+	      If a better way to identify groups of flows is
+	      discovered, then that method can be used.
+	    </t>
+	    <t>
+	      IP address hashing is not required, but use of this
+	      technique is strongly encouraged given the technique's
+	      long history of successful deployment.
+	    </t>
+	  </list>
+	</t>
+
       </section>
 
-      <section title="Composite Link in Control Plane">
+      <section anchor="sect.control-plane"
+	       title="Composite Link in Control Plane">
 
 	<t>
 	  A composite Link is advertised as a single logical interface
@@ -648,17 +680,18 @@
 	</t>
 
 	<t>
-	  Composite link capacity is aggregated capacity and MAY be
-	  larger than individual component link capacity.  Any
-	  aggregated LSP can determine a bounds on the largest
-	  microflow that could be carried and this constraint can be
-	  handled as follows.
+	  Composite link capacity is aggregated capacity.  LSP
+	  capacity MAY be larger than individual component link
+	  capacity.  Any aggregated LSP can determine a bounds on the
+	  largest microflow that could be carried and this constraint
+	  can be handled as follows.
 	</t>
 
 	<t>
 	  <list style="numbers">
 	    <t>
-	      If no other information is available the largest
+	      If no information is available through signaling,
+	      management plane, or configuration, the largest
 	      microflow is bound by one of the following:
 	      <list style="letters">
 		<t>
@@ -770,24 +803,26 @@
 
       </section>
 
-      <section title="Composite Link in Data Plane">
+      <section anchor="sect.data-plane"
+	       title="Composite Link in Data Plane">
 
 	<t>
-	  The traffic over a composite link is distributed over
-	  individual component links.  Traffic distribution may be
-	  determined by or constrained by control plane or management
-	  plane.  Traffic distribution may be changed due to component
-	  link status change, subject to constraints imposed by either
-	  the management plane or control plane.  The distribution
-	  function is local to the routers in which a composite link
-	  belongs to and is not specified here.
-	  <!-- corouted LSP are coordinated by the PATH/RESV -
-	    However, if a bi-directional LSP is required to be placed
-	    on the same component link in both directions, the routers
-	    at both composite link end points need incorporation in
-	    determining the component link for the LSP. The protocol
-	    extension of that is for further study.
-	  -->
+	  The data plane must first identify groups of flows.  Flow
+	  identification is covered in <xref target="sect.flow-id" />.
+	  Having identified groups of flows the groups must be placed
+	  on individual component links.  This second step is called
+	  traffic distribution or traffic placement.  The two steps
+	  together are known as traffic balancing or load balancing.
+	</t>
+
+	<t>
+	  Traffic distribution may be determined by or constrained by
+	  control plane or management plane.  Traffic distribution may
+	  be changed due to component link status change, subject to
+	  constraints imposed by either the management plane or
+	  control plane.  The distribution function is local to the
+	  routers in which a composite link belongs to and is not
+	  specified here.
 	</t>
 
 	<t>
@@ -795,34 +830,38 @@
 	  differentiate multicast traffic vs. unicast traffic.
 	</t>
 
-	<!-- protection already covered -
-	A component link in a composite link may fail
-	independently. The routers at a composite link MUST maintain
-	each component link status. Two routers may use the control
-	plane to sync up a component link state. When a component link
-	fails, the routers of a composite link MUST re-assign impacted
-	flows to other active component links in minimal disruptive
-	manner. This is local function and do not incorporate with LSP
-	head-end routers.
-	-->
+	<t>
+	  In order to maintain scalability, existing data plane
+	  forwarding retains state associated with the top label only.
+	  The use of flow group identification is in a second step in
+	  the forwarding process.  Data plane forwarding makes use of
+	  the top label to select a composite link, or a group of
+	  components within a composite link or for the case where an
+	  LSP is pinned (see <xref target="RFC4201" />), a specific
+	  component link.  For those LSP for which the LSP selects
+	  only the composite link or a group of components within a
+	  composite link, the load balancing makes use of the flow
+	  group identification.
+	</t>
 
-	<!-- remove this - add to CL-req in management section
-	The composite link functions provide component link fault
-	notification and composite link fault notification. Component
-	link fault notification MUST be sent to the management
-	plane. Composite link fault notification MUST be sent to
-	management plane and distribute via link state message in IGP.
-	-->
+	<t>
+	  The most common traffic placement techniques uses the a flow
+	  group identification as an index into a table.  The table
+	  provides an indirection.  The number of bits of hash is
+	  constrained to keep table size small.  While this is not the
+	  best technique, it is the most common.  Better techniques
+	  exist but they are outside the scope of this document and
+	  some are considered proprietary.
+	</t>
 
-	<!-- remove this - add to CL-req in management section
-	Operator may want to perform an optimization function such as
-	load balance or energy saving over a composite link, which may
-	conduct some traffic moving from one component link to
-	another. The process MUST support locally and gracefully
-	traffic movement process among component links. The protocol
-	that facilitates this process between two composite link end
-	points is for further study.
-	-->
+	<t>
+	  Requirements to limit frequency of load balancing can be
+	  adhered to by keeping track of when a flow group was last
+	  moved and imposing a minimum period before that flow group
+	  can be moved again.  This is straightforward for a table
+	  approach.  For other approaches it may be less
+	  straightforward but is acheivable.
+	</t>
 
       </section>
 
@@ -833,12 +872,20 @@
 
       <t>
 	Scalability and stability are critical considerations in
-	protocol design where protocols may be used in a large
-	network.  Composite Link is applicable to large networks, and
-	therefore scalability must be a major consideration.  Some of
-	the requirements of Composite Link require additional
-	information to be carried in situations where component links
-	differ in some significant way.
+	protocol design where protocols may be used in a large network
+	such as today's service provider networks.  Composite Link is
+	applicable to networks which are large enough to require that
+	traffic be split over multiple paths.  Scalability is a major
+	consideration for networks that reach a capacity large enough
+	to require Composite Link.
+      </t>
+
+      <t>
+	Some of the requirements of Composite Link could potentially
+	have a negative impact on scalability.  For example, Composite
+	Link requires additional information to be carried in
+	situations where component links differ in some significant
+	way.
       </t>
 
       <section anchor="sect.scalability"
@@ -851,15 +898,32 @@
 	  adequate to meet requirements.  Routing information is
 	  aggregated to reduce the amount of information exchange
 	  related to routing and to simplify route computation (see
-	  <xref target="sect.routing-tradeoff" />).  Reducing the
-	  amount of information allows the exchange of information
-	  during a large routing change to be accomplished more
-	  quickly, and simplifying route computation improves
-	  convergence time after very significant network faults which
-	  cannot be handled by preprovisioned or precomputed
-	  protection mechanisms.  Aggregating smaller LSP into larger
-	  LSP is a means to reduce RSVP-TE signaling (see <xref
-	  target="sect.signaling-tradeoff" />).
+	  <xref target="sect.routing-tradeoff" />).
+	</t>
+
+	<t>
+	  In an MPLS network large routing changes can occur when a
+	  single fault occurs.  For example, a single fault may impact
+	  a very large number of LSP traversing a given link.  As new
+	  LSP are signaled to avoid the fault, resources are consumed
+	  elsewhere, and routing protocol announcements must flood the
+	  resource changes.  If protection is in place, there is less
+	  urgency to converging quickly.  If multiple faults occur
+	  that are not covered by shared risk groups (SRG), then some
+	  protection may fail, adding urgency to converging quickly
+	  even where protection was deployed.
+	</t>
+
+	<t>
+	  Reducing the amount of information allows the exchange of
+	  information during a large routing change to be accomplished
+	  more quickly and simplifies route computation.  Simplifying
+	  route computation improves convergence time after very
+	  significant network faults which cannot be handled by
+	  preprovisioned or precomputed protection mechanisms.
+	  Aggregating smaller LSP into larger LSP is a means to reduce
+	  path computation load and reduce RSVP-TE signaling (see
+	  <xref target="sect.signaling-tradeoff" />).
 	</t>
 
 	<t>
@@ -905,7 +969,7 @@
 	  sensible choices regarding the amount of change to link
 	  parameters that require link readvertisement.  For example,
 	  if delay measurements include queuing delay, then a much
-	  more course granularity of delay measurement would be called
+	  more coarse granularity of delay measurement would be called
 	  for than if the delay does not include queuing and is
 	  dominated by geographic delay (speed of light delay).
 	</t>
@@ -1133,7 +1197,7 @@
 	  <t>
 	    path symmetry requires extensions and is particularly
 	    challenging for very large LSP (see
-	    target="sect.path-symmetry" />),
+	    <xref target="sect.path-symmetry" />),
 	  </t>
 	  <t>
 	    accommodating a very wide range of requirements among
@@ -1142,7 +1206,7 @@
 	    reduce scalability if a large number of aggregates are
 	    used to provide a too fine a reflection of the
 	    requirements in the contained LSP (see
-	    target="sect.contained-lsp" />),
+	    <xref target="sect.contained-lsp" />),
 	  </t>
 	  <t>
 	    backwards compatibility is somewhat limited due to the
@@ -1150,13 +1214,13 @@
 	    provide too little information regarding their configured
 	    default behavior, and legacy LSP which provide too little
 	    information regarding their requirements (see
-	    target="sect.compat" />),
+	    <xref target="sect.compat" />),
 	  </t>
 	  <t>
 	    data plane challenges include those of accommodating very
 	    large LSP, large microflows, traffic ordering constraints
 	    imposed by a subsent of LSP, and accounting for IP and LDP
-	    traffic (see target="sect.dp-challenge" />).
+	    traffic (see <xref target="sect.dp-challenge" />).
 	  </t>
 	</list>
       </t>
@@ -1391,59 +1455,14 @@
 	       title="Data Plane Challenges">
 
 	<t>
-	  Regardless of implementation choices, there are tradeoffs
-	  regarding the flow identification granularity.  If flow
-	  identification granularity is very course, such as using top
-	  lable only, LSP larger than the size of a component link are
-	  not feasible.  In practice using the MPLS label stack alone
-	  has proven too course to acheive a reasonably good load
-	  balance, due to bin-packing issues and discrpencies between
-	  signaled bandwidth and actual traffic loads on LSP.  If a
-	  finer granualrity is based on IP addresses, then a table
-	  approach is infeasible, due to the extremely large number of
-	  IP address pairs found in Internet traffic.  The preferred
-	  implementation has been to use a hash over the IP address
-	  pairs to provide a fine granuarity but with a feasible
-	  implementation.  Where hashing is used, the hash itself can
-	  be done at ingress and placed in a fat-pw label or entropy
-	  label (see <xref target="sect.entropy" />) to avoid
-	  performing the deeper MPLS header parse in the network core
-	  and the hash in the network core.
-	</t>
-
-	<t>
-	  Other implementations are possible, but must still face the
-	  problem that not using IP addresses provides a granularity
-	  which is too course and using IP host pairs yields a table
-	  size which is so impractical as to be considered an
-	  infeasible solution.  This section assumes that this problem
-	  is addressed, but makes no assumption as to the
-	  implementation.  Where a hash based solution is mentioned,
-	  such mention should be considered a reference approach, not
-	  a requirement.
-	</t>
-
-	<t>
-	  In order to maintain scalability, existing data plane
-	  forwarding retains state associated with the top label only.
-	  Data plane forwarding makes use of the top label to select a
-	  composite link, or a group of components within a composite
-	  link or for the case where an LSP is pinned (see <xref
-	  target="RFC4201" />), a specific component link.  For those
-	  LSP for which the LSP selects only the composite link or a
-	  group of components within a composite link, the load
-	  balancing may make use of the entire label stack and in some
-	  cases may make use of information in the payload, though no
-	  state on specific contained LSP is retained.
-	</t>
-
-	<t>
-	  Load balancing makes use of techniques which allow large
-	  sets of flows to be moved to rearrange traffic.  These large
-	  sets of flows may be at a finer granularity than contained
-	  LSP.  Requirements to limit frequency of load balancing
-	  rearrangement can be adhered to by constraining the
-	  frequency at which these large sets of flows are moved.
+	  Flow identification is briefly discussed in
+	  <xref target="sect.flow-id" />.
+	  Traffic distribution is briefly discussed in
+	  <xref target="sect.data-plane" />.
+
+	  This section discusses issues specific to particular
+	  requirements specified in
+	  <xref target="I-D.ietf-rtgwg-cl-requirement" />.
 	</t>
 
 	<section anchor="sect.large-lsp"
@@ -1453,8 +1472,18 @@
 	    Very large LSP may exceed the capacity of any single
 	    component of a composite link.  In some cases contained
 	    LSP may exceed the capacity of any single component.
-	    These LSP require the use of the equivalent of the
-	    all-ones component of a link bundle.
+	    These LSP may the use of the equivalent of the all-ones
+	    component of a link bundle, or may use a subset of
+	    components which meet the LSP requirements.
+	  </t>
+
+	  <t>
+	    Very large LSP can be accommodated as long as they can be
+	    subdivided (see <xref target="sect.large-flows" />).  A
+	    very large LSP cannot have a requirement for symetric
+	    paths unless complex protocol extensions are proposed (see
+	    <xref target="sect.control-plane" /> and <xref
+	    target="sect.path-symmetry" />).
 	  </t>
 
 	</section>
@@ -1464,8 +1493,8 @@
 
 	  <t>
 	    Within a very large LSP there may be very large
-	    microflows, or very large flows which cannot be further
-	    subdivided for other reasons.  Flows which cannot be
+	    microflows.  A very large microflow is a very large flows
+	    which cannot be further subdivided.  Flows which cannot be
 	    subdivided must be no larger that the capacity of any
 	    single component.
 	  </t>
@@ -2122,8 +2151,8 @@
 	  <t>
 	    FR#11 explicitly calls for dynamic load balancing similar
 	    to existing adaptive multipath.  In implementations where
-	    flow identification uses a course granularity, the
-	    adjustments would have to be equally course, in the worst
+	    flow identification uses a coarse granularity, the
+	    adjustments would have to be equally coarse, in the worst
 	    case moving entire LSP.  The impact of flow identification
 	    granularity and potential adaptive multipath approaches
 	    may need to be documented in greater detail than provided
@@ -2506,9 +2535,22 @@
 
       <t>
 	Authors would like to thank Adrian Farrel, Fred Jounay, Yuji
-	Kamite for his extensive comments and suggestions, Ron Bonica,
-	Nabil Bitar, Eric Gray, Lou Berger, and Kireeti Kompella for
-	their reviews and great suggestions.
+	Kamite for his extensive comments and suggestions regarding
+	early versions of this document, Ron Bonica, Nabil Bitar,
+	Eric Gray, Lou Berger, and Kireeti Kompella for their reviews
+	of early versions and great suggestions.
+      </t>
+      <t>
+	Authors would like to thank Iftekhar Hussain for review and
+	suggestions regarding recent versions of this document.
+      </t>
+      <t>
+	In the interest of full disclosure of affiliation and in the
+	interest of acknowledging sponsorship, past affiliations of
+	authors are noted.  Much of the work done by Ning So occurred
+	while Ning was at Verizon.  Much of the work done by Curtis
+	Villamizar occurred while at Infinera.  Infinera continues to
+	sponsor this work on a consulting basis.
       </t>
 
     </section>

---

From: Iftekhar Hussain <[email-masked]>
Date: Fri, 15 Jun 2012 20:55:12 +0000

Curtis,

Thanks for the follow up. These changes address my comments.

Regards,
Iftekhar
-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Thursday, June 14, 2012 5:39 PM
To: [email-masked]
Cc: Iftekhar Hussain; [email-masked]
Subject: Re: draft-so-yong-rtgwg-cl-framework - part 2 - diffs

Here they are:

 
   <front>
 
     <author
 
@@ -107,12 +101,12 @@
 
@@ -380,7 +374,8 @@

 	<t>
 	  Traffic mapping to component links is a data plane @@ -399,17 +394,18 @@
 	</t>
 
 	<t>
 
 	<!--
 
 	<t>
@@ -474,29 +470,35 @@
 
 	<t>
 
 	<t>
 	  but is capable of identifying such a high number of distinct @@ -506,9 +508,39 @@
 
+	<t>
+	  In summary:
+	  <list style="numbers">
+	    <t>

 	<t>
 	  A composite Link is advertised as a single logical interface @@ -648,17 +680,18 @@
 	</t>
 
 	<t>
 
 	<t>
 	  <list style="numbers">
 	    <t>
 
       </section>

 	<t>
 
 	<t>
@@ -795,34 +830,38 @@

 
 
       </section>
 
@@ -833,12 +872,20 @@
 
       <t>

 	<t>
@@ -905,7 +969,7 @@
 	    accommodating a very wide range of requirements among @@ -1142,7 +1206,7 @@
 	    backwards compatibility is somewhat limited due to the @@ -1150,13 +1214,13 @@
 
 	<t>

 	</section>
@@ -1464,8 +1493,8 @@
 
 	  <t>
 	    may need to be documented in greater detail than provided @@ -2506,9 +2535,22 @@
 
       <t>
 
     </section>