Subject: Re: change to requirements (was Re: draft-so-yong-rtgwg-cl-framework)

From: Curtis Villamizar <[email-masked]>
Date: Fri, 01 Jun 2012 11:36:47 -0400

Iftekhar,

In response to your comment:

> I also think, it would be a good idea to indicate the consequences
> of such load balancing change (if this event is going to cause
> service disruption).  FR#4 and FR#12 might be relevant.

The only thing we could do to make this more clear is to put in a
cross reference any where that "minimally disruptive" is mentioned and
maybe even put "minimally disruptive" and "delay discontinuity" in the
Definitions section.

The proposed text so far is:

  [FR#N]   Load balancing MAY be used during sustained low traffic
           periods to reduce the number of active component links for
           the purpose of power reduction.

  As with any load balancing change, a change initiated for the
  purpose of power reduction may be minimally disruptive.  Typically
  the disruption is limited to a change in delay characteristics and
  the potential for a very brief period with traffic reordering.  The
  network operator when configuring a network for power reduction
  should weight the benefit of power reduction against the
  disadvantage of a minimal disruption.

I could change the cautioning paragraph to:

  As with any load balancing change, a change initiated for the
  purpose of power reduction may be minimally disruptive (See FR#12).
  Typically the disruption is limited to a change in delay
  characteristics and the potential for a very brief period with
  traffic reordering.  The network operator when configuring a network
  for power reduction should weight the benefit of power reduction
  against the disadvantage of a minimal disruption.

Note that the only change is (See FR#12).  I really can't see how this
could be more clear.  I also can't see how "The network operator when
configuring a network for power reduction should weight the benefit of
power reduction against the disadvantage of a minimal disruption"
could be more clear about the operator having a choice as to whether
to enable this feature and in configuring it.

What follows is more background on the use of the term "minimally
disruptive", though I think we've been through this before.

Minimally disruptive is the term we use to describe the impact of a
load balancing event.  The implication of "minimally disruptive" is a
very small disruption, as small as practical.  Given that there is
more than two decades of experience with this type of load balancing
technique, what we mean by "minimally disruptive" is well known by
those who have experience with multipath (ECMP, LAG, etc) and by those
who carefully read what has been written in the set of three documents
and in the references (the latter may be the null set or close to it).

Lets start with experience with this technique on the big Internet
(service provider core networks).  Briefly:

  1987 - T1 NSFNET - load balancing was done due to inability of PC-RT
      to process traffic at full data rate (we know that is pathetic)

  1990 - T3 NSFNET - anyone not needing a T3/DS3 (45 Mb/s) could make
      use of multiple T1/DS1 circuits at a lower cost than commercial
      pricing for fractional T3/DS3.

  1994 - Use of parallel T3/DS3 before OC3 was available and/or priced
      within reason.

  1996-2000 - use of parallel OC3, then OC12, then OC48 as Internet
      growth consistently outpaced available circuits on commercial
      market and/or router blades.

  2001-present - exhaustive use of parallel OC192, then 10GbE.  Note
      that OC192 and OC768 router blades are much more expensive than
      10GbE and therefore 40 Gb/s has not overtaken 10 Gb/s.  Today
      both are carried over OTN in many transport networks but this is
      referring to the IP/MPLS interfaces in use.

Note that I am not including Ethernet Link Aggregation above, though
it too is now extensively used in conjunction with ECMP.

Look in draft-symmvo-rtgwg-cl-use-cases (Composite Link Use Cases) at
Appendix B "Existing Multipath Standards and Techniques" for some
details on the techniques used in the past and present.

We had discussed what minimally disruptive means in a prior thread.

  draft-ietf-rtgwg-cl-requirement
  Composite Link Requirements

    4.3.  Parallel Component Links with Different Characteristics
    Page 8

       FR#12  When a traffic flow is moved from one component link to
              another in the same composite link between a set of
	      nodes (or sites), it MUST be done so in a minimally
	      disruptive manner.

              When a flow is moved from a current link to a target
	      link with different latency, reordering can occur if the
	      target link latency is less than that of the current or
	      clumping can occur if target link latency is greater
	      than that of the current.  Therefore, some flows (e.g.,
	      timing distribution, PW circuit emulation) are quite
	      sensitive to these effects, which may be specified in an
	      NPO or are needed to meet a user experience objective
	      (e.g. jitter buffer under/overrun).

  draft-so-yong-rtgwg-cl-framework
  Composite Link Framework

    2.2.  Composite Link in Control Plane
    Page 10

       Individual component link may fail independently.  Upon
       component link failure, a composite link MUST support a
       minimally disruptive local repair, preempting any LSP which can
       no longer be supported.  Available capacity in other component
       links MUST be used to carry impacted traffic.  The available
       bandwidth after failure MUST be advertised immediately to avoid
       looped crankback.

       When a composite link is not able to transport all LSP, it
       preempts some LSP based upon local management configuration and
       informs the control plane on these preempted LSP.  The
       composite link MUST support soft preemption [RFC5712].  This
       action ensures the remaining traffic is transported properly.
       FR#10 requires that the traffic be restored.  FR#12 requires
       that any change be minimally disruptive.  These two
       requirements are interpreted to include preemption among the
       types of changes that must be minimally disruptive.

    4.2.2.  Very Large Microflows
    Page 20

       Some techniques are susceptible to statistical collisions where
       an algorithm to distribute traffic is unable to disambiguate
       traffic among two or more very large microflow where their sum
       is in excess of the capacity of any single component.  Hash
       based algorithms which use too small a hash space are
       particularly susceptible and require a change in hash seed in
       the event that this were to occur.  A change in hash seed is
       highly disruptive, causing traffic reordering among all traffic
       flows over which the hash function is applied.

    7.2.9.  Minimally Disruption Load Balance
    Page 29

       The behavior of hash methods used in classic multipath needs to
       be described in terms of FR#12 which calls for minimally
       disruptive load adjustments.  For example, reseeding the hash
       violates FR#12.  Using modulo operations is significantly
       disruptive if a link comes or goes down, as pointed out in
       [RFC2992].  In addition, backwards compatibility with older
       hardware needs to be accommodated.

In past email we agreed to introduce the term "delay discontinuity"
which has been used for close to two decades to describe the one time
delay change that is referred to in FR#12.

I think we have been fairly clear about what "minimally disruptive"
means today in practice.  It means a "delay discontinuity" with zero
packet loss.  If the delay is reduced, then a brief period of
reordering can occur, where the duration of the period of reordering
is the difference in the delay (ie: typically a few milliseconds).

Curtis

In message <[email-masked]>
Iftekhar Hussain writes:

Curtis,

I also think, it would be a good idea to indicate the consequences of such load balancing change (if this event is going to cause service disruption).  FR#4 and FR#12 might be relevant.

Regards,
Iftekhar
-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Thursday, May 24, 2012 2:33 PM
To: Kireeti Kompella
Cc: [email-masked]; Iftekhar Hussain; [email-masked]
Subject: Re: change to requirements (was Re: draft-so-yong-rtgwg-cl-framework)

In message <[email-masked]>
Kireeti Kompella writes:
 
> On May 24, 2012, at 10:56 , Curtis Villamizar wrote:
>  
> > In the discussion of the CL framework, a suggestion was made to 
> > change the requirements.  Please comment on this suggestion.
> > 
> > The following would be added somewhere.
> > 
> >  Load balancing MAY be used during sustained low traffic periods to  
> > reduce the number of active component links for the purpose of power  
> > reduction.
>  
> Is the intent:
>  
>    Load balancing MAY be _changed_ during sustained low traffic
>    periods to reduce the number of active component links ...
>  
> ?
>  
> If so, a warning ("this may result in some packets being reordered, 
> and a change in delay and jitter of some flows") should probably be 
> added.
>  
> Kireeti.

Kireeti,

You are correct that any change would be minimally disruptive.  In the example I gave there could be no more than one change in 20 minutes, but still more than zero.

I personally don't think a warning is needed here, but if you and/or others feel it is needed I have no objections to adding it.  The text would then be:

  [FR#N]   Load balancing MAY be used during sustained low traffic
           periods to reduce the number of active component links for
           the purpose of power reduction.

  As with any load balancing change, a change initiated for the
  purpose of power reduction may be minimally disruptive.  Typically
  the disruption is limited to a change in delay characteristics and
  the potential for a very brief period with traffic reordering.  The
  network operator when configuring a network for power reduction
  should weight the benefit of power reduction against the
  disadvantage of a minimal disruption.

The first paragraph is a requirement.  The second paragraph is discussion and should not appear within a numbered list of requirements.

I would like comments from you and others.  Do we need to add this requirement?  If so do we need to add this warning?

Curtis