Subject: Ref: Composite link drafts update and request

From: Iftekhar Hussain <[email-masked]>
Date: Fri, 13 Apr 2012 21:16:55 +0000

Hi,

I have following clarification questions/comments about the https://tools.ietf.org/html/draft-ietf-rtgwg-cl-requirement-05 document

a)      Section 3. "Examples of a physical link are: Lambda, Ethernet PHY, and OTN".
 Should "OTN" be "OTU"?  Furthermore, can a "component link"  be formed by set of Lambda?

b)      Section 4.1.  In the context of "FR#1" from end-to-end perspective, should there be some sort of network scale related requirement(s)? For example, max number of pair of nodes connected via composite links traversed etc. If this aspect is covered somewhere a reference should be added.

c)       Section 4.2, "Communication of other  performance parameters (e.g., delay variation) is desirable" seems to contradict "FR#17" in section 4.3.

d)      Section 4.2, FR#8" appears to use "latency" in the end-to-end sense.  While the next requirement "FR#9" appears to refer to this term only between two nodes. I think, it would helpful to clarify the usage of term "latency" e.g., what component of a node transit delay it excludes or includes. Or add a reference to a document if this is covered in some other document.

e)      General comment section 4.3. "Many techniques have been developed to balance the distribution of flows across component links that  connect the same pair of nodes" and "...via composite links,  other techniques have been developed." It would good to add a reference(s) for each.

f)       Section 4.3,  "FR#12" is about latency while the example "...a user experience  objective (e.g. jitter buffer under/overrun)" appears to refer to delay variation. I think, the example should be consistent with the requirement.

g)      Section 4.3, "FR#17". It is not clear why the delay variation is considered in this context. Reading this requirement,  one gets the impression as if the composite link delay has two components (delay and delay variation). I had understood that delay variation (contributed through the nodal switching/queuing delays) is not considered in this document.

h)      DR#6, a minor typo "Solution" should be "solution"?

Thanks,
Iftekhar

"As was discussed in yesterday's rtgwg meeting, there are three
composite link drafts:

https://tools.ietf.org/html/draft-ietf-rtgwg-cl-requirement-05

https://tools.ietf.org/html/draft-so-yong-rtgwg-cl-framework-05

https://tools.ietf.org/html/draft-symmvo-rtgwg-cl-use-cases-00

You can read the slides I presented on these drafts at
https://tools.ietf.org/agenda/83/slides/slides-83-rtgwg-3.pdf (it's a
quick read!).

As Alia said, you are all requested to read and comment on these
drafts. In about a month's time, we will be requesting WG adoption of
draft-so and draft-symmvo, and it'll be great to have an informed
discussion at that time.

Thanks much,
Andy"

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 19 Apr 2012 22:09:44 -0400

Iftekhar,

Responses inline.  Some reformating of your email into plain text.

Thanks for the interest and for the comments.

Curtis

In message <[email-masked]>
Iftekhar Hussain writes:
 
> Hi,
>  
> I have following clarification questions/comments about the
> document
> a) Section 3. "Examples of a physical link are: Lambda, Ethernet PHY,
>    and OTN".  Should "OTN" be "OTU"?  Furthermore, can a "component
>    link" be formed by set of Lambda?

We are mixing physical and physical plus link layer in the examples.
The distinction is not important in this context but if you feel
strongly we could clarify.

If we put ODU then others would complain that ODU is carried in OTU
and then we run into the issue that OTU is a framing on an OPU and
perhaps someone would argue that OTU is not a physical layer and that
even OPU is not truly a physical layer.

It might be best to change this to:

  Examples of a physical link (physical layer plus link layer) are: a
  lambda with any link layer(s), Ethernet, OTN.

Examples of multiple link layers include POS/SONET,
10GbE/ODU2e/ODU4/OTU4/OPU4, etc.  In the same way, WDM could be
considered an example of multiple physical layers (the individual
lambda multiplex using WDM).  The problem is that any attempt to
strictly apply the antiquated ISO 7-layer model falls apart even is we
apply only the physical layer and link layer.  For WDM we have the
physical layer subdivided into a layer zero (optical) and a layer one
(modulated signal aka single lambda).  We have plenty of examples of
multiple link layer (layer two) protocols.

The important distinction is [CL running over] a physical link plus
link layer [used as a component link] vs [CL running over] MPLS [used
as a component link].  Rewritten without the []: The important
distinction is a physical link plus link layer vs MPLS.

> b) Section 4.1.  In the context of "FR#1" from end-to-end perspective,
>    should there be some sort of network scale related requirement(s)?
>    For example, max number of pair of nodes connected via composite
>    links traversed etc. If this aspect is covered somewhere a
>    reference should be added.

End-to-end in any MPLS context means edge-to-edge (and is not
mentioned at all in FR#1 or elsewhere).  Existing scaling techniques
used in MPLS networks still apply.  Scaling is the motivation for DR#5
(IGP extensions disallowed in multiple domain case).  Scaling is
directly addressed in DR#6 (fast convergence) and DR#7 (fast worst
case failure convergence).

Scalability and stability are covered in the framework document
(draft-so-yong-rtgwg-cl-framework).  For example in "3.1.  Scalability
Motivations" and "3.5.  Avoiding Route Oscillation".  Possible need
for further work is suggested in "7.2.11.  Performance, Scalability,
and Stability" in the framework document.

> c) Section 4.2, "Communication of other performance parameters (e.g.,
>    delay variation) is desirable" seems to contradict "FR#17" in
>    section 4.3.

The statement is:

   [...]  Communication of the latency performance
   parameter is a very important requirement.  Communication of other
   performance parameters (e.g., delay variation) is desirable.

If jitter was described as "undesireable" then it might be a
contradiction.  The point is delay is very important and delay
variation would be nice (maybe not IMHO).  What is important (ie:
normative) is what the implementation MUST support, and what the
operator MAY enable or disable.  What is unimportant is subjective
verbiage about relative importance among parameters.  The operator
will decide what is important by using or not using a feature.

FR#17 says:

   FR#17  The solution SHALL provide a means to indicate that a traffic
          flow shall select a component link with a maximum acceptable
          delay variation value as specified by protocol.

There are two differences between the statement in Section 4.2 and the
statement in FR#17.  First "communication of other performance
parameters" and "provide a means to indicate that a traffic flow shall
select a component link with a maximum acceptable delay variation
value" are two different things; the first is a IGP parameter (link
delay or link delay variation), the second is an RSVP parameter
(maximum allowable delay variation for a given LSP).  Second, the
wording in Section 4.2 is informative and the wording in FR#17 is
normative.

> d) Section 4.2, FR#8" appears to use "latency" in the end-to-end
>    sense.  While the next requirement "FR#9" appears to refer to this
>    term only between two nodes. I think, it would helpful to clarify
>    the usage of term "latency" e.g., what component of a node transit
>    delay it excludes or includes. Or add a reference to a document if
>    this is covered in some other document.

BTW- I prefer delay over latency, but the term latency is used.  The
two mean the same thing.  Neither word by itself does not indicate any
one type of delay (fabric delay, transmit delay, queuing delay,
geographic delay).  Neither word by itself implies intra-node delay,
or delay between adjacent nodes, or end-to-end delay.

FR#8 applies to any measurement of delay, whether it is a measurement
over a single physical link or over a component link carried over a
multihop MPLS LSP.  Also in FR#9 there is no indication that the nodes
are adjacent, and the phrase "when the path between these nodes
contains one or more pairs of nodes connected via a composite link"
clearly indicates that multihop paths are being considered as well as
delay between adjacent nodes.

There are some hints as to which delays the operators may consider
important in this context.  FR#8 states that "The precision of latency
reporting SHOULD be at least 10% of the one way latencies for latency
of 1 ms or more."  This excludes most equipment latencies which can be
expected to be well under 100 usec (10% of 1 msec).

The intent is to measure the predominant latency in most (well run)
service provider networks, which is geographic delay on the order of
msec or tens of msec and in some cases over 100 msec (intercontinental
traffic).  In a congested network (possibly not so well run network,
or maybe a well run network with a substantial outage) queuing delay
can be quite large.  In a congested network measuring the queuing
delay for the low priority traffic (the only traffic likely to be
experiencing congestion, even in an outage) is more likely to add to
problems by adding routing instability.

The only real question in FR#8 and FR#9 is whether queuing delay is
counted in the latency figure.  The argument for including queuing
delay is that it reflects the delay experienced by applications.  The
argument against including queuing delay is that it if used in routing
decisions it can result in routing instability.

There are quite a few documents that deal with this tradeoff.  For
example, in MPLS-TP it is noted that delay measurements should be made
using the highest priority COS value (which in effect very nearly
eliminates any queuing delay from the measurement).

For the requirements document, this tradeoff is not discussed, however
the stability and convergence requirements must be considered when
coming up with a solution.  Solutions should be described in the
framework document and not in the requirments document.

In the framework document (draft-so-yong-rtgwg-cl-framework) there is
discussion of this tradeoff in "3.5.  Avoiding Route Oscillation" and
elsewhere.  The framework document is a better place for discussion of
requirements tradeoffs.

> e) General comment section 4.3. "Many techniques have been developed
>    to balance the distribution of flows across component links that
>    connect the same pair of nodes" and "...via composite links, other
>    techniques have been developed." It would good to add a
>    reference(s) for each.

There are a few RFCs on the topic.  Inverse multiplexing, various
forms of ECMP, and Ethernet Link Aggregation are all well known.  The
statement would certainly not be controversial without references.

If you are looking for further information on this, the Use Cases
draft (draft-symmvo-rtgwg-cl-use-cases) has references.  "Appendix B.
Existing Multipath Standards and Techniques" was moved from the
requirements document to the Use Cases draft.

We could add a reference to the use cases draft in the requirements
draft if you think that would help.

> f) Section 4.3, "FR#12" is about latency while the example "...a user
>    experience objective (e.g. jitter buffer under/overrun)" appears to
>    refer to delay variation. I think, the example should be consistent
>    with the requirement.

The requirement in FR#12 is in the first paragraph.

   FR#12  When a traffic flow is moved from one component link to
          another in the same composite link between a set of nodes (or
          sites), it MUST be done so in a minimally disruptive manner.

The rest is discussion.

          When a flow is moved from a current link to a target link with
          different latency, reordering can occur if the target link
          latency is less than that of the current or clumping can occur
          if target link latency is greater than that of the current.
          Therefore, some flows (e.g., timing distribution, PW circuit
          emulation) are quite sensitive to these effects, which may be
          specified in an NPO or are needed to meet a user experience
          objective (e.g. jitter buffer under/overrun).

The discussion exists primarily for the reader who might otherwise not
realize why there would be a disruption at all (and therefore might
not understand why we have to say "minimally disruptive" instead of
"completely non-disruptive").  Perhaps the entire paragraph needs to
start with "For example, [...]".

In any case it is not inconsistent.  When delay changes on the current
big-bad-Internet (such as in VOIP), the change is often absorbed by a
jitter buffer as long as the change is small.  If a delay change of
many milliseconds occurs, then it is unlikely that the typical
relatively small jitter buffer can hide the delay change.

> g) Section 4.3, "FR#17". It is not clear why the delay variation is
>    considered in this context. Reading this requirement, one gets the
>    impression as if the composite link delay has two components (delay
>    and delay variation). I had understood that delay variation
>    (contributed through the nodal switching/queuing delays) is not
>    considered in this document.

FR#16 is about delay.  FR#17 is about delay variation.  The wording
"The solution SHALL provide a means" doesn't require the operator to
use that feature.  Therefore it is better to keep the two in separate
requirements as it is more clear that one could be enabled and the
other disabled.

> h) DR#6, a minor typo "Solution" should be "solution"?

Thanks for pointing this out.

> Thanks,
> Iftekhar
>  
> > As was discussed in yesterday's rtgwg meeting, there are three
> > composite link drafts:
> >  
> > Andy

---

From: Iftekhar Hussain <[email-masked]>
Date: Mon, 23 Apr 2012 17:51:51 +0000

Curtis,

Please see comments inline. 

Thanks for the detailed responses.

Iftekhar

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Thursday, April 19, 2012 7:10 PM
To: Iftekhar Hussain
Cc: [email-masked]
Subject: Re: Ref: Composite link drafts update and request

Iftekhar,

Curtis

We are mixing physical and physical plus link layer in the examples.
The distinction is not important in this context but if you feel strongly we could clarify.

If we put ODU then others would complain that ODU is carried in OTU and then we run into the issue that OTU is a framing on an OPU and perhaps someone would argue that OTU is not a physical layer and that even OPU is not truly a physical layer.

[Iftekhar] Yes the main confusion was it was specifically referring to physical links.  Your suggested change will help it clarify.

Examples of multiple link layers include POS/SONET, 10GbE/ODU2e/ODU4/OTU4/OPU4, etc.  In the same way, WDM could be considered an example of multiple physical layers (the individual lambda multiplex using WDM).  The problem is that any attempt to strictly apply the antiquated ISO 7-layer model falls apart even is we apply only the physical layer and link layer.  For WDM we have the physical layer subdivided into a layer zero (optical) and a layer one (modulated signal aka single lambda).  We have plenty of examples of multiple link layer (layer two) protocols.

The important distinction is [CL running over] a physical link plus link layer [used as a component link] vs [CL running over] MPLS [used as a component link].  Rewritten without the []: The important distinction is a physical link plus link layer vs MPLS.

>    reference should be added.

End-to-end in any MPLS context means edge-to-edge (and is not mentioned at all in FR#1 or elsewhere).  Existing scaling techniques used in MPLS networks still apply.  Scaling is the motivation for DR#5 (IGP extensions disallowed in multiple domain case).  Scaling is directly addressed in DR#6 (fast convergence) and DR#7 (fast worst case failure convergence).

Scalability and stability are covered in the framework document (draft-so-yong-rtgwg-cl-framework).  For example in "3.1.  Scalability Motivations" and "3.5.  Avoiding Route Oscillation".  Possible need for further work is suggested in "7.2.11.  Performance, Scalability, and Stability" in the framework document.

[Iftekhar] Okay. I will take a look at the " draft-so-yong-rtgwg-cl-framework". You might consider adding a reference to the framework document.

The statement is:

If jitter was described as "undesireable" then it might be a contradiction.  The point is delay is very important and delay variation would be nice (maybe not IMHO).  What is important (ie:
normative) is what the implementation MUST support, and what the operator MAY enable or disable.  What is unimportant is subjective verbiage about relative importance among parameters.  The operator will decide what is important by using or not using a feature.

FR#17 says:

There are two differences between the statement in Section 4.2 and the statement in FR#17.  First "communication of other performance parameters" and "provide a means to indicate that a traffic flow shall select a component link with a maximum acceptable delay variation value" are two different things; the first is a IGP parameter (link delay or link delay variation), the second is an RSVP parameter (maximum allowable delay variation for a given LSP).  Second, the wording in Section 4.2 is informative and the wording in FR#17 is normative.

[Iftekhar] Okay. I interpreted jitter as "desirable" meaning it is an objective i.e., it is not required. But then saw the requirement FR#17 which I read as a requirement to support delay variation.  For example in FR#17 I was expecting "MAY" or "SHOULD"  rather than "MUST". 

>    this is covered in some other document.

BTW- I prefer delay over latency, but the term latency is used.  The two mean the same thing.  Neither word by itself does not indicate any one type of delay (fabric delay, transmit delay, queuing delay, geographic delay).  Neither word by itself implies intra-node delay, or delay between adjacent nodes, or end-to-end delay.

FR#8 applies to any measurement of delay, whether it is a measurement over a single physical link or over a component link carried over a multihop MPLS LSP.  Also in FR#9 there is no indication that the nodes are adjacent, and the phrase "when the path between these nodes contains one or more pairs of nodes connected via a composite link"
clearly indicates that multihop paths are being considered as well as delay between adjacent nodes.

There are some hints as to which delays the operators may consider important in this context.  FR#8 states that "The precision of latency reporting SHOULD be at least 10% of the one way latencies for latency of 1 ms or more."  This excludes most equipment latencies which can be expected to be well under 100 usec (10% of 1 msec).

The intent is to measure the predominant latency in most (well run) service provider networks, which is geographic delay on the order of msec or tens of msec and in some cases over 100 msec (intercontinental traffic).  In a congested network (possibly not so well run network, or maybe a well run network with a substantial outage) queuing delay can be quite large.  In a congested network measuring the queuing delay for the low priority traffic (the only traffic likely to be experiencing congestion, even in an outage) is more likely to add to problems by adding routing instability.

The only real question in FR#8 and FR#9 is whether queuing delay is counted in the latency figure.  The argument for including queuing delay is that it reflects the delay experienced by applications.  The argument against including queuing delay is that it if used in routing decisions it can result in routing instability.

There are quite a few documents that deal with this tradeoff.  For example, in MPLS-TP it is noted that delay measurements should be made using the highest priority COS value (which in effect very nearly eliminates any queuing delay from the measurement).

For the requirements document, this tradeoff is not discussed, however the stability and convergence requirements must be considered when coming up with a solution.  Solutions should be described in the framework document and not in the requirments document.

In the framework document (draft-so-yong-rtgwg-cl-framework) there is discussion of this tradeoff in "3.5.  Avoiding Route Oscillation" and elsewhere.  The framework document is a better place for discussion of requirements tradeoffs.

[Iftekhar] Okay, will take a look at the framework document. 

>    techniques have been developed." It would good to add a

There are a few RFCs on the topic.  Inverse multiplexing, various forms of ECMP, and Ethernet Link Aggregation are all well known.  The statement would certainly not be controversial without references.

If you are looking for further information on this, the Use Cases draft (draft-symmvo-rtgwg-cl-use-cases) has references.  "Appendix B.
Existing Multipath Standards and Techniques" was moved from the requirements document to the Use Cases draft.

We could add a reference to the use cases draft in the requirements draft if you think that would help.

[Iftekhar] I believe the requirement document would read better and easier to follow if the references are added.

The rest is discussion.

The discussion exists primarily for the reader who might otherwise not realize why there would be a disruption at all (and therefore might not understand why we have to say "minimally disruptive" instead of "completely non-disruptive").  Perhaps the entire paragraph needs to start with "For example, [...]".

In any case it is not inconsistent.  When delay changes on the current big-bad-Internet (such as in VOIP), the change is often absorbed by a jitter buffer as long as the change is small.  If a delay change of many milliseconds occurs, then it is unlikely that the typical relatively small jitter buffer can hide the delay change.

[Iftekhar] Thanks for detailed response. To me then the requirement then is about change in latency (i.e., delay variation).

>    considered in this document.

FR#16 is about delay.  FR#17 is about delay variation.  The wording "The solution SHALL provide a means" doesn't require the operator to use that feature.  Therefore it is better to keep the two in separate requirements as it is more clear that one could be enabled and the other disabled.

[Iftekhar] I think it would be clear if somehow the meaning of the above sentence "i.e., doesn't require the operator...." is included in the text.

> > https://tools.ietf.org/agenda/83/slides/slides-83-rtgwg-3.pdf (it's 
> > a quick read!).
> >  
> > drafts. In about a month's time, we will be requesting WG adoption 
> > of draft-so and draft-symmvo, and it'll be great to have an informed 
> > discussion at that time.
> >  
> > Thanks much,
> > Andy