Subject: [PWE3] WGLC comments on: draft-ietf-pwe3-congcons-02.txt

From: "Black, David" <[email-masked]>
Date: Fri, 8 Aug 2014 09:59:43 -0400

As we're now moving quickly to try to complete this draft , I agreed
w/Yaakov that my author comments would be sent in as WG Last Call comments ...
... so here they are:

Major issues:

The reference to a transport circuit breaker in section 3 is a bit optimistic
and high level:

	We can accomplish this by
	employing a transport circuit breaker, by which we mean an automatic
	mechanism for terminating a flow to prevent negative impact on other
	flows and on the stability of the network
	[I-D.fairhurst-tsvwg-circuit-breaker].

This should be accompanied by at least an example of how to do this - past
discussion has suggested carrier monitoring of delivered service quality
in order to shut down PWs that are experiencing loss rates that result in
unacceptable service quality (as defined for us by the ITU ...).  I'd also
like to add some text to suggest that operators use this sort of circuit
breaker practice when TDM PWs compete with elastic traffic.

Figure 1 and Figure 2 indicate that based on using the TCP throughput
equation as a proxy for potential impact on elastic traffic (which is what
we're doing in this draft), for the same fixed bandwidth PW, a larger segment
size is less likely to cause impacts.  I'd like to add a suggestion to use
the larger PW segment sizes as long as they're consistent with delivery of
the service.  There's been a related (long-to-the-point-of-ratholed) debate
over in the Transport Area about whether packet-rate-limited/congestible
forwarding is still relevant.  My view is that such equipment exists, and
hence that this is a relevant consideration for this draft, and would not
mention that issue here.

Minor issues:

Section 2 is about Ethernet PWs that carry elastic traffic.  Should others
be mentioned as also covered by this analysis - e.g., ATM and Frame Relay
PWs carrying elastic traffic?

Section 2 does not cover Ethernet PWs carrying inelastic traffic - that would
be unusual, but should it be explicitly noted as out of scope, or just not
discussed (as is the case for the current text)?

Section 3 should say that the reason the draft focuses on TDM PWs is that
they're the inelastic PWs that are likely to be carried over IP in practice.
My understanding is that other potentially inelastic PWs are MPLS-only
in practice.

A terminology section to roll up all the acronym expansions (e.g., TDM, CBR)
in one place seems like a good idea.

This text on p.11 could be incendiary:

	Under this condition it is safe to place the TDM PW along with
	congestion - responsive traffic such as TCP, without causing
	additional congestion.

First of all to reduce the flammability, "safe" -> "reasonable" and there
are a number of other occurrences of "safe" that deserve similar treatment ;-).
Beyond that, this assertion is dependent on the operating conditions staying
within that reasonable operating envelope, which entails use of something
like a circuit breaker - I'd say that here, and reference the earlier
circuit breaker discussion.

Figures 7 and 8 could use some shading or lines to indicate the typical
deployed operating regions for the pseudo-wires to visibly show that they
fall under the curves, particularly for larger segment sizes.  I'd also
repeat or refer to the suggestion to use larger 

Nits:

References aren't allowed in abstracts, so remove reference to
[I-D.fairhurst-tsvwg-circuit-breaker]

Introduction, first paragraph, delete "PWE3" towards end of paragraph -

   preferably the 4 byte PWE3 control word.

In addition to its current general statements, the Security Considerations
section should point to the security considerations sections in a few
particularly relevant RFCs - I'd suggest 3985, 4553, 5086 and 5087.

Thanks,
--David

> -----Original Message-----
> From: pwe3 [mailto:[email-masked]] On Behalf Of Yaakov Stein
> Sent: Thursday, July 24, 2014 4:52 PM
> To: [email-masked]
> Subject: Re: [PWE3] I-D Action: draft-ietf-pwe3-congcons-02.txt
> 
> Please look at the PDF version only
> (the txt version does not have the graphs).
> 
> Y(J)S
> 
>

---

From: "Black, David" <[email-masked]>
Date: Sat, 9 Aug 2014 22:32:24 -0400

Following up on this ...

> w/Yaakov that my author comments would be sent in as WG Last Call comments ...

... with some suggested new text ...
 
> breaker practice when TDM PWs compete with elastic traffic.

After this text in Section 3:

   We can accomplish this by
   employing a transport circuit breaker, by which we mean an automatic
   mechanism for terminating a flow to prevent negative impact on other
   flows and on the stability of the network
   [I-D.fairhurst-tsvwg-circuit-breaker].  Note that a transport circuit
   breaker is intended as a protection mechanism of last resort, just as
   an electrical circuit breaker is only triggered when absolutely
   necessary.  

I suggest adding:

   For TDM pseudowires, such a circuit breaker can be realized via
   operator monitoring of drop rates and consequent on the quality of
   TDM service delivered and shutting down TDM pseudowires that persistently
   fail to deliver acceptable TDM service (as defined by [G775] and
   [G826]).  This document explains why a TDM PW that is causing congestion
   problems via interference with elastic traffic cannot be delivering
   acceptable TDM service (see the rest of this Section, plus Appendices
   A and B).

And breaking the paragraph after the added text.

> hence that this is a relevant consideration for this draft, and would not

Ignoring the rathole invitation ;-).  This is still in section 3, after:

   We see that in general for large packet sizes, short delays, and low
   packet loss rates, the TDM pseudowires consume much less bandwidth
   than TCP would under identical conditions.  Only for small packets,
   long delays, and high packet loss ratios, do TDM PWs potentially
   consume more bandwidth, and even then only marginally.

I suggest adding:

   Comparing figures 1 and 2, use of a larger segment size for a T1 or E1
   TDM PW moves the operational behavior of the PW away from the boundary
   at which the TCP throughput equation suggests that undue interference with
   elastic traffic may start to occur.  We suggest that larger segment
   sizes be preferred for T1 and E1 TDM PWs, but note that other
   considerations may favor smaller segment sizes (e.g., jitter buffer
   size, bounding the impact of an isolated packet drop event).

and again, breaking the paragraph after that added text.

I hope that latter sentence ("We suggest that ...") is acceptable to the WG.

> PWs carrying elastic traffic?

After:

   In this section we consider Ethernet PWs that primarily carry
   congestion-responsive traffic.  We show that we automatically obtain
   the desired congestion avoidance behavior, and that additional
   mechanisms are not needed.

I suggest adding:

   This discussion is applicable to any PW that carries congestion-
   responsive traffic (e.g., ATM and Frame Relay PWs); Ethernet PWs are
   used here for clarity, as they are a common example.
 
> discussed (as is the case for the current text)?

I suggest doing nothing here, and planning to deal with this concern
if it should arise later (e.g., IETF Last Call or IESG Evaluation).
 
> My understanding is that other potentially inelastic PWs are MPLS-only

After:

   Inelastic PWs, such as TDM PWs ([RFC4553][RFC5086][RFC5087]), are
   potentially more problematic than the elastic PWs of the previous
   section.  Being constant bit-rate (CBR), TDM PWs can not be made
   responsive to congestion.  On the other hand, being CBR, they also do
   not attempt to capture additional bandwidth when neighboring TCP
   flows back off.

I suggest adding:

   We focus on TDM PWs here as they are the only inelastic PWs that are
   likely to be IP-encapsulated in practice.  Most other inelastic PWs
   tend to only be used with MPLS.

[I'd ask Yaakov and other PW experts to double-check the latter sentence,
please.]

> in one place seems like a good idea.

These four acronyms would be good to put into a terminology section:

CBR Constant Bit Rate
PSN Packet Switched Network
PW Pseudowire
TDM Time Division Multiplexing

> circuit breaker discussion.

Global substitution: "safe" -> "reasonable", starting with the abstract.

Then for this specific text on p.11:

OLD
 	Under this condition it is safe to place the TDM PW along with
 	congestion - responsive traffic such as TCP, without causing
 	additional congestion.
NEW
	Under this condition it is reasonable to place the TDM PW in competition
      with congestion-responsive traffic such as TCP, as the result should not
      produce unacceptable congestion effects.  That assumes that the operating
      conditions remain within the reasonable operating envelope - as discussed
      above, a transport circuit breaker is a suggested means of managing a TDM
      PW so that it is shut down if operating conditions remain persistently
      outside the reasonable operating envelope for the PW (e.g., due to a
      high packet loss rate).

> particularly relevant RFCs - I'd suggest 3985, 4553, 5086 and 5087.

After:

   This document does not introduce any new congestion-specific
   mechanisms and thus does not introduce any new security
   considerations above those present for PWs in general.

I suggest adding a new paragraph:

   For PW security considerations, see the security considerations sections
   of the cited RFCS; the security considerations in [RFC3985], [RFC4553],
   [RFC5086] and [RFC5087] are particularly applicable to this document.

> _______________________________________________
> pwe3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pwe3

---

From: "Black, David" <[email-masked]>
Date: Sun, 10 Aug 2014 14:45:28 -0400

Ugh, the spelling checker "helped" - "consequent on the quality" :-(.

Instead of this proposed new text from my original message:

This would be better (also reformatted as a list for clarity):

   For TDM pseudowires, such a circuit breaker can be realized via:

      1)  network operator monitoring of drop rates and the resulting
            quality of TDM service delivered; and
      2)  a network operational practice of shutting down TDM
            pseudowires that persistently fail to deliver
            acceptable TDM service (as defined by [G775] and [G826]).

   This document explains why a TDM PW that is causing congestion
   problems via interference with elastic traffic is not delivering
   acceptable TDM service (see the rest of this section, plus Appendices
   A and B).

Thanks,
--David

> From: pwe3 [mailto:[email-masked]] On Behalf Of Black, David
> Sent: Saturday, August 09, 2014 10:32 PM
> To: Yaakov Stein; [email-masked]
> Subject: Re: [PWE3] WGLC comments on: draft-ietf-pwe3-congcons-02.txt
> 
> Following up on this ...
> 
> > are a number of other occurrences of "safe" that deserve similar treatment
> ;-
> > ).

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 Aug 2014 18:01:41 +0100

Yaakov,

This time adding pointer to all my nits and 
including the list, which I hadn't noticed wasn't 
cc'd when I replied to all. More inline within my own previous email..

At 14:02 14/08/2014, Bob Briscoe wrote:
>To: Yaakov Stein <[email-masked]>
>From: Bob Briscoe <[email-masked]>
>Subject: RE: [PWE3] WGLC comments on: draft-ietf-pwe3-congcons-02.txt
>Cc: "Andrew G. Malis" <[email-masked]>, BOCCI 
>Matthew <[email-masked]>, 
>"Black, David" <[email-masked]>
>
>Yaakov,
>
>I have just re-read it thoroughly. I have 
>noticed about two dozen editorial nits where 
>correcting them would not change the intended 
>sense (altho sometimes the sense needs to change 
>to say what we intended!). Examples:
>- Unexpanded abbreviations (David's suggestion would help, but there are more)
>- Missing words, tautologies, random punctuation problems etc
>- Uses of the wrong word, even tho we know what you mean
>- The caveat about dynamics vs steady state 
>needs to shift down a para, so that it follows 
>into the para that says what to do about dynamics.
>
>Should I take the token, and edit the XML 
>directly, then you can see a diff to check if 
>you want to accept my suggestions?

I've provided xml source of my suggested edits (and a diff) here:
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03a-02.html>
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03a.txt>
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03a.xml>

I'm not implying that this is now the master 
version - it's just an easy way to describe edits 
(and you can use it as the basis for a new master 
if you want). Similarly to David, pls take my 
comments as co-author into consideration in parallel with the WGLC responses.

I couldn't alter the figures. I would ask for the following changes:
* All Figs: s/PLR/PLP/
* Figs 1-4:
   - s/delay/one-way delay.
* Fig 1: max at 10 not 50 on y axis
* Figs 5-8:
   - s/safe/undeniably safe/
* Figs 7-8: max at 0.5% on x axis
* Fig 9: s/Pd/PLP/

I incorporated those nits I could see on the list 
(Sasha, Stewart & David), some of which I had noticed myself.

>Below are six thoughts that go slightly beyond 
>'editorial'. 1-4 could be considered 
>'editorial'. But 5&6 are the more significant "techno-political" points.

Suggested resolutions of these six bigger issues 
are now also in my xml suggestion above too.

>1. The phrase "mechanisms for provisioning PWs 
>in service provider networks" is too opaque (the 
>PWE audience might understand this scoping 
>sentence, but a L4 reader might not understand 
>what we're excluding here, which could make them suspicious).
>CURRENT:
>    Mechanisms for provisioning PWs
>    in service provider networks are well understood and will not be
>    discussed further here.
>SUGGESTED:
>    The case where the service provider network 
> arranges sufficient capacity for a PW is well 
> understood and will not be discussed further 
> here. The concern is over PWs deployed 
> independently of the service provider network's 
> traffic engineering or capacity planning.
>
>2. The justification for using OWD of a few milliseconds misses the point.
>"  The one-way delay
>    of a native TDM service consists of the physical time-of-flight plus
>    125 microseconds for each TDM switch traversed; and is thus very
>    small as compared to typical PSN network-crossing latencies.
>"
>The 125us per switch is irrelevant. The point 
>that needs making is that PWs are generally 
>deployed with a short physical time of flight.
>
>3. Related point: the captions on the graphs do 
>not clearly indicate that they are OWD, not RTT. 
>This could be clarified in the text, or on the graphs.
>
>4. Change packet loss rate (and PLR) to packet 
>loss probability throughout, including on the 
>graph axes. PLR is an ambiguous term, even 
>though it is widely used in the industry. Packet 
>loss rate could be measured in packet per 
>second, whereas packet loss probability is 
>unambiguously dimensionless. Indeed, in the 
>appendices, packet loss rates with respect to time are calculated.
>
>5. Rather than David's "reasonable" I suggest we 
>use the phrase "unambiguously safe" not just 
>"safe" for the headings of Figs 5-8 and in the 
>text, as below. Otherwise, we are implying that 
>it is unsafe outside the region.

I thought of a better phrase: "undeniably safe" 
instead of "unambiguously safe".
I slightly prefer this over these other possibilities:
- unarguably safe
- indisputably safe
- unconditionally safe

>CURRENT:
>    One may view this condition as defining an operating envelope for a
>    TDM PW, as a TDM PW that occupies no more bandwidth than a TCP flow
>    causes no more congestion than that TCP flow would.  Under this
>    condition it is safe to place the TDM PW along with congestion-
>    responsive traffic such as TCP, without causing additional
>    congestion.  on the other hand, were the TDM PW to consume
>    significantly more bandwidth a TCP flow, it could contribute
>    disproportionately to congestion, and its mixture with congestion-
>    responsive traffic might be inappropriate.
>SUGGESTED:
>    One may view this condition as defining an 
> unambiguously safe operating envelope for a
>    TDM PW, as a TDM PW that occupies no more bandwidth than a TCP flow
>    causes no more congestion than that TCP flow 
> would.  This does not draw us into a debate 
> over whether TCP-friendliness is a valid 
> constraint; it simply says that there can be no 
> question that a PW is safe if it does no more 
> harm than a single TCP flow. Under this
>    condition it is unambiguously safe to place 
> the TDM PW along with congestion-
>    responsive traffic such as TCP.  On the 
> other hand, were the TDM PW to consume
>    significantly more bandwidth a TCP flow, it could contribute
>    disproportionately to congestion, and its mixture with congestion-
>    responsive traffic might start to cause concern.
>
>6. The analysis in Appendices A & B needs to 
>mention that loss bursts arise primarily from 
>tail drop buffers, and the introduction of 
>active queue management (AQM) makes losses more 
>independent of each other (less bursty). (I have 
>references showing evidence of this.) In the 
>comparisons with TCP, we have assumed 
>independent losses anyway, because they are the 
>worst case (and fortunately they are easier to 
>analyse). On a similar point, in appendix B, 
>where we say "(bursty packet loss was also 
>simulated but is not reported here)", we ought 
>to give a reason otherwise readers will be 
>suspicious. A good reason would be that uniformly random is the worst case.
>
>I've also responded to a couple of David's 
>suggestions inline (I agree with the others)...

I did /not/ incorporate the more extensive text 
requested/suggested by Stewart/David. I didn't 
want to step all over your call as editor on this.

As you'll see my responses to David's points are 
still inline below. I agree with them all except:
* I don't think we should go into any detail on a 
circuit-breaker mechanism in this draft, which 
should stay on the purely informational high ground.
* Also I don't think we need to recommend a PW 
should prefer larger packets; given we've just 
shown there isn't a problem, we don't need to propose any rememdy.
* I'm not convinced refs for Security 
Considerations are necessary, but I wouldn't mind if you added them either.

Cheers

Bob

>At 17:44 13/08/2014, Yaakov Stein wrote:
>>Bob told me on Friday of the IETF that he was 
>>going to look over the final version and comment.
>>
>>Bob – ping ?
>>
>>Y(J)S
>>
>>From: Andrew G. Malis [mailto:[email-masked]]
>>Sent: 13 August, 2014 17:02
>>To: Yaakov Stein
>>Cc: Bob Briscoe; [email-masked]; BOCCI Matthew; Black, David
>>Yaakov,
>>I still haven't heard from Bob, but you should 
>>incorporate David's updates into a new 
>>revision, and I'll use that version for my final Shepherd's review.
>>
>>Thanks,
>>Andy
>>
>>On Mon, Aug 11, 2014 at 11:16 AM, Andrew G. 
>>Malis <<mailto:[email-masked]>[email-masked]> wrote:
>>Actually, I'm going to end the LC as soon as I 
>>hear back from Bob on the IPR call. I sent him a reminder earlier this morning.
>>
>>Cheers,
>>Andy
>>
>>On Sun, Aug 10, 2014 at 9:53 AM, Andrew G. 
>>Malis <<mailto:[email-masked]>[email-masked]> wrote:
>>David,
>>
>>That's perfect, thanks! I'm going to end the LC tomorrow.
>>
>>Cheers,
>>Andy
>>
>>On Sat, Aug 9, 2014 at 10:32 PM, Black, David 
>><<mailto:[email-masked]>[email-masked]> wrote:
>>Following up on this ...
>>
>I don't think we should go there. We need to 
>keep this draft firmly as informational. If 
>someone thinks a mechanism is needed because of 
>the information in this draft, then they can write about a mechanism.
>
>
>There is no need to recommend any remedial 
>action, when we have just shown there is not a problem.
>
>Further, the "marginal problem" only applies 
>when the TCP traffic uses packets smaller than 
>128B, which we only used as a hypothetical 
>worst-case for the analysis. But in practice 
>this doesn't happen with long-running TCP flows 
>- they use packets that are as large as possible 
>(irrespective of what the PW uses). So there really is /no/ problem to remedy.
>
>
>> > are a number of other occurrences of "safe" 
>> that deserve similar treatment ;-
>> > ).
>No need to go there. See my suggestions earlier.
>
>
>I wanted to suggest this myself, but it is hard 
>to depict fuzzy boundaries. If you can do this, pls do.
>
>Given the PW operating regions are bounded by v 
>low loss probability, the x axis in Figs 7&8 
>ought to be scaled much smaller as well.
>
>Related point: the y axis in Fig 1 ought to max at 10, not 50.
>
>Regards
>
>
>
>Bob
>
>> >
>> > Nits:
>> >
>> > > To: <mailto:[email-masked]>[email-masked]
>> > <mailto:[email-masked]>[email-masked]
>><mailto:[email-masked]>[email-masked]
>________________________________________________________________
>Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: "Black, David" <[email-masked]>
Date: Thu, 14 Aug 2014 14:39:34 -0400

And forwarding my views to the list ...

... On Bob's six thoughts (I have problems with 2 & 5 - the other 4 are basically ok w/me) plus follow-ups on Bob's concerns w/my two major issues, here goes:

-- 1 -- I agree in principle, but I think Bob's suggested new text is too black/white and should be generalized.

Bob suggests:

> The case where the service provider network arranges sufficient capacity for a PW is well understood and
> will not be discussed further here. The concern is over PWs deployed independently of the service provider
> network's traffic engineering or capacity planning.

I want to include the case where capacity has been arranged, but may not be sufficient, I suggest rewriting Bob's second sentence as:

The concern is over PWs that share network capacity with elastic or congestion-responsive traffic, independent of whether that sharing is a result of the service provider network's traffic engineering and capacity planning or independent deployment of PWs by entities other than network operators.

-- 2 -- Here, I disagree.  One of the significant contributions of this draft is that TDM PWs don't cause congestion problems in practice, and one of the major reasons for that is the low latency requirements for acceptable TDM service delivery, both in ITU-T standards and in actual TDM PW deployment.  The 125us behavior of native TDM switches is relevant to the latter, and hence I prefer to include mention of it.

-- 3 -- I agree; clear captions are always a feature, not a bug ;-).

-- 4 -- I agree with changing "packet loss rate" to "packet loss probability".

-- 5 -- I firmly disagree with Bob here, although he's right that this is "techno-political" ;-).

I am concerned that S-A-F-E is likely to be a 4-letter word in wider Transport Area review, and hence, I want to use a word that does not imply anything remotely resembling an absolute guarantee that nothing can possibly ever go wrong (yes, I know Bob didn't mean to go to quite that extreme an interpretation, but S-A-F-E is liable to be read that way by those with an "axe to grind"), even though there's lots of solid operational experience that suggests all is well here.

-- 6 -- I think I'm ok with what Bob proposes to better characterize loss burstiness, but want to see specific text.

-- Circuit Breaker Mechanism --

>> [David] I suggest adding:
>>
> [Bob] I don't think we should go there. We need to keep this draft firmly as informational. If someone
> thinks a mechanism is needed because of the information in this draft, then they can write about a mechanism.

[David] I firmly disagree, but Bob and I might be talking past each other, as I do want to stay informational ;-).

This text is intended to describe an example or possibility, not to specify an implementation requirement.  The new text was intended as an OAM/OSS practice recommendation to the operator, not as a normative protocol spec.

I'm ok with inserting "example", "possibility" or something similar if that helps w/Bob's concern, but I strongly oppose doing nothing here, and note that this is also needed to respond to one of Stewart Bryant's WG LC comments.

-- PW segment size --

>> [David] Ignoring the rathole invitation ;-).  This is still in section 3, after:
>>
> [Bob] There is no need to recommend any remedial action, when we have just shown there is not a problem.

[David] Not exactly - Figure 1 shows that there is a potential concern at a combination of the high end of the range of TDM PW latencies (4ms), and higher packet loss ratios.

> Further, the "marginal problem" only applies when the TCP traffic uses packets smaller than 128B, which we only used
> as a hypothetical worst-case for the analysis. But in practice this doesn't happen with long-running TCP flows -
> they use packets that are as large as possible (irrespective of what the PW uses). So there really is /no/ problem to remedy.
Ok, Bob can relax - his transport credentials are intact, as this is an entry point to the bit-rate-congestible vs. packet-rate-congestible resource rathole that I want to ignore :-).

I believe that larger TDM PW segment sizes are a good thing for both this operating envelope boundary concern as well as per-packet processing overheads in TDM PW functionality, modulo other considerations, e.g., jitter buffers, and so I prefer to keep this addition.

Thanks,
--David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Thursday, August 14, 2014 9:03 AM
To: Yaakov Stein
Cc: Andrew G. Malis; BOCCI Matthew; Black, David
Subject: RE: [PWE3] WGLC comments on: draft-ietf-pwe3-congcons-02.txt

Yaakov,

I have just re-read it thoroughly. I have noticed about two dozen editorial nits where correcting them would not change the intended sense (altho sometimes the sense needs to change to say what we intended!). Examples:
- Unexpanded abbreviations (David's suggestion would help, but there are more)
- Missing words, tautologies, random punctuation problems etc
- Uses of the wrong word, even tho we know what you mean
- The caveat about dynamics vs steady state needs to shift down a para, so that it follows into the para that says what to do about dynamics.

Should I take the token, and edit the XML directly, then you can see a diff to check if you want to accept my suggestions?

Below are six thoughts that go slightly beyond 'editorial'. 1-4 could be considered 'editorial'. But 5&6 are the more significant "techno-political" points.

1. The phrase "mechanisms for provisioning PWs in service provider networks" is too opaque (the PWE audience might understand this scoping sentence, but a L4 reader won't understand what we're excluding here, which will make them suspicious).
CURRENT:
   Mechanisms for provisioning PWs
   in service provider networks are well understood and will not be
   discussed further here.
SUGGESTED:
   The case where the service provider network arranges sufficient capacity for a PW is well understood and will not be discussed further here. The concern is over PWs deployed independently of the service provider network's traffic engineering or capacity planning.

The 125us per switch is irrelevant. The point that needs making is that PWs are generally deployed with a short physical time of flight.

3. Related point: the captions on the graphs do not clearly indicate that they are OWD, not RTT. This could be clarified in the text, or on the graphs.

4. Change packet loss rate (and PLR) to packet loss probability throughout, including on the graph axes. PLR is an ambiguous term, even though it is widely used in the industry. Packet loss rate could be measured in packet per second, whereas packet loss probability is unambiguously dimensionless. Indeed, in the appendices, packet loss rates with respect to time are calculated.

5. Rather than David's "reasonable" I suggest we use the phrase "unambiguously safe" not just "safe" for the headings of Figs 5-8 and in the text, as below. Otherwise, we are implying that it is unsafe outside the region.

CURRENT:
   One may view this condition as defining an operating envelope for a
   TDM PW, as a TDM PW that occupies no more bandwidth than a TCP flow
   causes no more congestion than that TCP flow would.  Under this
   condition it is safe to place the TDM PW along with congestion-
   responsive traffic such as TCP, without causing additional
   congestion.  on the other hand, were the TDM PW to consume
   significantly more bandwidth a TCP flow, it could contribute
   disproportionately to congestion, and its mixture with congestion-
   responsive traffic might be inappropriate.
SUGGESTED:
   One may view this condition as defining an unambiguously safe operating envelope for a
   TDM PW, as a TDM PW that occupies no more bandwidth than a TCP flow
   causes no more congestion than that TCP flow would.  This does not draw us into a debate over whether TCP-friendliness is a valid constraint; it simply says that there can be no question that a PW is safe if it does no more harm than a single TCP flow. Under this
   condition it is unambiguously safe to place the TDM PW along with congestion-
   responsive traffic such as TCP.  On the other hand, were the TDM PW to consume
   significantly more bandwidth a TCP flow, it could contribute
   disproportionately to congestion, and its mixture with congestion-
   responsive traffic might start to cause concern.

6. The analysis in Appendices A & B needs to mention that loss bursts arise primarily from tail drop buffers, and the introduction of active queue management (AQM) makes losses more independent of each other (less bursty). (I have references showing evidence of this.) In the comparisons with TCP, we have assumed independent losses anyway, because they are the worst case (and fortunately they are easier to analyse). On a similar point, in appendix B, where we say "(bursty packet loss was also simulated but is not reported here)", we ought to give a reason otherwise readers will be suspicious. A good reason would be that uniformly random is the worst case.

I've also responded to a couple of David's suggestions inline (I agree with the others)...

At 17:44 13/08/2014, Yaakov Stein wrote:
Bob told me on Friday of the IETF that he was going to look over the final version and comment.

Bob - ping ?

Y(J)S

From: Andrew G. Malis [ mailto:[email-masked]]
Sent: 13 August, 2014 17:02
To: Yaakov Stein
Cc: Bob Briscoe; [email-masked]<mailto:[email-masked]>; BOCCI Matthew; Black, David
Subject: Re: [PWE3] WGLC comments on: draft-ietf-pwe3-congcons-02.txt

Yaakov,

I still haven't heard from Bob, but you should incorporate David's updates into a new revision, and I'll use that version for my final Shepherd's review.

Thanks,
Andy

On Mon, Aug 11, 2014 at 11:16 AM, Andrew G. Malis <[email-masked]<mailto:[email-masked]>> wrote:
Actually, I'm going to end the LC as soon as I hear back from Bob on the IPR call. I sent him a reminder earlier this morning.

Cheers,
Andy

On Sun, Aug 10, 2014 at 9:53 AM, Andrew G. Malis <[email-masked]<mailto:[email-masked]>> wrote:
David,

Cheers,
Andy

On Sat, Aug 9, 2014 at 10:32 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:
Following up on this ...

I suggest adding:

I don't think we should go there. We need to keep this draft firmly as informational. If someone thinks a mechanism is needed because of the information in this draft, then they can write about a mechanism.

I suggest adding:

There is no need to recommend any remedial action, when we have just shown there is not a problem.

Further, the "marginal problem" only applies when the TCP traffic uses packets smaller than 128B, which we only used as a hypothetical worst-case for the analysis. But in practice this doesn't happen with long-running TCP flows - they use packets that are as large as possible (irrespective of what the PW uses). So there really is /no/ problem to remedy.

After:

I suggest adding:

After:

I suggest adding:

CBR Constant Bit Rate

OLD

> repeat or refer to the suggestion to use larger

I wanted to suggest this myself, but it is hard to depict fuzzy boundaries. If you can do this, pls do.

Given the PW operating regions are bounded by v low loss probability, the x axis in Figs 7&8 ought to be scaled much smaller as well.

Regards

Bob

After:

> > From: pwe3 [ mailto:[email-masked]] On Behalf Of Yaakov Stein
> > To: [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
pwe3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/pwe3

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 15 Aug 2014 11:03:14 +0100

David,

I decided to think on a better word for 'safe' 
overnight, and allow others to comment. But 
no-one has, so here's my responses to all your outstanding points...

I've provided slightly updated xml source of my 
suggested edits (and a diff) here:
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03b-02.html>
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.txt>
<http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.xml>
Again note: xml is purely for precision, not 
implying I have taken over editorship.

1. Happy with your new scoping text, except the 
the first 'independent' clangs with the second, and it's unnecessary anyway:
"  The concern is over PWs that share network 
capacity with elastic or congestion-responsive 
traffic, [independent of] whether that sharing is 
a result of the service provider network's 
traffic engineering and capacity planning or 
independent deployment of PWs by entities other than network operators.
"

2. My point wasn't to remove the reference to the 
125us (in my suggested text I didn't). It was 
that the 125us stuff /alone/ doesn't warrant the 
'thus' in the next clause; the logic needs to be 
connected by talking about the sum of the flight 
time /and/ the per-node processing time.
CURRENT:
    The one-way delay
    of a native TDM service consists of the physical time-of-flight plus
    125 microseconds for each TDM switch traversed; and is thus very
    small as compared to typical PSN network-crossing latencies.
SUGGESTED:
    The one-way delay
    of a native TDM service consists of the physical time-of-flight plus
    125 microseconds for each TDM switch 
traversed; so the total one-way delay for most 
TDM PWs has to be small compared to typical PSN network-crossing latencies.

5. I understand and agree - 'S-A-F-E region' is 
incendiary. But I don't like 'reasonable region' 
- it's slightly /too/ meaningless.

Other possibilities are:
- admissable region (problem: implies outside the region is inadmissable)
- harmonious region (problem: too Californian)
So I suggest two words:
- friendly or acceptable, each used depending on context.

On the graphs, I suggest 'operating region 
(friendly under curve)' because that is precisely 
where the curves come from - 'friendly' does what 
it says on the tin, and the implied 'unfriendly' 
above the curve correctly avoids implying that the sky will fall.

Elsewhere, I suggest the following replacements for the S-word:
"It is thus worthwhile specifying under what 
conditions such competition is acceptable"
"One may view this condition as defining a 
'friendly' operating envelope for a TDM PW"
"Under this condition it is acceptable to place 
the TDM PW along with ... TCP. "
"comparison with something widely considered to 
offer an acceptable response to congestion"

7. Circuit breaker
Scope creep. Rat-hole. Stop digging.
Yes, I know Stewart has asked us to dig out 
another rat as well, but the appropriate response is "Out of scope".

You might think your text is uncontroversial, but 
if we are tempted to take Stewart's bait (rat 
metaphor intended), it will probably start new 
debates that this doc doesn't deserve: "What does 
persistent mean?", "Which traffic is more important/ valuable?", yada yada.

I suggest the text merely makes it clear that 
this debate is in scope for elsewhere, not here. 
I've also just realised that Yaakov has written 
the sentences around circuit breaker with the 
wrong logical sequence. The circuit breaker needs 
to be introduced as something the PW uses when 
its /own/ quality has failed, which should avoid 
any need to shut down because it is threatening elastic services:

CURRENT:
       If the bandwidth consumed by a TDM PW is
       considered detrimental, the only available remedy is to completely
       shut down the PW, by using a transport circuit breaker mechanism.
       However, we will show that even before such an action is
       warranted, the PW will become unable to faithfully emulate the
       native TDM service; for example, when a TDM service is carrying
       voice grade telephony channels, the voice quality will degrade to
       below useful levels.
SUGGESTED:
       We will show that, before a PW becomes 
deterimental to elastic traffic, it will become 
unable to faithfully emulate the
       native TDM service; for example, when a TDM service is carrying
       voice grade telephony channels, the voice quality will degrade to
       below useful levels. Once the quality of 
the emulated service is persistently 
unacceptable, if a PW shuts down selected flows 
or shuts itself down completely, it will 
automatically avoid detriment to elastic traffic. 
A transport circuit breaker 
[I-D.fairhurst-tsvwg-circuit-breaker] could be 
used, but mechanism is beyond the scope of the present document.

8. Segment size.
Everyone's in a big group-hug because we've 
written a draft showing there's is no need to 
change anything. Then just at the end of an 
uneventful WGLC, we hear David's voice from the 
bottom of a deep hole shouting, "avoid the 
rat-hole invitation", while furiously digging for a rat.

There is no need to add a recommendation that PWs 
need to be changed to use larger packets. No 
matter how hedged and soft this recommendation 
would be, it's not needed anyway. Stop digging.

Yaakov's new text is fine as it stands. "Only for 
small packets, long delays, and high packet loss 
ratios, do TDM PWs potentially consume more 
bandwidth, and even then only marginally. "

In order to justify your sentence, we would have 
to include graphs with packet-rate (not bit-rate) 
on the vertical axis and go into a whole treatise 
as to why packet-rate comparison might be 
important as well. Then the graph would show one 
plot for all TCP segment sizes and multiple plots 
for PWs with different packet sizes. Then we'd 
see that the corner of the two smallest packet 
PWs just clipped the TCP curve at loss 
probabilities that are stupidly large anyway. So 
even after all this, it would hardly justify a need for your sentence.

All that digging would not be worth it, just to 
discover (again) that there is no rat.

Bob

At 19:39 14/08/2014, Black, David wrote:
>Content-Language: en-US
>Content-Type: multipart/alternative;
> 
>boundary="_000_8D3D17ACE214DC429325B2B98F3AE712077B95212CMX15Acorpemcc_"
>
>... On Bob’s six thoughts (I have problems with 
>2 & 5 - the other 4 are basically ok w/me) plus 
>follow-ups on Bob’s concerns w/my two major issues, here goes:
>
>-- 1 -- I agree in principle, but I think Bob’s 
>suggested new text is too black/white and should be generalized.
>
>Bob suggests:
>
> arranges sufficient capacity for a PW is well understood and
> > will not be discussed further here. The 
> concern is over PWs deployed independently of the service provider
>I want to include the case where capacity has 
>been arranged, but may not be sufficient, I 
>suggest rewriting Bob’s second sentence as:
>
>The concern is over PWs that share network 
>capacity with elastic or congestion-responsive 
>traffic, independent of whether that sharing is 
>a result of the service provider network's 
>traffic engineering and capacity planning or 
>independent deployment of PWs by entities other than network operators.
>
>-- 2 -- Here, I disagree.  One of the 
>significant contributions of this draft is that 
>TDM PWs don’t cause congestion problems in 
>practice, and one of the major reasons for that 
>is the low latency requirements for acceptable 
>TDM service delivery, both in ITU-T standards 
>and in actual TDM PW deployment.  The 125us 
>behavior of native TDM switches is relevant to 
>the latter, and hence I prefer to include mention of it.
>
>-- 4 -- I agree with changing “packet loss rate” to “packet loss probability”.
>
>-- 5 -- I firmly disagree with Bob here, 
>although he’s right that this is “techno-political” ;-).
>
>I am concerned that S-A-F-E is likely to be a 
>4-letter word in wider Transport Area review, 
>and hence, I want to use a word that does not 
>imply anything remotely resembling an absolute 
>guarantee that nothing can possibly ever go 
>wrong (yes, I know Bob didn’t mean to go to 
>quite that extreme an interpretation, but 
>S-A-F-E is liable to be read that way by those 
>with an “axe to grind”), even though there’s 
>lots of solid operational experience that suggests all is well here.
>
>-- 6 -- I think I’m ok with what Bob proposes to 
>better characterize loss burstiness, but want to see specific text.
>
> > [Bob] I don't think we should go there. We 
> need to keep this draft firmly as informational. If someone
> > thinks a mechanism is needed because of the 
> information in this draft, then they can write about a mechanism.
>
>[David] I firmly disagree, but Bob and I might 
>be talking past each other, as I do want to stay informational ;-).
>
>This text is intended to describe an example or 
>possibility, not to specify an implementation 
>requirement.  The new text was intended as an 
>OAM/OSS practice recommendation to the operator, 
>not as a normative protocol spec.
>
>I’m ok with inserting “example”, “possibility” 
>or something similar if that helps w/Bob’s 
>concern, but I strongly oppose doing nothing 
>here, and note that this is also needed to 
>respond to one of Stewart Bryant’s WG LC comments.
>
>-- PW segment size --
>
> > [Bob] There is no need to recommend any 
> remedial action, when we have just shown there is not a problem.
>
>[David] Not exactly - Figure 1 shows that there 
>is a potential concern at a combination of the 
>high end of the range of TDM PW latencies (4ms), 
>and higher packet loss ratios.
>
> when the TCP traffic uses packets smaller than 128B, which we only used
> > as a hypothetical worst-case for the 
> analysis. But in practice this doesn't happen with long-running TCP flows -
> > they use packets that are as large as 
> possible (irrespective of what the PW uses). So 
> there really is /no/ problem to remedy.
>Ok, Bob can relax - his transport credentials 
>are intact, as this is an entry point to the 
>bit-rate-congestible vs. packet-rate-congestible 
>resource rathole that I want to ignore :-).
>
>I believe that larger TDM PW segment sizes are a 
>good thing for both this operating envelope 
>boundary concern as well as per-packet 
>processing overheads in TDM PW functionality, 
>modulo other considerations, e.g., jitter 
>buffers, and so I prefer to keep this addition.
>
>Thanks,
>--David
>
>From: Bob Briscoe [<mailto:[email-masked]>mailto:[email-masked]]
>sentence, but a L4 reader won't understand what 
>we're excluding here, which will make them suspicious).
>CURRENT:
>Cc: Bob Briscoe; 
><mailto:[email-masked]>[email-masked]; 
>BOCCI Matthew; Black, David
>incorporate David's updates into a new revision, 
>and I'll use that version for my final Shepherd's review.
>
>Thanks,
>Andy
>
>On Sun, Aug 10, 2014 at 9:53 AM, Andrew G. Malis 
><<mailto:[email-masked]>[email-masked]> wrote:
>David,
>

---

From: "Black, David" <[email-masked]>
Date: Mon, 18 Aug 2014 08:44:39 -0400

Bob,

1-4 & 6: ok

5: "acceptable" is a good replacement for S-A-F-E, but I'm not sure about "friendly".  The key concept here is assurance that there are no problems.  How about "clearly acceptable"?  That may need some additional text to say that acceptable network and service behavior may be delivered outside the clearly acceptable region, but one cannot rely on the analysis in this draft (based on the TCP throughput formula) to reach such a conclusion.

7: I disagree - the tsvwg circuit breaker draft is rather general, and we now have two WG LC comments on this draft asking how to apply a circuit breaker to PWs.  I think we're on solid ground with the references to the ITU-T service quality standards and expecting operator monitoring of service quality.  OTOH, the word "persistently" seemed to be a concern, and I have no problem removing it (anyone who asks about that can be directed to G.775 and G.826).  New suggested text:

   For TDM pseudowires, such a circuit breaker can be realized via
   operator monitoring of drop rates and the quality of
   TDM service delivered in order to shut down TDM pseudowires that
   fail to deliver acceptable TDM service (as defined by [G775] and
   [G826]).  This document explains why a TDM PW that is causing congestion
   problems via interference with elastic traffic is also not delivering
   acceptable TDM service (see the rest of this Section, plus Appendices
   A and B).

8: I believe that figures 1 and 2 suffice to make the point, but I see that a recommendation may be a bit of a reach for this draft.  How about the following modified form of my first sentence, which just states the inference that can be drawn from Figures 1 and 2? -

   Figures 1 and 2 indicate that a larger segment size for a T1 or E1
   TDM PW moves the operational behavior of the PW away from the boundary
   at which the TCP throughput equation suggests that undue interference with
   elastic traffic may start to occur.

Thanks,
--David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, August 15, 2014 6:03 AM
To: Black, David
Cc: [email-masked]
Subject: Re: [PWE3] FW: WGLC comments on: draft-ietf-pwe3-congcons-02.txt

David,

I decided to think on a better word for 'safe' overnight, and allow others to comment. But no-one has, so here's my responses to all your outstanding points...

I've provided slightly updated xml source of my suggested edits (and a diff) here:
< http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03b-02.html >
< http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.txt >
< http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.xml >
Again note: xml is purely for precision, not implying I have taken over editorship.

1. Happy with your new scoping text, except the the first 'independent' clangs with the second, and it's unnecessary anyway:
"  The concern is over PWs that share network capacity with elastic or congestion-responsive traffic, [independent of] whether that sharing is a result of the service provider network's traffic engineering and capacity planning or independent deployment of PWs by entities other than network operators.
"

2. My point wasn't to remove the reference to the 125us (in my suggested text I didn't). It was that the 125us stuff /alone/ doesn't warrant the 'thus' in the next clause; the logic needs to be connected by talking about the sum of the flight time /and/ the per-node processing time.
CURRENT:
   The one-way delay
   of a native TDM service consists of the physical time-of-flight plus
   125 microseconds for each TDM switch traversed; and is thus very
   small as compared to typical PSN network-crossing latencies.
SUGGESTED:
   The one-way delay
   of a native TDM service consists of the physical time-of-flight plus
   125 microseconds for each TDM switch traversed; so the total one-way delay for most TDM PWs has to be small compared to typical PSN network-crossing latencies.

5. I understand and agree - 'S-A-F-E region' is incendiary. But I don't like 'reasonable region' - it's slightly /too/ meaningless.

Other possibilities are:

On the graphs, I suggest 'operating region (friendly under curve)' because that is precisely where the curves come from - 'friendly' does what it says on the tin, and the implied 'unfriendly' above the curve correctly avoids implying that the sky will fall.

Elsewhere, I suggest the following replacements for the S-word:
"It is thus worthwhile specifying under what conditions such competition is acceptable"
"One may view this condition as defining a 'friendly' operating envelope for a TDM PW"
"Under this condition it is acceptable to place the TDM PW along with ... TCP. "
"comparison with something widely considered to offer an acceptable response to congestion"

7. Circuit breaker
Scope creep. Rat-hole. Stop digging.
Yes, I know Stewart has asked us to dig out another rat as well, but the appropriate response is "Out of scope".

You might think your text is uncontroversial, but if we are tempted to take Stewart's bait (rat metaphor intended), it will probably start new debates that this doc doesn't deserve: "What does persistent mean?", "Which traffic is more important/ valuable?", yada yada.

I suggest the text merely makes it clear that this debate is in scope for elsewhere, not here. I've also just realised that Yaakov has written the sentences around circuit breaker with the wrong logical sequence. The circuit breaker needs to be introduced as something the PW uses when its /own/ quality has failed, which should avoid any need to shut down because it is threatening elastic services:

CURRENT:
      If the bandwidth consumed by a TDM PW is
      considered detrimental, the only available remedy is to completely
      shut down the PW, by using a transport circuit breaker mechanism.
      However, we will show that even before such an action is
      warranted, the PW will become unable to faithfully emulate the
      native TDM service; for example, when a TDM service is carrying
      voice grade telephony channels, the voice quality will degrade to
      below useful levels.
SUGGESTED:
      We will show that, before a PW becomes deterimental to elastic traffic, it will become unable to faithfully emulate the
      native TDM service; for example, when a TDM service is carrying
      voice grade telephony channels, the voice quality will degrade to
      below useful levels. Once the quality of the emulated service is persistently unacceptable, if a PW shuts down selected flows or shuts itself down completely, it will automatically avoid detriment to elastic traffic. A transport circuit breaker [I-D.fairhurst-tsvwg-circuit-breaker] could be used, but mechanism is beyond the scope of the present document.

8. Segment size.
Everyone's in a big group-hug because we've written a draft showing there's is no need to change anything. Then just at the end of an uneventful WGLC, we hear David's voice from the bottom of a deep hole shouting, "avoid the rat-hole invitation", while furiously digging for a rat.

There is no need to add a recommendation that PWs need to be changed to use larger packets. No matter how hedged and soft this recommendation would be, it's not needed anyway. Stop digging.

Yaakov's new text is fine as it stands. "Only for small packets, long delays, and high packet loss ratios, do TDM PWs potentially consume more bandwidth, and even then only marginally. "

In order to justify your sentence, we would have to include graphs with packet-rate (not bit-rate) on the vertical axis and go into a whole treatise as to why packet-rate comparison might be important as well. Then the graph would show one plot for all TCP segment sizes and multiple plots for PWs with different packet sizes. Then we'd see that the corner of the two smallest packet PWs just clipped the TCP curve at loss probabilities that are stupidly large anyway. So even after all this, it would hardly justify a need for your sentence.

All that digging would not be worth it, just to discover (again) that there is no rat.

Bob

Content-Language: en-US

-- PW segment size --

Thanks,
--David

Yaakov,

CURRENT:

Bob - ping ?

Y(J)S

Yaakov,

Thanks,
Andy

Cheers,
Andy

Cheers,
Andy

I suggest adding:

I suggest adding:

After:

I suggest adding:

After:

I suggest adding:

CBR Constant Bit Rate

OLD

Regards

Bob

After:

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 20 Aug 2014 19:35:35 +0100

David,

Sry for delay. inline...

(Despite end of WGLC, we can still continue our 
public intra-author ping-pong :)

At 13:44 18/08/2014, Black, David wrote:
>Bob,
>
>1-4 & 6: ok
>
>5: “acceptable” is a good replacement for 
>S-A-F-E, but I’m not sure about “friendly”.  The 
>key concept here is assurance that there are no 
>problems.  How about “clearly acceptable”?  That 
>may need some additional text to say that 
>acceptable network and service behavior may be 
>delivered outside the clearly acceptable region, 
>but one cannot rely on the analysis in this 
>draft (based on the TCP throughput formula) to reach such a conclusion.

I wanted a different word from 'acceptable' for 
the adjective in the captions above the Figures 
describing the regions, because there's no 
opportunity for more text there to explain that 
we do not mean that outside the region is 
unacceptable. I thought about various qualifiers 
with acceptable (like clearly, unarguably, etc), 
but I don't think these are right either.

I figured that the curves that delineate the 
region are defined by the TCP-friendly equation, 
so 'friendly region' says on the tin what it does 
(and it is OK to imply that outside the region is 
unfriendly, because unfriendly doesn't imply 
"Don't go there" like 'unacceptable' would).

Let's stop batting back & forth until we see what Yaakov (or others) think.

>7: I disagree - the tsvwg circuit breaker draft 
>is rather general, and we now have two WG LC 
>comments on this draft asking how to apply a 
>circuit breaker to PWs.  I think we’re on solid 
>ground with the references to the ITU-T service 
>quality standards and expecting operator 
>monitoring of service quality.  OTOH, the word 
>“persistently” seemed to be a concern, and I 
>have no problem removing it (anyone who asks 
>about that can be directed to G.775 and G.826).  New suggested text:
>

If we're going to say a bit about how to realize 
a circuit-breaker, I'd be happier with the word 
persistently than without. I'd still rather we 
didn't allow this draft to mission creep into mechanism mode.

In fact, on reflection, whatever we say about 
circuit-breakers, I'd prefer not to mention 
circuit-breakers in the abstract, which implies 
this draft says stuff about mechanism, which was never the intent.

>
>8: I believe that figures 1 and 2 suffice to 
>make the point, but I see that a recommendation 
>may be a bit of a reach for this draft.  How 
>about the following modified form of my first 
>sentence, which just states the inference that 
>can be drawn from Figures 1 and 2? -
>

Not so - that's the problem. On these graphs, the 
op behaviour plots won't move at all if the PW 
uses larger packets. These are bit-rate plots, so 
the only way to move anything is to change the 
TCP segment size (which moves the TCP plots). To 
justify what you want to say, either:
a) these would have to be packet-rate plots, or
b) we would have to turn the assumption we've 
made about equal packet sizes into an operational 
requirement, not just an analytical assumption.

To go in either of these directions would require a spade and vermin.

Bob

PS. GOing offline until next Tue. Might check email, but no guarantee.

>< 
>http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03b-02.html  
> >
>< 
>http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.txt  
> >
>< 
>http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.xml  
> >
>traffic, [independent of] whether that sharing 
>is a result of the service provider network's 
>2. My point wasn't to remove the reference to 
>the 125us (in my suggested text I didn't). It 
>was that the 125us stuff /alone/ doesn't warrant 
>the 'thus' in the next clause; the logic needs 
>to be connected by talking about the sum of the 
>flight time /and/ the per-node processing time.
>CURRENT:
>    The one-way delay
>(friendly under curve)' because that is 
>precisely where the curves come from - 
>'friendly' does what it says on the tin, and the 
>implied 'unfriendly' above the curve correctly 
>avoids implying that the sky will fall.
>
>You might think your text is uncontroversial, 
>but if we are tempted to take Stewart's bait 
>(rat metaphor intended), it will probably start 
>new debates that this doc doesn't deserve: "What 
>does persistent mean?", "Which traffic is more 
>important/ valuable?", yada yada.
>
>wrong logical sequence. The circuit breaker 
>needs to be introduced as something the PW uses 
>when its /own/ quality has failed, which should 
>avoid any need to shut down because it is threatening elastic services:
>
>CURRENT:
> automatically avoid detriment to elastic 
> traffic. A transport circuit breaker 
>There is no need to add a recommendation that 
>PWs need to be changed to use larger packets. No 
>Yaakov's new text is fine as it stands. "Only 
>for small packets, long delays, and high packet 
>loss ratios, do TDM PWs potentially consume more 
>to include graphs with packet-rate (not 
>bit-rate) on the vertical axis and go into a 
>whole treatise as to why packet-rate comparison 
>might be important as well. Then the graph would 
>show one plot for all TCP segment sizes and 
>multiple plots for PWs with different packet 
>sizes. Then we'd see that the corner of the two 
>smallest packet PWs just clipped the TCP curve 
>at loss probabilities that are stupidly large 
>anyway. So even after all this, it would hardly 
>justify a need for your sentence.
>
>From: Andrew G. Malis [ <mailto:[email-masked]>mailto:[email-masked]]
> > > From: pwe3 [ 
> <mailto:[email-masked]>mailto:[email-masked]] 
> On Behalf Of Yaakov Stein

---

From: "Black, David" <[email-masked]>
Date: Wed, 20 Aug 2014 14:59:03 -0400

Bob,

I think we're close - with luck, this email provides resolutions in principle to 5 and 7,
leaving 8, where I don't understand your argument.

-- On 5:

> I figured that the curves that delineate the region are defined by the TCP-friendly equation, so 'friendly region' says on
> the tin what it does (and it is OK to imply that outside the region is unfriendly, because unfriendly doesn't imply

"TCP-friendly" and "possibly TCP-unfriendly" would be fine with me -
I'm just uncomfortable with use of vanilla unqualified "friendly."

-- On 7, in reverse order:

> In fact, on reflection, whatever we say about circuit-breakers, I'd prefer not to mention circuit-breakers in the
> abstract, which implies this draft says stuff about mechanism, which was never the intent.

Removal of mention of circuit breakers from the abstract is fine with me.

> If we're going to say a bit about how to realize a circuit-breaker, I'd be happier with the word persistently
> than without. I'd still rather we didn't allow this draft to mission creep into mechanism mode.

I continue to agree with Stewart that something needs to be said here - the proposed
text leaves enormous flexibility to the operator to decide when to shut down TDM PWs.
Here it is with "persistently" restored:

   persistently fail to deliver acceptable TDM service (as defined by [G775] and

-- On 8, I think I see the start of our disagreement:

> On these graphs, the op behaviour plots won't move at all if the PW uses larger packets. These are
> bit-rate plots, so the only way to move anything is to change the TCP segment size (which moves the TCP plots).

Actually, they're bit-rate plots on the assumption that both the PW and TCP are using the
same segment size, which is why Figure 2 looks better for the PWs than Figure 1 (the PW
lines did not move, but the TCP curves did move due to the segment size change).

Or in other words:
- Figure 1 compares a PW using 64-byte segments to a TCP connection using 64-byte segments.
- Figure 2 compares a PW using 256-byte segments to a TCP connection using 256-byte segments.
- Nothing in the draft makes a comparison using different segment sizes.

So, "if the PW uses larger packets" then the approach in this draft compares it to a TCP
connection using larger segment, and the result looks like Figure 2 instead of Figure 1.

I do not like the idea of comparing a PW using 256-byte segments with a TCP connection
using 64-byte segments, which is what I understand that above "won't move at all" assertion
to be about.  The TCP curves do move between Figure 1 and Figure 2, based solely on a TCP
segment size change.

Thanks,
--David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Wednesday, August 20, 2014 2:36 PM
To: Black, David
Cc: [email-masked]
Subject: RE: [PWE3] FW: WGLC comments on: draft-ietf-pwe3-congcons-02.txt

David,

Sry for delay. inline...

(Despite end of WGLC, we can still continue our public intra-author ping-pong :)

Bob,

1-4 & 6: ok

I wanted a different word from 'acceptable' for the adjective in the captions above the Figures describing the regions, because there's no opportunity for more text there to explain that we do not mean that outside the region is unacceptable. I thought about various qualifiers with acceptable (like clearly, unarguably, etc), but I don't think these are right either.

I figured that the curves that delineate the region are defined by the TCP-friendly equation, so 'friendly region' says on the tin what it does (and it is OK to imply that outside the region is unfriendly, because unfriendly doesn't imply "Don't go there" like 'unacceptable' would).

If we're going to say a bit about how to realize a circuit-breaker, I'd be happier with the word persistently than without. I'd still rather we didn't allow this draft to mission creep into mechanism mode.

In fact, on reflection, whatever we say about circuit-breakers, I'd prefer not to mention circuit-breakers in the abstract, which implies this draft says stuff about mechanism, which was never the intent.

Not so - that's the problem. On these graphs, the op behaviour plots won't move at all if the PW uses larger packets. These are bit-rate plots, so the only way to move anything is to change the TCP segment size (which moves the TCP plots). To justify what you want to say, either:
a) these would have to be packet-rate plots, or
b) we would have to turn the assumption we've made about equal packet sizes into an operational requirement, not just an analytical assumption.

Bob

Thanks,
--David

From: Bob Briscoe [ mailto:[email-masked]]
Sent: Friday, August 15, 2014 6:03 AM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [PWE3] FW: WGLC comments on: draft-ietf-pwe3-congcons-02.txt

David,

Other possibilities are:

7. Circuit breaker

CURRENT:

8. Segment size.

Bob

Content-Language: en-US

-- PW segment size --

Thanks,
--David

Yaakov,

CURRENT:

Bob - ping ?

Y(J)S

Yaakov,

Thanks,
Andy

Cheers,
Andy

Cheers,
Andy

I suggest adding:

I suggest adding:

After:

I suggest adding:

After:

I suggest adding:

CBR Constant Bit Rate

OLD

Regards

Bob

After:

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Aug 2014 19:31:41 +0100

David,

At 19:59 20/08/2014, Black, David wrote:
>Bob,
>
>I think we’re close - with luck, this email 
>provides resolutions in principle to 5 and 7,
>leaving 8, where I don’t understand your argument.

...I'd like to hear from Yaakov before we start 
declaring consensus - it's been just two of us so far!

> region are defined by the TCP-friendly equation, so 'friendly region' says on
> > the tin what it does (and it is OK to imply 
> that outside the region is unfriendly, because unfriendly doesn't imply
>“TCP-friendly” and “possibly TCP-unfriendly” would be fine with me -
>I’m just uncomfortable with use of vanilla unqualified “friendly.”

I'm sold on TCP-friendly region.

> circuit-breakers, I'd prefer not to mention circuit-breakers in the
> > abstract, which implies this draft says stuff 
> about mechanism, which was never the intent.
>
> > If we're going to say a bit about how to 
> realize a circuit-breaker, I'd be happier with the word persistently
> > than without. I'd still rather we didn't 
> allow this draft to mission creep into mechanism mode.
>

That's my point. A draft that isn't about 
mechanism shouldn't tell a developer or an 
operator what they can or can't do about 
mechanism. If we want to talk about that, we will 
need another draft (perhaps a PWE example in the 
circuit-breakers draft, which has an intended status of Standards Track).

>Here it is with “persistently” restored:
>

I can live with this much text. But don't blame 
me if this causes the draft to get referred back 
or questioned as it goes through subsequent phases of the RFC process.

> > On these graphs, the op behaviour plots won't 
> move at all if the PW uses larger packets. These are
> > bit-rate plots, so the only way to move 
> anything is to change the TCP segment size (which moves the TCP plots).
>
>Actually, they’re bit-rate plots on the 
>assumption that both the PW and TCP are using the
>- Figure 1 compares a PW using 64-byte segments 
>to a TCP connection using 64-byte segments.
>- Figure 2 compares a PW using 256-byte segments 
>to a TCP connection using 256-byte segments.
>So, “if the PW uses larger packets” then the 
>approach in this draft compares it to a TCP
>using 64-byte segments, which is what I 
>understand that above “won’t move at all” assertion
>to be about.  The TCP curves do move between 
>Figure 1 and Figure 2, based solely on a TCP
>segment size change.

Correct. We're getting somewhere. The draft 
actually says two sentences about this assumption:
"   In order to quantitatively compare TDM PWs to TCP flows, we will
    compare the effect of TDM PW packets with that of TCP packets of the
    same packet size and sent at the same rate.  This is potentially an
    overly pessimistic comparison, as TDM PW packets are frequently
    configured to be short in order to minimize latency, while TCP
    packets are free to be much larger.
"

You are focusing solely on the first sentence, 
which we only used to simplify presentation of the analysis.

I am saying any recommendations should be based 
on the analysis /after/ having adjusted the 
outcome to reverse out any assumptions that 
otherwise make the analysis unrealistic (ie. take 
account of the second quoted sentence).

But my case is even stronger than that:

a) Even if we solely make the assumption of 
comparable packet sizes (for the sake of 
analysis) and ignore the second qualifying 
sentence (the reality), then as we continue 
through the whole doc we still get to the point 
where we show the problem is virtually non-existent. Given:
   * TCP-friendliness is not an absolute 
requirement - one bit-rate within half-a-dozen 
times the other is fine (I recall Sally FLoyd 
writing somewhere that x8 shouldn't be a problem)
   * the overlap only occurs because we've drawn 
the upper bound of loss probability for a PWE 
conservatively high (Appendix A says this).

So we don't want to then move straight into 
recommending a solution in the following 
paragraph, which makes it look like we don't 
believe our own conclusion - that PWEs are not a 
significant capacity sharing problem - even when 
both PWE and TCP use the same size packets.

b) If we then take note of the second qualifying 
sentence above, it really does make this an 
unequivocal non-problem. So we really shouldn't 
be proposing that operators change anything.

c) Actually, if we did state your recommendation, 
it wouldn't be correct to say the analysis 
supports it, because packet size problems require 
consideration of packet-congestion, not just 
bit-congestion. So we would have to repeat the 
full analysis for packet-congestion, not 
bit-congestion, and show packet-rate graphs as 
well as bit-rate graphs. Having done this, and 
having similarly taken points (a) and (b) into 
account, we would still come to the same 
conclusion - that there is not a problem 
significant enough to need to recommend any action.

So, for all these reasons, let's not make this recommendation.

Bob

>so 'friendly region' says on the tin what it 
>does (and it is OK to imply that outside the 
>region is unfriendly, because unfriendly doesn't 
>imply "Don't go there" like 'unacceptable' would).
>
>Not so - that's the problem. On these graphs, 
>the op behaviour plots won't move at all if the 
>PW uses larger packets. These are bit-rate 
>plots, so the only way to move anything is to 
>change the TCP segment size (which moves the TCP 
>plots). To justify what you want to say, either:
>made about equal packet sizes into an 
>operational requirement, not just an analytical assumption.
>
>Cc: <mailto:[email-masked]>[email-masked]
><http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03b-02.html>http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-DIFF03b-02.html  
> >
>< 
><http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.xml>http://www.bobbriscoe.net/projects/2020comms/pwe3-congcons/draft-ietf-pwe3-congcons-03b.xml  
> >

---

From: "Black, David" <[email-masked]>
Date: Thu, 28 Aug 2014 00:38:52 -0400

Bob,

I'll yield to your views on point 8 to get the draft done.  I continue to believe that a recommendation for larger PW segment sizes is appropriate, but it's at best secondary to the primary purpose of this draft, which is to explain why TDM PWs do not pose congestion problems in practice (as you point out).

Yaakov - when do you think you'll be able to submit a revised draft?  I'm about to vanish for 3 weeks of vacation w/little to no email access.

Thanks,
--David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Tuesday, August 26, 2014 2:32 PM
To: Black, David
Cc: [email-masked]; Black, David
Subject: RE: [PWE3] FW: WGLC comments on: draft-ietf-pwe3-congcons-02.txt

David,

Bob,

...I'd like to hear from Yaakov before we start declaring consensus - it's been just two of us so far!

That's my point. A draft that isn't about mechanism shouldn't tell a developer or an operator what they can or can't do about mechanism. If we want to talk about that, we will need another draft (perhaps a PWE example in the circuit-breakers draft, which has an intended status of Standards Track).

I can live with this much text. But don't blame me if this causes the draft to get referred back or questioned as it goes through subsequent phases of the RFC process.

Or in other words:

Correct. We're getting somewhere. The draft actually says two sentences about this assumption:

You are focusing solely on the first sentence, which we only used to simplify presentation of the analysis.

I am saying any recommendations should be based on the analysis /after/ having adjusted the outcome to reverse out any assumptions that otherwise make the analysis unrealistic (ie. take account of the second quoted sentence).

a) Even if we solely make the assumption of comparable packet sizes (for the sake of analysis) and ignore the second qualifying sentence (the reality), then as we continue through the whole doc we still get to the point where we show the problem is virtually non-existent. Given:
  * TCP-friendliness is not an absolute requirement - one bit-rate within half-a-dozen times the other is fine (I recall Sally FLoyd writing somewhere that x8 shouldn't be a problem)
  * the overlap only occurs because we've drawn the upper bound of loss probability for a PWE conservatively high (Appendix A says this).

So we don't want to then move straight into recommending a solution in the following paragraph, which makes it look like we don't believe our own conclusion - that PWEs are not a significant capacity sharing problem - even when both PWE and TCP use the same size packets.

b) If we then take note of the second qualifying sentence above, it really does make this an unequivocal non-problem. So we really shouldn't be proposing that operators change anything.

c) Actually, if we did state your recommendation, it wouldn't be correct to say the analysis supports it, because packet size problems require consideration of packet-congestion, not just bit-congestion. So we would have to repeat the full analysis for packet-congestion, not bit-congestion, and show packet-rate graphs as well as bit-rate graphs. Having done this, and having similarly taken points (a) and (b) into account, we would still come to the same conclusion - that there is not a problem significant enough to need to recommend any action.

Bob

Thanks,
--David

David,

Sry for delay. inline...

Bob,

1-4 & 6: ok

Bob

Thanks,
--David

David,

Other possibilities are:

7. Circuit breaker

CURRENT:

8. Segment size.

Bob

Content-Language: en-US

-- PW segment size --

Thanks,
--David

Yaakov,

CURRENT:

Bob - ping ?

Y(J)S

Yaakov,

Thanks,
Andy

Cheers,
Andy

Cheers,
Andy

I suggest adding:

I suggest adding:

After:

I suggest adding:

After:

I suggest adding:

CBR Constant Bit Rate

OLD

Regards

Bob

After: