Subject: Re: [PWE3] the congestion framework violates the PWE architecture

From: [email-masked]
Date: Tue, 1 Jul 2008 11:04:47 -0400

Yaakov,

> Thanks for your comments (Stewart and I were beginning to feel rather lonely here ...)

... and I've been feeling rather busy here courtesy of end-of-quarter ;-).

> I think you summed up my concerns rather well, but I will sum 
> them up again anyway :
> 
> 1) we need to address the "PW over the public Internet" case, 
>     but the more "PWE3" case of a service provider network should not be forgotten

Ok, I look forward to your draft on congestion management for
service provider networks.

> 2) I am concerned that mechanism needed for the unusual first case and not relevant
> 	for the more usual second will be mandated for all PWs

First of all, I strongly disagree with "not relevant."  I was going to
say "backhoe," but Stewart beat to the punch with his citation of G.8113,
which suggests that ITU-T considers the "backhoe" threat relevant to
service providers.

Based on João's comment that out-of-band management is not always feasible
for a service provider, I think the appropriate form of "mandated" may be:
	- MUST implement
	- SHOULD use
	- a "service provider network" (to be defined in your draft) with
		sufficient (out-of-band) management provides valid reasons
		to ignore the "SHOULD use" requirement.
For the latter, your forthcoming draft on service provider networks ought
to provide the vehicle to discuss the "full implications [that] must be
understood and carefully weighed before choosing a different course."
NOTE: I'm quoting liberally from the RFC 2119 definition of SHOULD here.

The "MUST implement" requirement is relevant to service providers for
unexpected "backhoe" events and the like.

> 3) there is a concern that PWs in the first environment will not be
>	properly "weighted" the draft posits the concept of "equivalent TCP flows" 
> 	without making the argument clear enough. I think that this should be
> 	clarified more

I agree in principle.  We had a related discussion on this list last October
where effectively suggested that it may be possible to compute this implicitly
- baseline the PW traffic under normal circumstances, and apply TFRC in some
fashion to that baseline traffic rate:

http://www.ietf.org/mail-archive/web/pwe3/current/msg09208.html

This is a handwave - there are a bunch of details to work out.

> 4) for the second environment I do not accept that the kludge of "equivalent TCP flows"
> 	is the correct measure at all when there is not exact policing, we need to define
> 	a new concept of fairness based on percent degradation as compared to a "committed
> 	rate". This can be done in a centralized and in a distributed manner.
> 	Simulation work should be done to ensure that the distributed method works
> 	as anticipated.

At this level of description, it sounds plausible for the constrained
environment (cf. Dan's message) in which such mechanisms are feasible.
I'll wait for your draft before commenting further.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
pwe3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pwe3

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 1 Jul 2008 19:10:53 +0300

David, 

Thanks for reminding us of the previous discussion.

However, at that time we were mostly concerned with the inelastic case
(and I think that Sasha and I, after numerous iterations with Allison,
came up with some good text for RFC 4553, 5086 and 5087).
Now the focus is on elastic, but not necessarily TCP, traffic.

I stand corrected (and Stewart already chided me on the phone)
for my wording of "not relevant". 
However, I still don't buy your "MUST implement" wording. 
A signicant amount (if not the vast majority)
of PW equipment is purely for completely closed SP networks 
(e.g., cellular backhaul cores) and will never send anything 
towards the public Internet. Never. No backhoes, fronthoes or sidehoes.
I understand that it might not be possible for a PW gateway to know
to what it is sending its traffic, but the equipment operator, purchaser and
manufacturer DO know. I can already see service providers demanding
RFC5xxx congestion control on all PW gateways, and field technicians
incorrectly enabling it.

BTW, regarding whether out-of-band management is feasible,
there seem to be 2 types of operators. Those who consider it infeasible,
and those who consider anything else infeasible.

Y(J)S

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, July 01, 2008 6:05 PM
To: Yaakov Stein
Cc: [email-masked]
Subject: RE: [PWE3] the congestion framework violates the PWE architecture

Yaakov,

> I think you summed up my concerns rather well, but I will sum them up 
> again anyway :
> 

Ok, I look forward to your draft on congestion management for service provider networks.

> 	for the more usual second will be mandated for all PWs

First of all, I strongly disagree with "not relevant."  I was going to say "backhoe," but Stewart beat to the punch with his citation of G.8113, which suggests that ITU-T considers the "backhoe" threat relevant to service providers.

Based on João's comment that out-of-band management is not always feasible for a service provider, I think the appropriate form of "mandated" may be:
	- MUST implement
	- SHOULD use
	- a "service provider network" (to be defined in your draft) with
		sufficient (out-of-band) management provides valid reasons
		to ignore the "SHOULD use" requirement.
For the latter, your forthcoming draft on service provider networks ought to provide the vehicle to discuss the "full implications [that] must be understood and carefully weighed before choosing a different course."
NOTE: I'm quoting liberally from the RFC 2119 definition of SHOULD here.

The "MUST implement" requirement is relevant to service providers for unexpected "backhoe" events and the like.

> 	without making the argument clear enough. I think that this should be

I agree in principle.  We had a related discussion on this list last October where effectively suggested that it may be possible to compute this implicitly
- baseline the PW traffic under normal circumstances, and apply TFRC in some fashion to that baseline traffic rate:

> 	Simulation work should be done to ensure that the distributed method works

At this level of description, it sounds plausible for the constrained environment (cf. Dan's message) in which such mechanisms are feasible.
I'll wait for your draft before commenting further.

Thanks,
--David

---

From: [email-masked]
Date: Tue, 1 Jul 2008 17:35:09 -0400

Yaakov,

> Now the focus is on elastic, but not necessarily TCP, traffic.

I believe the frameworks developed here ought to cover both types of
traffic.

> However, I still don't buy your "MUST implement" wording.

I also work on security - I have a thick hide (and a lot of
patience) from the corresponding "MUST implement" debates
around security measures.  At the moment, this is striking
me as analogous.

> A significant amount (if not the vast majority)

In my experience, "Never" tends to be a rather difficult position
to defend against mother nature and human error:
- Two words: "lightning bolt" (all-electric version of "backhoe").
- Two more words: "ship anchor" (cutting a transoceanic cable).
I look forward to the explanation in your draft about why
service provider networks are immune to backhoes ;-).

Seriously, are you asserting that in completely closed SP networks,
there is never an opportunity for a PW to take action on its own
to cope with an unexpected network failure event?  IMHO, that's
getting close to asserting that if there's a failure that causes
some CR not to be sustainable then it's ok to have arbitrary
damage to other network traffic occur as a result.

[... remainder snipped ...]

Thanks,
--David

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 2 Jul 2008 09:43:01 +0300

> I also work on security - I have a thick hide (and a lot of patience)
from the corresponding "MUST implement" debates around security
measures.  
> At the moment, this is striking me as analogous.

I too work on security. Too many times have I seen people ask for
mechanisms defined in various RFCs
without thinking whether they match the deployment scenario.
They end up spending tons of money and slowing everything down by
encryption,
while leaving numerous backdoors that can easily be exploited.
That is what happens when you put in too many MUSTs.

> In my experience, "Never" tends to be a rather difficult position to
defend against mother nature and human error:
> I look forward to the explanation in your draft about why service
provider networks are immune to backhoes ;-).

I look forward to your explanation as to when a lightning bolt would
cause a cellular backhaul network 
to route its customer traffic over the public Internet or to send its
calls overseas via underwater cables.
These scenarios simply do not make sense for closed SP networks.

Let's do a reductio ad absurdum. The PSTN is a network. Its
infrastructure can be effected by
lightning bolts, operator error, etc. Do you really believe that when
this happens there is a scenario
whereby magically VoIP gateways appear and instantaneously mystical
fibers attach them to the
public Internet, and all the traffic in a major city flows over the
Internet ?

Why do you believe that this could happen JUST because the traffic is
packetized to begin with ?
If there is no connectivity it is not going to happen.

That is what I meant by a closed SP network.

> there is never an opportunity for a PW to take action on its own to
cope with an unexpected network failure event? 

I am asserting that well engineered SP networks have built-in protection
mechanisms.
Up to now these mechanisms, based on OAM indications of faults, cut in
reasonably fast,
and avoided intolerable service unavailability. In order to convince
service providers to transition
to modern infrastructures we have to put in place similar mechanisms.

In less demanding environments (no SLA, low priced SLA) there may indeed
be
cases where default behavior of an IP network, or falling back to
Internet connectivity
may make sense. But this does not eliminate the first case.

> IMHO, that's getting close to asserting that if there's a failure that
causes some CR not to be sustainable 
> then it's ok to have arbitrary damage to other network traffic occur
as a result.

You are assuming "other network traffic" unknown to the network
operator.
This is not the proper asssumption for well-engineered service provider
networks.
The network operator known exactly what traffic is going through its
network,
and knows how to prioritize based on SLAs.
When a major fault occurs, traffic will be impacted based on the
economic damage caused
to the operator. Certainly "fair" when we take cost into account.

Y(J)S

---

From: [email-masked]
Date: Wed, 2 Jul 2008 12:05:10 -0400

Yaakov,

I think we may be talking past each other.  

> I look forward to your explanation as to when a lightning bolt would
cause a cellular
> backhaul network  to route its customer traffic over the public
Internet or to send
> its calls overseas via underwater cables. These scenarios simply do
not make sense
> for closed SP networks.

The damage done by loss of connectivity may be within the SP's own
network.
A lightning bolt can easily damage a cellular backhaul network (cell
towers
are obvious lightning targets, and the lightning protection measures
aren't
always perfect).

I think you're asserting that for the class of SP network that you're
interested in, the reaction to this sort of loss of connectivity is
always
via OAM mechanisms, making an individual PW reaction superfluous.  I
remain
to be convinced, and will want to see how much of the space of
applicability
of pseudo-wires is covered by this class of closed SP network that I
expect
your draft to define.

[ ... snip ...]

> mechanisms.  Up to now these mechanisms, based on OAM indications of
faults,
> cut in reasonably fast, and avoided intolerable service
unavailability.
> In order to convince service providers to transition to modern
> infrastructures we have to put in place similar mechanisms.
> 
> In less demanding environments (no SLA, low priced SLA) there may
> indeed be cases where default behavior of an IP network, or falling
> back to Internet connectivity may make sense. But this does not
> eliminate the first case.

I think I can agree with this.  The specifications have to cover all the
cases.

operator.
> This is not the proper assumption for well-engineered service provider
networks.
network,
> and knows how to prioritize based on SLAs.

I want to see the details, starting with the complete set of baseline
conditions that characterize what you consider a "service provider
network" to be.  An important consideration will be how much of the
service provider space satisfies that set of conditions.

> When a major fault occurs, traffic will be impacted based on the
economic
> damage caused to the operator. Certainly "fair" when we take cost into
account.

I think that's reasonable at a high level, the hard part is reflecting
potential "economic damage" appropriately in protocol mechanisms.

Thanks,
--David

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 02 Jul 2008 17:32:40 +0100

> potential "economic damage" appropriately in protocol mechanisms.
David

An approximation is amount of damage to packet flow the PE is configured to
withstand in the hope of a return to normal service before it gives up and
takes the PW down.

Whist we are considering SP networks, to what extent are you worried
about SPs running without congestion control, vs PEs running
without congestion control in other environments. i.e. to what extent
are you concerned about enforcement?

Stewart

---

From: [email-masked]
Date: Wed, 2 Jul 2008 13:41:28 -0400

Stewart,

economic
> >> damage caused to the operator. Certainly "fair" when we take cost
into account.
> withstand in the hope of a return to normal service before it gives up
and

I think that's a reflection of elasticity rather than economic value.
I would expect economic value to manifest in things like priority and
weighting, especially among otherwise-similar flows.  For this situation
I would expect economic value to be used to apportion the damage that
should be applied to each PW, and then each PE decides whether to
continue
in the face of its apportioned level of damage.  Of course things aren't
that simple, because knowing the damage threshold at which the PW will
shut down is an important factor in apportioning the damage.

> are you concerned about enforcement?

I'm not sure I completely understand the question.

For SP networks, I believe that there are situations where per-PW
congestion control is neither needed nor wanted courtesy of the OAM
framework handling things that go wrong.  OTOH, Yaakov has yet to
convince me that no SP in their right mind would ever even think
about turning on a PW congestion control mechanism because the OAM
framework will always take care of any problems (yes, I'm probably
exaggerating his position, but not by a lot, IMHO).  It's also good
to see Dan Romascanu (OPS [O&M] AD) taking an interest in this topic.

My current intelligent guess is that Yaakov's assertions about SP
networks don't apply to all SPs.  I'm expecting the draft that Yaakov
(I hope) will write to delineate the boundary between the situations
in which it's ok to rely on OAM mechanisms vs. situations in which
the PE should or must make its own decisions about reaction to
congestion.  For the latter class of situations, I'd want to see
fairly strong requirements language for use of congestion control.

Overall, I still think "MUST implement" is appropriate for congestion
control, but I want to see Yaakov's draft to better understand the
details of the SP networks he's concerned about.

Thanks,
--David