Subject: Re: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

From: Stewart Bryant <[email-masked]>
Date: Tue, 01 Jul 2008 11:22:03 +0100

Maarten Vissers wrote:
> Yaakov,
>
>   
>> We should be able to enable the PWs to be guaranteed their CR, and to
>>     
> enjoy
>   
>> excess traffic when BW is available. 
>>     
>
> I fully agree with you.
>
>   
>> When congestion occurs, we should be able to tell the ingress PEs that
>>     
> this
>   
>> is the case, and that they should police to their CR. 
>>     
>
> I very much disagree with you here.
>
> Any PW should have at its UNI-N a Traffic Conditioning function that
> enforces the CIR/EIR of the service. Furthermore, connection admission
> control and traffic engineering should guarantee that links inside the
> transport network do not have unexpected congestion conditions.
>
> When congestion occurs this is per network design; the excess rate traffic
> (marked as discard eligible) will be discarded. If the network design
> includes committed rate based oversubscription of links, then also committed
> rate traffic may experience congestion and some of those packets will be
> discarded as well. No need to do anything special in the PWs. In a well
> engineered network such conditions will be very rare. 
>
> Furthermore, netowrk management will monitor the actual load on the links
> and will receive indications when the load is increasing and/or crosses a
> threshold. Then either the link capacity will be increased or some of the
> PWs are rerouted over another link.
>
>   
Maarten

I agree that in a well designed network congestion will only occur very 
rarely, but
we do need to cope with those cases when severe congestion does occur. 
Indeed
G.8113 recognizes the need to include detection for such events.

In most IP networks the path is always shortest path and reroute to a 
lightly
loaded path is not an option. In those circumstances we need the PE to
take the load off the net by its own independent action, if necessary by
shutting down the PW.

Including this mechanism in an equipment deployed on a well designed
network does no hard, because on such a network it will only get invoked
when the network departs from the planed operational state.

Stewart

_______________________________________________
pwe3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pwe3

---

From: Maarten Vissers <[email-masked]>
Date: Tue, 01 Jul 2008 12:35:54 +0200

Stewart,

Congestion management is typically very complex to do it good.
ATM has its congestion management; I am told that it is complex and not well
functioning.
IEEE 802.1 is developing a congestion management extension, but this was
only supported when its use is strictly limited to a data center.

MS-PWs will provide world wide connectivity.
H-VPLS will also provide world wide connectivity.

Both MS-PW and H-VPLS instances will be carried over MPLS Tunnels.
Both MS-PW and H-VPLS needs thus to support congestion management.
H-VPLS isntances are VLAN instances, which use a MPLS Label Stack Entry
header instead of a VLAN Tag; i.e. a kind of a MPLS labelled VLAN (M-VLAN).

If you want to introduce congestion management, you have to do this also in
those M-VLAN (H-VPLS) instances. It is known that 802.1 does not support
congestion management beyond the data center. So any addition of congestion
management to MS-PWs does not solve the problem in the IP network.

It is better to solve this issue by not deploying MS-PW/H-VPLS instances in
such IP netowrk.

Regards,
Maarten

-----Original Message-----
From: Stewart Bryant [mailto:[email-masked]] 
Sent: 01 July 2008 12:22
To: Maarten Vissers
Cc: 'Yaakov Stein'; 'Stewart Bryant'; 'Danny McPherson'; [email-masked]
Subject: Re: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

Maarten Vissers wrote:
> When congestion occurs this is per network design; the excess rate 
> traffic (marked as discard eligible) will be discarded. If the network 
> design includes committed rate based oversubscription of links, then 
> also committed rate traffic may experience congestion and some of 
> those packets will be discarded as well. No need to do anything 
> special in the PWs. In a well engineered network such conditions will be
very rare.
>
> Furthermore, netowrk management will monitor the actual load on the 
> links and will receive indications when the load is increasing and/or 
> crosses a threshold. Then either the link capacity will be increased 
> or some of the PWs are rerouted over another link.
>
>   
Maarten

I agree that in a well designed network congestion will only occur very
rarely, but we do need to cope with those cases when severe congestion does
occur. 
Indeed
G.8113 recognizes the need to include detection for such events.

In most IP networks the path is always shortest path and reroute to a
lightly loaded path is not an option. In those circumstances we need the PE
to take the load off the net by its own independent action, if necessary by
shutting down the PW.

Including this mechanism in an equipment deployed on a well designed network
does no hard, because on such a network it will only get invoked when the
network departs from the planed operational state.

Stewart

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 1 Jul 2008 13:21:57 +0200

Making a Network Management application that monitors link load and
reprovisions the PWs part of the congestion management framework is a
very problematic design. There are many reasons why such a solution
should be discouraged, to start with the fact that the congestion can
impact the management traffic itself. 

My 2 agorot. 

Dan

> From: [email-masked] [mailto:[email-masked]] On 
> 
> Furthermore, netowrk management will monitor the actual load 
> on the links and will receive indications when the load is 
> increasing and/or crosses a threshold. Then either the link 
> capacity will be increased or some of the PWs are rerouted 
> over another link.
> 
> Regards,
> Maarten
>

---

From: Nadeau Thomas <[email-masked]>
Date: Tue, 1 Jul 2008 09:24:11 -0400

While this is true for in-band management, most SPs use an out-of- 
band management network, so this would in fact, not be a problem from  
that perspective.

	--Tom

---

From: <[email-masked]>
Date: Tue, 1 Jul 2008 14:36:12 +0100

My 2 cents,
My experience from cellular networks is that in many sites the out-band management is not feasible.
Many sites are remote and the idea is to use the same infrastructure to access the site.
I imagine that many other type of applications can be put in the same bag as this one.

João J. Horta,
SonaeCom
Fixed and Mobile Provider

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nadeau Thomas
Sent: 2008-07-01 14:24
To: Romascanu, Dan (Dan)
Cc: Stewart Bryant; [email-masked]; Yaakov Stein; Danny McPherson
Subject: Re: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

        --Tom

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 1 Jul 2008 15:48:33 +0200

So we are talking about constraint environments only? Out-of-band
management systems do not scale beyond limited domains. 

Dan

> From: Nadeau Thomas [mailto:[email-masked]] 
> Sent: Tuesday, July 01, 2008 4:24 PM
> To: Romascanu, Dan (Dan)
> Cc: Maarten Vissers; Yaakov Stein; Stewart Bryant; Danny 
> McPherson; [email-masked]
> 	While this is true for in-band management, most SPs use 
> an out-of- band management network, so this would in fact, 
> not be a problem from that perspective.
> 
> 	--Tom
> 
>

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 1 Jul 2008 09:55:42 -0400

So my point is that the point you made applies to a subset of the  
scenarios we want to handle rather than discounting all of them out-of- 
hand.

	--Tom

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 1 Jul 2008 18:55:07 +0300

Maarten

I think you are misunderstanding the PWe environment
when you talk of  a UNI-N Traffic Conditioning function enforcing
CIR/EIR. 

The current state of the standards specifically does NOT have CIR/EIR
parameters for PWs.
PWs are set up using (non-CR) LDP, and do not communicate any such
information.

Of course you could configure such things using a management interface,
but that would be a non-standards-based solution,
and is beyond our scope at this point.

Understanding the constraints, what I believe is the best solution
is to define new PWE control protocol interface parameters
as I mentioned in a previous email. Since these will not undergo
the full treatment of checking for resource availability, we can instead
do ball-park estimates and define a concept of fairness of the type
I mentioned. By using a variant of AIMD we can get a steady-state
approximation to the appropriate behavior.

An alternative approach is to reinstate CR-LDP, defining CIR/EIR
parameters
and proper CACing for the SP case. Do you intend writing an ID proposing
this ?

Y(J)S

-----Original Message-----
From: Maarten Vissers [mailto:[email-masked]] 
Sent: Tuesday, July 01, 2008 10:01 AM
To: Yaakov Stein; 'Stewart Bryant'; 'Danny McPherson'
Cc: [email-masked]
Subject: RE: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

Yaakov,

enjoy

I fully agree with you.

this

When congestion occurs this is per network design; the excess rate
traffic (marked as discard eligible) will be discarded. If the network
design includes committed rate based oversubscription of links, then
also committed rate traffic may experience congestion and some of those
packets will be discarded as well. No need to do anything special in the
PWs. In a well engineered network such conditions will be very rare. 

Furthermore, netowrk management will monitor the actual load on the
links and will receive indications when the load is increasing and/or
crosses a threshold. Then either the link capacity will be increased or
some of the PWs are rerouted over another link.

Regards,
Maarten

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Yaakov Stein
Sent: 29 June 2008 11:04
To: Stewart Bryant; Danny McPherson
Cc: [email-masked]
Subject: Re: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

> Folks,
> Please consider today the start of a two-week WG LC for this document.

Stewart et al.

I apologize for just getting around to reading this document.

I have so many issues with it, that I don't know where to begin.

So I guess I will start with the scope. 
The draft claims to address congestion control for PWs. 
Actually it addresses TCP friendliness for PWs over the Internet (or an
Internet-like environment).
This preassumes that the most interesting topic to be solved is how to
keep PWs from crashing the Internet.
I have no doubt that this is an important issue, since if we produce an
encapsulation, then someone may one day use it on the public Internet.
However, there is still a major jump between this legitimate concern and
calling the treating of this pathological case "congestion control for
PWs".

Setting aside the aforementioned unusual cases, the focus of PWE3 is,
and has always been, the service provider environment. No SP I have ever
seen would place individual user TCP flows inside an MPLS tunnel along
with PWs.
And if one did, then they would certainly have no problem with such an
individual user flow suffering from the PW traffic - and would certainly
not reduce the PW service level in order to protect such an individual
TCP flow.
(Note - I am not including
system TCP flows, such as LDP or BGP here. These would assumably be
appropriately prioritized).

So, I would have much fewer problems with calling this ID "congestion
control for PWs over the Internet", but definitely object to calling it
"congestion control for PWs".

So, what IS congestion control for PWs ?
Well, due to the unfortunate decision detailed in RFC 3468 and the
interpretation that this limits PWE3 from extending its control protocol
to include BW reservations, we have to accept the situation that PWs are
not properly BW limited.
So, we can't talk about a PW's committed rate (CR), and thus even if we
deny admission when we believe we don't have sufficient BW, we can't
properly enforce CR and excess rate schedules.

What should a congestion framework for a service provider network
specify ?
We should be able to enable the PWs to be guaranteed their CR, and to
enjoy excess traffic when BW is available. When congestion occurs, we
should be able to tell the ingress PEs that this is the case, and that
they should police to their CR. If congestion continues, we should
exercise fairness between the PWs - BUT this does NOT mean that all PWs
should get the same BW, rather that the percentage reduction from CR
should be the same !

There are many ways that this can be done. Of course, SP networks
usually have central management, and the management may directly
instruct ingress PEs as to their constraints.
However, we can also develop noncentralized mechanisms. In order to
obtain the kind of fairness required, we still us AIMD, but instead of
reducing the flow to half the PRESENT rate, we need to multiply it by a
percentage of the CR (or whatever we desire to call it here).

Second topic.
The tutorial material on TCP congestion control and TFRP are useful, but
I believe that they miss the main point for PWs.
TCP flows end up approximately equally sharing the BW (rather than the
fairness I defined above) because of the basic assumptions of the model,
not solely because of AIMD. 
Each TCP flow is assumed to be atomic,
all flows are assume equally important (i.e. to have the same CR), and
the TCP flows are assumed to emanate from an end-user box where the
back-pressure can be used to hold up the application.
It is this set of assumptions along with the AIMD mechanisn that leads
to the TCP rate equation.
Were an Ethernet PW carrying TCP/IP to be in parallel to TCP flows, then
loss of a single PW packet would translate to loss of a TCP packet of a
single socket pair, and thus lead to a much smaller reduction in PW
capacity (which I assume is what the authors mean by "equivalent number
of TCP flows").
Similarly, were we to weight the flow importance, we could get different
flow rates.

On the other hand PWs emanate from PEs that can not usually perform
back-pressure and need to police by dropping packets. Furthermore, as
stated above, fairness does not mean that we allot the same BW to the
PWs - instead we need to allot CRs and police to these. 
This is more direct and easier to understand and configure than the
nebulous concept of "equivalent number of TCP flows".

Now for some shorter issues.

There is a statement about MOST (99%?) PWs carrying TCP/IP traffic.
This certainly needs qualification. I have been involved in many network
deployments where NONE of the PWs carry TCP/IP. 

There is a statement about the possibility of sending PWs over IPsec
tunnels.
It is unclear to me what this means. IPsec is most commonly used for
TCP/IP traffic, and thus if you use IPsec then you probably already have
the behavior you need.
On the other hand, there are PW encapsulations that use UDP/IP which
could potentially be put in IPsec tunnels. But these are the TDM
encapsulations, that can not fall back at all.
Finally, perhaps the ID means some new encapsulation of PW over RFC4023
over IPsec.
If so, it would be nice to have this stated, and to explain how this
works.

The drawbacks the draft mentions regarding using the sequence numbers to
detect packet drop seem artificial. Yes, misordering can masquerade as
packet drop if the delayed packet is very late. But no method is
perfect.
BTW, for TDM PWs packet arrival times can also be used as a very
senstive mechanism to detect imminent congestion.

The draft needs a serious edit. There are numerous typos (newtork?) and
in several places whole words seems to have dropped (due to congestion,
no doubt).

Y(J)S

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 2 Jul 2008 10:48:21 +0200

Maarten,

We are probably hearing and reading different reports and we may want to
change notes one day :-)  However, the issue here is not in-band vs.
out-of-band management. My problem is not with the out-of-band
management systems, I do understand the logic of their architecture in
transport networks. My problem is with introducing management
applications as a mandatory component of the standard congestion
management framework and recommending that the retrieval of link
utilization information and reprovisioning of the routers by management
applications as a congestion management method at Internet scale.  This
concern stands whatever transport (in-band or out-of-band) is used
between the management applications and the network devices. 

Regards,

Dan

> Sent: Wednesday, July 02, 2008 9:06 AM
> To: Romascanu, Dan (Dan); 'Nadeau Thomas'
> Dan,
> > Out-of-band management systems do not scale beyond limited domains.
> 
> There is a world wide transport network that is managed by 
> using an out-of-band management network.
> 
> I hear reports from operators that packet transport networks 
> with in-band management do not scale beyond some limited 
> domains. That's one of the reasons to operate packet 
> transport networks (including MPLS based ones) also with 
> out-of-band management networks.
> 
> Regards,
> Maarten
> 
> From: Romascanu, Dan (Dan) [mailto:[email-masked]]
> Sent: 01 July 2008 15:49
> To: Nadeau Thomas
> So we are talking about constraint environments only? 
> Out-of-band management systems do not scale beyond limited domains. 
> 
> Dan
>   
> 
> > Cc: Maarten Vissers; Yaakov Stein; Stewart Bryant; Danny McPherson; 
> > [email-masked]
> an out-of-

---

From: Maarten Vissers <[email-masked]>
Date: Wed, 02 Jul 2008 08:43:55 +0200

Yaakov,

Thanks for clarifying the environment of PWs. I was not aware of this, while
I use the MEF and ITU-T specifications with their Traffic Conditioning
function in the UNI-N as basis. As I wrote in another email a couple of
minutes ago, many packet transport networks have Ethernet deployed in the
access and metro edge. The PWs are in those cases only used between e.g.
metro core to far end metro core nodes and have indeed no traffic
conditioning function involved (PWs are not UNI-N to UNI-N). This implies
that for congestion notification that congestion notification messages must
be continued in the Ethernet VLAN space to reach the UNI-N ports and the
traffic conditioning functions on those ports.

This combination of Ethernet at the edge controlled by network management
and MPLS (PW/VPLS) in the middle seems a standards (i.e. IEEE 802.1, MEF,
ITU-T and IETF) based solution. A weak part in this standards based solution
is indeed the use of LDP when you have MS-PW. For a SS-PW LDP is fine, as
the PW is represents a client encapsulation plus demultiplex identifier. I
would expect that CR-LDP (or GMPLS) is required for MS-PWs in packet
transport networks.

Regards,
Maarten

-----Original Message-----
From: Yaakov Stein [mailto:[email-masked]] 
Sent: 01 July 2008 17:55
To: Maarten Vissers; Stewart Bryant; Danny McPherson
Cc: [email-masked]
Subject: RE: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

Maarten

I think you are misunderstanding the PWe environment when you talk of  a
UNI-N Traffic Conditioning function enforcing CIR/EIR. 

Of course you could configure such things using a management interface, but
that would be a non-standards-based solution, and is beyond our scope at
this point.

Understanding the constraints, what I believe is the best solution is to
define new PWE control protocol interface parameters as I mentioned in a
previous email. Since these will not undergo the full treatment of checking
for resource availability, we can instead do ball-park estimates and define
a concept of fairness of the type I mentioned. By using a variant of AIMD we
can get a steady-state approximation to the appropriate behavior.

An alternative approach is to reinstate CR-LDP, defining CIR/EIR parameters
and proper CACing for the SP case. Do you intend writing an ID proposing
this ?

Y(J)S

-----Original Message-----
From: Maarten Vissers [mailto:[email-masked]]
Sent: Tuesday, July 01, 2008 10:01 AM
To: Yaakov Stein; 'Stewart Bryant'; 'Danny McPherson'
Cc: [email-masked]
Subject: RE: [PWE3] WG LC:draft-ietf-pwe3-congestion-frmwk-01.txt

Yaakov,

enjoy

I fully agree with you.

this

Regards,
Maarten

-----Original Message-----

Stewart et al.

So I guess I will start with the scope. 
The draft claims to address congestion control for PWs. 
Actually it addresses TCP friendliness for PWs over the Internet (or an
Internet-like environment).
This preassumes that the most interesting topic to be solved is how to keep
PWs from crashing the Internet.
I have no doubt that this is an important issue, since if we produce an
encapsulation, then someone may one day use it on the public Internet.
However, there is still a major jump between this legitimate concern and
calling the treating of this pathological case "congestion control for PWs".

Setting aside the aforementioned unusual cases, the focus of PWE3 is, and
has always been, the service provider environment. No SP I have ever seen
would place individual user TCP flows inside an MPLS tunnel along with PWs.
And if one did, then they would certainly have no problem with such an
individual user flow suffering from the PW traffic - and would certainly not
reduce the PW service level in order to protect such an individual TCP flow.
(Note - I am not including
system TCP flows, such as LDP or BGP here. These would assumably be
appropriately prioritized).

So, what IS congestion control for PWs ?
Well, due to the unfortunate decision detailed in RFC 3468 and the
interpretation that this limits PWE3 from extending its control protocol to
include BW reservations, we have to accept the situation that PWs are not
properly BW limited.
So, we can't talk about a PW's committed rate (CR), and thus even if we deny
admission when we believe we don't have sufficient BW, we can't properly
enforce CR and excess rate schedules.

What should a congestion framework for a service provider network specify ?
We should be able to enable the PWs to be guaranteed their CR, and to enjoy
excess traffic when BW is available. When congestion occurs, we should be
able to tell the ingress PEs that this is the case, and that they should
police to their CR. If congestion continues, we should exercise fairness
between the PWs - BUT this does NOT mean that all PWs should get the same
BW, rather that the percentage reduction from CR should be the same !

There are many ways that this can be done. Of course, SP networks usually
have central management, and the management may directly instruct ingress
PEs as to their constraints.
However, we can also develop noncentralized mechanisms. In order to obtain
the kind of fairness required, we still us AIMD, but instead of reducing the
flow to half the PRESENT rate, we need to multiply it by a percentage of the
CR (or whatever we desire to call it here).

Second topic.
The tutorial material on TCP congestion control and TFRP are useful, but I
believe that they miss the main point for PWs.
TCP flows end up approximately equally sharing the BW (rather than the
fairness I defined above) because of the basic assumptions of the model, not
solely because of AIMD. 
Each TCP flow is assumed to be atomic,
all flows are assume equally important (i.e. to have the same CR), and the
TCP flows are assumed to emanate from an end-user box where the
back-pressure can be used to hold up the application.
It is this set of assumptions along with the AIMD mechanisn that leads to
the TCP rate equation.
Were an Ethernet PW carrying TCP/IP to be in parallel to TCP flows, then
loss of a single PW packet would translate to loss of a TCP packet of a
single socket pair, and thus lead to a much smaller reduction in PW capacity
(which I assume is what the authors mean by "equivalent number of TCP
flows").
Similarly, were we to weight the flow importance, we could get different
flow rates.

On the other hand PWs emanate from PEs that can not usually perform
back-pressure and need to police by dropping packets. Furthermore, as stated
above, fairness does not mean that we allot the same BW to the PWs - instead
we need to allot CRs and police to these. 
This is more direct and easier to understand and configure than the nebulous
concept of "equivalent number of TCP flows".

There is a statement about the possibility of sending PWs over IPsec
tunnels.
It is unclear to me what this means. IPsec is most commonly used for TCP/IP
traffic, and thus if you use IPsec then you probably already have the
behavior you need.
On the other hand, there are PW encapsulations that use UDP/IP which could
potentially be put in IPsec tunnels. But these are the TDM encapsulations,
that can not fall back at all.
Finally, perhaps the ID means some new encapsulation of PW over RFC4023 over
IPsec.
If so, it would be nice to have this stated, and to explain how this works.

The drawbacks the draft mentions regarding using the sequence numbers to
detect packet drop seem artificial. Yes, misordering can masquerade as
packet drop if the delayed packet is very late. But no method is perfect.
BTW, for TDM PWs packet arrival times can also be used as a very senstive
mechanism to detect imminent congestion.

The draft needs a serious edit. There are numerous typos (newtork?) and in
several places whole words seems to have dropped (due to congestion, no
doubt).

Y(J)S