Subject: [PWE3] the congestion framework violates the PWE architecture

From: "Yaakov Stein" <[email-masked]>
Date: Mon, 30 Jun 2008 07:12:05 +0300

Stewart and co-authors
 
My previous email was rather long, and I didn't want to make it
intolerably so
by including a specific argument as to why the suggested framework
is a bad idea, even in the scope I delineated yesterday.
 
Let us start with the premise that a PW is supposed to provide a service

equivalent to the native service, to within limitations stated in the
relevant applicability statement. 
The following argument suggests that the congestion framework
contradicts this goal, 
or at least requires significant modification of the present
applicability statements.
 
Let's assume an Ethernet PW that has been rated as 10 TCP equivalent
flows.
 
Now the Ethernet PW traffic is MACsec encrypted, and so the PE has no
idea
what the Ethernet is carrying, but in fact it carries 10 TCP flows.
Thus, we have the happy case where we have correctly rated the PW.
 
Now, due to some congestion in the network a PW packet is dropped.
The relevant underlying TCP/IP flow feels this and cuts back to half its
original rate.
In addition, according to the proposed mechanism, the rate of the
10-equivalent flow PW
needs to be reduced by half of 10 percent, i.e. by 5%. This translates
to dropping
every twentieth packet.
 
So 20 packets from now the PE drops a PW packet. With 90 percent
probability
this packet belongs to a different TCP/IP flow than originally effected
by the packet loss,
and an additional TCP/IP flow cuts back by half. Note that we now have
two flows at half rate
(10 percent rate drop) and an additional 5% PW restriction, for a total
of 15% rate loss.
Of course a true single TCP/IP flow would have dropped by 50%, so this
still seems reasonable,
but it is 3 times what was intended by rating the PW as 10 flows.
 
Now, 20 more packets are emanated, and an additional packet is dropped.
If this packet belongs to a third flow, then yet another innocent
bystander is throttled.
If it belongs to one of the two already effected, then we cut this flow
down to 25%
of its original rate ! This would not have happened for a simple TCP/IP
flow.
 
We reach two conclusions.
1) If there IS TCP/IP inside the PW being controlled, then the rate
cutback is much higher
than that intended by the equivalent flow rating.
2) A TCP/IP flow inside a PW receives significantly poorer treatment
than TCP/IP in the open
on the same network.
Thus, for a given amount of congestion, the service given to a TCP/IP
flow is significantly
impaired as compared to the native service, much more so than we would
be led to believe
from the Ethernet PW applicability statement.

Y(J)S
 
_______________________________________________
pwe3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pwe3

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 30 Jun 2008 11:03:22 +0100

Yaakov Stein wrote:
> The following argument suggests that the congestion 
> framework contradicts this goal,
> Now the Ethernet PW traffic is MACsec encrypted, and so the PE has no idea
> The relevant underlying TCP/IP flow feels this and cuts back to half 
> its original rate.
> So 20 packets from now the PE drops a PW packet.
Doesn't this depend on how quickly the PW rate change cuts in relative
to the round trip time. If the PW mechanism is slower than the TCP
mechanism  the traffic will have already dropped to the  right level.
> With 90 percent probability
> this packet belongs to a different TCP/IP flow than originally 
> effected by the packet loss,
> (10 percent rate drop) and an additional 5% PW restriction, for a 
> total of 15% rate loss.
> If it belongs to one of the two already effected, then we cut this 
> flow down to 25%
> of its original rate ! This would not have happened for a simple 
> TCP/IP flow.
>  

> from the Ethernet PW applicability statement.
Can you propose a mechanism that has better properties?

Stewart

---

From: "Yaakov Stein" <[email-masked]>
Date: Mon, 30 Jun 2008 13:19:10 +0300

Stewart

> Doesn't this depend on how quickly the PW rate change cuts in relative
to the round trip time. 
> If the PW mechanism is slower than the TCP mechanism  the traffic will
have already dropped to the  right level.

I was assuming a PW which aggregates 10 users, and thus we can assume
that its
interpacket interval is low as compared to RTT.

> Can you propose a mechanism that has better properties?

I can (and perhaps will) write a draft on what to do in the SP scenario,
which your draft does not cover.

I don't see any real way of completely solving the (IMO not very
important) problem
of a PW over the public Internet. As soon as you perform any PE rate
cutback
in addition to that perfomed by the network, you will more detrimentally
impact TCP/IP.

You could purposely overestimate the number of equivalent flows,
but if the real number is not very large, you still have a nonzero
probability of dropping multiple packets from a single flow,
thus causing it to undergo severe rate drop.

Were it possible to identify individual internal flows, then you could
do selective drop. 
But if the PW contains TCP flows, then why bother ?

So we are left with:
1) if the PW is carrying ONLY TCP traffic
         then don't do anything at the PW level (everyting will work out
correctly)
2) if the PW is carrying NO TCP traffic, then
   2a) if it is carrying constant rate traffic and the path has not
changed, 
            then it is not greedy so leave it alone
   2b) if it is carrying constant rate traffic 
        and you have just dumped it on a new path which is congested
            then pull the PW plug
   2c) if it is carrying variable rate real-time traffic
            then perform selective packet drop taking perceptual effect
into account
   2d) if it is carrying non-real-time traffic over some non-TCP
transport protocol
            then learn how that protocol works and do something
specifically designed for it
3) if the PW is carrying a mixture of real-time and TCP traffic, FFS.

Y(J)S

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 30 Jun 2008 12:11:46 +0100

> interpacket interval is low as compared to RTT.
Exactly, so a fast TCP and a slow PW mechanism should have the right effect.
> which your draft does not cover.
The draft does say an SP can turn this mechanism off, but they need to be
careful.
> of a PW over the public Internet.
The transport area are reviewing this draft. If they confirm that this 
draft is
not needed, we will withdraw it. However the whole motivation
for this draft is to address their concerns.
>  As soon as you perform any PE rate
> cutback

> thus causing it to undergo severe rate drop.

Doesn't this happen in TCP anyway? There is no flow identification in 
the routers
to distribute the drops over multiple flows. The whole basis of TCP is a 
simple
mechanism that may be short term unfair, but is statistically fair in 
the long term.

> But if the PW contains TCP flows, then why bother ?
Because as we keep saying, you don't know if it contains TCP or 
something else.
>          then don't do anything at the PW level (everyting will work out
But how do you know that those ATM cells are carrying TCP only?

> 2) if the PW is carrying NO TCP traffic, then
But how do you know that?
> 3) if the PW is carrying a mixture of real-time and TCP traffic, FFS.
The draft does not preclude such action in response to a congestion 
notification from
the PW. All the draft does is to discuss the issues and set up the 
context for the
solutions drafts.

Stewart

---

From: "Yaakov Stein" <[email-masked]>
Date: Mon, 30 Jun 2008 14:38:33 +0300

Stewart

I will only respond to a few of your comments.

> Exactly, so a fast TCP and a slow PW mechanism should have the right
effect.

No, a slow PW mechanism will keep incorrectly dropping TCP packets
for a long time, thus adversely effecting many TCP flows.
If the PW recovered quickly, then the adverse effect would be minimized,
but the PW would grab more than its share of BW in the steady state.

>> I can (and perhaps will) write a draft on what to do in the SP 
>> scenario, which your draft does not cover.
>   
>The draft does say an SP can turn this mechanism off, but they need to
be careful.

What I meant was that we should define a mechanism for the service
provider environment.
Were we to have committed rates, then it would be easy. In the absence
of true commitments,
what we can have is each PW being set up with an expected maximum rate
(EMR), and an elasticity factor (EF).
TDM PWs have essentially zero elasticity, while Ethernet ones could be
given say 50%.
The PE admits PWs as long as it has room in the tunnel for the EMR.
When congestion hits, the rate is policed based on fairness in EMR
constrained by EF.

> The transport area are reviewing this draft. 
> If they confirm that this draft is not needed, we will withdraw it. 
> However the whole motivation for this draft is to address their
concerns.

Of course. I understand their concerns about PWs over the public
Internet.
I just think that this focus should be made clear by calling this draft
"congestion control framework for PWs over the Internet".

> Doesn't this happen in TCP anyway? 
> There is no flow identification in the routers to distribute the drops
over multiple flows. 
> The whole basis of TCP is a simple mechanism that may be short term
unfair, but is statistically fair in the long term.

The point I was trying to make in my first email was that the fairness
is of a particular type
and based on particular assumptions. When you have a large number of
equally important TCP flows
then you get approximate fairness in the sense of byte/sec (to within an
order to magnitude).
In the case I brought up, there is NOT approximate fairness between a
"naked" TCP flow
and one placed inside a PW. The TCP in a PW is biased down to a lower
rate.
If this happens, then customers will not want their TCP to be put inside
PWs !

Y(J)S

---

From: [email-masked]
Date: Mon, 30 Jun 2008 18:57:03 -0400

Stewart and Yaakov,

> > important) problem of a PW over the public Internet.
> 
> draft is not needed, we will withdraw it. However the whole motivation

Hi, I'm from the Transport Area and I'm here to help :-).
(If you're not from the US, you may miss the joke, see:
	http://www.quotationspage.com/quote/33742.html .)

Seriously, I'm a member of the Transport Directorate who has had some
interest and some level of involvement in pseudo-wire transport concerns
for a while now, including working with the authors of the FC
pseudo-wire
draft.

The core of Yaakov's concern around the framework appears to be the
possibility of over-reaction to congestion.  From the first message

flows.
idea
> The relevant underlying TCP/IP flow feels this and cuts back to half
its
> original rate.  In addition, according to the proposed mechanism, the
rate
> of the 10-equivalent flow PW needs to be reduced by half of 10
percent,
> i.e. by 5%. This translates to dropping every twentieth packet.

No it doesn't, but assuming that every 20th packet has to be dropped
sets up a nice exercise in strawman demolition.

An alternative that has different properties is to run a queue at
the pseudo-wire ingress, rate limit transmission from that queue to
95% of the PW bandwidth and use active queue management on that queue
(cf. RFC 2309 and the RED algorithm therein).  In essence, if the
pseudo-wire is a virtual "link" in an overlay network, this runs a
virtual "router" at the ingress, complete with AQM.

That's fine if the traffic through the pseudo-wire (or enough of it)
is congestion-responsive (e.g., TCP).  What if it's not?  Dealing
with this brings in Section 9 of the congestion framework draft, which
I think needs more attention.  In particular, there are (at least)
a couple of questions that deserve more exploration:
- At what loss threshold below CR should a tunnel be shut down
	because the assumptions under which it has been provisioned
	have been violated?  In this example, if the traffic is
	non-responsive and a 5% drop is required, the tunnel sustains
	a 5% loss rate.
- At what loss threshold below its constant rate should a constant
	rate PW be shut down as no longer able to deliver acceptable
	service?
At the end of the day, there's a bottom-line principle that 10 pounds
of <bleep> won't fit in a 5 pound sack, so if a 5% reduction in
traffic is needed, it has to come from somewhere, and that somewhere
may involve one or more constant rate PWs.

Having used the CR (Committed Rate) acronym above, I agree with the
importance that Yaakov attaches to it.  I tend to agree with him that
the service provider problem can be productively approached in terms
of policing to CRs.  The problem is that in the worst case, the CR
is zero, and if a constant bit rate PW is being used with a zero CR,
something needs to be available to reduce its traffic load (i.e.,
shut it down) if it's "part of the problem".

More coming in the other thread.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Stewart Bryant
> Sent: Monday, June 30, 2008 7:12 AM
> To: Yaakov Stein
> Cc: [email-masked]
> Subject: Re: [PWE3] the congestion framework violates the PWE 
> architecture
> 
> Yaakov Stein wrote:
> > Stewart
> >
> >   
> >> Doesn't this depend on how quickly the PW rate change cuts 
> in relative to the round trip time. 
> >> If the PW mechanism is slower than the TCP mechanism  the 
> traffic will have already dropped to
> >> the right level.
> >
> > I was assuming a PW which aggregates 10 users, and thus we 
> can assume that its
> The draft does say an SP can turn this mechanism off, but 
> they need to be careful.
> 
> > I don't see any real way of completely solving the (IMO not 
> very important) problem
> The transport area are reviewing this draft. If they confirm 
> that this draft is not needed,
> we will withdraw it. However the whole motivation for this 
> draft is to address their concerns.
> 
> >  As soon as you perform any PE rate cutback in addition to 
> that perfomed by the network,
> > you will more detrimentally impact TCP/IP.
> >   
> 
> Doesn't this happen in TCP anyway? There is no flow 
> identification in the routers
> to distribute the drops over multiple flows. The whole basis 
> of TCP is a simple
> mechanism that may be short term unfair, but is statistically 
> fair in the long term.
> 
> > Were it possible to identify individual internal flows, 
> then you could do selective drop. 
> >          then don't do anything at the PW level (everyting 
> will work out correctly)
> >   
> >    2a) if it is carrying constant rate traffic and the path 
> has not changed, 
> >             then perform selective packet drop taking 
> perceptual effect into account
> >    2d) if it is carrying non-real-time traffic over some 
> non-TCP transport protocol
> >             then learn how that protocol works and do 
> something specifically designed for it
> The draft does not preclude such action in response to a 
> congestion notification from
> the PW. All the draft does is to discuss the issues and set 
> up the context for the
> solutions drafts.
> 
> Stewart
> 
>

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 1 Jul 2008 08:03:56 +0300

David, 

Thanks for your comments (Stewart and I were beginning to feel rather
lonely here ...)

I think you summed up my concerns rather well, but I will sum them up
again anyway :

1) we need to address the "PW over the public Internet" case, 
    but the more "PWE3" case of a service provider network should not be
forgotten
2) I am concerned that mechanism needed for the unusual first case and
not relevant for the more usual second 
    will be mandated for all PWs
3) there is a concern that PWs in the first environment will not be
properly "weighted"
    the draft posits the concept of "equivalent TCP flows" without
making the argument clear enough
    I think that this should be clarified more
4) for the second environment I do not accept that the kludge of
"equivalent TCP flows" is the correct measure at all
    when there is not exact policing, we need to define a new concept of
fairness based on percent degradation 
    as compared to a "committed rate". This can be done in a centralized
and in a distributed manner.
    Simulation work should be done to ensure that the distributed method
works as anticipated.

I have written some new text clarifying and justifying the equivalent
flow concept and adding the SP case.

Y(J)S