Subject: RE: What is PWE3 chartered to do?

From: "David Allan" <[email-masked]>
Date: Tue, 1 Apr 2003 09:40:23 -0500

Stewart:

> 
> Correct order would not be guaranteed during a fast re-route 
> or a return to service.
> 

Do you envision this being a frequent occurrance ?

Dave

---

From: "White, Craig (Level 3)" <[email-masked]>
Date: Tue, 1 Apr 2003 09:01:25 -0700

Just to put some numbers around this....
 
The rule of thumb that I hear tossed around is 1 cut per 1000 Miles of fiber per year... So a US network will consist of 15,000-20,000 miles, that's 15-20 cuts per year (assuming Link rather than Node protection fast re-route and further assuming all routers never die :)
 
If you add in a couple of tech brain fades while connecting all these PWE's :) so say you can expect 2 cuts per month somewhere on the network. Obviously the impact of each cut won't effect all LSP's, but... 
 
-craig

-----Original Message-----
From: David Allan [mailto:[email-masked]]
Sent: Tuesday, April 01, 2003 7:40 AM
To: [email-masked]'; Peter Willis
Cc: George Swallow; [email-masked]; [email-masked]
Subject: RE: [PWE3] What is PWE3 chartered to do?

Dave

---

From: "David Allan" <[email-masked]>
Date: Tue, 1 Apr 2003 11:58:54 -0500

George:

> Peter -
> 
> > I agree with you, MPLS LDP delivers a CNLS service and may deliver  
> > packets out of order. Therefore if we use PWE3 over MPLS 
> LDP then we 
> > have to cater for out of order packets. My concern is if we 
> used PWE3 
> > over MPLS RSVP-TE (or similar) then we would assume packets 
> would be 
> > delivered in order and we wouldn't want to bear the cost of the 
> > re-sequencing mechanism in PWE3.
> 
> The whole point of this discussion on the first nibble is 
> that if you obey certain rules then most (all?) currently 
> available P nodes will deliver both the data and associated 
> OAM in sequence, 

It will????? News to me.

Dave

---

From: George Swallow <[email-masked]>
Date: Tue, 01 Apr 2003 12:19:30 -0500

> > available P nodes will deliver both the data and associated 

Not if you insist of using label 14.  But there are other way to skin
the cat!

...George

==================================================================
George Swallow       Cisco Systems                  (978) 936-1398
                     250 Apollo Drive
                     Chelmsford, Ma 01824

---

From: "David Allan" <[email-masked]>
Date: Tue, 1 Apr 2003 12:48:52 -0500

George:

Not parsing this.... if an OAM flow is an IP packet, as LSP-PING is, then is
gets forwarded independently of the majority of microflows it potentially
represents unless one is actually ICMPinging the microflow destination
impersonating the microflow source (whoa). If it is label 14 it has similar
problems although not being an IP packet, it cannot impersonate one ;-).

If your skinning the cat is a means of transporting OAM protocols hidden
from ECMP and without using reserved labels (a la Tom's VCCV proposal when
Router Alert label is not employed), doesn't that potentially make all OAM
protocols equal (as they can all be smuggled across the network hidden from
the OAM evils of ECMP)? And equally irrelevant to IP flows? 

So what that poor skinless cat represents is that
1) No one has yet codified a layer violation for PWs, give folks time, ECMP
based on an Ethernet MAC sec/dest would work wonderfully.
2) OAM is perceived as to only providing benefits to some applications of
MPLS. Too bad.

cheers
Dave

> From: George Swallow [mailto:[email-masked]] 
> Sent: Tuesday, April 01, 2003 12:20 PM
> To: Allan, David [CAR:NS00:EXCH]
> Cc: 'George Swallow'; Peter Willis; [email-masked]; 
> [email-masked]; [email-masked]
> Subject: Re: [PWE3] What is PWE3 chartered to do? 
> 
> 
> Not if you insist of using label 14.  But there are other way 
> to skin the cat!
> 
> ...George
>

---

From: George Swallow <[email-masked]>
Date: Tue, 01 Apr 2003 13:37:15 -0500

> the OAM evils of ECMP)? And equally irrelevant to IP flows? 

As far as I know either the inband or the router alert will work with
most equipment.  (I'd like very much to hear if I'm wrong).  The VCCV
draft puts the RA label next to the bottom, so if you hash on just the
bottom label, you get the same result.

> doesn't that potentially make all OAM
> the OAM evils of ECMP)? 

Yes!  You can get a PPP codepoint (or whatever we end up with) that
says the payload is MPLS OAM.  

...George

---

From: "David Allan" <[email-masked]>
Date: Tue, 1 Apr 2003 14:10:21 -0500

Hi Again:

> As far as I know either the inband or the router alert will 
> work with most equipment.  (I'd like very much to hear if I'm 
> wrong).  The VCCV draft puts the RA label next to the bottom, 
> so if you hash on just the bottom label, you get the same result.

Why is Router Alert somehow more immune than OAM Alert to the vagarities of
ECMP? As an aside, it appears simply to be overloaded as a programmable PID
negotiated with signalling (if there's an RA it must be ICMP).

Dave

---

From: George Swallow <[email-masked]>
Date: Tue, 01 Apr 2003 14:14:46 -0500

> negotiated with signalling (if there's an RA it must be ICMP).

Can't speak for others, but we has on the bottom label.  Y.1711 says
the label goes on the bottom.  Has nothing to do with the value.

...George

---

From: Shahram Davari <[email-masked]>
Date: Tue, 1 Apr 2003 11:31:57 -0800

>the label goes on the bottom.  Has nothing to do with the value.

George,

But as Tom said, shouldn't a generic solution be ECMP implementation independent?

-Shahram

---

From: George Swallow <[email-masked]>
Date: Tue, 01 Apr 2003 14:39:04 -0500

> But as Tom said, shouldn't a generic solution be ECMP implementation 

It would be nice.  But some folks are insisting that they run without
PWE headers.  And there's some hardware out there that can't inspect
newly defined header bits.  Otherwise we'd only have the inband
solution.

...George

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 01 Apr 2003 16:12:55 -0500

> > available P nodes will deliver both the data and associated

         If you don't use special labels (i.e.: label 14)
this will work fine.

         --Tom

>Dave

http://www.elsevier-international.com/catalogue/title.cfm?ISBN=155860751X

---

From: "David Allan" <[email-masked]>
Date: Tue, 1 Apr 2003 16:31:07 -0500

Tom:

I think George clarified that it's not the label value, its the position, at
least specifically for the Cisco implementation.

I love trying to solve problems in the presence of incomplete information.
So any other surprises you folks would care to share?

rgds
Dave

> From: Thomas D. Nadeau [mailto:[email-masked]] 
> Sent: Tuesday, April 01, 2003 4:13 PM
> To: Allan, David [CAR:NS00:EXCH]; 'George Swallow'; Peter Willis
> Cc: [email-masked]; [email-masked]
> > > The whole point of this discussion on the first nibble is that if 
> > > you obey certain rules then most (all?) currently 
> available P nodes 
> > > will deliver both the data and associated OAM in sequence,
> >
> >It will????? News to me.
>

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 01 Apr 2003 16:38:38 -0500

At 12:48 PM 4/1/2003 -0500, David Allan wrote:

>George:
>
>Not parsing this.... if an OAM flow is an IP packet, as LSP-PING is, then 
>is gets forwarded independently of the majority of microflows it 
>potentially represents unless one is actually ICMPinging the microflow 
>destination impersonating the microflow source (whoa). If it is label 14 
>it has similar problems although not being an IP packet, it cannot 
>impersonate one ;-).

         The point of impersonation is to be forwarded through the data plane
precisely as the normal data is.  Use of a different label *may* change the
forwarding behavior depending on ECMP.   The current version of LSP Ping
for instance, will detect and test all and only all of the actual paths
(LSP ECMPs). I don't see how the microflows inside of those ECMPs are
relevant, but I might be missing something.

>protocols equal

         I don't think so. 8)

>(as they can all be smuggled across the network hidden from the OAM evils 
>of ECMP)? And equally irrelevant to IP flows?

          VCCV is specified in such a way that the test traffic is
forwarded precisely as the data would be all the way to the disposition
point.  Again, in this case, the use of a special label might change ECMP
behavior which is what we don't want.

         --Tom

>1) No one has yet codified a layer violation for PWs, give folks time, 
>ECMP based on an Ethernet MAC sec/dest would work wonderfully.
>
> > From: George Swallow [<mailto:[email-masked]>mailto:[email-masked]]

---

From: Shahram Davari <[email-masked]>
Date: Tue, 1 Apr 2003 14:01:27 -0800

>>Router Alert label is not employed), doesn't that potentially 

Why?

>behavior which is what we don't want.

1) I think what Dave said is that you could do the same thing you did
with VCCV, and signal Y.1711 (without reserved label) and it would fine.

2) Isn't alert label a special reserved label? And wouldn't it change the
ECMP behavior of some implementations?

-Shahram

>> > From: George Swallow 
[<mailto:[email-masked]>mailto:[email-masked]]

_______________________________________________
pwe3 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pwe3

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 01 Apr 2003 17:05:26 -0500

>But as Tom said, shouldn't a generic solution be ECMP implementation 

         Please be careful when you paraphrase me. I did not say
that we need a generic OAM solution.

         --Tom

---

From: Shahram Davari <[email-masked]>
Date: Tue, 1 Apr 2003 14:10:51 -0800

>that we need a generic OAM solution.

Ok, any solution shouldn't be ECMP implementation dependent. 

-Shahram

---

From: Peter Willis <[email-masked]>
Date: Wed, 02 Apr 2003 09:56:08 +0100

>          Right. As Eric points out, if you want sequencing, add it
> to the PWE header and don't worry about the transport misordering.
> 

Which is fine if sequencing comes for free. However sequencing does not come 
for free. It consumes memory & processing. And its also a difficult algorithm 
to implement correctly (is it still broken in the PWE3 drafts?) --- the 
problem being telling the difference between misordering, drops & something 
delayed for a long time, and if you cater for the latter then your 
delay and jitter targets go out the window.

Now if you use MPLS correctly you will only get misordering very infrequently. 
So if I can use MPLS RSVP-TE (or similar) and not implement the sequencing stuff
wouldn't it be lower cost to implement? And wouldn't it give better 
performance?

Peter.

---

From: Peter Willis <[email-masked]>
Date: Wed, 02 Apr 2003 10:03:43 +0100

>          There is no free lunch. Setting up fixed tunnels everywhere is not 
> cheap
> operationally compared to using LDP.  But you are an operator and can 
> appreciate
> that.

As an operator we are well aware of the pros & cons of LDP.

Seems to me the tunnel problem is an OSS/config problem where as the 
resequencing problem is something that has to be done to every packet. My gut 
feeling is that something that has to be done per tunnel is cheaper than 
something that has to be done per packet.

Peter.

---

From: [email-masked]
Date: Fri, 4 Apr 2003 07:37:50 +0100

Peter, 

I have previously pointed out that we, as a service provider, require simple
client/server adaptation between arbitrary X/Y client/server layer network
relationships for reasons like you remark.....also required for client and
server independence.  These guys are creating the beginings of another
'in-between' layer network that will have to be managed by those you wish to
use it....though I could never see anyone here touching it.  However, all
these bolt-on functions in this 'in-between' layer primarily come about
because of LDP....so, since LDP does not provide any benefit (that cannot be
better addressed directly via or IP or using p2p LSPs), but only gives hard
management problems, then the right answer is simply to deprecate these and
not try and bodge some fixes for them.

regards, Neil

> From: Peter Willis [mailto:[email-masked]]
> Sent: 02 April 2003 09:56
> To: Thomas D. Nadeau
> Cc: George Swallow; Peter Willis; Harrison,N,Neil,IKL2 R; 
> [email-masked]

---

From: Peter Willis <[email-masked]>
Date: Fri, 04 Apr 2003 11:57:01 +0100

Seems to me PWE3 has an expensive solution engineered in to its charter!

What can we do?

We could lobby to get the PWE3 charter changed but most of the very few other
 network operators on this list have vested interests in the current solution.

Peter.

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Fri, 04 Apr 2003 10:15:57 -0500

Neil,

         In a utopian/white-board world that disregards the
reality of deployed kit, such a conclusion makes perfect sense.
So did similar reasoning to those in the past who reasoned that
Earth was flat because that argument too looked good at least
on paper. However, the reality is that 100s of thousands of LSRs
are currently deployed in real networks and run LDP, so it is not a
realistic option to just say lets stop using LDP today or in the future
because it is hard to manage.  There are plenty of other technologies
that are difficult to manage and use that are still widely deployed today
because they still get the job done most of the time. Perhaps you are
familiar with a technology called Windows?

         --tom

---

From: "Yaakov Stein" <[email-masked]>
Date: Sun, 6 Apr 2003 09:12:50 +0300

> Seems to me PWE3 has an expensive solution engineered in to 

PWE was not defined to be purely MPLS,
quite the opposite, it is supposed to be PSN-agnostic.

Quite a few of us are still providing most of our PW
services over IPv4.

So putting the sequencing (when needed) into the PW layer, 
rather than relying on the PSN services, 
simplifies the design by removing complex interdependencies
between the PW and the PSN.

Y(J)S

---

From: [email-masked]
Date: Sun, 6 Apr 2003 10:41:55 +0100

Thomas D. Nadeau wrote 04 April 2003 16:16
 
> reality of deployed kit, such a conclusion makes **perfect sense**.
NH=> Good.  Then you do at least admit LDP and PHP are 'mistakes', this is a
required initial 1st step, ie acknowledge the true source of the problem.

> on paper.
NH=> This is simply a Universal truth that was discovered......it is not a
man-made creation like LDP.  So I don't see the relevance of this statement.

> However, the reality is that 100s of thousands of LSRs
> because it is hard to manage.
NH=>  Agreed.  We all make mistakes in our professional/private lives.  The
only thing I am asking for is to accept these, learn from them (stop doing
kludge on kludge) and move on.  Sure, we have to find ways to manage the
stuff we now have, but I want to see a position where we recognise that this
chapter needs closing at some point.  To not learn from one's mistakes is
not acceptable engineering practice IMO.

> There are plenty of other technologies
NH=> Not sure the point you are trying to make here.  The only logical
conclusion I can draw is that there is no need for bodies like IETF to exist
because we simply let the dominant supplier to the industry-sector of
relevance set the de facto stds that we all just accept.

regards, Neil

---

From: "Monique Morrow (mmorrow)" <[email-masked]>
Date: Mon, 7 Apr 2003 10:08:07 +0200

Neil,

Reading the thread thus far and having worked for a service provider
myself in both network operations, engineering and service development
and having spoken with service providers throughout the world one of the
challenges we all have (service providers and vendors) is to balance
pragmatism, architectural efficiency and business impact (for carrier
customers and vendors).  This thread clearly points out the difficulty
is doing the above.  Service providers (specifically the organizations
who are developing and deploying services) are very sensitive to
customer impact to whatever technology, architecture, mechanism that may
be introduced in a production environment -- I believe we all understand
this point and are attempting to articulate mechanisms that are
evolutionary, that is with minimal disruption and yet may address
service and operational requirements moving forward.  Using terms such
as "mistakes" is useless and really has no value in this discussion as
numerous providers have developed services and business models based on
this technology.  However when one speaks about "deprecating LDP," I
suggest an operational and business risk assessment (for production
networks) on what this concretely means, what is operational impact to
my customers?  What do I need to do with existing services?  What is
service management impact?  What mechanisms do I require to evolve new
services?

I would like to suggest that we approach the problem in balancing
pragmatism, architectural efficiency and business impact.

Best regards,

Monique

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: 06 April 2003 11:42
To: Tom Nadeau (tnadeau); [email-masked]
Subject: RE: [PWE3] What is PWE3 chartered to do? 

> reality of deployed kit, such a conclusion makes **perfect sense**.
NH=> Good.  Then you do at least admit LDP and PHP are 'mistakes', this
is a
required initial 1st step, ie acknowledge the true source of the
problem.

> Earth was flat because that argument too looked good at least
NH=> This is simply a Universal truth that was discovered......it is not
a
man-made creation like LDP.  So I don't see the relevance of this
statement.

> because it is hard to manage.
NH=>  Agreed.  We all make mistakes in our professional/private lives.
The
only thing I am asking for is to accept these, learn from them (stop
doing
kludge on kludge) and move on.  Sure, we have to find ways to manage the
stuff we now have, but I want to see a position where we recognise that
this
chapter needs closing at some point.  To not learn from one's mistakes
is
not acceptable engineering practice IMO.

NH=> Not sure the point you are trying to make here.  The only logical
conclusion I can draw is that there is no need for bodies like IETF to
exist
because we simply let the dominant supplier to the industry-sector of
relevance set the de facto stds that we all just accept.

regards, Neil

---

From: Peter Willis <[email-masked]>
Date: Mon, 07 Apr 2003 14:34:01 +0100

> between the PW and the PSN.

Which is OK so long as the operators that have networks that deliver packets 
in order do not have to bear the costs associated with resequencing because 
some other operators have networks that can't deliver packets in order!

Peter.

---

From: Eric Rosen <[email-masked]>
Date: Mon, 07 Apr 2003 10:42:55 -0400

Peter> Which is OK so long as  the operators that have networks that deliver
Peter> packets  in order  do  not have  to  bear the  costs associated  with
Peter> resequencing because  some other  operators have networks  that can't
Peter> deliver packets in order! 

All of the  PSN technologies under consideration may  misorder packets under
normal  operation.  Implementations  of all  of the  PSN  technologies under
consideration  generally do strive  to keep  the packets  of a  microflow in
order, as those packets pass through a single router.  This suggests that at
the PWE3 layer,  the sequencing (for data packets,  at least) operations can
be  (a) made  optional,  and (b)  restricted  to the  discarding of  out-of-
sequence packets.   Which is  exactly the functionality  we have  built into
PWE3.  

It seems to  me that PWE3 has  clear requirements in this area,  and that it
meets its those requirements in this area.  The current thread does not seem
to provide any relevant content here.

---

From: Peter Willis <[email-masked]>
Date: Mon, 07 Apr 2003 17:11:32 +0100

Monique,

It is precisely all these issues we have considered and for us (i.e. a network 
operator with existing ATM, FR, SDH networks) we think that building PWE3 on
a PSN that does not use MPLS LDP, but rather MPLS RSVP-TE (or similar), would
be more efficient and have less impact on customers.

Have any network operators currently deployed PWE3 over MPLS LDP? If so why 
didn't those operators just do PWE3 over IP? And what has using MPLS LDP bought 
them? 

Peter.