Subject: [PWE3] Network builder capability

From: [email-masked]
Date: Thu, 9 Jun 2005 10:20:24 +0100

Thanks Yaakov for a considered response.
Note - I have changed the thread title to reflect the nature of the
issue being addressed here.

Let me try and respond by explaining the underlying
commercial/operational problem because this should set technical
direction.

Consider 2 operating parties A and B who require operational
independence (we have many customers who want this, and we ourselves
want this when we extend our networks beyond where we own
infrastructure).  To satisfy this requirement one requires (what we
call) a 'network builder service' capability....in other words the
network (all 3 planes of some technology) of one party is carried
transparently over the network (or more accurately the *traffic
data-plane* of some technology) of another party.

This can be easily satisfied in both the co-cs and cl-ps modes.

In the co-cs mode the nature of the constraints that are applied to the
link-connections (ie real/physical time/space/freq semantic, bit rate
known/quantised) impose some important properties.  For example:
-	it forces hierarchy....you cannot, for example, nest one SDH VC4
signal inside another SDH VC4 signal as there is a physical occupancy
conflict....and this in turn forces well-distinguished/independent layer
networks;
-	it only allows p2p and p2mp trail constructs....this is a
requirement of the co mode, but the physical occupancy constraints force
you to obey the requirement whether you want to or not;
-	it forces the control and management plane network OOB wrt the
traffic data-plane.....this has some important spin-offs in resiliency
design, stability and security.....note that the control/management
plane network(s) are very likely to use a cl mode running in their own
co-cs partition;
-	it removes the notion of QoS classes.  The consequence of this
is far more important than many folk would probably credit it on 1st
glance.....but I don't really want to get side-tracked by QoS issues
here;
-	wrt trail faults we can only have breaks and swaps (and then
only swaps between *exactly* alike trail entities).

In the cl-ps mode one does not have trails...so one is not constrained
by the above considerations (a key consequence that may not be
immediately obvious from the above is that a trail demands that the
Characteristic Information (CI as defined in G.805) must be identical at
all points along the trail from source to sink(s) and that the correct
ordering of traffic units must be preserved).  This is not a problem
however, it simply means the behaviour is different in the 2 modes...and
its the differences that are the important aspect that we need to
preserve because both modes are required.

Now if I make the statement 'IPinIP' (or indeed 'MACinMAC' for the
ethernet cl-ps case) this implies I have a functionally decoupled
client/server relationship like the co-cs mode case....except for the
fact we don't have trails....and, because of server->client performance
inheritance (which is recursive to the duct), this impacts whether one
considers this a suitable means to provide a network builder service or
not.  However, I don't want to get drawn into QoS issues here....the key
point I am making is that both the co-cs mode and cl-ps mode allow
functional decoupling of the client/server relationship which satisfies
the operational requirement noted at the outset.

Now let me consider the co-ps mode.  But before looking at the different
behaviours this allows let me make a few observations wrt to what is
going on in operator networks in the context of the network builder
service capability.

In the past a network builder capability was provided off PDH/SDH co-cs
mode technologies for a large range of BW sizes (indeed there is a large
set of layer networks here ranging from nx64k, through ~2M (E1/T1), to
~150M (VC4) and beyond and many in-between).  But there is a move now to
phase-out the smaller BW size co-cs layer networks, eg min VC4.  This
may however be still too large for some cases when leasing capacity.
Smaller BW network builder services can be supplied by FR or ATM
today...which are in reality nothing more that co-ps emulations of the
co-cs mode to all intents and purposes.  However, there is also a desire
to phase out these technologies.....the watchword being 'convergence'.
I would put this slightly differently however, by observing that it
makes a great deal of sense to converge on a single technology for each
of the 3 modes (because each technology maps to one of these modes in
varying degrees of 'goodness', so why duplicate?) but I really don't
believe one can have full functional convergence across the 3
modes.......both from technical considerations (ie the modes are
different) and operational/commercial considerations (ie the fact that
different parties want operational independence....which is where I came
in wrt to the need to satisfy a client/server network builder
capability).

So let's return to have a look at the co-ps mode now.  Let me initially
observe that the link-connections here are not constrained like the
co-cs mode (see above) but are allowed to take on an abstract identifier
(ie a 'label' which is a relative address of the link-connection) and
the bite rate is not fixed/quantised.  This allows one to do things to
the co-ps mode that are simply not possible in the co-cs mode, eg:
-	mp2p merging....which destroys the requirement for consistent
and correctly ordered CI at all points along the trail, and forces
demerging at a higher level...see G.8110 which explains this if not
clear
-	control/management is not forced OOB
-	one can introduce QoS classes (though I think this is largely a
waste of time/money)
-	one increases the fault cases to include misbranching and swaps
between any trails
-	and, perhaps most importantly of all, it allows one to introduce
the concept of a wrapper to create a form of hierarchy

Now prior co-ps technologies like X.25, FR and ATM did not do all these
things (ATM tried QoS and that was largely a waste of time...the ability
to choose the mode is a far more significant 'QoS enabler' IMO), but in
particular none of them had a wrapper behaviour.  However, and this is
key, we never tried to do a client/server network builder service in
these technologies where both the client and the server layer network
were the SAME technology {Aside - ATM had a pseudo-hierarchy of VP and
VC but this is not a proper client/server decoupling....it was simply an
artificial partitioning on the basis of the VPI/VCI and PTI codepoints
since there is no true encapsulation, ie the traffic unit is the same in
both cases....the cell}.  So, if party A wanted to run their ATM or FR
network over party B, they leased a PDH or SDH server layer from party B
and this ensured full functional/operational decoupling of the networks
of the 2 parties.

If we phase-out lower granularity co-cs layers networks and native
FR/ATM layer networks and we have the case of party A with an MPLS
wanting to lease capacity over party B's MPLS network then how are we
going to do this in a functionally decoupled manner?  Perhaps the most
obvious answer is to force a termination of the wrapper behaviour (by
setting S=1) in the server layer MPLS network.....and today this means
inserting some technology (TDM, ATM, whatever) as a 'layer network
breakwater' between the 2 MPLS networks...in other words a PW. {Aside I
have often thought we may need a 'MPLS PW' to avoid this.}  However,
this (i) means the PW starts to become a fully-fledged layer network in
its own right (I've seen this one coming for a long time) which is
departing from the goal of convergence to a single technology per mode
and (ii) we still have the implications of the wrapper.  Maybe this is
unavoidable?....at least in the near-term.  But I think we can do better
than this and its something I am encouraging folks here to think long
and hard about because we all know what the problem is, ie there is a
clear commercial and operational need for a client/server network
builder solution in the co-ps mode, and without it folks are blundering
into 'MPLS interconnects' when this is not what is really
required.....and this really concerns us.

Anyway, let me finish off by saying a little more about the wrapper...

When we have a wrapper behaviour and the effective networked depth is 1
then the wrapper behaviour is latent.  Both single-hop PWs and rfc2547
VPNs don't expose the wrapper.  This is because the semantic of the
higher LSP 'label' is no longer a link-connection ID but it is proxying
for some client source identifier.  Further, the higher layer LSP is a
single hop entity and can be regarded as a session entity between 2
fixed end-points....it is for sure NOT a networked entity (MH PWs change
this....a point I'll shortly come back on).  In the rfc2547 case the
higher label proxies as a PE/VRF identifier (for a given VPN), in the
case of PWs the label proxies for the client layer technology
link-connection source.  One needs this behaviour in any case to recover
from any lower layer merging that one may find with LDP mp2p or
PHP......this is just a fact.

However, the notion of Multi-Hop PWs is now exposing the wrapper.
Further we are now creating an unnecessary layer network (the PW layer
network), and this is such a shame IMO...we should have been able to do
this directly in MPLS, but this harks back to the origins of why MPLS
looks like it does (we don't treat all clients of MPLS as 'equals').
There is further twist here regarding topology creation and topology
stability in layer networks:  The link-connections in layer network N
are provided by trails in layer network N-1 (which is of course how one
creates the topology of layer network N).  This relationship recurses to
the duct.  The implicit assumption here is that those lower layer trails
are ALREADY pre-provided....because if they are not how the heck can you
calculate a route over a topology that does not exist?  So if we work
down the stack we could say (for example):
-	to calculate routes in the PW layer N network it assumes the
link-connections provided by layer N-1 LSPs (trails) are already present
-	to calculate routes in layer network N-1 it assumes the
link-connections provided by layer N-2 trails (whatever technology) are
already present
-	to calculate routes in layer network N-2 (whatever technology)
it assumes the link-connections provided by layer N-3 trails (whatever
technology) are already present
-	etc to SDH, to Optics, to Duct

Put simply, a digital wrapper makes an assumption that 'topology can be
created on the fly'....and it can't.  Note also that, although
unwritten, there has to be a tacit assumption that the holding times of
trails increases as one approaches the duct (this is not simply to
present a stable topology to the client, but it has important commercial
implications if you try and figure out the redundant build required to
give a certain GoS such that an 'on demand' trails can be set up, esp at
the lower network levels).  And remember not all these layer networks
will be owned by the same party...which is why I said at the outset that
we MUST be able to provide a functionally decoupled client/server
network builder capability as the starting point.

Given where things are now at, its seems to me that current direction
will force the creation of a new layer network....the PW layer.  This
means that all the things we need in a layer network will have to be
replicated here. IMO we could have done this a lot better and avoided
this if we had regarded MPLS differently to IP from the outset.  I don't
think the situation is lost however.  For example we could very easily
avoid some of the architectural violations we find in some forms of MPLS
and we could add the missing functional components/behaviour (eg right
OAM, consistent addressing of LSP access points....these are not IP
layer network addresses, and OOB control/management).  However, I also
think some thought needs to be given to how we address the network
builder capability between the MPLS networks of 2 parties...and as I
noted, perhaps a 'MPLS PW' is a possible near-term answer?

Aside - an alternative is to create a new co-ps solution....some folks
are already postulating this using ethernet as you may be aware.

regards, Neil

> -----Original Message-----
> From: Yaakov Stein [mailto:[email-masked]]
> Sent: 08 June 2005 16:14
> To: Harrison,N,Neil,IKM1 R; [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: RE: [PWE3] draft-martini-pwe3-pw-switching-03.txt as 
> WG draft?
> 
> 
> Neil,
> 
> I read your email several times,
> and must confess that I can't follow your arguments
> regarding MS PWs.
> 
> For the multisegment case, we terminate the MPLS layer
> of one network (we have to since we are going into another
> administrative domain or onto another PW technology) and are 
> left with the PW layer.
> 
> The PW layer is treated just as an MPLS layer would be,
> with the label being swapped in order to belong to the
> label space of the new domain.
> 
> The two domains are in a peering relationship
> and represent a G.805 partitioning of the PW layer network. In order 
> to ensure the integrity of the trail at the PW layer, we have VCCV 
> running as OAM.
> 
> In the new domain the PW is now placed onto
> a new server layer, which may happen to be MPLS
> (or may be something else such as carrier Ethernet or L2TPv3).
> 
> The MS PW signaling operates at the PW layer.
> At the PW layer we usually only signal between endpoints,
> relying on the underlying tunnel making placement trivial. 
> But here we have two separate tunnels that need to be joined, 
> so we need signaling that ensures that we reach the stitching 
> point, and that both directions go through the same stitching point.
> 
> So where do you see any architectural problem?
> We are not (here at least) suggesting any confusion between CO and CL 
> server layers. We are not performing any (new) G.805 layer violations.
> All we are doing is recognizing that the PW layer
> is a true layer network, one that in the MPLS case
> is simply a continuation of the G.8110 recursion.
> 
> Y(J)S
> 

_______________________________________________
pwe3 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pwe3

---

From: "Yaakov Stein" <[email-masked]>
Date: Thu, 9 Jun 2005 15:41:21 +0200

Neil, 

Thanks for the detailed response,
which certainly clarifies what you meant in your earlier email.

I think your concept of a "network builder" is important,
but don't understand the reasoning for this nomenclature.
If I understood correctly, by "network builder" you mean
the ability to transparently send all planes of one party
over some other party's network. By "transparently" I assume
that you mean "network interworking" (although not just user plane
network interworking) rather than peering.
Why is this a "builder" rather than a "borrower"?

I think there are only two basic differences between our viewpoints,
which are otherwise not that divergent.

1) I recognize the PW layer as being a full layer network
   and always have. This is why (as you well know) I have
   been pushing for all the requisite functionality (e.g. PW layer OAM).

2) For the MPLS case I consider the PW layer to be 
    simply another G.8110 layer. This solves the problem
    of the "MPLS PW" you raise, and additionally reduces
    the number of new mechanisms needed.

    As you may remember, a while ago Shahram and I suggested that
    MPLS stacks should be stackable. That is, you should be able to send

    your MPLS packet through someone elses MPLS network as an opaque
    payload. This was resisted due to the ECMP problem of having
    an MPLS shim header immediately after an S=1 header.
    I agree with you that by making the payload into something
    else, e.g. ATM or a PW by adding a CW, would solve the problem,
    but I don't see why this overhead should be needed.

    It is a pity that in the IETF the PW work was carried out so
separately
    from the MPLS work. There were perhaps good reasons for this in the
past
    (e.g. to reduce duplication of format definitions for the MPLS and
L2TPv3 cases),
    but we are now stuck with PWs not being "just a regular MPLS layer".
    This has caused the problem we are now seeing with the MS PWs.
    Were the PW layer a specific MPLS layer, then we would be able to
    use any signaling mechanism developed for MPLS, e.g. RSVP-TE
    for QoS enhanced PWs. As it is we are limited to LDP a la Martini.
   
Regarding the recursion of encapsulations, yes this is a new feature
and quite a strong one. It allows G.805 layering without inventing new
protocols for each layer. I agree that it becomes more tempting
to perform layer violations, but if the spirit is strong we can survive.

I am aware of Maarten's suggestion for Ethernet transport.
However, I really don't see a big difference here (I know that you do). 
>From my point of view the VLAN tag/p-bits and the MPLS label/EXPbits 
are similar in functionality and both are stackable.
The VLAN tag has a smaller space, but still large enough for transport
applications. So although I agree with Maarten that this using the
Ethernet
format is possible, I really don't want to have to build everything
twice.
And since the MPLS already has the signaling (and in fact too many ways
of performing the signaling) while Ethernet is mainly statically
provisioned,
I prefer MPLS.

Y(J)S

---

From: [email-masked]
Date: Thu, 9 Jun 2005 17:08:22 +0100

Thanks Yaakov....glad at least you read my mails (and even better
understand them!)

> which certainly clarifies what you meant in your earlier email.
NH=> Its a very important point (which I have been banging on about for
a long time) and its getting overlooked.  And this is now leading into
some folks blundering into issues that really concern us, like 'MPLS
interconnects' (what the heck does that really mean!)  
> over some other party's network.
NH=> Yes, with the further observation ALL the client planes only get
carried over the server layer *TRAFFIC* data-plane for this.

> By "transparently" I assume 
> that you mean "network interworking"
NH=> A truly dreadful term (just like L1, L2, L3
classifications).....not as bad a 'service i/w' however, which means
very little in practice, but if I was to explain it would take another
page and I think I have said enough recently.  When I say
'transparenetly' I mean no client compression, ie the client layer (all
3 planes) is transported without any modification.

> (although not just user 
> plane network interworking) rather than peering. Why is this 
> a "builder" rather than a "borrower"?
NH=> You could use that phrase.  The key point is that one party uses
this to 'build' their own networks without actually having to own the
infrastructure stack to the duct.
> 
> I think there are only two basic differences between our 
> viewpoints, which are otherwise not that divergent.
NH=> No, I think you are right...in fact most engineers I speak to (esp
ones who grok func arch stuff) understand exactly where we are coming
from and agree with us.
>    been pushing for all the requisite functionality (e.g. PW 

NH=> Yes, I know that....as I remarked earlier I saw this coming too
right from the outset.  My point is that this is quite unnecessary.
However, it has been an incremental series of arch misjudgements that
have created the current situation.  It starts with the erroneous view
that IP==MPLS....but to which I should add that the reality is that, as
a client of MPLS, IP is treated differently to MPLS (this is the wrapper
issue) and both are treated differently to everything else that could be
a client.  This was, I assume, to ensure XoverMPLS looks like
XoverIP(L2tpv3) to 'keep the faith'....but as we know this has diverged
(has too, they are not the same)....and it also creates a consequential
'peer-partition' i/w problem between XoverIP and XoverMPLS.

Then we have the arch flaws of merging (due to mp2p and PHP), builds
through ECMP and the CW hacks, adds way too complex OAM (because of the
previous points)...to which folks add per specific client OAM and
functions because of compression and ommision in the server....and then
we have the wrapper (which I have already explained).
>     the number of new mechanisms needed.
NH=> Well, not really....not unless the intent is to really define a
full-blown layer network called 'PW'.....but why?  If we had a the right
form of MPLS, let me call this 'transport MPLS', you would not need to
create a new layer network....just the *right* forms of adaptation to
it.  My suggestion of a 'MPLS PW' is kludge for the current situation.
>     payload.
NH=> Absolutely right.  You should NEVER compress the client IF this is
another layer network (I make a distinction with end-system applications
here, eg voice compression) unless there are very good reasons to do
this.  And believe me BW saving is not one of them....far from it in
fact (I'm so tempted to get onto the QoS illusion here, but I'll
resist).

> This was resisted due to the ECMP problem of having
>     It is a pity that in the IETF the PW work was carried out 
> so separately
>     from the MPLS work. There were perhaps good reasons for 
> this in the past
>     (e.g. to reduce duplication of format definitions for the 
> MPLS and L2TPv3 cases),
NH=> I'd like to see a strong synnergy between the control-plane
solution for the co-cs mode and the co-ps mode....I want to see best of
breed functional solutions emerge with maximal functional resuse that
also address serious arch requirements (like taking all internal
control/management protocols OOB wrt traffic).  I strongly favour
RSVP-TE not because I think its a great signalling protocol, in fact I
don't think it is, but it is at least a real NETWORKING signalling
protocol and it does allow us to get the required synnergy with the
co-cs mode.  This is why I supported Rahul when everyone else was saying
LDP.
>    
> Regarding the recursion of encapsulations, yes this is a new 
> feature and quite a strong one. It allows G.805 layering 
> without inventing new protocols for each layer. I agree that 
> it becomes more tempting to perform layer violations, but if 
> the spirit is strong we can survive.
NH=> I agree, IF there is the will.  My experience is that when I try
and explain stuff like this it either (i) gets ignored or (ii) there are
attempts to ridicule it.....I've been there before with the OAM stuff
and Y.1711.  But this really has to start with some honesty about the
nature of the 3 networks modes....I wish more folks understood this, esp
func arch stuff.
> The VLAN tag has a smaller space, but still large enough for 
> transport applications. So although I agree with Maarten that 
> this using the Ethernet format is possible, I really don't 
> want to have to build everything twice. And since the MPLS 
> already has the signaling (and in fact too many ways of 
> performing the signaling) while Ethernet is mainly statically 
> provisioned, I prefer MPLS.
NH=> I have to be careful what I say here.  In principle one can take
any pkt PDU and create a co-ps network....you can even do it with IP
using an OOB control-plane etc, we tested such a proposal from a certain
party a few years ago.  We have even proposed developing transport MPLS
ourselves, ie keep the bits that are OK, add the bits that are missing,
and I know some other folks have proposed similar.

regards, Neil