Subject: RE: on the mpls oam framework

From: "David Allan" <[email-masked]>
Date: Mon, 17 Nov 2003 11:11:39 -0500

Hi again:
<snipped>
> >I'm not. I'm sure I'm not the only person who has noted the oxymoron 
> >inherent to the requirement "must support undocumented proprietary 
> >feature". I'm suggesting it MUST be documented in order to proceed.
> 
> 	I am suggesting that doing so is not a scalable
> problem to solve.  

Not parsing?

> Also practically speaking, I doubt
> that anyone is going to divulge their exact algorithm.
> It might be an interesting informational draft to 
> produce such a compendium from anonymous sources, but
> this always leaves open the door for one more algorithm.

Then that's tough for those who do not share. If you want interoperable OAM,
you have to play. IMHO the minimum requirement is exact what fields are
examined, how much of the stack, interface IDs etc. I'm less concerned about
whether it is CRC-32 or BIP-16 or whatever. 

<snipped>
> >That's not the suggestion. The suggestion is that we only
> >support what we can see. 
> 
> 	That sounds like you want connection-oriented
> constructs only too. *)

Is this comment relevant?

<snipped>
> 	While I would of course agree that knowing all of the 
> algorithms would make things easier, I disagree that we 
> cannot move forward with solutions today. I think that 
> solutions can be 
> created to manage what is deployed today. There is clear 
> evidence of this already.

As I say, we all need to be able to innovate in this space. At the moment
the majority of the list is disenfranchized. This is a problem. Either the
WG sticks with the decision that this is not specified and recognizes that
the quality of what it gets for interoperable OAM is unknowable, or people
at least 'fess up informationally and we can all participate. Which version
do you think is the IETF way....

BTW, would be nice to get some other opinions, IMHO this is rather
fundamental....

cheers
Dave

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 08:39:01 -0800

>BTW, would be nice to get some other opinions, IMHO this is rather
One of the fundamental precepts of routing, is that each node 
understands the routing decisions each other node makes, this is to 
prevent black holing among other things. So for example, the SPF 
algorithm is not a secret, precisely so that each router in a network 
can make consistent decisions. As ECMP algorithms impact the forwarding 
direction of packets, then I would suggest while not on the same level 
of criticality as SPF, there is a similar logic that could be applied to 
this case. I have similar thoughts about end-to-end QoS (but that's way 
OT, just sited as another example).

In addition, having worked at a vendor trying to work out the best way 
to do ECMP (which is not easy task in an MPLS environment) it sure would 
have been nice to have an industry-agreed way of doing this. While I 
realise 802.1ad have gotten away with not agreeing to alogorithms, that 
is on a point-to-point basis. When an algorithm impacts end-to-end 
(within an area/AS at least) then I think the need to have an agreed 
upon standard is more important.

So bottom line is I would have to agree that two routers from different 
vendors can not communicate in the same network using ECMP unless the 
algorithm is known by both. So if ECMP is in an architecture or 
framework document, then it is not a proprietary feature, and therefore 
it should be standardized.

my 2 cents....

Mark

---

From: Rahul Aggarwal <[email-masked]>
Date: Mon, 17 Nov 2003 11:51:22 -0800 (PST)

Hi Dave,

> BTW, would be nice to get some other opinions, IMHO this is rather

To reiterate what has already been said on this thread. A MPLS OAM
framework document must a) Follow the MPLS architecture and b) Discuss
an OAM framework that concerns *practical* problems seen in MPLS deployments.

If folks have issues with MPLS architecture/usage, it should be documented
elsewhere, not in the OAM framework document.

Regards,
rahul

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 17 Nov 2003 15:26:38 -0500

In message <[email-masked]>, "
> BTW, would be nice to get some other opinions, IMHO this is rather

Dave,

MPLS Ping has a means defined to determine that an ECMP branch point
exists, and a means to provide the means to exercise each branch.
This is all independent of the algorithm used to accomplished the
split, hash based or other.  That is a move forward technically.  The
alternate of randomly spraying addresses across the 127/8 space has
also been suggested as a solution that from a practical standpoint
covers the problem, though with no certainty.

This is a solved problem.  Can we stop moving backwards.  If you can
improve on the solution, that would be welcome too.  Substituting
Y.1711 for the solutions being worked on in the IETF and complaining
that Y.1711 doesn't handle ECMP is not seen as an improvement.

Curtis

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 15:31:00 -0500 (EST)

> elsewhere, not in the OAM framework document.

Rahul,

Sounds like a fair point. But just so I am understanding you clearly, are you saying these issues should not be discussed in any OAM document, or just during the framework development stage?

Thanks,
Mark

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 15:33:47 -0500 (EST)

---- Curtis Villamizar <[email-masked]> wrote:
> 
> In message <[email-masked]>, mark seery writes:
> > David Allan wrote:
> > 
> That was never the case.
> Each OSPF node did not have to know whether the downstream node did
> ECMP and if it did whether it was a dumb per packet split or a hash
> based, or something else.  It only had to know that the next node
> downstream kept the packet going in the direction of lower OSPF cost.

Curtis, I respect your opinion and experience, but it seems to me you keep referring back to ECMP in a native IP environment where everyone is using the same hashing algorithm.

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 17 Nov 2003 15:31:11 -0500

That was never the case.

> it should be standardized.

There is plenty of existance proof that contradicts the assertion in
your last paragraph.

Curtis

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 16:04:30 -0500 (EST)

> There is plenty of existance proof that contradicts the assertion in

As soon as I sent this I knew the statement was too strong. My concern is that if two routers in the same network think the same packet is part of two different microflows because they use different algorithms, then undesireable results could emerge. I will add no more on this subject if it is the consensus of the group that either:

a) this is incorrect from a technical perspective
b) only known practical issues are worthy of discussion
c) this is not the time or place for this discussion

thanks,
mark

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 17 Nov 2003 16:05:25 -0500

In message <[email-masked]>, mark seery writes:
> Curtis, I respect your opinion and experience, but it seems to me you
> keep referring back to ECMP in a native IP environment where everyone
> is using the same hashing algorithm.

That is not true.  Early on Cisco used per packet load splitting and
did not use a hash based algorithm.  It is still not certain what hash
algorithm anyone is using for those that do a hash.  It also does not
matter whether the hash is based on src/dst only or also includes the
TCP port numbers for TCP.

Therefore it is not "one of the fundamental precepts of routing" that
each node understand the ECMP algorithms used by the other nodes.
Each router must behave within some constraints required to keep the
routing protocol loop free.

Curtis

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 16:11:13 -0500 (EST)

> each node understand the ECMP algorithms used by the other nodes.

Curtis, it is grossly misleading to restate what I said in that way. I find that a little disappointing. But I accept your point that there is an existence proof that different algorithms can work in the same network. We will see if this holds true when multi-protocol traffic is routed between different vendors as well, perhaps as you say it will.

Mark.

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 17 Nov 2003 16:17:37 -0500

In message <[email-masked]>, mark seery writes:
> As soon as I sent this I knew the statement was too strong. My concern is tha
> t if two routers in the same network think the same packet is part of two dif
> ferent microflows because they use different algorithms, then undesireable re
> sults could emerge. I will add no more on this subject if it is the consensus
>  of the group that either:
> 

If two routers in a path do a split (happens in IGP, LDP or heirarchy
in RSVP/TE) and use an identical hash, then the downstream split will
always see microflows filtered by the first split.  If both are 2:1
splits, then the second becomes a NOOP because all of the packets
falling into one side of the hash have already been removed.

It is therefore a requirement that the hash be seeded independently by
each node in the topology.  This hash seeding is usually random with
input from addressing and things such as time of day in fine
granularity (usec or nsec) or other pseudorandom seed.  No node knows
how any other nodes have seeded the hash.  Nothing breaks as a result
of this.

[That would be a) above.]

I think enough has been said about Dave's "fundamental precept of
routing".  Next topic please.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 17 Nov 2003 16:47:16 -0500

In message <[email-masked]>, mark seery writes:
> Curtis, it is grossly misleading to restate what I said in that way. I find t
> hat a little disappointing. But I accept your point that there is an existenc
> e proof that different algorithms can work in the same network. We will see i
> f this holds true when multi-protocol traffic is routed between different ven
> dors as well, perhaps as you say it will.
> 
> Mark.

Multiprotocol traffic is already being routed in multivendor networks
with ECMP.  It isn't pretty, but looking for a valid IP header and
falling back to per label split works.

All that is required is that each node recognize that traffic which
contains identifiable microflows, doesn't split traffic that does not
contain identifiable microflows, and that for traffic containing
identifiable microflows any given microflow takes a single path.
The PW PID formalizes the means to determine the traffic type.

Curtis

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 17:16:38 -0500 (EST)

Curtis, I appreciate the time you have taken to fill in the holes.

Some last comments below.

Mark

> identifiable microflows any given microflow takes a single path.

That makes a great deal of sense when there is an agreed upon mechanism to do this,........

> The PW PID formalizes the means to determine the traffic type.

....and this is the agreed upon mechanism, thanks. I had not made the leap that all MPLS traffic that wanted to make use of ECMP was now going to be PW-based; the lack of a PID being the key historical reason why I had trouble with this.

thanks again.
mark

---

From: Alia Atlas <[email-masked]>
Date: Mon, 17 Nov 2003 17:27:23 -0500

At 04:04 PM 11/17/2003, mark seery wrote:
>As soon as I sent this I knew the statement was too strong. My concern is 
>that if two routers in the same network think the same packet is part of 
>two different microflows because they use different algorithms, then 
>undesireable results could emerge. I will add no more on this subject if 
>it is the consensus of the group that either:

It is in fact extremely desirable for two routers to think of micro-flows 
differently, as long as they don't break up a micro-flow from the 
end-system perspective.  If two routers classify all packets identically 
into a set of micro-flows, then if router A sends micro-flow 1 to router B, 
router B can only send it on as a single micro-flow to one of its primary 
neighbors, instead of breaking it into different micro-flows and sending 
different micro-flows to each of B's primary neighbors.

Alia

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 18:55:12 -0500 (EST)

> different micro-flows to each of B's primary neighbors.

Hi Alia, that is a very good point and I understand what you are saying. My only point would be that the classification mechanism would ideally be consistent, leading to not only "as long as they don't break up a micro-flow from the end-system perspective" but also from a network perspective, i.e. in a way that no loops form, or even simply in such a way that traffic flows according to expectations. I am satisfied that the use of PID, as Curtis described, is constructive in reaching this goal.

My apologies to the group if this was wasted bandwidth.

Thanks,
Mark

---

From: Kireeti Kompella <[email-masked]>
Date: Mon, 17 Nov 2003 16:04:42 -0800 (PST)

Hi Mark,

(Please note change of subject.)

On Mon, 17 Nov 2003, mark seery wrote:

> understands the routing decisions each other node makes,

I don't agree.  For a simple example: in RIP, a router has no idea
what the next hop will do.  Another (before y'all jump on me for
saying that RIP is a routing protocol) is BGP.  Link state protocols
are different -- in a sense, they give you too much information.

> this is to
> prevent black holing among other things.

This is the job of routing protocols, not individual routers.

> So for example, the SPF
> algorithm is not a secret,

Actually, the SPF *algorithm* can be a secret.  The result (a shortest
path dag) is not a secret.  It can be (and has been) proven that if
each router computes a spdag based on the same topology info, then no
black holes arise.

> As ECMP algorithms impact the forwarding
> this case.

Different ECMP algorithms have happily co-existed in networks for a
long time now.  The algorithms can be different, but there must be a
consisten result; in this case, packets in a 'flow', whatever that is,
must not be reordered.

There are both IP and MPLS networks with multi-vendor equipment with
ECMP and they work just fine.

> to do ECMP

Whole 'nuther ball game.  Very very hard question.  If I had an
answer, I sure wouldn't share it :-)

> algorithm is known by both.

See above.

This is easy to see if you consider that an ECMP path is a set of
choices in the data plane that *could* correctly have been made
instead in the control plane; and that the set of choices is stable
for a given packet flow.

Kireeti.
-------

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 19:43:12 -0500 (EST)

Kireeti,

> (Please note change of subject.)

Noted - good idea, should have been done earlier, agreed.
> are different -- in a sense, they give you too much information.

Oh yeah, thanks for reminding me of the conistency of topology knowledge in those other protocols ;-) (clearly a case of sarcasm).

Seriously, if I am wrong that OSPF and IS-IS assume every router in an area are using the same algorithm, then would appreciate being corrected. That path and distance vector algorithms advertise routes and select best routes based on policy etc. is different than the case when a common algorithm is being used. In those cases (well certainly in BGPs) there are mechanisms to prevent looping that are added to the protocol - it just doesn't happen, you have to do something to prevent it. If your point is that you could build a non algorithmic network then I concede your point. If your point is that you can have a non-algorithmic solution in a network that is already heavily algorithmically based, then I guess I would probably need some further discussion with you.

> This is the job of routing protocols, not individual routers.

I think we would have a semantic debate if I responded to this, so let's not.

> each router computes a spdag based on the same topology info, then no

Don't understand how the same topology would lead to the same result if there was not some common assumptions about how to get there. I understand at the implementation level you might want to do incremental stuff or do something innovative at the coding level, but not sure if that means that Dijkstra's isn't still the fundamental essence of what you are doing (in the case of OSPF or IS-IS). If you can point me to some reading on the use of different algorithms to achieve the same result I would appreciate it.

> ECMP and they work just fine.

That is a good data point, thanks. Was there any interaction between the respective engineering teams to get it working?

> answer, I sure wouldn't share it :-)

Answer assumed ;-) You've worked with some of them so be careful now ;-)

Additionally, if it is true that at some time in history Juniper and Cisco chose a different hash, then it might not be as obvious as your playful comment suggests. Also at the time there was no existence of a PID. Since the existence of a PID has since been argued to be useful......

> instead in the control plane; and that the set of choices is stable

You know there is nothing in this paragraph that I disagree which makes me wonder what the discussion is about. I obviously was missing an understanding on how to achieve "..the set of choices is stable for a given packet flow". I am not sure if it is a standard yet, but at least I understand the fundamentals better.

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 17 Nov 2003 18:27:15 -0800

Mark,

> Oh yeah, thanks for reminding me of the conistency of topology knowledge in 
> those other protocols ;-) (clearly a case of sarcasm).
> 
> Seriously, if I am wrong that OSPF and IS-IS assume every router
> in an area are using the same algorithm, then would appreciate
> being corrected. 

As Kireeti mentioned before, both OSPF and ISIS assume that each
router computes shortest path (wrt OSPF/ISIS metric) from itself
to all other nodes in the network. Neither OSPF nor ISIS require a
*specific* single source all destinations shortest path algorithm to
be used for such computation.  There is *more than one* algorithm
to compute single source all destinations shortest path (see below
for the list of books that describe some of these algorithms).

> That path and distance vector algorithms advertise
> routes and select best routes based on policy etc. is different
> than the case when a common algorithm is being used. In those cases
> (well certainly in BGPs) there are mechanisms to prevent looping
> that are added to the protocol - it just doesn't happen, you have
> to do something to prevent it. If your point is that you could
> build a non algorithmic network then I concede your point. If your
> point is that you can have a non-algorithmic solution in a network
> that is already heavily algorithmically based, then I guess I would
> probably need some further discussion with you.

I think Kireeti's point is that assuming that there is one and only
one single source all destinations shortest path algorithm doesn't
match current reality.

> > > this is to prevent black holing among other things.
> > 
> > > So for example, the SPF algorithm is not a secret,
> > 
> Don't understand how the same topology would lead to the same result
> if there was not some common assumptions about how to get there. I
> understand at the implementation level you might want to do
> incremental stuff or do something innovative at the coding level,
> but not sure if that means that Dijkstra's isn't still the fundamental
> essence of what you are doing (in the case of OSPF or IS-IS).
> If you can point me to some reading on the use of different algorithms to 
> achieve the same result I would appreciate it.

Here are few books that describe various shortest path algorithms 
(with Dijksta being one of them):

  1. "Network Flows" by Ahuja, Magnanti, Orlin.
  2. "Graphs and Algorithms" by Gondran and Minoux
  3. "Linear Network Optimization" by Bertsekas.

Yakov.

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 21:35:59 -0500 (EST)

>   3. "Linear Network Optimization" by Bertsekas.

Yakov, thanks for the reading material and the other corrections you made to my comments. I am hopeful one of these books will discuss using two different algorithms in the same network (though I understand the essence of Curtis's point that as long as you are moving the ball forward you have done your job).

Mark

---

From: Kireeti Kompella <[email-masked]>
Date: Mon, 17 Nov 2003 19:25:07 -0800 (PST)

Hi Mark,

> Oh yeah, thanks for reminding me of the conistency of topology knowledge in those other protocols ;-) (clearly a case of sarcasm).

No sarcasm intended.  There have been proposals for less than full
topology exchange that still achieve consistent routing.

> Seriously, if I am wrong that OSPF and IS-IS assume every router in
> an area are using the same algorithm, then would appreciate being
> corrected.

Vendor 1 uses vanilla Dijkstra, vendor 2 uses incremental SPF.  Very
different algorithms.

One thing you may want to spend some cycles on: router A computes
shortest paths *from A to all others*.  Router B computes shortest
paths *from B to all others*.  Router B doesn't compute what A will do
with B's packets.  Why should this not result in routing loops?

Hint: see Curtis's mail about decreasing metrics.

> That is a good data point, thanks. Was there any interaction between the respective engineering teams to get it working?

None.

> Additionally, if it is true that at some time in history Juniper
> and Cisco chose a different hash, then it might not be as obvious as
> your playful comment suggests. Also at the time there was no
> existence of a PID. Since the existence of a PID has since been
> argued to be useful......

I don't know anything whatsoever about cisco's hash, and don't have
the least interest (from a 'will routing break?' point-of-view).  I am
willing to bet that J and C (and A) have very different hash
functions.  It Just Doesn't Matter (tm).

> You know there is nothing in this paragraph that I disagree which
> makes me wonder what the discussion is about. I obviously was
> missing an understanding on how to achieve "..the set of choices is
> stable for a given packet flow". I am not sure if it is a standard
> yet, but at least I understand the fundamentals better.

"Stable" means that a given router doesn't change either its hash
algorithm or the fields it's hashing on.  This is very different from
saying that all routers must use the same hash algorithm or hash fields.

Kireeti.
-------

---

From: mark seery <[email-masked]>
Date: Mon, 17 Nov 2003 23:12:47 -0500 (EST)

Hi Kireeti,

> topology exchange that still achieve consistent routing.

Without seeing the proposals I can only assume that if a router has a known set of possibilities for getting to a known point, it doesn't need to know all the junk after that - just that it is headed in the right direction. But a bit of a meaningless discussion unless I know to what you refer.

Do I trust you have given this some thought and study, and are a more advanced student on this subject that I, yes - clearly. 

> Vendor 1 uses vanilla Dijkstra, vendor 2 uses incremental SPF.  Very

But do they both have the same definition of what a shortest path is? Which is really the point I was trying to make at the beginning of this. We can argue about what an algorithm is (of course we can assert there is only one definition which wouldn't be entirely out of character for this group) or we can argue about whether some some common basis of defining a shortest path is understood. We know routing loops occur when two different routers have a different understanding of the topology. I guess the question I am asking is if two different algorithms have a different concept about what shortest path means, does that not mean in essence they have a different view of the topology?
 
> Hint: see Curtis's mail about decreasing metrics

I am not sure I understand the premise. Router B is in a sense suggesting to itself that it knows what some other router (ahead of it in the path) will do. Isn't that an inference of understanding the shortest path?

But I will hunt down Curtis's email and give it some more thought. Thanks for the brain teaser.

> snipped <

If this is true, then why has the use of a PID been suggested to bring in to play at least some commonality. Now it has been suggested to me offline that some of my comments are really only relevant in a highly multi-protocol world, and that world does not exist in the real world. If this is one of the causes of the disconnect, I understand where people are coming from.

> saying that all routers must use the same hash algorithm or hash fields.

So I agree there is difference. But just like if two algorithms have a different definition of shortest path my brain says that is a problem, my brain says (yes I am hearing voices at this point) if a network has multiple definitions of what a microflow is, then that is a problem. If you are all say you are doing ECMP with Ethernet, IP, CES and alike and it is not causing you any problems (without a PID), then there is nothing more to be said. Proof of something working is obviously more powerful than theory (that old running code thing I guess).

Mark

---

From: Naiming Shen <[email-masked]>
Date: Mon, 17 Nov 2003 20:47:23 -0800

ECMP does not affect the result of SPF for IGP. Actually in some of the
IGP implementation, a router only announce a single adjacency in LSA even
though there are multiple ECMP adjacencies exist to the same neighbor.
This completely relies on the fact that ECMP info is only relevant to
local router. The ECMP info is needed only in TE application, where
bandwidth of each link is involved and remote router's CSPF needs to know.

cheers.

 ]Hi Kireeti,
 ]
 ]---- Kireeti Kompella <[email-masked]> wrote:
 ]> > Oh yeah, thanks for reminding me of the conistency of topology knowledge 
in those other protocols ;-) (clearly a case of sarcasm).
 ]> 
 ]> No sarcasm intended.  There have been proposals for less than full
 ]> topology exchange that still achieve consistent routing.
 ]
 ]Without seeing the proposals I can only assume that if a router has a known s
et of possibilities for getting to a known point, it doesn't need to know all t
he junk after that - just that it is headed in the right direction. But a bit o
f a meaningless discussion unless I know to what you refer.
 ]
 ]Do I trust you have given this some thought and study, and are a more advance
d student on this subject that I, yes - clearly. 
 ]
 ]> > Seriously, if I am wrong that OSPF and IS-IS assume every router in
 ]> > an area are using the same algorithm, then would appreciate being
 ]> > corrected.
 ]> 
 ]> Vendor 1 uses vanilla Dijkstra, vendor 2 uses incremental SPF.  Very
 ]> different algorithms.
 ]
 ]But do they both have the same definition of what a shortest path is? Which i
s really the point I was trying to make at the beginning of this. We can argue 
about what an algorithm is (of course we can assert there is only one definitio
n which wouldn't be entirely out of character for this group) or we can argue a
bout whether some some common basis of defining a shortest path is understood. 
We know routing loops occur when two different routers have a different underst
anding of the topology. I guess the question I am asking is if two different al
gorithms have a different concept about what shortest path means, does that not
 mean in essence they have a different view of the topology?
 ] 
 ]> One thing you may want to spend some cycles on: router A computes
 ]> shortest paths *from A to all others*.  Router B computes shortest
 ]> paths *from B to all others*.  Router B doesn't compute what A will do
 ]> with B's packets.  Why should this not result in routing loops?
 ]> 
 ]> Hint: see Curtis's mail about decreasing metrics
 ]
 ]I am not sure I understand the premise. Router B is in a sense suggesting to 
itself that it knows what some other router (ahead of it in the path) will do. 
Isn't that an inference of understanding the shortest path?
 ]
 ]But I will hunt down Curtis's email and give it some more thought. Thanks for
 the brain teaser.
 ]
 ]> snipped <
 ]> I don't know anything whatsoever about cisco's hash, and don't have
 ]> the least interest (from a 'will routing break?' point-of-view).  I am
 ]> willing to bet that J and C (and A) have very different hash
 ]> functions.  It Just Doesn't Matter (tm).
 ]
 ]If this is true, then why has the use of a PID been suggested to bring in to 
play at least some commonality. Now it has been suggested to me offline that so
me of my comments are really only relevant in a highly multi-protocol world, an
d that world does not exist in the real world. If this is one of the causes of 
the disconnect, I understand where people are coming from.
 ]
 ]> "Stable" means that a given router doesn't change either its hash
 ]> algorithm or the fields it's hashing on.  This is very different from
 ]> saying that all routers must use the same hash algorithm or hash fields.
 ]
 ]So I agree there is difference. But just like if two algorithms have a differ
ent definition of shortest path my brain says that is a problem, my brain says 
(yes I am hearing voices at this point) if a network has multiple definitions o
f what a microflow is, then that is a problem. If you are all say you are doing
 ECMP with Ethernet, IP, CES and alike and it is not causing you any problems (
without a PID), then there is nothing more to be said. Proof of something worki
ng is obviously more powerful than theory (that old running code thing I guess)
.
 ]
 ]Mark

- Naiming

---

From: mark seery <[email-masked]>
Date: Tue, 18 Nov 2003 00:06:26 -0500 (EST)

> bandwidth of each link is involved and remote router's CSPF needs to know.

Hi Naiming. Thanks for the info about ECMP and TE, that is interesting, and it is in the context of a single network using both LDP and TE that those voices in my head start getting louder ;-)

As for a router that has multiple links to the same neighbor, that is not the kind of topological construct I am thinking of when I discuss ECMP, though understand why you mention it.

Thanks again.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 00:44:02 -0500

In message <[email-masked]>, mark seery writes:
> If this is true, then why has the use of a PID been suggested to bring in to 
> play at least some commonality. Now it has been suggested to me offline that 
> some of my comments are really only relevant in a highly multi-protocol world
> , and that world does not exist in the real world. If this is one of the caus
> es of the disconnect, I understand where people are coming from.

Strictly speaking, a microflow is a stream of data which cannot be
reordered (without some sort of bad things happenning).  For IP a
microflow can be identified by source and destination address and TCP
(or UDP) port numbers for TCP and UDP.

If a hash is performed on an IP source and destination address
(src/dst addr), and the path selected from a set of N choices using
modulo N (simple and common technique) then each path will have a set
of microflows, generally a very large set.  Any one microflow will
take only one path and therefore packets for that microflow will not
be reordered.

If you don't know that traffic is IP, then it is not generally safe to
try to split it up.  An LSP carrying the data stream for a single
leased line cannot be load split because reordering the packets would
have dire consequences.  When this type of LSP and some IP LSPs are
put inside another LSP (hierarchical LSPs) then there are two labels
in the label stack.  In this case, it is not possible for a midpoint
to tell by looking at the labels which packets are IP and which are
not.  Load split can be done on the inner label alone (the outer label
is constant) but the quality of the load split is generally
unsatisfactory.

It is desireable to be able to load split the IP traffic based on the
IP src/dst.  If the ISP knows that any non-IP traffic will never have
what looks like an IP header in the first four bytes, the ISP can
enable ECMP in which the first four bytes are examined to see if the
packet is IP, and labels are used for the split if not, and the IP
src/dst is used for the split if it is IP.  If the vast majority of
the traffic is IP, then this yields a high quality load split.

The PW PID just provides a means to insure that for PW the above
condition is true (it insures that no non-IP traffic will have the
first four bytes that look like an IP header).  That is the only link
between PW and this discussion.

I made some comment in private email to Mark about multiprotocol.  A
lot of ISP/carriers have in their MPLS traffic IP, usually VOIP (also
IP), maybe IP VPN (also IP), and maybe FR, ATM, and TDM.  ECMP is
guarenteed to not scramble microflows if the FR, ATM, and TDM go
inside PW with PW PID (or otherwise acquire a 4 byte shim that doesn't
look like an IP header in the first four bytes or first one byte which
always contains the version number, 4 or 6).  For multiprotocol in the
real world (of wide area networking) this is about it.  OSI, X.25,
ISDN, Novell, Appletalk, Netbios, SNA, DECNET, etc, just plain don't
exist in the core.  Yes, I did mention OSI in that list, but if an ISP
needs OSI for some dribble of control packets for some archane reason
(or any of the others), then they can enscapsulate it in IP to keep
its packet ordering intact in the presence of ECMP.

Curtis

---

From: mark seery <[email-masked]>
Date: Tue, 18 Nov 2003 01:02:18 -0500 (EST)

---- Curtis Villamizar <[email-masked]> wrote:
> <snipped to end>

Thanks Curtis, I appreciate you taking the time to explain it, especially if you have had to before.

Mark

---

From: Peter Willis <[email-masked]>
Date: Tue, 18 Nov 2003 09:50:57 +0000

If we don't document OAM issues with MPLS in the OAM framework document where 
do they get documented? 

I think the precedent for discussing Issues with MPLS in MPLS WG framework
 documents is established in RFC3353.

Regards the "*practical*" point I can see why the MPLS WG has problems with 
the top down nature of draft-allan-mpls-oam-frmwk-05.txt. Perhaps the usual 
modus operandi of documenting implementations and waiting for customers to
request features to fix operational problems works much better?

Peter.

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 18 Nov 2003 07:58:31 -0500

>If we don't document OAM issues with MPLS in the OAM framework 
>document where do they get documented? 

	Peter,

	No one is arguing that you should not have your
say in documenting your issues with MPLS. It has been
suggested numerous times now that this be put into
an informational draft.  

	--Tom

>I think the precedent for discussing Issues with MPLS in MPLS 
>WG framework documents is established in RFC3353.
>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 11:27:30 -0500

In message <[email-masked]>, Peter Willis wri
> request features to fix operational problems works much better?

An informational RFC on P2MP has nothing to do with this.  There was
very limited interest in P2MP and no one considered that framework to
be harmful.  If good concrete work comes of it that would be welcome.

If there are aspects of the architecture that impacts some methods of
OAM (and clearly there is), it should be covered.  Neither the method
of OAM should be called deficient, nor the architecture, it just
becomes an applicability issue.

There are some people who see the need for monitoring in networks that
have ECMP and have no intention of removing ECMP.  These are real
networks and therefore this is a real problem.  Some methods of
monitoring will work others won't.  That is an issue of applicability.

Some people (you, Neil, Dave, others) prefer aspects of one approach.
Some people prefer the broader applicability of other approaches.
These are tradeoffs.  If we accept these as tradeoffs, then we can
move forward.  If some people keep insisting that we need a document
that states that the architecture is broken we can make no progress.

Curtis

---

From: Peter Willis <[email-masked]>
Date: Tue, 18 Nov 2003 17:30:27 +0000

> becomes an applicability issue.

I'm happy with that approach.

Peter.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 18 Nov 2003 12:58:36 -0500

In message <[email-masked]>, Peter Willis wri

Peter,

Finally some compromise.

Thanks,

Curtis