Subject: Re: LFB topology reconfguration

From: "Yang, Lily L" <[email-masked]>
Date: Thu, 6 Nov 2003 14:33:57 -0800

Seems like most people agree that micro-change within one given service (protocol as Alan put it) does not result in topology change. so no argument there.

Now lets look at the macro-change which invovles change of service ("service provisiong" as Joel called it).

For example, suppose FE0 supports IPv4 forwarding, DiffServ and MPLS. But the operator only wants to use IPv4 at the beginning. Well, the FE can still be modeled by a topology that includes all the services it is capable of. So the CE can model the FE by a topology like this (or, FE can just report its topology to CE):
+------------------+
|classifier V  out1|-----> IPv4 path (subgraph)
|                  out2|-----> DiffServ path (subgraph)
|                  out3|-----> MPLS path (subgraph)
+------------------+
Note that each path by itself is a subgraph potentially involving multiple LFBs and multiple datapath.

Now, "classifier V" is only a virtual classifier. FE0 may not have a classifier at all. But we use "classifier V" to model FE0's ability to support multiple services. Now at the begining since only IPv4 is used, CE can configure "classifier V" such that packets all go to out1 (IPv4 path). 

Later (like a month later), the operator want to use MPLS instead, they can configure classifier V such that packets now flow to out3.

Another month later, if they want to use both IPv4 and DiffServ, sure.
But the topology remains static.

Now, what if the operator wants to do IPv6, but FE0 does not support that to begin with. Well, the operator either buys another FE (FE1) that support IPv6, or he takes FE0 down and loads up the code to do IPv6, plug it in, and start all over again.

Remember, all we do here is just modeling the "logical" bahavior of FE with a bunch of "logical" LFBs. I can't see why we can't model the service provisioning this way, without the need to reconfigure LFB topology.

Lily

> -----Original Message-----
> From: Forwarding and Control Element Separation
> [mailto:[email-masked]]On Behalf Of Alan DeKok
> Sent: Thursday, November 06, 2003 7:33 AM
> To: [email-masked]
> Subject: Re: LFB topology reconfguration
> 
> 
> "Joel M. Halpern" wrote:
> >
> > My guess is that
> > 1) To assign a subset of customer traffic to an existing 
> DiffServ service
> > will not require LFB reconfiguration on almost any FE.
> ...
> 
>   I would tend to agree.
> 
>   For me, the boundary between internal changes to an LFB's
> configuration, and changes to inter-LFB connections is most often a
> protocol boundary.  Provisioning new connections/services within a
> protocol requires updating entries in the LFB's which implement that
> protocol.  Provisioning a new protocol requires re-arranging LFB's, or
> creating new LFB's.
> 
>   Alan DeKok.
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 6 Nov 2003 17:49:20 -0500

I would argue that there is no such thing as "the diffserv path".  Diffserv
is a mechanism for offering a number of services.  While one can have an FE
that has a fixed toplogy of classifiers, markers, meters, queues and
schedulers, many FEs will have some flexibility in that topology.
It is quite likely that even if one could envision the topology for all
possible services, there are not enough LFBs to offer them all
simultaneously.  So the operator picks the ones he is offering now, and
uses some of the resources to offer those.  As he decides to offer more
services, he provisions them on the FE (via the CE) and makes them
available to customers.

I do not know of an operator who can determine at box installation time all
the services he will offer from that platform.
Equally, I do not know any operator who will be happy being told that new
service offerings will requrie that he disable the box.
And there seems to be no point in having some other protocol for
extablishing these.

Let me put it another way.
Someone has to tell the FE what that initially topology ought to be.  It
may be fixed by the factory.
If it is not fixed by the factory, it is clearly the CEs job to set it up.
It is clearly desirable to be able to change that setup during operation.
If the FE can support such a change, it is clearly the CEs job to make such
a change.

And it is not possible to enumerate all possible graphs, or all possible
services.

Yours,
Joel M. Halpern

At 02:33 PM 11/6/2003 -0800, you wrote:
>Seems like most people agree that micro-change within one given service
>(protocol as Alan put it) does not result in topology change. so no
>argument there.
>
>Now lets look at the macro-change which invovles change of service
>("service provisiong" as Joel called it).
>
>For example, suppose FE0 supports IPv4 forwarding, DiffServ and MPLS. But
>the operator only wants to use IPv4 at the beginning. Well, the FE can
>still be modeled by a topology that includes all the services it is
>capable of. So the CE can model the FE by a topology like this (or, FE can
>just report its topology to CE):
>+------------------+
>Now, "classifier V" is only a virtual classifier. FE0 may not have a
>classifier at all. But we use "classifier V" to model FE0's ability to
>support multiple services. Now at the begining since only IPv4 is used, CE
>can configure "classifier V" such that packets all go to out1 (IPv4 path).
>
>Later (like a month later), the operator want to use MPLS instead, they
>can configure classifier V such that packets now flow to out3.
>
>Now, what if the operator wants to do IPv6, but FE0 does not support that
>to begin with. Well, the operator either buys another FE (FE1) that
>support IPv6, or he takes FE0 down and loads up the code to do IPv6, plug
>it in, and start all over again.
>
>Remember, all we do here is just modeling the "logical" bahavior of FE
>with a bunch of "logical" LFBs. I can't see why we can't model the service
>provisioning this way, without the need to reconfigure LFB topology.
>
>Lily
>
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 6 Nov 2003 20:40:13 -0500

I believe we can represent enough about the capabilities of the FE to be
useful.  I would certainly agree that it would be a rare FE that could
support any and all topologies possible from the LFBs it supports.
I believe that using information similar to what is in the Diffserv PIB we
can create a useful set of FE level and LFB level capability information.
(That is, at the FE level there would be information such as whether the
topology can be changed, and which LFB types can follow which other LFB
types, via which ports.  At the LFB level there would be information about
how many of this LFB type can be  in existence, how large the arrays of
information within this LFB type for classifiers may be, etc.)
It is true that this will not perfectly describe all capability information.
That is why the document calls this a compromise.

Yours,
Joel

At 05:25 PM 11/6/2003 -0800, you wrote:
>It seems like what is being proposed here is for the case of a very
>generic FE that contains a lot of different kinds of LFBs, and the
>services that can be supported depend on the topology configured for
>those LFBs (or various subsets of the LFBs). In this case, I can see
>where expressing all of the possible topologies might not be feasible.
>
>However, in the case where all possible (or even reasonable) topologies
>for all of the services cannot be expressed, the operator must be aware
>of what the FE is capable of supporting and how to configure the FE to
>provide the desired service. The operator may not be able to depend
>solely on the FE protocol to determine its capabilities and the
>configuration that is required in order to support such a capability. It
>seems like the operator may need to use some vendor-supplied tools to
>add a service or new capability to a service. In this case, I'm not sure
>how much sense it makes to have a generic protocol.
>
>Regards,
>Ellen
>
>[mailto:[email-masked]] On Behalf Of Joel M. Halpern
>Sent: Thursday, November 06, 2003 2:49 PM
> >Note that each path by itself is a subgraph potentially involving
>multiple
> >LFBs and multiple datapath.
> >

---

From: "Yang, Lily L" <[email-masked]>
Date: Fri, 7 Nov 2003 09:54:04 -0800

I guess that largely depends on how you model LFB and the topology to start with. 
To model the LFB topology, we recommend using a combination of topological approach and encoded state approach in the FE model dradf -v01:
"The topological approach should primarily be used when the packet datapath forks into areas with distinct LFB classes (not just distinct parameterizations of the same LFB classes), and when the fan-outs do not require changes (adding/removing LFB outputs) at all or require only very infrequent changes.  Configuration information that needs to change frequently should preferably be expressed by the internal attributes of one or more LFBs (and hence using the encoded state approach). (Section 3.3.1)"

The granularity of LFB also matters. The finer grain the LFBs are, the more likely you run into dynamic topology change. 

But all in all, I still think setting up a new RSVP connection does not result in topology change. It seems adding a new entry in the LFB table would do it. 

Maybe if you draw out the topology example you have in your mind, that would certainly help.

Isn't it great that we have all these warm-up exercises on FE model before going to the IETF meeting next week? For those of you who will be at Minneapolis, lets make sure we continue this discussion face to face and (hopefully) reach some consensus.

> [mailto:[email-masked]]On Behalf Of Weiming Wang
> Sent: Thursday, November 06, 2003 5:53 PM
> Subject: Re: FW: LFB topology reconfguration
> 
> 
> Hi Lily,
> 
> I aslo would like to argue that even micro-change  in a same 
> servie may result in
> LFB and its connections change. Alex takes the QoS levels, I 
> just want to take a RSVP connection
> setup as a consideration. When we deploy resources for a new 
> RSVP connection, can we really do it only 
> just by changing some LFB attributes?  I think current 
> all-in-one routers actually have already made such
> resources dynamic deployments, which, when reflects into 
> ForCES architeture, are LFB changes. Actually I 
> think such changes are not as awful to do as we might think,  
> especially when the resources are already 
> managed by a center network processors. 
> 
> Regards,
> 
> Weiming
> 
> ----- Original Message ----- 
> From: "Alex Audu" <[email-masked]>
> To: <[email-masked]>
> Sent: Friday, November 07, 2003 7:14 AM
> Subject: Re: FW: LFB topology reconfguration
> 
> 
> > Hi Lily,
> > 
> > Not only service change from A to B, but also, service 
> level maintenance within
> > some constraints falls within bounds in my view. For 
> example, one may want
> > to maintain a certain level of QoS which may require some 
> sort of feedback
> > process to dynamically bring in more resources (LFBs) to 
> make up for increased
> > demands, or loss of resources in some other NE.  This sort 
> of change may need
> > to take place much more often than the once-in-a-while 
> service change.
> > 
> > Regards,
> > Alex.
> > 
> > "Yang, Lily L" wrote:
> > 
> > > Oops ... forgot to cc to the list. Too late into the night :-)
> > >
> > > > From:         Yang, Lily L
> > > > Sent: Wednesday, November 05, 2003 6:42 PM
> > > > To:   'Zsolt Haraszti'; Khosravi, Hormuzd M; 'Joel M. 
> Halpern'; Deleganes, Ellen M; 'Jamal Hadi Salim'
> > > > Subject:      LFB topology reconfguration
> > > >
> > > > Some clarifications:
> > > > 1) LFB topology (the "graph") reprensents state, not capability.
> > > > 2) There are different ways to represent topology 
> capability. One way is to list all the possible graphs and 
> name them up front. Then to reconfigure the LFB topology is 
> to switch between the different graphs. Another way is to 
> describe the linkage constraint (like LFB A can only be 
> followed by LFB B and C but not D).  We can argue which is 
> more flexible, efficient etc later.
> > > >
> > > > But I still have some questions of my own.
> > > > 1) How feasible & frequent it really is for an FE to 
> change from service A to B at runtime? I heard a few people 
> said "yes, we do that today". I would really like to hear a 
> few concrete examples and understand what is involved at the 
> FE and CE to accomplish that. I don't consider recompiling 
> codes as dynamic run time configuration.
> > > > The reason that I am asking this is that I don't want 
> to end up designing some feature that is cool but nobody can 
> really uses in  the products.
> > > >
> > > > 2) Is providing ability to switch between graphs 
> flexible enough for topology reconfiguration? I realize it is 
> limited. But is it good enough?
> > > >
> > > > 3) Do we really want the ultimate flexibility to 
> rearrange LFB by adding (instantiating) new LFB instances, 
> deleting LFB instances, or re-connecting two LFBs (by 
> deleting and then adding connections)?
> > > > Again, we can do this in ForCES, but only if people 
> really see the need/benefit of it.
> > > >
> > > > 4) I imagine before you change service on FE from A to 
> B, you probably have to stop sending packets to this FE all 
> together if you don't want to have unpredictable behavior or 
> loss of traffic temporarily. So it is almost equivalent to 
> take down an FE and plug in a new one later, isn't it?
> > > >
> > > > 5) For those of you who are invovled in NPF -- Is NPF 
> supporting dynamic LFB topology? We want to align ForCES and 
> NPF to the same direction.
> > > >
> > > > Lily
> > 
>

---

From: "Deleganes, Ellen M" <[email-masked]>
Date: Fri, 7 Nov 2003 13:54:10 -0800

Alan,

Thanks. I think that it will be helpful to understand where that "middle
ground" is and to have an idea of the bounds of flexibility that the
protocol is expected to support. It makes sense to try to narrow in on
the amount of flexibility that we expect the protocol and model to
support by discussing real applications and usage models.

Regards,
Ellen

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Alan DeKok
Sent: Friday, November 07, 2003 10:14 AM
To: [email-masked]
Subject: Re: LFB topology reconfguration

> where expressing all of the possible topologies might not be feasible.

  Some vendors systems are highly programmable (e.g. software), and thus
can do "anything".  It's difficult for those vendors to see why they
should restrict their solutions to being described in LFBs.  It's also
difficult for them to see why they need to describe their solutions,
because they can do "anything".

  At the other extreme, we have vendors with dedicated (and limited)
hardware.  Their systems are too simple to implement any ForCES
protocol, and can be easily described without resorting to LFB's.

  ForCES allows everyone to meet in the middle.  This means that most
vendors have to perform additional work to support ForCES, without clear
and immediate benefits.  Much of the conversations (and concerns) on the
list have been about agreeing on what, exactly, those benefits are.

  Alan DeKok.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 8 Nov 2003 01:11:11 -0500

Let me propose a very simple example system.
One has a classical Unix system with dynamic streams drivers.
One has a whole collection of streams modules available.
One can decide to add a policer / shaper to the streams stack to change the
service one is delivering.
It would seem that if such a machine were an FE being controlled by a CE,
one would want the CE to be able to give such an instruction.
One should not have had to anticipate the particular combination one wants
to deploy, and create that graph as an alternative in order to be able to
invoke it.

Yours,
Joel M. Halpern

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 8 Nov 2003 09:31:55 -0500

My own guess is that one would typically not need to reconfigure LFBs for a
new RSVP flow.  (I would not be surprised if most FEs that could support
reconfiguration could not support that rate of change.)

To try to fill that in a little bit, I will sketch LFBs that I think one
might use to support RSVP, and then an example of a different case where I
think one might need to change the LFB topology.

For RSVP support, I would expect that one would have a high capacity
classifier applying metadata marking indicating the bandwidth that was
accepted.
This would feed to a policer that would measure the rate per metadata
marked flow, and compare with the accepted rate.  Then all RSVP traffic
would go into a high priority queue.
If one were implementing RSVP support without any metadata support at all
one would either need to omit policing or one would indeed need to
reconfigure the LFBs to have a policer for each accepted flow.

A different case where LFB modification is almost certainly necessary would
come up when a service provider has an FE offering conventional IP
forwarding at access line rate.  Suppose that he then decides to offer a
reduced rate policed service (where packets over a contracted rate are
dropped).  It would be necessary to add a policer into the data path.  The
provider could not reasonably have known beforehand that he needed such an
element in his data path.  (Obviously he can only make this transition if
the FEs he is using support the change.)

Yours,
Joel M. Halpern

At 09:54 AM 11/7/2003 -0800, you wrote:
>To model the LFB topology, we recommend using a combination of topological
>approach and encoded state approach in the FE model dradf -v01:
>"The topological approach should primarily be used when the packet
>datapath forks into areas with distinct LFB classes (not just distinct
>parameterizations of the same LFB classes), and when the fan-outs do not
>require changes (adding/removing LFB outputs) at all or require only very
>infrequent changes.  Configuration information that needs to change
>frequently should preferably be expressed by the internal attributes of
>one or more LFBs (and hence using the encoded state approach). (Section 3.3.1)"
>
>The granularity of LFB also matters. The finer grain the LFBs are, the
>more likely you run into dynamic topology change.
>
>But all in all, I still think setting up a new RSVP connection does not
>result in topology change. It seems adding a new entry in the LFB table
>would do it.
>
>Isn't it great that we have all these warm-up exercises on FE model before
>going to the IETF meeting next week? For those of you who will be at
>Minneapolis, lets make sure we continue this discussion face to face and
>(hopefully) reach some consensus.
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 8 Nov 2003 09:40:55 -0500

many (not all) vendors support on the fly modification of router configs in
ways that change the delivered service.
hence, from an operators perspective he should not need to take down a
service delivery element (an FE) in order to provision new services.
Yes, with some vendors and some devices he will have to.
But the protocol ought to recognize that the more dynamic situation is both
common and desirable.

Yours,
Joel

At 03:58 PM 11/7/2003 -0800, you wrote:
>My impression is that the state of the art engineering approach
>is to bring down blades one at a time to upgrade their
>firmware/software/hardware functions, using active/passive or
>other HA mechanisms to retain non-stop forwarding in the face of
>upgrades.  In this sort of approach there is indeed FE LFB
>topology change,  but that change occurs while an FE is offline
>(and thus not during the post-association phase).

---

From: "Putzolu, David" <[email-masked]>
Date: Sat, 8 Nov 2003 23:50:15 -0800

[chair hat still off]
Joel Halpern wrote>
> many (not all) vendors support on the fly modification of 
> router configs in ways that change the delivered service.
> But the protocol ought to recognize that the more dynamic 
> situation is both common and desirable.

Hi Joel.

I agree that the delivered service a FE is configured for can 
changes at run time.  My question is more along the lines of 
whether such changes can only be supported by topology changes.  
In general I think a good design principle for a protocol or 
model is "just enough complexity and no more".  I think having 
dynamic topology is more complex than static topology, so I 
want to better understand where a static approach falls down.

Imagine a FE that has IPv4 and DiffServ features, and is represented
by a static LFB graph as shown below (obviously a real
graph would have a lot more LFBs, both in terms of additional
functions and in terms of higher granularity):

Ingress --> IPv4 forwarder --> DiffServ --> Egress

The operator might initially not be using differentiated services.
However, the FE, being capable of DiffServ, would expose the LFB
graph shown above.  The DiffServ LFB in this case would simply
be configured to treat all traffic identically @ best effort,
with no differentiated treatment of packets going on.

At some point in time the operator wants to change the delivered 
service.  To do so, the operator would use some means of configuring
the CE (e.g. telnet in and type in some CLI commands).  In response
to this, the CE would use ForCES, touching DiffServ LFB above 
by pushing additional configuration information into it (and
likely to other FEs in the NE as well).  The mirroring example
previously discussed is similar - a FE capable of mirroring will
have some known set of points where mirroring can occur.  Rather
than dynamically instantiating LFBs, why not just expose the exact
set of points where such mirroring can occur, since it is built
into the FE?

I did take a look at http://www.cyberus.ca/~hadi/patches/action/README
(the example Jamal provided).  The hooks appeared to be well known/
static locations in the packet processing sequence, so I think they 
fit the static model.  I assume the pipes capability mentioned is
where dynamic ordering of functions can occur.  However, the main
application I saw was what appeared to be a hierarchical CBQ 
functionality - which I thought there was already some consensus 
around being supportable via a static approach?

The blade down/upgrade blade/blade up I mentioned is in the
context of adding entirely new functions to a FE that where
not present before.  In that case I agree that topology change
is necessary - but my impression is that is a different case
than the one being suggested above.

Thanks,
David

> Sent: Saturday, November 08, 2003 6:41 AM
> Subject: Re: [FORCES] LFB topology reconfguration
> 
> 
> 
> Yours,
> Joel
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 9 Nov 2003 11:04:08 -0500

The issue is that there is no such thing as the "diffserv" graph
segment.  There are multiple different graphs that one can use to provide
diffserv baseed services (there also is not just one diffserv service).  If
there were only one such graph, one could expect it to pre-exist.
I agree with the constraint that the LFBs must pre-exist.  The creation of
new LFBs types is clearly outside our scope.

If we want to declare topologies changes to be outside our scope, we are
essentially declaring that all flexible service delivery will be meta-data
based.  I would prefer not to place such a constraint on our work.

Another area where it becomes clear that one needs some dynamics is when
one delves into complex queueing / scheduling disciplines.  On the surface
it appears that a single scheduler with multiple disciplines pulling from a
collection of queues can cover all that is needed.  (After all, that is
what most routers actually have.)  But if you try to model some of the
complex schedulers out there (Xedia is a good example) one needs multiple
levels of schedulers, and sometimes multiple levels of queues.  Hence, if
the topology is static then I am stuck with the queueing services I picked
at rollout (probably one level) rather than being able to add more
complexity later.

Another issue example is tunnelling.  I could easily install a box with
support for multiple tunneling protocols.  I might initially be using none
of them.  It could easily have constraints that it supports 6 different
kinds of tunnels, but can only use up to 4 of them in any config (internal
resource constraint).  I am then required to guess which 4 I will want
during my service evolution, or can I roll out with the one I need now, and
adjust the topology to add services as I go forward.

Another aspect here is consistent simplicity.  Many flexible FEs will come
up with a blank topology.  We could declare that setting the topology up to
match the CEs needs is outside the scope of the protocol.  But that will
significantly reduce the CE - FE interoperability.

I well understand that to get the work done we need to reduce scope.  But
unless we really are saying that the only important scope is the IP
forwarding table, I think removing topology change from scope will
seriously restrict our capabilities.

Yours,
Joel M. Halpern

---

From: "(Ram Gopal )" <[email-masked]>
Date: Mon, 10 Nov 2003 11:02:35 -0500

Hello Lily,

Comments are inline.

> [mailto:[email-masked]]On Behalf Of ext Yang, Lily L
> Sent: Wednesday, November 05, 2003 9:54 PM
> Subject: FW: LFB topology reconfguration
> 
> 
> > 1) How feasible & frequent it really is for an FE to change 
> from service A to B at runtime? I heard a few people said 
> "yes, we do that today". I would really like to hear a few 
> concrete examples and understand what is involved at the FE 
> and CE to accomplish that. I don't consider recompiling codes 
> as dynamic run time configuration.  

Let me try to provide in two parts.

1)
For example consider a wireless edge providing basic 3 services like gaming, video-on-demand and browsing. Operator provision this service and the rate of change of such service is very slow.   But there may be several thousands of users (subscribers) subscring to this service.  But operator don't have an exact number of  users  (subscriber)  going to use a particular service. When a user connects to the wireless depending upon the user's subscription the LFB instances in the FE are configured.  

It may happen that at a given instance that all the users subscribed are subscribed to the same service and connected via same FE will have one topology (processing path).

But if users are subscribed to various services and connected to the same FE then there may one or more processing path( LFB topology).  

Most of the subscriber provision has either time-out or some for teardown associated with it and changing from Service A to B   may require deleting or modifying existing LFB instances. To summarize,

- We can have situation where user(s) are subscribed to only one service. (say only IPv4 traffic) 
- It may also be possible that where the users are subscribed to more than one services. ( say IPv4 and IPv6 traffic)
In the later case, you may need to program two sets of LFB instances to achieve different processing (service) path.

2)

NPE has bunch of LFB, NPE should provide capabilities to manipulate LFB interconnection via LFB configuration. If this is not the case even if we LFBs in the but doesn't support such reorganization of stage processing we will not be able to provide the required service.
 
> > The reason that I am asking this is that I don't want to 
> end up designing some feature that is cool but nobody can 
> > 2) Is providing ability to switch between graphs flexible 
> enough for topology reconfiguration? I realize it is limited. 
> But is it good enough?
> > 
> > 3) Do we really want the ultimate flexibility to rearrange 
> LFB by adding (instantiating) new LFB instances, deleting LFB 
> instances, or re-connecting two LFBs (by deleting and then 
> adding connections)?
> > Again, we can do this in ForCES, but only if people really 
> see the need/benefit of it.

NPE may support static or semi-static or fully dynamic LFB topology. IMHO we have very little flexibility with current NPE's fully dynamic re-connecting LFB looks like beyond the scope of LFB. In the case of fully-reconnecting LFB we may need  like  topology discover etc because LFB's can be programmed in any manner by CE.  

May be we should restrict ourselves to only semi-dynamic topology where CE after getting  FE and LFB capaiblity, configures instances of LFB to accomudate new services (processing path).

Since we are working in FE model (software) and we can keep extending our model to any level. 

We should ask ourselves following question.
Does current NPEs supports fully dynamic LFB configuration, if so what extent?

I would suggest let David P  take this with NPF as we have liaison relationship and he can provide better answer.

End of my comment

Regards
Ramg

> > 
> > 4) I imagine before you change service on FE from A to B, 
> you probably have to stop sending packets to this FE all