Subject: Support for multicast in the ForCES protocol

From: Steven Blake <[email-masked]>
Date: Mon, 17 Nov 2003 16:53:46 -0500

The issue of support for multicast transport in the ForCES protocol has
been a recurring topic of dispute between the competing protocol teams.
Multicast is not explicitly called out as a protocol requirement in
draft-ietf-forces-requirements-10.txt, and it is not mentioned in the
protocol evaluation draft.  However, there are at least some people
(myself included) who believe that it is a pretty useful feature in some
system environments where ForCES could be applicable.

Rather than make an argument for or against multicast support (some
people on this list have already heard my opinion anyway), I am
wondering if there are any folks not directly involved in the protocol
teams who have an opinion either pro or con?

Apologies if I am exhuming a dead horse.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: Sandy Pavel <[email-masked]>
Date: Tue, 18 Nov 2003 10:07:56 -0500

Hi there!

I haven't followed this thread of discussions and therefore I apologize
if
I am repeating something that has been mentioned before.

First of all, I must mention that I am totally with Steven and the
others that ask for multicast support in the ForCES protocol. If
necessary, in order to convince people we can come up with a number of
examples and real world cases that would justify it.

I am not sure if it is assumed or not but I think we should add
broadcast next to multicast.

However, these would be transport technology options rather than ForCES
protocol message types. That is, we should leave the option of binding
different ForCES protocol messages to either unicast, multicast or
broadcast transport, as being platform (NE) specific. This is because
different implementations might have different resources available at
transport and physical level.

The question is: do we add some sort of "multicast group management" as
part of ForCES control protocol, or leave it to the transport layer
itself?

To conclude, considering the switching fabric and other interconnection
technologies proposed in the last few years especially for distributed,
multi-shelf core routers, I believe the only viable option is to include
multicast/broadcast as transport options for ForCES protocol(s).

Regard,
Sandy Pavel

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Steven Blake
Sent: November 17, 2003 4:54 PM
To: [email-masked]
Subject: Support for multicast in the ForCES protocol

Regards,

---

From: Alia Atlas <[email-masked]>
Date: Tue, 18 Nov 2003 10:36:36 -0500

Hi,

I'd also like to chime in on the importance of having multicast transport
in the FORCES protocol.  Just the idea of trying to download common updated
forwarding tables to dozens or more FEs individually and to do so extremely
quickly is disturbing.  As Sandy said, there are interconnection
technologies to allow the packet replication/multicast distribution.  It is
necessary to have multicast to enable scaling to large numbers of FEs and
non-trivial forwarding information (IPv4 prefixes, global ILM entries,
etc).  Unfortunately, multicast transport is not something which can be
efficiently layered on top of the FORCES protocol; it needs to be built in.

I've just started following FORCES, so I'm sorry if this is redundant.

Alia Atlas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Nov 2003 14:38:03 -0500

I find myself very torn on this issue.
So I am going to try to show my thinking in hopes that someone can either
point out aspects I am missing, or point out why certain aspects are more
important than I tend to treat them.

It would be nice to simply say "multicast is for later".
However, it is equally clear that if we are going to use multicast it is
much better to support it as an integral part of the protocol suite.  This
directly effects core protocol features.

At the same time, my gut feel is that the complexity is not worth the
value.  But that is not a good reason for anyone else to make a
decision.  It is not even a good reason for me to make a decision.

I think some of this has to do with rates of change of information vs
easily supportable load.

Clearly, service configuration does not change often enough to require
multicast.
So the question becomes what rate of change of next-hops are we likely to
need to support, what number of FEs per CE are we trying to communicate
those changes to, and is the resultant load likely to be such that we will
want multicast.
Clearly, in many environments the answer is no.  But that is not sufficient
to say we should not support it.
Equally clearly, there are some environments that we would like to support
eventually that will push the limits for unicast distribution.
But is that margin of applicability enough to justify the complexity.

Yours,
Joel

---

From: "T. Sridhar" <[email-masked]>
Date: Wed, 19 Nov 2003 12:04:05 -0800

Hi,

I am sure I am missing something but if multicasting is only done below the
ForCES protocol level (e.g UDP over a multicast delivery mechanism), what
are we giving up? Other than a group addressing scheme, what else do we need
to add to the ForCES protocol itself?

This does impose a group addressing and multicast delivery requirement on
the underlying transport but that is not an unreasonable requirement, I
feel.

Thanks,
Sridhar

----------------------------------------------
From: Forwarding and Control Element Separation
[mailto:[email-masked]]On Behalf Of Joel M. Halpern
Sent: Wednesday, November 19, 2003 11:38 AM
To: [email-masked]
Subject: Re: Support for multicast in the ForCES protocol

Yours,
Joel

---

From: John Renwick <[email-masked]>
Date: Wed, 19 Nov 2003 14:38:19 -0600

Joel,

> [stuff deleted, all of which I agree with]
>

As you say: somewhere, somebody will inevitably come up with a set of
numbers for link capacity, utilization, updates per second and number of
FEs that will force a requirement for flooding.  For instance, link
capacity 10000 transactions per second, 10% maximum utilization, 10
updates per second, and 100 FEs.  The problem is similar to the database
synchronization that link state protocols do, and in that environment
flooding is the accepted mechanism.  Why not use it in ForCES as well,
since it's something we're all comfortable with?

--
John Renwick                          Agere Systems
Software Technical Manager            1230 Northland Drive
Phone: (651) 675-2986                 Mendota Heights, MN 55120
FAX:   (707) 897-7928                 Cell:  (612) 669-9397
mailto:[email-masked]             http://www.agere.com

---

From: Sandy Pavel <[email-masked]>
Date: Wed, 19 Nov 2003 16:07:08 -0500

I agree. The ForCES protocol should not reinvent the wheel by
re-implementing a broadcast/multicast protocol. More, the actual
multicast mechanism would be technology dependent.

As mentioned before, having the _option_ in ForCES to _choose_ the
transport mechanism (and in the case of multicast, to specify the
multicast group), is at least in principle, most of what we need to do.
It might sound simplistic but we should also try to consider the
complexity and more importantly, the layer separation in the protocol
stack that concerns ForCES.

Static multicast group membership sounds to me as being acceptable and
therefore there is no need to have a group membership management
protocol.

Functionally the multicast/broadcast is not necessary; it is merely a
mechanism that allows for efficient use of the transport technology and
_scalable_ router architectures.

This is especially valid in core router space but less important in
enterprise.

Regards,
Sandy

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of T. Sridhar
Sent: November 19, 2003 3:04 PM
To: [email-masked]
Subject: Re: Support for multicast in the ForCES protocol

Hi,

I am sure I am missing something but if multicasting is only done below
the
ForCES protocol level (e.g UDP over a multicast delivery mechanism),
what
are we giving up? Other than a group addressing scheme, what else do we
need
to add to the ForCES protocol itself?

This does impose a group addressing and multicast delivery requirement
on
the underlying transport but that is not an unreasonable requirement, I
feel.

Thanks,
Sridhar

I find myself very torn on this issue.
So I am going to try to show my thinking in hopes that someone can
either
point out aspects I am missing, or point out why certain aspects are
more
important than I tend to treat them.

It would be nice to simply say "multicast is for later".
However, it is equally clear that if we are going to use multicast it is
much better to support it as an integral part of the protocol suite.
This
directly effects core protocol features.

Clearly, service configuration does not change often enough to require
multicast.
So the question becomes what rate of change of next-hops are we likely
to
need to support, what number of FEs per CE are we trying to communicate
those changes to, and is the resultant load likely to be such that we
will
want multicast.
Clearly, in many environments the answer is no.  But that is not
sufficient
to say we should not support it.
Equally clearly, there are some environments that we would like to
support
eventually that will push the limits for unicast distribution.
But is that margin of applicability enough to justify the complexity.

Yours,
Joel

---

From: Alan DeKok <[email-masked]>
Date: Wed, 19 Nov 2003 16:58:53 -0500

> therefore there is no need to have a group membership management

  Multicast issues can be addressed via administrative issues above
ForCES.

  That is, assuming that there *is* a multicast-capable transport under
ForCES, a "multicast" update can simple appear to ForCES to be another
FE.  The system administrator would be responsible for maintenance of
multicast group membership.

  This approach can avoid a lot of the issues surrounding multicast in
ForCES, but may not avoid all of them.

> This is especially valid in core router space but less important in

  It's also important for latency, which may be an issue.

  Alan DeKok.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 19 Nov 2003 17:25:26 -0500

Just in case people have missed it; netlink2 uses multicast
by simply running over UDP; UDP also supports broadcast. The multicast
groups or wires in netlink2 speak are dynamically discovered.
The CEM-FEM interface makes the call which bundles/wires to join.
There is absolutely no need for netlink2 (or a forces protocol) to be
aware of this.
As a fallback multicast or broadcast may be simulated via
multi-unicast-sockets; but then thats innefficient because you end up
using more resources (RAM, CPU and bandwidth) to achieve the same end
goal.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 19 Nov 2003 17:29:05 -0500

I think i am agreeing with you - not sure;->
Any design that puts an upper limits is doomed. What may be deemed an
upper limit in this hour may be obsoleted in the next.

cheers,
jamal

---

From: Zsolt Haraszti <[email-masked]>
Date: Wed, 19 Nov 2003 18:49:23 -0500

> multicast group membership.

It is important to keep in mind that the target audience of
potential multicast configuration updates are a set of LFBs of
identical type across one or more FEs, and not the FEs
themselves.  For example, just because the same LPM table must
be loaded into two FEs', it does not mean that all other
LFBs of the two FEs will also be programmed identically.  In
fact it is more probable that they won't.

To be even more precise, the entities that need to be addressed
by the same config information may not be even full LFBs, but
rather some attributes of (e.g., tables) of these LFBs.

It appears to me that using a layered approach, in which the ForCES
protocol simply relies on multicast support in the lower layer, does
not address the whole problem, since that only addresses the
efficient distribution to a group of FEs and not a group of LFBs
across these FEs.  (Am I wrong?)  That may be fine, provided that
the rest of the problem can also be solved.  But I am not sure it
can be without support from the ForCES protocol itself.

To explain what I mean by target audience of multicast config
requests, consider the following (small-scale) example.

CE---+---> FE1 ---+---> LFB1(*)
     |            +---> LFB2(*)
     |            +---> LFB3(LPM)--+--> prefixtab1 [VRF1]
     |            |                +--> prefixtab2
     |            +---> LFB4(*)
     |            +---> LFB5(LPM)--+--> prefixtab1
     |            |                +--> prefixtab2 [VRF1]
     |            :
     |
     +---> FE2 ---+---> LFB1(*)
     |            +---> LFB2(*)
     |            :
     |
     +---> FE3 ---+---> LFB1(LPM)-----> prefixtab1 [VRF1]
     |            +---> LFB2(*)
     |            +---> LFB3(LPM)--+--> prefixtab1
     |            |                +--> prefixtab2
     :            :                +--> prefixtab3 [VRF1]

The CE controls a number of FEs, each has a set of LFBs (the
FEs may not have the same LFB topology). LFB(LPM) denotes
LPM prefix lookup LFBs (which for simplicity  are instances
of the same LFB class). LFB(*) denotes other types
of LFBs.  In a VRF capable router the LPM LFBs may contain
multiple LPM tables, and the number of LPM tables may be
different from one LFB to the other (as indicated in the
example).  Suppose that the following tables are created to
provide LPM for the same VRF (e.g., VRF1 as marked in the
figure):

FE1 : LFB3 : prefixtab1
FE1 : LFB5 : prefixtab2
FE3 : LFB1 : prefixtab1
FE3 : LFB3 : prefixtab3

In a large system where there is a large number of such tables,
one should use the ForCES model in a clever way such that these
tables are kept identical (not only the prefixes, which are
obvious, but also the next hop identifiers associated with
each prefix to address next hop entries in another LFB).  If
the tables are identical, then the exact same methods can be
and should be sent to all of them to maintain the tables, and
that is a case for multicast support.  In other words,
the above list of tables are the real and final target of the
multicast messages.

I am not sure what part of the problem would the underlying
multicast layer solve, but I think it is only from the CE to
the FEs (which is useful, but not sufficient).  If so, there
must be additional mechanisms to set up the "rest of the tree"
on each FE, and I don't see how you can do that without explicit
support from the ForCES protocol.  What is needed is that the CE
should be able to say: "I want to have
FE1 : LFB3 : prefixtab1
FE1 : LFB5 : prefixtab2
FE3 : LFB1 : prefixtab1
FE3 : LFB3 : prefixtab3
to form a muticast group, accessible by a fictive address
FEx : LFBy : prefixtab_z".  To implement that the address
triplet must be registered at the CE, a CE-to-FE multicast
tree must be established (if one with identical membership
does not exist yet), and then a table must be set up at
the FE (and possibly even in each involved LFB).

My main point, again, is that CE-to-FE multicast is useless
unless the full tree can be set up (the leaves are tables within
LFBs), and I suspect the ForCES protocol will need to pitch in
to address the second and perhaps the third layer of the tree
(the third can actually be solved within the LFB-specific protocol
data, instead of the base protocol).

My $0.02.

Zsolt Haraszti

---

From: Robert Haas <[email-masked]>
Date: Thu, 20 Nov 2003 12:00:44 +0100

Zsolt,

This "second-level mutlicast" you mention is also captured in the 
netlink2 grouping mechanism that permits to create groups of LFBs on the 
same/different FEs. For simplicity, we assumed that group membership is 
statically configured at pre-association. If you remember, this is what 
I tried to illustrate in Vienna on pages 8 and 11 of  my presentation:
http://www.zurich.ibm.com/~rha/netlink2-1.pdf

Another key characterisitic in netlink2 is that one can address LFB by 
type, i.e., "I want to send a routing table to all LFBs of the 
IPv4_ROUTING type". That's a great simplification as opposed to having 
to find out the IDs assigned to each individual LFB.

Cheers,
-Robert

Zsolt Haraszti wrote:

>>  That is, assuming that there *is* a multicast-capable transport
>>    
>>
>under
>  
>

-- 
Robert Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: Patrick Droz <[email-masked]>
Date: Thu, 20 Nov 2003 13:46:57 +0100

Joel,

at least some time back I have heard from people that they
intended to build boxes with more than 2000 ports and about
100K routes. In such a case multicast is very helpful to
make ForCES scalable. I do not know if people still have
such can of plans due to the downturn but I guess it makes
sense to have ForCES future proof.

Regards,
Patrick

-- 
   Dr. Patrick Droz 			| [email-masked]
   IBM Zurich Research Laboratory 	| http://www.zurich.ibm.com/~dro
   Saumerstrasse 4 			| Tel. +41-1-724-85-25 CH-8803
   Rueschlikon/Switzerland 		| Fax. +41-1-724-85-78

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 Nov 2003 09:20:44 -0500

I believe people have said this.  However...

While the route*port product affects startup time, I would be reluctant to
include multicast for just startup.  The question is what the rate of
change of the routes is.

Remember also that what we deal with in ForCES are FIB entries, not RIB
entries.  Even in the internet core there are rarely 100K FIB entries.

Given that we are separating data and control concepts, one could argue
that the ratio "should" get arbitrarily large.  One can equally argue that
the ratio "should" be kept bounded by deploying additional control entities
as needed.   We have not established any agreement in the working group as
to what target we are after.  That does not mean that the 2K port full
internet table device is out of scope.  It also does not mean it is in scope.

Let me put this another way...
In order to use multicast, we are going to have to define our own
reliability / security /congestion control mechanisms (we will be able to
use work from other working groups, but those do not appear to be sufficient.)
That is a large effort for the working group.
We are unlikely to declare multicast a :must implement".  But doing a lot
of work on an optional aspect seems counter to the goal of effective
useable rapid progress.

So I am trying to understand what problem we NEED to solve that multicast
is important for.

Yours,
Joel M. Halpern

---

From: Zsolt Haraszti <[email-masked]>
Date: Thu, 20 Nov 2003 09:59:47 -0500

Rob,

On Thu, 2003-11-20 at 06:00, Robert Haas wrote:
> same/different FEs.

Sorry, my ignorance.

> For simplicity, we assumed that group membership is
> statically configured at pre-association.

If we allow dynamic topology changes in post association phase
(and we decided to do so in the model team), then forming multicast
groups should also be supported in post-association phase.

> If you remember, this is what

That is a nice feature for some specific cases/systems.

/zsolt

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 Nov 2003 10:11:16 -0500

I think that addressing by type is likely to be at best of limitted
use.  There are very few LFB types that can be changed in parallel.

People tend to assume that the "forwarding table" has this property.  Even
when the forwarding table is disjoint from the next hop table (a good idea
in its own right) there are many uses of a "forwarding table LFB" which
will not be populated with the same information.  For example, one might
use the same LFB type for policy forwarding as for routing protocol derived
forwarding.

Thus, to allow multicast to be used for routing protocol derived FIB
propagation one needs to be more specific than merely "update the
forwarding table LFB".  We can create mechanisms where for the few LFBs
whose contents are consistent there is a way to address them
consistently.  But type based addressing is not sufficient.

Yours,
Joel

PS: It has been suggested in passing that multicast is good for backup
FEs.  For the kinds of backup I think ForCES will want, just having the CE
install the necessary state (either in advance or when needed, as the
system chooses to operate) is sufficient and MUCH simpler.

PPS: Yes, I am finding myself seeing more complications and less benefits
from multicast as this conversation continues.  However, I will restrict
the rate of my comments as I do not want to accidentally discourage others
from speaking.  I could easily be wrong.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 20 Nov 2003 11:03:01 -0500

On Thu, 2003-11-20 at 10:11, Joel M. Halpern wrote:
> will not be populated with the same information.

I think thats a detail which belongs to the model.There will be
different types of IPV4 LFBs - some requiring the clear demarcation you
make.
We are armed with the NE topology (how the FEs are interconnected and
their respective topologies) - you could easily translate what the
nexthop means. In both case there is value to address "all IPV4
forwarding LFBs".
OTOH, neighbor mappings are a disjoint. You may have meant this later
part.

>  For example, one might

So in these case you are even more corase than the finer granular level
of using a separate addressing to the nexthop table.

> consistently.  But type based addressing is not sufficient.

Its semantics: You could still have an "all IPV4" address. Protocol
details dont diminish its need.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 20 Nov 2003 11:13:45 -0500

I dont think it is a startup challenge issue only. The challenge applies
to all applications that build state based on packet arrival events. And
there are many.
Think of an IDS under a DOS. Or a flow tracking firewall system.
100s of FEs, millions of flows (say a 100K new ones per second).
You certainly wanna distribute that info to the FEs.
Unless you wanna rule out such apps for ForCES.

I think it is a design flaw to say "we may need it to scale in the
future or when we have slower hardware but lets forget ir right now".

cheers,
jamal