Subject: Re: Support for multicast in the ForCES protocol

From: Tal Anker <[email-masked]>
Date: Tue, 18 Nov 2003 10:00:50 +0200

Well, 
I would vote strongly for multicast support... I think that it would obviously boost the performance of the CE (number of configurations passed to FEs per unit of time, etc). The main issue with the usage of multicast here is to deal correctly with the reliability of a "transaction" made between the CE and a set of FEs. Given that this is doable (and it obviously is by using well known building blocks) my belief is that not using multicat will probably degrade the overall router/switch performance when many FE's are involved...

my 2c$.

- Tal

-----Original Message-----
From: Steven Blake [mailto:[email-masked]]
Sent: Monday, November 17, 2003 11:54 PM
To: [email-masked]
Subject: Support for multicast in the ForCES protocol

The issue of support for multicast transport in the ForCES protocol has
been a recurring topic of dispute between the competing protocol teams.
Multicast is not explicitly called out as a protocol requirement in
draft-ietf-forces-requirements-10.txt, and it is not mentioned in the
protocol evaluation draft.  However, there are at least some people
(myself included) who believe that it is a pretty useful feature in some
system environments where ForCES could be applicable.

Rather than make an argument for or against multicast support (some
people on this list have already heard my opinion anyway), I am
wondering if there are any folks not directly involved in the protocol
teams who have an opinion either pro or con?

Apologies if I am exhuming a dead horse.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: Ligang Dong <[email-masked]>
Date: Wed, 19 Nov 2003 07:29:58 -0500

hi,
Yes, the multicast will improve the performance. I agree with Sandy Pavel
about "first broadcast, next multicast". In fact, the GRMP (draft-wang-
forces-grmp-00.txt) have already considered broadcast. By a careful design
about the CE/FE identifier, it is not difficult to support multicast in
GRMP.

ligang

On Tue, 18 Nov 2003 10:36:36 -0500, Alia Atlas <[email-masked]> wrote:

>Hi,
>
>I'd also like to chime in on the importance of having multicast transport
>in the FORCES protocol.  Just the idea of trying to download common updated
>forwarding tables to dozens or more FEs individually and to do so extremely
>quickly is disturbing.  As Sandy said, there are interconnection
>technologies to allow the packet replication/multicast distribution.  It is
>necessary to have multicast to enable scaling to large numbers of FEs and
>non-trivial forwarding information (IPv4 prefixes, global ILM entries,
>etc).  Unfortunately, multicast transport is not something which can be
>efficiently layered on top of the FORCES protocol; it needs to be built in.
>
>I've just started following FORCES, so I'm sorry if this is redundant.
>
>Alia Atlas
>
>
>At 10:07 AM 11/18/2003, Sandy Pavel wrote:
>>Hi there!
>>
>>I haven't followed this thread of discussions and therefore I apologize
>>if
>>I am repeating something that has been mentioned before.
>>
>>First of all, I must mention that I am totally with Steven and the
>>others that ask for multicast support in the ForCES protocol. If
>>necessary, in order to convince people we can come up with a number of
>>examples and real world cases that would justify it.
>>
>>I am not sure if it is assumed or not but I think we should add
>>broadcast next to multicast.
>>
>>However, these would be transport technology options rather than ForCES
>>protocol message types. That is, we should leave the option of binding
>>different ForCES protocol messages to either unicast, multicast or
>>broadcast transport, as being platform (NE) specific. This is because
>>different implementations might have different resources available at
>>transport and physical level.
>>
>>The question is: do we add some sort of "multicast group management" as
>>part of ForCES control protocol, or leave it to the transport layer
>>itself?
>>
>>To conclude, considering the switching fabric and other interconnection
>>technologies proposed in the last few years especially for distributed,
>>multi-shelf core routers, I believe the only viable option is to include
>>multicast/broadcast as transport options for ForCES protocol(s).
>>
>>Regard,
>>Sandy Pavel
>>
>>
>>
>>From: Forwarding and Control Element Separation
>>[mailto:[email-masked]] On Behalf Of Steven Blake
>>Sent: November 17, 2003 4:54 PM

---

From: Tal Anker <[email-masked]>
Date: Thu, 20 Nov 2003 02:13:36 +0200

I aggree (poor choice of wording in my behalf in the original mail). The reliability of the transport layer would probably help to simplify the application protocol. The transaction commit status is indeed a part of the application responsibility... any thoughts on the model there (2 phase/3 phase commit, etc)? There might be some transactions, when applied to a group of FEs that can be regarded as "commited" on a all-or-none basis...
 
- Tal

	-----Original Message----- 
	From: Steven Blake [mailto:[email-masked]] 
	Sent: Thu 11/20/2003 2:03 AM 
	To: [email-masked] 
	Cc: 
	Subject: Re: Support for multicast in the ForCES protocol

	On Wed, 2003-11-19 at 18:52, Tal Anker wrote:
	
	> One last issue... the semantics needed from the transport layer should be
	> carefully defined, as regular multicast is not reliable, and I expect the
	> protocol to rely on some reliable transport in order to know the CE-FE
	> transaction is commited... the RMT WG had defined tools for reliable multicast
	> that can be deployed here...
	
	This is actually orthogonal to multicast transport (reliable or
	otherwise).  The fact that a reliable transport delivered a command
	message from a CE to a FE is insufficient to indicate to the CE that the
	command can and has been executed successfully by the FE.  So the ForCES
	protocol will need to support application-layer acknowledgements of some
	variety.

	Regards,

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 20 Nov 2003 13:57:26 -0800

Zsolt,

So are you suggesting a combination of unicast and multicast to solve
the efficiency problem ?
That's interesting, and wasn't apparent to me in your previous email.

BTW, the assumption that I made is exactly the same assumption that I
thought you are also
making regarding LFBs sharing the same information across FEs :-) (This
is what I was
referring to in the beginning of my email)

Any thoughts on how to address the "command bundling" issue in a
multicast approach ?

Thanks
Hormuzd
-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Zsolt Haraszti
Sent: Thursday, November 20, 2003 12:53 PM
To: [email-masked]
Subject: Re: [FORCES] Support for multicast in the ForCES protocol

Hormuzd,

See my comment inline.

On Thu, 2003-11-20 at 14:19, Khosravi, Hormuzd M wrote:
> Hi Zsolt,
>
> I think you have brought up a very good point,
> although based on some assumptions.
> (For now lets consider the assumptions are true.)
> I have been resisting getting into this thread since it
> was meant for folks "not directly involved with protocols".
> However I don't see that as the case any more :)
>
> In any case, what you are suggesting is that in order
> To use multicast efficiently, you will need to have this
> Tree in which multicast groups are assigned FE:LFB basis.
> (I will try to keep things simple for now by assuming 1 table per LFB)
> I agree its an efficient way to use multicast.
> Now what this means is that, just in terms of no. of
> connections from CE-to-FE, the ratio of unicast:multicast,
> is the ratio of -> sum of Distinct LFBs per FE (M): no of FEs (N).
>
> So in case of a medium size Edge network element with 10 FEs
> but 15 distinct LFBs (since it has to handle many diff protocols),
> This ratio is 15:10, which means that there will be more multicast
> Groups than unicast connections ! This clearly has worse performance
> than unicast.
>

You make an assumption that does not hold: you assume that all
15 LFBS will share information with their counterpart in all
other FEs.  That is unlikely the case in practice.  In fact
only a very small number of LFBs will share information (though
they may be the ones updated most frequently with large amount
of information).  A more realistic numerical example would be
when 2 out of the 15 LFBs are part of multicast trees.  So in
this case I would have 10 unicast connections for normal LFB
configuration, and 2 multicast trees.

/zsolt

> However, in case of a core router with 100 FEs and 10 distinct LFBs,
> This ration is 10:1, which means less multicast connections but not
> By a huge ratio.
>
> Now, this brings another question to mind... as to how  "command
> Bundling" will be supported with this concept of multicast groups.
> Say I want to update entries in 3 LFBs (in 2 FEs) which belong to
> different groups. Using multicast groups, you can send 3 messages
> On the 3 groups but how do you bundle that in an elegant way ?
> One way which I can think of is to bundle all 3 messages in a single
> Message and send it on one multicast connection. This does not
> Seem to be very elegant and there is no performance gain that I can
> think of.
> (Instead of sending 3, x size msgs, you sent 1 3x size msg)
> The only other way is to use a 2-phase commit in which you will land
> Up sending 6 messages, 3 for request and 3 to commit. This will
> definitely
> Be bad in terms of performance, etc and doesn't even meet the command
> Bundling requirement in spirit (single message).
>
> Let me know if you can think of an elegant, efficient way of doing
this
> ?
> Thanks
> Hormuzd
> [mailto:[email-masked]] On Behalf Of Zsolt Haraszti
> Sent: Wednesday, November 19, 2003 3:49 PM
> Subject: Re: [FORCES] Support for multicast in the ForCES protocol
>
>
> On Wed, 2003-11-19 at 16:58, Alan DeKok wrote:
> >   That is, assuming that there *is* a multicast-capable transport
> under
> > ForCES, a "multicast" update can simple appear to ForCES to be
another
> > FE.  The system administrator would be responsible for maintenance
of
> > multicast group membership.
>
> It is important to keep in mind that the target audience of
> potential multicast configuration updates are a set of LFBs of
> identical type across one or more FEs, and not the FEs
> themselves.  For example, just because the same LPM table must
> be loaded into two FEs', it does not mean that all other
> LFBs of the two FEs will also be programmed identically.  In
> fact it is more probable that they won't.
>
> To be even more precise, the entities that need to be addressed
> by the same config information may not be even full LFBs, but
> rather some attributes of (e.g., tables) of these LFBs.
>
> It appears to me that using a layered approach, in which the ForCES
> protocol simply relies on multicast support in the lower layer, does
> not address the whole problem, since that only addresses the
> efficient distribution to a group of FEs and not a group of LFBs
> across these FEs.  (Am I wrong?)  That may be fine, provided that
> the rest of the problem can also be solved.  But I am not sure it
> can be without support from the ForCES protocol itself.
>
> To explain what I mean by target audience of multicast config
> requests, consider the following (small-scale) example.
>
> CE---+---> FE1 ---+---> LFB1(*)
>      |            +---> LFB2(*)
>      |            +---> LFB3(LPM)--+--> prefixtab1 [VRF1]
>      |            |                +--> prefixtab2
>      |            +---> LFB4(*)
>      |            +---> LFB5(LPM)--+--> prefixtab1
>      |            |                +--> prefixtab2 [VRF1]
>      |            :
>      |
>      +---> FE2 ---+---> LFB1(*)
>      |            +---> LFB2(*)
>      |            :
>      |
>      +---> FE3 ---+---> LFB1(LPM)-----> prefixtab1 [VRF1]
>      |            +---> LFB2(*)
>      |            +---> LFB3(LPM)--+--> prefixtab1
>      |            |                +--> prefixtab2
>      :            :                +--> prefixtab3 [VRF1]
>
>
> The CE controls a number of FEs, each has a set of LFBs (the
> FEs may not have the same LFB topology). LFB(LPM) denotes
> LPM prefix lookup LFBs (which for simplicity  are instances
> of the same LFB class). LFB(*) denotes other types
> of LFBs.  In a VRF capable router the LPM LFBs may contain
> multiple LPM tables, and the number of LPM tables may be
> different from one LFB to the other (as indicated in the
> example).  Suppose that the following tables are created to
> provide LPM for the same VRF (e.g., VRF1 as marked in the
> figure):
>
> FE1 : LFB3 : prefixtab1
> FE1 : LFB5 : prefixtab2
> FE3 : LFB1 : prefixtab1
> FE3 : LFB3 : prefixtab3
>
> In a large system where there is a large number of such tables,
> one should use the ForCES model in a clever way such that these
> tables are kept identical (not only the prefixes, which are
> obvious, but also the next hop identifiers associated with
> each prefix to address next hop entries in another LFB).  If
> the tables are identical, then the exact same methods can be
> and should be sent to all of them to maintain the tables, and
> that is a case for multicast support.  In other words,
> the above list of tables are the real and final target of the
> multicast messages.
>
> I am not sure what part of the problem would the underlying
> multicast layer solve, but I think it is only from the CE to
> the FEs (which is useful, but not sufficient).  If so, there
> must be additional mechanisms to set up the "rest of the tree"
> on each FE, and I don't see how you can do that without explicit
> support from the ForCES protocol.  What is needed is that the CE
> should be able to say: "I want to have
> FE1 : LFB3 : prefixtab1
> FE1 : LFB5 : prefixtab2
> FE3 : LFB1 : prefixtab1
> FE3 : LFB3 : prefixtab3
> to form a muticast group, accessible by a fictive address
> FEx : LFBy : prefixtab_z".  To implement that the address
> triplet must be registered at the CE, a CE-to-FE multicast
> tree must be established (if one with identical membership
> does not exist yet), and then a table must be set up at
> the FE (and possibly even in each involved LFB).
>
> My main point, again, is that CE-to-FE multicast is useless
> unless the full tree can be set up (the leaves are tables within
> LFBs), and I suspect the ForCES protocol will need to pitch in
> to address the second and perhaps the third layer of the tree
> (the third can actually be solved within the LFB-specific protocol
> data, instead of the base protocol).
>
> My $0.02.
>
> Zsolt Haraszti
>

---

From: Tal Anker <[email-masked]>
Date: Fri, 21 Nov 2003 01:32:55 +0200

But what about the simplest form of inserting a card into a chasis, or adding a unit into a stackable product?
This "hot-insertion" capability does not (usually) require sys admin intervention - other than just inserting the card/unit... why posing such restriction (if we will require group membership to be static configured...) from the beginning? We can define the group menagement as a service that can be added later on by just defining the protocol interface to such service (add member/remove member etc). We do not need to define the membership management at the moment, but rather just to be able to cope with dynamic additions/removals of FEs (to cope with events of removal/insertion where the mechanism and source of such events is _currently_ not specified).
 
- Tal

	-----Original Message----- 
	From: Alan DeKok [mailto:[email-masked]] 
	Sent: Thu 11/20/2003 7:20 PM 
	To: [email-masked] 
	Cc: 
	Subject: Re: Support for multicast in the ForCES protocol

	Tal Anker wrote:
	>
	> Alan,
	> Are you suggesting that a network element would not support the regular hot-insertion that is common feature in either chasis or stackable products?
	
	  Not at all.  I'm just saying that complex configurations are more
	easily managed by system administrators, than by the CE.
	
	  Alan DeKok.

---

From: "(Ram Gopal )" <[email-masked]>
Date: Fri, 21 Nov 2003 15:45:00 -0500

Hello Jamal,

> [mailto:[email-masked]]On Behalf Of ext Jamal Hadi Salim
> Sent: Thursday, November 20, 2003 11:14 AM
> I dont think it is a startup challenge issue only. The 
> challenge applies
> to all applications that build state based on packet arrival 
> events. And
> there are many.
> Think of an IDS under a DOS. Or a flow tracking firewall system.
> 100s of FEs, millions of flows (say a 100K new ones per second).
> You certainly wanna distribute that info to the FEs.
> Unless you wanna rule out such apps for ForCES.
> 

You may be refering to some  specific architecture issue and cannot be generalized. 

In IDS, you need to distinguish clearly about your placement of  probing and 
monitoring/metering activity. 
I would say that poor seperation is a design issue.

In (distributed architecture) firewall or TCP off-loading case - connection are pushed to forwarding blade  after successfull connection - mechanism like  SYN-Defender etc protect the equipments.   

We have hardware chips available today which does these things nicely and also scales well.

Regards
Ramg

---

From: Tal Anker <[email-masked]>
Date: Sun, 23 Nov 2003 02:13:52 +0200

I aggree that the IDS/DOD/flow tracking seems more suitable for 1->very-few FE's case. However, distributing L3 FIB or L2 FDB are good candidates for multicast, as they are (in most cases) - the same on all/most FEs. As one parameter of measuring router/switch performance is the way they handle many updates per second while still being able to perform some other usefull functions (maintaining management sessions, etc) having multicast at least as an option for forces seems a good thing. We may not need to do the full design at first phase, but we sure at least need to try to not do anything that will later make incorporation of multicast a hard thing. 
Someone on the list suggested (if I am not mistaken) to have the proper "hooks" for multicast, and for that we need to understand what is the model and requirement from such transport. 
 
- Tal

	-----Original Message----- 
	From: Joel M. Halpern [mailto:[email-masked]] 
	Sent: Sat 11/22/2003 11:57 PM 
	To: [email-masked] 
	Cc: 
	Subject: Re: Support for multicast in the ForCES protocol

	In most lookup cache, flow processing, or RSVP architectures I know of, at
	most two (and typically one) FE would need to be updated when there is a
	change.  Hence, I am unclear as to why this is being suggested as a reason
	for multicast.
	
	Yours,
	Joel M. halpern
	
	At 10:42 AM 11/22/2003 -0500, you wrote:
	>On Fri, 2003-11-21 at 15:04, Putzolu, David wrote:
	> > [chair hat off]
	> > Jamal Hadi Salim wrote>
	> > > events. And there are many.
	> > IDS and flow tracking firewall is a specialized and challenging
	> > application,
	>
	>Challenging is a good word ;->
	>
	> > and firewalls and load balancers are often architected
	> > differently from routers
	> >
	> > - is the CE:FE protocol in the
	> > ForCES architecture the right solution for such an app?
	> >
	>
	>I just gave those two as an example and i think they are valid.
	>The general class of the problem is events that are caused by packet
	>arrivals (or maybe even timers) that require distributed state updates.
	>I know some routers(since it seems thats the operative word
	>unfortunately) or router architectures that use LPM cache entries or
	>adjacencies to speed up lookups (i.e you have a FIB which is considered
	>the slow path and some form of cache as the fast path). Some people may
	>have given up those architectures, they are however very useful IMO
	>because they allow you to associate additional state to the simple FIB
	>entry added either at the beggining of the FIB + (mostly)srcIP tuple
	>being hit or adjacency being created. I dont want to sell their virtue
	>or argue their value; suffice it is to say they exist.
	>Now that i have used the operative word "router" - would those serve as
	>good example?
	>In regards to architectures being different:
	>I think that boundaries are greying - in particular at the edges of the
	>network. I havent seen a router(at least in my contact) in the last 2-3
	>years which didnt have some firewalling capability built in.
	>In fact quiet a few that i have seen are also adding loadbalancing
	>capability; pick a $10 router from foundry (not sure how commodity you
	>wanted to get) and you will see what i am talking about. I suspect the
	>trend is gonna head that way. And i suspect these devices are gonna
	>scale even further by going into chasis and chasis going further into
	>racks. Would you want these devices selling for $50 a chasis out
	>of Taiwan be able to use Forces? I want them to.
	>
	> > Tracking flow state in a coherent fashion across blades can require
	> > updating state as often as once/packet in each flow. Using ForCES,
	> > that would require an event to the CE, followed by the
	> > CE creating and broadcasting a flow update message
	> > to every FE - all occurring every time a TCP SYN, SYNACK,
	> > or FIN (at least) is received.
	>
	>Did you forget RST? ;-> just kidding.
	>You would have to do a few smart things, but this is the wrong place to
	>discuss that.
	>
	> > Wouldn't a FE:FE protocol
	> > be a much better fit for this class of applications?
	>
	>If you are multicast capable, it doesnt matter: Use CE-FE events.
	>
	>cheers,
	>jamal