Subject: Re: ForCES transport requirements WAS (Re: Support for multicast inthe ForCE...

From: Tal Anker <[email-masked]>
Date: Tue, 25 Nov 2003 01:35:23 +0200

Hi Alex,
In the environment I expect the protocol to work in, I expect that with good engineering most of the protocol packets will reach their destination. I would expect relatively a small number of retransmissions. Thus, I do not think that the fact that a packet is retransmitted via multicast would be a big issue (note that when retransmitting due to a NACK you can either send via multicast or directly using unicast, depending on the amount of FE's that had nacked the packet...). 
 
Also - security and multicast have received quite lots of attention - both in the IETF and in the research community... As the number of FEs would not be too big, there are nice and elegant techiques for performing authentication and efficient key exchange in such groups (I guess that a reasonable number would be on the order of couple of dozens in a realistic scenario, but hundreds will also work ok).
 
- Tal

	-----Original Message----- 
	From: Alex Audu [mailto:[email-masked]] 
	Sent: Mon 11/24/2003 7:02 PM 
	To: [email-masked] 
	Cc: 
	Subject: Re: ForCES transport requirements WAS (Re: Support for multicast inthe ForCE...

	To add to my previous comments:
	
	On the following question:
	> How can you have reliable communcation over UDP?
	>
	Jamal  wrote:
	  The mechanisms would be a retransmit timer and ability to get ACKs
	  or NACKs.
	  Note the definition of reliability includes a valid time. So this
	  validity is best controled by the app (the ForCES protocol). TCP
	  doesnt allow you to do this.
	
	My comments:
	So the retransmit timer fires, and then you do another broadcast or
	multicast?
	I can see how this does wonders for your CE-FE bandwidth efficiency.
	
	I don't see why TCP can't cope with sending data over zero, one or two hops
	(the environment for ForCES) in time to warrant the use of UDP. After
	all, it is not like we are sending packets from one end of the globe to
	another!
	
	And you can't just wave the issue of security away.   Routing security is
	quite
	important the IETF created the RPSEC WG to look into ways of making
	routing protocols more secure.  The outcome of that may make some demands
	on the router architectures. For example, separating less critical but
	attack-prone
	"Hello" message channel from ciritical LFB configuration and control message
	
	channel is a good revelation.
	
	Regards,
	Alex.
	
	Jamal Hadi Salim wrote:
	
	> On Sun, 2003-11-23 at 19:30, Alex Audu wrote:
	> > Hi All,
	> >
	> > All this talk about the need and benefits of multicast certainly makes
	> > ones
	> > feel it is a cure all for what is being termed as the ailment in
	> > ForCEs' requirements.
	> >
	>
	> It is useful to have; it is not listed as a requirement - just like
	> congestion control is not listed as a requirement in the non-global
	> scope but useful to have.
	>
	> > Look, it has to be told that you can't do multicast on the cheap. And
	>
	> You will have to show me a single switch fabric that cant do multicast
	> or broadcast. Pick any netgear or dlink $1 switch and show me it doesnt
	> support multicast. How cheap do you want to go?
	>
	> >  you have
	> > to answer some key questions if you really want to sell multicast in a
	> > critical
	> > environment,..viz:
	>
	> [deleted things which have been repeated in the last 10 emails on this
	> subject for brevity]
	>
	> >
	> > There is also this myth that all the FEs will have the same routing
	> > information
	> > always.
	>
	> All examples i have seen say that most not all.
	>
	> > I submit that this is not true.  One of the problems ForCEs is meant
	> > to
	> > solve is scalability. This certainly relates to the abilty to manage
	> > the rapid
	> > growth of routing table sizes by distributing the routing information
	> > as partial
	> > routes into various FE groups.  Granted there is still some scaling
	> > issues
	> > here,..but is definitely not as bad as people have been making it to
	> > be.
	> >
	>
	> I am not sure i followed. Are we back to the topic that the problem is a
	> startup issue?
	>
	> >      ..more comments in-line...
	>
	> Thanks for being brave and addresing these issues.
	>
	> > Regards,
	> > Alex.
	> >
	> > In a message dated 11/22/2003 12:03:24 PM Central Standard Time,
	> > [email-masked] writes:
	> >
	> >
	> > > Continuation of the last thread with a change in topic - i think
	> > > these
	> > > are the real issues that need to be addressed instead of the
	> > > distraction
	> > > to "multicast is evil".
	> > >
	> > > Lets really list the requirements needed for a successful underlying
	> > > transport protocol that will serve ForCES well. i may be
	> > > missing some - but i hope to get challenged so i can add to them;
	> > > I am thinking of turning this into a draft so people read it first
	> > > and
	> > > then post.
	> > > My previous email has some more claims which i will assume here.
	> > >
	> > > I. ForCES  mechanisms requiring reliability:
	> > >
	> > > a) time limit very sensitive (in other words obsolence period is
	> > > faster):
	> > > Example table updates;
	> > > I have argued TCP doesnt qualify in my previous email. SCTP-PR may
	> > > qualify.
	> > > Something like what netlink2 over UDP qualifies because it gives
	> > > control
	> > > to netlink2 to make such decisions.
	> >
	> >
	> > How can you have reliable communcation over UDP?
	> >
	>
	> The mechanisms would be a retransmit timer and ability to get ACKs
	> or NACKs.
	> Note the definition of reliability includes a valid time. So this
	> validity is best controled by the app (the ForCES protocol). TCP
	> doesnt allow you to do this.
	>
	> > > II.  mechanisms capable of tolerating unreliability: Example psamp.
	> > > TCP doesnt qualify.
	> > > something(like netlink2) over UDP or SCTP-PR.
	> >
	> >
	> > Why do you say TCP doesn't qualify here?  Any application that can
	> > live
	> > with un-reliable transport can sure do better with a relaible one.
	> >
	>
	> You are right. TCP qualifies and the other two as well.
	>
	> > > III. mechanisms that prefer timeliness over reliability: Ex
	> > > heartbeats
	> > > something (like netlink2) over UDP or SCTP-PR.
	> > >
	>
	> I take it you agree TCP fails if you really value timelines.
	>
	> > > Now on the choice of netlink2 UDP vs SCTP-PR
	> > >
	> > > 1) UDP is here today. SCTP-PR is not. When SCTP-PR becomes available
	> > > Netlink2 can run over it. For now we choose to be standalone: We can
	> > > run
	> > > over IP, over UDP or over ethernet, you take your pick.
	> >
	> > I believe SCTP-PR is now in IESG queue. Even when it is available, you
	> > have
	> > to run it in a reliable mode to guarantee that those critical
	> > configuration
	> > messages get to their destination safely. And it still doesn't buy you
	> > security.
	>
	> I think it has been issued RFC status.
	> SCTP-PR would be a better choice than TCP. Because it will give user
	> control of removing stale messages.
	>
	> >
	> > > 2) UDP provides multicast (and broadcast). SCTP-PR does not.
	> > > Since SCTP has turned to be a swiss knife (or maybe a multi faceted
	> > > beast), i am sure it will soon as well support multicast;->
	> > >
	> > > If multicast is there - we will use it. Sans that we will do
	> > > something
	> > > multicast like (Jon calls it unireplicate i think).
	> > > All the three cauthors have had pleasant experiences using multicast
	> > > for
	> > > ForCES like. We think it is a good thing to have. But we will not
	> > > torture you if you dont have multicast or broadcast ;->
	> > >
	> > > Why Multicast is useful:
	> > > --
	> > > I must apologize since i have to repeat this.
	> > > In the simple case of a single multicast wire: each of the CEs or
	> > > FEs
	> > > opens a single multicast socket/channel.
	> > >
	> > > Why is this good:
	> > >
	> > > 1) Bandwidth is less used
	> > > a)because one or more CEs can send a single message to all FEs.
	> > > I really dont want to be teaching people the hows but i will provide
	> > > one
	> > > example which David or maybe it was Joel didnt seem to make sense
	> > > out of
	> > > (but one that i have successfuly used)[1]
	> > > b) you can reduce the number of packets further by limiting on
	> > > ACKs (partial acks)
	> > >
	> > > 2) RAM on the embedded CE or FE is used less. There are less sockets
	> > > open and less messages being built
	> > >
	>
	> I take it you agree with the above two ;->
	>
	> > > 3) CPU is preserved more.
	> > > Since there are fewer messages to process and fewer (socket states)
	> > > states to track.
	> >
	> >
	> > As far as I am concerned, the more critical resource CPU-wise is FE.
	> > The
	> > mutlicast scheme Netlink2 proposes blasts all every packet to every
	> > FE.
	> > Each FE has to do work to filter out which packet doesn't belong to
	> > it. Thus,
	> > the FE spends a lot of CPU cycles filtering off unwanted
	> > packets,..cycles
	> > that should be best used in forwarding packets.
	> >
	>
	> I wasnt sure whether i should respond to this or not. I have addressed
	> it in previous emails and it is addresed in the draft.
	> For brevity: Yes this may become an issue, but there are solutions.
	>
	> >
	> > > 4) The ability for all listeners on the wire to see message
	> > > exchanges
	> > > even when not directed at them provides a cheap way to synchronize.
	> >
	> >
	> > And of course, if this is a malicious node, it will gladly gleam as
	> > much information
	> > as is enough to mount an attack. This is why secure multicast is
	> > necessary.
	>
	> The MSEC people showed up at the last WG meeting - it is too bad that
	> noone raised this issue. I dont wanna discuss this - its a separate
	> issue. It is solvable. Let me ask the following since you bring the
	> issue of scalability:
	> If you were to encrypt messages, would it be cheaper to encrypt two
	> multicasted flows
	> or 8 unireplicated flows? I have seen processors having problem with a
	> single flow.
	>
	> > > 5) At a transport level, (not the forces transaction level), having
	> > > access to multicast provides location transparency.
	> > >
	>
	> I take it you agree with this.
	>
	> > > 6) A lot of switch fabrics (shared memory designs or otherwise)
	> > > interconnecting FEs and CES have multicast capabilities; so it is
	> > > good
	> > > if we can take advantage of it where possible.
	> > >
	>
	> I take it you agree with this.
	>
	> > > The above is the simple case, by induction you could extend the
	> > > reasoning to the case where you have X sockets.
	> > >
	>
	> I take it you agree with this as well.
	>
	> cheers,
	> jamal

---

From: Steven Blake <[email-masked]>
Date: Mon, 24 Nov 2003 21:49:54 -0500

Here is a contrived, but representative example of the type of problem I
think that ForCES needs to be able to solve to make it more useful for
me than a home-grown solution.

Imagine a NE with one (active) CE and 64 FEs, each forwarding global
routes.  64K external prefixes are currently reachable via peer A.
Suddenly the peering to A fails, and now half the prefixes are reachable
via peer B and the other half via peer C.  No matter how cleverly you
have designed your datastructures, in the general case you are going to
have to update at least 32K table entries.

Suppose you can pack 64 table updates/ForCES packet (~1500 byte packet
with ~24 byte entries).  Then the CE will have to send 500 packets/FE *
64 FEs = 32K packets if ForCES is restricted to unicast transport.  That
works out to about 400 Mbits, which will take 400 msec to transmit at
full blast assuming the CE is attached via gigabit Ethernet.  Meanwhile
the NE might be forwarding several gigabits worth of traffic into a
black hole.

This solution simply doesn't meet my timeliness requirements.  So I can
choose to hope that the number of FEs in my NE increases slower than
Moore's Law, I can choose to try and come up with a proprietary
mechanism for splitting this update load across several CEs, or I can
choose to explore whether multicast can be utilized to minimize the
update latency.

For the moment I have chosen the latter, but I am willing to listen to
arguments as to why that is the wrong choice.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 24 Nov 2003 22:55:36 -0500

Actually, based on the responsiveness requirements for IP routing, the time
constraints for this are not very bad at all.
Taking even several seconds to adjust for BGP failure, and black holing
traffic in the mean time.  Is quite acceptable.  Taking even 10 seconds is
probably okay.  The routing system is actually designed to respond slightly
slowly.  (Responding too fast to events produces very interesting problems
of its own.)  For example, when using GigE thorugh a switch (hence not
having physical link down detection) the time to detect a neighbor failure
is itself several 10s of seconds.  Worrying about 1 second of response time
to an event that takes 30 seconds to detect does not seem necessary.

There are contexts where this is not desirable.  In those cases, one takes
a completely different approach.  Because if small number of seconds is too
slow, then one is trying for 50 msec, and one should think about the
problem completely differently.  (For example, one could create backup FIBs
with ones second choices already pre-loaded.  Yes, I know that this is a
limitted solution.  I am not trying to solve the problem, merely to suggest
that if it matters one approaches it differently.)

Yours,
Joel M. Halpern

---

From: Alex Audu <[email-masked]>
Date: Tue, 25 Nov 2003 10:34:32 -0600

Multicasting certainly helps. In the ForCEs environment, it has to be secure

and reliable to be really useful.  You have to expect packet loss/damage due

to collision if you are multicasting over UDP. And If you have to
re-transmit
even one packet, that really knocks a hole into your overall time gain.

This will always be an issue untill the a solution for relaible secure
multicast is
found. Meanwhile, you could do multicasting at ForCES protocol level (by
doing multi-unicasts).  This is not a bad solution. All your unicasts are
protected
by the underlining reliability and security mechanisms. And it will be much
faster
than than trying to send all the messages to the targets from the
application layer.

Some have suggested that a typical deployement environment for ForCES will
be
one of zero hop. Certainly, if you are doing zero hop NE, you can worry less

about security issues. But you still have to worry about reliability and
in-order delivery of messages,  especially if the LFBs you are configuring
are stateful.
But of course, any solution we come up with must be applicable to
one/two-hop
scenarios also. That is a ForCES requirement.

Regards,
Alex.