Subject: Re: ForCES transport requirements WAS (Re: Support for multicast in the ForCE...

From: Alex Audu <[email-masked]>
Date: Sun, 23 Nov 2003 19:30:27 EST

Hi All,

All this talk about the need and benefits of multicast certainly makes ones
feel it is a cure all for what is being termed as the ailment in ForCEs'
requirements.

Look, it has to be told that you can't do multicast on the cheap. And you
have
to answer some key questions if you really want to sell multicast in a
critical
environment,..viz:

1) How would you guarrantee the relaible delivery of the messages being
    blasted to all the targets? What if the messages don't arrive at all
their
    intended targets? How do you recover from this? This suggests you need
    a reliable multicast mechanism.  I am not sure you can do this over UDP
    as it is currently being done in NetLink2.  That is only good for
un-reliable
    multicast. I am not sure if you can trust critical configuration messages
    like routing table information or basic LFB service/tree structure
messages
    over such an unreliable mechanism.

 2) How do you deal with the issue of security, given that we don't have
secure
     multicast yet?  I have seen some people call for broadcast capability
also.
     One result of multicast/broadcast is you run the risk of a potential
attacker
     listening in on say, the routing information being blasted to the FEs. I
don't
     think multicasting over UDP has an answer to this.

There is also this myth that all the FEs will have the same routing
information
always. I submit that this is not true.  One of the problems ForCEs is meant
to
solve is scalability. This certainly relates to the abilty to manage the
rapid
growth of routing table sizes by distributing the routing information as
partial
routes into various FE groups.  Granted there is still some scaling issues
here,..but is definitely not as bad as people have been making it to be.

     ..more comments in-line...

Regards,
Alex.

In a message dated 11/22/2003 12:03:24 PM Central Standard Time,
[email-masked] writes:

> Continuation of the last thread with a change in topic - i think these
> are the real issues that need to be addressed instead of the distraction
> to "multicast is evil".
>
> Lets really list the requirements needed for a successful underlying
> transport protocol that will serve ForCES well. i may be
> missing some - but i hope to get challenged so i can add to them;
> I am thinking of turning this into a draft so people read it first and
> then post.
> My previous email has some more claims which i will assume here.
>
> I. ForCES  mechanisms requiring reliability:
>
> a) time limit very sensitive (in other words obsolence period is
> faster):
> Example table updates;
> I have argued TCP doesnt qualify in my previous email. SCTP-PR may
> qualify.
> Something like what netlink2 over UDP qualifies because it gives control
> to netlink2 to make such decisions.
>

How can you have reliable communcation over UDP?

> b) time limit best effort (obsolence takes longer)
> Sample apps: File transfers,etc.
> TCP qualifies but AFAIK, we dont have this need.
>
> II.  mechanisms capable of tolerating unreliability: Example psamp.
> TCP doesnt qualify.
> something(like netlink2) over UDP or SCTP-PR.
>

Why do you say TCP doesn't qualify here?  Any application that can live
with un-reliable transport can sure do better with a relaible one.

> III. mechanisms that prefer timeliness over reliability: Ex heartbeats
> something (like netlink2) over UDP or SCTP-PR.
>
> Clearly in the bigger picture TCP is not the best protocol.
> UDP and SCTP-PR qualify partially merely because they dont hinder
> progress not because they have some magic. Another non-IETF protocol
> that would qualify is TIPC. I havent carefully studied DCCP but i
> believe it also has a fit. I will exclude it because its not there yet.
>
> So, to summarize: the transport protocol that meets all the requirements
> For Forces has not been born.
>
> Choosing TCP is choosing an answer and then trying to justify the
> questions that it brings. It does sound like peer pressure instead of
> looking at requirements and then deciding what you need to do.
> I think we recognized this in netlink2 and made it clear that the choice
> of underlying protocol chosen is up to the pre-association phase.
> We compensate for "missing features" depending on the transport choice
> (read the draft).
> It's very few lines of code to provide that luxury. So the cost is low.
>
> Now on the choice of netlink2 UDP vs SCTP-PR
>
> 1) UDP is here today. SCTP-PR is not. When SCTP-PR becomes available
> Netlink2 can run over it. For now we choose to be standalone: We can run
> over IP, over UDP or over ethernet, you take your pick.
>

I believe SCTP-PR is now in IESG queue. Even when it is available, you have
to run it in a reliable mode to guarantee that those critical configuration
messages get to their destination safely. And it still doesn't buy you
security.

> 2) UDP provides multicast (and broadcast). SCTP-PR does not.
> Since SCTP has turned to be a swiss knife (or maybe a multi faceted
> beast), i am sure it will soon as well support multicast;->
>
> If multicast is there - we will use it. Sans that we will do something
> multicast like (Jon calls it unireplicate i think).
> All the three cauthors have had pleasant experiences using multicast for
> ForCES like. We think it is a good thing to have. But we will not
> torture you if you dont have multicast or broadcast ;->
>
> Why Multicast is useful:
> --
> I must apologize since i have to repeat this.
> In the simple case of a single multicast wire: each of the CEs or FEs
> opens a single multicast socket/channel.
>
> Why is this good:
>
> 1) Bandwidth is less used
> a)because one or more CEs can send a single message to all FEs.
> I really dont want to be teaching people the hows but i will provide one
> example which David or maybe it was Joel didnt seem to make sense out of
> (but one that i have successfuly used)[1]
> b) you can reduce the number of packets further by limiting on
> ACKs (partial acks)
>
> 2) RAM on the embedded CE or FE is used less. There are less sockets
> open and less messages being built
>
> 3) CPU is preserved more.
> Since there are fewer messages to process and fewer (socket states)
> states to track.
>

As far as I am concerned, the more critical resource CPU-wise is FE. The
mutlicast scheme Netlink2 proposes blasts all every packet to every FE.
Each FE has to do work to filter out which packet doesn't belong to it. Thus,
the FE spends a lot of CPU cycles filtering off unwanted packets,..cycles
that should be best used in forwarding packets.

> 4) The ability for all listeners on the wire to see message exchanges
> even when not directed at them provides a cheap way to synchronize.

And of course, if this is a malicious node, it will gladly gleam as much
information
as is enough to mount an attack. This is why secure multicast is necessary.

> 5) At a transport level, (not the forces transaction level), having
> access to multicast provides location transparency.
>
> 6) A lot of switch fabrics (shared memory designs or otherwise)
> interconnecting FEs and CES have multicast capabilities; so it is good
> if we can take advantage of it where possible.
>
> The above is the simple case, by induction you could extend the
> reasoning to the case where you have X sockets.
>
> I dont want to preach multicast as panacea; we address some of the
> limitations in the draft; however, i want it to be available when i need
> it and not shoved down the toilet.
>
> So far apart from Joel (who implemented on ATM) on real boxes and Ram
> (who did something very specific that needs unicast) the rest of the
> arguements i have seen are from people who see the need for this
> feature. You can choose to ignore it - and you may be influenced by some
> implementation you are familiar with - thats your loss. I do think TCP
> is valid for some cases (low latency and congestion), but i wouldnt
> claim it to be the answer weve all been waiting for.
> Again let me reiterate that Netlink2 does not need to have multicast (it
> is a MAY) but would love to have it when it is available. We can run on
> your favorite protocol TCP ;->
>
> cheers,
> jamal
>
> [1] I will give the example in another email - this one is already
> getting too long.
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 23 Nov 2003 22:50:35 -0500

On Sun, 2003-11-23 at 19:30, Alex Audu wrote:
> feel it is a cure all for what is being termed as the ailment in
> ForCEs' requirements.
>

It is useful to have; it is not listed as a requirement - just like
congestion control is not listed as a requirement in the non-global
scope but useful to have.

> Look, it has to be told that you can't do multicast on the cheap. And

You will have to show me a single switch fabric that cant do multicast
or broadcast. Pick any netgear or dlink $1 switch and show me it doesnt
support multicast. How cheap do you want to go?

>  you have

[deleted things which have been repeated in the last 10 emails on this
subject for brevity]

> always.

All examples i have seen say that most not all.

> I submit that this is not true.  One of the problems ForCEs is meant
> to
> solve is scalability. This certainly relates to the abilty to manage
> the rapid
> growth of routing table sizes by distributing the routing information
> as partial

I am not sure i followed. Are we back to the topic that the problem is a
startup issue?

>      ..more comments in-line...

Thanks for being brave and addresing these issues.

> How can you have reliable communcation over UDP?

The mechanisms would be a retransmit timer and ability to get ACKs
or NACKs.
Note the definition of reliability includes a valid time. So this
validity is best controled by the app (the ForCES protocol). TCP
doesnt allow you to do this.

> with un-reliable transport can sure do better with a relaible one.

You are right. TCP qualifies and the other two as well.

> > something (like netlink2) over UDP or SCTP-PR.

I take it you agree TCP fails if you really value timelines.

> messages get to their destination safely. And it still doesn't buy you

I think it has been issued RFC status.
SCTP-PR would be a better choice than TCP. Because it will give user
control of removing stale messages.

> > torture you if you dont have multicast or broadcast ;->

I take it you agree with the above two ;->

> that should be best used in forwarding packets.

I wasnt sure whether i should respond to this or not. I have addressed
it in previous emails and it is addresed in the draft.
For brevity: Yes this may become an issue, but there are solutions.

> And of course, if this is a malicious node, it will gladly gleam as
> much information

The MSEC people showed up at the last WG meeting - it is too bad that
noone raised this issue. I dont wanna discuss this - its a separate
issue. It is solvable. Let me ask the following since you bring the
issue of scalability:
If you were to encrypt messages, would it be cheaper to encrypt two
multicasted flows
or 8 unireplicated flows? I have seen processors having problem with a
single flow.

> > access to multicast provides location transparency.

I take it you agree with this.

> > if we can take advantage of it where possible.

I take it you agree with this.

> > reasoning to the case where you have X sockets.

I take it you agree with this as well.

cheers,
jamal

---

From: Patrick Droz <[email-masked]>
Date: Mon, 24 Nov 2003 09:28:53 +0100

[chair hat off]

see comments inline

Alex Audu wrote:

>      multicast yet?  I have seen some people call for broadcast 
> capability also.
>      One result of multicast/broadcast is you run the risk of a 
> potential attacker
>      listening in on say, the routing information being blasted to the 
> FEs. I don't

could you please provide some evidence of this? Keep in mind we are
moving from a centralized (single routing table) to a distributed
router architecture. Why should it change that the routing table
should be much different. Or do you want to go into the discussion
of how you avoid routing loops?

> always. I submit that this is not true.  One of the problems ForCEs is 
> meant to

Regards,
Patrick

-- 
   Dr. Patrick Droz 			| [email-masked]
   IBM Zurich Research Laboratory 	| http://www.zurich.ibm.com/~dro
   Saumerstrasse 4 			| Tel. +41-1-724-85-25 CH-8803
   Rueschlikon/Switzerland 		| Fax. +41-1-724-85-78

---

From: Alex Audu <[email-masked]>
Date: Mon, 24 Nov 2003 10:37:44 -0600

Multicast is great to have, but certainly not at the cost of reliability and

security!  Multicasting "Hello" messages is sure much less critical than
multicasting critical LFB configuration information where packets have to
arrive in sequence and without curruption.

Regards,
Alex.

---

From: Alex Audu <[email-masked]>
Date: Mon, 24 Nov 2003 11:02:10 -0600

To add to my previous comments:

> How can you have reliable communcation over UDP?
Jamal  wrote:

My comments:
So the retransmit timer fires, and then you do another broadcast or
multicast?
I can see how this does wonders for your CE-FE bandwidth efficiency.

I don't see why TCP can't cope with sending data over zero, one or two hops
(the environment for ForCES) in time to warrant the use of UDP. After
all, it is not like we are sending packets from one end of the globe to
another!

And you can't just wave the issue of security away.   Routing security is
quite
important the IETF created the RPSEC WG to look into ways of making
routing protocols more secure.  The outcome of that may make some demands
on the router architectures. For example, separating less critical but
attack-prone
"Hello" message channel from ciritical LFB configuration and control message

channel is a good revelation.

Regards,
Alex.

---

From: Alex Audu <[email-masked]>
Date: Mon, 24 Nov 2003 11:08:13 -0600

Further comments:

> as is enough to mount an attack. This is why secure multicast is

Jamal wrote the following:
  The MSEC people showed up at the last WG meeting - it is too bad that
  noone raised this issue. I dont wanna discuss this - its a separate
  issue. It is solvable. Let me ask the following since you bring the
  issue of scalability:
  If you were to encrypt messages, would it be cheaper to encrypt two
  multicasted flows or 8 unireplicated flows? I have seen processors having
  problem with a single flow.

My comments:
  I think the question is "how are you going to solve multicast security
problem?"
  You say it is solvable. I really will like to know.

Regards,
Alex.

---

From: Alex Audu <[email-masked]>
Date: Mon, 24 Nov 2003 13:16:37 -0600

Hello Patrick,

Please my comments in-line..

Regards,
Alex.

> of how you avoid routing loops?

One of the benefits of distributing the NE is the ability to handle many more
routes,..many more than the current core routers are capable of. Route
table sizes start becoming a problem at that scale.  Storing partial routes
in a logically hierarchical distributed entities can help. Of course, you have
to assign the default routes appropriately to prevent routing loops.

---

From: Patrick Droz <[email-masked]>
Date: Tue, 25 Nov 2003 11:34:44 +0100

[chair hat off]

Alex Audu wrote:

Hi Alex,

see my comments inline.

> to assign the default routes appropriately to prevent routing loops.

can you proof this statement? For what reason do you need such big
routing tables. I have not see much growth of routing table sizes.
A distributed router has to provide forwarding an classification
performance. I have a hard time that you have to distribute a router
because of the forwarding table size. Overall you need exactly the
same amount of memory. Why should a packet coming in on port A and
heading for DA X being routed differently than a packet arriving on
port B and also heading to DA X?

Regards,
Patrick

---

From: Alex Audu <[email-masked]>
Date: Tue, 25 Nov 2003 11:59:16 -0600

Hi Patrick,

The growth rate of routing table size has been of interest in the research
community and industry for some time. And research is activly on-going
at finding ways to contain that growth. See for example:

   http://www.antc.uoregon.edu/route-views/dynamics/

Certainly the routing table gorws as the number of route peers increases. This
particularly the case with table-driven IP routing where each machine employes
an IP Routing table that sotre information about possible destinations and how
to reach them.

Using the network portion of a destination address instead of the complete
host address makes routing efficient and keeps routing tables small. This is
the essence of Next-Hop Routing where a routing table table contains (N,R)
(where N is the Ip address of the destination network and R is the IP address
of the next router along the path to N. Router R is called the next hop. There
are various other methods including partitioning the CORE into partial cores
that reside in separate sites and using default routes to point to the peer site.
Of course, this results into a routing loops,..and that is not what we are talking
about here.

I would rather not say anymore about this, but you can certainly partition the
routing table in an NE into a tree with the root residing in the CE. This is a form
of hierarchical routing. Sure, it may take longer to resolve certain routes.

See also
    http://www-rp.lip6.fr/~ridoux/Manuel-NS2/node319.html
    http://www.research.att.com/~jrex/papers/filter.pdf

Regards,
Alex.

---

From: Patrick Droz <[email-masked]>
Date: Tue, 25 Nov 2003 19:09:34 +0100

[chair hat off]

yes, I know you can partition the routing table but many routers
don't do that. Also keep in mind that you still have to have the
full routing info at the CE for all the packets that cannot be
handled at the FE (e.g. IP option handling).

Regards,
Patrick

---

From: Alex Audu <[email-masked]>
Date: Tue, 25 Nov 2003 12:21:03 -0600

Yes, the full routing table could be in the CE but not in the FEs. The issue
was the time it takes to blast the full routing table to each of the FEs, but
I am saying, it doesn't have to be that way. You can take the scalability
that ForCES provides and partition the routing tables into smaller
sizes that are blasted to the FEs. And of course, that also means that the
route partitions are not necessarily the same in all the FEs. This implies
you can't blindly multicast routing tables to all the FEs

The other issue is the routing table in the CE may not have to be the full table.
But that is another research topic.

Regards,
Alex.

---

From: Sandy Pavel <[email-masked]>
Date: Tue, 25 Nov 2003 16:18:43 -0500

On a multi-blade NE architectures an FE can be an ingress or egress with
respect to the data flow of a routed packet. Typically, the actual route
look up is done on the ingress card. This means that every FE has to
know about all the routes that are reachable through the other n-1 FEs
(where n is the total number of FEs). Assuming a balanced distribution
of reachable networks per egress FE, it might not be worth the effort of
partitioning the routing table (not mentioning the cost) in order to
eliminate only 1/n of the routes, especially if multicast/broadcast is
available.

Things become a bit more interesting when an actual implementation
decides to use the information in the forwarding table on both ingress
and egress FEs (I am not going to justify this though).

As Patrick mentioned, there is the case of exception (e.g. IP option)
processing. The amount of exception traffic could also be considerable;
due to deeper packet inspection and more complicate processing rules it
might require a lot of processing power. Therefore, there is a tendency
in the new core router designs (running tens of interfaces at
multigigabit rates) of providing a slow (exception) path processing on a
CPU that is tightly coupled to the data path engines on the FE. Although
there still going to be exception traffic that can only be processed at
CE, this solution reduces considerable the load on the CE and provides a
good scalability mechanism. This would be another case (some would
consider it an exception!?) where global distribution of the forwarding
table would be required.

Regards,
Sandy

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Alex Audu
Sent: November 25, 2003 1:21 PM
To: [email-masked]
Subject: Re: ForCES transport requirements WAS (Re: Support for
multicastintheForCE...

Yes, the full routing table could be in the CE but not in the FEs. The
issue
was the time it takes to blast the full routing table to each of the
FEs, but
I am saying, it doesn't have to be that way. You can take the
scalability
that ForCES provides and partition the routing tables into smaller
sizes that are blasted to the FEs. And of course, that also means that
the
route partitions are not necessarily the same in all the FEs. This
implies
you can't blindly multicast routing tables to all the FEs

The other issue is the routing table in the CE may not have to be the
full table.
But that is another research topic.

Regards,
Alex.

> >>>>>feel it is a cure all for what is being termed as the ailment in
ForCEs'
> >>>>>   multicast. I am not sure if you can trust critical
configuration
> >>>>>Each FE has to do work to filter out which packet doesn't belong
to it.
> >>>>>Thus,