Subject: Re: Support for multicast in the ForCES protocol

From: "Putzolu, David" <[email-masked]>
Date: Fri, 21 Nov 2003 12:04:34 -0800

[chair hat off]
Jamal Hadi Salim wrote>
Joel Halpern wrote> >
> > Even when the forwarding table is disjoint from the next hop 
> > table (a good idea in its own right) there are many uses of 
> > a "forwarding table LFB" which will not be populated with the 
> > same information.
> 
> I think thats a detail which belongs to the model.  There will be
> different types of IPV4 LFBs - some requiring the clear 
> demarcation you make.

I'm not sure I follow here.  The justification for the added
complexity of multicast is that it will save bandwidth.  Evaluating
some realistic examples of saved bandwidth seems to me to
be an important part of the discussion rather than just a detail!

Aside from the issue of performance, there is also the question
of feasibility.  Whether unicast or multicast is used, the following
requirements must be met when downloading FIB entries:

1 Data must not be corrupted in transit. 
2 Data must not be reordered in transit. 
3 Once delivered to the FE, the success or failure of a route
  addition attempt must be explicitly acknowledged.
4 There must be a way for the FE to know which LFB the
  route pertains to (FE will have many LFBs, and may even
  have multiple forwarding LFBs).
5 Route additions may sometimes be part of a larger transaction
  on a FE (e.g. "Add this prefix entry to LFB1, AND add this
  nexthop entry to LFB2, and use all or nothing semantics.")
6 When used over multiple hops, route additions (and all 
  other forces messages) must use an existing RFC2914 L4 transport.
  (charter requirement)

For #1, TCP is satisfactory in unicast. In multicast,
it will be necessary to add some sort of CRC to packets,
or use an existing reliable multicast protocol.

For #2, TCP is satisfactory in unicast. In multicast,
sequence numbers will be necessary, or a reliable
multicast transport protocol will need to be used.

For #3, the same TCP connection can be used in unicast. In
multicast, either the same multicast connection is used
(this results in every FE seeing every other FE's ack - yuck!)
or a separate TCP connection must be maintained by the FE to
the CE.

For #4, either the protocol or its payload probably needs a
LFB identifier in the unicast case.  In the multicast case,
it will be necessary to come up with some sort of binding
protocol, where LFBs are dynamically associated at runtime 
with multicast groups.

For #5, in the unicast case, a single message could contain
transaction information and a set of sub-messages.  In the
multicast case I cannot off the top of my head come up with
a mechanism to even support transactioning.

For #6, in the unicast case, TCP is satisfactory.  In the
multicast case, a 2914 compliant multicast-based standard
protocol would be needed.  Are there existing, standard,
robust implementations of a 2914 compliant multicast protocol
available in the operating systems commonly used in CEs and FEs?

In my (technical - chair hat still off) opinion, all of the
above requirements have to be met for any solution (multicast
or unicast) to be feasible.  I can understand how the above
are satisfied by unicast, and would like to understand better
how the above can be satisfied when using multicast.

Thanks,
David

---

From: Jon Maloy <[email-masked]>
Date: Fri, 21 Nov 2003 15:45:38 -0500

/jon

Putzolu, David wrote:

[chair hat off]

Jamal Hadi Salim wrote>

Joel Halpern wrote> >

same information.

demarcation you make.

  (charter requirement)

the CE.

The multicast ack case does not sound that horrible to me, but there
_will_ have to be (some kind of) connections FE-CE anyway, - most
messages are still not multicast. The bottom line is, no base protocol
in this world can fulfil this requirement, the acks have to be sent
user-to-user, and we are back to the "ack storm" scenario.

with multicast groups.

Why is this different ? Each FE receiving the multicast with bundled
sub-messages performs defines a local transaction and reports the
result back to the sender. It may be tricky, but not different from the
TCP based dito.

Thanks,

David

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sat, 22 Nov 2003 11:11:03 -0500

On Fri, 2003-11-21 at 15:04, Putzolu, David wrote:
> complexity of multicast

Lets hope we can clear this confusion
Bear with me while i become as verbose as you or more - its saturday
morning, cold outside  - so plenty of time here.
Maybe we can have an internet draft to capture all these issues
which seem to have been beaten to death many times.

> is that it will save bandwidth.

And a few other things such as location transparency, ease of
synchronizing state etc.

>   Evaluating

Sure.

> 2 Data must not be reordered in transit.

You compute at the transaction level, so this statement is a little
misleading.
In any case i would think in a ForCES environment (single chasis or
within a rack with a few L2 Multihops) - the possibility of reordering
is slim to non-existent.

>   addition attempt must be explicitly acknowledged.

nod.

>   have multiple forwarding LFBs).

A given.

>   nexthop entry to LFB2, and use all or nothing semantics.")

Lets call this a transaction.

>   other forces messages) must use an existing RFC2914 L4 transport.

The most important law to design is to review the requirements first and
then throw terms around.

Lets look at the scenario deployment environments for ForCES.

1) A private dedicated link from CE-FE, mostly within a single chasis
but may cross chasis or racks. Essentially "the close proximity"
situation. Mostly a single L3 setup or even single hop L2 in a network
dedicated for this activity (switch fabric within a chasis or multi
switch fabrics across chasis interconnected by some hierachical L2
switch).
2) The above in addition croses a private Lan. Maybe one or two L2 hops
maybe one or two L3 hops.
3) In addition to crossing a private lan #2 croses the big bad internet.
Many L3 hops.

I claim #1 would be the most deployed environment. I can handwave this
would be 90% of the deployments. I claim #2 would take 9.5% again using
the handwaving clairvoyance.
The rest would fall under #3 (It may be useful for remote office
management, countering dDOS amongst participating sites etc). There will
be competing protocols for #3 so you may never see ForCES deployed for
such work.

RFC2914 L4 protocols aka multihop is applicable for #3 mostly and #2
partially and for those scenarios it MUST be used.
Using the "optimize for the most" design rules, there are other
requirements which are more important than need to be met
than simply invoking RFC2914. I personally dont care about people who
throw cliches around - i prefer reasoning. Again i am not downplaying
the need, i just think that in the most cases the term is abused. So
lets kill this arguement please.
[ A good parody for this is: http://inetpix.com/dare/ ]

> or use an existing reliable multicast protocol.

Take the pedagogical definition of reliability:

1) Message is delivered with integrity.
This means the mesage is delivered identical to what was sent.
Additionaly only _one_ copy is delivered.

2) A valid message is delivered.
And in this portion a timeliness connotation comes in.
The "valid" part infact defines a time limit where the message is
considered useful.

Let me exagerate a little and expand the time axis: if a message that
was valid last month (such as "route add") got delivered today then you
should be able to at least discard it since its validity time limit is
over.
Sure you wasted bandwidth by sending a useless transaction, but its ok
to junk it (sequence numbers protect against this).
TCP is know for its famous head-of-line blocking. Once a
message is queued, my friend pray to the lord to have mercy on the
packet  because it shall be sent - whether you like it or not. Whether
it is "valid" as described or not. For this reason TCP is in fact a
failure.
What was supposed to save you in the case of congestion control just
turned out to be your achilles heel. When you have no congestion, and
have low latencies, this problem doesnt exist. But since the big claim
is that TCP is good because it has congestion control is being thrown
around, I am pointing to you that infact that is a falacy. That may be
fine for a file transfer where the validity bounds are almost unlimited
(go and drink coffee while you wait for ftp to complete)
but not for a transaction that has expired.
To illustrate:
Think of a two phase commit of a route cmd to several FEs within an NE
from a CE(window slashed to two packets, heavy congestion):
A revert (assuming its in a single packet) is queued behind a commit
(assuming its in a single packet) even though the commit is irrelevant
at that point - then there is no need to reconsider sending either one
or maybe even both of them. There is no user supervision to control
this.
A better protocol would be SCTP-PR (only recently, last week or so has
it become a RFC). But you are not talking SCTP-PR, you insist TCP.
An alternative would be something that runs on UDP as what Netlink2 is
doing.

> multicast transport protocol will need to be used.

Even when you use TCP you still need transaction sequence numbers.
So this is moot. And i dont know why/how TCP guarantees ordering in the
network. The Forces protocol dealing at the transaction level requires
ordering, which all protocols i have seen proposed handle at the message
level - TCP does not provide magical panacea to solve this.

> (this results in every FE seeing every other FE's ack - yuck!)

Note the netlink2 draft talks about these issues:
You could make use of filtering even at hardware level. You could
use partial acks to control implosions. You could scale to more sockets
which do multicast etc (the total number of sockets used would be very
few in comparison still), you could choose to add a unicast socket
on the fly.
You make it sound ugly that FEs could see other FEs ACKs or see other
transactions directed at other FEs. In fact having ability to do so when
you so wish is a good thing for things such as almost-free-state
synchronization (you cant with unicast only without incuring more
overhead).

> or a separate TCP connection must be maintained by the FE to

You have that choice with netlink2.

> protocol, where LFBs are dynamically associated at runtime

All protocols proposed so far have LFB identification. I dont understand
what you mean by the "binding" part. I think this is one of those
mysterious claims being made. Can you explain?

> a mechanism to even support transactioning.

You obviously didnt read the netlink2 draft or maybe didnt listen too
closely to the presentations ;->
I am afraid i am gonna have to force you to go and read the netlink2
draft ;->

> available in the operating systems commonly used in CEs and FEs?

Please read my comments on this above.

> how the above can be satisfied when using multicast.

I hope the above is helpful.

I will split the real discussion into another email to avoid the
usenet-loop effect.

cheers,
jamal