Subject: Re: Support for multicast in the ForCES protocol

From: "Putzolu, David" <[email-masked]>
Date: Wed, 19 Nov 2003 17:02:07 -0800

Hi Steve.

Are you saying that transport reliability is 
* Necessary but not sufficient, or
* Unnecessary?

My impression was that having the transport (or the
protocol itself) preserve ordering and avoid corruption
of messages is necessary to ensure predictable behavior
of the FE.  So if using multicast, then either reliable 
multicast is needed as Tal mentioned, or one must 
implement application level support for these features.

Also, in regards to multicast itself - I agree that
application-layer acknowledgements are needed (I think
there is a requirement about this as well).  In the
envisioned use of multicast, would the same multicast
group be used to send the application layer 
acknowledgements?  That would cause every FE to see 
every other FE's acknowledgment - or is the idea to
use multicast in CE->FE and unicast in FE->CE messages?

Thanks,
David (chair hat is off)

> -----Original Message-----
> From: Forwarding and Control Element Separation 
> [mailto:[email-masked]] On Behalf Of Steven Blake
> Sent: Wednesday, November 19, 2003 4:04 PM
> To: [email-masked]
> Subject: Re: [FORCES] Support for multicast in the ForCES protocol
> 
> This is actually orthogonal to multicast transport (reliable or
> otherwise).  The fact that a reliable transport delivered a command
> message from a CE to a FE is insufficient to indicate to the 
> CE that the command can and has been executed successfully by the FE.
So 
> the ForCES protocol will need to support application-layer 
> acknowledgements of some variety.

---

From: Steven Blake <[email-masked]>
Date: Wed, 19 Nov 2003 20:37:23 -0500

> * Necessary but not sufficient, or

Strictly speaking I think it is unnecessary; otherwise it would be
impossible to run the ForCES protocol over bare Ethernet or UDP.

Now, you may ask yourself why you would want to be able to run over bare
Ethernet or UDP.  I don't have much need for the ability to run over
bare Ethernet personally (although I have heard that some AD expressed
this as a requirement; maybe that's a myth).  In the absence of an
IETF-standard reliable multicast transport, being able to run over UDP
is pretty important if you want to be able to support multicast.

> implement application level support for these features.

Right.  In a perfect world we would have an IETF-standard RMT protocol
that we could use for sequencing.  Although I need to engage in some
deep thinking to convince myself that transport-level sequencing is
sufficient to support application-layer transactions; we may not want to
impose the requirement that all application-layer messages are processed
in order of reception.

> use multicast in CE->FE and unicast in FE->CE messages?

Both, actually.  The former (using partial ACKs randomly
multicast back to the originator as described in the Netlink2 draft) is
actually a useful mechanism for reducing ACK load on the originator,
although the partial ACK semantic is not as strong as full ACKs (it is
a useful hint in the interim until a new multicast message with a bumped
sequence number can be sent and NACKed by receivers that missed the
earlier message).  Full ACKs unicast back to the originator are needed
if you need transaction semantics (although you don't necessarily need
to ACK every message in a transaction sequence).  Full ACKs are often
considered to lead to ACK implosion at the originator, but note that the
number of full ACKs is identical to what would be required if all
messages were unicast from the originator.

If full ACKs are used then multicast allows you to reduce the number of
exchanged messages by roughly half (assuming more than a handful of
FEs); the amount of data communicated is probably reduced by much more
than half in the command case.  That may appear as a substantial
optimization for some but not others.  The strongest argument for
multicast support in ForCES will depend on whether a use case can be
found for partial ACKs.  I'm pretty sure that the popular use case
people have mentioned for multicast in ForCES will work with partial
ACKs.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: Alistair Munro <[email-masked]>
Date: Thu, 20 Nov 2003 08:25:42 -0000

Hi All,

Picking up on a couple of issues in this discussion, (and repeating things
I said two years back although the world has moved on meantime):

DP:>>
>> * Necessary but not sufficient, or
SB:>
> Strictly speaking I think it is unnecessary; ....
>
DP:... So if using multicast, then either reliable
>> implement application level support for these features.
...  Although I need to engage in some
SB:> deep thinking to convince myself that transport-level sequencing is
> impose the requirement that all application-layer messages are processed

The NPF has published service level APIs that are basically asynchronous
and reliable (i.e. the callback completions are not lost but may,
potentially be processed/returned out of order) and allow calls to fail or
succeed in a variety of ways. I know this is one step removed from the
level we are discussing here but this model is being used at CE<->FE level
too.

This behaviour will have to be propagated into the protocol to give a
equivalent transactional capability that will convey the semantics of NPF
FE functional APIs to the CE across the intervening stack.

Transport reliability is useful but I think we risk muddling layer
functionality - it is application reliability that is the problem. It might
be solved by a session protocol (the late unlamented OSI standards did it
this way) but it's better to merge it with the application protocol I think.

For this reason, I wonder how much we gain by using multicast. Halving the
number of messages could be just the rare best case. As above, I don't
think reliability of multicast would be anything more than just a useful
behaviour.

Regards,

Alistair

--
Alistair Munro,
Principal Engineer, U4EA Technologies Ltd.
City Point, Temple Gate, Bristol, BS1 6PL, U.K.
E-mail: [email-masked]
Tel: (work) +44-117-373-6765, (home) +44-1275-462707, (mobile)
+44-7974-922-442;
Fax: +44-117-373-6751
Web: http://www.u4eagroup.com

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 20 Nov 2003 06:04:18 -0800

Just changing the topic slightly since this topic is getting thread time.

I would like to poll people who have deployed boxes; either single chasis
with multiple FEs and CEs or within a rack or multi-rack; I believe this
is what 90% of deployments that will happen with Forces.
if you havent done the above but have opinions, just say so but mention
that you havent deployed something forces-like at the level mentioned
above.

Did you deploy utilizing multicast or broadcast transport? If not why?
And what did you use instead that solved the problem for you.

-In the early days for example multicast support was not
widely available in hardware so you had to emulate it. For such cases,
you are forgiven.
- Another reason is maybe you found multicast/bcast to be too complex.
Explain why.

cheers,
jamal

PS: All cauthors of netlink2 have experience implementing a forces-like
protocol which used a variant of mcast/bcast in environments that
got deployed at product level - thats why we feel very strongly about
this issue. Note, we worked then and still work for different companies
now. So we may be biased.

PPS: This is a poll, it would be preferable to stick to the questions
posed so i dont have to change the subject again.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 Nov 2003 09:28:10 -0500

When the folks I worked for build and sold a distributed forwarder (multi
box), we used unicast mechanisms.  This was largely an architectural
choice, since the interconnect technology (ATM) had point-to-multipoint.
We had no difficulty maintaining the necessary update rate.

[Nopt sure if the following is within what you asked for on your poll.  If
no, ignore it.]
Of the earlier routers that I build, some used unicast communication of the
forwarding table, and some used a shared table (which can be thought of as
broadcast.)  The shared table provided inherently reliable multicast, and
was used because the entire device was a shared memory architecture.  The
device with actual communications channels used unicast for forwarding
table, although the backplane had broadcast capability.

Yours,
Joel M. Halpern